BQ QiuData Pipeline Backfill on AirflowI recently worked on backfilling a large amount of historical data (~a few years’ worth) and ran into some Airflow gotchas. Airflow is an…4 min read·Mar 3, 2023----
BQ QiuData Pipelining ConceptsI’ve been recently thinking about data pipeline designs. Here are some key concepts or “gotchas” that are frequently not handled…4 min read·May 8, 2022----
BQ QiuTrying out OpenAI Beta with Poetry Writing AppI signed up for the OpenAI API Beta when GPT-3 was all the rage last year. Having finally gotten out of the waitlist a month back, I…4 min read·Jun 19, 2021----
BQ QiuKubernetes pods terminating with error after job is doneI was debugging a behaviour that was not present before we migrated Spark from a custom-provisioned cluster to a Kubernetes cluster. After…3 min read·Apr 20, 2021----
BQ QiuUsing AWS temporary credentials with Hadoop S3 ConnectorA few days ago I was trying to get a Spark app to access another AWS account by calling the AWS STS AssumeRole API. Let’s say there are…4 min read·Mar 17, 2021----