Hi I am working on a project to display more visualization in my dashboard from the Airflow data. I am struggling on how can I query Airflow database directly to retrieve such data. Is there a way how can I retrieve data from Airflow database? Thanks
I am trying to develop a few data pipelines using Apache Airflow with scheduled Spark jobs.
I have multiple JSON files (10 TB ~) on a S3 bucket, and I need to organize these files by a date element present in every json document.
I am using docker-compose to set up a scalable airflow cluster. I based my approach off of this Dockerfile https://hub.docker.com/r/puckel/docker-airflow/
I would like to create a conditional task in Airflow as described in the schema below. The expected scenario is the following:
