4 Apache Airflow internals you may need missed

I’ve been working with Airflow for greater than three years now and total, I’m fairly assured with it. It’s a robust orchestrator that helps me construct information pipelines shortly and in a scalable style whereas for many issues I’m seeking to implement it comes with batteries included.

Not too long ago, and whereas making ready myself to get a certification for Airflow, I’ve come throughout many various issues I had actually no clue about. And this was primarily my motivation to put in writing this text and share with you a couple of Airflow internals which have completely blown my thoughts!

1. Scheduler solely parses recordsdata containing sure key phrases

The Airflow Scheduler will parse solely recordsdata containing airflow or dag within the code! Sure, you’ve heard this proper! If a file beneath the DAG folder doesn’t comprise at the least one among these two key phrases, it’s going to merely not be parsed by the scheduler.

If you wish to modify this rule such that that is not a requirement for the scheduler, you possibly can merely set DAG_DISCOVERY_SAFE_MODE configuration setting to False. In that case, the scheduler will parse all recordsdata beneath your DAG folder (/dags).

I wouldn’t suggest disabling this test although, since doing so doesn’t actually make any sense. A correct DAG file can have Airflow imports and DAG definition which suggests the necessities for parsing that file are met) however it’s price figuring out that this rule exists.

2. Variables with sure key phrases of their identify have their values hidden

We all know that by default, Airflow will disguise delicate info saved in a Connection (and extra particularly within the password subject), however what about Variables?

Properly, that is certainly potential and the thoughts blowing factor is that Airflow can do that mechanically for you. If a variable accommodates sure key phrases, that may probably point out delicate info, then its worth will mechanically be hidden.

Right here’s a listing of key phrases that can make a Variable qualify for having delicate info retailer as…