The way in which you retrieve variables from Airflow can influence the efficiency of your DAGs

What occurs if a number of information pipelines have to work together with the identical API endpoint? Would you actually should declare this endpoint in each pipeline? In case this endpoint adjustments within the close to future, you’ll have to replace its worth in each single file.

Airflow variables are easy but priceless constructs, used to forestall redundant declarations throughout a number of DAGs. They’re merely objects consisting of a key and a JSON serializable worth, saved in Airflow’s metadata database.

And what in case your code makes use of tokens or different kind of secrets and techniques? Hardcoding them in plain-text doesn’t appear to be a safe strategy. Past decreasing repetition, Airflow variables additionally assist in managing delicate data. With six other ways to outline variables in Airflow, deciding on the suitable methodology is essential for making certain safety and portability.

An usually neglected side is the influence that variable retrieval has on Airflow efficiency. It may doubtlessly pressure the metadata database with requests, each time the Scheduler parses the DAG recordsdata (defaults to thirty seconds).

It’s pretty straightforward to fall into this lure, except you perceive how the Scheduler parses DAGs and the way Variables are retrieved from the database.

Earlier than entering into the dialogue of how Variables are fetched from the metastore and what greatest practices to use with a view to optimise DAGs , it’s necessary to get the fundamentals proper. For now, let’s simply deal with how we will really declare variables in Airflow.

As talked about already, there are a number of other ways to declare variables in Airflow. A few of them develop into safer and moveable than others, so let’s study all and attempt to perceive their execs and cons.

1. Making a variable from the Person Interface

On this first strategy, we’re going to create a variable by the Person Interface. From the highest menu choose Admin → Variables → +