Python API Reference¶

Operators¶

Operators allow for generation of certain types of tasks that become nodes inthe DAG when instantiated. All operators derive fromBaseOperator andinherit many attributes and methods that way.

There are 3 main types of operators:

Operators that performs anaction, or tell another system toperform an action
Transfer operators move data from one system to another
Sensors are a certain type of operator that will keep running until acertain criterion is met. Examples include a specific file landing in HDFS orS3, a partition appearing in Hive, or a specific time of the day. Sensorsare derived fromBaseSensorOperator and run a pokemethod at a specifiedpoke_interval until it returnsTrue.

BaseOperator¶

All operators are derived fromBaseOperator and acquire muchfunctionality through inheritance. Since this is the core of the engine,it’s worth taking the time to understand the parameters ofBaseOperatorto understand the primitive features that can be leveraged in yourDAGs.

BaseSensorOperator¶

All sensors are derived fromBaseSensorOperator. All sensors inheritthetimeout andpoke_interval on top of theBaseOperatorattributes.

Operators packages¶

All operators are in the following packages:

Hooks¶

Hooks are interfaces to external platforms and databases, implementing a commoninterface when possible and acting as building blocks for operators. All hooksare derived fromBaseHook.

Hooks packages¶

All hooks are in the following packages:

Executors¶

Executors are the mechanism by which task instances get run. All executors arederived fromBaseExecutor.

Executors packages¶

All executors are in the following packages:

airflow.executors

Models¶

Models are built on top of the SQLAlchemy ORM Base class, and instances arepersisted in the database.

airflow.models

Exceptions¶

airflow.exceptions

Core and community package¶

Formerly the core code was maintained by the original creators - Airbnb. The codethat was in the contrib package was supported by the community. The projectwas passed to the Apache community and currently the entire code ismaintained by the community, so now the division has no justification,and it is only due to historical reasons. Currently, all new classes areadded only to the contrib package.

Secrets Backends¶

Airflow relies on secrets backends to retrieveConnection objects.All secrets backends derive fromBaseSecretsBackend.

Was this entry helpful?

Movatterモバイル変換