orbitra.dags.airflow.operators.aci
Run a Python callable inside an Azure Container Instances container group built from a pre-built ACR image.
Functions
container_group_name
[a-z0-9-] becomes one hyphen, and a
suffix is appended that hashes the task identity (DAG id, task id and the complete, untruncated prefix)
together with the run id, try number and map index. So neither parallel runs of the same task nor two tasks
whose prefixes only differ past the cut or in characters that normalization erases ever collide. The result
fits Azure’s limit of 63 characters.
Args:
prefix: Human-readable part (nameof the task or<dag_id>-<task_id>).dag_id: DAG id, part of the hashed identity even whenprefixis a custom name.task_id: Task id, likewise.run_id: Airflow run id.try_number: Task try number.map_index: Map index (-1for unmapped tasks).
- A name matching
[a-z0-9]([-a-z0-9]*[a-z0-9])?of at most 63 characters.
Classes
ContainerTimeoutError
Raised while polling when the container exceeded the task’s timeout.
AciConfig
The resolved ACI settings of a DAGs folder, validated for a run.
Attributes:
resource_group: Resource group of the container groups.region: Azure region.subnet_id: Delegated subnet the container group joins.registry: ACR login server.identity_id: User-assigned identity resource id.identity_client_id: Client id of that identity, forwarded to the container asORBITRA_ASSIGNED_IDENTITY_CLIENT_ID.image: Image reference to run.subscription_id: Subscription the resources live in.conn_id: Airflow connection for the hook.log_analytics_workspace_resource_id: Resource id of the Log Analytics workspace receiving the container logs, orNone. Its customer id and shared key are read from Azure when the task runs.managed_identity_client_id: Client id of the worker’s managed identity, when the platform settings name one.azure_log_level: Level of theazurelogger inside the container, forwarded asORBITRA_DAGS_AZURE_LOG_LEVEL.
from_settings
settings: Settings loaded for the DAGs folder.dags_folder: Where they were read from, for messages.
- The validated configuration.
AirflowException: If a required setting is missing, the subscription cannot be determined, or the Log Analytics setting is not a workspace resource id.
orbitra_environment_variables
orbitra-commons picks its credential from orbitra_assigned_identity_client_id: with it set, a
ManagedIdentityCredential for the container group’s identity; without it, an interactive login that
cannot happen in a container. The tenant’s published settings do not carry the client id of this
identity, so the worker forwards it from orbitra_dags_aci_identity_client_id. The environment is
not forwarded: a task that needs one other than the default sets orbitra_environment itself.
orbitra_dags_azure_log_level travels the same way: the runner sets the azure logger to it before
the task function runs, which keeps the Azure SDKs’ HTTP dumps out of the container logs.
Returns:
ORBITRA_ASSIGNED_IDENTITY_CLIENT_IDandORBITRA_DAGS_AZURE_LOG_LEVEL.
OrbitraContainerInstanceHook
Container Instances hook that pulls from ACR with the container group’s identity and enforces a deadline.
create_or_updateadds anImageRegistryCredential(server=<registry>, identity=<identity id>)to the container group when it has none, so the image is pulled with the user-assigned identity (no registry user/password connection). Every group it creates is remembered for cleanup.get_stateraises :class:ContainerTimeoutErroronce the deadline passes; the operator’s polling loop lets it through and the container group is deleted.- When the Airflow connection does not exist, or carries no
subscriptionId, the client is built fromDefaultAzureCredential(managed identity on a worker,az loginon a developer machine) and the subscription of the configured resources. get_log_analyticsreads the customer id and primary shared key of the Log Analytics workspace through the same client, so the container group’s diagnostics need no key in any setting.
azure_conn_id: Airflow connection id.subscription_id: Subscription used when the connection does not provide one.registry: ACR login server.identity_id: User-assigned identity resource id.managed_identity_client_id: Client id passed toDefaultAzureCredentialwhen the connection is absent.
create_or_update
get_conn
get_log_analytics
DefaultAzureCredential). The identity needs
Microsoft.OperationalInsights/workspaces/read and Microsoft.OperationalInsights/workspaces/sharedKeys/action
on the workspace. The key only exists in the container group definition sent to Azure: it is never a
setting and never logged.
Args:
workspace_resource_id:/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.OperationalInsights/workspaces/<name>.
- The
LogAnalyticsdiagnostics block of the container group.
AirflowException: If the workspace cannot be read; a 401/403 names the missing RBAC actions.
get_logs
Running; for a few seconds
Azure answers ContainerGroupDeploymentNotReady (an HttpResponseError the loop would log as an
unexpected exception with a traceback). An empty list keeps the loop quiet until the logs exist.
get_state
OrbitraContainerInstancesOperator
Run a function in an Azure Container Instances container group from a pre-built ACR image.
The function is sent to the container the same way @orbitra_task sends it to a venv: its source is
relocated to the DAG file (breakpoints and tracebacks name the real file) and travels, with the pickled
arguments, in the secured ORBITRA_TASK_PAYLOAD environment variable of the container group. The
container runs python -m orbitra.dags.airflow.aci.runner, which executes the function and prints the pickled
return value on a marker line; after the container terminates the operator reads the logs, decodes the
value and returns it (it becomes the task’s XCom). The container group is always deleted afterwards.
Only cpu, memory, name, timeout, tags and uv_project_dir are chosen per task. Image, resource
group, region, subnet, registry, identity, Log Analytics and connection come from the [tool.orbitra]
table of the task’s pyproject.toml, loaded on the worker through
orbitra.dags.airflow.settings.get_dags_settings. None of them is a secret: the Log Analytics shared key
is read from Azure with the worker’s credential when the task runs.
Orbitra inside the container. The image carries the tenant’s settings package (the orbitra.settings
entry point), so the Orbitra SDKs know the published settings URL; what they cannot know is which
credential to use. The worker always forwards the client id of the container group’s identity
(orbitra_dags_aci_identity_client_id) as ORBITRA_ASSIGNED_IDENTITY_CLIENT_ID, which makes
orbitra-commons authenticate with that managed identity.
Project discovery. uv_project_dir names the directory holding that pyproject.toml (the same
directory, with its uv.lock, the image is built from and the debug fallback syncs). It follows the
rules of @orbitra_task: absolute, relative to the DAG bundle root, or relative to the DAG file’s
directory; omitted, the operator walks up from the DAG file until it finds a pyproject.toml. Inside an
OrbitraDAG the DAG’s uv_project_dir is the default for uv and ACI tasks alike.
Tags. Every container group carries Azure resource tags, so its cost can be grouped in Azure Cost
Management and in the cost exports:
orbitra-product: alwaysdags; tells the container groups of this SDK apart from the other Orbitra products in the same subscription or resource group.orbitra-dag-id: the id of the DAG the container ran for.orbitra-task-id: the id of the task, task group prefix included. Every run, retry and mapped instance of a task shares it, so the tag adds up the cost of the task over time.
tags adds up to 5 more for dimensions Orbitra does not know (a team, a cost center). They are Azure
resource tags, not Airflow DAG tags: they never show in the Airflow UI.
Inside the container there is no Airflow: no Variables, Connections or XCom; tasks use the Orbitra SDKs
with the container group’s identity. A Variable.get, Connection.get or XCom pull fails at once with a hint
to pass the value as a templated argument instead, whether the image includes the Airflow package or not. Return values travel through the logs, so keep them small and put
large data in the lake. The image must run the same Python major.minor as the worker.
Args:
python_callable: Function to run inside the container. Must be a plaindefwith its imports inside.cpu: vCPUs of the container (default1.0).memory: GB of memory (default2.0).name: Prefix of the container group name (templated). Defaults to<dag_id>-<task_id>. A suffix derived from the task identity and the run is appended and the result is validated (lower case, digits, hyphens, 63 chars).timeout: Maximum container run time (default 30 minutes). Past it the task fails and the group is deleted.tags: Up to 5 Azure resource tags added to the container group next toorbitra-product,orbitra-dag-idandorbitra-task-id(see Tags above). Names and values must be strings Azure accepts: a name has 1 to 512 characters and no backslash or any of< > % & ? /, a value at most 256 characters. Names starting withorbitra-are reserved for Orbitra, and names starting withmicrosoft,azureorwindowsfor Azure. Not templated.uv_project_dir: Directory with thepyproject.tomlholding the ACI settings (templated).Noneenables automatic discovery as described above.op_args: Positional arguments for the callable.op_kwargs: Keyword arguments for the callable.templates_dict: Templated values made available in the callable’s context.show_return_value_in_logs: Whether to log the return value.**kwargs: Any otherBaseOperatorargument.
ValueError: If an argument owned by the settings (image,region,resource_group,ci_conn_id,registry_conn_id,subnet_ids,identity,diagnostics,command,environment_variables,secured_variables,deferrable) is passed, ortagsbreaks one of the rules above.
aci_config
execute; available on the worker only.
build_payload
configure_container_group
orbitra-product,
orbitra-dag-id and orbitra-task-id), image, region, resource group, connection, subnet, identity,
Log Analytics diagnostics, the runner command and the environment: the AIRFLOW_CTX_* context
variables, ORBITRA_CONTAINER_GROUP_NAME, ORBITRA_ASSIGNED_IDENTITY_CLIENT_ID and ORBITRA_DAGS_AZURE_LOG_LEVEL in the clear, plus
the payload as the only secured variable.
execute
execute_callable).
execute_callable
get_python_source
orbitra.dags.airflow._source.relocate_source).
hook
load_config
dags_folder and validate them for ACI (no secret is involved).
on_kill
partial
tags before building the partial of a mapped task.
partial(...).expand(...) builds a MappedOperator without running __init__, which would
otherwise postpone the tags error from DAG parsing to the moment the task is unmapped on the worker.
Args:
**kwargs: Arguments ofBaseOperator.partial.
- The
OperatorPartialto callexpandon.
ValueError: Iftagsbreaks one of the rules of the class.
resolve_dags_folder
pyproject.toml holds the ACI settings.
uv_project_dir is resolved like for uv tasks (absolute, relative to the bundle root, relative to the
DAG file’s directory); without it the walk goes up from the DAG file’s directory until a
pyproject.toml is found, stopping at the bundle root when the DAG lives inside one (otherwise at
the filesystem root). DAGs deployed as a zip are searched inside the extracted archive first.
Args:
bundle_path: The DAG bundle root, orNoneoutside a bundle.
- The directory containing the
pyproject.toml.
UvProjectNotFoundError: If nopyproject.tomlexists where the rules look for it.