Skip to main content

orbitra.dags.airflow.operators.aci

Run a Python callable inside an Azure Container Instances container group built from a pre-built ACR image.

Functions

container_group_name

Build a valid, unique container group name from a prefix. The prefix is lower-cased, every run of characters outside [a-z0-9-] becomes one hyphen, and a suffix is appended that hashes the task identity (DAG id, task id and the complete, untruncated prefix) together with the run id, try number and map index. So neither parallel runs of the same task nor two tasks whose prefixes only differ past the cut or in characters that normalization erases ever collide. The result fits Azure’s limit of 63 characters. Args:
  • prefix: Human-readable part (name of the task or <dag_id>-<task_id>).
  • dag_id: DAG id, part of the hashed identity even when prefix is a custom name.
  • task_id: Task id, likewise.
  • run_id: Airflow run id.
  • try_number: Task try number.
  • map_index: Map index (-1 for unmapped tasks).
Returns:
  • A name matching [a-z0-9]([-a-z0-9]*[a-z0-9])? of at most 63 characters.

Classes

ContainerTimeoutError

Raised while polling when the container exceeded the task’s timeout.

AciConfig

The resolved ACI settings of a DAGs folder, validated for a run. Attributes:
  • resource_group: Resource group of the container groups.
  • region: Azure region.
  • subnet_id: Delegated subnet the container group joins.
  • registry: ACR login server.
  • identity_id: User-assigned identity resource id.
  • identity_client_id: Client id of that identity, forwarded to the container as ORBITRA_ASSIGNED_IDENTITY_CLIENT_ID.
  • image: Image reference to run.
  • subscription_id: Subscription the resources live in.
  • conn_id: Airflow connection for the hook.
  • log_analytics_workspace_resource_id: Resource id of the Log Analytics workspace receiving the container logs, or None. Its customer id and shared key are read from Azure when the task runs.
  • managed_identity_client_id: Client id of the worker’s managed identity, when the platform settings name one.
  • azure_log_level: Level of the azure logger inside the container, forwarded as ORBITRA_DAGS_AZURE_LOG_LEVEL.
Methods:

from_settings

Validate the ACI settings and build the config. Args:
  • settings: Settings loaded for the DAGs folder.
  • dags_folder: Where they were read from, for messages.
Returns:
  • The validated configuration.
Raises:
  • AirflowException: If a required setting is missing, the subscription cannot be determined, or the Log Analytics setting is not a workspace resource id.

orbitra_environment_variables

Plain environment variables that make the Orbitra SDKs work inside the container. orbitra-commons picks its credential from orbitra_assigned_identity_client_id: with it set, a ManagedIdentityCredential for the container group’s identity; without it, an interactive login that cannot happen in a container. The tenant’s published settings do not carry the client id of this identity, so the worker forwards it from orbitra_dags_aci_identity_client_id. The environment is not forwarded: a task that needs one other than the default sets orbitra_environment itself. orbitra_dags_azure_log_level travels the same way: the runner sets the azure logger to it before the task function runs, which keeps the Azure SDKs’ HTTP dumps out of the container logs. Returns:
  • ORBITRA_ASSIGNED_IDENTITY_CLIENT_ID and ORBITRA_DAGS_AZURE_LOG_LEVEL.

OrbitraContainerInstanceHook

Container Instances hook that pulls from ACR with the container group’s identity and enforces a deadline.
  • create_or_update adds an ImageRegistryCredential(server=<registry>, identity=<identity id>) to the container group when it has none, so the image is pulled with the user-assigned identity (no registry user/password connection). Every group it creates is remembered for cleanup.
  • get_state raises :class:ContainerTimeoutError once the deadline passes; the operator’s polling loop lets it through and the container group is deleted.
  • When the Airflow connection does not exist, or carries no subscriptionId, the client is built from DefaultAzureCredential (managed identity on a worker, az login on a developer machine) and the subscription of the configured resources.
  • get_log_analytics reads the customer id and primary shared key of the Log Analytics workspace through the same client, so the container group’s diagnostics need no key in any setting.
Args:
  • azure_conn_id: Airflow connection id.
  • subscription_id: Subscription used when the connection does not provide one.
  • registry: ACR login server.
  • identity_id: User-assigned identity resource id.
  • managed_identity_client_id: Client id passed to DefaultAzureCredential when the connection is absent.
Methods:

create_or_update

Create the container group, pulling the image with the identity, and remember it for cleanup.

get_conn

Return the management client from the connection, or from the default credential when it is missing.

get_log_analytics

Return the diagnostics of a Log Analytics workspace: its customer id and primary shared key, read from Azure. Both requests go through the management client’s pipeline, so they use the credential of the container group operations (the Airflow connection, or DefaultAzureCredential). The identity needs Microsoft.OperationalInsights/workspaces/read and Microsoft.OperationalInsights/workspaces/sharedKeys/action on the workspace. The key only exists in the container group definition sent to Azure: it is never a setting and never logged. Args:
  • workspace_resource_id: /subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.OperationalInsights/workspaces/<name>.
Returns:
  • The LogAnalytics diagnostics block of the container group.
Raises:
  • AirflowException: If the workspace cannot be read; a 401/403 names the missing RBAC actions.

get_logs

Return the log tail, or nothing while Azure still reports the group as not ready for the operation. The provider’s polling loop asks for logs as soon as the container is Running; for a few seconds Azure answers ContainerGroupDeploymentNotReady (an HttpResponseError the loop would log as an unexpected exception with a traceback). An empty list keeps the loop quiet until the logs exist.

get_state

Return the container group state, unless the deadline has passed.

OrbitraContainerInstancesOperator

Run a function in an Azure Container Instances container group from a pre-built ACR image. The function is sent to the container the same way @orbitra_task sends it to a venv: its source is relocated to the DAG file (breakpoints and tracebacks name the real file) and travels, with the pickled arguments, in the secured ORBITRA_TASK_PAYLOAD environment variable of the container group. The container runs python -m orbitra.dags.airflow.aci.runner, which executes the function and prints the pickled return value on a marker line; after the container terminates the operator reads the logs, decodes the value and returns it (it becomes the task’s XCom). The container group is always deleted afterwards. Only cpu, memory, name, timeout, tags and uv_project_dir are chosen per task. Image, resource group, region, subnet, registry, identity, Log Analytics and connection come from the [tool.orbitra] table of the task’s pyproject.toml, loaded on the worker through orbitra.dags.airflow.settings.get_dags_settings. None of them is a secret: the Log Analytics shared key is read from Azure with the worker’s credential when the task runs. Orbitra inside the container. The image carries the tenant’s settings package (the orbitra.settings entry point), so the Orbitra SDKs know the published settings URL; what they cannot know is which credential to use. The worker always forwards the client id of the container group’s identity (orbitra_dags_aci_identity_client_id) as ORBITRA_ASSIGNED_IDENTITY_CLIENT_ID, which makes orbitra-commons authenticate with that managed identity. Project discovery. uv_project_dir names the directory holding that pyproject.toml (the same directory, with its uv.lock, the image is built from and the debug fallback syncs). It follows the rules of @orbitra_task: absolute, relative to the DAG bundle root, or relative to the DAG file’s directory; omitted, the operator walks up from the DAG file until it finds a pyproject.toml. Inside an OrbitraDAG the DAG’s uv_project_dir is the default for uv and ACI tasks alike. Tags. Every container group carries Azure resource tags, so its cost can be grouped in Azure Cost Management and in the cost exports:
  • orbitra-product: always dags; tells the container groups of this SDK apart from the other Orbitra products in the same subscription or resource group.
  • orbitra-dag-id: the id of the DAG the container ran for.
  • orbitra-task-id: the id of the task, task group prefix included. Every run, retry and mapped instance of a task shares it, so the tag adds up the cost of the task over time.
tags adds up to 5 more for dimensions Orbitra does not know (a team, a cost center). They are Azure resource tags, not Airflow DAG tags: they never show in the Airflow UI. Inside the container there is no Airflow: no Variables, Connections or XCom; tasks use the Orbitra SDKs with the container group’s identity. A Variable.get, Connection.get or XCom pull fails at once with a hint to pass the value as a templated argument instead, whether the image includes the Airflow package or not. Return values travel through the logs, so keep them small and put large data in the lake. The image must run the same Python major.minor as the worker. Args:
  • python_callable: Function to run inside the container. Must be a plain def with its imports inside.
  • cpu: vCPUs of the container (default 1.0).
  • memory: GB of memory (default 2.0).
  • name: Prefix of the container group name (templated). Defaults to <dag_id>-<task_id>. A suffix derived from the task identity and the run is appended and the result is validated (lower case, digits, hyphens, 63 chars).
  • timeout: Maximum container run time (default 30 minutes). Past it the task fails and the group is deleted.
  • tags: Up to 5 Azure resource tags added to the container group next to orbitra-product, orbitra-dag-id and orbitra-task-id (see Tags above). Names and values must be strings Azure accepts: a name has 1 to 512 characters and no backslash or any of < > % & ? /, a value at most 256 characters. Names starting with orbitra- are reserved for Orbitra, and names starting with microsoft, azure or windows for Azure. Not templated.
  • uv_project_dir: Directory with the pyproject.toml holding the ACI settings (templated). None enables automatic discovery as described above.
  • op_args: Positional arguments for the callable.
  • op_kwargs: Keyword arguments for the callable.
  • templates_dict: Templated values made available in the callable’s context.
  • show_return_value_in_logs: Whether to log the return value.
  • **kwargs: Any other BaseOperator argument.
Raises:
  • ValueError: If an argument owned by the settings (image, region, resource_group, ci_conn_id, registry_conn_id, subnet_ids, identity, diagnostics, command, environment_variables, secured_variables, deferrable) is passed, or tags breaks one of the rules above.
Methods:

aci_config

The configuration resolved by execute; available on the worker only.

build_payload

Encode the relocated function, its name and the (templated) arguments for the runner.

configure_container_group

Fill the parent operator’s fields from the settings and the run context. Sets the final container group name, its tags (the task’s own plus orbitra-product, orbitra-dag-id and orbitra-task-id), image, region, resource group, connection, subnet, identity, Log Analytics diagnostics, the runner command and the environment: the AIRFLOW_CTX_* context variables, ORBITRA_CONTAINER_GROUP_NAME, ORBITRA_ASSIGNED_IDENTITY_CLIENT_ID and ORBITRA_DAGS_AZURE_LOG_LEVEL in the clear, plus the payload as the only secured variable.

execute

Resolve the DAGs folder settings, then run the callable in a container group (via execute_callable).

execute_callable

Create the container group, wait for it, decode the result from its logs and delete it.

get_python_source

Return the callable’s source relocated to the DAG file (see orbitra.dags.airflow._source.relocate_source).

hook

Hook bound to the resolved settings (identity-based ACR pull, deadline).

load_config

Load the settings of dags_folder and validate them for ACI (no secret is involved).

on_kill

Delete the container group(s) this task created, if any.

partial

Validate tags before building the partial of a mapped task. partial(...).expand(...) builds a MappedOperator without running __init__, which would otherwise postpone the tags error from DAG parsing to the moment the task is unmapped on the worker. Args:
  • **kwargs: Arguments of BaseOperator.partial.
Returns:
  • The OperatorPartial to call expand on.
Raises:
  • ValueError: If tags breaks one of the rules of the class.

resolve_dags_folder

Find the directory whose pyproject.toml holds the ACI settings. uv_project_dir is resolved like for uv tasks (absolute, relative to the bundle root, relative to the DAG file’s directory); without it the walk goes up from the DAG file’s directory until a pyproject.toml is found, stopping at the bundle root when the DAG lives inside one (otherwise at the filesystem root). DAGs deployed as a zip are searched inside the extracted archive first. Args:
  • bundle_path: The DAG bundle root, or None outside a bundle.
Returns:
  • The directory containing the pyproject.toml.
Raises:
  • UvProjectNotFoundError: If no pyproject.toml exists where the rules look for it.