Skip to content

air: find the AI Runtime task anywhere in a multi-task job in air list and air get - #7632

Open
dimk0 wants to merge 2 commits into
mainfrom
cnxt-2622-air-multitask
Open

air: find the AI Runtime task anywhere in a multi-task job in air list and air get#7632
dimk0 wants to merge 2 commits into
mainfrom
cnxt-2622-air-multitask

Conversation

@dimk0

@dimk0 dimk0 commented Nov 14, 2026 •
edited by atlassian Bot
Loading

Copy link
Copy Markdown
Member

Changes

Multi-task AIR jobs are supported, and the official DAB example (universe ai-compute/dabs/examples/03_multitask_dag) runs a notebook prep task before its ai_runtime_task. air list and air get read tasks[0], so that run was omitted from the default air list, listed with null experiment/accelerators by air list --all-status, and rendered with N/A Experiment and Accelerators in air get.

Selection rule (also posted on CNXT-2622):

  • Discovery. Walk the run's tasks in task order and pick the first AI Runtime task: an ai_runtime_task, a legacy gen_ai_compute_task with a training script, or a for_each_task whose inner task is one. A run is an AIR run iff it has at least one such task. Non-AIR tasks (the prep notebook) no longer feed status, timing, experiment, accelerators or the MLflow lookup: the run is narrowed to the selected task's task runs (all attempts) before the display helpers read it.
  • Several AIR tasks. air list shows one row per AIR task. Rows share run_id; JSON gains task_key. Each row's fields and MLflow link come from its own task runs, and the experiment/accelerator filters apply per task. The index-path row cache stores the whole set per run (air-list-runs-v3). cancel --all cancels a multi-row run once.
  • air get. New --task-key flag; default is the first AIR task. A key that is not one of the run's AIR tasks is an INVALID_ARGS error listing them. JSON gains task_key; the text view adds a Task line only when the run has several AIR tasks. The air list detail pane opens the selected row's task.
  • foreach. Unchanged: a for_each_task wrapping an AIR task is that AIR task (sweep view / is_sweep). A job mixing a direct AIR task and a foreach AIR task now yields one row (or one selectable task) each, and air list and air get agree on the default (today list inspected tasks[0] while get picked any foreach).

Out of scope: air logs keeps its last-task behavior.

Why

Fixes CNXT-2622. Follows the single-task fix in #7587.

Tests

Unit tests for the five requested shapes (AIR task first, AIR task second, two AIR tasks, foreach AIR task, no AIR task) plus retries sharing a key, run narrowing, per-task list rows and filters, air get --task-key selection and error, the detail pane, the legacy gen_ai_compute_task path and the per-task row cache. New acceptance test acceptance/air/get-multitask; the air list fixture gains the notebook-first DAB shape.

Verified against a real run of 03_multitask_dag deployed to e2-dogfood (run 47834518189346) with a build of this branch and a build of main:

main this branch
air list --all-status -o json experiment_name: null, accelerator_type: null air-byot-multitask-dag, GPU_1xA10 ×1, task_key: train
air get Experiment N/A, Accelerators N/A air-byot-multitask-dag, 1x A10
air get --task-key prep n/a run … has no AI Runtime task "prep"; its AI Runtime tasks are: train

go test ./cmd/air, go test ./acceptance -run TestAccept/air and golangci-lint run ./cmd/air/... pass.

This pull request and its description were written by Isaac.

@dimk0 Isaac
air: find the AI Runtime task anywhere in a multi-task job in `air li…
6f1cb7c 
…st` and `air get`

Multi-task AIR jobs, such as the official DAB example that runs a notebook
prep task before its ai_runtime_task, were omitted from `air list` and
showed N/A experiment and accelerators in `air get`, because both read
tasks[0].

- Discover the first AIR task in task order: an ai_runtime_task, a legacy
  gen_ai_compute_task with a training script, or a foreach wrapping one.
  Non-AIR tasks no longer feed status, timing or the MLflow lookup.
- A job with several AIR tasks lists one row per task. Rows share the
  run_id; JSON output gains task_key. Each row reads its own task runs.
- `air get` gains --task-key (default: the first AIR task), its JSON gains
  task_key, and the text view adds a Task line when the run has several.
- The list detail pane opens the selected row's task; `cancel --all`
  cancels a multi-row run once.
- The index-path row cache stores one row per AIR task (air-list-runs-v3).

Verified against a real run of universe ai-compute/dabs/examples/03_multitask_dag
on e2-dogfood (run 47834518189346).

CNXT-2622

Co-authored-by: Isaac <[email protected]>
@dimk0
dimk0 requested review from a team as code owners November 14, 2026 04:37
@dimk0
dimk0 requested a review from rclarey November 14, 2026 04:37
@github-actions github-actions Bot added the AIR Databricks AI Runtime CLI label Nov 14, 2026
@github-actions

github-actions Bot commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

Commit: 52dfc193e | Workflow run

Regression test report

Selected 7 changed tests (limit=20, 0 not selected)

7 tests: regression=7 unreleased=0 coverage=0 failing=0
test branch main (2699068) latest (v1.26.0)
TestAccept/air/control-chars ✅ ❌ ❌
TestAccept/air/get ✅ ❌ ❌
TestAccept/air/get-ai-runtime ✅ ❌ ❌
TestAccept/air/get-multitask ✅ ❌ ❌
TestAccept/air/get-pending ✅ ❌ ❌
TestAccept/air/help ✅ ❌ ❌
TestAccept/air/list ✅ ❌ ❌
TestAccept/air/control-chars: branch ✅ | main (2699068) ❌ | latest (v1.26.0) ❌

main (2699068):

acceptance_test.go:1374: Diff:
        --- air/control-chars/output.txt
        +++ /tmp/TestAcceptaircontrol-chars2088619126/002/output.txt
        @@ -38,7 +38,6 @@
             "runs": [
               {
                 "run_id": "123",
        -        "task_key": "train",
                 "run_name": "qwen\u001b[2J-train",
                 "user": "user\u001b[[email protected]",
                 "status": "SUCCESS",

latest (v1.26.0):

acceptance_test.go:1374: Diff:
        --- air/control-chars/output.txt
        +++ /tmp/TestAcceptaircontrol-chars1902308178/002/output.txt
        @@ -11,9 +11,10 @@
         │  Retries       0                                               │
         │  Max Retries   0                                               │
         │  Duration      12s                                             │
        -│  Experiment    my-exp                                          │
        -│  MLflow Run    my-run                                          │
        -│  User          [email protected]                                │
        +│  Experiment    my�]0;pwned�-exp                                          │
        +│  MLflow Run    �]0;pwned�my�[2J-run
        +                                          │
        +│  User          user�[[email protected]                                │
         │  Accelerators  1x A10                                          │
         │  Environment   N/A                                             │
         │                                                                │
        @@ -26,7 +27,8 @@
         
         >>> [CLI] air list --all-users
            Run ID  Experiment  Status     Started              Duration  Progress  MLflow                  User              Accelerators
        -   123     my-exp      ● SUCCESS  [TIMESTAMP]       12s         -  my-run                  [email protected]  1x A10
        +   123     my�]0;pwned�-exp      ● SUCCESS  [TIMESTAMP]       12s         -  �]0;pwned�my�[2J-run
        +                  user�[[email protected]  1x A10
         
         === list (json): strings are passed through as the server sent them
         
        @@ -38,17 +40,11 @@
             "runs": [
               {
                 "run_id": "123",
        -        "task_key": "train",
                 "run_name": "qwen\u001b[2J-train",
                 "user": "user\u001b[[email protected]",
                 "status": "SUCCESS",
                 "started_at": "[TIMESTAMP]",
        -        "is_sweep": false,
        -        "experiment_name": "my\u001b]0;pwned\u0007-exp",
        -        "duration_seconds": 12,
        -        "accelerator_type": "GPU_1xA10",
        -        "num_accelerators": 1,
        -        "dashboard_url": "[DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]"
        +        "is_sweep": false
               }
             ]
           }
TestAccept/air/get: branch ✅ | main (2699068) ❌ | latest (v1.26.0) ❌

main (2699068):

acceptance_test.go:1374: Diff:
        --- air/get/output.txt
        +++ /tmp/TestAcceptairget1958698912/002/output.txt
        @@ -43,7 +43,6 @@
           "ts": "[TIMESTAMP]",
           "data": {
             "run_id": "123",
        -    "task_key": "train",
             "status": "SUCCESS",
             "started_at": "[TIMESTAMP]",
             "duration_seconds": 12,

latest (v1.26.0):

acceptance_test.go:1374: Diff:
        --- air/get/output.txt
        +++ /tmp/TestAcceptairget81737483/002/output.txt
        @@ -43,14 +43,13 @@
           "ts": "[TIMESTAMP]",
           "data": {
             "run_id": "123",
        -    "task_key": "train",
             "status": "SUCCESS",
             "started_at": "[TIMESTAMP]",
             "duration_seconds": 12,
             "attempt_number": 0,
             "experiment_name": "my-exp",
             "dashboard_url": "[DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]",
        -    "mlflow_url": "[DATABRICKS_URL]/ml/experiments/exp1/runs/run1/logs",
        +    "mlflow_url": "[DATABRICKS_URL]/ml/experiments/exp1/runs/run1/artifacts/logs/node_0",
             "est_remaining_seconds": null,
             "est_percent_complete": null
           }
TestAccept/air/get-ai-runtime: branch ✅ | main (2699068) ❌ | latest (v1.26.0) ❌

main (2699068):

acceptance_test.go:1374: Diff:
        --- air/get-ai-runtime/output.txt
        +++ /tmp/TestAcceptairget-ai-runtime323877121/002/output.txt
        @@ -43,7 +43,6 @@
           "ts": "[TIMESTAMP]",
           "data": {
             "run_id": "123",
        -    "task_key": "train",
             "status": "SUCCESS",
             "started_at": "[TIMESTAMP]",
             "duration_seconds": 12,
        
    acceptance_test.go:1152: LOG.import: 
    acceptance_test.go:1152: LOG.mkdirs:

latest (v1.26.0):

acceptance_test.go:1374: Diff:
        --- air/get-ai-runtime/output.txt
        +++ /tmp/TestAcceptairget-ai-runtime3271300407/002/output.txt
        @@ -43,14 +43,13 @@
           "ts": "[TIMESTAMP]",
           "data": {
             "run_id": "123",
        -    "task_key": "train",
             "status": "SUCCESS",
             "started_at": "[TIMESTAMP]",
             "duration_seconds": 12,
             "attempt_number": 0,
             "experiment_name": "my-exp",
             "dashboard_url": "[DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]",
        -    "mlflow_url": "[DATABRICKS_URL]/ml/experiments/exp1/runs/run1/logs",
        +    "mlflow_url": "[DATABRICKS_URL]/ml/experiments/exp1/runs/run1/artifacts/logs/node_0",
             "est_remaining_seconds": null,
             "est_percent_complete": null
           }
        
    acceptance_test.go:1152: LOG.import: 
    acceptance_test.go:1152: LOG.mkdirs:
TestAccept/air/get-multitask: branch ✅ | main (2699068) ❌ | latest (v1.26.0) ❌

main (2699068):

acceptance_test.go:1374: Diff:
        --- air/get-multitask/output.txt
        +++ /tmp/TestAcceptairget-multitask2870426324/002/output.txt
        @@ -6,16 +6,15 @@
         ╭─ Metadata ─────────────────────────────────────────────────────╮
         │                                                                │
         │  Run ID        123                                             │
        -│  Task          train                                           │
         │  Status        ● SUCCESS                                       │
         │  Submitted     2023-11-14 22:13 UTC                            │
         │  Retries       0                                               │
        -│  Max Retries   0                                               │
        -│  Duration      10s                                             │
        -│  Experiment    air-byot-multitask-dag                          │
        +│  Max Retries   2                                               │
        +│  Duration      8s                                              │
        +│  Experiment    N/A                                             │
         │  MLflow Run    matmul-smoke                                    │
         │  User          [email protected]                                │
        -│  Accelerators  1x A10                                          │
        +│  Accelerators  N/A                                             │
         │  Environment   5                                               │
         │                                                                │
         ╰────────────────────────────────────────────────────────────────╯
        @@ -26,88 +20,19 @@
         === get --task-key eval (text)
         
         >>> [CLI] air get 123 --task-key eval
        +Error: unknown flag: --task-key
         
        -╭─ Metadata ─────────────────────────────────────────────────────╮
        -│                                                                │
        -│  Run ID        123                                             │
        -│  Task          eval                                            │
        -│  Status        ● SUCCESS                                       │
        -│  Submitted     2023-11-14 22:13 UTC                            │
        -│  Retries       0                                               │
        -│  Max Retries   2                                               │
        -│  Duration      8s                                              │
        -│  Experiment    eval-exp                                        │
        -│  MLflow Run    matmul-smoke                                    │
        -│  User          [email protected]                                │
        -│  Accelerators  8x H100                                         │
        -│  Environment   5                                               │
        -│                                                                │
        -╰────────────────────────────────────────────────────────────────╯
        +Usage:
        +  databricks air get JOB_RUN_ID [flags]
         
        -Run URL:    [DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]
        -MLflow URL: [DATABRICKS_URL]/ml/experiments/exp1/runs/run1
        +Flags:
        +  -h, --help   help for get
         
        -=== get (json)
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
         
        ->>> [CLI] air get 123 -o json
        -{
        -  "v": 1,
        -  "ts": "[TIMESTAMP]",
        -  "data": {
        -    "run_id": "123",
        -    "task_key": "train",
        -    "status": "SUCCESS",
        -    "started_at": "[TIMESTAMP]",
        -    "duration_seconds": 10,
        -    "attempt_number": 0,
        -    "experiment_name": "air-byot-multitask-dag",
        -    "dashboard_url": "[DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]",
        -    "mlflow_url": "[DATABRICKS_URL]/ml/experiments/exp1/runs/run1/logs",
        -    "est_remaining_seconds": null,
        -    "est_percent_complete": null
        -  }
        -}
         
        -=== get --task-key eval (json)
        -
        ->>> [CLI] air get 123 --task-key eval -o json
        -{
        -  "v": 1,
        -  "ts": "[TIMESTAMP]",
        -  "data": {
        -    "run_id": "123",
        -    "task_key": "eval",
        -    "status": "SUCCESS",
        -    "started_at": "[TIMESTAMP]",
        -    "duration_seconds": 8,
        -    "attempt_number": 0,
        -    "experiment_name": "eval-exp",
        -    "dashboard_url": "[DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]",
        -    "mlflow_url": "[DATABRICKS_URL]/ml/experiments/exp1/runs/run1/logs",
        -    "est_remaining_seconds": null,
        -    "est_percent_complete": null
        -  }
        -}
        -
        -=== get --task-key prep: not an AI Runtime task
        -
        ->>> [CLI] air get 123 --task-key prep
        -Error: run 123 has no AI Runtime task "prep"; its AI Runtime tasks are: train, eval
        -
         Exit code: 1
        -
        -=== get --task-key prep (json)
        -
        ->>> [CLI] air get 123 --task-key prep -o json
        -{
        -  "v": 1,
        -  "ts": "[TIMESTAMP]",
        -  "error": {
        -    "code": "INVALID_ARGS",
        -    "kind": "PERMANENT",
        -    "message": "run 123 has no AI Runtime task \"prep\"; its AI Runtime tasks are: train, eval",
        -    "retryable": false
        -  }
        -}
        -
        -Exit code: 1

latest (v1.26.0): same output as main (2699068)

TestAccept/air/get-pending: branch ✅ | main (2699068) ❌ | latest (v1.26.0) ❌

main (2699068):

acceptance_test.go:1374: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1374
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1115
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:660
        	Error:      	Not equal: 
        	            	expected: "\n>>> [CLI] air get 123 -o json\n{\n  \"v\": 1,\n  \"ts\": \"[TIMESTAMP]\",\n  \"data\": {\n    \"run_id\": \"123\",\n    \"task_key\": \"train\",\n    \"status\": \"PENDING\",\n    \"started_at\": null,\n    \"duration_seconds\": null,\n    \"attempt_number\": 0,\n    \"experiment_name\": \"my-exp\",\n    \"dashboard_url\": \"[DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]\",\n    \"mlflow_url\": null,\n    \"status_message\": \"Waiting for accelerator compute capacity to become available...\",\n    \"est_remaining_seconds\": null,\n    \"est_percent_complete\": null\n  }\n}\n"
        	            	actual  : "\n>>> [CLI] air get 123 -o json\n{\n  \"v\": 1,\n  \"ts\": \"[TIMESTAMP]\",\n  \"data\": {\n    \"run_id\": \"123\",\n    \"status\": \"PENDING\",\n    \"started_at\": null,\n    \"duration_seconds\": null,\n    \"attempt_number\": 0,\n    \"experiment_name\": \"my-exp\",\n    \"dashboard_url\": \"[DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]\",\n    \"mlflow_url\": null,\n    \"status_message\": \"Waiting for accelerator compute capacity to become available...\",\n    \"est_remaining_seconds\": null,\n    \"est_percent_complete\": null\n  }\n}\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -7,3 +7,2 @@
        	            	     "run_id": "123",
        	            	-    "task_key": "train",
        	            	     "status": "PENDING",
        	Test:       	TestAccept/air/get-pending
        	Messages:   	air/get-pending/output.txt vs /tmp/TestAcceptairget-pending2361209803/002/output.txt

latest (v1.26.0):

acceptance_test.go:1374: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1374
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1115
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:660
        	Error:      	Not equal: 
        	            	expected: "\n>>> [CLI] air get 123 -o json\n{\n  \"v\": 1,\n  \"ts\": \"[TIMESTAMP]\",\n  \"data\": {\n    \"run_id\": \"123\",\n    \"task_key\": \"train\",\n    \"status\": \"PENDING\",\n    \"started_at\": null,\n    \"duration_seconds\": null,\n    \"attempt_number\": 0,\n    \"experiment_name\": \"my-exp\",\n    \"dashboard_url\": \"[DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]\",\n    \"mlflow_url\": null,\n    \"status_message\": \"Waiting for accelerator compute capacity to become available...\",\n    \"est_remaining_seconds\": null,\n    \"est_percent_complete\": null\n  }\n}\n"
        	            	actual  : "\n>>> [CLI] air get 123 -o json\n{\n  \"v\": 1,\n  \"ts\": \"[TIMESTAMP]\",\n  \"data\": {\n    \"run_id\": \"123\",\n    \"status\": \"RUNNING\",\n    \"started_at\": null,\n    \"duration_seconds\": null,\n    \"attempt_number\": 0,\n    \"experiment_name\": \"my-exp\",\n    \"dashboard_url\": \"[DATABRICKS_URL]/jobs/runs/123?o=[WORKSPACE_ID]\",\n    \"mlflow_url\": null,\n    \"est_remaining_seconds\": null,\n    \"est_percent_complete\": null\n  }\n}\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -7,4 +7,3 @@
        	            	     "run_id": "123",
        	            	-    "task_key": "train",
        	            	-    "status": "PENDING",
        	            	+    "status": "RUNNING",
        	            	     "started_at": null,
        	            	@@ -15,3 +14,2 @@
        	            	     "mlflow_url": null,
        	            	-    "status_message": "Waiting for accelerator compute capacity to become available...",
        	            	     "est_remaining_seconds": null,
        	Test:       	TestAccept/air/get-pending
        	Messages:   	air/get-pending/output.txt vs /tmp/TestAcceptairget-pending3708618622/002/output.txt
TestAccept/air/help: branch ✅ | main (2699068) ❌ | latest (v1.26.0) ❌

main (2699068):

acceptance_test.go:1374: Diff:
        --- air/help/output.txt
        +++ /tmp/TestAcceptairhelp1999594784/002/output.txt
        @@ -92,17 +92,11 @@
         
         JOB_RUN_ID is the job run ID returned by "databricks air run".
         
        -A job may run other tasks around its AI Runtime task (for example a notebook
        -that prepares data first); the view shows the first AI Runtime task in task
        -order. When a job has several AI Runtime tasks, pass --task-key to show another
        -one.
        -
         Usage:
           databricks air get JOB_RUN_ID [flags]
         
         Flags:
        -  -h, --help              help for get
        -      --task-key string   Show this AI Runtime task of the run (default: the first in task order)
        +  -h, --help   help for get
         
         Global Flags:
               --debug            enable debug logging
        
FAIL

latest (v1.26.0):

acceptance_test.go:1374: Diff:
        --- air/help/output.txt
        +++ /tmp/TestAcceptairhelp4135067714/002/output.txt
        @@ -2,31 +2,23 @@
         === help
         
         >>> [CLI] air --help
        -Run GPU workloads on Databricks serverless compute.
        +Run and manage AI Runtime training workloads on Databricks serverless GPU compute.
         
        -Describe a workload in a YAML file and submit it with "databricks air run".
        -Follow a run with "databricks air logs" and "databricks air get", or see your
        -active runs with "databricks air list".
        +These commands are under active development.
         
         Usage:
           databricks air [command]
         
        -Run workloads:
        -  run             Submit a workload from a YAML config
        -
        -Manage runs:
        +Available Commands:
           cancel          Cancel one or more runs
        -  get             Show a run's status, configuration, and timing
        -  list            List your active runs
        -  logs            Stream or fetch a run's logs
        +  convert-to-dabs Convert an AIR run YAML into a Databricks Asset Bundle
        +  get             Show status, configuration, and timing details for a specified run
        +  images          Manage container images for AI Runtime
        +  list            List active runs for the current user
        +  logs            Stream or fetch logs for a run
        +  pools           View provisioned GPU pools
        +  run             Submit a training workload from a YAML config
         
        -Package and deploy:
        -  convert-to-dabs Convert a workload YAML into a Databricks Asset Bundle
        -  images          Push container images for custom environments
        -
        -Provisioned capacity:
        -  pools           View provisioned GPU pools, if any (not required to run)
        -
         Flags:
           -h, --help   help for air
         
        @@ -38,12 +23,6 @@
         
         Use "databricks air [command] --help" for more information about a command.
         
        -Learn more:
        -  YAML fields and example  databricks air run -h config
        -  Quickstart (AWS)         https://docs.databricks.com/aws/en/machine-learning/ai-runtime/quickstart
        -  Quickstart (Azure)       https://learn.microsoft.com/azure/databricks/machine-learning/ai-runtime/quickstart
        -  Coding agent skill       databricks aitools install --skills-only --skills databricks-ai-runtime
        -
         === list help
         
         >>> [CLI] air list --help
        @@ -52,9 +31,7 @@
         Filters:
           Use --filter KEY=VALUE per condition. A run must match every condition.
         
        -  accelerator_type  Accelerator family or full YAML type. Examples:
        -                    A10, H100, GPU_1xA10, GPU_8xH100.
        -                    Workspace types: databricks air run -h config.compute.accelerator_type
        +  accelerator_type  Accelerator type, such as A10 or H100.
           experiment        Case-insensitive glob matched against the experiment name.
           num_accelerators  Total GPU count.
           user              Run creator identifier.
        @@ -65,8 +42,6 @@
         Examples:
           databricks air list --filter accelerator_type=H100
         
        -  databricks air list --filter accelerator_type=GPU_8xH100
        -
           databricks air list --all-status \
             --filter 'experiment=qwen*' \
             --filter num_accelerators=8 \
        @@ -88,21 +63,15 @@
         === get help
         
         >>> [CLI] air get --help
        -Show a run's status, configuration, and timing.
        +Show status, configuration, and timing details for a specified run.
         
        -JOB_RUN_ID is the job run ID returned by "databricks air run".
        +JOB_RUN_ID is the job run ID returned by "air run".
         
        -A job may run other tasks around its AI Runtime task (for example a notebook
        -that prepares data first); the view shows the first AI Runtime task in task
        -order. When a job has several AI Runtime tasks, pass --task-key to show another
        -one.
        -
         Usage:
           databricks air get JOB_RUN_ID [flags]
         
         Flags:
        -  -h, --help              help for get
        -      --task-key string   Show this AI Runtime task of the run (default: the first in task order)
        +  -h, --help   help for get
         
         Global Flags:
               --debug            enable debug logging
        @@ -113,18 +76,14 @@
         === pools help
         
         >>> [CLI] air pools --help
        -View GPU pools provisioned for your workspace.
        +View provisioned GPU pools
         
        -A GPU pool is dedicated accelerator capacity that your Databricks account team
        -provisions. Pools are optional: a run uses one only when its YAML sets
        -compute.pool_id.
        -
         Usage:
           databricks air pools [flags]
           databricks air pools [command]
         
         Available Commands:
        -  get         Show a GPU pool and its accelerator usage
        +  get         Show a GPU pool, including its accelerator usage (POOL_ID may be omitted when the workspace has exactly one pool)
           list        List the GPU pools available to the current workspace
         
         Flags:
        @@ -141,7 +94,7 @@
         === images help
         
         >>> [CLI] air images --help
        -Push container images for custom environments
        +Manage container images for AI Runtime
         
         Usage:
           databricks air images [flags]
        @@ -165,7 +118,7 @@
         
         >>> [CLI] air images push --help
         This CLI command is in Preview. For current feature status and requirements, see:
        -  AWS: https://docs.databricks.com/aws/en/machine-learning/ai-runtime/connecting/cli/docker-images
        +  AWS: https://docs.databricks.com/aws/en/machine-learning/ai-runtime/cli/docker-images
           Azure: https://learn.microsoft.com/azure/databricks/machine-learning/ai-runtime/connecting/cli/docker-images
         
         Push a container image to Databricks Artifact Registry under the specified Unity Catalog catalog and schema.
        @@ -201,7 +154,7 @@
         Stream logs from an active run until it completes, or fetch logs from a
         completed run.
         
        -JOB_RUN_ID is the job run ID returned by "databricks air run".
        +JOB_RUN_ID is the job run ID returned by "air run".
         
         Usage:
           databricks air logs JOB_RUN_ID [flags]
        
FAIL
TestAccept/air/list: branch ✅ | main (2699068) ❌ | latest (v1.26.0) ❌

main (2699068):

acceptance_test.go:1374: Diff:
        --- air/list/output.txt
        +++ /tmp/TestAcceptairlist2911671035/002/output.txt
        @@ -2,9 +2,8 @@
         === list (text)
         
         >>> [CLI] air list
        -   Run ID           Experiment              Status     Started              Duration  Progress  MLflow                  User                   Accelerators
        -   334747067049496  qwen-train              ● SUCCESS  [TIMESTAMP]       12s         -  qwen-train-001          [USERNAME]  8x H100
        -   334747067049500  air-byot-multitask-dag  ● SUCCESS  [TIMESTAMP]       20s         -  qwen-train-001          [USERNAME]  1x A10
        +   Run ID           Experiment  Status     Started              Duration  Progress  MLflow                  User                   Accelerators
        +   334747067049496  qwen-train  ● SUCCESS  [TIMESTAMP]       12s         -  qwen-train-001          [USERNAME]  8x H100
         
         === list (json)
         
        @@ -16,7 +15,6 @@
             "runs": [
               {
                 "run_id": "334747067049496",
        -        "task_key": "",
                 "run_name": "qwen-train",
                 "user": "[USERNAME]",
                 "status": "SUCCESS",
        @@ -27,20 +25,6 @@
                 "accelerator_type": "GPU_8xH100",
                 "num_accelerators": 8,
                 "dashboard_url": "[DATABRICKS_URL]/jobs/runs/334747067049496?o=[WORKSPACE_ID]"
        -      },
        -      {
        -        "run_id": "334747067049500",
        -        "task_key": "train",
        -        "run_name": "air-byot-multitask-dag",
        -        "user": "[USERNAME]",
        -        "status": "SUCCESS",
        -        "started_at": "[TIMESTAMP]",
        -        "is_sweep": false,
        -        "experiment_name": "air-byot-multitask-dag",
        -        "duration_seconds": 20,
        -        "accelerator_type": "GPU_1xA10",
        -        "num_accelerators": 1,
        -        "dashboard_url": "[DATABRICKS_URL]/jobs/runs/334747067049500?o=[WORKSPACE_ID]"
               }
             ]
           }
        @@ -62,7 +46,6 @@
             "runs": [
               {
                 "run_id": "334747067049496",
        -        "task_key": "",
                 "run_name": "qwen-train",
                 "user": "[USERNAME]",
                 "status": "SUCCESS",

latest (v1.26.0):

acceptance_test.go:1374: Diff:
        --- air/list/output.txt
        +++ /tmp/TestAcceptairlist1991512940/002/output.txt
        @@ -2,9 +2,8 @@
         === list (text)
         
         >>> [CLI] air list
        -   Run ID           Experiment              Status     Started              Duration  Progress  MLflow                  User                   Accelerators
        -   334747067049496  qwen-train              ● SUCCESS  [TIMESTAMP]       12s         -  qwen-train-001          [USERNAME]  8x H100
        -   334747067049500  air-byot-multitask-dag  ● SUCCESS  [TIMESTAMP]       20s         -  qwen-train-001          [USERNAME]  1x A10
        +   Run ID           Experiment  Status     Started              Duration  Progress  MLflow                  User                   Accelerators
        +   334747067049496  qwen-train  ● SUCCESS  [TIMESTAMP]       12s         -  qwen-train-001          [USERNAME]  8x H100
         
         === list (json)
         
        @@ -16,31 +15,11 @@
             "runs": [
               {
                 "run_id": "334747067049496",
        -        "task_key": "",
                 "run_name": "qwen-train",
                 "user": "[USERNAME]",
                 "status": "SUCCESS",
                 "started_at": "[TIMESTAMP]",
        -        "is_sweep": false,
        -        "experiment_name": "qwen-train",
        -        "duration_seconds": 12,
        -        "accelerator_type": "GPU_8xH100",
        -        "num_accelerators": 8,
        -        "dashboard_url": "[DATABRICKS_URL]/jobs/runs/334747067049496?o=[WORKSPACE_ID]"
        -      },
        -      {
        -        "run_id": "334747067049500",
        -        "task_key": "train",
        -        "run_name": "air-byot-multitask-dag",
        -        "user": "[USERNAME]",
        -        "status": "SUCCESS",
        -        "started_at": "[TIMESTAMP]",
        -        "is_sweep": false,
        -        "experiment_name": "air-byot-multitask-dag",
        -        "duration_seconds": 20,
        -        "accelerator_type": "GPU_1xA10",
        -        "num_accelerators": 1,
        -        "dashboard_url": "[DATABRICKS_URL]/jobs/runs/334747067049500?o=[WORKSPACE_ID]"
        +        "is_sweep": false
               }
             ]
           }
        @@ -62,17 +37,11 @@
             "runs": [
               {
                 "run_id": "334747067049496",
        -        "task_key": "",
                 "run_name": "qwen-train",
                 "user": "[USERNAME]",
                 "status": "SUCCESS",
                 "started_at": "[TIMESTAMP]",
        -        "is_sweep": false,
        -        "experiment_name": "qwen-train",
        -        "duration_seconds": 12,
        -        "accelerator_type": "GPU_8xH100",
        -        "num_accelerators": 8,
        -        "dashboard_url": "[DATABRICKS_URL]/jobs/runs/334747067049496?o=[WORKSPACE_ID]"
        +        "is_sweep": false
               }
             ]
           }

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 14, 2026

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 52dfc19

Run: 43557747252

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 270 22 3:28
💚​ aws-windows-latest-4core-16gb 1 272 20 2:50
💚​ azure linux-2core-8gb 1 269 22 3:28
💚​ azure-windows-latest-4core-16gb 1 271 20 3:28
💚​ gcp linux-2core-8gb 1 270 22 3:32
💚​ gcp-windows-latest-4core-16gb 1 272 20 2:41
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 3 slowest tests (at least 2 minutes):
duration env testname
3:24 azure-windows-latest-4core-16gb TestAccept
2:45 aws-windows-latest-4core-16gb TestAccept
2:37 gcp-windows-latest-4core-16gb TestAccept

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rclarey rclarey Awaiting requested review from rclarey rclarey is a code owner automatically assigned from databricks/eng-deco-cli

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

AIR Databricks AI Runtime CLI

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@dimk0 @eng-dev-ecosystem-bot