Job Cluster
Controls how Databricks job tasks are assigned compute resources.
| Value | Behavior |
|---|---|
Serverless (default) | Uses Databricks serverless compute; the generated job tasks carry no cluster configuration and set disable_auto_optimization: true |
A databricks.yml variable name | The job defines a job cluster (job_cluster_key with new_cluster: ${var.<name>}) from the named complex variable: Spark version, node type, scaling and other cluster attributes; the sample databricks.yml includes such a variable |
| An existing cluster ID | Used when Use Existing Cluster is enabled; the tasks reference the cluster with existing_cluster_id |
When to Change This Setting
Keep the default Serverless for the simplest operation with no cluster definitions to manage. Point it at a databricks.yml cluster variable when workloads need a specific cluster configuration, or pair it with Use Existing Cluster to run against an always-on cluster you manage yourself.
Related Settings
| Setting | Interaction |
|---|---|
| Use Existing Cluster | Switches this value from a cluster variable to an existing cluster ID |
| Utils Driver | JDBC is the driver option compatible with serverless compute |
| Git Source | The companion databricks.yml variable reference for notebook source |
Notes
- This setting is part of the Databricks settings category.
- The default value for this setting is
Serverless.