Databricks

Integrate W&B with Databricks for experiment tracking, metric logging, and model management on Spark clusters.

W&B integrates with Databricks by customizing the W&B Jupyter notebook experience in the Databricks environment.

Configure Databricks

1. Install wandb in the cluster

   Navigate to your cluster configuration, choose your cluster, click Libraries. Click Install New, choose PyPI, and add the package wandb.

2. Set up authentication

   To authenticate your W&B account you can add a Databricks secret which your notebooks can query.

   # install databricks cli
   pip install databricks-cli
   
   # Generate a token from databricks UI
   databricks configure --token
   
   # Create a scope with one of the two commands (depending if you have security features enabled on databricks):
   # with security add-on
   databricks secrets create-scope --scope wandb
   # without security add-on
   databricks secrets create-scope --scope wandb --initial-manage-principal users
   
   # Create an API key at https://wandb.ai/settings
   databricks secrets put --scope wandb --key api_key
   

Examples

Simple example

import os
import wandb

api_key = dbutils.secrets.get("wandb", "api_key")
wandb.login(key=api_key)

with wandb.init() as run:
    run.log({"foo": 1})

Sweeps

Setup required (temporary) for notebooks attempting to use wandb.sweep() or wandb.agent():

import os

# These will not be necessary in the future
os.environ["WANDB_ENTITY"] = "my-entity"
os.environ["WANDB_PROJECT"] = "my-project-that-exists"