How to Configure Dask Labextension Cluster

The Dask JupyterLab extension package provides a JupyterLab extension to manage Dask clusters, as well as to embed Dask’s dashboard plots directly into JupyterLab panes.

The ~/.config/dask/jobqueue-coffea-casa.yaml or /etc/dask/jobqueue-coffea-casa.yaml files are usually the default configuration files used for CoffeaCasaCluster:

Example of a file:

jobqueue:
  coffea-casa:

    # Dask worker options, taken from https://github.com/dask/dask-jobqueue/tree/master/dask_jobqueue
    cores: 4                 # Total number of cores per job
    memory: "6 GiB"                # Total amount of memory per job
    processes: null                # Number of Python processes per jobs
    worker-image: "coffeateam/coffea-casa-analysis:0.xx.xx"

    # Comunication settings
    interface: null             # Network interface to use like eth0 or ib0
    death-timeout: 60           # Number of seconds to wait if a worker can not find a scheduler
    local-directory: null       # Location of fast local storage like /scratch or $TMPDIR
    extra: []

    # HTCondor Resource Manager options
    disk: "5 GiB"          # Amount of disk per worker job
    env-extra: []
    job-extra: {}          # Extra submit attributes
    log-directory: null
    shebang: "#!/usr/bin/env condor_submit -spool"

    # Scheduler options
    scheduler-options: {}
    name: dask-worker

To configure a cluster that is launched using it, you should adjust the Dask configuration file, typically stored at ~/.config/dask/labextension.yaml or /etc/dask/labextension.yaml.

labextension:
  factory:
    module: 'coffea_casa'
    class: 'CoffeaCasaCluster'
    args: []
    kwargs: {}

    default:
      workers: 1
      adapt:
        minimum: 5
        maximum: 10

Users can edit kwargs: {} to change a CoffeaCasaCluster constructor directly (see more details in Coffea-Casa Setup Without Dask Labextention).

To get an address of scheduler that will be used during client connection, try right-clicking on the cluster in the sidebar:

And then pressing Inject Dask Client Connection Code, as is shown in example below:

from dask.distributed import Client
client = Client("tls://oksana-2eshadura-40cern-2ech.dask.coffea.casa:8786")
client

or, more simply:

from dask.distributed import Client
client = Client("tls://localhost:8786")
client