Running on a Cluster#

A flowgraph is a set of independent tasks with declared dependencies, which is exactly the shape a job scheduler wants. Handing the nodes to a cluster instead of running them on your laptop is one option:

project.option.scheduler.set_name("slurm")

Everything else – the design, the flow, the target – is unchanged. The same script runs locally, on a cluster, or in containers.

What the schedulers are#

Value	Runs each node
(unset)	As a process on the machine you launched from. The default.
`slurm`	As a Slurm job. Needs a reachable `slurmctld`.
`docker`	In its own container – see Docker

Note

lsf and sge are accepted by the schema but have no implementation – only slurm and docker are dispatched. Setting either is not silently ignored: the run fails at start-up, before any node executes, with

Unsupported scheduler 'lsf' for node <step>/<index>

The same applies when it is set on a single node (set_name("lsf", step="route")) rather than on the whole run. If you need one of them, say so on Discussions; the dispatch layer is small and the Slurm one is the template.

Important

The build directory must be on shared storage. Nodes pass results to each other through build/, so every host that might run a node has to see the same filesystem at the same path. This is the single most common reason a working local build fails on a cluster.

project.option.set_builddir("/shared/scratch/me/build")

Asking for resources#

Per-node resource requests translate to the scheduler’s own switches:

project.option.scheduler.set_cores(16)
project.option.scheduler.set_memory(64000)      # MB
project.option.scheduler.set_queue("bigmem")    # partition

All three take step=/index=, which is usually what you want – routing needs a different machine than linting does:

project.option.scheduler.set_cores(32, step="route")
project.option.scheduler.set_memory(128000, step="route")

Anything the accessors do not cover goes through add_options(), which passes switches to the scheduler verbatim.

Bounding the fan-out#

On a cluster the limit stops being your core count and starts being what you are allowed to occupy:

project.option.scheduler.set_maxnodes(50)    # concurrent nodes
project.option.scheduler.set_maxthreads(8)   # threads per tool

These compose with the flow’s own width. A sweep of 10 jobs, each with syn_np=4, is 40 concurrent tool invocations before either of these applies – see Parallel Job Execution, which is worth reading first if you are about to point this at a shared machine.

Being told when it finishes#

A cluster run is one you walk away from:

project.option.scheduler.add_msgcontact("[email protected]")
project.option.scheduler.add_msgevent("end")     # also: "begin", "timeout", "fail"

Job status emails covers what arrives.

Delaying the start#

set_defer() takes a time and passes it to Slurm as --begin, so the job is queued but does not start until then – useful for holding a long build until a shared machine is quiet.

Note

This does not detach the run. SiliconCompiler still waits for the job to finish, so the process stays alive for the delay and the build. There is currently no fire-and-forget submission; use your shell or the scheduler directly if you need one.

Setting up Slurm#

If you do not already have a cluster, Slurm setup walks through configuring a single machine as a one-host cluster, which is the right way to test that a script works under a scheduler before asking for real resources. Add hosts afterwards.