Job Control with Slurm
The new Palmetto 2 cluster uses the Slurm Workload Manager to schedule jobs.
Lifecycle of a Slurm Job
The life of a job begins when you submit the job to the scheduler. If accepted, it will enter the Pending state.
Thereafter, the job may move to other states, as defined below:
- Pending - the job has been accepted by the scheduler and is eligible for execution; waiting for resources.
- Held - the job is not eligible for execution because it was held by user request, administrative action, or job dependency.
- Running - the job is currently executing on the compute node(s).
- Completed - the job finished executing successfully.
- Failed - the job finished executing, but returned an error.
- Timeout - the job was executing, but was terminated because it exceeded the amount of wall time requested.
- Out of Memory - the job was executing, but was terminated because it exceeded the amount of memory requested.
- Canceled - the job was canceled by the requestor or a system administrator.
This diagram below provides a visual representation of these states:
For more details, you can review the Job State Codes in the Slurm Documentation.
Slurm Job Control Commands
This section provides essential commands and approaches to efficiently monitor and manage jobs within the Slurm workload manager on Palmetto.
Listing Slurm jobs with squeue
You can list the status of your current jobs using the squeue
command:
squeue -u <username>
where you replace <username>
with your Palmetto user name. This returns basic
information for your queued and running jobs. You can control the output fields
and format using the --format
flag. For instance,
squeue -u <username> --format "%all"
will show all information available for each job.
For more information, see the official
Slurm squeue
documentation. Find
additional examples at the bottom of the page. You can also access this
documentation by running man squeue
from a terminal.
Checking on Slurm jobs with scontrol
You can list detailed information about a specific job using the scontrol
command:
scontrol show job <job-id>
where you replace <job-id>
with the ID of a running or queued job. Retrieve
this ID using the squeue
command as described above. The scontrol show job
command gives detailed information about your job including information about
the resources allocated to the job.
In this example, we used scontrol
to show job information. However, this
powerful command also interacts with other aspects of the slurm configuration
including nodes and partitions. For instance, you can view detailed node
information with scontrol show node <hostname>
where you replace <hostname>
with the host name for a specific node.
For more information, see the official
Slurm scontrol
documentation. You
can also access this documentation by running man scontrol
from a terminal.
See the output of a Slurm job during execution
Slurm logs your job outputs to a text file. By default, this log file is located
in the working directory used when creating the job and has the naming format
slurm-<job-id>.out
, where <job-id>
matches the job ID returned by squeue
.
The name and location of the log file can be changed when
submitting a job. You can view the
output of your running job using standard linux commands. For instance, you can
display live updates using the tail
command:
tail -f path/to/slurm-<job-id>.out
The -f
(follow) flag gives continuous updates as the file changes.
Canceling a Slurm job with scancel
You can terminate a running Slurm job and free the associated resources using
the scancel
command:
scancel <job-id>
where <job-id>
matches the job ID returned by squeue
.
For more information, see the official
Slurm scancel
documentation. You can
also access this documentation by running man scancel
from a terminal.
Connecting to a running Slurm job
Sometimes you may wish to connect to a running job for performance monitoring or debugging purposes. For instructions, refer to our job monitoring documentation.
Viewing available Slurm partitions to run your job
The sinfo -s
command allows you to see all
partitions you have access to. For example:
sinfo -s
Example output could be:
PARTITION AVAIL TIMELIMIT NODES(A/I/O/T) NODELIST
work1* up 3-00:00:00 3/16/0/19 node[0401,0403-0419,0421]
rcde up infinite 3/18/0/21 node[0400-0401,0403-0419,0421,1036]
rcdeprio up infinite 1/3/0/4 node[0400-0401,0404,1036]
training up 2:00:00 0/1/0/1 node0400
This would mean I have access to the work1
(The *
indicates this is the
default), rcde
, rcdeprio
, and training
partitions. To run a job with a
particular partition, use -p <partition-name>
on
srun
,
sbatch
, or
salloc
.