seff and sacct usage
seff
Slurm comes installed with a simple job efficiency script, seff. With it we
can gather an estimate on how much resources were used.
To run seff, just pass the job ID:
seff <job-id>
It will print out summary statistics and efficiency information about the job:
[dndawso@slogin001 ~]$ seff 2917
Job ID: 2917
Cluster: palmetto2
User/Group: dndawso/cuuser
State: COMPLETED (exit code 0)
Nodes: 4
Cores per node: 4
CPU Utilized: 01:06:03
CPU Efficiency: 38.76% of 02:50:24 core-walltime
Job Wall-clock time: 00:10:39
Memory Utilized: 21.93 GB
Memory Efficiency: 17.14% of 128.00 GB
GPU Usage is not tracked by the scheduler and thus not available in seff.
While you can run seff while the job is running, it likely won't be accurate
until the job is complete. Slurm does not update accounting statistics very
often.
Why does seff differ from jobperf in overall statistics?
seff differ from jobperf in overall statistics?The CPU usage should be the same in each. If there are differences, please let
us know. Memory differences are expected, however. The Slurm database does not
store a total, max memory used. Instead it stores the max memory used by any
task for each step (TresUsageInMax column in salloc).
The seff algorithm is as follows: For each step, calculate approximate max
memory usage by multiplying the memory from TresUsageInMax times the number of
tasks. It then reports the highest memory calculated of any step.
This works well in most cases.
- For single step, single task jobs, it is accurate.
- For single step, multiple task jobs, it may overestimate, but should work well if most of the step tasks are doing the same thing.
- For multiple non-overlapping step, single task jobs, it is accurate.
- For multiple non-overlapping step, multiple task jobs, it may overestimate, but should work well if most of the step tasks are doing the same thing.
- For overlapping steps, it may be quite inaccurate.
Jobperf uses a slightly different algorithm:
- Collect all start and end times for each step.
- For each collected time, t:
- Find all steps that were running (step start time <= t && end time > t)
- For each running step, compute approx max memory use as TresUsageInMax times number of tasks.
- Record this time's memory as the sum of all steps max memory used.
- Report the highest memory usage of all times.
It should provide the same numbers as seff for single step jobs, but will have
higher numbers for multiple steps if they overlap.
sacct
The sacct utility displays accounting
data from the accounting database for jobs. Once a job starts running, entries
are added to the accounting database for each step (each invocation of srun
within a job).
By default, running sacct with no arguments will show you all your jobs and
steps from today. You can look further back by passing the --starttime flag.
For example, to see jobs ran in the past 5 days, run:
sacct --starttime now-5days
You can also pass --job flag to look at just steps related to a job. For
example, if you want to see information on steps from job ID 3016, run:
sacct --job 3016
By default, sacct does not show information related to resource utilization.
We'll need to adjust the columns. These are the most useful:
TotalCPU: total CPU time used by the step.NTasks: total number of tasks in the step.TRESUsageInMax: maximum usage of resources for each task of the stop.
These options can be passed as arguments to the --format parameter of sacct.
For example, to check the usage of job ID 2917, I can use
sacct --job 2917 --format JobID,JobName,NTasks,TotalCPU,TRESUsageInMax%80
That produces the following output:
JobID JobName NTasks TotalCPU TRESUsageInMax
------------ ---------- -------- ---------- --------------------------------------------------------------------------------
2917 clarityBe+ 01:06:02
2917.batch batch 1 01:18.736 cpu=00:01:19,energy=0,fs/disk=500405988,mem=338256K,pages=1940,vmem=694264K
2917.extern extern 4 00:00.002 cpu=00:00:00,energy=0,fs/disk=2332,mem=0,pages=0,vmem=0
2917.0 jobperf 1 00:00.045 cpu=00:00:00,energy=0,fs/disk=59720,mem=4676K,pages=58,vmem=12176K
2917.1 jobperf 1 00:00.046 cpu=00:00:00,energy=0,fs/disk=60320,mem=4680K,pages=0,vmem=4680K
2917.2 jobperf 1 00:00.044 cpu=00:00:00,energy=0,fs/disk=59263,mem=4676K,pages=58,vmem=12176K
2917.3 jobperf 1 00:00.044 cpu=00:00:00,energy=0,fs/disk=59252,mem=4676K,pages=58,vmem=12172K
2917.4 svclaunch+ 1 20:52.665 cpu=00:20:53,energy=0,fs/disk=729913299,mem=22998628K,pages=524,vmem=23075964K
2917.5 svclaunch+ 1 16:21.333 cpu=00:16:20,energy=0,fs/disk=492817603,mem=21843172K,pages=490,vmem=21912144K
2917.6 svclaunch+ 1 13:14.594 cpu=00:13:13,energy=0,fs/disk=361008588,mem=22477272K,pages=566,vmem=22545836K
2917.7 svclaunch+ 1 14:15.009 cpu=00:14:14,energy=0,fs/disk=378631749,mem=21848644K,pages=543,vmem=21925064K
With this I can see step 2917.4 used 20 minutes, and 52.665 seconds of
CPU Time. It also had a task that
maxed out at 21848644 KiB (from the mem within TRESUsageInMax) = 20.83 Gib.
This is the only task within the step (NTasks=1) so the max memory used by the
step is 20.83Gib. For this particular job, steps 2917.4-2917.7 were all run at
the same time, so to get total memory used for the whole job I add the memory
usages up to get about 85GiB. Many times, steps are not run at the same time,
they are run sequentially which changes the math (you'd look at the max memory
used rather than sum).
Steps may have multiple tasks (e.g. an MPI step). For example, this step had 20 tasks:
JobID JobName NTasks TotalCPU TRESUsageInMax
------------ ---------- -------- ---------- --------------------------------------------------------------------------------
2428.0 pw.x 20 02:18:36 cpu=00:06:57,energy=0,fs/disk=4018983,mem=337392K,pages=241,vmem=3079476K
The usage listed in TRESUsageInMax is max over all tasks so to estimate total
usage, we can multiply 337392 KiB times 20 to get 6.4 GiB total.