Temporary Scratch Space
Palmetto provides a space to store intermediate scratch data that is accessible on every node.
The two current scratch file systems are: /scratch, and /local_scratch.
Take care not to leave precious research results or programming assignments in scratch space for very long. Move files you need to keep to a permanent storage medium.
Scratch spaces are never backed up. Files deleted or automatically purged from scratch space are permanently destroyed. Our support team cannot recover files stored in scratch space.
Shared Scratch (/scratch)
A large amount of scratch space is available to all users on the cluster through the shared scratch space.
We have designed the shared scratch file system to be resilient against hardware failure. If a disk fails, in most cases the system can continue operating without data loss. However, understand that no backups are available.
Shared Scratch Limits
Every Palmetto user has their own directory in /scratch under their username
(/{filesystem}/{username}). There are larger limits (compared to your home
directory) to the number and size of the files you can store in your scratch
space.
| Limit Description | Maximum Value |
|---|---|
| Per-User Storage Quota | 5 TB |
| Per-User File Count | 5 million files |
Need a higher limit on /scratch for your account?
If you need to increase your Storage Limit or File Count Limit on /scratch
please reach out to us by
submitting a support ticket.
Include what you'd like the new usage to be and a comprehensive explanation as to why it is needed. We may ask to discuss your workflow to see if it can be changed so that you fit within the limits.
How do I check how many files I have in shared scratch?
You can use the following commands to check your file count on the scratch system. These should be run from a compute node.
srun bash -c 'find -type f /scratch/$USER | wc -l'
How do I check my disk space usage on shared scratch?
You can use the checkquota command to check your file count on the scratch
file system. This command must be run from the login node.
[rcdstudent@vm-slurm-p-login01 ~]$ checkquota
Quota infomation for rcdstudent
Path : /home/rcdstudent
Total Capacity : 250.0 GiB
Used : 1.64 GiB
------------------------------------
Path : /scratch/rcdstudent
Total Capacity : 5.0 TiB
Used : 451.62 GiB
------------------------------------
Shared Scratch Purge Schedule
To ensure that space remains available for everyone, our system will periodically purge old files.
This purge happens every day at 1:00 AM EDT.
During the purge, files on scratch will be deleted if ALL of the following are true:
- the file has not been accessed/read within the last 30 days
- the file has not been modified within the last 30 days
- the file has not had a metadata change within the last 30 days
How can I check the date a file was last accessed or modified?
You can check the time which an individual file was last accessed or modified
using the stat command.
user@login001 $ stat /scratch/user/file1
File: file1
Size: 0 Blocks: 0 IO Block: 524288 regular empty file
Device: 2dh/45d Inode: 73496411419974474 Links: 1
Access: (0644/-rw-r--r--) Uid: (342429/ bfgodfr) Gid: (10000/ cuuser)
Access: 2023-01-23 10:44:10.000000000 -0500
Modify: 2023-01-23 10:44:10.000000000 -0500
Change: 2023-01-23 10:44:10.000000000 -0500
Birth: -
The following explains each timestamp:
| Output | Meaning |
|---|---|
Access: | the date and time the file was last accessed |
Modify: | the date and time the file was last modified |
Change: | the date and time of the last metadata change on this file |
You can see a list of all of your files, sorted by their access date, by running this command:
srun find /scratch/$USER -type f -printf "%A+ %p\n" | sort -n
You can see a list of all of your files, sorted by their modify date, by running this command:
srun find /scratch/$USER -type f -printf "%A+ %p\n" | sort -n
Can I get a list of files that are at risk of being purged?
To see a list of your files on /scratch that are at risk of being purged
within the next day, you can run:
srun find /scratch/$USER -type f -a \( -atime +29 -a -mtime +29 -a -ctime +29 \)
To see a list that may be purged in the next week, you can look at the same, but for 22 days ago:
srun find /scratch/$USER -type f -a \( -atime +22 -a -mtime +22 -a -ctime +22 \)
Can I just touch or copy my files to avoid the purge?
Altering or copying files for the express purpose of circumventing the purge system is prohibited. If our administrators detect this behavior, your files may be manually purged without notice.
The intent is for scratch file systems to only be used as temporary space. We encourage users to move intermediate and final results off of scratch space as soon as possible when they are not being used.
If you are struggling to make your workflow function within the purge window, please submit a support ticket. We want to make sure our users are able to complete their research and keep valuable data, while also keeping the system available for other users. Our team can work with you to find a solution.
Local Scratch (/local_scratch)
Every Palmetto compute node has storage on the node itself and is referred to as
/local_scratch. This scratch space can only be accessed by a job running on
the node. The file system for /local_scratch has no hardware failure
protection and is never backed up.
To access this space in your Slurm job script, use the $TMPDIR environment
variable or the location /local_scratch/slurm.$SLURM_JOBID. If you are using
multiple nodes, further setup is required.
Files in /local_scratch will be purged automatically when your job terminates.
Copy files elsewhere before your job ends.
You can see the amount of local scratch storage using the df command:
user@node $ df --human-readable --output=size /local_scratch/
Size
414G
Local scratch space is shared between all jobs running on a node at the same time.
How do I set up local scratch for use on multiple nodes?
When running a multiple node job and you want to access this local storage on each node, the Slurm job script must first set up the local storage on each node in the following way:
# copy data TO /local_scratch FROM your source ON EACH NODE in the job
srun --ntasks-per-node=1 \
cp /path/to/remote/dir/{input files} $TMPDIR/{input files}
# ... run your other commands here ...
# copy data FROM /local_scratch BACK TO your source ON EACH NODE in the job
srun --ntasks-per-node=1 \
cp $TMPDIR/{output files} /path/to/remote/dir/$node/{output files}