Temporary Scratch Space
Palmetto 2 provides a space to store intermediate scratch data that is accessible on every node.
The two current scratch file systems are: shared scratch and local scratch.
Take care not to leave precious research results or programming assignments in scratch space for very long. Move files you need to keep to a permanent storage medium.
Scratch spaces are never backed up. Files deleted or automatically purged from scratch space are permanently destroyed. Our support team cannot recover files stored in scratch space.
Shared Scratch (/scratch)
A large amount of scratch space is available to all users on the cluster through the shared scratch space.
We have designed the shared scratch file system to be resilient against hardware failure. If a disk fails, in most cases the system can continue operating without data loss. However, understand that no backups are available.
Shared Scratch Limits
Every Palmetto 2 user has their own directory in /scratch under their
username (/{filesystem}/{username}). There are larger limits (compared to your
home directory) to the number and size of the files you can store in your
scratch space.
| Limit Description | Maximum Value |
|---|---|
| Per-User Storage Quota | 10 TB |
| Per-User File Count | 10 million files |
Need a higher limit on /scratch for your account?
If you need to increase your storage limit or file count limit on /scratch
please reach out to us by
submitting a support ticket.
Include what you'd like the new usage to be and a comprehensive explanation as to why it is needed. We may ask to discuss your workflow to see if it can be changed so that you fit within the limits.
How do I check how many files I have in shared scratch?
You can use the following commands to check your file count on the scratch system. These should be run from a compute node.
srun bash -c 'find -type f /scratch/$USER | wc -l'
How do I check my disk space usage on shared scratch?
You can use the checkquota command to check your file count on the scratch
file system.
The checkquota command is only available on the login nodes.
$ checkquota
Quota infomation for rcdstudent
Path : /home/rcdstudent
Total Capacity : 250.0 GiB
Used : 1.64 GiB
------------------------------------
Path : /scratch/rcdstudent
Total Capacity : 5.0 TiB
Used : 451.62 GiB
------------------------------------
Shared Scratch Purge Schedule
To ensure that space remains available for everyone, our system will periodically purge old files.
This purge happens every day at 1:00 AM EDT.
During the purge, files on scratch will be deleted if ALL of the following are true:
- the file has not been accessed/read within the last 30 days
- the file has not been modified within the last 30 days
- the file has not had a metadata change within the last 30 days
How can I check when a file was last accessed or modified?
You can check when an individual file was last accessed or modified using the
stat command.
$ stat /scratch/user/file1
File: file1
Size: 0 Blocks: 0 IO Block: 524288 regular empty file
Device: 2dh/45d Inode: 73496411419974474 Links: 1
Access: (0644/-rw-r--r--) Uid: (342429/ bfgodfr) Gid: (10000/ cuuser)
Access: 2023-01-23 10:44:10.000000000 -0500
Modify: 2023-01-23 10:44:10.000000000 -0500
Change: 2023-01-23 10:44:10.000000000 -0500
Birth: -
The following explains each timestamp:
| Output | Meaning |
|---|---|
Access: | the date and time the file was last accessed |
Modify: | the date and time the file was last modified |
Change: | the date and time of the last metadata change on this file |
You can see a list of all of your files, sorted by their access date, by running this command:
srun find /scratch/$USER -type f -printf "%A+ %p\n" | sort -n
You can see a list of all of your files, sorted by their modify date, by running this command:
srun find /scratch/$USER -type f -printf "%A+ %p\n" | sort -n
Can I get a list of files that are at risk of being purged?
To see a list of your files on /scratch that are at risk of being purged
within the next day, you can run:
srun find /scratch/$USER -type f -a \( -atime +29 -a -mtime +29 -a -ctime +29 \)
To see a list that may be purged in the next week, you can look at the same, but for 22 days ago:
srun find /scratch/$USER -type f -a \( -atime +22 -a -mtime +22 -a -ctime +22 \)
Can I just touch or copy my files to avoid the purge?
Altering or copying files for the express purpose of circumventing the purge system is prohibited. If our administrators detect this behavior, your files may be manually purged without notice.
The intent is for scratch file systems to only be used as temporary space. We encourage users to move intermediate and final results off of scratch space as soon as possible when they are not being used.
If you are struggling to make your workflow function within the purge window, please submit a support ticket. We want to make sure our users are able to complete their research and keep valuable data, while also keeping the system available for other users. Our team can work with you to find a solution.
Local Scratch ($TMPDIR)
Every Palmetto 2 compute node has a local disk within the node itself for temporary files, which is referred to as local scratch. This scratch space can only be accessed by a job running on the node. The file system for local scratch has no hardware failure protection and is never backed up.
Slurm will automatically set up a local scratch directory for you when your job
begins and place the path in the $TMPDIR environment variable.
The path for local scratch may change, so be sure to always use the $TMPDIR
variable to determine the correct location.
Files in local scratch will be purged automatically when your job terminates. Copy files elsewhere before your job ends. Files left behind in local scratch are lost forever and cannot be recovered.
Why do I see the same path in $TMPDIR across jobs?
$TMPDIR across jobs?Prior to the August 2026 maintenance, users may have noticed that the $TMPDIR
variable contained a subdirectory of /local_scratch with their job ID.
To make it easier to use local scratch and handle cases where programs were not
designed to read $TMPDIR, we now use Linux mount namespaces to transparently
map common system temporary file locations to the underlying job-specific
directories.
As a result, you cannot directly access or cd into your job's directory in
/local_scratch anymore.
Users should use the $TMPDIR variable in their scripts to ensure they are
always using the correct location for temporary files, since the path is subject
to change again in the future.
How much space is available in local scratch?
Since Palmetto is a heterogeneous cluster with varying hardware specifications, the size of the local disk in each node may be different.
You can see the amount of local scratch storage available on your current node
using the df command:
$ df --human-readable --output=size "$TMPDIR"
Size
414G
Note that while each job gets its own separate local scratch space, the amount of space available is shared between all jobs running on a node at the same time.
How do I set up local scratch for use on multiple nodes?
If you want to access local storage on each node, when running a multiple node job, the Slurm job script must first set up local storage on each node in the following way:
# copy data TO local scratch FROM your source ON EACH NODE in the job
srun --ntasks-per-node=1 \
cp /path/to/remote/dir/{input files} $TMPDIR/{input files}
# ... run your other commands here ...
# copy data FROM local scratch BACK TO your source ON EACH NODE in the job
srun --ntasks-per-node=1 \
cp $TMPDIR/{output files} /path/to/remote/dir/$node/{output files}