Temporary Scratch Space
Palmetto provides a space to store intermediate scratch data that is accessible on every node.
The two current scratch file systems are: /scratch, and /local_scratch.
Take care not to leave precious research results or programming assignments in scratch space for very long. Move files you need to keep to a permanent storage medium.
Scratch spaces are never backed up. Files deleted or automatically purged from scratch space are permanently destroyed. Our support team cannot recover files stored in scratch space.
Shared Scratch: /scratch
A large amount of scratch space is available to all users on the cluster through the shared scratch space.
We have designed the shared scratch file system to be resilient against hardware failure. If a disk fails, in most cases the system can continue operating without data loss. However, understand that no backups are available.
Shared Scratch Limits
Every Palmetto user has their own directory in /scratch under their username
(/{filesystem}/{username}). There are larger limits (compared to your home
directory) to the number and size of the files you can store in your scratch
space.
| Per-User Storage Limit | Per-User File Count Limit |
|---|---|
| 5 TB | 5 million files |
Need a higher limit on /scratch for your account?
If you need to increase your Storage Limit or File Count Limit on /scratch
please reach out to us by
submitting a support ticket.
Include what you'd like the new usage to be and a comprehensive explanation as to why it is needed. We may ask to discuss your workflow to see if it can be changed so that you fit within the limits.
How do I check how many files I have in shared scratch?
You can use the following commands to check your file count on the scratch system. These should be run from a compute node.
find -type f /scratch/$USER | wc -l
How do I check my disk space usage on shared scratch?
You can use the following commands to check your file count on the scratch file system. These should be run from a compute node.
du -sh /scratch/$USER
Shared Scratch Purge Schedule
To ensure that space remains available for everyone, our system will periodically purge old files. Any files that have not been modified within a defined number of days will be automatically deleted.
| Purge Criteria | Purge Schedule |
|---|---|
| 30 days since last modified timestamp | 11:00 PM EDT daily |
Accessing or reading a file will not prevent it from being purged. The last modified timestamp determines purge eligibility.
How can I check the date a file was last modified?
Files are purged based on when they were last modified, regardless of when they were last accessed.
You can check the time which an individual file was last modified using the
stat command.
user@login001 $ stat /scratch/user/file1
File: file1
Size: 0 Blocks: 0 IO Block: 524288 regular empty file
Device: 2dh/45d Inode: 73496411419974474 Links: 1
Access: (0644/-rw-r--r--) Uid: (342429/ bfgodfr) Gid: (10000/ cuuser)
Access: 2023-01-23 10:44:10.000000000 -0500
Modify: 2023-01-23 10:44:10.000000000 -0500
Change: 2023-01-23 10:44:10.000000000 -0500
Birth: -
The Modify timestamp shows the time that it was last modified.
You can see a list of all of your files, sorted by their modified date, by running these commands from a compute node:
find /scratch/$USER -type f -printf "%Tc %p\n" | sort -n
Can I get a list of files that are at risk of being purged?
To see a list of your files on /scratch that are at risk of being purged in
the next week, you can run:
find /scratch/$USER -type f -a \( -atime +30 -a -mtime +30 -a -ctime +30 \)
Can I just touch or copy my files to avoid the purge?
Altering or copying files for the express purpose of circumventing the purge system is prohibited. If our administrators detect this behavior, your files may be manually purged without notice.
The intent is for scratch file systems to only be used as temporary space. We encourage users to move intermediate and final results off of scratch space as soon as possible when they are not being used.
If you are struggling to make your workflow function within the purge window, please submit a support ticket. We want to make sure our users are able to complete their research and keep valuable data, while also keeping the system available for other users. Our team can work with you to find a solution.
Local Scratch
Every Palmetto compute node has storage on the node itself and is referred to as
/local_scratch. This scratch space can only be accessed by a job running on
the node. The file system for /local_scratch has no hardware failure
protection and is never backed up.
To access this space in your PBS script, use the $TMPDIR environment variable
or the location /local_scratch/pbs.$PBS_JOBID. If you are using multiple
nodes, further setup is required.
Files in /local_scratch will be purged automatically when your job terminates.
Copy files elsewhere before your job ends.
You can see the amount of storage by looking at the hardware table from the login node:
user@login $ cat /etc/hardware-table
The hardware table identifies compute nodes by phases. Each phase has a specific
disk medium and size to house its /local_scratch.
Local scratch space is shared between all jobs running on a node at the same time. If your job requires the entire space, then your job must request the entire node. To request the entire node, specify all the cpu cores for the particular phase in your PBS script. For example:
#PBS select=1:ncpus=16:mem=4gb:phase=8b
How do I set up local scratch for use on multiple nodes?
When running a multiple node job and you want to access this local storage on each node, the PBS script must first set up the local storage on each node in the following way:
# ensure that /local_scratch is created on all nodes
# pbsdsh issues the sleep command on each node assigned
# to your job, which in turn signals PBS to create the local storage
# directory on each node before running the sleep command. Any
# command may be executed; sleep is just a quick command to
# process.
module load gcc/4.8.1 openmpi/1.10.3
pbsdsh sleep 10
module purge
#copy data to local_scratch
for node in `uniq $PBS_NODEFILE`
do
ssh $node cp /path/to/remote/dir/{input files} \
$TMPDIR/{input files}
done
# ….other commands
#copy data from local_scratch
for node in `uniq $PBS_NODEFILE`
do
ssh $node cp $TMPDIR/{output files} \
/path/to/remote/dir/$node/{output files}
done
Deprecated Scratch Spaces
Both /scratch1 and /fastscratch are being replaced with the single shared
scratch service /scratch. The older scratch systems were hosted on aging
equipment and are scheduled to be removed from the cluster in the summer 2024
maintenance.
Both of these systems are no longer provisioned for new users, instead they
must make use of the newer /scratch system. Older users should transition
their workflows over to use the new scratch system before /scratch1 and
/fastscratch are removed.
For the sake of completeness, we include the details for /scratch1,
/fastscratch below.
Limits
| File System | Per-User Storage Limit | Per-User File Count Limit |
|---|---|---|
/fastscratch | None | please keep below 1 million files |
/scratch1 | None | please keep below 1 million files |
Purge cycle
| File System | Purge Criteria | Purge Schedule |
|---|---|---|
/fastscratch | 14 days since last modified timestamp | 11:00 PM EDT daily |
/scratch1 | 30 days since last modified timestamp | 11:00 PM EDT daily |
To see a list of your files on /scratch1 that are at risk of being purged in
the next week, you can run:
find /scratch1/$USER -type f -a \( -atime +30 -a -mtime +30 -a -ctime +30 \)
To see a list of your files on /fastscratch that are at risk of being purged
in the next week, you can run:
find /fastscratch/<user> -type f -a \( -atime +14 -a -mtime +14 -a -ctime +14 \)