There are many ways to move data in and out of Palmetto for your research. Selecting the correct method is important, since using the wrong method might impact the performance of shared resources for other users.
Which method to use?
Globus and SFTP via the data transfer nodes (DTNs) are the recommended methods to transfer data between filesystems, as well as into and out of the cluster. The DTNs contain the fastest connections to all storage systems and are dedicated resources for transferring data.
Below are the supported methods to transfer data into Palmetto.
Excessive SFTP data transfer activity on the login and Open OnDemand resources may make these shared systems slow or unusable for other users. If you exceed the limits below, system administrators may terminate your data transfer without warning to protect these shared resources.
|Method||Maximum Transfer Size|
|SFTP (Data Transfer Nodes)||All Transfers|
|SFTP (login nodes)||Small files (< 100 MB total)|
|Open OnDemand||Very small files (< 50 MB total)|
For larger files, we recommend using the Globus file transfer application. Here, we demonstrate how to use Globus Online to transfer files between Palmetto and a local machine (laptop). However, Globus can be used for file transfers to/from other locations as well.
You must have a Globus account to use their services.
If you do not already have one, you can visit the Globus app to create an account.
Select Clemson University on the organizational login screen to take advantage of single sign on.
Globus works by transferring files between collections. The data transfer nodes provide a collection for data stored on Palmetto. The collection is named Clemson RCD Data Transfer.
Search for Clemson RCD to locate the collection.
Installing Globus Connect Personal
To transfer files between your local computer and Palmetto, you must install the Globus Connect Personal utility and create a collection for your local file system. Below is a list of instructions by platform for installing the utility:
- How to Install Globus Connect Personal for Mac OS X
- How to Install Globus Connect Personal for Windows
- How to Install Globus Connect Personal for Linux
Transferring Data with Globus
To begin transferring files, navigate to the Globus online transfer utility.
At the top left of the File Manager, search for the collection name that you selected for your local computer.
To the right, search for the collection name for one of the Palmetto data transfer nodes.note
You can copy and paste the collection names for the Palmetto data transfer nodes from the collections section above.
Click the Transfer or Sync button in the center of the screen to begin your transfer session.
You can now use the Globus interface to transfer files between your local computer and the Palmetto cluster.
Video demonstration of Globus transfer on macOS
Transfers can be completed via SFTP through your SSH client by connecting to one of Palmetto's dedicated data transfer nodes. These nodes are set up to ensure data transfer activity does not impact other users.
- Data Transfer Node 1 -
- Data Transfer Node 2 -
If one data transfer node is unavailable, try using the other one.
For transfers smaller than 100 MB total, you may transfer them via the login nodes.
SFTP on Windows
On Windows machines, using the MobaXterm SSH client, the built-in file browser can be used (SCP tab of the side window). Using the Upload (green arrow pointing up) and and Download (blue arrow pointing down) buttons at the top of the SCP tab, you can easily transfer small files between Palmetto and your local computer.
Video demonstration of MobaXTerm file transfers
SFTP on macOS/Linux
On Unix systems, you can use the
scp (secure copy) command to perform file
transfers. The general form of the
scp command is:
scp <path_to_source> firstname.lastname@example.org:<path_to_destination>
scp <path_to_source> email@example.com:<path_to_destination>
For example, here is the
scp command to copy a file from the current directory
on your local machine to your
/home/username directory on Palmetto (this
command is entered into a terminal when not logged-in to Palmetto):
scp myfile.txt firstname.lastname@example.org:/home/username
scp myfile.txt email@example.com:/home/username
... and to do the same in reverse, i.e., copy from Palmetto to your local machine. (again, from a terminal running on your local machine, not on Palmetto):
scp firstname.lastname@example.org:/home/username/myfile.txt .
scp email@example.com:/home/username/myfile.txt .
. represents the working directory on the local machine.
For folders, include the
scp -r myfolder firstname.lastname@example.org:/home/username
scp -r myfolder email@example.com:/home/username
Open OnDemand provides a web interface for browsing file systems on Palmetto. You can upload and download files through your web browser without installing any extra software.
To keep the OpenOD web server performant, the maximum upload size is restricted to 50 MB total.
For more information, see the OpenOD Files Dashboard page.