Overview
There are many ways to move data in and out of Palmetto for your research. Selecting the correct method is important, since using the wrong method might impact the performance of shared resources for other users.
Which method to use?
Globus and SFTP via the data transfer nodes (DTNs) are the recommended methods to transfer data between filesystems, as well as into and out of the cluster. The DTNs contain the fastest connections to all storage systems and are dedicated resources for transferring data.
Below are the supported methods to transfer data into Palmetto.
Excessive SFTP data transfer activity on the login and Open OnDemand resources may make these shared systems slow or unusable for other users. If you exceed the limits below, system administrators may terminate your data transfer without warning to protect these shared resources.
| Method | Maximum Transfer Size |
|---|---|
| Globus | All Transfers |
| SFTP (Data Transfer Nodes) | All Transfers |
| SFTP (login nodes) | Small files (< 100 MB total) |
| Open OnDemand | Very small files (< 50 MB total) |
| rclone | All Transfers |
Globus
For larger files, we recommend using the Globus file transfer application. Globus provides a secure, unified interface for research data and gracefully handles transfers of terabytes of data.
In addition to transfers from your personal computer, Globus can also:
- transfer data to and from Box
- securely share your data on Indigo with external users
SFTP
Transfers can be completed via SFTP through your SSH client by connecting to one of Palmetto's dedicated data transfer nodes. These nodes are set up to ensure data transfer activity does not impact other users.
Open OnDemand
Open OnDemand provides a web interface for browsing file systems on Palmetto. You can upload and download files through your web browser without installing any extra software.
To keep the OpenOD web server performant, the maximum upload size is restricted to 50 MB total.
For more information, see the OpenOD Files Dashboard page.
Rclone
Rclone is a command line tool for transferring files to and from cloud storage products (over 70 cloud storage providers are supported). We have several examples of using Rclone from the Data Transfer Node to move files from Google Drive, Microsoft OneDrive, and Box.