Data Transfer

There are many ways to move data in and out of Palmetto for your research. Selecting the correct method is important, since using the wrong method might impact the performance of shared resources for other users.

Which method to use?

Globus and SFTP via the data transfer nodes (DTNs) are the recommended methods to transfer data between filesystems, as well as into and out of the cluster. The DTNs contain the fastest connections to all storage systems and are dedicated resources for transferring data.

Below are the supported methods to transfer data into Palmetto.

danger

Excessive SFTP data transfer activity on the login and Open OnDemand resources may make these shared systems slow or unusable for other users. If you exceed the limits below, system administrators may terminate your data transfer without warning to protect these shared resources.

Method	Maximum Transfer Size
Globus	All Transfers
SFTP (Data Transfer Nodes)	All Transfers
SFTP (login nodes)	Small files (< 100 MB total)
Open OnDemand	Very small files (< 50 MB total)
rclone	All Transfers

Globus

For larger files, we recommend using the Globus file transfer application. Here, we demonstrate how to use Globus Online to transfer files between Palmetto and a local machine (laptop). However, Globus can be used for file transfers to/from other locations as well.

Globus Accounts

You must have a Globus account to use their services.

If you do not already have one, you can visit the Globus app to create an account.

tip

Select Clemson University on the organizational login screen to take advantage of single sign on.

Globus Collections

Globus works by transferring files between collections. The data transfer nodes provide a collection for data stored on Palmetto. The collection is named Clemson RCD Data Transfer.

Search for Clemson RCD to locate the collection.

Installing Globus Connect Personal

To transfer files between your local computer and Palmetto, you must install the Globus Connect Personal utility and create a collection for your local file system. Below is a list of instructions by platform for installing the utility:

• How to Install Globus Connect Personal for Mac OS X
• How to Install Globus Connect Personal for Windows
• How to Install Globus Connect Personal for Linux

Transferring Data with Globus

Prerequisites

Before you proceed, make sure that you have already created a Globus account and installed Globus Connect Personal.

1. To begin transferring files, navigate to the Globus online transfer utility.

2. At the top left of the File Manager, search for the collection name that you selected for your local computer.

3. To the right, search for the collection name for one of the Palmetto data transfer nodes.

   note

   You can copy and paste the collection names for the Palmetto data transfer nodes from the collections section above.

4. Click the Transfer or Sync button in the center of the screen to begin your transfer session.

5. You can now use the Globus interface to transfer files between your local computer and the Palmetto cluster.

Video demonstration of Globus transfer on macOS

SFTP

Transfers can be completed via SFTP through your SSH client by connecting to one of Palmetto's dedicated data transfer nodes. These nodes are set up to ensure data transfer activity does not impact other users.

• Data Transfer Node 1 - hpcdtn01.rcd.clemson.edu
• Data Transfer Node 2 - hpcdtn02.rcd.clemson.edu

If one data transfer node is unavailable, try using the other one.

For transfers smaller than 100 MB total, you may transfer them via the login nodes.

SFTP on Windows

On Windows machines, using the MobaXterm SSH client, the built-in file browser can be used (SCP tab of the side window). Using the Upload (green arrow pointing up) and and Download (blue arrow pointing down) buttons at the top of the SCP tab, you can easily transfer small files between Palmetto and your local computer.

Video demonstration of MobaXTerm file transfers

SFTP on macOS/Linux

On Unix systems, you can use the scp (secure copy) command to perform file transfers. The general form of the scp command is:

scp <path_to_source> username@hpcdtn01.rcd.clemson.edu:<path_to_destination>
scp <path_to_source> username@hpcdtn02.rcd.clemson.edu:<path_to_destination>

For example, here is the scp command to copy a file from the current directory on your local machine to your /home/username directory on Palmetto (this command is entered into a terminal when not logged-in to Palmetto):

scp myfile.txt username@hpcdtn01.rcd.clemson.edu:/home/username
scp myfile.txt username@hpcdtn02.rcd.clemson.edu:/home/username

... and to do the same in reverse, i.e., copy from Palmetto to your local machine. (again, from a terminal running on your local machine, not on Palmetto):

scp username@hpcdtn01.rcd.clemson.edu:/home/username/myfile.txt .
scp username@hpcdtn02.rcd.clemson.edu:/home/username/myfile.txt .

The . represents the working directory on the local machine.

For folders, include the -r switch:

scp -r myfolder username@hpcdtn01.rcd.clemson.edu:/home/username
scp -r myfolder username@hpcdtn02.rcd.clemson.edu:/home/username

Open OnDemand

Open OnDemand provides a web interface for browsing file systems on Palmetto. You can upload and download files through your web browser without installing any extra software.

To keep the OpenOD web server performant, the maximum upload size is restricted to 50 MB total.

For more information, see the OpenOD Files Dashboard page.