Skip to main content

Cluster Architecture

Understanding the different hardware components of Palmetto and how they are connected or related is critical to making the best use of resources.

Overview​

You may have heard the term "supercomputer" used to describe Palmetto. While not inaccurate, a better description would be that Palmetto is a high-performance computing (HPC) cluster.

The cluster is made up of over a thousand separate computers, called nodes. These nodes are connected through many networks, described as interconnects.

The diagram below shows a high-level view of the cluster.

Compute Nodes​

Most of the nodes in the cluster are compute nodes. These nodes have powerful hardware that can perform fast calculations on large amounts of data.

However, you may have noticed that the diagram has more than just compute nodes. The other nodes in the cluster support the compute nodes.

Special Nodes​

However, a cluster needs more than just compute nodes to be functional. Other special nodes in the cluster provide critical services to keep the cluster running or provide user access to resources.

Login Nodes​

The login nodes are the interface between Palmetto and outside networks. Compute nodes are isolated from the internet, so you must connect to a login node first to gain access to them.

caution

The login node is intended to be used only for scheduling jobs, connecting to compute nodes, and other light maintenance tasks. Performing intensive tasks on the login node may cause it to become unusable for other users.

You can access the login nodes via SSH connections.

Data Transfer Nodes​

The data transfer nodes allow users to move large amounts of data in or out of Palmetto.

You can access the data transfer nodes via SSH file transfer protocol (SFTP) connections.

Scheduler Node​

The scheduler node keeps track of what resources are available on the compute nodes and assigns those resources to people who request them.

You can request time on a compute node by submitting a job to the scheduler. The scheduler will add your job to the queue and determine which nodes are able to run your job and when they will be available.

Interconnects​

Recall that the nodes in the cluster are connected via computer networks. These network connections, called interconnects are frequently classified by their type.

info

A single node can have multiple interconnects.

Each type of interconnect has a certain amount of bandwidth, which is measured in gigabits per second (Gbps). Depending on how much data you are moving between nodes, you will want to select an appropriate interconnect.

Ethernet​

Ethernet is the slowest interconnect, but it is available on every node.

note

Ethernet is the only interconnect available on the oldest nodes in the cluster.

Bandwidth varies depending on the model of network interface card installed in the node. However, the bandwidth will be the same for all nodes in a given phase.

One of the following will be available on every node:

  • 1g - 1 Gbps ethernet
  • 10ge - 10 Gbps ethernet
  • 25ge - 25 Gbps ethernet

FDR Infiniband​

The nodes on phases 7 through 17 have FDR Infiniband interconnects, which have a bandwidth of 56 Gbps. This is also referred to as 56g.

HDR Infiniband​

The newest nodes on phases 18 through 29 have HDR Infiniband interconnects, which have a bandwidth of 100 Gbps. This is also referred to as 100g.