Data Transfer Services

 

Data Transfers for Small Data Sets

In some scenarios, a research workflow requires the movement of a few small input files to a computational cluster and/or a few small output files back to a local desktop for storage, final reporting and/or analysis.   For this case, CHPC suggests four options:

• Open OnDemand: Graphical user interface that makes transfering small data sets from your local computer to your chpc home directory and vice versa easy.
• WinSCP (Windows), CyberDuck (Macs): The use of simple graphical tools to move data to/from a local machine.  In these cases you can "drag and drop" files from one system to the other.
• The use of Linux commands and tools (scp, rsnyc over ssh, sftp, wget, curl are examples) on CHPC interactive nodes or on the specialized Data Transfer Nodes (DTNs).  Note the DTN usage is described further down on this page. Detailed information of the use of these commands can readily be obtained either from the main pages links or searching 'DTNs' in our search bar.
• CurlWget: To get a download link for data on a website with no direct link, e.g. for Dropbox or Sharefile.

Data Transfers for Large Data Sets

For science workflows that transfer very large files and/or very large data sets between national labs, industry partners, or peer institutions, users require more advanced parallel transfer tools running on tuned endpoint devices such as Data Transfer Nodes (DTNs).  CHPC supports various parallel transfer tools that support the known science workflows at the University of Utah.  If a research group requires the support of an additional transfer tool, the group may request it through helpdesk@chpc.utah.edu.

Data Transfer Nodes

Campus network traffic usually routes through a firewall, sufficient for small transfers. However, research computing requires high bandwidth and connections, overwhelming the firewall and impacting general campus use. To address this, a Science DMZ was created, a separate network with optimized performance and security. CHPC provides Data Transfer Nodes (DTNs) utilizing this DMZ for efficient research data transfers.

For more information visit "What makes a Data Transfer Node?".

 DTNs can be used with standard cli or gui data transfer tools (e.g., rsync, wget, WinSCP, etc.) to move data to and from CHPC resources. These nodes are also set up to leverage all the parallel transfer tools that CHPC supports. All CHPC DTNs are also available as Globus Endpoint Clusters, which allows for concurrent and fault-tolerant transfer of large data sets with lots of files (see Globus section below for CHPC endpoints). DTN's could also be used to transfer data to and from the cloud. 

The CHPC also supports additional specialized  tools for moving data to/from cloud storage.  Some of these tools are specific to a single cloud storage provider (such as s3cmd for Amazon cloud services), whereas others, such as rclone, work with different cloud storage providers.

The CHPC also supports a number of group-owned DTNs.  If you need any information about an existing group owned DTN or are interested in having a DTN dedicated for your group, please contact us through helpdesk@chpc.utah.edu.

General Environment DTNs

Data transfer nodes in the General Environment are split into two groups, internal DTNs (intdtn) and external DTNs. Internal DTNs are connected at 40 Gbps inside the campus firewall and are best for transfers inside campus (e.g., from campus to CHPC). External DTNs (dtn) are connected at 100 Gbps to the Science DMZ and are suited for transfers external to campus (e.g., downloading datasets from the internet, transferring from external locations/institutions) and large, multi-terabyte transfers.

The DTNs can also be utilized in jobs via Slurm, both in the General Environment on notchpeak and the Protected Environment on redwood. See Data Transfer Node Access via Slurm for details.

CHPC General Environment DTNs:

• intdtn01.chpc.utah.edu
• intdtn02.chpc.utah.edu
  
• intdtn03.chpc.utah.edu
• intdtn04.chpc.utah.edu
• dtn05.chpc.utah.edu
• dtn06.chpc.utah.edu
  
• dtn07.chpc.utah.edu
• dtn08.chpc.utah.edu

Protected Environment DTNs

The CHPC’s DTNs in the PE are connected at 40 Gbps and require the University of Utah VPN to connect from off campus. The PE DTNs have the same data transfer tools as the General Environment but have more restrictive security settings for remote connections. Tools such as screen and tmux can be used to keep transfers on the console running in the background even if an SSH connection is terminated.

CHPC Protected Environment DTNs:

• pe-dtn03.chpc.utah.edu
• pe-dtn04.chpc.utah.edu

pe-dtn.chpc.utah.edu can be used to connect to the servers with round-robin balancing.

Parallel Transfer Tools

There are a number of options available to use for large scale data transfer.  In the following we list ones that we have installed on CHPC resources. See the application database for more information on CHPC installations of these tools.  These can be used on the Data Transfer Nodes mentioned in the above section (recommended for any large file transfers), or from the other CHPC maintained resources that have access to the CHPC applications. 

Globus (recommended): The Globus service is a set of tools developed to facilitate parallel, load-balanced, fault tolerant data transfers. There are a few steps involved in getting set up to use this service at CHPC, which are detailed on the Globus Software Documentation page.  For further information please visit: https://www.globus.org/quickstart.

Rclone (recommended):  Rclone is a command-line program that supports file transfers and syncing of files between local storage and Google Drive as well as a number of other storage services, including Dropbox, OneDrive, and Swift/S3-based services. Rclone offers options to optimize a transfer and reach higher transfer speeds than other common transfer tools such as scp and rsync. For more information on using rclone as well as configuring it for both transferring data to google/onedrive drive storage and to the CHPC archive storage, see the CHPC Rclone Software Documentation page.

Fpsync:  Fpsync, part of the fpart package, is a shell script that wraps fpart and rsync to launch multiple rsyncs in parallel

            To use:

             module load fpsync

             fpsync -h 

Aspera:  Aspera is IBM's high-performance file transfer software which allows for the transfer of large files and data sets with predictable, reliable and secure delivery regardless of file size or transfer distance from a location that has the aspera transfer server running.  The NCBI recommends the use of Aspera for transfer of data sets from their site.  The command line client is available by loading the ascp module.

To use the command-line client:

   module load aspera

   ascp --help

See the CHPC Aspera software page for additional information.

BBCP: BBCP is a point-to-point network file copy application developed at SLAC National Accelerator Laboratory that allows for sending data in multiple simultaneous streams, developed for the BABAR as a particle physics experiment. Complete documentation on bbcp usage can be found at BBCP website.

To use: /uufs/chpc.utah.edu/sys/pkg/bbcp/std/bin/bbcp --help  or visit the BBCP site.

FDT: FDT, Fast Data Transport, is a Java based file transfer application. CHPC has an installation that is located at /uufs/chpc.utah.edu/sys/pkg/fdt/0.9.20.  FDT can be used in three modes: server, client and SCP.

•  In Servermode the FDT will start listening for incoming client connections. The server may or may not stop after the last client finishes the transfer.
• In Client mode the client will connect to the specified host, where an FDT Server is expected to be running. The client can either read or write files from/to the server.
• In the SCP (Secure Copy) mode the local FDT instance will use SSH to start/stop the FDT server and/or client.  The security is based on ssh credentials. The server started in this mode will accept connections ONLY from the "SCP" client.
• To use: java -jar /uufs/chpc.utah.edu/sys/pkg/fdt/0.9.20/fdt.jar --help or visit the FDT site.

Direct Mounts of CHPC File Systems

The CHPC doesn't offer direct mounts of PE file servers outside of the PE. We recommend that users consider Globus, SFTP (FileZilla), SCP (WinSCP), rclone, and the PE Data Transfer Nodes.

For usage situations where you do not need a second copy on a local machine, but instead only need to access a file, you can do so by having the CHPC file system in which the file is located mounted on your local machine. CHPC allows mounts of home directories as well as group owned file systems on local machines. In addition we allow mounts of other group owned file systems; however, we do not allow mounts of the general CHPC scratch file systems such as /scratch/general/nfs1 and /scratch/general/vast. If the local machine is off campus, you must access via the University of Utah VPN. Below is information on how to do the mounts for home and group spaces on Windows, Mac and Linux systems. Note that there is also a short training video that covers this topic. In all of the following you must replace <uNID>  with your unid. 

 

For home directories:  The information needed to mount your home directory space exists in the Account Creation Notification email sent to new users. Alternately, you can also get this information when on a CHPC resource, by issuing a df or df | grep <uNID> command. As an example, if the df command returns  eth.vast.chpc.utah.edu:/home/XX/<uNID>, then using the value of XX and your unid you would map the network drive using the following paths (NOTE - when authenticating, your username is ad\<uNID>):

On Windows: \\samba.chpc.utah.edu\XX-home\<uNID> On a Mac:  smb://samba.chpc.utah.edu/XX-home/<uNID>   On Linux: If you have root, you can cifs mount CHPC file spaces by (you will be prompted for your password): mount.cifs //samba.chpc.utah.edu/XX-home/<uNID>  /mnt -o domain=AD,user=<uNID>

For group directories:  When the PI obtains group space, CHPC provides the name of the space along with the information on mounting and using the group spaces.  (NOTE - when authenticating, your username is ad\<uNID>):

On Windows: \\samba.chpc.utah.edu\<name-groupnumber> On a MAC:  smb://samba.chpc.utah.edu/<name-groupnumber> On Linux: If you have root, you can cifs mount  CHPC  file spaces by (you will be prompted for your password): mount.cifs //samba.chpc.utah.edu/<name-groupnumber>  /mnt -o domain=AD,user=<uNID>

 

Temporary Guest Transfer Service

CHPC provides a mechanism for our users to transfer files to and/or from individuals without CHPC accounts. This service is called guest-transfer.

What is it for?

• At times, CHPC users need to transfer files to or from individuals that don't have CHPC accounts. These files are often quite large (many gigabytes), and thus unsuitable for other transport mechanisms (email, DVD).
• These file transfers often need to happen with little or no warning. They may also need to occur outside CHPC's support hours. Thus, the guest-transfer service must function without intervention or assistance from CHPC staff.

What is it not for?

• The guest transfer service is not for repeated events.
• The guest transfer service is not for long-term data storage.
• The guest transfer service is not for restricted (PHI/HIPAA/HITECH/FERPA) or sensitive data.
• The guest transfer service is not for huge data transfers (it's currently restricted to approximately 5 terabytes).

How do I get a guest account?

• When you need to use the guest transfer service, visit  https://guest-transfer.chpc.utah.edu/ and fill out the form. This form creates a guest transfer account. You then give the guest account username and password to your colleague. You and your colleague can now share files.

How do I use the service?

• Once you have created a guest-transfer account and given it to your colleague, you and your colleague can copy your files to guest-transfer.chpc.utah.edu with your scp/sftp client (scp, sftp, WinSCP, etc). Files should be transferred to /scratch on guest-transfer.chpc.utah.edu.
  
• The CHPC user can then transfer the files from /scratch on guest-transfer.chpc.utah.edu to other CHPC resources.

Things to remember:

• The process is completely automatic. You fill out the form, it immediately gives you a guest account.
• Only CHPC users can create accounts.
• The person who creates the guest account is responsible for all activity of the guest account.
• This guest account is only usable for the guest transfer service. It provides no access to any other CHPC or University resources.
• Files are transferred via scp/sftp. Interactive logins are disabled.
• Files are automatically removed after 90 days (based on last-access time).
• Files in the guest-transfer service can be read or changed by any other user.
• Consider using encryption to protect and verify your files.
• DO NOT USE THIS SERVICE FOR SENSITIVE OR RESTRICTED DATA!