Data Transfer

User Information Home
Compute Resources
Software
Data
Security
Allocations
Consulting
Training
Strategic Applications Program

Table of Contents

1. Data Transfer Overview
   Data Transfer Protocols
   Data Transfer Clients
   NCSA-TeraGrid Data Transfer Resources
   NCSA Mass Storage System Transfers
   Data Transfer Software Installation
   Transfer Performance Considerations
   Data Transfer Examples

NCSA Mass Storage System Transfers

Connectivity of each cluster into Mass Storage System (MSS) varies. In general, multiple transfer streams will achieve the best aggregate transfer rates. To maximize the efficiency of each individual transfer, use the following guidelines when writing to MSS. (See also, MSS Transfer Examples.)

• Use msscmd or mssftp or uberftp.
• Use "active" mode.
• Avoid globus-url-copy.

When you retrieve data from MSS, be mindful of the archive retrieval process. A "get" command will request the staging of a file, if it resides on the tape archive, then return without retrieving the file. A subsequent "get" command must be issued after tape retrieval is completed.

NCSA HPC System Transfers to Remote Locations

Moving data effectively to/from an NCSA system from/to an external system can complicate factors outside of NCSA's control. Data transfer methods that work well within the NCSA environment may not be feasible at an external site. Below are guidelines and recommendations for establishing connectivity and, hopefully, good data throughput to and from NCSA's resources.

Connecting to a Remote Location

So-called "clear-text passwords" are unencrypted and unscrambled and thus vulnerable to detection; therefore, NCSA has eliminated all clear-text passswords. NCSA requires that connections made to its high-performance systems be made using encryption software — SSH, Kerberos-enabled Telnet or GSI-based clients — which eliminates clear-text logins and passwords.

Enabling Password-less Login via Kerberos

To enable password-less login via Kerberos, follow these steps:

1. Verify that your system uses Kerberos-enabled SSH, FTP and has Kerberos installed. Most newer Linux distributions come with Kerberos and Kerberos-enabled SSH, including Fedora Core 3, or newer.
2. Obtain the NCSA Kerberos install package (see, Software: Kerberos and Secure Shell). For systems without Kerberos already installed, install as directed. For Linux systems with Kerberos already installed, simply replace /etc/krb5.conf with the NCSA version.
3. To enable password-less SSH logins to NCSA resources, edit /etc/ssh/ssh_config or create a ~/.ssh/config file that contains the following lines:

           Host *
                   GSSAPIAuthentication yes
           

For more information about login requirements, see the "Encryption Software" section on the NCSA Allocations and Access to NCSA HPC Systems page.

High-Performance SSH/SCP - HPN-SSH

All NCSA systems run the OpenSSH server with the High Performanc Network (HPN) patches to allow high-speed transfers over long network connections. Client and server software must be installed and patched in order to access the full feature set of HPN-SSH. This patch is also available in the GSI-OpenSSH Globus package.

Delegating Grid Credentials to a Remote Workstation

To use GSI grid authentication from a remote workstation or non-TeraGrid cluster, you must have the Globus Toolkit (or at least a subset therein) installed. Grid credentials can then be passed to the remote client machine by using an existing TeraGrid- or NCSA-accepted X.509 certificate as the initial proxy.

Valid proxies can be issued and stored on a TeraGrid or NCSA MyProxy server and then delegated to a remote system. Refer to NCSA's MyProxy Server page for instructions on configuring a local installation of MyProxy to connect to the NCSA server. Once a valid proxy certificate exists on a correctly configured host, GSI authentication tools will automatically connect to hosts for which the user has been granted access.

Note: MyProxy is included in the Globus Toolkit.

For more information, see Getting an NCSA Certificate and Accessing Machines on the Grid.

Remote Access of MSS

MSS access is only availiable through FTP. The FTP client used to access MSS must be capable of either Kerberos or GSI authentication.