Table of Contents
- Data Transfer Overview
Data Transfer Protocols
Data Transfer Clients
NCSA-TeraGrid Data Transfer Resources
NCSA Mass Storage System Transfers
Data Transfer Software Installation
Transfer Performance Considerations
Data Transfer Examples
Data Transfer Software Installation
When attempting to transfer data to or from a high-performance computing
center, such as NCSA, it is sometimes advisable to use the same software
tools that are installed at the center on the external source or destination
computer. We recommend that you install HPN-SSH/SCP and the data transfer
components of the Globus Toolkit. With these tools in place at an external
site, achieving the best possible performance for the given transfer
path is much easier.
Installing HPN-SSH/SCP
A Unix/Linux SCP client patched with the HPN enhancements can be installed
on a remote system in order to facilitate good transfer rates to and
from HPN-enabled servers at NCSA or at other Grid sites. These clients
can either be built from the OpenSSH
source or clients can be obtained from the Globus Toolkit or Virtual
Data Toolkit.
Installing Data Components of the Globus Toolkit
These instructions provide information for installing UberFTP (a full-featured,
grid-enabled FTP client), globus-url-copy (a command line copy tool),
and gsiscp (an OpenSSH patched with Grid, GSI, authentication and HPN-SSH
capability). See Available Transfer Clients for
more information about these and other data transfer clients installed
on NCSA systems.
Note: A full installation of Globus will
contain gsiscp and globus-url-copy. If you
already have a working Globus installation, skip to Step 2 to obtain
UberFTP.
If using the example installation below as a reference, you should
update versions as needed. It is recommended that you always install
the latest incremental release of the Globus Toolkit components.
Installation Procedure
- Install MyProxy and GSISSH
http://grid.ncsa.uiuc.edu/myproxy/tgsso.html#install
- Install UberFTP using the GPT that was set up automatically in the
MyProxy and GSISSH install above:
http://dims.ncsa.uiuc.edu/set/uberftp/install.html
Example Installation
Below are the exact commands used to install on a Fedora Core 3 box,
using a non-root user with a bash shell. Installation location can be
set to $HOME/globus. (--prefix=$HOME/globus GLOBUS_LOCATION=$HOME/globus
GPT_LOCATION=$HOME/globus)
Installation Space: 307 MB (313504 KB)
Installation Directory: Can be wherever the user has
write permissions. (For this example, we gave our non-root user write
permissions to /usr/local/GridClient/globus.)
Commands
mkdir grid-client-software
cd grid-client-software
wget http://www-unix.globus.org/ftppub/gt4/4.0/4.0.3/installers/ \
bin/gt4.0.3-x86_fc_3-installer.tar.gz
wget http://dims.ncsa.uiuc.edu/set/uberftp/download/ \
uberftp-client-1.22.tar.gz
tar xzf gt4*-installer.tar.gz
cd gt4*-installer
./configure --prefix=/usr/local/GridClient/globus
make gsi-myproxy gsi-openssh gridftp
make install
GLOBUS_LOCATION=/usr/local/GridClient/globus
GPT_LOCATION=/usr/local/GridClient/globus
MYPROXY_SERVER=myproxy.ncsa.uiuc.edu
MYPROXY_SERVER_PORT=7512
export GLOBUS_LOCATION GPT_LOCATION MYPROXY_SERVER MYPROXY_SERVER_PORT
. $GLOBUS_LOCATION/etc/globus-user-env.sh
cd ../
gpt-build -verbose uberftp-client-1.22.tar.gz gcc32
mkdir -p ~/.globus/certificates
cd ~/.globus/certificates
wget http://ca.ncsa.uiuc.edu/4a6cd8b1.0
wget http://ca.ncsa.uiuc.edu/4a6cd8b1.signing_policy
### # End # ###
TCSH shell users
setenv GLOBUS_LOCATION /usr/local/GridClient/globus
setenv GPT_LOCATION /usr/local/GridClient/globus
setenv MYPROXY_SERVER myproxy.ncsa.uiuc.edu
setenv MYPROXY_SERVER_PORT 7512
source $GLOBUS_LOCATION/etc/globus-user-env.csh
=== TEST INSTALLATION ===
$ myproxy-logon -T -l jdoe
Enter MyProxy pass phrase:
A credential has been received for user jdoe in /tmp/x509up_u20707.
Trust roots have been installed in /home/jdoe/.globus/certificates/.
$ grid-proxy-info
subject : /C=US/O=National Center for Supercomputing Applications/ \
CN=John Doe
issuer : /C=US/O=National Center for Supercomputing Applications/ \
CN=Certification Authority
identity : /C=US/O=National Center for Supercomputing Applications/ \
CN=John Doe
type : end entity credential
strength : 1024 bits
path : /tmp/x509up_u20707
timeleft : 11:59:52
$ uberftp mss.ncsa.teragrid.org
220-
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
***************************************************************
NCSA's Mass Storage System
Running EMC's DiskXtender for Unix 2.9
for IRIX on a SGI Origin 3900
For information on using NCSA's archive storage system see:
http://www.ncsa.uiuc.edu/UserInfo/Resources/Hardware/UniTree/
***************************************************************
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
UNIX Archive FTP server (DiskXtender Version 2.9) active. Checking \
DiskXtender.conf
220 UNIX Archive FTP server ready.
230 User jdoe logged in.
uberftp>
