Hoffman2:Data Transfer

Back to all things Hoffman2

So you got a whole bunch of data that's sitting on your hard drive or somewhere else, and you need it on hoffman2. How do you get that gigs and gigs of data onto the cluster? Sorry there's no magical genie that instantly does it for you. You're going to have to slowly transfer it through your network on to Hoffman2. PLEASE NOTE: TRANSFER IS ONLY AS FAST AS YOUR INTERNET CONNECTION!

Here's several ways of pushing that cow up the hill.

Globus Online

Globus Online is a tool that abstracts a lot of complexity from the data transfer process. It is a GUI system that is capable of starting a transfer, having the internet connection broken and re-established, and then continuing and finishing the transfer automatically.

It is the fastest way to transfer data to Hoffman2 because it has a faster connection than other Hoffman2 nodes.

For more information, see the ATS instructions here Globus Online

scp

A command line tool for copying files using secure encrypted channels.

General Usage

scp username@FROM:location username@TO:location

Hoffman2 Examples

Copy a file from local computer to Hoffman2

scp /path/to/local/file username@dtn2.hoffman2.idre.ucla.edu:/path/to/destination/for/copy

Copy a file from Hoffman2 to local computer

scp username@dtn2.hoffman2.idre.ucla.edu:/path/to/file /path/to/destination/for/copy

Copy a directory from local computer to Hoffman2

scp -r /path/to/local/directory username@dtn2.hoffman2.idre.ucla.edu:/path/to/destination/for/copy

Copy a directory from Hoffman2 to local computer

scp -r username@dtn2.hoffman2.idre.ucla.edu:/path/to/directory /path/to/destination/for/copy

sftp

Another command line tool that uses secure encrypted channels. There are also GUIs that use this protocol (like Cyberduck or Filezilla).

General Usage

Log in to the server, then punch in your password when prompted and you'll be logged in.

sftp USERNAME@SERVERADDRESS

Pull down a file from the server

get /path/to/server/file /path/to/destination/for/copy

Push a file to the server

put /path/to/local/file /path/to/destination/for/copy/on/server

Log out.

bye

Hoffman2 Example

sftp USERNAME@dtn2.hoffman2.idre.ucla.edu
get /server/file /local/file
put /local/file /server/file
bye

rsync

Implied in the name, syncs folder/files between filesystems. It also makes use of secure encrypted channels.

General Usage

rsync [OPTION] … SRC [SRC] … [USER@]HOST:DEST

Hoffman2 Example

We recommend using something like

rsync -av /PATH/TO/SRC/FILES/HERE USERNAME@dtn2.hoffman2.idre.ucla.edu:~

to upload /PATH/TO/SRC/FILES/HERE from your local machine to your home directory on Hoffman2

External Links

• Man page for scp
• Man page for sftp
• Man page for rsync
• Globus Online