Hoffman2:Data Transfer: Difference between revisions
No edit summary |
|||
Line 18: | Line 18: | ||
You will receive an email with a link which you need to follow to confirm your new Globus account. | You will receive an email with a link which you need to follow to confirm your new Globus account. | ||
For more information, see the IDRE instructions here | For more information, see the IDRE instructions here |
Revision as of 05:22, 25 May 2014
So you got a whole bunch of data that's sitting on your hard drive or somewhere else, and you need it on hoffman2. How do you get that gigs and gigs of data onto the cluster? Sorry there's no magical genie that instantly does it for you. You're going to have to slowly transfer it through your network on to Hoffman2. PLEASE NOTE: TRANSFER IS ONLY AS FAST AS YOUR INTERNET CONNECTION!
Here's several ways of pushing that cow up the hill.
Globus Online
Globus Online is a tool that abstracts a lot of complexity from the data transfer process. It is a GUI system that is capable of starting a transfer, having the internet connection broken and re-established, and then continuing and finishing the transfer automatically.
It is the fastest way to transfer data to Hoffman2 because it has a faster connection than other Hoffman2 nodes.
Create a Globus Account
First you need to create a free, Globus account (one-time):
- Point your browser at http://www.globus.org and click Sign Up.
- On the Create an Account page, fill in the information (your name, email address, username, password, etc.) and read the terms, then click Register.
You will receive an email with a link which you need to follow to confirm your new Globus account.
For more information, see the IDRE instructions here Globus Online
General Usage=
scp
A command line tool for copying files using secure encrypted channels.
General Usage
scp username@FROM:location username@TO:location
Hoffman2 Examples
Copy a file from local computer to Hoffman2
scp /path/to/local/file username@dtn2.hoffman2.idre.ucla.edu:/path/to/destination/for/copy
Copy a file from Hoffman2 to local computer
scp username@dtn2.hoffman2.idre.ucla.edu:/path/to/file /path/to/destination/for/copy
Copy a directory from local computer to Hoffman2
scp -r /path/to/local/directory username@dtn2.hoffman2.idre.ucla.edu:/path/to/destination/for/copy
Copy a directory from Hoffman2 to local computer
scp -r username@dtn2.hoffman2.idre.ucla.edu:/path/to/directory /path/to/destination/for/copy
sftp
Another command line tool that uses secure encrypted channels. There are also GUIs that use this protocol (like Cyberduck or Filezilla).
General Usage
Log in to the server, then punch in your password when prompted and you'll be logged in.
sftp USERNAME@SERVERADDRESS
Pull down a file from the server
get /path/to/server/file /path/to/destination/for/copy
Push a file to the server
put /path/to/local/file /path/to/destination/for/copy/on/server
Log out.
bye
Hoffman2 Example
sftp USERNAME@dtn2.hoffman2.idre.ucla.edu get /server/file /local/file put /local/file /server/file bye
rsync
Implied in the name, syncs folder/files between filesystems. It also makes use of secure encrypted channels.
General Usage
rsync [OPTION] … SRC [SRC] … [USER@]HOST:DEST
Hoffman2 Example
We recommend using something like
rsync -av /PATH/TO/SRC/FILES/HERE USERNAME@dtn2.hoffman2.idre.ucla.edu:~
to upload /PATH/TO/SRC/FILES/HERE from your local machine to your home directory on Hoffman2