Hoffman2:Data Transfer: Difference between revisions
No edit summary |
(No difference)
|
Revision as of 00:40, 18 April 2012
So you got a whole bunch of data that's sitting on your hard drive or somewhere else, and you need it on hoffman2. How do you get that gigs and gigs of data onto the cluster? Sorry there's no magical genie that instantly does it for you. You're going to have to slowly transfer it through your network on to hoffman. PLEASE NOTE: TRANSFER IS ONLY AS FAST AS YOUR INTERNET CONNECTION!
Here's several ways of pushing that cow up the hill.
scp
General Usage:
scp username@FROM:location username@TO:location
Copy a file from local to Hoffman2
scp FILE username@hoffman2.idre.ucla.edu:PathFromHome/DESTINATION
Copy a file from Hoffman2 to local
scp username@hoffman2.idre.ucla.edu:FILE DESTINATION
Copy a folder from local to Hoffman2
scp -r FOLDER username@hoffman2.idre.ucla.edu:DESTINATION
Copy a folder from Hoffman2 to local
scp -r username@hoffman2.idre.ucla.edu:PathFromHome/FOLDER DESTINATION
sftp
With your favorite GUI (like Cyberduck or Filezilla), you can move data from one location to another.
rsync
Implied in the name, syncs folder/files between filesystems. General Usage:
rsync [OPTION] … SRC [SRC] … [USER@]HOST:DEST
We recommend using something like
rsync -rpltv SRC/FILES/HERE USERNAME@hoffman2.idre.ucla.edu:~
to upload SRC/FILES/HERE from your local machine to your home directory on Hoffman2
Globus Online
Globus Online is a tool that abstracts a lot of complexity from the data transfer process. It is a GUI system that is capable of starting a transfer, having the internet connection broken and re-established, and then continuing and finishing the transfer automatically.
For more information, see the ATS instructions here Globus Online