Hoffman2:Data Transfer: Difference between revisions

From Center for Cognitive Neuroscience
Jump to navigation Jump to search
No edit summary
 
No edit summary
(9 intermediate revisions by 3 users not shown)
Line 1: Line 1:
[[Hoffman2|Back to all things Hoffman2]]
[[Hoffman2|Back to all things Hoffman2]]


So you got a whole bunch of data that's sitting on your hard drive or somewhere else, and you need it on hoffman2. How do you get that gigs and gigs of data onto the cluster? Sorry there's no magical genie that instantly does it for you. You're going to have to slowly transfer it through your network on to hoffman. PLEASE NOTE: TRANSFER IS ONLY AS FAST AS YOUR INTERNET CONNECTION!
So you got a whole bunch of data that's sitting on your hard drive or somewhere else, and you need it on hoffman2. How do you get that gigs and gigs of data onto the cluster? Sorry there's no magical genie that instantly does it for you. You're going to have to slowly transfer it through your network on to Hoffman2. PLEASE NOTE: TRANSFER IS ONLY AS FAST AS YOUR INTERNET CONNECTION!


Here's several ways of pushing that cow up the hill.
Here's several ways of pushing that cow up the hill.
Line 7: Line 7:




==scp==
==Globus Online==
General Usage:
Globus Online is a tool that abstracts a lot of complexity from the data transfer process. It is a GUI system that is capable of starting a transfer, having the internet connection broken and re-established, and then continuing and finishing the transfer automatically.
scp username@FROM:location username@TO:location
 
'''It is the fastest way to transfer data to Hoffman2''' because it has a faster connection than other Hoffman2 nodes.
 
====Create a Globus Account====
First you need to create a free, Globus account (one-time):
*Point your browser at http://www.globus.org and click Sign Up.
*On the Create an Account page, fill in the information (your name, email address, username, password, etc.) and read the terms, then click Register.
You will receive an email with a link which you need to follow to confirm your new Globus account.
 
====General Usage====
After creating an account, you need to setup endpoints to transfer from/to.
*Hoffman2 Endpoint
Hoffman2 Endpoint is already setup on Globus, and you just need to type '''hoffman2#ucla''' into the endpoint field.
*CASS Endpoint
[http://www.cass.idre.ucla.edu/ (Cloud Archival Storage Service)] You Need to Set this up on Globus before you can use it.
[[Hoffman2:Using_Globus#Adding_a_CASS_Endpoint | Directions Here.]]
*Local Desktop Endpoint
If you want to transfer files between your local desktop, you need to download the Globus Desktop Client and set it up on Globus before you can transfer any files.
[[Hoffman2:Using_Globus#Globus_Connect_Software | Directions Here.]]
 
====Transfer Files====
*[[Hoffman2:Using_Globus#Transfer_-_Cluster_to_CASS| Transfer -> Cluster to CASS]]
 
*[[Hoffman2:Using_Globus#Transfer_-_Local_to_Cluster| Transfer -> Local to Cluster]]
 
 
For more information, see the IDRE instructions here
[http://hpc.ucla.edu/hoffman2/file-transfer/gol.php  Globus Online]




Copy a file from local to Hoffman2
scp FILE username@hoffman2.idre.ucla.edu:PathFromHome/DESTINATION


==scp==
A command line tool for copying files using secure encrypted channels.


Copy a file from Hoffman2 to local
====General Usage====
  scp username@hoffman2.idre.ucla.edu:FILE DESTINATION
  scp username@FROM:location username@TO:location


====Hoffman2 Examples====
Copy a file from local computer to Hoffman2
scp /path/to/local/file username@dtn2.hoffman2.idre.ucla.edu:/path/to/destination/for/copy


Copy a folder from local to Hoffman2
Copy a file from Hoffman2 to local computer
  scp -r FOLDER username@hoffman2.idre.ucla.edu:DESTINATION
  scp username@dtn2.hoffman2.idre.ucla.edu:/path/to/file /path/to/destination/for/copy


Copy a directory from local computer to Hoffman2
scp -r /path/to/local/directory username@dtn2.hoffman2.idre.ucla.edu:/path/to/destination/for/copy


Copy a folder from Hoffman2 to local
Copy a directory from Hoffman2 to local computer
  scp -r username@hoffman2.idre.ucla.edu:PathFromHome/FOLDER DESTINATION
  scp -r username@dtn2.hoffman2.idre.ucla.edu:/path/to/directory /path/to/destination/for/copy






==sftp==
==sftp==
With your favorite GUI (like [http://cyberduck.ch/ Cyberduck] or [http://filezilla-project.org/ Filezilla]), you can move data from one location to another.
Another command line tool that uses secure encrypted channels.  There are also GUIs that use this protocol (like [http://cyberduck.ch/ Cyberduck] or [http://filezilla-project.org/ Filezilla]).
 
====General Usage====
Log in to the server, then punch in your password when prompted and you'll be logged in.
sftp USERNAME@SERVERADDRESS
Pull down a file from the server
get /path/to/server/file /path/to/destination/for/copy
Push a file to the server
put /path/to/local/file /path/to/destination/for/copy/on/server
Log out.
bye
 
====Hoffman2 Example====
sftp USERNAME@dtn2.hoffman2.idre.ucla.edu
get /server/file /local/file
put /local/file /server/file
bye






==rsync==
==rsync==
Implied in the name, syncs folder/files between filesystems. General Usage:
Implied in the name, syncs folder/files between filesystems. It also makes use of secure encrypted channels.
 
====General Usage====
  rsync [OPTION] … SRC [SRC] … [USER@]HOST:DEST
  rsync [OPTION] … SRC [SRC] … [USER@]HOST:DEST


====Hoffman2 Example====
We recommend using something like
We recommend using something like
  rsync -rpltv SRC/FILES/HERE USERNAME@hoffman2.idre.ucla.edu:~
  rsync -av /PATH/TO/SRC/FILES/HERE USERNAME@hoffman2.idre.ucla.edu:~
to upload ''SRC/FILES/HERE'' from your local machine to your home directory on Hoffman2
to upload ''/PATH/TO/SRC/FILES/HERE'' from your local machine to your home directory on Hoffman2
 
 
 
==Globus Online==
Globus Online is a tool that abstracts a lot of complexity from the data transfer process. It is a GUI system that is capable of starting a transfer, having the internet connection broken and re-established, and then continuing and finishing the transfer automatically.
 
For more information, see the ATS instructions here
[http://www.ats.ucla.edu/clusters/hoffman2/gol.htm  Globus Online]





Revision as of 16:50, 10 October 2019

Back to all things Hoffman2

So you got a whole bunch of data that's sitting on your hard drive or somewhere else, and you need it on hoffman2. How do you get that gigs and gigs of data onto the cluster? Sorry there's no magical genie that instantly does it for you. You're going to have to slowly transfer it through your network on to Hoffman2. PLEASE NOTE: TRANSFER IS ONLY AS FAST AS YOUR INTERNET CONNECTION!

Here's several ways of pushing that cow up the hill.


Globus Online

Globus Online is a tool that abstracts a lot of complexity from the data transfer process. It is a GUI system that is capable of starting a transfer, having the internet connection broken and re-established, and then continuing and finishing the transfer automatically.

It is the fastest way to transfer data to Hoffman2 because it has a faster connection than other Hoffman2 nodes.

Create a Globus Account

First you need to create a free, Globus account (one-time):

  • Point your browser at http://www.globus.org and click Sign Up.
  • On the Create an Account page, fill in the information (your name, email address, username, password, etc.) and read the terms, then click Register.

You will receive an email with a link which you need to follow to confirm your new Globus account.

General Usage

After creating an account, you need to setup endpoints to transfer from/to.

  • Hoffman2 Endpoint

Hoffman2 Endpoint is already setup on Globus, and you just need to type hoffman2#ucla into the endpoint field.

  • CASS Endpoint

(Cloud Archival Storage Service) You Need to Set this up on Globus before you can use it. Directions Here.

  • Local Desktop Endpoint

If you want to transfer files between your local desktop, you need to download the Globus Desktop Client and set it up on Globus before you can transfer any files. Directions Here.

Transfer Files


For more information, see the IDRE instructions here Globus Online


scp

A command line tool for copying files using secure encrypted channels.

General Usage

scp username@FROM:location username@TO:location

Hoffman2 Examples

Copy a file from local computer to Hoffman2

scp /path/to/local/file username@dtn2.hoffman2.idre.ucla.edu:/path/to/destination/for/copy

Copy a file from Hoffman2 to local computer

scp username@dtn2.hoffman2.idre.ucla.edu:/path/to/file /path/to/destination/for/copy

Copy a directory from local computer to Hoffman2

scp -r /path/to/local/directory username@dtn2.hoffman2.idre.ucla.edu:/path/to/destination/for/copy

Copy a directory from Hoffman2 to local computer

scp -r username@dtn2.hoffman2.idre.ucla.edu:/path/to/directory /path/to/destination/for/copy


sftp

Another command line tool that uses secure encrypted channels. There are also GUIs that use this protocol (like Cyberduck or Filezilla).

General Usage

Log in to the server, then punch in your password when prompted and you'll be logged in.

sftp USERNAME@SERVERADDRESS

Pull down a file from the server

get /path/to/server/file /path/to/destination/for/copy

Push a file to the server

put /path/to/local/file /path/to/destination/for/copy/on/server

Log out.

bye

Hoffman2 Example

sftp USERNAME@dtn2.hoffman2.idre.ucla.edu
get /server/file /local/file
put /local/file /server/file
bye


rsync

Implied in the name, syncs folder/files between filesystems. It also makes use of secure encrypted channels.

General Usage

rsync [OPTION] … SRC [SRC] … [USER@]HOST:DEST

Hoffman2 Example

We recommend using something like

rsync -av /PATH/TO/SRC/FILES/HERE USERNAME@hoffman2.idre.ucla.edu:~

to upload /PATH/TO/SRC/FILES/HERE from your local machine to your home directory on Hoffman2


External Links