Current Events - Revision history

Dmargolis: Blanked the page

2017-05-26T20:13:28Z

Blanked the page

← Older revision		Revision as of 20:13, 26 May 2017
Line 1:		Line 1:
	~~=Miles Decommissioned=~~
	~~Due to a change in ATS policy, the miles cluster is scheduled for retirement at the end of July along with many other small to mid-sized clusters at UCLA.~~

	~~==What does this mean to me as a researcher on miles?==~~
	This means that all on going projects need to be migrated to Hoffman or onto your local lab computers no later than August 1st. Though I strongly recommend looking at the [http://www.ats.ucla.edu/clusters/hoffman2 Hoffman2 Cluster]. It's an amazing resource and asset to the UCLA research community.

	~~==How do I get my data onto Hoffman?==~~
	~~<b><font color='red'>WARNING! this method gladly overwrites any existing data of the same name in the destination directory.~~
	~~Make sure that if this is not your intention, you are transferring to an empty destination directory!!!!</font></b>~~

	First, you need to get a Hoffman account. If you do not yet have one, you need to go through the application process. For a walk through on this process please see our [[Getting_a_Miles_or_Hoffman2_Account \| Getting a Miles or Hoffman2 Account]] HowTo.

	Once you have a Hoffman2 account, you need to transfer your files from miles to Hoffman2. If the project you're transferring is a collaborative project you likely want to take advantage of the group data directories created for this purpose. The data directories are located at the top level of your group folder. For example, if your username is <i>foo</i> and you are a member of the <i>bar</i> group your home directory is likely located at
	~~/u/home9/FMRI/sbook/hoff_user~~
	~~The data directory for your group would be located at:~~
	~~/u/home9/FMRI/sbook/data~~

	~~Finally, this is the 6 step process to moving data from miles to hoffman~~

	*Log into miles
	*Change into your project directory. If you wish to move all projects at once, change into your top level data directory. A typical example might be,
	~~$ cd /Volumes/Songbook2/data/bookheimerlab/ad_projects~~
	*Choose a destination directory, e.g. /u/home9/FMRI/sbook/data/ad_projects
	*Start [[Using_Screen \| GNU Screen]]
	~~$ screen~~
	*Disable extended attributes (don't know what they are? that's ok, disable them anyway :) )
	~~$ export COPYFILE_DISABLE=true~~
	*execute the following command to move the data
	~~$ tar -zvcf - ./ \| ssh hoffman_username@hoffman2.idre.ucla.edu tar -C /u/home9/FMRI/sbook/data/ad_projects -zxf -~~

	Another common task might be moving data from one's miles home directory into their hoffman home directory. Below is the sequence of commands for a fictional user called "bobbie" who is a member of the sbook hoffman group.
	~~$ ssh bobbie@fmri00-miles.hosted.ats.ucla.edu~~
	~~$ cd~~
	~~$ tar -zvcf - * \| ssh bobbie@hoffman2.idre.ucla.edu tar -C /u/home9/FMRI/sbook/bobbie/ -zxf -~~

	~~==Why do I have to do screen first?==~~
	Projects can be extremely large and take a considerable amount of time to transfer to hoffman2. . . even with our blazing fast connections. Various glitches could occur such as loss of connection on your local machine or maybe you just get tired of staring at text for 4 hours in a row and wish to go home. By using screen, you ensure that as long as miles and hoffman2 are up (near 100% of the time) the transfer happily continues.

	~~You can safely turn off or close your laptop without worrying about canceling the transfer.~~

	~~==Why this funky tar command instead of just a regular scp?==~~
	The majority of projects on miles are FSL projects. These project directories have some peculiar features that require the above command. For details, please see our [[FAQ#Why_do_I_keep_getting_directories_like_.2Ffoo.2Ffoo.2Ffoo.2Ffoo.2Ffoo_after_scp.27ing_my_data.3F \| FAQ]] on the topic.

	~~==I want to use a gui, can I?==~~
	You can, however I've found guis to be somewhat unreliable for very large transfers of this nature. Also, by using a gui you must keep your computer on for the duration of the transfer. For many projects on miles this might approach 6 hours or more. The above method produces a more reliable and flexible transfer method.

Ccn admin: 6 revisions

2014-01-16T03:13:12Z

6 revisions

← Older revision	Revision as of 03:13, 16 January 2014
(No difference)

Elau: Undo revision 2769 by Aenalb (talk)

2011-07-07T00:20:39Z

Undo revision 2769 by Aenalb (talk)

New page

=Miles Decommissioned=
Due to a change in ATS policy, the miles cluster is scheduled for retirement at the end of July along with many other small to mid-sized clusters at UCLA.

==What does this mean to me as a researcher on miles?==
This means that all on going projects need to be migrated to Hoffman or onto your local lab computers no later than August 1st. Though I strongly recommend looking at the [http://www.ats.ucla.edu/clusters/hoffman2 Hoffman2 Cluster]. It's an amazing resource and asset to the UCLA research community.

==How do I get my data onto Hoffman?==
WARNING! this method gladly overwrites any existing data of the same name in the destination directory.
Make sure that if this is not your intention, you are transferring to an empty destination directory!!!!

First, you need to get a Hoffman account. If you do not yet have one, you need to go through the application process. For a walk through on this process please see our [[Getting_a_Miles_or_Hoffman2_Account | Getting a Miles or Hoffman2 Account]] HowTo.

Once you have a Hoffman2 account, you need to transfer your files from miles to Hoffman2. If the project you're transferring is a collaborative project you likely want to take advantage of the group data directories created for this purpose. The data directories are located at the top level of your group folder. For example, if your username is foo and you are a member of the bar group your home directory is likely located at
/u/home9/FMRI/sbook/hoff_user
The data directory for your group would be located at:
/u/home9/FMRI/sbook/data

Finally, this is the 6 step process to moving data from miles to hoffman

*Log into miles
*Change into your project directory. If you wish to move all projects at once, change into your top level data directory. A typical example might be,
$ cd /Volumes/Songbook2/data/bookheimerlab/ad_projects
*Choose a destination directory, e.g. /u/home9/FMRI/sbook/data/ad_projects
*Start [[Using_Screen | GNU Screen]]
$ screen
*Disable extended attributes (don't know what they are? that's ok, disable them anyway :) )
$ export COPYFILE_DISABLE=true
*execute the following command to move the data
$ tar -zvcf - ./ | ssh hoffman_username@hoffman2.idre.ucla.edu tar -C /u/home9/FMRI/sbook/data/ad_projects -zxf -

Another common task might be moving data from one's miles home directory into their hoffman home directory. Below is the sequence of commands for a fictional user called "bobbie" who is a member of the sbook hoffman group.
$ ssh bobbie@fmri00-miles.hosted.ats.ucla.edu
$ cd
$ tar -zvcf - * | ssh bobbie@hoffman2.idre.ucla.edu tar -C /u/home9/FMRI/sbook/bobbie/ -zxf -

==Why do I have to do screen first?==
Projects can be extremely large and take a considerable amount of time to transfer to hoffman2. . . even with our blazing fast connections. Various glitches could occur such as loss of connection on your local machine or maybe you just get tired of staring at text for 4 hours in a row and wish to go home. By using screen, you ensure that as long as miles and hoffman2 are up (near 100% of the time) the transfer happily continues.

You can safely turn off or close your laptop without worrying about canceling the transfer.

==Why this funky tar command instead of just a regular scp?==
The majority of projects on miles are FSL projects. These project directories have some peculiar features that require the above command. For details, please see our [[FAQ#Why_do_I_keep_getting_directories_like_.2Ffoo.2Ffoo.2Ffoo.2Ffoo.2Ffoo_after_scp.27ing_my_data.3F | FAQ]] on the topic.

==I want to use a gui, can I?==
You can, however I've found guis to be somewhat unreliable for very large transfers of this nature. Also, by using a gui you must keep your computer on for the duration of the transfer. For many projects on miles this might approach 6 hours or more. The above method produces a more reliable and flexible transfer method.