Every Day UNIX and Working with Hoffman

The Trifecta of Every Day Unix

Working with UNIX and UNIX like systems via the command line can be intimidating. The good news is a normal user only needs to have a firm understanding of three principle actions to efficiently use UNIX for their every day work.

1. POSIX Permissions
2. File System Navigation
3. File & Shell Management

Each of these areas require knowledge of a few small programs and how they behave. In the following sections, each program and utility is covered in detail. When appropriate, a link to the relevant wiki page on the topic is provided.

Each section is followed by an itemized list of the programs covered and some links to further information on that particular utility as well as the command to bring up the man page for the utility contained with parthesis.

Where appropriate, links to external tutorials deemed suitable and apropriate to the topic are provided rather than reproducing the work here.

Learning to Fish or The Man Pages

However, possibly even more important is being able to find information on one's own. In the spirit of this, the instructional is concluded with a detailed coverage of the built-in unix manual pages. Virtually every program installed on a UNIX like system provied a manual page for reference. Even the most proficient of UNIX users could never retain the dizzying array of programs and their options in their working memory and with the built-in manual pages, they'll never need to.

The man pages (for manual) are the be all end all reference on UNIX systems. When reading a man page becomes natural, you know there is nothing you can't get done quickly and efficiently.

A Beginners Guide to man Pages is an excellent introduction into how to move around a man page easily and understand what it's telling you.

POSIX Permissions

POSIX permissions are what determines who and to what degree users can access a file. The key terminology and function of the POSIX permission system is found on the UNIX Permissions wiki page.

List of Utilities Covered

• chmod (man chmod)
  • Dartmouth Tutorial
  • Exhaustive tutorial including min-quizzes CatCode Tutorial
• chgrp
• umask
• newgrp

File System Navigation

The following series of tutorials provide a very basic introduction to file system navigation on unix like systems without any assumptions of prior knowledge on the topics.

• Listing files & directories, making directories, changing directories, the . and .. directories, pathnames, the "home" directories
  • Tutorial One
• Copying files, moving files, removing files and directories, displaying the contents of a file, searching the contents of a file
  • Tutorial Two

List of Utilities Covered

• ls
• mkdir
• cd
• pwd
• cp
• mv
• rm
• cat
• less
• head & tail
• grep

File & Shell Management

This is where your career on a UNIX type system can be made or crippled. Sure you know how to move around, list files, find out where you are, and display the contents of files. But now you have to do something with those files. And let's face it, there are a whole lot of files.

It is highly recommended that the reader look over Tutorial Four on how to use wildcards for matching before preceding with this section.

The good news is, using the above utilties we just learned about we can accomplish almost anything we want to do using a very handy utility called find.

As the name might imply, find, well, finds things. What it finds is up to you. find has many, many options. All laid out in its man page. However, for most purposes only a few are needed. We'll cover those here.

A basic find command looks like

$ find /path/to/directory -name 'filename.txt'

This command looks at all files in /path/to/directory and in all directories therein for a file named 'filename.txt'.

Common Options

-type

Specifies the type of file we're looking for. e.g. text file, directory, link, etc.

-name

Specifies the name of the file. Case Sensitive

-iname

Specifies the name of the file. Case Insensitive

-or

Joins the precedeing and following terms by the boolean OR

-and

Joins the preceding and following terms by the boolean AND

-not

negates the next term. e.g. -not -empty means "is not empty"

-exec

excutes a shell command for each file found. The only 'trick' is to replace the actual file name with {} and end the command with a \;. This should become clear when reviewing the examples below.

-empty

The file or directory is empty.

We can combine the above options to preform complex searches on the file system and, even better, execute commands on those search terms. For a list of all options and their arguments, please see Find Man Page.

Examples

1. Find all directories named 'tsplot' in the current directory

$ find . -name tsplot -type d 

1. Find all empty directories in the directory /u/home9/foo/data

$ find /u/home9/foo/data -empty 

1. Find all empty files or directories named 'tsplot' in the current directory

$ find . -name tsplot -or -type d -empty

Using commands we've already learned to perform actions on the above.

1. Find all directories named 'tsplot' in the current directory and delete them using rm

$ find . -name tsplot -type d -exec rm -rf {} \;

1. Find all empty directories in the directory /u/home9/foo/data and delete them

$ find . -type d -empty -exec rmdir {} \; # using rmdir for empty directories is safe as it will refuse to delete a directory with a file in it

1. Find all empty files or directories named 'tsplot' in the current directory and delete them

$ find . -name tsplot -or -type d -empty -exec rm -rf {} \;

But we're not restricted to rm, of course, we can use any of the utilities we've learned about as the -exec argument.

1. Find all files named design.fsf and look for a subject named 'foo'

$ find . -name design.fsf -type f -exec grep "foo" {} \;

With a little reading of the man page, we find a new option called -user, which surprisingly finds all files that belong to the specified user

1. Find all files owned by user 'foo' and change their permissions

$ find . -type f -user foo -exec chmod -R ug+rwX {} \;

Setting up your Bash Profile

This is the easiest part. Just place this one, single line in your ~/.bash_profile:

source /u/home/FMRI/apps/etc/profile

If you wish to grant group write permissions, use the umask command discussed in detail in the basic unix section above

umask 007

Or you can cut and paste this into your ~/.bash_profile

source /u/home/FMRI/apps/etc/profile
umask 007

Save, then log out and back in. This is all that is needed to gain full access to all FMRI applications. Nothing else should be put in this file, ever, unless you 100% know what you are doing and why, don't take someone else's word for it either.

Cyberduck: An SFTP client

Cyberduck installs like most other mac application, just drag and drop. Please see our screen cast on using Cyberduck.

MacFuse installation

1. Download MacFuse and install it. (double click the MacFuse.mpkg file)
2. Download the appropriate (tiger or leopard) sshfs from the MacFuse wiki
3. Un-archive the downloaded sshfs (double click)
4. Copy the sshfs binary to your /usr/local/bin directory or anywhere else on your path
5. Make sure the sshfs binary has executable permissions

Example Workflow

This assumes the user has downloaded and installed or unarchived the needed packages listed above. The below example assumes the leopard sshfs binary was downloaded.

$ tar xzvf sshfs-static-leopard.gz
$ sudo mv sshfs-static-leopard /usr/local/bin/sshfs
$ sudo chmod +x /usr/local/bin/sshfs

Example Mounting of a Hoffman2 Directory

Of course, you can name the /Volumes/<directory> anything you want or even put it anywhere you want. For example, I have /Volumes/Hoffman/sbook, /Volumes/Hoffman/my_home, and /Volumes/Hoffman/data

# Mounting the SSH file system
$ mkdir /Volumes/MyHoffmanHome
$ sshfs your_hoffman_username@hoffman2.idre.ucla.edu:/path/to/directory/you/want/to/mount /Volumes/MyHoffmanHome -oauto_cache,reconnect

SPM

SPM8 as well as the older SPM5 are on hoffman. In order to use these, you must add the path ("Add with Subfolders") in MATLAB.

The path to spm8 is:

/u/home/FMRI/apps/spm8

Likewise, the path to spm5 is:

/u/home/FMRI/apps/spm5

FAQ

How do I change the permissions for all files I own in a certain directory?

As a reference, please see the above section on [#File & Shell Management | File & Shell Management]. We can do this using find and chmod.

$ find . -user myusername -type f -not -perm -g+w -exec chmod g+w {} \;

There is also a helper script which will set group permissions on all files in the current directory that are owned by user called fix_perms.sh. To use this, simply

$ fix_perms.sh

How do I transfer data to and from hoffman?

Please see our general FAQ on how to properly transfer data between two machines found here FAQ: How do I properly transfer my data between two machines on the network

How do I switch FSL Versions?

Please see the help output of the following utility on hoffman which eases this process:

$ switch_fsl --help
Description: switch_fsl easily and cleanly switches between various fsl versions.
Usage: switch_fsl [OPTIONS]
Options:
    -h, --help                       Display this help/usage message
    -v, --verbose                    Verbose output
    -s, --switch VERSION             Designate the appropriate fsl version. 
					If switching between versions 3.x and 4.x, you must log out
					then back it to reset up your environment.
					Available versions are: 4.0.4, 4.1.0, 4.1.1, 4.1.3
    -c, --check                      Checks for current fsl version being used
Examples: 
	switch_fsl -s 4.1.0
	switch_fsl --switch 4.0.4
	switch_fsl -h
	switch_fsl --check

External Links

• Hoffman2 FAQ (Frequently Asked Questions
• Hoffman2 Home Page
• Hoffman2 Announcements and News
• UCLA Grid Portal
• Hoffman2 Security Policy
• Using the Cluster from the UCLA Grid (web) Portal
• Logging into Hoffman2 head node over SSH
• Computing on the Hoffman2 Cluster