Hoffman2:Linux Tutorial

Back to all things Hoffman2

Working with UNIX and UNIX like systems via the command line can be intimidating. The good news is a normal user only needs to have a firm understanding of some principle actions to efficiently use UNIX for their every day work.

1. POSIX Permissions
2. File System Navigation
3. File & Shell Management
4. Environment variables

Each of these areas requires knowledge of a few small programs and how they behave. In the following sections, each program and utility is covered in detail. When appropriate, a link to the relevant wiki page on the topic is provided.

Each section is followed by an itemized list of the programs covered and some links to further information on that particular utility as well as the command to bring up the man page for the utility contained within parthesis.

Where appropriate, links to external tutorials deemed suitable and appropriate to the topic are provided rather than reproducing the work here.

POSIX Permissions

POSIX permissions are what determines who and to what degree users can access a file.

The key terminology and function of the POSIX permission system is found here:

UNIX Permissions

List of Utilities Covered

• ls
• chmod (man chmod)
  • Dartmouth Tutorial
  • Exhaustive tutorial including min-quizzes CatCode Tutorial
• chgrp
• umask
• newgrp

File System Navigation

The following series of tutorials provide a very basic introduction to file system navigation on unix like systems without any assumptions of prior knowledge on the topics.

• Listing files & directories, making directories, changing directories, the . and .. directories, pathnames, the "home" directories
  • Tutorial One
• Copying files, moving files, removing files and directories, displaying the contents of a file, searching the contents of a file
  • Tutorial Two

List of Utilities Covered

• ls
• mkdir
• cd
• pwd
• cp
• mv
• rm
• cat
• less
• head & tail
• grep

File & Shell Management

This is where your career on a UNIX type system can be made or crippled. Sure you know how to move around, list files, find out where you are, and display the contents of files. But now you have to do something with those files. And let's face it, there are a whole lot of files.

It is highly recommended that the reader look over Tutorial Four on how to use wildcards for matching before preceding with this section.

The good news is, using the above utilties we just learned about we can accomplish almost anything we want to do using a very handy utility called find.

As the name might imply, find, well, finds things. What it finds is up to you. find has many, many options. All laid out in its man page. However, for most purposes only a few are needed. We'll cover those here.

A basic find command looks like

$ find /path/to/directory -name 'filename.txt'

This command looks at all files in /path/to/directory and in all directories therein for a file named 'filename.txt'.

Common Options

-type

Specifies the type of file we're looking for. e.g. text file, directory, link, etc.

-name

Specifies the name of the file. Case Sensitive

-iname

Specifies the name of the file. Case Insensitive

-or

Joins the precedeing and following terms by the boolean OR

-and

Joins the preceding and following terms by the boolean AND

-not

negates the next term. e.g. -not -empty means "is not empty"

-exec

excutes a shell command for each file found. The only 'trick' is to replace the actual file name with {} and end the command with a \;. This should become clear when reviewing the examples below.

-empty

The file or directory is empty.

We can combine the above options to preform complex searches on the file system and, even better, execute commands on those search terms. For a list of all options and their arguments, please see Find Man Page.

Examples

1. Find all directories named 'tsplot' in the current directory

    $ find . -name tsplot -type d 

2. Find all empty directories in the directory /u/home9/foo/data

    $ find /u/home9/foo/data -empty 

3. Find all empty files or directories named 'tsplot' in the current directory

    $ find . -name tsplot -or -type d -empty 

Using commands we've already learned to perform actions on the above.

1. Find all directories named 'tsplot' in the current directory and delete them using rm

    $ find . -name tsplot -type d -exec rm -rf {} \; 

2. Find all empty directories in the directory /u/home9/foo/data and delete them

    $ find . -type d -empty -exec rmdir {} \; # using rmdir for empty directories is safe as it will refuse to delete a directory with a file in it 

3. Find all empty files or directories named 'tsplot' in the current directory and delete them

    $ find . -name tsplot -or -type d -empty -exec rm -rf {} \; 

But we're not restricted to rm, of course, we can use any of the utilities we've learned about as the -exec argument.

1. Find all files named design.fsf and look for a subject named 'foo'

    $ find . -name design.fsf -type f -exec grep "foo" {} \; 

With a little reading of the man page, we find a new option called -user, which surprisingly finds all files that belong to the specified user

1. Find all files owned by user 'foo' and change their permissions

    $ find . -type f -user foo -exec chmod -R ug+rwX {} \; 

Environment Variables

UNIX uses environment variables to pass information to various tools during a session. These variables are named in all capitals by convention. You can see all of the environment variables and their values by using the command

$ env

To simply see the value of one variable, you can echo its value

$ echo $VARIABLENAME

There are a few environment variables that you should be familiar with...

USER

This has the username of the current user, which should usually be you. See what the value is

$ echo $USER

HOME

This is the path to the home directory of the current user. The tilde symbol (~) is also recognized as shorthand for this home directory.

$ echo $HOME

PATH

This is a list of directories, separated by colons, in which the operating system should check for commands that you type. For instance, when you were using the ls, grep, or find commands in previous tutorials, the operating system started looking through the directories in your PATH variable to find the first command that matched that name and tried executing it. See the directories that the operating system will check for you by typing

$ echo $PATH

Making additions to this variable can be important. Let's say you have a personal set of scripts you have created and you store them in a directory ~/scripts. Every time you want to use one of those scripts, you have to type out the full path to it

$ ~/scripts/my-first-script.sh

or

$ $HOME/scripts/my-first-script.sh

This can get tiring. If you added your scripts directory to your path, you wouldn't have to type that extra bit every time. The best way to do this would be to edit your Bash Profile with a text editor. e.g.

$ vim ~/.bash_profile

and add the line

export PATH=$PATH:~/scripts

to the end of the file and saved it. This will "export" the variable named PATH to the environment and set its value equal to whatever was already in PATH plus the directory ~/scripts. The next time you login, you can do

$ echo $PATH

and see that at the end of the list of directories to search, your directory ~/scripts has been added. Now you can be anywhere in the filesystem and call the command

$ my-first-script.sh

to run that same script from before.

Collisions on the PATH

If you get in the habit of naming your scripts the same thing (e.g. my-script.sh) and placing them in different directories, you may run into a collision on your PATH. This is a case where you think you are running one script, but the operating system is actually running another. Let's look at an example.

Continuing from the previous example where we have the script ~/scripts/my-first-script.sh, let's say that we make another directory called analyze and we are working with some data there and make a processing script coincidentally called my-first-script.sh. So we have the files

~/scripts/my-first-script.sh
~/analyze/my-first-script.sh
~/analyze/data-file-1
~/analyze/data-file-2
...

And we have amended our .bash_profile so that ~/scripts is at the end of our PATH environment variable.

If we change to the analyze directory

$ cd ~/analyze

and wish to run the processing script my-first-script.sh on the data, you may think we can execute

$ my-first-script.sh

and call it a day. But this will actually run the file ~/scripts/my-first-script.sh because it is the first file the operating system found in the directories of PATH that matched that name. If you wanted to verify this, you can execute

$ which my-first-script.sh

This command will search your PATH variable for the first instance of my-first-script.sh and return the full path to it, something like this

~/scripts/my-first-script.sh

To run the script we had intended, we would need to execute

$ ./my-first-script.sh

The period and slash specify that the operating system should look in the current directory for this script.

If something seems weird or script isn't working, a good starting point is to check that you are running the script you think you are. Use which to find out

Learning to Fish or The Man Pages

However, possibly even more important is being able to find information on one's own. In the spirit of this, the instructional is concluded with a detailed coverage of the built-in unix manual pages. Virtually every program installed on a UNIX like system provied a manual page for reference. Even the most proficient of UNIX users could never retain the dizzying array of programs and their options in their working memory and with the built-in manual pages, they'll never need to.

The man pages (for manual) are the be all end all reference on UNIX systems. When reading a man page becomes natural, you know there is nothing you can't get done quickly and efficiently.

A Beginners Guide to man Pages is an excellent introduction into how to move around a man page easily and understand what it's telling you.

External Links

• UNIX Tutorial