The bash shell and common command line tools

Rob Marissen Center for Proteomics and Metabolomics

Introduction Overview • Command line editing tricks • Common commands and examples • What is a shell, what is bash? • Using bash • Redirecting/piping • Concluding remarks

https://humgenprojects.lumc.nl/trac/humgenprojects/wiki/scripting Linux and basic scripting

1/22

Monday, January 26, 2015

Command line editing tricks Keyboard command line editing Most Linux users only use the command line for file based (and many other) operations. It is usually much faster than graphical tools!

Linux and basic scripting

2/22

Monday, January 26, 2015

Command line editing tricks Keyboard command line editing • Command completion: TAB • Stop entering line: Ctrl+c • Navigate commands with (Ctrl) arrow keys • Start/end of line: HOME END • Repeat last item from previous command: Alt+.

Linux and basic scripting

3/22

Monday, January 26, 2015

Command line editing tricks Quick copy/paste • Select text • Press middle button to paste

Works also in other programs on Linux In bash, paste happens at the location of the cursor Linux and basic scripting

4/22

Monday, January 26, 2015

Command line editing tricks Command history Commands are kept for later re-use. To show ’all’ the commands entered so far: history To show the commands containing ls: history | grep ls

Linux and basic scripting

5/22

Monday, January 26, 2015

Common commands and examples

Frequently used programs: • Task specific (proteomics/genomics/...) • File system (pwd, cd, mkdir, ls, rm) • Viewing files (less, cat, head, tail) • Generic (man, echo, wc, grep, find, history) • Remote connection (wget, ssh) • Scripting (awk, sed) • Sytem administration (sudo, apt-get, ...) • There are many more. In general, use

(with

’linux’ and ’command line’ in de search box)

Linux and basic scripting

6/22

Monday, January 26, 2015

Common commands and examples man man shows a user manual of the command

While looking at a manual page, you can search by pressing slash (/) This also works in less and many other programs Example: search for the word directory while looking at a manual page: /directory man -k lists commands whos manual page contains

keyword Example: list commands that let you edit something 1

$ man -k edit

Linux and basic scripting

7/22

Monday, January 26, 2015

Common commands and examples echo • Print text to screen • Useful in scripts to show what is going on • Used in piping/redirecting: echo "this is a test"> myfile.txt

Linux and basic scripting

8/22

Monday, January 26, 2015

Common commands and examples wc Count number of lines, words and characters 1

$ wc uniprot-human.fasta

Linux and basic scripting

9/22

Monday, January 26, 2015

Common commands and examples ps/kill Show programs running on your computer To show all programs, with some extra information: 1

$ ps aux

Second column shows PID. To terminate a program (for instance if it hangs) assuming its PID is 12345: 1 2 3

$ kill 12345 # Ask program to stop $ kill -KILL 12345 # Forcefull stop program $ killall firefox # kill by name

Linux and basic scripting

10/22

Monday, January 26, 2015

Common commands and examples find • Search for files and directories • Many ways to specify selection: check man page

Example: search for all files ending with ’.fasta’ in directory ’mascot’ that where last modified more than 2 years ago: 1 2 3 4

$ find /nfs/mascot -type f -name \*.fasta -mtime +730 /nfs/mascot/sequence/Cdifficile630/current/ 25374.C_difficile_630.fasta /nfs/mascot/sequence/Sman_GP_0808/current/ GeneDB_Smansoni_Proteins.fasta /nfs/mascot/sequence/Sman_GP_4.0h/old/ GeneDB_Smansoni_Proteins.v4.0h.fasta

Linux and basic scripting

11/22

Monday, January 26, 2015

Common commands and examples wget • Get data from the internet (http/ftp) • Store data in file or process it directly (using piping) • Many options, e.g. download recursively

Example: Download the yeast protein fasta file from uniprot 1 2 3 4

$ wget ftp://ftp.uniprot.org/pub/databases/uniprot\ /current_release/knowledgebase/proteomes/\ YEAST.fasta.gz $ gunzip YEAST.fasta.gz

Linux and basic scripting

12/22

Monday, January 26, 2015

What is a shell, what is bash? What is a shell

In general: a shell surrounds/protects something inside In computer terms, a shell surrounds the operating system

Linux and basic scripting

13/22

Monday, January 26, 2015

What is a shell, what is bash? What is a shell • enter/edit commands • execute programs • execute scripts • execute control statements (loops/conditional exection)

Most commands are external programs. E.g. firefox can simply be started from command line. which shows the location of a program 1 2 3 4

$ which ls /bin/ls $ ls -al /bin/ls -rwxr-xr-x 1 root root 110080 Jan 14 04:50 /bin/ls

Linux and basic scripting

14/22

Monday, January 26, 2015

What is a shell, what is bash? What is a bash • bash is a shell • several shells are available in Linux

The name bash is a pun: Bourne Again shell

Linux and basic scripting

15/22

Monday, January 26, 2015

Using bash Wildcards Wildcards can be used to specify one or more files without typing the whole filename • The * character matches zero or more characters • The ? character matches one character • Wildcards are handled by bash: the program that you

call sees only the expanded form: 1 2 3 4

$ ls a_file.txt $ echo a* a_file.txt

another_file.txt

yet_another_file.txt

another_file.txt

• Take care tomorrow at the regular expression lecture:

wildcards are not regular expressions! Linux and basic scripting

16/22

Monday, January 26, 2015

Using bash Handling strange filenames • Many characters have special meaning to bash, e.g.

space, quotes (’, ” and ‘), *, $ • Almost all characters are valid in file/directory names • Try to avoid using these characters • Even better: handle them correctly • Put a backslash \ before special characters

Linux and basic scripting

17/22

Monday, January 26, 2015

Using bash Command line options Program behaviour can be modified with command line options, e.g. ls -al • Not standardized between programs • Check man page for exact meaning • Most start with a hyphen (-) • Take care: filenames can also start with a • Filenames usually are at the end of command lines • (Usually) good practice to put -- before filenames. E.g. rm -- -delete-me-

Linux and basic scripting

18/22

Monday, January 26, 2015

Redirecting/piping Redirecting output to files Instead of printing output on screen, you can send it to a file • > creates a new file, overwriting existing one. • >> Appends to a file, preserving the data. 1 2 3 4 5 6 7 8 9 10

$ ls Docs Downloads $ ls > list.txt; cat list.txt Docs Downloads $ ls >> list.txt; cat list.txt Docs Downloads Docs Downloads

Linux and basic scripting

19/22

Monday, January 26, 2015

Redirecting/piping Pipes • Pipes are useful to send output from one command to

another command • Check if output is valid input for the next command • Use a hyphen - for a command which requires a file 1 2 3 4 5

$ ls abba_poster.jpg dancing_queen.FLAC list.txt sos.mp3 waterloo.mp3 $ ls | grep ’mp3’ > my_music_collection.txt $ ls | grep ’FLAC’ >> my_music_collection.txt

Linux and basic scripting

20/22

Monday, January 26, 2015

Concluding remarks

• There are many more cool things you can do with a shell • There are many more cool things command line

programs to use • Use Google to find solutions for your problem

Linux and basic scripting

21/22

Monday, January 26, 2015

Questions?

¨ Tugce Guler Zuotian Tatum Maarten van Iterson Martijn Vermaat Rob Marissen Magnus Palmblad Jeroen Laros

https://humgenprojects.lumc.nl/trac/humgenprojects/wiki/scripting Linux and basic scripting

22/22

Monday, January 26, 2015