The bash shell and common command line tools
Rob Marissen Center for Proteomics and Metabolomics
Introduction Overview • Command line editing tricks • Common commands and examples • What is a shell, what is bash? • Using bash • Redirecting/piping • Concluding remarks
https://humgenprojects.lumc.nl/trac/humgenprojects/wiki/scripting Linux and basic scripting
1/22
Monday, January 26, 2015
Command line editing tricks Keyboard command line editing Most Linux users only use the command line for file based (and many other) operations. It is usually much faster than graphical tools!
Linux and basic scripting
2/22
Monday, January 26, 2015
Command line editing tricks Keyboard command line editing • Command completion: TAB • Stop entering line: Ctrl+c • Navigate commands with (Ctrl) arrow keys • Start/end of line: HOME END • Repeat last item from previous command: Alt+.
Linux and basic scripting
3/22
Monday, January 26, 2015
Command line editing tricks Quick copy/paste • Select text • Press middle button to paste
Works also in other programs on Linux In bash, paste happens at the location of the cursor Linux and basic scripting
4/22
Monday, January 26, 2015
Command line editing tricks Command history Commands are kept for later re-use. To show ’all’ the commands entered so far: history To show the commands containing ls: history | grep ls
Linux and basic scripting
5/22
Monday, January 26, 2015
Common commands and examples
Frequently used programs: • Task specific (proteomics/genomics/...) • File system (pwd, cd, mkdir, ls, rm) • Viewing files (less, cat, head, tail) • Generic (man, echo, wc, grep, find, history) • Remote connection (wget, ssh) • Scripting (awk, sed) • Sytem administration (sudo, apt-get, ...) • There are many more. In general, use
(with
’linux’ and ’command line’ in de search box)
Linux and basic scripting
6/22
Monday, January 26, 2015
Common commands and examples man man shows a user manual of the command
While looking at a manual page, you can search by pressing slash (/) This also works in less and many other programs Example: search for the word directory while looking at a manual page: /directory man -k lists commands whos manual page contains
keyword Example: list commands that let you edit something 1
$ man -k edit
Linux and basic scripting
7/22
Monday, January 26, 2015
Common commands and examples echo • Print text to screen • Useful in scripts to show what is going on • Used in piping/redirecting: echo "this is a test"> myfile.txt
Linux and basic scripting
8/22
Monday, January 26, 2015
Common commands and examples wc Count number of lines, words and characters 1
$ wc uniprot-human.fasta
Linux and basic scripting
9/22
Monday, January 26, 2015
Common commands and examples ps/kill Show programs running on your computer To show all programs, with some extra information: 1
$ ps aux
Second column shows PID. To terminate a program (for instance if it hangs) assuming its PID is 12345: 1 2 3
$ kill 12345 # Ask program to stop $ kill -KILL 12345 # Forcefull stop program $ killall firefox # kill by name
Linux and basic scripting
10/22
Monday, January 26, 2015
Common commands and examples find • Search for files and directories • Many ways to specify selection: check man page
Example: search for all files ending with ’.fasta’ in directory ’mascot’ that where last modified more than 2 years ago: 1 2 3 4
$ find /nfs/mascot -type f -name \*.fasta -mtime +730 /nfs/mascot/sequence/Cdifficile630/current/ 25374.C_difficile_630.fasta /nfs/mascot/sequence/Sman_GP_0808/current/ GeneDB_Smansoni_Proteins.fasta /nfs/mascot/sequence/Sman_GP_4.0h/old/ GeneDB_Smansoni_Proteins.v4.0h.fasta
Linux and basic scripting
11/22
Monday, January 26, 2015
Common commands and examples wget • Get data from the internet (http/ftp) • Store data in file or process it directly (using piping) • Many options, e.g. download recursively
Example: Download the yeast protein fasta file from uniprot 1 2 3 4
$ wget ftp://ftp.uniprot.org/pub/databases/uniprot\ /current_release/knowledgebase/proteomes/\ YEAST.fasta.gz $ gunzip YEAST.fasta.gz
Linux and basic scripting
12/22
Monday, January 26, 2015
What is a shell, what is bash? What is a shell
In general: a shell surrounds/protects something inside In computer terms, a shell surrounds the operating system
Linux and basic scripting
13/22
Monday, January 26, 2015
What is a shell, what is bash? What is a shell • enter/edit commands • execute programs • execute scripts • execute control statements (loops/conditional exection)
Most commands are external programs. E.g. firefox can simply be started from command line. which shows the location of a program 1 2 3 4
$ which ls /bin/ls $ ls -al /bin/ls -rwxr-xr-x 1 root root 110080 Jan 14 04:50 /bin/ls
Linux and basic scripting
14/22
Monday, January 26, 2015
What is a shell, what is bash? What is a bash • bash is a shell • several shells are available in Linux
The name bash is a pun: Bourne Again shell
Linux and basic scripting
15/22
Monday, January 26, 2015
Using bash Wildcards Wildcards can be used to specify one or more files without typing the whole filename • The * character matches zero or more characters • The ? character matches one character • Wildcards are handled by bash: the program that you
call sees only the expanded form: 1 2 3 4
$ ls a_file.txt $ echo a* a_file.txt
another_file.txt
yet_another_file.txt
another_file.txt
• Take care tomorrow at the regular expression lecture:
wildcards are not regular expressions! Linux and basic scripting
16/22
Monday, January 26, 2015
Using bash Handling strange filenames • Many characters have special meaning to bash, e.g.
space, quotes (’, ” and ‘), *, $ • Almost all characters are valid in file/directory names • Try to avoid using these characters • Even better: handle them correctly • Put a backslash \ before special characters
Linux and basic scripting
17/22
Monday, January 26, 2015
Using bash Command line options Program behaviour can be modified with command line options, e.g. ls -al • Not standardized between programs • Check man page for exact meaning • Most start with a hyphen (-) • Take care: filenames can also start with a • Filenames usually are at the end of command lines • (Usually) good practice to put -- before filenames. E.g. rm -- -delete-me-
Linux and basic scripting
18/22
Monday, January 26, 2015
Redirecting/piping Redirecting output to files Instead of printing output on screen, you can send it to a file • > creates a new file, overwriting existing one. • >> Appends to a file, preserving the data. 1 2 3 4 5 6 7 8 9 10
$ ls Docs Downloads $ ls > list.txt; cat list.txt Docs Downloads $ ls >> list.txt; cat list.txt Docs Downloads Docs Downloads
Linux and basic scripting
19/22
Monday, January 26, 2015
Redirecting/piping Pipes • Pipes are useful to send output from one command to
another command • Check if output is valid input for the next command • Use a hyphen - for a command which requires a file 1 2 3 4 5
$ ls abba_poster.jpg dancing_queen.FLAC list.txt sos.mp3 waterloo.mp3 $ ls | grep ’mp3’ > my_music_collection.txt $ ls | grep ’FLAC’ >> my_music_collection.txt
Linux and basic scripting
20/22
Monday, January 26, 2015
Concluding remarks
• There are many more cool things you can do with a shell • There are many more cool things command line
programs to use • Use Google to find solutions for your problem
Linux and basic scripting
21/22
Monday, January 26, 2015
Questions?
¨ Tugce Guler Zuotian Tatum Maarten van Iterson Martijn Vermaat Rob Marissen Magnus Palmblad Jeroen Laros
https://humgenprojects.lumc.nl/trac/humgenprojects/wiki/scripting Linux and basic scripting
22/22
Monday, January 26, 2015