Unix/Linux Utility Commands

subshell

Unix subshell allows to scope the execution of command and the effect to a group of commands. For example changing directories or manipulating environment variables.

Example:

Change directory in subshell and execute command without switching directory in main shell

( cd ../ && pwd ) & ( cd ../../ && pwd ) & pwd

Add new environment variable to a command

( export NEW_VARIABLE=123 && echo $NEW_VARIABLE ) && echo $NEW_VARIABLE

Remove an environment variable to a command

( unset JAVA_HOME && echo $JAVA_HOME ) && echo $JAVA_HOME

Change an environment variable to a command

( export HOME=$HOME/another && echo $HOME ) && echo $HOME

The same can be achieved using -c option with sh or bash or zsh shell explicitly

sh -c ' export HOME=$HOME/another && echo $HOME ' && echo $HOME

bash -c ' export HOME=$HOME/another && echo $HOME ' && echo $HOME

zsh -c ' export HOME=$HOME/another && echo $HOME ' && echo $HOME

start multiple commands in parallel using subshell and wait

$ ( ( command1 2>&1 | tee command1.log | sed 's/^/[command1]/g' ) & ( command2 2>&1 | tee command2.log | sed 's/^/[command2]/g' ) & wait )

Press Ctrl+C to stop all command and wait gracefully for commands to shutdown

Outputs of each command is printed in console and captured in the individual log files

Same command in formatted style for readability

( \

( command1 2>&1 | tee command1.log | sed 's/^/[command1]/g' ) & \

( command2 2>&1 | tee command2.log | sed 's/^/[command2]/g' ) & \

wait )

Eg:

$ ( ( npm run dev 2>&1 | tee dev.log | sed 's/^/[dev]/g' ) & ( npm run cypress 2>&1 | tee cypress.log | sed 's/^/[cypress]/g' ) & wait )

test command

test command is used to test mathematical expressions or file or directory related verifications. The expression can also be evaluated using double square brackets [[ ]] which is useful with if else statements.

test 100 -gt 99 && echo 'Yes' || echo 'No'

[[ 100 -gt 99 ]] && echo 'Yes' || echo 'No'

Prints Yes as the expression evaluates to true

test -f /tmp/testfile.txt && echo 'Exists' || echo 'Not Exists'

[[ -f /tmp/testfile.txt ]] && echo 'Exists' || echo 'Not Exists'

Prints the message based on the expression result

test -f /tmp/testfile.txt -a -d /tmp && echo 'True' || echo 'False'

[[ ( -f /tmp/testfile.txt ) && ( -d /tmp ) ]] && echo 'True' || echo 'False'

Using and operator to verify multiple expressions

test ! -f /tmp/testfile.log && echo 'Not Exists'

[[ ! ( -f /tmp/testfile.log ) ]] && echo 'Not Exists'

Using not operator to negate the expression result

The full list of options can be listed by

man test

Some common options:

-e check if file or directory or symbolic link exists

-f check if regular file exists

-d check if regular directory exists

-L check if symbolic link exists

-O check if user owns the file or directory

-r check if file or directory is readable

-w check if file or directory is writtable

-x check if file is executable

-s check if the file is non empty

-a AND && operation

-o OR || operation

! NOT operation

which command

Prints the location of the executable

which java

pbcopy and pbpaste

Copy the output of a command to clipboard

ls | pbcopy

Print the clipboard value in terminal

pbcopy

Copy the clipboard value to a variable to reuse

tmp=$(pbpaste)

echo $tmp

sleep command

sleep or block the terminal for 5 seconds

sleep 5

sleep or block the terminal for 0.4 seconds

sleep 0.4

Adding header and footer line to stream of lines

Adding header line

cat sample.txt | (printf 'header\n' ; cat - ; printf '\n')

Adding header and footer line

cat sample.txt | (printf 'header\n' ; cat - ; printf '\nfooter\n')

cut

$ printf "Hello\tworld\nHi\tthere\n" | cut -f 2

Outputs the 2nd element after splitting the tab separated columns. -f value starts from 1

$ echo 'key=value' | cut -d "=" -f 2

Outputs the 2nd element after splitting the string using delimiter

$ cat input.tsv | cut -f 1,2,4

Outputs first, second and fourth column of tab separated file

$ cat input.csv | cut -d "," -f 2,4

Outputs second and fourth column of comma separated file

comm

Compares the files which contains list and prints the differences

$ comm -23 <(cat list_a.txt | sort | uniq) <(cat list_b.txt | sort | uniq)

Prints the items present in list_a.txt and missing in list_b.txt

$ comm -13 <(cat list_a.txt | sort | uniq) <(cat list_b.txt | sort | uniq)

Prints the items present in list_b.txt and missing in list_a.txt

$ comm -12 <(cat list_a.txt | sort | uniq) <(cat list_b.txt | sort | uniq)

Prints the common items present in both list_a.txt and list_b.txt

$ comm -3 <(cat list_a.txt | sort | uniq) <(cat list_b.txt | sort | uniq) | xargs -n1

Performs XOR operation between two list_a.txt and list_b.txt. That is print the lines which are not present in both files

$ comm <(cat list_a.txt | sort | uniq) <(cat list_b.txt | sort | uniq) | xargs -n1

Performs OR operation between two list_a.txt and list_b.txt. That is print the lines which is present in either of both files

xargs

$ printf "1\n2" | xargs -I {} echo "Hello {}"

Passes each line from first echo command to second echo command. -I argument indicates the symbol to substitute in the command to be executed by xargs for each line

$ printf "1\n2" | xargs -I{} echo "echo 'Hello {}' && echo 'Welcome {}'" | bash

Execute multiple commands for value in each line using bash or sh or zsh shell

$ printf "1\n2" | xargs -n1 sh -c 'echo "Hello $0"'

$ printf "1\t2\n3\t4" | xargs -n2 sh -c 'echo "Hello $1 $0"'

$ printf "1 2\n3 4" | xargs -n2 sh -c 'echo "Hello $1 $0"'

$ printf "1,2\n3,4" | tr ',' ' ' | xargs -n2 sh -c 'echo "Hello $0 $1"'

Passes multiple arguments to anonymous shell script to execute arbitrary commands with parameter substitution. The parameters can be tab or space separated. The xargs will execute the anonymous shell for each input line

xargs also trims the white spaces or tabs in the input line and prints only the trimmed text

$ printf "\t Hello \t there\t" | xargs

$ printf " Hello there " | xargs

grep

$ grep -ir 'sample text' .

i - ignore case
r - recursive in directory
v - invert match

include - include the file extension (multiple allowed)

exclude - exclude the file extension (multiple allowed)
n - print the line number of the matching line
E - extended regular expression
F - treat the search string as literal string and not use is as regex
f - load the search pattern / string from the given file
l - print file names that has matching text
L - print file names that does not match text
o - print only matched text (doesn't print the entire line)

c - print only the count of matched lines
H - print filename at the beginning of each line (default)
h - do not print filename at the beginning
n - print line number
A NUM - no of lines after matching line
B NUM - no of line before matching line

Practical examples of grep:

Extract all image url from a html page

$ curl -s "https://example.com" | grep -oE 'src="http[s]?[^"]"' | grep -oE 'http[s]?[^"]' | grep -vE '\.(js|css)$' | sort | uniq

Print all lines matching the search texts (one per line) in a given file

$ grep -F -f search_keywords.txt file-to-be-searched.csv

Print all lines not matching the search texts (one per line) in a given file

$ grep -Fv -f search_keywords.txt file-to-be-searched-for-non-matching-lines.csv

Exclude or include directories in search

$ grep -ir 'someText' --include-dir='main' --exclude-dir='test' --exclude-dir='target' .

Search for file within certain extension types with given keyword

$ grep -ir 'someText' --include='.xml' --include='.json' .

Search for file excluding certain extension types with given keyword

$ grep -ir 'someText' --exclude='.log' --exclude='.json' .

Search for text in files and display lines before (-B) and after (-A) the matching line

$ grep -B 2 -A 2 -ir 'someText' .

Search for number of occurrence of a word in a file once per line

$ grep -ic 'someText' file.txt

Search for number of total occurrence of a word in a file (even if repeated multiple times in a line)

$ grep -io 'someText' file.txt | grep -ic 'someText'

Chaining multiple greps and list file names containing or not containing texts

$ grep -lir 'firstText' . | xargs -I {} grep -Hi 'secondText' {}

$ grep -lir 'firstText' . | xargs -I {} grep -li 'secondText' {} | xargs -I {} grep -Hi 'thirdText' {}

$ grep -lir 'textToMatch' . | xargs -I {} grep -li 'furtherToMatch' {} | xargs -I {} grep -Li 'notToMatch' {}

Search for multiple words or patterns using |

$ grep -E 'pattern1|pattern2' file.txt

Group line item by quantity as csv and sort it by quantity in descending order

$ cat line-items.txt | sort | uniq | xargs -I{} echo "printf '{},' && grep -c '^{}$' line-items.txt" | bash | sort -nr -k 2 -t ,

sed

Prints 2nd line from file or stream

$ printf " Line 1 \n Line 2\nLine 3 " | sed '2!d'

Prints 2nd to 4th line from file or stream

$ printf "Line 1\nLine 2\nLine 3\nLine 4\nLine 5\n" | sed '2,4!d'

Delete the 1st line from the file or stream

$ printf "Header Line\nLine 1\nLine 2\n" | sed '1d'

Deletes 2nd to 4th line from file or stream

$ printf "Line 1\nLine 2\nLine 3\nLine 4\nLine 5\n" | sed '2,4d'

Merge 2 consecutive lines into 1 line

$ printf "Line 1\nLine 2\nLine 3\nLine 4\n" | sed 'N;s/\n/ /g'

Merge 3 consecutive lines into 1 line

$ printf "Line 1\nLine 2\nLine 3\nLine 4\nLine 5\nLine 6\n" | sed 'N;N;s/\n/ /g'

Deletes all lines which has white space at the beginning of the line

$ printf " Line 1 \n Line 2\nLine 3 " | sed -E "/^[[:space:]]+.*$/d"

Strips out any white spaces in the front and the end and prints only the non white space characters.

$ printf " Line 1 \n Line 2\nLine 3 " | sed -E "s/^([[:space:]])(.)([[:space:]]*)$/\2/g"

Swaps the second matching group in the regex with the first matching group

$ printf "key1=value1\nkey2=value2" | sed -E "s/^(.)=(.)$/\2=\1/g"

Remove n characters from front of every line

$ printf '1234XX' | sed -E 's/^.{4}//g'

Remove n character from last of every line

$ printf '1234XX' | sed -E 's/.{2}$//g'

Add text to the start and end of each line, example add port as suffix and protocol as prefix for each line

$ printf "localhost\n127.0.0.1" | sed -E "s|$|:80/|g" | sed -E "s|^|http://|g"

Transforms the date format from dd/MM/yyyy to yyyy-MM-dd

$ printf "start_date,end_date\n13/01/2020,16/02/2020\n04/05/2020,07/05/2020" | sed -E "s|([0-9]{2})/([0-9]{2})/([0-9]{4})|\3-\2-\1|g"

Search and replace all occurences of 'key' with 'cache'

$ printf "key1=key-value1\nkey2=key-value2" | sed -E "s/key/cache/g"

Search and replace all occurences of 'key' with 'cache' as case-insensitive

$ printf "key1=Key-value1\nkey2=Key-value2" | sed -E "s/[kK][eE][yY]/cache/g"

Multiple search and replace seperated by semicolon

$ printf "key1=Key-value1\nkey2=Key-value2" | sed -E "s/key/cache/g;s/Key/Cache/g"

Escaping delimiter in search and replace text can be done by escaping the '/' character or by changing the delimiter itself. In the following examples, add/sub will be addition/subtraction

$ printf "Text add/sub" | sed -E "s/add\/sub/addition\/subtraction/g"

$ printf "Text add/sub" | sed -E "s#add/sub#addition/subtraction#g"

Search and replace all occurrences of text in a file without generating backup file

$ sed -i '' -E 's/world/WORLD/g' sample.txt

Search and replace all occurrences of text in a file with backup file

$ sed -i '.bkp' -E 's/world/WORLD/g' sample.txt

Search and replace all occurrences of date format in a file using regex

$ sed -i '' -E "s|([0-9]{2})/([0-9]{2})/([0-9]{4})|\3-\2-\1|g" sample.txt

Text transformation

Transform space separated value to new line

echo "arg1 arg2 'arg with space'" | xargs printf '%s\n'

Transform upper case to lower case

echo "Hello" | tr '[:upper:]' '[:lower:]'

Transform lower case to upper case

echo "Hello" | tr '[:lower:]' '[:upper:]'

Transform comma separated values to each item per line

echo "1,2,4,5" | tr ',' '\n'

Squeeze multiple whitespaces into single

echo "hello there" | tr -s ' '

Converts horizontal list to vertical columns separated by tab

printf "1\n2\n3\n" | paste -s -

Converts horizontal list to comma separated values

printf "1\n2\n3\n" | paste -sd, -

Total the list of values numbers

printf "1\n2\n3\n" | paste -sd+ - | bc -l

Combines two lines into one line

printf '1\n2\n3\n4\n5' | paste -d ' ' - -

Generate UUID

$ uuidgen

$ echo `uuidgen`

Base64 encode and decode

Encode a string

$ printf "username:password" | base64

Decode a string on unix / linux

$ printf "dXNlcm5hbWU6cGFzc3dvcmQ=" | base64 --decode

tail

tail -F myfile.txt

Outputs the last 10 lines of myfile.txt, and monitors myfile.txt for updates; tail then continues to output any new lines that are added to myfile.txt

tail -F *.log

Prints and follows the logs in all log files in the directory

tail -F */logs*/*.log

Prints and follows the logs in all log files in the matching sub directory using wildcard

find . -name '*.log' | xargs tail -F

Prints and follows the logs in all log files in the all sub directories recursively

tail -f access.log | grep 24.10.160.10

This is a useful example of using tail and grep to selectively monitor a log file in real time.

cat input.csv | tail -r

Reverses the lines in the input.csv and prints in the console

tac input.csv

Reverses the lines in a file using tac command (opposite of cat)

cat input.csv | tail -n +2

Ignores the header (first line) in the file and prints rest of the lines

cat input.csv | tail -r | tail -n +2 | tail -r

Ignores the last line in the file and prints rest of the lines in same order as it appears in the input.csv. Works only in Mac OS

head

$ cat input.csv | head -n 3

Prints only the first 3 lines of the file

$ cat input.csv | head -n -l

Ignores the last line in the file and prints rest of the lines. Doesn't work in Mac OS

wc (word count)

print total number of lines in a file

$ wc -l filename.txt

print total number of words in a file

$ wc -w filename.txt

print total count of a text in a file

$ grep -o ‘some text’ filename.txt | wc -w

find

Searches through the directories for files and directories with a given name

$ find . -name "*.txt"

Searches through the directories for files and directories with a given name (case insensitive)

$ find . -iname "*.txt"

Searches through the directories for files and directories matching the given regex

$ find . -regex '\.\/a[01][.].*'

Searches through the directories for files and directories matching the given regex (case insensitive)

$ find . -regex '\.\/a[01][.].*'

Search through the directories for files with extension 'js' and excludes files with extension '.min.js'

$ find . $ -name ".js" ! -name ".min.js" $

Search through the directories for files with extension 'js' or 'css'

$ find . $ -name ".js" -o -name ".css" $

Search through the directories for files with extension 'js' or 'css' and exclude minified files

$ find . $ -name ".js" -o -name ".css" $ ! $ -name ".min.js" -o -name ".min.css" $

Search for all files and directories matching “*.txt”, starting at the current directory (.) and working through all sub-directories, then printing the name of the file to the screen

$ find . -name "*.txt" -ls

Searches only for directories matching “some-dir-*”

$ find . -type d -name "some-dir-*"

Searches only for files matching “some-file-*”

$ find . -type f -name "some-file-*"

Excludes or hides hidden files and directories from the list

$ find . -not -path "/./" -name ".txt"

Includes only src directory and excludes test or target or build directories

$ find . -name '*.json' -path '*/src/*' -not -path '*/test/*' -not -path '*/target/*' -not -path '*/build/*'

Perform non-recursive find (doesn't search files in subdirectory)

$ find . -maxdepth 1 -name "*.txt"

Searches for text file in the specified directory

$ find /some/directory -name "*.txt"

Finds all the txt files in sub directories recursively and prints each file using cat command

$ find . -name "*.txt" | xargs cat

-print0 uses ASCII null character instead of new line to separate file names, which can be used in conjunction with xargs -0

$ find . -name "*.txt" -print0 | xargs -0 -I {} echo "File: {}"

Searches for files and ignores error message

$ find . -name "*.txt" 2>/dev/null

Searches for files and redirects error output to standard output

$ find . -name "*.txt" 2>&1

rm (remove file or directory or symbolic link)

remove a file or directory or link

$ rm -rf dirName

$ rm -rf symbolicLinkName

$ rm -rf fileName.txt

mkdir (make directory)

$ mkdir -p rootDirectory/subDirectory1/subDirectory2

creates all the above directory if it doesn’t exists and ignores if it already exists

redirection (<, >, >>)

< - read from the file
> - write into the file
>> - append at the end of the file

tee

capture stdout to a log file and also prints the log in the console

$ command | tee log.txt

capture stdout and stderr to a log file and also prints the log in the console

$ command 2>&1 | tee log.txt

use tee to append to file instead of overwrite

$ command | tee -a log.txt

cat (concatenate)

read and print the content of a file on the scree

$ cat filename.txt

sort

sort the input passed

lexically sorts the values in a file and write it to a new file

$ cat input.txt | sort > sortedOutput.txt

lexically sorts the values of a file in reverse order and displays the result

$ cat input.txt | sort -r

numerically sorts the values and displays the result

$ printf "1\n3\n01\n02" | sort -n

numerically sorts the values in reverse order and displays the result

$ printf "1\n3\n01\n02" | sort -nr

numerically sorts the space or tab separated values in reverse order by second column

$ cat sample.tsv | sort -nr -k 2

numerically sorts the csv values in rever order by second column

$ cat sample.csv | sort -nr -k 2 -t ,

uniq

capture the uniq values from a list of values

captures the unique values to a new file

$ cat input.txt | sort | uniq > uniqueOutput.txt

capture the unique values with count and displays the result

$ printf "ab\ncd\nab\nbc\ncd" | sort | uniq -c

capture the unique values with count and displays the result in ascending order

$ printf "ab\ncd\nab\nbc\ncd" | sort | uniq -c | sort -r

capture the unique values with count and displays the result in descending order

$ printf "ab\ncd\nab\nbc\ncd" | sort | uniq -c | sort -nr

seq

Creates a sequence of numbers from 1 to 5 (both inclusive)

$ seq 5

Creates a sequence of numbers from 0 to 9 (both inclusive)

$ seq 0 9

Creates a sequence of numbers by skipping 5 numbers in between

$ seq -5 5 25

Repeat a command n times with index

$ seq 5 | xargs -I {} echo "Index: {}"

Repeat multiple command n times with index

$ seq 0 4 | xargs -n1 sh -c ' echo "Command 1 : $1" && echo "Command 2 : $1" ' sh

chmod (change a file mode)

Symbol	Meaning
u	user
g	group
o	other
a	all
r	read
w	write (and delete)
x	execute (and access directory)
+	add permission
-	take away permission

$ chmod go-rwx biglist

To remove read write and execute permissions on the file/directory biglist for the group and others

$ chmod a+rw biglist

To give read and write permissions on the file/directory biglist to all,

$ chmod u+rwx,g+rw,o-rwx test.file

Set multiple permission in single command

$ chmod a+rwx directory/*

To give permission to all files in a directory

$ chmod -R a+rwx directory/*

To give permission to all files/directories in a directory and its sub directories

ls (list directory)

$ ls

Display all directory contents as tab separated values

$ ls -1

Display all directory contents each on new line

$ ls -l

Display all contents as detailed list

$ ls -al

Display all contents including hidden files and directories (starting with .)

$ ls -alt

Display contents by sorting on last modified time, with recently modified files on top

$ ls -altr

Display contents by sorting on last modified time, with recently modified files at bottom

ln -s (soft link or symbolic link)

ln -s /opt/source/some_executable /usr/local/bin/some_executable

Creates a soft link for some_executable located in /opt/source to /usr/local/bin/some_executable

ln -sfn /opt/source/some_executable /usr/local/bin/some_executable

Creates or overwrites any existing soft link

Information about Linux

uname

prints operation system name or kernel name

uname -m

prints hardware arch type of the linux. x86_64 stands for x64 architecture

uname -a

prints all information about hardware and software

cat /etc/os-release

prints linux distribution type, e.g: ubuntu, centos, fedora etc

grep MemTotal /proc/meminfo

prints the total RAM size of the machine

cat /proc/cpuinfo

prints CPU processor and available cores

Processes and Jobs

ps -Ae

prints parent process status for all users with minimal formatting. Doesn't display uid and parent pid

ps -Ae | grep -i intellij | grep -v grep

search for process using grep

ps -Aef

prints process status for all users with full formatting

ps -Aef | grep -i intellij | grep -v grep | tr -s ' ' | cut -d ' ' -f2

extracts and prints the pids of the processes matching grep

kills the processes matching grep by extracting the pids and killing with kill command

kills the processes using sudo

ps aux --sort -%cpu | head -n 12

Lists the processes using most cpu on Linux

ps aux -r | head -n 12

Lists the processes using most cpu on Mac

ps aux --sort -%mem | head -n 12

Lists the processes using most ram memory on Linux

ps aux -m | head -n 12

Lists the processes using most ram memory on Mac

# on Mac

top -o MEM

# on Linux

top -o %MEM

Lists the processes sorted by Memory. Press q to quit

# on Mac

top -o CPU

# on Linux

top -o %CPU

Lists the processes sorted by Memory. Press q to quit

kill -9 26152

Force kills the process with process id 26152

pgrep -i idea

Search for processes as case-insensitive and print the process id

pgrep -ilf intellij

Search for processes as case-insensitive with full form of process with path and parameters

pkill -i idea

Kills the processes matching the process with case-insensitive pattern

pkill -ilf intellij

Kills the processes as case-insensitive with full form of process with path and parameters

command 2>&1

run command and send error output to standard output

command &

run command in background

command > /dev/null &

run command in background and hide the standard output of background process from terminal (redirect it to /dev/null)

command > /dev/null 2>&1 &

run command in background and hide both standard and error output of background process from terminal (redirect it to /dev/null)

nohup command > /dev/null 2>&1 &

run command in background and detach it from the terminal. Both standard and error output will be redirected to /dev/null (external sink)

sudo su

nohup command > /dev/null 2>&1 &

exit

run command in background as sudo user

kill the job running in the foreground

suspend the job running in the foreground

background the suspended job

jobs

foreground job number 1

kill %1

kill job number 1

df

df -h

reports file system disk space usage in a human readable format

du

estimates file space usage

$ du -s *

reports size of each sub directory (summary)

$ du -sh *

reports size of each sub directory in human readable format

$ du -sh * | sort -h

reports size of each sub directory in human readable format and sort it by ascending order

$ du -sh * | sort -rh

reports size of each sub directory in human readable format and sort it by descending order

$ du -sh .

reports size of current directory in human readable format

$ du -h * | sort -h

reports size of each directory and files and sorts the result in ascending order by size

$ du -h * | sort -rh

reports size of each directory and files and sorts the result in descending order by size

$ du -ah * | sort -rh

reports size of individual files and sorts the result in descending order by size

lsof

$ lsof

Lists all open files used by different processes in the system

$ sudo lsof -s | awk '{print $7,$2,$9}'| sort -rn | less -S

List all open files and sort the result in descending order of file size and print the result with <file size> <pid> <file name / location>

$ sudo lsof -p 9240

List all open files by the given process id

gzip & gunzip (compression)

gzip filename.txt

Compresses the file and create a new file as filename.txt.gz

gunzip filename.txt.gz

Decompresses the gz file and extracts the actual file

zip & unzip

zip -r ./output_file.zip ./source_dir

zips the source_dir with all its contents (recursively)

zip -u ./output_file.zip ./file1.txt ./file2.txt

zips the given files

unzip ./input_file.zip -d ./target_dir

unzips the file in the target directory

tar

Creation commands:

$ tar -cvf /target/directory/file.tar /source/directory

Creates a tar file

$ tar -zcvf /target/directory/file.tar.gz /source/directory

Creates a tar file zipped with gzip format

Extraction commands:

$ tar -xvf archive.tar

Decompresses the tar file and extracts the actual file / directory in current location

$ tar -xvzf archive.tar.gz

Decompresses the tar file zipped with gzip format in current location

$ tar -xvf archive.tar -C /target/directory

Decompresses the tar file and extracts the actual file / directory in specified location

zcat

will read the compressed file without needing them to uncompress

zcat science.txt.gz

The output can be piped to less to read

zcat science.txt.gz | less

touch

Create an empty file as

touch ./blank_file.txt

Modify the last updated timestamp of a file without modifying its content

touch ./existing_file.txt

wget (download files from url)

$ wget "https://www.example.com/download.zip" -P /download/directory

Downloads the binary file from the url to the path specified by -P argument

$ wget "https://www.example.com/download.zip" -O /directory/filename.zip

Downloads the binary file from the url to the path and filename specified by -O argument

$ wget "https://www.example.com/download.zip" -P /download/directory --user-agent="User Agent String"

Override user agent in the request by wget command

$ wget "https://www.example.com/download.zip" -P /download/directory --no-cookies --header="Cookie: name1=value1; name2=value2"

Make wget command to pass cookies (session or other authentication) in the request

$ wget "https://www.example.com/download.zip" -P /download/directory --wait=60

Make wget to wait for 60 seconds before download

$ wget "https://www.example.com/download.zip" -P /download/directory --waitretry=60

Make wget to wait for 60 seconds before download on a failed request

$ wget "https://www.example.com/download.zip" -P /download/directory --wait=1 --random-wait

Make wget to wait for random times between 0.5 and 1.5 * wait in seconds. In this case 1 second

$ wget "https://www.example.com/download.zip" -P /download/directory -e http_proxy=127.0.0.1:8080 -e https_proxy=127.0.0.1:8080 --proxy-user=user --proxy-password=password

Make wget use proxy with authentication to download the file

file

Display file, directory and its type in the current directory

$ file *

diff

Compares the content of two files and displays the differences

$ diff file1 file2

Lines beginning with a < denotes file1 and lines beginning with a > denotes file2

Display the differences differences side by side

$ diff -y file1 file2

Display the differences by ignoring white space

$ diff -wy file1 file2

If the diff output is too long then pipe it to less

$ diff -wy file1 file2 | less -S

history

Shows command history list

history -cw

Clears history of commands from current session

history -c

Clears all history of commands written to history file in bash

history -p

Clears all history of commands written to history file in zsh

rm -rf ~/.zsh_history && history -p && unset HISTFILE && killall Terminal

Clears all history on Mac OS using zsh

!!

recall last command

!-3

recall 3rd most recent command (from bottom of the list)

!5

recall 5th command in the list (from top of the list)

!grep

recall last command starting with grep

sudo !-4

recall 4th most recent command with sudo

scp

$ scp username@host:~/directory1/directory2/file.txt .

$ scp username@host:directory1/directory2/file.txt .

$ scp username@host:/directory1/directory2/file.txt .

Copy a file from remote server to current directory

$ scp local_file.txt username@host:~/directory1/directory2

$ scp local_file.txt username@host:directory1/directory2

$ scp local_file.txt username@host:/directory1/directory2

Copy a file from current directory to remote server

$ scp -r username@host:~/directory1/directory2 .

$ scp -r username@host:directory1/directory2 .

$ scp -r username@host:/directory1/directory2 .

Copy a directory and all its sub directories from remote server to current directory

$ scp -r local_directory username@host:~/directory1/directory2

$ scp -r local_directory username@host:directory1/directory2

$ scp-r local_directory username@host:/directory1/directory2

Copy a directory and all its sub directories from current directory to remote server

$ scp -i ~/.ssh/different-private-key.pem remote-host:/remote-directory/file.txt /local-directory

Use the specified private key to transfer files from remote server.

cp (Copy command)

$ cp dir1/file1.txt dir2

Copy file1.txt from dir1 to dir2

$ cp dir1/file1.txt dir2/file2.txt

Copy file1.txt from dir1 as file2.txt in dir2

$ cp -r dir1/temp dir2

Copy temp directory and all sub directory to dir2

$ cp -r dir1/temp1 dir2/temp2

Copy temp1 directory and all sub directory to dir2 with new directory name as temp2

mv (Move command)

$ mv dir1/file1.txt dir2

Move file1.txt from dir1 to dir2

$ mv dir1/file1.txt dir2/file2.txt

Move file1.txt from dir1 as file2.txt in dir2

$ mv dir1/temp dir2

Move temp directory to dir2

$ mv dir1/temp1 dir2/temp2

Move temp1 directory and all sub directory to dir2 with new directory name as temp2

date

Print UTC date and time

$ date -u +%FT%TZ

Print UTC date and time in offset format

$ date -u +%FT%T%z

Print local offset date and time

$ date +%FT%T%z

Print current utc date

$ date -u +%F

Print current local date

$ date +%F

Print current epoch seconds

$ date +%s

Print epoch seconds minus 60 seconds

$ current_epoch=$(date +%s) && echo "$current_epoch - 60" | bc

Print current epoch milliseconds

$ current_epoch=$(date +%s) && echo "$current_epoch * 1000" | bc

Print current epoch milliseconds minus 60 seconds

$ current_epoch=$(date +%s) && echo "($current_epoch - 60) * 1000" | bc

Convert epoch seconds to UTC date time

$ date -r 1541408691 -u +%FT%TZ

Convert Offset date time to UTC date time

$ date -r $(date -jf '%Y-%m-%dT%H:%M:%S%z' '2018-11-04T14:00:00+0530' +%s) -u +%FT%TZ

Convert Offset date time to UTC offset date time

$ date -r $(date -jf '%Y-%m-%dT%H:%M:%S%z' '2018-11-04T14:00:00+0530' +%s) -u +%FT%T%z

Convert Offset date time to epoch seconds

$ date -jf '%Y-%m-%dT%H:%M:%S%z' '2018-11-04T14:00:00+0530' +%s

Convert UTC date time to epoch seconds

$ date -jf '%Y-%m-%dT%H:%M:%SZ' '2018-11-04T14:00:00Z' +%s

Convert UTC date time to epoch milliseconds

$ epoch=$(date -jf '%Y-%m-%dT%H:%M:%SZ' '2018-11-04T14:00:00Z' +%s) && echo "$epoch * 1000" | bc

Calculate UTC date time minus 10 days

$ current_epoch=$(date +%s) && past_epoch=$(echo "$current_epoch - (10 * 24 * 60 * 60)" | bc) && date -r $past_epoch -u +%FT%TZ

Calculate UTC date time plus 10 days

$ current_epoch=$(date +%s) && future_epoch=$(echo "$current_epoch + (10 * 24 * 60 * 60)" | bc) && date -r $future_epoch -u +%FT%TZ

Environment Variables

Print an environment variable

$ echo $OSTYPE

Examples of environment variables

USER (your login name)
HOME (the path name of your home directory)
HOST (the name of the computer you are using)
ARCH (the architecture of the computers processor)
DISPLAY (the name of the computer screen to display X windows)
PRINTER (the default printer to send print jobs)
PATH (the directories the shell should search to find a command)

List all environment variables

$ printenv | less -N -S

Shell Variables

List all shell variables

$ set | less -S

Set a shell variable

$ set history = 200

Print a shell variable

$ echo $history

Conditional if then else

If then
The following statement checks if a file or directory exists, then executes the action

$ if [ -e /tmp/testdir ]; then echo "'/tmp/testdir' exists"; fi

If not then
The following statement checks if a file or directory doesn't exists, then executes the action

$ if [ ! -e /tmp/testdir/testfile.txt ]; then echo "'/tmp/testdir/testfile.txt' doesn't exists"; fi

If then else
The following statement checks for presennce of a file or directory, then executes the action accordingly

$ if [ -e /tmp/testdir ]; then echo "'/tmp/testdir' exists"; else echo "'/tmp/testdir/' doesn't exist"; fi

While loop

Iterate on each line in a file

$ cat /tmp/text.txt | (while read line; do echo "Hello $line"; done;)

Iterate on each line in a variable

$ printf "line1\nline2\n" | (while read line; do echo "Hello $line"; done;)

Infinite loop (until Ctrl+C is pressed)

( while true; do ./my_script.sh; sleep 5; done; )

For loop

For loop with counter

$ for (( c=0; c<5; c=$c+1 )); do echo $c; done;

printf

Prints lines formatted with carriage return

$ printf "line1\nline2\n"

Prints lines formatted with columns separated by tabs

$ printf "1,1\t1,2\n2,1\t2,2\n"

Padding number with zero

$ x=25; padded_x=$(printf '%04d' $x); echo $padded_x

Pretty Print Json

Pretty print json content

$ cat some_json_file.json | jq .

$ cat some_json_file.json | python -m json.tool

Reformat all json files

$ find . -type f -name '*.json' | xargs -I {} sh -c ' cat {} | jq . > {}.bkp && mv {}.bkp {} '

Pretty Print Xml

Pretty print xml content

$ cat some_xml_file.xml | xmllint --format -

$ cat some_xml_file.xml | tidy -iq -xml

passwd

Command to change the password of the currently logged in user

Loops with pipe streams

Simple loop which iterates through the stream and duplicates each line from the csv file

$ cat file.csv | { while read CSV; do echo ${CSV}; echo ${CSV}; done }

Chaining multiple loops using pipe, which transforms given input to upper case and displays only first column

$ cat file.csv | { while read CSV; do echo ${CSV} | tr '[:lower:]' '[:upper:]'; done } | { while read CSV; do echo ${CSV} | cut -d, -f1; done }

Nested loop which iterates through the stream and joins the upper case transformed value of first file with each line of second file

$ cat file1.csv | { while read CSV1; do cat file2.csv | { while read CSV2; do UP_VAR=$(echo ${CSV1} | tr '[:lower:]' '[:upper:]'); echo "${UP_VAR},${CSV2}"; done } done }

Download all jpg images in a sequence of pages

$ seq 1 5 | xargs -I{} echo "https://www.example.com/page-{}.html" | xargs -I{} echo "sleep 0.5 && curl -s {}" | bash | grep -oE 'src="http[s]?[^"]*"' | grep -oE 'http[s]?[^"]*' | grep -E '\.jpg$' | xargs -I{} echo "sleep 0.25 && wget {} -N -P ~/Downloads/test" | bash

Aliases

Creating an alias composed of multiple commands

$ alias git_pull='git fetch origin && git checkout master && git pull'

Usage

$ git_pull

Creating an alias with multiple arguments

$ alias git_create_tag='f(){ git tag $1 $2 && git push origin $1; unset -f f; }; f'

Usage

$ git_create_tag tag_name commit_id

Creating an alias with unbounded arguments

$ alias echo_args='f(){ echo "Args: $@"; unset -f f; }; f'

Usage

$ echo_args arg1 arg2 arg3