Bash Input Output Magic

Filter and Modify

Today's Messages from /var/log/syslog

cat /var/log/syslog | grep ^"`date +%b %d`"

Format of date output

# date +'%Y%m%d_%H%M%S'

Filter Empty Lines and Comments

grep -ve ^ *#'|'^$ /etc/*.conf

Highlight what machted

echo "ABC123DEF456GHI" | grep --color [0-9H]

List each match

echo "ABC123DEF456GHI" | grep -E --only-matching [0-9H]+

List only the names of the files where hits where found

grep --files-with-match foo *

References in sed to machted parts

echo "ABC123DEF456GHI" | sed 's/[0-9H]+/_&_/g'

echo "ABC223DEF456GHI" | sed -E s/"([0-9]+)"/"_\\1_"/


use awk instead of grep

If you use grep to find lines and the awk to print columns

grep error /var/log/syslog | awk '{ print $3 }'

you can save the grep call as awk can also grep for you

awk '/error/ { print $3 }' /var/log/syslog

Regular Expression Lookahead

If you have the following String


and want to replace the second M with a P.

Positiv Lookahead

Search for an M which is followed by an L


The difference to this one

is that the the L in the first case would not be replaced, but in the second case it would be replaced

Negativ Lookahead

Search for an M, which is not followed by an M


Search for a String which does not start with CA


See also Regular Expressions Lookaround


Get the 2nd column when ; is the seperator

echo "a;bc;d" | cut -d';' -f2


Use different field separator for awk

awk 'BEGIN {FS=" - "} {print $1 $2 $3}'
awk 'BEGIN {FS="t"}  {print $3}

Split a String with awk at a given separator

echo foo,bar,test | awk 'BEGIN {FS=","} {split($0,a,","); for (i=1; i<=NF; i++) print a[i]; }'

Calculate with awk

echo 42 11 | awk '{print $1*1000, $2}'

Add all values in a column

cat foo | awk '{mycol2 += $2; mycol3 += $3}; END {print mycol2":"mycol3}'

Only deal with lines which match a given condition

Only deal with lines where the first column is larger than 45

cat foo | awk '$1>45 {print $0}'

Only deal with lines beginning with line number 12

cat foo | awk 'NR>11 { print $0 }'

Only deal with lines where in the second column is the word "FooBar":

cat foo | awk '$2~/FooBar/ {print $0}'

Define output format for awk

echo 1 2 | awk '{ printf("%.3d %sn", $1, $2)  }'


echo Cool | sed s/c/T/i

Change Upper and Lower Case

echo fOo | tr "[:lower:]" "[:upper:]" # FOO
echo fOo | tr "[:upper:]" "[:lower:]" # foo

Sort after special columns

First after the 3. one, then the 2. and finally the 6. columns (the last one numeric).

sort -k 3,3 -k 2,2 -k 6,6n

Do not only choose which column is used for sorting, but also define the column separator (here ":"):

# echo "1:1:1
1:1:0" | sort -t : -k 3



Sync Two Files (Directories)

/usr/bin/rsync --rsync-path=/usr/local/bin/rsync -rlpe ssh QUELLE ZIEL
rsync --compress --progress --no-whole-file -rlpe ssh remote:/path/file file

List Files Sorted by Filesize

ls -lS

Get file attributes

Get file attributes which can be easily parsed

stat myfile.txt

unix2dos, dos2unix

alias unix2dos='recode lat1..ibmpc'
alias dos2unix='recode ibmpc..lat1'

Convert file encoding from ISO to Unicode

iconv --from-code=ISO-8859-1 --to-code=UTF-8 ./oldfile.htm > ./newfile.html

pdf2ps (with Password)

gs -sPDFPassword=password -sDEVICE=pswrite -dNOPAUSE in.pdf

Convert some files in a pretty looking file suitable for printing

a2ps --medium A4dj -E -o /tmp/ A B C D ...
Unicode / utf-8: u2ps ...([[ | gnome-u2ps ]])

Create a Postscript Book

psbook | psnup -2 -m25 >


find / -printf "%CY%Cm%Cd%CI%CM | %d | %i | %s | %h | %fn"

find can spawn a new command for each file or start the command once and give it all found files

find / -exec echo  '{}' +
find / -exec echo  '{}' ';'

Find old files

List files which are at least 165 days old, exactly 165 days old, not older than 165 days

find /foo -type f -ctime +165
find /foo -type f -ctime  165
find /foo -type f -ctime -165

Find large files

List files which are at least 3MB large

find /foo -size +3000000c
find /foo -size +3M

Us find to find files with given file permissions

CommandDescriptionWill findWill not find
# find . -perm 400<br /># find . -perm u=rFinds files with exactly these permissions-r---------r--r-----
# find . ! -perm 400<br /># find . ! -perm u=rFinds files whose permissions are not equal to these permissions-rwxrwxrwx-r--------
# find . -perm -440# find . -perm -u=r,g=rAt least these permissions have to be there-rwxrwxrwx<br />-rw-r--r--<br />lrwxrwxrwx-r--------<br />----r-----
# find . -perm /440<br /># find . -perm /u=r,g=rAt least one of these permissions have to be there-rwxrwxrwx<br />-r--------<br />----r-------w-------

Find dangling symlinks

find /tmp/tg1/ -type l | while read link
 if [ -h "$link" -a ! -e "$link" ];
  then echo "$link";
find /tmp/tg1/ -printf '%Y %p n' | grep ^N  | sed s/^N //

List all files which are linked to the given file

find /mnt/mnt/ -L -samefile /mnt/mnt/bin/foo

Pretty Print JSON

sudo apt-get install yajl-tools
cat demo.json | json_reformat

This runs with basic python be reorders elements in json

cat demo.json | python -mjson.tool


Make Interactive Services Scripable with expect

set myserver ""
set myport   "21"
send "Try to connect to server $myserver port $myportn"
spawn telnet $myserver $myport
expect   "220 " { send "USER ftpn" }
expect   "331 " { send "PASS foon" }
expect   "230 " { send "PASVn" }
expect   "227 " { send "HELPn" }
expect   "214 " { interact }

Download a File and Split It on the Fly

wget ftp://... -nv -O - | split ....


View File Without Linewarp

less -S

If you do a lowercase search in less the search will ignore upper and lower case

export LESS="-i"

Differences between two files

diff -ubB FOO BAR

Rename files with mmv

Rename *a* to *b*:

mmv "*a*" "#1b#2"

cp -r for some files

Copy recursively some files and create the folder structure, too.


Bayes: How Many Mails Have Been Learned

sa-learn --dump | grep non-token data: | grep nspam$'|'nham$

Bayes: List with All Learned Words and if They Were Found in a Spam Message

sa-learn --dump | grep -v non-token data: | sort -n

Send mails from the command line

(echo "Hello world"; uuencode file1.txt ReadThis.txt; uuencode file2.txt AndThis.txt) | mailx -s subject

echo "Hello world" | mutt -a file1.txt


Multiple ssh connections to one host share an reuse one channel

Host *
IdentityFile ~/.ssh/id_rsa
ForwardAgent yes
ControlMaster auto
ControlPath /tmp/control-%r@%h:%p
ControlPersist 18000
ServerAliveInterval 60

Upload files with sftp

If you can not use scp for any reason

sftp -o "batchmode no"  -b meinebefehle.sftp user@computer
lcd /home/foo/4upload/
cd  /tmp/uploaded/
put Backup1.img
put Backpu2.img


Count from ALPHA to OMEGA


Console Calculation

echo "1 + 2" | bc
echo "1 / 3" | bc -l

Restart process and print its output for each restart

watch -n3 cat /proc/mdstat

Test a string vs a regular expression

expr FooBar-5.2 : ".*-5.2"
# shopt -s globstar
# cp -v --parents -r **/*.jpeg **/*.jpg **/*.JPG **/*.png **/*.tif **/*.gif /mnt/mnt/backup_of_pictures/

Bash Array

Define and initialise an array in the Bash shell. Afterwards various outputs of the array.

# fruits=(apple banana strawberry)

# echo ${fruits}

# echo ${fruits[1]}

# echo ${fruits[*]}
apple banana strawberry

# echo ${#fruits[*]}

# echo ${fruits[*]}
apple banana strawberry

# for ((idx=(${#fruits[*]}-1); idx>=0; idx--)); do echo ${fruits[$idx]}; done

Bash Maps

(requires bash 4)

declare -A animals
animals=( ["moo"]="cow" ["woof"]="dog")

# Show all keys
echo "${!animals[@ ]}"

# Show all values
echo "${animals[@ ]}"

# Iterate
for sound in "${!animals[@ ]}"; do
 echo "$sound - ${animals[$sound]}";

If you want to fill an array or a map via a (while) loop you can not use the ... | while read line syntax but have to use the while read line ... done < <(cat ...) syntax

Run things in parallel

Use xargs to start processes in parallel

I very often process lots of things (e.g. files) in bash commands. Sometimes that is not every fast. Either because IO is involved (so you often have to wait) or because it is CPU intensive and you only use one thread of your CPU. Surprisingly xargs can be very helpful to achieve this. Let's assume you process many things. You want to have always 8 threads running in parallel (maybe because your CPU can handle 8 threads) and speed things up, every thread you get a batch of 10 things to process. at once. For this example we generate the input with seq (which just counts) but I could be anything, files, a rows of file, output of other commands, etc.

seq 1 100 | xargs -n10 -P8 /tmp/

That would start 8 instances of the script, and give each instance 10 numbers. The first 1-10, seconds 11-20, etc. As soon as one of the 8 scripts finishes, the next one is started. So you always have 8 threads running in parallel, no more, no less. Until you run out of input. If each thread should only handle one input, use -n1, if you only want to have one thread, use -P1.

This here for example zips all the files, one by one, but does 5 files in parallel

One gzip for each file, only 5 run in parallel

ls *.txt | xargs -n1 -P5 gzip

Get the number of CPU cores

# nproc

make in parallel

Use this information to start make with multiple threads

# make -j $(nproc)