No More Syslog Messages on Your Console

dmesg -n level

Start a New Xserver

startx fvwm2 -- :2 -depth 16

xlock as Screensaver

xautolock -exit 2>/dev/null; xautolock -corners "+---" -time 10 -notify 30 -locker "xlock -mode random"

Make Interactive Services Scripable with expect

#!/usr/bin/expect
set myserver "10.0.0.1"
set myport   "21"
send "Try to connect to server $myserver port $myportn"
spawn telnet $myserver $myport
expect   "220 " { send "USER ftpn" }
expect   "331 " { send "PASS foon" }
expect   "230 " { send "PASVn" }
expect   "227 " { send "HELPn" }
expect   "214 " { interact }

Download a File and Split It on the Fly

wget ftp://... -nv -O - | split ....

Sync Two Files (Directories)

/usr/bin/rsync --rsync-path=/usr/local/bin/rsync -rlpe ssh QUELLE ZIEL
rsync --compress --progress --no-whole-file -rlpe ssh remote:/path/file file

List Files Sorted by Filesize

ls -lS

Get file attributes

Get file attributes which can be easily parsed

stat myfile.txt

Count from ALPHA to OMEGA

seq ALPHA OMEGA

Text filter

Today's Messages from /var/log/syslog

cat /var/log/syslog | grep ^"`date +%b %d`"

Format of date output

# date +'%Y%m%d_%H%M%S'
20081103_1759026

Filter Empty Lines and Comments

grep -ve ^ *#'|'^$ /etc/*.conf

Highlight what machted

echo "ABC123DEF456GHI" | grep --color [0-9H]
ABC'''123'''DEF'''456'''G'''H'''I

List each match

echo "ABC123DEF456GHI" | grep -E --only-matching [0-9H]+
123
456
H

List only the names of the files where hits where found

grep --files-with-match foo *

References in sed to machted parts

echo "ABC123DEF456GHI" | sed 's/[0-9H]+/_&_/g'

ABC_123_DEF_456_G_H_I
echo "ABC223DEF456GHI" | sed s/"([0-9]+)"/"_1_"/

ABC_223_DEF456GHI

use awk instead of grep

If you use grep to find lines and the awk to print columns

grep error /var/log/syslog | awk '{ print $3 }'

you can save the grep call as awk can also grep for you

awk '/error/ { print $3 }' /var/log/syslog

Regular Expression Lookahead

If you have the following String

EXAMMLE

and want to replace the second M with a P.

Positiv Lookahead

Search for an M which is followed by an L

M(?=L)

The difference to this one

is that the the L in the first case would not be replaced, but in the second case it would be replaced

Negativ Lookahead

Search for an M, which is not followed by an M

M(?!M)

Search for a String which does not start with CA

^(?!CA)

See also Regular Expressions Lookaround

awk

Use different field separator for awk

awk 'BEGIN {FS=" - "} {print $1 $2 $3}'
awk 'BEGIN {FS="t"}  {print $3}

Split a String with awk at a given separator

echo foo,bar,test | awk 'BEGIN {FS=","} {split($0,a,","); for (i=1; i<=NF; i++) print a[i]; }'

Calculate with awk

echo 42 11 | awk '{print $1*1000, $2}'

Add all values in a column

cat foo | awk '{mycol2 += $2; mycol3 += $3}; END {print mycol2":"mycol3}'

Only deal with lines which match a given condition

Only deal with lines where the first column is larger than 45

cat foo | awk '$1>45 {print $0}'

Only deal with lines beginning with line number 12

cat foo | awk 'NR>11 { print $0 }'

Only deal with lines where in the second column is the word "FooBar":

cat foo | awk '$2~/FooBar/ {print $0}'

Define output format for awk

echo 1 2 | awk '{ printf("%.3d %sn", $1, $2)  }'

Follow (file) content

View File Without Linewarp

less -S

Restart process and print its output for each restart

watch -n3 cat /proc/mdstat

Console Calculation

echo "1 + 2" | bc
echo "1 / 3" | bc -l

Differences between two files

diff -ubB FOO BAR

Test a string vs a regular expression

expr FooBar-5.2 : ".*-5.2"

Rename files with mmv

Rename *a* to *b*:

mmv "*a*" "#1b#2"

cp -r for some files

Copy recursively some files and create the folder structure, too.

# shopt -s globstar
# cp -v --parents -r **/*.jpeg **/*.jpg **/*.JPG **/*.png **/*.tif **/*.gif /mnt/mnt/backup_of_pictures/

Mount SMB Share

mount -t smbfs -o username=uname,password=passwd //smbserv/share /destination

Label fat32 vfat Filesystem

mlabel -i /dev/sdc1 ::FOOBAR

Kernel

Build Kernel (Traditionally)

nice --5 make dep clean bzImage modules bzlilo modules_install

Kernel Bootparameter

/usr/src/linux/Documentation/kernel-parameters.txt

Create Xface

48x48 ppm -> pgm -> pbm -> xbm
cat file.xbm | xbm2ikon > file.48
compface file.48 file.face

pdf2ps (with Password)

gs -sPDFPassword=password -sDEVICE=pswrite -dNOPAUSE -sOutputFile=out.ps in.pdf

Convert some files in a pretty looking file suitable for printing

a2ps --medium A4dj -E -o /tmp/pretty.ps A B C D ...
Unicode / utf-8: u2ps ...([[http://packages.debian.org/gnome-u2ps | gnome-u2ps ]])

Create a Postscript Book

psbook normal.ps | psnup -2 -m25 > book.ps

Multimedia

vob2avi

Convert VOB files with Transcode into another video format.

transcode -i myVobFolder -o movie.avi -y xvid -x vob

Scale

Select a different audio channel, don't select a non existing audio channel or your VOB files will only be read once and no content will be generated

Encoder bitrate

-w 1800

Clipping, this cuts the first and last 16 rows of

-j 16,0

Linear interpolation for interlaced videos

Cut the results

avisplit -c -i movie.avi -t 00:00:00.00-00:05:10.00,00:11:59.00-00:22:41.00

Use tcaud

-y xvid,tcaud

Create DivX files

-y ffmpeg -F mpeg4

Use mencoder instead of transcode

cat *.vob | mencoder -oac mp3lame -ovc xvid -xvidencopts pass=1             - -o movie.avi;
cat *.vob | mencoder -oac mp3lame -ovc xvid -xvidencopts pass=2:bitrate=700 - -o movie.avi;

mkv2avi

mencoder -oac pcm -ovc copy  -ofps 30000/1001 -o Movi.avi Movi.mkv

Extract audio from video

nice transcode -i vts_01_6.vob -o soundtrack.wav -y null,wav -x vob -e 44000
mplayer -ao pcm:fast:file=audio.wav -vo null -vc null video.avi

jpg2avi

Mplayer

mencoder 'mf://*.jpg'                      -mf fps=10:type=jpg -xvidencopts bitrate=1600 -ovc xvid -oac copy -o output.avi
mencoder mf:///tmp/foo/*.jpg -vf rotate=1 -mf  fps=5:type=jpg -lavcopts vcodec=mpeg4    -ovc lavc -oac copy -o output.avi

ImageMagick

animate -delay 10 *.jpg
animate -help

convert -delay 10 *.jpg out.gif

Automated picture manipulation

ImageMagick

convert large.jpg -resize 120x120 small.jpg
convert large.jpg -resize 50% small.jpg

Got pictures from different cameras from one event? Use the exif header in the JPEG pictures to extract the date and time and put it into the filename so you can sort the files chronologically. Make a backup and check you don't overwrite files!

for i in *.jpg; do d=`exif -t 0x9003 "$i" | grep Value | sed s/^"  Value: "// | sed s/ /_/g | sed s/:/-/g`; mv "$i" "$d""_""$i"; done

Change the date from 2005 to 2006

for i in *;
 do
  n=`exif            -t 0x9004                    "$i"  | grep ^"  Value: " | sed s/^"  Value: "// | sed s/2005/2006/`;
  x="exif --ifd=EXIF -t 0x9004 --set-value="$n" $i";
  eval "$x";
done
 

Increase time by one hour

for i in *.jpg;
do
 value=`exif -t 0x9003 "$i" | grep Value | sed s/^"  Value: "//;`
 value_date=`echo "$value" | awk '{ print $1 }'`
 value_time=`echo "$value" | awk '{ print $2 }'`
 value_date_yyyy=`echo "$value_date" | awk 'BEGIN {FS=":"} {print $1}'`
 value_date_mm=`echo "$value_date" | awk 'BEGIN {FS=":"} {print $2}'`
 value_date_dd=`echo "$value_date" | awk 'BEGIN {FS=":"} {print $3}'`
 value_time_hh=`echo "$value_time" | awk 'BEGIN {FS=":"} {print $1}'`
 value_time_mm=`echo "$value_time" | awk 'BEGIN {FS=":"} {print $2}'`
 value_time_ss=`echo "$value_time" | awk 'BEGIN {FS=":"} {print $3}'`

 value_time_hh=`expr $value_time_hh + 1`
 if [ $value_time_hh -lt 10 ]; then
  value_time_hh="0$value_time_hh"
 fi

 newvalue="$value_date_yyyy:$value_date_mm:$value_date_dd $value_time_hh:$value_time_mm:$value_time_ss"

 echo "$i" "$value" " -> "  "$newvalue"

 exif --ifd=EXIF -t 0x9003 --set-value "$newvalue" "$i"
done

unix2dos, dos2unix

alias unix2dos='recode lat1..ibmpc'
alias dos2unix='recode ibmpc..lat1'

Convert file encoding from ISO to Unicode

iconv --from-code=ISO-8859-1 --to-code=UTF-8 ./oldfile.htm > ./newfile.html

ACL

setfacl -m u:foo:r-- DEST
setfacl -x u:foo DEST
setfacl -m g:bar:r-- DEST
setfacl -x g:bar DEST
setfacl -m d:u:foo:r-- DIR
setfacl -m d:g:bar:r-- DIR
setfacl -m m:r-- DEST
setfacl -m d:m:r-- DIR

Be Able to Start Several Instances of Mozilla

alias my_mozilla='mozilla -remote "ping()" && mozilla -remote "openURL (,new-window)" || mozilla -splash'

Spamassassin

Bayes: How Many Mails Have Been Learned

sa-learn --dump | grep non-token data: | grep nspam$'|'nham$

Bayes: List with All Learned Words and if They Were Found in a Spam Message

sa-learn --dump | grep -v non-token data: | sort -n

Send mails from the command line

(echo "Hello world"; uuencode file1.txt ReadThis.txt; uuencode file2.txt AndThis.txt) | mailx -s subject foo@example.com

echo "Hello world" | mutt -a file1.txt foo@example.com

Upload files with sftp

If you can not use scp for any reason

sftp -o "batchmode no"  -b meinebefehle.sftp user@computer
meinebefehle.sftp:
lcd /home/foo/4upload/
cd  /tmp/uploaded/
put Backup1.img
put Backpu2.img
quit

Archiving / Backup

Size of some storage media

Name(Bytes)
DVD4590200
CDFIXME

Backup Remotehost with tar

ssh root@otherhost -c blowfish 'cd /YOURPATH; tar -f - -p -P -c --exclude=/YOURPATH/tmp --exclude=/YOURPATH/var/cache /' | dd of=/daten/backup.tar

tar multi volume

tar --multi-volume --tape-length=1024 -cf archive.tar foo // n archive.001.tar n ...
tar --multi-volume --tape-length=1024 -c -f archive.tar -f archive001.tar -f archive002.tar -f ... foo
tar --multi-volume --tape-length=1024 --new-volume-script=/usr/local/bin/tar_changer.sh -cf archive.tar foo
tar --multi-volume                    -t -f archive.tar -f archive001.tar -f archive002.tar -f ...

tar_changer.sh:

#!/bin/sh

new_vol=`echo $TAR_VOLUME | awk '{printf "%04dn", $1;}'`

new_name="$TAR_ARCHIVE"
new_name="`echo "$new_name" | sed s/"."tar$//`"
new_name="`echo "$new_name" | sed s/"."[0-9][0-9]*$//`"
new_name="$new_name"".""$new_vol"".tar"

echo "For volume $TAR_VOLUME of $TAR_ARCHIVE the next file name will be: $new_name"

if [ -e "$new_name" ]; then
 echo "Error: file already exists!"
 exit 1
else
 echo "$new_name" >&$TAR_FD
fi

Add redundancy to files

par2 create myparfiles foo*

Create a DVD image larger than 2GB

mkisofs -udf -allow-limited-size -o foo.img foo

Complete Compressed Partionbackup Into an Image

mount /dev/hda1 / -o remount,rw;
dd if=/dev/zero of=/delme1 bs=4048; rm /delme1;
mount /dev/hda1 / -o remount,ro;

Local

dd if=/dev/hda1 bs=4048 | bzip2 > /mountpoint/image_hda1.bz;

Remote


ssh host -c blowfish 'dd if=/dev/hda1 bs=4048 | bzip2' | dd of=/image_hda1.bz

Software Raid

Turn Beep Off (e.g. while autocomplete)

xterm: xset b off
setterm -blength
echo 'set bell-style visible' >> ~/.inputrc

$TERM

vt100
xterm
linux

chmod

chmod ABCD FILE

4=user ID, 2=group ID, 1=sticky

B(owner), C(group), D(other):

4=r, 2=w, 1=x

Restore Bootmanager

Backup your Bootsector

if=/dev/hda of=/tmp/mbrbackupwithouttable.img bs=446 count=1
if=/dev/hda of=/tmp/mbrbackupwithtable.img    bs=512 count=1

Lilo

Boot from Floppy/CD, mount /dev/linuxrootpartition /mnt; chroot /mnt; lilo; exit; umount /mnt; reboot

Grub

  • Start the Grub shell
  • Find out where your Linux partition is (e.g. use find to search one of your files) and tell grub with the root command where it is.
grub> find /boot/grub/stage1
grub> root (hd0,3)
  • Write Grub into the masterbootsector or in one of your partitions
grub> setup (hd0)
grub> setup (hd0,3)

yaboot

ybin

less

export LESS="-i"

Upper- and Lower Case

less

If you do a lowercase search in less the search will ignore upper and lower case

export LESS="-i"

sed

echo Cool | sed s/c/T/i

Change Upper and Lower Case

echo fOo | tr "[:lower:]" "[:upper:]" # FOO
echo fOo | tr "[:upper:]" "[:lower:]" # foo

Subversion

Subversion

Sort after special columns

First after the 3. one, then the 2. and finally the 6. columns (the last one numeric).

sort -k 3,3 -k 2,2 -k 6,6n

Do not only choose which column is used for sorting, but also define the column separator (here ":"):

# echo "1:1:1
1:1:0" | sort -t : -k 3

1:1:0
1:1:1

Bash

Bash Array

Define and initialise an array in the Bash shell. Afterwards various outputs of the array.

# fruits=(apple banana strawberry)

# echo ${fruits}
apple

# echo ${fruits[1]}
banana

# echo ${fruits[*]}
apple banana strawberry

# echo ${#fruits[*]}
3

# echo ${fruits[*]}
apple banana strawberry

# for ((idx=(${#fruits[*]}-1); idx>=0; idx--)); do echo ${fruits[$idx]}; done
strawberry
banana
apple

Bash history

BashTricks

find

find / -printf "%CY%Cm%Cd%CI%CM | %d | %i | %s | %h | %fn"

find can spawn a new command for each file or start the command once and give it all found files

find / -exec echo  '{}' +
find / -exec echo  '{}' ';'

Find old files

List files which are at least 165 days old, exactly 165 days old, not older than 165 days

find /foo -type f -ctime +165
find /foo -type f -ctime  165
find /foo -type f -ctime -165

Find large files

List files which are at least 3MB large

find /foo -size +3000000c
find /foo -size +3M

Us find to find files with given file permissions

CommandDescriptionWill findWill not find
# find . -perm 400<br /># find . -perm u=rFinds files with exactly these permissions-r---------r--r-----
# find . ! -perm 400<br /># find . ! -perm u=rFinds files whose permissions are not equal to these permissions-rwxrwxrwx-r--------
# find . -perm -440# find . -perm -u=r,g=rAt least these permissions have to be there-rwxrwxrwx<br />-rw-r--r--<br />lrwxrwxrwx-r--------<br />----r-----
# find . -perm /440<br /># find . -perm /u=r,g=rAt least one of these permissions have to be there-rwxrwxrwx<br />-r--------<br />----r-------w-------

Find dangling symlinks

find /tmp/tg1/ -type l | while read link
do
 if [ -h "$link" -a ! -e "$link" ];
  then echo "$link";
 fi;
done
find /tmp/tg1/ -printf '%Y %p n' | grep ^N  | sed s/^N //

List all files which are linked to the given file

find /mnt/mnt/ -L -samefile /mnt/mnt/bin/foo

OSX

Display and Modify Routes in OSX

netstat -rn
route delete default
route add default 1.2.3.4

Battery status

ioreg -l | grep IOBa

Detect interactive shells

if shopt -q login_shell ; then
 ...
else
 ...
fi

Jump to recently visited directories

Push a new directory to the stack and change to that directory

# pwd
/tmp

# pushd /etc/network/

# pushd /var/log/
/var/log /etc/network /tmp

# pushd /usr/sbin/
/usr/sbin /var/log /etc/network /tmp

# dirs
/usr/sbin /var/log /etc/network /tmp

A simple pushd swaps the topmost directory on the stack with the next one and changes the current directory to the new topmost one. Very handy to jump between two directories.

# pushd
/var/log /usr/sbin /etc/network /tmp

# pwd
/var/log

# pushd
/usr/sbin /var/log /etc/network /tmp

# pwd
/usr/sbin
 

With +x or -x you can select the x. directory (count starts with zero) counted from the left or the right side to be the new topmost directory.

# pushd +2
/etc/network /tmp /usr/sbin /var/log

/# pwd
/etc/network
 

With popd the topmost directory is removed from the stack and the current directory will be set to the new topmost directory. With -n the current directory is not changed.

# dirs
/etc/network /usr/sbin /var/log

# popd
/usr/sbin /var/log

# pwd
/usr/sbin

# popd
/var/log

# pwd
/var/log

With +x and -x you can again remove the x. directory counted from left or from right. If you remove the topmost directory (+0) the current directory is change to the new topmost directory.

You can also get the last directory you were before the current directory with the following variable

echo "$OLDPWD"

Command Substitution

Repeat last token of the last command

# echo /etc/group /etc/passwd &gt;/dev/null
# echo !$
echo /dev/null

With

you get the last token at the current cursor position.

Repeat the last argument of the last command

# cp backup2008a.tar /mnt/streamer/backup2008a.tar 2&gt;/dev/null
# echo $_
/mnt/streamer/backup2008a.tar

Repeat the last command with a substitution

# cp backup2008.tar /mnt/streamer/
# ^2008^2009^
cp backup2009.tar /mnt/streamer/

However, this substitutes only the first occurrence, which also applies to the long version

# cp backup2008a.tar backup2008b.tar /mnt/streamer/
# !!:s/2008/2009/
cp backup2009a.tar backup2008b.tar /mnt/streamer/

The long version can also substitute all occurrences

# cp -v backup2008a.tar backup2008b.tar /mnt/streamer/
# !!:gs/2008/2009/

Change output of a shellscript and restore output again later

exec 3>&amp;1
exec 1>>/var/log/mylogfile.txt
...
exec 1>&amp;3
exec 3&gt;&amp;-

Change file content without temporary files with sponge

Usually you have to use temporary files if you want to change the content of a file, as you can not read and write to a file simultaneously. This can be very inconvenient, especially as secure creation of temp files is not too easy. One solution for this problem is sponge:

cat foo.txt | sed s/t/T/ | sponge foo.txt

However, this does only work if sponge writes to a file. If you redirect its output this will not work. In Debian sponge can be found in the moreutils package.

cygwin complains about too open permissions

If you use ssh in cygwin it might happen that you get a complain that the permissions of your private ssh key are too open.

$ ssh foo@bar.example.com
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Permissions 0644 for '/cygdrive/p/.ssh/id_dsa' are too open.
It is recommended that your private key files are NOT accessible by others.
This private key will be ignored.
bad permissions: ignore key: /cygdrive/p/.ssh/id_dsa
Permission denied (publickey).

You can overwrite this warning like this

$ CYGWIN=nontsec ssh foo@bar.example.com

System Infos

# free -m
# df -ah
# top -n1
htop
# vmstat 5 3
smartctl --all /dev/sda
memstat -w

Who blocks a mounted drive

If you can not unmount a mounted drive find out who blocks it

lsof +D /mnt/foo
fuser /mnt/foo

Get the number of CPU cores

# nproc
6

Use this information to start make with multiple threads

# make -j $(nproc)

Benchmarks

Use zcav to measure the speed of a mass storage device depending on the position on the drive you are currently reading from

zcav /dev/sdb > zcav.txt
gnuplot
 set yrange [0:*]
 set format y '%g Blocks/s'
 set format x '%g MB'
 plot "zcav.txt" using 1:2 title "Foo"

Benchmark a drive with bonnie++

bonnie++ -u root -d /mnt/mnt/

Solaris

More verbose ps in Solaris

In Solaris there is the normal ps with

ps -ef
prstat -a
truss -f /bin/date
truss -fp PID

but you also have another ps which can be more verbose

/usr/ucb/ps -auxww

Solaris 10 md5sum

digest -v -a md5 foo

Perl

Install Perl module

# perl -MCPAN -e 'install HTML::CalendarMonthSimple'

# cpan
cpan> install HTML::CalendarMonthSimple

Test if a module is already installed

require HTML::CalendarMonthSimple;

Perl foreach

If you iterate in Perl of something

my @mystuff = ("foo", "bar");

there is automatically a variable $_ with the current iteration in it

foreach(@mystuff)
{
 print "$_";
}

Some commands use this variable if you don't name one

foreach (@mystuff)
{
   /foo/ and print "Found foo!";
}

Man kann aber natürlich auch eine Variable angeben

foreach my $x(@mystuff)
{
 print "$x";
}

See also Perl foreach looping and Perl variables.

Perl Debugger

perl -d myPerlScript.pl

s execute the next command (and step also in method calls)

n next command (but skip method calls)

Return repeat last selection

x $FOO prints the value of the variable

r run until the end of the method (or the next breakpoint)

c run until the end (or the next breakpoint)

l (small L) print the next 10 lines of code

- print the previous 10 lines of code

S print all method names

S main print all method names

b METHODENNAME Brakepoint for this method

b NUMMER Brakepoint for this line number (use l (small L) to get the line numbers)

t print the executed lines

q quit debugger

R Restart program

T Show Stacktrace

See also Perl Debugger

Use xargs to start processes in parallel

One gzip process gets a long List of files to work with

ls *.txt | xargs gzip

One gzip for each file

ls *.txt | xargs -n1 gzip

One gzip for each file, all start in parallel

ls *.txt | xargs -n1 gzip &amp;

One gzip for each file, only 5 run in parallel

ls *.txt | xargs -n1 -P5 gzip