Saturday 25 May 2013

Text Processing Tools in Linux

The following command are often used when writing scripts to automate tasks.

The diff command :- to compare the content of two files for differences.

diff file1.conf file2.conf > output.txt
cat output.txt
Options are -c, -u and -r.

The patch command :- ued to apply simple patch file to a single file.

patch -b file1.conf < output.txt

The grep command :- displays the lines in a files that match pattern.

grep root /etc/passwd
ps -aux | grep sshd
Options are -i, -n, -r, -c, -v and -l.

The cut command :- ued to cut fields or colomns of text from a file and display standard output.

cut -f3 -d: /etc/passwd
/sbin/ip addr | grep 'inet' | cut -d ' ' -f6 | cut -d / -f1
Options are -d, -f and -c.

The head command :- displays first few lines of a file.

head /etc/passwd
head -n 3 /etc/passwd
Options is -n.

The tail command :- displays last few lines of a file.

tail -n 3 /etc/passwd
tail -f /etc/passwd
tail -f will continue to show updates until Ctrl+c is pressed.

The wc command :- counts the number of lines(l), words(w), bytes(c) and characters(m) in a file.

wc -l file1.conf
ls /tmp | wc -l
Options are -l, -w, -c and -m.

The sort command :- used to sort text data.

grep bash /etc/passwd | cut -d: -f1 | sort
options are -n, -k and -t.

The uniq command :- removes duplicate adjacent lines from a file.

cut -d: -f7 /etc/passwd | sort | uniq -c
Options are -u and -d

The echo command :-  output strings.

echo This is a test.
echo This is a test. > output.txt
cat output.txt

The cat command :- output or concatenate files.

cat file1.conf > output.txt
cat output.txt
cat file1.conf file2.conf | less
Options are -b, -n, -s, -v, -t, -e and -A.

The paste command :- join multiple files horizontally.

paste file1.conf file2.conf
Options are -d and -s.

The split command :- split files based on context lines.

split -l 500 myfile segment
split -b 40k myfile segment
Options are -l (line no) and -b(bytes).

The comm command :- to compare two files for common and distinct lines.

comm file1.conf file2.conf

The dirname command :-  it will delete any suffix beginning with the last slash ('/') character and return the result.

dirname /etc/httpd/conf/httpd.conf

The fold command :- used for for making a file with long lines more readable on a limited width terminal.
fold -w 30 file.txt

The sed command :- reads text input, line by line, and allows midfication.

sed 'word;wordtoreplace' < file1.conf > output.txt
sed '/word/ d' filename > output.txt
sed 's/word/ /g' filename > output.txt
sed 's/firstword//g; s/secondword//g' yourfile > output.txt

The awk command :- used as a data extraction and reporting tool.

awk "/word/" filename > output.txt

The less command :- used to view the contents of a text file one screen at a time.

less /etc/passwd

Friday 24 May 2013

How to scp, ssh and rsync without prompting for password

For example, login in to some host linux box as root and pass the command as,

ssh-keygen -t rsa

Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/
The key fingerprint is:
35:50:28:70:2b:b0:9f:01:9a:8b:cb:0e:17:89:1d:a2 root@InVImFTSrv
The key's randomart image is:
+--[ RSA 2048]----+
|  o ... .o.      |
| o + .....       |
|+ o o ..  o      |
|o= + +   . .     |
|E + o   S        |
|.. .             |
|o..              |
|o.               |
| .               |

now login to the host linux box as root and copy,

scp -r root@ /root/.ssh/

cd /root/.ssh/
cat >> authorized_keys
chmod 700 /root/.ssh/authorized_keys

now form try ssh, scp or rsync as,

scp -r root@ /usr/local/src/

It won't prompt for any password.