Thursday, 4 August 2016

How To Fetch Specific Field In Multi Character Delimiter File Using AWK

In earlier articles we have seen awk command to fetch a specific field where we have single character delimiter. Here we will see how can we fetch specific field where data is separated by multi character delimiter.

$ echo "1001***A1***HR***5000" | awk -F'***' '{print $1}' 

Output:
awk: There is a regular expression error.
        ?, *, or + not preceded by valid regular expression


$ echo "1001***A1***HR***5000" | awk -F'[*][*][*]' '{print $1}' 
Output:
1001

You can achieve the same by cut command as given below:

$ echo "1001***A1***HR***5000" | cut -d'***' -f1

Output:
1001

Keep Reading, Keep Learning, Keep Sharing...!!

Friday, 12 February 2016

How To Send Email In HTML Formatting In Unix/Linux

In This article we will see how to send an email notification with HTML formatting. We will see how to create and send following table type structure in an email body.


#----------------------------------------START OF THE SCRIPT--------------------------------------
$ cat Email_In_Html_Format.sh

#!/usr/bin/sh

V_MAIL_TO="Director@kw.com"
V_EMAIL_BODY="Email_Body.html"

#--Below code will be used to generate the HTML code
echo "<html>" > ${V_EMAIL_BODY}
echo "<head></head>" >> ${V_EMAIL_BODY}
echo "<body>" >> ${V_EMAIL_BODY}
echo "<STYLE TYPE="text/css">" >> ${V_EMAIL_BODY}
echo "<!--"  >> ${V_EMAIL_BODY}
echo "TD{font-family: Arial; font-size: 9pt;}" >> ${V_EMAIL_BODY}
echo "--->"  >> ${V_EMAIL_BODY}
echo "</STYLE>" >> ${V_EMAIL_BODY}

#--Below echo is used to create a HTML table
echo "<TABLE border="2" frame="border" rules="all">" >> ${V_EMAIL_BODY}

#--Below echo is used to create table header.
echo "<tr><th bgcolor=#BDBDBD><font size="2">EMP_ID</font></th>
<th bgcolor=#BDBDBD><font size="2">EMP_NAME</font></th>
<th bgcolor=#BDBDBD><font size="2">EMAP_DEPT</font></th>
<th bgcolor=#BDBDBD><font size="2">EMP_SAL</font></th></tr>" >> ${V_EMAIL_BODY}

#--Below echo is used to generate the data in tabular format.
echo "<tr><td>1001</td><td>A1</td><td>HR</td><td>50000</td></tr>          <tr><td>1002</td><td>A2</td><td>MAN</td><td>60000</td></tr>
      <tr><td>1003</td><td>A3</td><td>VP</td><td>100000</td></tr>         <tr><td>1003</td><td>A3</td><td>CEO</td><td>150000</td></tr>" 
      >> ${V_EMAIL_BODY}
echo "</table>" >> ${V_EMAIL_BODY}
echo "</body>" >> ${V_EMAIL_BODY}
echo "</html>" >> ${V_EMAIL_BODY}

#--Below code will be used to send email notification
export V_CONTENT="${V_EMAIL_BODY}"
(
echo "Subject: Employees Report"
echo "To: ${V_MAIL_TO}"
echo "MIME-Version: 1.0"
echo "Content-Type: text/html"
echo "Content-Disposition: inline"
echo ${V_CONTENT}

) | /usr/sbin/sendmail ${V_MAIL_TO}

#-------------------------------------------END OF THE SCRIPT---------------------------------------

Output: Above Script will send the email to the user Director@kw.com with the tabular format. We can use such emails in notifying details with more clarification and readability.

Keep Reading, Keep Learning, Keep Sharing..!!!

How To Send Email In Unix

In This post we will see how to send email using mailx command available in most of the Unix like operating system.

SYNTAX: mailx <Switch> "<Subject>" <Email Of Address Recipient>

Email With subject

mailx -s "Test" "A1@kw.com"

Output: Will send an email to A1@kw.com with subject line as test
To:  A1@kw.com

Email With CC and BCC 

We can also use cc and bcc email option by using -c and -b

mailx -s "test" -c "A2@kw.com" -b "A3@kw.com" "A1@kw.com"

Output: Email recipients will appear as shown below in email. 
To:  A1@kw.com
Cc:  A2@kw.com
Bcc: A3@kw.com

Email With body

We can do that in multiple ways:

echo "Hello User" | mailx -s "test" "A1@kw.com"

Output:In this case, email body will have following text: "Hello User"
  
We can also use the content of a file as an email body, for example we have a file F_Msg_File.txt

cat F_Msg_File.txt
Hi User,

Please Note, We will be discussing mailx command.

Thanks
Knowledge Warehouse
   
(cat F_Msg_File.txt) | mailx -s "test" "A1@kw.com"

[OR]

mailx -s "test" "A1@kw.com" < F_Msg_File.txt

Output: Above command will send an email to the user with the file content as email body.

Email With Attachment

We can attach(log file or anything) an attachment in the email by using uuencode command.

(uuencode F_Log_File.txt F_Log_File.txt) | mailx -s "test" "A1@kw.com"

Email With Attachment & body

(cat F_Msg_File.txt; uuencode F_Error_File.txt F_Error_File.txt) | mailx -s "test" A1@kw.com

Output: Will attach F_Error_File.txt in email as attachment and display the content of F_Msg_File.txt as email body.

Note: Observe carefully, we have provided F_Log_File.txt wice in uuencode command.


Keep Reading, Keep Learning, Keep Sharing...!!

How To Use Alias In Unix/Linux

Many a times we face situation where we have to move in different directories to analyse the issue or checkout the multiple objects involved. At times it becomes very lengthy and time consuming as you have type a lot to change the directories involved.

For example, let say you have following directories available for your project:
  • Source File Directory which keeps the source file
  • Target File Directory which keeps the files generated after some processing.
  • Scripts, executable logics which perform some action
  • Log File Directory, keeps the log of all the activities
Now, if you have are doing some impact analysis it will become very time consuming to navigate in these directories. Alias command provides solution of such situations. How? Let see...!!

Source File Directory: /home/baba/kw/SourceFiles/ABC
Target File Directory: /home/baba/kw/TargetFiles/ABC
Script File Directory: /home/baba/kw/Scripts/ABC
Log    File Directory: /home/baba/kw/Logs/ABC

alias Source='cd /home/baba/kw/SourceFiles/ABC'
alias Target='cd /home/baba/kw/TargetFiles/ABC'
alias Script='cd /home/baba/kw/Scripts/ABC'
alias Log='cd /home/baba/kw/Logs/ABC'
  
Now if you type log at terminal, it will directly take you to the log directory, similarly if you want to go to source directory while you are still in log directory, just type source
you will be in source file directory.

alias command is active till the session is active. if you want it to be permanent declare it in user profile file (.profile  or bash profile)


It is not like you can only navigate in directory using alias, you can almost use it in all the repetitive work. For example, you want to count number of line in a file.

alias cnt='wc -l'

$ cnt F_Input_File

It will return you the number of line available in F_Input_File

alias clr='clear'

now by only typing c your screen will be cleared.

If you want to remove alias, use unalias command to remove the same.

unalias clr

will remove the alias clr.


Conclusion: You can create alias for almost all the day to day activity so utilize this feature to make your life ease....!!!

Keep Reading, Keep Learning, Keep Sharing...!!

Thursday, 11 February 2016

How To Rename Multiple Files In Unix/Linux

Let say we have below file in home directory /home/baba

$ ls -ltr


-rw-r--r--  1 baba None   9 Jan 26 14:11 A1.txt
-rw-r--r--  1 baba None  32 Jan 26 14:20 A2.txt
-rw-r--r--  1 baba None  32 Jan 26 14:20 A3.txt
-rw-r--r--  1 baba None  10 Jan 26 14:20 A4.txt
-rw-r--r--  1 baba None  92 Feb  2 22:38 A5.txt
-rw-r--r--  1 baba None 278 Feb  2 22:43 A6.txt

We want to rename all these files and want to concatenate _bkp in file name

#!bin/sh
for i in `ls`
do
echo "Current File Name is...:"${i}
mv ${i} ${i}"_bkp"
done

Output:Will rename all the objects in the current directory by concatenating _bkp

We can archive the multiple files in one go by keeping the original fles, for example we want to keep archive of files in archive directory.

#!bin/sh
for i in `ls`
do
echo "Current File Name is...:"${i}
cp ${i} /home/baba/Archive/${i}"_bkp"
done

Output:
cp command will create a copy of the existing file in archive directory and will keep the original file in same directory.


Always use the path while reading the file name in for loop instead of giving only ls command.

#!bin/sh
for i in `ls /home/baba`
do
echo "Current File Name is...:"${i}
mv ${i} /home/baba/Archive/${i}"_bkp"
done


Output: Above all command will rename all the files from /home/baba and move it to /home/baba/Archive/ directory.

-rw-r--r--  1 baba None   9 Feb 12 14:11 A1.txt_bkp
-rw-r--r--  1 baba None  32 Feb 12 14:12 A2.txt_bkp
-rw-r--r--  1 baba None  32 Feb 12 14:13 A3.txt_bkp
-rw-r--r--  1 baba None  10 Feb 12 14:15 A4.txt_bkp
-rw-r--r--  1 baba None  92 Feb 12 14:30 A5.txt_bkp
-rw-r--r--  1 baba None 278 Feb 12 14:40 A6.txt_bkp

Keep Reading, Keep Learning, Keep Sharing.

Tuesday, 2 February 2016

How To Read A Delimited File In Unix Script

In this article we will see how to read a delimited files in shell script. We will use input fileF_Input_File.txt for demonstration purpose.

Input File: F_Input_File.txt
1001|A1|HR|5000|Y
1002|A2|FIN|6000|Y
1003|A3|HR|6000|Y
1004|A4|HR|7000|Y


We can read a file using loop (while, for), here we are reading data from F_Input_File.txt.
V_Line holds the complete record and by using cut we are fetching all the fields.

while read V_Line 
do

V_Id=`echo ${V_Line} | cut -d"|" -f1`
V_Name=`echo ${V_Line} | cut -d"|" -f2`
V_Dept=`echo ${V_Line} | cut -d"|" -f3`
V_Sal=`echo ${V_Line} | cut -d"|" -f4`
V_Actv_Ind=`echo ${V_Line} | cut -d"|" -f5`

echo "Employee Id is.........:" ${V_Id}
echo "Employee Name is.......:" ${V_Name}
echo "Employee Department is.:" ${V_Dept}
echo "Employee Salary is.....:" ${V_Sal}
echo "Employee Status is.....:" ${V_Actv_Ind}

done < F_Input_File.txt

Output:
Employee Id is.........: 1001
Employee Name is.......: A1
Employee Department is.: HR
Employee Salary is.....: 5000
Employee Status is.....: Y
Employee Id is.........: 1002
Employee Name is.......: A2
Employee Department is.: FIN
Employee Salary is.....: 6000
Employee Status is.....: Y
Employee Id is.........: 1003
Employee Name is.......: A3
Employee Department is.: HR
Employee Salary is.....: 6000
Employee Status is.....: Y
Employee Id is.........: 1004
Employee Name is.......: A4
Employee Department is.: HR
Employee Salary is.....: 7000
Employee Status is.....: Y


We can get the same output by using IFS keyword (Internal Field Separator), here data is stored in associated variables separated by pipe(|). By using IFS, we can reduce the code and implement the same with the ease.

IFS="|"
while read V_Id V_Name V_Dept V_Sal V_Actv_Ind
do

echo "Employee Id is.........:" ${V_Id}
echo "Employee Name is.......:" ${V_Name}
echo "Employee Department is.:" ${V_Dept}
echo "Employee Salary is.....:" ${V_Sal}
echo "Employee Status is.....:" ${V_Actv_Ind}

done < F_Input_File.txt

Output:
$ sh a.sh
Employee Id is.........: 1001
Employee Name is.......: A1
Employee Department is.: HR
Employee Salary is.....: 5000
Employee Status is.....: Y
Employee Id is.........: 1002
Employee Name is.......: A2
Employee Department is.: FIN
Employee Salary is.....: 6000
Employee Status is.....: Y
Employee Id is.........: 1003
Employee Name is.......: A3
Employee Department is.: HR
Employee Salary is.....: 6000
Employee Status is.....: Y
Employee Id is.........: 1004
Employee Name is.......: A4
Employee Department is.: HR
Employee Salary is.....: 7000
Employee Status is.....: Y

Conclusion: By using similar method we can read any kind of delimited file like CSV file (comma delimited)

Keep Reading, Keep Learning, Keep Sharing...!!!

Tuesday, 26 January 2016

How To Zip/Unzip A File In Unix/Linux

In this article we will see how to zip/compress files as well as how to unzip/decompress the files. We will divide this article in three categorise as explained below:

Category:1-> Switches which will be applied on Normal files (unzipped file)
Category:2-> Switches which will be applied on zipped files.
Category:3-> gunzip to unzip the files.

We will see some of the below listed switches, you can get the list from man gzip or gzip -h.

-d, --decompress  decompress
-h, --help        give this help
-k, --keep        keep (don't delete) input files
-l, --list        list compressed file contents
-q, --quiet       suppress all warnings
-r, --recursive   operate recursively on directories
-t, --test        test compressed file integrity
-v, --verbose     verbose mode
-1, --fast        compress faster
-9, --best        compress better

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<CATEGORY:1>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

ZIP The Input File

$ gzip F_Input_File1.txt

Output: Will create a zip file
F_Input_File.txt1.gz

We can zip multiple files together as shown below:
$ gzip F_Input_File1.txt F_Input_File2.txt F_Input_File3.txt

Keep The Original File With Zipped File

The original file be lost once the zip file is created,if you want to keep the original file with the zipped file, use -k switch.

$ gzip -k F_Input_Fil1e.txt

Output: Will zip the file and keep the original file as well.
$ ls
F_Input_File1.txt
F_Input_File1.txt.gz

What gzip is Doing

$ gzip -v F_Input_File1.txt

Output: It will provide real time activities(verbose) which gzip is doing.

Zip All The File Recursively

$ gzip -v -r /home/baba

Output: The -r switch has forced gzip to perform recursive compression(zip).
/home/baba/F_Input_File1.txt:    11.1% -- replaced with /home/baba/F_Input_File1.txt.gz
/home/baba/F_Input_File2.txt:    51.6% -- replaced with /home/baba/F_Input_File2.txt.gz
/home/baba/Imp/F_Input_File1.txt:       -12.5% -- replaced with /home/baba/Imp/F_Input_File1.txt.gz

Zip All The File Recursively, Keeping Original Files

We can also do recursive zip keeping the original files.

$ gzip -v -k -r /home/baba

Output:
/home/baba/F_Input_File1.txt:    11.1% -- replaced with /home/baba/F_Input_File1.txt.gz
/home/baba/F_Input_File2.txt:    51.6% -- replaced with /home/baba/F_Input_File2.txt.gz
/home/baba/Imp/F_Input_File1.txt:       -12.5% -- replaced with /home/baba/Imp/F_Input_File1.txt.gz

$ ls -ltr
total 7
-rw-r--r--  1 baba None   52 Jan 26 12:50 F_Input_File1.txt.gz
-rw-r--r--  1 baba None   36 Jan 26 12:50 F_Input_File1.txt
-rw-r--r--  1 baba None  604 Jan 26 13:08 F_Input_File2.txt.gz
-rw-r--r--  1 baba None 1206 Jan 26 13:08 F_Input_File2.txt
drwxr-xr-x+ 1 baba None    0 Jan 26 13:24 Imp

How To Do Fast Zipping/Compression

$ gzip -v -1 F_Input_File1.txt
[OR]
$ gzip -v --fast F_Input_File1.txt

How To Do best Zipping/Compression

$ gzip -v -9 F_Input_File1.txt
[OR]
$ gzip -v --best F_Input_File1.txt

-1 or --fast indicates the fastest compression method (less compression) and -9 or --best indicates the slowest compression method (best compression). The default compression level is -6 (that is, biased towards high compression at expense of speed).

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<CATEGORY:2>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Unzip/Decompress The Zipped file

We have a separate command, gunzip, to unzip the files but we can do the same by using gzip itself. We will see gunzip as well.

$ gzip -d F_Input_File.txt.gz
$ gzip --decompress F_Input_File.txt.gz
$ gzip --uncompress F_Input_File.txt.gz

Output: All the 3 will unzip the file.

Unzip The Zipped Files Recursively

$ gzip -v -d -r /home/baba

Output: -d switch will unzip, -r switch will perform recursion, -v verbose
/home/baba/F_Input_File1.txt.gz:         11.1% -- replaced with /home/baba/F_Input_File1.txt
/home/baba/F_Input_File2.txt.gz:         51.6% -- replaced with /home/baba/F_Input_File2.txt
/home/baba/Imp/F_Input_File1.txt.gz:    -12.5% -- replaced with /home/baba/Imp/F_Input_File1.txt

Statistics Of Zipped File

$ gzip -v -l F_Input_File.txt.gz

Output: Will lsit down the statistics of zipped file.
method  crc     date  time           compressed        uncompressed  ratio uncompressed_name
defla 69cf461f Jan 26 12:50                  52                  36  11.1% a

How To Read A Zip/Compressed File

$ gcat F_Input_File1.txt.gz

Output: Will display the content of the file F_Input_File1.txt

How To Check Whether File Is Compressed Correctly

$ gzip -v -t F_Input_File1.txt.gz

Output: -t switch is used to test the integrity.
F_Input_File1.txt.gz:    OK

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<CATEGORY:3>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

We can use Most of the switches of gzip in gunzip will see the same in upcoming example.

Unzip/ Decompress The Zipped File using gunzip 

$ gunzip F_Input_File1.txt.gz
$ gunzip -v F_Input_File1.txt.gz

Output: Will unzip the file
F_Input_File1.txt

$ gunzip -d F_Input_File1.txt.gz
$ gunzip -v -d F_Input_File1.txt.gz

Output: -d switch will decompress the zip file, we have already seen the same with gzip file.
F_Input_File1.txt

Unzip/ Decompress The Zipped File Recursively using gunzip 

$ gunzip -r /home/baba/
$ gunzip -v -r /home/baba/

Output: Will unzip/decompress all the zipped files recursively.

Unzip/ Decompress The Zipped File, keeping the original zipped files.

$ gunzip -k F_Input_File1.txt.gz F_Input_File2.txt.gz
$ gunzip -v -k F_Input_File1.txt.gz F_Input_File2.txt.gz

Output: Will unzip the files as well as keep the copy of zip files. 

Note: If you observe we gzip and gunzip are very similar in functionality as well as in working. We should always use -v (verbose option), it provides the clear picture about what is going on.

Conclusion: Go with anyone gzip or gunzip, They are almost doing same kind of thing so why to use both, master one!!!

Keep Reading, Keep Learning, keep Sharing...!!!

Saturday, 23 January 2016

How To Insert, Append, Replace Lines In A File Using Sed

Some time we face situation where we need to either insert some records or line separator  in file or need to change the complete line which matches the criteria.

Let See how to implement the same using sed command.

Input_File: F_Input_File
1001|A00121|NOKIA-1100|Y

SYNTAX:
sed '<N/Pattern> [a/i/c] <Line To Be Appended /Inserted /Replaced>' F_Input_File.txt

N -> Line Number After Which File will be appended.
a -> Append After the Matching line or line number
i -> Insert Before the Matching line or line number
c -> change the matching line or line number
Pattern-> Pattern Matching/Regular Expression


Append: Add A line After The Match 

$ sed '1 a "*************************"' F_Input_File.txt
[OR]
$ sed '/A00121/ a "*************************"' F_Input_File.txt

Output:
1001|A00121|NOKIA-1100|Y
*************************

Insert: Add A line Before The Match

$ sed '1 i "-------------------------"' F_Input_File.txt
[OR]
$ sed '/A00121/ i "-------------------------"' F_Input_File.txt

Output:
-------------------------
1001|A00121|NOKIA-1100|Y

Change: Change Matching Line(s)

$ sed '1 c "****PRODUCT OUT OF STOCK**"' F_Input_File.txt
[OR]
$ sed '/A00121/ c "****PRODUCT OUT OF STOCK**"' F_Input_File.txt

Output:
****PRODUCT OUT OF STOCK**"


The above all method will display the updated/changed data on terminal but the changes will not be permanent in the original file.To make changes permanent, we need to use -i switch, as shown below:

$ sed -i '/A00121/ a "*************************"' F_Input_File.txt
$ sed -i '/A00121/ i "-------------------------"' F_Input_File.txt
$ sed -i '/A00121/ c "***PRODUCT OUT OF STOCK**"' F_Input_File.txt


And again, if your version of sed is not supporting -i switch then we can always go the basics to achieve the same using redirection operator. Already explained in following post

Keep Reading, Keep Learning, Keep Sharing..!!!

How To Update A File Keeping Original File Using Sed In Unix/Linux

In this article we will see how to update file using sed instead of using temporary table or displaying data at terminal. We will see how can we keep the backup of original file, that can be used in case of any failure or data retrieval. We will use input file F_Input_File.txt for the same. Let see how to achieve it...!!

$ cat F_Input_File.txt
1001|A1|HR|5000|Y
1002|A2|FIN|6000|Y
1003|A3|HR|6000|Y
1004|A4|HR|7000|Y

$ sed -i .bak 's/FIN/HR' F_Input_File.txt
[OR]
$ sed -i.bak 's/FIN/HR' F_Input_File.txt


Output: It Will produce two files in the same directory, one with updated data (in Green) and one with original data (.bak file,  in blue)

$ ls
F_Input_File.txt
F_Input_File.txt.bak

$ cat F_Input_File.txt
1001|A1|HR|5000|Y
1002|A2|HR|6000|Y
1003|A3|HR|6000|Y
1004|A4|HR|7000|Y 

$ cat F_Input_File.txt.bak
1001|A1|HR|5000|Y
1002|A2|FIN|6000|Y
1003|A3|HR|6000|Y
1004|A4|HR|7000|Y

Note: In some of the Linux/Unix system first command may not work, in that case please use 2nd command. (there should not be any space between -i switch and extension)

Most Imp: If NO extension is given, no backup will be saved.

$ sed -i 's/FIN/HR' F_Input_File.txt

Output: Will update the file F_Input_File.txt and there will be no file available with original data (backup file)

If -i switch is not working in your version of Unix/linux, we can always go to the basics and achieve the same by using redirection.

$ sed 's/FIN/HR' F_Input_File.txt > F_Input_File.txt_tmp


Updated data is now available in F_Input_File.txt_tmp, we can rename the original file (F_Input_File.txt) to backup (F_Input_File.txt.bkp) and then rename the temporary file (updated file:F_Input_File.txt_tmp,) to original file(F_Input_File.txt).

$ mv F_Input_File.txt F_Input_File.txt.bkp
$ mv F_Input_File.txt_tmp F_Input_File.txt


Keep Reading, Keep Learning, Keep Sharing...!!!
Related Posts Plugin for WordPress, Blogger...