Do you know Bash Shell Script Framework

When working with shell script programming, you should ask yourself why this one is so basic (but powerful), and you need to do everything from beginning. For example, logging to file, stripping double quote, call database script, …

These things are easy and full support in modern web programming language, but in shell script programming, what do you have to tackle your jobs?

Fortunately, there are some bash shell script framework that you can use for your work and contribute to get it powerful.

bashinator
Bashinator support logging, printing, emailing with colorful output also with call stack traces and lockfile handling. It’s flexible and powerful.

shesfw
Shesfw is a collection of bash functions composing a API for Shell applications. It includes:

  1. Print output and input functions (with several interfaces, textual or graphical)
  2. Automatic build/remove of temporary directories
  3. Log function
  4. Argument treatment
  5. Functions for configuration files

The main goal is provide less effort on shell development.

log4sh
log4sh is an advanced logging framework for shell scripts (eg. sh, bash) that works similar to the logging products available from the Apache Software Foundataion (eg. log4j, log4perl).

oobash
oo-style framework for bash 4. It provides tools for rapid script development and huge libraries. Written in bash

bangsh
This framework is intended to help on easy bash script development. It is totally modularized. It helps you developing new Bash Script programs by forcing you to modularize and organize your code in functions, so that your program can be tested.

blp
This bash library project is a library of bash functions.

Bash-Toolbox
Bash-Toolbox is a simple set of functions and instructions that help develop script applications in Bash.

How to Read a File Line by Line in a Shell Script

There are many ways to handle any task on a Unix platform,  but some
techniques that are used to process a file waste a lot of CPU time.
Most of the wasted time is spent in unnecessary variable assignment and
continuously opening and closing the same file over and over. Using a
pipe also has a negative impact on the timing.

 In this article I will explain various techniques for parsing a file
line by line. Some techniques are very fast and some make you wait for
half a day. The techniques used in this article are measurable, and I
tested each technique with time command so that you can see which tec-
hniques suits your needs.

 I don't explain in depth every thing, but if you know basic shell
scripting, I hope you can understand easily.

 I extracted last five lines from my /etc/passwd file, and stored in a
file "file_passwd".

[root@www blog]# tail -5 /etc/passwd > file_passwd
[root@www blog]# cat file_passwd
venu:x:500:500:venu madhav:/home/venu:/bin/bash
padmin:x:501:501:Project Admin:/home/project:/bin/bash
king:x:502:503:king:/home/project:/bin/bash
user1:x:503:501::/home/project/:/bin/bash
user2:x:504:501::/home/project/:/bin/bash
 I use this file whenever a sample file required.

Method 1: PIPED while-read loop


#!/bin/bash
# SCRIPT: method1.sh
# PURPOSE: Process a file line by line with PIPED while-read loop.

FILENAME=$1
count=0
cat $FILENAME | while read LINE
do
let count++
echo “$count $LINE”
done

echo -e “\nTotal $count Lines read”

 With catting a file and piping the file output to a while read loop a
single line of text is read into a variable named LINE on each loop
iteration. This continuous loop will run until all of the lines in the
file have been processed one at a time.

 Bash can sometimes start a subshell in a PIPED "while-read" loop. So
the variable set within the loop will be lost (unset) outside of the
loop. Therefore, $count would return 0, the initialized value outside
the loop.

Output:

[root@www blog]# sh method1.sh file_passwd
1 venu:x:500:500:venu madhav:/home/venu:/bin/bash
2 padmin:x:501:501:Project Admin:/home/project:/bin/bash
3 king:x:502:503:king:/home/project:/bin/bash
4 user1:x:503:501::/home/project/:/bin/bash
5 user2:x:504:501::/home/project/:/bin/bash

Total 0 Lines read


Method 2: Redirected “while-read” loop


#!/bin/bash
#SCRIPT: method2.sh
#PURPOSE: Process a file line by line with redirected while-read loop.

FILENAME=$1
count=0

while read LINE
do
let count++
echo “$count $LINE”

done < $FILENAME

echo -e “\nTotal $count Lines read”

 We still use the while read LINE syntax, but this time we feed the
loop from the bottom (using file redirection) instead of using a pipe.
You will find that this is one of the fastest ways to process each
line of a file. The first time you see this it looks a little unusual,
but it works very well.

 Unlike method 1, with method 2 you will get total number of lines out
side of the loop.

Output:

[root@www blog]# sh method2.sh file_passwd
1 venu:x:500:500:venu madhav:/home/venu:/bin/bash
2 padmin:x:501:501:Project Admin:/home/project:/bin/bash
3 king:x:502:503:king:/home/project:/bin/bash
4 user1:x:503:501::/home/project/:/bin/bash
5 user2:x:504:501::/home/project/:/bin/bash

Total 5 Lines read

Note: In some older shell scripting languages, the redirected loop
would also return as a subshell.

Method 3:while read LINE Using File Descriptors

 A file descriptor is simply a number that the operating system assigns
to an open file to keep track of it. Consider it a simplified version
of a file pointer. It is analogous to a file handle in C.

 There are always three default "files" open, stdin (the keyboard),
stdout (the screen), and stderr (error messages output to the screen).
These, and any other open files, can be redirected. Redirection simply
means capturing output from a file, command, program, script, or even
code block within a script  and sending it as input to another file,
command, program, or script.

 Each open file gets assigned a file descriptor. The file descriptors
for stdin,stdout, and stderr are 0,1, and 2, respectively. For opening
additional files, there remain descriptors 3 to 9 (may be vary depend-
ing on OS). It is sometimes useful to assign one of these additional
file descriptors to stdin, stdout, or stderr as a temporary duplicate
link. This simplifies restoration to normal after complex redirection
and reshuffling .

 There are two steps in the method we are going to use. The first step
is to close file descriptor 0 by redirecting everything to our new file
descriptor 3. We use the following syntax for this step:

  exec 3<&0

 Now all of the keyboard and mouse input is going to our new file des-
criptor 3. The second step is to send our input file, specified by the
variable $FILENAME, into file descriptor 0 (zero), which is standard
input. This second step is done using the following syntax:

    exec 0<$FILENAME

 At this point any command requiring input will receive the input from
the $FILENAME file. Now is a good time for an example.

#!/bin/bash
#SCRIPT: method3.sh
#PURPOSE: Process a file line by line with while read LINE Using
#File Descriptors

FILENAME=$1
count0=
exec 3<&0
exec 0< $FILENAME

while read LINE
do
let count++
echo “$count $LINE”
done

exec 0<&3
echo -e “\nTotal $count Lines read”

 while loop reads one line of text at a time.But the beginning of this
script does a little file descriptor redirection. The first exec comm-
and redirects stdin to file descriptor 3. The second exec command red-
irects the $FILENAME file into stdin, which is file descriptor 0. Now
the while loop can just execute without our having to worry about how
we assign a line of text to the LINE variable. When the while loop
exits we redirect the previously reassigned stdin, which was sent to
file descriptor 3, back to its original file descriptor 0.

 exec 0<&3

In other words we set it back to the system’s default value.

Output:

[root@www tempdir]# sh method3.sh file_passwd 
1 venu:x:500:500:venu madhav:/home/venu:/bin/bash
2 padmin:x:501:501:Project Admin:/home/project:/bin/bash
3 king:x:502:503:king:/home/project:/bin/bash
4 user1:x:503:501::/home/project/:/bin/bash
5 user2:x:504:501::/home/project/:/bin/bash

Total 5 Lines read

Method 4: Process file line by line using awk

 awk is pattern scanning and text processing language. It is useful
for manipulation of data files, text retrieval and processing. Good
for manipulating and/or extracting fields (columns) in structured
text files.

 Its name comes from the surnames of its authors: Alfred Aho, Peter
Weinberger, and Brian Kernighan.

 I am not going to explain everything here.To know more about awk just
Google it.

At the command line, enter the following command:

$ awk '{ print }' /etc/passwd 

 You should see the contents of your /etc/passwd file appear before
your eyes.Now, for an explanation of what awk did. When we called awk,
we specified /etc/passwd as our input file. When we executed awk, it
evaluated the print command for each line in /etc/passwd, in order.All
output is sent to stdout, and we get a result identical to catting
/etc/passwd. Now, for an explanation of the { print } code block. In
awk, curly braces are used to group blocks of code together, similar
to C. Inside our block of code,we have a single print command. In awk,
when a print command appears by itself, the full contents of the curr-
ent line are printed.

Here is another awk example that does exactly the same thing:

$ awk '{ print $0 }' /etc/passwd 

 In awk, the $0 variable represents the entire current line, so print
and print $0 do exactly the same thing. Now is a good time for an
example.

#!/bin/bash
#SCRIPT: method4.sh
#PURPOSE: Process a file line by line with awk

FILENAME=$1

awk ‘{kount++;print kount, $0}
END{print “\nTotal ” kount ” lines read”}’ $FILENAME

Output:

[root@www blog]# sh method4.sh file_passwd
1 venu:x:500:500:venu madhav:/home/venu:/bin/bash
2 padmin:x:501:501:Project Admin:/home/project:/bin/bash
3 king:x:502:503:king:/home/project:/bin/bash
4 user1:x:503:501::/home/project/:/bin/bash
5 user2:x:504:501::/home/project/:/bin/bash

Total 5 lines read

Awk is really good at handling text that has been broken into multiple
logical fields, and allows you to effortlessly reference each individ-
ual field from inside your awk script. The following script will print
out a list of all user accounts on your system:

awk -F":" '{ print $1 "\t " $3  }' /etc/passwd

 Above, when we called awk, we use the -F option to specify ":" as the
field separator. By default white space (blank line) act as filed sep-
arator. You can set new filed separator with -F option. When awk proc-
esses the print $1 "\t " $3 command, it will print out the first and
third fields that appears on each line in the input file. "\t" is used
to separate field with tab.

Method 5: Little tricky with head and tail
commands


#!/bin/bash
#SCRIPT: method5.sh
#PURPOSE: Process a file line by line with head and tail commands

FILENAME=$1
Lines=`wc -l < $FILENAME`

count=0

while [ $count -lt $Lines ]
do
let count++
LINE=`head -n $count $FILENAME | tail -1`
echo “$count $LINE”
done
echo -e “\nTotal $count lines read”

On each iteration head command extracts top $count lines, then tail
command extracts bottom line from that lines. A very stupid method,
but some people still using it.

Output:

[root@www blog]# sh method5.sh file_passwd
1 venu:x:500:500:venu madhav:/home/venu:/bin/bash
2 padmin:x:501:501:Project Admin:/home/project:/bin/bash
3 king:x:502:503:king:/home/project:/bin/bash
4 user1:x:503:501::/home/project/:/bin/bash
5 user2:x:504:501::/home/project/:/bin/bash

Total 5 lines read


Time Comparison for the Five Methods

 Now take a long breath, we are going test each technique. Before you
get into test each method of parsing a file line by line create a large
file that has the exact number of lines that you want to process.
Use bigfile.sh script to create a large file.

$ sh bigfile.sh 900000 

bigfile.sh with 900000 lines as an argument,it has taken more than two
hours to generate bigfile.4227. I don't know exactly how much time it
has taken. This file is extremely large to parse a file line by line,
but I needed a large file to get the timing data greater than zero.

[root@www blog]# du -h bigfile.4227
70M bigfile.4227
[root@www blog]# wc -l bigfile.4227
900000 bigfile.4227

[root@www blog]# time ./method1.sh bigfile.4227 >/dev/null

real 6m2.911s
user 2m58.207s
sys 2m58.811s
[root@www blog]# time ./method2.sh bigfile.4227 > /dev/null

real 2m48.394s
user 2m39.714s
sys 0m8.089s
[root@www blog]# time ./method3.sh bigfile.4227 > /dev/null

real 2m48.218s
user 2m39.322s
sys 0m8.161s
[root@www blog]# time ./method4.sh bigfile.4227 > /dev/null

real 0m2.054s
user 0m1.924s
sys 0m0.120s
[root@www blog]# time ./method5.sh bigfile.4227 > /dev/null
I waited more than half day, still i didn’t get result, then I created
a 10000-line file to test this method.
[root@www tempdir]# time ./method5.sh file.10000 > /dev/null

real 2m25.739s
user 0m21.857s
sys 1m12.705s

Method 4 came in first place,it has taken very less time 2.05 seconds,
but we can't compare Method 4 with other methods, because awk is not
just a command, but a programming language too.

 Method 2 and method 3 are tied for second place, they  produce mostly
the same real execution time at 2 minutes and 48 seconds . Method 1
came in third at 6 minutes and 2.9 seconds.

 Method 5 has taken more than half a day. 2 minutes 25 seconds to pro-
cess just a 10000 line file, how stupid it is.

Note: If file contain escape characters, use read -r instead of read,
then Backslash does not act as an escape character. The back-slash is
considered to be part of the line. In particular, a backslash-newline
pair may not be used as a line continuation.

How to execute a MySQL Command from a Linux Bash Shell?

Sometimes it is necessary to runMySQL query directly from theLinux Command Line without actually going into the interactive MySQL prompt.

For example, when you want to schedule a backup of MySQL databases or automate a creation of MySQL databases and users with some Bash Script.

Use one of the following commands to run a MySQL query from a Linux command line.

MySQL Command From a Bash Shell in One Line

Use the following command for quickly execution of MySQL query from a Linux Bash Shell :
# mysql -u [user] -p[pass] -e “[mysql commands]”
Example :
# mysql -u root -pSeCrEt -e “show databases”

Run a MySQL Query From a Bash script using EOF

Use the following syntax in your Bash scripts for running MySQL commands :
mysql -u [user] -p[pass] << EOF
[mysql commands]
EOF

Example :

#!/bin/bash
mysql -u root -pSeCrEt << EOF
use mysql;
show tables;
EOF

Execute a MySQL Command Remotely

Use -h option to specify a MySQL server’s IP address:
# mysql -h [ip] -u [user] -p[pass] -e “[mysql commands]”
Example :
# mysql -h 192.168.1.10 -u root -pSeCrEt -e “show databases”

Specify a Database to Use

Use -D option to specify the name of MySQL database :
# mysql -D [db name] -u [user] -p[pass] -e “[mysql commands]”
Example :
# mysql -D clients -u root -pSeCrEt -e “show tables”