Practical Shell Scripting

Contents
TCSH
Regular Expressions	Practical
Redirection
foreach	Shell
Backticks
Ex: SAS errors	Scripting
Ex: Families
Ex: Columns of Data
Script Files
How To Do...
Ex: 3 line compare
	Terry Gliedt March 13, 2000

If you print this with IE 5.5 or higher or Netscape 6.0, each of the sections will print on a separate page.

Top TCSH Regular
Expressions Redirection foreach Backticks Ex: SAS errors Ex: Families Ex: Columns of Data How To Do... Ex: 3 Line Compare

TCSH

C shell with file name completion and command line editing

Completely compatible with CSH
Reduces your typing, fewer errors

File completion>
Command completion

Uses ~/.tcshrc else uses ~/.cshrc
Current TCSH users: tpg, chli
Ask to Use It!

Top TCSH Regular
Expressions Redirection foreach Backticks Ex: SAS errors Ex: Families Ex: Columns of Data CSH Scripts How To Do... Ex: 3 Line Compare

Regular Expressions

* Match zero or more characters


example/1> echo chr*
chr1.txt chr10.txt chr11.txt chr12.txt chr13.txt chr2.txt
  chr21.txt chr22.txt chr23.txt chr29.txt chr3.txt chr4.txt

example/1> cat chr* > junk


Note: Data in 'junk' from files in order shown above.

?	Match any single character


example/1> echo chr?.txt
chr1.txt chr2.txt chr3.txt chr4.txt

example/1> cat chr?.txt chr1?.txt chr2?.txt > junk


note: data in 'junk' from files in alphabetical order.

" "	Double quotes preserve space, Allow substitution of variables


example/1> echo hi there
hi there

example/1> echo hi   there
hi there

example/1> echo "hi  there"
hi  there

example/1> echo "hi, $user"
hi, tpg

' '	Single quotes preserve space, Do not allow substitution of variables


example/1> echo 'hi  there'
hi  there

example/1> echo 'hi, $user'
hi, $user

\	Escape special shell characters


example/1> echo These are special: \" \' \` \~ \[ \] \{ \} \| \& \? \! \^ \\
These are special: " ' ` ~ [ ] { } | & ? ! ^ \

example/1> echo "In CSH one may not escape \" within double quotes"
Unmatched ".

example/1> echo 'In CSH one may not escape \' within single quotes'
Unmatched ".

example/1> echo 'Use "quotes" within single quotes'
Use "quotes" within single quotes

example/1> echo "Use 'quotes' within double quotes"
Use 'quotes' within double quotes

[ ]	Matches any one of a set of characters


example/1> echo chr[12].txt
chr1.txt chr2.txt

example/1> echo chr[1-4].txt
chr1.txt chr2.txt chr3.txt chr4.txt

Top TCSH Regular
Expressions Redirection foreach Backticks Ex: SAS errors Ex: Families Ex: Columns of Data CSH Scripts How To Do... Ex: 3 Line Compare

Redirection

All programs start with these file HANDLES setup

STDIN - set to terminal
STDERR - set to console
STDOUT - set to console

<	STDIN redirect from a file
>	STDOUT redirect to a file
>&	STDOUT and STDERR redirect to a file
>!	STDOUT redirect to a file, override noclobber if set
>&!	STDOUT and STDERR redirect to a file, override noclobber if set
>>	STDOUT appended to a file use 'touch' to create file
>>&	STDOUT and STDERR appended to a file use 'touch' to create file
>>!	STDOUT and STDERR appended to a file, override noclobber if set use 'touch' to create file
>>&!	STDOUT and STDERR appended to a file, override noclobber if set use 'touch' to create file

Top TCSH Regular
Expressions Redirection foreach Backticks Ex: SAS errors Ex: Families Ex: Columns of Data CSH Scripts How To Do... Ex: 3 Line Compare

Foreach

foreach Loop through a list of values, executing commands


example/1> foreach n ( 1 2 3 )
foreach? echo Step $n       First command, substitute for $n
foreach? ls chr$n.txt       Second command, substitute for $n
foreach? end

Step 1
chr1.txt
Step 2
chr2.txt
Step 3
chr3.txt




example/1> foreach f ( chr1.txt chr2.txt chr3.txt )
foreach? ls -l $f
foreach? end

-rw-r--r--   1 tpg      staff         19 Mar  8 16:15 chr1.txt
-rw-r--r--   1 tpg      staff         19 Mar  8 16:15 chr2.txt
-rw-r--r--   1 tpg      staff         19 Mar  8 16:15 chr3.txt

Top TCSH Regular
Expressions Redirection foreach Backticks Ex: SAS errors Ex: Families Ex: Columns of Data CSH Scripts How To Do... Ex: 3 Line Compare

Backticks

` ` Capture STDOUT of some command, newlines converted to a space


example/1> set now = `date`
example/1> echo $now

Thu Mar 9 10:56:42 EST 2000


example/1> rm -f chr.lst
example/1> touch chr.lst
example/1> ls chr?.txt

chr1.txt   chr2.txt   chr3.txt   chr4.txt

example/1> foreach f (`ls chr?.txt`)
foreach? echo "Appending file '$f'..."
foreach? cat $f >> chr.lst
foreach? endq

Appending file 'chr1.txt'...
Appending file 'chr2.txt'...
Appending file 'chr3.txt'...
Appending file 'chr4.txt'...

example/1> ls -l chr.lst
-rw-r--r--   1 tpg      staff         76 Mar  9 11:01  chr.lst

Top TCSH Regular
Expressions Redirection foreach Backticks Ex: SAS errors Ex: Families Ex: Columns of Data CSH Scripts How To Do... Ex: 3 Line Compare

Find all cases where a SAS run failed

cd /group/boehnke/fusion/ppwhite

find . -name \*.log -print


./OldFiles/studyid.log
./OldFiles/LA/March99/datchk99.log
./OldFiles/LA/May99/datchk99.log
.
..
...
./studyid.log
./LA/March99/datchk99.log
./LA/May99/datchk99.log
./LA/visit2.log
.
..
...




find . -name \*.log -print | grep -v OldFiles


./studyid.log
./LA/March99/datchk99.log
./LA/May99/datchk99.log
./LA/visit2.log
.
..
...


find . -name \*.log -print | grep -v OldFiles >! ~/j.txt

foreach f ( `cat j.txt` )
foreach? echo "========= $f"
foreach? grep "because of errors" $f
foreach? end


========= ./Hotlist/PhenoInt/10and5/10and5.log
========= ./Hotlist/PhenoInt/10and6/10and6.log
========= ./Hotlist/PhenoInt/10and7/10and7.log
========= ./Hotlist/Peaks/0550T/poslod.log
========= ./Hotlist/Peaks/0550T/assoc0550T.log
NOTE: The SAS System stopped processing this step because of errors.
========= ./Hotlist/Peaks/0550T/pheno0550T.log
========= ./Hotlist/Peaks/1040A/poslod.log
========= ./Hotlist/Peaks/1040A/assoc1040A.log
NOTE: The SAS System stopped processing this step because of errors.
========= ./Hotlist/Peaks/1040A/pheno1040A.log
========= ./Hotlist/Peaks/0525A/poslod.log

Top TCSH Regular
Expressions Redirection foreach Backticks Ex: SAS errors Ex: Families Ex: Columns of Data CSH Scripts How To Do... Ex: 3 Line Compare

Example: Families

                       TABLE OF FAMID BY FAMMEM2

         FAMID     FAMMEM2

         Frequency|mother  |proband |sib     |  Total
         ---------+--------+--------+--------+
         0162     |      0 |      1 |      1 |      7
         ---------+--------+--------+--------+
         0163     |      1 |      1 |      1 |      7
         ---------+--------+--------+--------+
         0164     |      0 |      1 |      1 |      4
         ---------+--------+--------+--------+
         0165     |      0 |      1 |      1 |      2
         ---------+--------+--------+--------+
         0166     |      0 |      1 |      1 |      2
         ---------+--------+--------+--------+
         0167     |      0 |      1 |      1 |      2
         ---------+--------+--------+--------+
         0168     |      1 |      1 |      1 |      3
         ---------+--------+--------+--------+
         0169     |      0 |      1 |      1 |      5
         ---------+--------+--------+--------+
         0170     |      0 |      1 |      2 |      3
         ---------+--------+--------+--------+
         0171     |      0 |      1 |      1 |      6
         ---------+--------+--------+--------+
         0172     |      0 |      1 |      2 |      9

How many families (0-199) are there where the mother and more than one sibling is affected.

Step 1. Reduce families.lst to a list of lines of families we want

example/2> grep 01.. families.lst

         0001     |      0 |      0 |      0 |      0 |      4
         0010     |      0 |      3 |      0 |      0 |      7
         0011     |      0 |      1 |      1 |      0 |      5
.
..
         0019     |      0 |      1 |      2 |      3
         0100     |      0 |      1 |      1 |      5
         0101     |      0 |      1 |      1 |      2
.
..

example/2> grep ' 0[0-9][0-9][0-9] ' families.lst

         0100     |      0 |      2 |      1 |      0 |      5
         0101     |      0 |      0 |      0 |      0 |      2
.
..
         0198     |      0 |      0 |      0 |      0 |      2
         0100     |      0 |      1 |      1 |      5   
         0101     |      0 |      1 |      1 |      2
.
..

Note extra columns. Be sure this subset is what you want!
Maybe edit input file manually to reduce it.

Step 2. Reduce list of families to just those with affected mothers.

example/2> grep '0[0-9][0-9][0-9]     |      1 |' families2.lst

         0010     |      1 |      1 |      2 |      7
         0013     |      1 |      1 |      2 |      8
         0044     |      1 |      1 |      1 |      6
         0070     |      1 |      1 |      1 |     10
.
..
...


Step 3. Reduce list of families to just those with > 1 affected siblings.
example/2> grep '0[0-9][0-9][0-9]     |      1 |      1 |      [2-9]' families2.lst

         0010     |      1 |      1 |      2 |      7
         0013     |      1 |      1 |      2 |      8
         0375     |      1 |      1 |      2 |      4
         0381     |      1 |      1 |      2 |      4
         0436     |      1 |      1 |      2 |      4
         0442     |      1 |      1 |      3 |      9



Step 4. Count number of families.
example/2> grep '0[0-9][0-9][0-9]     |      1 |      1 |      [2-9]'
families2.lst | wc -l

      6








  Top
  TCSH
  Regular
Expressions
  Redirection
  foreach
  Backticks
  Ex: SAS errors
  Ex: Families
  Ex: Columns of Data
  CSH Scripts
  How To Do...
  Ex: 3 Line Compare







Example: Columns of Data





Input Files
marker01.lst
   BETA3 fusion
   D10S1208 cidr
   D10S1213 cidr
   D10S1218 cidr
   D10S1239 cidr
   D10S1267 fusion
   D10S1412 cidr
   D10S1435 cidr
   D10S1652 fusion
   D10S1720 fusion
   D10S1765 fusion
   D10S185 fusion


infoall.txt
  1QTEL19     D1S3739  Unknown      0.71     0   0     0   0     0   0
  ACT1B03a    D1S1586  G07765       0.69    91 118   112 112    91  91
  AFM016xh3   D1S411   Z23283       0.62   194 204   204 196   204 198
  AFM024xd2   D1S447   Z23291       0.77   123 141   135 129   129 123
  AFM024xf8   D1S456   Z23292       0.72   197 211   211 209   201 199
  AFM031xd12  D1S412   Z23298       0.71   185 207   199 185   205 197
  AFM036xc5   D1S189   Z16458       0.78   124 136   134 132   136 128
  AFM042xe3   D1S423   Z23307       0.61   162 167   164 164   164 164
  AFM046xh10  D1S191   Z16475       0.73   153 169   163 161   169 161
  AFM051xh8   D1S192   Z16482       0.66   203 211   207 207   205 205
  .
  ..


Create Output Files
fusion01.cfg
  C01_AFMa301xe1=D10S1267
  C01_AFMa162zh9=D10S1652
  C01_AFMb317wc1=D10S1720
  C01_AFM337xf9=D10S1765
  C01_AFM019th6=D10S185

cidr01.cfg
  C01_ATA5A04=D10S1208
  C01_GGAA5D10=D10S1213
  C01_ATA10G07=D10S1218
  C01_GATA64A09=D10S1239
  C01_ATA31G11=D10S1412
  C01_GATA88F09=D10S1435



Step 1. Reduce marker01.lst to just cidr markers
example/4> grep cidr marker01.lst

D10S1208 cidr
D10S1213 cidr
D10S1218 cidr
D10S1239 cidr
D10S1412 cidr
D10S1435 cidr



Step 2. Get just marker names from this list (first word)
example/4> grep cidr marker01.lst | awk '{ print $1 }' > cidr.01

D10S1208
D10S1213
D10S1218
D10S1239
D10S1412
D10S1435



Step 3. Search infoall.txt for each marker
example/4> foreach m (`cat cidr.01`)
foreach? grep -w $m infoall.txt
foreach? end

  ATA5A04     D10S1208 G08781       0.67   179 200   197 194   194 179
  GGAA5D10    D10S1213 G08823       0.79    93 121   121 101   121 117
  ATA10G07    D10S1218 G08766       0.46   212 233   218 218   218 218
  GATA64A09   D10S1239 G08799       0.75   160 184   184 172   180 180
  ATA31G11    D10S1412 G08780       0.73   151 166   160 151   157 151
  GATA88F09   D10S1435 G08819       0.70   256 276   268 264   268 268



Step 4. Search infoall.txt for each marker, getting AFM name
example/4> foreach m (`cat cidr.01`)
foreach? grep $m infoall.txt | awk '{ print $1 }'
foreach? end

ATA5A04
GGAA5D10
ATA10G07
GATA64A09
ATA31G11
GATA88F09



Step 5. Format these properly
example/4> foreach m (`cat cidr.01`)
foreach? grep $m infoall.txt | awk '{ printf("C01_%s=%s\n", $1, $2)}'
foreach? end

C01_ATA5A04=D10S1208
C01_GGAA5D10=D10S1213
C01_ATA10G07=D10S1218
C01_GATA64A09=D10S1239
C01_ATA31G11=D10S1412
C01_GATA88F09=D10S1435








  Top
  TCSH
  Regular
Expressions
  Redirection
  foreach
  Backticks
  Ex: SAS errors
  Ex: Families
  Ex: Columns of Data
  CSH Scripts
  How To Do...
  Ex: 3 Line Compare







CSH Scripts





To create a shell script, simply put the commands you can issue at the
terminal into a file, chmod it and execute it.


Convention: Make first line of file: #!/bin/csh -f



  
#!/bin/csh -f

#

#       Example one

#

foreach n ( 1 2 3 )

echo Step $n

ls chr$n.txt

end






example/1> chmod +x ex1.sh
example/1> ./ex1.sh


Step 1
chr1.txt
Step 2
chr2.txt
Step 3
chr3.txt






  
#!/bin/csh -fx

#

#       Example one

#

foreach n ( 1 2 3 )

echo Step $n

ls chr$n.txt

end






example/1> ./ex1.sh


foreach n ( 1 2 3 )
echo Step 1
Step 1
ls chr1.txt
chr1.txt
end
echo Step 2
Step 2
ls chr2.txt
chr2.txt
end
echo Step 3
Step 3
ls chr3.txt
chr3.txt
end












  Top
  TCSH
  Regular
Expressions
  Redirection
  foreach
  Backticks
  Ex: SAS errors
  Ex: Families
  Ex: Columns of Data
  CSH Scripts
  How To Do...
  Ex: 3 Line Compare







How To ...







  Prompt For Input









#!/bin/csh -f

#       Example 3a

echo -n "Please enter your name: "

set name = $<

if ( "x$name" == "x") then

  echo "No name provided, stopping"

  exit 3

endif

echo "Hello, $name, Welcome to FUSION"





example/3> ./ex3a.sh
Please enter your name: 

No name provided, stopping

example/3> ./ex3a.sh
Please enter your name: Terry

Hello, Terry, Welcome to FUSION

example/3> ./ex3a.sh
Please enter your name:  Terry Gliedt

Hello, Terry Gliedt, Welcome to FUSION







  Using Parameters









#!/bin/csh -f

#

#       Example 3b

#

if ( $#argv  < 1 ) then

  echo -n "Please enter your name: "

  set name = $<

  if ( "x$name" == "x") then

    echo "No name provided, stopping"

    exit 3

  endif

else

  set name = $1

endif

echo "Hello, $name, Welcome to FUSION ($0 called with $#argv parms)"





example/3> ./ex3b.sh
Please enter your name: 

No name provided, stopping

example/3> ./ex3b.sh
Please enter your name: Terry

Hello, Terry, Welcome to FUSION (./ex3b.sh called with 0 parms)



example/3> ./ex3b.sh
Please enter your name:  Terry Gliedt

Hello, Terry Gliedt, Welcome to FUSION (./ex3b.sh called with 0 parms)

example/3> ./ex3b.sh Terry

Hello, Terry, Welcome to FUSION (./ex3b.sh called with 1 parms)

example/3> ./ex3b.sh Terry Gliedt

Hello, Terry, Welcome to FUSION (./ex3b.sh called with 2 parms)
   (Note last name not picked up)








  Test for a file









#!/bin/csh -f

#       Example 3c

set f = /tmp/$USER.tmp

rm -f $f $f.ro $f.nothere



touch $f

touch $f.ro

chmod 444 $f.ro

echo "Created files"



foreach k ( $f $f.ro $f.nothere )

  if ( -r $k ) then

    echo "File '$k' is readable"

  endif

  if ( -w $k ) then

    echo "File '$k' is writable"

  endif

  if (! -e $k ) then

    echo "File '$k' does not exist"

  endif

end





example/3> ./ex3c.sh

Created files
File '/tmp/tpg.tmp' is readable
File '/tmp/tpg.tmp' is writable
File '/tmp/tpg.tmp.ro' is readable
File '/tmp/tpg.tmp.nothere' does not exist






  Test for a Directory









#!/bin/csh -f

#

#       Example 3d

#

set d = $HOME/temp



if (! -d $d) then

  mkdir $d

  if ( $status != 0 ) then

    echo "Unable to create '$d'"

    exit 3

  endif

  echo "Created directory '$d'"

else

  echo "Directory '$d' exists"

endif






example/3> ./ex3d.sh

Created directory '/afs/sph.umich.edu/user/t/tpg/temp'

example/3> ./ex3d.sh

Directory '/afs/sph.umich.edu/user/t/tpg/temp' exists









  Handle Options

    Usage: ex3e.sh [-bell] [-number N] infile









#!/bin/csh -f

#       Example 3e

#       ex3e.sh [-bell] [-number N] infile

set b = 'not set'

set n = 5

set infile = ''



echo "$#argv parameters found"

while ($#argv > 0)

  switch ($argv[1])

    case -b*:

      set b = 'set'

      breaksw

    case -n*:

      set n = $argv[2]

      shift

      breaksw

    case -*:

      echo "Unknown option '$argv[1]'"

      exit 4

    default:            # Not an option, must be file

      set infile = $argv[1]

  endsw

  shift

end

echo "Option b=$b"

echo "Option n=$n"

echo "Infile=$infile"






example/3> ./ex3e.sh

0 parameters found
Option b=not set
Option n=5
Infile=

example/3> ./ex3e.sh in.file

1 parameters found
Option b=not set
Option n=5
Infile=in.file

example/3> ./ex3e.sh -b in.txt

2 parameters found
Option b=set
Option n=5
Infile=in.txt

example/3> ./ex3e.sh -n 88 -b in.txt

4 parameters found
Option b=set
Option n=88
Infile=in.txt













  Top
  TCSH
  Regular
Expressions
  Redirection
  foreach
  Backticks
  Ex: SAS errors
  Ex: Families
  Ex: Columns of Data
  CSH Scripts
  How To Do...
  Ex: 3 Line Compare







3 Line Compare





Input Files
D20S103  PROBAND  UNAFFECT   3.592   1.028   1.028   0.812   1.663  99  99  91
   20                       -0.695   1.686   1.686   1.828   1.158
  0.0 cM                   0.73475 0.89207 0.74740 0.83270 0.73067
D20S117  PROBAND  UNAFFECT  12.030   4.501   4.501   1.258   4.292 177 177 197
   20                       -0.190   7.299   7.299   1.243   1.269
  0.5 cM                   0.53250 0.30840 0.19323 0.91678 0.71270
D20S906  PROBAND  UNAFFECT   6.030   2.055   1.472   2.055   3.055 217 231 217
   20                       -0.700   1.194   2.948   1.194   1.221
  5.1 cM                   0.76160 0.71560 0.79160 0.47232 0.57705
D20S193  PROBAND  UNAFFECT  23.239   8.375   3.346   8.375  11.806 159 165 159
   20                        2.294   2.156   2.927   2.156   1.514
  9.1 cM                   0.02070 0.03135 0.33125 0.01230 0.03275
D20S889  PROBAND  UNAFFECT  18.962   3.805   3.805   2.468   9.302 226 226 204
   20                        0.336   2.017   2.017   4.947   1.467
 10.0 cM                   0.32760 0.54832 0.38755 0.62552 0.38638
D20S482  PROBAND  UNAFFECT   7.846   4.641   0.893   4.641   1.915 167 143 167
   20                       -0.038   1.918   1.763   1.918   1.190
 11.8 cM                   0.45190 0.20505 0.90453 0.09585 0.71675
D20S849  PROBAND  UNAFFECT   9.800   2.076   1.458   2.076   4.166 214 235 214
   20                       -0.628   1.219   2.919   1.219   1.282
 14.8 cM                   0.76900 0.79002 0.88248 0.56942 0.51372
D20S905  PROBAND  UNAFFECT   9.733   2.734   2.486   2.734   1.107  79  87  79
   20                       -0.060   5.479   1.931   5.479   1.128
 18.6 cM                   0.47130 0.62035 0.47642 0.38338 0.91287








#!/bin/csh -f

#

#   Read a file in 3 line chunks and check certain values

#   that are less than some threshold.

#

set me = `basename $0`

set threshhold = "0.50"         # Show lines < this



#   Identify the file to be read

if ( "x$1" == "x-h" ) then

  echo "Usage: $me < infile"

  echo ""

  echo "Read 'infile' in 3 line chunks and show lines"

  echo " with certain values less than some threshhold"

  exit 1

endif




#  Read until EOF in three line chunks. If the file is "short"

#  the data will come from the terminal.

set eof = "n"

while ( $eof == "n" )

  set line1 = $<

  if ( "x$line1" == "x" ) then

    break

  endif

  set line2 = $<                # Not actually used

  if ( "x$line2" == "x" ) then

    echo "$me - Premature end of file"

  exit 3

  endif

  set line3 = $<                # E.g. 0.0 cM 0.92 0.96 0.88 0.88 0.87

  if ( "x$line3" == "x" ) then

    echo "$me - Premature end of file"

    exit 3

  endif



  #  Check 3rd, 5th ns 7th values against the threshhold

  set l = ( $line3 )            # Tokenize this line

  #echo "==>$l[3] $l[5] $l[7]"

  set rc3 = `expr $threshhold \>= $l[3]`

  set rc5 = `expr $threshhold \>= $l[5]`

  set rc7 = `expr $threshhold \>= $l[7]`

  if ( $rc3 != 0 || $rc5 != 0 || $rc7 != 0 ) then

    echo $line1

    echo $line3

  endif

end







example/5> ./ex5a.sh -h

Usage: ex5a.sh < infile
 
Read 'infile' in three line chunks and show those lines
 with certain values less than some threshhold

example/5> ./ex5a.sh < chr20.sho

D20S117 PROBAND UNAFFECT 12.030 4.501 4.501 1.258 4.292 177 177 197
0.5 cM 0.53250 0.30840 0.19323 0.91678 0.71270
D20S193 PROBAND UNAFFECT 23.239 8.375 3.346 8.375 11.806 159 165 159
9.1 cM 0.02070 0.03135 0.33125 0.01230 0.03275
D20S889 PROBAND UNAFFECT 18.962 3.805 3.805 2.468 9.302 226 226 204
10.0 cM 0.32760 0.54832 0.38755 0.62552 0.38638
.
..
...