Example: Extracting information from a text file

A practice to get some information out of text file.

• This commands:

  ms 30 4 -t 3.0 | sample_stats
  

  prints out:

  pi:	4.232	ss:	13	D:	0.956	thetaH:	3.285	H:	0.947
  pi:	1.314	ss:	5	D:	0.114	thetaH:	0.478	H:	0.836
  pi:	2.540	ss:	8	D:	0.783	thetaH:	1.942	H:	0.597
  pi:	2.726	ss:	14	D:	-0.762	thetaH:	1.549	H:	1.177
  

  We are making a script to extract desired columns (only the values).

• Simple perl script (cleanSampleStatsSimple.pl ):

  #!/usr/bin/perl -w
  
  while(<>) {            # read in each line
      if (/pi:\s+(\S+)\s+ss:\s+(\S+)/) { # Regular Expression
  	my $pi = $1;
  	my $ss = $2;
  
  	print join("\t", ($pi, $ss));
  	print "\n";
      } else {
  	warn "WARN: This line does not follow the normal output pattern, " .
  	    "Ignored...:\n$_";
      }
  }
  
  exit;
  

• This can be used in the following way:

  ms 30 4 -t 3.0 | sample_stats | ./cleanSampleStatsSimple.pl
  

• You can get the same results with:

  ms 30 4 -t 3.0 | sample_stats | cut -f 2,4
  

Details: line-by-line processing, conditional, regular expressions

• Diamond operator

  while(<>) {
   ....;
   ....;
  }
  

  • This is used to process each line by line.
  • A special variable $_ contains a single line.
  • The process will keep looping until all lines are read.

  Example:

  print "Starting\n";
  $counter = 1;
  while (<>) {
    print "$counter : $_";
    $counter = $counter + 1;
  }
  print "End of file\n";
  

• Conditional:

  if (condition){
    do this 1;
    do this 2;
  } else {
    do this 3;
    do this 4;
  }
  

  If the condition (e.g. $num < 0) is true, it will do 1 & 2, and skip 3 & 4. If the condition is false, it will do 3 & 4 only.

  Example:

  $num = - 0.4
  if ($num < 0) {
    print "$num is negative\n";
  } else {
    print "$num is NOT negative\n";
  }
  

• Pattern matching with Regular Expressions

  if (/ ..... /) {
    do this 1;
  } else  {
    do this 2;
  }
  

  • For the condition, you can use some pattern enclosed by / /.
  • It checks if the special variable $_ contains the specified pattern.
  • Examples;
    • Text string

      /dog/
      

      This matches if $_ = "hotdog eating champ". Or any text containing dog.
    • Multiplier

      /Hmm+/
      

      Plus (+) means one or more of the immediately previous character.
      It mathces "Hmm" or "Hmmm" or "Hmmmmmmm".
      But not "Hm"
    • Special character

      /\s/   # A "space" character, space, tab, newline
      /\S/   # A non-space character (A, a, 1, 0, :, _, etc 
      /\d/   # A digit (0, 1, ... 9)
      /./    # matches any single character
      

    • Extraction
      

      Parentheses in a regular expression is used to extract information

      # extract hours, minutes, seconds
      if ($time =~ /(\d\d):(\d\d):(\d\d)/) {  # match hh:mm:ss format
        $hours = $1;
        $minutes = $2;
        $seconds = $3;
      }
      

• More information about regular expressions.
  • Official tutorial
  • Shorter tutorial

Exercise:

Modify the script to extract all five summary statistics.

Answer: Don't click this until you do it by yourself, cleanSampleStats.pl