Cluster Dump

Cluster Dump

for

the CSG

Terry Gliedt
Department of Biostatistics
School of Public Health
University of Michigan

CLUSTER DUMP - MACC + DC

What clusters
How do MOSIX clusters work ?
How do I run my programs on MOSIX ?
Running programs defensively
Running lots of programs

References

http://csg.sph.umich.edu/docs/cluster/
Contains links to

Introduction
Pictures
Using MOSIX
Performance details
Cluster Dump Presentation

What Clusters

CSG a.k.a. snowwhite/fantasia or MACC Cluster

Remote facility on State Street
+ a couple of machines in SPH (e.g. peterpan)
Funded by Mike B oehnke and Goncalo Abecasis
Gateways:

snowwhite 64 bit, Boehnke folks, FUSION + PRITZKER
fantasia 64 bit, Goncalo folks
wonderland 64 bit, Goncalo friends
peterpan 32 bit, Goncalo folks

Depression Center a.k.a. Frodo Cluster

Remote facility at Depression Center off Plymouth Road
Funded by Sebastian Zoellner
Gateways:

frodo 64 bit, Zoellner folks

How do MOSIX clusters work ?

Run program: preface your normal command with 'runon'
E.g. runon demo1.pl

demo1.pl process is spawned on gateway, runs a very short time
MOSIX stops process, finds another node, moves memory to that machine, processing continues
demo1.pl on client issues 'system call'
System call and data is returned to gateway where it is executed
Results are returned to client, continue running

  #!/usr/bin/perl
  #   demo1.pl - a form of the Perl Dump 'universal program'
  use warnings;
  use strict;
  my $file = 'mydata.input';          # File to read
  open(IN,$file) ||
      die "Unable to read file '$file': $!\n";
  my @data = ();
  while (my $line = <IN>) {
      chomp($line);
      push @data,$line;
  }
  close(IN);

  $file = 'mydata.output';            # File to create
  open(OUT,'>' . $file) ||
      die "Unable to create file '$file': $!\n";
  foreach my $l (reverse @data) {
      print OUT $l + 1 . "\n";
  }
  close(OUT);
  exit;

How do MOSIX clusters work ?

Launch a bunch of work (original from Liming)

  #!/usr/bin/perl
  #   Launch a mach1 run for each chromosome. Note all details are hardcoded
  use warnings;
  use strict;

  foreach my $chr (1..22) {
      my $command = "runon mach1 -d ~/AD_meta/combined/dat/Berlin_Kiel_Munich_combined_chr_$chr.dat " .
          "-p ~/AD_meta/combined/Berlin_Kiel_Munich_combined_400random.ped --autoFlip " .
          "-h ~/asthma/imputation/r3_haplo/chr".$chr."-character.r3.haplo " .
          "-s ~/asthma/imputation/r3_snps/chr$chr.r3.snps " .
          "--greedy -r 100 --prefix chr$chr > printout_chr$chr.txt &";
      print "Submitting Chr $chr\n";
       #print($command."\n");
      system($command);
      sleep 10;               # Give MOSIX a chance to work
  }

mon - How do I run my programs on MOSIX ?

'top' will show your process on gateway, but it gets no time
mon - to see how busy things are

Node numbers are not obvious. The first one (DC) or four (MACC) are gateways, rest are clients
mon accepts keyboard input to show other details
Enter 'h' to get summary of meanings of keys
Avoid submitting work if the cluster is 'too full'

Full =~ all processors busy (e.g. mon load near 1.0 on each node)
In practice we sometimes cannot even tolerate that much
Goncalo will tell you (maybe after he's stopped your work)

mosps - How do I run my programs on MOSIX ?

mosps -aeu - to see what's running
WHERE is number of node where it is running. This can change.
'man mosps' will explain options

~> mosps -aeu
USER       PID WHERE FROM CLASS FRZ TTY START CMD
ylwtx    21755 3     here local  -  ?   09:17 /usr/bin/mosrun -b -e -t mach1 -d dat/chr3.dat -p
ylwtx    21771 3     here local  -  ?   09:17 /usr/bin/mosrun -b -e -t mach1 -d dat/chr6.dat -p
ylwtx    21777 3     here local  -  ?   09:17 /usr/bin/mosrun -b -e -t mach1 -d dat/chr3.dat -p
ylwtx    21783 3     here local  -  ?   09:18 /usr/bin/mosrun -b -e -t mach1 -d dat/chr6.dat -p
wguan    24110 73    here local  -  ?   12:19 mosrun -m2048 /home/wguan/similarity/scripts/sour


~> mosps -aeu | grep wguan
wguan    24110 73    here local  -  ?         12:19 mosrun -m2048 /home/wguan/similarity/scripts/sour

Running programs defensively

MOSIX Weaknesses

No support for threads
Certain system calls are not supported - cannot do 'runon sas ...'
If there is MOSIX problem and we are lucky, only the running command is killed
When gateway fails, all processes on clients from that gateway are stopped

Gateway memory overwhelmed

Servicing too many system calls
Too much data in system calls

Programming failure in MOSIX (never these days)

Make your work resumable

If a single merlin run is stopped, all is lost
If running 100 merlin runs sequencially, don't start from the beginning

Running programs defensively - Simple

Launch a bunch of work (original from Jun)

  #!/bin/csh -f
  date > mout/nsp.6.startLog
  runon ~/bin/mach1-strand -d panel_in_HMP/dat/nsp.6.dat -t panel_in_HMP/strand/nsp.6.strand \
    -p panel_in_HMP/ped/nsp.6.only.ped -s ~/imputation_GWA/unionDat/r3_snps/chr6.r3.snps \
    -h ~/imputation_GWA/HMP2/r3_haplo/chr6.r3.haplo --greedy -r 10 -e 0.01 \
    --mle -o mout/nsp.6 > mout/nsp.6.mlog 
  date > mout/nsp.7.startLog
  runon ~/bin/mach1-strand -d panel_in_HMP/dat/nsp.7.dat -t panel_in_HMP/strand/nsp.7.strand \
    -p panel_in_HMP/ped/nsp.7.only.ped -s ~/imputation_GWA/unionDat/r3_snps/chr7.r3.snps 
    -h ~/imputation_GWA/HMP2/r3_haplo/chr7.r3.haplo --greedy -r 10 -e 0.01 \
    --mle -o mout/nsp.7 > mout/nsp.7.mlog 
  date > mout/nsp.8.startLog
  runon ~/bin/mach1-strand -d panel_in_HMP/dat/nsp.8.dat -t panel_in_HMP/strand/nsp.8.strand \
    -p panel_in_HMP/ped/nsp.8.only.ped -s ~/imputation_GWA/unionDat/r3_snps/chr8.r3.snps \
    -h ~/imputation_GWA/HMP2/r3_haplo/chr8.r3.haplo --greedy -r 10 -e 0.01 \
    --mle -o mout/nsp.8 > mout/nsp.8.mlog 

  etc...

Keep looking at mlog files to find out if something fails

Running programs defensively - Resumable

Lines in this color implement resumability

  #!/bin/csh -f 
  #   Do $n_sim gene-dropping simulations using merlin and
  #   save the lod and zmean values for a specific marker ($marker)
  #   to a results directory called ~/merlin_results
  #   Requires a file (random.numbers) with random seeds, one per line
  #
  #   Usage:
  #       demosimple.sh             # Normal start
  #       demosimple.sh  restart    # Continue from curr_simulation
  #
  set restart = "0"
  if ( "x$1" == "xrestart ) then
    set restart = 1
  endif
  set file = "asp"
  set marker = "MRK11"
  set work_dir = "~/merlin_work"
  set save_dir = "~/merlin_results"
  mkdir -p $work_dir      # Set up working and result directories
  mkdir -p $save_dir

  cp $file.ped $work_dir
  cp $file.dat $work_dir
  cp $file.map $work_dir

  #   Do actual computation in temporary work directory
  cd $work_dir
  echo "LOD" > lods
  echo "ZMEAN" > zmeans

  @ n_sims = 100
  if ($restart == "0") then
     @ sim = 1
  else
     @ sim = `sed -n '1p" curr_simulation"
  endif

  while ( $sim <= $n_sims )
     #    Save index for current simulation to file for restart in case of crash
     echo "$sim" > curr_simulation
     #    Select the ith random number from random.numbers
     set random = `sed -n "$sim q;d" random.numbers` 
     merlin -p $file.ped -d $file.dat -m $file.map \
        --npl --simulate --markerNames -r $random > merlin.output
     #    Save lod and zmean value for  $marker 
     set line = `grep $marker_name pairs`   
     set lod = `echo $line | tr -s ' ' ' ' | cut -d' ' -f5`
     set zmean = `echo $line | tr -s ' ' ' ' | cut -d' ' -f2`
     echo "$lod" >> lods
     echo "$zmean" >> zmeans 
     @ sim ++
  end
  rm -f merlin.output

  #   Write results to results directory
  mv lods $save_dir/z_means_$file
  mv zmeans $save_dir/lods_$file

  # Get rid of the working directory
  if ( $work_dir != "" ) then
      rm -rf $work_dir
  }

Running Lots of Programs - myjobs.pl

Hard coded approach
Copy a Perl program, modify it for your needs. Run it.
Runs all commands concurrently.
Could be used to create a log of what happened.
The sample program (myjobs.pl) lives in /usr/cluster/bin

  #!/usr/bin/perl -w
  #################################################################
  #
  # Name:	myjobs.pl
  #
  # Description:
  #   Everything here is hardcoded. Each command to run is
  #   almost exactly like the others with some small change
  #   like a chromosome number etc.
  #
  #   Copy and modify this program every time you want to run 
  #   a set of commands on the cluster.
  #
  #   This approach was inspired by Liming Liang (lianglim@umich.edu)
  #
  # ChangeLog:
  #   $Log: myjobs.pl,v $
  #   Revision 1.1  2008-09-10 17:08:24  tpg
  #   Initial version
  #
  #
  # This is free software; you can redistribute it and/or modify it under the
  # terms of the GNU General Public License as published by the Free Software
  # Foundation; See http://www.gnu.org/copyleft/gpl.html
  #
  ################################################################
  use strict;
  use warnings;
  use CSG::Cluster;               # Doc available with 'perldoc CSG::Cluster'

  #   If you want a log file, set it here, else set it to ''
  my $logfile = "/tmp/$ENV{USER}.log";

  #--------------------------------------------------------------
  #   Set up optional logfile
  #--------------------------------------------------------------
  if ($logfile) {
      #   Second parameter says 'replace $logfile if it exists'.
      #   You can also specify 'append' as the second parameter.
      CSG::Cluster::SetClusterLog($logfile, 'replace');
  }

  #--------------------------------------------------------------
  #   Run all the commands at once
  #   Each command is built from a model line
  #--------------------------------------------------------------
  my $model = 'mach1 -d ~/AD_meta/combined/dat/Berlin_Kiel_Munich_combined_chr_XX.dat ' .
      '-p ~/AD_meta/combined/Berlin_Kiel_Munich_combined_400random.ped ' .
      '--autoFlip -h ~/asthma/imputation/r3_haplo/chrXX-character.r3.haplo ' .
      '-s ~/asthma/imputation/r3_snps/chrXX.r3.snps ' .
      '--greedy -r 100 --prefix chrXX > printout_chrXX.txt';
  foreach my $n (1..22) {
      my $cmd = $model;
      $cmd =~ s/XX/$n/g;
      CSG::Cluster::RunOn($cmd);
  }

  if ($logfile) { print "Commands submitted, progress is logged to '$logfile'\n"; }

  #   Now wait for commands started by Runon above to complete
  CSG::Cluster::WaitForCommandsToComplete();
  exit;

Running Lots of Programs - runjobs.pl

Create file of commands to run in a file

#   File mymach.txt
#   Here are commands I want to run
mach1 -d ~/AD_meta/combined/dat/Berlin_Kiel_Munich_combined_chr_01.dat -p ~/AD_meta/combined/Berlin_Kiel_Munich_combined_400random.ped --autoFlip -h ~/asthma/imputation/r3_haplo/chr01-character.r3.haplo -s ~/asthma/imputation/r3_snps/chr01.r3.snps --greedy -r 100 --prefix chr01 > printout_chr01.txt
mach1 -d ~/AD_meta/combined/dat/Berlin_Kiel_Munich_combined_chr_02.dat -p ~/AD_meta/combined/Berlin_Kiel_Munich_combined_400random.ped --autoFlip -h ~/asthma/imputation/r3_haplo/chr02-character.r3.haplo -s ~/asthma/imputation/r3_snps/chr02.r3.snps --greedy -r 100 --prefix chr02 > printout_chr02.txt
mach1 -d ~/AD_meta/combined/dat/Berlin_Kiel_Munich_combined_chr_03.dat -p ~/AD_meta/combined/Berlin_Kiel_Munich_combined_400random.ped --autoFlip -h ~/asthma/imputation/r3_haplo/chr03-character.r3.haplo -s ~/asthma/imputation/r3_snps/chr03.r3.snps --greedy -r 100 --prefix chr03 > printout_chr03.txt
[lines truncated]

Invoke runjobs.pl with some options to run the commands
runjobs.pl lives in /usr/cluster/bin

  runjobs.pl -h

  runjobs.pl -poolsize 5  -replace -logfile terry.mach1.log   mymach.txt

Runs a few to start and then others as commands complete
Typically will create a logfile

tail -f terry.mach1.log         # Show lines as they are added to file

2008/09/10 12:27:33 : ####### Log started #######
2008/09/10 12:27:33 : Start runjoblist.pl t.jobs
2008/09/10 12:27:36 : Issuing: mach1 -d ~/AD_meta/combined/dat/Berlin_Kiel_Munich_combined_chr_01.dat ...
2008/09/10 12:27:40 : Issuing: mach1 -d ~/AD_meta/combined/dat/Berlin_Kiel_Munich_combined_chr_02.dat ...
2008/09/10 12:27:43 : Issuing: mach1 -d ~/AD_meta/combined/dat/Berlin_Kiel_Munich_combined_chr_03.dat ...
2008/09/10 12:27:46 : Issuing: mach1 -d ~/AD_meta/combined/dat/Berlin_Kiel_Munich_combined_chr_04.dat ...
2008/09/10 12:27:50 : Issuing: mach1 -d ~/AD_meta/combined/dat/Berlin_Kiel_Munich_combined_chr_05.dat ...
2008/09/10 13:35:06 : Command complete: runon mach1 -d ~/AD_meta/combined/dat/Berlin_Kiel_Munich_combined_chr_04.dat ...
2008/09/10 13:35:06 : Issuing: mach1 -d ~/AD_meta/combined/dat/Berlin_Kiel_Munich_combined_chr_06.dat ...
2008/09/10 13:55:22 : Command complete: runon mach1 -d ~/AD_meta/combined/dat/Berlin_Kiel_Munich_combined_chr_05.dat ...
2008/09/10 13:35:06 : Issuing: mach1 -d ~/AD_meta/combined/dat/Berlin_Kiel_Munich_combined_chr_07.dat ...
[lines deleted]
2008/09/11 03:15:11 : Command complete: runon mach1 -d ~/AD_meta/combined/dat/Berlin_Kiel_Munich_combined_chr_21.dat ...
2008/09/11 03:15:11 : ####### Log ended #######

Cluster Dump	Cluster Dump	unit1.htm
Terry Gliedt (tpg@hps.com)	Cluster Dump for BioStat

Cluster Dump for the CSG

Cluster Dump

for

the CSG