Wednesday, November 24, 2010

enterprise storage woes [part one] - who is pounding bob?

It is 3am. The nagios pages are going bonkers. The emails from the faculty are already coming in thick and fast. It is raining cats and dogs outside. You are 25 miles away from the data centre and your OBM systems seem to have all failed on you! Who ya' gonna call? Storage busters? Hrrm no, sorry captain: it's all on you mr. sysadm buddy! Fortunately, there is a noticeable correlation to each alert and event:

Nov 24, 2010 ... kernel: nfs server "bob" not responding, still trying...

Oh great! So it's that time again! Someone or something is kicking the everliving daylights out of poor old file server bob again (don't know why, but it is always bob that gets himself a serious pounding). We've all been there, seen it, bought the t-shirts etc... It never ends well when there are ca. 2,000+ running accounts and 12,000 possible cpu to hunt through to find out who, or what is the culprit.  Time to start a digging, we have to save bob, and the day! 

First up 
nfswatch to the rescue!  Never, ever leave home without it kids!  Oh but, hang on, this is not one of our regular ghetto linux nfs boxes... pants!

This time, fileserver bob is part of a huge, honking EMC NS960. bob is a very lovely machine, he can scale to 1.8PB raw, and has 8x10GB/s network intercepts with failover and all sorts of magic. He is a really big boy, capable of wonderful transfer rates. Unfortunately we also have 12,000 hungry clients asking for both tiny and large files for a cosmology simulation. Hrrm, there exists no such thing as nfswatch for bob. 

However, he does have a lovely java $1.5MM web two dot fail browser to look at all sorts of other things - but alas we can't tell which client is beating upon on his disks. It is now time to bust out some perl. It is time to try a totally different, much lower budget track to nail this issue...

or why this sketchy perl script helped our protagonist save the day! - coming soon to a cinema near you!

[root@ithaca-ctrl nasadmin]# cat /usr/bin/
if ($ARGV[0] eq ""){die "Usage: server_2 or ALL\n"};
# map is based on:
%servermap = (
        server_2 => 'root_vdm_1',
        server_3 => 'root_vdm_2',
        server_4 => 'root_vdm_3',
        server_5 => 'root_vdm_4',
        server_6 => 'root_vdm_5',
        server_7 => 'root_vdm_6',
        server_8 => 'root_vdm_7',
if ($ARGV[0] eq "ALL"){
 foreach $host (sort keys %servermap){
else {capture()};
sub capture {
  $rootloc = $servermap{$thisserver};
  ($server,$slot) = split("_",$thisserver);
  $remotecapfile = "/$rootloc/jctmp.cap";
  $localcapfile = "/nas/rootfs/slot_$slot$remotecapfile";
  system ("/nas/sbin/server_tcpdump $thisserver -start fsn0 -s 1500 -max 20000 -w $remotecapfile > /dev/null");
  # sleep a while to let the packetz capture ;-)
  print "Standby: we is waiting for teh packetz from $thisserver...\n";
  sleep 4;
  system ("/nas/sbin/server_tcpdump $thisserver -stop fsn0 > /dev/null");
  open (IN, "/usr/sbin/tcpdump -r $localcapfile src port nfs or src port microsoft-ds |");
    @array=split(/ /,$_);
    $clport=pop @client;
    $slport=pop @server;
    if($clport == "nfs" || $clport == "microsoft"){
      $hashline = "$clport: $cl\t\t$sr";
      $hashline = sprintf ("%-20s %-32s %-32s",$clport,$cl,$sr);
      $msg{$hashline} = $msg{$hashline} ? $msg{$hashline} + 1 : 1; 
  printf ("\n\n%12s %-20s %-32s %-32s\n","Packetz","Proto","Virtual Server Name","Client Address");
  print "+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n";
  for(sort { $msg{$b} <=> $msg{$a} } keys %msg) { 
        ($j,$j,$keepclient[$i])=split(" ",$_);
        printf ("%12s %s\n",$msg{$_},$_); 
  print "+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n\n\n";
  undef %msg;
  undef @keepclient;
  undef @array;
  undef @client;
  undef @server;

[any opinions here are all mine, and have absolutely nothing to do with my employer]
(c) 2011 James Cuff