Tuesday, January 10, 2012

eyes on the ball...

Today was one of those tough manager moments.

We have an amazing team, but often we need folks to be available off hours and able to pick up on when computers fail, and yet science still needs to happen. Most often this is late on at night or on a weekend or more recently during a holiday, or even more so all of the above!

We wanted to have a super fair, and super simple algorithm so that folks on the team were not penalized or over burdened. We came up with this simple approach, 9-5 is already covered, we all do that already. However for those hours between 6-11pm and the weekends we basically need a "lookout".

The role of this individual is to be the "keep eyes peeled for strange things". This job is not for those monster huge "oh my the sky is falling" critical incidents - we have other ways of dealing with those :-)

More for those requests that would be really awesome to answer before the morning, say because a researcher could really do with their nodes online because of a grant deadline, or maybe a simple binary fix to get an algorithm back up and running again.

So we came up with the idea that folks would be able to rotate in and out of this role once a week. We have quite a few head count so the burden was not all that high. We also wanted this job to be a triage and learning opportunity. The algorithm for this "eyeballs on call" is:

while(EyesOnBall == 1)
  watch_rt_nagios_alerts();
  take_ticket(attempt fix);
  if (Can'tFixItOnYourOwn) 
     try raise_a_team_member();
  else record_ticket(forMorning == 1);
  document_ticket(withInfoFound);
loop;

Then really, really simply we had perl come to the rescue to build the schema for this idiot proof rotation! Our only challenge right now is how to get this "calendar" to import into Exchange, but then again everything we do has issues with Exchange, so no harm no foul I guess :-)

use Date::Calc qw(Monday_of_Week);
All hail! The all powerful Date::Calc!

We ended up wil an incredibly simple script to print out the Monday of the week that the "guard would change", and who on the rotation would be taking care of being the fresh eyeballs for the upcoming week:
#!/usr/bin/perl
use Date::Calc qw(Monday_of_Week Weeks_in_Year);
@s=("Staff A","Staff B","Staff C","Staff D","Staff E"); $z=0;
for ($i=1;$i<=Weeks_in_Year($ARGV[0]);$i++){
   printf ("%-15s %-4d %-4d %-4d\n",$s[$z],Monday_of_Week($i,$ARGV[0]));
   $z++; if ($z>$#s){$z=0};
}
http://micro.jcuff.net/desperation


toofan: jcuff$ ./rot.pl 2012
Staff A         1    2    2012
Staff B         1    9    2012
Staff C         1    16   2012
Staff D         1    23   2012
Staff E         1    30   2012
Staff A         2    6    2012
Staff B         2    13   2012
Staff C         2    20   2012
Staff D         2    27   2012
Staff E         3    5    2012
Staff A         3    12   2012
Staff B         3    19   2012
Staff C         3    26   2012
Staff D         4    2    2012
Staff E         4    9    2012
Staff A         4    16   2012
Staff B         4    23   2012
Staff C         4    30   2012
Staff D         5    7    2012
Staff E         5    14   2012
Staff A         5    21   2012
Staff B         5    28   2012
Staff C         6    4    2012
Staff D         6    11   2012
Staff E         6    18   2012
Staff A         6    25   2012
Staff B         7    2    2012
Staff C         7    9    2012
Staff D         7    16   2012
Staff E         7    23   2012
Staff A         7    30   2012
Staff B         8    6    2012
Staff C         8    13   2012
Staff D         8    20   2012
Staff E         8    27   2012
Staff A         9    3    2012
Staff B         9    10   2012
Staff C         9    17   2012
Staff D         9    24   2012
Staff E         10   1    2012
Staff A         10   8    2012
Staff B         10   15   2012
Staff C         10   22   2012
Staff D         10   29   2012
Staff E         11   5    2012
Staff A         11   12   2012
Staff B         11   19   2012
Staff C         11   26   2012
Staff D         12   3    2012
Staff E         12   10   2012
Staff A         12   17   2012
Staff B         12   24   2012

It is so simple it hurts, but I personally think it totally reflects the consumate style of research computing that it is not funny :-)

I'll let you know how we get on with this, it's basically what I'm calling "on call lite"!



[any opinions here are all mine, and have absolutely nothing to do with my employer]
(c) 2011 James Cuff