Thursday, October 23, 2014

OdyBot And Pointy Haired Alerting! AKA: Grumpy old man shakes fist at web two dot oh!

Hi all,

It's been ages since I last posted. We have been super busy in the day job running our monster high performance computing infrastructure and keeping lots of petabytes spinning and many scientists and researchers happy. I wanted to quickly note that we recently had a new addition to our group, supporting the Odyssey cluster.

Let me introduce:


Behind the scenes there are all sorts of fun activities, like checking that our data centers are neat and tidy and doing lots of awesome science:

from Harvard FAS Research Computing on Vimeo.

and sometimes just chilling out riding a skateboard around the yard:

OdyBot Gets Schooled
from Harvard FAS Research Computing on Vimeo.

You can find out all about OdyBot over at

Meanwhile we have had a couple of integration issues back at the ranch making sure that our awesome RC support staff are able to answer questions as our community asks them, and I think my old school methods finally got the better of me yesterday...

So, we use two online web services Userlike and hipchat to provide our external "voice" for OdyBot and for our internal communications. We wanted a quick way to post an alert to our main chat room when the operator count became zero.

Simple eh?

Well, kinda... and given that I'm not a child of the web 2.0 world I went about it in true UNIX style. A python script, with a unix pipe to a perl script... I'm sure sometimes I just do this stuff to wind up my team. :-) Anyway, here's the hipchat part, based on the awesome script with the following 2 second changes to allow it to read from stdin, and quote out the <CR>'s for all that down stream JSON cleverness...
bash-3.2$ diff 
>             "message=s"=> \$optionMessage,
< while (<>){
<   $optionMessage .= $_;
< }
< $optionMessage=~s/\n/\\n/g;

The nice thing about this is you can quickly post random stdin stuffs to you chat rooms:

Then I busted out some extremely suspect python:
bash-3.2$ cat 
import httplib2
import os
import json

API_HOST = os.environ.get('API_HOST') or ''
c = 0 
h = httplib2.Http()

resp, content = h.request(
        'Authorization': API_TOKEN
if resp.status == 200:
    data = json.loads(content)

    print 'Error status=%s' % resp.status

for x in data:
    if (x['slots']['online'] == 1):
      c = c + 1

cc = 1;

if (c>0):
    print "OdyBot Operators Online..."
    for x in data:
      if (x['slots']['online'] == 1):
        print ("%d) %s" % (cc, x['name']))
        cc = cc + 1

    print "Warning!  No OdyBot Ops Are Online!!"

Which works by just querying the Userlike API:
bash-3.2$ ./ 
OdyBot Operators Online...
1) James Cuff
2) Bob Freeman
3) John Noss
4) Dan Caunt

And there you have it - simple alerting to the main chat room when you need it to tell you that folks are not on the wire taking our OdyBot support requests! This is actually pretty important, we absolutely don't want to have our community waiting, and we have invested a lot of time and effort into the OdyBot concept so that our community can contact us. We also have an open OdyBot community list that folks inside and outside of Harvard can use to post questions, tips and techniques, although it's only just starting to ramp up right now.

Question is:

Could you do this integration in an even more ghetto/bandaid/bailing wire fashion :-)

p.s. we are also testing SLACK as a replacement for hipchat as I type. For pretty much the exact same reasons we had to replace Zopim with Userlike yesterday - as you can see, even with shoddy perl and python scripts, it is all about the integration these days!

No matter how ghetto the methods ;-) #allhailtheunixpipe

[any opinions here are all mine, and have absolutely nothing to do with my employer]
(c) 2011 James Cuff