Tuesday, February 28, 2012


A few days ago I was asked what one ought do to get a job as a system admin.

I no longer hire systems admins

However, I wrote the following rant ^H^H^H^H advice to a very talented CS major currently taking finals at a high quality university... I've not seen a response from them yet.

Question: Was I out of line?

From:  James Cuff 
Date:  Feb 21 (8 days ago)


So the new trend these days is all about what they are calling
"devops".  The days of the lone systems admin (to me at least) are
pretty much dead.  This is where you have a win - if you can code you
are already 100% more employable than a regular old systems admin

These days if you can't code and stand up automatic systems and then 
come to me for a job, I'll never even give you a start.  Sorry.

If you look at technology like Puppet, Chef, and the new
eucalyptus/openstack stuff this is where the world is heading.  

Given all the big boys are on "scale out", knowing how to stand up an
openstack environment running SGE or say OpenLava that is all
automatically provisioned using say puppet would be a very marketable
thing these days...

Look at those big boys, they are moving to commodity equipment, the
systems are basically just rip and replace.  

But that software? - Oh that is money!

FB/google/MS/Apple etc. they are all about the scale out.  

An individual system in an orchestrated cloud has become the way of 
what systems integrators used to do back in the day... 

Michael Dell killed off all those sys integration boys, and devops 
is equally going to kill off the sysadm...

If you wanted a project, you could not go far wrong designing a small
cluster (5 or 6 vm's) and getting them all orchestrated with
openstack.  Other hot topics are parallel file systems - Hadoop,
Lustre, gluster, ceph are a few that are big - some posix, some
object based.  These are the new scale out challenges for sure!

Anyway, here are some links to whet your appetite!


Hope this helps!


dr. james cuff, director of research computing & chief technology
architect harvard university | faculty of arts and sciences
slightly edited for style and content

Update from Philip Durbin!

Sunday, February 19, 2012

what's in a name?

For a while computers have been anthropomorphized(*). We know they are finite state devices, they are supposed to only simply add numbers together right?

Wrong, especially when they are clustered they become almost human.

I'm not kidding - they start to take on strange quirks and attributes that make them cease to behave in a linear fashion. They come "alive". Not quite the whole 2001 space odyssey "dave i'm sorry i can't do that" sort of alive, but they are certainly aware, and they just know you are trying to fix them. They will resist. My take:

Computers: yeah they just want to reach higher levels of entropy(*)!

So we attempt to solve some of this fear by giving them names. The names represent the spirit of what we are trying to achieve, we attempt to render them more harmless. They are often also pretty obvious. This is not a new thing. Think of animals, cats we call "kitty", dogs we call "spot" (if they have spots) and rabbits well we often call "thumper", my childhood bunny was "bugsy". Anyway you get the gist.

We humans are pretty crappy when it comes to giving stuff names.

There are times we get really quite "meta" about the whole thing. For example, the monster consolidation of equipment I have been attempting in my day job I give the meta cluster name:


Odyssey | ˈädəsē|
a Greek epic poem traditionally ascribed to Homer, describing the travels of Odysseus during his ten years of wandering after the fall of Troy. He eventually returned home to Ithaca and killed the suitors who had plagued his wife Penelope during his absence.

Yeah I'm totally original eh? How long did it take to come up with that idea. heheheh. It got way worse when we stupidly named our storage arrays based on the same theme... we ended up with this utter fiasco:

iliad, hero, ismaros, phaiakia, troy, charybdis, helios, skylla, aiolos

Word to the wise. Please don't ever do this. you will end up with all your storage hung off "troy". Why? Because no admin (or sane person) can be arsed to type out or remember how to spell "phaiakia" it literally happened to us, I kid you not.

It's not just me. I was literally fixing a set of machines for Oceanographic Earth Science tonight. What were there names then? Oh go on... you will never guess... this one is a tough one...

swell, wave and crest!

Oh hells yeah - there we go again with our rampant originality! We are totally cooking on gas eh? I worked on a large cancer genomics project once. We had a two node DS20 cluster. They worked as a pair. What were their names?

Machine one: Benson

wait for it....

Machine two: Hedges

Oh how we laughed... :-)

Other famous machines at the EBI were called "Gin" and "Tonic"

Michele and I even went through a phase of calling machines proper names, it started off so simple, we called them:

dave, bernard, steve, bill and bob

it was all fun and games until we realized we had no physical mapping to where each machine was... we literally lost dave for about three weeks...! We eventually found him wandering around a sketchy and rather dodgy district inside some random cloud. (cough).

On the theme of proper names, we even named one of our top priority queues "nancy" after our previous Dean of Finance and Administration. She got a real good kick out of knowing the faculty when running their jobs would have to "submit to nancy". heh.

And so it is now. And so it will ever be...

We have been working with our pals in Clean Energy for a while now, and are about to add some more kit to their cluster inside our Odyssey environment. We always get some fun names from these guys, they are pretty creative...

so if you think you have new ideas let either @A_Aspuru_Guzik or @jamesdotcuff know. Here's Alan's original tweet:


(*) pseudo posh word glossary follows:
anthropomorphize | ˌanθrəpəˈmôrˌfizəm | verb
the attribution of human characteristics or behavior to a god, animal, or object

entropy | ˈentrəpē | noun Physics
a thermodynamic quantity representing the unavailability of a system's thermal energy for conversion into mechanical work, often interpreted as the degree of disorder or randomness in the system

Update June 10th
Rodrigo Lopez, an old chum from the EBI reminded me that we had another set of machines called "Gin" and "Tonic" at the EBI back in the day...

The name game will always be fun!

[any opinions here are all mine, and have absolutely nothing to do with my employer]
(c) 2011 James Cuff