Sunday, November 28, 2010

diaspora on aws cloud install...

Wanted to try and see if an aws m1.tiny is a good enough development instance for diaspora. It is a little small in the way of resources, but was certainly worth a shot to see if it will run. Summary, it does run, but needs a little more horsepower in my opinion, but if you are cheap and can wait a bit for all the ruby stuff to spin up, you will be in good shape! The stuff below was all executed on our same (m1.tiny) amazon instance from the previous post over at:

So here we go, using this wonderful guide as a reference:

Saturday, November 27, 2010

i've been watching this one grow for a while now

[] - repost from my micro blog site

I just set up a test account on one of the public pods:

There are more searching tools over here @

The alpha is also taking off more publicly any day soon! Hope we see more "pods" starting to spin up, this coupled with some xmpp could be what the IRC killer google wave never quite managed to be. 

*fingers crossed!*

(via email)

found tumblr

Friday, November 26, 2010

found flickr

In the words of Jesse{*}: 

This week I shall be mainly using flickr  ;-)  


update: recently discovered all previous flickr bets may well be off tumblr + clearly == epic win!

400Gb/s photo sharing

Inside a 400Gbit / second 2U switch
we have this bad boy on an eval for HPC right now

my first IEEE talk!

As a former molecular biophysicist, I'm inordinately excited to be talking over at IEEE Boston.

Computer Society and GBC/ACM 

7:00 PM, Thursday, 16 December

MIT building E51, room 345
(corner of Wadsworth, Amherst and Memorial Dr.)

"Introducing Energy Efficient Green Computing while expanding resources to support exponential growth in research computing data"

Hope to see some of you there! 

Thursday, November 25, 2010

ghetto lustre bug nearly donates my whole group a set of fresh pink slips

To: Prof. of famous discovery; Prof. of very famous things;
Cc: Some Assorted Deans

From: James Cuff
Subject: Groveling 

Dear Professors of all career holding...

We are still inordinately sorry about that whole missing set of galaxies thing, and those few lost irreplaceable genomes that kinda happened as a result of our scratch disks conking out. Although I do have a wizard idea where we can most likely write this all up together in Science as a new complex black hole / mythical dark matter or junk dna phenomenon? Yeah, I know maybe not; you each always told me that we best leave such rabid extrapolations to our atmospheric chemistry department...

Anyway, I digress. I've attached our latest in-depth analysis from our best top industry experts below. I hope this is enough information to not get my whole team fired?

Wednesday, November 24, 2010

enterprise storage woes [part one] - who is pounding bob?

It is 3am. The nagios pages are going bonkers. The emails from the faculty are already coming in thick and fast. It is raining cats and dogs outside. You are 25 miles away from the data centre and your OBM systems seem to have all failed on you! Who ya' gonna call? Storage busters? Hrrm no, sorry captain: it's all on you mr. sysadm buddy! Fortunately, there is a noticeable correlation to each alert and event:

Nov 24, 2010 ... kernel: nfs server "bob" not responding, still trying...

Oh great! So it's that time again! Someone or something is kicking the everliving daylights out of poor old file server bob again (don't know why, but it is always bob that gets himself a serious pounding). We've all been there, seen it, bought the t-shirts etc... It never ends well when there are ca. 2,000+ running accounts and 12,000 possible cpu to hunt through to find out who, or what is the culprit.  Time to start a digging, we have to save bob, and the day! 

First up 
nfswatch to the rescue!  Never, ever leave home without it kids!  Oh but, hang on, this is not one of our regular ghetto linux nfs boxes... pants!

This time, fileserver bob is part of a huge, honking EMC NS960. bob is a very lovely machine, he can scale to 1.8PB raw, and has 8x10GB/s network intercepts with failover and all sorts of magic. He is a really big boy, capable of wonderful transfer rates. Unfortunately we also have 12,000 hungry clients asking for both tiny and large files for a cosmology simulation. Hrrm, there exists no such thing as nfswatch for bob. 

However, he does have a lovely java $1.5MM web two dot fail browser to look at all sorts of other things - but alas we can't tell which client is beating upon on his disks. It is now time to bust out some perl. It is time to try a totally different, much lower budget track to nail this issue...

or why this sketchy perl script helped our protagonist save the day! - coming soon to a cinema near you!

oops cloud goes pop, rains a little

So, the cloud was looking totally awesome!!! Michele and I (Michele is working extremely hard, and enjoying life as a senior scientific consultant over @ bioteam) were quickly installing her super nifty new minilims laboratory management code.  It was supposed to be a simple proof of concept for Michele and I to test out the t1.micro instance to let us see how mini, tiny, or micro a "cloud node" really needs to be to run her fully fledged LIMS application she designed for next generation sequence data. It was all going frightfully well until we started the part that needed a smidgen of java for her installation code:

sudo apt-get install openjdk-6-jdk

results in an epic AWS kernel lock up @ 100% CPU fail with this dmesg error:
[103545.754170] kernel BUG at /build/buildd/linux-2.6.35/mm/mmap.c:2391!
[103545.754177] invalid opcode: 0000 [#2] SMP
[103545.754183] last sysfs file:
[103545.754191] Modules linked in: acpiphp

Oh dear - I really do know that one, it's a kernel bug!

Tuesday, November 23, 2010

hey you get on my tiny micro aws cloud ;-)

So, it took me a while to get with this whole cloud thing.  The hype seemed bigger than the reality for HPC at least, so time to dig in and see what's occurring.  Turns out it's pretty sweet for a cheap guy (did I tell you that I have a tiny budget?).  

Time to take up the offer for a t1.micro instance over at the amazon free tier for a whirl.  Micro is 2 cents an hour, less than the 8.5 cent "small" offering.  However they let you run up an instance for a month for no charge, and they even bundle some extra gadgets to make it all work out with a hint of extra awesome.  Our great colleagues on the interwebs helped me make sure that you can also spin up a 10.10 Ubuntu Maverick release in under 10GB of elastic block storage - I did mine in slightly over 5GB - hurray!.

Many, many thanks to Scott Moser with his epic link over here for <15GB Ubuntu 10.10 installs!

Monday, November 15, 2010

ghetto HFS+ file recovery

So, this one is pretty sweet. My wonderful wife had a minor accident with a customer's sequence data and a slightly botched tar cvf, (never forget the - option!) that resulted in a clobbered 10GB file of non recoverable data that was shipped to us on a honking big USB drive. The disk was a 2TB HFS+ filesystem (always a brute to recover from - most folks say that it is pretty much impossible).

So we set about and downloaded Data Rescue III (very cool program). However, it was predicting >150 hours to first scan the drive for lost files. Even if successful, it would only recover 10MB in demo mode, so $100 would have been needed to get this back, and possibly a week of scan time. However, Michele had an idea of what the file contained so was able to reduce this into a few seconds by doing a random search through the disk with dd. 
Really rather useful - and yet again proves that dd is one of the true swiss army knife cli applications of UNIX!

Sunday, November 14, 2010

welcome to!


I'm attempting to capture the fun and crazy world of High Performance Computing in an academic setting.  I intend to post various bits and pieces that I happen to stumble across in my travels through modern HPC, GPGPU systems and the complete and utter bane of my existence - 'enterprise' storage arrays.

We are primarily a large scale UNIX shop with a focus on trying to help our scientists and faculty get their work and research done without us or, more importantly the computer systems getting in the way.  More often than not the majority of the systems we end up putting together are somewhat erm... 'creative' to say the very least!

It ought to be fun!



(c) 2018 James Cuff