Wednesday, November 24, 2010

oops cloud goes pop, rains a little

So, the cloud was looking totally awesome!!! Michele and I (Michele is working extremely hard, and enjoying life as a senior scientific consultant over @ bioteam) were quickly installing her super nifty new minilims laboratory management code.  It was supposed to be a simple proof of concept for Michele and I to test out the t1.micro instance to let us see how mini, tiny, or micro a "cloud node" really needs to be to run her fully fledged LIMS application she designed for next generation sequence data. It was all going frightfully well until we started the part that needed a smidgen of java for her installation code:

sudo apt-get install openjdk-6-jdk

results in an epic AWS kernel lock up @ 100% CPU fail with this dmesg error:
[103545.754170] kernel BUG at /build/buildd/linux-2.6.35/mm/mmap.c:2391!
[103545.754177] invalid opcode: 0000 [#2] SMP
[103545.754183] last sysfs file:
/sys/devices/system/cpu/cpu0/cache/index2/shared_cpu_map
[103545.754191] Modules linked in: acpiphp

Oh dear - I really do know that one, it's a kernel bug!




Turns out: https://bugs.launchpad.net/ubuntu/+source/linux-ec2/+bug/634487 has all the answers, and our good friend Scott Moser is already on the case *yet again* from the launchpad logs - that dude is everywhere, and it is all good ;-)!

one of the comments in the launchpad suggests:

   ec2-stop-instance ${IID}

   ec2-modify-instance-attribute --instance-type m1.small ${IID}
   ec2-start-instances ${IID}
   ssh to instance and do whatever you want
   ec2-stop-instances ${IID}
   ec2-modify-instance-attribute --instance-type t1.micro ${IID}

   ec2-start-instances ${IID}

has to be worth a shot right?  
So, ok then here we go:

jcuff@shuttle:~$ cd amazon/

jcuff@shuttle:~/amazon$ source setup.sh


I remembered that from the last post, now to get the instances.

jcuff@shuttle:~/amazon$ iid=`ec2-describe-instances | grep INSTANCE |

awk '{print $2}'`


stop and then restart the instance with the modified parameters (small is larger than micro)

jcuff@shuttle:~/amazon$ ec2-stop-instances ${iid}

jcuff@shuttle:~/amazon$ ec2-modify-instance-attribute --instance-type m1.small ${iid}
jcuff@shuttle:~/amazon$ ec2-start-instances ${iid}

ssh to instance, fix things that broke, dpkg is your only friend here after a fail...

ubuntu@ip-10-202-215-103:~$ sudo dpkg --configure -a

Setting up openjdk-6-jre-headless (6b20-1.9.1-1ubuntu3) ...
Setting up libaccess-bridge-java (1.26.2-5) ...
Setting up icedtea-6-jre-cacao (6b20-1.9.1-1ubuntu3) ...
Setting up ca-certificates-java (20100412) ...
creating /etc/ssl/certs/java/cacerts...
done.
Setting up libaccess-bridge-java-jni (1.26.2-5) ...
Setting up openjdk-6-jre (6b20-1.9.1-1ubuntu3) ...

(hurray!)


restart the instance...


ubuntu@ip-10-202-215-103:~$ sudo shutdown -h now

ubuntu@ip-10-202-215-103:~$
Broadcast message from ubuntu@ip-10-202-215-103
       (/dev/pts/0) at 1:01 ...


twiddle the instance back to micro from small


jcuff@shuttle:~/amazon$ ec2-stop-instances ${iid}

jcuff@shuttle:~/amazon$ ec2-modify-instance-attribute --instance-type
t1.micro ${iid}

and restart it, and log in with the private key (ssh -i is epic):

jcuff@shuttle:~/amazon$ ec2-start-instances ${iid}

jcuff@shuttle:~/amazon$ ssh -i key.pem
ubuntu@somewhere.compute-1.amazonaws.com
Linux ip-10-244-14-44 2.6.35-22-virtual #35-Ubuntu SMP Sat Oct 16
23:57:40 UTC 2010 i686 GNU/Linux
Ubuntu 10.10

Welcome to Ubuntu!

ubuntu@ip-10-244-14-44:~$ uptime
 01:05:10 up 0 min,  1 user,  load average: 0.00, 0.00, 0.00

ubuntu@ip-10-244-14-44:~$ java -version

java version "1.6.0_20"
OpenJDK Runtime Environment (IcedTea6 1.9.1) (6b20-1.9.1-1ubuntu3)
OpenJDK Client VM (build 17.0-b16, mixed mode, sharing)

All done!



EPIC WIN!




[any opinions here are all mine, and have absolutely nothing to do with my employer]
(c) 2011 James Cuff