Saturday, April 27, 2013

openstack day - live blog...

Today some pals got together to play with openstack and generally hang out and have some nerdy fun. For those in the tl;dr crowd, I've put a summary of what we learned today right here. For brave souls feel free to read on, should also be hilarious for those that actually know what they are doing!

Things we learned today

- DevStack is awesome for single use single box dev
- Fedora16 images seem to be a bit busted
- Tim never backs up his shit ;-)
- PackStack for RHEL is pretty rad!
- PackStack does some sweet things with Puppet.
- We had our best luck with RHEL6.4, but that's because Tim was here! ;-)
- DNS it turns out is rather important
- Networking is hard
- Dominos build your own artisan pizza is teh epic!
- VMWare Fusion migration to VirtualBox is cake
- Ethernet name problems in Linux now look like Windows .reg hacks
- If we knew what we were doing we would be really dangerous!
- n+2 seems to be a bit of an issue
- You probably need trained professionals to help you!
- This thing called EC2 is actually pretty rad... #someoneelseproblem ;-)

Off we go... #openstackday starting at the crack of 12:30pm!

You can follow along with Tim here:

I'm doing the Ubuntu track, Tim is on REL64, and Michele is on the Fedora style.

First up:

In theory this should all be simple enough... right?
apt-get update
apt-get install -qqy git
git clone
cd devstack
echo ADMIN_PASSWORD=letmein > localrc
echo MYSQL_PASSWORD=letmein >> localrc
echo RABBIT_PASSWORD=letmein >> localrc
echo SERVICE_PASSWORD=letmein >> localrc
echo SERVICE_TOKEN=letmein >> localrc

top - 12:38:35 up 13 min,  4 users,  load average: 2.51, 2.92, 1.80
Tasks: 245 total,   1 running, 244 sleeping,   0 stopped,   0 zombie
%Cpu(s): 18.6 us, 32.4 sy,  0.0 ni, 45.1 id,  1.0 wa,  0.0 hi,  2.9 si,  0.0 st
KiB Mem:   1026896 total,   862920 used,   163976 free,    41364 buffers
KiB Swap:   522236 total,    55080 used,   467156 free,   144960 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND           
28145 rabbitmq  20   0 98.2m  22m 2204 S 249.4  2.2   0:02.93 beam.smp     

Looks like this is the problem...

And sure enough...
jcuff@jcuff-virtualbox:/opt/stack$ sudo rabbitmqctl report | grep -A3 file_descriptors

jcuff@jcuff-virtualbox:/opt/stack$ sudo su -
root@jcuff-virtualbox:~# ulimit -n 10000
root@jcuff-virtualbox:~# /etc/init.d/rabbitmq-server restart
 * Restarting message broker rabbitmq-server                             [ OK ] 

root@jcuff-virtualbox:~# rabbitmqctl report | grep -A3 file_descriptors

Then we got stuck here:
stack 31892 0.0 0.1 5216 1396 pts/2 S+ 12:44 0:00 bash
/opt/stack/devstack/tools/ -PA --target-dir /opt/stack/devstack/accrc

Killed the process, tried to restart ./ oops!

Ok clearly time for a new VM... this time I'm going to snapshot it before I start playing with ./, as this vm has been seriously scribbled on :-)

Back later, once I make a new VM... doh! Pretty much everything was wrong with this setup!


Starting over again with 12.04.2 LTS 64bit Server version, and this time with enough disk space :-) So glad we have an epic fast network connection here, and this SSD drive on the laptop sure is helping out a lot!


Ok, new VM base created, I'm going to freeze this one so I don't ever have to do that part of the shenanigans again, if there are openstack folks looking on at this as we go try not to laugh too much, thus far it's all been a bit of a string of self inflicted PBKAC issues :-)
[James-Cuffs-MacBook-Pro]$ cd /Users/jcuff/VirtualBox\ VMs/
[James-Cuffs-MacBook-Pro]$ ls -ltra
total 0
drwx--x---+ 111 jcuff  staff  3774 Apr 26 14:05 ..
drwxr-xr-x    7 jcuff  staff   238 Apr 27 13:14 Ubuntu
drwxr-xr-x    4 jcuff  staff   136 Apr 27 13:14 .
drwxr-xr-x    6 jcuff  staff   204 Apr 27 13:25 OpenStack
[James-Cuffs-MacBook-Pro]$ du -sh OpenStack/
1.6G OpenStack/
[James-Cuffs-MacBook-Pro]$ cp -r OpenStack/ OpenStack.Golden

Ok, so that's going to help, not have to revisit that one! Time for a quick break, before going at again :-) Ok, off we go again!

jcuff@openstack:~$ git clone git://
Cloning into 'devstack'...
remote: Counting objects: 9763, done.
remote: Compressing objects: 100% (3366/3366), done.
remote: Total 9763 (delta 6804), reused 9151 (delta 6305)
Receiving objects: 100% (9763/9763), 1.71 MiB | 2.12 MiB/s, done.
Resolving deltas: 100% (6804/6804), done.
jcuff@openstack:~$ cd devstack/
jcuff@openstack:~/devstack$ ./

Good news, no 200% rabbitmq this time, that's great. So far so good! Lots of downloading to do now, may need another snapshot if this round of ./ succeeds.

Oh snap! Wrong MYSQL password... Doh!
An unexpected error prevented the server from fulfilling your request. (OperationalError) (1045, "Access denied for user 'root'@'localhost' (using password: YES)") None None (HTTP 500)
Here we go again...
jcuff@openstack:~/devstack$ ./
You are already running a session.
To rejoin this session type 'screen -x stack'.
To destroy this session, type './'.

jcuff@openstack:~/devstack$ ./
* Stopping web server apache2
Volume group "stack-volumes" not found
Skipping volume group stack-volumes
Ok time to edit: jcuff@openstack:~/devstack$ vi localrc

And try again!

Holy shit! It worked!

Horizon is now available at
Keystone is serving at
Examples on using novaclient command line is in
The default users are: admin and demo
The password: letmein
This is your host ip: completed in 112 seconds.

Time for a well deserved cup of tea before the next steps!

Tea made, first image launched!!

Yup all good:
jcuff@openstack:~/devstack$ ssh cirros@
cirros@'s password:

$ uname -a
Linux cirros 3.2.0-37-virtual #58-Ubuntu SMP Thu Jan 24 15:48:03 UTC 2013 x86_64 GNU/Linux

$ uptime
11:58:17 up 6 min, 2 users, load average: 0.07, 0.04, 0.03

$ cat /proc/meminfo | grep Mem
MemTotal: 503520 kB
MemFree: 473728 kB

$ cat /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 6
model : 2
model name : QEMU Virtual CPU version 1.0
stepping : 3
cpu MHz : 2212.096
cache size : 512 KB
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm up nopl pni cx16 popcnt hypervisor lahf_lm svm abm sse4a
bogomips : 4424.19
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

$ ifconfig eth0
eth0 Link encap:Ethernet HWaddr FA:16:3E:18:60:30
inet addr: Bcast: Mask:
inet6 addr: fe80::f816:3eff:fe18:6030/64 Scope:Link
RX packets:1153 errors:0 dropped:0 overruns:0 frame:0
TX packets:699 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:118307 (115.5 KiB) TX bytes:116076 (113.3 KiB)

$ hostname


Ok, time to snap this install again... oh my have we grown!
jcuff@openstack:~/devstack$ sudo shutdown -h now

Broadcast message from jcuff@openstack
(/dev/pts/1) at 14:18 ...

The system is going down for halt NOW!
jcuff@openstack:~/devstack$ Connection to closed by remote host.
Connection to closed.

[James-Cuffs-MacBook-Pro]$ pwd
/Users/jcuff/VirtualBox VMs
[James-Cuffs-MacBook-Pro]$ rm -rf OpenStack.Golden/

[James-Cuffs-MacBook-Pro]$ time cp -r OpenStack/ OpenStack.Golden

real 0m20.886s
user 0m0.007s
sys 0m3.656s

[James-Cuffs-MacBook-Pro]$ du -sh OpenStack.Golden/
3.0G OpenStack.Golden/

Sure was a whole lot easier than the folsom guide looks!

Tim has also had success!

Sure is a whole lot going on in here!

-M pc-1.0 
-m 512 
-smp 1,sockets=1,cores=1,threads=1 
-name instance-00000001 
-uuid c9c91966-e460-4131-bcf8-9e830c7854f8

-smbios type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=2013.2,serial=245343e1-c87c-4c83-a08a-26bbd6d2756a,uuid=c9c91966-e460-4131-bcf8-9e830c7854f8 
-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000001.monitor,server,nowait 
-mon chardev=charmonitor,id=monitor,mode=control 
-rtc base=utc 

-kernel /opt/stack/data/nova/instances/c9c91966-e460-4131-bcf8-9e830c7854f8/kernel 
-initrd /opt/stack/data/nova/instances/c9c91966-e460-4131-bcf8-9e830c7854f8/ramdisk 
-append root=/dev/vda console=ttyS0 

-drive file=/dev/disk/by-path/,if=none,id=drive-virtio-disk0,format=raw,serial=bc1a9e79-cf0d-4efe-9fe3-95da882f0277,cache=none 

-device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -netdev tap,fd=18,id=hostnet0 

-device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:71:bb:de,bus=pci.0,addr=0x3 

-chardev file,id=charserial0,path=/opt/stack/data/nova/instances/c9c91966-e460-4131-bcf8-9e830c7854f8/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 

-device usb-tablet,id=input0 
-k en-us 
-vga cirrus 
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5

But that's cake compared to the database structure we need ;-)
for i in cinder keystone glance nove ; do echo "show tables"|mysql -u root -pxxxxxx $i; done |wc -l


Some folks seemed to be concerned with HA. We will leave that for another day!

Anyhoo - now a fancy one, with new ssh keypairs working... look mom, no passwords!
jcuff@openstack:~/devstack$ ssh cirros@
$ uname -a
Linux cirros 3.2.0-37-virtual #58-Ubuntu SMP Thu Jan 24 15:48:03 UTC 2013 x86_64 GNU/Linux

Cool, and this one is running off of block storage via iscsi!
-drive file=/dev/disk/by-path/

And yes, it really is iSCSI!
jcuff@openstack:~$ sudo iscsiadm -m discovery -t st -p,1

Time to get fancy with a real ubuntu install.
jcuff@openstack:~$ wget

2013-04-27 15:14:19 (3.50 MB/s) - `raring-server-cloudimg-amd64-disk1.img' saved [234618880/234618880]

jcuff@openstack:~$ file raring-server-cloudimg-amd64-disk1.img 
raring-server-cloudimg-amd64-disk1.img: QEMU QCOW Image (v2), 2147483648 bytes

Very cool that you can also load img files directly from the web...

Yay! This is awesome... we have a working real ubuntu inside openstack
ubuntu@ubuntu:~$ uname -a
Linux ubuntu 3.8.0-19-generic #29-Ubuntu SMP Wed Apr 17 18:16:28 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

This ssh chain is getting a little carried away...
[James-Cuffs-MacBook-Pro]$ ssh James-Cuffs-MacBook-Pro.local
Last login: Sat Apr 27 15:43:02 2013 from
[James-Cuffs-MacBook-Pro]$ ssh jcuff@
Welcome to Ubuntu 12.04.2 LTS (GNU/Linux 3.5.0-23-generic x86_64)

* Documentation:

System information as of Sat Apr 27 15:43:59 EDT 2013

System load:  0.05               Processes:             197
Usage of /:   21.2% of 17.74GB   Users logged in:       1
Memory usage: 74%                IP address for virbr0:
Swap usage:   4%                 IP address for br100:

Graph this data and manage this system at

51 packages can be updated.
35 updates are security updates.

Last login: Sat Apr 27 15:40:38 2013 from
jcuff@openstack:~$ ssh ubuntu@
Welcome to Ubuntu 13.04 (GNU/Linux 3.8.0-19-generic x86_64)

* Documentation:

System information as of Sat Apr 27 19:44:13 UTC 2013

System load:  0.15              Processes:           73
Usage of /:   35.9% of 1.93GB   Users logged in:     1
Memory usage: 25%               IP address for eth0:
Swap usage:   0%

Graph this data and manage this system at

Get cloud support with Ubuntu Advantage Cloud Guest:

Use Juju to deploy your cloud instances and workloads:

0 packages can be updated.
0 updates are security updates.

Last login: Sat Apr 27 19:41:01 2013 from
OSX (physical) > Ubuntu 12.04.2 (virtual box) > Ubuntu 13.04 (openstack)

A cloud inside a cloud inside a cloud in a box :-)

Pizza arrives! If you look close Tim's laptop has lots of green [OK] going on!

Shockingly, discovered the openstack instigated VM can connect to the internet just fine!
ubuntu@ubuntu:~$ telnet 80
Connected to
Escape character is '^]'.
HTTP/1.0 200 OK
Date: Sat, 27 Apr 2013 19:47:54 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1


Starting some more fun with NOVA, and instance types:
jcuff@openstack:/opt/stack/nova$ ./bin/nova-manage flavor list

m1.medium: Memory: 4096MB, VCPUS: 2, Root: 40GB, Ephemeral: 0Gb,
FlavorID: 3, Swap: 0MB, RXTX Factor: 1.0, public, ExtraSpecs {}

m1.micro: Memory: 128MB, VCPUS: 1, Root: 0GB, Ephemeral: 0Gb,
FlavorID: 84, Swap: 0MB, RXTX Factor: 1.0, public, ExtraSpecs {}

m1.tiny: Memory: 512MB, VCPUS: 1, Root: 0GB, Ephemeral: 0Gb,
FlavorID: 1, Swap: 0MB, RXTX Factor: 1.0, public, ExtraSpecs {}

m1.large: Memory: 8192MB, VCPUS: 4, Root: 80GB, Ephemeral: 0Gb,
FlavorID: 4, Swap: 0MB, RXTX Factor: 1.0, public, ExtraSpecs {}

m1.nano: Memory: 64MB, VCPUS: 1, Root: 0GB, Ephemeral: 0Gb,
FlavorID: 42, Swap: 0MB, RXTX Factor: 1.0, public, ExtraSpecs {}

m1.small: Memory: 2048MB, VCPUS: 1, Root: 20GB, Ephemeral: 0Gb,
FlavorID: 2, Swap: 0MB, RXTX Factor: 1.0, public, ExtraSpecs {}

m1.xlarge: Memory: 16384MB, VCPUS: 8, Root: 160GB, Ephemeral: 0Gb,
FlavorID: 5, Swap: 0MB, RXTX Factor: 1.0, public, ExtraSpecs {}

What is in the cli is in the gui... nice!

Ok, next up trying to turn one of Tim's laptop VMs into a nova compute engine, by end of day we would love to look something like this. Very much doubt that we will get there, but we will give it the good old college try :-)

While debugging noticed that a simple "screen -r" gets you to an awesome screen debug with 17 windows! Count em all if you can.

Pretty cool this, here you can see how it was set up:
jcuff@openstack:~/devstack$ cat stack-screenrc | grep tuff
sudo tail -f /var/log/apache2/horizon_error.log
sg libvirtd /opt/stack/nova/bin/nova-compute

Update, lots of fannying about with Tim's laptop - was not configured with bridge and NAT sure makes it "interesting" to do computer to computer networking. While changing this we discovered that nova.conf and a whole pile of other things are all hard coded into the machine. Time to spin up fresh VMs rather than attempt to unpick this rats nest of static and hardcoded stuff. Even the fanciest of code seem to have a boatload of hardcoded links in it - this one even more than many others. Good job Tim is stella at the devops, onto bridged networking, we think we may even get a two membered set up going soon.

DevStack is great, but really only designed as we used it for single box runs. We wanted to do multiple boxes, and using automatic scripts to provision Ubuntu from RHEL was clearly a bad idea... ;-) Now copying Tim's RH64 version over so we can do multi compute images. Also now need to test out our Fusion to VirtualBox fu... this also should be a bit of a laugh! First up - go go gadget wifi!
[James-Cuffs-MacBook-Pro]$ pwd
/Users/jcuff/VirtualBox VMs

[James-Cuffs-MacBook-Pro]$ scp -r alaric@"/Users/alaric/Documents/Virtual\ Machines.localized/RH\ OS\ Lab\ -2\ .vmwarevm" ./RH64

RH OS Lab -2 .vmxf                           100%   3433   3.4KB/s   00:00    
Virtual Disk-s001.vmdk                       100%   58MB   8.3MB/s   00:07    
Virtual Disk-s002.vmdk                       100%  320KB 320.0KB/s   00:01    
Virtual Disk-s003.vmdk                       100% 1363MB   8.7MB/s   02:36    
Virtual Disk-s004.vmdk                       100%  442MB   8.7MB/s   00:51    
Virtual Disk-s005.vmdk                       100%  272MB   8.5MB/s   00:32    
Virtual Disk-s006.vmdk                       100%  391MB   8.5MB/s   00:46    

Update on Fedora, Michele tried a solid and eventually gave up, three downloads, corruption and other buckets of fail. She is now on the Ubuntu wagon!

In other news - oh my lord! Tim's copy to my laptop worked first time! Yay integration with virtualbox... I remember days when this was no way near as easy to do as this! Proof:

Oh my - RHEL bites us in the arse again! Be sure that you look at:
[paxindustria@localhost ~]$ cat /etc/udev/rules.d/70-persistent-net.rules
# This file was automatically generated by the /lib/udev/write_net_rules
# program, run by the persistent-net-generator.rules rules file.
# You can modify it, as long as you keep each rule on a single
# line, and change only the value of the NAME= key.

# PCI device 0x8086:0x100e (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="08:00:27:92:66:1e", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

# PCI device 0x8086:0x100e (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="08:00:27:b1:87:64", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

Because we moved the instance to virtual box, and the packstack code expects to see "eth0", and "eth1".

Certainly gets very pissed if it happens to see "eth2" and "eth3"... le sigh. Anyhoo, chopped it out and now we are looking pretty awesome again! Tim is booting up the two laptop cluster as we speak.

Yay! Tim successfully got nova-compute installed on my laptop from his with packstack:
nova  4213  2.2  2.8 1117324 53948 ? Sl 15:38   0:01 /usr/bin/python /usr/bin/nova-compute --config-file /etc/nova/nova.conf --logfile /var/log/nova/compute.log

le sigh - clearly having network issues here, we have been bouncing firewalls and machines all over the place, google searches are turning up a few questions about this... still hacking away here, but starting to hit more roadblocks than we thought. Mind you it is always a problem with the network. hehehe!
2013-04-27 16:19:52.193 4047 TRACE nova.compute.manager [instance: 709c127a-0f8f-4751-a496-f47e05ac73be]     raise rpc_common.Timeout()
2013-04-27 16:19:52.193 4047 TRACE nova.compute.manager [instance: 709c127a-0f8f-4751-a496-f47e05ac73be] Timeout: Timeout while waiting on RPC response.

8:07pm Tim is running a fresh install now we have sorted out our DNS issues.

8:22pm And we have SUCCESS! We have a two node cluster running across the network!


And that my friends is what we call a WRAP! Here ends our inaugral #openstackday

Hope you enjoyed the journey as much as we did.

Saturday, April 20, 2013

little known technologies that helped save the human genome

What is that chunky wiring monstrosity?

That my friends, is the utter magic that happens when you have to get creative to solve problems as a team. Back in the day, there were no clouds, amazon was still selling books. There was also no "devops", I mean heck we hardly had functioning cell phones!

So we faced a problem.  We had no space, not enough kit, and a race to further inform the human genome. These were the days before "map reduce" or any of that clever stuff. We wanted to find a way to get DNA on multiple computer clusters in a real hurry. Tim Cutts on the team with Guy Coates came up with this epic system that in essence relies on that set of little wires you see above. 

We were using RLX Technologies blades which could stack 24 little motherboards into a 3U chassis! These were some interesting days as we stacked up 768 of these puppies in 2 racks - we used 52U monster racks back then, that needed a stepladder to get to the top blades! However, this did allow us to fit 16 chassis or 384 individual computers in a single rack! Each blade had two network ports that were presented out the back via a nifty RJ21 connection...

You can see that the networking could get very quickly out of hand... 12*4 wires is 48 ethernet connections out of a single box!

Yeah it got out of hand fast! To run the 768 machines we had purchased we would have needed a network switch with 1,536 ports.  Remember this is 2002, that would have been a crazy switch to buy... even today it would be pretty nuts to do it...

So enter Tim Cutts, and his magic wires!

Because we were basically doing a file system distribution and image run (we talked about it later here). We could effectively make a next to next to next network taking each node and hooking it up to his friend next door.  This way we had the ability to make a network that was perfect to distribute data from one end of the chain all the way to the end of the chain.

We called this a "Distribution Area Network".

We basically took Dolly as our inspiration:

     +--------+  Data   +----------+  Data  +----------+
     | Master |-------->| Client 1 |------->| Client 2 |
     +--------+         +----------+        +----------+
         ^                   |                   |
         | Data              | Data              | Data
         |                   V                   V
      +------+            +------+            +------+
      | Disk |            | Disk |            | Disk |
      +------+            +------+            +------+

We were using the local 2.5" disks as a software raid stripe of 2 x 40G disks so there were 1,536 little spindles (60TB total) in there, each with local copies of the human genome and analysis data needed to get our science out of the door.

This all kinda looks a bit like something released not that long ago eh?   Seeing these recent announcements of so called "high density CPU servers" reminds me of that phrase...

What was once old. Is new again. I guess?

As a foot note, which to me was even more staggering. Tim sent the folks over at the manufacturers what was basically an upgraded beer mat design. We, much like the world famous Dr. Birney also loved to design things at the pub :-) The first example "wire" came back we plugged it in on a test chassis, set up some IP addresses... sent some pings... crossed our fingers!

and the bloody thing went and worked the FIRST TIME!

I still so much remember laughing in our ISG portakabin.

While researching this article, I stumbled across proof I also used Pine on Tru64 ;-)

Wednesday, April 17, 2013

IaaS services uptick in .edu

Great to stumble across and finally see this happening:

[James-MacBook-Pro]$ host has address mail is handled by 20 mail is handled by 20

for those wondering what "" is, here it is...

[James-MacBook-Pro]$ nslookup

Non-authoritative answer: name =

Awesome to see a major .edu IT org using not one but three outsourced services!

Tuesday, April 16, 2013

you may want to consider utility supercomputing on the cloud when...

In my new gig I was trying to come up with a single, unified image to capture when you really should probably consider the cloud as an option for your high performance computing...

Aww Yeah! Totally Nailed It!!

With thx to slashgear ca. 2009 ;-)

(c) 2018 James Cuff