Saturday, April 27, 2013

openstack day - live blog...

Today some pals got together to play with openstack and generally hang out and have some nerdy fun. For those in the tl;dr crowd, I've put a summary of what we learned today right here. For brave souls feel free to read on, should also be hilarious for those that actually know what they are doing!

Things we learned today

- DevStack is awesome for single use single box dev
- Fedora16 images seem to be a bit busted
- Tim never backs up his shit ;-)
- PackStack for RHEL is pretty rad!
- PackStack does some sweet things with Puppet.
- We had our best luck with RHEL6.4, but that's because Tim was here! ;-)
- DNS it turns out is rather important
- Networking is hard
- Dominos build your own artisan pizza is teh epic!
- VMWare Fusion migration to VirtualBox is cake
- Ethernet name problems in Linux now look like Windows .reg hacks
- If we knew what we were doing we would be really dangerous!
- n+2 seems to be a bit of an issue
- You probably need trained professionals to help you!
- This thing called EC2 is actually pretty rad... #someoneelseproblem ;-)

Off we go... #openstackday starting at the crack of 12:30pm!

You can follow along with Tim here:

I'm doing the Ubuntu track, Tim is on REL64, and Michele is on the Fedora style.

First up:

In theory this should all be simple enough... right?
apt-get update
apt-get install -qqy git
git clone
cd devstack
echo ADMIN_PASSWORD=letmein > localrc
echo MYSQL_PASSWORD=letmein >> localrc
echo RABBIT_PASSWORD=letmein >> localrc
echo SERVICE_PASSWORD=letmein >> localrc
echo SERVICE_TOKEN=letmein >> localrc

top - 12:38:35 up 13 min,  4 users,  load average: 2.51, 2.92, 1.80
Tasks: 245 total,   1 running, 244 sleeping,   0 stopped,   0 zombie
%Cpu(s): 18.6 us, 32.4 sy,  0.0 ni, 45.1 id,  1.0 wa,  0.0 hi,  2.9 si,  0.0 st
KiB Mem:   1026896 total,   862920 used,   163976 free,    41364 buffers
KiB Swap:   522236 total,    55080 used,   467156 free,   144960 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND           
28145 rabbitmq  20   0 98.2m  22m 2204 S 249.4  2.2   0:02.93 beam.smp     

Looks like this is the problem...

And sure enough...
jcuff@jcuff-virtualbox:/opt/stack$ sudo rabbitmqctl report | grep -A3 file_descriptors

jcuff@jcuff-virtualbox:/opt/stack$ sudo su -
root@jcuff-virtualbox:~# ulimit -n 10000
root@jcuff-virtualbox:~# /etc/init.d/rabbitmq-server restart
 * Restarting message broker rabbitmq-server                             [ OK ] 

root@jcuff-virtualbox:~# rabbitmqctl report | grep -A3 file_descriptors

Then we got stuck here:
stack 31892 0.0 0.1 5216 1396 pts/2 S+ 12:44 0:00 bash
/opt/stack/devstack/tools/ -PA --target-dir /opt/stack/devstack/accrc

Killed the process, tried to restart ./ oops!

Ok clearly time for a new VM... this time I'm going to snapshot it before I start playing with ./, as this vm has been seriously scribbled on :-)

Back later, once I make a new VM... doh! Pretty much everything was wrong with this setup!


Starting over again with 12.04.2 LTS 64bit Server version, and this time with enough disk space :-) So glad we have an epic fast network connection here, and this SSD drive on the laptop sure is helping out a lot!


Ok, new VM base created, I'm going to freeze this one so I don't ever have to do that part of the shenanigans again, if there are openstack folks looking on at this as we go try not to laugh too much, thus far it's all been a bit of a string of self inflicted PBKAC issues :-)
[James-Cuffs-MacBook-Pro]$ cd /Users/jcuff/VirtualBox\ VMs/
[James-Cuffs-MacBook-Pro]$ ls -ltra
total 0
drwx--x---+ 111 jcuff  staff  3774 Apr 26 14:05 ..
drwxr-xr-x    7 jcuff  staff   238 Apr 27 13:14 Ubuntu
drwxr-xr-x    4 jcuff  staff   136 Apr 27 13:14 .
drwxr-xr-x    6 jcuff  staff   204 Apr 27 13:25 OpenStack
[James-Cuffs-MacBook-Pro]$ du -sh OpenStack/
1.6G OpenStack/
[James-Cuffs-MacBook-Pro]$ cp -r OpenStack/ OpenStack.Golden

Ok, so that's going to help, not have to revisit that one! Time for a quick break, before going at again :-) Ok, off we go again!

jcuff@openstack:~$ git clone git://
Cloning into 'devstack'...
remote: Counting objects: 9763, done.
remote: Compressing objects: 100% (3366/3366), done.
remote: Total 9763 (delta 6804), reused 9151 (delta 6305)
Receiving objects: 100% (9763/9763), 1.71 MiB | 2.12 MiB/s, done.
Resolving deltas: 100% (6804/6804), done.
jcuff@openstack:~$ cd devstack/
jcuff@openstack:~/devstack$ ./

Good news, no 200% rabbitmq this time, that's great. So far so good! Lots of downloading to do now, may need another snapshot if this round of ./ succeeds.

Oh snap! Wrong MYSQL password... Doh!
An unexpected error prevented the server from fulfilling your request. (OperationalError) (1045, "Access denied for user 'root'@'localhost' (using password: YES)") None None (HTTP 500)
Here we go again...
jcuff@openstack:~/devstack$ ./
You are already running a session.
To rejoin this session type 'screen -x stack'.
To destroy this session, type './'.

jcuff@openstack:~/devstack$ ./
* Stopping web server apache2
Volume group "stack-volumes" not found
Skipping volume group stack-volumes
Ok time to edit: jcuff@openstack:~/devstack$ vi localrc

And try again!

Holy shit! It worked!

Horizon is now available at
Keystone is serving at
Examples on using novaclient command line is in
The default users are: admin and demo
The password: letmein
This is your host ip: completed in 112 seconds.

Time for a well deserved cup of tea before the next steps!

Tea made, first image launched!!

Yup all good:
jcuff@openstack:~/devstack$ ssh cirros@
cirros@'s password:

$ uname -a
Linux cirros 3.2.0-37-virtual #58-Ubuntu SMP Thu Jan 24 15:48:03 UTC 2013 x86_64 GNU/Linux

$ uptime
11:58:17 up 6 min, 2 users, load average: 0.07, 0.04, 0.03

$ cat /proc/meminfo | grep Mem
MemTotal: 503520 kB
MemFree: 473728 kB

$ cat /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 6
model : 2
model name : QEMU Virtual CPU version 1.0
stepping : 3
cpu MHz : 2212.096
cache size : 512 KB
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm up nopl pni cx16 popcnt hypervisor lahf_lm svm abm sse4a
bogomips : 4424.19
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

$ ifconfig eth0
eth0 Link encap:Ethernet HWaddr FA:16:3E:18:60:30
inet addr: Bcast: Mask:
inet6 addr: fe80::f816:3eff:fe18:6030/64 Scope:Link
RX packets:1153 errors:0 dropped:0 overruns:0 frame:0
TX packets:699 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:118307 (115.5 KiB) TX bytes:116076 (113.3 KiB)

$ hostname


Ok, time to snap this install again... oh my have we grown!
jcuff@openstack:~/devstack$ sudo shutdown -h now

Broadcast message from jcuff@openstack
(/dev/pts/1) at 14:18 ...

The system is going down for halt NOW!
jcuff@openstack:~/devstack$ Connection to closed by remote host.
Connection to closed.

[James-Cuffs-MacBook-Pro]$ pwd
/Users/jcuff/VirtualBox VMs
[James-Cuffs-MacBook-Pro]$ rm -rf OpenStack.Golden/

[James-Cuffs-MacBook-Pro]$ time cp -r OpenStack/ OpenStack.Golden

real 0m20.886s
user 0m0.007s
sys 0m3.656s

[James-Cuffs-MacBook-Pro]$ du -sh OpenStack.Golden/
3.0G OpenStack.Golden/

Sure was a whole lot easier than the folsom guide looks!

Tim has also had success!

Sure is a whole lot going on in here!

-M pc-1.0 
-m 512 
-smp 1,sockets=1,cores=1,threads=1 
-name instance-00000001 
-uuid c9c91966-e460-4131-bcf8-9e830c7854f8

-smbios type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=2013.2,serial=245343e1-c87c-4c83-a08a-26bbd6d2756a,uuid=c9c91966-e460-4131-bcf8-9e830c7854f8 
-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000001.monitor,server,nowait 
-mon chardev=charmonitor,id=monitor,mode=control 
-rtc base=utc 

-kernel /opt/stack/data/nova/instances/c9c91966-e460-4131-bcf8-9e830c7854f8/kernel 
-initrd /opt/stack/data/nova/instances/c9c91966-e460-4131-bcf8-9e830c7854f8/ramdisk 
-append root=/dev/vda console=ttyS0 

-drive file=/dev/disk/by-path/,if=none,id=drive-virtio-disk0,format=raw,serial=bc1a9e79-cf0d-4efe-9fe3-95da882f0277,cache=none 

-device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -netdev tap,fd=18,id=hostnet0 

-device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:71:bb:de,bus=pci.0,addr=0x3 

-chardev file,id=charserial0,path=/opt/stack/data/nova/instances/c9c91966-e460-4131-bcf8-9e830c7854f8/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 

-device usb-tablet,id=input0 
-k en-us 
-vga cirrus 
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5

But that's cake compared to the database structure we need ;-)
for i in cinder keystone glance nove ; do echo "show tables"|mysql -u root -pxxxxxx $i; done |wc -l


Some folks seemed to be concerned with HA. We will leave that for another day!

Anyhoo - now a fancy one, with new ssh keypairs working... look mom, no passwords!
jcuff@openstack:~/devstack$ ssh cirros@
$ uname -a
Linux cirros 3.2.0-37-virtual #58-Ubuntu SMP Thu Jan 24 15:48:03 UTC 2013 x86_64 GNU/Linux

Cool, and this one is running off of block storage via iscsi!
-drive file=/dev/disk/by-path/

And yes, it really is iSCSI!
jcuff@openstack:~$ sudo iscsiadm -m discovery -t st -p,1

Time to get fancy with a real ubuntu install.
jcuff@openstack:~$ wget

2013-04-27 15:14:19 (3.50 MB/s) - `raring-server-cloudimg-amd64-disk1.img' saved [234618880/234618880]

jcuff@openstack:~$ file raring-server-cloudimg-amd64-disk1.img 
raring-server-cloudimg-amd64-disk1.img: QEMU QCOW Image (v2), 2147483648 bytes

Very cool that you can also load img files directly from the web...

Yay! This is awesome... we have a working real ubuntu inside openstack
ubuntu@ubuntu:~$ uname -a
Linux ubuntu 3.8.0-19-generic #29-Ubuntu SMP Wed Apr 17 18:16:28 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

This ssh chain is getting a little carried away...
[James-Cuffs-MacBook-Pro]$ ssh James-Cuffs-MacBook-Pro.local
Last login: Sat Apr 27 15:43:02 2013 from
[James-Cuffs-MacBook-Pro]$ ssh jcuff@
Welcome to Ubuntu 12.04.2 LTS (GNU/Linux 3.5.0-23-generic x86_64)

* Documentation:

System information as of Sat Apr 27 15:43:59 EDT 2013

System load:  0.05               Processes:             197
Usage of /:   21.2% of 17.74GB   Users logged in:       1
Memory usage: 74%                IP address for virbr0:
Swap usage:   4%                 IP address for br100:

Graph this data and manage this system at

51 packages can be updated.
35 updates are security updates.

Last login: Sat Apr 27 15:40:38 2013 from
jcuff@openstack:~$ ssh ubuntu@
Welcome to Ubuntu 13.04 (GNU/Linux 3.8.0-19-generic x86_64)

* Documentation:

System information as of Sat Apr 27 19:44:13 UTC 2013

System load:  0.15              Processes:           73
Usage of /:   35.9% of 1.93GB   Users logged in:     1
Memory usage: 25%               IP address for eth0:
Swap usage:   0%

Graph this data and manage this system at

Get cloud support with Ubuntu Advantage Cloud Guest:

Use Juju to deploy your cloud instances and workloads:

0 packages can be updated.
0 updates are security updates.

Last login: Sat Apr 27 19:41:01 2013 from
OSX (physical) > Ubuntu 12.04.2 (virtual box) > Ubuntu 13.04 (openstack)

A cloud inside a cloud inside a cloud in a box :-)

Pizza arrives! If you look close Tim's laptop has lots of green [OK] going on!

Shockingly, discovered the openstack instigated VM can connect to the internet just fine!
ubuntu@ubuntu:~$ telnet 80
Connected to
Escape character is '^]'.
HTTP/1.0 200 OK
Date: Sat, 27 Apr 2013 19:47:54 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1


Starting some more fun with NOVA, and instance types:
jcuff@openstack:/opt/stack/nova$ ./bin/nova-manage flavor list

m1.medium: Memory: 4096MB, VCPUS: 2, Root: 40GB, Ephemeral: 0Gb,
FlavorID: 3, Swap: 0MB, RXTX Factor: 1.0, public, ExtraSpecs {}

m1.micro: Memory: 128MB, VCPUS: 1, Root: 0GB, Ephemeral: 0Gb,
FlavorID: 84, Swap: 0MB, RXTX Factor: 1.0, public, ExtraSpecs {}

m1.tiny: Memory: 512MB, VCPUS: 1, Root: 0GB, Ephemeral: 0Gb,
FlavorID: 1, Swap: 0MB, RXTX Factor: 1.0, public, ExtraSpecs {}

m1.large: Memory: 8192MB, VCPUS: 4, Root: 80GB, Ephemeral: 0Gb,
FlavorID: 4, Swap: 0MB, RXTX Factor: 1.0, public, ExtraSpecs {}

m1.nano: Memory: 64MB, VCPUS: 1, Root: 0GB, Ephemeral: 0Gb,
FlavorID: 42, Swap: 0MB, RXTX Factor: 1.0, public, ExtraSpecs {}

m1.small: Memory: 2048MB, VCPUS: 1, Root: 20GB, Ephemeral: 0Gb,
FlavorID: 2, Swap: 0MB, RXTX Factor: 1.0, public, ExtraSpecs {}

m1.xlarge: Memory: 16384MB, VCPUS: 8, Root: 160GB, Ephemeral: 0Gb,
FlavorID: 5, Swap: 0MB, RXTX Factor: 1.0, public, ExtraSpecs {}

What is in the cli is in the gui... nice!

Ok, next up trying to turn one of Tim's laptop VMs into a nova compute engine, by end of day we would love to look something like this. Very much doubt that we will get there, but we will give it the good old college try :-)

While debugging noticed that a simple "screen -r" gets you to an awesome screen debug with 17 windows! Count em all if you can.

Pretty cool this, here you can see how it was set up:
jcuff@openstack:~/devstack$ cat stack-screenrc | grep tuff
sudo tail -f /var/log/apache2/horizon_error.log
sg libvirtd /opt/stack/nova/bin/nova-compute

Update, lots of fannying about with Tim's laptop - was not configured with bridge and NAT sure makes it "interesting" to do computer to computer networking. While changing this we discovered that nova.conf and a whole pile of other things are all hard coded into the machine. Time to spin up fresh VMs rather than attempt to unpick this rats nest of static and hardcoded stuff. Even the fanciest of code seem to have a boatload of hardcoded links in it - this one even more than many others. Good job Tim is stella at the devops, onto bridged networking, we think we may even get a two membered set up going soon.

DevStack is great, but really only designed as we used it for single box runs. We wanted to do multiple boxes, and using automatic scripts to provision Ubuntu from RHEL was clearly a bad idea... ;-) Now copying Tim's RH64 version over so we can do multi compute images. Also now need to test out our Fusion to VirtualBox fu... this also should be a bit of a laugh! First up - go go gadget wifi!
[James-Cuffs-MacBook-Pro]$ pwd
/Users/jcuff/VirtualBox VMs

[James-Cuffs-MacBook-Pro]$ scp -r alaric@"/Users/alaric/Documents/Virtual\ Machines.localized/RH\ OS\ Lab\ -2\ .vmwarevm" ./RH64

RH OS Lab -2 .vmxf                           100%   3433   3.4KB/s   00:00    
Virtual Disk-s001.vmdk                       100%   58MB   8.3MB/s   00:07    
Virtual Disk-s002.vmdk                       100%  320KB 320.0KB/s   00:01    
Virtual Disk-s003.vmdk                       100% 1363MB   8.7MB/s   02:36    
Virtual Disk-s004.vmdk                       100%  442MB   8.7MB/s   00:51    
Virtual Disk-s005.vmdk                       100%  272MB   8.5MB/s   00:32    
Virtual Disk-s006.vmdk                       100%  391MB   8.5MB/s   00:46    

Update on Fedora, Michele tried a solid and eventually gave up, three downloads, corruption and other buckets of fail. She is now on the Ubuntu wagon!

In other news - oh my lord! Tim's copy to my laptop worked first time! Yay integration with virtualbox... I remember days when this was no way near as easy to do as this! Proof:

Oh my - RHEL bites us in the arse again! Be sure that you look at:
[paxindustria@localhost ~]$ cat /etc/udev/rules.d/70-persistent-net.rules
# This file was automatically generated by the /lib/udev/write_net_rules
# program, run by the persistent-net-generator.rules rules file.
# You can modify it, as long as you keep each rule on a single
# line, and change only the value of the NAME= key.

# PCI device 0x8086:0x100e (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="08:00:27:92:66:1e", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

# PCI device 0x8086:0x100e (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="08:00:27:b1:87:64", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"

Because we moved the instance to virtual box, and the packstack code expects to see "eth0", and "eth1".

Certainly gets very pissed if it happens to see "eth2" and "eth3"... le sigh. Anyhoo, chopped it out and now we are looking pretty awesome again! Tim is booting up the two laptop cluster as we speak.

Yay! Tim successfully got nova-compute installed on my laptop from his with packstack:
nova  4213  2.2  2.8 1117324 53948 ? Sl 15:38   0:01 /usr/bin/python /usr/bin/nova-compute --config-file /etc/nova/nova.conf --logfile /var/log/nova/compute.log

le sigh - clearly having network issues here, we have been bouncing firewalls and machines all over the place, google searches are turning up a few questions about this... still hacking away here, but starting to hit more roadblocks than we thought. Mind you it is always a problem with the network. hehehe!
2013-04-27 16:19:52.193 4047 TRACE nova.compute.manager [instance: 709c127a-0f8f-4751-a496-f47e05ac73be]     raise rpc_common.Timeout()
2013-04-27 16:19:52.193 4047 TRACE nova.compute.manager [instance: 709c127a-0f8f-4751-a496-f47e05ac73be] Timeout: Timeout while waiting on RPC response.

8:07pm Tim is running a fresh install now we have sorted out our DNS issues.

8:22pm And we have SUCCESS! We have a two node cluster running across the network!


And that my friends is what we call a WRAP! Here ends our inaugral #openstackday

Hope you enjoyed the journey as much as we did.

[any opinions here are all mine, and have absolutely nothing to do with my employer]
(c) 2011 James Cuff