Today some pals got together to play with openstack and generally hang out and have some nerdy fun. For those in the tl;dr crowd, I've put a summary of what we learned today right here. For brave souls feel free to read on, should also be hilarious for those that actually know what they are doing!
Things we learned today
-
DevStack is awesome for single use single box dev
- Fedora16 images seem to be a bit busted
- Tim
never backs up his shit ;-)
-
PackStack for RHEL is pretty rad!
- PackStack does some sweet things with
Puppet.
- We had our best luck with RHEL6.4, but that's because Tim was here! ;-)
- DNS it turns out is rather important
- Networking is hard
-
Dominos build your own artisan pizza is teh epic!
- VMWare Fusion migration to VirtualBox is cake
- Ethernet name problems in Linux now look like Windows .reg hacks
- If we knew what we were doing we would be really dangerous!
- n+2 seems to be a bit of an issue
- You probably need
trained professionals to help you!
- This thing called
EC2 is actually pretty rad... #someoneelseproblem ;-)
12:25pm
Off we go... #openstackday starting at the crack of 12:30pm!
You can follow along with Tim here:
http://personalskynet.blogspot.com/2013/04/dr-openstack-or-how-i-learned-to-stop.html
I'm doing the Ubuntu track, Tim is on REL64, and Michele is on the Fedora style.
First up:
http://devstack.org/guides/single-vm.html
In theory this should all be simple enough... right?
#!/bin/sh
apt-get update
apt-get install -qqy git
git clone https://github.com/openstack-dev/devstack.git
cd devstack
echo ADMIN_PASSWORD=letmein > localrc
echo MYSQL_PASSWORD=letmein >> localrc
echo RABBIT_PASSWORD=letmein >> localrc
echo SERVICE_PASSWORD=letmein >> localrc
echo SERVICE_TOKEN=letmein >> localrc
./stack.sh
Rotroh!
top - 12:38:35 up 13 min, 4 users, load average: 2.51, 2.92, 1.80
Tasks: 245 total, 1 running, 244 sleeping, 0 stopped, 0 zombie
%Cpu(s): 18.6 us, 32.4 sy, 0.0 ni, 45.1 id, 1.0 wa, 0.0 hi, 2.9 si, 0.0 st
KiB Mem: 1026896 total, 862920 used, 163976 free, 41364 buffers
KiB Swap: 522236 total, 55080 used, 467156 free, 144960 cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
28145 rabbitmq 20 0 98.2m 22m 2204 S 249.4 2.2 0:02.93 beam.smp
Looks like this is the problem...
http://lzone.de/Solving+100%25+CPU+usage+of+Chef
And sure enough...
jcuff@jcuff-virtualbox:/opt/stack$ sudo rabbitmqctl report | grep -A3 file_descriptors
{file_descriptors,[{total_limit,924},
{total_used,924},
{sockets_limit,829},
{sockets_used,19}]},
jcuff@jcuff-virtualbox:/opt/stack$ sudo su -
root@jcuff-virtualbox:~# ulimit -n 10000
root@jcuff-virtualbox:~# /etc/init.d/rabbitmq-server restart
* Restarting message broker rabbitmq-server [ OK ]
root@jcuff-virtualbox:~# rabbitmqctl report | grep -A3 file_descriptors
{file_descriptors,[{total_limit,9900},
{total_used,1033},
{sockets_limit,8908},
{sockets_used,7}]},
Then we got stuck here:
stack 31892 0.0 0.1 5216 1396 pts/2 S+ 12:44 0:00 bash
/opt/stack/devstack/tools/create_userrc.sh -PA --target-dir /opt/stack/devstack/accrc
Killed the process, tried to restart ./stack.sh... oops!
Ok clearly time for a new VM... this time I'm going to snapshot it before I start playing with ./stack.sh, as this vm has been seriously scribbled on :-)
Back later, once I make a new VM... doh! Pretty much everything was wrong with this setup!
1:08pm
Starting over again with 12.04.2 LTS 64bit Server version, and this time with enough disk space :-) So glad we have an epic fast network connection here, and this SSD drive on the laptop sure is helping out a lot!
1:22pm
Ok, new VM base created, I'm going to freeze this one so I don't ever have to do that part of the shenanigans again, if there are openstack folks looking on at this as we go try not to laugh too much, thus far it's all been a bit of a string of self inflicted PBKAC issues :-)
[James-Cuffs-MacBook-Pro]$ cd /Users/jcuff/VirtualBox\ VMs/
[James-Cuffs-MacBook-Pro]$ ls -ltra
total 0
drwx--x---+ 111 jcuff staff 3774 Apr 26 14:05 ..
drwxr-xr-x 7 jcuff staff 238 Apr 27 13:14 Ubuntu
drwxr-xr-x 4 jcuff staff 136 Apr 27 13:14 .
drwxr-xr-x 6 jcuff staff 204 Apr 27 13:25 OpenStack
[James-Cuffs-MacBook-Pro]$ du -sh OpenStack/
1.6G OpenStack/
[James-Cuffs-MacBook-Pro]$ cp -r OpenStack/ OpenStack.Golden
[James-Cuffs-MacBook-Pro]$
Ok, so that's going to help, not have to revisit that one! Time for a quick break, before going at stack.sh again :-) Ok, off we go again!
1:32pm
jcuff@openstack:~$ git clone git://github.com/openstack-dev/devstack.git
Cloning into 'devstack'...
remote: Counting objects: 9763, done.
remote: Compressing objects: 100% (3366/3366), done.
remote: Total 9763 (delta 6804), reused 9151 (delta 6305)
Receiving objects: 100% (9763/9763), 1.71 MiB | 2.12 MiB/s, done.
Resolving deltas: 100% (6804/6804), done.
jcuff@openstack:~$ cd devstack/
jcuff@openstack:~/devstack$ ./stack.sh
Good news, no 200% rabbitmq this time, that's great. So far so good! Lots of downloading to do now, may need another snapshot if this round of ./stack.sh succeeds.
Oh snap! Wrong MYSQL password... Doh!
An unexpected error prevented the server from fulfilling your request. (OperationalError) (1045, "Access denied for user 'root'@'localhost' (using password: YES)") None None (HTTP 500)
+ KEYSTONE_SERVICE=
Here we go again...
jcuff@openstack:~/devstack$ ./stack.sh
You are already running a stack.sh session.
To rejoin this session type 'screen -x stack'.
To destroy this session, type './unstack.sh'.
jcuff@openstack:~/devstack$ ./unstack.sh
* Stopping web server apache2
Volume group "stack-volumes" not found
Skipping volume group stack-volumes
Ok time to edit: jcuff@openstack:~/devstack$ vi localrc
And try again!
Holy shit! It worked!
1:46pm
Horizon is now available at http://10.0.1.12/
Keystone is serving at http://10.0.1.12:5000/v2.0/
Examples on using novaclient command line is in exercise.sh
The default users are: admin and demo
The password: letmein
This is your host ip: 10.0.1.12
stack.sh completed in 112 seconds.
Time for a well deserved cup of tea before the next steps!
Tea made, first image launched!!
Yup all good:
jcuff@openstack:~/devstack$ ssh cirros@10.0.0.2
cirros@10.0.0.2's password:
$ uname -a
Linux cirros 3.2.0-37-virtual #58-Ubuntu SMP Thu Jan 24 15:48:03 UTC 2013 x86_64 GNU/Linux
$ uptime
11:58:17 up 6 min, 2 users, load average: 0.07, 0.04, 0.03
$ cat /proc/meminfo | grep Mem
MemTotal: 503520 kB
MemFree: 473728 kB
$ cat /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 6
model : 2
model name : QEMU Virtual CPU version 1.0
stepping : 3
cpu MHz : 2212.096
cache size : 512 KB
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm up nopl pni cx16 popcnt hypervisor lahf_lm svm abm sse4a
bogomips : 4424.19
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
$ ifconfig eth0
eth0 Link encap:Ethernet HWaddr FA:16:3E:18:60:30
inet addr:10.0.0.2 Bcast:10.0.0.255 Mask:255.255.255.0
inet6 addr: fe80::f816:3eff:fe18:6030/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1153 errors:0 dropped:0 overruns:0 frame:0
TX packets:699 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:118307 (115.5 KiB) TX bytes:116076 (113.3 KiB)
$ hostname
cirros
2:20pm
Ok, time to snap this install again... oh my have we grown!
jcuff@openstack:~/devstack$ sudo shutdown -h now
Broadcast message from jcuff@openstack
(/dev/pts/1) at 14:18 ...
The system is going down for halt NOW!
jcuff@openstack:~/devstack$ Connection to 10.0.1.12 closed by remote host.
Connection to 10.0.1.12 closed.
[James-Cuffs-MacBook-Pro]$ pwd
/Users/jcuff/VirtualBox VMs
[James-Cuffs-MacBook-Pro]$ rm -rf OpenStack.Golden/
[James-Cuffs-MacBook-Pro]$ time cp -r OpenStack/ OpenStack.Golden
real 0m20.886s
user 0m0.007s
sys 0m3.656s
[James-Cuffs-MacBook-Pro]$ du -sh OpenStack.Golden/
3.0G OpenStack.Golden/
Sure was a whole lot easier than the folsom guide looks!
https://github.com/mseknibilel/OpenStack-Folsom-Install-guide/blob/master/OpenStack_Folsom_Install_Guide_WebVersion.rst
Tim has also had success!
2:53pm
Sure is a whole lot going on in here!
/usr/bin/qemu-system-x86_64
-S
-M pc-1.0
-no-kvm
-m 512
-smp 1,sockets=1,cores=1,threads=1
-name instance-00000001
-uuid c9c91966-e460-4131-bcf8-9e830c7854f8
-smbios type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=2013.2,serial=245343e1-c87c-4c83-a08a-26bbd6d2756a,uuid=c9c91966-e460-4131-bcf8-9e830c7854f8
-nodefconfig
-nodefaults
-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000001.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control
-rtc base=utc
-no-shutdown
-kernel /opt/stack/data/nova/instances/c9c91966-e460-4131-bcf8-9e830c7854f8/kernel
-initrd /opt/stack/data/nova/instances/c9c91966-e460-4131-bcf8-9e830c7854f8/ramdisk
-append root=/dev/vda console=ttyS0
-drive file=/dev/disk/by-path/ip-10.0.1.12:3260-iscsi-iqn.2010-10.org.openstack:volume-bc1a9e79-cf0d-4efe-9fe3-95da882f0277-lun-1,if=none,id=drive-virtio-disk0,format=raw,serial=bc1a9e79-cf0d-4efe-9fe3-95da882f0277,cache=none
-device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -netdev tap,fd=18,id=hostnet0
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:71:bb:de,bus=pci.0,addr=0x3
-chardev file,id=charserial0,path=/opt/stack/data/nova/instances/c9c91966-e460-4131-bcf8-9e830c7854f8/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1
-usb
-device usb-tablet,id=input0
-vnc 127.0.0.1:0
-k en-us
-vga cirrus
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
But that's cake compared to the database structure we need ;-)
for i in cinder keystone glance nove ; do echo "show tables"|mysql -u root -pxxxxxx $i; done |wc -l
159
Some folks
seemed to be concerned with HA. We will leave that for another day!
Anyhoo - now a fancy one, with new ssh keypairs working... look mom, no passwords!
jcuff@openstack:~/devstack$ ssh cirros@10.0.0.2
$ uname -a
Linux cirros 3.2.0-37-virtual #58-Ubuntu SMP Thu Jan 24 15:48:03 UTC 2013 x86_64 GNU/Linux
Cool, and this one is running off of block storage via iscsi!
-drive file=/dev/disk/by-path/ip-10.0.1.12:3260-iscsi-iqn.2010-10.org.openstack:volume-bc1a9e79-cf0d-4efe-9fe3-95da882f0277-lun-1
And yes, it really is iSCSI!
jcuff@openstack:~$ sudo iscsiadm -m discovery -t st -p 10.0.1.12
10.0.1.12:3260,1 iqn.2010-10.org.openstack:volume-bc1a9e79-cf0d-4efe-9fe3-95da882f0277
3:24pm
Time to get fancy with a real ubuntu install.
jcuff@openstack:~$ wget http://uec-images.ubuntu.com/raring/current/raring-server-cloudimg-amd64-disk1.img
2013-04-27 15:14:19 (3.50 MB/s) - `raring-server-cloudimg-amd64-disk1.img' saved [234618880/234618880]
jcuff@openstack:~$ file raring-server-cloudimg-amd64-disk1.img
raring-server-cloudimg-amd64-disk1.img: QEMU QCOW Image (v2), 2147483648 bytes
Very cool that you can also load img files directly from the web...
Yay! This is awesome... we have a working real ubuntu inside openstack
ubuntu@ubuntu:~$ uname -a
Linux ubuntu 3.8.0-19-generic #29-Ubuntu SMP Wed Apr 17 18:16:28 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
3:34pm
This ssh chain is getting a little carried away...
[James-Cuffs-MacBook-Pro]$ ssh James-Cuffs-MacBook-Pro.local
Last login: Sat Apr 27 15:43:02 2013 from 10.0.1.2
[James-Cuffs-MacBook-Pro]$ ssh jcuff@10.0.1.12
Welcome to Ubuntu 12.04.2 LTS (GNU/Linux 3.5.0-23-generic x86_64)
* Documentation: https://help.ubuntu.com/
System information as of Sat Apr 27 15:43:59 EDT 2013
System load: 0.05 Processes: 197
Usage of /: 21.2% of 17.74GB Users logged in: 1
Memory usage: 74% IP address for virbr0: 192.168.122.1
Swap usage: 4% IP address for br100: 10.0.0.1
Graph this data and manage this system at https://landscape.canonical.com/
51 packages can be updated.
35 updates are security updates.
Last login: Sat Apr 27 15:40:38 2013 from 10.0.1.2
jcuff@openstack:~$ ssh ubuntu@10.0.0.3
Welcome to Ubuntu 13.04 (GNU/Linux 3.8.0-19-generic x86_64)
* Documentation: https://help.ubuntu.com/
System information as of Sat Apr 27 19:44:13 UTC 2013
System load: 0.15 Processes: 73
Usage of /: 35.9% of 1.93GB Users logged in: 1
Memory usage: 25% IP address for eth0: 10.0.0.3
Swap usage: 0%
Graph this data and manage this system at https://landscape.canonical.com/
Get cloud support with Ubuntu Advantage Cloud Guest:
http://www.ubuntu.com/business/services/cloud
Use Juju to deploy your cloud instances and workloads:
https://juju.ubuntu.com/#cloud-raring
0 packages can be updated.
0 updates are security updates.
Last login: Sat Apr 27 19:41:01 2013 from 10.0.0.1
ubuntu@ubuntu:~$
A cloud inside a cloud inside a cloud in a box :-)
3:50pm
Pizza arrives! If you look close Tim's laptop has lots of green
[OK] going on!
Shockingly, discovered the openstack instigated VM can connect to the internet just fine!
ubuntu@ubuntu:~$ telnet www.google.com 80
Trying 74.125.26.104...
Connected to www.google.com.
Escape character is '^]'.
GET /
HTTP/1.0 200 OK
Date: Sat, 27 Apr 2013 19:47:54 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
4:07pm
Starting some more fun with NOVA, and instance types:
jcuff@openstack:/opt/stack/nova$ ./bin/nova-manage flavor list
m1.medium: Memory: 4096MB, VCPUS: 2, Root: 40GB, Ephemeral: 0Gb,
FlavorID: 3, Swap: 0MB, RXTX Factor: 1.0, public, ExtraSpecs {}
m1.micro: Memory: 128MB, VCPUS: 1, Root: 0GB, Ephemeral: 0Gb,
FlavorID: 84, Swap: 0MB, RXTX Factor: 1.0, public, ExtraSpecs {}
m1.tiny: Memory: 512MB, VCPUS: 1, Root: 0GB, Ephemeral: 0Gb,
FlavorID: 1, Swap: 0MB, RXTX Factor: 1.0, public, ExtraSpecs {}
m1.large: Memory: 8192MB, VCPUS: 4, Root: 80GB, Ephemeral: 0Gb,
FlavorID: 4, Swap: 0MB, RXTX Factor: 1.0, public, ExtraSpecs {}
m1.nano: Memory: 64MB, VCPUS: 1, Root: 0GB, Ephemeral: 0Gb,
FlavorID: 42, Swap: 0MB, RXTX Factor: 1.0, public, ExtraSpecs {}
m1.small: Memory: 2048MB, VCPUS: 1, Root: 20GB, Ephemeral: 0Gb,
FlavorID: 2, Swap: 0MB, RXTX Factor: 1.0, public, ExtraSpecs {}
m1.xlarge: Memory: 16384MB, VCPUS: 8, Root: 160GB, Ephemeral: 0Gb,
FlavorID: 5, Swap: 0MB, RXTX Factor: 1.0, public, ExtraSpecs {}
What is in the cli is in the gui... nice!
Ok, next up trying to turn one of Tim's laptop VMs into a nova compute engine, by end of day we would love to look something like this. Very much doubt that we will get there, but we will give it the good old college try :-)
4:23pm
While debugging noticed that a simple "screen -r" gets you to an awesome screen debug with 17 windows! Count em all if you can.
Pretty cool this, here you can see how it was set up:
jcuff@openstack:~/devstack$ cat stack-screenrc | grep tuff
/opt/stack/keystone/bin/keystone-all
sudo tail -f /var/log/apache2/horizon_error.log
/opt/stack/glance/bin/glance-registry
/opt/stack/glance/bin/glance-api
/opt/stack/nova/bin/nova-api
/opt/stack/nova/bin/nova-conductor
sg libvirtd /opt/stack/nova/bin/nova-compute
/opt/stack/nova/bin/nova-cert
/opt/stack/nova/bin/nova-network
/opt/stack/nova/bin/nova-scheduler
/opt/stack/nova/bin/nova-novncproxy
/opt/stack/nova/bin/nova-xvpvncproxy
/opt/stack/nova/bin/nova-consoleauth
/opt/stack/nova/bin/nova-objectstore
/opt/stack/cinder/bin/cinder-api
/opt/stack/cinder/bin/cinder-volume
/opt/stack/cinder/bin/cinder-scheduler
Update, lots of fannying about with Tim's laptop - was not configured with bridge and NAT sure makes it "interesting" to do computer to computer networking. While changing this we discovered that nova.conf and a whole pile of other things are all hard coded into the machine. Time to spin up fresh VMs rather than attempt to unpick this rats nest of static and hardcoded stuff. Even the fanciest of code seem to have a boatload of hardcoded links in it - this one even more than many others. Good job Tim is stella at the devops, onto bridged networking, we think we may even get a two membered set up going soon.
5:57pm
DevStack is great, but really only designed as we used it for single box runs. We wanted to do multiple boxes, and using automatic scripts to provision Ubuntu from RHEL was clearly a bad idea... ;-) Now copying Tim's RH64 version over so we can do multi compute images. Also now need to test out our Fusion to VirtualBox fu... this also should be a bit of a laugh! First up - go go gadget wifi!
[James-Cuffs-MacBook-Pro]$ pwd
/Users/jcuff/VirtualBox VMs
[James-Cuffs-MacBook-Pro]$ scp -r alaric@10.0.1.11:"/Users/alaric/Documents/Virtual\ Machines.localized/RH\ OS\ Lab\ -2\ .vmwarevm" ./RH64
RH OS Lab -2 .vmxf 100% 3433 3.4KB/s 00:00
Virtual Disk-s001.vmdk 100% 58MB 8.3MB/s 00:07
Virtual Disk-s002.vmdk 100% 320KB 320.0KB/s 00:01
Virtual Disk-s003.vmdk 100% 1363MB 8.7MB/s 02:36
Virtual Disk-s004.vmdk 100% 442MB 8.7MB/s 00:51
Virtual Disk-s005.vmdk 100% 272MB 8.5MB/s 00:32
Virtual Disk-s006.vmdk 100% 391MB 8.5MB/s 00:46
...
Update on Fedora, Michele tried a solid and eventually gave up, three downloads, corruption and other buckets of fail. She is now on the Ubuntu wagon!
In other news - oh my lord! Tim's copy to my laptop worked first time! Yay integration with virtualbox... I remember days when this was no way near as easy to do as this! Proof:
6:28pm
Oh my - RHEL bites us in the arse again! Be sure that you look at:
[paxindustria@localhost ~]$ cat /etc/udev/rules.d/70-persistent-net.rules
# This file was automatically generated by the /lib/udev/write_net_rules
# program, run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single
# line, and change only the value of the NAME= key.
# PCI device 0x8086:0x100e (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="08:00:27:92:66:1e", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"
# PCI device 0x8086:0x100e (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="08:00:27:b1:87:64", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"
Because we moved the instance to virtual box, and the packstack code expects to see "eth0", and "eth1".
Certainly gets very pissed if it happens to see "eth2" and "eth3"... le sigh. Anyhoo, chopped it out and now we are looking pretty awesome again! Tim is booting up the two laptop cluster as we speak.
Yay! Tim successfully got nova-compute installed on my laptop from his with packstack:
nova 4213 2.2 2.8 1117324 53948 ? Sl 15:38 0:01 /usr/bin/python /usr/bin/nova-compute --config-file /etc/nova/nova.conf --logfile /var/log/nova/compute.log
le sigh - clearly having network issues here, we have been bouncing firewalls and machines all over the place,
google searches are turning up a few questions about this... still hacking away here, but starting to hit more roadblocks than we thought. Mind you it is always a problem with the network. hehehe!
2013-04-27 16:19:52.193 4047 TRACE nova.compute.manager [instance: 709c127a-0f8f-4751-a496-f47e05ac73be] raise rpc_common.Timeout()
2013-04-27 16:19:52.193 4047 TRACE nova.compute.manager [instance: 709c127a-0f8f-4751-a496-f47e05ac73be] Timeout: Timeout while waiting on RPC response.
Tim is running a fresh install now we have sorted out our DNS issues.
8:22pm And we have SUCCESS! We have a two node cluster running across the network!
Tada!!
And that my friends is what we call a WRAP! Here ends our inaugral #openstackday
Hope you enjoyed the journey as much as we did.
.png)
