Thursday, September 29, 2011

they call it smart for a reason...

I need a faster piece of glass into my home basement, 25mbit was the only rate determining issue here... 1 min 48 seconds is just way too long! ;-)

jcuff@srv:~$ wget
Saving to: `smartos-20110926T021612Z.iso'
100%[====================================>] 282,716,160 2.84M/s   in 1m 48s  

2011-09-29 15:11:34 (2.50 MB/s) - `smartos-20110926T021612Z.iso' 
[ one ]

/usr/bin/kvm -S -M pc-0.14 -enable-kvm -m 512 -smp 1,sockets=1,cores=1,threads=1 -name smartos -drive file=/home/jcuff/smartos-20110926T021612Z.iso,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -usb -device usb-tablet,id=input0 -vnc -vga cirrus -device AC97,id=sound0,bus=pci.0,addr=0x3 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4...... 
[ two ]

jcuff@srv:~$ xvncviewer localhost
Authentication successful
Desktop name "QEMU (smartos)"
VNC server default format:
32 bits per pixel.
[ three ]

[ ! done ! ]!/jasonh/status/119462126829056000

Sunday, September 25, 2011

ubuntu libvirt yeah it really is that good (tm)...

Watching this twitter thread:!/fak3r/status/118162368776777729

and remembering that the pfsense hackathon is soon, and pfsense 2.0 is alive!

Reminded me how simple it is now to spin up a VM, even one on a busted mac!

Check it:
root@isrv:~/Downloads# uname -a
Linux isrv 2.6.38-11-generic #50-Ubuntu SMP Mon Sep 12 21:18:14 UTC 2011

root@isrv:~/Downloads# cat /etc/lsb-release

From there it is cake:
root@isrv:~/Downloads# apt-get install kvm
root@isrv:~/Downloads# apt-get install libvirt-bin
root@isrv:~/Downloads# apt-get install virt-manager
root@isrv:~/Downloads# virt-manager
(follow some GUI prompts, point the GUI at the uncompressed .img file)


And it will load the kernel module with no effort!

root@isrv:~/Downloads# lsmod | grep kvm
kvm_intel              47162  3 
kvm                   317577  1 kvm_intel

In the style of A1 steak sauce - yeah it's that good!

Thanks Scott!

Thursday, September 22, 2011

beautiful two factor desktop client

Since working on two factor auth for a while we wanted to make an awesomely beautiful client that could work with google authenticator on the desktop and be secure.

Michele worked on this a bit... I poked at some parts to help, and we nailed it! We now have an undecorated, draggable, copyable and most awesome simple client for two factor and for the win!

googleplus and twitter have some more of the devops fun!!/jamesdotcuff/status/117056018625728512!/jamesdotcuff/status/117058135507730433

Code will be released soon once we test some more - we did have the mythical "off by one" error which anyone who has worked in genomic science knows it to be a classic!

Oh Michele just squashed that bug, as I type this and has committed the code up to her GIT repo as I type this!

"whoop there it is!"

Vote in the comments for the one that has a better use of real estate ;-)

the foreman's job

Our team here have been going gangbusters on puppet / cobbler and foreman recently. Turns out when you have ca. 13-14 thousand processors you really do need a configuration manager ;-)

Here's a little snap as we continue to develop the infrastructure, we still have a way to go, but we are getting there.

As you can see we are pushing forward with CentOS, not much 6.0 but we are getting there, 986 copies of CentOS 5.5 to move. Lots of changes in 6.0 that affect production environments esp LDAP, puppet manifests change and multiple modifications often needed. Seems we also quite like 8 cpu boxes, couple of fun sized 48's in there though. Oddly we only seem to have one type of architecture... and one type of environment - x86_64_PRODUCTION ;-)

Not sure why but every time I log into our provisioning server the "Foreman's Job" tune goes through my head - it is an all time favorite of my father's. Seemingly very popular back when he worked at the nuclear fuel factory, as being promoted to foreman was something to behold!

Foreman is still a pretty powerful figure today - certainly for "pointy hairs" as you can see the whole environment (and modify it) from a simple set of pages. Very swish.

Amazingly this crazy foreman's job tune exists on the internet!


To the tune of:

Wednesday, September 21, 2011

of tokens, car keys and security

Lots of concern recently on the net about secure token storage - especially since googleauthenticator hit the ground for google apps. One great write up of the issue can be found here:

I do have to say that "cuteauthenticator" is way more flash than my reference proof of concept implementation they did a great job! At some point we ought build a nicer java GUI for my quick lash up!

So, anyway long story short albeit you can grab the HTOP keys from a decent QR reader for your google auth key for gmail, or generate your own if you are using the libpam module for ssh or drupal auth you will end up with:

or a file by default it is copied to:


You basically have a simple 16 character string that defines the key. That's it nothing else is needed. Clearly time to put this little string of letters and numbers on a more secure system:
toofan:~ jcuff$ hdiutil create -encryption -size 10m secure.dmg -fs HFS+J -volname Secure

Enter a new password to secure "secure.dmg": 
Re-enter new password: 
created: /Users/jcuff/secure.dmg

Now you can open up the dmg it automatically prompts you for a password:
toofan:~ jcuff$ open secure.dmg

toofan:~ jcuff$ df -H /Volumes/Secure/
Filesystem     Size   Used  Avail Capacity  Mounted on
/dev/disk4s1    10M   774k   9.7M     8%    /Volumes/Secure

Simple - then you can write your token file in there, and you favorite desktop client, copy the dmg to a flash stick and you are golden - nice simple and fairly secure. As you can see one nice thing about using the OSX DMG files is they integrate nicely with your keychain.

jcuff@srv:/Volumes/Secure $ javac -cp ./ Authenticator/*.java

Run (point at your auth file):
jcuff@srv:/Volumes/Secure $ java -classpath ./ Authenticator.Main /Volumes/Secure/.google_authenticator

Authenticator Started!
:       Code Wait Time       :  Code  :
+++++++++++++++++++++++++++++: 964323 :
..............+++++++++++++++: 523907 :
.............................: 867553 :
.............................: 053547 :

Remember DMG encryption can be a little sketchy. If you need a platform independent way to do this there is always the most awesome truecrypt!

However, if you take a really quick look at the current most popular comercial way to store tokens that use a desktop client code for tokens...
/Users/jcuff/.RSA/RSA SecurID Software Token Library:
-rw-------  1 jcuff  staff  3072 Sep 20 16:13 RSASecurIDStorage

bash-3.2$ file RSASecurIDStorage 
RSASecurIDStorage: SQLite 3.x database

Ohh I remember that file type, in another life I found they were really quite popular inside the iPhone *ahem* ;-)
bash-3.2$ sqlite3 RSASecurIDStorage

SQLite version 3.6.12
Enter ".help" for instructions
Enter SQL statements terminated with a ";"

sqlite> .schema


James|Cuff|jcuff|"ahem cough"

Oh hello there!

The rest I'll leave as an exercise to the reader as to how to "move" this token file around, but suffice to say, let us use this analogy as a working hypothesis for all token activities:

"If you leave the keys to your car on a table in a public place be they for a Ferrari or a Skoda; someone will eventually come along and "borrow" them if they get a chance to!"

NOTE: It also *really* helps if the keys you leave around have the equivalent of the registration number and on which floor of the car park you left the car (uid, firstname, lastname) attached to the little keyring.

So clearly token files need to be looked after more seriously - luckily there are all sorts of "flash" ways to do this so you can keep your keys in your pocket and not out on the table ;-)

Saturday, September 3, 2011

why great partners matter

Large scale anything is tough. Really tough. Does not really matter what field you are in, if you run anything at scale: complexity increases. In my day job I'm responsible to our faculty, researchers and scientists each and every day trying our very best to be as good at supporting them as they are at pushing their craft.

It is a challenge. Tens of thousands of processors, multiple petabytes of data over 100's of different disciplines each day brings new surprises. Hardware suffers, software does not execute as you would expect and infrastructure can go wobbly at a moments notice. No one organization can pull this off single handedly - you need friends. Really, really good friends!

Product placement, endorsement and the like are frowned upon in most organizations (rightly so), and I constantly worry about how to reflect upon the success we have with our external and internal companies and partners.

They all know individually how hard they all work to make sure my success is achieved - I'm someone who gives feedback freely, they get it. I do however want to talk about two particular examples that have helped me, my team get things done.

"the junior members of staff"

This week I took a flight down to Austin to participate in an executive summit of IT professionals with the team that runs the IT business behind the scenes that in turn runs Dell. I met so many high rollers, it was a fabulous meeting. While there with the all grand fromage, I even managed to snag ten mins 1:1 with Michael himself - that was a lot of fun! However, I also finally got to meet some of our inside sales team. This is the team that I and my folks have been working with directly and depending upon for the last past five years.

I only ever knew them by email addresses, and the odd phone call. Andy and John have always made sure we always got everything we needed, and of course we normally needed it yesterday! It was so fabulous finally meeting them in person in their monster cube farm down in Round Rock.

On the giant metaphorical totem pole of a 100,000 person, 26 billion market capitalized organization these two boys are clearly vertically challenged to say the least! But without them, there is no chance either a company of that size or, more importantly we would have any degree of success. These two boys could work for any computer company on the planet. Any company that has folks like John and Andy will be successful, some companies lack the Johns and Andys, and they fail - it really is just that simple.

"the senior members of staff"

One of the other set of folks we work with run a data center facility in downtown Boston. Our organization asks for the moon on a stick daily, our growth has been substantial and is extremely unpredictable. Luckily as with the large computer manufacture I mentioned previously I'm fortunate to also have a homologue to "John and Andy".

This time it is the chap that actually owns the company. Jeff is by all definitions of the term a "senior chap", but he has the same drive tenacity and purpose that John and Andy have. I ask him for 400KW of cooling, he is on it. If infrastructure goes wobbly, he is on it.

John, Andy and Jeff are all made of the same stuff.

You may call it the right stuff, I simply call it awesome!

Bottom line is that every day I'm eternally grateful for our partners at every level and in every company we work with. You can see it on my face in the snap below of Jeff and I standing infront of some of our 14,000 processors that John and Andy helped to deliver up here in Boston.

Pretty cool eh?

(photo credit Carl Brooks)

(c) 2018 James Cuff