Friday, July 31, 2015

How does your cluster sound?

So I was on the airplane coming back from XSEDE15 in St. Louis, and got to thinking about all the amazing visualizations that were on display. I wondered. What would a cluster sound like? On our HPC cluster we have millions of jobs running each month, and often 10-20,000 running simultaneously. So I decided to go on a hunt for a MIDI player and a MIDI file format generator.

Found both in seconds, the internet is awesome!

First up a player for OSX (you don't do the --with-libsndfile, it won't work at the end):
brew install libsndfile lame
brew install --with-libsndfile fluidsynth

Now download a soundfont (wow this takes me back!)

wget http://www.schristiancollins.com/soundfonts/GeneralUser_GS_1.44-FluidSynth.zip

and we have the musics!

Jamess-MacBook-Pro:GeneralUser GS 1.44 FluidSynth jcuff$ fluidsynth -i ./GeneralUser\ GS\ FluidSynth\ v1.44.sf2 demo\ MIDIs/All\ Night\ Long.mid
FluidSynth version 1.1.6
Copyright (C) 2000-2012 Peter Hanappe and others.
Distributed under the LGPL license.
SoundFont(R) is a registered trademark of E-mu Systems, Inc.

Ok so we can play a midi file from the CLI. Time to write one now. And yet again, the internet provides:

http://code.google.com/p/midiutil/

So here we go:
Jamess-MacBook-Pro:~ jcuff$ wget http://midiutil.googlecode.com/files/MIDIUtil-0.89.zip
Jamess-MacBook-Pro:~ jcuff$ unzip MIDIUtil-0.89.zip 
Jamess-MacBook-Pro:~ jcuff$ cd MIDIUtil-0.89
Jamess-MacBook-Pro:~ jcuff$ ls
Jamess-MacBook-Pro:~ jcuff$ python ./setup.py install
Jamess-MacBook-Pro:~ jcuff$ sudo python ./setup.py install
Jamess-MacBook-Pro:~ jcuff$ python ./examples/single-note-example.py 
Jamess-MacBook-Pro:~ jcuff$ file output.mid 

output.mid: Standard MIDI data (format 1) using 1 track at 1/960

Jamess-MacBook-Pro:MIDIUtil-0.89 jcuff$ fluidsynth -v -i ../GeneralUser\ GS\ 1.44\ FluidSynth/GeneralUser\ GS\ FluidSynth\ v1.44.sf2 output.mid 
FluidSynth version 1.1.6
Copyright (C) 2000-2012 Peter Hanappe and others.
Distributed under the LGPL license.
SoundFont(R) is a registered trademark of E-mu Systems, Inc.

fluidsynth: noteon 0 60 100 00000 0.975 1.113 0.000 0
fluidsynth: noteoff 0 60 0 00000 1.612 1


Ok - so we can play notes from the command line. Time to knock up a parser from sacct data, let's use -p and -l so we can actually parse the data that comes out… :-)

[root@sa01 tmp]# sacct -p -l > /tmp/sacct.dat
[root@sa01 tmp]# wc -l /tmp/sacct.dat
58651 /tmp/sacct.dat

Ok so we have some rich sacct data for our 58,651 jobs in the system right now. Let's look at the ones that completed, and ran for one day, write some weird chord generator, and then add Michele Clamp's epic python under a 1 hour rather extreme pair programing episode and we give you…


THE SOUND OF ODYSSEY!



Here's the code for your enjoyment and modification.

So what does your cluster sound like?

:-)

You can get after your sound file to post to your sound cloud site with this:
Jamess-MacBook-Pro:MIDIUtil-0.89 jcuff$ python ./the_sound_of_odyssey.py -f tt -s 1 -e 4


Jamess-MacBook-Pro:MIDIUtil-0.89 jcuff$ fluidsynth -F out.wav -i ../GeneralUser\ GS\ 1.44\ FluidSynth/GeneralUser\ GS\ FluidSynth\ v1.44.sf2 output.mid 
FluidSynth version 1.1.6
Copyright (C) 2000-2012 Peter Hanappe and others.
Distributed under the LGPL license.
SoundFont(R) is a registered trademark of E-mu Systems, Inc.

Rendering audio to file 'out.wav'..


Jamess-MacBook-Pro:MIDIUtil-0.89 jcuff$ lame out.wav 
LAME 3.99.5 64bits (http://lame.sf.net)
Using polyphase lowpass filter, transition band: 16538 Hz - 17071 Hz
Encoding out.wav to out.mp3
Encoding as 44.1 kHz j-stereo MPEG-1 Layer III (11x) 128 kbps qval=3
    Frame          |  CPU time/estim | REAL time/estim | play/CPU |    ETA 
 27776/27776 (100%)|    0:17/    0:17|    0:18/    0:18|   40.386x|    0:00 
-------------------------------------------------------------------------------------------------------------------------------------------
   kbps        LR    MS  %     long switch short %
  128.0       50.4  49.6        99.6   0.3   0.2
Writing LAME Tag...done
ReplayGain: +10.1dB

Jamess-MacBook-Pro:MIDIUtil-0.89 jcuff$ open out.mp3 

And finally, here's the code!
Jamess-MacBook-Pro:MIDIUtil-0.89 jcuff$ cat ~/Downloads/the_sound_of_odyssey-2.py 

# wget http://midiutil.googlecode.com/files/MIDIUtil-0.89.zip
# unzip MIDIUtil-0.89.zip 
# cd MIDIUtil-0.89
# python sudo ./setup.py install

# Fluidsynth has better noises :
# brew install --with-libsndfile fluidsynth
# wget http://www.schristiancollins.com/soundfonts/GeneralUser_GS_1.44-FluidSynth.zip
# unzip this into a directory

# python the_sound_of_odyssey.py -f sacct.dat -s 17 -e 25  (Writes into output.mid)

# fluidsynth -F output.wav -i ./GeneralUser\ GS\ 1.44\ FluidSynth/GeneralUser\ GS\ FluidSynth\ v1.44.sf2 output.mid

from argparse  import ArgumentParser
from random    import randint

import math
import re

from midiutil.MidiFile import MIDIFile

parser        = ArgumentParser(description = 'Convert sacct data to midi')

parser.add_argument('-f','--file'     ,      help="The sacct data file")
parser.add_argument('-s','--programstart'  , help="The instrument program start")
parser.add_argument('-e','--programend'  ,   help="The instrument program end")
parser.add_argument('-b','--bpm'  ,          help="Beats per minute")

args = parser.parse_args()

programstart = 1 
programend   = 1
bpm          = 120

if args.programstart is not None:
   programstart = int(args.programstart)

if args.programend is not None:
   programend= int(args.programend)

if args.bpm is not None:
   bpm = int(args.bpm)

fh = open(args.file)

MyMIDI = MIDIFile(1)

track = 0
time  = 0

MyMIDI.addTrackName(track,time,"Sample Track ")
MyMIDI.addTempo(track,time, bpm)

lnum = 0 
daysecs = 24*60*60

maxtime  = 0

for line in fh:
   lnum = lnum + 1

   if lnum == 1:
     continue

   line    = line.rstrip('\n')
   ff      = line.split('|')

   cores   = int(ff[21])
   elapsed = ff[22]
   status  = ff[23]

   if lnum%10 != 0:
      continue

   if status != "COMPLETED":
      continue

   tt = elapsed.split(':')
      
   channel  = 0

   duration = cores+1
   volume   = 90

   if len(tt) == 3 and elapsed > 0 and cores != 0 and '-' not in elapsed:

     program = randint(programstart,programend)
     
     secs    = int(tt[0])*60*60 + int(tt[1])*60 + int(tt[2])
     newsecs = 10 + int(secs*(127-10)/daysecs)
     #time    = secs*120.0/float(daysecs)
     #pitch   = cores
     pitch    = newsecs 

     MyMIDI.addProgramChange(track,channel, time, program)

     print tt[0],tt[1],tt[2],secs,newsecs,daysecs,program,time

     MyMIDI.addNote(track,channel,int(pitch),time,duration,volume)
     MyMIDI.addNote(track,channel,int(pitch-randint(1,4)),time+0.2,duration,volume)
     MyMIDI.addNote(track,channel,int(pitch-randint(4,8)),time+0.3,duration,volume)

     tmppitch = pitch + 12
     if tmppitch > 127:
       tmppitch = 127

     MyMIDI.addNote(track,channel,int(tmppitch),time,duration,volume)

     if time > maxtime:
        maxtime = time

     tmp = float((math.sqrt(cores))/2.0)

     if tmp > 10.0:
       tmp = 10.0
 
     time = float(time) + tmp

print maxtime

i =  0

MyMIDI.addProgramChange(track,10,1,116)

while i < maxtime:
   MyMIDI.addNote(track,10,60,i,1,100)
   MyMIDI.addNote(track,10,60,i+0.5,1,70)
   i = i + 1

# And write it to disk.
binfile = open("output.mid", 'wb')
MyMIDI.writeFile(binfile)
binfile.close()


[any opinions here are all mine, and have absolutely nothing to do with my employer]
(c) 2011 James Cuff