Sunday, July 29, 2012

GCE denied.

I have to say sorry to our community of researchers.

Without access to google compute engine, we can't help...

This tweet below pretty much sums it up!

To wait a whole month, only to be told no will never fly in our high velocity research shop. I hope #GCE can get their act together... 600,000 core genome demos don't mean an awful lot if folks are not able to get access to this mythical system.

Thursday, July 19, 2012

part two: scientific software as a service sprawl

Seppo asked a great question in the comment field for the last post...

So I pulled out some desperate perl and used our database that tracks usage of module load commands. The results are pretty fascinating when you ignore version numbers and look only at the raw main binary (i.e. versions of matlab don't count we just wanted to know the main programs). It all looks like fun... here you go:

First up our checkmodule code:
[jcuff@odyssey ~]$ checkmodule -u jcuff | head

Name     Module              Count     Time
jcuff    hpc/rc              19        Thu Jul 19 13:45:53 2012
jcuff    viz/gnuplot-4.5     135       Thu Jul 19 13:45:45 2012
jcuff    hpc/blat-34         150       Thu Jul 19 13:45:45 2012
jcuff    hpc/blastall        146       Thu Jul 19 13:45:45 2012
jcuff    hpc/abyss-1.2.1     242       Thu Jul 19 13:45:45 2012
jcuff    hpc/openmpi-intel   421       Thu Jul 19 13:45:45 2012
jcuff    bio/samtools-0.1.7  127       Thu Jul 19 13:45:45 2012
jcuff    bio/bowtie-0.11.3   150       Thu Jul 19 13:45:45 2012


As you can see it tots up the total modules that I've run, the list goes on a bit. Then we execute this evil to get a total over all possible users. Yeah our db layout is a bit odd we could have done this in a much better way, but this is research computing after all, it should have a little bit of cowboy by default!
[jcuff@odyssey ~]$ checkmodule | awk '{print$2" " $3}'| perl -pe 'while(<>){ ($n,$c)=split(/ /,$_,2);($m,$j)=split(/\-/,$n);$k{$m}=$k{$m}+$c;}foreach $h (keys %k){print "$k{$h} $h\n";}' | sort -rn 

QED, we then get a lovely list:
108040 hpc/openmpi
80607 math/matlab
59438 hpc/python
58890 hpc/matlab
49866 hpc/hdf5
46042 hpc/gsl
41291 hpc/intel
40882 hpc/gv
39281 hpc/netcdf
37982 hpc/IDL
29805 math/R
24183 hpc/fftw2
19133 hpc/svn
18334 hpc/xv
17978 hpc/mathematica
17365 hpc/gnuplot
16972 hpc/imagemagick
15314 hpc/fftw3
15196 hpc/numpy
14637 hpc/ds9
14218 hpc/gcc
13302 hpc/git

I guess we do a lot of hard sums here, it's openmpi and matlab all the time, plus all the cool kids now use python and not a lot of perl, perl5mods was pulled in 3,574 times in comparison to 59,438 lots of python... not an exact science this, but I'd say that python is getting legs ;-)

And Seppo was right, it has a tail alright ;-)

Wednesday, July 18, 2012

scientific software as a service sprawl...

Lots of chatter in HPC circles about how fast your machine is, how many petabytes of storage you have, how low is your latency, how fast is your fabric, yada, yada, yada, the list goes on and on. One thing folks forget is how hard it is to build open source software with hundreds of dependencies and interactions with system libraries and interpreted language sub systems.

Oh yeah, and perl, python ruby: I'm looking at each of ya'll ;-)).

So, I took a quick audit today of our own software system, we use modules to be able to pull in and manage various software, Fourier transform libraries, math, blas, compilers and big old honking perl pipelines. You know the sort of thing, foo requires bah depends on blergh, must be linked with piffle....

I knew the team had been adding software at a crazy rate, I just had no idea quite how far this had gone :-) Our faculty and researchers do love this stuff, we build it once, invest the time and then the codes are available to all quickly and easily as a module load hpc/matlab style operation. It is a huge cost savings, certainly to my organisation and the folks we support.

Do it once, reuse many!

[jcuff@hpc ~]$ module avail |& wc -l

That's just shy of two thousand, two hundred pieces of software loaded on our cluster!

[jcuff@hpc ~]$ module avail |& awk -F "/" '{print $1}' | sort | uniq -c | sort -nr
   1054 hpc
    618 bio
    194 math
    134 viz
    116 devel
     28 ncf
     28 chem
     14 geo
      2 mia32

However, industry folks worry a lot about server spraw, VM sprawl, disk sprawl.

So I guess my question is:

How are folks managing this new beast that is clearly "scientific software sprawl"?

If we are to believe the dream of SaaS models, it's only going to get way worse!

(c) 2018 James Cuff