Friday, March 16, 2012

local /scratch turned up to 11: thanks much mr rdma!

Update on: http://blog.jcuff.net/2012/03/glusterfs-80tb-in-seconds-flat.html

Here's another recipe:

Take six little 500G spindles that each manage about 100MB/s on six little compute boxes add 3oz of our new found friend RDMA, and a soup├žon of IB network we just happened to find lying around the kitchen ;-)
[root@itc011 gfs]# gluster volume info ibgfs

Volume Name: ibgfs
Type: Stripe
Status: Started
Number of Bricks: 6
Transport-type: rdma
Bricks:
Brick1: itc031:/scratch/gfs
Brick2: itc032:/scratch/gfs
Brick3: itc012:/scratch/gfs
Brick4: itc021:/scratch/gfs
Brick5: itc011:/scratch/gfs
Brick6: itc022:/scratch/gfs

We love it when things scale in mostly straight lines for size + scale
[root@itc071 /test]# df -H .
Filesystem             Size   Used  Avail Use% Mounted on
glusterfs#itc022:/ibgfs
                       2.8T    54G   2.6T   3% /test

[root@itc071 /test]# time dd if=/dev/zero of=test3.dat bs=1024k count=10000
10000+0 records in
10000+0 records out
10485760000 bytes (10 GB) copied, 14.1454 seconds, 741 MB/s


[any opinions here are all mine, and have absolutely nothing to do with my employer]
(c) 2011 James Cuff