Thursday, January 6, 2011

many s3 disk log files

Be careful with those there s3 logs kids!

You can end up with a whole lot of stuff that you can't delete from the GUI (yeah and before you say real men don't use GUIs... I was all EBS and had no idea S3 existed on this account). I also did not know my bucket got set up accidentally with the log option, and it was rather busy after only a month or so!

I had "one or two", ok 8,336 files (glad I spotted it when I did!):
jcuff@shuttle:~/amazon/s3mount$ ls
{snip many!}

jcuff@shuttle:~/amazon/s3mount$ ls | wc -l

time to bust out some serious s3fs action, I needed 1.19 as my ubuntu fuse was too old:
Requested 'fuse >= 2.8.4' but version of fuse is 2.8.1

also need to make sure you have these pkgs:
jcuff@shuttle:~/amazon/s3fs-1.33$ sudo apt-get install g++
jcuff@shuttle:~/amazon/s3fs-1.33$ sudo apt-get install libxml2-dev
jcuff@shuttle:~/amazon/s3fs-1.33$ sudo apt-get install libfuse-dev
jcuff@shuttle:~/amazon/s3fs-1.33$ sudo apt-get install libcurl4-openssl-dev

you can now pull down s3fs 1.19, that does not have the recent 2.8.4 fuse deps:
jcuff@shuttle:~/amazon$ wget
jcuff@shuttle:~/amazon$ tar zxvf s3fs-1.19.tar.gz
jcuff@shuttle:~/amazon/s3fs-1.19$ ./configure
jcuff@shuttle:~/amazon/s3fs-1.19$ make
Making all in src
make[1]: Entering directory `/home/jcuff/amazon/s3fs-1.19/src'
jcuff@shuttle:~/amazon/s3fs-1.19$ sudo make install

and set up these env vars (they are like your normal vars, but w/o the underscores:

then you may mount the s3 disk (my bucket is called wickedpsyched), and remove the log files:
jcuff@shuttle:~/amazon$ s3fs wickedpsyched ./s3mount
jcuff@shuttle:~/amazon/s3mount$ rm log-*
jcuff@shuttle:~/amazon/s3mount$ ls
jcuff@shuttle:~/amazon/s3mount$ df -H .
Filesystem Size Used Avail Use% Mounted on
s3fs 282T 0 282T 0% /home/jcuff/amazon/s3mount
it sure does take a while, but is worth doing - all as a result of serious logging - be careful when you happen to tick that "Enabled" box:

[any opinions here are all mine, and have absolutely nothing to do with my employer]
(c) 2011 James Cuff