Thursday, January 6, 2011

many s3 disk log files

Be careful with those there s3 logs kids!

You can end up with a whole lot of stuff that you can't delete from the GUI (yeah and before you say real men don't use GUIs... I was all EBS and had no idea S3 existed on this account). I also did not know my bucket got set up accidentally with the log option, and it was rather busy after only a month or so!



I had "one or two", ok 8,336 files (glad I spotted it when I did!):
jcuff@shuttle:~/amazon/s3mount$ ls
log-wickedpsyched2010-11-22-19-30-46-23BD9FFEE0F31DED
log-wickedpsyched2010-12-14-21-22-10-A7F2A970769192A9
log-wickedpsyched2010-11-22-19-32-41-C4B0F50214C73614
log-wickedpsyched2010-12-14-21-25-00-00B4C457DCD202DF
log-wickedpsyched2010-11-22-19-35-41-BCC060705BE7A3EF
log-wickedpsyched2010-12-14-21-25-42-2EA205A6EE223BF3
{snip many!}

jcuff@shuttle:~/amazon/s3mount$ ls | wc -l
8336

time to bust out some serious s3fs action, I needed 1.19 as my ubuntu fuse was too old:
Requested 'fuse >= 2.8.4' but version of fuse is 2.8.1

also need to make sure you have these pkgs:
jcuff@shuttle:~/amazon/s3fs-1.33$ sudo apt-get install g++
jcuff@shuttle:~/amazon/s3fs-1.33$ sudo apt-get install libxml2-dev
jcuff@shuttle:~/amazon/s3fs-1.33$ sudo apt-get install libfuse-dev
jcuff@shuttle:~/amazon/s3fs-1.33$ sudo apt-get install libcurl4-openssl-dev

you can now pull down s3fs 1.19, that does not have the recent 2.8.4 fuse deps:
jcuff@shuttle:~/amazon$ wget http://s3fs.googlecode.com/files/s3fs-1.19.tar.gz
jcuff@shuttle:~/amazon$ tar zxvf s3fs-1.19.tar.gz
jcuff@shuttle:~/amazon/s3fs-1.19$ ./configure
jcuff@shuttle:~/amazon/s3fs-1.19$ make
Making all in src
make[1]: Entering directory `/home/jcuff/amazon/s3fs-1.19/src'
jcuff@shuttle:~/amazon/s3fs-1.19$ sudo make install

and set up these env vars (they are like your normal vars, but w/o the underscores:
export AWSACCESSKEYID=SEKR!TK3Y
export AWSSECRETACCESSKEY=MUCHL0NG3RS3KR1TK3Y

then you may mount the s3 disk (my bucket is called wickedpsyched), and remove the log files:
jcuff@shuttle:~/amazon$ s3fs wickedpsyched ./s3mount
jcuff@shuttle:~/amazon/s3mount$ rm log-*
jcuff@shuttle:~/amazon/s3mount$ ls
jcuff@shuttle:~/amazon/s3mount$ df -H .
Filesystem Size Used Avail Use% Mounted on
s3fs 282T 0 282T 0% /home/jcuff/amazon/s3mount
it sure does take a while, but is worth doing - all as a result of serious logging - be careful when you happen to tick that "Enabled" box:



[any opinions here are all mine, and have absolutely nothing to do with my employer]
(c) 2011 James Cuff