Monday, June 16, 2008

Analyzing and cleaning out Linux/Unix filesystems that fill up

So I have this problem from time to time at work: bastid users or custom processes that either go rogue and start filling up filesystem space at an alarming rate, or crappy code that doesn't clean itself up.

Here's a couple of my favorite commands to help out with that:

`find` Command

If you aren't using `find`, you should. My favorite one-liners for stuff like this:

Find all files in a directory with a size between (for example) 1KB and 10MB that last changed 1 minute(s) ago:

[root@testbox]# find . -type f -size +1024k -and -size -10240k -cmin 1 -exec du -sb '{}' \;


Find all files in a directory with a size at least (for example) 75MB that last changed 1 minute(s) ago:

[root@testbox]# find . -type f -size +76800k -cmin 1 -exec du -sb '{}' \;


Find all files in a directory with a size between (for example) 1MB to 100MB, owned by 'frank' (UID: 501) that last changed 1 minute(s) ago:

[root@testbox]# find . -type f -size +1024k -and -size -102400k -uid 501 -cmin 1 -exec du -sb '{}' \;


...the execution of `du` in my commands will definitely add some overhead, so feel free to leave it off. It's a nice quick human readable presentation if you're in a hurry, but I wouldn't do it if you have a shitload of files that are going match your search.

`lsof` Command

Simply: list open files. This is your friend you can use in tandem with the `find` commands above to narrow down what process(es) are pillaging your filesystem.

What I do is take the output of a particular find command if I see growing files out there but cannot narrow down what process(es) it could be and just simply run:

[root@testbox]# lsof | grep file_name

...where obviously, file_name is some output you got from the `find` command(s).

Those are just some of many tricks to do to track down that type of ridiculousness that happens when things go bad late at night when you're on-call.