Undeleting opened files on ext3
I'm a specialist when it comes to delete files I didn't intend to. 3 years ago, I did an rm -rf that cost me most of my ${HOME}, and I wrote some software (which i really should work on, BTW) which helped recover most of it (it's tricky to get files back from ext3 once they are removed from the filesystem, since ext3 clears the block pointers from the inodes then).
Today, I somehow managed to remove log files. Fortunately the log files were still in use by a process, so they were still in the filesystem. Unfortunately they were still in use by a process, filling and filling and filling... and in no way i wanted to lose all this data.
Let's consider, for the needs of the demonstration, the following case:
mh@namakemono:/mnt/test$ while true; do date; sleep 1; done > log
That fills a 'log' file every second with the current date.
mh@namakemono:/mnt/test$ tail -2 log
2006å¹´ 7月 4æ—¥ ç«æ›œæ—¥ 20:48:02 CEST
2006å¹´ 7月 4æ—¥ ç«æ›œæ—¥ 20:48:03 CEST
The file keeps growing and growing. Now, let's remove it.
mh@namakemono:/mnt/test$ rm log
The file is not in the directory any more, but remains in the filesystem. It is actually readable through /proc:
mh@namakemono:/mnt/test$ ls -l /proc/10724/fd
åˆè¨ˆ 5
lrwx------ 1 mh users 64 2006-07-04 20:52 0 -> /dev/pts/1
l-wx------ 1 mh users 64 2006-07-04 20:52 1 -> /mnt/test/log (deleted)
lrwx------ 1 mh users 64 2006-07-04 20:52 10 -> /dev/pts/1
lrwx------ 1 mh users 64 2006-07-04 20:45 2 -> /dev/pts/1
lrwx------ 1 mh users 64 2006-07-04 20:52 255 -> /dev/pts/1
mh@namakemono:/mnt/test$ tail -2 /proc/10724/fd/1
2006å¹´ 7月 4æ—¥ ç«æ›œæ—¥ 20:52:45 CEST
2006å¹´ 7月 4æ—¥ ç«æ›œæ—¥ 20:52:46 CEST
As you can see, it kept filling while deleted. The usual technique to recover such a file is to copy it from /proc. I happen to have done so some times before, it works pretty well for static files. But that doesn't in our case: we'll get a new copy of the file, frozen at some point in time. Here, we'd want to 're-link' the file.
Basically, that can be done with debugfs. The problem is that I didn't find a clean way to get the inode number from the /proc/10724/fd/1 file. But that can be worked around. You need root access (what a surprise), and hope your filesystem was mostly clean before removing the file.
namakemono:/mnt/test# e2fsck -n /dev/namakemono/tmp
e2fsck 1.39 (29-May-2006)
Warning! /dev/namakemono/tmp is mounted.
Warning: skipping journal recovery because doing a read-only filesystem check.
/dev/namakemono/tmp contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Deleted inode 113570 has zero dtime. Fix? no
(...)
Don't worry, the -n option will make the filesystem opened read-only, so there's no risk for your data.
The interesting part of the output is obvious: the number of the deleted inode. You now only need to link the inode:
namakemono:/mnt/test# debugfs -w /dev/namakemono/tmp
debugfs 1.39 (29-May-2006)
debugfs: cd test
debugfs: ln <113570> log
debugfs: q
And see how it worked:
namakemono:/mnt/test# tail -2 log
2006å¹´ 7月 4æ—¥ ç«æ›œæ—¥ 20:58:40 CEST
2006å¹´ 7月 4æ—¥ ç«æ›œæ—¥ 20:58:41 CEST
You can tail again, it will show you the file is being appended to.
I guess you can do something along these lines to restore a file from ext2. I don't know for other filesystems.
If someone knows how to get the inode number without e2fsck, comments are opened :)
Update: Well, it didn't take long for me to actually find a better way:
namakemono:/mnt/test$ lsof -p 10724 -a -d 1
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
bash 10724 mh 1w REG 254,3 13552 113570 /mnt/test/toto (deleted)
Basically, when the file can be found in /proc/$pid/fd/$fd, run lsof -p $pid -a -d $fd.
2006-07-04 21:17:08+0900
