Archive for the 'ext3rminator' Category

ADSL woes

For 10 days now, I’ve had ADSL problems. Basically, there is something fishy somewhere between my end and the DSLAM. That can be anything, and for the moment, all I can do is wait for either my ADSL provider or France Telecom, whichever is responsible for the problem, to fix this. Anyways, my network connectivity is sometimes working (though quite slowly, especially on uploads), but more often not, with sometimes connectivity for a few seconds (enough to download small files or pop mail).

The glandium.org server being behind this ADSL line, it means that you’re probably not able to see this post. Or maybe a feed-reader/planet/whatever got it while the line was somehow working. Note I’ve been able to setup another MX server, so that mail sent to my domain go somewhere I can fetch them. Don’t worry about sending me messages, they will reach me. Just that it may take time for me to be able to see and/or answer them.

Anyways, the main downside is that it makes it harder to handle the upcoming xulrunner transition.

On the other hand, the upside is that I finally could take some time to work on ext3rminator again, basically rewriting the code from scratch for reasons I’ll explain when it will be ready for a release. It’s good to see that some new APIs in libext2fs, added since 2002~2003, when I first wrote ext3rminator, make some of the work easier. It’s still sad there is nothing to handle reading the journal. Not that it’s difficult (though not documented much), but that would downsize my code some more ;).

Update: It seems to be back to normal.

2008-06-08 16:39:56+0200

ext3rminator, p.d.o, website, xulrunner | Comments Off

Recovering files from ext3

Carlo Wood wrote a nice piece of documentation about how to recover files from ext3, based on his experience after deleting 3GB.

This actually happened to me 5 years ago, now, and is how ext3rminator came to exist. I’ve been meaning to update it for a while (and journal parsing has been on the top of the TODO list for a while), but package maintenance is unfortunately time consuming. And I have a big tendancy to do something else, too (like coding on git).

2008-03-13 21:05:09+0200

ext3rminator | 2 Comments »

Playing evil with VMWare ESX Server, part 3

For a reminder of the situation, see also:

I finally had time at work to implement a solution for the problem. The result is a bit less than 400 lines of code, which I hope to be able to make free-as-speech. I’ll probably have tons of paperwork to do with my employer before that can happen…

The program implements a fuse filesystem that you feed with a raw device image file (but I’m willing to implement more image file formats), and that shows the individual partitions as separate files, named “partn.fstype“. These files can then be mounted via loopback devices the standard way (which was not possible on vmfs for the reason you can find in my first post on the subject), with the benefit of not requiring offset adjustments (as when you want to mount partitions from a disk image), or some loopback device hack.

Additionally, it has an internal Copy-On-Write mechanism so that it is possible to mount “dirty” ext3 filesystems (e.g. a snapshot of a mounted filesystem) without modifying the original disk image. Note that there is no way, yet, to keep these writes after unmounting the fuse filesystem.

It uses libparted to handle the partition table reading, which means it will read any disk label types parted supports, such as BSD labels, sun partition tables, and so on. It doesn’t support LVM, though.

Unfortunately, VMWare ESX server 3.0 being based on RedHat Enterprise Linux 3, only an ancient version of libparted is available. By the way, this version (1.6.3) had a pretty bad bug that made it impossible to use it for regular files instead of devices: it was trying to do a BLKGETSIZE (or was it HDIO_GETGEO ?) ioctl on it. To workaround this, I implemented my own PedArchitecture. It was somehow a revelation, because with similar mechanism, I can implement support for different disk image types :).

Anyways, there have been quite some API changes in-between, so some updates will have to be done to the code…

In other news, I started working on ext3rminator again. There may be a new release in a few weeks.

Update: Thinking again about the ioctl issue, I’m wondering if vmfs couldn’t be responsible (again) for the failure, actually… I’ll check that on monday.

2007-01-12 21:58:39+0200

diskimgfs, ext3rminator | 1 Comment »

Undeleting opened files on ext3

I’m a specialist when it comes to delete files I didn’t intend to. 3 years ago, I did an rm -rf that cost me most of my ${HOME}, and I wrote some software (which i really should work on, BTW) which helped recover most of it (it’s tricky to get files back from ext3 once they are removed from the filesystem, since ext3 clears the block pointers from the inodes then).

Today, I somehow managed to remove log files. Fortunately the log files were still in use by a process, so they were still in the filesystem. Unfortunately they were still in use by a process, filling and filling and filling… and in no way i wanted to lose all this data.

Let’s consider, for the needs of the demonstration, the following case:
mh@namakemono:/mnt/test$ while true; do date; sleep 1; done > log

That fills a 'log' file every second with the current date.
mh@namakemono:/mnt/test$ tail -2 log
2006年 7月 4日 火曜日 20:48:02 CEST
2006年 7月 4日 火曜日 20:48:03 CEST

The file keeps growing and growing. Now, let’s remove it.
mh@namakemono:/mnt/test$ rm log

The file is not in the directory any more, but remains in the filesystem. It is actually readable through /proc:
mh@namakemono:/mnt/test$ ls -l /proc/10724/fd
合計 5
lrwx------ 1 mh users 64 2006-07-04 20:52 0 -> /dev/pts/1
l-wx------ 1 mh users 64 2006-07-04 20:52 1 -> /mnt/test/log (deleted)
lrwx------ 1 mh users 64 2006-07-04 20:52 10 -> /dev/pts/1
lrwx------ 1 mh users 64 2006-07-04 20:45 2 -> /dev/pts/1
lrwx------ 1 mh users 64 2006-07-04 20:52 255 -> /dev/pts/1
mh@namakemono:/mnt/test$ tail -2 /proc/10724/fd/1
2006年 7月 4日 火曜日 20:52:45 CEST
2006年 7月 4日 火曜日 20:52:46 CEST

As you can see, it kept filling while deleted. The usual technique to recover such a file is to copy it from /proc. I happen to have done so some times before, it works pretty well for static files. But that doesn’t in our case: we’ll get a new copy of the file, frozen at some point in time. Here, we’d want to ‘re-link’ the file.

Basically, that can be done with debugfs. The problem is that I didn’t find a clean way to get the inode number from the /proc/10724/fd/1 file. But that can be worked around. You need root access (what a surprise), and hope your filesystem was mostly clean before removing the file.

namakemono:/mnt/test# e2fsck -n /dev/namakemono/tmp
e2fsck 1.39 (29-May-2006)
Warning! /dev/namakemono/tmp is mounted.
Warning: skipping journal recovery because doing a read-only filesystem check.
/dev/namakemono/tmp contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Deleted inode 113570 has zero dtime. Fix? no
(...)

Don’t worry, the -n option will make the filesystem opened read-only, so there’s no risk for your data.
The interesting part of the output is obvious: the number of the deleted inode. You now only need to link the inode:
namakemono:/mnt/test# debugfs -w /dev/namakemono/tmp
debugfs 1.39 (29-May-2006)
debugfs: cd test
debugfs: ln <113570> log
debugfs: q

And see how it worked:
namakemono:/mnt/test# tail -2 log
2006年 7月 4日 火曜日 20:58:40 CEST
2006年 7月 4日 火曜日 20:58:41 CEST

You can tail again, it will show you the file is being appended to.

I guess you can do something along these lines to restore a file from ext2. I don’t know for other filesystems.

If someone knows how to get the inode number without e2fsck, comments are opened :)

Update: Well, it didn’t take long for me to actually find a better way:
namakemono:/mnt/test$ lsof -p 10724 -a -d 1
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
bash 10724 mh 1w REG 254,3 13552 113570 /mnt/test/toto (deleted)

Basically, when the file can be found in /proc/$pid/fd/$fd, run lsof -p $pid -a -d $fd.

2006-07-04 21:17:08+0200

ext3rminator, miscellaneous | 4 Comments »

How-to: switch your partitions to LVM and lose everything

(almost)

I’ve had some problems with disk space for a while, so much that I was “creating space” by creating an lvm logical volume of 5GB over 3 files (through loop devices) on 3 different partitions (/usr, /var, and /home) that all had less than 5GB free. But I needed 5GB free.

So, instead of continuing relying on such brain damaged solutions, I decided to just switch everything on lvm, which, incidentally, would allow me to use lvm snapshots for pbuilder or similar tools (pbuilder doesn’t support lvm snapshot, as far as I know). Since my / partition is quite small (so small that I sometimes have space problems at kernel upgrade), I just decided to keep it and just transform my /usr, /var, and /home partitions into a big lvm physical volume.

The challenge was to move all the files without any resort to external devices. Yes, I have other devices on other PCs, but they all are in about the same shape : full as hell. No way I could possibly free 40GB for backup. I couldn’t even burn some data : my DVD burner died a few weeks ago and I still haven’t bought a new one. I could’ve waited but well, I wanted to do it.

So, the week-end before last, I decided to go on, and do it the painful way. I used my swap partition to move /usr and /var around, so that I could free the space from these 2 partitions to create the premises of the lvm physical volume. First trick: not enough space in the swap partition, so I just deleted a bunch of files (mostly /usr/share/doc/ and /usr/share/locale files) so that all would fit. After moving all around, I’d just unpack all installed debian packages so that the files are there again.

Then, I managed to move everything by packs of 4~5GB, which was laborious. It involved creating temporary lvm physical volumes on temporary partitions and then moving the logical volumes’s chunks from physical volumes to other physical volumes on the same disk. Brain damage.

Anyway, the big fright came when I had this stupid idea to delete partitions and create them again. Why ? because neither fdisk (no surprise) nor parted can resize a partition. Well, parted can, but it apparently can’t without playing around with the underlying filesystem.

So, I deleted and recreated the partitions. And guess what ? They didn’t end up on the same sectors… The second stupid idea was to run pvcreate on the newly created not quite the same partitions… I thought I just lost everything. Pictures, personal repositories (especially 6 months worth (well, it took 6 months of real time, not of work time, but still) of work on xulrunner), everything.

It’s actually not the first time I (almost) lose everything. I once did rm -rf / in a chroot… in which /home was mounted bound… and wrote a program to scan the file-system for the precious files (that’s the story of ext3rminator, which I still have to release once and for all). That was 3 years ago.

First, calm down. Second, the bits are still on the disk (except maybe the bits overwritten by pvcreate). Third, calm down. Fourth, think.

All I obviously had to do was to find the correct location of the partitions, and pray that the pvcreate didn’t blow everything. Which was almost okay. I could find 2 of the LVM volumes out of 3, while the third one seemed in a pretty bad shape. A bit of search with my friend google on another computer hinted me on vgcfgrestore. So I pvcreated once more over this broken volume, and used vgcfgrestore to get the volume group correct. It seemed okay. I could mount the volume. That was close.

Well, to be sure, let’s run a fsck… ouch… inconsistency in some inodes. Okay, correct it. Once, twice… a couple times. Waw, is it that fucked up ? It can’t be that fucked up.

So I took another look and discovered that the third volume was offsetted by a sector. 512 octets. Ouch, letting the fsck correct things was definitely not a good idea. Correcting again the partition table, pvcreating and vgcfgrestoring again, fsck, and let’s rock and roll.

Conclusion(s):
– I only lost 150MB of MP3. I don’t care.
– Never ever use parted again, and never ever use fdisk in basic mode again.
– Avoid tricky procedures with stuff you are trying for the first time.
– Backups are for the weakest.

2006-02-14 21:56:19+0200

ext3rminator, miscellaneous | Comments Off