Huge rotational disk(s) + dpkg = disaster

Since a few versions now, dpkg has been made "safer" in that it makes sure everything is written to disk after unpacking a package. It does so with sync(), which means all kernel buffer caches are being flushed. Between every package install. All kernel buffer caches. Imagine what it does when you have two 2TB hard drives (though they are far from being full). Update: The use of btrfs might not be helping either.

I have 2 concurrent pbuilder instances that have been installing iceweasel build dependencies for more than 40 minutes, and still not finished yet. I don't expect the builds themselves, including test suites, to take more than 20 minutes on that machine...

I should really start using eatmydata...

Update: with eatmydata, the package installation is almost instantaneous...

Update 2: It might just be btrfs. And the best part yet is that since sync() is global, even when running pbuilder over, say, tmpfs, it doesn't change a damn thing.

2010-10-20 01:48:25+0900

p.d.o

You can leave a response, or trackback from your own site.

9 Responses to “Huge rotational disk(s) + dpkg = disaster”

  1. John Hughes Says:
    2010-10-20 06:39:31+0900

    It’s not the size of the discs – rather the amount of dirty cache, surely?

    If the discs aren’t full how come you have so much dirty cache?

    Also, does apt/dpkg/whatever sync all filesystems, or only the ones it writes? Maybe you need to have a smaller / and put all your space in /home, /usr/local, /var/local &c.

  2. Marcus Says:
    2010-10-20 08:43:53+0900

    Now I kinda feel like installing that program as well. The risk that software may not be crash-free, does that risk include file-system corruption and such?

  3. John Hughes Says:
    2010-10-20 16:36:27+0900

    I said “Also, does apt/dpkg/whatever sync all filesystems, or only the ones it writes?”

    Silly me, sync() syncs everything. There is no “sync this fs”.

  4. Matthew W. S. Bell Says:
    2010-10-20 18:16:45+0900

    It’s not dpkg’s fault. It’s the endlessly discussed shortcomings of the POSIX interface.

  5. Ken Bloom Says:
    2010-10-20 22:32:44+0900

    Once I saw this, I started wondering whether this affects my ext3/4 filesystem (ext3 mounted with the ext4 driver, but none of the advanced ext4 features enabled) as well. So in a cowbuilder chroot (with all of the packages pre-cached by apt-cacher-ng):

    # time eatmydata apt-get install –no-install-recommends openoffice.org
    0 upgraded, 142 newly installed, 0 to remove and 0 not upgraded.
    …
    real 0m57.682s
    user 0m37.030s
    sys 0m7.220s

    # time apt-get install –no-install-recommends openoffice.org
    0 upgraded, 142 newly installed, 0 to remove and 0 not upgraded.
    …
    real 3m17.158s
    user 0m37.186s
    sys 0m11.057s

    Over three times as long.

  6. Ken Bloom Says:
    2010-10-20 23:12:47+0900

    I think they chose sync() because they were worrying about issues in the kernel that were getting fixed upstream at about the time that they made these dpkg changes. I’d like to know whether they can go back to using fsync() again (or just rename into place) and count on things working quickly and reliably.

  7. MoDaX Says:
    2010-10-21 00:09:24+0900

    It’s btrfs (and ext4, and probably whatever else that uses delayed allocation). My benchmarks:

    http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=588254

  8. Debian Wiki Says:
    2010-12-29 18:03:07+0900

    M68k/Cowbuilder…

    … It’s also good to install eatmydata as it yields a (small, but on m68k we’re thankful for every bit) speed-up (whose cost is the potential loss of data integrity, but as the chroots are thrown away after building, this doesn’t matter). ……

  9. su Says:
    2012-09-21 09:06:33+0900

    (In case someone finds this…)
    One can also:
    echo “force-unsafe-io” > /etc/dpkg/dpkg.cfg.d/force-unsafe-io
    to disable sync()

Leave a Reply