Archive for the 'miscellaneous' Category

Dear lazyweb

I would like to replace my current blog with a system that mostly generates static pages, with support for comments. I’d like it to take files as input for blog posts (I’d like to store them in git), instead of database tables, and to have a flexible markup language (flexible in that it’d allow to customize the HTML output), and flexible templates.

Ikiwiki might come close to that, though I haven’t looked into details. Dear lazyweb, would you know other software that’d fulfill my needs, or come close?

2011-01-16 10:12:53+0100

miscellaneous, p.d.o, p.m.o | 17 Comments »

Efficient way to get files off the page cache

There’s this great feature in modern operating systems called the page cache. Simply put, it keeps in memory what normally is stored on disk and helps both read and write performance. While it’s all nice for a day to day use, it often gets in the way when one wants to track performance issues with “cold cache” (when the files you need to access are not in the page cache yet).

A commonly used command to flush the Linux page cache is the following:

# echo 3 > /proc/sys/vm/drop_caches

Unfortunately, its effect is broad, and it flushes the whole page cache. When working on cold startup performance of an application like Firefox, what you really want is to have your page cache in a state close to what it was before you started the application.

One way to get in a better position than flushing the entire page cache is to reboot: the page cache will be filled with system and desktop environment libraries during the boot process, making the application startup closer to what you want. But it takes time. A whole lot of it.

In one of my “what if” moments, I wondered what happens to the page cache when using posix_fadvise with the POSIX_FADV_DONTNEED hint. Guess what? It actually reliably flushes the page cache for the given range in the given file. At least it does so with Debian Squeeze’s Linux kernel. Provided you have a list of files your application loads that aren’t already in the page cache, you can flush these files and only these from the page cache.

The following source code compiles to a tool to which you give a list of files as arguments, and that flushes these files:

#include <unistd.h>
#include <fcntl.h>
#include <sys/stat.h>
int main(int argc, char *argv[]) {
  int i, fd;
  for (i = 1; i < argc; i++) {
    if ((fd = open(argv[i], O_RDONLY)) != -1) {
      struct stat st;
      fstat(fd, &st);
      posix_fadvise(fd, 0, st.st_size, POSIX_FADV_DONTNEED);
      close(fd);
    }
  }
  return 0;
}

It’s actually a pretty scary feature, especially on multi-user environments, because any user can flush any file she can open, repeatedly, possibly hitting system performance. By the way, on systems using lxc (and maybe other containers, I don’t know), running the echo 3 > /proc/sys/vm/drop_caches command from a root shell in a container does flush the host page cache, which could be dangerous for VPS hosting services using these solutions.

Update: I have to revise my judgment, it appears posix_fadvise(,,,POSIX_FADV_DONTNEED) doesn’t flush (parts of) files that are still in use by other processes, which still makes it very useful for my usecase, but also makes it less dangerous than I thought. The drop_cache problem is still real with lxc, though.

2010-12-29 19:34:37+0100

miscellaneous, p.d.o, p.m.o | 11 Comments »

Recommended for You

There is a part on the Youtube home page which recommends some videos based on what you already watched.

I’m still trying to understand what kind of link there can be.

2010-05-22 11:33:24+0100

miscellaneous, p.m.o | No Comments »

日本語入力をibusで

最近まで、scimを使っていましたが、一週間前ibusを使ってみました。Input Method自体がまだanthyですけれどナントナクibusの方が使い易いというか楽というか。

とにかく、Debianでインストールするのは簡単です。最初はrootで:

$ apt-get install ibus-anthy ibus-gtk ibus-qt4 (Gnome/Gtk+のソフトだけを使ってる場合はibus-gtkだけでいいですし、KDE/Qtの場合はもちろんibus-qt4だけでもいいです)

次にユーザで:

$ im-switch -s ibus

2010-04-26 09:11:26+0100

miscellaneous, p.d.o | No Comments »

ssh root@guitar

After the coffee machine, would you have expected to be able to login on a guitar ? Well, now, you can, on the Misa digital. It is not only a digital guitar, but it runs (Gentoo) Linux, and… an ssh server.

2010-01-20 17:14:48+0100

miscellaneous, p.d.o | 1 Comment »

VMware + X.org + gnome-screensaver + strong password = FAIL

Guess what happens when you use software that can fuck up your keyboard mappings (VMware Remote Console), in combination with software that uses these mappings to be able to switch back to a text console (X.org) and a screen saver that locks your screen (gnome-screensaver, but that would worl equally well with xlock or anything else similar) ? A recipe for FAIL.

It so happens that VMware Remote Console, and apparently other VMware products such as Player or Workstation not only are unable to do anything useful with special keys (try installing Debian without the arrows keys, for example), but they are also able to remap keys (such as ctrl, shift and caps lock) to nothing.

It also happens that X.org uses its keyboard mappings when dealing with the ctrl+alt+Fn key combinations that allow to get back to a text console. Yes, that means you can’t switch to a text console after VMware fucked up your keyboard mappings.

On top of all that, add a X session locking program, that won’t allow you back until you type your password, and a password that, well, you just can’t type without shift of caps-lock, because it is somehow strong. The X session locking program won’t allow you to fix your keyboard mappings, you can’t switch to a text console either, and you can’t unlock for obvious reasons.

The only solution that didn’t involve a reboot or losing everything under the X session was to ssh in, change the password to one that can be typed without shift and unlock.

It is said on the interwebs that adding “xkeymap.nokeycodeMap = true” in the ~/.vmware/config file solves the issue. At least, it works for the arrows. I’ll see if it also prevents the special keys to be remapped.

2009-12-09 00:13:15+0100

miscellaneous, p.d.o | 5 Comments »

Chrome OS

Chrome OS has been announced today, and from a quick look at this video, it very much looks like it is some kind of Coreboot (formerly LinuxBIOS), X and Chrome combination. As such, it seems it’s not something you’re going to fully enjoy on any computer.

Update: Apparently, they use their own firmware and a Linux 2.6.30 kernel. If they went the custom firmware route, and wanted a fast boot, I wonder why they didn’t start with Coreboot, instead.

2009-11-19 21:32:56+0100

miscellaneous, p.d.o | 3 Comments »

How not to provide robust clustered storage with Linux and GFS

(The title is a bit strong, on purpose)

LWN links to an article describing how to provide robust clustered storage with Linux and GFS.

While explaining how to setup GFS can be nice, the incentive made me jump.

The author writes:

Load balancing is difficult; often we need to share file systems via NFS or other mechanisms to provide a central location for the data. While you may be protected against a Web server node failure, you are still sharing fate with the central storage node. Using GFS, the free clustered file system in Linux, you can create a truly robust cluster that does not depend on other servers. In this article, we show you how to properly configure GFS.

In case you don’t know, GFS is not exactly a “clustered storage”. It is more a “shared storage”. You have one storage, and several clients accessing it. You have one storage array, compared to the NFS case, where you have one central server for the data. But what is a storage array except a special (and expensive) kind of server ? You don’t depend on other servers, but you depend on other servers ? How is that supposed to be different ?

Conceptually, a clustered file system allows multiple operating systems to mount the same file system, and write to it at the same time. There are many clustered file systems available including Sun’s Lustre, OCFS from Oracle, and GFS for Linux.

OCFS and GFS are the same class of file systems, but Lustre is definitely out of league and would, actually, provide a truly robust cluster that does not depend on other servers. Lustre is a truly clustered filesystem, that distributes data on several nodes such that losing some nodes don’t make you lose access to the data.

With the incentive given by the author, and considering he lists Lustre as an example, I would actually have preferred an article about setting up Lustre.

2009-04-07 20:29:04+0100

miscellaneous, p.d.o | 13 Comments »

A revolution happening at the W3C ?

It seems undergoing discussion is leading towards a free (as in speech) HTML5 specification.

That would really be great.

2009-04-02 21:31:00+0100

miscellaneous, p.d.o | Comments Off

Emptying a deleted file

Yesterday, at work, we had the typical case where df would say there is (almost) no space left on some device, while du doesn’t see as much data present as you would expect from this situation. This happens when you delete a file that another process has opened (and, obviously, not yet closed).

In typical UNIX filesystems, files are actually only entries in a directory, pointing (linking) to the real information about the content, the inode.

The inode contains the information about how many such links exist on the filesystem, the link count. When you create a hard link (ln without -s), you create another file entry in some directory, linking to the same inode as the “original” file. You also increase the link count for the inode.

Likewise, when removing a file, the entry in the directory is removed (though most of the time, really only skipped, but that’s another story), and the link count decreased. When the link count is zero, usually, the inode is marked as deleted.

Except when the usage count is not zero.

When a process opens a file, the kernel keeps a usage count for the corresponding inode in memory. When some process is reading from a file, it doesn’t really expect it to disappear suddenly. So, as long as the usage count is not null, even when the link count in the inode is zero, the content is kept on the disk and still takes space on the filesystem.

On the other hand, since there is no entry left in any directory linking to the inode, the size for this content can’t be added to du‘s total.

Back to our problem, the origin was that someone had to free some space on a 1GB filesystem, and thought a good idea would be to delete that 860MB log file that nobody cares about. Except that it didn’t really remove it, but he didn’t really check.

Later, the “filesystem full” problem came back at someone else, who came to ask me what files from a small list he could remove. But the files were pretty small, and that wouldn’t have freed enough space. That gave me the feeling that we probably were in this typical case I introduced this post with, which du -sk confirmed: 970MB used on the filesystem according to df, but only 110MB worth of data…

Just in case you would need to find the pid of the process having the deleted file still opened, or even better, get access to the file itself, you can use the following command:

find -L /proc/*/fd -type f -links 0

(this works on Linux ; remove -L on recent Solaris ; on other OSes, you can find the pid with lsof)

Each path this command returns can be opened and its content accessed with a program, such as cat. That will give access to the deleted content.

I already adressed how to re-link such a file, which somehow works under Linux, but in my case, all that mattered was to really remove the file, this time. But we didn’t know if it was safe to stop the process still holding the file, nor how to properly restart it. We were left without a possible resolution, but still needed to come up with something before the filesystem gets really full while waiting to be able to deal with the root of the problem.

The first crazy idea I had was to attach a debugger to the process, and use it to close the file descriptor and open a new file instead (I think you can find some examples with google). But there was no debugger installed.

So, I had this other crazy idea: would truncate() work on these /proc/$pid/fd files?

You know what? It does work. So I bought us some time by running:

perl -e 'truncate("/proc/$pid/fd/$fd", 0);'

(somehow, there is no standard executable to do a truncate(), so I always resort to perl)

Afterwards, I also verified the same works under Linux (where you wouldn’t really know what it’d do with these files that are symbolic links to somewhere that doesn’t exist).

The even simpler following command works, too.

> /proc/$pid/fd/$fd

It doesn’t truncate() but open() with O_WRONLY | O_CREAT | O_TRUNC, and close() right after (to simplify), which has the same effect.

Good to know, isn’t it?

2008-11-06 22:23:33+0100

miscellaneous, p.d.o | 5 Comments »