Use unionfs instead of LVM snapshots

[ghudson]

Right now we use LVM snapshots for copy-on-write functionality in two places: on the build server (linux-build-10) and for login chroots. This works pretty well, but it works at the storage layer, which is a somewhat heavyweight approach.

It is also possible to do this at the filesystem layer using unionfs. I do not know a lot about unionfs at this time, but tabbott reports that this approach performs better for large numbers of snapshots.

Unfortunately, schroot does not have support for unionfs at this time. There is a patch at:

 http://lists.alioth.debian.org/pipermail/buildd-tools-devel/2008-January/001964.html

tabbott has a port of it to schroot 1.2.1 if we want to pursue this.

[jdreed]

Is unionfs support better in Jaunty and/or Karmic? If so, we should start looking at this again. Between this and #321, it would be awesome if cluster logins were faster.

[broder]

I can't keep track of all this filesystem stuff. Isn't aufs the new hotness, though?

[tabbott]

aufs seems to be the preferred choice. When I last looked at it, unionfs-fuse didn't work (e.g. you could not delete a directory).

Also, Unionfs support was merged into schroot master recently, and will go into a 1.3.0 release in the near future (they did an -rc1 a couple days ago).

[geofft]

schroot aufs support appears to be mostly stable; we should hack up reactivate to use schroot 1.3.0rc1 (or the latest version in git) and aufs, and make sure it works reaonably so we're ready when 1.3.0 is released.

Since LVM's slowness is probably Debathena's most annoying bug, especially on quickstations, I'm increasing the priority.

Incidentally, I have a feature-ish request into schroot that affects aufs use with multiple users:
 http://lists.alioth.debian.org/pipermail/buildd-tools-devel/2009-August/003622.html
Fixing this would let us do something crazy like a dialup with aufs chroots, but we don't need to care about multiple simultaneous users for clusters.

[broder]

If someone works on this, they need to keep in mind that (a)ufs can not be used as a simple drop-in replacement for the LVM snapshots. In particular, the auto-update mechanism only works currently because of the assumption that we can modify the master filesystem without it affecting the snapshot.

[geofft]

I've started working on this in r24125; I've tested this on lola-granola and it works spiffily. It does require a couple of changes to the schroot package -- two to the build system, one to schroot itself to permit union mounts of / -- that I plan on sending upstream. There's also a fair amount of cleanup possible in that code before we send it to production.

[geofft]

I've finished up most of what I want this code to do, and uploaded reactivate 2 and schroot 1.3 to -development (and the auto-update that it depends on to -proposed). I've also upgraded next5 and next6.

The one thing we should consider doing at some point before pushing this to production is making use of the remaining disk space since we're no longer using it for LVM snapshots. One option very much in the spirit of the original partitioning is to make it swap space, and make the tmpfs overlay as big as that partition, so it can swap as necessary.

[broder]

I move this to proposed today.

[broder]

Just kidding.

I moved this back to development, because schroot was prompting about the schroot.conf changes on opus.

I think what happened was that I uploaded schroot to proposed before uploading reactivate, so a handful of workstations grabbed the schroot update (along with its prompt) before getting the reactivate update that un-frobbed schroot.conf.

I'm not sure if uploading the packages in the reverse order would have the desired effect, but even if it does, that's a bit fragile.

Maybe we want to upload a new version of reactivate 1 that moves the athena chroot into /etc/schroot/chroot.d instead of /etc/schroot/schroot.conf, and give that a few days to propogate.

[broder]

Oh right - I moved this back into proposed on Sunday night. Geoffrey assured me that the transition works correctly so long as reactivate 2 is uploaded to the apt repo before schroot

[broder]

Right! I can close tickets.

We moved this to production at 9:59 PM tonight.