Ticket #463 (new enhancement)

Opened 15 years ago

Last modified 12 years ago

Pay attention to the state of union mounts and overlayfs

Reported by: geofft Owned by:
Priority: normal Milestone: The Distant Future
Component: login chroot Keywords:
Cc: Fixed in version:
Upstream bug:  http://valerieaurora.org/union/


The upstream Linux kernel community prefers the approach of union mounts (adding explicit support for unioning at the VFS/mount syscall layer) to union filesystems (a filesystem that behind-the-scenes combines the contents of one or more other filesystems mounted elsewhere), so they've rejected merging things like unionfs and aufs. People are hacking on union mounts, and at some point it will be done. When this happens we probably want to switch to it and we probably want schroot to support it.

Change History

comment:1 Changed 14 years ago by geofft

Call for maintainer: Neither Jan nor I can give this patch set the
long term love that it needs. If you are interested in becoming the
maintainer for union mounts at some point in the future, please drop
me an email.


Any volunteers? :)

comment:2 Changed 14 years ago by geofft

Looks like Valerie Aurora has gotten almost all the work done and a thorough code review from Al Viro. There's a good  overview of the current state at LWN (currently subscriber-only content; like any LWN article it'll be free in a few days), and highlights include  the most recent patchset.

It sounds like she's gotten far enough with it that we can start doing some concrete work -- build a kernel from her tree, play around with union mounts, and get schroot to support them in addition to aufs.

comment:3 Changed 14 years ago by broglek

While the kernel support for union mounts is looking like it is on its way to being mainlined, there is currently only support for ext2, jffs, and tmpfs filesystems.

So for this to be completely integrated:

1) Kernel patches will have to be accepted
2) patches to util-linux-ng (most notably mount and umount) will have to be accepted
3) patches to various filesystems will have to be written/accepted to enable support for whiteouts and fallthrus.

In short: I'm still playing with the patched kernel and utilities, but there is a long way to go before this is practical.

comment:4 Changed 13 years ago by geofft

  • Summary changed from Pay attention to the state of union mounts to Pay attention to the state of union mounts and overlayfs

There's a  refreshed patchset against 2.6.36-rc5, which should be enough to get it to work on Maverick and probably Lucid's kernels, but not Natty or later. (This is the same as currently in  git://git.kernel.org/pub/scm/linux/kernel/git/val/linux-2.6.git ext2_cleanup.) I e-mailed to ask if there was anything newer.

There's also this new thing called  overlayfs, which seems less ambitious (and marginally less "right") -- it hooks open() and does a bit of copying of dentries and inode structures -- but closer to being complete and to being taken upstream. It seems like it's currently usable now: see also the  pull request for v7 and  for v8. Might also be worth a try.

comment:5 Changed 13 years ago by geofft

  • Milestone changed from Upstream Utopia to The Distant Future

Overlayfs has been merged into Ubuntu's kernel in  3.0.0-7.8. We can now give it a try and target overlayfs for the Q cluster release.

comment:6 Changed 13 years ago by geofft

  • Milestone changed from The Distant Future to Animal-that-starts-with-P Alpha

Er, make that P (12.04).

comment:7 Changed 13 years ago by geofft

Some random dude named Evan Broder seems to have  submitted a patch for overlayfs support in schroot, which has landed in Ubuntu's schroot package (and presumably will land in upstream at some point soon).

I'm planning on playing around with a local VM running Precise and seeing how builds go.

comment:8 Changed 12 years ago by geofft

 v2 union mount patches were just posted; doesn't look like they're requesting a merge into mainline though.

Also, regarding overlayfs' usability,  from #ubuntu-devel:

[02:00] <broder> oh what the heck. why does insserv ftbfs, but only in sbuild
[02:00] <broder> ...with fopen(file_the_test_suite_just_wrote_out): Operation not permitted
[02:00] <infinity> broder: In your local sbuild, you mean?
[02:00] <broder> infinity: yeah
[02:00] <infinity> broder: Is that running on top of an overlayfs-using schroot?
[02:00] <infinity> broder: If so, that's your answer.
[02:00] <broder> ...seriously?
[02:01] <broder> ugh
[02:01] <infinity> broder: The good news is that the buildds won't fail that way. :P
[02:01] <infinity> broder: The bad news is you need to test without overlayfs locally.
[02:01] <broder> wait, but why does it not happen if i build it with schroot dpkg-buildpackage, but not using sbuild?
[02:02] <broder> oh, i guess that would be hitting one of the bind mounts
[02:02] <broder> what is overlayfs doing wrong?
[02:02] <infinity> broder: overlayfs plays fast and loose with inodes in ways that can confuse tools that assume that inodes don't (or do) change based on certain actions.
[02:03] <broder> oww
[02:03] <infinity> broder: The tools (or, in this case, test suite) are usually wrong, not overlayfs, but...
[02:04] <broder> ok. since i *can* build it without copying the source into the overlayfs, i'll go ahead and move forward
[02:04] <geofft> infinity: out of curiosity, is there documentation of usual overlayfs issues?
[02:04] <geofft> I have a local buildd that I was just about to switch from LVM chroots to overlayfs one of these weekends...
[02:04] <infinity> geofft: Mostly in apw's head, and scattered around to others who've had to deal with it. :P
[02:05] <infinity> geofft: If you have a working LVM setup, I see no compelling reason to switch.
[02:05] <geofft> infinity: "working" is a bit much, it randomly panics during rebuilds every so often
[02:05] <infinity> Special.
[02:06] <broder> also LVM snapshot-based chroots are borderline unusably slow, especially when there are a lot of them in action
[02:08] <infinity> Anyhow, other than "some software makes stupid assumptions about filesystems that it shouldn't" and "overlayfs doesn't support inotify", I can't think of any other known gotchas.

and  later:

[01:53] <broder> a dpkg trigger wouldn't be an unreasonable way to work around overlayfs+inotify sucking
[01:54] <SpamapS> GrueMaster: no build will work if it depends on an upstart job AFAICT
[01:54] <SpamapS> broder: well in mk-sbuild chroots, policy.d denies starting jobs anyway
[01:54] <broder> SpamapS: right, but there's a livecd issue
[01:54] <SpamapS> policy-rc.d rather.. or whatever its called
[01:55] <SpamapS> broder: I think we should actually be able to fix overlayfs.. very troubling to me that its broken.
[01:55]  * SpamapS wanders off to eat dinner.
[01:56] <broder> good plan :)
[02:09] <infinity> SpamapS: Fixing inotify in overlayfs is a much tougher problem than it looks like at first blush.  apw and I have talked circles around it a few times.

So, we may want to wait out union mounts for actual logins as opposed to just the buildd, sadly.

comment:9 Changed 12 years ago by jdreed

  • Milestone changed from Precise Beta to The Distant Future

I lost track of whether this is for zulu, or for reactivate? I thought for the latter, but then geofft talked about zulu in the IRC logs. If it's for zulu, we've stopped caring. If the latter, will it buy us anything, considering our issues with that are specific to chroots in general, regardless of the backing, AFAICT?

comment:10 Changed 12 years ago by jdreed

I've played with overlayfs chroots on precise, and they seem...not terrible. I added support for them to make-chroot, and I may want to spin up a new zulu this fall running precise, and we can put it through some stress testing (a quantal build, for example) and see if overlayfs is what we want.

comment:11 Changed 12 years ago by jdreed

So, this is done in the context of the build server. Do we want to explore this for cluster?

Note: See TracTickets for help on using tickets.