Ticket #775 (closed enhancement: fixed)

Opened 11 years ago

Last modified 9 years ago

Cluster logouts are slow because of schroot stupidity

Reported by: broder Owned by:
Priority: normal Milestone: Precise Release
Component: login chroot Keywords:
Cc: Fixed in version:
Upstream bug:

Description

/etc/schroot/setup/15killprocs in schroot is responsible for making sure that no processes are still running in a chroot when that chroot is torn down.

It does this by looping over the processes still running in the chroot. For each process, it sends a SIGTERM, then immediately checks to see if the process is still running. If not, it then sleeps for 1 second before trying again.

The problem with this approach is that most programs try to handle SIGTERM gracefully, and therefore do not exit out immediately. When geofft instrumented the logout process, it looked like there were about 15 such processes, causing 15 seconds worth of sleep before the logout would continue.

This is the major cause of logout slowness on cluster machines.

When we used LVM snapshots for the login chroots, we had code to use fuser(1) to first SIGTERM and then eventually SIGKILL all processes running within the chroots, but we removed it from snapshot-run when we switched to aufs chroots.

Bringing that code back and adding it before snapshot-run calls schroot -e would eliminate this issue.

Change History

comment:1 follow-up: ↓ 2 Changed 11 years ago by broder

Filing a bug against schroot to change /etc/schroot/setup/15killprocs to use fuser instead would also be a reasonable course of action.

comment:2 in reply to: ↑ 1 Changed 11 years ago by andersk

Replying to broder:

Filing a bug against schroot to change /etc/schroot/setup/15killprocs to use fuser instead would also be a reasonable course of action.

I wrote that patch in December, but then I realized that fuser kills any process with an open file descriptor in the mountpoint, not just ones running in the chroot—so an attacker could send an fd to his favorite privileged processes, and then arrange for schroot to tell fuser to SIGKILL it before it has a chance to close the fd.

But it would be reasonable to rearrange the loops such that the 5 second delay applies to all processes in parallel instead of each process in series.

comment:3 Changed 10 years ago by jdreed

I briefly looked at this. Rather than put fuser(1) calls in snapshot-run, wouldn't it be easier to simply divert 15killprocs (or even simpler, ship 14killprocs-noreally that does what we want)?

Someone should still file the upstream bug to re-arrange the loops so that it TERMs all the processes, sleeps $n, and then kills them all.

comment:4 Changed 10 years ago by jdreed

  • Milestone changed from The Distant Future to Precise Beta

comment:5 Changed 10 years ago by jdreed

FWIW, here are the processes that don't immediately shut down 1 second after a TERM:

  • zwgc (2 secs)
  • dbus-launch (3 secs)

pulseaudio almost never shuts down cleanly and requires a kill.

Even after a SIGKILL, pulseaudio and dbus-daemon can take up to 2 seconds to actually die, and I'm fairly certain that's what's triggering our reboot code more often than we expect. The new reactivate logging agrees with me.

Refactoring the loops takes our logout time down to 7 seconds, which is noticeably better. We should still try and figure out why pulseaudio is sticking around, though.

comment:6 Changed 9 years ago by jdreed

  • Status changed from new to committed

comment:7 Changed 9 years ago by jdreed

  • Status changed from committed to development

comment:8 Changed 9 years ago by jdreed

  • Status changed from development to closed
  • Resolution set to fixed
Note: See TracTickets for help on using tickets.