Ticket #1156 (closed enhancement: fixed)

Opened 12 years ago

Last modified 11 years ago

Entirely new thirdparty infrastructure

Reported by: jdreed Owned by: bbaren
Priority: high Milestone: Current Semester
Component: -- Keywords:
Cc: Fixed in version:
Upstream bug:


We spend way too much time and effort on updating thirdparty, and have to play stupid games to work around broken dependency resolvers, and the whole thing is just too fragile. I propose a new design for thirdparty that makes everyone's lives easier (where "everyone" is me and alexp), and fixes #372:

Basically, we supply a list of packages (possibly separated into dependencies and recommendations), but I think we can do away with those concepts. This list can be a global list, or it can be per-distro. debathena-thirdparty is a package that knows where to find these lists, copies them locally, and iterates over the packages, apt-get installing each one (presumably not in its postinst, but rather as a cron job, or a trigger at the end of auto-update, or something). If a package fails, it retries. After 3 consecutive failures, it whines, either via athinfo (which we can add a new Nagios check for, or use update-status, or something) or syslog, or whatever. When someone uninstalls thirdparty, we can either remove those packages (again, failing if there are dependency issues), or perhaps mark them as auto-installed or something?


  • Changing the membership of thirdparty is as simple as editing a plain text file, no rebuilds are necessary.
  • We don't have to play stupid dependency games
  • Removal of metapackages is far less likely.
  • Fixes #372
  • Installs possibly go faster as we can relegate this to firstboot or something?


  • We're reverting to what are essentially control files. (Counterpoint: They worked.)
  • There's no longer a hard guarantee that all of thirdparty must be installed for a machine to call itself cluster.

I'm fully aware that using APT is the Right(tm) way to do this. But it causes more trouble than it's worth, IMHO.

Change History

comment:1 Changed 12 years ago by jdreed

  • Status changed from new to accepted
  • Owner set to jdreed

OK, initial proof-of-concept committed in r25654. I'd appreciate feedback. Nothing pulls this in yet, and I'm not even close to building it. Nor does it correctly Conflict/Break/Whatever? the old thirdparty stuff.

comment:2 Changed 12 years ago by bbaren

  • Status changed from accepted to assigned
  • Owner changed from jdreed to bbaren

comment:3 Changed 12 years ago by bbaren

Architecturally, this sounds pretty reasonable to me.

I've looked at the implementation, and it looks reasonable too. The shell script looks sane, and while I don't speak Perl all that well, I don't see any issues there either.

comment:4 Changed 11 years ago by jdreed

  • Status changed from assigned to committed

OK, I rewrote this to be simpler in r25843. The lists aren't pulled from AFS dynamically, they're generated at build time. So a change to thirdparty is still a rebuild of this package, to ensure it goes through our workflow. But this is much simpler than the idiom we had for Depend-If-Exists. Feedback would be appreciated.

comment:5 Changed 11 years ago by jdreed

I'm back at looking at this again, but running up against a wall that the chroots do not have multiverse enabled, so packages in multiverse get removed from the list as they have no installation candidate. I could whitelist certain packages as "depend but don't verify", but that kind of defeats the purpose of the package. geofft suggests a dummy apt-config and lists, but we'll see how well libapt-pkg-perl plays with that.

comment:6 Changed 11 years ago by jdreed

comment:7 Changed 11 years ago by jdreed

One thing that comes to mind in testing is that there's no way to remove packages we no longer want from thirdparty, because everything will be marked as manually installed. We could play stupid games with apt-mark, or attempt to generate diffs in the post/preinst, but I'm not terribly excited about that. We could also decide that machines get reinstalled annually, and supply a list of specific packages to remove if we actually need to _remove_ something (vs simply no longer needing to have it installed). I think I favor that solution. Unless there's a whole better way I'm not seeing. (Remember, part of the goal here is solve #372)

comment:8 Changed 11 years ago by jdreed

It's possible the right answer here is to simply make -thirdparty not a dependency of -cluster. I think I still favor a monolithic thirdparty, possibly with the same features as this that it generates a list at build time, but *shrug*

comment:9 Changed 11 years ago by jdreed

  • Status changed from committed to review

OK, this has been tested, and I think is ready to go to -development, but I'd like some review.


comment:10 Changed 11 years ago by jdreed

  • Status changed from review to committed

comment:11 Changed 11 years ago by jdreed

Do we want to demote thirdparty to a recommendation of cluster? That should, in theory, allow it to be uninstalled in the chroots, but lets us get away with not having to change the installer, and lets us not have to use an update-hook to apt-mark thirdparty as auto-installed.

comment:12 Changed 11 years ago by jdreed

  • Status changed from committed to proposed

comment:13 Changed 11 years ago by jdreed

  • Status changed from proposed to closed
  • Resolution set to fixed
Note: See TracTickets for help on using tickets.