We need a better plan for manual-config

[broder]

Our current system for handling the manual-config packages is broken in a lot of ways.

First of all, because the manual-config packages are built by hand when someone remembers and/or cares enough to do so, they're always horribly out of date.

Second of all, the manual-config packages don't even always work. Because the packages are named debathena-manual-foo-config and only provide debathena-foo-config, they can't satisfy any versioned dependency.

Finally, very, very few people are using the manual-config packages. I'm sure somebody will point out the lack of documentation, but I don't think we've gotten any support load from people whose problems would be solved by the manual-config packages.

I see a couple of plausible options for manual-config going forward:

• Punt the manual-config packages.
• If we're not going to punt the packages, then hook them into some automated build and upload infrastructure, like the autodebathenificator

Given that config-package-dev already comes with a mechanism for the local sysadmin to undo the packages' configuration without installing them, I'm personally in favor of punting the manual-config packages, and maybe coming up with tools to, say, re-symlink all files diverted by a particular package or something.

[broder]

Some data to back up the points:

Replying to broder:

First of all, because the manual-config packages are built by hand when someone remembers and/or cares enough to do so, they're always horribly out of date.

For instance, right now, out of 38 config packages, 10 of their manual-config counterparts are out of date. And 5 more of the packages just don't have a manual-config counterpart.

# Information on version skew gathered with:
grep-dctrl '' -n -s Package,Version /mit/debathena/apt/dists/hardy/debathena-config/binary-amd64/Packages  | fmt | sed -ne '/./p' | while read package version; do
  grep -q -e '-config$' || continue
  version=${version%~*}
  printf "%s: %s " $package $version
  manual_package=${package/debathena-/debathena-manual-}
  grep-dctrl -P $manual_package -n -s Version /mit/debathena/apt/dists/hardy/debathena-manual-config/binary-amd64/Packages
done

Second of all, the manual-config packages don't even always work. Because the packages are named debathena-manual-foo-config and only provide debathena-foo-config, they can't satisfy any versioned dependency.

Currently 5 of our non-manual-config binary packages have such a dependency on a -config package, eliminating 6 of the manual-config packages from being usable alongside those other packages.

# Information on versioned dependencies gathered with
grep-dctrl -FDepends --eregex 'debathena-[^ ]*-config \(' -s Package,Depends /mit/debathena/apt/dists/hardy/{debathena,debathena-config,debathena-system}/binary-amd64/Packages

Finally, very, very few people are using the manual-config packages. I'm sure somebody will point out the lack of documentation, but I don't think we've gotten any support load from people whose problems would be solved by the manual-config packages.

Looking at our install logs, I see 57 downloads of any manual-config packages (which includes upgrades). Those downloads come from 12 distinct IP addresses and are for 14 distinct packages.

# Information on users of manual-config packages gathered with
grep -h manual-config /mit/debathena/web_scripts/log/log{,-old*} | grep -v '/apt/dists/'

[jdreed]

Could we provide manual-config packages on-demand? I mean, if only 14 packages are seeing use, then perhaps those are the important ones. We could perhaps update the documentation to encourage people to request new ones if they need them. We could also document which ones have a -config dependency (and why).

[jdreed]

Given that config-package-dev already comes with a mechanism for the local sysadmin to undo the packages' configuration without installing them, I'm personally in favor of punting the manual-config packages, and maybe coming up with tools to, say, re-symlink all files diverted by a particular package or something.

Can we just document this method, and move on? Is there anything that manual-config provides that this method would not?

[andersk]

config-package-dev comes with a mechanism to override any diversion symlink with a symlink to the original file. This doesn’t help with any other forms of configuration, such as files edited in place, files dropped in /etc/foo.d directories, files diverted away without being replaced, or debconf values modified, all of which are things that we do. It doesn’t help avoid the existence of the symlink and the extra .debathena* files, which matters in some cases. It doesn’t help when a config package is removed and reinstalled. And it doesn’t help when a config package is upgraded to include new diversions.

As far as I can see, the manual-config status quo works well enough for everyone that needs it. Why don’t we just document _that_ and move on? (Oh wait, because it’s already documented.)

If it breaks because a package needs to be added or upgraded, I’m happy to take two minutes to go run the script that automatically does that.

[jdreed]

Having never used manual-config, I have no data. Is the following still an issue?

Because the packages are named debathena-manual-foo-config and only provide debathena-foo-config, they can't satisfy any versioned dependency.

I think if we're going to continue with manual-config, some sort of automated build is the right thing to do.

[andersk]

Replying to jdreed:

Having never used manual-config, I have no data. Is the following still an issue?

Because the packages are named debathena-manual-foo-config and only provide debathena-foo-config, they can't satisfy any versioned dependency.

I use several manual-config packages and I’ve never run into that problem, because the only versioned dependencies we have on config packages are from packages that aren’t in debathena-standard.

I think if we're going to continue with manual-config, some sort of automated build is the right thing to do.

Sure, we could throw make-manual-config into a cron job.

[jdreed]

"I bet I know why some things are missing from manual-config":
zcat /mit/debathena/apt/dists/etch/debathena-config/binary-i386/Packages.gz

BITD, we talked about making the manual-config packages build as part of their -config cousins. That is, foo-config also builds foo-manual-config. This ensures that they're kept up to date. This also gives us a better idea over which package has a manual-config component. Certainly it's easier to s/etch/precise/, but I slightly favor simplifying the build infrastructure. We should ensure that sbuildhack/daupload correctly deal with the manual-config component, though.

[jdreed]

So, I looked harder at the idea of having the config packages generate the manual-config packages:

1) The stupid/simple way is to just do it, but then you run the risk of dependency skew between config and manual-config (because someone typo'd). A comment in the file saying "Change this down there too" might help.
2) A slightly less stupid way is to list the files by hand and then have the dependencies copied at build time with something like: echo "debathena-lightdm-config-depends=$(shell dpkg-awk --rec_sep '#' -f debian/control -- 'Depends' | tr -s '[[:space:]]' ' ' | cut -d '#' -f 2 | cut -d ':' -f 2)" >> debian/debathena-manual-lightdm-config.substvars but that's klunky and would have to be copied and pasted into each rules file.
3) We write dh_manualconfig and a corresponding manual-config-packge.mk, but then packages can't be built without that. (As compared to packages using c-p-d which is upstream. OTOH, config-package-dev is ancient in lucid, so you arguably can't build the current source tree on older releases.

Thoughts? Unless there are serious objections, I'm leaning towards #3, primarily because it will force me to learn more about debhelper.

[geofft]

Without commenting on the relative merits of the proposals yet (though I think I lean towards the third one),

2) A slightly less stupid way is to list the files by hand and then have the dependencies copied at build time with something like: echo "debathena-lightdm-config-depends=$(shell dpkg-awk --rec_sep '#' -f debian/control -- 'Depends' | tr -s '[[:space:]]' ' ' | cut -d '#' -f 2 | cut -d ':' -f 2)" >> debian/debathena-manual-lightdm-config.substvars but that's klunky and would have to be copied and pasted into each rules file.

 A blog post from a dpkg developer suggests using the override_dh_gencontrol target to pass the -V option, for generating substvars in the rules file. This strikes me as less fragile than hardcoding the .substvars file and appending to it. That doesn't really address any of the other issues you bring up with that solution, though.

[jdreed]

I looked at pkg-create-dbgsym (as suggested by geofft) and it's really ugly. I don't love the ideal of building a source package not listed in the source control file. I think the substvars solution is the way to go?

[jdreed]

Thinking even harder about this, I think I now actually prefer option 1: "Just do it" and maybe leave a comment in the control file. In addition to making our lives easier, the dependencies are not necessarily the same between regular and manual-config. Let's say some wrapper script is written in Perl (ignore the fact that perl-base is an essential package). The config package would depend on Perl, the manual-config package need not.

[jdreed]

Of course this is harder than it seems, because you're not supposed to have a Source package generate binary packages in different components, so lintian and reprepro complain. We'd also need a lintian override because the manual-config packages should not have ${misc:Depends}. We can either ignore the warnings, or I guess give up and fix make-manual-config to not use Etch as its canonical repo and shove it in a cron job?