ssh changes behavior from Athena 9, does not delegate tickets by default

[jmorzins]

Hi,

When trying out a debathena machine today, I noticed that the
behavior of ssh has changed.

On an Athena 9 computer, if I "ssh" to another kerberized system,
ssh does not ask for my password. It uses my kerberos tickets,
and I connect.

On the machine I tried today, ssh seemed not to use my kerberos
tickets. I had to type a password in order to connect to the
remote host.

For ssh to require passwords is a fairly visible behavior change,
at least to the people who use ssh. If a lot of people use ssh,
I anticipate getting a lot of questions about the change.

(I commented about my observations on zephyr today, and was told
that this ssh change may have been a deliberate choice, but was
asked to file a ticket in trac.)

Regards,

-Jacob Morzinski

[andersk]

There are two things going on here:

• Athena 9 customizes the SSH client to force GSSAPIDelegateCredentials on. This option causes ssh to delegate your Kerberos tickets to any remote machine, rather than just using them to authenticate. Debathena does not, so by default it only uses Kerberos to authenticate, unless you specify ssh -K.

• Athena 9 customizes the SSH server to reject Kerberos authentication attempts that do not include such delegations. Debathena does not, so you can successfully log in with only authentication (but you won’t have tickets or tokens).

Both of these choices were absolutely deliberate.

In order to delegate your Kerberos tickets from a Debathena client to a remote server that you trust (as is necessary for an Athena 9 server to log in at all, and for a Debathena server to have tickets and tokens), you just need to use ssh -K some-server instead of ssh some-server. If you don’t want to type the -K every time, you can configure your ~/.ssh/config to trust a particular server by default:

Host some-server
    GSSAPIDelegateCredentials yes

or even to trust all machines by default, if you really want that:

Host *
    GSSAPIDelegateCredentials yes

Changing these default on a system-wide basis, for any or all classes of Debathena systems, would be a mistake for several reasons.

• Kerberos is increasingly used to authenticate sysadmins to their servers, without necessarily being used remotely once logged in (since the servers have local accounts rather than using Athena accounts). Debathena and XVM make this very easy to set up. The fact that those machines trust your Kerberos principal for authentication should not imply any kind of trust in the reverse direction. An attacker or another administrator on one of your servers should not be able to compromise your Athena account, and with that, compromise every other server you run as well, etc.

• In particular, root instances especially should _never_ be delegated to remote servers, if they are to be considered secure.

• SIPB runs kerberized SSH services, like scripts.mit.edu and the xvm.mit.edu console server, that explicitly request you to _not_ trust them with delegated credentials. Especially on scripts.mit.edu, where unmaintained PHP applications are compromised by script kiddies on a regular basis, it is important for the server’s accounts to be held at a lower privilege level than even an average Athena account.

• Many common actions (e.g. logging into linerva to attach a screen with your Zephyr session) do not require delegation, and for some people it is even useful to be able to perform these actions using alternate authentication mechanisms like SSH public keys.

• The current Debathena behavior is consistent with the Debian, Ubuntu, and upstream SSH defaults (which I expect were chosen carefully for much the same reasons as above).

The Debathena behavior is a change from Athena 9, and definitely needs to be documented. (I thought we had this on the Debathena website already, but I only see documentation on the  Linerva website and a passing reference on the customization page.)

There are, in addition, some code changes we could make to nudge users in the right direction. For example, we could configure dialups to print a warning when you log in to an Athena account with Kerberos without delegating credentials, rather than silently rejecting or silently accepting the login. So far, Debathena and linerva have not received a great number of support requests about this behavior (a few, but not too many, and they’re easy to deal with), so we have not found it necessary.

[jdreed]

Replying to andersk:

• Athena 9 customizes the SSH client to force GSSAPIDelegateCredentials on. This option causes ssh to delegate your Kerberos tickets to any remote machine, rather than just using them to authenticate. Debathena does not, so by default it only uses Kerberos to authenticate, unless you specify ssh -K.

Kerberos Extras does this for you on Mac OS X too. I understand why it could be considered dangerous, but I would like to keep the option of enabling this for -workstation and -cluster on the table if we get enough complaints from users. Should we go that route, users who take their security seriously enough can always have a ~/.ssh/config that overrides the system defaults.

• SIPB runs kerberized SSH services, like scripts.mit.edu and the xvm.mit.edu console server, that explicitly request you to _not_ trust them with delegated credentials. Especially on scripts.mit.edu, where unmaintained PHP applications are compromised by script kiddies on a regular basis, it is important for the server’s accounts to be held at a lower privilege level than even an average Athena account.

Can't these services be configured to reject delegated credentials, rather than using it as a rationale for a widespread user behavior change?

• Many common actions (e.g. logging into linerva to attach a screen with your Zephyr session) do not require delegation, and for some people it is even useful to be able to perform these actions using alternate authentication mechanisms like SSH public keys.

I don't thing these are common actions across the entire user Athena base, rather I think they're common for Linerva users, which is to some degree a self-selecting group. Additionally, if your homedir is not system:anyuser list, you end up in the root directory, which seems like a terrible idea. Refusing conections that don't delegrate credentials was a deliberate choice, because it's extremely confusing, and results in a flurry of calls from users saying "Help, it says permission denied when I try to access my files."

• The current Debathena behavior is consistent with the Debian, Ubuntu, and upstream SSH defaults (which I expect were chosen carefully for much the same reasons as above).

It's also consistent with the default OS X behavior. However, as noted above, the decision was made to change that. We should be aware of what factors went into that decision and why rather than simply dismissing it as a poor security choice. We should probably solicit input from krbdev/macdev/whoever.

There are, in addition, some code changes we could make to nudge users in the right direction. For example, we could configure dialups to print a warning when you log in to an Athena account with Kerberos without delegating credentials, rather than silently rejecting or silently accepting the login.

If we decide to stick with the current configuration, I think we definitely need to do this, ideally for debathena-login and higher. The warning should also, as succinctly as possible, inform users of the consequences of this (ie: no dotfiles, no homedir access, no running locker software, no zephyr, etc). Perhaps the right thing to do is to say "Warning: you have no Kerberos tickets or AFS tokens. For more information, see  http://whatever"

We should also talk to ops about the possibility of enabling this on the existing dialups. Given that the dialups will be Athena 9 for quite a while, if enabling this is not possible, I'd really like to configure -cluster and -workstation to at least delegate credentials to the Athena dialups. If those are compromised, we're pretty doomed anyway, so I think it's reasonable to trust them.

[andersk]

Replying to jdreed:

Kerberos Extras does this for you on Mac OS X too.

Really? That seems extremely poor, especially since I don’t see any documentation for that change, at least on their website. I agree we should get macdev to comment on this.

Should we go that route, users who take their security seriously enough can always have a ~/.ssh/config that overrides the system defaults.

Users shouldn’t have to change how they do things unrelated to Debathena in order to remain secure after installing Debathena.

Can't these services be configured to reject delegated credentials, rather than using it as a rationale for a widespread user behavior change?

Not without sshd code changes. Furthermore, that doesn’t solve the problem, because even if the service was normally configured to reject delegation, an attacker who compromises the service could change that configuration. A secure distributed computing infrastructure cannot be built upon hosts with a “gentleman’s agreement” not to attack each other.

If we decide to stick with the current configuration, I think we definitely need to do this, ideally for debathena-login and higher. The warning should also, as succinctly as possible, inform users of the consequences of this (ie: no dotfiles, no homedir access, no running locker software, no zephyr, etc). Perhaps the right thing to do is to say "Warning: you have no Kerberos tickets or AFS tokens. For more information, see  http://whatever"

That seems reasonable. Obviously, there are some details we’d want to think about: the message should go to stderr, only for nonlocal accounts, possibly only for interactive logins, and possibly only if your {.bashrc,.cshrc} isn’t otherwise readable without tokens (since otherwise you probably took some explicit action to make this work); such restrictions don’t seem like they would diminish the message’s usefulness to its target audience.

Given that the dialups will be Athena 9 for quite a while, if enabling this is not possible, I'd really like to configure -cluster and -workstation to at least delegate credentials to the Athena dialups. If those are compromised, we're pretty doomed anyway, so I think it's reasonable to trust them.

For -cluster and -workstation, I would be open to considering this, but I worry that the inconsistency (between -standard and -cluster, as well as between athena.dialup and other hosts) may be just as confusing as the difference from Athena 9 in the first place.

[jdreed]

Replying to andersk:

Kerberos Extras does this for you on Mac OS X too.

Really? That seems extremely poor, especially since I don’t see any documentation for that change, at least on their
website. I agree we should get macdev to comment on this.

 http://web.mit.edu/macdev/KfM/Common/Documentation/osx-kerberos-extras.html says "In addition the Mac OS X 10.4 (Tiger) and 10.5 (Leopard) Kerberos Extras will also modify your ssh configuration to allow the use of Kerberos by default. Apple disabled Kerberized ssh in 10.4.9 (Tiger) by default to solve performance problems on networks without DNS."

and I have /etc/ssh_config.backup.1 correspdoning the date I installed Kerberos Extras (based on the package receipt), and /etc/ssh_config says:

# System-wide defaults set by MIT Kerberos Extras
Host *
  GSSAPIAuthentication yes
  GSSAPIDelegateCredentials yes
  GSSAPIKeyExchange yes

Given that the dialups will be Athena 9 for quite a while, if enabling this is not possible, I'd really like to configure -cluster
and -workstation to at least delegate credentials to the Athena dialups. If those are compromised, we're pretty doomed
anyway, so I think it's reasonable to trust them.

For -cluster and -workstation, I would be open to considering this, but I worry that the inconsistency (between -standard
and -cluster, as well as between athena.dialup and other hosts) may be just as confusing as the difference from Athena 9

in the first place.

Given that it will have a fixed time period, we could bill it as a transition feature, so I'm not too concerned about the consistency. Once we have a date for the transition of athena.dialup to Linux, we can explicitly say we will desupport this feature on that date, and users will then have to use ssh -K regardless.

[jdreed]

Mail sent to macdev soliciting feedback on their decision to enable GSSAPIDelegateCredentials.

[fawkes]

As noted on -c debathena, the description of debathena-ssh-client-config states that:

This package configures the SSH client to automatically forward
Kerberos credentials.

but it actually only changes GSSAPIAuthentication to on and does not change the GSSAPIDelegatCredentials setting.

Either the package should be modified to work as advertised (which would resolve this ticket) or the description should be modified to be accurate.

[tlyu]

I asked around a few places. Between Tiger and Leopard, SWRT took ownership of the Kerberos Extras from Macdev. Around the Leopard release, Apple disabled GSS authentication by default because it was causing long timeouts in non-Kerberos environments. SWRT changed the Kerberos Extras to reinstate the Tiger behavior of the ssh client, which did GSS authentication as well as credential delegation.

Also, my recollection is that an earlier release of Athena 9, possibly 9.3, sshd would refuse GSS authentication if credentials were not delegated, and this may have been a motivation for turning on credential delegation. This sshd behavior may have changed in 9.4.

[kaduk]

I believe that this behavior of disallowing GSS authentication without credential delegation is also present in Athena 9.4, having checked on multics.mit.ede, portnoy.mit.edu, and athena.dialup.mit.edu.

[kcarnold]

How about split the difference:

Host linerva.mit.edu linerva linux.mit.edu linux
    GSSAPIDelegateCredentials yes

(This subtly encourages people to use linerva... not sure if that's a bug or a feature.)

[jdreed]

I think trusting linerva is reasonable, as is athena.dialup.mit.edu, at least for the cluster machines, and probably -workstation. Certainly we shouldn't be making any trust decisions for -login-graphical and lower.

[broder]

Marking specific hosts as trusted makes me uncomfortable, although I can't exactly place why. I think part of it is that I don't like the inconsistently of certain hosts getting tickets and others not - it seems to me like it would be more confusing.

[xavid]

It really seems like the official dialups should work out of the box.

[kaduk]

I find myself agreeing with both Evan and Xavid.
I could probably live with {athena,x}.dialup and linerva special-cased, though; that
certainly seems better to me than delegating to all of *.mit.edu would, as aesthetically pleasing as that might be.

[andersk]

I strongly agree with Evan, as I have already noted at length. But I will reiterate that we are not preventing the dialups from “working out of the box”; they work just fine if you use ssh -K or configure your .ssh/config. Taking an explicit action to trust a remote server (either per-command or one-time) just isn’t too much to ask of our users. So as I see it, the only reason to consider adding implicit trusts by default is if improved documentation and warning messages will not be sufficient to avert an unacceptable support load generated by users who assume the insecure Athena 9 behavior. But I think adding a warning message as suggested by jdreed will be more than sufficient.

[jdreed]

Well, the warning message was suggested for Debathena. Implementing a warning message for Athena 9 machines is going to be difficult at best, because it would involve a patch release, and also a manual process for the dialups.

On the other hand, for the fall semester, the dialups are going to be very important, as they will be the only public Sun machines available. Once the dialups are running Debathena, I'm less concerned about people needing to take explicit action to connect to Athena 9 machines, because Athena 9 machines will be crufty and desupported at that point.

There are really two issues here:

a) It is now possible to log in to "Athena machines" (Debathena) without tickets and thus end up with no home directory and no tokens.
b) It is not possible to log in to the dialups without taking explicit action to change the login procedure (ie: adding -K).

I think (a) will be the far more confusing of the two, however we have a solution to that -- some sort of warning message to users saying "You are currently logged in without tickets and tokens. To log in correctly, please disconnect now and use "ssh -K" instead. For more information, see  http://debathena/blah". I'd like to see that warning message be a blocker for the Fall Release.

For (b), I'd like to suggest the following compromise: For -workstation and higher, we configure ssh to delegate tickets. We also include an ssh wrapper that displays one line at the top that says something like "Now delegating tickets, please see  http://whatever". That stays in place until the dialups are transitioned (IAP, hopefully). At that point, we configure ssh to no longer delegate credentials, and people will then have to take explicit action to trust remote servers. I think that provides a good balance between security and not drastically changing behavior out from under people with no warning.

I also plan to provide some sort of script in the consult locker (which I'm tempted to name "fixssh", but won't) that users can run which will trust *.mit.edu (after explaining the implications of doing so).

[jdreed]

After discussing this at release team, I think we only need to move forward on "(a)".

Can we update the dotfiles on -login and higher to check for the existence of tickets and tokens and warn the user. Something like:

Warning: You have no Kerberos tickets or AFS tokens. Please see http://debathena.mit.edu/ssh

I will ensure that URL works shortly. Can we get this in place by the 24th?

[jdreed]

http://debathena.mit.edu/ssh now works. We should add the message somewhere, probably in /usr/share/debathena{bash,tcsh}-config/{bash,csh}rc.debathena

Something like this (for tcsh) should suffice:

klist -s
if ($? != 0) then

echo "You do not have valid Kerberos tickets. See http://debathena.mit.edu/ssh"

endif
tokens | grep -q athena\.mit\.edu
if ($? != 0) then

echo "You do not have valid AFS tokens. See http://debathena.mit.edu/ssh"

endif

We could check for "mit.edu" in tokens' output instead, but users with homedirs outside the athena cell are expected to know how to deal.

[andersk]

Replying to jdreed:

Something like this (for tcsh) should suffice:

Keep in mind my comment earlier:

“Obviously, there are some details we’d want to think about: the message should go to stderr, only for nonlocal accounts, possibly only for interactive logins, and possibly only if your {.bashrc,.cshrc} isn’t otherwise readable without tokens (since otherwise you probably took some explicit action to make this work); such restrictions don’t seem like they would diminish the message’s usefulness to its target audience.”

[broder]

Ok - change is now in proposed, and should be Anders-compliant.

[broder]

And now in production.