[14.04] ssh server on thin client broken by default

On Thu, May 29, 2014 at 01:48:12PM -0000, Jakob Unterwurzacher wrote:
> The line
> etc/ssh/ssh_host_*_key
> in
> /etc/ltsp/ltsp-update-image.excludes
> excludes the ssh host key from the nbd image. This means that an installed ssh server is broken by default, and you get lines like
> sshd[2378]: error: Could not load host key: /etc/ssh/ssh_host_rsa_key
> in the syslog.

True. By default, the clients do not have a functional ssh server. The keys
could be regenerated at boot (which means there would be no way to verify them)
or copied from some other location at boot (which means they would not be
secure).

> The reason for excluding it probably is that the host key is secret, and
> everything that goes in the nbd image is essentially public.
>
> However, exposing the host key does NOT compromise ssh sessions ( http://security.stackexchange.com/questions/54852/server-side-ssh-keys-compromised ).

After reading the article referenced, this seems like a misinformed conclusion...
There are many ways those keys could be used to compromise both the server and
the clients in question. The most obvious is MITM attacks, but you could also
gain significant data by capturing all network traffic and processing the
conversations later...

> Exploiting the exposed key would need a full man-in-the-middle attack. And if the attacker can to that, he can also set up a rogue DHCP server and own everything.

A major purpose of public key crypto in ssh is to detect and mitigate the
impact of man in the middle attacks, so this seems a bit backwards
thinking...

While DHCP server is a weak point in the security of an LTSP installation,
we don't have to throw out all attempts at securing the sessions because
there's a weak point in the chain. Not to say we shouldn't be aware of the
weaknesses in our security, but it's not a good excuse to further weaken the
security.

> What IS a problem, however, is password authentication via ssh. As the
> /etc/shadow is public in the nbd image, it can be cracked and then used
> to connect to running thin clients.
>
> Instead of excluding the ssh host keys, I suggest to append
> "PasswordAuthentication no" to the thin client's /etc/ssh/sshd_config .

I think storing the shadow file in the image by default is a security issue,
unrelated to weather or not sshd on the client allows password-authenticated
connections... That seems like a policy decision for the network
administrator on a per-site basis, not the LTSP developers.

live well,
  vagrant

On 29.05.2014 18:57, Vagrant Cascadian wrote:
>> However, exposing the host key does NOT compromise ssh sessions (
>> http://security.stackexchange.com/questions/54852/server-side-ssh-keys-compromised
>> ).
>
> After reading the article referenced, this seems like a misinformed
> conclusion... There are many ways those keys could be used to
> compromise both the server and the clients in question. The most
> obvious is MITM attacks, but you could also gain significant data by
> capturing all network traffic and processing the conversations
> later...

Hmm, the post is quite clear on that:

"Due to the use of the Diffie-Hellman key exchange, knowing the server's
private key does not help a passive-only attacker. If the attacker wants
to learn the data, then he must go active."

So no, recording is not going to do anything.

> That seems like a policy decision for the network administrator on a
> per-site basis, not the LTSP developers.

Well, yes. But wasn't

# apt-get install openssh-server

that administrator decision?

I mean, it's nice to prevent people from shooting themselves in the
foot, however, as it is right now this is what happens:

1) Install ssh server
2) Find out it does not work
3) Enable root access via password (!) to debug the issue

It would be nice to get the error earlier, during ltsp-update-image for
example. That would skip step number 3.

On Thu, May 29, 2014 at 06:46:07PM -0000, Jakob Unterwurzacher wrote:
> On 29.05.2014 18:57, Vagrant Cascadian wrote:
> >> However, exposing the host key does NOT compromise ssh sessions (
> >> http://security.stackexchange.com/questions/54852/server-side-ssh-keys-compromised
> >> ).
> >
> > After reading the article referenced, this seems like a misinformed
> > conclusion... There are many ways those keys could be used to
> > compromise both the server and the clients in question. The most
> > obvious is MITM attacks, but you could also gain significant data by
> > capturing all network traffic and processing the conversations
> > later...
>
> Hmm, the post is quite clear on that:
>
> "Due to the use of the Diffie-Hellman key exchange, knowing the server's
> private key does not help a passive-only attacker. If the attacker wants
> to learn the data, then he must go active."
>
> So no, recording is not going to do anything.

Perhaps, but I cannot see what you gain by making a server's private keys
publicly available... that is a bad idea. MITM attacks are a big enough reason
alone and it is far too easy to miss some other security issue by publishing
private keys.

> > That seems like a policy decision for the network administrator on a
> > per-site basis, not the LTSP developers.
>
> Well, yes. But wasn't
>
> # apt-get install openssh-server
>
> that administrator decision?

I was referring to the decision to disable or enable password authentication in
SSH; that decision should be left to the administrator, entirely unrelated to
LTSP. In a default LTSP environment, there are no valid passwords on the
clients, for what it's worth.

I think the excludes in ltsp-update-image.excludes are there so when used with
with ltsp-pnp mode (where the server's root filesystem is used to create the
NBD image), you're not compromising the server's private keys.

I can see that using ltsp-update-image when building the NBD image from
a chroot it would make sense to copy the ssh keys. This still means a shared
host key between all the clients, which allows MITM of any client. Knowing the
risks, an administrator may add or remove anything they want from
ltsp-update-image.excludes, which is a configuration file.

ltsp-update-image could also be changed to have a more comprehensive list of
excludes when used with --cleanup (typically used with ltsp-pnp mode), which I
could see the case for...

Another workaround would be to use NFS, which doesn't exclude those files.

It would be trivial to add an init-ltsp.d hook that generates or copies ssh
keys at boot or at image build time to be able to run sshd on the thin clients.
It could store the client's ssh keys in some alternate location (either in the
server's or client chroot's /etc/ltsp). Patches along these lines might make
sense for inclusion in LTSP. Alternately, some structure to copy arbitrary
files into place at boot might be another way to accomplish what you were
looking for.

So I can see several options worth exploring to workaround or even fix this
issue...

live well,
  vagrant

Ahhh, ltsp-pnp! So that's the reason they are excluded by default - and it makes a lot of sense there. I was only thinking about the classic setup, with a separate chroot for the thin clients.

I'll try to come up with a patch so ltsp-update-image at least emits a warning in that case.

Thanks Vagrant!

ltsp-pnp uses a cleanup phase so it's much more flexible than ltsp-update-image.excludes.
For example, /usr/share/ltsp/cleanup.d/51-sshd regenerates the server keys so that they're unique for the generated squashfs image. That's still not good enough though, something better is needed for ssh keys.
/usr/share/ltsp/51-remove-users removes user entries from passwd and shadow, and so on.

So while ltsp-pnp is currently using that ltsp-update-image.excludes line, it can very easily do the "rm" in the cleanup phase over its cow mount. I.e. feel free to remove that line it if you want.

The cleanup phase is not specific to ltsp-pnp; it can also be used in Virtual Machines, e.g. ltsp-update-image --cleanup /path/to/vbox/image.vdi (the code to loop-mount the .vdi has not yet been committed upstream though), and also in chroots, e.g. ltsp-update-image --cleanup (no path, /opt/ltsp/i386 is assumed).

But, if the ssh line is removed from ltsp-update-image.excludes, I think that with the same reasoning, the epoptes certificate, the mysql password, the network manager system connections etc should be kept as well, i.e. ltsp-update-image.excludes then should only have the lines about temporary directories, not the lines about configuration files.

Wow, that infrastructure is really nice. You even deal with the "user passwords in the shadow file" problem! Well, almost. You miss the root password as far as I can see? But just adding "passwd --delete root" should fix that.

Regarding the other config files: Having them in ltsp-update-image.excludes means you don't have to handle it in cleanup.d. If there is no clear use case for having it on the thin client, it's probably ok to just keep it excluded.

I think an ssh server has a clear use case for debugging, so I'd remove the lines:

    etc/ssh/ssh_host_*_key (sshd does not work without a host key)
    root/.* (public key authentification does not work without .ssh/authorized_keys)

but add

    root/.ssh/id* (no clear use case for having personal private keys on the thin client)

> You miss the root password as far as I can see?

On purpose, many people set a root password for local troubleshooting on vt1.

> If there is no clear use case for having it on the thin client, it's probably ok to just keep it excluded.

It's the same as in sshd; if those files are there in a chroot and --cleanup isn't used, that means that the sysadmin probably wants them *included*. For example, if the epoptes certificate is there, the sysadmin probably wants to run epoptes from a fat client.

> remove root/.* (public key authentification does not work without .ssh/authorized_keys)

I think by default we should omit the root's private configuration files, shouldn't we? To protect people using `ltsp-chroot`... It's not just about ssh but about all apps that save sensitive data...
Let me also throw the idea of introducing "SSH_AUTHORIZED_KEY_nn" lts.conf variables, and having a /usr/share/ltsp/init-ltsp.d/50-ssh-authorized-keys script put them in /root/.ssh/authorized_keys. That way you can even give to different people, root access to different clients.

Which of the following options sounds saner to you guys?
1) Use a different ltsp-update-image--cleanup.excludes file when --cleanup is used.
2) Use an additional ltsp-update-image--cleanup.excludes file when --cleanup is used.
3) [proposed] Remove from ltsp-update-image.excludes the lines we've talked about, and re-implement them as --cleanup scripts.

For ltsp-pnp, doesn't that mean that you are exposing the root password of the ltsp *server* (as it is identical)?

Yeah, excluding /root/.* while including /root/.ssh/authorized key would be the right thing to do, but mksquashfs syntax can't do it. Having SSH_AUTHORIZED_KEY_nn instead sounds great, as does option (3).

> For ltsp-pnp, doesn't that mean that you are exposing the root password of the ltsp *server* (as it is identical)?

It's unset by default on Ubuntu. True, when other distributions start using ltsp-pnp, such things should be fixed.

As a workaround, use Epoptes to get shell from the client.

The way openSUSE does it is like this, missing keys are created on the fly when the sshd is launched so ltsp image does not need those keys beforehand:

cat /usr/lib/systemd/system/sshd.service
[Unit]
Description=OpenSSH Daemon
After=network.target

[Service]
EnvironmentFile=-/etc/sysconfig/ssh
ExecStartPre=/usr/sbin/sshd-gen-keys-start
ExecStart=/usr/sbin/sshd -D $SSHD_OPTS
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=always

[Install]
WantedBy=multi-user.target

cat /usr/sbin/sshd-gen-keys-start
#!/bin/sh
if ! grep -q '^[[:space:]]*HostKey[[:space:]]' /etc/ssh/sshd_config; then
    echo "Checking for missing server keys in /etc/ssh"
    ssh-keygen -A
fi

On Ubuntu:
grep '^[[:space:]]*HostKey[[:space:]]' /etc/ssh/sshd_config
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
root@server:/mss# echo $?
0

So ExecStartPre=ssh-keygen -A should work

on openSUSE:
prime:~ # grep '^[[:space:]]*HostKey[[:space:]]' /etc/ssh/sshd_config
prime:~ # echo $?
1

prime:~ # grep HostKey /etc/ssh/sshd_config
# HostKey for protocol version 1
#HostKey /etc/ssh/ssh_host_key
# HostKeys for protocol version 2
#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_dsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key

Wow, getting this right has become even more difficult in Ubuntu 16.04.

You need to
1) Remove the "etc/ssh/ssh_host_*_key" line from /etc/ltsp/ltsp-update-image.excludes
2) Remove the "root/.*" line from the same file, otherwise your authorized_keys will get excluded
3) Add "KEEP_SYSTEM_SERVICES=ssh" to lts.conf

Since cleanup.d already contains code to remove the ssh_host_*_key, i think we can safely remove it from ltsp-update-image.excludes. It also regenerates keys, and Alkis and I have been debating/discussing the merits of regenerating a new key with each ltsp-update-image. One the one hand, having keys present makes it possible to log in, on the other hand, having them change every time could introduce alarming security warnings and teach people to ignore them even more than they already do...

I'm less thrilled about shipping /root files in the image, though, as root could have other files in their homedir that shouldn't be exported. Exposing which keys are able to log in to the server is also a bit undesireable. You could also adjust sshd_config using the Authorized_keys directive to allow alternate locations for the keys. Or remove the line from ltsp-update-image.excludes

Requiring KEEP_SYSTEM_SERVICES=ssh works as intended, so plan to keep it that way.

We might implement some sort of overlay dir for files to add to the image, and then you could explicitly add the ssh_host_*_key* and authorized_keys to that.

live well,
  vagrant

Fix committed in http://bazaar.launchpad.net/~ltsp-upstream/ltsp/ltsp-trunk/revision/2768.

Fix released in LTSP 5.5.9.

I'm trying to enable openssh-server on an ltsp client with no luck. I've tried all of the settings here, exclude file, everything.

Wasted several days on this now. Is this broken again?

Thanks,
Jon Herr

Hi, I've build new LTSP image today in ubuntu 16.04, and SSH is not working (is working in an old image).

Can someone gime a tip of how to make it work? (as well as any services I want to keep, since KEEP_SYSTEM_SERVICES is not working for me...).

Thanks.