ssh host key fingerprint no longer available in the console log

Going through the /var/log/syslog file, the fingerprint is printed there. However there is a line related to rsyslog just after the line printed in the console:

rsyslogd-2039: Could no open output file '/dev/xconsole' [try http://www.rsyslog.com/e/2039 ]

Upon more investigation, it seems that the default rsyslog configuration is incorrect for ec2:

/etc/rsyslog.d/50-default.conf:

daemon.*;mail.*;\
        news.err;\
        *.=debug;*.=info;\
 *.=notice;*.=warn |/dev/xconsole

Replacing /dev/xconsole with /dev/xvc0 will fix the problem. All syslog messages will be printed to the console and thus be available via the console-output command.

We may wanna restrict what will be send to /dev/xvc0 (or add an ec2 specific configuration file in /etc/rsyslog.d/) and adjust the logger call in the ec2-init script to match the filters.

I've verified that putting the following stanza into /etc/rsyslog.d/60-ec2.conf fixes the problem:

daemon.*;mail.*;\
 news.err;\
 *.=debug;*.=info;\
 *.=notice;*.=warn |/dev/xvc0

we do still need to trim it back a bit, but that can be easily enough written by vm-builder if we dont see a more appropriate way.

Ok, prepare for long windedness.

I've tried to determine if the regression shown here was due to a kernel
change (including ramdisk) or a user space change. To do that, I ran
several different combinations of root and initrd and checked console
output.

aliases:
- ec2-krd = aki-a71cf9ce / aki-9c04e4f5
  ec2-vmlinuz-2.6.21.7-2.fc8xen.i386
  ec2-initrd-2.6.21.7-2.fc8xen.i386
- karmic-krd = aki-841efeed / ari-9a1efef3
  ubuntu-kernel-2.6.31-300-ec2-xen-i386-20090910-test-02
  ubuntu-ramdisk-2.6.31-300-ec2-xen-i386-20090910-test-02.manifest.xml
- intrepid-krd = aki-714daa18 / ari-7e4daa17
  vmlinuz-2.6.27-23-xen-i386
  initrd.img-2.6.27-23-xen-i386
- alpha5 = ami-3520c05c
  karmic-i386-alpha5.manifest.xml
- alpha5.1 = ami-a40fefcd
  karmic-i386-alpha5.1.manifest.xml
- alpha6 = ami-fa658593
  karmic-i386-alpha6.manifest.xml

- Note, aki-9c04e4f5 / ari-9e04e4f7 == aki-841efeed / ari-9a1efef3
  I simply re-published the same manifest in a different bucket generating
  the different numbers

root_________| kern/rd | instance | result
alpha5_______| amazon-ec2 | i-c374a0ab | Yes : init messages get to console
alpha5______ | karmic-krd | i-ab73a7c3 | No : no console output past kernel
alpha5.1_____| karmic-krd | i-9573a7fd | No : no console output past kernel
alpha6_______| amazon-ec2 | i-0778ac6f | Yes : init messages ("Deactivating swap", "init:")
alpha6_______| karmic-krd | i-c726f2af | No : no console output past kernel

Note, the amazon-ec2 kernels dont completly boot with our user space, so
the output isn't as clear as you'd like, but you do see output from the
init process and sysvinit.

Its not completely straightforward, but I believe that the above shows a regression in the kernel/initrd rather than in userspace init or general makeup of image.

While it appears there is some kernel interaction it isn't just a kernel issue. I did further testing and came up with some interesting results.

     ami | aki | ari | full boot | init message go to console
fedora ami-2547a34c | fedora aki-b51cf9dc | fedora ari-b31cf9da | Yes | Yes
ubuntu ami-1a658573 | fedora aki-b51cf9dc | fedora ari-b31cf9da | No | Yes
ubuntu ami-1a658573 | aki-9c1efef5 | ari-901efef9 | Yes | No
fedora ami-2547a34c | aki-9c1efef5 | ari-901efef9 | Yes | Yes

A fedora userspace with the karmic kernel/initrd will boot and display init messages to the console.

> A fedora userspace with the karmic kernel/initrd will boot and display
> init messages to the console.

Interesting. Thank you for your effort. You have any clues?

I have a theroy that I am testing right now

Hi,

So my testing so far has been the following:

I think there might be a bug with the interaction of new xen framebuffer drivers and the older versions of Xen that Amazon runs.

I have added a kernel patch to my local ec2 tree which changes the console from xvc0 to tty1. This will allow console messages to be displayed on ec2 so that users can see the ssh keys, boot up messages, etc. I will send the patch today to the kernel team mailing list for approval to be included in the karmic kernel.

Regards
chuck

Hi,

Can you try with with aki-305dbd59 and ari-365dbd5f.

I was able to get a proper boot console with the above kernel and ramdisk.

Thanks
chuck

This combination worked for me as far as seeing the host ssh key in the console output:

  ami-fa658593 + aki-305dbd59 + ari-365dbd5f

I did note that it spent almost 5 minutes in "pending" state which would be a concern if it continues to happen.

There were also a few more warning type messages than I am used to seeing which somebody might want to investigate.

Console output attached.

> I did note that it spent almost 5 minutes in "pending" state which would
> be a concern if it continues to happen.

I had noticed longer 'pending' times today then normal myself, and not
with this new kernel, but other things that i was testing. Its something
to watch for, but I think probably unrelated to any of our changes.

This bug was fixed in the package linux-ec2 - 2.6.31-300.3

---------------
linux-ec2 (2.6.31-300.3) karmic; urgency=low

  [ Chuck Short ]

  * SAUCE: ec2: Default domU console to tty.
    - LP: #431103

  [ Tim Gardner ]

  * [Config] Remove dependency on wireless-crda
    - LP: #434755
  * [Config] Drop bootloaders as a recommendation
    - LP: #434755
  * [Config] Drop virtual sub flavours

 -- Tim Gardner <email address hidden> Fri, 25 Sep 2009 15:18:34 -0600

I just uploaded to canonical-testing-us:
  aki-80a84be9 ubuntu-kernel-2.6.31-300.3-ec2-i386.manifest.xml
  aki-8aa84be3 ubuntu-kernel-2.6.31-300.3-ec2-x86_64.manifest.xml
  ari-90a84bf9 ubuntu-ramdisk-2.6.31-300.3-ec2-i386.manifest.xml
  ari-8ea84be7 ubuntu-ramdisk-2.6.31-300.3-ec2-x86_64.manifest.xml

I then booted the i386 build karmic-i386-20090926.1 (ami-28a34041) with:
  run-instances ami-28a34041 --kernel aki-80a84be9 --ramdisk ari-90a84bf9

The happy result is we have console output again, including the ssh keys.

So, with linux-image-2.6.31-300-ec2 at version 2.6.31-300.3,
a.) this should be fixed in beta
b.) There is no ec2-init task required

Just marking here, the beta release has proper console output. Thanks zul and kernel team.