ec2 kernel needs CONFIG_BLK_DEV_LOOP=y and other config changes

just for the record, the -ec2 kernel should be similar to either -server or -virtual (which have the same .config). The filtered list of modules in the -virtual kernel would probably be nice as having loads of hardware drivers in an ec2 image is not useful.

On Mon, Sep 14, 2009 at 03:01:30PM -0000, Scott Moser wrote:
> just for the record, the -ec2 kernel should be similar to either -server
> or -virtual (which have the same .config). The filtered list of modules
> in the -virtual kernel would probably be nice as having loads of
> hardware drivers in an ec2 image is not useful.

A config more similar to -virtual would definitely be preferable to a config
similar to -server.

--
 - mdz

One other config option we don't want to see lost, that is currently not in the kernel image we have:

# CONFIG_EXT4_FS is not set

I realize that the above isn't a very good sentence. What I meant to say, was that the current kernel (with alpha6 -- linux-image-2.6.31-300-ec2) has :
# CONFIG_EXT4_FS is not set

We want to have ext4 support in ec2 kernels 2.6.31-10-generic has:
CONFIG_EXT4_FS=y
# CONFIG_EXT4DEV_COMPAT is not set
CONFIG_EXT4_FS_XATTR=y
CONFIG_EXT4_FS_POSIX_ACL=y
CONFIG_EXT4_FS_SECURITY=y

Per John (#ubuntu-server)

10/02/09 19:15:06 <jjohansen> anyway, here they are aki=aki-c6896aaf, ari=ari-d8896ab1
10/02/09 19:15:44 <jjohansen> i386: aki-e2896a8b ari-e4896a8d
10/02/09 19:16:14 <jjohansen> there are also, almost identical kernels with Hz=250 and kexec
10/02/09 19:16:48 <jjohansen> x86_64: aki-5e896a37 ari-50896a39
10/02/09 19:17:01 <jjohansen> i386: aki-00896a69 ari-02896a6b
10/02/09 19:17:51 <jjohansen> the Hz=250 was to compare that vs. standard config for VOIP

Hz=250 would be for bug 365233

The main wart here is the lack of ext4 support. This is not thought to cause major issues before release. Therefore the plan is to get those fixed for early upload via SRU after release. With a bigger review following that.

The lastest versions of the test kernel can be found using
x86_64: aki-1fcd2e76 ari-11cd2e78
i386: aki-43cd2e2a ari-45cd2e2c

unfortunately in my limited testing, they are experiencing the following bug

[43080.857643] BUG: soft lockup - CPU#1 stuck for 112s! [swapper:0]
[43080.857651] Modules linked in: fbcon tileblit font bitblit softcursor
[43080.857651] CPU 1:
[43080.857651] Modules linked in: fbcon tileblit font bitblit softcursor
[43080.857651] Pid: 0, comm: swapper Not tainted 2.6.31-302-ec2 #7
[43080.857651] RIP: e030:[<ffffffff801063aa>] [<ffffffff801063aa>] 0xffffffff801063aa
[43080.857651] RSP: e02b:ffff8801df853ec8 EFLAGS: 00000246
[43080.857651] RAX: 0000000000000000 RBX: ffffffff8085b3a8 RCX: ffffffff801063aa
[43080.857651] RDX: ffff8801df853eb0 RSI: 0000000000000000 RDI: 0000000000000001
[43080.857651] RBP: ffff8801df853ee0 R08: 0000000000000000 R09: 0000000000000000
[43080.857651] R10: 0000000000000005 R11: 0000000000000246 R12: 0000000000000000
[43080.857651] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[43080.857651] FS: 00007f0496b796f0(0000) GS:ffffc90000010000(0000) knlGS:0000000000000000
[43080.857651] CS: e033 DS: 002b ES: 002b CR0: 000000008005003b
[43080.857651] CR2: 00007f0496698230 CR3: 0000000000101000 CR4: 0000000000000660
[43080.857651] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[43080.857651] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000000
[43080.857651] Call Trace:
[43080.857651] [<ffffffff8010c565>] ? xen_safe_halt+0x15/0x40
[43080.857651] [<ffffffff801102a6>] xen_idle+0x46/0xf0
[43080.857651] [<ffffffff805e81d5>] ? atomic_notifier_call_chain+0x15/0x20
[43080.857651] [<ffffffff8010855c>] cpu_idle+0x6c/0xb0
[43080.857651] [<ffffffff805e2025>] cpu_bringup_and_idle+0xe/0x10

What is the configuration of the kernel which is triggering this bug, and how does it differ from the current linux-ec2 config (which presumably doesn't trigger the bug)?

Any chance these kernels can be uploaded to the eu region for testing too?

On further checking, it seems there are newer kernels/ramdisks in EU now, but I can't seem to launch an instance with one...

$ ec2-run-instances ami-7a5e750e -k rossg --kernel=aki-da5972ae --ramdisk=ari-c45972b0 -z eu-west-1b --region eu-west-1
Client.InvalidManifest: HTTP 403 (Forbidden) response for URL http://ubuntu-kernels-testing-eu.s3.amazonaws.com:80/ubuntu-karmic-i386-linux-image-2.6.31-302-ec2-v-2.6.31-302.7-ramdisk.img.manifest.xml: check your S3 ACLs are correct.

Ross, The kernel you pointed at (ubuntu-kernels-testing-eu/ubuntu-karmic-i386-linux-image-2.6.31-302-ec2-v-2.6.31-302.7-ramdisk.img.manifest.xml) is not comparible to the kernel John loaded to ec2 US. It will not have ext4 in it.

That said, it *should* launch, as far as I can tell you should have launch permission, I even went ahead and re-ran the modify-image-attribute command to give it to all:
  ec2-modify-image-attribute --region eu-west-1 --launch-permission --add all ari-c45972b0

I believe this must be a bug in amazon's ec2-migrate-image. That command just must not work for kernel and ramdisk. I'll have to change my publish tools to only use migrate on images. Thank you for pointing this out.

It appears my guess above was correct. I've modified the publish tools to use migrate-image only for 'image', and also deregistered all the images that were registered using 'migrate-image'.

There is a new kernel and ramdisk in eu-west-1 at aki-0c5e7578 and ari-0e5e757a respectively.

The amd64 version of these should get automatically published here in about an hour, so they'll likely be there, along with a 20091018 image by the time you read this.

This is the config patch that the test kernels from comment #4 where built from and that is resulting in the cpu lockup seen in comment #7. This config is based off of the virtual config.

This is a minimal small config patch that enables only ext4, sound dummy (for VOIP), and block loop.

The test kernels for this patch are:
x86-64: kernel aki-13dc3f7a ramdisk ari-15dc3f7c
i386: kernel aki-1ddc3f74 ramdisk ari-1fdc3f76

these kernels don't appear to be having the same issues, but further testing is needed.

I guess those kernels aren't available in the EU region :(

# ec2-run-instances ami-544c6720 -k rossg -z eu-west-1b --region eu-west-1 --kernel aki-1ddc3f74 --ramdisk ari-1fdc3f76
Client.InvalidAMIID.NotFound: The AMI IDs 'ami-1ddc3f74, ami-1fdc3f76' do not exist

No, they aren't yet. I will look into setting them up for the EU region too.

Just an update here. John is hoping to get a SRU kernel out to address this.
In the mean time, the largest lack of functionality in the available kernels is ext4. To combat that, I've built an ext4 kernel module and made those available at
http://people.canonical.com/~smoser/bugs/428692/ . See the README.txt there for more info.

hi there,
sorry, but i think you might have the wrong 'scott moser'
have you tried '<email address hidden>?'

sorry.
scott

On Wed, Jan 27, 2010 at 8:33 PM, Scott Moser <email address hidden> wrote:

> Just an update here. John is hoping to get a SRU kernel out to address
> this.
> In the mean time, the largest lack of functionality in the available
> kernels is ext4. To combat that, I've built an ext4 kernel module and made
> those available at
> http://people.canonical.com/~smoser/bugs/428692/<http://people.canonical.com/%7Esmoser/bugs/428692/>. See the README.txt there for more info.
>
> --
> ec2 kernel needs CONFIG_BLK_DEV_LOOP=y and other config changes
> https://bugs.launchpad.net/bugs/428692
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in “linux-ec2” package in Ubuntu: Triaged
> Status in “linux-ec2” source package in Karmic: Triaged
>
> Bug description:
> As seen in bug 418130 comment 25, the ec2 kernel doesn't have
> CONFIG_BLK_DEV_LOOP=y set. There are probably other options that differ
> from the other ubuntu kernels.
>
> In general, I think we want to configure the ec2 kernel as much like the
> others as reasonably possible. Specifically, we should at least enable
> CONFIG_BLK_DEV_LOOP=y.
>
>
>
> To unsubscribe from this bug, go to:
> https://bugs.launchpad.net/ubuntu/+source/linux-ec2/+bug/428692/+subscribe
>

--
Post-Doctoral Research Fellow
Nuffield College
Oxford, OX1 1NF
Office: +44 (0)1865 278623
Fax: +44 (0)1865 278621

Please get this into lucid ASAP. Bugs need to be fixed and tested in the development release before SRUing them.

Accepted linux-ec2 into karmic-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Martin, regarding comment 20, this is fixed in lucid release. The lucid release -ec2 kernel configs are much more similar to "standard".

Ok, please mark the task as fixed then. :) (done)

The Karmic proposed kernels have been tested on EC2. They boot and everything appears to be working correctly.

This bug was fixed in the package linux-ec2 - 2.6.31-304.11

---------------
linux-ec2 (2.6.31-304.11) karmic-proposed; urgency=low

  [ John Johansen ]

  * [Config] enable ext4 and block loop
    - LP: #428692
 -- Leann Ogasawara <email address hidden> Mon, 01 Feb 2010 11:27:41 -0800

The kernel listed above is now available in "ubuntu-kernels-testing" buckets. In the next week or two we hope to have fully released versions of images with these kernels. Thanks John!

Adding CONFIG_BLK_DEV_LOOP=y will prevent loopAES to be used. It was "=m" in the last LTS release and will break all installations using loopAES.

On Thu, Mar 11, 2010 at 02:22:28AM -0000, ulpfr wrote:
> Adding CONFIG_BLK_DEV_LOOP=y will prevent loopAES to be used. It was
> "=m" in the last LTS release and will break all installations using
> loopAES.

Are you aware of anyone using loopAES on EC2?

--
 - mdz

> Are you aware of anyone using loopAES on EC2?
I do, for one.