"pad block corrupted" error when trying to register an image with 2.6.34 kernel

Seems to (oddly) be a kernel regression. Reverting to a Lucid kernel on Maverick resolves this issue.

Hi Dave,

Please be sure to confirm this issue exists with the latest development release of Ubuntu. ISO CD images are available from http://cdimage.ubuntu.com/daily-live/current/ . If the issue remains, please run the following command from a Terminal (Applications->Accessories->Terminal). It will automatically gather and attach updated debug information to this report.

apport-collect -p linux 588861

Also, if you could test the latest upstream kernel available that would be great. It will allow additional upstream developers to examine the issue. Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Once you've tested the upstream kernel, please remove the 'needs-upstream-testing' tag. This can be done by clicking on the yellow pencil icon next to the tag located at the bottom of the bug description and deleting the 'needs-upstream-testing' text. Please let us know your results.

Thanks in advance.

    [This is an automated message. Apologies if it has reached you inappropriately; please just reply to this message indicating so.]

I can confirm on my rig.
Other Eucalyptus users have seen it on other platforms (RedHat/Eucalyptus/Xen), and also linked it to the 2.6.34 kernel, see:
http://open.eucalyptus.com/forum/decrypting-image-exception

For the record, root cause is:
ERROR javax.crypto.BadPaddingException: pad block corrupted
javax.crypto.BadPaddingException: pad block corrupted
 at org.bouncycastle.jce.provider.JCEBlockCipher.engineDoFinal(Unknown Source)
 at javax.crypto.Cipher.doFinal(Cipher.java:1708)
 at edu.ucsb.eucalyptus.cloud.ws.WalrusImageManager.decryptImage(WalrusImageManager.java:801)

Decryption works alright for kernel bundle, but not for the image bundle:

Decrypting image: //var/lib/eucalyptus/bukkits/ISOtest-20100616103000/lucid-server-uec-amd64-vmlinuz-virtual-HBfS9Es..tgz
Done decrypting: //var/lib/eucalyptus/bukkits/ISOtest-20100616103000/lucid-server-uec-amd64-vmlinuz-virtual-HBfS9Es..tgz
Decrypting image: //var/lib/eucalyptus/bukkits/ISOtest-20100616103000/lucid-server-uec-amd64.img-XcuxtCo..tgz
javax.crypto.BadPaddingException: pad block corrupted

Might be size-related.

Tried with current maverick 2.6.35-3-server, same issue.
Registering the image is sufficient to trigger the above error, no need to try to run an instance.

From my testing, the bigger the image, the more chances you have of hitting that error. If I take the lucid image and gradually truncate it, it starts working when its size is <=430Mb. I also have a 200Mb file that fails, so its not an absolute size thing.

I also tested bouncycastle 1.45 and it does fail the same.

Chris Cheney-

Please bisect the kernels at:
 * http://kernel.ubuntu.com/~kernel-ppa/mainline
and try to identify which kernel introduces this regression.

I am attaching a tarball with a stand-alone Java program that exercises BouncyCastle in the same exact way as Walrus does for image decryption. Included with the code are a couple of the JARs it needs and three input files: an encrypted compressed image (originally a ~1-GB Ubuntu image), the manifest for the image, and a cloud private key necessary for decryption. We are unable to reproduce this bug on our setup, but perhaps someone else will be able to, either with the included image or with a different one. To use the included image:

tar zxvf TestWalrusDecryption.tgz
cd TestWalrusDecryption
javac -classpath bcprov-jdk16-145.jar:xalan-2.7.1.jar TestWalrusDecryption.java
java -classpath bcprov-jdk16-145.jar:xalan-2.7.1.jar:. TestWalrusDecryption encrypted.img manifest.xml key.pem

To use with a different image, start with a working Eucalyptus installation and obtain user credentials. Using the credentials bundle (but not necessarily upload or register) an image of your choice. In the directory where bundling took place there will be a manifest file and several encrypted parts. Cat the parts together in the right order to obtain the encrypted image. Finally, extract the cloud private key from the cloud controller as follows:

openssl pkcs12 -in ${EUCALYPTUS}/var/lib/eucalyptus/keys/euca.p12 \
-name eucalyptus -name "eucalyptus" \
-password pass:eucalyptus -passin pass:eucalyptus -passout pass:eucalyptus \
-nodes | \
grep -A30 "friendlyName: eucalyptus" | \
grep -A26 "BEGIN RSA" > ${EUCALYPTUS}/var/lib/eucalyptus/keys/cloud-pk.pem

The cloud-pk.pem file is what you pass as the third parameter.

works:

Linux beagle 2.6.32-020632-generic #020632 SMP Thu Dec 3 10:09:58 UTC 2009 x86_64 GNU/Linux
Linux beagle 2.6.32-0206321505-generic #0206321505 SMP Wed Jun 2 19:09:40 UTC 2010 x86_64 GNU/Linux

bad:

Linux beagle 2.6.33-020633-generic #020633 SMP Thu Feb 25 10:10:03 UTC 2010 x86_64 GNU/Linux

Hmm I tried 2.6.33 again just to be certain and now it is working on 2.6.33, hmm :-\

I now can't get it to crash anymore. I am upgrading from maverick alpha 1 to current and then will do further testing.

I tried upgrading to current and now I can't see my nc anymore. :-\

After changing my test to just be registering the images per Dave's recommendation I was able to come up with this list:

I tested the following kernels, 10 times each:

Passed:
v2.6.32 (10/10)
v2.6.32.15.5-lucid (10/10)

Failed:
v2.6.33 (10/10)
v2.6.34-rc3-maverick (10/10)
v2.6.34-lucid (09/10) (weird python failure instead for 1 of them, see below)

So it looks like it broken between 2.6.32.15.5 and 2.6.33.

--

Sun Jun 20 23:57:09 CDT 2010: ====== extracting image ======
Warning: no ramdisk found, assuming '--ramdisk none'
kernel : lucid-server-uec-amd64-vmlinuz-virtual
ramdisk: none
image : lucid-server-uec-amd64.img
Sun Jun 20 23:57:17 CDT 2010: ====== bundle/upload kernel ======
Sun Jun 20 23:57:19 CDT 2010: ====== bundle/upload image ======
Traceback (most recent call last):
  File "/usr/bin/euca-bundle-image", line 231, in <module>
    main()
  File "/usr/bin/euca-bundle-image", line 209, in main
    encrypted_file, key, iv, bundled_size = euca.encrypt_image(tgz_file)
  File "/usr/lib/python2.6/dist-packages/euca2ools/__init__.py", line 579, in encrypt_image
    k=EVP.Cipher(alg='aes_128_cbc', key=unhexlify(key), iv=unhexlify(iv), op=1)
TypeError: Odd-length string
failed to bundle image lucid-server-uec-amd64.img
failed bundle/upload/register of image
Encrypting image

--

Tracked down to occurring in 2.6.33rc1, so will be checking between that and 2.6.32-15.5

Fails with 701791cc3c8fc6dd83f6ec8af7e2541b4a316606

Works with 41440ffe21f29bdb985cab76b2d0b06d83e63b19 (required minor compile fix)

Works with 23eb3b64b5e44680c867e165fe1cd18e57fba255 (required compile fix, probably the same one as Tim mentioned earlier).

works with 23eb3b64b5e44680c867e165fe1cd18e57fba255 (required minor compile fix)

Fails with 79a56ed0e11c7d924762062a0e2a46b87014498d

Fails with e33c01972239fee4696679ae5f7d1f340f424999

Fails with 951c30d135390a108f102b0f6e3cfa6241f2a1aa

Works with 5db5d64277bf390056b1a87d0bb288c8b8553f96

Works with 4f570f995f68ef77aae7e5a441222f59232f2d0e

Works with 150e6c67f4bf6ab51e62defc41bd19a2eefe5709

works with f21121cde6e617b90cd03ce083652ca543004dc2

rtg@zinc:~/linux-2.6$ git bisect good
cc56f7de7f00d188c7c4da1e9861581853b9e92f is first bad commit
commit cc56f7de7f00d188c7c4da1e9861581853b9e92f
Author: Changli Gao <email address hidden>
Date: Wed Nov 4 09:09:52 2009 +0100

    sendfile(): check f_op.splice_write() rather than f_op.sendpage()

    sendfile(2) was reworked with the splice infrastructure, but it still
    checks f_op.sendpage() instead of f_op.splice_write() wrongly. Although
    if f_op.sendpage() exists, f_op.splice_write() always exists at the same
    time currently, the assumption will be broken in future silently. This
    patch also brings a side effect: sendfile(2) can work with any output
    file. Some security checks related to f_op are added too.

    Signed-off-by: Changli Gao <email address hidden>
    Signed-off-by: Jens Axboe <email address hidden>

:040000 040000 c10964cc42af214af80ecfd5cc3e9e873c09b8d5 a1fecaea1c9a54c62471cdfe3b6540ca6619e2cb M fs

He appears to have made another update related to sendfile on May 26. However, I am not sure if it fixes this problem or that it has been applied to mainline yet.

http://marc.info/?a=119980668900012&r=1&w=2

A Ubuntu-2.6.35-6.7 kernel with cc56f7de7f00d188c7c4da1e9861581853b9e92f reverted also appears to clear up the error.

sendfile is used by openjdk-6 directly in the following file:

http://pastebin.com/tceugS7r

We mentioned this bug during the 20100625 release meeting.

The A2 milestone will ship with an affected kernel, but Tim hopes to be able to nail the issue before the milestone release.

If he does, we'll releasenote that UEC users should dist-upgrade before registering their first image.

If no fixed kernel is available in the archive by then, we'll set up a PPA with a kernel with the problematic commit reverted, and releasenote instructions for people to use it if they want to run Alpha2 UEC.

During test of the following kernel it prints the following to dmesg once for each pass:

ae0f36f0b964caf916c2ffc9f84b28c0f91c18a2-maverick

"[ 365.268713] /home/rtg/kern/maverick/kern-64/ubuntu-maverick/fs/read_write.c:849 should this happen?"

0a87a0c1b12f56bd556fd4506041966717c87fb1-maverick

failed as expected

[ 129.980324] /home/rtg/kern/maverick/kern-64/ubuntu-maverick/fs/read_write.c:849 ffffffff816256e0 (null)
[ 178.931445] /home/rtg/kern/maverick/kern-64/ubuntu-maverick/fs/read_write.c:849 ffffffff816256e0 (null)
[ 178.946722] /home/rtg/kern/maverick/kern-64/ubuntu-maverick/fs/read_write.c:849 ffffffff816256e0 (null)
[ 178.956955] /home/rtg/kern/maverick/kern-64/ubuntu-maverick/fs/read_write.c:849 ffffffff816256e0 (null)
[ 178.966482] /home/rtg/kern/maverick/kern-64/ubuntu-maverick/fs/read_write.c:849 ffffffff816256e0 (null)
[ 178.975832] /home/rtg/kern/maverick/kern-64/ubuntu-maverick/fs/read_write.c:849 ffffffff816256e0 (null)
[ 178.985221] /home/rtg/kern/maverick/kern-64/ubuntu-maverick/fs/read_write.c:849 ffffffff816256e0 (null)
[ 178.995507] /home/rtg/kern/maverick/kern-64/ubuntu-maverick/fs/read_write.c:849 ffffffff816256e0 (null)
[ 179.004812] /home/rtg/kern/maverick/kern-64/ubuntu-maverick/fs/read_write.c:849 ffffffff816256e0 (null)
[ 179.014215] /home/rtg/kern/maverick/kern-64/ubuntu-maverick/fs/read_write.c:849 ffffffff816256e0 (null)
[ 179.023585] /home/rtg/kern/maverick/kern-64/ubuntu-maverick/fs/read_write.c:849 ffffffff816256e0 (null)
[ 179.032934] /home/rtg/kern/maverick/kern-64/ubuntu-maverick/fs/read_write.c:849 ffffffff816256e0 (null)
[ 179.042205] /home/rtg/kern/maverick/kern-64/ubuntu-maverick/fs/read_write.c:849 ffffffff816256e0 (null)
[ 179.051533] /home/rtg/kern/maverick/kern-64/ubuntu-maverick/fs/read_write.c:849 ffffffff816256e0 (null)
[ 179.060871] /home/rtg/kern/maverick/kern-64/ubuntu-maverick/fs/read_write.c:849 ffffffff816256e0 (null)
[ 179.070252] /home/rtg/kern/maverick/kern-64/ubuntu-maverick/fs/read_write.c:849 ffffffff816256e0 (null)
[ 179.079538] /home/rtg/kern/maverick/kern-64/ubuntu-maverick/fs/read_write.c:849 ffffffff816256e0 (null)
[ 179.089084] /home/rtg/kern/maverick/kern-64/ubuntu-maverick/fs/read_write.c:849 ffffffff816256e0 (null)
[ 179.099589] /home/rtg/kern/maverick/kern-64/ubuntu-maverick/fs/read_write.c:849 ffffffff816256e0 (null)
[ 179.110479] /home/rtg/kern/maverick/kern-64/ubuntu-maverick/fs/read_write.c:849 ffffffff816256e0 (null)

---

afef312909fa10e603a05e030b2ee2feebde8d9f-maverick

works

Upstream response:

On Sat, Jun 26, 2010 at 4:18 AM, Tim Gardner <email address hidden> wrote:
> > Fat fingered the original email and didn't get the fsdevel address correct.
> >
> > -----------------------
> > Hi,
> >
> > I have bisected a strange regression down to this commit that you made. If I
> > apply the attached patch (which restores the original 2 lines of code), then
> > everything works correctly. I apologize, but I don't have a simple
> > reproducer. The gist of the problem is that Java cannot decrypt a file under
> > certain circumstances, e.g., "ERROR javax.crypto.BadPaddingException: pad
> > block corrupted". There is more information in the bug report at
> > http://bugs.launchpad.net/bugs/588861 , but not all of it is helpful.
> >
> > Any thoughts?
> >
It seems the target of sendfile(2) isn't a socket, and the current
sendfile(2) code can't handle non-socket target correctly. Please try
this patch: http://marc.info/?l=linux-fsdevel&m=127488508224173&w=2 .

-- Regards, Changli Gao(<email address hidden>)

The fix from upstream appears to work. This bug can be closed out once the fix goes into Maverick and will be in a ppa for alpha-2 testing from what Tim has said on IRC.

Thanks for all the help Tim!

The above referenced patch appears to fix the issue. Pull request sent.

https://lists.ubuntu.com/archives/kernel-team/2010-June/011352.html

This was a kernel regression.

A test kernel is available from the kernel PPA at http://ppa.launchpad.net/kernel-ppa/ppa/ubuntu in the maverick release pocket: linux 2.6.35-6.8~maverick1

The test kernel is actually linux 2.6.35-6.9~maverick1

Ok, so we've committed the fix to our git repo and will upload the kernel to the archive late Wed. I suggest the release note recommend using the kernel provided in the PPA and then update to the 2.6.35-6.9 kernel once it's available.

From #ubuntu-release on FreeNode:
[14:37:47] <ogasawara> cjwatson, ttx: in the release team meeting last friday we agreed, "if on Wednesday bug 588861 is not fixed in kernel, then can ogasawara prepare the PPA and then ttx can release note the PPA, otherwise ttx to release note to dist-upgrade"
[14:38:09] <ogasawara> cjwatson, ttx: we've already uploaded a kernel which contains the fix to our PPA
[14:38:51] <ogasawara> cjwatson, ttx: do you want me to wait to upload the official kernel to the archive until immediately after Alpha2 is released?
[14:39:34] <ogasawara> cjwatson, ttx: or would you prefer I upload the kernel sooner (ie now) so it's immediately available for people to dist-upgrade?
[14:41:55] <ScottK> ogasawara: It might be nice to wait until the buildds aren't quite so clogged if it's not going to be accepted until after Alpha 2.
[14:43:38] <ogasawara> ScottK: makes sense
[14:44:58] <cjwatson> I think late Wednesday would be OK
[14:57:20] <ogasawara> cjwatson: ack

This bug was fixed in the package linux - 2.6.35-6.9

---------------
linux (2.6.35-6.9) maverick; urgency=low

  [ Tim Gardner ]

  * [Upstream] direct_splice_actor() should not use pos in sd
    - LP: #588861
 -- Leann Ogasawara <email address hidden> Mon, 28 Jun 2010 12:35:49 -0700

Opening task for Lucid as it seems this issue is now showing there. (as per dupe Bug #813266)

Reconfirmed on lucid 10.04.3 candidate iso; also proposed kernel fix which resolved the issue.

This fix has been committed in a kernel to proposed. Updating the status of the bug to reflect that.

This bug was fixed in the package linux - 2.6.32-33.71

---------------
linux (2.6.32-33.71) lucid-proposed; urgency=low

  [Andy Whitcroft]

  * Release Tracking Bug
    - LP: #813507

  [ Upstream Kernel Changes ]

  * splice: direct_splice_actor() should not use pos in sd
    - LP: #588861
 -- Andy Whitcroft <email address hidden> Wed, 20 Jul 2011 14:31:11 +0100