usb storage with ext4 does not work in lucid on imx51 hardware

seems that this issue starts with "unhandled modesense"

sd 2:0:0:0: [sdb] Unhandled sense code

I can not reproduce this issue on my side. EXT4 USB disk works fine.

Hardware: Babbage Board 2.5
Image: Alpha2 release + apt-get dist-upgrade
Kernel:
--
$ uname -a
Linux babbage 2.6.31-602-imx51 #4-Ubuntu Mon Jan 11 17:05:57 UTC 2010 armv7l GNU/Linux
--

I created a 1G EXT4 USB disk on my Karmic desktop and mounted it on my Babbage board. Wrote 10M file to the USB disk and can read out it on my desktop machine.

--
roc@babbage:/media$ cd 1G_EXT4/
roc@babbage:/media/1G_EXT4$ ls
lost+found
roc@babbage:/media/1G_EXT4$ dd if=/dev/zero of=data.bin bs=1M count=10
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.140188 s, 74.8 MB/s
roc@babbage:/media/1G_EXT4$ cd ..
roc@babbage:/media$ umount 1G_EXT4/
roc@babbage:/media$ ls
4B4D-D0D4 cdrom cdrom0
roc@babbage:/media$ dmesg
Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Linux version 2.6.31-602-imx51 (buildd@huito) (gcc version 4.4.3 20100108 (prerelease) (Ubuntu 4.4.2-9ubuntu3) ) #4-Ubuntu Mon Jan 11 17:05:57 UTC 2010 (Ubuntu 2.6.31-602.4-imx51)
CPU: ARMv7 Processor [412fc081] revision 1 (ARMv7), cr=10c53c7f
CPU: VIPT nonaliasing data cache, VIPT nonaliasing instruction cache
Machine: Freescale MX51 Babbage Board
Memory policy: ECC disabled, Data cache writeback
On node 0 totalpages: 122880
free_area_init_node: node 0, pgdat c060cc24, node_mem_map c0682000
  DMA zone: 128 pages used for memmap
  DMA zone: 0 pages reserved
  DMA zone: 16256 pages, LIFO batch:3
  Normal zone: 832 pages used for memmap
  Normal zone: 105664 pages, LIFO batch:31
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 121920
Kernel command line: root=UUID=fd8c99d5-8e53-4d99-9d4e-75b8c96e7b94 ro quiet splash
PID hash table entries: 2048 (order: 11, 8192 bytes)
Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
allocated 2457600 bytes of page_cgroup
please try 'cgroup_disable=memory' option if you don't want memory cgroups
Memory: 480MB = 480MB total
Memory: 468552KB available (5744K code, 692K data, 176K init, 0K highmem)
SLUB: Genslabs=11, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
NR_IRQS:272
MXC IRQ initialized
Console: colour dummy device 80x30
console [tty0] enabled
Calibrating delay loop... 799.53 BogoMIPS (lpj=3997696)
Security Framework initialized
AppArmor: AppArmor initialized
Mount-cache hash table entries: 512
Initializing cgroup subsys ns
Initializing cgroup subsys cpuacct
Initializing cgroup subsys memory
Initializing cgroup subsys freezer
Initializing cgroup subsys net_cls
CPU: Testing write buffer coherency: ok
regulator: core version 0.5
NET: Registered protocol family 16
CPU is i.MX51 Revision 2.0
MXC GPIO hardware
iomux_config_mux: Warning: iomux pin config changed, reg=fb0a8134, prev=0x1 new=0x3
IRAM READY
mxc_init_dvfs_per initialised
Using SDMA I.API
MXC DMA API initialized
bio: create slab <bio-0> at 0
SCSI subsystem initialized
libata version 3.00 loaded.
CSPI: mxc_spi-0 probed
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
MXC I2C driver
MXC I2C driver
MXC HS I2C driver
PMIC Li...

Bryan, do you have a bbg3 to test this? On bbg3 the behaviour was odd: in lucid it wouldnt work at all ... in karmic it first didnt work, but i was able to make this work by plugging the disk in an external, powered usb hub.

So i wouldnt rule out that there are issues with the usb on board.

Alex,

Too bad, I just have a BB2.5 board and USB ext4 works fine on my lucid. Maybe we need other guys to confirm this.

Confirmed on an intel iMac 8,1

klaus@klaus-imac:~$ uname -a
Linux klaus-imac 2.6.32-11-generic #15-Ubuntu SMP Tue Jan 19 20:38:41 UTC 2010 x86_64 GNU/Linux

Scratch that - I must've encountered a different bug that didn't let me mount my ext4 usv-drive while another external ext3 drive was mounted at the same time. Sorry.

@Alexander -- where are we at with this, have we put this down to bad h/w? or do we have an issue?

I can confirm that this is happening on my babbage 3 as well. I had pretty nasty problems trying to install to a usb device, including the usb errors mentioned below. However, upon switching to a sata target, everything installed fine.

Ok we have a major set of updates to the BSP from freescale, can you retest with the latest kernel in the archive and let us know if it is still present.

I'm not running lucid, but it _seems_ that the combination of ext4 on USB with karmic makes my machine instable and eventually crashing. ICH8 family USB. Can this be related?

do you use an imx51 based board (note that this bug is agains this specific kernel which differs from the other ubuntu kernels quite a lot) ?

@Andy,
I tested on my board with latest kernel and failed to reproduce this bug on my side as before.

@Mirar,
As Oliver posted, this bug is for Freescale i.MX51 hardware not for your PC. Apparently ICH8 is a Intel chipset including USB host controller.

@Oliver,
Do you meet this issue on your hardware?

Thanks,
-Bryan

no, i have never seen it on a babbage 2.5, it might be a 3.0 specific thing

Okay, it's me again, we DO have the same bug. I think it's either a hardware or a driver error regarding the usb enclosure detected as "05e3:0718 Genesys Logic, Inc. ". I have two of these and both fail to be mounted in lucid (on two different machines, one imac and a Lenovo T500). I don't think it's lucid specific, though, as a vanilla 2.6.32-7 kernel also fails to mount the external disk. In Karmic they work just fine, even though I have seen odd behaviour there too when it wouldn't mount the disk on the first try.

I'm not having a whole lot of luck reproducing this (with previous kernel, haven't tried the new one yet), except at install time. This completely seems to prevent me from installing to usb, but despite several attempts at mounting, reading, and writing to ext4+usb (the same one I was having trouble installing to) I can't reproduce it that way. It'll be easier to test the new kernel as part of an image at install time from my perspective.

i gave this usb storage disk to bryan now ... checking with him the power issue i previously referred to maybe being a bbg3 issue is expected. So for now it seems to be software issue aka kernel as it works on karmic

Alex,

Unfortunately, I compiled a Karmic kernel from our latest Karmic kernel fsl-imx51 branch. Then installed that kernel with my Lucid userspace. After testing the USB disk you gave to me, it is the same result as our latest Lucid kernel. It is failed to mount, since this:
--
roc@babbage:~$ uname -a
Linux babbage 2.6.31-107-imx51 #18 Wed Feb 3 20:36:51 EST 2010 armv7l GNU/Linux
roc@babbage:~$ usb 1-1.1: device not accepting address 4, error -71
hub 1-1:1.0: unable to enumerate USB device on port 1
sd 1:0:0:0: [sdb] Assuming drive cache: write through
sd 1:0:0:0: [sdb] Assuming drive cache: write through
sd 1:0:0:0: [sdb] Assuming drive cache: write through
sd 2:0:0:0: [sdb] Assuming drive cache: write through
sd 2:0:0:0: [sdb] Assuming drive cache: write through
end_request: I/O error, dev sdb, sector 0
Buffer I/O error on device sdb, logical block 0
ldm_validate_partition_table(): Disk read failed.
sd 2:0:0:0: [sdb] Assuming drive cache: write through

roc@babbage:~$ end_request: I/O error, dev sdb, sector 0
Buffer I/O error on device sdb, logical block 0
Buffer I/O error on device sdb, logical block 1
Buffer I/O error on device sdb, logical block 2
Buffer I/O error on device sdb, logical block 3
Buffer I/O error on device sdb, logical block 4
Buffer I/O error on device sdb, logical block 5
Buffer I/O error on device sdb, logical block 6
Buffer I/O error on device sdb, logical block 7
Buffer I/O error on device sdb, logical block 8
sd 2:0:0:0: rejecting I/O to offline device
end_request: I/O error, dev sdb, sector 240
sd 3:0:0:0: [sdb] Assuming drive cache: write through
sd 3:0:0:0: [sdb] Assuming drive cache: write through
sd 3:0:0:0: [sdb] Assuming drive cache: write through
--

Can you tell me which version Karmic kernel works when you test this?

Thanks,
-Bryan

Very interesting, I fresh installed the Karmic image on my Babbage 2.5 board. The USB storage disk works in Karmic. I will keep this broken device to continue my debugging.

-Bryan

I think I might have this problem, this is an extract form /var/log/messages. I am running ubuntu karmic, using a external usb hard drive as my root disk.

Feb 15 10:40:07 chris-desktop kernel: [ 43.928026] usb 1-3: reset high speed USB device using ehci_hcd and address 2
Feb 15 10:40:08 chris-desktop kernel: [ 44.315724] sd 2:0:0:0: [sdb] Unhandled sense code
Feb 15 10:40:08 chris-desktop kernel: [ 44.315730] sd 2:0:0:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Feb 15 10:40:08 chris-desktop kernel: [ 44.315735] sd 2:0:0:0: [sdb] Sense Key : Medium Error [current]
Feb 15 10:40:08 chris-desktop kernel: [ 44.315741] sd 2:0:0:0: [sdb] Add. Sense: Unrecovered read error
Feb 15 10:40:15 chris-desktop kernel: [ 51.928038] usb 1-3: reset high speed USB device using ehci_hcd and address 2

People interested in this bug should also take a look at:

https://bugs.launchpad.net/ubuntu/+source/linux-fsl-imx51/+bug/523293 (Cache incoherency may corrupt code pages read from filesystems on USB or PIO mass storage)

I think i find the root cause of this bug now. Actually it is the same issue as bug http://bugs.launchpad.net/bugs/431963. This JMircon USB storage disk uses the same USB-TO-SATA bridge controller as our Babbage board, but it gives us different USB VID and PID.

We need to skip ATA PASS THROUGH command for this device as well as our Babbage on board controller.

--
roc@babbage:~$ dmesg
Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Linux version 2.6.31-605-imx51 (root@emerald) (gcc version 4.4.1 (Sourcery G++ Lite 2009q3-67) ) #8 Wed Feb 24 02:11:14 EST 2010 (Ubuntu 2.6.31-605.8-imx51)
CPU: ARMv7 Processor [412fc081] revision 1 (ARMv7), cr=10c53c7f
CPU: VIPT nonaliasing data cache, VIPT nonaliasing instruction cache
Machine: Freescale MX51 Babbage Board
Memory policy: ECC disabled, Data cache writeback
On node 0 totalpages: 122880
free_area_init_node: node 0, pgdat c06692cc, node_mem_map c06eb000
  DMA zone: 128 pages used for memmap
  DMA zone: 0 pages reserved
  DMA zone: 16256 pages, LIFO batch:3
  Normal zone: 832 pages used for memmap
  Normal zone: 105664 pages, LIFO batch:31
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 121920
Kernel command line: root=UUID=e35b9b48-cd90-45ce-9dd5-3a186f4f9a7b ro quiet splash
PID hash table entries: 2048 (order: 11, 8192 bytes)
Dentry cache hash table entries: 65536 (order: 6, 262144 bytes)
Inode-cache hash table entries: 32768 (order: 5, 131072 bytes)
allocated 2457600 bytes of page_cgroup
please try 'cgroup_disable=memory' option if you don't want memory cgroups
Memory: 480MB = 480MB total
Memory: 468132KB available (6104K code, 744K data, 184K init, 0K highmem)
SLUB: Genslabs=11, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
NR_IRQS:272
MXC IRQ initialized
Console: colour dummy device 80x30
console [tty0] enabled
Calibrating delay loop... 799.53 BogoMIPS (lpj=3997696)
Security Framework initialized
AppArmor: AppArmor initialized
Mount-cache hash table entries: 512
Initializing cgroup subsys ns
Initializing cgroup subsys cpuacct
Initializing cgroup subsys memory
Initializing cgroup subsys freezer
Initializing cgroup subsys net_cls
CPU: Testing write buffer coherency: ok
tmpfs: No value for mount option 'mode'
devtmpfs: initialized
regulator: core version 0.5
NET: Registered protocol family 16
CPU is i.MX51 Revision 2.0
MXC GPIO hardware
iomux_config_mux: Warning: iomux pin config changed, reg=fb0a8134, prev=0x1 new=0x3
IRAM READY
mxc_init_dvfs_per initialised
Using SDMA I.API
MXC DMA API initialized
bio: create slab <bio-0> at 0
SCSI subsystem initialized
libata version 3.00 loaded.
CSPI: mxc_spi-0 probed
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
MXC I2C driver
MXC I2C driver
MXC HS I2C driver
IPU DMFC NORMAL mode: 1(0~1), 5B(4,5), 5F(6,7)
PMIC Light driver loading...
Bluetooth: Core ver 2.15
NET: Registered protocol family 31
Bluetooth: HCI device and connection manager initialized
Bluetooth: HCI socket layer initialized
NET: Registered protocol family 8
NET: Registered protocol family 20
NetLabel: Initializing
NetLabel: domain hash size = 128
NetLabel: proto...

That's exactly what Alan Stern told me on linux-usb when he looked at my trace:

(quoting from here: https://bugs.launchpad.net/ubuntu/+source/udev/+bug/515023 I filed it against udev because Alan suggested some rule(s) there might be the cause of it.)

I took the problem to the linux-usb list and submitted a usbmon trace. Alan Stern looked at it and he found these entries suspicious:

> ffff88001b083e40 2655717029 S Bo:2:015:2 -115 31 = 55534243 21000000 00000000 00001085 06200005 00fe0000 00000000 40ef00
> ffff88001b083e40 2655717090 C Bo:2:015:2 0 31 >
> ffff88001b083e40 2655717123 S Bi:2:015:1 -115 13 <
> ffff88001b083e40 2655731738 C Bi:2:015:1 0 13 = 55534253 21000000 00000000 00
> ffff88001b083e40 2655732007 S Bo:2:015:2 -115 31 = 55534243 22000000 00020000 80001085 082e0000 00000000 00000000 40ec00
> ffff88001b083e40 2655732099 C Bo:2:015:2 0 31 >
> ffff8800afe263c0 2655732126 S Bi:2:015:1 -115 512 <
> ffff8800afe263c0 2655748859 C Bi:2:015:1 -121 13 = 55534253 22000000 00020000 00
> ffff88001b083e40 2655748906 S Bi:2:015:1 -115 13 <
> ffff88001b083e40 2663266469 C Bi:2:015:1 -104 0

From Alan's comment on it:
"The shows a sequence of two ATA pass-through commands being sent to the
device. I don't know what those commands are or what program was
responsible for sending them; as far as I'm aware nothing in the kernel
will do it. Perhaps some program started by udev is responsible.
Looking through your udev rules might pinpoint the culprit.

Anyway, the first command doesn't seem to cause any difficulty, but the
second command fails completely. Instead of sending data in its
response, the device sends a status message. The kernel gets this,
thinks it is data, and then waits for the status to come -- which of
course never happens. So the kernel resets the device and tries
issuing the same command again, with the same result."

Klaus,

Thanks a lot for this useful information. I also have some concern about this patch solution, it looks like we added some regression between the karmic and lucid, not only on fsl-imx51 but also on x86.

From Alan's comment, I think we need to find out who send out the second ATA pass-through command.

Cheers,
-Bryan

I just tested the JMicron USB Disk on my Lucid machine. It fails to work as well as on our imx51 babbage board. So I suspect that this issue is common for x86 and imx51, it is a duplicated bug as Klaus mentioned.

Since it works on Karmic, I also tried Karmic kernel on my Lucid imx51 babbage board. We might take a look at udev or other user space changes.

-Bryan

This bug was fixed in the package linux-fsl-imx51 - 2.6.31-605.9

---------------
linux-fsl-imx51 (2.6.31-605.9) lucid; urgency=low

  [ Andy Whitcroft ]

  * Revert "include modules.builtin in the binary debs"
  * include modules.builtin in the binary debs

  [ Bryan Wu ]

  * SAUCE: IMX51: skip ATA PASS THROUGH command for USB storage disk from
    JMicron
    - LP: #499881

  [ Upstream Kernel Changes ]

  * ARM: 5677/1: ARM support for
    TIF_RESTORE_SIGMASK/pselect6/ppoll/epoll_pwait
    - LP: #319729
  * leds/mc13892: Use workqueue for setting LED brightness
    - LP: #531696
  * ARM: 5882/1: ARM: Fix uncompress code compile for different defines of
    flush(void)
    - LP: #319240
  * ARM: 5884/1: arm: Fix DCC console for v7
    - LP: #319240
  * ARM: 5885/1: arm: Flush TLB entries in setup_mm_for_reboot()
    - LP: #319240
  * ARM: 5886/1: arm: Fix cpu_proc_fin() for proc-v7.S and make kexec work
    - LP: #319240
  * ARM: 5888/1: arm: Update comments in cacheflush.h and remove
    unnecessary V6 and V7 comments
    - LP: #319240
  * arm: disable L2 cache in the v7 finish function
  * arm: invalidate TLBs when enabling mmu
 -- Andy Whitcroft <email address hidden> Tue, 09 Mar 2010 13:52:16 +0000