Storage devices are detected in a different order when upgrading from woody

Can you confirm that you definitely ran yabootconfig from warty's yaboot (1.3.13-1)?

Created an attachment (id=18)
yaboot.conf generated from yabootconfig (1.3.13-1)

Yes, that is the version, and this is the generated yaboot.conf.

Incidentally, I had the same problem on i386 with d-i RC1. Upgrading to 2.6.7
choked, but going to 2.6.8 worked.

Does this still apply to linux-image-2.6.8.1-1-powerpc in warty?

This also applies to linux-image-2.6.8.1-1-powerpc.

Booting with "devfs=mount" as suggested by mdz resulted in the same panic,
though "Mounting devfs on /dev" was printed twice in a row.

Running init with xtrace reveals the following

try_name hda7
[ -f /sys/block/hda7/dev ]
return 1
part=7
root=hda
[ -z hda ]
try_name hda 7
[ -f /sys/block/hda/dev ]
return 1
return 0
mknod dev2/root2 b 0 0
device=/dev2/root2
mount_device
unset flags fstype
set -f
set +f root=/dev/hda7 ro ramdisk_size=8192
[blah blah, attempt to mount /dev2/root2 to /mnt with ext3, ext2, iso9660, hfs,
cramfs, auto, then umount sys, devfs, dev2, proc, cd to mnt, then...]
pivot_root . initrd
pivot_root: No such file or directory

This is probably due to

ide0: Found Apple Heathrow ATA controller, bus ID 0, irq 13
Probing IDE interface ide0
hda: [cd-rom]
hdb: [atapi floppy]
[no hd mentioned]

Where is the hard disk attached? Since root is configured as /dev/hda7, I'd
assume that it was the first disk on the first IDE bus, but that's showing up as
a CD-ROM?

In order to boot with linux-2.6.8.1-powerpc, I had to force "modprobe -k cmd64x"
into loadmodules in the initrd, boot with "Linux root=/dev/hdc7" (since hda had
become hdc), endure several screenfuls of ieee1394/pcilynx spew, and manually
load the Ethernet driver (bmac).

So it seems that cmd64x should have been loaded by initrd-tools, but was not.
It is listed in loadmodules on my i386 system, even though I don't have a
cmd64x, but on a powerpc, I only see unix and ide-disk.

(In reply to comment #11)
> So it seems that cmd64x should have been loaded by initrd-tools, but was not.
> It is listed in loadmodules on my i386 system, even though I don't have a
> cmd64x, but on a powerpc, I only see unix and ide-disk.

What version of initrd-tools is this? This looks like an old bug that's already
fixed in my private tree. AFAIC all the changes in my private tree should also
be in Debian.

Our initrd-tools is a branch from 0.1.70.

Looking at the changelog since that version, I think this may be
http://bugs.debian.org/252466, fixed in 0.1.71.

It looks like this was fixed by moving the:

if [ -d "$MODULEDIR/kernel/drivers/ide/pci" ]; then

block above the case statement, rather than below it. Is that correct?

(In reply to comment #13)
> It looks like this was fixed by moving the:
>
> if [ -d "$MODULEDIR/kernel/drivers/ide/pci" ]; then
>
> block above the case statement, rather than below it. Is that correct?

Yep.

Thanks; Colin, please take it from here

This is still a problem with linux-image-2.6.8.1-3-powerpc 2.6.8.1-10 and
initrd-tools 0.1.70ubuntu9

Can you test the suggested fix in this bug? It hasn't been integrated into
initrd-tools yet.

Created an attachment (id=336)
correct patch?

I will attempt to test the attached patch within 24 hours.

Yes, correct. Thanks for testing, let us know if it fixes things.

The patched initrd-tools produces the following loadmodules file:

modprobe -k unix 2> /dev/null
modprobe -k sd_mod
modprobe -k pdc202xx_new > /dev/null 2>&1
modprobe -k aec62xx > /dev/null 2>&1
modprobe -k cmd64x > /dev/null 2>&1
modprobe -k generic > /dev/null 2>&1
modprobe -k hpt34x > /dev/null 2>&1
modprobe -k hpt366 > /dev/null 2>&1
modprobe -k ns87415 > /dev/null 2>&1
modprobe -k pdc202xx_old > /dev/null 2>&1
modprobe -k sc1200 > /dev/null 2>&1
modprobe -k siimage > /dev/null 2>&1
modprobe -k sl82c105 > /dev/null 2>&1
modprobe -k trm290 > /dev/null 2>&1
modprobe -k via82cxxx > /dev/null 2>&1
modprobe -k ide-disk

It now has cmd64x, which was missing before, right? Does it work?

(In reply to comment #21)
> It now has cmd64x, which was missing before, right? Does it work?

Yes, it does load cm64x and detect the hard drive. No, it doesn't work;

hda has become hdc so yaboot and fstab become incorrect
bmac is not loaded so therefore eth0 does not work
the (USB) keyboard and mouse do not work

Let's take it one problem at a time; this bug is about it failing to boot. So
after the initrd-tools fix, we still have the problem of the modules being
loaded in a different order than they were during the install.

(In reply to comment #23)
> Let's take it one problem at a time; this bug is about it failing to boot. So
> after the initrd-tools fix, we still have the problem of the modules being
> loaded in a different order than they were during the install.

Correction--these drivers are being loaded in a different order than in woody's 2.2
and 2.4 kernels; in 2.4.18-newpmac, both IDE drivers are compiled-in, and the IDE
devices are arranged thusly:
hda: IBM-DTTA-351290, ATA DISK drive
hde: MATSHITA CR-587, ATAPI CD/DVD-ROM drive
hdf: IOMEGA ZIP 100 ATAPI, ATAPI FLOPPY drive

(ide0, ide1 are the cmd646, ide2 is the Apple Heathrow ATA/pmac_ide)

with the current warty 2.6 setup, pmac_ide is compiled in, but cmd64x is a module;
then hda become hdc, hde becomes hda, and hdf becomes hdb.

This is only an issue for upgrades from woody; on a fresh install,
it will use hdc consistently.

The specific problem of cmd64x not being loaded is solved by the patch,
and this particular bug should probably be closed when that patch is applied
to the initrd-tools.

(In reply to comment #24)
> Correction--these drivers are being loaded in a different order than in
woody's 2.2
> and 2.4 kernels; in 2.4.18-newpmac, both IDE drivers are compiled-in, and the IDE

Indeed. That's a nasty change. IMHO we should change the initialisation order
so that it's identical to the order when the drivers are all built-in.

Unfortunately unless Debian does that as well, it'll cause problems for people
who switch between Ubuntu/Debian kernels. Of course they already have problems
switching between Debian/custom kernels today if they choose to build the
drivers in.

Matt, please make a call on this.

We can't fix the ordering permanently unless we make the Ubuntu kernel more
modular, because if pmac_ide is compiled in, it is always initialized before any
drivers which are loaded as modules, no?

Now I'm concerned about the initrd-tools patch itself, because it will change
the ordering for existing installs. Someone who installed Ubuntu with root on a
device managed by one of the drivers listed after cmd64x in Clint's list would
have their system rendered unbootable by this change, would they not?

(In reply to comment #26)
> We can't fix the ordering permanently unless we make the Ubuntu kernel more
> modular, because if pmac_ide is compiled in, it is always initialized before any
> drivers which are loaded as modules, no?

Actually you can postpone the initialisation of drivers built-in to ide-core. So this can be
done easily.

> Now I'm concerned about the initrd-tools patch itself, because it will change
> the ordering for existing installs. Someone who installed Ubuntu with root on a
> device managed by one of the drivers listed after cmd64x in Clint's list would
> have their system rendered unbootable by this change, would they not?

The patch to initrd-tools doesn't change the ordering so much as actually making the modules
load.

Clint's problem comes from the fact that his system was installed from a kernel that had both
chip drivers built-in. He'd have the same problem when upgrading to a Debian ppc kernel.

However if we make the proposed change then yes it will break Ubuntu systems installed prior to
the change.

(In reply to comment #27)
> (In reply to comment #26)
> > We can't fix the ordering permanently unless we make the Ubuntu kernel more
> > modular, because if pmac_ide is compiled in, it is always initialized before any
> > drivers which are loaded as modules, no?
>
> Actually you can postpone the initialisation of drivers built-in to ide-core.
 So this can be
> done easily.

I see. Do we currently do this postponing?

> > Now I'm concerned about the initrd-tools patch itself, because it will change
> > the ordering for existing installs. Someone who installed Ubuntu with root on a
> > device managed by one of the drivers listed after cmd64x in Clint's list would
> > have their system rendered unbootable by this change, would they not?
>
> The patch to initrd-tools doesn't change the ordering so much as actually
making the modules
> load.

Right. But for some systems, such as Clint's, this could amount to a change in
ordering.

> Clint's problem comes from the fact that his system was installed from a
kernel that had both
> chip drivers built-in. He'd have the same problem when upgrading to a Debian
ppc kernel.
>
> However if we make the proposed change then yes it will break Ubuntu systems
installed prior to
> the change.

What changes would we need to make in order to solve the ordering issue
permanently? Would we need additional changes on top of that in order to solve
the woody upgrade problem?

(In reply to comment #28)
> > Actually you can postpone the initialisation of drivers built-in to ide-core.
> So this can be
> > done easily.
> I see. Do we currently do this postponing?

We do it for PCI drivers. I'm proposing that this be extended to all chip drivers.

> Right. But for some systems, such as Clint's, this could amount to a change in
> ordering.

It's not the initrd-tools patch that caused the reordering. It's the fact that kernel had his
driver compiled as a module.

> What changes would we need to make in order to solve the ordering issue
> permanently? Would we need additional changes on top of that in order to solve
> the woody upgrade problem?

I'm afraid that this is going to be an arbitrary choice. Basically there are two possible
orderings to choose from. We can either go with the woody/vanilla/custom kernel ordering which
is what I proposed, or we can stick with the current ordering which is the same as Debian.

Going from one ordering to another will inevitably break existing systems.

(In reply to comment #24)
> This is only an issue for upgrades from woody; on a fresh install,
> it will use hdc consistently.

About that: I've been struggling with getting a fresh install to work on a very
similar blue/white G3. Others have had similar experiences. Please see
http://www.ubuntuforums.org/showthread.php?t=8050 for further discussion. I've
contributed to the discussion there as Random Juju.

The relevant part of that discussion is that I've modified the loadmodules file
as suggested here, but am still unable to boot. Booting with the modified
initrd.img yields:

[tux logo]
PCI: Cannot allocate resource region 0 of device 0000:01:02.0
audit(1104724407.122:0): initialized
mount: mount point proc does not exist
cat: proc/sys/kernel/real-root-dev: No such file or directory
/linuxrc: 9: cannot create proc/sys/kernel/real-root-dev: Directory nonexistent
Kernel panic: VFS: Unable to mount root fs on unknown-block(0,0)

I don't know if this is a symptom of the same problem, but it seems that it may
be causing further problems with fresh installs.

(In reply to comment #30)
> [tux logo]
> PCI: Cannot allocate resource region 0 of device 0000:01:02.0
> audit(1104724407.122:0): initialized
> mount: mount point proc does not exist
> cat: proc/sys/kernel/real-root-dev: No such file or directory
> /linuxrc: 9: cannot create proc/sys/kernel/real-root-dev: Directory nonexistent
> Kernel panic: VFS: Unable to mount root fs on unknown-block(0,0)
>
> I don't know if this is a symptom of the same problem, but it seems that it may
> be causing further problems with fresh installs.

Doesn't look like the same problem; your initrd seems to be broken (missing
/proc). Please upload a copy of /boot/initrd.img somewhere that we can see it,
and check whether (e.g.) the install ran out of disk space

(In reply to comment #31)
> Doesn't look like the same problem; your initrd seems to be broken (missing
> /proc). Please upload a copy of /boot/initrd.img somewhere that we can see it,
> and check whether (e.g.) the install ran out of disk space

Oh, and please follow up elsewhere (not to this bug), since your problem is
unrelated.

*** Bug 9996 has been marked as a duplicate of this bug. ***

Just installed the hoary preview release on my B&W G3 and didn't have the
loadmodules problem.

My apologies, after reading this post:
http://ubuntuforums.org/showthread.php?postid=101454#poststop, I've realized
that I installed the 2.6.10-4-powerpc kernel by mistake. The above post
suggests that perhaps the problem has regressed in kernel 2.6.10-5-powerpc. I
will try a current daily asap. Note that I did not, however, have the
loadmodules problem when upgrading from 2.6.10-4 to 2.6.10-5.

The only way to guarantee the order is by putting the modules in
/etc/mkinitramfs/modules and regenerating. Because we scan by PCI ID, we rely
on the machine to give us the information.

The best solution if you're having troubles in Breezy is to use LVM. Another
solution post-Breezy will be to use /dev/by-uuid, or /dev/by-name which is
available in the udev that came out after UpstreamVersionFreeze.

Closing this bug as WONTFIX.