failure to boot groovy daily (again)

Also fails to boot on
hp dc7900 (c2d-e8400, 4gb, intel 4 series integrated i915)

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1899308

I zsync'd Xubuntu groovy (daily) and tested

Xubuntu fails on same boxes, ie.

hp dc7700 (c2d-e6320, 5gb, nvidia quadro nvs 290)
hp dc7900 (c2d-e8400, 4gb, intel 4 series integrated i915)

but boots on boxes where Lubuntu also booted.

Ubuntu groovy (daily 20201010) desktop won't boot on the same boxes

hp dc7700 (c2d-e6320, 5gb, nvidia quadro nvs 290)
hp dc7900 (c2d-e8400, 4gb, intel 4 series integrated i915)

but boots as expected on other boxes tested (identical results to Lubuntu & Xubuntu).

@ Chris,

This bug might correspond to an observation, that I made years ago with persistent live drives made by mkusb.

Some old (at that time maybe middle-aged) HP computers refuse to boot

. in BIOS mode
- from a USB drive
- via grub
- in a GUID partition table (GPT).

That made me add an entry to the settings menu, where the default in GPT, but the user can select MSDOS partition table.

This is described in the following link,

https://askubuntu.com/questions/1076951/what-are-the-mkusb-options-msdos-upefi-and-d-n-i

If this is the case, it should work for you to create persistent live drives to work around this bug, when you select 'MSDOS partition table' in the settings menu of mkusb. But it should fail, if you keep the 'default GPT'.

I have access to one old and slow HP computer, a Probook 6450b with an Intel i5-M520 CPU. But it seems to be too new to be affected by this bug. It boots in BIOS mode with this new Groovy iso file.

My son has another and older HP Elitebook, and this bug needed MSDOS partition table to boot via grub from USB, but its graphics stopped working some time ago. Maybe I can make it work via an external screen ...

typo fix in comment #6

this bug needed MSDOS partition table --->
this computer needed MSDOS partition table

I could get a signal to an external screen from the HP Elitebook 8560p.

As I expected, it failed to boot to the current Lubuntu Groovy iso file.

In mkusb's settings menu for a persistent live drive I selected

- MSDOS partition table
- usb-pack-efi

and I could boot to this persistent live system.

So my conclusion is that we forgot about these old HP computers, when we accepted a GPT for the Groovy iso files. They simply refuse to boot via grub and GPT in BIOS mode.

-o-

I am part of this problem: I should have remembered and warned about it, but maybe because my son's computer is broken, it disappeared from my memory until I read this bug report.

Status changed to 'Confirmed' because the bug affects multiple users.

Hi,

so the world might begin to split into incompatible camps.
New Lenovos versus old HPs.
Debian still has the "mac" amd64 ISOs which simply have no EFI equipment
in order to please some old Apple machines:
  https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/

But this GPT allergy would have to be verified by repacking a GPT Ubuntu
ISO to a MBR Ubuntu ISO which the Lenovos seem to hate.
Continuing the repack endeavors of
  https://bugs.launchpad.net/ubuntu-cdimage/+bug/1886148/comments/174
one may do

  original=groovy-desktop-amd64_20201010_gpt.iso

  test -f test_mbr.iso && rm test_mbr.iso

  xorriso \
    -indev "$original" \
    -outdev test_mbr.iso \
    -boot_image any replay \
    -boot_image any appended_part_as=mbr

Partition editors should report MBR/DOS partition table and boot flag

  $ /sbin/fdisk -l test_mbr.iso
  ...
  Disklabel type: dos
  ...
  Device Boot Start End Sectors Size Id Type
  test_mbr.iso1 * 64 5724655 5724592 2.7G 83 Linux
  test_mbr.iso2 5724656 5734607 9952 4.9M ef EFI (FAT-12/16/32)

If this does not boot then it is not about MBR or boot flag, but about
something else.

If it boots, it's maybe only the addiction to MBR boot flags, for which
there is a barely legal solution:
One may create a pseudo partition with boot flag which is on the other
side invisible enough not to disturb GPT recognition.

  test -r test_gpt_bootflag.iso && rm test_gpt_bootflag.iso

  xorriso \
    -indev "$original" \
    -outdev test_gpt_bootflag.iso \
    -boot_image any replay \
    -boot_image any appended_part_as=gpt \
    -boot_image any mbr_force_bootable=on

(If partition editors complain about some Protective MBR mismatch in
 test_gpt_bootflag.iso: I am investigating this. It happens only with
 replay runs. Let's hope it does not influence the outcome ot the test.)

Have a nice day :)

Thomas

Today's daily (Lubuntu 20201011) was written with GNOME-DISKS with same results.

Failure to boot on

hp dc7700 (c2d-e6320, 5gb, nvidia quadro nvs 290)
hp dc7900 (c2d-e8400, 4gb, intel 4 series integrated i915)

but i can boot it on other dell boxes (& more modern HP)

@sudodus
The dc7700 has no booting OS on it; last a failed xubuntu qa-test install but I bet it'd legacy/BIOS/MBR, and I know the dc7900 has legacy/MBR (no GPT) as my boxes are mostly 80-160GB disks & I don't bother with GPT

This morning (CET) I tested a cloned system from a modified iso file according to Thomas Schmitt;

  original=groovy-desktop-amd64_20201010_gpt.iso

  test -f test_mbr.iso && rm test_mbr.iso

  xorriso \
    -indev "$original" \
    -outdev test_mbr.iso \
    -boot_image any replay \
    -boot_image any appended_part_as=mbr

in the HP Elitebook 8560p. With an external monitor and in recovery mode I could get it to the desktop. Actually it is enough for this debug task to get it to the grub menu, which fails with the original iso file because of the GUID partition table.

So I am sure that we have identified the problem. Thomas Schmitt's method is a second work-around, more direct and maybe more convenient than creating a persistent live drive with mkusb.

A solution would be to revert to the boot structure of 20.04.x LTS, which provides iso files that boot for all Ubuntu family iso files in all amd64 type computers, that we have tested.

@guiverc,

> The dc7700 has no booting OS on it; last a failed xubuntu qa-test install but I bet
> it'd legacy/BIOS/MBR, and I know the dc7900 has legacy/MBR (no GPT) as my boxes are
> mostly 80-160GB disks & I don't bother with GPT

The problem is that the current Groovy *iso files contain a GPT*. This is independent of what you have in the internal drive.

Also, please notice that those old HP computers are willing to boot via grub in GPT in drives connected via SATA (or eSATA). The bug affects boot drives connected via USB.

Hi,

> So I am sure that we have identified the problem.

There is still the possibility that only a boot flag is missed by the
firmware.

So please test also an ISO made by
    ...
    -boot_image any replay \
    -boot_image any appended_part_as=gpt \
    -boot_image any mbr_force_bootable=on

If this does not boot, it could be because of xorriso's inconsistency
between Protective MBR size and GPT size when "replay" is used.
(I am hunting this in the code since yesterday.)

So in case of failure, try a method which is very near to the way how
original Ubuntu ISOs get built.
The number 5725652 is the start block of the original EFI partition in the
ISO from 10 october. If you use a different ISO as base, you need to ask
xorriso or a partition editor for the start block of partition 2 and use
that number. The partition size 9952 seems to be stable over all new
Ubuntu ISOs i have seen in the last days:

  original=groovy-desktop-amd64_20201010_gpt.iso
  mount_dir=/mnt/iso

  mbr=groovy-desktop-amd64_20201010_gpt.mbr.img
  efi=groovy-desktop-amd64_20201010_gpt.efi.img
  efi_part_start=5725652

  dd if="$original" of="$mbr" bs=512 count=1
  dd if="$original" of="$efi" bs=512 skip="$efi_part_start" count=9952

  sudo mount "$original" "$mount_dir"

  xorriso -as mkisofs \
     -o test.iso \
     -J -joliet-long -l \
     -b boot/grub/i386-pc/eltorito.img \
        -no-emul-boot -boot-load-size 4 -boot-info-table --grub2-boot-info \
     --grub2-mbr "$mbr" \
     -append_partition 2 0xef "$efi" \
     -eltorito-alt-boot \
     -e --interval:appended_partition_2:all:: \
        -no-emul-boot \
     -partition_offset 16 -r \
     -appended_part_as_gpt \
     --mbr-force-bootable \
     -no-pad \
     "$mount_dir"

Have a nice day :)

Thomas

You are a wizard Thomas,

This tweak:

  original=groovy-desktop-amd64_20201010_gpt.iso

  test -f test_mbr.iso && rm test_mbr.iso

  xorriso \
    -indev "$original" \
    -outdev test_mbr.iso \
    -boot_image any replay \
    -boot_image any appended_part_as=gpt \
    -boot_image any mbr_force_bootable=on

makes the cloned USB drive work both in the HP Elitebook 8560p and the Lenovo V130 :-)

Hi,

> You are a wizard Thomas,

I lurk at boot loader mailing lists since more than a decade.

But as sysadmin i am lame enough to not knowing how to make ASUS ProWS
C246-ACE EFI firmware 1401 offer me any external storage device for
booting, or how to disable its Secure Boot setting.
I will have to ask the provider of the machine how he installed a
Debian 10 base system to it a few months ago.

(It will be my next workstation. But due to some hardware flaws i began
to use it for kernel experiments. Now the flaws are gone and i should
invest more time into accomodating it to my oldfashioned window manager
desires.)

> -boot_image any mbr_force_bootable=on
> makes the cloned USB drive work both in the HP Elitebook 8560p and
> the Lenovo V130 :-)

That means we should ask Steve Langasek to add option

  --mbr-force-bootable

to the xorriso -as mkisofs command that is used for ISO production.

Then the waiting for complaints will begin again.

From comment #12:
> A solution would be to revert to the boot structure of 20.04.x LTS,

That would mean to revive ISOLINUX as BIOS boot loader.
I assume there were compelling reasons to switch to GRUB for that job.

(My current theory for Lenovo V130 is that its firmware recognizes
ISOLINUX isohybrid and then drops its usual demand for valid GPT.
The V14 could have the flaw that it offers to disable Secure Boot but
nevertheless fails if no Secure Boot signature is found.
I am not aware that we got confirmation that V14 boots with old Ubuntu
ISOs. We have a report in #176 of 1886148 that Manjaro and openSUSE fail.)

Have a nice day :)

Thomas

Hi Steve,

I hope there is still time to improve booting of Groovy iso files, to make also it work in old HP computers in BIOS mode. Thomas Schmitt suggested a modification of the iso files, and according to my tests, it works without destroying anything for other computers.

Hi,

this is a one-time tangent about the xorriso replay run which i advised
in comment #10.

I found at least the trigger of the xorriso replay flaw about the size
values in ISO 9660 superblock, protective MBR, and GPT header block.
It is the multi-session emulation for overwritable media which is not
enabled when xorriso works by its mkisofs emulation.

A xorriso replay run for an ISO which was made by the mkisofs emulation
should disable the multi-session emulation by command

  -compliance no_emul_toc

in order to get the same result (and in order to avoid the bug which i
still have to catch).

Further i advise for machines with lots of RAM to use
  -stdio_sync off
which avoids the periodic waiting for the hard disk to have taken all
written data and the end sync of the mkisofs emulation.
(Whose RAM is fast enough to beat 1200x DVD speed ?)

So a repack run which shall yield consistent size values in ISO 9660, MBR,
and GPT is currently advised to look like:

  test -r test_gpt_bootflag.iso && rm test_gpt_bootflag.iso

  xorriso \
    -stdio_sync off \
    -compliance no_emul_toc \
    -indev "$original" \
    -outdev test_gpt_bootflag.iso \
    -boot_image any replay \
    \
    -boot_image any appended_part_as=gpt \
    -boot_image any mbr_force_bootable=on

with the commands after "-boot_image any replay" being the modifications of
the imported boot setup of "$original". (See man xorriso, -boot_image)

(I added
  -boot_image anyappended_part_as=gpt
although the 20201010 ISO brings this setting, because the proposal shall
also work with ISOs like 20201007.1 which have MBR partition table.)

Have a nice day :)

Thomas

Thank you @Thomas Schmitt (scdbackup) & @sudodus

Using today's Lubuntu groovy ISO 2020.10.12, I ran
(based on @sudodus' comment #15)
--
#!/bin/bash
  original=groovy-desktop-amd64.iso
  test -f test_mbr.iso && rm test_mbr.iso
  xorriso \
    -indev "$original" \
    -outdev test_mbr.iso \
    -boot_image any replay \
    -boot_image any appended_part_as=gpt \
    -boot_image any mbr_force_bootable=on
--
to create new 'test_mbr.iso'. It was written with
`sudo dd if=test_mbr.iso of=/dev/sdb bs=4M oflag=sync

it booted fine on (not impacted by this bug)

hp 8200 elite sff (i5-2400, 8gb, nvidia quadro 600)
dell [optiplex] 755 (c2d-e6850, 5gb, amd/ati radeon rv516/x1300/x1550)
dell [optiplex] 755 (c2d-e8300, 8gb, amd/ati radeon rv610/radeon hd2400 pro/xt)
(it appeared to boot slow on 755-8 or 2nd 755, but subjective)
dell [optiplex] 780 (c2q-q9400, 8gb, amd/ati cedar radeon hd 5000/6000/7350/8350)

HOWEVER it did NOT boot on

hp dc7700 (c2d-e6320, 5gb, nvidia quadro nvs 290)
hp dc7900 (c2d-e8400, 4gb, intel 4 series integrated i915)

Please feel free to verify what I did & highlight any mistakes.
(keep instruction dead-simple for me please)

On Mon, Oct 12, 2020 at 09:20:14AM -0000, Thomas Schmitt wrote:
> That means we should ask Steve Langasek to add option

> --mbr-force-bootable

> to the xorriso -as mkisofs command that is used for ISO production.

> Then the waiting for complaints will begin again.

I have applied this change to the build scripts and triggered a manual build
of the Ubuntu Desktop daily image, which should be done soon at
<http://cdimage.ubuntu.com/ubuntu/daily-live/20201013/>.

Hi,

> xorriso \
> ...
> -outdev test_mbr.iso \
> ...
> -boot_image any appended_part_as=gpt \
> ...
> HOWEVER it did NOT boot on
> hp dc7700 (c2d-e6320, 5gb, nvidia quadro nvs 290)
> hp dc7900 (c2d-e8400, 4gb, intel 4 series integrated i915)

The only flaw i see is that the ISO's name "test_mbr.iso" does not match
its partition table type "gpt". But that is not visible any more when the
ISO is on the USB stick.

So these machines probably really don't like the GPT.

What happens if you give them an ISO with MBR partition table:

  xorriso \
    -stdio_sync off \
    -compliance no_emul_toc \
    -indev "$original" \
    -outdev test_mbr.iso \
    -boot_image any replay \
    \
    -boot_image any appended_part_as=mbr

Have a nice day :)

Thomas

@ Chris,

I was almost sure, that the bug was squashed. But I cannot see anything wrong with what you describe in comment #19, so it seems there more to do until we have iso files that can boot all our computers (if at all possible before the release of 20.10).

Anyway, I look forward to both your results and my result from testing the old HP computers when cloning from the new iso file by Steve Langasek according to comment #20.

Finally, I look forward to your test results from an iso file with 'mbr' (dos partition table) according to Thomas Schmitt in comment #21.

The current daily Ubuntu Desktop iso file

-rw------- 1 sudodus sudodus 2938152960 okt 13 06:25 groovy-desktop-amd64.iso

$ sha256sum groovy-desktop-amd64.iso
a1ee702fd38036ac3b22a34657c94a9f1f7cb476bc13b4712ee34dad20499fc2 groovy-desktop-amd64.iso

$ md5sum groovy-desktop-amd64.iso
541b509f09d79da944275dc2eaeed137 groovy-desktop-amd64.iso

when cloned does *not* boot in my son's HP Elitebook 8560p (It boots in the Lenovo V130).

-o-

I double-checked that the tweaked iso file, that worked yesterday (according to comment #15) works also today. I cloned it again to the same USB pendrive, that failed with the current daily Ubuntu iso file. And it works in both the HP Elitebook 8560p and the Lenovo V130.

The working iso file was tweaked from

original=groovy-desktop-amd64_20201010_gpt.iso

which was pointing to the daily Ubuntu Desktop iso file dated 2020-10-10. I have saved that file now as a separate file (so that it will not be overwritten at the next zsync). And I have verified that I can create a drive from it via the tweak in comment #15.

Please notice that the option

-compliance no_emul_toc

may cause problems. Anyway, the tweak in comment #21 failed for me.

-o-

I also tested the tweak in comment #15 on the current Ubuntu Groovy iso file. When cloned the USB drive fails to boot the Lenovo V130, maybe because something is 'set twice'. There is not even any 'Linpus Lite' boot option in the temporary menu. (It boots in the HP Elitebook 8560p.)

The output of gdisk is changed from

$ sudo gdisk groovy-desktop-amd64.iso
GPT fdisk (gdisk) version 1.0.3

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.

to

$ sudo gdisk test_mbr.iso
GPT fdisk (gdisk) version 1.0.3

Partition table scan:
  MBR: MBR only
  BSD: not present
  APM: not present
  GPT: present

Found valid MBR and GPT. Which do you want to use?
 1 - MBR
 2 - GPT
 3 - Create blank GPT

-o-

@ Thomas,

Would it be useful for you, if I upload the modified iso file that works both in the Lenovo V130 and in the HP Elitebook 8560p?

Or can you create such an iso file yourself and do the analysis of its head end?

It seems I was too early to test the new current daily Ubuntu Desktop iso file. Now I found this one:

$ ls -l groovy-desktop-amd64.iso
-rw------- 1 sudodus sudodus 2894153728 okt 13 08:36 groovy-desktop-amd64.iso

$ sha256sum groovy-desktop-amd64.iso
3b92cb0d427998b5a7f58377d0204ef56e3f21e8748331bef4005f8f244e814f groovy-desktop-amd64.iso

and it works much better than the previous one :-)

When cloned this iso file boots in my son's HP Elitebook 8560p and also in the Lenovo V130 (in UEFI mode with secure boot). To make the test more general I checked that it boots in the Dell Precision M4800 too (and it works in BIOS mode and UEFI mode).

Hi,

sudodus wrote:
> I also tested the tweak in comment #15 on the current Ubuntu Groovy iso file.
> When cloned the USB drive fails to boot the Lenovo V130,
> ...
> $ sudo gdisk test_mbr.iso
> MBR: MBR only

Obviously the production of "test_mbr.iso" did what the file name says
and made a MBR partition table. Your reported run in comment #15 is
supposed to produce GPT for the Lenovos.

> Anyway, the tweak in comment #21 failed for me.

That one is aimed to produce an MBR table in order to let Chris Guiver
test whether this works with his HPs.
We know that this does not work for the Lenovos.

> maybe because something is 'set twice'.

Hardly. If settings in xorriso are in conflict, the later command overrides
the setting of the earlier command.
E.g. in the comment #15 with groovy-desktop-amd64_20201010_gpt.iso as base
  -boot_image any replay
sets appended_part_as=mbr, but the later command
  -boot_image any appended_part_as=gpt
overrides this.
When the production of the ISO filesystem begins, the setting is "gpt".

> Please notice that the option -compliance no_emul_toc may cause problems.

Do you have any tangible indications for this (beyond the fact that MBR
table does not work for the Lenovos) ?

> Would it be useful for you, if I upload the modified iso file

It would suffice to show the exact xorriso run arguments, the URL of
the original ISO, and the inspecting program runs of xorriso or partition
editors with their results.

-------------------------------------------------------------------------

Whatever, we need to know whether Chris' hp dc7700 and hp dc7700 boot
by an MBR partition table. See comment #21.

Further we need confirmation that they still not boot with the newest
daily groovy ISO. (The one with GPT which has the boot flag attributed to
a nearly non-existing second MBR partition.)

If it turns out that the HPs only boot via MBR table, and if Ubuntu still
wants to have its ISO booting on those machines and on Lenovos, then i
fear that two ISO variations are needed:

- For most machines: GPT without that fabricated boot flag, which is a
  hack that is barely UEFI specs compliant.

- For the HPs and maybe other old machines: MBR partition table.
  (That variant has a boot flag by default.)

Most machines will work with both ISOs.

As said, Debian does similar for machines which are allergic to the normal
Debian EFI boot equipment. (The "mac" netboot ISOs are simply BIOS-only.
No special Apple parts inside.)

If only a single ISO is possible, then it should stay as it is now (GPT and
boot flag hack) which tosses some of the HPs out of the game.
(We now know how a user can repack that ISO to a HP-friendly one.)

Have a nice day :)

Thomas

Hi Thomas,

> > Please notice that the option -compliance no_emul_toc may cause problems.

> Do you have any tangible indications for this (beyond the fact that MBR
> table does not work for the Lenovos) ?

I wrote maybe (which includes maybe not). I may have drawn faulty conclusions when I thought that the previous Ubuntu Groovy iso file of today was the 'real thing', with modified boot structure. I am really not sure, and I will do more testing, only if you think it is important. In that case, please advice what to look for. Otherwise, now that the current daily Ubuntu Desktop iso file works in both the HP Elitebook 8560p and also in the Lenovo V130, it may not be necessary.

Likewise, now we have an uploaded iso file (the current daily Ubuntu Desktop iso file), that [when cloned] works in the HP Elitebook 8560p and also in the Lenovo V130. For that reason I don't think we need to bother with 'my' iso file tweaked according to comment #15.

I can confirm that my Lenovo V14 IIL boots successfully on todays Ubuntu Groovy ISO -

http://cdimage.ubuntu.com/daily-live/20201013.1/groovy-desktop-amd64.iso

in UEFI+secure boot and BIOS (legacy) modes. I have not had any problems with any of my other boxes.

The current Lubuntu Groovy iso file

$ ls -l groovy-desktop-amd64.iso
-rw------- 1 sudodus sudodus 1894162432 okt 13 16:56 groovy-desktop-amd64.iso

$ sha256sum groovy-desktop-amd64.iso
2bc2b6bf2a14a4a293ce5280e754ed9ef4913873a529c99bfe0bcfe222aff2b5 groovy-desktop-amd64.iso

when cloned does *not* boot in my son's HP Elitebook 8560p but it boots in the Lenovo V130.

This does *not* match the behaviour of the current Ubuntu Groovy iso file, that works also in the HP Elitebook 8560p.

-o-

The current Lubuntu iso file '20201013' was created later than that of Ubuntu '20201013.1', but the number in the ISO tracker is lower. So I guess that the Lubuntu iso file lacks the final tweak of the boot structure (adding mbr flag: --mbr-force-bootable).

Lubuntu QA-tests using groovy daily 20201013
(iso written with `dus` or mkusb live)

Boots & completed install on

- hp dc7700 (c2d-e6320, 5gb, nvidia quadro nvs 290)

but FAILS TO BOOT on

- hp dc7900 (c2d-e8400, 4gb, intel 4 series integrated i915)

It's been a ~three hours since my last comment (#29)
and the thumb-drive no longer boots on
- hp dc7700 (c2d-e6320, 5gb, nvidia quadro nvs 290)

I booted it with *live* to test for http://launchpad.net/bugs/1899719 where I quote myself on saying ot occurred on 3 boxes; the purpose of this boot was to see if this box was one of them... but it won't boot??

I ejected thumb-drive, re-inserted, flashing of thumb-drive indicates to me (subjectively) it's being read, then ignored & hdd of 'recent' install loads (and fast-flashing stops on thumb-drive)

thumb-drive boots on other devices still, tested on
- hp 8200 elite sff (i5-2400, 8gb, nvidia quadro 600)

Hi Chris,

Please test Ubuntu Desktop (standard Ubuntu) Groovy version dated Okt 13 08:36 from zsync,

- '20201013.1' in the testing tracker

$ ls -l groovy-desktop-amd64.iso
-rw------- 1 sudodus sudodus 2894153728 okt 13 08:36 groovy-desktop-amd64.iso

$ sha256sum groovy-desktop-amd64.iso
3b92cb0d427998b5a7f58377d0204ef56e3f21e8748331bef4005f8f244e814f groovy-desktop-amd64.iso

This iso file works for me, also in my son's old HP Elitebook 8560p (where the current Lubuntu iso file fails).

If the booting is flaky try either to shutdown and wait for several seconds before booting *or* reboot. (The computer may act differently in these two cases.)

@ Steve Langasek (vorlon),

The current Ubuntu Desktop (standard Ubuntu) Groovy version dated Okt 13 08:36 from zsync,

- '20201013.1' in the testing tracker

$ ls -l groovy-desktop-amd64.iso
-rw------- 1 sudodus sudodus 2894153728 okt 13 08:36 groovy-desktop-amd64.iso

$ sha256sum groovy-desktop-amd64.iso
3b92cb0d427998b5a7f58377d0204ef56e3f21e8748331bef4005f8f244e814f groovy-desktop-amd64.iso

when cloned to USB can boot my son's old HP Elitebook 8560p.

The current Groovy iso files of Lubuntu and Xubuntu have a later date and time (from zsync), yet they fail to boot the HP Elitebook 8560p.

Is this because the latest tweak suggested by Thomas Schmitt is only present in the Ubuntu Groovy iso file, or is there some other cause of this difference?

The current Ubuntu Desktop (standard Ubuntu) Groovy version dated Okt 14 08.30 from zsync,

- '20201014' in the testing tracker

$ sha256sum groovy-desktop-amd64.iso
20b9b69600470e6641406a6ab975f985903bde4555b552f3155693aa10a862a1 groovy-desktop-amd64.iso

when cloned to USB can boot an old HP Elitebook 8560p as well as a newer Lenovo V130 (like the previous iso file).

On Wed, Oct 14, 2020 at 08:04:19AM -0000, sudodus wrote:
> The current Groovy iso files of Lubuntu and Xubuntu have a later date
> and time (from zsync), yet they fail to boot the HP Elitebook 8560p.

> Is this because the latest tweak suggested by Thomas Schmitt is only
> present in the Ubuntu Groovy iso file, or is there some other cause of
> this difference?

All Ubuntu flavors use the same bootloader logic; there is no distinction
here.

And I have checked that the latest xubuntu image has the same partition
setup.

Ubuntu 20201014:

$ fdisk groovy-desktop-amd64.iso

Welcome to fdisk (util-linux 2.34).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Command (m for help): p
Disk groovy-desktop-amd64.iso: 2.76 GiB, 2939303936 bytes, 5740828 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: B4C6B0BE-D645-4E64-8EB9-B4C4C650029D

Device Start End Sectors Size Type
groovy-desktop-amd64.iso1 64 5730211 5730148 2.7G Microsoft basic data
groovy-desktop-amd64.iso2 5730212 5740163 9952 4.9M EFI System
groovy-desktop-amd64.iso3 5740164 5740763 600 300K Microsoft basic data

Command (m for help): M
Entering protective/hybrid MBR disklabel.

Command (m for help): p
Disk groovy-desktop-amd64.iso: 2.76 GiB, 2939303936 bytes, 5740828 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Device Boot Start End Sectors Size Id Type
groovy-desktop-amd64.iso1 1 5740827 5740827 2.8G ee GPT
groovy-desktop-amd64.iso2 * 0 0 1 512B 0 Empty

Partition table entries are not in disk order.

Command (m for help): q

$

Xubuntu 20201014:

$ fdisk groovy-desktop-amd64.iso

Welcome to fdisk (util-linux 2.34).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Command (m for help): p
Disk groovy-desktop-amd64.iso: 1.71 GiB, 1814708224 bytes, 3544352 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 5F95C427-857C-4F99-A8DA-0BDAD813137F

Device Start End Sectors Size Type
groovy-desktop-amd64.iso1 64 3533735 3533672 1.7G Microsoft basic data
groovy-desktop-amd64.iso2 3533736 3543687 9952 4.9M EFI System
groovy-desktop-amd64.iso3 3543688 3544287 600 300K Microsoft basic data

Command (m for help): M
Entering protective/hybrid MBR disklabel.

Command (m for help): p
Disk groovy-desktop-amd64.iso: 1.71 GiB, 1814708224 bytes, 3544352 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Device Boot Start End Sectors Size Id Type
groovy-deskto...

Hi Thomas,

> All Ubuntu flavors use the same bootloader logic; there is no
> distinction here.
>
> And I have checked that the latest xubuntu image has the same
> partition setup.
...
> So a failure to boot Xubuntu where Ubuntu works is not explained
> by missing/different xorriso options.

Thanks Steve,

Then we must conclude that booting the old HP computers is flaky with the current Groovy iso files.

Let us hope that Thomas can help us with another tweak. Otherwise I think that these old HP computers will be left behind at least in Groovy.

-o-

There is evidence that some old Dells might also be affected by this bug. See this link by kansasnoob

https://ubuntuforums.org/showthread.php?t=2447379&page=3&p=13992621#post13992621

---
Neither Latitude E6400's nor an E6510 will even recognize the image on USB. The E6400's are C2D (both P series and T series cpu's fail), and the E6510 has an i7 M620. I've tried creating the live USB's both with Focal's startup disc creator and uNetbootin - no dice! I'll try a DVD tomorrow as time allows. Prior images work fine so it's not hardware settings - usually you just press F12 repeatedly to display the one time boot menu and select USB, but in this case no USB appears to be found.
---

Hi,

i confirm that

  groovy-desktop-amd64.iso1 1 5740827 5740827 2.8G ee GPT
  groovy-desktop-amd64.iso2 * 0 0 1 512B 0 Empty

is the indication that the --mbr-force-bootable hack is applied.
A nearly non-existend MBR partition not overlapping the partition of
type EE and carrying the boot flag. It is not really compliant to the
idea of a Protective MBR in UEFI specs about GPT. But the same specs say
(quoted from UEFI 2.8):
  "A Partition Record that contains an OSType value of zero or
   a SizeInLBA value of zero may be ignored."

As for more tweaks: Since i began to participate in bug 1886148 we tried
everything that makes sense with GRUB for PC-BIOS.
We began with a lame mock-up of the old barely legal layout that was used
with ISOLINUX isohybrid. Then we moved to a neat MBR partition table only
to learn that the Lenovos don't like it. Then we moved to GPT and learned
that several old HPs don't like it. Then we used the hack that was invented
five years ago for https://savannah.gnu.org/bugs/?46716 . This helped with
about half of the HPs ... sometimes.

Now we are out of alternatives, unless we revive ISOLINUX for BIOS and
the multi-hack layout of the released Ubuntu ISOs.

Have a nice day :)

Thomas

Thanks for all the improvements, Thomas and Steve :-)

Although we feel frustrated now, we must remember that the Groovy iso files have improved a lot and can boot many more computers than when we started.

-o-

There are workarounds to make also the remaining computers boot: for example to make a persistent live drive with mkusb and the settings 'msdos partition table' and 'usb-pack-efi' for the old HP computers. I tested right now that it works with the current Lubuntu Groovy and the HP Elitebook 8560p.

Ubuntu MATE Groovy Oct 14, 2020 daily does not boot on HP Pavilion Elite HPE series Desktop PC.

ISO: groovy-desktop-amd64.iso

Downloaded from: http://cdimage.ubuntu.com/ubuntu-mate/daily-live/current/

sha256sum:
bdf86534a8ef8c353f45a0c82776aaf8b0cc57eac85cc05b9d80b302f9e7cade

I flashed/prepared the USB drive using from Ubuntu MATE Focal 20.04 using:

(1) Etcher 1.5.109 ( Balena Etcher https://github.com/balena-io/etcher/releases ),

and

(2) MultiBootUSB 9.2.0 ( https://github.com/mbusb/multibootusb/releases )

Upon reboot, and selecting the USB drive from BIOS's "One-time boot menu" option,

cursor blinks once or twice in upper left corner of the screen,

and then system displays Ubuntu MATE Focal 20.04 Grub (Grub options are: 2 kernels of Ubuntu MATE Focal and Windows 10).

So far I have tried 2 different USB ports.

Also, using the same USB drive (prepared through either Etcher or MultiBootUSB) I can boot Ubuntu MATE Focal 20.04 on the same HP desktop just fine.

Please let me know, if you want me to run any tests or something.

Thanks.

Lubuntu groovy daily (2020-10-14) is reacting the same as yesterdays (written with `dus` (mkusb live))

first boot on
- hp dc7700 (c2d-e6320, 5gb, nvidia quadro nvs 290)
worked, but all subsequent boots made (reboot, or turning off, waiting & turning back on) have failed.

no tries on
- hp dc7900 (c2d-e8400, 4gb, intel 4 series integrated i915)
booted

This is identical to what occurred yesterday.
(I'll grab latest Ubuntu next which I wasn't able to test yesterday)

Ubuntu daily groovy (2020-10-14) written with `dus` (mkusb in 'live')
responds identically to Lubuntu daily.

First attempt on
- hp dc7700 (c2d-e6320, 5gb, nvidia quadro nvs 290)
worked, but subsequent reboots all failed.
Refer http://iso.qa.ubuntu.com/qatracker/milestones/413/builds/221721/testcases/1303/results
for some details, but I attempted more than I listed on iso.qa.ubu
Different USB ports tried, waits as long as ~90 mins between off & back on for next try were tried as well as many reboots...
Identical results to Lubuntu over last two days.

Also failed on
- hp dc7900 (c2d-e8400, 4gb, intel 4 series integrated i915)

Hi Chris,

There seems to be a difference between the first boot and the subsequent boots in your

hp dc7700 (c2d-e6320, 5gb, nvidia quadro nvs 290)

It might depend on the creation of the ext4 partition behind the image of the iso file, which happens only in the first attempt to boot. Another difference is that the partition is available for reading and writing at once in the following attempts to boot.

If you have the time, you could wipe the whole USB drive (with mkusb, slow process) and then clone from the current iso file again, and when booting add the boot option 'nopersistent' before continuing to boot. A more convenient method would be to replace the cloning with creating a 'nopersistent' boot drive using mkusb-plug (but it replaces 'quiet splash' with 'nopersistent').

This way, the drive should look the same during all subsequent boots. The question is: will the computer boot every time or fail every time? Or will there still be differences between the boot attempts?

Hi Jags Desai,

It is possible that your problem depends on the graphics, if there is nvidia graphics. Have you tried to boot in recovery mode or with the boot option 'nomodeset'?

But if your HP does not even reach grub, you may need a workaround for the boot process. Please try to install mkusb (mkusb-dus) and make with a persistent live drive with the settings

- 'msdos partition table'
- 'usb-pack-efi'

for your HP computer. Using 'msdos partition table' should work in BIOS mode (alias CSM alias legacy mode) but you can expect it to fail in UEFI mode.

Hi @sudodus

(1) The HP desktop have AMD radeon 290X graphics card.

(2) Ubuntu MATE Focal 20.04 with kernel 5.9.0 and 5.8.14 runs just fine on the same HP PVILION eLITE DESKTOP.

(3) Yes, as described in my previous post, when I try to boot the system from Ubuntu MATE Groovy desktop Oct 14, 2020 Live USB (prepared either with Etcher or MultiBootUSB), I do NOT get Grub from Live USB at all.

Cursor blinks once or twice and immediately I get "installed version" of the Grub from Ubunbtu MATE Focal 20.04

(4) Regarding the "mkusb". Is this the correct PPA to install it from?

https://launchpad.net/~mkusb/+archive/ubuntu/ppa

Because I just added the PPA (and added the signing key / fingerprint) but afterwards I only get "mkusb" in Synaptic.

I can not find "mkusb-dus". Or are they one and same package?

(5) I will try once again with mkusb with the settings you've suggested.

Thanks alot.

- Jags

Hi Jags Desai,

Yes, mkusb-dus (alias dus) is part of the mkusb package.

You need the package 'usb-pack-efi' too.

sudo add-apt-repository ppa:mkusb/ppa # and press Enter
sudo apt update
sudo apt install mkusb usb-pack-efi

@sudodus/Nio

- hp dc7700 (c2d-e6320, 5gb, nvidia quadro nvs 290)

thumb-drive was created as requested by @sudodus comment #41
Ubuntu 2020-10-14 groovy daily ISO (same as comment #40)

(1) wiped thumb-drive
(2) `mkusb-plug groovy-desktop-amd64.iso` (option N or non-persistent was selected)

It's been booted TWICE successfully on the box. I'm out of time so will have to continue testing later.

Of note on shutdown: IO errors on shutdown, sdb . async page read
Are those (now) errors what corrupts the normal written `dus groo<tab>` ISO? data is written to device?

@sudodus

Using mkusb-dus / usb-pack-efi to prepare the live usb, I was able to boot the "Persistant Live" of Ubuntu MATE Groovy Desktop 20.10 Oct 14, 2020 ISO.

Though the system booted fine, and I got to see the desktop screen, I have not actually installed the OS yet as I need to free up a partition for that.

Many thanks for all the help.

Hi Chris,

1. In the meantime I tested the current Lubuntu Groovy iso file in the HP Elitebook 8560p.

It boots when I made a nopersistent live drive using mkusb-plug, and I could repeat it.

2. I don't know if the error that you see at shutdown causes failure at the next boot (when there is an ext partition). Maybe. I think we help from Steve or Thomas to dig deeper into that problem.

Hi,

i now am boss of my mainboard ASUS ProWS C246-ACE EFI firmware 1401,
at least as long i do not want to disable Secure Boot.
The hardware provider was of not of help with USB sticks, but remembers
to have inserted Debian 10.4.0 netinst on CD-RW when SSD and HDD were
still without bootable system.

The EFI confused me old BIOS owner by not attributing external devices
a high boot priority. I have to go into the EFI setup and move the
offered drive (USB stick or CD reader) to the top of the list.
Then it boots.

This setting gets forgotten as soon as i boot again.

Next stumblestone was that my large 128 GB stick causes a long delay
with booting, is then not presented as boot device, and does not show
up in a booted Debian 10, if i plug it into a blue USB socket.
In a black socket it works for Linux and is recognized by EFI after i
powered down the machine after unplugging from blue and plugging into
black.
(An old 2 GB stick with Debian 10.4.0 netinst on it works at blue USB
sockets.)

I tested from USB stick successfully up to the GRUB menu:

  groovy-desktop-amd64.iso 20201007.1 MBR-only

  groovy-desktop-amd64.iso 20201010 GPT without boot flag

  groovy-desktop-amd64.iso 20201015 GPT with boot flag

The 20201010 version was also tested from DVD+RW. (There is no difference
between the three ISO as far as the El Torito catalog is concerned.)

Have a nice day :)

Thomas

Hi again Jags Desai,

I'm glad that you managed to boot when using mkusb. When you want to install Ubuntu-MATE, please read the tips at this link:

https://help.ubuntu.com/community/mkusb/persistent#Installing_-_using_the_Install_icon_on_the_desktop

FYI-

I do not have any HP boxes to test on re: bug 1899308 booting prob.

my mixture is dell/lenovo/acer/mac/IntelNuc and InWin - none of these with boot issues.

Further to my comment #46
I booted again (still same thumb-drive) without issue on
- hp dc7700 (c2d-e6320, 5gb, nvidia quadro nvs 290)
and booted successfully on
hp dc7900 (c2d-e8400, 4gb, intel 4 series integrated i915)
(ubuntu groovy daily 2020-10-14, these are not recorded on iso.qa.ubu as it's no longer the latest ISO)

Hi,

Chris Guiver (guiverc) wrote:
> (1) wiped thumb-drive
> (2) `mkusb-plug groovy-desktop-amd64.iso` (option N or non-persistent
> was selected)

Is (2) equivalent to a plain copy by dd or cp ?

> I booted again (still same thumb-drive) without issue on
> - hp dc7700
> and booted successfully on hp dc7900

Did they previously fail with plain copies of the Ubuntu ISOs ?

If so, then it could be that step (1) erased an old GPT backup at the
end of the USB stick, which made the whole stick suspicious when there
was a valid but not matching GPT at its start.
With MBR partition table in the ISOs, there would have been no need to
look for a backup GPT at the end.

This theory would be wrong if the plain copy methods included a wipe
of the last block of the USB stick, as i let xorriso-dd-target do.

@sudodus: Can you confirm or refute this theory from the reports in
this thread ?

If the theory stays valid:
Have those machines, which demanded a boot flag, been tested with no GPT
backup at the end of the USB stick ?

It could be that the boot flag keeps EFI from looking for a backup GPT
and thus from taking offense if it does not match the main GPT.
(I confess that this is far fetched. But we already went through most of
the not so wayward possibilities.)

---

Always valid is the theory about hardware allergies. It is not explainable
to me why my "128 GB" Intenso Ultra Line rev 1.00 does not work in the
USB 3 sockets of my new machine, although it was sold and is advertised as
USB 3.0 device.

Have a nice day :)

Thomas

@ Steve,

Booting 'nopersistent' is a mild tweak, and it seems to improve things a lot.

Please consider if this should be the default (in other words modify the iso file accordingly). At least now for Groovy, so that we can release a set of iso files, that work in the [amd64] computers, that we have tested.

Later on, in the HH series, you can revert back from 'nopersistent' and let the system create an ext4 partition again (in a USB drive behind the image of the iso file), if you think it is an important feature.

@ Thomas,

1. I know that it can improve things to start from a wiped drive, but I don't know why. I know that before you got involved and started to improve the booting, the systems could not create the ext4 partition in my computers, unless the USB drive was wiped, and I *think* this was independent of what was at the end of the drive. Maybe I am wrong there. Anyway see

https://bugs.launchpad.net/ubuntu/+source/casper/+bug/1895329

and for example comment #4.

Earlier today I tested drives *with 'nopersistent' without prior wiping*, and if I understood correctly it made a difference when booting the HP Elitebook 8560p. But the behaviour of cloned drives was flaky, so it is not easy to be sure what happens by chance and what is really reproducible.

I think we should do some more tests before we can be sure of what happens. That way we can find out if the backup partition table at the end of the drive makes a difference in this case.

2. I think USB pendrives are not well standardized. They are tested to work as data storage devices with Windows and MacOS. Booting is not important enough for the manufacturers of USB pendrives to do thorough tests. Maybe the same is true for the USB systems in computers. Anyway most computers can be booted by most USB pendrives, but I have seen and read about many exceptions.

I had an own case when a USB drive did not boot when connected directly to the computer, but was happy to boot the same computer when connected via an external USB hub. It is also my experience that booting is more likely to succeed via USB2 ports than via USB3 ports, and it was a trial and error adventure to find a USB3 hub, that works well with booting.

@ everybody,

In the light of the lasts findings, questions and suggestions, I did some more tests, this time with yesterday's Xubuntu Groovy iso file, that failed booting yesterday.

I think I started testing the cloned USB drive in the Lenovo V130 (which worked), and continued testing in the HP Elitebook 8560p.

Anyway, today I started with a cloned drive (without prior wiping) and testing booting in the HP Elitebook 8560p.

- Successful boot first time, and an ext4 partition was created
- Failed boot after that: the drive was not recognized (could be repeated)

Then I created a 'nopersistent' drive with mkusb-plug in the same drive (without prior wiping) and testing booting in the HP Elitebook 8560p.

- Successful boot first time. No ext4 partition was created
- Successful boot after that (could be repeated)

I conclude that the crucial thing is the existence of the ext4 partition (and maybe other things associated with it - corruption, backup gpt table at the tail end of the drive ...)

Hi,

sudidus wrote:
> Later on, in the HH series, you can revert back from 'nopersistent' and
> let the system create an ext4 partition again (in a USB drive behind the
> image of the iso file),

So the software in the ISO fiddles with the USB stick partitioning ?
That would be a prime suspect for MBR addiction.

How does the stick look in partition editors after this happened ?
Is still a GPT recognized ? Did it mutate to MBR ("dos") ?

If this "persistent" feature stems from the time of Ubuntu ISOs with
ISOLINUX isohybrid, then it had to deal with a MBR partition table and
had to ignore the invalid GPT.
Is it now able to handle the new and neat GPT ?

> 2. I think USB pendrives are not well standardized. They are tested to
> work as data storage devices with Windows and MacOS.

Their only job is to take storage blocks (usually 512 byte per block) and
to give their unaltered content back on demand.
Partitioning and other boot lures are in the data of these blocks.

Problems arise if the USB stick cannot communicate properly with the
computer or the operating system. In my case a "USB 3.0" stick appears
useless at two USB 3 sockets, where an old stick from 2005 does work fine.

> It is also my experience that booting is more
> likely to succeed via USB2 ports than via USB3 ports,

If the port belongs to the main board, then the firmware should be able to
operate it. Less likely if the port belongs to an add-on card.
Hubs add complexity. (But are fine to protect the computer's sockets from
wear-off.)

Have a nice day :)

Thomas

Hi Thomas,

1. Yes, the software in the ISO fiddles with the USB stick partitioning. I think this was introduced in Ubuntu 19.10. I think it is used to log the installer (to help debug failures) and to store crash dumps. (I would be happy to turn it off.)

Please tell me which partition editor commands to run and I will upload the output. You can do the same in your computer and check if there is any difference.

Well, after your tweaks it seems to work again in most computers also with GPT, but for a long time the partition was created, but the ext4 file system failed to be created.

2. I agree with you about how it should be. But my experience is contrary to that in some cases.

Hi,

> Please tell me which partition editor commands to run

One might need superuser autority for reading the device file:

  device=/dev/sdd

  /sbin/fdisk -l "$device"

  xorriso -indev stdio:"$device" -report_system_area plain

Have a nice day :)

Thomas

I've manually run through the steps of casper's persistent partition creation on a local image. The partition creation operation, 'sfdisk --no-reread -q $DEVICE -a", properly creates the partition on the GPT, not on the MBR.

However, in the process it also "fixes" the protective MBR, removing the empty bootable partition.

We could adjust casper to recreate this MBR entry after creating the GPT partition.

Hi Thomas,

The output of the partition editor commands is in the attached file.

Hi,

Steve Langasek (vorlon) wrote:
> I've manually run through the steps of casper's persistent partition
> creation [...]
> it also "fixes" the protective MBR, removing the empty bootable partition.

That explains why the machines which demand the boot flag boot exactly
once.

sudodus (nio-wiklund) wrote :
> p-ed-commands.txt

In the cases 4, 7, 11, 12, and 14 the bootable but nearly non-existing
partition has vanished, because its MBR slot became all 0.
This matches what Steve found out.

-----------------------------------------------------------------------

> https://code.launchpad.net/~ubuntu-installer/casper/+git/casper/+merge/392318

> sfdisk --no-reread -q $DEVICE -A 1 -X dos || true

If this attributes the boot flag to the first partion of type EE, then
this is a semi-violation of EFI specs and known to be punished by some
firmwares. UEFI 2.8 says:

  "Table 20. Protective MBR Partition Record protecting the entire disk
   BootIndicator 0 1 Set to 0x00 to indicate a non-bootable partition
                           [...]
                           Must be ignored by UEFI implementations."

It turned out back in 2015 that some EFIs interpret the latter statement
as instruction to ignore the device if BootIndicator is present in the
only existing partition. That's why --mbr-force-bootable was invented to
mark a nearly non-existent partition as bootable.

The changelog description is wrong in these aspects:

> "If our partition table is GPT, we *may* have a protective MBR;"

We *must* have a protective MBR to have a valid GPT.

> "we mark the iso9660 partition as bootable"

The ISO 9660 partition is number 1 in the GPT. Partition 1 in MBR is not
mountable as ISO 9660 because it starts at LBA 1. 0 or 64 would be ok for
mounting, but then it would not mark a valid GPT any more.

-----------------------------------------------------------------------

Casper needs to bring these bytes (decimal) into bytes 462 to 477 of
the ISO image (i.e. the MBR):

  128 0 1 0 0 0 1 0 0 0 0 0 1 0 0 0

With bash i could do this by help of dd if=/dev/zero and echo -n $'\x80'
$'\x01'. But with dash ...

Would a temporary 16-byte binary file by acceptable for Casper ?

Extract it before manipulation of the original GPT of the ISO:

  dd if="$DEVICE" bs=1 skip=462 count=16 of=/tmp/boot_flag_bytes_$$

and instead of the sfdisk -A 1 -X dos, insert the extracted file:

  dd if=/tmp/boot_flag_bytes_$$ conv=notrunc bs=1 seek=462 of="$DEVICE"

-----------------------------------------------------------------------

Have a nice day :)

Thomas

On Thu, Oct 15, 2020 at 05:31:08PM -0000, Thomas Schmitt wrote:

> > sfdisk --no-reread -q $DEVICE -A 1 -X dos || true

> If this attributes the boot flag to the first partion of type EE, then
> this is a semi-violation of EFI specs and known to be punished by some
> firmwares. UEFI 2.8 says:
>
> "Table 20. Protective MBR Partition Record protecting the entire disk
> BootIndicator 0 1 Set to 0x00 to indicate a non-bootable partition
> [...]
> Must be ignored by UEFI implementations."
>
> It turned out back in 2015 that some EFIs interpret the latter statement
> as instruction to ignore the device if BootIndicator is present in the
> only existing partition. That's why --mbr-force-bootable was invented to
> mark a nearly non-existent partition as bootable.
>
> The changelog description is wrong in these aspects:
>
> > "If our partition table is GPT, we *may* have a protective MBR;"
>
> We *must* have a protective MBR to have a valid GPT.
>
> > "we mark the iso9660 partition as bootable"
>
> The ISO 9660 partition is number 1 in the GPT. Partition 1 in MBR is not
> mountable as ISO 9660 because it starts at LBA 1. 0 or 64 would be ok for
> mounting, but then it would not mark a valid GPT any more.
>
> -----------------------------------------------------------------------
>
> Casper needs to bring these bytes (decimal) into bytes 462 to 477 of
> the ISO image (i.e. the MBR):
>
> 128 0 1 0 0 0 1 0 0 0 0 0 1 0 0 0
>
> With bash i could do this by help of dd if=/dev/zero and echo -n $'\x80'
> $'\x01'. But with dash ...

This works under dash (and bash):

echo -n '\0200\00\01\00\00\00\01\00\00\00\00\00\01\00\00\00' | dd of="$DEVICE" bs=1 seek=462 conv=notrunc

I'll switch to using that.

Thanks,
--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer https://www.debian.org/
<email address hidden> <email address hidden>

Also seems to effect Dell Latitude E6400 with C2D CPU's and an E6510 with i7 M620 CPU laptops using 20201015 Ubuntu image created with Startup Disk Creator in 20.04. The same USB with the same image works OK in my old Intel Desktop so I'm reasonably certain it's hardware specific.

Typical behavior for these Latitude laptops is needing to press F12 while the BIOS screen is displayed and a boot menu will be displayed. Doesn't matter if you have USB set as first boot option in BIOS - sort of a buggy BIOS I guess, but in this case the USB is not even recognized with a recent 20.10 Live USB inserted.

If I go ahead and boot into an installed Ubuntu system on one of these effected systems with the USB still inserted a message is displayed saying:

 "Error mounting /dev/sdb3 at /media/lance/Ubuntu 20.10 amd64: Command-line `mount -t "iso9660" -o "uhelper=udisks2,nodev,nosuid,uid=1000,gid=1000,iocharset=utf8,mode=0400,dmode=0500" "/dev/sdb3" "/media/lance/Ubuntu 20.10 amd64"' exited with non-zero exit status 32: mount: wrong fs type, bad option, bad superblock on /dev/sdb3,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

If in fact that USB drive is at fault it's odd that it will boot in my old desktop computer, but I will try a different drive just to be certain. I'm really at a loss as what else to try at this point.

Ubuntu daily (2020-10-15)
written to thumb-drive using `gnome disks`

First boot works on
- hp dc7700 (c2d-e6320, 5gb, nvidia quadro nvs 290)

But subsequent reboots, even off & back on fail on same box and another box
- hp dc7700 (c2d-e6320, 5gb, nvidia quadro nvs 290)
- hp dc7900 (c2d-e8400, 4gb, intel 4 series integrated i915)
(see http://iso.qa.ubuntu.com/qatracker/milestones/413/builds/221768/testcases/1303/results/ for specifics of tests performed)

Thumb-drive was written again (2020-10-15) this time with my usual `dus` (mkusb), and it was tried first on
- hp dc7900 (c2d-e8400, 4gb, intel 4 series integrated i915)
After initial boot, a second re-try on same box failed
- hp dc7900 (c2d-e8400, 4gb, intel 4 series integrated i915)

This bug was fixed in the package casper - 1.455

---------------
casper (1.455) groovy; urgency=medium

  * Use a tmpfs in tests; overlay is not compatible with zfs.
  * Make tests work when host kernel does not match that in the test env.
  * Properly handle echo -e for dash vs busybox in initramfs.
  * Limit our dd to only change 16 bytes in the MBR regardless of input,
    to avoid major damage from any future shell bugs.

 -- Steve Langasek <email address hidden> Fri, 16 Oct 2020 09:51:20 -0700

The image at http://cdimage.ubuntu.com/ubuntu/daily-live/20201016.1/ has been built with this new casper.

As time is short before release, please test this asap and let us know of any problems. This new casper passes its autopkgtests, so it does successfully modify the partition table of the booted image; hopefully that means we've covered all the scenarios now.

http://cdimage.ubuntu.com/ubuntu/daily-live/20201016.1/groovy-desktop-amd64.manifest still shows casper 1.452

Is that correct or is the manifest just not updated? It's still not working on the Dell latitude E6510 and E6400 previously mentioned. On the E6510 I tried both legacy and UEFI modes and no go in either (USB not even detected). The E6400 does not offer UEFI as an option.

Sorry, not sure why casper is still out-of-date there. Respinning now.

Note that this only changes the behavior on SECOND boot of the image. If you have systems that wouldn't boot the previous image at all, this doesn't do anything to correct that.

Lubuntu daily (2020-10-16 ISO)

first boot worked on
- hp dc7700 (c2d-e6320, 5gb, nvidia quadro nvs 290)
but subsequent boots on box failed.

casper is 1.452 still on lubuntu daily :(

http://cdimage.ubuntu.com/ubuntu/daily-live/20201016.4/groovy-desktop-amd64.iso has now been built with the correct casper.

I am not respinning a lubuntu test image for this issue, since the boot logic is the same across all flavors. Please test specifically with the above image, not with other flavors.

Worst case scenario on those older Latitude laptops a DVD still works fine.

Ubuntu groovy daily (2020-10-16.4) written (using mkusb/dus)

booted first on
- hp dc7700 (c2d-e6320, 5gb, nvidia quadro nvs 290)

shutdown, then booted again twice on same dc7700 box

then thumb-drive taken to
- hp dc7900 (c2d-e8400, 4gb, intel 4 series integrated i915)
and it booted

http://iso.qa.ubuntu.com/qatracker/milestones/413/builds/221858/testcases/1303/results/

(I hadn't read emails or noticed above when I tested Lubuntu sorry; comment #70)

Although not intentionally so, this now works fine on both the E6400 and E6510 laptops. Both initial boots, and subsequent boots. Just works perfect.

The current Ubuntu Desktop (standard Ubuntu) Groovy version dated Okt 17 08:30 from zsync,

- '20201017' in the testing tracker

ls -l groovy-desktop-amd64.iso
-rw------- 1 sudodus sudodus 2942062592 okt 17 08:30 groovy-desktop-amd64.iso

$ sha256sum groovy-desktop-amd64.iso
f572ff624215e00b398233be6a9e482d41b706c24266ed05bd019a6ab75bf53d groovy-desktop-amd64.iso

when cloned to a USB3 pendrive, can boot

- an old HP Elitebook 8560p (in BIOS mode) - This was problematic in the previous daily iso file, it booted only the first time. Now it booted successfully 3 times in a row. The 300K partition is preserved and the ext4 partition is created correctly (and preserved between boots).

- a middle-aged Dell Precision M4800 (in BIOS mode and UEFI mode) - Works as usual.

- a newer Lenovo V130 (in UEFI mode with secure boot) - Works since GUID partition table.

-o-

I tested with a second USB drive, this time an SSD connected via a USB3 to SATA adapter and cloned from the iso file.

- I started booting the Lenovo V130 in UEFI mode with secure boot (and let it create the ext4 partition). The 300K partition is preserved and the ext4 partition is created correctly.

- Then I tested with the previously problematic HP Elitebook 8560p (in BIOS mode). It booted as it should.

-o-

It looks like you have squashed this bug. Thanks Thomas and Steve :-)

The current Ubuntu Desktop Groovy iso file works with both the old HPs and old Dells and the newer computers that we have tested.

However there are many other computer brands and models, that are not tested yet. Let us hope that there will be no severe backlash.

I tested also the current Xubuntu Groovy version dated Okt 17 02:07 from zsync,

Conclusion: This bug seems squashed with casper version 1.455 :-)