Linux fails to read GPT(EFI) partition table created by both GNU parted and OSX diskutil

Eugene,
      Please run
apport-collect -p linux 538100
form the teerminal so that we can gather and post logs to this bug.

Thanks!

~JFo

I must mention that I am using Lucid daily AMD64 ISO for testing and looks like apport is currently pretty broken.
What information may be helpful?

Meanwhile I'll attach kern.log.

Reconstructed on Karmic i386 (2.6.31-14-generic).

Eugene,
    It may be necessary to run 'sudo apt-get install apport' I had another user experience issue with apport today and running the install seems to have sorted him.

Thanks!

~JFo

Jeremy: What do you mean by install? In case you mean finish install and then report from fresh installation, then it's impossible since system may not be formatted...

Apport is still broken in latest daily ISO.
It crashes with python errors in parsing urls or something (under user and root).
Updating apport from repository fails also due to readonly root.

Anyways, I think issue might be related to:
1. http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-lucid.git;a=commit;h=949c2699b38575324dd8522af6348b63df3e72dd
2. http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-lucid.git;a=commit;h=f28d4f1e0a524ed8ec7039059ea6bc6a4b25a88b

Looks like second commit pretty destroys improvements first commit adds.

BTW: It might not looks like, but that bug is a show-stopper for many mac and 2TB+ storage users.

I think I've localized the problem.
Looks Linux unable to deal with GPT pure partition tables (without synced MBR).
In past a patch (gptsync.patch) was added to parted to solve that issue. (Not sure ubiquity should enjoy from that fix also)
But it looks like parted refuses to sync GPT and MBR, even tried to force it by "PARTED_GPT_APPLE=1 parted".

Above also correct for OSX Graphical Disk Utility. Console version of OSX diskutil does create GPT with synced MBR.

Since the issue is not related to specific system, incomplete status should be removed.
Hope there is a solution floating in upstream...

It's not out of the question that this is actually a parted bug. I merged the Apple gptsync patch up to 2.2 more or less blindly, and I could have broken something.

I think you're analysing this incorrectly, though. What I'm seeing in the kernel is that you need a valid *protective* MBR, as specified in EFI. The gptsync patch in parted is to deal with the fact that Apple diverged from the GPT spec in requiring (approximately) a copy of the first four GPT partitions in MBR-speak rather than a dummy protective MBR whose only purpose is to say "hi, this is really GPT", so on Apple systems (only), parted has to act differently. A synced MBR is usually also a valid protective MBR as far as the kernel is concerned, but only if you have an EFI System Partition.

Therefore, I think the problem might well be that you haven't created an EFI System Partition, only a random FAT partition. Could you try this modification to your recipe?

  mklabel gpt
  mkpart primary fat 0 50
  set 1 boot on
  quit

See also bug 538536 for similar installer problems.

Colin: Thanks for prompt respond.

Additional testing showed that problem is more severe then I assumed initially.
Testcase:
1) Zero HDD header.
2) Format HDD using OSX installer (3 partitions: fat32@EFI-Boot, hfs+@OSX, fat32@LINUX).
3) Reboot into OSX, partition table looks perfect.
3) Reboot into Daily Lucid. parted and fdisk shows expected output, correct Hybryd GPT+MBR
4) Use parted, ubiquity or partman to modify type and format of LINUX partiton.
5) Linux instantly reports inability to recognise partition table, partman and ubiquity fails.
    fdisk shows empty MBR. parted shows expected partitions layout but not sure about binary correctness.
6) Reboot and find that OSX cannot boot also!

Conclusion: parted and friends in Lucid makes GPT partition table not usable. System unbootable and data loss!

We will have to split the problem:
1) I suppose there is no reason why Linux shouldn't be able to work with bare GPT volumes. That part should be forwarded upstream for inspection.
2) parted and friends must be fixed ASAP. In their current state, they are source of destruction in GPT/EFI environments. We may consider merging that part with https://bugs.launchpad.net/bugs/538536

I have tested GPT on non Mac system (VirtualBox).
1) Used gparted LiveCD to create GPT with 4 partitions. As result MBR contained protective MBR with single partition covering whole suface.
2) Booted in to latest Daily Lucid. Instructed installer to change type of 1 of 4 partitions and use 3 of 4 availbale.
3) Ubiquity finished successfuly.
4) Rebooted into fresh system.
5) Found that GPT is ok but MBR now contain 2 partitions:
    First one was previosly available protective
    Second was: Bootable, start=1, end=1, size=0, id=0 and type=Empty. What is that?

From all above I can conclude that issue is severe on Macs but non-Macs are not too safe either.
I think gpt-sync patch in parted must be reviewed ASAP, before people start testing Lucid betas on Macs...

Notice how the other bug I referenced is already targeted to beta-1 :-)

Hope issue will be solved soon.
Since my Laptop is currently OS-less (at least for now) I am able to perform some testing if required.

Indeed, I'm working on it.

On "bare GPT volumes": the EFI specification requires that all GPT disks have a Protective MBR in the first LBA. I think Linux is entitled to check for this, if nothing else to avoid accidental misdetection of the partition table format.

I'd appreciate logs from the non-Mac case (specifically /var/log/installer/syslog and /var/log/installer/partman). I haven't been able to reproduce this behaviour in an isolated test of parted, and I'd like to see the sequence of operations that the installer is performing.

Reconstructed the problem (with parted 2.2-1ubuntu2):
1. Clean GPT with 3 partitions (Gparted)
2. Start ubiquity (Lucid Daily)
    * Change type of 3rd partition
    * Format 2st and 3rd partitions (ext4 and swap)
    * Finish install
3. After reboot found partition table in state as below:
==========================================
# fdisk -l
WARNING: GPT (GUID Partition Table) detected on '/dev/sda'! The util fdisk doesn't support GPT. Use GNU Parted.
Disk /dev/sda: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical / optimal IO): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot Start End Blocks Id System
/dev/sda1 1 1045 8388607+ ee GPT
/dev/sda2 * 1 1 0 0 Empty
Partition 2 does not end on cylinder boundary.
==========================================
# parted -l
Model: ATA VBOX HARDDISK (scsi)
Disk /dev/sda: 8590MB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number Start End Size File system Name Flags
 1 17.4kB 206MB 206MB fat32 boot
 2 206MB 4401MB 4195MB ext4
 3 4401MB 8587MB 4187MB linux-swap(v1)
===========================================

Logs are attached.

BTW: Above didn't happen when disabled installation of grub into MBR, so it might be grub related issue...

That's a grub-installer bug. Fixed and uploaded, pending release team approval.

Is there still a bug to handle against linux here? I don't think there's been any clear identification of a kernel bug, aside from the question of GPT labels without protective MBRs which I think I addressed in a previous comment.

This bug was fixed in the package grub-installer - 1.49ubuntu7

---------------
grub-installer (1.49ubuntu7) lucid; urgency=low

  * Fix GPT detection to account for changed fdisk output (LP: #538100).
 -- Colin Watson <email address hidden> Mon, 15 Mar 2010 10:00:46 +0000

I assume updated parted and grub-installer are still expected to handle Mac partitioning demands, right?
After they will land in Daily Lucid, I'll test them on Mac and report.

About Linux, since Linux supports only Hibryd GPT, we would expect every partitionnig tool to prepare such, right?
Is it correct from EFI perspective?

On Mon, Mar 15, 2010 at 12:15:45PM -0000, Eugene San wrote:
> I assume updated parted and grub-installer are still expected to
> handle Mac partitioning demands, right?

Yes, should do.

> After they will land in Daily Lucid, I'll test them on Mac and report.

Thanks.

> About Linux, since Linux supports only Hibryd GPT, we would expect every partitionnig tool to prepare such, right?
> Is it correct from EFI perspective?

The EFI specification (which is also the specification for GPT) does not
define the term "Hybrid GPT". It specifies that all GPT disks have a
protective MBR. I don't think that makes them "hybrid" in any
significant sense - it's simply a part of the GPT specification.

Is there some more authoritative reference that you're thinking of here?

I can't think of any authoritative contact.
Anyway, If existence of MBR (at least protective MBR) is vital part of GPT/EFI specification then it's ok for Linux to demand it.
Linux could just ignore absence of MBR in GPT, but we can't treat its behavior as bug.

Results of testing on Mac:
  * parted do not create MBR when empty GPT is created.
  * ubiquity:
     a) When starting from empty GPT and creating partitions on GPT it creates synced MBR, but Linux then reads MBR instead of GPT.
     b) Created by ubiquity partitions created without alignment. Only last partition created with some spare space at end of device, while OSX Disk utility does insert holes and parted refuses to create without holes.
     c) When editing attributes of existing partitions it behaves normally, but only if partitions layout was created previously by ubuquty and synced to MBR using parted. In that case ubiquity do not destroy MBR.
     d) When starting with correct GPT (prepared by gparted or OSX Disk Utility), ubiquity converts holes in between first 3 partitions, to partitions in MBR.

Unfortunately ubiquity crashed few times and it's partitioning behavior was too hard to "catch", so there were no useful/sequential logs to provide.

In additiion I would like get help with definition of bugs, for two issues I have found while testing:
  * ubiquity do not provide ability to create new GPT, only MBR.
  * ubiquity do not provide ability to use grub-efi when installing to devices with GPT.