Merge lp:~paelzer/curtin/fix-bug-1513085-partition-alignment-v3 into lp:~curtin-dev/curtin/trunk

[Christian Ehrhardt ]

Partition alignment and sizing fixes:
- end sector calculation so that partition size matches what was requested
- proper size of extended partition size calculation to be able to hold as much
  as it was specified in the yaml
- proper logical partition start calculation and realignment
- prepare curtin for the appearance of 4K logical sector size disk.
- by that implicitly fixing our alignment issues as long as sizes are aligned
  which they usually are as e.g. MAAS exposes only 1MB units
- extends one of the existing tests to cover 2G+3G logical in 5G extended

First partitions start at 2048 by default (which is parted default as well)
As I pointed out we have to understand that when currently creating a 1024
byte partition (small size makes no sense but good for the example) we do
2048 - 2048 + (1024/512)
2048 - 2050
That means we span 2048, 2049 2050 => that is 1536 byte

Also logical/extended partitions did wrong sector calculations by adding the
size of the extended (2 sectors) to the start - that only accidentially worked
most of the time.
Extended and Logical can't share the start sector and subsequent logical
partitions can't go head-to-head with former logical partitions.
While all that goes on a human would assume all that just works.
Also one will assume that e.g. a 5G extended partition can hold a 3G and a 2G
logical which so far it couldn't for all these reasons.

All that also broke alignment.

This patch now fixes:
- end sector calculation so that partition size matches what was requested
- proper size of extended partition size calculation to be able to hold as much
  as it was specified in the yaml
- proper logical partition start calculation and realignment
- prepare curtin for the appearance of 4K logical sector size disk.
- by that implicitly fixing our alignment issues as long as sizes are aligned
  which they usually are as e.g. MAAS exposes only 1MB units
- extends one of the existing tests to cover 2G+3G logical in 5G extended

[Scott Moser (smoser)]

This looks good. Thank you Christian.
We really should start to write some doc to say things like "1MB gap" and such.
and also document what we leave at the end.

=== modified file 'curtin/commands/block_meta.py'
--- curtin/commands/block_meta.py	2015-10-30 13:00:28 +0000
+++ curtin/commands/block_meta.py	2015-11-10 12:47:04 +0000
@@ -480,6 +480,14 @@
         make_dname(info.get('id'), storage_config)
 
 
+def getnumberoflogicaldisks(device, storage_config):
+    logicaldisks = 0
+    for key, item in storage_config.items():
+        if item.get('device') == device and item.get('flag') == "logical":
+            logicaldisks = logicaldisks + 1
+    return logicaldisks
+
+
 def partition_handler(info, storage_config):
     device = info.get('device')
     size = info.get('size')
@@ -494,11 +502,18 @@
     disk = get_path_to_storage_volume(device, storage_config)
     partnumber = determine_partition_number(info.get('id'), storage_config)
 
-    # Offset is either 1 sector after last partition, or near the beginning if
-    # this is the first partition
+    disk_kname = os.path.split(
+        get_path_to_storage_volume(device, storage_config))[-1]
+    # consider the disks logical sector size when calculating sectors
+    try:
+        prefix = "/sys/block/%s/queue/" % disk_kname
+        with open(prefix + "logical_block_size", "r") as f:
+            l = f.readline()
+            logical_block_size_bytes = int(l)
+    except:
+        logical_block_size_bytes = 512
+
     if partnumber > 1:
-        disk_kname = os.path.split(
-            get_path_to_storage_volume(device, storage_config))[-1]
         if partnumber == 5 and disk_ptable == "msdos":
             for key, item in storage_config.items():
                 if item.get('type') == "partition" and \
@@ -515,12 +530,41 @@
         with open(os.path.join(previous_partition, "size"), "r") as fp:
             previous_size = int(fp.read())
         with open(os.path.join(previous_partition, "start"), "r") as fp:
-            offset_sectors = previous_size + int(fp.read()) + 1
+            previous_start = int(fp.read())
+
+    # Align to 1M at the beginning of the disk and at logical partitions
+    alignment_offset = (1 << 20) / logical_block_size_bytes
+    if partnumber == 1:
+        # start of disk
+        offset_sectors = alignment_offset
     else:
-        offset_sectors = 2048
+        # further partitions
+        if disk_ptable == "gpt" or flag != "logical":
+            # msdos primary and any gpt part start after former partition end
+            offset_sectors = previous_start + previous_size
+        else:
+            # msdos extended/logical partitions
+            if flag == "logical":
+                if partnumber == 5:
+                    # First logical partition
+                    # start at extended partition start + alignment_offset
+                    offset_sectors = previous_start + alignment_offset
+                else:
+                    # Further logical partitions
+                    # start at former logical partition end + alignment_offset
+                    offset_sectors = (previous_start + previous_size
+                                      + alignment_offset)
 
     length_bytes = util.human2bytes(size)
-    length_sectors = int(length_bytes / 512)
+    # start sector is part of the sectors that define the partitions size
+    # so length has to be "size in sectors - 1"
+    length_sectors = int(length_bytes / logical_block_size_bytes) - 1
+    # logical partitions can't share their start sector with the extended
+    # partition and logical partitions can't go head-to-head, so we have to
+    # realign and for that increase size as required
+    if info.get('flag') == "extended":
+        logdisks = getnumberoflogicaldisks(device, storage_config)
+        length_sectors = length_sectors + (logdisks * alignment_offset)
 
     # Handle preserve flag
     if info.get('preserve'):

=== modified file 'examples/tests/lvm.yaml'
--- examples/tests/lvm.yaml	2015-08-12 17:42:11 +0000
+++ examples/tests/lvm.yaml	2015-11-10 12:47:04 +0000
@@ -24,7 +24,7 @@
         device: sda
       - id: sda3
         type: partition
-        size: 2G
+        size: 3G
         flag: logical
         device: sda
       - id: volgroup1