Ubuntu
cloud-utils package

Merge ~chad.smith/ubuntu/+source/cloud-utils:upload-xenial-handle-missing-overlayfs into ubuntu/+source/cloud-utils:ubuntu/xenial-updates

  1. Git
  2. lp:~chad.smith/ubuntu/+source/cloud-utils
  3. upload-xenial-handle-missing-overlayfs
  4. Merge into ubuntu/xenial-updates
Proposed by Chad Smith
Status: Needs review
Proposed branch: ~chad.smith/ubuntu/+source/cloud-utils:upload-xenial-handle-missing-overlayfs
Merge into: ubuntu/+source/cloud-utils:ubuntu/xenial-updates
Diff against target: 404 lines (+376/-0)
4 files modified
debian/changelog (+11/-0)
debian/patches/lp-1493188-support-overlay-filesystem (+240/-0)
debian/patches/lp-1630274-mount-overlay-first (+123/-0)
debian/patches/series (+2/-0)
Reviewer Review Type Date Requested Status
Paride Legovini (community) Approve
Robert C Jennings (community) Approve
Scott Moser (community) Approve
Andreas Hasenack Pending
Review via email: mp+390318@code.launchpad.net

Description of the change

Backport two separate upstream commits to handle overlayfs support on xenial to fix image builds per request https://bugs.launchpad.net/ubuntu/+source/cloud-initramfs-tools/+bug/1493188/comments/16

Pushed deb pkg to

https://launchpad.net/~chad.smith/+archive/ubuntu/cloud-utils-uploads

Verification procedure:
sudo add-apt-repository ppa:chad.smith/cloud-utils-uploads -y
sudo apt-get update
sudo apt-get install cloud-utils

To post a comment you must log in.
Revision history for this message
Scott Moser (smoser) wrote :

Assuming you can verify this works, +1.

review: Approve
Reply
Revision history for this message
Robert C Jennings (rcj) wrote :

@chad.smith I have used the cloud-utils in your ppa to build Xenial maas-images on a bionic host to confirm that things are working now.

review: Approve
Reply
Revision history for this message
Paride Legovini (paride) wrote :

Hi,

Per standup discussion, this should probably be moved to the `ubuntu/xenial` branch in the upstream GitHub repository.

review: Needs Fixing
Reply
Revision history for this message
Paride Legovini (paride) wrote :

Well maybe not really, given the low number of SRUs in the past years?

review: Approve
Reply

Unmerged commits

62f74f1... by Chad Smith

0.27-0ubuntu25.2 (patches unapplied)

Backported two upstream commits for overlay support

Notes (changelog):
      * debian/patches/lp-1493188-support-overlay-filesystem:
        mount-image-callback: support 'overlay' filesystem type rather
        than ubuntu specific 'overlayfs' (LP: #1493188)
      * debian/patches/lp-1630274-mount-overlay-first:
        mount-image-callback: try mounting overlay rather than checking first
        (LP: #1630274)

a7f1afd... by Chad Smith

backport overlay filesystem mount first (patches unapplied)

upstream commit 252183d916125231f4225eee2fa36c95ff294507
bzr-revno: 304

LP: #1630274

07d3ca8... by Chad Smith

backport overlay filesystem support (patches unapplied)

upstream commit 3f4b72e12b5739f74b6077a24df5f6f51a970699
bzr-revno: 302.1.1

LP: #1493188

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
diff --git a/debian/changelog b/debian/changelog
index ce53783..0b25adc 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,14 @@
1cloud-utils (0.27-0ubuntu25.2) xenial; urgency=medium
2
3 * debian/patches/lp-1493188-support-overlay-filesystem:
4 mount-image-callback: support 'overlay' filesystem type rather
5 than ubuntu specific 'overlayfs' (LP: #1493188)
6 * debian/patches/lp-1630274-mount-overlay-first:
7 mount-image-callback: try mounting overlay rather than checking first
8 (LP: #1630274)
9
10 -- Chad Smith <chad.smith@canonical.com> Fri, 04 Sep 2020 08:47:08 -0600
11
1cloud-utils (0.27-0ubuntu25.1) xenial; urgency=medium12cloud-utils (0.27-0ubuntu25.1) xenial; urgency=medium
213
3 * debian/patches/lp-1762748-growpart-grow-past-2TB-disks.patch:14 * debian/patches/lp-1762748-growpart-grow-past-2TB-disks.patch:
diff --git a/debian/patches/lp-1493188-support-overlay-filesystem b/debian/patches/lp-1493188-support-overlay-filesystem
4new file mode 10064415new file mode 100644
index 0000000..5dabb68
--- /dev/null
+++ b/debian/patches/lp-1493188-support-overlay-filesystem
@@ -0,0 +1,240 @@
1Description: mount-image-callback: support 'overlay' as a filesystem type
2 mount-image-callback only knew of 'overlayfs' as a filesystem
3 type before, but that was ubuntu specific patch.
4Bug: https://launchpad.net/bugs/1493188
5Origin: upstream, bzr-revno-303
6Applied-upstream: commit: revno 302.1.1
7
8=== modified file 'bin/mount-image-callback'
9Index: cloud-utils/bin/mount-image-callback
10===================================================================
11--- cloud-utils.orig/bin/mount-image-callback
12+++ cloud-utils/bin/mount-image-callback
13@@ -138,11 +138,30 @@ get_partition() {
14 fi
15 }
16
17+has_fs_support() {
18+ local fstype="$1" tab=' '
19+ grep -q "${tab}${fstype}$" /proc/filesystems
20+}
21+
22+find_overlay_fstype() {
23+ local fstype
24+ set -- overlay overlayfs
25+ for fstype in "$@"; do
26+ has_fs_support "$m" && _RET="$fstype" && return 0
27+ done
28+ for modname in "$@"; do
29+ if modprobe --quiet --use-blacklist "$modname"; then
30+ has_fs_support "$modname" && _RET="$modname" && return 0
31+ fi
32+ done
33+ return 1
34+}
35+
36 mount_callback_umount() {
37 local img_in="$1" dev="" out="" mp="" ret="" img="" readonly=""
38 local opts="" bmounts="" system_resolvconf=false ptnum=auto
39 local cd_mountpoint=false fmt="" mp_is_tmp=false overlay=false
40- local img_mp="" workd=""
41+ local img_mp="" workd="" overlay_fstype=""
42
43 short_opts="CdhmPpsv"
44 long_opts="cd-mountpoint,dev,help,format:,mountpoint:,overlay,partition:,proc,read-only,sys,system-mounts,system-resolvconf,verbose"
45@@ -350,7 +369,7 @@ mount_callback_umount() {
46 sleep .1
47 done
48
49- if ( set -f; mount ${ro:+-o ${ro}} $opts "$mdev" "$img_mp" ) &&
50+ if ( set -f; mount ${readonly:+-o ${readonly}} $opts "$mdev" "$img_mp" ) &&
51 UMOUNTS[${#UMOUNTS[@]}]="$img_mp"; then
52 debug 1 "mounted $mdev via qemu-nbd $nbd"
53 else
54@@ -366,6 +385,13 @@ mount_callback_umount() {
55 fi
56
57 if $overlay; then
58+ find_overlay_fstype || {
59+ error "Unable to find a filesystem type for overlay."
60+ error "You need support for overlayfs or overlay in your kernel."
61+ return 1
62+ }
63+ overlay_fstype="$_RET"
64+
65 local olayopts="lowerdir=$img_mp,upperdir=$mp"
66 workdir="${TEMP_D}/workdir"
67 mkdir "$workdir"
68@@ -374,8 +400,8 @@ mount_callback_umount() {
69 2*|3.1[01234567]*|3.[0-9].*) :;;
70 *) olayopts="${olayopts},workdir=$workdir";;
71 esac
72- mount -t overlayfs -o "$olayopts" "$img_mp" "$mp" || {
73- error "failed mount -t overlayfs -o '$olayopts' '$img_mp' '$mp'"
74+ mount -t "${overlay_fstype}" -o "$olayopts" "$img_mp" "$mp" || {
75+ error "failed mount -t ${overlay_fstype} -o '$olayopts' '$img_mp' '$mp'"
76 return 1;
77 }
78 UMOUNTS[${#UMOUNTS[@]}]="$mp"
79Index: cloud-utils/test/test-mic
80===================================================================
81--- /dev/null
82+++ cloud-utils/test/test-mic
83@@ -0,0 +1,157 @@
84+#!/bin/bash
85+
86+set -e
87+
88+[ "$(id -u)" = "0" ] ||
89+ { echo "sorry, must be root"; exit 1; }
90+
91+PT_TYPE="${PT_TYPE:-gpt}" # dos or gpt
92+
93+cleanup() {
94+ [ ! -d "${TEMP_D}" ] || rm -Rf "${TEMP_D}"
95+}
96+rq() {
97+ local out="${TEMP_D}/out"
98+ "$@" > "$out" 2>&1 || { error "FAILED:" "$@"; cat "$out"; return 1; }
99+}
100+fail() { echo "$@" 1>&2; exit 1; }
101+error() { echo "$@" 1>&2; }
102+msg() { error "$@"; }
103+
104+do_pt_test() {
105+ local img="$1" pt="$2" data="$3" info="$4" out=""
106+ msg "testing partition $pt in $img"
107+ mount-image-callback --cd "--part=$pt" "$img" -- \
108+ sh -c "echo '$data' > data.txt" ||
109+ { error "failed writing to partition $pt in $img"; return 1; }
110+ local expected=$(printf "%s\n%s\n" "$info" "$data")
111+ out=$(mount-image-callback --read-only --cd "--part=$pt" "$img" -- \
112+ sh -c "cat info.txt data.txt") ||
113+ { error "failed mounting part1 for verification"; return 1; }
114+
115+ [ "$expected" = "$out" ] || {
116+ error "expected $pt to have:"
117+ error "$expected"
118+ error "found:"
119+ error "$out"
120+ }
121+}
122+
123+
124+TEMP_D=$(mktemp -d ${TMPDIR:-/tmp}/${0##*/}.XXXXXX)
125+trap cleanup EXIT
126+
127+pt1="${TEMP_D}/pt1.img"
128+pt2="${TEMP_D}/pt2.img"
129+prept="${TEMP_D}/header.img"
130+postpt="${TEMP_D}/foot.img"
131+img="${TEMP_D}/disk.img"
132+MB=$((1024*1024))
133+SSIZE=512
134+pt1_size=$((100*$MB))
135+pt2_size=$((200*$MB))
136+prept_size=$MB
137+postpt_size=$MB
138+
139+pt1_d="${TEMP_D}/pt1"
140+pt2_d="${TEMP_D}/pt2"
141+mkdir -p "$pt1_d" "$pt2_d"
142+echo "partition 1" > "$pt1_d/info.txt"
143+echo "file 1" > "$pt1_d/file1.txt"
144+echo "partition 2" > "$pt2_d/info.txt"
145+
146+## Stage 1
147+## Create 2 un-partitioned images, put a filesystem on them.
148+## And then mount them write a file, and then mount and
149+## read the file to verify its there.
150+truncate "--size=$pt1_size" "$pt1"
151+truncate "--size=$pt2_size" "$pt2"
152+rq mkfs.ext4 -F "${pt1}"
153+rq mkfs.ext4 -F "${pt2}"
154+
155+## Stage 1.5: Verify
156+msg "testing partition 1 image"
157+mount-image-callback "$pt1" -- cp -r "$pt1_d/"* _MOUNTPOINT_ ||
158+ fail "copying file to pt1 mount failed"
159+
160+out=$(mount-image-callback --read-only "$pt1" --cd -- cat info.txt) &&
161+ [ "$out" = "partition 1" ] ||
162+ fail "failed verification of pt1 contents"
163+
164+out=$(mount-image-callback --read-only --cd "$pt1" -- cat file1.txt)
165+[ "$out" = "file 1" ] ||
166+ fail "found unexpected contents in file1.txt on pt1: $out"
167+
168+msg "testing partition 2 image"
169+mount-image-callback "$pt2" -- cp -r "$pt2_d/"* _MOUNTPOINT_ ||
170+ fail "copying file to pt2 mount failed"
171+
172+out=$(mount-image-callback --read-only "$pt2" --cd -- cat info.txt) &&
173+ [ "$out" = "partition 2" ] ||
174+ fail "failed verification of pt2 contents"
175+
176+## Stage 1.6: Mount with overlay
177+copy_out="${TEMP_D}/copy-out"
178+mkdir -p "$TEMP_D/copy-out"
179+mount-image-callback --overlay --cd "$pt1" -- \
180+ sh -ec 't="$1";
181+ echo hi > new-file.txt; echo xxx > file1.txt;
182+ cp -r * "$t"' -- "$copy_out"
183+read found < "$copy_out/file1.txt"
184+[ "$found" = "xxx" ] ||
185+ fail "unexpected contents in file1.txt from overlay: $found."
186+
187+[ -f "$copy_out/new-file.txt" ] ||
188+ fail "new file created during overlay does not exist in copy out"
189+read found <"$copy_out/new-file.txt"
190+[ "$found" = "hi" ] ||
191+ fail "unexpected contents in new-file.txt from overlay: $found."
192+
193+## verify the overlay mount changes did not change partition
194+out=$(mount-image-callback --cd "$pt1" -- cat file1.txt)
195+[ "$out" = "file 1" ] ||
196+ fail "found unexpected contents in pt1:file1.txt after overlay: $out"
197+
198+## Stage 2
199+## Create a full disk image with those 2 partition images inside
200+## Then add a GPT partition table that points at the 2 partitions.
201+truncate "--size=$prept_size" "$prept"
202+truncate "--size=$postpt_size" "$postpt"
203+
204+msg "writing hunks to disk image $img"
205+for hunk in "$prept" "$pt1" "$pt2" "$postpt"; do
206+ rq dd bs=1M conv=notrunc oflag=append "if=$hunk" "of=$img" ||
207+ fail "failed adding $hunk to disk image"
208+done
209+
210+#
211+sfdisk_in="$TEMP_D/ptable_in"
212+curstart=0
213+(
214+echo unit: sectors
215+for pair in "-:${prept_size}" "1:${pt1_size}" "2:${pt2_size}" "-:${postpt_size}"; do
216+ op=${pair%%:*}
217+ size="${pair#*:}"
218+ if [ "$op" != "-" ]; then
219+ echo "${curstart} $((size/$SSIZE)) L"
220+ fi
221+ curstart=$(($curstart+($size/$SSIZE)))
222+done
223+) > "$sfdisk_in"
224+
225+msg "partitioning disk image $img"
226+rq sfdisk "$img" <"$sfdisk_in"
227+
228+
229+## Stage 2.5
230+## Verify we can mount each partition with '--part=N'
231+## write data, then remount and read expected data.
232+do_pt_test "$img" 1 "foo1" "partition 1" ||
233+ fail "failed testing partition 1 on $img"
234+
235+do_pt_test "$img" 2 "foo2" "partition 2" ||
236+ fail "failed testing partition 2 on $img"
237+
238+error "Finished tests."
239+
240+# vi: ts=4 noexpandtab
diff --git a/debian/patches/lp-1630274-mount-overlay-first b/debian/patches/lp-1630274-mount-overlay-first
0new file mode 100644241new file mode 100644
index 0000000..5271292
--- /dev/null
+++ b/debian/patches/lp-1630274-mount-overlay-first
@@ -0,0 +1,123 @@
1Description: mount-image-callback: try mounting overlay first
2 The logic to determine if mount would work was flawed in it assumed
3 that both or neither of the following would succeed:
4 modprobe --quiet --use-blacklist fstype
5 mount -t fstype would succeed
6 .
7 That logic failed when modprobe could not load the module due to
8 being in a chroot or some other reason. The change here is to just
9 do try the mount. In order to avoid confusing output about trying
10 to mount overlayfs we first try mounting overlay. If that fails
11 and 'overlay' seems present in /proc/filesystems, then do not bother
12 trying to mount overlayfs.
13Bug: https://launchpad.net/bugs/1630274
14Origin: upstream, bzr-revno-304
15
16=== modified file 'bin/mount-image-callback'
17Index: cloud-utils/bin/mount-image-callback
18===================================================================
19--- cloud-utils.orig/bin/mount-image-callback
20+++ cloud-utils/bin/mount-image-callback
21@@ -4,6 +4,8 @@ VERBOSITY=0
22 TEMP_D=""
23 UMOUNTS=( )
24 QEMU_DISCONNECT=""
25+CR=$'\n'
26+TAB=$'\t'
27
28 error() { echo "$@" 1>&2; }
29
30@@ -138,23 +140,37 @@ get_partition() {
31 fi
32 }
33
34-has_fs_support() {
35- local fstype="$1" tab=' '
36- grep -q "${tab}${fstype}$" /proc/filesystems
37-}
38-
39-find_overlay_fstype() {
40- local fstype
41- set -- overlay overlayfs
42- for fstype in "$@"; do
43- has_fs_support "$m" && _RET="$fstype" && return 0
44- done
45- for modname in "$@"; do
46- if modprobe --quiet --use-blacklist "$modname"; then
47- has_fs_support "$modname" && _RET="$modname" && return 0
48+mount_overlay() {
49+ local lower="$1" upper="$2" workdir="$3"
50+ local olayopts="lowerdir=$lower,upperdir=$upper"
51+ # 3.18+ require 'workdir=' option.
52+ case "$(uname -r)" in
53+ 2*|3.1[01234567]*|3.[0-9].*) :;;
54+ *) olayopts="${olayopts},workdir=$workdir"
55+ mkdir -p "$workdir" ||
56+ { _ERR="Failed to create workdir '$workdir'"; return 1; }
57+ ;;
58+ esac
59+
60+ local cmd="" fstype="" ret="" out="" fsfile="/proc/filesystems"
61+ _ERR=""
62+ for fstype in overlay overlayfs; do
63+ cmd=( mount -t "$fstype" -o "$olayopts" "$lower" "$upper" )
64+ debug 2 "attempting '$fstype' mount with: ${cmd[*]}"
65+ out=$("${cmd[@]}" 2>&1)
66+ ret=$?
67+ if [ $ret -eq 0 ]; then
68+ debug 1 "mounted '$fstype' via $fstype: ${cmd[*]}"
69+ return 0
70+ fi
71+ _ERR="${_ERR}Failed [$ret]: ${cmd[*]}:${CR}"
72+ _ERR="${_ERR}$out${CR}"
73+ if [ -r "$fsfile" ] && grep -q "${TAB}${fstype}$" "$fsfile"; then
74+ # this failed and we have support in kernel. do not try further.
75+ return $ret
76 fi
77 done
78- return 1
79+ return $ret
80 }
81
82 mount_callback_umount() {
83@@ -321,7 +337,7 @@ mount_callback_umount() {
84 fi
85
86 local nptnum=""
87- debug 1 "connected $img_in ($fmt) to $nbd. waiting for device."
88+ debug 1 "connected ${readonly:+${readonly} }$img_in ($fmt) to $nbd. waiting for device."
89 i=0
90
91 local out=""
92@@ -385,27 +401,14 @@ mount_callback_umount() {
93 fi
94
95 if $overlay; then
96- find_overlay_fstype || {
97- error "Unable to find a filesystem type for overlay."
98- error "You need support for overlayfs or overlay in your kernel."
99+ mount_overlay "$img_mp" "$mp" "${TEMP_D}/workdir" || {
100+ [ -n "${_ERR}" ] && error "${_ERR}"
101+ error "Unable to mount overlay filesystem. Maybe no kernel support?"
102 return 1
103 }
104- overlay_fstype="$_RET"
105-
106- local olayopts="lowerdir=$img_mp,upperdir=$mp"
107- workdir="${TEMP_D}/workdir"
108- mkdir "$workdir"
109- # 3.18+ require 'workdir=' option.
110- case "$(uname -r)" in
111- 2*|3.1[01234567]*|3.[0-9].*) :;;
112- *) olayopts="${olayopts},workdir=$workdir";;
113- esac
114- mount -t "${overlay_fstype}" -o "$olayopts" "$img_mp" "$mp" || {
115- error "failed mount -t ${overlay_fstype} -o '$olayopts' '$img_mp' '$mp'"
116- return 1;
117- }
118 UMOUNTS[${#UMOUNTS[@]}]="$mp"
119 fi
120+
121 local bindmp=""
122 for bindmp in $bmounts; do
123 [ -d "$mp${bindmp}" ] || mkdir "$mp${bindmp}" ||
diff --git a/debian/patches/series b/debian/patches/series
index 00726c3..9c3d8be 100644
--- a/debian/patches/series
+++ b/debian/patches/series
@@ -1,3 +1,5 @@
1sync-to-trunk.patch1sync-to-trunk.patch
2lp-1741300-fix-qcow-nbd.patch2lp-1741300-fix-qcow-nbd.patch
3lp-1762748-growpart-grow-past-2TB-disks.patch3lp-1762748-growpart-grow-past-2TB-disks.patch
4lp-1493188-support-overlay-filesystem
5lp-1630274-mount-overlay-first

Subscribers

People subscribed via source and target branches