Ubuntu
cloud-utils package

Merge ~chad.smith/ubuntu/+source/cloud-utils:upload-xenial-handle-missing-overlayfs into ubuntu/+source/cloud-utils:ubuntu/xenial-updates

  1. Git
  2. lp:~chad.smith/ubuntu/+source/cloud-utils
  3. upload-xenial-handle-missing-overlayfs
  4. Merge into ubuntu/xenial-updates
Proposed by Chad Smith
Status: Needs review
Proposed branch: ~chad.smith/ubuntu/+source/cloud-utils:upload-xenial-handle-missing-overlayfs
Merge into: ubuntu/+source/cloud-utils:ubuntu/xenial-updates
Diff against target: 404 lines (+376/-0)
4 files modified
debian/changelog (+11/-0)
debian/patches/lp-1493188-support-overlay-filesystem (+240/-0)
debian/patches/lp-1630274-mount-overlay-first (+123/-0)
debian/patches/series (+2/-0)
Reviewer Review Type Date Requested Status
Paride Legovini (community) Approve
Robert C Jennings (community) Approve
Scott Moser (community) Approve
Andreas Hasenack Pending
Review via email: mp+390318@code.launchpad.net

Description of the change

Backport two separate upstream commits to handle overlayfs support on xenial to fix image builds per request https://bugs.launchpad.net/ubuntu/+source/cloud-initramfs-tools/+bug/1493188/comments/16

Pushed deb pkg to

https://launchpad.net/~chad.smith/+archive/ubuntu/cloud-utils-uploads

Verification procedure:
sudo add-apt-repository ppa:chad.smith/cloud-utils-uploads -y
sudo apt-get update
sudo apt-get install cloud-utils

To post a comment you must log in.
Revision history for this message
Scott Moser (smoser) wrote :

Assuming you can verify this works, +1.

review: Approve
Reply
Revision history for this message
Robert C Jennings (rcj) wrote :

@chad.smith I have used the cloud-utils in your ppa to build Xenial maas-images on a bionic host to confirm that things are working now.

review: Approve
Reply
Revision history for this message
Paride Legovini (paride) wrote :

Hi,

Per standup discussion, this should probably be moved to the `ubuntu/xenial` branch in the upstream GitHub repository.

review: Needs Fixing
Reply
Revision history for this message
Paride Legovini (paride) wrote :

Well maybe not really, given the low number of SRUs in the past years?

review: Approve
Reply

Unmerged commits

62f74f1... by Chad Smith

0.27-0ubuntu25.2 (patches unapplied)

Backported two upstream commits for overlay support

Notes (changelog):
      * debian/patches/lp-1493188-support-overlay-filesystem:
        mount-image-callback: support 'overlay' filesystem type rather
        than ubuntu specific 'overlayfs' (LP: #1493188)
      * debian/patches/lp-1630274-mount-overlay-first:
        mount-image-callback: try mounting overlay rather than checking first
        (LP: #1630274)

a7f1afd... by Chad Smith

backport overlay filesystem mount first (patches unapplied)

upstream commit 252183d916125231f4225eee2fa36c95ff294507
bzr-revno: 304

LP: #1630274

07d3ca8... by Chad Smith

backport overlay filesystem support (patches unapplied)

upstream commit 3f4b72e12b5739f74b6077a24df5f6f51a970699
bzr-revno: 302.1.1

LP: #1493188

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/debian/changelog b/debian/changelog
2index ce53783..0b25adc 100644
3--- a/debian/changelog
4+++ b/debian/changelog
5@@ -1,3 +1,14 @@
6+cloud-utils (0.27-0ubuntu25.2) xenial; urgency=medium
7+
8+ * debian/patches/lp-1493188-support-overlay-filesystem:
9+ mount-image-callback: support 'overlay' filesystem type rather
10+ than ubuntu specific 'overlayfs' (LP: #1493188)
11+ * debian/patches/lp-1630274-mount-overlay-first:
12+ mount-image-callback: try mounting overlay rather than checking first
13+ (LP: #1630274)
14+
15+ -- Chad Smith <chad.smith@canonical.com> Fri, 04 Sep 2020 08:47:08 -0600
16+
17 cloud-utils (0.27-0ubuntu25.1) xenial; urgency=medium
18
19 * debian/patches/lp-1762748-growpart-grow-past-2TB-disks.patch:
20diff --git a/debian/patches/lp-1493188-support-overlay-filesystem b/debian/patches/lp-1493188-support-overlay-filesystem
21new file mode 100644
22index 0000000..5dabb68
23--- /dev/null
24+++ b/debian/patches/lp-1493188-support-overlay-filesystem
25@@ -0,0 +1,240 @@
26+Description: mount-image-callback: support 'overlay' as a filesystem type
27+ mount-image-callback only knew of 'overlayfs' as a filesystem
28+ type before, but that was ubuntu specific patch.
29+Bug: https://launchpad.net/bugs/1493188
30+Origin: upstream, bzr-revno-303
31+Applied-upstream: commit: revno 302.1.1
32+
33+=== modified file 'bin/mount-image-callback'
34+Index: cloud-utils/bin/mount-image-callback
35+===================================================================
36+--- cloud-utils.orig/bin/mount-image-callback
37++++ cloud-utils/bin/mount-image-callback
38+@@ -138,11 +138,30 @@ get_partition() {
39+ fi
40+ }
41+
42++has_fs_support() {
43++ local fstype="$1" tab=' '
44++ grep -q "${tab}${fstype}$" /proc/filesystems
45++}
46++
47++find_overlay_fstype() {
48++ local fstype
49++ set -- overlay overlayfs
50++ for fstype in "$@"; do
51++ has_fs_support "$m" && _RET="$fstype" && return 0
52++ done
53++ for modname in "$@"; do
54++ if modprobe --quiet --use-blacklist "$modname"; then
55++ has_fs_support "$modname" && _RET="$modname" && return 0
56++ fi
57++ done
58++ return 1
59++}
60++
61+ mount_callback_umount() {
62+ local img_in="$1" dev="" out="" mp="" ret="" img="" readonly=""
63+ local opts="" bmounts="" system_resolvconf=false ptnum=auto
64+ local cd_mountpoint=false fmt="" mp_is_tmp=false overlay=false
65+- local img_mp="" workd=""
66++ local img_mp="" workd="" overlay_fstype=""
67+
68+ short_opts="CdhmPpsv"
69+ long_opts="cd-mountpoint,dev,help,format:,mountpoint:,overlay,partition:,proc,read-only,sys,system-mounts,system-resolvconf,verbose"
70+@@ -350,7 +369,7 @@ mount_callback_umount() {
71+ sleep .1
72+ done
73+
74+- if ( set -f; mount ${ro:+-o ${ro}} $opts "$mdev" "$img_mp" ) &&
75++ if ( set -f; mount ${readonly:+-o ${readonly}} $opts "$mdev" "$img_mp" ) &&
76+ UMOUNTS[${#UMOUNTS[@]}]="$img_mp"; then
77+ debug 1 "mounted $mdev via qemu-nbd $nbd"
78+ else
79+@@ -366,6 +385,13 @@ mount_callback_umount() {
80+ fi
81+
82+ if $overlay; then
83++ find_overlay_fstype || {
84++ error "Unable to find a filesystem type for overlay."
85++ error "You need support for overlayfs or overlay in your kernel."
86++ return 1
87++ }
88++ overlay_fstype="$_RET"
89++
90+ local olayopts="lowerdir=$img_mp,upperdir=$mp"
91+ workdir="${TEMP_D}/workdir"
92+ mkdir "$workdir"
93+@@ -374,8 +400,8 @@ mount_callback_umount() {
94+ 2*|3.1[01234567]*|3.[0-9].*) :;;
95+ *) olayopts="${olayopts},workdir=$workdir";;
96+ esac
97+- mount -t overlayfs -o "$olayopts" "$img_mp" "$mp" || {
98+- error "failed mount -t overlayfs -o '$olayopts' '$img_mp' '$mp'"
99++ mount -t "${overlay_fstype}" -o "$olayopts" "$img_mp" "$mp" || {
100++ error "failed mount -t ${overlay_fstype} -o '$olayopts' '$img_mp' '$mp'"
101+ return 1;
102+ }
103+ UMOUNTS[${#UMOUNTS[@]}]="$mp"
104+Index: cloud-utils/test/test-mic
105+===================================================================
106+--- /dev/null
107++++ cloud-utils/test/test-mic
108+@@ -0,0 +1,157 @@
109++#!/bin/bash
110++
111++set -e
112++
113++[ "$(id -u)" = "0" ] ||
114++ { echo "sorry, must be root"; exit 1; }
115++
116++PT_TYPE="${PT_TYPE:-gpt}" # dos or gpt
117++
118++cleanup() {
119++ [ ! -d "${TEMP_D}" ] || rm -Rf "${TEMP_D}"
120++}
121++rq() {
122++ local out="${TEMP_D}/out"
123++ "$@" > "$out" 2>&1 || { error "FAILED:" "$@"; cat "$out"; return 1; }
124++}
125++fail() { echo "$@" 1>&2; exit 1; }
126++error() { echo "$@" 1>&2; }
127++msg() { error "$@"; }
128++
129++do_pt_test() {
130++ local img="$1" pt="$2" data="$3" info="$4" out=""
131++ msg "testing partition $pt in $img"
132++ mount-image-callback --cd "--part=$pt" "$img" -- \
133++ sh -c "echo '$data' > data.txt" ||
134++ { error "failed writing to partition $pt in $img"; return 1; }
135++ local expected=$(printf "%s\n%s\n" "$info" "$data")
136++ out=$(mount-image-callback --read-only --cd "--part=$pt" "$img" -- \
137++ sh -c "cat info.txt data.txt") ||
138++ { error "failed mounting part1 for verification"; return 1; }
139++
140++ [ "$expected" = "$out" ] || {
141++ error "expected $pt to have:"
142++ error "$expected"
143++ error "found:"
144++ error "$out"
145++ }
146++}
147++
148++
149++TEMP_D=$(mktemp -d ${TMPDIR:-/tmp}/${0##*/}.XXXXXX)
150++trap cleanup EXIT
151++
152++pt1="${TEMP_D}/pt1.img"
153++pt2="${TEMP_D}/pt2.img"
154++prept="${TEMP_D}/header.img"
155++postpt="${TEMP_D}/foot.img"
156++img="${TEMP_D}/disk.img"
157++MB=$((1024*1024))
158++SSIZE=512
159++pt1_size=$((100*$MB))
160++pt2_size=$((200*$MB))
161++prept_size=$MB
162++postpt_size=$MB
163++
164++pt1_d="${TEMP_D}/pt1"
165++pt2_d="${TEMP_D}/pt2"
166++mkdir -p "$pt1_d" "$pt2_d"
167++echo "partition 1" > "$pt1_d/info.txt"
168++echo "file 1" > "$pt1_d/file1.txt"
169++echo "partition 2" > "$pt2_d/info.txt"
170++
171++## Stage 1
172++## Create 2 un-partitioned images, put a filesystem on them.
173++## And then mount them write a file, and then mount and
174++## read the file to verify its there.
175++truncate "--size=$pt1_size" "$pt1"
176++truncate "--size=$pt2_size" "$pt2"
177++rq mkfs.ext4 -F "${pt1}"
178++rq mkfs.ext4 -F "${pt2}"
179++
180++## Stage 1.5: Verify
181++msg "testing partition 1 image"
182++mount-image-callback "$pt1" -- cp -r "$pt1_d/"* _MOUNTPOINT_ ||
183++ fail "copying file to pt1 mount failed"
184++
185++out=$(mount-image-callback --read-only "$pt1" --cd -- cat info.txt) &&
186++ [ "$out" = "partition 1" ] ||
187++ fail "failed verification of pt1 contents"
188++
189++out=$(mount-image-callback --read-only --cd "$pt1" -- cat file1.txt)
190++[ "$out" = "file 1" ] ||
191++ fail "found unexpected contents in file1.txt on pt1: $out"
192++
193++msg "testing partition 2 image"
194++mount-image-callback "$pt2" -- cp -r "$pt2_d/"* _MOUNTPOINT_ ||
195++ fail "copying file to pt2 mount failed"
196++
197++out=$(mount-image-callback --read-only "$pt2" --cd -- cat info.txt) &&
198++ [ "$out" = "partition 2" ] ||
199++ fail "failed verification of pt2 contents"
200++
201++## Stage 1.6: Mount with overlay
202++copy_out="${TEMP_D}/copy-out"
203++mkdir -p "$TEMP_D/copy-out"
204++mount-image-callback --overlay --cd "$pt1" -- \
205++ sh -ec 't="$1";
206++ echo hi > new-file.txt; echo xxx > file1.txt;
207++ cp -r * "$t"' -- "$copy_out"
208++read found < "$copy_out/file1.txt"
209++[ "$found" = "xxx" ] ||
210++ fail "unexpected contents in file1.txt from overlay: $found."
211++
212++[ -f "$copy_out/new-file.txt" ] ||
213++ fail "new file created during overlay does not exist in copy out"
214++read found <"$copy_out/new-file.txt"
215++[ "$found" = "hi" ] ||
216++ fail "unexpected contents in new-file.txt from overlay: $found."
217++
218++## verify the overlay mount changes did not change partition
219++out=$(mount-image-callback --cd "$pt1" -- cat file1.txt)
220++[ "$out" = "file 1" ] ||
221++ fail "found unexpected contents in pt1:file1.txt after overlay: $out"
222++
223++## Stage 2
224++## Create a full disk image with those 2 partition images inside
225++## Then add a GPT partition table that points at the 2 partitions.
226++truncate "--size=$prept_size" "$prept"
227++truncate "--size=$postpt_size" "$postpt"
228++
229++msg "writing hunks to disk image $img"
230++for hunk in "$prept" "$pt1" "$pt2" "$postpt"; do
231++ rq dd bs=1M conv=notrunc oflag=append "if=$hunk" "of=$img" ||
232++ fail "failed adding $hunk to disk image"
233++done
234++
235++#
236++sfdisk_in="$TEMP_D/ptable_in"
237++curstart=0
238++(
239++echo unit: sectors
240++for pair in "-:${prept_size}" "1:${pt1_size}" "2:${pt2_size}" "-:${postpt_size}"; do
241++ op=${pair%%:*}
242++ size="${pair#*:}"
243++ if [ "$op" != "-" ]; then
244++ echo "${curstart} $((size/$SSIZE)) L"
245++ fi
246++ curstart=$(($curstart+($size/$SSIZE)))
247++done
248++) > "$sfdisk_in"
249++
250++msg "partitioning disk image $img"
251++rq sfdisk "$img" <"$sfdisk_in"
252++
253++
254++## Stage 2.5
255++## Verify we can mount each partition with '--part=N'
256++## write data, then remount and read expected data.
257++do_pt_test "$img" 1 "foo1" "partition 1" ||
258++ fail "failed testing partition 1 on $img"
259++
260++do_pt_test "$img" 2 "foo2" "partition 2" ||
261++ fail "failed testing partition 2 on $img"
262++
263++error "Finished tests."
264++
265++# vi: ts=4 noexpandtab
266diff --git a/debian/patches/lp-1630274-mount-overlay-first b/debian/patches/lp-1630274-mount-overlay-first
267new file mode 100644
268index 0000000..5271292
269--- /dev/null
270+++ b/debian/patches/lp-1630274-mount-overlay-first
271@@ -0,0 +1,123 @@
272+Description: mount-image-callback: try mounting overlay first
273+ The logic to determine if mount would work was flawed in it assumed
274+ that both or neither of the following would succeed:
275+ modprobe --quiet --use-blacklist fstype
276+ mount -t fstype would succeed
277+ .
278+ That logic failed when modprobe could not load the module due to
279+ being in a chroot or some other reason. The change here is to just
280+ do try the mount. In order to avoid confusing output about trying
281+ to mount overlayfs we first try mounting overlay. If that fails
282+ and 'overlay' seems present in /proc/filesystems, then do not bother
283+ trying to mount overlayfs.
284+Bug: https://launchpad.net/bugs/1630274
285+Origin: upstream, bzr-revno-304
286+
287+=== modified file 'bin/mount-image-callback'
288+Index: cloud-utils/bin/mount-image-callback
289+===================================================================
290+--- cloud-utils.orig/bin/mount-image-callback
291++++ cloud-utils/bin/mount-image-callback
292+@@ -4,6 +4,8 @@ VERBOSITY=0
293+ TEMP_D=""
294+ UMOUNTS=( )
295+ QEMU_DISCONNECT=""
296++CR=$'\n'
297++TAB=$'\t'
298+
299+ error() { echo "$@" 1>&2; }
300+
301+@@ -138,23 +140,37 @@ get_partition() {
302+ fi
303+ }
304+
305+-has_fs_support() {
306+- local fstype="$1" tab=' '
307+- grep -q "${tab}${fstype}$" /proc/filesystems
308+-}
309+-
310+-find_overlay_fstype() {
311+- local fstype
312+- set -- overlay overlayfs
313+- for fstype in "$@"; do
314+- has_fs_support "$m" && _RET="$fstype" && return 0
315+- done
316+- for modname in "$@"; do
317+- if modprobe --quiet --use-blacklist "$modname"; then
318+- has_fs_support "$modname" && _RET="$modname" && return 0
319++mount_overlay() {
320++ local lower="$1" upper="$2" workdir="$3"
321++ local olayopts="lowerdir=$lower,upperdir=$upper"
322++ # 3.18+ require 'workdir=' option.
323++ case "$(uname -r)" in
324++ 2*|3.1[01234567]*|3.[0-9].*) :;;
325++ *) olayopts="${olayopts},workdir=$workdir"
326++ mkdir -p "$workdir" ||
327++ { _ERR="Failed to create workdir '$workdir'"; return 1; }
328++ ;;
329++ esac
330++
331++ local cmd="" fstype="" ret="" out="" fsfile="/proc/filesystems"
332++ _ERR=""
333++ for fstype in overlay overlayfs; do
334++ cmd=( mount -t "$fstype" -o "$olayopts" "$lower" "$upper" )
335++ debug 2 "attempting '$fstype' mount with: ${cmd[*]}"
336++ out=$("${cmd[@]}" 2>&1)
337++ ret=$?
338++ if [ $ret -eq 0 ]; then
339++ debug 1 "mounted '$fstype' via $fstype: ${cmd[*]}"
340++ return 0
341++ fi
342++ _ERR="${_ERR}Failed [$ret]: ${cmd[*]}:${CR}"
343++ _ERR="${_ERR}$out${CR}"
344++ if [ -r "$fsfile" ] && grep -q "${TAB}${fstype}$" "$fsfile"; then
345++ # this failed and we have support in kernel. do not try further.
346++ return $ret
347+ fi
348+ done
349+- return 1
350++ return $ret
351+ }
352+
353+ mount_callback_umount() {
354+@@ -321,7 +337,7 @@ mount_callback_umount() {
355+ fi
356+
357+ local nptnum=""
358+- debug 1 "connected $img_in ($fmt) to $nbd. waiting for device."
359++ debug 1 "connected ${readonly:+${readonly} }$img_in ($fmt) to $nbd. waiting for device."
360+ i=0
361+
362+ local out=""
363+@@ -385,27 +401,14 @@ mount_callback_umount() {
364+ fi
365+
366+ if $overlay; then
367+- find_overlay_fstype || {
368+- error "Unable to find a filesystem type for overlay."
369+- error "You need support for overlayfs or overlay in your kernel."
370++ mount_overlay "$img_mp" "$mp" "${TEMP_D}/workdir" || {
371++ [ -n "${_ERR}" ] && error "${_ERR}"
372++ error "Unable to mount overlay filesystem. Maybe no kernel support?"
373+ return 1
374+ }
375+- overlay_fstype="$_RET"
376+-
377+- local olayopts="lowerdir=$img_mp,upperdir=$mp"
378+- workdir="${TEMP_D}/workdir"
379+- mkdir "$workdir"
380+- # 3.18+ require 'workdir=' option.
381+- case "$(uname -r)" in
382+- 2*|3.1[01234567]*|3.[0-9].*) :;;
383+- *) olayopts="${olayopts},workdir=$workdir";;
384+- esac
385+- mount -t "${overlay_fstype}" -o "$olayopts" "$img_mp" "$mp" || {
386+- error "failed mount -t ${overlay_fstype} -o '$olayopts' '$img_mp' '$mp'"
387+- return 1;
388+- }
389+ UMOUNTS[${#UMOUNTS[@]}]="$mp"
390+ fi
391++
392+ local bindmp=""
393+ for bindmp in $bmounts; do
394+ [ -d "$mp${bindmp}" ] || mkdir "$mp${bindmp}" ||
395diff --git a/debian/patches/series b/debian/patches/series
396index 00726c3..9c3d8be 100644
397--- a/debian/patches/series
398+++ b/debian/patches/series
399@@ -1,3 +1,5 @@
400 sync-to-trunk.patch
401 lp-1741300-fix-qcow-nbd.patch
402 lp-1762748-growpart-grow-past-2TB-disks.patch
403+lp-1493188-support-overlay-filesystem
404+lp-1630274-mount-overlay-first

Subscribers

People subscribed via source and target branches