Merge ~smoser/ubuntu/+source/initramfs-tools:bug/1714308-initramfs-dns into ~usd-import-team/ubuntu/+source/initramfs-tools:ubuntu/devel

[Scott Moser]

networking: write netplan files in /run/netplan, add some tests.

Render /run/netplan/*.conf files for each device configured in the
initramfs.

The netinfo_to_netplan function handles reading 'netinfo' style files
(those written by klibcs 'ipconfig' or the dhclient for the ipv6 path)
and writing /run/netplan/<device>.conf files.

Note the following use cases that are not covered:
 a. user has files in the root /etc/netplan that conflict with the
    generated /run/netplan/*.conf files.
 b. the networking configured by 'configure_networking' was only desired
    to be used in the initramfs and not afterward.

Note also that the order of arguments passed to netinfo_to_netplan is
important and will have implications on things such as nameservers/search
keys.

Also, adds some tests for the netplan rendering.
This extends the tests/run-tests that I previously added
and puts a new tests/test_netinfo.sh and tests/test_netinfo.d.

LP: #1713803

[Scott Moser (smoser)]

I thought some after I put this up.
I think it might make sense for the netplan path to have the initramfs write netplan configuration to /run/netplan/ (I think it reads there).

[Scott Moser (smoser)]

OK.
Now this branch is writing resolv.conf in the initramfs and also writing /run/netplan/<device>.yaml files.
I've added some tests to verify output.

I've not integration tested, but it seems that there is a fair shot now that
if we have this code boot with 'ip=' on the command line then it will
write /run/netplan/eth0.yaml that would then get moved from the initramfs over
to the real root.

since the /run/netplan files will have dns info in them i think that netplan might
do the right thing. The thing i'm not sure about is if netplan will bounce the
interfaces or not.

1d11695... by Scott Moser on 2017-09-05

scripts/functions: support netplan writing

aabc1e0... by Scott Moser on 2017-09-05

add netplan tests.

6b8d55b... by Scott Moser on 2017-09-05

update changelog (netplan writing)

9f473ce... by Scott Moser on 2017-09-19

Import patches-unapplied version 0.125ubuntu11 to ubuntu/artful-proposed

Imported using git-ubuntu import.

Publish parent: 48b52ea2b8d6699c5c09e01967bae4e276bc3ca9

New changelog entries:
  * hook-functions, mkinitramfs, scripts/functions: support usage of dns
    after configure_networking. (LP: #1714308)

3063dc1... by Scott Moser on 2017-09-19

Update ubuntu/devel from 0.125ubuntu10 to 0.125ubuntu11

Prior ubuntu/devel commit: 3112eb4aedac6024286b9aa02685ef168abcfb66
New ubuntu/devel commit: 9f473ce57fc209e76962c33bfdb0c287097b3a0e

0b98524... by Scott Moser on 2017-09-29

netinfo_to_resolv_conf: allow writing to '-'

if file name provided is '-', then write to stdout.

[Scott Moser (smoser)]

In a discussion with Dimitri, it was suggested that one way to transition
the link to systemd-networkd more fully would be to write files like:
  /run/systemd/netif/links/20
and
  /run/systemd/netif/leases/20

Then it is believed that systemd-networkd would even manage the dhcp link
correctly.

One question this brings up in my mind, is what should/will happen when a
dhcp lease is up on an interface when a network filesystem or block
device resides on that link? This isn't specific at all to initramfs
transition as it would apply to any dhcp-configured network device that
mounts network filesystem.

As an example, of content, in an lxc container I have:

# cat /run/systemd/netif/leases/20
# This is private data. Do not parse.
ADDRESS=10.75.205.194
NETMASK=255.255.255.0
ROUTER=10.75.205.1
SERVER_ADDRESS=10.75.205.1
NEXT_SERVER=10.75.205.1
BROADCAST=10.75.205.255
T1=1800
T2=3150
LIFETIME=3600
DNS=10.75.205.1
DOMAINNAME=lxd
HOSTNAME=a1
CLIENTID=ffdeb2597a00020000ab1156b04a522c37f831

# cat /run/systemd/netif/links/20
# This is private data. Do not parse.
ADMIN_STATE=configured
OPER_STATE=routable
NETWORK_FILE=/run/systemd/network/10-netplan-eth0.network
DNS=10.75.205.1 fd42:eee5:7c43:3d62::1 fe80::9cb9:7aff:fea4:429
NTP=
DOMAINS=lxd
ROUTE_DOMAINS=
LLMNR=yes
MDNS=no
ADDRESSES=fd42:eee5:7c43:3d62:b106:9773:17c6:52ca/128 10.75.205.194/24
ROUTES=::/0/0/100/254/4186785205 fd42:eee5:7c43:3d62::/64/0/100/254/5986785205 0.0.0.0/0/0/100/254/18446744073709551615 10.75.205.1/32/0/100/254/18446744073709551615
DHCP4_ADDRESS=10.75.205.194
DHCP_LEASE=/run/systemd/netif/leases/20

Unmerged commits

6b8d55b... by Scott Moser on 2017-09-05

update changelog (netplan writing)

aabc1e0... by Scott Moser on 2017-09-05

add netplan tests.

0b98524... by Scott Moser on 2017-09-29

netinfo_to_resolv_conf: allow writing to '-'

if file name provided is '-', then write to stdout.

1d11695... by Scott Moser on 2017-09-05

scripts/functions: support netplan writing

3063dc1... by Scott Moser on 2017-09-19

Update ubuntu/devel from 0.125ubuntu10 to 0.125ubuntu11

Prior ubuntu/devel commit: 3112eb4aedac6024286b9aa02685ef168abcfb66
New ubuntu/devel commit: 9f473ce57fc209e76962c33bfdb0c287097b3a0e

9f473ce... by Scott Moser on 2017-09-19

Import patches-unapplied version 0.125ubuntu11 to ubuntu/artful-proposed

Imported using git-ubuntu import.

Publish parent: 48b52ea2b8d6699c5c09e01967bae4e276bc3ca9

New changelog entries:
  * hook-functions, mkinitramfs, scripts/functions: support usage of dns
    after configure_networking. (LP: #1714308)

48b52ea... by Andy Whitcroft on 2017-08-30

Import patches-unapplied version 0.125ubuntu10 to ubuntu/artful-proposed

Imported using git-ubuntu import.

Publish parent: c210559c01d5a59930831f5509d061b566c3458f

New changelog entries:
  * hook-functions: include backported bnxt driver. (LP: #1712491)

c210559... by LaMont Jones on 2016-12-15

Import patches-unapplied version 0.125ubuntu9 to ubuntu/artful

Imported using usd-importer.

Publish parent: e0f307c4463f01c2ce92a76bb96d4c6f2226c75e

e0f307c... by LaMont Jones on 2016-12-15

Import patches-unapplied version 0.125ubuntu9 to ubuntu/zesty

Imported using usd-importer.

Publish parent: 9fcb9e51603f604f2e621d9db90d5c986aaef1a9
Changelog parent: 59cf727709343de7314164e01370850136c19f95

59cf727... by LaMont Jones on 2016-12-15

Import patches-unapplied version 0.125ubuntu9 to ubuntu/zesty-proposed

Imported using usd-importer.

Publish parent: 15a3e4b44a9af92961b05d87adc20fde799bcc00

New changelog entries:
  [ Mathieu Trudel-Lapierre ]
  * Use configure_networking from yakkety-proposed (SRU), addressing various
    review feedback. dhclient -4 will be reconsidered later. LP: #1621507

diff --git a/debian/changelog b/debian/changelog
index 93a1ef8..5a757ca 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,10 @@
+initramfs-tools (0.125ubuntu12) UNRELEASED; urgency=medium
+
+  * scripts/functions: write netplan config files to /run/netplan for
+    network devices configured with configure_networking. (LP: #1713803)
+
+ -- Scott Moser <[e-mail]>  Wed, 20 Sep 2017 10:54:36 -0400
+
 initramfs-tools (0.125ubuntu11) artful; urgency=medium
 
   * hook-functions, mkinitramfs, scripts/functions: support usage of dns
diff --git a/scripts/functions b/scripts/functions
index e5e1029..b84a9cf 100644
--- a/scripts/functions
+++ b/scripts/functions
@@ -388,6 +388,8 @@ configure_networking()
 
 	netinfo_to_resolv_conf /etc/resolv.conf \
 		/run/net-${DEVICE}.conf /run/net-*.conf /run/net6-*.conf
+	netinfo_to_netplan /run/netplan \
+		/run/net-${DEVICE}.conf /run/net-*.conf /run/net6-*.conf
 }
 
 netinfo_to_resolv_conf() {
@@ -398,10 +400,8 @@ netinfo_to_resolv_conf() {
 		( _in_subshell=1; netinfo_to_resolv_conf "$@" )
 		return
 	fi
-
 	local output="$1" search="" ns="" f="" n=""
 	shift
-	{
 	for f in "$@"; do
 		[ -f "$f" ] || continue
 		unset IPV4DNS0 IPV4DNS1 IPV6DNS0 IPV6DNS1
@@ -427,13 +427,182 @@ netinfo_to_resolv_conf() {
 		search=${search# }
 		ns=${ns# }
 	done
+
+	local rconf=""
 	for n in ${ns}; do
-		echo "nameserver $n"
+		rconf="${rconf}nameserver $n${CR}"
 	done
 	if [ -n "${search}" ]; then
-		echo "search ${search}"
+		rconf="${rconf}search ${search}${CR}"
+	fi
+	if [ -z "$rconf" ]; then
+		echo "no search or nameservers found in $*" 1>&2
+	fi
+	if [ "$rconf" = "-" ]; then
+		echo -n "$rconf"
+	else
+		echo -n "$rconf" > "$output"
+	fi
+}
+
+mask2cidr() {
+	# https://forum.openwrt.org/viewtopic.php?pid=220781#p220781
+	# Assumes there's no "255." after a non-255 byte in the mask
+	local x=${1##*255.}
+	set -- 0^^^128^192^224^240^248^252^254^ $(( (${#1} - ${#x})*2 )) ${x%%.*}
+	x=${1%%$3*}
+	echo $(( $2 + (${#x}/4) ))
+}
+
+_declare_sh_append_var() {
+	# append_var(name, skip, strings)
+	# write a declaration of name that will append to any existing
+	local name="$1" skip="$2" add="" n=""
+	shift 2
+	for n in "$@"; do
+		[ -n "$n" -a "$n" != "$skip" ] || continue
+		add="$add $n"
+	done
+	add=${add# }
+	[ -n "$add" ] || return 0
+	echo "$name=\"\${${name}:+\${${name}} }${add}\""
+}
+
+_declare_ip_info() {
+	# declare_ip_info(version, proto, address, netmask, gateway)
+	local version="$1" proto="$2" address="$3" netmask="$4" gateway="$5"
+	local netprefix=""
+	if [ "$proto" = "dhcp" -o "$proto" = "dhcp4" -o "$proto" = "dhcp6" ]; then
+		echo "dhcp${version}=true"
+	elif [ "$proto" = "static" ]; then
+		if [ -n "$address" ]; then
+			netprefix=$netmask
+			if [ "$version" = "4" ]; then
+				netprefix=$(mask2cidr "$netmask")
+			fi
+			_declare_sh_append_var addresses "" "$address/$netprefix"
+		fi
+		if [ -n "$gateway" ]; then
+			echo "gateway${version}=$gateway"
+		fi
+	fi
+}
+
+_render_netplan() {
+	# write a netplan stanza for the given device.
+	local name="$1" mac="$2" dhcp4="$3" dhcp6="$4" addrs="$5" \
+		gateway4="$6" gateway6="$7" ns_addrs="$8" ns_search="$9"
+	local n found=""
+	echo "$name:"
+	if [ -n "$mac" ]; then
+		echo "  match:"
+		echo "    macaddress: \"$mac\""
+		echo "  set-name: $name"
+	fi
+	[ -n "$dhcp4" ] && echo "  dhcp4: $dhcp4"
+	[ -n "$dhcp6" ] && echo "  dhcp6: $dhcp6"
+	if [ -n "$addrs" ]; then
+		echo "  addresses:"
+		found=","
+		for n in $addrs; do
+			# remove dups
+			[ "${found#*,$n,}" = "${found}" ] || continue
+			found="${found}$n,"
+			echo "    - \"$n\""
+		done
+	fi
+	[ -n "$gateway4" ] && echo "  gateway4: \"$gateway4\""
+	[ -n "$gateway6" ] && echo "  gateway6: \"$gateway6\""
+
+	if [ -n "$ns_addrs" ]; then
+		local alist="[" slist=""
+		for n in $ns_addrs; do
+			# do not put in duplicates
+			[ "${alist#*\"$n\"}" = "$alist" ] || continue
+			alist="${alist}\"$n\", ";
+		done
+		alist="${alist%, }]"
+
+		if [ -n "$ns_search" ]; then
+			slist="["
+			for n in ${ns_search}; do
+				# do not put in duplicates
+				[ "${slist#*\"$n\"}" = "$slist" ] || continue
+				slist="${slist}\"$n\", ";
+			done
+			slist="${slist%, }]"
+		fi
+		echo "  nameservers:"
+		echo "    addresses: $alist"
+		[ -n "$slist" ] && echo "    search: $slist"
+	fi
+}
+
+netinfo_to_netplan() {
+	# read /run/net-* files write netplan config.
+	if [ "${_in_subshell:-0}" = "0" ]; then
+		# subshell to avoid modification of variables by '.'
+		( _in_subshell=1; netinfo_to_netplan "$@" )
+		return
 	fi
-	} > "$output"
+	local out_d="$1" tmpd
+	if command -v mktemp >/dev/null 2>&1; then
+		tmpd=$(mktemp -d "${TMPDIR:-/tmp}/${0##*/}.XXXXXX")
+	else
+		tmpd="${TMPDIR:-/tmp}/${0##*/}.niinfo.$$"
+		mkdir -p "$tmpd" || return
+	fi
+
+	local devices="" pre="" mac=""
+	# we go through all the files presented and create per-device files in
+	# a tmpdir that are shell sourceable and closer to the netplan that
+	# we want to render. Then render those to netplan stanzas.
+	for f in "$@"; do
+		[ -f "$f" ] || continue
+		unset DEVICE DEVICE6 PROTO IPV6PROTO
+		unset IPV6ADDR IPV6NETMASK IPV6GATEWAY
+		unset IPV4ADDR IPV4NETMASK IPV4GATEWAY
+		. "$f" || { echo "WARN: failed '. \"$f\"'" 1>&2; return 1; }
+		local name=""
+		name=${DEVICE:-${DEVICE6}}
+		[ -n "$name" ] || {
+			echo "WARN: $f did not define DEVICE or DEVICE6" 1>&2;
+			return 1;
+		}
+		case " ${devices} " in
+			*\ ${name}\ *) :;;
+			*) devices="${devices} ${name}"
+		esac
+		if [ ! -e "$tmpd/$name" -a -r "/sys/class/net/$name/address" ]; then
+			read mac < /sys/class/net/$name/address &&
+				echo "macaddress=$mac" > "$tmpd/$name"
+		fi
+
+		{
+		if [ -n "$DEVICE" ]; then
+			_declare_ip_info 4 "$PROTO" "$IPV4ADDR" "$IPV4NETMASK" "$IPV4GATEWAY"
+		elif [ -n "$DEVICE6" ]; then
+			_declare_ip_info 6 "$IPV6PROTO" "$IPV6ADDR" "$IPV6NETMASK" \
+				"$IPV6GATEWAY"
+		fi
+		_declare_sh_append_var ns_addresses "0.0.0.0" \
+			"${IPV4DNS0}" "${IPV4DNS1}" "${IPV6DNS0}" "${IPV6DNS1}"
+		_declare_sh_append_var ns_search "" "$DOMAINSEARCH" "$IPV6DOMAINSEARCH"
+		} >> "$tmpd/$name"
+	done
+
+	[ -d "$out_d" ] || mkdir -p "$out_d" ||
+		{ echo "WARN: failed mkdir $out_d"; return 1; }
+
+	for name in $devices; do
+		local macaddress="" dhcp4="" dhcp6="" addresses=""
+		local gateway4="" gateway6="" ns_addresses="" ns_search=""
+		. "$tmpd/$name"
+		_render_netplan "$name" "$macaddress" "$dhcp4" "$dhcp6" "$addresses" \
+			"$gateway4" "$gateway6" "$ns_addresses" "$ns_search" \
+			> "${out_d}/$name.yaml"
+	done
+	rm -Rf "$tmpd"
 }
 
 # Wait for queued kernel/udev events
diff --git a/tests/test_netinfo.d/ipv4/netplan/eth1.yaml b/tests/test_netinfo.d/ipv4/netplan/eth1.yaml
new file mode 100644
index 0000000..3537d9b
--- /dev/null
+++ b/tests/test_netinfo.d/ipv4/netplan/eth1.yaml
@@ -0,0 +1,5 @@
+eth1:
+  dhcp4: true
+  nameservers:
+    addresses: ["192.168.122.1"]
+    search: ["foo.com"]
diff --git a/tests/test_netinfo.d/ipv4and6/netplan/eth0.yaml b/tests/test_netinfo.d/ipv4and6/netplan/eth0.yaml
new file mode 100644
index 0000000..3c5a729
--- /dev/null
+++ b/tests/test_netinfo.d/ipv4and6/netplan/eth0.yaml
@@ -0,0 +1,6 @@
+eth0:
+  dhcp4: true
+  dhcp6: true
+  nameservers:
+    addresses: ["192.168.122.1", "2001:67c:1562:8010::2:1"]
+    search: ["foo.com", "example.com", "ubuntu.com"]
diff --git a/tests/test_netinfo.d/ipv4static/netplan/eth1.yaml b/tests/test_netinfo.d/ipv4static/netplan/eth1.yaml
new file mode 100644
index 0000000..600fea8
--- /dev/null
+++ b/tests/test_netinfo.d/ipv4static/netplan/eth1.yaml
@@ -0,0 +1,4 @@
+eth1:
+  addresses:
+    - "10.0.0.2/24"
+  gateway4: "10.0.0.1"
diff --git a/tests/test_netinfo.d/ipv6/netplan/eno1.yaml b/tests/test_netinfo.d/ipv6/netplan/eno1.yaml
new file mode 100644
index 0000000..303e591
--- /dev/null
+++ b/tests/test_netinfo.d/ipv6/netplan/eno1.yaml
@@ -0,0 +1,5 @@
+eno1:
+  dhcp6: true
+  nameservers:
+    addresses: ["2001:67c:1562:8010::2:1"]
+    search: ["example.com", "ubuntu.com"]
diff --git a/tests/test_netinfo.sh b/tests/test_netinfo.sh
index bd92173..2ebc554 100755
--- a/tests/test_netinfo.sh
+++ b/tests/test_netinfo.sh
@@ -52,6 +52,18 @@ run_resolvconf() {
    pass_on_nodiff "$tname" "$expected" "$found"
 }
 
+run_netinfo_to_netplan() {
+   local bname="$1" testdir="$2" results_d="$3"
+   local tname="$bname-netinfo-to-netplan"
+   shift 3
+   local found_d="$results_d/netplan"
+   local expected_d="$testdir/netplan"
+   [ -d "$expected_d" ] || return $RET_SKIP
+
+   _run_in_functions "$tname" netinfo_to_netplan "$found_d" "$@" || return
+   pass_on_nodiff "$tname" "$expected_d" "$found_d"
+}
+
 record() {
    local ret="$1"
    case "$ret" in
@@ -73,6 +85,9 @@ for testdir in "${data_dir}"/*; do
    run_resolvconf "$dname" "$testdir" "$results_d" $confs
    record $?
 
+   run_netinfo_to_netplan "$dname" "$testdir" "$results_d" $confs
+   record $?
+
    rm -Rf "${results_d}"
 done