Merge ~ahasenack/ubuntu/+source/base-files:eoan-cloud-id-support into ubuntu/+source/base-files:ubuntu/devel

[Andreas Hasenack]

client-side change to implement MOTD for clouds.

This simply adds a cloud_id string to the existing user-agent. While making that change, I noticed that curl wasn't called with -f, which meant that should the server fail and present an error page, that would be taken as valid motd text. With -f, curl will silently fail in that case, and we will have an empty motd instead of garbage.

cloud-id comes from cloud-init, but rather than adding a dependency, I'm assuming it will be available in ubuntu cloud instances, but of course I look for it first.

It's unknown if an external attacker can manipulate the output of cloud-id. If that's possible, then this cloud-id usage is a security hole, as it's taken unfiltered. I could add a "tr" call to filter out some things, like shell special characters and control characters like it's done for the motd text itself. I can easily be convinced to do something like this, but I want to get some eyes on this implementation sooner rather than later.

The server side bit that uses this user agent string is basically this snippet:
   RewriteEngine on
   # list all clouds in the regexp
   RewriteCond "%{HTTP_USER_AGENT}" "cloud_id/(lxd|openstack|....)"
   RewriteRule "^/$" "/index-%1.txt" [L]

One could also use RewriteMap and point to a text file which lists the clouds, so no apache config change is needed when a cloud is added or removed. And I suppose there are other ways in Apache to serve a specific file depending on the user agent, other than mod_rewrite. Simpler suggestions accepted.

The respective index-$cloud.txt files are expected to come from cloud-specific branches, which the mojo spec[1] that deploys this service will fetch, as it does today for the single index.txt. Something like this, as a first cut:
$ cat collect-ubuntu-motd
ubuntu-motd lp:ubuntu-motd
ubuntu-motd-aws lp:ubuntu-motd-aws
ubuntu-motd-gce lp:ubuntu-motd-gce
...
$ cat manifest-upload
collect config=collect-ubuntu-motd
collect config=collect-ubuntu-motd-aws
collect config=collect-ubuntu-motd-gce
...
script config=upload-motd.sh
include config=manifest-verify

With this system in place, cloud-specific index files will override the default index.txt one.

If an instance on cloud X comes alive, and there is an index-x.txt file, that will be the sole motd. If there is no index-x.txt file, then index.txt is served (index.txt is also the 404 error document in the apache config).

With this scheme, in order for the common index.txt file be again served as motd for a cloud instance, the index-$cloud.txt file must be deleted.

1. https://bazaar.launchpad.net/~canonical-is/canonical-mojo-specs/trunk/files/head%3A/is/motd-ubuntu/

[Andreas Hasenack (ahasenack)]

Pushed

[Andreas Hasenack (ahasenack)]

I could add this filter to cloud-id's output:
 | tr -c -d '[:alnum:]'

Example:

$ cat ~/foo -vet
"'$^M^I\`()&#1achACH2^?"<>."'$\`()&#$

$ cat ~/foo | tr -c -d '[:alnum:]'
1achACH2

It will delete (-d) anything that is not (-c) alphanumeric ([:alnum:]).

[Steve Langasek (vorlon)]

Is there a specification for what 'cloud-id' returns across clouds? Is this guaranteed to always be the cloud "name"? (E.g. 'azure' or 'gce' or 'openstack' and nothing more) If so, that seems reasonable to me, in general terms it isn't exposing any information that couldn't be correlated by other means, it's just doing so more efficiently.

review: Approve

[Andreas Hasenack (ahasenack)]

There is no such clear cut specification. I asked cloud-init, but there wasn't a single document or file that had such a list. The closest I got was:

- fgrep -r 'dsname =' cloudinit/sources (result is still mangled, at least with a .lower())
- https://hackmd.io/aAJFerDRRNaoL0Q_vDRYBg?both

But with a remark that "Cloud metadata services can override though if they provide a "cloud-name" key in their metadata"

[Andreas Hasenack (ahasenack)]

Got some input from the security team as well, so I'm going with the filtering approach.

Tagging and uploading 65bc858eb5ec56b2061f570fcdba48a3fcae1dce

$ git push pkg upload/10.2ubuntu5
Enumerating objects: 36, done.
Counting objects: 100% (36/36), done.
Delta compression using up to 2 threads
Compressing objects: 100% (31/31), done.
Writing objects: 100% (31/31), 3.26 KiB | 477.00 KiB/s, done.
Total 31 (delta 22), reused 0 (delta 0)
To ssh://git.launchpad.net/~usd-import-team/ubuntu/+source/base-files
 * [new tag] upload/10.2ubuntu5 -> upload/10.2ubuntu5

$ dput ubuntu ../base-files_10.2ubuntu5_source.changes
Checking signature on .changes
gpg: ../base-files_10.2ubuntu5_source.changes: Valid signature from AC983EB5BF6BCBA9
Checking signature on .dsc
gpg: ../base-files_10.2ubuntu5.dsc: Valid signature from AC983EB5BF6BCBA9
Uploading to ubuntu (via ftp to upload.ubuntu.com):
  Uploading base-files_10.2ubuntu5.dsc: done.
  Uploading base-files_10.2ubuntu5.tar.xz: done.
  Uploading base-files_10.2ubuntu5_source.buildinfo: done.
  Uploading base-files_10.2ubuntu5_source.changes: done.
Successfully uploaded packages.

diff --git a/debian/changelog b/debian/changelog
index d550558..5054e04 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,9 @@
+base-files (10.2ubuntu5) eoan; urgency=medium
+
+  * motd/50-motd-news: add cloud_id to user-agent string (LP: #1840946)
+
+ -- Andreas Hasenack <[e-mail]>  Thu, 15 Aug 2019 14:57:39 -0300
+
 base-files (10.2ubuntu4) eoan; urgency=medium
 
   * debian/motd-news.timer: Change the timer to use an OnCalendar entry
diff --git a/motd/50-motd-news b/motd/50-motd-news
index e031cdf..026429f 100644
--- a/motd/50-motd-news
+++ b/motd/50-motd-news
@@ -84,13 +84,18 @@ codename="$DISTRIB_CODENAME"
 platform="$(uname -o)/$(uname -r)/$(uname -m)"
 arch="$(uname -m)"
 cpu="$(grep -m1 "^model name" /proc/cpuinfo | sed -e "s/.*: //" -e "s:\s\+:/:g")"
+if [ -x /usr/bin/cloud-id ]; then
+    cloud_id=$(/usr/bin/cloud-id 2>/dev/null | cut -c -40 | tr -c -d '[:alnum:]')
+else
+    cloud_id="unknown"
+fi
 
 # Some messages may only be pertinent before or after some amount of uptime
 read up idle < /proc/uptime
 uptime="uptime/$up/$idle"
 
 # Piece together the user agent
-USER_AGENT="curl/$curl_ver $lsb $platform $cpu $uptime"
+USER_AGENT="curl/$curl_ver $lsb $platform $cpu $uptime cloud_id/$cloud_id"
 
 # Loop over any configured URLs
 for u in $URLS; do