Merge ~ubuntu-docker-images/ubuntu-docker-images/+git/utils:update-and-check-images into ~ubuntu-docker-images/ubuntu-docker-images/+git/utils:master

[Bryce Harrington]

Towards the objective of flagging merge opportunities for OCI image updates, and to assist in mechanizing at least the simpler refreshes, this adds a pair of scripts:

update-images.sh handles the mechanics of updating the Dockerfile, building the image, and running the unit tests for verification. It uploads the images to the packager's dockerhub repos, but does not push the images to production (that's a job for push-images.sh). The packager is also responsible for pushing the git branch and filing the MP.

check-images.sh extracts version numbers from the official images, as well as local images such as generated from update-images.sh. This first version of the script is limited to five packages, so will need further development to make it more broadly useful.

I'm not wedded to these names for the scripts.

These have been developed rather independently from push-images.sh, however there are some similar functional needs between all of them. I expect that the commonly used functionality could be broken out to a shared include file. There are likely other adjustments that can be made to make all the scripts consistent with code style, variable naming, etc. as followup work.

[Lucas Kanashiro (lucaskanashiro)]

Thanks for the scripts Bryce. I did not take a deep look into your implementation but based on your description I would say that update-images.sh does not need to upload the image to DockerHub under the packager's namespace, you could simply update the image locally and run the tests against the local image. I can see no gain in uploading it to a personal DockerHub repository, do you have a reason to do it?

review: Needs Information

[Bryce Harrington (bryce)]

The reason was just in case it facilitated easier testing during review. However, if not then I agree that's probably out of scope for this script.

[Sergio Durigan Junior (sergiodj)]

I agree with Lucas: we can run the tests locally, so the part about pushing the tags to a developer namespace seems like it's not needed.

I have a few comments/questions about the script, but other than that it looks great. Thanks for doing it.

review: Needs Fixing

[Bryce Harrington (bryce)]

Discussion on the comments below.

I've dropped the push statement. I'll work on the function codedocs and other refinements for the next rev.

[Bryce Harrington (bryce)]

On Fri, Feb 19, 2021 at 10:19:00PM -0000, Sergio Durigan Junior wrote:
> > +
> > + cd "${REPOS_DIR}/${software_name}" \
> > + || die "Could not chdir to ${REPOS_DIR}/${software_name}"
> > +
> > + git checkout -f edge
>
> Ops, scratch that. Of course you will need to be inside a valid branch to create another one from it. I understand why you're using edge here now. But just as the edge tag will move as we update the images, I think the branch ought to move as well. Maybe the way to go here is to recreate the edge branch off of the current one, once the script finishes the update.
>

Yeah, like I mentioned I expect this section of code will be expanded on
as we sort out how we want to handle branch updates, but I'm not sure
how this should work yet. Next time we do updates to these branches
we'll need to majorly rework this section.

39a0596... by Bryce Harrington on 2021-03-01

check-images: Add support for updating redis

2649b8f... by Bryce Harrington on 2021-03-01

update-images: Resync shared functions with check-images

b76c036... by Bryce Harrington on 2021-03-04

Rename check-images.sh to software-versions.sh and make it sourceable

Also document each of the routines

6542054... by Bryce Harrington on 2021-03-04

Make update-versions.sh source software-versions.sh

This allows avoiding code duplication. Routines pertaining to software
version determination are consolodated to software-versions.sh.

Also document each of the routines.

c86e57d... by Bryce Harrington on 2021-03-04

update-images.sh: Don't create new branches

The original code handled the rename of 'edge' branches to the new
naming scheme. Now that this renaming is completed, there is no need
for this logic.

For now, simply expect that the branch we are going to work in already
exists. This might need further development to create new branches when
there are new major versions of software, but for now assume we'll
handle those situations manually.

f3edcd8... by Bryce Harrington on 2021-03-04

update-images.sh: Also refresh the docs

[Bryce Harrington (bryce)]

I've reworked both scripts a fair bit, and added code docs as requested.

Both scripts still run (for me at least). The software-versions.sh script outputs:

Software Distro Current-Image Proposed-Image
apache2 2.4.46-2ubuntu1 2.4.41-4ubuntu3.1 2.4.46-2ubuntu1
memcached 1.6.9+dfsg-1 1.6.9+dfsg-1 1.6.9+dfsg-1
mysql-8.0 8.0.23-3build1 8.0.23-3build1 8.0.23-1ubuntu2
nginx 1.18.0-6ubuntu4 1.18.0-6ubuntu4 -
postgresql-13 13.2-1 - -
redis 5:6.0.11-1 5:6.0.9-4 -

I don't consider either script 'finished' by any stretch, but this is a good checkpoint for review purposes.

Future work includes:
* Add support for telegraf(?) and the snap-based images(???)
* More error checking
* Use log()/error() from push-images.sh instead of die()
* Output version data as JSON (for feeding into merge opportunity report)
* Ability to update branches for stable Ubuntu releases too (i.e. not just the devel release)
* Ability to appropriately create branches as needed for new major versions
* Less hard-coding of package-specific information, perhaps by relying on data/<package>.yaml
* Add use of getopts and better usage()

3eb648b... by Bryce Harrington on 2021-03-15

software-versions.sh: Special-case postgres' software version

Minor version bumps of postgres are allowed in stable Ubuntu releases,
per security team policy. Thus for tagging OCI images we should
reference only the major version in this particular case.

[Bryce Harrington (bryce)]

I've done a reimplementation on software-versions.sh, with the goal of establishing a better git history for review, and to further generalize handling of package-specific stuff. I'll reintroduce this script with a new MP.

Regarding the update-images.sh script, there has been a bit of fluidity in tag naming policies and in our requirements for how things need updated. Due to this I don't feel we're at a point we can really nail down the requirements for this script. I'm going to withdraw this script for now; it can be reimplemented and reintroduced when the maintenance situation is more stable.

Unmerged commits

3eb648b... by Bryce Harrington on 2021-03-15

software-versions.sh: Special-case postgres' software version

Minor version bumps of postgres are allowed in stable Ubuntu releases,
per security team policy. Thus for tagging OCI images we should
reference only the major version in this particular case.

f3edcd8... by Bryce Harrington on 2021-03-04

update-images.sh: Also refresh the docs

c86e57d... by Bryce Harrington on 2021-03-04

update-images.sh: Don't create new branches

The original code handled the rename of 'edge' branches to the new
naming scheme. Now that this renaming is completed, there is no need
for this logic.

For now, simply expect that the branch we are going to work in already
exists. This might need further development to create new branches when
there are new major versions of software, but for now assume we'll
handle those situations manually.

6542054... by Bryce Harrington on 2021-03-04

Make update-versions.sh source software-versions.sh

This allows avoiding code duplication. Routines pertaining to software
version determination are consolodated to software-versions.sh.

Also document each of the routines.

b76c036... by Bryce Harrington on 2021-03-04

Rename check-images.sh to software-versions.sh and make it sourceable

Also document each of the routines

2649b8f... by Bryce Harrington on 2021-03-01

update-images: Resync shared functions with check-images

39a0596... by Bryce Harrington on 2021-03-01

check-images: Add support for updating redis

087c41b... by Bryce Harrington on 2021-02-19

Add initial version of check-images.sh script

This script can be used to examine the versions of software installed in
the docker images. Currently, this script only works with apache2,
memcached, mysql, nginx, and postgres, and has only been lightly tested
even with them.

Future work includes cli support to allow specifying the software to
be examined, better error checking, and ability to output in some
structured file format (i.e. JSON and/or YAML). Better command line
argument handling would likely also help...

Further, this copies some functionality from update-images.sh that
should be moved into a common file and sourced.

7638fb8... by Bryce Harrington on 2021-02-19

Add preliminary script 'update-images.sh'

This script is intended to facilitate simple refreshes of docker images
using the version in the current devel release of Ubuntu. This has only
been tested against apache2, memcached, mysql, nginx, and postgres.
Other images might work, but YMMV.

In particular, if adjustments need to be made to the Dockerfile to
facilitate the new software, this script probably won't be sufficient to
do the update.

Note that the docker image name, test script name, source package name,
and binary package name for a given piece of software may be
considerably different. To address that, we just brute force hack
around it by hardcoding some name translations; hopefully later on this
can be done better.

In particular, some software names include major version numbers
(e.g. to indicate API levels), such as mysql-8.0 and postgresql-13.
This script includes some logic to figure these numbers out, which may
or may not work more broadly. Again, it's hoped this can be replaced by
something more robust eventually.

diff --git a/software-versions.sh b/software-versions.sh
new file mode 100755
index 0000000..4172ce8
--- /dev/null
+++ b/software-versions.sh
@@ -0,0 +1,229 @@
+#!/usr/bin/env bash
+# -*- coding: utf-8; mode: sh -*-
+
+# software-versions.sh:  Determine versions of packages installed in OCI images
+
+progname=$(basename "${0}")
+progdir=$(dirname "$(readlink -f "${0}")")
+
+if [ -z "${DOCKER_USER}" ]; then
+    DOCKER_USER=$(docker info | grep Username: | cut -d: -f2 | xargs)
+fi
+if [ -z "${DISTRO_CODENAME}" ]; then
+    DISTRO_CODENAME=$(distro-info --devel)
+fi
+
+die() {
+    echo "Error:  ${*}" 1>&2
+    exit 1
+}
+
+
+# Converts a package version number to equivalent upstream version.
+# Strips off epoch, if any, and drops the debian version.
+#
+# :param v: Package version number to convert
+upstream_version() {
+    local v="${1#*:}"
+    echo "${v%-*}"
+}
+
+# Look up version of package in a given Ubuntu release
+#
+# @param source_package_name: Name of the source package to look up
+get_source_package_version() {
+    local source_package_name="${1}"
+
+    rmadison -s "${DISTRO_CODENAME}" "${source_package_name}" | cut -d\| -f2 | xargs
+}
+
+# Provides the "version family" of the software
+#
+# For a given Ubuntu stable release, updates will be limited to certain
+# ranges of version numbers based on API and stability policies.  For
+# example, some may get patch-level updates (i.e. M.N.1->M.N.2->M.N.3)
+# and thus have an "M.N" version family, others may get minor-level
+# updates (such as postrgres, which in focal had 12.4->12.5->12.6) so
+# have an "M" version family.
+#
+# This routine determines this from the debian version by extracting the
+# first two numbers from upstream version portion.
+#
+# @param software_name: Formal name of the upstream software
+get_source_package_major() {
+    # TODO: This routine should be named get_software_version_family
+    #       or get_software_main_version or something similar
+    local software_name="${1}"
+    local package_version=""
+
+    # For apache2, just return 2.4
+    if [ "${software_name}" = "apache2" ]; then
+	package_version=$(get_source_package_version apache2)
+
+    # For postgres return 12, 13, etc.
+    elif [ "${software_name}" = "postgres" ]; then
+	pg_api_version=$(get_source_package_version postgresql)
+	pg_api_package="postgresql-${pg_api_version%+*}"
+	package_version=$(get_source_package_version "${pg_api_package}")
+
+    # For mysql return 5.7, 8.0, etc.
+    elif [ "${software_name}" = "mysql" ]; then
+	package_version=$(get_source_package_version mysql-server)
+
+    elif [ "${software_name}" = "redis" ]; then
+	package_version=$(get_source_package_version redis-server)
+    fi
+
+    if [ -n "${package_version}" ]; then
+	version=$(upstream_version "${package_version}")
+	local v=( ${version//./ } )
+	local version_major="${v[0]}"
+	local version_minor="${v[1]}"
+	if [ "${software_name}" = "postgres" ]; then
+	    echo "${version_major}"
+	else
+	    echo "${version_major}.${version_minor}"
+	fi
+    fi
+
+    # For everything else, return blank
+    echo ""
+}
+
+# Determine the source package for the given software name
+#
+# In many cases, the software name is identical to the source package's
+# name, however there are cases where the spelling may vary
+# (e.g. postgres vs. postgresql), or where the source package includes
+# part of the version number.  This routine does this mapping.
+#
+# @param software_name: Formal name of the upstream software
+get_software_source_package() {
+    local software_name="${1}"
+    local source_package_name=""
+
+    # Look to see if there is a <software>-N.M package
+    local package_major=$(get_source_package_major "${software_name}")
+    if [ -n "${package_major}" ]; then
+	local prefix="${software_name}"
+	if [ "${software_name}" = "postgres" ]; then
+	    prefix="postgresql"
+	fi
+
+	source_package_name=$(rmadison -s "${DISTRO_CODENAME}" "${prefix}-${package_major}" | cut -d\| -f1 | xargs)
+	if [ -n "${source_package_name}" ] && [ "${source_package_name}" != "${prefix}-${package_major}" ]; then
+	    die "error: ${source_package_name} is not ${prefix}-${package_major} as expected"
+	fi
+    fi
+
+    # If not, default to just <software>
+    if [ -z "${source_package_name}" ]; then
+	source_package_name="${software_name}"
+    fi
+
+    echo "${source_package_name}"
+    return 0
+}
+
+# Get primary binary package for given software
+#
+# For some software, the binary package names differ from the source
+# package name, so when checking installed versions we need to be careful
+# to use the appropriate binary package name.  For now, we just manually
+# translate a few specific cases.
+#
+# @param software_name: Formal name of the upstream software
+get_software_binary_package() {
+    local software_name="${1}"
+    local package_major=$(get_source_package_major "${software_name}")
+
+    if [ "${software_name}" = "mysql" ]; then
+	# mysql-client-core-N and mysql-client-N are also installed
+	echo "mysql-server-core-${package_major}"
+    elif [ "${software_name}" = "postgres" ]; then
+	echo "postgresql-${package_major}"
+    elif [ "${software_name}" = "nginx" ]; then
+	# nginx-common is also installed
+	echo "nginx-full"
+    elif [ "${software_name}" = "redis" ]; then
+	echo "redis-server"
+    else
+	echo "${software_name}"
+    fi
+}
+
+# Lookup a binary package version present in a given Docker image
+#
+# Uses apt-cache policy to determine installed version of a specific
+# binary package.
+#
+# @param image_id: The name of the image to examine
+# @param binary_package_name: The binary package to look up version info
+get_image_installed_package_version() {
+    local image_id="${1}"
+    local binary_package_name="${2}"
+
+    [ -n "${image_id}" ] || return 1
+
+    docker run --rm \
+	   --name "${image_id}-version-check" \
+	   "${image_id}" \
+	   apt-cache policy "${binary_package_name}" \
+	| grep "Installed:" | cut -d: -f2- | xargs
+}
+
+# Display a table entry for the given software
+#
+# This is an aide for summarizing versions of software packages in the
+# distro, vs. what's in the existing ubuntu-namespaced Docker images.
+# It also displays versions of the package in the user's namespace
+# as a convenience for development work.
+#
+# @param software_name: Formal name of the upstream software
+get_software_versions() {
+    # For given software_name:
+    #    Create container from the image
+    #    Log into the container
+    #    Retrieve the version for software_name
+    software_name="${1}"
+    source_package_name="$(get_software_source_package ${software_name})"
+    binary_package_name="$(get_software_binary_package ${software_name})"
+    proposed_image_name="${DOCKER_USER}/${software_name}"
+    current_image_name="ubuntu/${software_name}"
+
+    proposed_image_id=$(docker image ls | grep ^${proposed_image_name} | grep edge | awk '{print $3}' | uniq)
+    current_image_id=$(docker image ls | grep ^${current_image_name} | grep edge | awk '{print $3}' | uniq)
+    package_version_distro=$(get_source_package_version "${source_package_name}")
+    package_version_current=$(get_image_installed_package_version "${current_image_id}" "${binary_package_name}")
+    package_version_proposed=$(get_image_installed_package_version "${proposed_image_id}" "${binary_package_name}")
+
+    printf "%-20s %-30s %-30s %-30s\n" "${source_package_name}" \
+	   "${package_version_distro:--}" "${package_version_current:--}" "${package_version_proposed:--}"
+}
+
+# Tabular report of Canonical Server's debian-based docker images
+#
+# Creates a summary of package version numbers for software installed
+# in docker images that the canonical server team maintains.
+display_version_report() {
+    docker image pull ubuntu/apache2:edge
+    docker image pull ubuntu/memcached:edge
+    docker image pull ubuntu/mysql:edge
+    docker image pull ubuntu/nginx:edge
+    docker image pull ubuntu/postgres:edge
+    docker image pull ubuntu/redis:edge
+
+    printf "%-20s %-30s %-30s %-30s\n" "Software" "Distro" "Current-Image" "Proposed-Image"
+    get_software_versions "apache2"
+    get_software_versions "memcached"
+    get_software_versions "mysql"
+    get_software_versions "nginx"
+    get_software_versions "postgres"
+    get_software_versions "redis"
+}
+
+# If we're being run directly (i.e. not sourced), then
+# print out the current software versions.
+if [[ "${0}" == "${BASH_SOURCE[0]}" ]]; then
+    display_version_report
+fi
diff --git a/update-images.sh b/update-images.sh
new file mode 100755
index 0000000..ecea08a
--- /dev/null
+++ b/update-images.sh
@@ -0,0 +1,150 @@
+#!/usr/bin/env bash
+# -*- coding: utf-8; mode: sh -*-
+
+# update-images.sh:  Refresh software versions for Ubuntu's OCI images
+
+progname=$(basename "${0}")
+progdir=$(dirname "$(readlink -f "${0}")")
+
+# shellcheck source=/dev/null disable=SC1090
+source "${progdir}/software-versions.sh"
+
+if ! groups | grep docker > /dev/null; then
+    echo "Error: Must be in docker group" 1>&2
+    exit 1
+fi
+
+if [ -z "${TESTS_DIR}" ]; then
+    TESTS_DIR="${progdir}/../server-test-scripts/oci-unit-tests"
+fi
+if [ -z "${REPOS_DIR}" ]; then
+    REPOS_DIR="${progdir}/../"
+fi
+
+usage() {
+    cat >&2 <<EOF
+Usage: ${progname} <package-name> [package-name ...]
+
+EOF
+    # shellcheck source=/dev/null disable=SC2086
+    exit "${1}"
+}
+
+# Modifies contents of the Dockerfile in the current directory
+#
+# @param software_name:  Formal name of the upstream software for the Dockerfile
+update_dockerfile_parameters() {
+    local software_name="${1}"
+    local dockerfile="./Dockerfile"
+
+    # TODO: Given that we're in the git repository, can we determine the
+    #       software_name from just that?  If so, then this can run without
+    #       argument $1.
+
+    if [ ! -e "${dockerfile}" ]; then
+	die "error: Dockerfile ${dockerfile} not found at ${PWD}"
+    fi
+
+    # Update global parameters
+    sed -i "/FROM ubuntu:/c\FROM ubuntu:${DISTRO_CODENAME}" "${dockerfile}"
+
+    # Update per-package parameters
+    if [ "${software_name}" = "postgres" ]; then
+	pg_major=$(get_source_package_major postgresql)
+	sed -i "/ENV PG_MAJOR/c\ENV PG_MAJOR ${pg_major}" "${dockerfile}"
+    fi
+
+    return 0
+}
+
+# Creates branch for a new Ubuntu distro series and updates package to match
+#
+# This routine can be run to create an updated image for a new version
+# of Ubuntu.  It sets the Dockerfile to the new Ubuntu version, and
+# rebuilds it with the version of the software's package from that
+# release.
+#
+# @param software_name:  Formal name of the upstream software for the Dockerfile
+update_image() {
+    [ -n "${1}" ] \
+	|| usage 1
+
+    local software_name="${1}"
+    local package_name=$(get_software_source_package "${software_name}")
+    local package_version=$(get_source_package_version "${package_name}")
+    local package_major=$(get_source_package_major "${software_name}")
+    local upstream_version=$(upstream_version "${package_version}")
+    # TODO: Remove +dfsg
+    local v=( ${upstream_version//./ } )
+    local version_major="${v[0]}"
+    local version_minor="${v[1]}"
+    local docker_image="${DOCKER_USER}/${software_name}:21.04"
+    local test_script="${TESTS_DIR}/${software_name}_test.sh"
+    local distro_release=$(distro-info --series "${DISTRO_CODENAME}" -r)
+    local branch_name="${version_major}.${version_minor}-${distro_release}"
+
+    # Hack, due to postgres/postgresql discrepancy
+    if [ "${software_name}" = "postgres" ]; then
+	test_script="${TESTS_DIR}/postgresql_test.sh"
+    fi
+
+    echo
+    echo "########################################################"
+    echo "# distro:       ${DISTRO_CODENAME} (${distro_release})"
+    echo "# software:     ${software_name} -> ${package_name} ${package_major}"
+    echo "# docker_image: ${docker_image}"
+    echo "# test_script:  ${test_script}"
+    echo "# version:      ${package_version}"
+    echo "# upstream:     ${upstream_version}"
+    echo "# branch_name:  ${branch_name}"
+    echo "########################################################"
+
+    cd "${REPOS_DIR}/${software_name}" \
+	|| die "Could not chdir to ${REPOS_DIR}/${software_name}"
+
+    git checkout "${branch_name}" \
+       || die "Could not checkout ${branch_name} from git"
+
+    # Update Dockerfile parameters
+    echo
+    echo "### Updating Dockerfile ###"
+    update_dockerfile_parameters "${software_name}"
+
+    # Update the docs
+    make all-doc \
+	|| die "Could not update the documentation"
+
+    # Build the image
+    echo
+    echo "### Building Image ###"
+    docker build --tag "${docker_image}" . \
+	|| die "Failed to build ${docker_image} for ${package_name}"
+
+    git commit -a -m "Update docker image to ${package_version} for ${DISTRO_CODENAME}"
+
+    # Run the test
+    echo
+    echo "### Testing Image ###"
+    if [ ! -e "${test_script}" ]; then
+	die "Test script for ${software_name} not found at ${test_script}"
+    fi
+    cd "${TESTS_DIR}" \
+	|| die "Could not chdir to ${TESTS_DIR}"
+    DOCKER_IMAGE="${docker_image}" bash ${test_script} \
+	|| die "Failed ${docker_image}'s test for ${package_name}"
+
+    exit 0
+}
+
+if [[ "${0}" == "${BASH_SOURCE[0]}" ]]; then
+    [ -n "${1}" ] \
+	|| usage 1
+
+    for package_name in ${@}; do
+	echo "Updating ${package_name}"
+	update_image "${package_name}"
+	echo
+	echo "---"
+	echo
+    done
+fi