Merge ~ubuntu-docker-images/ubuntu-docker-images/+git/utils:software-versions into ~ubuntu-docker-images/ubuntu-docker-images/+git/utils:master

[Bryce Harrington]

This is not functionally complete yet but is at a good checkpoint to review.

This is a top-down rewrite of my original POC, with commits split up and organized better for review. The implementation focus for this part is determining the Ubuntu archive version information, for both focal and hirsute. (The other part still to come is determining the package version in the corresponding OCI package, but that will be followup.)

The script can be run in one of two ways:

    $ software-versions.sh --format text
    $ software-versions.sh --format json

The first form is the default if no args are provided. Both should provide equivalent data, just displayed differently. The latter is intended for machine consumption so must always generate valid JSON data. There is a json-check script in ubuntu-helpers that can do this verification.

While this only prints out info for hirsute and focal, I am developing it with an eye towards eventually needing to add maybe xenial + bionic + i-series, although I've not tested those yet.

One other possible future change is that currently the list of packages is hardcoded. I hope to maybe move package-specific data into data files, maybe included in the package oci repositories. Thoughts on that would be appreciated. At this point there's just two package-specific bits - the M.N style needed for the source package name, and the identity of the primary binary package installed from the source package.

[Sergio Durigan Junior (sergiodj)]

Thanks for the MP, Bryce.

The very first thing I did was to run shellcheck on the script. I've been trying to keep all of these scripts shellcheck-free (except the "push-images.sh", which is an old script and might not even be needed anymore). Could you fix the shellcheck warnings/notices, please?

Now, about the feature itself: I like it. I think it's a nice addition to our swiss-knife kit. I have a few comments/questions on the code, let me know what you think.

c7fd886... by Bryce Harrington on 2021-04-13

software_versions: Fix shellcheck issues

Mostly changing the local declarations, and a few quoted variables.
There was also a typo in the num_software decremental.

f030a57... by Bryce Harrington on 2021-04-13

software-versions: Add mention of --format to usage()

fb76f34... by Bryce Harrington on 2021-04-13

software-versions: Omit prometheus and other snap-based packages

(For now anyway...)

328e350... by Bryce Harrington on 2021-04-13

software-versions.sh: Set $CODENAMES as read-only

[Bryce Harrington (bryce)]

Thanks for reviewing!

On Tue, Apr 13, 2021 at 01:21:05AM -0000, Sergio Durigan Junior wrote:
> Thanks for the MP, Bryce.
>
> The very first thing I did was to run shellcheck on the script. I've been trying to keep all of these scripts shellcheck-free (except the "push-images.sh", which is an old script and might not even be needed anymore). Could you fix the shellcheck warnings/notices, please?
>

Done. One of the shellcheck flags actually revealed a legit bug.

> Now, about the feature itself: I like it. I think it's a nice addition to our swiss-knife kit. I have a few comments/questions on the code, let me know what you think.
>
> Diff comments:
>
> > diff --git a/software-versions.sh b/software-versions.sh
> > new file mode 100755
> > index 0000000..895982a
> > --- /dev/null
> > +++ b/software-versions.sh
> > @@ -0,0 +1,273 @@
> > +#!/bin/bash
> > +# -*- coding: utf-8; mode: sh -*-
> > +#
> > +# software-versions.sh: Determine versions of packages installed in OCI images
> > +
> > +progname=$(basename "${0}")
> > +progdir=$(dirname "$(readlink -f "${0}")")
> > +
> > +# shellcheck disable=SC1090
> > +source "${progdir}/helpers/log.sh"
> > +
> > +###############
> > +### Options ###
> > +###############
> > +
> > +declare -a ARGS
> > +
> > +# Whether to pull latest versions of the images
> > +opt_report_format="text"
> > +
> > +## Explains how to use this command.
> > +usage() {
> > + cat > /dev/stderr <<EOF
> > +${progname} - Determine versions of packages installed in OCI images.
> > +
> > +This script can either be run stand-alone to display a text report or
> > +JSON data structure, or can be sourced into other bash scripts for its
> > +version functionality.
> > +
> > +Usage:
> > +
> > + ${progname} [options]
> > +
> > +Arguments:
> > + -h|--help Display this usage
>
> I think you can mention the "--format" option here.

thx

> > +EOF
> > +}
> > +
> > +## Parses command line arguments
> > +##
> > +## @param *: Command line to be parsed.
> > +parse_args() {
> > + ARGS=()
> > +
> > + # Transform long options to short ones
> > + for arg in "${@}"; do
> > + shift
> > + case "${arg}" in
> > + "--help") set -- "${@}" "-h" ;;
> > + "--format") set -- "${@}" "-F" ;;
> > + "--*") usage; exit 1 ;;
> > + *) set -- "${@}" "${arg}" ;;
> > + esac
> > + done
> > +
> > + # Parse short options
> > + while [ ${#} -gt 0 ]; do
> > + local OPTIND=1
> > + while getopts "hF:\?" opt "${@}"; do
> > + case "${opt}" in
> > + h | \?) usage; exit 0; ;;
> > + F) opt_report_format="${OPTARG}" ;;
> > + *) error "Unknown option '${opt}'" ;;
> > + esac
> > + done
> > + shift "$((OPTIND - 1))"
> > + ARGS+=("${1}")
> > + shift
> > + done
> > + return 0
> > +}
> > +
> > +
> > +###############################
> > +### Source Package Versions ###
> > +###############################
> > +
> > +## Converts a package version number to equivalent upstream version.
> > +## Strips off epoch, if any, and drops the debian version.
> > +##
> > +## @param v:...

[Sergio Durigan Junior (sergiodj)]

Awesome, thanks Bryce. This version LGTM. +1.

review: Approve

[Athos Ribeiro (athos-ribeiro)]

The multi-arch-tagger script has its own README file describing some script dependencies.

Should we have something similar for this new script (i.e., somewhere documenting it needs distro-info installed) or maybe get it to fail gracefully in case the distro-info is not available in the current $PATH?

[Bryce Harrington (bryce)]

Athos, yes you're right, thanks. I'll include that task in the next MP for this script.

diff --git a/software-versions.sh b/software-versions.sh
new file mode 100755
index 0000000..1bb3125
--- /dev/null
+++ b/software-versions.sh
@@ -0,0 +1,273 @@
+#!/bin/bash
+# -*- coding: utf-8; mode: sh -*-
+#
+# software-versions.sh:  Determine versions of packages installed in OCI images
+
+progname=$(basename "${0}")
+progdir=$(dirname "$(readlink -f "${0}")")
+
+# shellcheck disable=SC1090
+source "${progdir}/helpers/log.sh"
+
+###############
+### Options ###
+###############
+
+declare -a ARGS
+
+# Whether to pull latest versions of the images
+opt_report_format="text"
+
+## Explains how to use this command.
+usage() {
+    cat > /dev/stderr <<EOF
+${progname} - Determine versions of packages installed in OCI images.
+
+This script can either be run stand-alone to display a text report or
+JSON data structure, or can be sourced into other bash scripts for its
+version functionality.
+
+Usage:
+
+  ${progname} [options]
+
+Arguments:
+    -h|--help                            Display this usage
+    --format [text|json]                 Whether to print results as JSON or text (default)
+EOF
+}
+
+## Parses command line arguments
+##
+## @param *: Command line to be parsed.
+parse_args() {
+    ARGS=()
+
+    # Transform long options to short ones
+    for arg in "${@}"; do
+	shift
+	case "${arg}" in
+	    "--help")     set -- "${@}" "-h"                 ;;
+	    "--format")   set -- "${@}" "-F"                 ;;
+	    "--*")        usage; exit 1                      ;;
+	    *)            set -- "${@}" "${arg}"             ;;
+	esac
+    done
+
+    # Parse short options
+    while [ ${#} -gt 0 ]; do
+	local OPTIND=1
+	while getopts "hF:\?" opt "${@}"; do
+	    case "${opt}" in
+		h | \?)   usage; exit 0;                     ;;
+		F)        opt_report_format="${OPTARG}"      ;;
+		*)        error "Unknown option '${opt}'"    ;;
+	    esac
+	done
+	shift "$((OPTIND - 1))"
+	ARGS+=("${1}")
+	shift
+    done
+    return 0
+}
+
+
+###############################
+### Source Package Versions ###
+###############################
+
+## Converts a package version number to equivalent upstream version.
+## Strips off epoch, if any, and drops the debian version.
+##
+## @param v: Package version number to convert
+upstream_version() {
+    local v="${1#*:}"
+    [ -n "${v}" ] || return 1
+
+    local v2="${v%-*}"
+    echo "${v2%+*}"
+}
+
+## Look up version of source or binary package in a given Ubuntu release.
+##
+## @param package_name: Name of the source or binary package to look up.
+## @param distro_codename: Ubuntu release name to check.
+## @param type: Lookup either "source" or "binary" packages.
+package_version() {
+    local package_name="${1}"
+    local distro_codename="${2}"
+    local type="${3:-source}"
+    [ -n "${package_name}" ] || return 1
+    [ -n "${type}" ] || return 1
+    [ -n "${distro_codename}" ] || return 1
+
+    local suites="${distro_codename}"
+    if [ "${distro_codename}" != "$(distro-info --devel)" ]; then
+	# Not a devel version, need to consider -updates and -security, too
+	suites="${distro_codename},${distro_codename}-updates,${distro_codename}-security"
+    fi
+
+    local source_or_binary
+    if [ "${type}" = "source" ]; then
+	source_or_binary="-a \"source\""
+    fi
+
+    rmadison \
+	"${source_or_binary}" \
+	-s "${suites}" \
+	"${package_name}" \
+	| cut -d\| -f2 \
+	| sort --version-sort \
+	| tail -n 1 \
+	| xargs
+}
+
+source_package() {
+    local template v M N
+    template="${1}"
+    v=( "${2//./ }" )
+    M="${v[0]}"
+    N="${v[1]}"
+
+    echo "${template}" | \
+	sed -e "s/{M}/${M}/g" | \
+	sed -e "s/{N}/${N}/g" | \
+	sed -e "s/{M.N}/${M}.${N}/g"
+}
+
+
+#########################
+### Software Versions ###
+#########################
+
+## Returns the M.N version from a package version.
+##
+## We define "software version" to mean the initial two values in the
+## upstream package version number, i.e. the 'major' and 'minor' version
+## numbers.  Epoch numbers, debian revisions, alpha/beta indicators, and
+## other such components are excluded.
+##
+## @param str version: The package version to extract M.N from.
+## @stdout: The M.N version string.
+software_version() {
+    local version v version_major version_minor
+    version=$(upstream_version "${1}")
+
+    v=( "${version//./ }" )
+    version_major="${v[0]}"
+    version_minor="${v[1]}"
+
+    echo "${version_major}.${version_minor}"
+}
+
+############
+### Main ###
+############
+
+## Top level entry point for the 'software-versions' script
+##
+## Expects to be passed the command line parameter string.
+main() {
+    # Parse options
+    parse_args "${@}" \
+	|| error "Could not process options"
+    set -- "${ARGS[@]}"
+
+    # Debian packages in focal
+    declare -A SOFTWARE=(
+	# Software    Source                               Binary
+	[apache2]="   apache{M}                            apache2"
+	[memcached]=" memcached                            memcached"
+	[mysql]="     mysql-{M.N}                          mysql-server"
+	[nginx]="     nginx                                nginx"
+	[postgres]="  postgresql-{M}                       postgresql"
+	[redis]="     redis                                redis-server"
+    )
+
+    declare -ar CODENAMES=(
+	"focal"
+	"hirsute"
+    )
+
+    case "${opt_report_format}" in
+	"text")
+	    # Generate text table report
+	    printf "%-25s " "Software"
+	    printf "%-50s " "${CODENAMES[@]::${#CODENAMES[@]}-1}"
+	    printf "%s "    "${CODENAMES[-1]}"
+	    printf "\n"
+	    for software_name in "${!SOFTWARE[@]}"; do
+		local RECORD=() source_package_template binary_package_name
+
+		source_package_template=$(echo "${SOFTWARE[${software_name}]}" | awk '{print $1}')
+		binary_package_name=$(echo "${SOFTWARE[${software_name}]}" | awk '{print $2}')
+
+		local codename
+		for codename in "${CODENAMES[@]}"; do
+		    local binary_package_version software_version source_package version version_upstream
+		    binary_package_version=$(package_version "${binary_package_name}" "${codename}" "binary" )
+		    software_version=$(software_version "${binary_package_version}")
+		    source_package=$(source_package "${source_package_template}" "${software_version}")
+		    version=$(package_version "${source_package}" "${codename}" "source")
+		    version_upstream=$(upstream_version "${version}")
+		    RECORD+=("${version} (${version_upstream})")
+		done
+
+		printf "%-25s " "${software_name}:"
+		printf "%-50s " "${RECORD[@]::${#RECORD[@]}-1}"
+		printf "%s "    "${RECORD[-1]}"
+		printf "\n"
+	    done
+	    ;;
+	"json")
+	    # Dump data as JSON data
+	    echo "["
+	    local num_software=${#SOFTWARE[@]}
+	    for software_name in "${!SOFTWARE[@]}"; do
+		local RECORD=() source_package_template binary_package_name
+		source_package_template=$(echo "${SOFTWARE[${software_name}]}" | awk '{print $1}')
+		binary_package_name=$(echo "${SOFTWARE[${software_name}]}" | awk '{print $2}')
+		echo "    {"
+		echo "	\"software_name\":            \"${software_name}\","
+
+		local codename
+		local num_codenames=${#CODENAMES[@]}
+		for codename in "${CODENAMES[@]}"; do
+		    local binary_package_version software_version source_package version version_upstream
+		    binary_package_version=$(package_version "${binary_package_name}" "${codename}" "binary" )
+		    software_version=$(software_version "${binary_package_version}")
+		    source_package=$(source_package "${source_package_template}" "${software_version}")
+		    version=$(package_version "${source_package}" "${codename}" "source")
+		    version_upstream=$(upstream_version "${version}")
+		    echo "	\"${codename}\": {
+		\"source_package\":   \"${source_package}\",
+		\"software_version\": \"${software_version}\",
+		\"version\":          \"${version}\",
+		\"version_upstream\": \"${version_upstream}\""
+		    if [ "${num_codenames}" = "1" ]; then
+			echo "		}"
+		    else
+			echo "		},"
+			num_codenames=$(( num_codenames - 1 ))
+		    fi
+		done
+		if [ "${num_software}" -eq 1 ]; then
+		    echo "    }"
+		else
+		    echo "    },"
+		    num_software=$(( num_software - 1 ))
+		fi
+	    done
+	    echo "]"
+	    ;;
+	"*")
+	    error "Unknown report format '${opt_report_format}' (expected 'text')"
+	    ;;
+    esac
+}
+
+# If we're being run directly (i.e. not sourced), then
+# print out the current software versions.
+if [[ "${0}" == "${BASH_SOURCE[0]}" ]]; then
+    main "${@}"
+fi