Merge ~virtustom/ubuntu-docker-images/+git/utils:azurecr into ~ubuntu-docker-images/ubuntu-docker-images/+git/utils:main

[Tomáš Virtus]

Add support for Azure

Tested on personal registry vubuntu.azurecr.io with image ubuntu tagged with focal-20.04_edge.

Tested running all ./list-* commands on said Azure CR (ACR) registry and my personal Docker Hub registry. Also tested ./tag-images.sh script on ACR registry which created following tags:

20.04
20.04_beta
20.04_candidate
20.04_edge
20.04_stable
focal
focal-20.04_edge

[Sergio Durigan Junior (sergiodj)]

Thanks for the MP, Tomáš.

Overall this LGTM, but I'm leaving a comment about the changes you've made to the functions using trap. Let me know what you think.

I confess I haven't tested these changes, so I'm trusting your report saying that tests went fine.

Regarding the push-images.sh script, that's just a helper I wrote when we were creating the initial images. I haven't maintained it properly and it could even be deleted from the repo TBH.

Anyway, I will wait for your reply before giving my +1. Thanks.

review: Needs Information

[Tomáš Virtus (virtustom)]

Sergio, I've addressed your comments. For the Azure CR authentication dance removal, I've created https://warthogs.atlassian.net/browse/ROCKS-65 to replace registry interaction with Skopeo.

EDIT: Tested as in description.

[Tomáš Virtus (virtustom)]

Sergio, I updated the commit. If you have no other comments, can I merge?

Tested tagging and running scripts on on Azure CR and Docker Hub.

[Sergio Durigan Junior (sergiodj)]

Thanks.

LGTM, +1.

review: Approve

diff --git a/Dockerfile b/Dockerfile
index a28196d..aea8a42 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -24,6 +24,11 @@ RUN ./aws/install
 RUN rm -rf ./aws awscliv2.zip
 COPY . /src
 RUN mkdir -p /root/.aws
+RUN . /etc/os-release && curl -O https://packages.microsoft.com/config/ubuntu/$VERSION_ID/packages-microsoft-prod.deb
+RUN apt-get install -y ./packages-microsoft-prod.deb
+RUN apt-get update
+RUN apt-get install -y azure-cli
+RUN rm -f packages-microsoft-prod.deb
 RUN DEBIAN_FRONTEND=noninteractive apt-get remove --purge --auto-remove -y
 RUN rm -rf /var/lib/apt/lists/*
 FROM scratch
diff --git a/README.multi-arch-tagger.md b/README.multi-arch-tagger.md
index 40a4d5d..6553d6c 100644
--- a/README.multi-arch-tagger.md
+++ b/README.multi-arch-tagger.md
@@ -35,6 +35,7 @@ order to run the scripts:
 - jq
 - curl
 - awscli
+- azure-cli
 
 You don't need `docker` installed, since we use API endpoints when
 communicating with Dockerhub.  AWS doesn't support most of the API
@@ -67,3 +68,15 @@ profile name or configure two profiles (named like the used namespace):
 
 Each profile needs to be properly configured using the corresponding
 access keys, of course.
+
+Configuring azure-cli for tag-images.sh
+---------------------------------------
+
+For scripts to work with Azure you need to have `az` tool from azure-cli in
+path. To install the azure-cli Debian package follow the [official
+documentation]( https://docs.microsoft.com/en-us/cli/azure/install-azure-cli).
+
+To login into Azure, you can either export `AZURE_CLIENT_ID`,
+`AZURE_CLIENT_SECRET` and `AZURE_TENAND_ID`, or login via `az login` command
+beforehand. For more details see [official documentation](
+https://docs.microsoft.com/en-us/cli/azure/install-azure-cli).
diff --git a/helpers/registry-login.sh b/helpers/registry-login.sh
index 5e23314..382c966 100644
--- a/helpers/registry-login.sh
+++ b/helpers/registry-login.sh
@@ -76,7 +76,7 @@ _login_docker_registry1 ()
 	exit 1
     fi
 
-    export REGISTRY_URL="$DOCKERHUB_REGISTRY_URL"
+    export REGISTRY_NAMESPACE_URL="$DOCKERHUB_REGISTRY_URL/v2/$NAMESPACE"
 }
 
 _login_aws ()
@@ -116,7 +116,78 @@ _login_aws_registry1 ()
 	exit 1
     fi
 
-    REGISTRY_URL="$AWS_REGISTRY_URL"
+    REGISTRY_NAMESPACE_URL="$AWS_REGISTRY_URL/v2/$NAMESPACE"
+}
+
+_login_azure ()
+{
+    if ! command -v az >& /dev/null; then
+	error "You must have the az program from azure-cli installed."
+	exit 1
+    fi
+
+    if [[ -v AZURE_CLIENT_ID && -v AZURE_CLIENT_SECRET && -v AZURE_TENANT_ID ]]; then
+	debug "Logging to Azure as service principal"
+	az login -o none --service-principal --tenant "$AZURE_TENANT_ID" \
+	    -u "$AZURE_CLIENT_ID" -p "$AZURE_CLIENT_SECRET"
+    fi
+}
+
+_login_azure_registry1 ()
+{
+    local image=$1 unset_pipefail=0 ret=0
+
+    if [[ ! -o pipefail ]]; then
+	set -o pipefail
+	unset_pipefail=1
+    fi
+
+    REGISTRY_NAMESPACE_URL="https://$NAMESPACE.azurecr.io/v2"
+
+    # Why are we doing whole authentication dance manually instead of this
+    # seemingly obvious choice?
+    #
+    #    REGISTRY_AUTH_TOKEN=$(az acr login -n "$NAMESPACE" --expose-token \
+    #	     -o tsv --query accessToken --only-show-errors)
+    #
+    # With this token the API endpoints for manifests and tags reply with 401
+    # response. Apparently "az acr login" only gives "refresh token" and one has
+    # to obtain "access token". Docker and Skopeo are smart enough to use the
+    # refresh token to get access token on 401 but here it would require
+    # checking for 401 on multiple places. The login command also calls "docker
+    # login" (not sure if even with --expose-token) which is not what we care
+    # about. Just get the access token directly. The flow was reverse-engineered
+    # with Wireshark and SSLKEYLOGFILE by OAuth-ignorant person (@virtustom).
+
+    REGISTRY_AUTH_TOKEN=$(
+	az account get-access-token \
+	| jq -r --arg namespace $NAMESPACE \
+	    '[  "grant_type=access_token",
+		"service=\($ARGS.named.namespace).azurecr.io",
+		"tenant=\(.tenant)",
+		"access_token=\(.accessToken)"
+	    ] | join("&")' \
+	| curl -fsS -d @- https://$NAMESPACE.azurecr.io/oauth2/exchange \
+	| jq -r --arg namespace $NAMESPACE --arg image $image \
+	    '[  "grant_type=refresh_token",
+		"service=\($ARGS.named.namespace).azurecr.io",
+		"scope=\("repository:\($ARGS.named.image):*" | @uri)",
+		"refresh_token=\(.refresh_token)"
+	    ] | join("&")' \
+	| curl -fsS -d @- https://$NAMESPACE.azurecr.io/oauth2/token \
+	| jq -r .access_token
+    )
+    ret=$?
+
+    if [[ $unset_pipefail -ne 0 ]]; then
+	set +o pipefail
+    fi
+
+    if [[ $ret -ne 0 ]]; then
+	error "There was an error while logging into the registry."
+    fi
+
+    return $ret
 }
 
 do_login ()
diff --git a/helpers/validate-args.sh b/helpers/validate-args.sh
index 84ccd6b..954c889 100644
--- a/helpers/validate-args.sh
+++ b/helpers/validate-args.sh
@@ -10,6 +10,12 @@ export progname progdir
 # shellcheck disable=SC1090
 . "${progdir}/lib/image.sh"
 
+readonly -A namespace_registry_regexes=(
+    [ubuntu]='docker|aws'
+    [lts]='aws'
+    [canonical]='azure'
+)
+
 validate_var ()
 {
     local varname="$1"
@@ -28,40 +34,27 @@ validate_var ()
 validate_registry ()
 {
     local registry="$1"
+    local registry_regex
 
-    case "$registry" in
-	"docker"|"aws")
-	    :
-	    ;;
-	*)
-	    error "Invalid registry: '$registry'."
-	    return 1
-	    ;;
-    esac
+    for registry_regex in "${namespace_registry_regexes[@]}"; do
+	if egrep -xq "$registry_regex" <<<$REGISTRY; then
+	    return 0
+	fi
+    done
 
-    return 0
+    error "Invalid registry: '$registry'."
+    return 1
 }
 
 validate_namespace ()
 {
     local namespace="$1"
+    local registry_regex=${namespace_registry_regexes[$namespace]}
 
-    case "$namespace" in
-	"ubuntu")
-	    :
-	    ;;
-	"lts")
-	    # shellcheck disable=SC2153
-	    if [ "$REGISTRY" = "docker" ]; then
-		error "The lts namespace is not available on dockerhub yet."
-		return 1
-	    fi
-	    ;;
-	*)
-	    error "Invalid namespace: '$namespace:'."
-	    return 1
-	    ;;
-    esac
+    if egrep -vxq "$registry_regex" <<<$REGISTRY; then
+	error "The $namespace namespace is not available on $REGISTRY yet."
+	return 1
+    fi
 
     return 0
 }
diff --git a/lib/image.sh b/lib/image.sh
index ed2664b..eaca0d9 100644
--- a/lib/image.sh
+++ b/lib/image.sh
@@ -24,6 +24,13 @@ _print_list_of_images_aws ()
 	| jq -r '.repositories[].repositoryName'
 }
 
+_print_list_of_images_azure ()
+{
+    [ -n "${NAMESPACE}" ] || return 1
+
+    az acr repository list --name "$NAMESPACE" -o tsv
+}
+
 print_list_of_images ()
 {
     [ -n "${REGISTRY}" ] || return 1
@@ -62,6 +69,16 @@ _print_tags_for_image_aws ()
     done
 }
 
+_print_tags_for_image_azure ()
+{
+    local image="$1"
+
+    [ -n "${NAMESPACE}" ] || return 1
+    [ -n "${image}" ] || return 1
+
+    az acr repository show-tags --name "$NAMESPACE" --repository "$image" -o tsv
+}
+
 print_tags_for_image ()
 {
     local image="$1"
@@ -78,26 +95,20 @@ print_manifest_for_image ()
 {
     local image="$1"
     local tag="$2"
+    local ret=0
 
     [ -n "${REGISTRY_AUTH_TOKEN}" ] || return 1
-    [ -n "${REGISTRY_URL}" ] || return 1
-    [ -n "${NAMESPACE}" ] || return 1
+    [ -n "${REGISTRY_NAMESPACE_URL}" ] || return 1
     [ -n "${image}" ] || return 1
     [ -n "${tag}" ] || return 1
 
-    local outfile
-    outfile=$(mktemp)
-
-    trap 'rm -f ${outfile}' 0 INT QUIT ABRT PIPE TERM
-
-    ret=$(curl -s -H "Authorization: Bearer ${REGISTRY_AUTH_TOKEN}" \
-	       -H "Accept: application/vnd.docker.distribution.manifest.list.v2+json" \
-	       "${REGISTRY_URL}/v2/${NAMESPACE}/${image}/manifests/${tag}" \
-	       -s -o "${outfile}" -w "%{http_code}" || true)
+    curl -fsS -H "Authorization: Bearer ${REGISTRY_AUTH_TOKEN}" \
+	-H "Accept: application/vnd.docker.distribution.manifest.list.v2+json" \
+	"${REGISTRY_NAMESPACE_URL}/${image}/manifests/${tag}" || ret=$?
 
-    if [ "$ret" -ne 200 ]; then
-	error "Unable to obtain manifest file for image (return code $ret)"
+    if [[ $ret -ne 0 ]]; then
+	error "Unable to obtain manifest file for image"
     fi
 
-    jq . "${outfile}"
+    return $ret
 }
diff --git a/lib/tag.sh b/lib/tag.sh
index 5ae0c98..81eb153 100644
--- a/lib/tag.sh
+++ b/lib/tag.sh
@@ -14,6 +14,7 @@ tag_image ()
     local image="$1"
     local source_tag="$2"
     local tag="$3"
+    local ret=0
 
     [ -n "${REGISTRY}" ] || return 1
     [ -n "${NAMESPACE}" ] || return 1
@@ -21,42 +22,35 @@ tag_image ()
     [ -n "${source_tag}" ] || return 1
     [ -n "${tag}" ] || return 1
     [ -n "${REGISTRY_AUTH_TOKEN}" ] || return 1
-    [ -n "${REGISTRY_URL}" ] || return 1
+    [ -n "${REGISTRY_NAMESPACE_URL}" ] || return 1
 
     info "Tagging ${NAMESPACE}/${image}:${source_tag} as ${NAMESPACE}/${image}:${tag} (on ${REGISTRY})"
 
     local tmpfile
-    local ret
-
     tmpfile=$(mktemp)
-    trap 'rm -f ${tmpfile}' 0 INT QUIT ABRT PIPE TERM
 
     if ! print_manifest_for_image "${image}" "${source_tag}" > "${tmpfile}"; then
 	error "Unable to print manifest for ${image}:${source_tag}"
+	rm "$tmpfile"
+	return 1
     fi
 
     if [ "$DRY_RUN" -eq 1 ]; then
 	info "dry-run: not tagging the image"
+	rm "$tmpfile"
 	return
     fi
 
-    curl_err=$(mktemp)
-    trap 'rm -f ${curl_err}' 0 INT QUIT ABRT PIPE TERM
-
-    # We use "|| true" here because we want the command to complete
-    # even if it fails.  We check for the HTTP response code below.
-    ret=$(curl -X PUT -H "Authorization: Bearer ${REGISTRY_AUTH_TOKEN}" \
-	       -H "Content-Type: application/vnd.docker.distribution.manifest.list.v2+json" \
-	       "${REGISTRY_URL}/v2/${NAMESPACE}/${image}/manifests/${tag}" \
-	       -d "@${tmpfile}" \
-	       -s -o "${curl_err}" -w "%{http_code}" || true)
+    curl -fsS -X PUT -H "Authorization: Bearer ${REGISTRY_AUTH_TOKEN}" \
+	-H "Content-Type: application/vnd.docker.distribution.manifest.list.v2+json" \
+	"${REGISTRY_NAMESPACE_URL}/${image}/manifests/${tag}" \
+	-d "@${tmpfile}" || ret=$?
 
     rm "$tmpfile"
 
-    if [ "$ret" -ne 201 ]; then
-	error "Image tagging failed with status code $ret"
-	cat "${curl_err}" > /dev/stderr
+    if [[ $ret -ne 0 ]]; then
+	error "Image tagging failed"
     fi
 
-    rm "${curl_err}"
+    return $ret
 }
diff --git a/list-all-images.sh b/list-all-images.sh
index 63493a4..2e6df71 100755
--- a/list-all-images.sh
+++ b/list-all-images.sh
@@ -24,7 +24,7 @@ $progname -- Obtain the list of images from a certain repository.
 
 Usage:
 
-  $progname [-a <TOKEN>] -r <docker|aws> -n <ubuntu|lts> [-d]
+  $progname [-a <TOKEN>] -r <docker|aws|azure> -n <ubuntu|lts|canonical> [-d]
 
 Arguments:
 
@@ -34,10 +34,10 @@ Arguments:
 					script will ask for your user/password.
 
   -r REGISTRY|--registry REGISTRY	Registry to consult.
-     			 		Possible values: docker, aws
+     			 		Possible values: docker, aws, azure
 
   -n NAMESPACE|--namespace NAMESPACE:	Namespace to consult.
-     			   		Possible values: ubuntu, lts
+     			   		Possible values: ubuntu, lts, canonical
 
   -d|--debug				Print debug statements.  This includes printing
   					the authentication token if using Dockerhub.
diff --git a/list-manifest-for-image-and-tag.sh b/list-manifest-for-image-and-tag.sh
index dbcf08d..57dd832 100755
--- a/list-manifest-for-image-and-tag.sh
+++ b/list-manifest-for-image-and-tag.sh
@@ -30,12 +30,16 @@ same username/password you use for docker.
 
 Usage (when using Dockerhub):
 
-  $progname [-a <TOKEN>] -r docker -n <ubuntu|lts> -i <IMAGE> -t <TAG> [-u <USERNAME>] [-p <PASSWORD>] [-d]
+  $progname [-a <TOKEN>] -r docker -n ubuntu -i <IMAGE> -t <TAG> [-u <USERNAME>] [-p <PASSWORD>] [-d]
 
 Usage (when using AWS):
 
   $progname [-a <TOKEN>] -r aws -n <ubuntu|lts> -i <IMAGE> -t <TAG> [-d]
 
+Usage (when using Azure):
+
+  $progname [-a <TOKEN>] -r azure -n canonical -i <IMAGE> -t <TAG> [-d]
+
 Arguments:
 
   -a TOKEN|--auth-token TOKEN		Specify the authorization token that
@@ -49,10 +53,10 @@ Arguments:
 					below.
 
   -r REGISTRY|--registry REGISTRY	Registry to consult.
-     			 		Possible values: docker, aws
+     			 		Possible values: docker, aws, azure
 
   -n NAMESPACE|--namespace NAMESPACE	Namespace to consult.
-     			   		Possible values: ubuntu, lts
+     			   		Possible values: ubuntu, lts, canonical
 
   -i IMAGE|--image IMAGE		Name of an image from the repository (without
      		   			namespace), for example 'postgres' or 'redis'.
diff --git a/list-tags-for-image.sh b/list-tags-for-image.sh
index 67380f8..efefd6c 100755
--- a/list-tags-for-image.sh
+++ b/list-tags-for-image.sh
@@ -24,7 +24,7 @@ $progname -- Obtain the list of tags from a certain image.
 
 Usage:
 
-  $progname [-a <TOKEN>] -r <docker|aws> -n <ubuntu|lts> -i <IMAGE> [-d]
+  $progname [-a <TOKEN>] -r <docker|aws|azure> -n <ubuntu|lts|canonical> -i <IMAGE> [-d]
 
 Arguments:
 
@@ -34,10 +34,10 @@ Arguments:
 					script will ask for your user/password.
 
   -r REGISTRY|--registry REGISTRY	Registry to consult.
-     			 		Possible values: docker, aws
+     			 		Possible values: docker, aws, azure
 
   -n NAMESPACE|--namespace NAMESPACE	Namespace to consult.
-     			   		Possible values: ubuntu, lts
+     			   		Possible values: ubuntu, lts, canonical
 
   -i IMAGE|--image IMAGE		Image to obtain tags from.
 
diff --git a/multi-arch-tagger.sh b/multi-arch-tagger.sh
index c51bb4d..2e3dbb4 100755
--- a/multi-arch-tagger.sh
+++ b/multi-arch-tagger.sh
@@ -30,12 +30,16 @@ same username/password you use for docker.
 
 Usage (when using Dockerhub):
 
-  $progname [-a <TOKEN>] -r docker -n <ubuntu|lts> -i <IMAGE> -s <SOURCE_TAG> -t <NEW_TAG> [-u <USERNAME>] [-p <PASSWORD>] [-d]
+  $progname [-a <TOKEN>] -r docker -n ubuntu -i <IMAGE> -s <SOURCE_TAG> -t <NEW_TAG> [-u <USERNAME>] [-p <PASSWORD>] [-d]
 
 Usage (when using AWS):
 
   $progname [-a <TOKEN>] -r aws -n <ubuntu|lts> -i <IMAGE> -s <SOURCE_TAG> -t <NEW_TAG> [-d]
 
+Usage (when using Azure):
+
+  $progname [-a <TOKEN>] -r azure -n canonical -i <IMAGE> -s <SOURCE_TAG> -t <NEW_TAG> [-d]
+
 Arguments:
 
   -a TOKEN|--auth-token TOKEN		Specify the authorization token that
@@ -49,10 +53,10 @@ Arguments:
 					below.
 
   -r REGISTRY|--registry REGISTRY	Registry to consult.
-     			 		Possible values: docker, aws
+     			 		Possible values: docker, aws, azure
 
   -n NAMESPACE|--namespace NAMESPACE	Namespace to consult.
-     			   		Possible values: ubuntu, lts
+     			   		Possible values: ubuntu, lts, canonical
 
   -i IMAGE|--image IMAGE		Name of an image from the repository (without
      		   			namespace), for example 'postgres' or 'redis'.
diff --git a/tag-images.sh b/tag-images.sh
index 92ccaf3..dfce404 100755
--- a/tag-images.sh
+++ b/tag-images.sh
@@ -69,15 +69,15 @@ all architectures using a single machine.
 
 Usage:
 
-  $progname [-a <TOKEN>] -r <docker|aws> -n <ubuntu|lts> [-d] [-- IMAGE_1 IMAGE_2 ... IMAGE_N]
+  $progname [-a <TOKEN>] -r <docker|aws|azure> -n <ubuntu|lts|canonical> [-d] [-- IMAGE_1 IMAGE_2 ... IMAGE_N]
 
 Arguments:
 
   -r REGISTRY|--registry REGISTRY	Registry to consult.
-     			 		Possible values: docker, aws
+     			 		Possible values: docker, aws, azure
 
   -n NAMESPACE|--namespace NAMESPACE	Namespace to consult.
-     			   		Possible values: ubuntu, lts
+     			   		Possible values: ubuntu, lts, canonical
 
   -f|--force				Force tagging, even if the script would think
   					that it's not needed.  This mostly applies for