Merge ~virtustom/ubuntu-docker-images/+git/utils:azurecr into ~ubuntu-docker-images/ubuntu-docker-images/+git/utils:main

Proposed by Tomáš Virtus
Status: Merged
Merged at revision: db92f1c142d90037004455ff12532a50d9916846
Proposed branch: ~virtustom/ubuntu-docker-images/+git/utils:azurecr
Merge into: ~ubuntu-docker-images/ubuntu-docker-images/+git/utils:main
Diff against target: 489 lines (+170/-75)
11 files modified
Dockerfile (+5/-0)
README.multi-arch-tagger.md (+13/-0)
helpers/registry-login.sh (+73/-2)
helpers/validate-args.sh (+19/-26)
lib/image.sh (+25/-14)
lib/tag.sh (+12/-18)
list-all-images.sh (+3/-3)
list-manifest-for-image-and-tag.sh (+7/-3)
list-tags-for-image.sh (+3/-3)
multi-arch-tagger.sh (+7/-3)
tag-images.sh (+3/-3)
Reviewer Review Type Date Requested Status
Sergio Durigan Junior Approve
Review via email: mp+416326@code.launchpad.net

Commit message

Add support for Azure

Description of the change

Tested on personal registry vubuntu.azurecr.io with image ubuntu tagged with focal-20.04_edge.

Tested running all ./list-* commands on said Azure CR (ACR) registry and my personal Docker Hub registry. Also tested ./tag-images.sh script on ACR registry which created following tags:

20.04
20.04_beta
20.04_candidate
20.04_edge
20.04_stable
focal
focal-20.04_edge

To post a comment you must log in.
Revision history for this message
Sergio Durigan Junior (sergiodj) wrote :

Thanks for the MP, Tomáš.

Overall this LGTM, but I'm leaving a comment about the changes you've made to the functions using trap. Let me know what you think.

I confess I haven't tested these changes, so I'm trusting your report saying that tests went fine.

Regarding the push-images.sh script, that's just a helper I wrote when we were creating the initial images. I haven't maintained it properly and it could even be deleted from the repo TBH.

Anyway, I will wait for your reply before giving my +1. Thanks.

review: Needs Information
Revision history for this message
Tomáš Virtus (virtustom) :
Revision history for this message
Tomáš Virtus (virtustom) :
Revision history for this message
Tomáš Virtus (virtustom) wrote (last edit ):

Sergio, I've addressed your comments. For the Azure CR authentication dance removal, I've created https://warthogs.atlassian.net/browse/ROCKS-65 to replace registry interaction with Skopeo.

EDIT: Tested as in description.

Revision history for this message
Sergio Durigan Junior (sergiodj) :
Revision history for this message
Sergio Durigan Junior (sergiodj) :
Revision history for this message
Sergio Durigan Junior (sergiodj) :
Revision history for this message
Tomáš Virtus (virtustom) wrote (last edit ):

Sergio, I updated the commit. If you have no other comments, can I merge?

Tested tagging and running scripts on on Azure CR and Docker Hub.

Revision history for this message
Sergio Durigan Junior (sergiodj) wrote :

Thanks.

LGTM, +1.

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/Dockerfile b/Dockerfile
2index a28196d..aea8a42 100644
3--- a/Dockerfile
4+++ b/Dockerfile
5@@ -24,6 +24,11 @@ RUN ./aws/install
6 RUN rm -rf ./aws awscliv2.zip
7 COPY . /src
8 RUN mkdir -p /root/.aws
9+RUN . /etc/os-release && curl -O https://packages.microsoft.com/config/ubuntu/$VERSION_ID/packages-microsoft-prod.deb
10+RUN apt-get install -y ./packages-microsoft-prod.deb
11+RUN apt-get update
12+RUN apt-get install -y azure-cli
13+RUN rm -f packages-microsoft-prod.deb
14 RUN DEBIAN_FRONTEND=noninteractive apt-get remove --purge --auto-remove -y
15 RUN rm -rf /var/lib/apt/lists/*
16 FROM scratch
17diff --git a/README.multi-arch-tagger.md b/README.multi-arch-tagger.md
18index 40a4d5d..6553d6c 100644
19--- a/README.multi-arch-tagger.md
20+++ b/README.multi-arch-tagger.md
21@@ -35,6 +35,7 @@ order to run the scripts:
22 - jq
23 - curl
24 - awscli
25+- azure-cli
26
27 You don't need `docker` installed, since we use API endpoints when
28 communicating with Dockerhub. AWS doesn't support most of the API
29@@ -67,3 +68,15 @@ profile name or configure two profiles (named like the used namespace):
30
31 Each profile needs to be properly configured using the corresponding
32 access keys, of course.
33+
34+Configuring azure-cli for tag-images.sh
35+---------------------------------------
36+
37+For scripts to work with Azure you need to have `az` tool from azure-cli in
38+path. To install the azure-cli Debian package follow the [official
39+documentation]( https://docs.microsoft.com/en-us/cli/azure/install-azure-cli).
40+
41+To login into Azure, you can either export `AZURE_CLIENT_ID`,
42+`AZURE_CLIENT_SECRET` and `AZURE_TENAND_ID`, or login via `az login` command
43+beforehand. For more details see [official documentation](
44+https://docs.microsoft.com/en-us/cli/azure/install-azure-cli).
45diff --git a/helpers/registry-login.sh b/helpers/registry-login.sh
46index 5e23314..382c966 100644
47--- a/helpers/registry-login.sh
48+++ b/helpers/registry-login.sh
49@@ -76,7 +76,7 @@ _login_docker_registry1 ()
50 exit 1
51 fi
52
53- export REGISTRY_URL="$DOCKERHUB_REGISTRY_URL"
54+ export REGISTRY_NAMESPACE_URL="$DOCKERHUB_REGISTRY_URL/v2/$NAMESPACE"
55 }
56
57 _login_aws ()
58@@ -116,7 +116,78 @@ _login_aws_registry1 ()
59 exit 1
60 fi
61
62- REGISTRY_URL="$AWS_REGISTRY_URL"
63+ REGISTRY_NAMESPACE_URL="$AWS_REGISTRY_URL/v2/$NAMESPACE"
64+}
65+
66+_login_azure ()
67+{
68+ if ! command -v az >& /dev/null; then
69+ error "You must have the az program from azure-cli installed."
70+ exit 1
71+ fi
72+
73+ if [[ -v AZURE_CLIENT_ID && -v AZURE_CLIENT_SECRET && -v AZURE_TENANT_ID ]]; then
74+ debug "Logging to Azure as service principal"
75+ az login -o none --service-principal --tenant "$AZURE_TENANT_ID" \
76+ -u "$AZURE_CLIENT_ID" -p "$AZURE_CLIENT_SECRET"
77+ fi
78+}
79+
80+_login_azure_registry1 ()
81+{
82+ local image=$1 unset_pipefail=0 ret=0
83+
84+ if [[ ! -o pipefail ]]; then
85+ set -o pipefail
86+ unset_pipefail=1
87+ fi
88+
89+ REGISTRY_NAMESPACE_URL="https://$NAMESPACE.azurecr.io/v2"
90+
91+ # Why are we doing whole authentication dance manually instead of this
92+ # seemingly obvious choice?
93+ #
94+ # REGISTRY_AUTH_TOKEN=$(az acr login -n "$NAMESPACE" --expose-token \
95+ # -o tsv --query accessToken --only-show-errors)
96+ #
97+ # With this token the API endpoints for manifests and tags reply with 401
98+ # response. Apparently "az acr login" only gives "refresh token" and one has
99+ # to obtain "access token". Docker and Skopeo are smart enough to use the
100+ # refresh token to get access token on 401 but here it would require
101+ # checking for 401 on multiple places. The login command also calls "docker
102+ # login" (not sure if even with --expose-token) which is not what we care
103+ # about. Just get the access token directly. The flow was reverse-engineered
104+ # with Wireshark and SSLKEYLOGFILE by OAuth-ignorant person (@virtustom).
105+
106+ REGISTRY_AUTH_TOKEN=$(
107+ az account get-access-token \
108+ | jq -r --arg namespace $NAMESPACE \
109+ '[ "grant_type=access_token",
110+ "service=\($ARGS.named.namespace).azurecr.io",
111+ "tenant=\(.tenant)",
112+ "access_token=\(.accessToken)"
113+ ] | join("&")' \
114+ | curl -fsS -d @- https://$NAMESPACE.azurecr.io/oauth2/exchange \
115+ | jq -r --arg namespace $NAMESPACE --arg image $image \
116+ '[ "grant_type=refresh_token",
117+ "service=\($ARGS.named.namespace).azurecr.io",
118+ "scope=\("repository:\($ARGS.named.image):*" | @uri)",
119+ "refresh_token=\(.refresh_token)"
120+ ] | join("&")' \
121+ | curl -fsS -d @- https://$NAMESPACE.azurecr.io/oauth2/token \
122+ | jq -r .access_token
123+ )
124+ ret=$?
125+
126+ if [[ $unset_pipefail -ne 0 ]]; then
127+ set +o pipefail
128+ fi
129+
130+ if [[ $ret -ne 0 ]]; then
131+ error "There was an error while logging into the registry."
132+ fi
133+
134+ return $ret
135 }
136
137 do_login ()
138diff --git a/helpers/validate-args.sh b/helpers/validate-args.sh
139index 84ccd6b..954c889 100644
140--- a/helpers/validate-args.sh
141+++ b/helpers/validate-args.sh
142@@ -10,6 +10,12 @@ export progname progdir
143 # shellcheck disable=SC1090
144 . "${progdir}/lib/image.sh"
145
146+readonly -A namespace_registry_regexes=(
147+ [ubuntu]='docker|aws'
148+ [lts]='aws'
149+ [canonical]='azure'
150+)
151+
152 validate_var ()
153 {
154 local varname="$1"
155@@ -28,40 +34,27 @@ validate_var ()
156 validate_registry ()
157 {
158 local registry="$1"
159+ local registry_regex
160
161- case "$registry" in
162- "docker"|"aws")
163- :
164- ;;
165- *)
166- error "Invalid registry: '$registry'."
167- return 1
168- ;;
169- esac
170+ for registry_regex in "${namespace_registry_regexes[@]}"; do
171+ if egrep -xq "$registry_regex" <<<$REGISTRY; then
172+ return 0
173+ fi
174+ done
175
176- return 0
177+ error "Invalid registry: '$registry'."
178+ return 1
179 }
180
181 validate_namespace ()
182 {
183 local namespace="$1"
184+ local registry_regex=${namespace_registry_regexes[$namespace]}
185
186- case "$namespace" in
187- "ubuntu")
188- :
189- ;;
190- "lts")
191- # shellcheck disable=SC2153
192- if [ "$REGISTRY" = "docker" ]; then
193- error "The lts namespace is not available on dockerhub yet."
194- return 1
195- fi
196- ;;
197- *)
198- error "Invalid namespace: '$namespace:'."
199- return 1
200- ;;
201- esac
202+ if egrep -vxq "$registry_regex" <<<$REGISTRY; then
203+ error "The $namespace namespace is not available on $REGISTRY yet."
204+ return 1
205+ fi
206
207 return 0
208 }
209diff --git a/lib/image.sh b/lib/image.sh
210index ed2664b..eaca0d9 100644
211--- a/lib/image.sh
212+++ b/lib/image.sh
213@@ -24,6 +24,13 @@ _print_list_of_images_aws ()
214 | jq -r '.repositories[].repositoryName'
215 }
216
217+_print_list_of_images_azure ()
218+{
219+ [ -n "${NAMESPACE}" ] || return 1
220+
221+ az acr repository list --name "$NAMESPACE" -o tsv
222+}
223+
224 print_list_of_images ()
225 {
226 [ -n "${REGISTRY}" ] || return 1
227@@ -62,6 +69,16 @@ _print_tags_for_image_aws ()
228 done
229 }
230
231+_print_tags_for_image_azure ()
232+{
233+ local image="$1"
234+
235+ [ -n "${NAMESPACE}" ] || return 1
236+ [ -n "${image}" ] || return 1
237+
238+ az acr repository show-tags --name "$NAMESPACE" --repository "$image" -o tsv
239+}
240+
241 print_tags_for_image ()
242 {
243 local image="$1"
244@@ -78,26 +95,20 @@ print_manifest_for_image ()
245 {
246 local image="$1"
247 local tag="$2"
248+ local ret=0
249
250 [ -n "${REGISTRY_AUTH_TOKEN}" ] || return 1
251- [ -n "${REGISTRY_URL}" ] || return 1
252- [ -n "${NAMESPACE}" ] || return 1
253+ [ -n "${REGISTRY_NAMESPACE_URL}" ] || return 1
254 [ -n "${image}" ] || return 1
255 [ -n "${tag}" ] || return 1
256
257- local outfile
258- outfile=$(mktemp)
259-
260- trap 'rm -f ${outfile}' 0 INT QUIT ABRT PIPE TERM
261-
262- ret=$(curl -s -H "Authorization: Bearer ${REGISTRY_AUTH_TOKEN}" \
263- -H "Accept: application/vnd.docker.distribution.manifest.list.v2+json" \
264- "${REGISTRY_URL}/v2/${NAMESPACE}/${image}/manifests/${tag}" \
265- -s -o "${outfile}" -w "%{http_code}" || true)
266+ curl -fsS -H "Authorization: Bearer ${REGISTRY_AUTH_TOKEN}" \
267+ -H "Accept: application/vnd.docker.distribution.manifest.list.v2+json" \
268+ "${REGISTRY_NAMESPACE_URL}/${image}/manifests/${tag}" || ret=$?
269
270- if [ "$ret" -ne 200 ]; then
271- error "Unable to obtain manifest file for image (return code $ret)"
272+ if [[ $ret -ne 0 ]]; then
273+ error "Unable to obtain manifest file for image"
274 fi
275
276- jq . "${outfile}"
277+ return $ret
278 }
279diff --git a/lib/tag.sh b/lib/tag.sh
280index 5ae0c98..81eb153 100644
281--- a/lib/tag.sh
282+++ b/lib/tag.sh
283@@ -14,6 +14,7 @@ tag_image ()
284 local image="$1"
285 local source_tag="$2"
286 local tag="$3"
287+ local ret=0
288
289 [ -n "${REGISTRY}" ] || return 1
290 [ -n "${NAMESPACE}" ] || return 1
291@@ -21,42 +22,35 @@ tag_image ()
292 [ -n "${source_tag}" ] || return 1
293 [ -n "${tag}" ] || return 1
294 [ -n "${REGISTRY_AUTH_TOKEN}" ] || return 1
295- [ -n "${REGISTRY_URL}" ] || return 1
296+ [ -n "${REGISTRY_NAMESPACE_URL}" ] || return 1
297
298 info "Tagging ${NAMESPACE}/${image}:${source_tag} as ${NAMESPACE}/${image}:${tag} (on ${REGISTRY})"
299
300 local tmpfile
301- local ret
302-
303 tmpfile=$(mktemp)
304- trap 'rm -f ${tmpfile}' 0 INT QUIT ABRT PIPE TERM
305
306 if ! print_manifest_for_image "${image}" "${source_tag}" > "${tmpfile}"; then
307 error "Unable to print manifest for ${image}:${source_tag}"
308+ rm "$tmpfile"
309+ return 1
310 fi
311
312 if [ "$DRY_RUN" -eq 1 ]; then
313 info "dry-run: not tagging the image"
314+ rm "$tmpfile"
315 return
316 fi
317
318- curl_err=$(mktemp)
319- trap 'rm -f ${curl_err}' 0 INT QUIT ABRT PIPE TERM
320-
321- # We use "|| true" here because we want the command to complete
322- # even if it fails. We check for the HTTP response code below.
323- ret=$(curl -X PUT -H "Authorization: Bearer ${REGISTRY_AUTH_TOKEN}" \
324- -H "Content-Type: application/vnd.docker.distribution.manifest.list.v2+json" \
325- "${REGISTRY_URL}/v2/${NAMESPACE}/${image}/manifests/${tag}" \
326- -d "@${tmpfile}" \
327- -s -o "${curl_err}" -w "%{http_code}" || true)
328+ curl -fsS -X PUT -H "Authorization: Bearer ${REGISTRY_AUTH_TOKEN}" \
329+ -H "Content-Type: application/vnd.docker.distribution.manifest.list.v2+json" \
330+ "${REGISTRY_NAMESPACE_URL}/${image}/manifests/${tag}" \
331+ -d "@${tmpfile}" || ret=$?
332
333 rm "$tmpfile"
334
335- if [ "$ret" -ne 201 ]; then
336- error "Image tagging failed with status code $ret"
337- cat "${curl_err}" > /dev/stderr
338+ if [[ $ret -ne 0 ]]; then
339+ error "Image tagging failed"
340 fi
341
342- rm "${curl_err}"
343+ return $ret
344 }
345diff --git a/list-all-images.sh b/list-all-images.sh
346index 63493a4..2e6df71 100755
347--- a/list-all-images.sh
348+++ b/list-all-images.sh
349@@ -24,7 +24,7 @@ $progname -- Obtain the list of images from a certain repository.
350
351 Usage:
352
353- $progname [-a <TOKEN>] -r <docker|aws> -n <ubuntu|lts> [-d]
354+ $progname [-a <TOKEN>] -r <docker|aws|azure> -n <ubuntu|lts|canonical> [-d]
355
356 Arguments:
357
358@@ -34,10 +34,10 @@ Arguments:
359 script will ask for your user/password.
360
361 -r REGISTRY|--registry REGISTRY Registry to consult.
362- Possible values: docker, aws
363+ Possible values: docker, aws, azure
364
365 -n NAMESPACE|--namespace NAMESPACE: Namespace to consult.
366- Possible values: ubuntu, lts
367+ Possible values: ubuntu, lts, canonical
368
369 -d|--debug Print debug statements. This includes printing
370 the authentication token if using Dockerhub.
371diff --git a/list-manifest-for-image-and-tag.sh b/list-manifest-for-image-and-tag.sh
372index dbcf08d..57dd832 100755
373--- a/list-manifest-for-image-and-tag.sh
374+++ b/list-manifest-for-image-and-tag.sh
375@@ -30,12 +30,16 @@ same username/password you use for docker.
376
377 Usage (when using Dockerhub):
378
379- $progname [-a <TOKEN>] -r docker -n <ubuntu|lts> -i <IMAGE> -t <TAG> [-u <USERNAME>] [-p <PASSWORD>] [-d]
380+ $progname [-a <TOKEN>] -r docker -n ubuntu -i <IMAGE> -t <TAG> [-u <USERNAME>] [-p <PASSWORD>] [-d]
381
382 Usage (when using AWS):
383
384 $progname [-a <TOKEN>] -r aws -n <ubuntu|lts> -i <IMAGE> -t <TAG> [-d]
385
386+Usage (when using Azure):
387+
388+ $progname [-a <TOKEN>] -r azure -n canonical -i <IMAGE> -t <TAG> [-d]
389+
390 Arguments:
391
392 -a TOKEN|--auth-token TOKEN Specify the authorization token that
393@@ -49,10 +53,10 @@ Arguments:
394 below.
395
396 -r REGISTRY|--registry REGISTRY Registry to consult.
397- Possible values: docker, aws
398+ Possible values: docker, aws, azure
399
400 -n NAMESPACE|--namespace NAMESPACE Namespace to consult.
401- Possible values: ubuntu, lts
402+ Possible values: ubuntu, lts, canonical
403
404 -i IMAGE|--image IMAGE Name of an image from the repository (without
405 namespace), for example 'postgres' or 'redis'.
406diff --git a/list-tags-for-image.sh b/list-tags-for-image.sh
407index 67380f8..efefd6c 100755
408--- a/list-tags-for-image.sh
409+++ b/list-tags-for-image.sh
410@@ -24,7 +24,7 @@ $progname -- Obtain the list of tags from a certain image.
411
412 Usage:
413
414- $progname [-a <TOKEN>] -r <docker|aws> -n <ubuntu|lts> -i <IMAGE> [-d]
415+ $progname [-a <TOKEN>] -r <docker|aws|azure> -n <ubuntu|lts|canonical> -i <IMAGE> [-d]
416
417 Arguments:
418
419@@ -34,10 +34,10 @@ Arguments:
420 script will ask for your user/password.
421
422 -r REGISTRY|--registry REGISTRY Registry to consult.
423- Possible values: docker, aws
424+ Possible values: docker, aws, azure
425
426 -n NAMESPACE|--namespace NAMESPACE Namespace to consult.
427- Possible values: ubuntu, lts
428+ Possible values: ubuntu, lts, canonical
429
430 -i IMAGE|--image IMAGE Image to obtain tags from.
431
432diff --git a/multi-arch-tagger.sh b/multi-arch-tagger.sh
433index c51bb4d..2e3dbb4 100755
434--- a/multi-arch-tagger.sh
435+++ b/multi-arch-tagger.sh
436@@ -30,12 +30,16 @@ same username/password you use for docker.
437
438 Usage (when using Dockerhub):
439
440- $progname [-a <TOKEN>] -r docker -n <ubuntu|lts> -i <IMAGE> -s <SOURCE_TAG> -t <NEW_TAG> [-u <USERNAME>] [-p <PASSWORD>] [-d]
441+ $progname [-a <TOKEN>] -r docker -n ubuntu -i <IMAGE> -s <SOURCE_TAG> -t <NEW_TAG> [-u <USERNAME>] [-p <PASSWORD>] [-d]
442
443 Usage (when using AWS):
444
445 $progname [-a <TOKEN>] -r aws -n <ubuntu|lts> -i <IMAGE> -s <SOURCE_TAG> -t <NEW_TAG> [-d]
446
447+Usage (when using Azure):
448+
449+ $progname [-a <TOKEN>] -r azure -n canonical -i <IMAGE> -s <SOURCE_TAG> -t <NEW_TAG> [-d]
450+
451 Arguments:
452
453 -a TOKEN|--auth-token TOKEN Specify the authorization token that
454@@ -49,10 +53,10 @@ Arguments:
455 below.
456
457 -r REGISTRY|--registry REGISTRY Registry to consult.
458- Possible values: docker, aws
459+ Possible values: docker, aws, azure
460
461 -n NAMESPACE|--namespace NAMESPACE Namespace to consult.
462- Possible values: ubuntu, lts
463+ Possible values: ubuntu, lts, canonical
464
465 -i IMAGE|--image IMAGE Name of an image from the repository (without
466 namespace), for example 'postgres' or 'redis'.
467diff --git a/tag-images.sh b/tag-images.sh
468index 92ccaf3..dfce404 100755
469--- a/tag-images.sh
470+++ b/tag-images.sh
471@@ -69,15 +69,15 @@ all architectures using a single machine.
472
473 Usage:
474
475- $progname [-a <TOKEN>] -r <docker|aws> -n <ubuntu|lts> [-d] [-- IMAGE_1 IMAGE_2 ... IMAGE_N]
476+ $progname [-a <TOKEN>] -r <docker|aws|azure> -n <ubuntu|lts|canonical> [-d] [-- IMAGE_1 IMAGE_2 ... IMAGE_N]
477
478 Arguments:
479
480 -r REGISTRY|--registry REGISTRY Registry to consult.
481- Possible values: docker, aws
482+ Possible values: docker, aws, azure
483
484 -n NAMESPACE|--namespace NAMESPACE Namespace to consult.
485- Possible values: ubuntu, lts
486+ Possible values: ubuntu, lts, canonical
487
488 -f|--force Force tagging, even if the script would think
489 that it's not needed. This mostly applies for

Subscribers

People subscribed via source and target branches