Merge ~ubuntu-docker-images/ubuntu-docker-images/+git/utils:multi-arch-tagger into ~ubuntu-docker-images/ubuntu-docker-images/+git/utils:master

Proposed by Sergio Durigan Junior
Status: Merged
Merged at revision: 75a4c7e37ead57073f4020eb71df9c053724650f
Proposed branch: ~ubuntu-docker-images/ubuntu-docker-images/+git/utils:multi-arch-tagger
Merge into: ~ubuntu-docker-images/ubuntu-docker-images/+git/utils:master
Diff against target: 1196 lines (+1148/-0)
8 files modified
README.multi-arch-tagger.md (+67/-0)
helpers/log.sh (+49/-0)
helpers/registry-login.sh (+122/-0)
helpers/validate-args.sh (+85/-0)
list-all-images.sh (+129/-0)
list-tags-for-image.sh (+141/-0)
multi-arch-tagger.sh (+220/-0)
tag-images.sh (+335/-0)
Reviewer Review Type Date Requested Status
Paride Legovini Approve
Bryce Harrington Approve
Review via email: mp+399681@code.launchpad.net

Description of the change

Implement a multi-architecture tagger framework.

This MP consists of a series of scripts that work together to implement the multi-architecture tagging, which is one of the "missing pieces" for our OCI work.

The scripts are a bit complex, and they all work independently. The main script here is the "tag-images.sh", which encapsulates all the logic needed to properly create the "latest" and the "M.N-XX.YY_beta" tags (not to mention the special "ubuntu" base image).

I've tested this on some images and it's working OK, but most of my test was done using "echo" statements to make sure that the right commands would be invoked. We're still not ready to mass-tag everything, but I don't foresee any problems.

The aws-cli tool needs to be installed and configured locally in order to communicate with the AWS service. I've written some instructions on how to configure it in the README file.

To post a comment you must log in.
Revision history for this message
Bryce Harrington (bryce) wrote :

Initial pass of mostly just stylistic comments. I also looked for typos or coding errors but didn't find any! So nothing that actually "needs" fixed, but more just suggestions.

I haven't tried actually running the code and studying its execution, but will give that a shot for the next pass, but that'll have be for tomorrow.

I like that you broke out the log.sh to be a sourced file. I had already been thinking of breaking them out so I could use them, so that's cool.

Revision history for this message
Sergio Durigan Junior (sergiodj) wrote :

Thanks for the initial review, Bryce. Comments inline, and new version force-pushed.

Revision history for this message
Bryce Harrington (bryce) wrote :
Download full text (3.7 KiB)

Ok, I've had a chance to play with the script in earnest for a bit (just enough to be usefully confused), and have some usage suggestions.

Before I get too far into it, I'll mention upfront that if this is going to be used entirely non-interactively, then a lot of the remarks in this comment are probably going to be irrelevant. However, if you anticipate anyone might be running this by hand, then they may be worthwhile.

And first, here's how I'm running things:

stirling:~/pkg/Oci/utils$ ./list-all-images.sh -r docker -n ubuntu
Username: bryceharrington
Password:
ubuntu/nginx
ubuntu/redis
ubuntu/postgres
ubuntu/mysql
ubuntu/apache2
ubuntu/memcached
ubuntu/grafana
ubuntu/prometheus
ubuntu/cortex
ubuntu/telegraf
ubuntu/prometheus-alertmanager

$ ./list-tags-for-image.sh -r docker -n ubuntu -i redis
Username: bryceharrington
Password:
ubuntu/redis:latest
ubuntu/redis:6.0-21.04_beta
ubuntu/redis:edge
ubuntu/redis:6.0-21.04_edge
ubuntu/redis:5.0-20.04_beta
ubuntu/redis:5.0-20.04_edge

One thing that's a bit confusing is after running ./list-all-images.sh, I figured I could just cut and paste a line from its input to pass to ./list-tags-for-image.sh, however, that's wrong:

stirling:~/pkg/Oci/utils$ ./list-tags-for-image.sh -r docker -n ubuntu -i ubuntu/redis
Username: bryceharrington
Password:
E: Invalid image name 'ubuntu/redis'.

I'd probably just chalk that up to user error, but wouldn't be surprised if other rookie users make the same mistake. Again, if this is mainly intended to be used in an automated workflow, it's probably better to not have too many smarts in the argument parsing logic. You've got the --no-prefix argument if someone really needs the bare image name.

stirling:~/pkg/Oci/utils$ ./multi-arch-tagger.sh -t foobar -n ubuntu -r docker -b 20.04 -u bryceharrington -p <my-docker-password> -i redis
Username: bryceharrington
Password:
I: Tagging ubuntu/redis:20.04 as ubuntu/redis:foobar (on docker)
ERROR: Image tagging failed with status code 400

So first comment is I'm not entirely certain this is the right way to be calling this tool. Given the large number of required parameters it would help a lot to have an example or two in the help text (or in a man page). For this tool, maybe one cut-and-paste command line to test it, and then one real-world usage.

Second, I'd kind of like to be able to run this against a throwaway image in my own namespace, just to test it, but passing '-n bryceharrington' gives an error. If that's intended/desired then a --dry-run parameter might be handy to add.

Third, I was initially a bit confused why there are --username and --password arguments, yet it still prompts me for username and password. From the help text I got the impression they were two different sets of usernames and passwords, but still a bit confusing what they might be. Also, taking a password on the command line might have some security implications, although I gather this is just for testing convenience and the auth token is used more in practice?

And fourth, the error message with status code 400 is a bit cryptic. That said, I'll bet it's not really going to be possible to give a better error message since that c...

Read more...

review: Approve
Revision history for this message
Sergio Durigan Junior (sergiodj) wrote :
Download full text (7.3 KiB)

On Tuesday, March 16 2021, Bryce Harrington wrote:

> Review: Approve
>
> Ok, I've had a chance to play with the script in earnest for a bit (just enough to be usefully confused), and have some usage suggestions.

Thanks for the thorough review, Bryce.

I'm replying by email because it's easier to do inline comments.

> Before I get too far into it, I'll mention upfront that if this is
> going to be used entirely non-interactively, then a lot of the remarks
> in this comment are probably going to be irrelevant. However, if you
> anticipate anyone might be running this by hand, then they may be
> worthwhile.

Yeah, I think the only script that will be "heavily" used in an
interactive way is the "tag-images.sh", and *perhaps* the
"multi-arch-tagger.sh". I'm not expecting the other scripts to be used
interactively much, although I wanted them to support that as well.

> And first, here's how I'm running things:
>
> stirling:~/pkg/Oci/utils$ ./list-all-images.sh -r docker -n ubuntu
> Username: bryceharrington
> Password:
> ubuntu/nginx
> ubuntu/redis
> ubuntu/postgres
> ubuntu/mysql
> ubuntu/apache2
> ubuntu/memcached
> ubuntu/grafana
> ubuntu/prometheus
> ubuntu/cortex
> ubuntu/telegraf
> ubuntu/prometheus-alertmanager

Cool!

> $ ./list-tags-for-image.sh -r docker -n ubuntu -i redis
> Username: bryceharrington
> Password:
> ubuntu/redis:latest
> ubuntu/redis:6.0-21.04_beta
> ubuntu/redis:edge
> ubuntu/redis:6.0-21.04_edge
> ubuntu/redis:5.0-20.04_beta
> ubuntu/redis:5.0-20.04_edge

Cool again! :-)

> One thing that's a bit confusing is after running
> ./list-all-images.sh, I figured I could just cut and paste a line from
> its input to pass to ./list-tags-for-image.sh, however, that's wrong:
>
> stirling:~/pkg/Oci/utils$ ./list-tags-for-image.sh -r docker -n ubuntu -i ubuntu/redis
> Username: bryceharrington
> Password:
> E: Invalid image name 'ubuntu/redis'.

Yeah, that doesn't work because you're already specifying the namespace
via the "-n" parameter, and that will prepended to the image name when
composing the URL used to retrieve the information. I mean, I can
understand the initial confusion, but once you have a clearer picture of
what an "image", a "namespace" and a "registry" are, I think it makes
more sense.

On a side note, I've been thinking about reversing the "--no-prefix"
option and making it the default, because that's what we usually want as
an output.

> I'd probably just chalk that up to user error, but wouldn't be
> surprised if other rookie users make the same mistake. Again, if this
> is mainly intended to be used in an automated workflow, it's probably
> better to not have too many smarts in the argument parsing logic.
> You've got the --no-prefix argument if someone really needs the bare
> image name.

I'm expecting some of these scripts to be used interactively, as I
mentioned above, so maybe it's a good idea to write a more comprehensive
documentation. I don't think nothing fancy is needed; maybe just
extending the README file will be enough.

> stirling:~/pkg/Oci/utils$ ./multi-arch-tagger.sh -t foobar -n ubuntu -r docker -b 20.04 -u bryceharrington -p <my-docker-password> -i redis
> Username: bryceharrington
...

Read more...

Revision history for this message
Sergio Durigan Junior (sergiodj) :
Revision history for this message
Bryce Harrington (bryce) :
Revision history for this message
Paride Legovini (paride) wrote :

Great work with these scripts Sergio! I did a shellcheck run and they're almost clean, just with some nits like:

  SC2086: Double quote to prevent globbing and word splitting.

here and there, mostly about "safe" variables known to have no spaces. There are also some false positives I think . Given the high quality of the scripts it may be worth to make shellcheck fully happy, and test future MPs (possibly in CI) to keep quality high.

review: Approve
Revision history for this message
Sergio Durigan Junior (sergiodj) wrote :

Thanks for the review, Paride :-).

I have silenced everything that shellcheck flagged, and I have also addressed most (if not all) of the comments that Bryce has made.

I am merging the script now, because it is ready to be used. We can always come back and improve it if needed.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
diff --git a/README.multi-arch-tagger.md b/README.multi-arch-tagger.md
0new file mode 1006440new file mode 100644
index 0000000..dd33eee
--- /dev/null
+++ b/README.multi-arch-tagger.md
@@ -0,0 +1,67 @@
1Multi-arch tagger
2=================
3
4The following scripts are part of the multi-architecture tagger
5application. They are used to tag multiple architectures of an OCI
6image without having to actually run `docker tag` in those
7architectures.
8
9- `list-all-images.sh`: This script can be used to list all images
10 from a certain registry/namespace combination.
11
12- `list-tags-for-image.sh`: This script can be used to list all tags
13 pertaining to a certain image from a registry/namespace.
14
15- `multi-arch-tagger.sh`: This script can be used to perform the
16 multi-architecture tagging of a certain image from a
17 registry/namespace.
18
19- `tag-images.sh`: This script is a wrapper on top of
20 `multi-arch-tagger.sh`; it iterates over the available images from a
21 certain registry/namespace combination and tags them.
22
23All of these scripts accept a `--help` argument and print their
24usages, so it should not be hard to understand how to use them.
25
26Most of the time, you will actually just use the `tag-images.sh`
27script, which invokes all the other scripts in order to do its job.
28
29Requirements to run these scripts
30---------------------------------
31
32You have to have the following programs installed in your system in
33order to run the scripts:
34
35- jq
36- curl
37- awscli
38
39You don't need `docker` installed, since we use API endpoints when
40communicating with Dockerhub. AWS doesn't support most of the API
41endpoints yet, so that is why you need `awscli`.
42
43Configuring awscli for tag-images.sh
44-------------------------------------
45
46The tag-images.sh script requires that you install and configure
47`awscli` in order to tag images on AWS.
48
49To install the package:
50
51```
52 $ apt install awscli
53```
54
55To configure the software, you will need to create two profiles in
56order to use the tagger.
57
58```
59 $ aws configure --profile ubuntu
60 ...
61
62 $ aws configure --profile lts
63 ...
64```
65
66Each profile needs to be properly configured using the corresponding
67access keys, of course.
diff --git a/helpers/log.sh b/helpers/log.sh
0new file mode 10064468new file mode 100644
index 0000000..e3e6b9b
--- /dev/null
+++ b/helpers/log.sh
@@ -0,0 +1,49 @@
1#!/bin/bash
2
3# Generic logging function.
4_log ()
5{
6 local type="$1"
7 local msg="$2"
8 local redir=/dev/stdout
9
10 case "$type" in
11 "E"|"D")
12 redir=/dev/stderr
13 ;;
14 "I"|"W")
15 :
16 ;;
17 *)
18 # Can't call error here, can we?
19 echo "E: Internal error in the '_log' function!" \
20 > /dev/stderr
21 ;;
22 esac
23
24 printf "%s: %s\n" "$type" "$msg" > $redir
25}
26
27# Error logging.
28error ()
29{
30 _log "E" "$1"
31}
32
33# Warn logging.
34warn ()
35{
36 _log "W" "$1"
37}
38
39# Info logging.
40info ()
41{
42 _log "I" "$1"
43}
44
45# Debug logging.
46debug ()
47{
48 _log "D" "$1"
49}
diff --git a/helpers/registry-login.sh b/helpers/registry-login.sh
0new file mode 10064450new file mode 100644
index 0000000..fa68362
--- /dev/null
+++ b/helpers/registry-login.sh
@@ -0,0 +1,122 @@
1#!/bin/bash
2
3# Log into a registry.
4
5set -e
6
7readonly AWS_URL="https://public.ecr.aws"
8readonly AWS_REGISTRY_URL="https://public.ecr.aws"
9export AWS_URL AWS_REGISTRY_URL
10
11readonly DOCKERHUB_URL="https://hub.docker.com"
12readonly DOCKERHUB_REGISTRY_URL="https://registry-1.docker.io"
13export DOCKERHUB_URL DOCKERHUB_REGISTRY_URL
14
15# Log into dockerhub, asking the user and the password.
16#
17# This function doesn't take arguments. It sets the AUTH_TOKEN
18# environment variable to the Authentication Token that can be used on
19# further requests.
20_login_docker ()
21{
22 if [ -z "$USERNAME" ]; then
23 read -rp "Username: " USERNAME
24 fi
25 if [ -z "$PASSWORD" ]; then
26 read -rsp "Password: " PASSWORD
27 echo > /dev/stderr
28 fi
29
30 AUTH_TOKEN=$(curl -s \
31 -H "Content-Type: application/json" \
32 -X POST -d '{"username": "'"${USERNAME}"'", "password": "'"${PASSWORD}"'"}' \
33 "${DOCKERHUB_URL}/v2/users/login/" | jq -r .token)
34
35 if [ -z "$AUTH_TOKEN" ]; then
36 error "There was an error while logging into the service."
37 exit 1
38 fi
39
40 if [ -n "$DEBUG" ]; then
41 debug "This is the Dockerhub authentication token I got, in case you want to reuse it:"
42 printf 'AUTH_TOKEN=%s\n' "$AUTH_TOKEN" > /dev/stderr
43 fi
44
45 REGISTRY_USERNAME="$USERNAME"
46 REGISTRY_PASSWORD="$PASSWORD"
47}
48
49# Log into dockerhub's registry1, asking the user and the password.
50#
51# It takes the list of images to be used as the scope as argument. It
52# assumes that the image names are not prefixed with "$NAMESPACE/"
53_login_docker_registry1 ()
54{
55 local IMAGES_FOR_SCOPE="$1"
56 local REGISTRY_USERNAME="$2"
57 local REGISTRY_PASSWORD="$3"
58 local URL="https://auth.docker.io/token?service=registry.docker.io"
59
60 if [ -z "$REGISTRY_USERNAME" ] || [ -z "$REGISTRY_PASSWORD" ]; then
61 error "You must specify the registry username and password."
62 exit 1
63 fi
64
65 for image in $IMAGES_FOR_SCOPE; do
66 URL+="&scope=repository:${NAMESPACE}/${image}:pull,push"
67 done
68
69 REGISTRY_AUTH_TOKEN=$(curl -s \
70 -u "${REGISTRY_USERNAME}:${REGISTRY_PASSWORD}" \
71 "$URL" | jq -r .token)
72 export REGISTRY_AUTH_TOKEN
73
74 if [ -z "$REGISTRY_AUTH_TOKEN" ]; then
75 error "There was an error while logging into the registry."
76 exit 1
77 fi
78
79 export REGISTRY_URL="$DOCKERHUB_REGISTRY_URL"
80}
81
82_login_aws ()
83{
84 # No need to login with aws since we will be using aws-cli.
85 if ! command -v aws >& /dev/null; then
86 error "You must have the aws-cli program installed."
87 exit 1
88 fi
89
90 if ! aws configure list --profile "$NAMESPACE" >& /dev/null; then
91 error "You must configure aws-cli with a profile named '$NAMESPACE'."
92 error "Please refer to the README file for instructions."
93 exit 1
94 fi
95
96 export AWS_PROFILE="$NAMESPACE"
97}
98
99_login_aws_registry1 ()
100{
101 REGISTRY_AUTH_TOKEN=$(aws ecr-public get-authorization-token \
102 --region us-east-1 \
103 --output=text \
104 --query 'authorizationData.authorizationToken')
105
106
107 if [ -z "$REGISTRY_AUTH_TOKEN" ]; then
108 error "There was an error while logging into the registry."
109 exit 1
110 fi
111
112 REGISTRY_URL="$AWS_REGISTRY_URL"
113}
114
115do_login ()
116{
117 if [ -n "$AUTH_TOKEN" ]; then
118 return
119 fi
120
121 _login_"${REGISTRY}"
122}
diff --git a/helpers/validate-args.sh b/helpers/validate-args.sh
0new file mode 100644123new file mode 100644
index 0000000..551c6f7
--- /dev/null
+++ b/helpers/validate-args.sh
@@ -0,0 +1,85 @@
1#!/bin/bash
2
3progname=$(basename "${0}")
4progdir=$(dirname "$(readlink -f "${0}")")
5
6export progname progdir
7
8# shellcheck disable=SC1090
9. "${progdir}/helpers/log.sh"
10
11validate_var ()
12{
13 local varname="$1"
14 local varvalue
15
16 varvalue=$(eval "echo \$${varname}")
17
18 if [ -z "$varvalue" ]; then
19 error "You must specify the ${varname} parameter."
20 return 1
21 fi
22
23 return 0
24}
25
26validate_registry ()
27{
28 local registry="$1"
29
30 case "$registry" in
31 "docker"|"aws")
32 :
33 ;;
34 *)
35 error "Invalid registry: '$registry'."
36 return 1
37 ;;
38 esac
39
40 return 0
41}
42
43validate_namespace ()
44{
45 local namespace="$1"
46
47 case "$namespace" in
48 "ubuntu")
49 :
50 ;;
51 "lts")
52 # shellcheck disable=SC2153
53 if [ "$REGISTRY" = "docker" ]; then
54 error "The lts namespace is not available on dockerhub yet."
55 return 1
56 fi
57 ;;
58 *)
59 error "Invalid namespace: '$namespace:'."
60 return 1
61 ;;
62 esac
63
64 return 0
65}
66
67validate_image ()
68{
69 local image="$1"
70 local -a ALL_IMAGES
71
72 # shellcheck disable=SC2153
73 # Fill the ALL_IMAGES array.
74 readarray -t ALL_IMAGES < <("${progdir}/list-all-images.sh" \
75 -r "$REGISTRY" \
76 -n "$NAMESPACE" \
77 -a "$AUTH_TOKEN")
78
79 if ! grep -Fwq "$image" <<< "${ALL_IMAGES[@]}"; then
80 error "Invalid image name '$image'."
81 return 1
82 fi
83
84 return 0
85}
diff --git a/list-all-images.sh b/list-all-images.sh
0new file mode 10075586new file mode 100755
index 0000000..c82b50c
--- /dev/null
+++ b/list-all-images.sh
@@ -0,0 +1,129 @@
1#!/bin/bash
2
3set -e
4
5progname=$(basename "${0}")
6progdir=$(dirname "$(readlink -f "${0}")")
7
8# shellcheck disable=SC1090
9. "${progdir}/helpers/validate-args.sh"
10# shellcheck disable=SC1090
11. "${progdir}/helpers/log.sh"
12# shellcheck disable=SC1090
13. "${progdir}/helpers/registry-login.sh"
14
15# Obtain the list of images for a certain repository.
16
17# Usage.
18usage ()
19{
20 cat > /dev/stderr <<EOF
21$progname -- Obtain the list of images from a certain repository.
22
23Usage:
24
25 $progname [-a <TOKEN>] -r <docker|aws> -n <ubuntu|lts> [-d]
26
27Arguments:
28
29 -a TOKEN|--auth-token TOKEN Specify the authorization token that
30 should be used when connecting to the
31 registry's API. If not provided, the
32 script will ask for your user/password.
33
34 -r REGISTRY|--registry REGISTRY Registry to consult.
35 Possible values: docker, aws
36
37 -n NAMESPACE|--namespace NAMESPACE: Namespace to consult.
38 Possible values: ubuntu, lts
39
40 -d|--debug Print debug statements. This includes printing
41 the authentication token if using Dockerhub.
42
43 -h|--help Display this usage
44EOF
45}
46
47validate_args ()
48{
49 local ret=0
50
51 validate_var REGISTRY || ret=1
52 validate_var NAMESPACE || ret=1
53
54 validate_registry "$REGISTRY" || ret=1
55 validate_namespace "$NAMESPACE" || ret=1
56
57 if [ "$ret" -eq 1 ]; then
58 error "Failed to validate arguments."
59 exit 1
60 fi
61
62 do_login
63}
64
65print_list_of_images_docker ()
66{
67 for image in $(curl -s \
68 -H "Authorization: Bearer ${AUTH_TOKEN}" \
69 "${DOCKERHUB_URL}/v2/repositories/${NAMESPACE}/?page_size=10000" \
70 | jq -r '.results|.[]|.name'); do
71 echo "$image"
72 done
73}
74
75print_list_of_images_aws ()
76{
77 aws --region us-east-1 ecr-public describe-repositories \
78 | jq -r '.repositories[].repositoryName'
79}
80
81print_list_of_images ()
82{
83 print_list_of_images_"${REGISTRY}"
84}
85
86if [ $# -lt 2 ]; then
87 error "You need to provide at least the registry (-r) and the namespace (-n) arguments."
88 usage
89 exit 1
90fi
91
92while [ -n "$1" ]; do
93 case "$1" in
94 "-a"|"--auth-token")
95 AUTH_TOKEN="$2"
96 shift 2
97 ;;
98
99 "-r"|"--registry")
100 REGISTRY="$2"
101 shift 2
102 ;;
103
104 "-n"|"--namespace")
105 NAMESPACE="$2"
106 shift 2
107 ;;
108
109 "-d"|"--debug")
110 export DEBUG=1
111 shift
112 ;;
113
114 "-h"|"--help")
115 usage
116 exit 0
117 ;;
118
119 *)
120 error "Invalid argument '$1'."
121 usage
122 exit 1
123 ;;
124 esac
125done
126
127validate_args
128
129print_list_of_images
diff --git a/list-tags-for-image.sh b/list-tags-for-image.sh
0new file mode 100755130new file mode 100755
index 0000000..305c9be
--- /dev/null
+++ b/list-tags-for-image.sh
@@ -0,0 +1,141 @@
1#!/bin/bash
2
3set -e
4
5progname=$(basename "${0}")
6progdir=$(dirname "$(readlink -f "${0}")")
7
8# shellcheck disable=SC1090
9. "${progdir}/helpers/validate-args.sh"
10# shellcheck disable=SC1090
11. "${progdir}/helpers/log.sh"
12# shellcheck disable=SC1090
13. "${progdir}/helpers/registry-login.sh"
14
15# Obtain the list of tags for a certain image.
16
17# Usage.
18usage ()
19{
20 cat > /dev/stderr <<EOF
21$progname -- Obtain the list of tags from a certain image.
22
23Usage:
24
25 $progname [-a <TOKEN>] -r <docker|aws> -n <ubuntu|lts> -i <IMAGE> [-d]
26
27Arguments:
28
29 -a TOKEN|--auth-token TOKEN Specify the authorization token that
30 should be used when connecting to the
31 registry's API. If not provided, the
32 script will ask for your user/password.
33
34 -r REGISTRY|--registry REGISTRY Registry to consult.
35 Possible values: docker, aws
36
37 -n NAMESPACE|--namespace NAMESPACE Namespace to consult.
38 Possible values: ubuntu, lts
39
40 -i IMAGE|--image IMAGE Image to obtain tags from.
41
42 -d|--debug Print debug statements. This includes printing
43 the authentication token if using Dockerhub.
44
45 -h|--help Display this usage
46EOF
47}
48
49validate_args ()
50{
51 local ret=0
52
53 validate_var REGISTRY || ret=1
54 validate_var NAMESPACE || ret=1
55 validate_var IMAGE || ret=1
56
57 validate_registry "$REGISTRY" || ret=1
58 validate_namespace "$NAMESPACE" || ret=1
59
60 if [ "$ret" -eq 1 ]; then
61 error "Failed to validate arguments."
62 exit 1
63 fi
64
65 do_login
66
67 validate_image "$IMAGE"
68}
69
70print_tags_for_image_docker ()
71{
72 for tag in $(curl -s \
73 -H "Authorization: JWT ${AUTH_TOKEN}" \
74 "${DOCKERHUB_URL}/v2/repositories/${NAMESPACE}/${IMAGE}/tags/?page_size=10000" \
75 | jq -r '.results|.[]|.name'); do
76 echo "${tag}"
77 done
78}
79
80print_tags_for_image_aws ()
81{
82 for tag in $(aws --region us-east-1 ecr-public describe-image-tags --repository-name "${IMAGE}" \
83 | jq -r '.imageTagDetails[].imageTag'); do
84 echo "${tag}"
85 done
86}
87
88print_tags_for_image ()
89{
90 print_tags_for_image_"${REGISTRY}"
91}
92
93if [ $# -lt 2 ]; then
94 error "You need to provide at least the registry (-r), the namespace (-n) and the image (-i) arguments."
95 usage
96 exit 1
97fi
98
99while [ -n "$1" ]; do
100 case "$1" in
101 "-a"|"--auth-token")
102 AUTH_TOKEN="$2"
103 shift 2
104 ;;
105
106 "-r"|"--registry")
107 REGISTRY="$2"
108 shift 2
109 ;;
110
111 "-n"|"--namespace")
112 NAMESPACE="$2"
113 shift 2
114 ;;
115
116 "-i"|"--image")
117 IMAGE="$2"
118 shift 2
119 ;;
120
121 "-d"|"--debug")
122 export DEBUG=1
123 shift
124 ;;
125
126 "-h"|"--help")
127 usage
128 exit 0
129 ;;
130
131 *)
132 error "Invalid argument '$1'."
133 usage
134 exit 1
135 ;;
136 esac
137done
138
139validate_args
140
141print_tags_for_image
diff --git a/multi-arch-tagger.sh b/multi-arch-tagger.sh
0new file mode 100755142new file mode 100755
index 0000000..a493fa2
--- /dev/null
+++ b/multi-arch-tagger.sh
@@ -0,0 +1,220 @@
1#!/bin/bash
2
3set -e
4
5progname=$(basename "${0}")
6progdir=$(dirname "$(readlink -f "${0}")")
7
8# shellcheck disable=SC1090
9. "${progdir}/helpers/validate-args.sh"
10# shellcheck disable=SC1090
11. "${progdir}/helpers/log.sh"
12# shellcheck disable=SC1090
13. "${progdir}/helpers/registry-login.sh"
14
15# Tag all architectures of an image.
16
17# Usage.
18usage ()
19{
20 cat > /dev/stderr <<EOF
21$progname -- Tag all architectures of an image.
22
23Due to the way docker's API works, you can provide an authentication
24token for the hub.docker.com service (if you don't, the script will
25ask for you user and password), but you *must* provide your username
26and password for the registry1.docker.io service. This is likely the
27same username/password you use for docker.
28
29Usage (when using Dockerhub):
30
31 $progname [-a <TOKEN>] -r docker -n <ubuntu|lts> -i <IMAGE> -s <SOURCE_TAG> -t <NEW_TAG> [-u <USERNAME>] [-p <PASSWORD>] [-d]
32
33Usage (when using AWS):
34
35 $progname [-a <TOKEN>] -r aws -n <ubuntu|lts> -i <IMAGE> -s <SOURCE_TAG> -t <NEW_TAG> [-d]
36
37Arguments:
38
39 -a TOKEN|--auth-token TOKEN Specify the authorization token that
40 should be used when connecting to the
41 registry's API. If not provided, the
42 script will ask for your user/password.
43 This is **NOT** the authentication token
44 for the registry (i.e., not for e.g.
45 registry1.docker.io, if you're using
46 docker). See the -u and -p options
47 below.
48
49 -r REGISTRY|--registry REGISTRY Registry to consult.
50 Possible values: docker, aws
51
52 -n NAMESPACE|--namespace NAMESPACE Namespace to consult.
53 Possible values: ubuntu, lts
54
55 -i IMAGE|--image IMAGE Name of an image from the repository (without
56 namespace), for example 'postgres' or 'redis'.
57
58 -s SOURCE_TAG|--source-tag SOURCE_TAG Use SOURCE_TAG as the source tag when
59 tagging the image.
60
61 -t NEW_TAG|--tag NEW_TAG Tag image using NEW_TAG.
62
63 -u USERNAME|--username USERNAME This is your Dockerhub registry username. Only
64 required when using Dockerhub as the registry.
65 This will be the same as your regular Dockerhub
66 username.
67
68 -p PASSWORD|--password PASSWORD This is your Dockerhub registry password. Only
69 required when using Dockerhub as the registry.
70 This will be the same as your Dockerhub username.
71
72 -d|--debug Print debug statements. This includes printing
73 the authentication token if using Dockerhub.
74
75 -h|--help Display this usage
76EOF
77}
78
79do_tag_image ()
80{
81 info "Tagging ${NAMESPACE}/${IMAGE}:${SOURCE_TAG} as ${NAMESPACE}/${IMAGE}:${TAG} (on ${REGISTRY})"
82
83 local tmpfile
84 local ret
85
86 tmpfile=$(mktemp)
87 trap 'rm -f ${tmpfile}' 0 INT QUIT ABRT PIPE TERM
88
89 curl -s -H "Authorization: Bearer ${REGISTRY_AUTH_TOKEN}" \
90 -H "Accept: application/vnd.docker.distribution.manifest.list.v2+json" \
91 "${REGISTRY_URL}/v2/${NAMESPACE}/${IMAGE}/manifests/${SOURCE_TAG}" \
92 > "$tmpfile"
93
94 # We use "|| true" here because we want the command to complete
95 # even if it fails. We check for the HTTP response code below.
96 ret=$(curl -X PUT -H "Authorization: Bearer ${REGISTRY_AUTH_TOKEN}" \
97 -H "Content-Type: application/vnd.docker.distribution.manifest.list.v2+json" \
98 "${REGISTRY_URL}/v2/${NAMESPACE}/${IMAGE}/manifests/${TAG}" \
99 -d "@${tmpfile}" \
100 -s -o /dev/null -w "%{http_code}" || true)
101
102 rm "$tmpfile"
103
104 if [ "$ret" -ne 201 ]; then
105 echo "ERROR: Image tagging failed with status code $ret"
106 exit 1
107 fi
108}
109
110do_registry_login ()
111{
112 _login_"${REGISTRY}"_registry1 "$IMAGE" "$REGISTRY_USERNAME" "$REGISTRY_PASSWORD"
113}
114
115validate_args ()
116{
117 local ret=0
118
119 validate_var REGISTRY || ret=1
120 validate_var NAMESPACE || ret=1
121 validate_var IMAGE || ret=1
122 validate_var SOURCE_TAG || ret=1
123 validate_var TAG || ret=1
124
125 validate_registry "$REGISTRY" || ret=1
126 validate_namespace "$NAMESPACE" || ret=1
127
128 if [ "$ret" -eq 1 ]; then
129 error "Failed to validate arguments."
130 exit 1
131 fi
132
133 if [ -n "$REGISTRY_USERNAME" ]; then
134 export USERNAME="$REGISTRY_USERNAME"
135 fi
136 if [ -n "$REGISTRY_PASSWORD" ]; then
137 export PASSWORD="$REGISTRY_PASSWORD"
138 fi
139
140 do_login
141
142 validate_image "$IMAGE" || exit 1
143
144 if [ "$REGISTRY" = "docker" ]; then
145 validate_var REGISTRY_USERNAME || exit 1
146 validate_var REGISTRY_PASSWORD || exit 1
147 fi
148
149 do_registry_login
150}
151
152if [ $# -lt 2 ]; then
153 error "You need to provide arguments to the script."
154 usage
155 exit 1
156fi
157
158while [ -n "$1" ]; do
159 case "$1" in
160 "-a"|"--auth-token")
161 export AUTH_TOKEN="$2"
162 shift 2
163 ;;
164
165 "-r"|"--registry")
166 REGISTRY="$2"
167 shift 2
168 ;;
169
170 "-n"|"--namespace")
171 NAMESPACE="$2"
172 shift 2
173 ;;
174
175 "-i"|"--image")
176 IMAGE="$2"
177 shift 2
178 ;;
179
180 "-s"|"--source-tag")
181 SOURCE_TAG="$2"
182 shift 2
183 ;;
184
185 "-t"|"--tag")
186 TAG="$2"
187 shift 2
188 ;;
189
190 "-u"|"--username")
191 REGISTRY_USERNAME="$2"
192 shift 2
193 ;;
194
195 "-p"|"--password")
196 REGISTRY_PASSWORD="$2"
197 shift 2
198 ;;
199
200 "-d"|"--debug")
201 export DEBUG=1
202 shift
203 ;;
204
205 "-h"|"--help")
206 usage
207 exit 0
208 ;;
209
210 *)
211 error "Invalid argument '$1'."
212 usage
213 exit 1
214 ;;
215 esac
216done
217
218validate_args
219
220do_tag_image
diff --git a/tag-images.sh b/tag-images.sh
0new file mode 100755221new file mode 100755
index 0000000..874673a
--- /dev/null
+++ b/tag-images.sh
@@ -0,0 +1,335 @@
1#!/bin/bash
2
3set -e
4
5progname=$(basename "${0}")
6progdir=$(dirname "$(readlink -f "${0}")")
7
8# shellcheck disable=SC1090
9. "${progdir}/helpers/validate-args.sh"
10# shellcheck disable=SC1090
11. "${progdir}/helpers/log.sh"
12# shellcheck disable=SC1090
13. "${progdir}helpers/registry-login.sh"
14
15# Tag images from a repository.
16
17# The list of images to tag.
18IMAGES=()
19
20# The list of all available images.
21ALL_IMAGES=()
22
23# Default source tag is "edge".
24SOURCE_TAG=edge
25
26# Usage.
27usage ()
28{
29 cat > /dev/stderr <<EOF
30$progname -- Tag images from a repository (multi-architecture).
31
32This script will tag images from a repository using a certain tag the
33source tag.
34
35******************************* DISCLAIMER **************************************
36** This script will only work with the 'latest' and the 'M.N-XX.YY_beta' tags! **
37*********************************************************************************
38
39This script is needed because of a few particularities in the tagging
40policy for our OCI images.
41
42The first is the fact that Launchpad will not tag anything using the
43"latest" tag. When it finishes a build, the only tags it will
44generate are the "edge" and the "M.N-XX.YY_edge" tags. This means
45that we need to manually tag everything using "latest" after the
46images have been uploaded to the registries. We also need to create
47the "M.N-XX.YY_beta" tag, since that is not automatically created by
48Launchpad either.
49
50The other problem we have to consider is that, when tagging an image,
51we must tag all of the architectures that were uploaded. If we were
52to do that using "docker", we would need to run it on several
53machines, one for each supported architectures, which is not always
54possible/feasible. By using API calls directly, we are able to tag
55all architectures using a single machine.
56
57Usage:
58
59 $progname [-a <TOKEN>] -r <docker|aws> -n <ubuntu|lts> -s <SOURCE_TAG> [-d] [-- IMAGE_1 IMAGE_2 ... IMAGE_N]
60
61Arguments:
62
63 -r REGISTRY|--registry REGISTRY Registry to consult.
64 Possible values: docker, aws
65
66 -n NAMESPACE|--namespace NAMESPACE Namespace to consult.
67 Possible values: ubuntu, lts
68
69 -s TAG|--source-tag SOURCE_TAG Use SOURCE_TAG as the source tag when tagging
70 the images. If not specified, the default
71 is 'edge'.
72
73 -f|--force Force tagging, even if the script would think
74 that it's not needed. This mostly applies for
75 the 'M.N-XX.YY_beta' tag, which is only created
76 if it doesn't exist.
77
78 -d|--debug Print debug statements. This includes printing
79 the authentication token if using Dockerhub.
80
81 -- IMAGE_1 IMAGE_2 ... IMAGE_N Images to tag. If not specified,
82 the script will obtain the list of
83 all images from REGISTRY/NAMESPACE
84 and tag them.
85
86 -h|--help Display this usage
87EOF
88}
89
90# Tag the 'ubuntu' base image.
91#
92# We perform the tagging of '_stable', '_beta', '_candidate', 'XX.YY'
93# and 'RELEASE_NAME'.
94do_tag_base_ubuntu_image ()
95{
96 local image="ubuntu"
97 local -a TAG_SUFFIXES=( "stable" "beta" "candidate" )
98
99 if [ "$REGISTRY" != "aws" ]; then
100 error "The 'ubuntu' base image is only present in the 'aws' registry."
101 exit 1
102 fi
103
104 local -a IMAGE_TAGS EDGE_TAGS
105
106 readarray -t IMAGE_TAGS < <("${progdir}./list-tags-for-image.sh" \
107 -a "$AUTH_TOKEN" \
108 -r "$REGISTRY" \
109 -n "$NAMESPACE" \
110 -i "$image")
111
112 readarray -t EDGE_TAGS < <(printf '%s\n' "${IMAGE_TAGS[@]}" \
113 | grep '_edge$')
114
115 for edgetag in "${EDGE_TAGS[@]}"; do
116 # We support retagging images tagged as DISTRONAME-XX.YY_edge.
117 local EDGE_TAG_PREFIX
118 EDGE_TAG_PREFIX=$(sed -ne 's@\([[:alpha:]]\+-[[:digit:]]\+\.[[:digit:]]\+\)_edge@\1@p' <<< "$edgetag")
119 if [ -z "$EDGE_TAG_PREFIX" ]; then
120 warn "Unable to obtain prefix for tag '$edgetag'."
121 continue
122 fi
123
124 local IMG_DISTRONAME
125 local IMG_RELEASEVER
126
127 IMG_DISTRONAME=$(cut -d'-' -f1 <<< "$EDGE_TAG_PREFIX")
128 IMG_RELEASEVER=$(cut -d'-' -f2 <<< "$EDGE_TAG_PREFIX")
129 for tagsuffix in "${TAG_SUFFIXES[@]}"; do
130 # Tag each suffix.
131 if [ -n "$FORCE" ] || ! grep -Fwq "${IMG_RELEASEVER}_${tagsuffix}" <<< "${IMAGE_TAGS[@]}"; then
132 info "Invoking multi-arch tagger for ${NAMESPACE}/${image}:${IMG_RELEASEVER}_${tagsuffix} (source tag: ${EDGE_TAG_PREFIX}_edge) (on ${REGISTRY})"
133 "${progdir}./multi-arch-tagger.sh" \
134 -a "$AUTH_TOKEN" \
135 -r "$REGISTRY" \
136 -n "$NAMESPACE" \
137 -i "$image" \
138 -s "${EDGE_TAG_PREFIX}_edge" \
139 -t "${IMG_RELEASEVER}_${tagsuffix}" \
140 -u "$USERNAME" \
141 -p "$PASSWORD"
142 fi
143 done
144
145 # Tag the DISTRONAME.
146 if [ -n "$FORCE" ] || ! grep -Fwq "${IMG_DISTRONAME}" <<< "${IMAGE_TAGS[@]}"; then
147 info "Invoking multi-arch tagger for ${NAMESPACE}/${image}:${IMG_DISTRONAME} (source tag: ${EDGE_TAG_PREFIX}_edge) (on ${REGISTRY})"
148 "${progdir}./multi-arch-tagger.sh" \
149 -a "$AUTH_TOKEN" \
150 -r "$REGISTRY" \
151 -n "$NAMESPACE" \
152 -i "$image" \
153 -s "${EDGE_TAG_PREFIX}_edge" \
154 -t "${IMG_DISTRONAME}" \
155 -u "$USERNAME" \
156 -p "$PASSWORD"
157 fi
158
159 # Tag the RELEASEVER
160 if [ -n "$FORCE" ] || ! grep -Fwq "${IMG_RELEASEVER}" <<< "${IMAGE_TAGS[@]}"; then
161 info "Invoking multi-arch tagger for ${NAMESPACE}/${image}:${IMG_RELEASEVER} (source tag: ${EDGE_TAG_PREFIX}_edge) (on ${REGISTRY})"
162 "${progdir}./multi-arch-tagger.sh" \
163 -a "$AUTH_TOKEN" \
164 -r "$REGISTRY" \
165 -n "$NAMESPACE" \
166 -i "$image" \
167 -s "${EDGE_TAG_PREFIX}_edge" \
168 -t "${IMG_RELEASEVER}" \
169 -u "$USERNAME" \
170 -p "$PASSWORD"
171 fi
172 done
173}
174
175do_tag_images ()
176{
177 # Tag images using the 'latest' tag.
178 for image in "${IMAGES[@]}"; do
179 info "Invoking multi-arch tagger for ${NAMESPACE}/${image}:latest (source tag: ${SOURCE_TAG}) (on ${REGISTRY})"
180 "${progdir}./multi-arch-tagger.sh" \
181 -a "$AUTH_TOKEN" \
182 -r "$REGISTRY" \
183 -n "$NAMESPACE" \
184 -i "$image" \
185 -s "$SOURCE_TAG" \
186 -t "latest" \
187 -u "$USERNAME" \
188 -p "$PASSWORD"
189 done
190
191 # Now comes the fun part. We have to determine whether the image
192 # needs a 'M.N-XX.YY_beta' tag, and which number should N, M, XX
193 # and YY be.
194 #
195 # Basically, if there is a tag named 'M.N-XX.YY_edge' without a
196 # corresponding '_beta' tag, then we assume that the image will
197 # need to receive the '_beta' tag. This is the usual case
198 # (currently) with images built using Launchpad.
199 for image in "${IMAGES[@]}"; do
200 if [ "$image" = "ubuntu" ]; then
201 # The ubuntu image is special.
202 do_tag_base_ubuntu_image
203 continue
204 fi
205
206 local -a IMAGE_TAGS EDGE_TAGS
207
208 readarray -t IMAGE_TAGS < <("${progdir}./list-tags-for-image.sh" \
209 -a "$AUTH_TOKEN" \
210 -r "$REGISTRY" \
211 -n "$NAMESPACE" \
212 -i "$image")
213
214 readarray -t EDGE_TAGS < <(printf '%s\n' "${IMAGE_TAGS[@]}" \
215 | grep '_edge$' | sort -r)
216
217 for edgetag in "${EDGE_TAGS[@]}"; do
218 local EDGE_TAG_PREFIX
219 EDGE_TAG_PREFIX=$(sed -ne 's@\([[:digit:]]\+\(\.[[:digit:]]\+\)\?-[[:digit:]]\+\.[[:digit:]]\+\)_edge@\1@p' <<< "$edgetag")
220 if [ -z "$EDGE_TAG_PREFIX" ]; then
221 warn "Unable to obtain prefix for tag '$edgetag'."
222 continue
223 fi
224 if [ -n "$FORCE" ] || ! grep -Fwq "${EDGE_TAG_PREFIX}_beta" <<< "${IMAGE_TAGS[@]}"; then
225 # If we're here, it means that there isn't a
226 # corresponding 'M.N-XX.YY_beta' tag associated with
227 # the '_edge' tag, so we need to create it.
228 info "Invoking multi-arch tagger for ${NAMESPACE}/${image}:${EDGE_TAG_PREFIX}_beta (source tag: ${EDGE_TAG_PREFIX}_edge) (on ${REGISTRY})"
229 "${progdir}./multi-arch-tagger.sh" \
230 -a "$AUTH_TOKEN" \
231 -r "$REGISTRY" \
232 -n "$NAMESPACE" \
233 -i "$image" \
234 -s "${EDGE_TAG_PREFIX}_edge" \
235 -t "${EDGE_TAG_PREFIX}_beta" \
236 -u "$USERNAME" \
237 -p "$PASSWORD"
238 fi
239 done
240 done
241}
242
243validate_args ()
244{
245 local ret=0
246
247 validate_var REGISTRY || ret=1
248 validate_var NAMESPACE || ret=1
249 validate_var SOURCE_TAG || ret=1
250
251 validate_registry "$REGISTRY" || ret=1
252 validate_namespace "$NAMESPACE" || ret=1
253
254 if [ "$ret" -eq 1 ]; then
255 error "Failed to validate arguments."
256 exit 1
257 fi
258
259 do_login
260
261 # Fill the ALL_IMAGES array.
262 readarray -t ALL_IMAGES < <("${progdir}./list-all-images.sh" \
263 -r "$REGISTRY" \
264 -n "$NAMESPACE" \
265 -a "$AUTH_TOKEN")
266
267 if [ "${#IMAGES[@]}" -eq 0 ]; then
268 # The user hasn't provided any image names, so just use all of
269 # them.
270 IMAGES=( "${ALL_IMAGES[@]}" )
271 else
272 # Validate the image names the user has provided.
273 for img in "${IMAGES[@]}"; do
274 validate_image "$img" || exit 1
275 done
276 fi
277}
278
279if [ $# -lt 2 ]; then
280 error "You need to provide at least the registry (-r), the namespace (-n), and the source tag (-s)."
281 usage
282 exit 1
283fi
284
285while [ -n "$1" ]; do
286 case "$1" in
287 "-r"|"--registry")
288 REGISTRY="$2"
289 shift 2
290 ;;
291
292 "-n"|"--namespace")
293 NAMESPACE="$2"
294 shift 2
295 ;;
296
297 "-s"|"--source-tag")
298 SOURCE_TAG="$2"
299 shift 2
300 ;;
301
302 "-f"|"--force")
303 FORCE=1
304 shift
305 ;;
306
307 "-d"|"--debug")
308 export DEBUG=1
309 shift
310 ;;
311
312 "-h"|"--help")
313 usage
314 exit 0
315 ;;
316
317 "--")
318 shift
319 while [ -n "$1" ]; do
320 IMAGES+=( "$1" )
321 shift
322 done
323 ;;
324
325 *)
326 error "Invalid argument '$1'."
327 usage
328 exit 1
329 ;;
330 esac
331done
332
333validate_args
334
335do_tag_images

Subscribers

People subscribed via source and target branches

to all changes: