Merge ~ubuntu-docker-images/ubuntu-docker-images/+git/templates:add_script_for_ubuntu into ~ubuntu-docker-images/ubuntu-docker-images/+git/templates:main

[Samir Akarioh]

feat : add Ubuntu yaml file generator

This commit add a script which permit to generate
the ubuntu yaml file

Co-Signed-Off: Samir Akarioh and Cristovao Cordeiro

[Cristovao Cordeiro (cjdc)]

I guess PyYaml needs to be installed. If so, please adjust the readme and .devcontainer accordingly

review: Needs Fixing

[Samir Akarioh (samiraka)]

by .devcontainer you want to says the dockerfile ?

[Cristovao Cordeiro (cjdc)]

> by .devcontainer you want to says the dockerfile ?

yes

[Cristovao Cordeiro (cjdc)]

please make the script executable

review: Needs Fixing

[Samir Akarioh (samiraka)]

no need it's already install with renderdown : https://github.com/ValentinViennot/RenderDown/blob/master/requirements.txt

[Samir Akarioh (samiraka)]

> please make the script executable
What do you mean by that?

[Cristovao Cordeiro (cjdc)]

please format the file with black and isort and flake8

review: Needs Fixing

[Samir Akarioh (samiraka)]

Done

[Cristovao Cordeiro (cjdc)]

review: Needs Fixing

[Cristovao Cordeiro (cjdc)]

> no need it's already install with renderdown :
> https://github.com/ValentinViennot/RenderDown/blob/master/requirements.txt

and so is boto3

[Samir Akarioh (samiraka)]

it's better to do a MP on github to add boto3 no ?

[Cristovao Cordeiro (cjdc)]

> it's better to do a MP on github to add boto3 no ?

does renderdown need it?

[Samir Akarioh (samiraka)]

no

[Cristovao Cordeiro (cjdc)]

> no

then it shouldn't go there. let's update the README and .devcontainer in this project instead

[Samir Akarioh (samiraka)]

i updated it

[Cristovao Cordeiro (cjdc)]

ok thanks. pls let me know once the other comments from https://code.launchpad.net/~ubuntu-docker-images/ubuntu-docker-images/+git/templates/+merge/431915/comments/1144725 are addressed

review: Needs Fixing

[Samir Akarioh (samiraka)]

Done

[Samir Akarioh (samiraka)]

review: Needs Fixing

[Samir Akarioh (samiraka)]

I review your script, you want me to do the correction or ?

[Cristovao Cordeiro (cjdc)]

thanks. fixed

[Samir Akarioh (samiraka)]

review: Needs Information

[Samir Akarioh (samiraka)]

review: Approve

[Cristovao Cordeiro (cjdc)]

review: Approve

[Cristovao Cordeiro (cjdc)]

I think you can merge it now Samir

diff --git a/.devcontainer/Dockerfile b/.devcontainer/Dockerfile
index fa24741..3c8c2b3 100644
--- a/.devcontainer/Dockerfile
+++ b/.devcontainer/Dockerfile
@@ -6,6 +6,6 @@ RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
     && apt-get -y install --no-install-recommends python3-mako python3-yaml
 
 RUN cd /usr/bin && git clone https://github.com/valentinviennot/RenderDown \
-    && pip3 --disable-pip-version-check --no-cache-dir install -r /usr/bin/RenderDown/requirements.txt \ 
+    && pip3 --disable-pip-version-check --no-cache-dir install -r /usr/bin/RenderDown/requirements.txt boto3 \ 
     && rm -rf /tmp/pip-tmp
 ENV RENDERDOWN /usr/bin/RenderDown/renderdown.py
diff --git a/README.md b/README.md
index 0166dea..29d4e38 100644
--- a/README.md
+++ b/README.md
@@ -15,8 +15,39 @@ The DevContainer will provide you with a working environment out of the box. **Y
 ```bash
 git clone https://github.com/misterw97/RenderDown
 sudo apt update && sudo apt install -y python3-mako python3-yaml
+pip install boto3 # if you want to run the generate_ubuntu_yaml file
 ```
 
+#### Generate_ubuntu_yaml
+
+This script allows to generate the ubuntu.yaml file in order to use it by the RenderDown script. It uses the template ubuntu.yaml located in the template folder.
+
+Here are the available arguments and examples of commands: 
+
+```
+usage: generate_ubuntu_yaml.py [-h] [--provider PROVIDER] [--username USERNAME] [--password PASSWORD] [--data-dir DATA_DIR] [--unpublished-suite UNPUBLISHED_SUITE]
+                               [--unpublished-tags UNPUBLISHED_TAGS] [--unpublished-archs UNPUBLISHED_ARCHS]
+
+Generate documentation about Ubuntu for ECR and DockerHub
+
+options:
+  -h, --help            show this help message and exit
+  --provider PROVIDER   aws or docker
+  --username USERNAME   Username of provider
+  --password PASSWORD   Password of provider
+  --data-dir DATA_DIR   Where you will find the output template (folder), If it does not exist then it is created
+  --unpublished-suite UNPUBLISHED_SUITE
+                        an Ubuntu Suite (e.g. jammy).
+  --unpublished-tags UNPUBLISHED_TAGS
+                        list of tags separated by comma (e.g. 'kinetic 22.10 22.10_edge',kinetic)
+  --unpublished-archs UNPUBLISHED_ARCHS
+                        list of archs separated by comma (e.g amd64,arm)
+```
+
+Example: `./generate_ubuntu_yaml.py --provider docker --username admin --password admin`
+
+If you give the --unpublished-suite argument you need to give --unpublished-tags and --unpublished-archs. When these options are given, the script will take your information into the YAML file without checking the registry.
+
 ## Running
 
 Create README files for all registries and namespaces:
diff --git a/generate_ubuntu_yaml.py b/generate_ubuntu_yaml.py
 new file mode 100755
index 0000000..b7ca361
--- /dev/null
+++ b/generate_ubuntu_yaml.py
@@ -0,0 +1,525 @@
+#!/usr/bin/env python3
+
+import argparse
+import datetime
+import json
+import logging
+import os
+import subprocess
+from typing import Dict, List
+
+import boto3
+import requests
+import sys
+import yaml
+
+logging.basicConfig(stream=sys.stdout, level=logging.INFO)
+NOW = datetime.datetime.now()
+SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
+
+
+def cli_args() -> argparse.ArgumentParser:
+    """Argument parser"""
+    parser = argparse.ArgumentParser(
+        description="Generate documentation about Ubuntu for ECR and DockerHub"
+    )
+
+    parser.add_argument(
+        "--provider",
+        default="docker",
+        dest="provider",
+        help="aws or docker",
+        required=True,
+    )
+    parser.add_argument(
+        "--username",
+        default="admin",
+        dest="username",
+        help="either the Docker Hub username, or the AWS access key ID",
+        required=True,
+    )
+    parser.add_argument(
+        "--password",
+        default="admin",
+        dest="password",
+        help="either the Docker Hub password/token, or the AWS secret access key",
+        required=True,
+    )
+    parser.add_argument(
+        "--token-docker",
+        dest="dockertoken",
+        default=None,
+        help="JWT token for Docker Hub authentication. \
+            Only useful for the 'docker' provider.",
+    )
+    parser.add_argument(
+        "--repository-basename",
+        dest="repository",
+        default=None,
+        help="repository basename of the ubuntu images. \
+            Used to infer existing information.",
+    )
+    parser.add_argument(
+        "--data-dir",
+        default="data",
+        dest="data_dir",
+        help="""The path of the folder
+                where the data file will be
+                saved ( if not exist, the script
+                will create the folder)""",
+    )
+    parser.add_argument(
+        "--unpublished-suite",
+        dest="unpublished_suite",
+        help="""an Ubuntu Suite (e.g. jammy).
+                if given we will take the
+                tags pass on command lines (required)
+                and the arches for this section
+                of the yaml file.
+            """,
+    )
+    parser.add_argument(
+        "--unpublished-tags",
+        dest="unpublished_tags",
+        help="""list of tags
+        (e.g. 'kinetic 22.10 22.10_edge kinetic)""",
+    )
+    parser.add_argument(
+        "--unpublished-archs",
+        dest="unpublished_archs",
+        help="list of archs (e.g amd64 arm)",
+    )
+
+    return parser
+
+
+def validate_args(
+    parser: argparse.ArgumentParser,
+) -> argparse.ArgumentParser.parse_args:
+    """Parse and validate the CLI arguments"""
+    args = parser.parse_args()
+    if any(
+        [
+            args.unpublished_suite is None,
+            args.unpublished_tags is None,
+            args.unpublished_archs is None,
+        ]
+    ) and not all(
+        [
+            args.unpublished_suite is None,
+            args.unpublished_tags is None,
+            args.unpublished_archs is None,
+        ]
+    ):
+        parser.error(
+            """--unpublished-suite need
+                        --unpublished-archs and --unpublished_tags"""
+        )
+
+    return args
+
+
+def build_image_endpoint(provider: str, repo_base: str = None) -> (str, str):
+    """Define the image's registry URL"""
+    if provider == "aws":
+        registry_url = "docker://public.ecr.aws/"
+        staging_repo = "rocksdev"
+    else:
+        registry_url = "docker://docker.io/"
+        staging_repo = "rocksdev4staging"
+
+    if repo_base is None:
+        logging.warning("Using staging repository")
+        url = f"{registry_url}{staging_repo}/ubuntu"
+        namespace = staging_repo
+    else:
+        url = f"{registry_url}{repo_base}/ubuntu"
+        namespace = repo_base
+
+    logging.info(f"Using {url} to collect information")
+
+    return url, namespace
+
+
+def add_yaml_representer():
+    def str_presenter(dumper, data):
+        """
+        Permit to format
+        multiline string into
+        yaml file
+        """
+
+        c = "tag:yaml.org,2002:str"
+        if len(data.splitlines()) > 1:  # check for multiline string
+            return dumper.represent_scalar(c, data, style="|")
+        return dumper.represent_scalar(c, data)
+
+    yaml.add_representer(str, str_presenter)
+    yaml.representer.SafeRepresenter.add_representer(str, str_presenter)
+
+
+def _process_run(command: List[str], **kwargs) -> str:
+    """Run a command and handle its output."""
+    logging.info(f"Execute process: {command!r}, kwargs={kwargs!r}")
+    try:
+        out = subprocess.run(
+            command,
+            **kwargs,
+            capture_output=True,
+            check=True,
+            universal_newlines=True,
+        )
+    except subprocess.CalledProcessError as err:
+        msg = f"Failed to run command: {err!s}"
+        if err.stderr:
+            msg += f" ({err.stderr.strip()!s})"
+        raise Exception(msg) from err
+
+    return out.stdout.strip()
+
+
+def get_arches(release: str, image_url: str) -> List[str]:
+    """
+    Permit to get the arches associated to the release
+    """
+    logging.info(f"Getting the arches for {release}")
+    command = ["skopeo", "inspect", f"{image_url}:{release}", "--raw"]
+    manifest = json.loads(_process_run(command))["manifests"]
+    arches = []
+    for arch in manifest:
+        arches.append(arch["platform"]["architecture"])
+    return arches
+
+
+def get_dockerhub_token(username: str, password: str) -> str:
+    """
+    Permit to get the token  associated to the docker account
+    """
+    logging.info("Getting the token form Docker")
+
+    url_token = "https://hub.docker.com/v2/users/login"
+    data = {"username": username, "password": password}
+    get_token = requests.post(url_token, json=data)
+    get_token.raise_for_status()
+    return get_token.json()["token"]
+
+
+def get_tags_docker(
+    release: str, token: str, image_url: str, image_namespace: str
+) -> List[str]:
+    """
+    Permit to get the tags associated to the release
+    """
+    logging.info(f"Getting the tags from Docker for {release}")
+    tags = []
+    command = [
+        "skopeo",
+        "inspect",
+        f"{image_url}:{release}",
+        "--raw",
+    ]
+    result_json = _process_run(command)
+    digest = json.loads(result_json)["manifests"][0]["digest"]
+
+    url_dockerhub = "https://hub.docker.com/v2/repositories/"
+    url_dockerhub += f"{image_namespace}/ubuntu/tags/?page_size=999"
+    Headers = {"Authorization": f"JWT {token}"}
+    get_the_tags = requests.get(url_dockerhub, headers=Headers)
+    get_the_tags = get_the_tags.json()["results"]
+    for image in get_the_tags:
+        for info_image in image["images"]:
+            if info_image["digest"] == digest and image["name"] not in tags:
+                tags.append(image["name"])
+
+    return tags
+
+
+def get_tags_aws(release: str, client: boto3.Session, image_url: str) -> List[str]:
+    """
+    Permit to get the tags associated to the release
+    """
+    logging.info(f"Getting the tags from AWS for {release}")
+
+    tags = []
+    command = [
+        "skopeo",
+        "inspect",
+        f"{image_url}:{release}",
+    ]
+    result_json = _process_run(command)
+    digest = json.loads(result_json)["Digest"]
+    response = client.describe_image_tags(repositoryName="ubuntu")
+
+    for image in response["imageTagDetails"]:
+        if (
+            image["imageDetail"]["imageDigest"] == digest
+            and image["imageTag"] not in tags
+        ):
+            tags.append(image["imageTag"])
+    return tags
+
+
+def get_fullname(release: str) -> str:
+    """
+    Permit to get the full name associated to the release
+    """
+    logging.info(f"Getting full name of {release} ")
+
+    command = ["ubuntu-distro-info", f"--series={release}", "-f"]
+    result_json = _process_run(command)
+    return result_json.replace("Ubuntu", "").strip()
+
+
+def get_support(series: str, is_lts: bool) -> Dict[str, Dict[str, str]]:
+    """Calculates the end of support dates for a given Ubuntu series"""
+    logging.info(f"Getting support information for the {series}")
+
+    base_cmd = ["ubuntu-distro-info", "--series", series]
+    eol_cmd = base_cmd + ["--day=eol"]
+
+    eol = int(_process_run(eol_cmd))
+    eol_date = NOW + datetime.timedelta(days=eol)
+
+    support = {"support": {"until": f"{eol_date.month:02d}/{eol_date.year}"}}
+
+    if not is_lts:
+        return support
+
+    # The it is LTS, and lts_until=until
+    support["support"]["lts_until"] = support["support"]["until"]
+
+    eol_esm_cmd = base_cmd + ["--day=eol-esm"]
+
+    eol_esm = int(_process_run(eol_esm_cmd))
+    eol_esm_date = NOW + datetime.timedelta(days=eol_esm)
+    eol_esm_value = f"{eol_esm_date.month:02d}/{eol_esm_date.year}"
+    support["support"]["esm_until"] = eol_esm_value
+
+    return support
+
+
+def get_deprecated(series: str) -> Dict[str, Dict[str, object]]:
+    """
+    Calculated the deprecation date
+    and upgrade path for a deprecated release
+    """
+    logging.info(f"Getting support information for the {series}")
+
+    eol_cmd = ["ubuntu-distro-info", "--series", series, "--day=eol"]
+
+    eol = int(_process_run(eol_cmd))
+    eol_date = NOW + datetime.timedelta(days=eol)
+    # For now, the upgrade path is always the next release
+
+    this_release_cmd = ["ubuntu-distro-info", "--series", series, "--day=release"]
+    this_release = int(_process_run(this_release_cmd))
+    # add 60 days to the release date, to get the next development version
+    next_date = NOW + datetime.timedelta(days=this_release + 60)
+
+    following_dev_series_cmd = [
+        "ubuntu-distro-info",
+        "-d",
+        f"--date={next_date.year}-{next_date.month}-{next_date.day}",
+    ]
+    development_suite_at_eol = _process_run(following_dev_series_cmd)
+
+    upgrade_path_cmd = [
+        "ubuntu-distro-info",
+        "--series",
+        development_suite_at_eol,
+        "-r",
+    ]
+    upgrade_path = _process_run(upgrade_path_cmd).strip(" LTS")
+
+    return {
+        "deprecated": {
+            "date": f"{eol_date.month:02d}/{eol_date.year}",
+            "path": {"track": upgrade_path},
+        }
+    }
+
+
+def is_deprecated(series: str) -> bool:
+
+    """Checks whether a series is completely deprecated (both LTS and ESM)"""
+    logging.info(f"Checking is {series} is deprecated")
+    supported_cmd = "ubuntu-distro-info --supported"
+    supported_esm_cmd = supported_cmd + "-esm"
+    all_supported = _process_run(supported_cmd.split(" ")) + _process_run(
+        supported_esm_cmd.split(" ")
+    )
+    return series not in all_supported
+
+
+def is_lts(series: str) -> bool:
+
+    """Checks if a given series is LTS"""
+    logging.info(f"Checking is {series} is lts")
+
+    cmd = ["ubuntu-distro-info", "--series", series, "-f"]
+
+    return "LTS" in _process_run(cmd)
+
+
+def get_lowest_risk(tags: List[str]) -> str:
+    """
+    Get the lowest risk associated with the release
+    """
+    risk_sorted = ["stable", "candidate", "beta", "edge"]
+
+    all_tags_str = " ".join(tags)
+    for risk in risk_sorted:
+        if risk in all_tags_str:
+            return risk
+
+    return "edge"
+
+
+def get_release(series: str) -> str:
+    command = ["ubuntu-distro-info", f"--series={series}", "-r"]
+
+    return _process_run(command)
+
+
+def infer_registry_user(
+    provider: str, username: str, password: str, dh_token: str = None
+) -> object:
+    user = None
+    if provider == "aws":
+        logging.info("Connecting to AWS")
+        session = boto3.Session(
+            region_name="us-east-1",
+            aws_access_key_id=username,
+            aws_secret_access_key=password,
+        )
+        user = session.client("ecr-public")
+    else:
+        logging.info("Fetching Docker Hub token")
+        if dh_token:
+            user = dh_token
+        else:
+            user = get_dockerhub_token(username, password)
+
+    return user
+
+
+def build_releases_data(
+    list_of_series: List[str],
+    all_tags: List[str],
+    image_url: str,
+    image_ns: str,
+    arguments: argparse.ArgumentParser.parse_args,
+    registry_user: object,
+) -> Dict:
+    """Build the releases info data structure"""
+    releases = []
+    for count, series in enumerate(list_of_series):
+        if series not in all_tags and series != arguments.unpublished_suite:
+            logging.warning(
+                f"Series {series} does not exist in {image_url}. Skipping it..."
+            )
+            continue
+
+        release_data = {}
+
+        release = get_release(series)
+        if "LTS" in release:
+            release_data["type"] = "LTS"
+
+        release_data["track"] = release.rstrip(" LTS")
+
+        if arguments.unpublished_suite and arguments.unpublished_suite == series:
+            release_data["architectures"] = arguments.unpublished_archs.split()
+            release_data["version"] = get_fullname(arguments.unpublished_suite)
+            release_data["risk"] = get_lowest_risk(arguments.unpublished_tags.split())
+            release_data["tags"] = arguments.unpublished_tags.split()
+        else:
+            release_data["architectures"] = get_arches(series, image_url)
+            release_data["version"] = get_fullname(series)
+            if arguments.provider == "docker":
+                release_data["tags"] = get_tags_docker(
+                    series, registry_user, image_url, image_ns
+                )
+            else:
+                release_data["tags"] = get_tags_aws(series, registry_user)
+            release_data["risk"] = get_lowest_risk(release_data["tags"])
+
+        if is_deprecated(series):
+            release_data["deprecated"] = get_deprecated(series)
+        else:
+            release_data["support"] = get_support(series, is_lts(series))
+
+        releases.append(release_data)
+
+    return releases
+
+
+def read_ubuntu_data_template() -> Dict:
+    """Reads and parses the YAML contents of the data template"""
+    template_file = f"{SCRIPT_DIR}/templates/ubuntu.yaml"
+    logging.info(f"Opening the template file {template_file}")
+    with open(template_file) as file:
+        try:
+            return yaml.safe_load(file)
+        except yaml.YAMLError as exc:
+            logging.error("Error when loading the ubuntu template file")
+            raise exc
+
+
+def create_data_dir(path: str):
+    """Create data dir if it doesn't exist"""
+    if not os.path.exists(path):
+        logging.info(f"Creating the {path} folder")
+
+        os.makedirs(path)
+
+
+def write_ubuntu_data_file(file_path: str, content: Dict):
+    """Write the YAML content into the ubuntu file path"""
+    with open(file_path, "w") as file:
+        logging.info(f"Create the yaml file {file_path}")
+        yaml.dump(content, file)
+
+
+def main():
+    arguments = validate_args(cli_args())
+    registry_user = infer_registry_user(
+        arguments.provider,
+        arguments.username,
+        arguments.password,
+        arguments.dockertoken,
+    )
+
+    add_yaml_representer()
+    url, ns = build_image_endpoint(arguments.provider, repo_base=arguments.repository)
+
+    logging.info(f"Getting all tags from {url}")
+    command_tags = ["skopeo", "list-tags", url]
+    existing_tags = json.loads(_process_run(command_tags))["Tags"]
+
+    logging.info("Getting all the series from ubuntu-distro-info")
+    command_suites = ["ubuntu-distro-info", "--all"]
+    series_names = _process_run(command_suites).split("\n")
+
+    if arguments.unpublished_suite and arguments.unpublished_suite not in series_names:
+        logging.error(
+            f"The provided unpublished suite {arguments.unpublished_suite}"
+            "is not recognized. Ignoring it"
+        )
+
+    logging.info("Building releases info")
+    releases = build_releases_data(
+        series_names, existing_tags, url, ns, arguments, registry_user
+    )
+
+    dict_file = read_ubuntu_data_template()
+    dict_file["releases"] = releases
+
+    create_data_dir(arguments.data_dir)
+
+    ubuntu_data_file = f"{arguments.data_dir}/ubuntu.yaml"
+    write_ubuntu_data_file(ubuntu_data_file, dict_file)
+
+
+if __name__ == "__main__":
+    main()