Merge ~bdrung/autopkgtest-cloud/+git/autopkgtest-package-configs:sort into ~ubuntu-release/autopkgtest-cloud/+git/autopkgtest-package-configs:main

[Benjamin Drung]

I don't like keeping an sorted list sorted when adding new entries. Therefore add a sort script to sort big_packages and long_tests and sort those two files.

[Steve Langasek (vorlon)]

Uses python3 natsort module which is not part of the python3 package and I see no declaration of this added dependency anywhere useful.

Seems rather complicated for something that could be done just as well for our purposes with the 'sort' command in the coreutils package. (I don't think the use of natsort really matters here, the point is that they *be* sorted for ease of maintenance, not that they use any particular collation order.)

review: Disapprove

[Benjamin Drung (bdrung)]

I tried with the sort command, but there is no option for naturally sorting. Would a Python script acceptable that does not need the natsort module?

[Steve Langasek (vorlon)]

On Mon, Aug 01, 2022 at 07:39:22AM -0000, Benjamin Drung wrote:

> I tried with the sort command, but there is no option for naturally
> sorting. Would a Python script acceptable that does not need the natsort
> module?

Why do you consider natural sorting important here? As I said I don't think
the sort method is important.

[Benjamin Drung (bdrung)]

1. The files are currently kindish sorted naturally.

2. I prefer to have openjdk-8 sorted before openjdk-13 and python3.10 after python3.9. This is less confusing when adding new version or dropping old ones.

3. "./sort" is shorter than "LANG=C sort big_packages | sponge big_packages" (okay, that could be put in a short shell script).

[Skia (hyask)]

This currently doesn't work due to comments in `big_packages` (yes, they are supported [1]).
Steve's comments about the `python3-natsort` dependency still stands.

I'll add:
- ideally, introducing any kind `sort` script should come with enforcing it in the CI
- any decent text editor has a `sort` function too

[1]: https://git.launchpad.net/autopkgtest-cloud/tree/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker/worker#n275

review: Disapprove

Unmerged commits

142b3ad... by Benjamin Drung on 2022-07-27

Add sort script

Add sort script to sort big_packages and long_tests.

Signed-off-by: Benjamin Drung <email address hidden>

79e293b... by Benjamin Drung on 2022-07-27

Sort big_packages and long_tests

Signed-off-by: Benjamin Drung <email address hidden>

diff --git a/README.md b/README.md
index 7b38276..4d46aaf 100644
--- a/README.md
+++ b/README.md
@@ -20,11 +20,15 @@ Entries here run on the `package_size_big` Openstack flavour as opposed to the
 which run on Openstack (not `lxd`). Entries here also get a longer
 `--timeout-test`.
 
+The entries in `big_packages` are kept sorted with the `sort` script.
+
 ## long_tests
 
 Entries here get a longer timeout, currently defined in the code of
 `autopkgtest-cloud`.
 
+The entries in `long_tests` are kept sorted with the `sort` script.
+
 ## never_run
 
 Entries here are never run. A worker which encounters one of these will
diff --git a/big_packages b/big_packages
index 6e1e32c..a2dc162 100644
--- a/big_packages
+++ b/big_packages
@@ -11,9 +11,9 @@ androguard/armhf
 androguard/i386
 androguard/ppc64el
 androguard/s390x
+asmjit/arm64
 assimp/amd64
 assimp/i386
-asmjit/arm64
 asymptote/amd64
 asymptote/arm64
 asymptote/ppc64el
@@ -66,9 +66,9 @@ cl-ironclad/amd64
 cl-ironclad/arm64
 cl-ironclad/i386
 cod-tools/ppc64el
+cross-toolchain-base/ppc64el
 cross-toolchain-base-mipsen/amd64
 cross-toolchain-base-mipsen/i386
-cross-toolchain-base/ppc64el
 cwltool/arm64
 cwltool/ppc64el
 dask/amd64
@@ -171,18 +171,18 @@ link-grammar/arm64
 link-grammar/ppc64el
 link-grammar/s390x
 lintian/arm64
-linux-*/amd64
-linux-*/arm64
-linux-*/armhf
-linux-*/i386
-linux-*/ppc64el
-linux-*/s390x
 linux/amd64
 linux/arm64
 linux/armhf
 linux/i386
 linux/ppc64el
 linux/s390x
+linux-*/amd64
+linux-*/arm64
+linux-*/armhf
+linux-*/i386
+linux-*/ppc64el
+linux-*/s390x
 lttng-modules/amd64
 lttng-modules/arm64
 lttng-modules/armhf
@@ -245,6 +245,12 @@ octave-parallel/i386
 octave-parallel/ppc64el
 octave-parallel/s390x
 onetbb/arm64
+openjdk-8/amd64
+openjdk-8/arm64
+openjdk-8/armhf
+openjdk-8/i386
+openjdk-8/ppc64el
+openjdk-8/s390x
 openjdk-13/amd64
 openjdk-13/i386
 openjdk-13/ppc64el
@@ -253,12 +259,6 @@ openjdk-14/amd64
 openjdk-14/i386
 openjdk-14/ppc64el
 openjdk-14/s390x
-openjdk-8/amd64
-openjdk-8/arm64
-openjdk-8/armhf
-openjdk-8/i386
-openjdk-8/ppc64el
-openjdk-8/s390x
 openjdk-lts/amd64
 openjdk-lts/arm64
 openjdk-lts/armhf
@@ -292,6 +292,16 @@ pyresample/ppc64el
 pyresample/s390x
 pyside2/amd64
 pyside2/arm64
+python2.7/amd64
+python2.7/i386
+python3.6/amd64
+python3.6/i386
+python3.7/amd64
+python3.7/i386
+python3.8/amd64
+python3.8/i386
+python3.9/ppc64el
+python3.10/ppc64el
 python-cffi/amd64
 python-cffi/arm64
 python-cffi/armhf
@@ -326,16 +336,6 @@ python-ruffus/ppc64el
 python-ruffus/s390x
 python-scipy/amd64
 python-scipy/i386
-python2.7/amd64
-python2.7/i386
-python3.6/amd64
-python3.6/i386
-python3.7/amd64
-python3.7/i386
-python3.8/amd64
-python3.8/i386
-python3.9/ppc64el
-python3.10/ppc64el
 q2-feature-classifier/amd64
 q2-feature-classifier/arm64
 q2-feature-classifier/i386
diff --git a/sort b/sort
new file mode 100755
index 0000000..f494852
--- /dev/null
+++ b/sort
@@ -0,0 +1,48 @@
+#!/usr/bin/python3
+
+"""Sort files naturally."""
+
+import argparse
+import logging
+import os
+import sys
+
+import natsort
+
+LOG_FORMAT = "%(name)s %(levelname)s: %(message)s"
+
+
+def sort(filename: str, encoding: str = "utf-8"):
+    """Sort given file naturally."""
+    logger = logging.getLogger(os.path.basename(sys.argv[0]))
+    with open(filename, encoding=encoding) as textfile:
+        content = textfile.read()
+    sorted_content = "".join(
+        "/".join(x)
+        for x in natsort.natsorted(
+            [x.split("/") for x in content.splitlines(keepends=True)]
+        )
+    )
+    if sorted_content == content:
+        logger.info("`%s' is already sorted.", filename)
+        return
+    logger.info("Writing sorted `%s'...", filename)
+    with open(filename, "w", encoding=encoding) as textfile:
+        textfile.write(sorted_content)
+
+
+def main():
+    """Sort files naturally."""
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        "file", nargs="*", default=["big_packages", "long_tests"]
+    )
+    args = parser.parse_args()
+    logging.basicConfig(format=LOG_FORMAT, level=logging.INFO)
+
+    for filename in args.file:
+        sort(filename)
+
+
+if __name__ == "__main__":
+    main()