Merge ~hyask/autopkgtest-cloud:skia/fix_push_amqp into autopkgtest-cloud:master

[Skia]

Fix pull/push-amqp workflow.

[Tim Andersson (andersson123)]

LGTM, looks like there's a typo in the commit msg (red->read) but other than that this looks great to me :)

review: Approve

[Paride Legovini (paride)]

Hmm, I think we should figure out a way to keep the escapes literal on the pull-amqp side, and avoid printing the b'...'. Dealing with the literal b'...' doesn't feel right to me.

[Skia (hyask)]

Yes, I agree, but as I explained in the commit log, I don't see a simple way forward to do this right now. Would you prefer that I give a try to the `\0` solution? That's way less common in a shell pipeline, although not completely non-existing, with `find -print0`.

[Skia (hyask)]

I've found a way to print the strings with escaped endline **and** without converting to bytes: `repr()`. We could indeed use that in `pull-amqp` to avoid printing bytes, but then I couldn't solve the problem of interpreting the `\n` as a real single character endline. We could in the end hack that with a `.replace("\\n", "\n")` (or `.replace(r"\n", "\n")`, it's the same), but I find it even more hacky since we don't really cover everything, and I think passing the information with bytes is more robust.

As we discussed, we need to build a plan to change the message format to be easier to work with.

Also discussed, I've added a log and a skip condition if the heuristic for bytes fails, meaning the message is probably malformed.

[Paride Legovini (paride)]

Thanks for trying the alternative approaches. This is hacky but good enough for now, as mentioned already the right thing to fix is the weird format of the queue entries.

review: Approve

diff --git a/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/push-amqp b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/push-amqp
index ecc1355..dff98a7 100755
--- a/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/push-amqp
+++ b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/tools/push-amqp
@@ -1,6 +1,7 @@
 #!/usr/bin/python3
 
 import argparse
+import ast
 import configparser
 import sys
 
@@ -67,6 +68,22 @@ def main():
     else:
         while True:
             message = sys.stdin.readline()
+            if message.startswith("b'") and message.endswith("'"):
+                # this is most likely bytes that we need to interpret as a string
+                # `literal_eval` is far safer than a true `eval`: the warnings
+                # around it in the documentation mostly mention denial of
+                # service, and possible huge memory consumption with evil
+                # content, but that function should be safe from a code
+                # execution point of view.
+                # DoS is no big deal in a CLI tool
+                message = ast.literal_eval(message).decode("utf-8")
+            else:
+                print(
+                    "WARNING: Message was not detected as bytes, it may be malformed. "
+                    f"Skipping. (msg: `{message}`)",
+                    file=sys.stderr,
+                )
+                continue
             if not message:
                 break
             try: