Merge ~alexmurray/ubuntu-security-tools:umt-grep-source-map into ubuntu-security-tools:master

[Alex Murray]

This allows to find packages listed in the various subprojects even if they don't exist in the Ubuntu archive.

Should be useful for https://docs.google.com/document/d/13wpBtK2uySdBeZXbpLIp-oO3C5kYkXw-8bL8KEjOd1c/edit#

[Alex Murray (alexmurray)]

Any chance someone can review this?

[Eduardo Barretto (ebarretto)]

Overall it looks good to me, but I'm just wondering if we need to do source_map.load() without skipping eol releases. Here's an example

In current UST master:
$ umt grep ffmpeg
ffmpeg
ffmpeg-php
ffmpeg2theora
ffmpegfs
ffmpegthumbnailer
ffmpegthumbs
gstreamer0.10-ffmpeg
kffmpegthumbnailer
kodi-inputstream-ffmpegdirect
ocaml-ffmpeg

with your changes applied:
$ umt grep ffmpeg
ffmpeg
ffmpeg2theora
ffmpegfs
ffmpegthumbnailer
ffmpegthumbs
kffmpegthumbnailer
kodi-inputstream-ffmpegdirect
ocaml-ffmpeg

You can see that ffmpeg-php and gstreamer0.10-ffmpeg.
I did a grep on ffmpeg-php, this is a precise source, so not sure it matters at all.
I did the same for gstreamer0.10-ffmpeg, this is a precise and trusty source, since trusty is still in ESM, should we list trusty source on umt grep?

If yes, then doing a source_map.load(skip_eol_releases=False) might be enough.

571be21... by Alex Murray on 2022-02-25

umt grep: Avoid temporary file and pipe stdin to grep

Signed-off-by: Alex Murray <email address hidden>

Unmerged commits

571be21... by Alex Murray on 2022-02-25

umt grep: Avoid temporary file and pipe stdin to grep

Signed-off-by: Alex Murray <email address hidden>

851ed12... by Alex Murray on 2022-01-31

umt grep: Search package names from source_map by default

This then allows umt grep to find packages in any other configured
subprojects rather than just the Ubuntu archive (eg. this finds kafka in
esm-apps whereas the current umt grep does not).

Signed-off-by: Alex Murray <email address hidden>

diff --git a/build-tools/umt b/build-tools/umt
index ee74cbe..44baa1f 100755
--- a/build-tools/umt
+++ b/build-tools/umt
@@ -2023,7 +2023,7 @@ def cmd_grep():
 
     config = source_map.cve_lib.read_config()
 
-    mirror_path = config["packages_mirror"]
+    mirror_path = None
     if opt.repo == "debian":
         mirror_path = config["debian_mirror"]
     elif opt.repo == "partner":
@@ -2033,8 +2033,20 @@ def cmd_grep():
         parser.print_help()
         sys.exit(1)
 
-    (rc, report) = runcmd(["grep", args[0], os.path.join(mirror_path, "sources")])
-    print(report.rstrip())
+    if mirror_path is not None:
+        _, report = runcmd(["grep", args[0], os.path.join(mirror_path, "sources")])
+        print(report.rstrip())
+    else:
+        # search the configured releases via source_map
+        src_map = source_map.load()
+        srcs = set()
+        for rel in src_map:
+            srcs = srcs | set(src_map[rel].keys())
+        # use grep to do the actual search so we keep the same semantics
+        # when interpreting the input regex etc
+        input = "\n".join(sorted(list(srcs))).encode()
+        _, report = runcmd(["grep", args[0], '-'], input=input, stdin=subprocess.PIPE)
+        print(report.rstrip())
 
 def cmd_search():
     '''Search for best source packages for each release'''