Merge ~rodrigo-zaiden/ubuntu-security-tools:fix-build-source-list into ubuntu-security-tools:master

[Rodrigo Figueiredo Zaiden]

UST: build-source-list: change build ESM PPAs

 From the ubuntu-security-tools.conf file, there are different ways
 to add an ESM release, if an user has a release in the format
 'release/esm-x' it was not building correctly.
 If the format 'esm-x/release' is used, it is not being handled.
 The exception case is for Precise and Trusty that should be created
 in the format 'release/esm'

I've seen some different ways to describe an ESM release in
~/.ubuntu-security-tools.conf, such as:
`esm-apps/focal` or `focal/esm-apps`

I'm using the first and I noticed that it was not building the PPAs
correctly, since this case was not handled.
I couldn't find a "correct" format, and as far as I checked, both seems
to be accepted.
In this commit, I'm trying to accept both and handle both.
I understand that in the ideal world, I could be generating the auth part
in the correct file, but for now the warning note remains, I'm just trying
to make this script usable in case someone tries it.

The only special case is for trusty and precise that should be handled
as release/esm since they only have one component. Actually, it could be
time to remove Precise, but I decided to first push with it as it was
still there and if someone thinks that we should remove it, I can create
a new commit.

Thank you!

[Eduardo Barretto (ebarretto)]

we should not support the format 'release/esm-{infra/apps}', please follow the new CVE file format which is:
PRODUCT/RELEASE

so it should be:
esm/precise [1]
esm/trusty [1]
esm-infra/xenial
esm-apps/xenial
esm-apps/bionic
esm-apps/focal
esm-apps/jammy

[1] Please note that precise and trusty we do have alias setup to accept trusty/esm and precise/esm and this was a decision so we don't have to touch all our CVEs and infrastructure as their ESM came before the new CVE file format.

review: Needs Fixing

[Rodrigo Figueiredo Zaiden (rodrigo-zaiden)]

> we should not support the format 'release/esm-{infra/apps}', please follow the
> new CVE file format which is:
> PRODUCT/RELEASE
>
> so it should be:
> esm/precise [1]
> esm/trusty [1]
> esm-infra/xenial
> esm-apps/xenial
> esm-apps/bionic
> esm-apps/focal
> esm-apps/jammy
>
> [1] Please note that precise and trusty we do have alias setup to accept
> trusty/esm and precise/esm and this was a decision so we don't have to touch
> all our CVEs and infrastructure as their ESM came before the new CVE file
> format.

Thanks Eduardo!

I've made a new commit where I support the format release/esm{-infra/-apps}
In the end, it is a cleaner code.
If you could give a new look at it, I appreciate.
Thanks!

[Steve Beattie (sbeattie)]

On Wed, Apr 27, 2022 at 06:25:04AM -0000, Eduardo Barretto wrote:
> we should not support the format 'release/esm-{infra/apps}', please follow the new CVE file format which is:
> PRODUCT/RELEASE
>
> so it should be:
> esm/precise [1]
> esm/trusty [1]
> esm-infra/xenial
> esm-apps/xenial
> esm-apps/bionic
> esm-apps/focal
> esm-apps/jammy

Cool, I missed when this was communicated. With the change that
Rodrigo is proposing, `esm/trusty` has to be the format used in
~/.ubuntu-security-tools.conf in order for a sources.list to be
generated that references the trusty ppa for ESM, having `trusty/esm`
results in it being skipped.

> [1] Please note that precise and trusty we do have alias setup to
> accept trusty/esm and precise/esm and this was a decision so we don't
> have to touch all our CVEs and infrastructure as their ESM came before
> the new CVE file format.

The esm/trusty alias does not work correctly for umt download:

$ umt search tzdata | grep trusty
trusty/esm: 2022a-0ubuntu0.14.04+esm1, Pocket: release, Component: main
trusty: 2019a-0ubuntu0.14.04, Pocket: updates, Component: main
$ umt download tzdata -r esm/trusty
Skipping release 'esm/trusty': package not found.
$ umt download tzdata -r trusty/esm
Downloading 'tzdata' version '2022a-0ubuntu0.14.04+esm1' for release 'trusty/esm'.

(This happens regardless of whether ~/.ubuntu-security-tools.conf
contains esm/trusty or trusty/esm in `release_list`.)

--
Steve Beattie
<email address hidden>

[Steve Beattie (sbeattie)]

Hey Rodrigo,

On Wed, Apr 27, 2022 at 01:29:44PM -0000, Rodrigo Figueiredo Zaiden wrote:
> > we should not support the format 'release/esm-{infra/apps}', please follow the
> > new CVE file format which is:
> > PRODUCT/RELEASE
> >
> > so it should be:
> > esm/precise [1]
> > esm/trusty [1]
> > esm-infra/xenial
> > esm-apps/xenial
> > esm-apps/bionic
> > esm-apps/focal
> > esm-apps/jammy
> >
> > [1] Please note that precise and trusty we do have alias setup to accept
> > trusty/esm and precise/esm and this was a decision so we don't have to touch
> > all our CVEs and infrastructure as their ESM came before the new CVE file
> > format.
>
> I've made a new commit where I support the format release/esm{-infra/-apps}
> In the end, it is a cleaner code.
> If you could give a new look at it, I appreciate.

I think there's a bit of confusion about what Eduardo was trying to say.

For xenial and newer, only the formats `esm-{infra,apps}/RELEASE`
should be accepted. For trusty `esm/trusty` and `trusty/esm` both
need to work (in particular the latter format); in this case either
should generate the private ppa entry for trusty. With the revised
change it's only generating a trusty ppa line if `esm/trusty` is used.

That said, feel free to drop references to precise. It is gone and not
coming back.

Thanks!

--
Steve Beattie
<email address hidden>

[Rodrigo Figueiredo Zaiden (rodrigo-zaiden)]

Thanks Steve and Eduardo for discussing the idea.

Just pushed a new review adding support to trusty/esm and removing precise.
Decided to go in a "hardcoded" style for trysty/esm for a better readability instead of trying to play with regex for all cases: trusty/esm and esm[-infra|-apps]/release.
esm/trusty is still supported although it can break stuff around `umt download`.

Thanks for revisiting it whenever possible.

Unmerged commits

64d9697... by Rodrigo Figueiredo Zaiden on 2022-04-28

UST: build-source-list: build ESM PPAs

    Handle trusty/esm and esm{-infra/-apps}/release,
    including esm/trusty and remove precise esm.

49f76a5... by Rodrigo Figueiredo Zaiden on 2022-04-27

UST: build-source-list: change build ESM PPAs

  As per the new CVE format, the accepted is esm{-infra/-apps/release}
  So we can clean a chunck for code and handle only esm{}/x cases.

f9597dd... by Rodrigo Figueiredo Zaiden on 2022-04-26

UST: build-source-list: change build ESM PPAs

 From the ubuntu-security-tools.conf file, there are different ways
 to add an ESM release, if an user has a release in the format
 'release/esm-x' it was not building correctly.
 If the format 'esm-x/release' is used, it is not being handled.
 The exception case is for Precise and Trusty that should be created
 in the format 'release/esm'

diff --git a/build-tools/build-sources-list b/build-tools/build-sources-list
index bb91469..baa0711 100755
--- a/build-tools/build-sources-list
+++ b/build-tools/build-sources-list
@@ -114,23 +114,25 @@ EOM
         ppa_overlay="image"
         emit_ppa_overlay "snappy-dev" "$ppa_overlay" "${ppa_overlay_release}"
 
-    # ESM overlay ppas
-    elif echo "$r" | grep -q "/esm" ; then
-        ppa_overlay_release=$(echo "$r" | cut -f 1 -d '/')
-        if [ "$ppa_overlay_release" = "precise" ] ; then
-            ppa_overlay="esm"
-            emit_private_ppa_overlay "ubuntu-esm" "$ppa_overlay" "${ppa_overlay_release}"
-	    echo "# Requires gpg keyid DBB1FC89762BF6B96707C4059BC0A1A1622CF918"
-        else
-            # newer ESM releases are in the esm-infra-security and
-            # esm-apps-security ppas
+    # Trusty ESM overlay ppa
+    elif echo "$r" | grep -q "trusty/esm" ; then
+        emit_private_ppa_overlay "ubuntu-esm" "esm-infra-security" "trusty"
+        echo "# Requires gpg keyid DBB1FC89762BF6B96707C4059BC0A1A1622CF918"
+
+    # ESM Infra/Apps overlay ppas
+    # in the format "esm{-infra|-apps}/<release>"
+    elif echo "$r" | grep -qe "esm.*/" ; then
+        ppa_overlay_release=$(echo "$r" | cut -f 2 -d '/')
+        if [ "$ppa_overlay_release" = "trusty" ] ; then
+            # Trusty ESM release is in ubuntu-esm/esm-infra-security
             ppa_overlay="esm-infra-security"
-            emit_private_ppa_overlay "ubuntu-esm" "$ppa_overlay" "${ppa_overlay_release}"
-	    echo "# Requires gpg keyid DBB1FC89762BF6B96707C4059BC0A1A1622CF918"
-            ppa_overlay="esm-apps-security"
-            emit_private_ppa_overlay "ubuntu-esm" "$ppa_overlay" "${ppa_overlay_release}"
-	    echo "# Requires gpg keyid DBB1FC89762BF6B96707C4059BC0A1A1622CF918"
+        else
+            ppa_esm_component=$(echo "$r" | cut -f 1 -d '/')
+            ppa_overlay="$ppa_esm_component-security"
         fi
+        emit_private_ppa_overlay "ubuntu-esm" "$ppa_overlay" "${ppa_overlay_release}"
+        echo "# Requires gpg keyid DBB1FC89762BF6B96707C4059BC0A1A1622CF918"
+
     # normal ubuntu releases.
     else
         if [ "$r" = "precise" ] ; then