Merge ~philroche/maas-images:feature/esm-image-sync-support into maas-images:master

[Philip Roche]

Add support for syncing images that are no longer receiving regular support

.. but are receiving ESM support. I found this issue with when trying to sync a
final Trusty release after it entered ESM support.

Add support for syncing images that are no longer receiving regular support

.. but are receiving ESM support. I found this issue with when trying to sync a
final Trusty release after it entered ESM support.

[David Krauser (davidkrauser)]

review: Approve

[Robert C Jennings (rcj)]

Looks good to me Phil. Thanks for ensuring we can deliver these during Trusty ESM as necessary.

review: Approve

[Andres Rodriguez (andreserl)]

Can you expand what do you mean by "when trying to sync a final Trusty release" ? Where are you syncing images and how? Because we have not dropped trusty from the available images for MAAS this code produces.

IS there a bug report where to understand the context?

review: Needs Information

[Philip Roche (philroche)]

@andreserl The issue is that no further Trusty images would be added to streams or synced to http://images.maas.io/ephemeral-v3/daily/trusty/

See

https://cloud-images-jenkins.canonical.com/job/MAAS_v3_MasterBuilder/ARCH_TYPE=amd64/258/console

"ignoring unsupported release: trusty",

I have not filed a bug report. I work on the Canonical CPC team so I'm not a consumer of these images. I help in producing them.

[Andres Rodriguez (andreserl)]

Thanks for the context. I, however, have one more question.

As far as I understand, we are not continuing to produce new Trusty images under ESM, as there are no new images that include ESM related packages in them. As such, why would we want to allow syncing new images of ESM releases (trusty/precise in this case) if we are no longer producing new images and these images are already available in the MAAS streams?

[Andres Rodriguez (andreserl)]

Also, comment inline.

review: Needs Information

[Andres Rodriguez (andreserl)]

Ok, so now I see what the issue is here. The last trusty image built for MAAS was on April 11, 2019 [1]. CPC has produced a new image on April 29, 2019 (published May 1st) [2]. The job wants to build and produce a new MAAS trusty images based on this last update.

On that basis, I'm +1 on this branch. Just the comment inline should be addressed if possible.

[1]: https://images.maas.io/ephemeral-v3/daily/trusty/amd64/
[2]: http://cloud-images.ubuntu.com/trusty/20190429/ # published may 1st.

review: Approve

4287d3f... by Philip Roche on 2019-05-02

Add support_esm_eol key with details on when a releases ESM support will end

If a release does not receive ESM support (all non-LTS releases) then the esm_eol
is the same as the regular support eol.

[Philip Roche (philroche)]

@andreserl I have made the requested change but would like another review from you as I wasn't sure if when a release is not an ESM release (all non-LTS releases) the esm_eol should be "" or the eol date.

[Lee Trager (ltrager)]

I think this is fine for allowing new ESM images to be built in MAAS. Do you think this should be including the support_esm_eol field in the stream? I'm fine either way but currently that value is not propagated to the stream itself [1].

[1] http://paste.ubuntu.com/p/X8DM7xnYm6/

review: Approve

[Lee Trager (ltrager)]

I spoke with Andres on IRC. We agreed to land this as is and I've added another commit to include support_esm_eol in the stream.

[Philip Roche (philroche)]

@ltrager Thank you. We have open MPs internally in CPC about adding support_esm_eol to non MAAS streams but we have held off on merging until a final decision on how this should look across all streams is made. I believe this is to be discussed at the next management sprint in May.

diff --git a/meph2/commands/cloudimg_sync.py b/meph2/commands/cloudimg_sync.py
index e05a060..c7aec39 100755
--- a/meph2/commands/cloudimg_sync.py
+++ b/meph2/commands/cloudimg_sync.py
@@ -81,7 +81,8 @@ class CloudImg2Meph2Sync(mirrors.BasicMirrorWriter):
 
         self.releases = []
         for r in [k['release'] for k in cfgdata['releases']]:
-            if r not in ubuntu_info.SUPPORTED:
+            if r not in ubuntu_info.SUPPORTED and \
+                    r not in ubuntu_info.SUPPORTED_ESM:
                 LOG.info("ignoring unsupported release: %s", r)
             else:
                 self.releases.append(r)
diff --git a/meph2/ubuntu_info.py b/meph2/ubuntu_info.py
index d7ab49c..032ee5f 100644
--- a/meph2/ubuntu_info.py
+++ b/meph2/ubuntu_info.py
@@ -60,6 +60,7 @@ def get_ubuntu_info(date=None):
                 getall(result="release", date=date)]
     full_codenames = [x.split('"')[1] for x in fullnames]
     supported = udi.supported(date=date)
+    supported_esm = udi.supported_esm(date=date)
     try:
         devel = udi.devel(date=date)
     except distro_info.DistroDataOutdated as e:
@@ -82,11 +83,20 @@ def get_ubuntu_info(date=None):
     for i, codename in enumerate(codenames):
         title = "%s LTS" % versions[i] if lts[i] else versions[i]
         eol = hack_all[codename]['eol'].strftime("%Y-%m-%d")
+
+        if hack_all[codename]['eol_esm']:
+            eol_esm = hack_all[codename]['eol_esm'].strftime("%Y-%m-%d")
+        else:
+            # If eol_esm is None then this release does not receive ESM support
+            # As such we should set the esm_eol to the same as eol
+            eol_esm = eol
         release_date = hack_all[codename]['release'].strftime("%Y-%m-%d")
         ret.append({'lts': lts[i], 'version': versions[i],
                     'supported': codename in supported,
+                    'supported_esm': codename in supported_esm,
                     'codename': codename,
                     'support_eol': eol,
+                    'support_esm_eol': eol_esm,
                     'release_codename': full_codenames[i],
                     'release_date': release_date,
                     'devel': bool(codename == devel),
@@ -108,6 +118,7 @@ REL2VER = {k['codename']: k for k in get_ubuntu_info()}
 
 LTS_RELEASES = [d for d in REL2VER if REL2VER[d]['lts']]
 SUPPORTED = {d: v for d, v in REL2VER.items() if v['supported']}
+SUPPORTED_ESM = {d: v for d, v in REL2VER.items() if v['supported_esm']}
 
 if __name__ == '__main__':
     import json