Merge ~litios/ubuntu-cve-tracker:feature/data-lib-rel-storage into ubuntu-cve-tracker:master

[David Fernandez Gonzalez]

Add ReleaseStorage class to datalib.

This class will handle the Release objects, avoiding duplication and handling the links among themselves.
Furthermore, it will provide convenient features for retrieving specific sets of releases.

As an extra, package cache loading is not sequential by default, due to the high memory consumption when performed in parallel.

[David Fernandez Gonzalez (litios)]

Testing script:

from datalib import *
release_storage = ReleaseStorage()

trusty = release_storage.get_release('trusty')
print('Trusty release loaded as "trusty"', trusty)
trusty = release_storage.get_release('ubuntu/trusty')
print('Trusty release loaded as "ubuntu/trusty"',trusty)
print(' * ESM Apps', trusty.esm_apps_release)
print(' * ESM Infra', trusty.esm_infra_release)
print(' * ESM Infra parent', trusty.esm_infra_release.parent)

bionic = release_storage.get_release('bionic')
print('Bionic release loaded as "bionic"',bionic)
bionic = release_storage.get_release('ubuntu/bionic')
print('Bionic release loaded as "ubuntu/bionic"',bionic)
print(' * ESM Apps', bionic.esm_apps_release)
print(' * ESM Infra', bionic.esm_infra_release)
print(' * ESM Infra parent', bionic.esm_infra_release.parent)
print(' * ESM Apps parent', bionic.esm_apps_release.parent)

jammy = release_storage.get_release('jammy')
print('Jammy release loaded as "jammy"',jammy)
jammy = release_storage.get_release('ubuntu/jammy')
print('Jammy release loaded as "ubuntu/jammy"',jammy)
print(' * ESM Apps', jammy.esm_apps_release)
print(' * ESM Infra', jammy.esm_infra_release)
print(' * ESM Apps parent', jammy.esm_apps_release.parent)

print('All supported releases', release_storage.supported_releases)
print('ESM Apps releases', release_storage.esm_apps_releases)
print('ESM Infra releases', release_storage.esm_infra_releases)

print('Ubuntu product releases', release_storage.get_product_releases('ubuntu'))
print('Ubuntu product releases -- supported only', release_storage.get_product_releases('ubuntu', supported_only=True))

print('Devel release:', release_storage.get_devel_release())

import time

now = time.time()

cve_storage = UCTCVEStorage()
cve_storage.link_release_storage(release_storage)
cve_storage.load()

package_storage = UCTPackageStorage()
package_storage.link_release_storage(release_storage)
package_storage.load(filter_releases=release_storage.supported_releases)

usn_storage = USNStorage()
usn_storage.link_release_storage(release_storage)
usn_storage.load()

cve_storage.link_pkg_storage(package_storage)
usn_storage.link_pkg_storage(package_storage)
usn_storage.link_cve_storage(cve_storage)

later = time.time()
print('Total loading time: ', later - now)

# gpac = package_storage.get_package('python-idna')
# for version in gpac.source_versions:
# print(gpac, gpac.source_info[version]['release'], version)
# for binary in gpac.binaries[version]:
# print(f' - {binary.name} {binary.version} {binary.arches}')

# print(list(usn_storage.sns.keys()))

print()
print('------------------')
print('Showing CVE-2023-1625')
print('------------------')
cve = cve_storage.get_cve('CVE-2023-1625')
print(cve.id, '\n')
print('Description:', cve.description, '\n')
print('References:', cve.references, '\n')
print('Notes:', cve.notes,'\n')
for entry in cve.pkg_entries:
    print(' -', entry.cve, entry.pkg, entry.release, entry.status, entry.note)

print()
print('------------------')
print('Loading jammy into the storage')
package_storage.load_release('jammy')
cve = cve_storage.get_cve('CVE-2024-3651')
print('------------------')
print('Showing CVE entries now')
for entry in ...

[David Fernandez Gonzalez (litios)]

Output without subprojects:

Trusty release loaded as "trusty" ubuntu/trusty
Trusty release loaded as "ubuntu/trusty" ubuntu/trusty
 * ESM Apps None
 * ESM Infra esm/trusty
 * ESM Infra parent ubuntu/trusty
Bionic release loaded as "bionic" ubuntu/bionic
Bionic release loaded as "ubuntu/bionic" ubuntu/bionic
 * ESM Apps esm-apps/bionic
 * ESM Infra esm-infra/bionic
 * ESM Infra parent ubuntu/bionic
 * ESM Apps parent esm-infra/bionic
Jammy release loaded as "jammy" ubuntu/jammy
Jammy release loaded as "ubuntu/jammy" ubuntu/jammy
 * ESM Apps esm-apps/jammy
 * ESM Infra None
 * ESM Apps parent ubuntu/jammy
All supported releases [esm-apps/jammy, fips-updates/focal, esm/trusty, esm-apps/focal, ros-esm/melodic, fips/xenial, esm-apps/bionic, fips/bionic, ubuntu/jammy, esm-apps/xenial, esm-infra/xenial, ubuntu/mantic, esm-infra/bionic, fips-updates/bionic, fips-updates/xenial, ubuntu/noble, ros-esm/kinetic, fips/focal, esm-apps/noble, snap, ubuntu/focal]
ESM Apps releases [esm-apps/jammy, esm-apps/noble, esm-apps/focal, esm-apps/bionic, esm-apps/xenial]
ESM Infra releases [esm-infra/bionic, esm/trusty, esm-infra/xenial]
Ubuntu product releases [ubuntu/warty, ubuntu/hoary, ubuntu/breezy, ubuntu/dapper, ubuntu/edgy, ubuntu/feisty, ubuntu/gutsy, ubuntu/hardy, ubuntu/intrepid, ubuntu/jaunty, ubuntu/karmic, ubuntu/lucid, ubuntu/maverick, ubuntu/natty, ubuntu/oneiric, ubuntu/precise, ubuntu/quantal, ubuntu/raring, ubuntu/saucy, ubuntu/trusty, ubuntu/utopic, ubuntu/vivid, ubuntu/wily, ubuntu/xenial, ubuntu/yakkety, ubuntu/zesty, ubuntu/artful, ubuntu/bionic, ubuntu/cosmic, ubuntu/disco, ubuntu/eoan, ubuntu/focal, ubuntu/groovy, ubuntu/hirsute, ubuntu/impish, ubuntu/jammy, ubuntu/kinetic, ubuntu/lunar, ubuntu/mantic, ubuntu/noble, ubuntu/oracular]
Ubuntu product releases -- supported only [ubuntu/jammy, ubuntu/mantic, ubuntu/noble, ubuntu/focal]
Devel release: ubuntu/oracular
Loading 50236 CVEs
Loading 21 releases
Couldn't find package cache for release ros-esm/melodic with filename ros-esm_melodic
Couldn't find package cache for release ros-esm/kinetic with filename ros-esm_kinetic
Couldn't find package cache for release snap with filename snap
Loading 5171 USNs
Total loading time: 27.690572500228882

------------------
Showing CVE-2023-1625
------------------
CVE-2023-1625

Description: An information leak was discovered in OpenStack heat. This issue could allow a remote, authenticated attacker to use the 'stack show' command to reveal parameters which are supposed to remain hidden. This has a low impact to the confidentiality, integrity, and availability of the system.

References: ['https://ubuntu.com/security/notices/USN-6066-1', 'https://ubuntu.com/security/notices/USN-6293-1', 'https://www.cve.org/CVERecord?id=CVE-2023-1625']

Notes: [['mdeslaur', 'fixed in jammy-updates and kinetic-updates as a SRU, but not\nyet in the -security pocket.']]

  - CVE-2023-1625 heat ubuntu/focal released 1:14.2.0-0ubuntu1.1
  - CVE-2023-1625 heat ubuntu/jammy released 1:18.0.1-0ubuntu1.1
  - CVE-2023-1625 heat ubuntu/mantic not-affected 1:20.0.0-0ubuntu1
  - CVE-2023-1625 heat ubuntu/noble not-affected 1:20.0.0-0ubuntu1

------------------
Loading jammy into the ...

9bafe15... by David Fernandez Gonzalez on 2024-06-13

[DATALIB] Fix release loading to include parents

Signed-off-by: David Fernandez Gonzalez <email address hidden>

c6132b2... by David Fernandez Gonzalez on 2024-06-13

[DATALIB] Don't include embargoed dir

Signed-off-by: David Fernandez Gonzalez <email address hidden>

915557b... by David Fernandez Gonzalez on 2024-06-13

[DATALIB] Infer USN ESM release by pocket instead of version

Signed-off-by: David Fernandez Gonzalez <email address hidden>

c50841a... by David Fernandez Gonzalez on 2024-06-13

[DATALIB] Add DirtyCache class

This commits add a cache system to prevent relinking objects
if the storage classes haven't been updated since the last time
they were linked.

Signed-off-by: David Fernandez Gonzalez <email address hidden>

[David Fernandez Gonzalez (litios)]

More changes:

* Fix release loading so that when asking for a release, the parents are also loaded.
* Prevent the embargoed CVE dir from being loaded by mistake.
* Since USNs don't specify ESM releases, do best effort to actually figure out the ESM release based on binary pockets
* Implement cache system to avoid relinking objects if the storage classes have not changed since the object was linked before. When loading all USNs, total time went from +70 sec to 30sec.

Testing:

Loading 50228 CVEs
Loading 3 releases
Loading 5219 USNs
*** Done loading
*** Loading USN
Cache:: Marking everything dirty
USN 6812-1 :: Dirty - relink
Cache:: Marking everything dirty
CVE CVE-2024-21012 :: Dirty - relink
CVE CVE-2024-21011 :: Dirty - relink
CVE CVE-2024-21094 :: Dirty - relink
CVE CVE-2024-21068 :: Dirty - relink
*** Loading CVE
CVE CVE-2023-1625 :: Dirty - relink
*** Loading again; cache should reuse
*** Loading USN
USN 6812-1 :: Clean - reuse
*** Loading CVE
CVE CVE-2023-1625 :: Clean - reuse
*** Loading jammy
Loading 1 releases
*** Again with new release; it should relink
*** Loading USN
Cache:: Marking everything dirty
USN 6812-1 :: Dirty - relink
Cache:: Marking everything dirty
CVE CVE-2024-21012 :: Dirty - relink
CVE CVE-2024-21011 :: Dirty - relink
CVE CVE-2024-21094 :: Dirty - relink
CVE CVE-2024-21068 :: Dirty - relink
*** Loading CVE
CVE CVE-2023-1625 :: Dirty - relink
CVE CVE-2024-21011 :: Clean - reuse

[Eduardo Barretto (ebarretto)]

lgtm, thanks!

review: Approve

diff --git a/scripts/datalib/storage.py b/scripts/datalib/storage.py
index fc5c271..308be59 100644
--- a/scripts/datalib/storage.py
+++ b/scripts/datalib/storage.py
@@ -12,13 +12,50 @@
 from datalib.uct_models import SourcePackage
 from .models import *
 
+class DirtyCache:
+    """Cache class for handling modifications"""
+    dirty_cache: dict[str: bool]
+    last_updated: int
+    others_last_updated: list[int]
+
+    def __init__(self) -> None:
+        self.dirty_cache = {}
+        self.others_last_updated = []
+        self.last_updated = 0
+
+    def is_dirty(self, id: str, others_last_updated: list[int]) -> bool:
+        if not self.others_last_updated:
+            self.mark_dirty()
+        else:
+            for index in range(len(others_last_updated)):
+                if self.others_last_updated[index] < others_last_updated[index]:
+                    self.mark_dirty()
+                    break
+        self.others_last_updated = others_last_updated
+        return self.dirty_cache[id]
+
+    def add(self, id: str) -> None:
+        self.dirty_cache[id] = True
+
+    def make_clean(self, id: str) -> None:
+        self.dirty_cache[id] = False
+
+    def mark_dirty(self) -> None:
+        for item in self.dirty_cache:
+            self.dirty_cache[item] = True
+
+    def increment_last_updated(self) -> None:
+        self.last_updated += 1
+
 class PackageStorage:
     """
     Storage class for handling packages.
     """
     packages: dict[str: SourcePackage]
+    dirty_cache: DirtyCache
 
     def __init__(self) -> None:
+        self.dirty_cache = DirtyCache()
         self.packages = {}
 
     def _order_packages(self) -> None:
@@ -34,11 +71,13 @@ class CVEStorage:
     """
     cves: dict[str: CVE]
     package_storage: PackageStorage
+    dirty_cache: DirtyCache
 
     # TODO: add a changed cache so we can cache CVEs and only update when something got loaded
 
     def __init__(self) -> None:
         self.cves = {}
+        self.dirty_cache = DirtyCache()
 
     def link_pkg_storage(self, package_storage: PackageStorage) -> None:
         self.package_storage = package_storage
@@ -50,6 +89,7 @@ class CVEStorage:
     def _init_cve(self, cve: CVE, pkg_data: dict) -> None:
         """Internal function to add CVE object to the class storage"""
         self.cves[cve.id] = (cve, pkg_data)
+        self.dirty_cache.add(cve.id)
 
     def load_cve(self, cve_id: str) -> None:
         """Load a CVE into the storage"""
@@ -68,9 +108,15 @@ class SNStorage:
     sns: dict[str: SN]
     package_storage: PackageStorage
     cve_storage: CVEStorage
+    dirty_cache: DirtyCache
 
     def __init__(self) -> None:
         self.sns = {}
+        self.dirty_cache = DirtyCache()
+
+    def _init_sn(self, sn_object: SN, cves: set, pkgs: dict):
+        self.sns[sn_object.id] = (sn_object, cves, pkgs)
+        self.dirty_cache.add(sn_object.id)
 
     def _order_sns(self) -> None:
         self.sns = dict(sorted(self.sns.items(),
diff --git a/scripts/datalib/uct_models.py b/scripts/datalib/uct_models.py
index 7b9fe01..31568f1 100644
--- a/scripts/datalib/uct_models.py
+++ b/scripts/datalib/uct_models.py
@@ -20,10 +20,13 @@ class Release:
     codename: str
     product: str
     series: str
+    is_devel: bool
     is_esm: bool
     is_active: bool
     requires_oval: bool
     parent: "Release"
+    esm_infra_release: "Release"
+    esm_apps_release: "Release"
 
     def __init__(self, name: str) -> None:
         canon, product, series, details = get_subproject_details(name)
@@ -39,19 +42,27 @@ class Release:
         self.parent = None
         self.version = None
         self.codename = None
+        self.esm_infra_release = None
+        self.esm_apps_release = None
+        self.is_devel = False
         if not details: return
         if 'oval' in details:
             self.requires_oval = details['oval']
         else:
             self.requires_oval = True
         if 'parent' in details:
-            self.parent = Release(details['parent'])
+            self.parent = details['parent']
         if 'version' in details:
             self.version = details['version']
         if 'codename' in details:
             self.codename = details['codename']
+        if 'codename' in details:
+            self.codename = details['codename']
+        if 'devel' in details:
+            self.is_devel = details['devel']
 
     def __eq__(self, value: object) -> bool:
+        if not hasattr(value, 'name'): return False
         return self.name == value.name
 
     def __repr__(self) -> str:
@@ -60,21 +71,6 @@ class Release:
     def __hash__(self):
         return hash(self.name)
 
-    def get_esm_release(self, component: str) -> "Release":
-        esm_releases = get_active_esm_releases()
-        potential_releases = list(filter(lambda release: self.series in release, esm_releases))
-        if len(potential_releases) == 1:
-            return Release(potential_releases[0])
-        elif len(potential_releases) == 2:
-            if component in ['main', 'restricted']:
-                target = 'esm-infra'
-            elif component in ['universe', 'multiverse']:
-                target = 'esm-apps'
-            
-            if target in potential_releases[0]: return Release(potential_releases[0])
-            else: return Release(potential_releases[1])
-
-        return None
     def is_parent_release(self, release: "Release") -> bool:
         return release in self.get_all_parents()
     
diff --git a/scripts/datalib/uct_storage.py b/scripts/datalib/uct_storage.py
index 14f7b98..28cacea 100644
--- a/scripts/datalib/uct_storage.py
+++ b/scripts/datalib/uct_storage.py
@@ -12,7 +12,6 @@
 from .storage import *
 from .uct_models import *
 from .config import PKG_CACHE_DIR, USN_DATABASE
-from .utils import get_active_ubuntu_releases
 from .config import UCT
 
 import json
@@ -20,6 +19,77 @@ import os
 import cve_lib
 from multiprocessing import Pool
 
+class ReleaseStorage():
+    releases: list[str: Release]
+    supported_releases: list[Release]
+    esm_apps_releases: list[Release]
+    esm_infra_releases: list[Release]
+
+    def __init__(self) -> None:
+        self.releases = {}
+        self.supported_releases = []
+        self.esm_apps_releases = []
+        self.esm_infra_releases = []
+
+        for release in cve_lib.all_releases:
+            rel_obj = Release(release)
+            self.releases[str(rel_obj)] = rel_obj
+
+        # Link parents
+        for release in self.releases.values():
+            if release.parent: release.parent = self.releases[release.parent]
+
+        esm_releases = get_active_esm_releases()
+        supported_releases_names = list(set(cve_lib.all_releases).difference(set(cve_lib.eol_releases)).difference(set([cve_lib.devel_release])))
+        esm_infra_releases_names = list(filter(lambda release: release == 'trusty/esm' or 'esm-infra/' in release, esm_releases))
+        esm_apps_releases_names = list(filter(lambda release: 'esm-apps/' in release, esm_releases))
+    
+        for release in supported_releases_names:
+            self.supported_releases.append(self.get_release(release))
+        for release in esm_infra_releases_names:
+            rel_obj = self.get_release(release)
+            self.esm_infra_releases.append(rel_obj)
+            rel_obj.get_oldest_parent().esm_infra_release = rel_obj
+        for release in esm_apps_releases_names:
+            rel_obj = self.get_release(release)
+            self.esm_apps_releases.append(rel_obj)
+            rel_obj.get_oldest_parent().esm_apps_release = rel_obj
+
+    def get_all_releases(self) -> list[Release]:
+        return list(self.releases.values())
+
+    def get_release(self, release_name: str) -> Release:
+        if release_name == 'devel':
+            return self.get_devel_release()
+
+        release = self.releases.get(release_name, None)
+        if not release: 
+            release = self.releases.get(release_name.split('/')[0], None)
+
+        if not release: 
+            for cve_lib_rel in cve_lib.subprojects:
+                if 'alias' in cve_lib.subprojects[cve_lib_rel] and \
+                    cve_lib.subprojects[cve_lib_rel]['alias'] == release_name:
+                    release = self.releases.get(cve_lib_rel, None)
+                    break
+
+        return release
+
+    def get_product_releases(self, product: str, supported_only: bool = False) -> list [Release]:
+        """Get all releases that belong to a product"""
+        if supported_only:
+            releases = list(filter(lambda release: release.product == product, self.supported_releases))
+        else:
+            releases = list(filter(lambda release: release.product == product, self.releases.values()))
+        return releases
+
+    def get_devel_release(self) -> Release:
+        for release in self.releases.values():
+            if release.is_devel:
+                return release
+        
+        return None
+
 class UCTPackageStorage(PackageStorage):
     """
     Storage class for handling UCT packages.
@@ -27,41 +97,59 @@ class UCTPackageStorage(PackageStorage):
     packages: dict[str: UbuntuSourcePackage]
     latest_date_created: dict[Release:datetime.datetime]
     loaded_releases: set[Release]
+    release_storage: ReleaseStorage
 
     def __init__(self) -> None:
         self.loaded_releases = set()
         self.latest_date_created = {}
+        self.release_storage = None
         super().__init__()
     
-    def load(self, filter_releases: list = []) -> None:
-        final_releases = []
-        active_ubuntu_releases = get_active_ubuntu_releases()
-        for release in active_ubuntu_releases:
-            if filter_releases and release not in filter_releases: continue
-            final_releases.append(release)
-
-        with Pool(processes=os.cpu_count()) as pool:
-            print(f'Loading {len(final_releases)} releases')
-            package_data = pool.map(self._load_release_from_cache, final_releases)
+    def load(self, filter_releases: list[Release] = [], multithreading: bool = False) -> None:
+        self.dirty_cache.increment_last_updated()
+        releases = self.release_storage.supported_releases if not filter_releases else filter_releases
+        final_releases = set()
+
+        for release in releases:
+            final_releases.add(release)
+            while release.parent:
+                release = release.parent
+                if release not in final_releases:
+                    final_releases.add(release)
+
+        print(f'Loading {len(final_releases)} releases')
+        if multithreading:
+            with Pool(processes=os.cpu_count()) as pool:
+                package_data = pool.map(self._load_release_from_cache, final_releases)
+        else:
+            package_data = []
+            for release in final_releases:
+                package_data.append(self._load_release_from_cache(release))
 
         for release, release_data, last_date_created in package_data:
             self._init_release(release, release_data, last_date_created)
 
         self._order_packages()
 
-    def _load_release_from_cache(self, release: str)  -> dict:
+    def _load_release_from_cache(self, release: Release)  -> dict:
         data = {}
+        pkg_cache_filename = []
+        if release.name == 'esm/trusty':
+            pkg_cache_filename = 'trusty_esm'
+        elif release.product == 'ubuntu':
+            pkg_cache_filename = release.series
+        else:
+            pkg_cache_filename = release.name.replace('/', '_')
 
         try:
-            with open(os.path.join(os.path.expanduser(PKG_CACHE_DIR), release.replace('/', '_') + '-pkg-cache.json')) as f:
+            with open(os.path.join(os.path.expanduser(PKG_CACHE_DIR), pkg_cache_filename + '-pkg-cache.json')) as f:
                 data = json.load(f)
-        except FileNotFoundError as ex:
-            print(ex)
+        except FileNotFoundError:
+            print(f'Couldn\'t find package cache for release {release.name} with filename {pkg_cache_filename}')
             return (release, None, None)
 
         release_data = {}
         latest_date_created = None
-        release_obj = Release(release)
         for src_pkg_name, src_pkg_versions in data.items():
             if src_pkg_name == 'latest_date_created':
                 latest_date_created = datetime.datetime.fromtimestamp(src_pkg_versions)
@@ -71,7 +159,7 @@ class UCTPackageStorage(PackageStorage):
             release_data[src_pkg_name] = source_package
             for src_pkg_version, src_pkg_version_data in src_pkg_versions.items():
                 src_pkg_version_obj = UbuntuVersion(src_pkg_version)
-                source_package.add_source_version(src_pkg_version_obj, release_obj,
+                source_package.add_source_version(src_pkg_version_obj, release,
                                                   src_pkg_version_data['component'], 
                                                   src_pkg_version_data['pocket'])
 
@@ -82,7 +170,7 @@ class UCTPackageStorage(PackageStorage):
                                                                   source_package,
                                                                   binary_data['component']))
 
-        return (release_obj, release_data, latest_date_created)
+        return (release, release_data, latest_date_created)
     
     def _init_release(self, release: Release, release_data: dict, latest_date_created: str) -> None:
         if not release_data: return
@@ -98,13 +186,13 @@ class UCTPackageStorage(PackageStorage):
     def get_package(self, pkg_name: str) -> UbuntuSourcePackage:
         return super().get_package(pkg_name)
 
-    def load_release(self, release:str) -> None:
-        if Release(release) in self.loaded_releases: return
+    def load_release(self, release: Release) -> None:
+        if self.release_storage.get_release(release) in self.loaded_releases: return
         release_obj, data, date = self._load_release_from_cache(release)
         self._init_release(release_obj, data, date)
 
-    def unload_release(self, release: str) -> None:
-        release = Release(release)
+    def unload_release(self, release: Release) -> None:
+        release = self.release_storage.get_release(release)
         if release not in self.loaded_releases: return
         self.loaded_releases.remove(release)
 
@@ -113,6 +201,9 @@ class UCTPackageStorage(PackageStorage):
             for version in original_source_versions:
                 ver_rel = package.get_version_release(version)
                 if ver_rel == release: package.remove_version(version)
+    
+    def link_release_storage(self, release_storage: ReleaseStorage) -> None:
+        self.release_storage = release_storage
 
 class UCTCVEStorage(CVEStorage):
     """
@@ -120,10 +211,18 @@ class UCTCVEStorage(CVEStorage):
     """
     cves: dict[str: UbuntuCVE]
     package_storage: UCTPackageStorage
+    release_storage: ReleaseStorage
 
     # TODO: add a changed cache so we can cache CVEs and only update when something got loaded
+    def __init__(self) -> None:
+        self.release_storage = None
+        super().__init__()
+
+    def link_release_storage(self, release_storage: ReleaseStorage) -> None:
+        self.release_storage = release_storage
 
     def load(self, cves_filter: list = []) -> None:
+        self.dirty_cache.increment_last_updated()
         final_cves = []
         cves_ids = self.get_uct_cve_ids()
         for cve_id in cves_ids:
@@ -171,11 +270,15 @@ class UCTCVEStorage(CVEStorage):
         
         return (cve, data['pkgs'])
 
-    def get_uct_cve_ids(self) -> list:
+    def get_uct_cve_ids(self, embargoed: bool = False) -> list:
         """Return all CVE IDs identified from UCT"""
 
         cve_ids = []
-        for dir in cve_lib.cve_dirs:
+        cve_dirs = [cve_lib.active_dir, cve_lib.retired_dir, cve_lib.ignored_dir]
+        if embargoed:
+            cve_dirs.append(cve_lib.embargoed_dir)
+
+        for dir in cve_dirs:
             for (_, _, filenames) in os.walk(dir):
                 for filename in filenames:
                     if not re.search(CVE.ID_REGEX, filename): continue
@@ -212,7 +315,8 @@ class UCTCVEStorage(CVEStorage):
             package = self.package_storage.get_package(package_name)
             if not package: continue
             for release, status in package_entries.items():
-                release = Release(release)
+                release = self.release_storage.get_release(release)
+                if not release: continue
                 if not package.release_exists(release, include_parents=True): continue
                 cve.add_cve_entry(UbuntuCVEPkgEntry(package, cve, status[0], status[1], release))
 
@@ -228,23 +332,36 @@ class UCTCVEStorage(CVEStorage):
         if cve_id not in self.cves: return None
 
         cve, _ = self.cves[cve_id]
+        if not self.dirty_cache.is_dirty(cve_id, [self.package_storage.dirty_cache.last_updated]):
+            return cve
+
         cve.pkg_entries = list()
         if not with_pkg_links: return cve
         self.populate_cve_entries(cve_id)
 
+        self.dirty_cache.make_clean(cve_id)
         return cve
 
 class USNStorage(SNStorage):
     sns: dict[str: tuple[USN, list, dict]]
     package_storage: UCTPackageStorage
     cve_storage: UCTCVEStorage
+    release_storage: ReleaseStorage
+
+    def __init__(self) -> None:
+        self.release_storage = None
+        super().__init__()
+
+    def link_release_storage(self, release_storage: ReleaseStorage) -> None:
+        self.release_storage = release_storage
 
     def load(self) -> None:
+        self.dirty_cache.increment_last_updated()
         usns_data = self._get_usn_database_data()
         print(f'Loading {len(usns_data)} USNs')
         for usn_id, usn_data in usns_data.items():
             _, _, cves, lp_bugs, pkgs = self.process_usn_data((usn_id, usn_data))
-            self.add_usn(usn_id, usn_data, cves, lp_bugs, pkgs)
+            self._init_sn(USN(usn_id, usn_data, set(), dict(), lp_bugs), cves, pkgs)
 
         self._order_sns()
 
@@ -265,7 +382,21 @@ class USNStorage(SNStorage):
             # Get package
             for pkg, pkg_info in info['sources'].items():
                 pkgs.setdefault(pkg, [])
-                pkgs[pkg].append((pkg_info['version'], rel))
+                pocket = None
+                # Best effort, there are too many corner cases
+                # since the format is not always the same for all
+                # USNs
+                if 'allbinaries' in info:
+                    for binary_data in info['allbinaries'].values():
+                        if 'source' in binary_data and binary_data['source'] != pkg: continue
+                        if 'pocket' not in binary_data: continue
+                        if pocket and binary_data['pocket'] != pocket:
+                            # Different pocket in binaries, default to parent
+                            pocket = None
+                            break
+                        elif not pocket:
+                            pocket = binary_data['pocket']
+                pkgs[pkg].append((pkg_info['version'], pocket, rel))
 
         # CVE loading
         for cve_text in usn_data['cves']:
@@ -276,9 +407,6 @@ class USNStorage(SNStorage):
                 cves.add(cve_text)
 
         return usn_id, usn_data, cves, lp_bugs, pkgs 
-        
-    def add_usn(self, usn_id: str, usn_data: dict, cves: set, lp_bugs: set, pkgs: dict) -> None:
-        self.sns[usn_id] = (USN(usn_id, usn_data, set(), dict(), lp_bugs), cves, pkgs)
 
     def get_usns_by_cve(self, cve_id: str) -> list[USN]:
         target_usns = []
@@ -291,6 +419,11 @@ class USNStorage(SNStorage):
     def get_usn(self, usn_id: str) -> USN:
         if usn_id not in self.sns: return None
         usn, cves, pkgs = self.sns[usn_id]
+        if not self.dirty_cache.is_dirty(usn_id,
+                                         [self.cve_storage.dirty_cache.last_updated,
+                                          self.package_storage.dirty_cache.last_updated]):
+            return usn
+
         cve_mapping = set()
         pkg_mapping = dict()
 
@@ -302,22 +435,22 @@ class USNStorage(SNStorage):
         for pkg_name in pkgs:
             pkg = self.package_storage.get_package(pkg_name)
             if not pkg: continue
-            for version, release in pkgs[pkg_name]:
-                release = Release(release)
+            for version, pocket, release in pkgs[pkg_name]:
+                release = self.release_storage.get_release(release)
+                if not release: continue
                 if not pkg.release_exists(release, include_parents=True): continue
                 pkg_mapping.setdefault(pkg, [])
                 version = UbuntuVersion(version)
-                if version.is_esm():
-                    for pkg_version in pkg.source_versions:
-                        if pkg.get_version_release(pkg_version) == release:
-                            release = release.get_esm_release(pkg.get_version_component(pkg_version))
-                            break
+                if pocket == 'esm-apps':
+                    release = release.esm_apps_release
+                elif pocket == 'esm-infra':
+                    release = release.esm_infra_release
 
                 pkg_mapping[pkg].append((version, release))
 
         usn.cves = cve_mapping
         usn.package_fixed_versions = pkg_mapping
         
+        self.dirty_cache.make_clean(usn_id)
         return usn
 
-
diff --git a/scripts/datalib/utils.py b/scripts/datalib/utils.py
deleted file mode 100644
index cc072e0..0000000
--- a/scripts/datalib/utils.py
+++ /dev/null
@@ -1,20 +0,0 @@
-#!/usr/bin/python3
-# -*- coding: utf-8 -*-
-# Module containing classes that represent CVE and Package data 
-#
-# Author: David Fernandez Gonzalez <[e-mail]>
-# Copyright (C) 2024 Canonical Ltd.
-#
-# This script is distributed under the terms and conditions of the GNU General
-# Public License, Version 2 or later. See http://www.gnu.org/copyleft/gpl.html
-# for details.
-#
-import cve_lib
-
-def get_active_ubuntu_releases():
-    active_ubuntu_releases = []
-    for release in cve_lib.all_releases:
-        if release not in cve_lib.eol_releases or release in cve_lib.get_active_releases_with_esm():
-            active_ubuntu_releases.append(release)
-    
-    return active_ubuntu_releases
\ No newline at end of file