Merge ~litios/ubuntu-cve-tracker:json-pkg-gen into ubuntu-cve-tracker:master

[David Fernandez Gonzalez]

This is the JSON Generator as per https://warthogs.atlassian.net/browse/SEC-2930.

This generator follows the same pattern as the OVAL generators. It reuses the loading capabilities of the main OVALGenerator, but it uses the SSN generator as it also relies on the SSNs being loaded. It also implements the parent support from OVAL.

Generation is done through the generate-oval script, by adding another type `--type json-pkg`.
The output filename is 'com.ubuntu.RELEASE.pkg.json'.

Unit testing has been added to test/test_json_generation.py.

[David Fernandez Gonzalez (litios)]

It also modifies the following OVAL behavior:

* Package won't fail if sources are not provided when loading.
* USNs now support lp_bugs.
* Function _get_parent_releases added to the main OVAL generator since it was needed for collecting the needed sources in the JSON generator. This refactors the code embedded in _load, it doesn't add new code.

83d24ee... by David Fernandez Gonzalez on 2024-03-05

[JSON] Output the right filename

Signed-off-by: David Fernandez Gonzalez <email address hidden>

93f67ba... by David Fernandez Gonzalez on 2024-03-06

[JSON PKG] Account for Packages/USNs/CVEs in parent releases

The objects might not being present in the child release, but
be in the parent. We need to include them.

Source packages take a different treatment because what we want
to merge is all the sources.

Also, create a shared function to retrieve all objects affecting
all parent releases + selected release.

Tests updated to account for parent cases.

Signed-off-by: David Fernandez Gonzalez <email address hidden>

114b27e... by David Fernandez Gonzalez on 2024-03-06

[OVAL] Use new expand system

Signed-off-by: David Fernandez Gonzalez <email address hidden>

[Eduardo Barretto (ebarretto)]

a few things to fix, nothing major

review: Needs Fixing

670af96... by David Fernandez Gonzalez on 2024-03-13

[JSON] Styling issues

Signed-off-by: David Fernandez Gonzalez <email address hidden>

[David Fernandez Gonzalez (litios)]

Hey Eduardo, thanks for the review. I fixed the issues mentioned in a new commit, except some of them:

* I believe there's an alignment issue here.
- This seems to be already in the current code, I just fixed it.

* I believe we can drop the fixed_only argument. It doesn't fit the schema.
- Since this works from the _load function, we could generate the file only with the CVEs that have been fixed instead of all. We are not going to use it in production, but since it works without really touching anything, I didn't see any benefit in removing it. Let me know if you still think we should get rid of it and I will delete it.

* could we just rely on get_pocket and avoid this check?
- Not right now, because the pocket will be either security, release, etc. They want the pocket to be esm-infra/apps if that's where the version is, which is a custom feature for this format. If we migrate that to get_pocket, this will also be the case for regular OVAL.

b710fca... by David Fernandez Gonzalez on 2024-03-13

[OVAL] More styling issues

Signed-off-by: David Fernandez Gonzalez <email address hidden>

ebcfa6f... by David Fernandez Gonzalez on 2024-03-13

[JSON] Account for cases with parent release and /

Signed-off-by: David Fernandez Gonzalez <email address hidden>

59fad01... by David Fernandez Gonzalez on 2024-03-13

[JSON] Add JSON schema

Signed-off-by: David Fernandez Gonzalez <email address hidden>

2618ee3... by David Fernandez Gonzalez on 2024-03-13

[JSON] Clean spaces in test_json_generation

Signed-off-by: David Fernandez Gonzalez <email address hidden>

3dd0c8e... by David Fernandez Gonzalez on 2024-03-14

[JSON] Update schema to the last available

Signed-off-by: David Fernandez Gonzalez <email address hidden>

319f3ab... by David Fernandez Gonzalez on 2024-03-14

[JSON] Update tests to match current format

Signed-off-by: David Fernandez Gonzalez <email address hidden>

[Eduardo Barretto (ebarretto)]

lgtm, thanks for addressing all the comments!
If possible in a later PR, I think it would be good to validate the generated test json against the schema.

review: Approve

diff --git a/.launchpad.yaml b/.launchpad.yaml
index 9bcdb5d..e672a85 100644
--- a/.launchpad.yaml
+++ b/.launchpad.yaml
@@ -26,6 +26,8 @@ jobs:
       - python3-pytest-cov
       - python3-yaml
       - shellcheck
+      - python3-dateutil
+      - python3-pytest-mock
     run-before: |
       # configure a basic ~/.ubuntu-cve-tracker.conf
       echo plb_authentication=/dev/null > ~/.ubuntu-cve-tracker.conf
@@ -42,7 +44,7 @@ jobs:
       echo "reporting syntax issues on scripts"
       make check-syntax-scripts
       echo "Running unit tests..."
-      pytest-3 --cov=scripts ./scripts/test_*.py ./test/test_oval_lib_unit.py
+      pytest-3 --cov=scripts ./scripts/test_*.py ./test/test_oval_lib_unit.py ./test/test_json_generation.py
   check-cves:
     series: jammy
     architectures: amd64
diff --git a/scripts/generate-oval b/scripts/generate-oval
index cc44ee7..d01329f 100755
--- a/scripts/generate-oval
+++ b/scripts/generate-oval
@@ -45,7 +45,7 @@ def main():
 
     # parse command line options
     parser = argparse.ArgumentParser(description='Generate OVAL for CVE, PKG or USNs.')
-    parser.add_argument('--type', required=True, choices=['cve', 'pkg', 'usn'],
+    parser.add_argument('--type', required=True, choices=['cve', 'pkg', 'usn', 'json-pkg'],
                         help='OVAL format')
     parser.add_argument('--cves', nargs='*', default=['active/CVE-*', 'retired/CVE-*'],
                         help='pathname patterns (globs) specifying CVE '
@@ -104,7 +104,7 @@ def main():
         for r in set(all_releases).difference(set(eol_releases)).difference(set([devel_release])):
             if needs_oval(r):
                 releases.append(r)
-        
+
     out_releases = releases
 
     # for each release we need to get its parent to also
@@ -124,8 +124,9 @@ def main():
         out_releases = set(out_releases) - parent_releases
 
     if args.type == 'usn':
-            generate_oval_usn(args.output_dir, args.usn_number, releases,
-                              args.cves, args.usn_db_dir, args.no_progress, args.oci_prefix, args.oci_output_dir)
+        generate_oval_usn(args.output_dir, args.usn_number, releases,
+                          args.cves, args.usn_db_dir, args.no_progress,
+                          args.oci_prefix, args.oci_output_dir)
     else:
         cache = {}
         for release in releases:
@@ -136,6 +137,12 @@ def main():
             generate_oval_package(out_releases, args.output_dir, args.cves, cache, cve_cache, args.oci, args.no_progress, args.packages, args.fixed_only, args.oci_output_dir, args.expand)
         elif args.type == 'cve':
             generate_oval_cve(out_releases, args.output_dir, args.cves, cache, cve_cache, args.oci, args.no_progress, args.packages, args.fixed_only, args.oci_output_dir, args.expand)
+        elif args.type == 'json-pkg':
+            usn_database = get_usn_database(args.usn_db_dir)
+            if not usn_database:
+                error("Error getting local USN database. Please, run '$UCT/scripts/fetch-db database.json.bz2' to retrieve the database and try again.")
+
+            generate_json_pkg_oval(out_releases, args.output_dir, args.cves, cache, cve_cache, usn_database, args.no_progress, args.packages, args.fixed_only, args.expand)
 
 
 def warn(message):
@@ -292,5 +299,17 @@ def generate_oval_cve(releases, outdir, cves, pkg_cache, cve_cache, oci, no_prog
     if not no_progress:
         print(f'[X] Done generating OVAL CVE for packages in releases {", ".join(releases)}')
 
+
+def generate_json_pkg_oval(releases, output_dir, cves, cache, cve_cache, usn_database, no_progress, packages, fixed_only, expand):
+    if not no_progress:
+        print(f'[*] Generating JSON PKG for packages in releases {", ".join(releases)}')
+
+    gen = oval_lib.JSONPkgGenerator(releases, cves, packages, not no_progress, cache, usn_database, fixed_only, cve_cache, output_dir, expand)
+
+    gen.generate_json()
+
+    if not no_progress:
+        print(f'[X] Done generating JSON PKG for packages in releases {", ".join(releases)}')
+
 if __name__ == '__main__':
     main()
diff --git a/scripts/oval_lib.py b/scripts/oval_lib.py
index 6273d08..7105732 100755
--- a/scripts/oval_lib.py
+++ b/scripts/oval_lib.py
@@ -29,6 +29,8 @@ import tempfile
 import collections
 import glob
 import xml.etree.cElementTree as etree
+import json
+from dateutil import parser, tz
 from xml.dom import minidom
 from typing import Tuple # Needed because of Python < 3.9 and to also support < 3.7
 
@@ -374,21 +376,25 @@ class Package:
         self.rel = rel
         self.description = cve_lib.lookup_package_override_description(pkgname)
 
-        if not self.description:
-            if 'description' in sources[rel][pkgname]:
-                self.description = sources[rel][pkgname]['description']
-            elif pkgname in source_map_binaries[rel] and \
-                'description' in source_map_binaries[rel][pkgname]:
-                self.description = source_map_binaries[rel][pkgname]['description']
-            else:
-                # Get first description found
-                if 'binaries' in sources[self.rel][self.name]:
-                    for binary in sources[self.rel][self.name]['binaries']:
-                        if binary in source_map_binaries[self.rel] and 'description' in source_map_binaries[self.rel][binary]:
-                            self.description = source_map_binaries[self.rel][binary]["description"]
-                            break
-
-        self.section = sources[rel][pkgname]['section']
+        if rel in sources and pkgname in sources[rel]:
+            if not self.description:
+                if 'description' in sources[rel][pkgname]:
+                    self.description = sources[rel][pkgname]['description']
+                elif pkgname in source_map_binaries[rel] and \
+                    'description' in source_map_binaries[rel][pkgname]:
+                    self.description = source_map_binaries[rel][pkgname]['description']
+                else:
+                    # Get first description found
+                    if 'binaries' in sources[self.rel][self.name]:
+                        for binary in sources[self.rel][self.name]['binaries']:
+                            if binary in source_map_binaries[self.rel] and 'description' in source_map_binaries[self.rel][binary]:
+                                self.description = source_map_binaries[self.rel][binary]["description"]
+                                break
+
+            self.section = sources[rel][pkgname]['section']
+        else:
+            self.section = 'main'
+
         self.versions_binaries = versions_binaries if versions_binaries else {}
         self.earliest_version = self.get_earliest_version()
         self.latest_version = self.get_latest_version()
@@ -480,12 +486,14 @@ class Package:
         return self.__str__()
 
 class USN:
-    def __init__(self, data, cve_objs, pkgs_by_rel):
+    def __init__(self, data, cve_objs, pkgs_by_rel, lp_bugs):
         for item in ['description', 'releases', 'title', 'timestamp', 'summary', 'action', 'id', 'isummary']:
             if item in data:
                 setattr(self, item, data[item])
             else:
                 setattr(self, item, None)
+
+        self.lp_bugs = lp_bugs
         self.cves = cve_objs
         self.pkgs = self._generate_pkg_fixed_ver_tuple_dict(pkgs_by_rel)
 
@@ -498,6 +506,13 @@ class USN:
                 tup_dict[rel][src_name] = (pkg_object, fixed_ver)
         return tup_dict
 
+    def get_pkg_fixed_version(self, package: Package):
+        if not self.is_package_present(package): return None
+        return self.pkgs[package.rel][package.name][1]
+
+    def is_package_present(self, package: Package):
+        return package.rel in self.pkgs and package.name in self.pkgs[package.rel]
+
     def __str__(self) -> str:
         # return f'description: {self.description}\nid: {self.id}\ncves: {self.cves}\npkgs: {self.pkgs}\n' # TODO: remove this ugly debug print
         return self.id
@@ -531,14 +546,7 @@ class OvalGenerator:
         self.release_codename = cve_lib.release_progenitor(self.release) if cve_lib.release_progenitor(self.release) else self.release
         self.release_codename = self.release_codename.replace('/', '_')
         self.release_name = cve_lib.release_name(self.release)
-
-        self.parent_releases = list()
-        current_release = self.release
-        while(cve_lib.release_parent(current_release)):
-            current_release = cve_lib.release_parent(current_release)
-            if current_release != self.release and \
-                current_release not in self.parent_releases:
-                self.parent_releases.append(current_release)
+        self.parent_releases = self._get_parent_releases(self.release)
 
         self.ns = 'oval:com.ubuntu.{0}'.format(self.release_codename)
         self.id = 100
@@ -572,6 +580,25 @@ class OvalGenerator:
 
         return False
 
+    def _get_parent_releases(self, release) -> list[str]:
+        parent_releases = list()
+        current_release = release
+        while(cve_lib.release_parent(current_release)):
+            current_release = cve_lib.release_parent(current_release)
+            if current_release != release and \
+                current_release not in parent_releases:
+                parent_releases.append(current_release)
+
+        return parent_releases
+
+    def _get_objects_including_parents(self, objects) -> dict:
+        all_objs = dict()
+        for parent_release in self.parent_releases[::-1]:
+            all_objs.update(objects[parent_release])
+
+        all_objs.update(objects[self.release])
+        return all_objs
+
     def _add_structure(self, root) -> None:
         structure = {}
         for element in self.supported_oval_elements:
@@ -1508,11 +1535,7 @@ class OvalGeneratorPkg(OvalGenerator):
                 running_kernel_id = self.definition_id
                 self._increase_id(is_definition=True)
 
-            all_pkgs = dict()
-            for parent_release in self.parent_releases[::-1]:
-                all_pkgs.update(self.packages[parent_release])
-
-            all_pkgs.update(self.packages[self.release])
+            all_pkgs = self._get_objects_including_parents(self.packages)
 
             for pkg in all_pkgs:
                 if self._ignore_source_package(pkg): continue
@@ -1760,12 +1783,7 @@ class OvalGeneratorCVE(OvalGenerator):
             accepted_releases = self.parent_releases.copy()
             accepted_releases.insert(0, self.release)
 
-            all_cves = self.cves[self.release]
-            for parent_release in self.parent_releases:
-                for cve in self.cves[parent_release]:
-                    if cve not in all_cves:
-                        all_cves[cve] = self.cves[parent_release][cve]
-
+            all_cves = self._get_objects_including_parents(self.cves)
             all_cves = dict(sorted(all_cves.items()))
 
             for cve in all_cves:
@@ -1775,21 +1793,24 @@ class OvalGeneratorCVE(OvalGenerator):
             self._write_oval_xml(xml_tree, root_element)
 
 class OvalGeneratorUSNs(OvalGenerator):
-    def __init__(self, releases, cve_paths, packages, progress, pkg_cache, usn_database, fixed_only=True, cve_cache=None, outdir='./', oval_format='dpkg') -> None:
-        super().__init__('usn', releases, cve_paths, packages, progress, pkg_cache, fixed_only, cve_cache, outdir, oval_format)
+    def __init__(self, releases, cve_paths, packages, progress, pkg_cache, usn_database, fixed_only=True, cve_cache=None, outdir='./', oval_format='dpkg', expand=False) -> None:
+        super().__init__('usn', releases, cve_paths, packages, progress, pkg_cache, fixed_only, cve_cache, outdir, oval_format, expand)
         self.usns = self._load_usns(usn_database)
 
     def _load_usns(self, usn_database):
         usns = {}
         # go thru every USN in the JSON
-        for usn_id, usn_data in usn_database.items():
+        for usn_id, usn_data in dict(sorted(usn_database.items())).items():
             # take existing CVE and Package objects
             cve_objs = {}
             pkg_objs_by_rels = {}
+            lp_bugs = []
             for rel, info in usn_data['releases'].items():
                 # CVE stays the same across releases
                 for cve in usn_data['cves']:
-                    if cve_objs.get(cve) is None:
+                    if 'launchpad' in cve:
+                        lp_bugs.append(cve)
+                    elif cve_objs.get(cve) is None:
                         try:
                             cve_objs[cve] = self.cves[rel][cve]
                         except KeyError:
@@ -1804,7 +1825,7 @@ class OvalGeneratorUSNs(OvalGenerator):
                     pkg_objs_by_rels[rel] = pkg_objs
             # create a USN object with fields in the USN and
             # corresponding CVEs and Packages
-            usns[usn_id] = USN(usn_data, cve_objs, pkg_objs_by_rels)
+            usns[usn_id] = USN(usn_data, cve_objs, pkg_objs_by_rels, lp_bugs)
         return usns
 
     def _generate_advisory(self, usn: USN) -> etree.Element:
@@ -2702,6 +2723,183 @@ class OvalGeneratorUSN():
         if not os.listdir(self.tmpdir):
             os.rmdir(self.tmpdir)
 
+class JSONPkgGenerator(OvalGeneratorUSNs):
+    """Class to generate JSON representation of OVAL"""
+
+    def __init__(self, releases, cve_paths, packages, progress, pkg_cache, usn_database,fixed_only=True, cve_cache=None, outdir='./', expand=False) -> None:
+        self.json_data = dict()
+        super().__init__(releases, cve_paths, packages, progress, pkg_cache, usn_database, fixed_only, cve_cache, outdir, expand=expand)
+        self.generator_type = 'json_pkg'
+
+    def _generate_usn_info(self, usn: USN):
+        return {
+            'description': usn.description,
+            'published_at': datetime.fromtimestamp(usn.timestamp).isoformat(timespec="seconds"),
+            'related_cves': list(usn.cves.keys()),
+            'related_launchpad_bugs': usn.lp_bugs
+        }
+
+    def _generate_usns_info(self):
+        usns_json = {}
+        # USNs should probably follow the same structure as self.packages and self.cves
+        # For now, doing this instead of self._get_objects_including_parents
+        all_releases = self.parent_releases.copy()
+        all_releases.append(self.release)
+
+        for usn_id, usn in self.usns.items():
+            affects = False
+            for usn_release in usn.releases:
+                if usn_release in all_releases:
+                    affects = True
+                    break
+
+            if not affects: continue
+
+            usns_json[f'USN-{usn_id}'] = self._generate_usn_info(usn)
+        return usns_json
+
+    def _generate_cve_info(self, cve: CVE) -> dict:
+        notes = []
+
+        for note in cve.notes:
+            notes.append(f'{note[0]}> {note[1]}')
+
+        return {
+            'description': cve.description.strip(),
+            'published_at': parser.parse(cve.public_date,
+                                         tzinfos={'PDT': tz.gettz('America/Los_Angeles'),
+                                                  'PST': tz.gettz('America/Los_Angeles')
+                                                  }
+                                        ).isoformat(timespec="seconds"),
+            'notes': notes,
+            'mitigation': cve.mitigation,
+            'cvss_severity': cve.cvss[0]['baseSeverity'].lower() if cve.cvss else None,
+            'cvss_score': float(cve.cvss[0]['baseScore']) if cve.cvss else None
+        }
+
+    def _generate_cves_info(self) -> dict:
+        cves_json = {}
+        for cve_number, cve in self._get_objects_including_parents(self.cves).items():
+            cves_json[cve_number] = self._generate_cve_info(cve)
+
+        return cves_json
+
+    def _generate_cve_pkg_info(self, package: Package) -> dict:
+        pkg_cves = {}
+        for cve in package.cves:
+            cve_pkg_rel_entry = None
+
+            for pkg_rel_entry_id in cve.pkg_rel_entries:
+                pkg_rel_entry_package, pkg_rel_entry_rel = pkg_rel_entry_id.split('/', maxsplit=1)
+                if pkg_rel_entry_package == package.name and \
+                    pkg_rel_entry_rel == package.rel:
+                    cve_pkg_rel_entry = cve.pkg_rel_entries[pkg_rel_entry_id]
+                    break
+
+            if not cve_pkg_rel_entry:
+                print(f'CVE entry for {package.name} - {package.rel} in {cve.number} not found')
+                continue
+
+            pkg_cves[cve.number] = {
+                'source_fixed_version': cve_pkg_rel_entry.fixed_version,
+                'ubuntu_priority': cve.priority,
+                'status': cve_pkg_rel_entry.status
+            }
+
+        return pkg_cves
+
+    def _generate_usn_pkg_info(self, package: Package) -> dict:
+        usns_pkg_info = dict()
+        usns_regression_pkg_info = dict()
+        for usn_id, usn in self.usns.items():
+            if usn.is_package_present(package):
+                usn_pkg_info = {
+                        'source_fixed_version': usn.get_pkg_fixed_version(package)
+                    }
+
+                if 'regression' in usn.title.lower():
+                    usns_regression_pkg_info[f'USN-{usn_id}'] = usn_pkg_info
+                else:
+                    usns_pkg_info[f'USN-{usn_id}'] = usn_pkg_info
+
+        return usns_pkg_info, usns_regression_pkg_info
+
+    def _generate_package_source_info(self, package: Package, source_package_info: dict):
+        for source_version in package.versions_binaries:
+            if 'esm' in source_version:
+                pocket = 'esm-apps' if package.section == 'universe' else 'esm-infra'
+            else:
+                _, pocket = get_pocket(self.pkg_cache, package.name, source_version, package.rel)
+
+            source_package_info.setdefault(source_version, {
+                'binary_packages': dict(),
+                'pocket': pocket.lower()
+            })
+            for binary_version in package.versions_binaries[source_version]:
+                for binary in package.versions_binaries[source_version][binary_version]:
+                    source_package_info[source_version]['binary_packages'][binary] = binary_version
+
+    def _generate_package_info(self, package: Package) -> dict:
+        source_package_info = dict()
+
+        usns_pkg_info, usns_reg_pkg_info = self._generate_usn_pkg_info(package)
+        self._generate_package_source_info(package, source_package_info)
+
+        for release in self.parent_releases:
+            if not package.name in self.packages[release]: continue
+            parent_package = self.packages[release][package.name]
+            self._generate_package_source_info(parent_package, source_package_info)
+
+        return {
+            'source_versions': source_package_info,
+            'ubuntu_security_notices': usns_pkg_info,
+            'ubuntu_security_notices_regressions': usns_reg_pkg_info,
+            'cves': self._generate_cve_pkg_info(package)
+        }
+
+    def _generate_packages_info(self) -> dict:
+        json_packages = {}
+        for package_name, package in self._get_objects_including_parents(self.packages).items():
+            json_packages[package_name] = self._generate_package_info(package)
+
+        return json_packages
+
+    def _generate_json_headers(self) -> dict:
+        return {
+            'format': '1',
+            'published_at': datetime.now().isoformat(timespec="seconds"),
+            'release': self.release_codename,
+            'packages': dict(),
+            'security_issues': dict()
+        }
+
+    def _write_json(self, json_structure: dict) -> None:
+        with open(os.path.join(self.output_dir, self.output_file), 'w+') as file:
+            json.dump(json_structure, file)
+
+    def _generate_structure(self) -> dict:
+        json_structure = self._generate_json_headers()
+        json_structure['packages'] = self._generate_packages_info()
+        json_structure['security_issues']['cves'] = self._generate_cves_info()
+        json_structure['security_issues']['usns'] = self._generate_usns_info()
+
+        return json_structure
+
+    def _init_ids(self, release):
+        self.release = release
+        self.release_codename = cve_lib.release_progenitor(release) if cve_lib.release_progenitor(release) else release
+        self.release_codename = self.release_codename.replace('/', '_')
+        self.parent_releases = self._get_parent_releases(release)
+        if self._expand_release(self.release, self.expand):
+             self.output_file = f'com.ubuntu.{self.release.replace("/", "_")}.pkg.json'
+        else:
+             self.output_file = f'com.ubuntu.{self.release_codename}.pkg.json'
+
+    def generate_json(self) -> None:
+        for release in self.releases:
+            self._init_ids(release)
+            self._write_json(self._generate_structure())
+
 def find_release_codename(release):
     return cve_lib.release_progenitor(release) if cve_lib.release_progenitor(release) else release.replace('/', '_')
 
diff --git a/test/json-gen-schema.json b/test/json-gen-schema.json
 new file mode 100644
index 0000000..6d1a833
--- /dev/null
+++ b/test/json-gen-schema.json
@@ -0,0 +1,123 @@
+{
+  "$schema": "http://json-schema.org/draft/2020-12/schema#",
+  "type": "object",
+  "properties": {
+    "format": {"type": "string"},
+    "published_at": {"type": "string", "format": "date-time"},
+    "release": {"type": "string"},
+    "packages": {
+      "type": "object",
+      "additionalProperties": {"$ref": "#/$defs/package"}
+    },
+    "security_issues": {
+      "type": "object",
+      "properties": {
+        "cves": {
+          "type": "object",
+          "additionalProperties": {"$ref": "#/$defs/cve"}
+        },
+        "usns": {
+          "type": "object",
+          "additionalProperties": {"$ref": "#/$defs/usn"}
+        }
+      },
+      "required": ["cves", "usns"]
+    }
+  },
+  "$defs": {
+    "package": {
+      "type": "object",
+      "properties": {
+        "source_versions": {
+          "type": "object",
+          "additionalProperties": {"$ref": "#/$defs/sourceVersion"}
+        },
+        "ubuntu_security_notices": {
+          "type": "object",
+          "additionalProperties": {"$ref": "#/$defs/securityNotice"}
+        },
+        "ubuntu_security_notices_regressions": {
+          "type": "object",
+          "additionalProperties": {"$ref": "#/$defs/securityNoticeRegression"}
+        },
+        "cves": {
+          "type": "object",
+          "additionalProperties": {"$ref": "#/$defs/cvePkg"}
+        }
+      },
+      "required": ["source_versions"]
+    },
+    "sourceVersion": {
+      "type": "object",
+      "properties": {
+        "binary_packages": {
+          "type": "object",
+          "additionalProperties": {"type": "string"}
+        },
+        "pocket": {"type": "string"}
+      },
+      "required": ["binary_packages", "pocket"]
+    },
+    "securityNotice": {
+      "type": "object",
+      "properties": {
+        "source_fixed_version": {"type": ["string", "null"]}
+      },
+      "required": ["source_fixed_version"]
+    },
+    "securityNoticeRegression": {
+      "type": "object",
+      "properties": {
+        "source_fixed_version": {"type": ["string", "null"]}
+      },
+      "required": ["source_fixed_version"]
+    },
+    "cvePkg": {
+      "type": "object",
+      "properties": {
+        "source_fixed_version": {"type": ["string", "null"]},
+        "ubuntu_priority": {
+          "type": "string",
+          "enum": ["untriaged", "negligible", "low", "medium", "high", "critical"]
+        },
+        "status": {
+          "type": "string",
+          "enum": ["not-vulnerable", "vulnerable", "fixed"]
+        }
+      },
+      "required": ["source_fixed_version", "ubuntu_priority", "status"]
+    },
+    "cve": {
+      "type": "object",
+      "properties": {
+        "description": {"type": "string"},
+        "published_at": {"type": "string", "format": "date-time"},
+        "notes": {
+          "type": "array",
+          "items": {"type": "string"}
+        },
+        "mitigation": {"type": ["string", "null"]},
+        "cvss_severity": {"type": ["string", "null"]},
+        "cvss_score": {"type": ["number", "null"]}
+      },
+      "required": ["description", "published_at", "cvss_severity", "cvss_score"]
+    },
+    "usn": {
+      "type": "object",
+      "properties": {
+        "description": {"type": "string"},
+        "published_at": {"type": "string", "format": "date-time"},
+        "related_cves": {
+          "type": "array",
+          "items": {"type": "string"}
+        },
+        "related_bugs": {
+          "type": "array",
+          "items": {"type": "string"}
+        }
+      },
+      "required": ["description", "published_at"]
+    }
+  },
+  "required": ["format", "published_at", "release", "packages", "security_issues"]
+}
diff --git a/test/test_json_generation.py b/test/test_json_generation.py
 new file mode 100644
index 0000000..a7eec6c
--- /dev/null
+++ b/test/test_json_generation.py
@@ -0,0 +1,837 @@
+import pytest
+from oval_lib import JSONPkgGenerator, Package, CVE, USN
+from dateutil import parser, tz
+from datetime import datetime
+
+def generate_mock_cve(pkgs, cve_id='CVE-0001-0001',
+                       description='test',
+                       priority='medium',
+                       public_date='2020-08-04 17:00:00 UTC',
+                       cvss = [{
+                           'baseScore': '8.8',
+                           'vector': '3.0/AV:N/AC:L/PR:N/UI:R/S:U/C:H/I:H/A:H',
+                           'baseSeverity': 'HIGH'
+                           }],
+                        assigned_to='foo',
+                        discovered_by='johndoe',
+                        notes=[
+                            ('test', 'note'),
+                            ('test2', 'note2')
+                            ],
+                        mitigation='test',
+                        references= 'http://example.com\nhttp://example2.com\n',
+                        bugs='http://bug1.com\nhttp://bug2.com\n',
+                        ):
+    cve = CVE(
+            number=cve_id,
+            info={
+                'Description': description,
+                'Priority': [priority],
+                'PublicDate': public_date,
+                'CVSS': cvss,
+                'Assigned-to': assigned_to,
+                'Discovered-by': discovered_by,
+                'Notes': notes,
+                'Mitigation': mitigation,
+                'References': references,
+                'Bugs': bugs
+            }
+        )
+
+    for pkg, info in pkgs.items():
+        cve.add_pkg(pkg, info['release'], info['state'], info['note'])
+
+    return cve
+
+def generate_mock_pkg(version_binaries,
+                      pkgname='foo',
+                      rel='jammy',
+                      component='main',
+                      ):
+    pkg = Package(pkgname, rel, version_binaries)
+    pkg.section = component
+
+    return pkg
+
+def generate_mock_usn(description=None, releases=None, title=None, timestamp=None, summary=None, action=None, id=None, isummary=None, cve_objs={}, pkgs_by_rel={}, lp_bugs=[]):
+    default_values = {
+        'description': 'Default Description',
+        'releases': ['jammy'],
+        'title': 'Default Title',
+        'timestamp': datetime.now().isoformat(timespec="seconds"),
+        'summary': 'Default Summary',
+        'action': 'Default Action',
+        'id': 'UNS-0001-1',
+        'isummary': 'Default ISummary'
+    }
+
+    # Update default values with provided values
+    default_values.update({
+        k: v for k, v in locals().items() if k in default_values and v is not None
+    })
+
+    # Create and return instance of MyClass with provided or default values
+    return USN(default_values, cve_objs, pkgs_by_rel, lp_bugs)
+
+
+def generate_version_binaries(source_depth, binaries, different_binary_versions=False, esm=False):
+    """
+    "0ad": {
+    "0.0.25b-1": {
+      "binaries": {
+        "0ad": {
+          "arch": [
+            "amd64",
+            "arm64",
+            "armhf"
+          ],
+          "component": "universe",
+          "version": "0.0.25b-1"
+        },
+        "0ad-dbgsym": {
+          "arch": [
+            "amd64",
+            "arm64",
+            "armhf"
+          ],
+          "component": "universe",
+          "version": "0.0.25b-1"
+        }
+      },
+      "component": "universe",
+      "pocket": "Release"
+    },
+
+    """
+    version_binaries = {}
+    for base_source_version in range(source_depth):
+        source_version = str(base_source_version)
+        if esm:
+            source_version = source_version + '+' + 'esm1'
+
+        version_binaries[source_version] = {}
+        for binary_version in range(len(binaries)):
+            binary_version_modifier = 1
+            version = str(binary_version)
+            if different_binary_versions:
+                version = version + '+' + str(binary_version_modifier)
+                binary_version_modifier += 1
+            if esm:
+                version = version + '+' + 'esm1'
+            version_binaries[source_version][version] = []
+            version_binaries[source_version][version].append(binaries[binary_version])
+
+    return version_binaries
+
+
+class EmptyJSONPkgGenerator(JSONPkgGenerator):
+    def __init__(self, releases = ['jammy']) -> None:
+        self.releases = releases
+        self.pkg_cache = None
+        self.expand = True
+        pass
+
+#### TESTS ####
+
[e-mail]("mock_cve", [
+    generate_mock_cve(pkgs=dict(), cve_id='CVE-0001-0001'),
+    generate_mock_cve(pkgs=dict(), cve_id='CVE-0001-0002', public_date='2020-08-04 17:00:00 PDT'),
+    generate_mock_cve(pkgs=dict(), cve_id='CVE-0001-0003', public_date='2020-08-04 17:00:00 PST'),
+    generate_mock_cve(pkgs=dict(), cve_id='CVE-0001-0004', cvss=None)
+])
+def test_generate_cve_info(mock_cve):
+    json_gen = EmptyJSONPkgGenerator()
+    generated_info = json_gen._generate_cve_info(mock_cve)
+    assert generated_info['description'] == mock_cve.description.strip()
+    assert generated_info['published_at'] == parser.parse(mock_cve.public_date,
+                                         tzinfos={'PDT': tz.gettz('America/Los_Angeles'),
+                                                  'PST': tz.gettz('America/Los_Angeles')
+                                                  }
+                                        ).isoformat(timespec="seconds")
+
+    notes = []
+    for note in mock_cve.notes:
+        notes.append(f"{note[0]}> {note[1]}")
+
+    assert generated_info['notes'] == notes
+    assert generated_info['mitigation'] == mock_cve.mitigation
+    assert generated_info['cvss_severity'] == (mock_cve.cvss[0]['baseSeverity'].lower() if mock_cve.cvss else None)
+    assert generated_info['cvss_score'] == (float(mock_cve.cvss[0]['baseScore']) if mock_cve.cvss else None)
+
+def test_generate_cves_info():
+    cves = {
+        'CVE-0001-0001': generate_mock_cve(pkgs=dict(), cve_id='CVE-0001-0001'),
+        'CVE-0001-0002': generate_mock_cve(pkgs=dict(), cve_id='CVE-0001-0002', public_date='2020-08-04 17:00:00 PDT'),
+        'CVE-0001-0003': generate_mock_cve(pkgs=dict(), cve_id='CVE-0001-0003', public_date='2020-08-04 17:00:00 PST'),
+        'CVE-0001-0004': generate_mock_cve(pkgs=dict(), cve_id='CVE-0001-0004', cvss=None)
+    }
+
+    json_gen = EmptyJSONPkgGenerator()
+    json_gen.cves = {'jammy': {}}
+
+    for cve_id, cve in cves.items():
+        json_gen.cves[cve_id] = cve
+
+    json_gen._init_ids('jammy')
+    generated_info_cves = json_gen._generate_cves_info()
+
+    for cve_id, generated_info in generated_info_cves.items():
+        mock_cve = cves[cve_id]
+        assert generated_info['description'] == mock_cve.description.strip()
+        assert generated_info['published_at'] == parser.parse(mock_cve.public_date,
+                                            tzinfos={'PDT': tz.gettz('America/Los_Angeles'),
+                                                    'PST': tz.gettz('America/Los_Angeles')
+                                                    }
+                                            ).isoformat(timespec="seconds")
+        assert generated_info['notes'] == mock_cve.notes
+        assert generated_info['mitigation'] == mock_cve.mitigation
+        assert generated_info['cvss_severity'] == (mock_cve.cvss[0]['baseSeverity'].lower() if mock_cve.cvss else None)
+        assert generated_info['cvss_score'] == (float
+        (mock_cve.cvss[0]['baseScore']) if mock_cve.cvss else None)
+
+def test_generate_cves_info_parents():
+    cves_jammy = {
+        'CVE-0001-0001': generate_mock_cve(pkgs=dict(), cve_id='CVE-0001-0001'),
+        'CVE-0001-0002': generate_mock_cve(pkgs=dict(), cve_id='CVE-0001-0002', public_date='2020-08-04 17:00:00 PDT'),
+    }
+
+    cves_jammy_esm_apps = {
+        'CVE-0001-0003': generate_mock_cve(pkgs=dict(), cve_id='CVE-0001-0003', public_date='2020-08-04 17:00:00 PST'),
+    }
+
+    cves_jammy_esm_infra = {
+        'CVE-0001-0004': generate_mock_cve(pkgs=dict(), cve_id='CVE-0001-0004', cvss=None)
+    }
+
+    json_gen = EmptyJSONPkgGenerator()
+    json_gen.cves = {'jammy': {}, 'esm-apps/jammy': {}, 'esm-infra/jammy': {}}
+
+    for cve_id, cve in cves_jammy.items():
+        json_gen.cves['jammy'][cve_id] = cve
+
+    for cve_id, cve in cves_jammy_esm_apps.items():
+        json_gen.cves['esm-apps/jammy'][cve_id] = cve
+
+    for cve_id, cve in cves_jammy_esm_infra.items():
+        json_gen.cves['esm-infra/jammy'][cve_id] = cve
+
+
+    json_gen._init_ids('jammy')
+    generated_info_cves = json_gen._generate_cves_info()
+    assert len(generated_info_cves) == 2
+    for cve_id in cves_jammy.keys():
+        assert cve_id in generated_info_cves.keys()
+
+    json_gen._init_ids('esm-infra/jammy')
+    json_gen.parent_releases = ['jammy']
+    generated_info_cves = json_gen._generate_cves_info()
+    assert len(generated_info_cves) == 3
+    for cve_id in list(cves_jammy.keys()) + list(cves_jammy_esm_infra.keys()):
+        assert cve_id in generated_info_cves.keys()
+
+
+    json_gen._init_ids('esm-apps/jammy')
+    json_gen.parent_releases = ['esm-infra/jammy', 'jammy']
+    generated_info_cves = json_gen._generate_cves_info()
+    assert len(generated_info_cves) == 4
+    for cve_id in list(cves_jammy.keys()) + list(cves_jammy_esm_infra.keys()) + list(cves_jammy_esm_apps.keys()):
+        assert cve_id in generated_info_cves.keys()
+
+
[e-mail]("status,note,json_status,fixed_version", [
+    ("ignored", "not for us", "vulnerable", None),
+    ("needed", "", "vulnerable", None),
+    ("needs-triage", "", "vulnerable", None),
+    ("deferred", "1-2-3", "vulnerable", None),
+    ("not-affected", "code not present", "not-vulnerable", None),
+    ("not-affected", "1.2.3", "fixed", "1.2.3"),
+    ("released", "1.2.3", "fixed", "1.2.3"),
+])
+def test_generate_cve_pkg_info(status, note, json_status, fixed_version):
+    package = generate_mock_pkg(generate_version_binaries(
+        2, ['bar', 'foo']
+    ))
+
+    cve = generate_mock_cve(pkgs=dict(), cve_id='CVE-0001-0001', priority='critical')
+    cve.add_pkg(package, 'jammy', status, note)
+    json_gen = EmptyJSONPkgGenerator()
+
+    info = json_gen._generate_cve_pkg_info(package)
+    info = info['CVE-0001-0001']
+
+    assert info['source_fixed_version'] == fixed_version
+    assert info['ubuntu_priority'] == 'critical'
+    assert info['status'] == json_status
+
[e-mail]("usn", [
+    generate_mock_usn(
+        description='this is a test description',
+        timestamp=1708590783,
+        cve_objs={'CVE-0001-0001': None, 'CVE-0001-0002': None},
+        lp_bugs=['http://bug1.com', 'http://bug2.com']
+    ),
+    generate_mock_usn(
+        description='this is a test description',
+        timestamp=1708590783,
+        lp_bugs=['http://bug1.com', 'http://bug2.com']
+    ),
+    generate_mock_usn(
+        description='this is a test description',
+        timestamp=1708590783,
+        cve_objs={'CVE-0001-0001': None, 'CVE-0001-0002': None}
+    ),
+])
+def test_generate_usn_info(usn):
+    # 1708590783 is ~2024-02-22T09:33:03+00:00
+    date = datetime.fromtimestamp(1708590783).isoformat(timespec="seconds")
+    json_gen = EmptyJSONPkgGenerator()
+
+    info = json_gen._generate_usn_info(usn)
+    assert info['description'] == usn.description
+    assert info['published_at'] == date
+    assert info['related_cves'] == list(usn.cves.keys())
+    assert info['related_launchpad_bugs'] == usn.lp_bugs
+
+def test_generate_usns_info():
+    date = datetime.fromtimestamp(1708590783).isoformat(timespec="seconds")
+
+    usns = [generate_mock_usn(
+        id='1000-1',
+        description='this is a test description',
+        timestamp=1708590783,
+        cve_objs={'CVE-0001-0001': None, 'CVE-0001-0002': None},
+        lp_bugs=['http://bug1.com', 'http://bug2.com']
+    ),
+    generate_mock_usn(
+        id='1000-2',
+        description='this is a test description',
+        timestamp=1708590783,
+        lp_bugs=['http://bug1.com', 'http://bug2.com']
+    ),
+    generate_mock_usn(
+        id='1000-3',
+        description='this is a test description',
+        timestamp=1708590783,
+        cve_objs={'CVE-0001-0001': None, 'CVE-0001-0002': None}
+    )]
+
+    wrong_release_usn = generate_mock_usn(
+        id='2000-1',
+        description='this is a test description',
+        timestamp=1708590783,
+        releases=['bionic'],
+        cve_objs={'CVE-0001-0001': None, 'CVE-0001-0002': None}
+    )
+
+    final_usns  = usns.copy()
+    final_usns.append(wrong_release_usn)
+    usns_dict = dict(map(lambda usn: (usn.id, usn), final_usns))
+    json_gen = EmptyJSONPkgGenerator()
+    json_gen.usns = usns_dict
+    json_gen._init_ids('jammy')
+    info = json_gen._generate_usns_info()
+
+    assert 'USN-2000-1' not in info
+
+    for usn in usns:
+        assert f'USN-{usn.id}' in info
+        assert info[f'USN-{usn.id}']['description'] == usn.description
+        assert info[f'USN-{usn.id}']['published_at'] == date
+        assert info[f'USN-{usn.id}']['related_cves'] == list(usn.cves.keys())
+        assert info[f'USN-{usn.id}']['related_launchpad_bugs'] == usn.lp_bugs
+
+def test_generate_usns_info_parents():
+    jammy_usns = [generate_mock_usn(
+        id='1000-1',
+        description='this is a test description',
+        timestamp=1708590783,
+        cve_objs={'CVE-0001-0001': None, 'CVE-0001-0002': None},
+        lp_bugs=['http://bug1.com', 'http://bug2.com']
+    ),
+    generate_mock_usn(
+        id='1000-2',
+        description='this is a test description',
+        timestamp=1708590783,
+        lp_bugs=['http://bug1.com', 'http://bug2.com']
+    ),
+    generate_mock_usn(
+        id='1000-3',
+        description='this is a test description',
+        timestamp=1708590783,
+        cve_objs={'CVE-0001-0001': None, 'CVE-0001-0002': None}
+    )]
+
+    jammy_infra_usns = [
+        generate_mock_usn(
+            id='2000-1',
+            description='this is a test description',
+            timestamp=1708590783,
+            releases=['esm-infra/jammy'],
+            cve_objs={'CVE-0001-0001': None, 'CVE-0001-0002': None}
+        )
+    ]
+
+    jammy_apps_usns = [
+        generate_mock_usn(
+            id='3000-1',
+            description='this is a test description',
+            timestamp=1708590783,
+            releases=['esm-apps/jammy'],
+            cve_objs={'CVE-0001-0001': None, 'CVE-0001-0002': None}
+        )
+    ]
+
+    final_usns  = jammy_usns + jammy_infra_usns + jammy_apps_usns
+    usns_dict = dict(map(lambda usn: (usn.id, usn), final_usns))
+    json_gen = EmptyJSONPkgGenerator()
+    json_gen.usns = usns_dict
+    json_gen._init_ids('jammy')
+    info = json_gen._generate_usns_info()
+    assert len(info.keys()) == 3
+    for usn in jammy_usns:
+        assert f'USN-{usn.id}' in info
+
+    json_gen._init_ids('esm-infra/jammy')
+    json_gen.parent_releases = ['jammy']
+    info = json_gen._generate_usns_info()
+    assert len(info.keys()) == 4
+    for usn in jammy_usns + jammy_infra_usns:
+        assert f'USN-{usn.id}' in info
+
+    json_gen._init_ids('esm-apps/jammy')
+    json_gen.parent_releases = ['esm-infra/jammy', 'jammy']
+    info = json_gen._generate_usns_info()
+    assert len(info.keys()) == 5
+    for usn in jammy_usns + jammy_infra_usns + jammy_apps_usns:
+        assert f'USN-{usn.id}' in info
+
+def test_generate_usn_pkg_info():
+    releases_1 = {
+        'jammy': {
+            'sources': {
+                'foo': {
+                    'version': '1.0.0'
+                }
+            }
+        }
+    }
+
+    releases_2 = {
+        'jammy': {
+            'sources': {
+                'foo': {
+                    'version': '2.0.0'
+                }
+            }
+        }
+    }
+
+    pkg = generate_mock_pkg(generate_version_binaries(
+        1, ['bar', 'foo']
+    ))
+
+    pkg_not = generate_mock_pkg(generate_version_binaries(
+        1, ['bar', 'foo']
+    ), rel='bionic')
+
+    usn = generate_mock_usn(
+        id='1000-1',
+        description='this is a test description',
+        timestamp=1708590783,
+        cve_objs={'CVE-0001-0001': None, 'CVE-0001-0002': None},
+        lp_bugs=['http://bug1.com', 'http://bug2.com'],
+        pkgs_by_rel={'jammy': {'foo': pkg}},
+        releases=releases_1
+    )
+
+    usn_regression = generate_mock_usn(
+        id='1000-2',
+        title='Regression',
+        description='this is a test description',
+        timestamp=1708590783,
+        cve_objs={'CVE-0001-0001': None, 'CVE-0001-0002': None},
+        lp_bugs=['http://bug1.com', 'http://bug2.com'],
+        pkgs_by_rel={'jammy': {'foo': pkg}},
+        releases=releases_2
+    )
+
+    json_gen = EmptyJSONPkgGenerator()
+    usns_dict = dict(map(lambda usn: (usn.id, usn), [usn, usn_regression]))
+    json_gen.usns = usns_dict
+
+    usn_info, usn_reg_info = json_gen._generate_usn_pkg_info(pkg)
+
+    assert 'USN-1000-1' in usn_info
+    assert 'USN-1000-2' in usn_reg_info
+    assert 'USN-1000-1' not in usn_reg_info
+    assert 'USN-1000-2' not in usn_info
+
+    assert usn_info['USN-1000-1']['source_fixed_version'] == '1.0.0'
+    assert usn_reg_info['USN-1000-2']['source_fixed_version'] == '2.0.0'
+
+    usn_info, usn_reg_info = json_gen._generate_usn_pkg_info(pkg_not)
+
+    assert 'USN-1000-1' not in usn_info
+    assert 'USN-1000-2' not in usn_reg_info
+    assert 'USN-1000-1' not in usn_reg_info
+    assert 'USN-1000-2' not in usn_info
+
+
[e-mail]("package,pocket", [
+    (generate_mock_pkg(generate_version_binaries(
+        1, ['bar', 'foo']
+    )), "security"),
+    (generate_mock_pkg(generate_version_binaries(
+        2, ['bar', 'foo'], different_binary_versions=True
+    )), "release"),
+    (generate_mock_pkg(generate_version_binaries(
+        3, ['bar', 'foo'], esm=True,
+    )), "esm-infra"),
+    (generate_mock_pkg(generate_version_binaries(
+        4, ['bar', 'foo'], esm=True
+    ), component='universe'), "esm-apps"),
+])
+def test_generate_package_source_info(mocker, package, pocket):
+    mocker.patch('oval_lib.get_pocket', return_value=('',pocket))
+    json_gen = EmptyJSONPkgGenerator()
+    info = dict()
+    json_gen._generate_package_source_info(package, info)
+
+    for source_version in package.versions_binaries:
+        assert source_version in info
+        for binary_version in package.versions_binaries[source_version]:
+            for binary in package.versions_binaries[source_version][binary_version]:
+                assert binary in info[source_version]['binary_packages']
+                assert info[source_version]['binary_packages'][binary] == binary_version
+
+        assert info[source_version]['pocket'] == pocket
+
+def test_generate_package_info(mocker):
+    mocker.patch('oval_lib.get_pocket', return_value=('','release'))
+
+    releases_1 = {
+        'jammy': {
+            'sources': {
+                'foo': {
+                    'version': '1.0.0'
+                }
+            }
+        }
+    }
+
+    releases_2 = {
+        'bionic': {
+            'sources': {
+                'bar': {
+                    'version': '2.0.0'
+                }
+            }
+        }
+    }
+
+    pkg = generate_mock_pkg(generate_version_binaries(
+        1, ['bar', 'foo']
+    ))
+
+    pkg2 = generate_mock_pkg(generate_version_binaries(
+        2, ['bar', 'foo']
+    ), rel='bionic', pkgname='bar')
+
+    usn = generate_mock_usn(
+        id='1000-1',
+        description='this is a test description',
+        timestamp=1708590783,
+        cve_objs={'CVE-0001-0001': None, 'CVE-0001-0002': None},
+        lp_bugs=['http://bug1.com', 'http://bug2.com'],
+        pkgs_by_rel={'jammy': {'foo': pkg}},
+        releases=releases_1
+    )
+
+    usn_regression = generate_mock_usn(
+        id='1000-2',
+        title='Regression',
+        description='this is a test description',
+        timestamp=1708590783,
+        cve_objs={'CVE-0001-0001': None, 'CVE-0001-0002': None},
+        lp_bugs=['http://bug1.com', 'http://bug2.com'],
+        pkgs_by_rel={'bionic': {'bar': pkg2}},
+        releases=releases_2
+    )
+
+
+    cve = generate_mock_cve(pkgs=dict(), cve_id='CVE-0001-0001', priority='critical')
+    cve2 = generate_mock_cve(pkgs=dict(), cve_id='CVE-0001-0002', priority='medium')
+
+    cve.add_pkg(pkg, 'jammy', 'needed', '')
+    cve2.add_pkg(pkg2, 'bionic', 'not-affected', '1')
+    json_gen = EmptyJSONPkgGenerator()
+    usns_dict = dict(map(lambda usn: (usn.id, usn), [usn, usn_regression]))
+    json_gen.usns = usns_dict
+    json_gen.cves = dict()
+    json_gen.cves['CVE-0001-0001'] = cve
+    json_gen.cves['CVE-0001-0002'] = cve2
+
+    json_gen.packages = {}
+    json_gen.packages['jammy'] = {
+        'foo': pkg
+    }
+
+    json_gen.packages['bionic'] = {
+        'bar': pkg2
+    }
+
+    json_gen._init_ids('jammy')
+    info = json_gen._generate_package_info(pkg)
+    assert '0' in info['source_versions']
+    assert 'USN-1000-1' in info['ubuntu_security_notices']
+    assert len(info['ubuntu_security_notices_regressions']) == 0
+    assert 'CVE-0001-0001' in info['cves']
+    assert 'CVE-0001-0002' not in info['cves']
+
+    json_gen._init_ids('bionic')
+    info = json_gen._generate_package_info(pkg2)
+    assert '0' in info['source_versions']
+    assert 'USN-1000-2' in info['ubuntu_security_notices_regressions']
+    assert len(info['ubuntu_security_notices']) == 0
+    assert 'CVE-0001-0002' in info['cves']
+    assert 'CVE-0001-0001' not in info['cves']
+
+def test_generate_packages_info(mocker):
+    mocker.patch('oval_lib.get_pocket', return_value=('','release'))
+
+    releases_1 = {
+        'jammy': {
+            'sources': {
+                'foo': {
+                    'version': '1.0.0'
+                }
+            }
+        }
+    }
+
+    releases_2 = {
+        'bionic': {
+            'sources': {
+                'bar': {
+                    'version': '2.0.0'
+                }
+            }
+        }
+    }
+
+    pkg = generate_mock_pkg(generate_version_binaries(
+        1, ['bar', 'foo']
+    ))
+
+    pkg2 = generate_mock_pkg(generate_version_binaries(
+        2, ['bar', 'foo']
+    ), rel='bionic', pkgname='bar')
+
+    usn = generate_mock_usn(
+        id='1000-1',
+        description='this is a test description',
+        timestamp=1708590783,
+        cve_objs={'CVE-0001-0001': None, 'CVE-0001-0002': None},
+        lp_bugs=['http://bug1.com', 'http://bug2.com'],
+        pkgs_by_rel={'jammy': {'foo': pkg}},
+        releases=releases_1
+    )
+
+    usn_regression = generate_mock_usn(
+        id='1000-2',
+        title='Regression',
+        description='this is a test description',
+        timestamp=1708590783,
+        cve_objs={'CVE-0001-0001': None, 'CVE-0001-0002': None},
+        lp_bugs=['http://bug1.com', 'http://bug2.com'],
+        pkgs_by_rel={'bionic': {'bar': pkg2}},
+        releases=releases_2
+    )
+
+
+    cve = generate_mock_cve(pkgs=dict(), cve_id='CVE-0001-0001', priority='critical')
+    cve2 = generate_mock_cve(pkgs=dict(), cve_id='CVE-0001-0002', priority='medium')
+
+    cve.add_pkg(pkg, 'jammy', 'needed', '')
+    cve2.add_pkg(pkg2, 'bionic', 'not-affected', '1')
+    json_gen = EmptyJSONPkgGenerator()
+    usns_dict = dict(map(lambda usn: (usn.id, usn), [usn, usn_regression]))
+    json_gen.usns = usns_dict
+    json_gen.cves = dict()
+    json_gen.cves['CVE-0001-0001'] = cve
+    json_gen.cves['CVE-0001-0002'] = cve2
+    json_gen.packages = {
+        'jammy': {
+            'foo': pkg
+        },
+        'bionic': {
+            'bar': pkg2
+        }
+    }
+
+    json_gen._init_ids('jammy')
+    info = json_gen._generate_packages_info()
+    assert 'foo' in info
+    assert 'bar' not in info
+
+    json_gen._init_ids('bionic')
+    info = json_gen._generate_packages_info()
+    assert 'foo' not in info
+    assert 'bar' in info
+
+def test_generate_packages_info_parents(mocker):
+    mocker.patch('oval_lib.get_pocket', return_value=('','release'))
+
+    releases_jammy = {
+        'jammy': {
+            'sources': {
+                'foo': {
+                    'version': '1.0.0'
+                }
+            }
+        }
+    }
+
+    releases_infra_jammy = {
+        'esm-infra/jammy': {
+            'sources': {
+                'bar': {
+                    'version': '2.0.0'
+                }
+            }
+        }
+    }
+
+    releases_apps_jammy = {
+        'esm-apps/jammy': {
+            'sources': {
+                'dodo': {
+                    'version': '3.0.0'
+                }
+            }
+        }
+    }
+
+    pkg = generate_mock_pkg(generate_version_binaries(
+        1, ['bar', 'foo']
+    ))
+
+    pkg2 = generate_mock_pkg(generate_version_binaries(
+        2, ['bar', 'fofa']
+    ), rel='esm-infra/jammy', pkgname='bar')
+
+
+    pkg3 = generate_mock_pkg(generate_version_binaries(
+        2, ['dodo', 'foo']
+    ), rel='esm-apps/jammy', pkgname='dodo')
+
+    pkg4 = generate_mock_pkg(generate_version_binaries(
+        3, ['bar', 'foo']
+    ), rel='esm-apps/jammy')
+
+
+    usn = generate_mock_usn(
+        id='1000-1',
+        description='this is a test description',
+        timestamp=1708590783,
+        cve_objs={'CVE-0001-0001': None, 'CVE-0001-0002': None},
+        lp_bugs=['http://bug1.com', 'http://bug2.com'],
+        pkgs_by_rel={'jammy': {'foo': pkg}},
+        releases=releases_jammy
+    )
+
+    usn_infra = generate_mock_usn(
+        id='2000-1',
+        title='Infra',
+        description='this is a test description',
+        timestamp=1708590783,
+        cve_objs={'CVE-0001-0001': None, 'CVE-0001-0002': None},
+        lp_bugs=['http://bug1.com', 'http://bug2.com'],
+        pkgs_by_rel={'esm-infra/jammy': {'bar': pkg2}},
+        releases=releases_infra_jammy
+    )
+
+    usn_apps = generate_mock_usn(
+        id='3000-1',
+        title='Apps',
+        description='this is a test description',
+        timestamp=1708590783,
+        cve_objs={'CVE-0001-0001': None, 'CVE-0001-0002': None},
+        lp_bugs=['http://bug1.com', 'http://bug2.com'],
+        pkgs_by_rel={'esm-apps/jammy': {'dodo': pkg3}},
+        releases=releases_apps_jammy
+    )
+
+    cve = generate_mock_cve(pkgs=dict(), cve_id='CVE-0001-0001', priority='critical')
+    cve2 = generate_mock_cve(pkgs=dict(), cve_id='CVE-0001-0002', priority='medium')
+    cve3 = generate_mock_cve(pkgs=dict(), cve_id='CVE-0001-0003', priority='medium')
+
+    cve.add_pkg(pkg, 'jammy', 'needed', '')
+    cve.add_pkg(pkg4, 'esm-apps/jammy', 'not-affected', '4')
+    cve2.add_pkg(pkg2, 'esm-infra/jammy', 'not-affected', '1')
+    cve3.add_pkg(pkg3, 'esm-apps/jammy', 'released', '2')
+
+    json_gen = EmptyJSONPkgGenerator()
+    usns_dict = dict(map(lambda usn: (usn.id, usn), [usn, usn_infra, usn_apps]))
+    json_gen.usns = usns_dict
+    json_gen.cves = {'jammy': {}, 'esm-apps/jammy': {}, 'esm-infra/jammy': {}}
+    json_gen.cves['jammy']['CVE-0001-0001'] = cve
+    json_gen.cves['esm-infra/jammy']['CVE-0001-0002'] = cve2
+    json_gen.cves['esm-apps/jammy']['CVE-0001-0003'] = cve3
+
+    json_gen.packages = {
+        'jammy': {
+            'foo': pkg
+        },
+        'esm-infra/jammy': {
+            'bar': pkg2
+        },
+        'esm-apps/jammy': {
+            'dodo': pkg3,
+            'foo': pkg4
+        }
+    }
+
+    json_gen._init_ids('jammy')
+    info = json_gen._generate_packages_info()
+    assert 'foo' in info
+    assert 'bar' not in info
+    assert 'dodo' not in info
+    assert info['foo']['cves']['CVE-0001-0001']['status'] == 'vulnerable'
+
+    json_gen._init_ids('esm-infra/jammy')
+    json_gen.parent_releases = ['jammy']
+    info = json_gen._generate_packages_info()
+    assert 'foo' in info
+    assert 'bar' in info
+    assert 'dodo' not in info
+    assert info['foo']['cves']['CVE-0001-0001']['status'] == 'vulnerable'
+    assert info['bar']['cves']['CVE-0001-0002']['status'] == 'fixed'
+    assert info['bar']['ubuntu_security_notices']['USN-2000-1']['source_fixed_version'] == '2.0.0'
+
+    json_gen._init_ids('esm-apps/jammy')
+    json_gen.parent_releases = ['esm-infra/jammy', 'jammy']
+    info = json_gen._generate_packages_info()
+    assert 'foo' in info
+    assert 'bar' in info
+    assert 'dodo' in info
+    assert info['foo']['cves']['CVE-0001-0001']['status'] == 'fixed'
+    assert info['bar']['cves']['CVE-0001-0002']['status'] == 'fixed'
+    assert info['dodo']['cves']['CVE-0001-0003']['status'] == 'fixed'
+    assert info['foo']['cves']['CVE-0001-0001']['source_fixed_version'] == '4'
+    assert info['dodo']['ubuntu_security_notices']['USN-3000-1']['source_fixed_version'] == '3.0.0'
+
[e-mail]("expand,release,filename", [
+    (False, 'esm-apps/bionic', 'com.ubuntu.bionic.pkg.json'),
+    (True, 'esm-apps/bionic', 'com.ubuntu.esm-apps_bionic.pkg.json'),
+    (False, 'esm-infra/bionic', 'com.ubuntu.bionic.pkg.json'),
+    (True, 'esm-infra/bionic', 'com.ubuntu.esm-infra_bionic.pkg.json'),
+    (False, 'bionic', 'com.ubuntu.bionic.pkg.json'),
+    (True, 'bionic', 'com.ubuntu.bionic.pkg.json')
+])
+def test_expand(expand, release, filename):
+    json_gen = EmptyJSONPkgGenerator()
+    json_gen.expand = expand
+    json_gen._init_ids(release)
+
+    assert json_gen.output_file == filename