Merge ~jslarraz/review-tools:container-size-checks into review-tools:master

Proposed by Jorge Sancho Larraz
Status: Merged
Merged at revision: 97ce21fc1803c0e84e77f866486944fb8e745ddf
Proposed branch: ~jslarraz/review-tools:container-size-checks
Merge into: review-tools:master
Diff against target: 909 lines (+481/-160)
9 files modified
reviewtools/available.py (+7/-2)
reviewtools/common.py (+0/-142)
reviewtools/containers/base_container.py (+149/-4)
reviewtools/containers/snap_container.py (+44/-0)
reviewtools/containers/squashfs_container.py (+111/-0)
reviewtools/sr_common.py (+2/-2)
reviewtools/sr_tests.py (+1/-2)
reviewtools/tests/containers/test_base_container.py (+94/-8)
reviewtools/tests/containers/test_squashfs_container.py (+73/-0)
Reviewer Review Type Date Requested Status
Alex Murray Approve
Review via email: mp+466463@code.launchpad.net

Commit message

Add size check functions to base_container

To post a comment you must log in.
Revision history for this message
Alex Murray (alexmurray) wrote :

Can you provide some more rationale for this change?

review: Needs Information
Revision history for this message
Jorge Sancho Larraz (jslarraz) wrote (last edit ):

So in a general case, we may want to set up the valid size limits for the container file and for the unpacked content. It is already true for snaps.

For snaps we are checking the size via a set of functions in common.py (see inline comments), however this functionality is intrinsic to the container and only makes sense in the context of it. Thus, I think it should be part of the base_container class.
The specific values for {min,max}_{,un}packed_size limits will be defined in each subclass (such as SnapContainer, ComponentContainer), but the functionality will be the same

There is some interaction among check_container_size and unpack, so I'm leaving the original functionality in comomon.py for now and it will removed in a follow up MR where unpack funtionality will be moved to the corresponding container class

Revision history for this message
Alex Murray (alexmurray) wrote :

Thanks for the pointers but perhaps I am missing something, but I don't see any users of these new methods - as such, perhaps it would help to also include the changes required to port the existing size checking / limits code to use this new interface?

Revision history for this message
Jorge Sancho Larraz (jslarraz) wrote :

Updated to move unpack to SquashfsContainer, which uses size checks

Revision history for this message
Alex Murray (alexmurray) wrote :

In general this looks really good - just a few minor things to address if you can thanks @jslarraz

review: Needs Fixing
Revision history for this message
Jorge Sancho Larraz (jslarraz) wrote :

Comments addressed in https://git.launchpad.net/~jslarraz/review-tools/commit/?id=de2de8651408f287c5c1b0a27787e2e3511b6057

Fix for container size not being checked in SnapContainer in https://git.launchpad.net/~jslarraz/review-tools/commit/?id=0cd400e35ad2b1637c02fd5463e5ca9bfaf559d9

> "I assume all containers would want to check their format - as such, should this be done in the BaseContainer constructor instead?"

I fully agree with you but it will require some non-obvious changes in test_base_container, thus I would prefer to do it in a followup MR

Revision history for this message
Alex Murray (alexmurray) wrote :

> I fully agree with you but it will require some non-obvious changes in test_base_container, thus I would prefer to do it in a followup MR

Sounds good - thanks @jslarraz - LGTM!

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
diff --git a/reviewtools/available.py b/reviewtools/available.py
index eafa366..5b0dd3c 100755
--- a/reviewtools/available.py
+++ b/reviewtools/available.py
@@ -19,6 +19,7 @@ import shutil
19import tempfile19import tempfile
2020
21from reviewtools.common import (21from reviewtools.common import (
22 error,
22 debug,23 debug,
23 warn,24 warn,
24 open_file_write,25 open_file_write,
@@ -26,7 +27,6 @@ from reviewtools.common import (
26 MKDTEMP_PREFIX,27 MKDTEMP_PREFIX,
27 _add_error, # make a class28 _add_error, # make a class
28 get_rock_manifest,29 get_rock_manifest,
29 get_snap_manifest,
30)30)
31import reviewtools.email as email31import reviewtools.email as email
3232
@@ -37,6 +37,7 @@ from reviewtools.store import (
37 get_faked_build_and_stage_packages,37 get_faked_build_and_stage_packages,
38)38)
39from reviewtools.usn import read_usn_db39from reviewtools.usn import read_usn_db
40from reviewtools.containers.snap_container import SnapContainer, ContainerException
4041
41email_update_required_text = """A scan of this %s shows that it was built with packages from the Ubuntu42email_update_required_text = """A scan of this %s shows that it was built with packages from the Ubuntu
42archive that have since received security updates. """43archive that have since received security updates. """
@@ -484,7 +485,11 @@ def scan_store(
484def scan_snap(secnot_db_fn, snap_fn, with_cves=False, ignore_pockets=list()):485def scan_snap(secnot_db_fn, snap_fn, with_cves=False, ignore_pockets=list()):
485 """Scan snap for packages with security notices"""486 """Scan snap for packages with security notices"""
486 out = ""487 out = ""
487 (man, dpkg) = get_snap_manifest(snap_fn)488 try:
489 snap = SnapContainer(snap_fn)
490 (man, dpkg) = snap.get_snap_manifest()
491 except ContainerException as e:
492 error(str(e))
488 man = get_faked_build_and_stage_packages(man)493 man = get_faked_build_and_stage_packages(man)
489494
490 # Use dpkg.list when the snap ships it in a spot we honor. This is limited495 # Use dpkg.list when the snap ships it in a spot we honor. This is limited
diff --git a/reviewtools/common.py b/reviewtools/common.py
index e2e80ff..d2c15b8 100644
--- a/reviewtools/common.py
+++ b/reviewtools/common.py
@@ -424,30 +424,6 @@ def cmdIgnoreErrorStrings(command, ignoreErrorStrings):
424 return [rc, out]424 return [rc, out]
425425
426426
427def _unpack_cmd(cmd_args, d, dest):
428 """Low level unpack helper"""
429 curdir = os.getcwd()
430 os.chdir(d)
431
432 (rc, out) = cmdIgnoreErrorStrings(cmd_args, UNSQUASHFS_IGNORED_ERRORS)
433
434 os.chdir(curdir)
435
436 if rc != 0:
437 if os.path.isdir(d):
438 recursive_rm(d)
439 error("unpacking failed with '%d':\n%s" % (rc, out))
440
441 if dest is None:
442 dest = d
443 else:
444 # Recursively move unpacked content to original dir, keeping
445 # permissions
446 move_dir_content(d, dest)
447
448 return dest
449
450
451# The original directory might have restrictive permissions, so save427# The original directory might have restrictive permissions, so save
452# them off, chmod the dir, do the move and reapply428# them off, chmod the dir, do the move and reapply
453def move_dir_content(d, dest):429def move_dir_content(d, dest):
@@ -621,24 +597,6 @@ def unsquashfs_lln_parse(lln_out):
621 return hdr, entries597 return hdr, entries
622598
623599
624def _calculate_snap_unsquashfs_uncompressed_size(snap_pkg):
625 """Calculate size of the uncompressed snap"""
626 (rc, out) = cmd(["unsquashfs", "-lln", snap_pkg])
627 if rc != 0:
628 error("unsquashfs -lln '%s' failed: %s" % (snap_pkg, out))
629
630 size = 0
631 for line in out.splitlines():
632 if not line.startswith("-"): # skip non-regular files
633 continue
634 try:
635 size += int(line.split()[2])
636 except ValueError: # skip non-numbers
637 continue
638
639 return size
640
641
642def _calculate_rock_untar_uncompressed_size(rock_pkg):600def _calculate_rock_untar_uncompressed_size(rock_pkg):
643 """Calculate size of the uncompressed tar"""601 """Calculate size of the uncompressed tar"""
644 size = 0602 size = 0
@@ -656,34 +614,6 @@ def unsquashfs_supports_ignore_errors():
656 return "-ig[nore-errors]" in out614 return "-ig[nore-errors]" in out
657615
658616
659def _unpack_snap_squashfs(snap_pkg, dest, items=[]):
660 """Unpack a squashfs based snap package to dest"""
661 size = _calculate_snap_unsquashfs_uncompressed_size(snap_pkg)
662
663 snap_max_size = MAX_UNCOMPRESSED_SIZE * 1024 * 1024 * 1024
664 valid_size, error_msg = is_pkg_uncompressed_size_valid(
665 snap_max_size, size, snap_pkg
666 )
667 if valid_size:
668 global MKDTEMP_PREFIX
669 global MKDTEMP_DIR
670 d = tempfile.mkdtemp(prefix=MKDTEMP_PREFIX, dir=MKDTEMP_DIR)
671 cmd = ["unsquashfs", "-no-progress", "-f", "-d", d]
672
673 # If unsquashfs supports it, pass "-ignore-errors -quiet"
674 if unsquashfs_supports_ignore_errors():
675 cmd.append("-ignore-errors")
676 cmd.append("-quiet")
677
678 cmd.append(os.path.abspath(snap_pkg))
679
680 if len(items) != 0:
681 cmd += items
682 return _unpack_cmd(cmd, d, dest)
683 else:
684 error(error_msg)
685
686
687def is_pkg_uncompressed_size_valid(pkg_max_size, size, pkg):617def is_pkg_uncompressed_size_valid(pkg_max_size, size, pkg):
688 st = os.statvfs(pkg)618 st = os.statvfs(pkg)
689 avail = st.f_bsize * st.f_bavail * 0.9 # 90% of available space619 avail = st.f_bsize * st.f_bavail * 0.9 # 90% of available space
@@ -751,21 +681,6 @@ def _unpack_rock_tar(rock_pkg, dest):
751 error(error_msg)681 error(error_msg)
752682
753683
754def unpack_pkg(fn, dest=None, items=[]):
755 """Unpack package"""
756 pkg = check_fn(fn)
757 check_dir(dest)
758
759 # Limit the maximimum size of the package
760 check_max_pkg_size(pkg)
761
762 # check if its a squashfs based snap
763 if is_squashfs(pkg):
764 return _unpack_snap_squashfs(fn, dest, items)
765
766 error("Unsupported package format (not squashfs)")
767
768
769def check_dir(dest):684def check_dir(dest):
770 if dest is not None and os.path.exists(dest):685 if dest is not None and os.path.exists(dest):
771 error("'%s' exists. Aborting." % dest)686 error("'%s' exists. Aborting." % dest)
@@ -790,13 +705,6 @@ def check_max_pkg_size(pkg):
790 )705 )
791706
792707
793def is_squashfs(filename):
794 """Return true if the given filename as a squashfs header"""
795 with open(filename, "rb") as f:
796 header = f.read(10)
797 return header.startswith(b"hsqs")
798
799
800def unpack_rock(fn, dest=None):708def unpack_rock(fn, dest=None):
801 """Unpack rock"""709 """Unpack rock"""
802 pkg = check_fn(fn)710 pkg = check_fn(fn)
@@ -1078,56 +986,6 @@ def read_file_as_json_dict(fn):
1078 return raw986 return raw
1079987
1080988
1081def get_snap_manifest(fn):
1082 if "SNAP_USER_COMMON" in os.environ and os.path.exists(
1083 os.environ["SNAP_USER_COMMON"]
1084 ):
1085 MKDTEMP_DIR = os.environ["SNAP_USER_COMMON"]
1086 else:
1087 MKDTEMP_DIR = tempfile.gettempdir()
1088
1089 man = "snap/manifest.yaml"
1090 os_dpkg = "usr/share/snappy/dpkg.list"
1091 snap_dpkg = "snap/dpkg.list"
1092 # unpack_pkg() fails if this exists, so this is safe
1093 dir = tempfile.mktemp(prefix=MKDTEMP_PREFIX, dir=MKDTEMP_DIR)
1094 unpack_pkg(fn, dir, [man, os_dpkg, snap_dpkg])
1095
1096 man_fn = os.path.join(dir, man)
1097 if not os.path.isfile(man_fn):
1098 recursive_rm(dir)
1099 error("%s not in %s" % (man, fn))
1100
1101 with open_file_read(man_fn) as fd:
1102 try:
1103 man_yaml = yaml.safe_load(fd)
1104 except Exception:
1105 recursive_rm(dir)
1106 error("Could not load %s. Is it properly formatted?" % man)
1107
1108 os_dpkg_fn = os.path.join(dir, os_dpkg)
1109 snap_dpkg_fn = os.path.join(dir, snap_dpkg)
1110 dpkg_list = None
1111 if os.path.isfile(os_dpkg_fn):
1112 with open_file_read(os_dpkg_fn) as fd:
1113 try:
1114 dpkg_list = fd.readlines()
1115 except Exception:
1116 recursive_rm(dir)
1117 error("Could not load %s. Is it properly formatted?" % os_dpkg)
1118 elif os.path.isfile(snap_dpkg_fn):
1119 with open_file_read(snap_dpkg_fn) as fd:
1120 try:
1121 dpkg_list = fd.readlines()
1122 except Exception:
1123 recursive_rm(dir)
1124 error("Could not load %s. Is it properly formatted?" % snap_dpkg)
1125
1126 recursive_rm(dir)
1127
1128 return (man_yaml, dpkg_list)
1129
1130
1131def get_rock_manifest(fn):989def get_rock_manifest(fn):
1132 if "SNAP_USER_COMMON" in os.environ and os.path.exists(990 if "SNAP_USER_COMMON" in os.environ and os.path.exists(
1133 os.environ["SNAP_USER_COMMON"]991 os.environ["SNAP_USER_COMMON"]
diff --git a/reviewtools/containers/base_container.py b/reviewtools/containers/base_container.py
index 9e6f104..15d4d7e 100644
--- a/reviewtools/containers/base_container.py
+++ b/reviewtools/containers/base_container.py
@@ -8,11 +8,19 @@ from reviewtools.common import (
8 MKDTEMP_PREFIX,8 MKDTEMP_PREFIX,
9 MKDTEMP_DIR,9 MKDTEMP_DIR,
10 debug,10 debug,
11 unpack_pkg,
12 recursive_rm,11 recursive_rm,
13)12)
1413
1514
15# Add base_container private attributes to documentation
16__pdoc__ = {
17 "BaseContainer._min_packed_size": True,
18 "BaseContainer._max_packed_size": True,
19 "BaseContainer._min_unpacked_size": True,
20 "BaseContainer._max_unpacked_size": True,
21}
22
23
16class ContainerException(Exception):24class ContainerException(Exception):
17 """This class represents exceptions"""25 """This class represents exceptions"""
1826
@@ -42,6 +50,30 @@ class BaseContainer:
42 "application/x-pie-executable",50 "application/x-pie-executable",
43 ]51 ]
4452
53 _min_packed_size = None
54 """
55 Minimum size of the container file in bytes. It will be used to check the
56 container size during initialization.
57 """
58
59 _max_packed_size = None
60 """
61 Maximum size of the container file in bytes. It will be used to check the
62 container size during initialization.
63 """
64
65 _min_unpacked_size = None
66 """
67 Minimum size of the unpacked directory in bytes. It will be used to check the
68 container size during initialization.
69 """
70
71 _max_unpacked_size = None
72 """
73 Maximum size of the unpacked directory in bytes. It will be used to check the
74 container size during initialization.
75 """
76
45 _unpack_dir = None77 _unpack_dir = None
46 _files = None78 _files = None
47 _bin_files = None79 _bin_files = None
@@ -59,6 +91,9 @@ class BaseContainer:
59 else:91 else:
60 raise ContainerException("Could not find '%s'" % fn)92 raise ContainerException("Could not find '%s'" % fn)
6193
94 # Check max/min size if defined
95 self.check_container_size()
96
62 @property97 @property
63 def unpack_dir(self) -> str:98 def unpack_dir(self) -> str:
64 """99 """
@@ -67,8 +102,7 @@ class BaseContainer:
67 will be returned.102 will be returned.
68 """103 """
69 if self._unpack_dir is None:104 if self._unpack_dir is None:
70 self._unpack_dir = ".".join(self.filename.split(".")[:-1])105 self._unpack_dir = self.unpack()
71 unpack_pkg(self.filename, dest=self._unpack_dir)
72 return self._unpack_dir106 return self._unpack_dir
73107
74 @property108 @property
@@ -104,6 +138,117 @@ class BaseContainer:
104 if os.path.exists(self.tmp_dir):138 if os.path.exists(self.tmp_dir):
105 recursive_rm(self.tmp_dir)139 recursive_rm(self.tmp_dir)
106140
141 def check_container_format(self):
142 """
143 Checks that the container file has the expected format.
144
145 Raises:
146 NotImplementedError: Calling this method will raise an exception as
147 format is specific to the container type. This method needs to be
148 implemented by the specific subclasses.
149 """
150 raise NotImplementedError("Must be implemented by subclasses")
151
152 def get_container_size(self) -> int:
153 """
154 Uses os.path.getsize() to return the size of the container file.
155
156 Retruns:
157 int: Size in bytes of the container file
158 """
159 return os.path.getsize(self.filename)
160
161 def check_container_size(self):
162 """
163 Checks that the container size is within the range specified by
164 min_packed_size and max_packed_size. If any of those values is not specified
165 (i.e. it is None), the respective check will be skipped.
166
167 Raises:
168 ContainerException: If container size is less than minSize or greater
169 than maxSize, a ContainerException will be raised.
170 """
171 size = self.get_container_size()
172 if self._min_packed_size and (size < self._min_packed_size):
173 raise ContainerException(
174 "container file is too small (%dM < %dM)"
175 % (size / 1024 / 1024, self._min_packed_size / 1024 / 1024)
176 )
177 if self._max_packed_size and (size > self._max_packed_size):
178 raise ContainerException(
179 "container file is too large (%dM > %dM)"
180 % (size / 1024 / 1024, self._max_packed_size / 1024 / 1024)
181 )
182
183 def calculate_unpacked_size(self) -> int:
184 """
185 Estimates the space that the unpacked directory will use without unpacking.
186
187 Returns:
188 int: Estimated size in bytes of the unpacked directory
189
190 Raises:
191 NotImplementedError: Calling this method will raise an exception as the
192 estimation method is specific to the container type. This method
193 needs to be implemented by the specific subclasses.
194 """
195 raise NotImplementedError("Must be implemented by subclasses")
196
197 def check_unpacked_size(self):
198 """
199 Checks that the container size is within the range specified by
200 min_unpacked_size and max_unpacked_size. If any of those values is not
201 specified (i.e. it is None), the respective check will be skipped.
202
203 Raises:
204 ContainerException: If unpacked size is less than minSize or greater
205 than maxSize, a ContainerException will be raised.
206 """
207 size = self.calculate_unpacked_size()
208 if self._min_unpacked_size and (size < self._min_unpacked_size):
209 raise ContainerException(
210 "unpacked directory is too small (%dM < %dM)"
211 % (size / 1024 / 1024, self._min_unpacked_size / 1024 / 1024)
212 )
213 if self._max_unpacked_size and (size > self._max_unpacked_size):
214 raise ContainerException(
215 "unpacked directory is too large (%dM > %dM)"
216 % (size / 1024 / 1024, self._max_unpacked_size / 1024 / 1024)
217 )
218
219 def check_disk_space(self):
220 """
221 Checks that there is sufficient disk space available for the container to be
222 unpacked, i.e. that it will not consume more than 90% of the available disk
223
224 Raises:
225 ContainerException: If estimated unpacked size exceed the 90% of the
226 available space, an exception will be raised.
227 """
228 size = self.calculate_unpacked_size()
229 st = os.statvfs(self.tmp_dir)
230 avail = st.f_bsize * st.f_bavail * 0.9 # 90% of available space
231 if size > avail * 0.9:
232 raise ContainerException(
233 "insufficient available disk space (%dM > %dM)"
234 % (size / 1024 / 1024, avail / 1024 / 1024)
235 )
236
237 def unpack(self) -> str:
238 """
239 Extracts the content of the container file and returns the absolute path to
240 the newly unpacked directory.
241
242 Returns:
243 str: absolute path to the newly created unpack dir
244
245 Raises:
246 NotImplementedError: Calling this method will raise an exception as the
247 estimation method is specific to the container type. This method
248 needs to be implemented by the specific subclasses.
249 """
250 raise NotImplementedError("Must be implemented by subclasses")
251
107 def get_file(self, fn: str) -> Union[str, None]:252 def get_file(self, fn: str) -> Union[str, None]:
108 """253 """
109 Reads and return the raw content of the specified file.254 Reads and return the raw content of the specified file.
@@ -117,7 +262,7 @@ class BaseContainer:
117262
118 Raises:263 Raises:
119 ContainerException: Calling this method will raise an exception if264 ContainerException: Calling this method will raise an exception if
120 the requested file is not found.265 the requested file is not found.
121 """266 """
122 if not os.path.exists(os.path.join(self.unpack_dir, fn)):267 if not os.path.exists(os.path.join(self.unpack_dir, fn)):
123 raise ContainerException("Could not find '%s'" % fn)268 raise ContainerException("Could not find '%s'" % fn)
diff --git a/reviewtools/containers/snap_container.py b/reviewtools/containers/snap_container.py
124new file mode 100644269new file mode 100644
index 0000000..ff1ee57
--- /dev/null
+++ b/reviewtools/containers/snap_container.py
@@ -0,0 +1,44 @@
1import yaml
2from reviewtools.containers.squashfs_container import (
3 SquashfsContainer,
4 ContainerException,
5)
6
7
8class SnapContainer(SquashfsContainer):
9 # TODO: review-tools was not checking for it. We still have some snaps in
10 # "tests/" that doesn't fit this limit and will need to be regenerated before
11 # this check can be enforced.
12 # _min_packed_size = 16 * 1024
13 _max_packed_size = 5 * 1024 * 1024 * 1024
14 _max_unpacked_size = 25 * 1024 * 1024 * 1024
15
16 def get_snap_manifest(self):
17 man = "snap/manifest.yaml"
18 os_dpkg = "usr/share/snappy/dpkg.list"
19 snap_dpkg = "snap/dpkg.list"
20
21 if self._unpack_dir is None:
22 self.unpack(items=[man, os_dpkg, snap_dpkg])
23
24 man_content = self.get_file(man).decode()
25 try:
26 man_yaml = yaml.safe_load(man_content)
27 except Exception:
28 raise ContainerException(
29 "Could not load %s. Is it properly formatted?" % man
30 )
31
32 dpkg_list = None
33 try:
34 dpkg_list = self.get_file(os_dpkg).decode().split("\n")
35 return (man_yaml, dpkg_list)
36 except ContainerException:
37 pass
38
39 try:
40 dpkg_list = self.get_file(snap_dpkg).decode().split("\n")
41 except ContainerException:
42 pass
43
44 return (man_yaml, dpkg_list)
diff --git a/reviewtools/containers/squashfs_container.py b/reviewtools/containers/squashfs_container.py
0new file mode 10064445new file mode 100644
index 0000000..631fe76
--- /dev/null
+++ b/reviewtools/containers/squashfs_container.py
@@ -0,0 +1,111 @@
1import os
2import shutil
3from reviewtools.common import cmd, cmdIgnoreErrorStrings, UNSQUASHFS_IGNORED_ERRORS
4from reviewtools.containers.base_container import BaseContainer, ContainerException
5
6
7class SquashfsContainer(BaseContainer):
8 """
9 A container is a file that encapsulates a file system. This class extends
10 BaseContainer to handle functionality that is specific to Squashfs containers.
11 """
12
13 def __init__(self, fn):
14 """
15 If SquashfsContainer is initialized with a container file, it will also
16 verify that the provided file is actually a squashfs, in addition to the
17 checks performed by the BaseContainer.
18 """
19 super().__init__(fn)
20
21 # Check is squash file
22 self.check_container_format()
23
24 def check_container_format(self):
25 """
26 Checks that the container file has the expected format.
27
28 Raises:
29 ContainerException: This method will raise an exception if the
30 container file is not a squashfs container.
31 """
32 with open(self.filename, "rb") as f:
33 header = f.read(10)
34 if not header.startswith(b"hsqs"):
35 raise ContainerException("Unsupported package format (not squashfs)")
36
37 def calculate_unpacked_size(self) -> int:
38 """
39 Estimates the space that the unpacked directory will use by adding the
40 individual files size reported by "unsquashfs -lln" command. It may not
41 be completely accurate, as this command does not consider that each folder
42 requires at least one complete block
43
44 Returns:
45 int: Estimated size in bytes of the unpacked directory
46 """
47 (rc, out) = cmd(["unsquashfs", "-lln", self.filename])
48 if rc != 0:
49 raise RuntimeError("unsquashfs -lln '%s' failed: %s" % (self.filename, out))
50
51 size = 0
52 for line in out.splitlines():
53 if not line.startswith("-"): # skip non-regular files
54 continue
55 try:
56 size += int(line.split()[2])
57 except ValueError: # skip non-numbers
58 continue
59
60 return size
61
62 def unpack(self, items: list = [], force: bool = False) -> str:
63 """
64 Extracts the content of the container file and returns the absolute path to
65 the newly unpacked directory.
66
67 Args:
68 force (bool, optional): If ``True``, the unpack_dir will be overwritten
69 if it already exists. Defaults to ``False``
70
71 Returns:
72 str: absolute path to the newly created unpack dir
73
74 Raises:
75 ContainerException: This method will raise an ContainerException if the
76 unpack_dir already exists and this method was called without the
77 force argument set to True
78 RuntimeError: This method will raise a RuntimeError under the execution
79 of the "unsquashfs" command returns with an error code other than 0.
80 """
81 if self._unpack_dir is not None and not force:
82 raise ContainerException("'%s' exists. Aborting." % self._unpack_dir)
83
84 # Check disk space and space available in disk
85 self.check_unpacked_size()
86 self.check_disk_space()
87
88 # Unpack the container
89 stage_dir = os.path.join(self.tmp_dir, "stage")
90 cmd = [
91 "unsquashfs",
92 "-no-progress",
93 "-ignore-errors",
94 "-quiet",
95 "-f",
96 "-d",
97 stage_dir,
98 self.filename,
99 ] + items
100 (rc, out) = cmdIgnoreErrorStrings(cmd, UNSQUASHFS_IGNORED_ERRORS)
101 if rc != 0:
102 if os.path.isdir(stage_dir):
103 shutil.rmtree(stage_dir)
104 raise RuntimeError("unpacking failed with '%d':\n%s" % (rc, out))
105
106 # Update on success
107 if self._unpack_dir is not None and os.path.exists(self._unpack_dir):
108 shutil.rmtree(self._unpack_dir)
109 self._unpack_dir = ".".join(self.filename.split(".")[:-1])
110 shutil.move(stage_dir, self._unpack_dir)
111 return self._unpack_dir
diff --git a/reviewtools/sr_common.py b/reviewtools/sr_common.py
index 16790cc..9b961e2 100644
--- a/reviewtools/sr_common.py
+++ b/reviewtools/sr_common.py
@@ -30,7 +30,7 @@ from reviewtools.common import (
30 unsquashfs_lln_parse,30 unsquashfs_lln_parse,
31 verify_type,31 verify_type,
32)32)
33from reviewtools.containers.base_container import BaseContainer, ContainerException33from reviewtools.containers.snap_container import SnapContainer, ContainerException
34from reviewtools.overrides import interfaces_attribs_addons34from reviewtools.overrides import interfaces_attribs_addons
3535
3636
@@ -398,7 +398,7 @@ class SnapReview(ReviewBase):
398 ReviewBase.__init__(self, review_type, overrides=overrides)398 ReviewBase.__init__(self, review_type, overrides=overrides)
399399
400 try:400 try:
401 self.snap = BaseContainer(fn)401 self.snap = SnapContainer(fn)
402 except ContainerException as e:402 except ContainerException as e:
403 error(str(e))403 error(str(e))
404 # TODO: update code to directly use self.snap.xxx instead of assigning here404 # TODO: update code to directly use self.snap.xxx instead of assigning here
diff --git a/reviewtools/sr_tests.py b/reviewtools/sr_tests.py
index 6caae54..bfbf375 100644
--- a/reviewtools/sr_tests.py
+++ b/reviewtools/sr_tests.py
@@ -87,7 +87,7 @@ def create_patches():
87 # child.87 # child.
88 patches = []88 patches = []
89 patches.append(89 patches.append(
90 patch("reviewtools.sr_common.BaseContainer.__init__", _BaseContainer.__init__)90 patch("reviewtools.sr_common.SnapContainer.__init__", _BaseContainer.__init__)
91 )91 )
92 patches.append(92 patches.append(
93 patch("reviewtools.sr_common.SnapReview._extract_snap_yaml", _extract_snap_yaml)93 patch("reviewtools.sr_common.SnapReview._extract_snap_yaml", _extract_snap_yaml)
@@ -98,7 +98,6 @@ def create_patches():
98 _extract_snap_manifest_yaml,98 _extract_snap_manifest_yaml,
99 )99 )
100 )100 )
101 patches.append(patch("reviewtools.common.unpack_pkg", _mock_func))
102 # sr_common101 # sr_common
103 patches.append(102 patches.append(
104 patch("reviewtools.sr_common.SnapReview._get_unpack_dir", __get_unpack_dir)103 patch("reviewtools.sr_common.SnapReview._get_unpack_dir", __get_unpack_dir)
diff --git a/reviewtools/tests/containers/test_base_container.py b/reviewtools/tests/containers/test_base_container.py
index 740e942..1b4e022 100644
--- a/reviewtools/tests/containers/test_base_container.py
+++ b/reviewtools/tests/containers/test_base_container.py
@@ -3,29 +3,43 @@ import shutil
3import unittest3import unittest
4from tempfile import mkdtemp4from tempfile import mkdtemp
5from reviewtools.containers.base_container import BaseContainer, ContainerException5from reviewtools.containers.base_container import BaseContainer, ContainerException
6from reviewtools.tests.utils import make_snap2
76
87
9class TestBaseContainer(unittest.TestCase):8class TestBaseContainer(unittest.TestCase):
10 """Tests for reviewtools.containers.base_container.BaseContainer"""9 """Tests for reviewtools.containers.base_container.BaseContainer"""
1110
12 expected_files = ["meta/snap.yaml", "meta/icon.png", "nonexecstack.bin"]11 expected_files = ["meta/snap.yaml", "meta/icon.png", "bin/ls"]
13 expected_binaries = ["nonexecstack.bin"]12 expected_binaries = ["bin/ls"]
1413
15 @classmethod14 @classmethod
16 def setUpClass(cls):15 def setUpClass(cls):
17 cls.tmp_dir = mkdtemp()16 cls.tmp_dir = mkdtemp()
18 cls.fn = make_snap2(17 cls.fn = cls._create_container(cls)
19 output_dir=cls.tmp_dir,18 cls.unpack_dir = cls._create_unpackdir(cls)
20 yaml="Test",
21 extra_files=["/bin/ls:nonexecstack.bin"],
22 )
2319
24 @classmethod20 @classmethod
25 def tearDownClass(cls):21 def tearDownClass(cls):
26 if os.path.exists(cls.tmp_dir):22 if os.path.exists(cls.tmp_dir):
27 shutil.rmtree(cls.tmp_dir)23 shutil.rmtree(cls.tmp_dir)
2824
25 def _create_container(self, container_size: int = 10000):
26 fn = os.path.join(self.tmp_dir, "container.test")
27 with open(fn, "w") as f:
28 f.truncate(container_size)
29 return fn
30
31 def _create_unpackdir(self):
32 unpack_dir = os.path.join(self.tmp_dir, "unpack_dir")
33 for file in self.expected_files:
34 unpacked_file = os.path.join(unpack_dir, file)
35 os.makedirs(os.path.dirname(unpacked_file), exist_ok=True)
36 if file not in self.expected_binaries:
37 with open(unpacked_file, "w+b") as fd:
38 fd.write(b"Test")
39 else:
40 shutil.copyfile(os.path.join("/", file), unpacked_file)
41 return unpack_dir
42
29 # Test class initialization43 # Test class initialization
30 def test_initialization_container__happy(self):44 def test_initialization_container__happy(self):
31 try:45 try:
@@ -43,10 +57,75 @@ class TestBaseContainer(unittest.TestCase):
43 in context.exception.value57 in context.exception.value
44 )58 )
4559
60 # Test deletion
61 def test_check_cleanup(self):
62 try:
63 container = BaseContainer(self.fn)
64 tmp_dir = container.tmp_dir
65 del container
66 self.assertFalse(os.path.exists(tmp_dir))
67 except Exception:
68 self.fail("An unexpected error occurred during deletion test")
69
70 # Test check container size
71 def test_check_container_size__happy(self):
72 container = BaseContainer(self.fn)
73 container._min_packed_size = 0
74 container._max_packed_size = 10**6
75 try:
76 container.check_container_size()
77 except ContainerException:
78 self.fail("An unexpected error occurred while checking file size")
79
80 def test_check_container_size__too_big(self):
81 container = BaseContainer(self.fn)
82 container._max_packed_size = 1
83 with self.assertRaises(ContainerException) as context:
84 container.check_container_size()
85 self.assertTrue("container file is too large " in context.exception.value)
86
87 def test_check_container_size__too_small(self):
88 container = BaseContainer(self.fn)
89 container._min_packed_size = 10**6
90 with self.assertRaises(ContainerException) as context:
91 container.check_container_size()
92 self.assertTrue("container file is too small " in context.exception.value)
93
94 # Test check unpacked size
95 def _mock_calculate_unpacked_size(self):
96 return 10**3
97
98 def test_check_unpacked_size__happy(self):
99 container = BaseContainer(self.fn)
100 container.calculate_unpacked_size = self._mock_calculate_unpacked_size
101 container._min_unpacked_size = 0
102 container._max_unpacked_size = 10**6
103 try:
104 container.check_unpacked_size()
105 except ContainerException:
106 self.fail("An unexpected error occurred while checking file size")
107
108 def test_check_unpacked_size__too_big(self):
109 container = BaseContainer(self.fn)
110 container.calculate_unpacked_size = self._mock_calculate_unpacked_size
111 container._max_unpacked_size = 1
112 with self.assertRaises(ContainerException) as context:
113 container.check_unpacked_size()
114 self.assertTrue("container file is too large " in context.exception.value)
115
116 def test_check_unpacked_size__too_small(self):
117 container = BaseContainer(self.fn)
118 container.calculate_unpacked_size = self._mock_calculate_unpacked_size
119 container._min_unpacked_size = 10**6
120 with self.assertRaises(ContainerException) as context:
121 container.check_unpacked_size()
122 self.assertTrue("container file is too small " in context.exception.value)
123
46 # Test get file124 # Test get file
47 def test_get_file__happy(self):125 def test_get_file__happy(self):
48 try:126 try:
49 container = BaseContainer(self.fn)127 container = BaseContainer(self.fn)
128 container._unpack_dir = self.unpack_dir
50 snap_yaml = container.get_file("meta/snap.yaml")129 snap_yaml = container.get_file("meta/snap.yaml")
51 self.assertEqual(snap_yaml.decode(), "Test")130 self.assertEqual(snap_yaml.decode(), "Test")
52 except ContainerException:131 except ContainerException:
@@ -55,6 +134,7 @@ class TestBaseContainer(unittest.TestCase):
55 def test_get_file__missing_file(self):134 def test_get_file__missing_file(self):
56 with self.assertRaises(ContainerException) as context:135 with self.assertRaises(ContainerException) as context:
57 container = BaseContainer(self.fn)136 container = BaseContainer(self.fn)
137 container._unpack_dir = self.unpack_dir
58 container.get_file("non/existent/file")138 container.get_file("non/existent/file")
59 self.assertTrue("Could not find" in context.exception.value)139 self.assertTrue("Could not find" in context.exception.value)
60140
@@ -62,6 +142,7 @@ class TestBaseContainer(unittest.TestCase):
62 def test_get_files_list__happy(self):142 def test_get_files_list__happy(self):
63 try:143 try:
64 container = BaseContainer(self.fn)144 container = BaseContainer(self.fn)
145 container._unpack_dir = self.unpack_dir
65 self.assertCountEqual(146 self.assertCountEqual(
66 container.get_files_list(),147 container.get_files_list(),
67 self.expected_files,148 self.expected_files,
@@ -72,6 +153,7 @@ class TestBaseContainer(unittest.TestCase):
72 def test_get_files_list_abs__happy(self):153 def test_get_files_list_abs__happy(self):
73 try:154 try:
74 container = BaseContainer(self.fn)155 container = BaseContainer(self.fn)
156 container._unpack_dir = self.unpack_dir
75 self.assertCountEqual(157 self.assertCountEqual(
76 container.get_files_list(abs=True),158 container.get_files_list(abs=True),
77 [159 [
@@ -85,6 +167,7 @@ class TestBaseContainer(unittest.TestCase):
85 def test_files__happy(self):167 def test_files__happy(self):
86 try:168 try:
87 container = BaseContainer(self.fn)169 container = BaseContainer(self.fn)
170 container._unpack_dir = self.unpack_dir
88 self.assertCountEqual(171 self.assertCountEqual(
89 container.files,172 container.files,
90 self.expected_files,173 self.expected_files,
@@ -96,6 +179,7 @@ class TestBaseContainer(unittest.TestCase):
96 def test_get_compiled_binaries_list__happy(self):179 def test_get_compiled_binaries_list__happy(self):
97 try:180 try:
98 container = BaseContainer(self.fn)181 container = BaseContainer(self.fn)
182 container._unpack_dir = self.unpack_dir
99 self.assertCountEqual(183 self.assertCountEqual(
100 container.get_compiled_binaries_list(), self.expected_binaries184 container.get_compiled_binaries_list(), self.expected_binaries
101 )185 )
@@ -107,6 +191,7 @@ class TestBaseContainer(unittest.TestCase):
107 def test_get_compiled_binaries_list_abs__happy(self):191 def test_get_compiled_binaries_list_abs__happy(self):
108 try:192 try:
109 container = BaseContainer(self.fn)193 container = BaseContainer(self.fn)
194 container._unpack_dir = self.unpack_dir
110 self.assertCountEqual(195 self.assertCountEqual(
111 container.get_compiled_binaries_list(abs=True),196 container.get_compiled_binaries_list(abs=True),
112 [197 [
@@ -122,6 +207,7 @@ class TestBaseContainer(unittest.TestCase):
122 def test_bin_files__happy(self):207 def test_bin_files__happy(self):
123 try:208 try:
124 container = BaseContainer(self.fn)209 container = BaseContainer(self.fn)
210 container._unpack_dir = self.unpack_dir
125 self.assertCountEqual(container.bin_files, self.expected_binaries)211 self.assertCountEqual(container.bin_files, self.expected_binaries)
126 except ContainerException:212 except ContainerException:
127 self.fail(213 self.fail(
diff --git a/reviewtools/tests/containers/test_squashfs_container.py b/reviewtools/tests/containers/test_squashfs_container.py
128new file mode 100644214new file mode 100644
index 0000000..7d19a77
--- /dev/null
+++ b/reviewtools/tests/containers/test_squashfs_container.py
@@ -0,0 +1,73 @@
1import os
2import unittest
3from tempfile import mkstemp, mkdtemp
4from reviewtools.containers.squashfs_container import (
5 SquashfsContainer,
6 ContainerException,
7)
8from reviewtools.tests.utils import make_snap2
9
10
11class TestSquashfsContainer(unittest.TestCase):
12 """Tests for reviewtools.containers.squashfs_container"""
13
14 @classmethod
15 def setUpClass(cls):
16 cls.tmp_dir = mkdtemp()
17 cls.fn = make_snap2(
18 output_dir=cls.tmp_dir,
19 yaml="Test",
20 extra_files=["/bin/ls:bin/ls"],
21 )
22
23 # Test initialization
24 def test_check_initialization__happy(self):
25 try:
26 SquashfsContainer(self.fn)
27 except Exception:
28 self.fail(
29 "An unexpected error occurred during SquashfsContainer initialization with container file"
30 )
31
32 def test_check_initialization__invalid_format(self):
33 with self.assertRaises(ContainerException) as context:
34 fd, plain_file = mkstemp()
35 SquashfsContainer(plain_file)
36 self.assertEqual(
37 "unsupported package format (not squashfs)", context.exception.value
38 )
39 os.unlink(plain_file)
40
41 # Test check format
42 def test_calculate_unpacked_size__happy(self):
43 try:
44 # unpacked size calculated is a bit smaller than actual size as it does not consider that
45 # folders require at least one complete block
46 container = SquashfsContainer(self.fn)
47 size = container.calculate_unpacked_size()
48 self.assertTrue(isinstance(size, int))
49 except Exception:
50 self.fail("An unexpected error occurred during unpacked size calculation")
51
52 # Test unpack
53 def test_unpack__happy(self):
54 try:
55 container = SquashfsContainer(self.fn)
56 container.unpack()
57 except Exception:
58 self.fail("An unexpected error occurred during unpack")
59
60 def test_unpack__force(self):
61 try:
62 container = SquashfsContainer(self.fn)
63 container.unpack()
64 container.unpack(force=True)
65 except Exception:
66 self.fail("An unexpected error occurred during unpack")
67
68 def test_unpack__no_force(self):
69 with self.assertRaises(ContainerException) as context:
70 container = SquashfsContainer(self.fn)
71 container.unpack()
72 container.unpack()
73 self.assertTrue(" exists. Aborting." in context.exception.value)

Subscribers

People subscribed via source and target branches