lp:~alexmurray/ubuntu-security-tools

Author: Alex Murray
Author Date: 2024-12-03 04:27:40 UTC

umt logs: unify with find_build_log and add to usage

Remove the copy-pasta from find_build_log() and instead refactor that out into a
new find_build_logs() function which can be used by either cmd_logs() or
find_build_log() and also add this log command to print_usage() so it is
somewhat documented.

Signed-off-by: Alex Murray <alex.murray@canonical.com>

Author: Alex Murray
Author Date: 2024-11-27 01:32:54 UTC

umt qrt: fix incorrect handling of test scripts with periods

Signed-off-by: Alex Murray <alex.murray@canonical.com>

Author: Alex Murray
Author Date: 2024-11-07 03:34:57 UTC

build-tools/umt check: only warn if no CVEs in changelog for security updates

Signed-off-by: Alex Murray <alex.murray@canonical.com>

Author: Alex Murray
Author Date: 2024-10-21 00:46:14 UTC

umt check-log: rename logfile to buildlog

This more accurately describes this variable - thanks to @sespiros for the
suggestion.

Signed-off-by: Alex Murray <alex.murray@canonical.com>

Author: Alex Murray
Author Date: 2024-10-16 19:44:26 UTC

umt testflinger: fix installing QRT packages when using skeleton

The commands we provide to be executed in the testflinger backend get manged so
we can't use single-quotes as normal in a shell command, so instead don't quote
the sed invocation at all and instead escape the spaces to ensure this command
gets executed properly via testflinger.

I have also tested that this still works as expected when using umt qrt directly
as well.

Signed-off-by: Alex Murray <alex.murray@canonical.com>

Author: Alex Murray
Author Date: 2024-09-18 01:17:22 UTC

.launchpad.yaml: update to support running new umt unit test

Signed-off-by: Alex Murray <alex.murray@canonical.com>

Author: Alex Murray
Author Date: 2024-08-16 02:45:32 UTC

umt build: always try generate debdiff when doing a --merge

--merge is useful when preparing an update that requires a multiple changelog
entries etc (ie. when it needs to be respun to include additional changes). So
always try and generate the debdiff, not just when this looks like a standard
merge with a "ubuntu1" version.

Signed-off-by: Alex Murray <alex.murray@canonical.com>

Author: Alex Murray
Author Date: 2024-07-31 07:35:13 UTC

audits/uaudit.py: use grep to find binary blobs for LP: #2073158

Use grep to find all non-text files and capture this list for manual analysis by
the reviewer.

Signed-off-by: Alex Murray <alex.murray@canonical.com>

Author: Alex Murray
Author Date: 2024-07-26 10:22:16 UTC

package-tools/test_increment_version.py: fix testcase using interim release

Don't hard-code the interim release name since this can change (e.g. when mantic
reach EOL recently) - instead programatically find one to use in the test
case. This should fix the failing lpci test.

Signed-off-by: Alex Murray <alex.murray@canonical.com>

Author: Alex Murray
Author Date: 2024-07-26 05:23:29 UTC

umt compare-log: support using a URI for --previous-logfile

If --previous-logfile looks like a URI, fetch it and compare against that. This
allows to point to a log file from launchpad if the previous version of the
package can no longer be built locally without having to manually download the
log file.

Signed-off-by: Alex Murray <alex.murray@canonical.com>

Author: Alex Murray
Author Date: 2024-07-25 01:27:07 UTC

umt delete: document in print_usage()

Signed-off-by: Alex Murray <alex.murray@canonical.com>

Author: Alex Murray
Author Date: 2024-06-28 05:13:02 UTC

umt qrt: remote --prev-ppa argument for now

This is broken and doesn't work since any packages that we copy here get
shadowed in the local repo by the actual packages under test (as there can be
only one version of a package in a given APT repo).

Instead we perhaps need to look into using multiple pockets or similar within
the local repo to allow more than one version of a package to be present in the
repo and then we need to appropriately configure these different pockets via apt
priorities or similar so that we can use both the new and old versions of the
package during the tests.

Signed-off-by: Alex Murray <alex.murray@canonical.com>

Author: Alex Murray
Author Date: 2024-06-19 07:54:10 UTC

Merge remote-tracking branch 'bruce-cable/testbed_failure_info'

Signed-off-by: Alex Murray <alex.murray@canonical.com>

Author: Alex Murray
Author Date: 2024-05-23 03:36:05 UTC

umt changelog: allow to --repeat with -v etc

Signed-off-by: Alex Murray <alex.murray@canonical.com>

Author: Alex Murray
Author Date: 2024-05-17 00:53:28 UTC

check-source-package: make superceded a warning if local versions present

In this case, it is still useful to warn the user since they need to remember
to actually publish all these local versions

Signed-off-by: Alex Murray <alex.murray@canonical.com>

Author: Alex Murray
Author Date: 2024-05-15 12:46:58 UTC

check-source-package: warn if distro-info doesn't know about a release

As suggested by @litios

Signed-off-by: Alex Murray <alex.murray@canonical.com>

Author: Alex Murray
Author Date: 2024-04-15 06:29:19 UTC

audits/uaudit: pass through command-line arguments

Signed-off-by: Alex Murray <alex.murray@canonical.com>

Author: Alex Murray
Author Date: 2024-02-25 23:29:51 UTC

build-tools/umt: fix invalid escape sequence in regexes

With Python 3.12 (default now in noble) the python interpreter complains about
these at runtime, so fix them up to avoid the noise - see example as follows:

[amurray:/tmp] $ umt download gtk+3.0 -r noble
/home/amurray/bin/umt:129: SyntaxWarning: invalid escape sequence '\d'
  (rc4, _) = runcmd(['grep', '-P', '\d+:\s+[0-9A-F]+:0050', "/proc/net/tcp" ])
/home/amurray/bin/umt:130: SyntaxWarning: invalid escape sequence '\d'
  (rc6, _) = runcmd(['grep', '-P', '\d+:\s+[0-9A-F]+:0050', "/proc/net/tcp6" ])
/home/amurray/bin/umt:1067: SyntaxWarning: invalid escape sequence '\+'
  pattern = re.compile(r' %s( |_)%s ' % (pkg.replace('+', '\+'), ver.replace('+', '\+')))
/home/amurray/bin/umt:1067: SyntaxWarning: invalid escape sequence '\+'
  pattern = re.compile(r' %s( |_)%s ' % (pkg.replace('+', '\+'), ver.replace('+', '\+')))
/home/amurray/bin/umt:1536: SyntaxWarning: invalid escape sequence '\/'
  if re.search('[WE]: Failed to fetch .*\/debian-installer\/', report):
/home/amurray/bin/umt:1946: SyntaxWarning: invalid escape sequence '\+'
  esm_version_match = re.search("[\+~]esm\d+", details['version'])
/home/amurray/bin/umt:4338: SyntaxWarning: invalid escape sequence '\-'
  build_script = build_script + '''
/home/amurray/bin/umt:4351: SyntaxWarning: invalid escape sequence '\-'
  build_script = build_script + '''
/home/amurray/bin/umt:4503: SyntaxWarning: invalid escape sequence '\s'
  build_script = build_script + '''
/home/amurray/bin/umt:4738: SyntaxWarning: invalid escape sequence '\.'
  if re.search('^[a-z0-9][a-z0-9+\.\-]+$', details['package']) == None:
/home/amurray/bin/umt:4754: SyntaxWarning: invalid escape sequence '\.'
  magic = '^[^-]+-(.*~ppa[0-9]+|(0(\.0)?ubuntu)?(1|0\.1|0\.(6.06|6.10|7.04|7.10|8.04|8.10|9.04)(\.1)?))$'
/home/amurray/bin/umt:5047: SyntaxWarning: invalid escape sequence '\s'
  changelog_text = snip(report, '^Changes:\s*$', '^[^ ]')
/home/amurray/bin/umt:5053: SyntaxWarning: invalid escape sequence '\+'
  if re.search('^ ' + details['source'].replace('+', '\+'), line):
/home/amurray/bin/umt:5062: SyntaxWarning: invalid escape sequence '\s'
  changes_text = snip(changes.read(), '^Changes:\s*$', '^[^ ]')
/home/amurray/bin/umt:5065: SyntaxWarning: invalid escape sequence '\+'
  if re.search('^ ' + details['source'].replace('+', '\+'), line):
/home/amurray/bin/umt:5609: SyntaxWarning: invalid escape sequence '\$'
  if re.search('^\$maintainer_name', line):
/home/amurray/bin/umt:5610: SyntaxWarning: invalid escape sequence '\g'
  keyid = re.sub(".*<(.*)>.*", "\g<1>", line.strip())
Downloading 'gtk+3.0' version '3.24.40-2ubuntu1' for release 'noble'.

Signed-off-by: Alex Murray <alex.murray@canonical.com>

Author: Alex Murray
Author Date: 2024-02-09 00:47:45 UTC

build-tools/umt: add --disable-proposed option when building

This can be used to override the automatic enablement of the -proposed pocket
when building for devel.

Signed-off-by: Alex Murray <alex.murray@canonical.com>

Author: Alex Murray
Author Date: 2024-02-08 10:51:17 UTC

umt qrt: fix typo in comment

Signed-off-by: Alex Murray <alex.murray@canonical.com>

Author: Alex Murray
Author Date: 2024-02-08 01:54:35 UTC

build-tools/build-sources-list: remove support for EOL releases

precise for Ubuntu and sarge, etch, lenny and squeeze for Debian are EOL so we
won't be needing these anymore (plus neither sec-buildenv nor the
BuildEnvironment wiki page refer to these either for configuring them in
~/.ubuntu-security-tools.conf so it is safe to remove them from
build-sources-list).

Signed-off-by: Alex Murray <alex.murray@canonical.com>

Author: Alex Murray
Author Date: 2024-02-06 04:12:09 UTC

audits/uaudit.py: always generate coverity.txt even if no defects

Signed-off-by: Alex Murray <alex.murray@canonical.com>

Author: Alex Murray
Author Date: 2023-12-07 03:36:17 UTC

Add a basic .launchpad.yaml for lpci unit test goodness

Also move the unit tests for increment_version.py into the same
directory so that we can run them easily via pytest.

Signed-off-by: Alex Murray <alex.murray@canonical.com>

Author: Alex Murray
Author Date: 2023-09-20 00:42:24 UTC

umt: offer suggestions on invalid input

if the provided command is not known then try suggest a similar one, and same
for the package name provided to umt search.

Signed-off-by: Alex Murray <alex.murray@canonical.com>

Author: Alex Murray
Author Date: 2023-09-14 01:40:18 UTC

umt search: treat unknown packages as an error

Signed-off-by: Alex Murray <alex.murray@canonical.com>

Author: Alex Murray
Author Date: 2023-09-07 01:49:04 UTC

umt search: add column headings to reduce extraneous info

Signed-off-by: Alex Murray <alex.murray@canonical.com>

Author: Alex Murray
Author Date: 2023-02-14 00:52:06 UTC

uaudit: Update template for rubocop but limit to only security checks

Signed-off-by: Alex Murray <alex.murray@canonical.com>

Author: Alex Murray
Author Date: 2023-02-12 01:04:01 UTC

check-source-package: Add checks for devel release semantics

Versioning in devel is different than stable releases so make sure we invoke the
new python based sdch with -d in this case, plus also check that uploads for the
devel release do not contain -security pocket in the distribution.

Signed-off-by: Alex Murray <alex.murray@canonical.com>

Author: Alex Murray
Author Date: 2023-01-22 23:07:32 UTC

Initial WIP attempt to integrate ikos with uaudit/umt

Signed-off-by: Alex Murray <alex.murray@canonical.com>

Author: Alex Murray
Author Date: 2022-09-08 01:33:50 UTC

umt: Enable use of make's --output-sync

recurse is the best option overall to reduce parallel interleavings but does
cause the build output to pause for a significant amount of time.

Signed-off-by: Alex Murray <alex.murray@canonical.com>

Author: Alex Murray
Author Date: 2022-08-18 01:18:35 UTC

umt upload: Show full overridden destination name in warning message

Signed-off-by: Alex Murray <alex.murray@canonical.com>

Author: Alex Murray
Author Date: 2022-08-10 00:52:47 UTC

umt: Support overriding ulimit as unlimited for libbluray

It seems libbluray needs more than the previous memory limit to build for
kinetic so set this to unlimited in ulimit_pkg_exceptions and ensure this
is a supported value when handled by the code which sets this overridden
value.

Signed-off-by: Alex Murray <alex.murray@canonical.com>

Author: Alex Murray
Author Date: 2022-07-18 12:42:08 UTC

umt testflinger: Support private PPAs for testflinger package source

When a ppa is specified via --repo, lookup the PPA signing key and
subscription URL so that it can be used more seamlessly via
add-apt-repository on the target device.

Signed-off-by: Alex Murray <alex.murray@canonical.com>

Author: Alex Murray
Author Date: 2022-07-18 12:41:32 UTC

uaudit: Add additional python static analysis via dlint

Signed-off-by: Alex Murray <alex.murray@canonical.com>

Author: Alex Murray
Author Date: 2022-06-27 11:46:27 UTC

umt: Query apt-cache / download via chdist

This avoids having to have a global
/etc/apt/sources.list.d/ubuntu-security.list so that user's can configure
their own local chdist environments instead.

Signed-off-by: Alex Murray <alex.murray@canonical.com>

Author: Alex Murray
Author Date: 2022-06-08 01:07:59 UTC

uaudit: Refactor static analysis tools handling

Introduce a StaticAnalysisTool() class to encapsulate the logic needed for
each different static analysis tool and then remove all the copy-pasta'd
code which used to handle these. This should make adding new static
analysis tools as simple as defining a new instance of this object and
adding it to the static_analysis_tools list.

Signed-off-by: Alex Murray <alex.murray@canonical.com>

Author: Alex Murray
Author Date: 2022-02-28 04:07:59 UTC

umt check: Use a regex to parse LP bug URLs (thanks sarnold)

Signed-off-by: Alex Murray <alex.murray@canonical.com>

Author: Alex Murray
Author Date: 2022-02-25 04:17:32 UTC

umt grep: Avoid temporary file and pipe stdin to grep

Signed-off-by: Alex Murray <alex.murray@canonical.com>

Author: Alex Murray
Author Date: 2021-11-16 05:41:14 UTC

check-source-package: Use both full and per release rmadison output

Only call rmadison once where possible and store this as $rmadout and then
also store a new variable $rmadoutrel which is $rmadout scoped to just the
current release - this then allows us to use $rmadout for the upgradeable
version check whilst using $rmadoutrel in other checks as before.

Author: Alex Murray
Author Date: 2021-10-21 05:32:42 UTC

umt: Handle latest cve_lib API change and remove old unused PPA cruft

Author: Alex Murray
Author Date: 2021-07-09 05:13:28 UTC

Keep umt and ceviche working with customer-ppa-tracking-2 of UCT

Author: Alex Murray
Author Date: 2021-09-13 07:06:36 UTC

umt qrt: Add support for creating a VM if none already exists

Since both uvt and lxc can create VMs, first check if one exists with the
given name, and if not offer to create one.

Author: Alex Murray
Author Date: 2021-09-08 06:14:28 UTC

umt adt: Slight refactor of adt results handling to be more pythonic

Author: Alex Murray
Author Date: 2021-08-24 02:33:57 UTC

umt qrt: Use argparse to validate possible backend choices

Author: Alex Murray
Author Date: 2021-08-23 06:20:08 UTC

umt qrt: Push files to home on backend by default if none specified

This ensures that QRT tests which run as root get placed in /root on the
target VM etc and then can be run successfully.

Author: Alex Murray
Author Date: 2021-08-02 04:30:32 UTC

umt: Only output hyperlinks when we are sure they are supported

As per https://gist.github.com/egmontkob/eb114294efbcd5adb1944c9f3cb5feda
it looks like VTE versions >= 0.50.0 support this so check for that and
only output in that case.

Author: Alex Murray
Author Date: 2020-11-25 01:38:45 UTC

umt testflinger: Capture lsb_release and uname details for devices

Author: Alex Murray
Author Date: 2020-11-04 01:07:20 UTC

check-source-package: Don't recommend to use -proposed

This is not the done-thing anymore

Author: Alex Murray
Author Date: 2020-01-09 06:21:04 UTC

umt autopkgtest: Fix missing arguments

Author: Alex Murray
Author Date: 2019-12-10 04:19:13 UTC

umt: Remove extra junk left over from previous commit

Author: Alex Murray
Author Date: 2019-10-21 00:58:20 UTC

check-source-package: Add version check to ensure we don't block upgrades

For the current package version, we need to check that it will not block
future upgrades - ie if a package has version X in release N, then it can't
have version X' in release N' where X' < X and N' > N.