lp:~dannf/canonical-server-hwe-utils

Author: dann frazier
Author Date: 2023-03-23 18:46:23 UTC

Clear contents and update README.md to point to new repository

Author: dann frazier
Author Date: 2023-03-23 18:46:23 UTC

Add a test script to verify system access after the Tremont->Needham move

Author: dann frazier
Author Date: 2022-09-09 15:29:52 UTC

gen_conserver_cf.py: Grant access to some kernel team hosts

Requested by Sean Feole:
  https://chat.canonical.com/canonical/pl/sx1z74uo3trkjxgbah7ugurm4w

Author: dann frazier
Author Date: 2022-08-16 17:12:06 UTC

jenkins-scripts: weekly-rota-init: Replace result w/ timestamp in task summary

As Alexandre pointed out, job numbers get reset when we redeploy scalebot,
so there is a risk of collision. Let's add the timestamp (well, datestamp)
to the summary to increase its uniqueness. Drop the result string to keep
the length from becoming unwieldy.

Author: dann frazier
Author Date: 2022-01-27 19:38:55 UTC

jenkins-rss-to-trello-md.py: Delete; superseded by weekly-rota-init.py

Author: dann frazier
Author Date: 2022-01-27 19:08:33 UTC

weekly-rota-init: Use the jenkins API instead of parsing RSS files

This changes the command line syntax to take a Jenkins URL, username
and password instead of an RSS file. RSS file parsing support is dropped -
I see no reason to retain it.

Author: dann frazier
Author Date: 2022-01-14 16:49:46 UTC

weekly-rota-init.py: 0-pad dates

cosmetic improvement only

Author: dann frazier
Author Date: 2021-12-15 18:35:10 UTC

Introduce weekly-rota-init.py

Scan a Jenkins RSS feed and use the JIRA API to automatically generate cards
for any failures.

Author: dann frazier
Author Date: 2021-09-14 21:41:47 UTC

Add command to clear System Event Log

Author: dann frazier
Author Date: 2021-09-14 21:53:16 UTC

labkey: Handle default branch rename

We're renaming the default branch from "master" to "main". Handle the
migration for existing labkey users.

Author: dann frazier
Author Date: 2021-07-29 20:08:42 UTC

installer-test-matrix: Drop xenial release

xenial has reached end of standard support.

Author: dann frazier
Author Date: 2021-05-18 14:36:37 UTC

sbsh: Add support for executing commands specified after "--"

Some examples:

$ sbsh -l tremont -- date
Tue May 18 14:43:00 UTC 2021
Connection to 10.228.68.60 closed.
Shared connection to 10.228.0.2 closed.

$ sbsh -l 1ss -- juju status
Model Controller Cloud/Region Version SLA Timestamp Notes
controller jenkins scalebot/default 2.7.6 unsupported 14:42:33Z upgrade available: 2.7.8

Machine State DNS Inst id Series AZ Message
0 started 10.229.192.54 cwht73 bionic default Deployed

Connection to 10.229.193.21 closed.
Shared connection to 10.229.0.101 closed.

Author: dann frazier
Author Date: 2021-05-07 17:59:03 UTC

gen_conserver_cf.py: Deactivate sol before activating it

Some BMCs have an issue where they randomly drop an active sol session,
and then refuse to start a new one claiming another sol session is
already active. Aftert his, console clients and console log files will
fail to show any console output, and just have a series of errors:

Info: SOL payload already active on another session
Info: SOL payload already active on another session
Info: SOL payload already active on another session
Info: SOL payload already active on another session

To recover, we've had to send an sol deactivate command out-of-band of
conserver so that conserver's next reconnect attempt can succeed. Let's
just teach conserver to send the deactivate command itself before every
reconnect attempt.

Author: dann frazier
Author Date: 2021-05-05 23:17:01 UTC

sysadmin-tools: update_conserver: Rewrite in python

We currently hardcode the IP addresses of the conservers for each lab,
but we could instead just look them up in the lab.yaml. But that
requires parsing YAML. I don't know of a good YAML parser utility,
so let's rewrite in python and use its parser.

This commit is simply a rewrite of the existing tool in python - no
functional differences are intended.

Author: dann frazier
Author Date: 2021-04-30 17:51:46 UTC

Introduce sbsh - a command to get you a shell in the scalebot environment

In order to do scalebot maintenance tasks, such as cleaning up models
from failed tests, you first have to get a shell in the scalebot environment
as the jenkins user. This is a pain, and typically looks like:

  dannf@laptop:~$ ssh ubuntu@10.229.0.101
  ubuntu@bates:~$ juju status | grep scalebot # find the unit name
    scalebot/0* active idle [...]
  ubuntu@bates:~$ juju ssh scalebot/0
  ubuntu@juju-fe2b5c-0:~$ sudo bash
  root@juju-fe2b5c-0:~# su - jenkins
  jenkins@juju-fe2b5c-0:~$

This tool automates these steps:

  dannf@laptop:~$ sbsh -l 1ss
  jenkins@juju-fe2b5c-0:~$

Author: dann frazier
Author Date: 2021-05-04 14:34:32 UTC

update_conserver.sh: Add a check that we're in the correct directory

Author: dann frazier
Author Date: 2021-04-21 18:13:03 UTC

labkey: console: Add IPMI serial-over-lan support

In labs that do not have a conserver, support connecting to IPMI BMCs
directly. Unlike conserver, IPMI SoL does not support multiple users
at the same time, so this also adds a "force" argument to the console
subcommand that will cause labkey to first disconnect any existing users.

Author: dann frazier
Author Date: 2021-04-21 17:11:05 UTC

labkey: Remove duplicate code, causing console to restart after exit

For the longest time I've been annoyed that I have to send console's
exit sequence (~.) twice to make it exit when called via labkey. I
just assumed there was some weird interaction between the labkey and
console process that caused it. Nope. I had literally just duplicated
the callout to console - presumably a debug cut & paste gone wrong.

Author: dann frazier
Author Date: 2021-04-20 19:43:03 UTC

installer-test-matrix: Add the Gigabyte R120-T33 server

Although it is essentially a CRB1S, we certified these servers w/
acpi=force on the command line, so we should also test that.

Author: dann frazier
Author Date: 2021-03-31 17:10:15 UTC

installer-test-matrix: Add hirsute

Author: dann frazier
Author Date: 2021-02-01 17:44:39 UTC

pre-commit: Don't use mutable references

pre-commit complains about 'rev: master' not being supported:

[WARNING] The 'rev' field of repo 'https://github.com/psf/black' appears to be a mutable reference (moving tag / branch). Mutable references are never updated after first install and are not supported. See https://pre-commit.com/#using-the-latest-version-for-a-repository for more details.
[WARNING] The 'rev' field of repo 'https://gitlab.com/pycqa/flake8' appears to be a mutable reference (moving tag / branch). Mutable references are never updated after first install and are not supported. See https://pre-commit.com/#using-the-latest-version-for-a-repository for more details.

Apparently we should just run 'pre-commit autoupdate' periodically, and
commit the config it generates.

Author: dann frazier
Author Date: 2021-02-01 16:18:29 UTC

installer-test-matrix: Add HWE option to focal config

Author: dann frazier
Author Date: 2021-01-26 20:00:55 UTC

jenkins-rss-to-trello-md.py: Initial commit

This tool is intended to help generate trello checklists for our weekly
scalebot job analysis rotations. The idea being that you can point it
at a copy of Jenkins' "Atom feed for just latest builds", and you'll
get some markdown you can cut & paste into a Trello checklist that has
clickable links, status info, etc.

Author: dann frazier
Author Date: 2020-10-27 22:24:38 UTC

Revert "Allow console access from all clients to work around LP#1887174"

The conserver packages in Ubuntu are now fixed, so this workaround is
no longer necessary. LP: #1887174

This reverts commit 65836177ea6b33c8580dd46924a0c010d8b67d0a.

Author: dann frazier
Author Date: 2020-10-23 16:52:07 UTC

Use the more concise yaml.safe_load() instead of Loader=yaml.SafeLoader

Author: dann frazier
Author Date: 2020-10-20 14:25:52 UTC

Correct spelling: s/sabre/saber/

Now that marketing info is available on Marvell's website, it is clear
that the correct spelling of the platform is "Saber".

Ref: https://www.marvell.com/content/dam/marvell/en/public-collateral/server-processors/marvell-server-processors-thunderx2-micron-nvme-ssd-high-performance-ceph-object-stores-white-paper-2018.pdf

Author: dann frazier
Author Date: 2020-10-08 13:26:09 UTC

labkey: Remove accidentally left-in debug code

Author: dann frazier
Author Date: 2020-10-06 20:52:34 UTC

Add hint to install conserver-client when console command not found

Author: dann frazier
Author Date: 2020-08-17 16:25:06 UTC

Omit the taishan2280v2 from xenial HWE tests

While it should work fine w/ xenial/HWE, we never certified it with xenial.
It was certified w/ bionic/GA:
  https://certification.ubuntu.com/hardware/201907-27213

Author: dann frazier
Author Date: 2020-08-11 01:03:06 UTC

Add a tool to generate a random ISO test matrix

In the past we have spent a lot of time considering which tests to run
on release candidates for upcoming point releases. I figured it might
help if we encapsulated those decisions in code so we don't have to
revisit those decisions each time.

This implementation will produce a set of tests that meet the following
criteria:

 - For "full coverage" systems, we want both ISO and PXE tests. If there
   is an HWE kernel available, we want to test that in both modes as well.
 - For "spot coverage" systems, we just want one test. It might be PXE
   or ISO, GA or (if available) HWE.
 - We try to evenly distribute PXE/ISO tests and GA/HWE tests.

A config file is used to describe platforms and releases. We'll need
to update it for new platforms, new releases, and when an existing
release gains an HWE kernel. The config file can also be used to
exclude platforms from certain tests, such as releases that are too old.

We might want to add other options in the future. For example, for
interim releases we may want a --spot-only. Also, while the "boot-options"
are limited to "GA" and "HWE" right now, that could be expanded in the
future to include things like ACPI mode, optimized kernels, 64K kernels,
etc.

Author: dann frazier
Author Date: 2020-05-07 20:33:05 UTC

Add pre-commit hooks for black, pyflakes3

Author: dann frazier
Author Date: 2020-01-24 17:18:43 UTC

Add update_conserver.sh

This script is meant to be ran on a user's laptop. It will locally
generate a new conserver.cf, copy it to the target server and reload
that conserver daemon.

Author: dann frazier
Author Date: 2019-11-27 21:29:11 UTC

Add lp-scripts/dgx2-performace-regression-googlesheet.py

Author: dann frazier
Author Date: 2019-11-26 17:22:57 UTC

Add console support

The conserver client (console) can be easily configured to point at a single
server, or presumably multiple servers, but it isn't able to figure out which
server a given machine is connected to. I've proposed that each lab describe
it's conserver config in a new 'lab.yaml' file. With this, labkey can
generated the appropriate 'console' client command. Presumably we could
extend this to support other console-attach methods - though I'm hoping
we'll be able to find a way to have conserver attach to everything.

Author: dann frazier
Author Date: 2019-11-04 22:50:25 UTC

gen_conserver_cf.py: Leave a trailing '\n'

Author: dann frazier
Author Date: 2019-10-03 14:33:07 UTC

Rename dump-subscribers.py to dump-bug-subscribers.py

Makes it easier (at least for me) to guess what the script does.

Author: dann frazier
Author Date: 2019-10-02 21:22:04 UTC

Add kunpeng920 project

Author: dann frazier
Author Date: 2019-10-02 14:02:32 UTC

Add clone-project-milestones.py

This script will copy series/milestones/releases from one project to another.

Author: dann frazier
Author Date: 2019-08-20 01:07:48 UTC

labkey: set explicit yaml loader

Fixes the following runtime warning:

labkey:42: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  machines.update(yaml.load(stream))

Author: dann frazier
Author Date: 2019-06-09 20:27:35 UTC

Convert ubuntu-18.04.3-ga-risky to more granular tags

We now have:
  ubuntu-18.04.3-ga-risk-high
  ubuntu-18.04.3-ga-risk-low
  ubuntu-18.04.3-ga-wont-fix
  ubuntu-18.04.3-ga-risk-tbd

Author: dann frazier
Author Date: 2019-06-05 21:03:37 UTC

lp-bulk-action.py: Add update-milestone command

Change the expected release date for a milestone across projects.

Author: dann frazier
Author Date: 2019-06-03 23:10:40 UTC

Add pearl2-tag-ubuntu-18.04.3-risky.py

Look for all bugs that have an ubuntu-18.04-hwe task that either has not
yet been given a milestone, or the milestone has an estimated release
date after the release date of ubuntu-18.04.3. These bugs get the
ubuntu-18.04.3-ga-risky tag applied. All other bugs get it removed.

Author: dann frazier
Author Date: 2019-03-05 18:10:41 UTC

lp-bulk-action.py: Require a subcommand

When called w/o an argument, lp-bulk-action.py gives a traceback:

$ ./lp-bulk-action.py
Traceback (most recent call last):
  File "./lp-bulk-action.py", line 77, in <module>
    if args.cmd == 'create-milestone':
AttributeError: 'Namespace' object has no attribute 'cmd'

Enforce the subcommand requirement so that we get a more useful message:

$ ./lp-bulk-action.py
usage: lp-bulk-action.py [-h]
                         {create-milestone,release-milestone,create-series}
                         ...
lp-bulk-action.py: error: the following arguments are required: cmd

Author: dann frazier
Author Date: 2019-02-22 10:11:10 UTC

Add mc-reset command to perform a cold reset of the BMC

IPMI also supports a "warm" reset, but I don't generally have a
use case for that.

Author: dann frazier
Author Date: 2018-10-22 18:56:17 UTC

Add release-milestone subcommand

Author: dann frazier
Author Date: 2018-09-26 20:13:23 UTC

Call out Incomplete bugs with the "needs-sru-justification" tag

Author: dann frazier
Author Date: 2018-08-28 20:52:14 UTC

Create a milestone for each enablement projects w/ associated series

Author: dann frazier
Author Date: 2018-08-13 22:57:42 UTC

Show "Incomplete" bugs as needing input from HiSilicon

Author: dann frazier
Author Date: 2018-07-12 23:49:56 UTC

ubuntu-18.04-sru-4 is now planned to occur before 18.04.1 GA

Author: dann frazier
Author Date: 2018-07-12 23:42:15 UTC

labkey: dispense with --maxdepth optimization

I added --maxdepth=1 thinking it'd save on download time, but it appears
to break 'labkey update':

$ labkey update
fatal: refusing to merge unrelated histories