Merge lp:~paelzer/britney/hints-ubuntu-bionic-linux-fails-march-2019 into lp:~ubuntu-sru/britney/hints-ubuntu-bionic

Proposed by Christian Ehrhardt 
Status: Merged
Merged at revision: 3120
Proposed branch: lp:~paelzer/britney/hints-ubuntu-bionic-linux-fails-march-2019
Merge into: lp:~ubuntu-sru/britney/hints-ubuntu-bionic
Diff against target: 27 lines (+5/-0)
2 files modified
apw (+3/-0)
ubuntu-release (+2/-0)
To merge this branch: bzr merge lp:~paelzer/britney/hints-ubuntu-bionic-linux-fails-march-2019
Reviewer Review Type Date Requested Status
Ubuntu Stable Release Updates Team Pending
Review via email: mp+365045@code.launchpad.net
To post a comment you must log in.
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Hi,
I'm not insisting on actually merging the changes, but wanted to have something to start a discussion on.

I happened to find when checking some of my ongoing SRUs that in Bionic a lot seems to be blocked on kernel fails, so I took a look.

On one hand there seem to be "new" custom kernels that need to be added
http://autopkgtest.ubuntu.com/packages/l/linux-gcp-edge/bionic/amd64
http://autopkgtest.ubuntu.com/packages/l/linux-oracle/bionic/amd64

And on the other hand the i386 runs fail for quite a while now.
http://autopkgtest.ubuntu.com/packages/l/linux/bionic/i386

I was wondering if we should add force-badtests as suggested in the MP or if in general there is another plan to resolve this for Bionic SRUs in general?

I set APW as reviewer being in both, the kernel and SRU team.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Sometimes it helps to check if only a few subtests usually fail to mask those, here an info on the stats overall of the last 40 runs:
$ check-autopkgtest-stats.sh -p linux -r bionic -c 40 -a i386 -P "^TestFail: Test failed"
Check last 40 test results for src:linux on releases 'bionic' on architectures 'i386'
Fetch Data to /tmp/tmp.dFoc5ZtuBN
Of the 40 last tests, we had these subtest failing per release/arch:

bionic
  i386
      1 rebuild ( 2.50%) ....................F...................
     40 ubuntu-regression-suite (100.00%) FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF

Pattern found, sum/order found patterns
     17 TestFail: Test failed for ftrace
     17 TestFail: Test failed for net
     17 TestFail: Test failed for seccomp
     15 TestFail: Test failed for timers

So those seem to be involved at least half the time of the last series of 100% fails.
I also found some test timeouts and some breaking on "/mnt/vfat-test-4902 is already mounted" and such.

Anyway - the TL;DR is that atm the i386 test for src:linux seems not useful and gates a lot of packages (I guess there will be more over time).

3118. By Christian Ehrhardt 

Use explcit versions for i386 hints

The kernel Team wants to resolve the issues or at least understand
what is going on. For now only mark the former few versions which
block things in proposed.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

After a discussion this morning on IRC.

[10:30] <cpaelzer> apw: have you seen the merge above ^^ ?
[10:30] <cpaelzer> that tries to hint a few unreliable kernel autopkgtest on bionic
[10:31] <apw> i saw you pointed at it, i think the second hunk is a no-brainer, the i386 thing less obviously ok
[10:33] <cpaelzer> apw: less ok might be true, but the results at http://autopkgtest.ubuntu.com/packages/l/linux/bionic/i386 seem clearly indicating that atm it is not working
[10:33] <cpaelzer> apw: we could hint i386 with a version
[10:33] <cpaelzer> to let things through
[10:33] <apw> cpaelzer, right and wanting to find out why it is being ignored is mattering to me
[10:33] <apw> cpaelzer, right, make it a version and i think it is acceptable
[10:33] <cpaelzer> ok, I'll make a version for now
[...]
[10:40] <cpaelzer> apw: I have checked what versions currently block things on SRUs, that is why I added the last four versions
[10:40] <apw> cpaelzer, ack thanks
[11:06] <cpaelzer> apw: you said ack to the britney MP but it isn't merged yet - will that be done later and I can forget it, or are there other things that need to resolve first?
[11:06] <apw> cpaelzer, mearly time for my fingers to get round to it

Due to that - anyone of the SRU Team has the permissions to do so, I have modified the review slot for other SRU members to help here - this doesn't have to be only on Andy's shoulders anymore.

Revision history for this message
Robie Basak (racb) wrote :

Merging based on apw's ack above.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'apw'
2--- apw 2018-05-02 07:04:45 +0000
3+++ apw 2019-04-08 08:38:29 +0000
4@@ -44,3 +44,6 @@
5 unblock linux-meta/4.15.0.20.23
6 force-skiptest linux-meta-raspi2/4.15.0.1010.9
7 unblock linux-meta-raspi2/4.15.0.1010.9
8+
9+# failing continuously since 4.15.0-37.40 to 4.15.0-47.50
10+force-badtest linux/4.15.0-48.51/i386 linux/4.15.0-47.50/i386 linux/4.15.0-46.49/i386 linux/4.15.0-45.48/i386
11
12=== modified file 'ubuntu-release'
13--- ubuntu-release 2019-02-22 06:30:25 +0000
14+++ ubuntu-release 2019-04-08 08:38:29 +0000
15@@ -80,10 +80,12 @@
16 force-badtest linux-azure-edge/all
17 force-badtest linux-euclid/all
18 force-badtest linux-gcp/all
19+force-badtest linux-gcp-edge/all
20 force-badtest linux-gke/all
21 force-badtest linux-hwe/all
22 force-badtest linux-hwe-edge/all
23 force-badtest linux-oem/all
24+force-badtest linux-oracle/all
25
26 # flaky test due to race with ipv6 RA due to deliberate networkd change
27 force-badtest netplan.io/0.36.1

Subscribers

People subscribed via source and target branches