Merge ~canonical-kernel-team/+git/rt-hints:krzk/hints into ~canonical-kernel-team/+git/rt-hints:main

[Krzysztof Kozlowski]

[Stefan Bader (smb)]

I am not that confident that I already know all to be able to do good review here. So just as a comment, it would be nice if you would explain (ideally in the commit message as that will survive the longest) why LTP syscals is using that different comment format. Is there more advantage to it beyond readability? Maybe would be something to put into an example hint file which explains the various possibilities.

[Kleber Sacilotto de Souza (kleber-souza)]

In the case of ubuntu_ltp_syscalls I'm wondering if we should ever hint all versions given that we don't have the test-case granularity to hint the individual failures. The same happens with kernel_selftests.net. We run thousands of ltp_syscalls tests and if any of those start to fail we won't notice because everything is flagged as flaky for all future versions.

[Krzysztof Kozlowski (krzk)]

> I am not that confident that I already know all to be able to do good review
> here. So just as a comment, it would be nice if you would explain (ideally in
> the commit message as that will survive the longest) why LTP syscals is using
> that different comment format. Is there more advantage to it beyond
> readability? Maybe would be something to put into an example hint file which
> explains the various possibilities.

It's the same format - YAML string. If you mean why it is multi-line - because I mentioned multiple LP issues.

[Krzysztof Kozlowski (krzk)]

> In the case of ubuntu_ltp_syscalls I'm wondering if we should ever hint all
> versions given that we don't have the test-case granularity to hint the
> individual failures. The same happens with kernel_selftests.net. We run
> thousands of ltp_syscalls tests and if any of those start to fail we won't
> notice because everything is flagged as flaky for all future versions.

For other cases when we hope to fix the issue behind hint, I agree that we could limit it to specific version. However here the issues most likely won't be fixed, so the hinting will be done every release. I am not sure if the hinting person will actually check for new test failures instead of just yank-yank-paste... The proper solution is actually to be able to hint specific failures, e.g. regex on messages? Maybe we should implement it instead in autotest?

[Po-Hsu Lin (cypressyew)]

> > In the case of ubuntu_ltp_syscalls I'm wondering if we should ever hint all
> > versions given that we don't have the test-case granularity to hint the
> > individual failures. The same happens with kernel_selftests.net. We run
> > thousands of ltp_syscalls tests and if any of those start to fail we won't
> > notice because everything is flagged as flaky for all future versions.
>
For the test granularity, I think I can fix ubuntu_ltp_syscalls like how we test the kvm-unit-test, we build the test first, and read from the test plan file to make it one test per page.

> For other cases when we hope to fix the issue behind hint, I agree that we
> could limit it to specific version. However here the issues most likely won't
> be fixed, so the hinting will be done every release. I am not sure if the
> hinting person will actually check for new test failures instead of just yank-
> yank-paste... The proper solution is actually to be able to hint specific
> failures, e.g. regex on messages? Maybe we should implement it instead in
> autotest?

I have done this regex thing before [1], this is somewhat a predecessor of rt-hinting. It cost some effort to maintain this db.
With more people looking on LTP tests, I think maybe we can focus on fixing failures now, and hint those known issues (or even blacklist it in the ubuntu_ltp_syscalls/testcase_blacklist.py). We will need to come up with a solution for the kselftests/net suite.

[1] https://git.launchpad.net/~cypressyew/+git/regression-testing-reviewer

diff --git a/xenial-linux-oracle.yaml b/xenial-linux-oracle.yaml
 new file mode 100644
index 0000000..a22a31f
--- /dev/null
+++ b/xenial-linux-oracle.yaml
@@ -0,0 +1,51 @@
+DEFAULTS:
+  series: xenial
+  source: linux-oracle
+  cloud: oracle
+
+ubuntu_ltp_syscalls:
+  - version: 4\.15\.0.*
+    # TODO: narrow test to fallocate06
+    test-case: syscalls
+    state: FLAKY
+    comment: |
+      fallocate06
+      lp:#1866323 - Fix would require backport for btrfs, no plans for backport to Xenial
+      fanotify09
+      lp:#1876684 - No plans for backport to Xenial
+      ptrace10
+      lp:#1900951 - No plans for fixing on Xenial
+
+ubuntu_kvm_unit_tests:
+  - version: 4\.15\.0.*
+    test-case: apic
+    state: FLAKY
+    comment: 'lp:#1918689 (x2apic id matches cpuid)'
+  - version: 4\.15\.0.*
+    test-case: vmx_host_state_area
+    state: FLAKY
+    comment: Log notes that this test can fail on older Intel HW
+  - version: 4\.15\.0.*
+    test-case: vmx_apic_passthrough_thread
+    state: FLAKY
+    comment: 'lp:#1918692 - ubuntu-kvm-unit-test/vmx_apic_passthrough_thread reports assertion failure'
+  - version: 4\.15\.0.*
+    test-case: vmx_hlt_with_rvi_test
+    state: FLAKY
+    comment: 'lp:#1923284 - vmx_host_state_area / vmx_intr_window_test / vmx_nmi_window_test / vmx_hlt_with_rvi_test fails with timeout on Bionic'
+  - version: 4\.15\.0.*
+    test-case: vmx_host_state_area
+    state: FLAKY
+    comment: 'lp:#1923284 - vmx_host_state_area / vmx_intr_window_test / vmx_nmi_window_test / vmx_hlt_with_rvi_test fails with timeout on Bionic'
+  - version: 4\.15\.0.*
+    test-case: vmx_intr_window_test
+    state: FLAKY
+    comment: 'lp:#1923284 - vmx_host_state_area / vmx_intr_window_test / vmx_nmi_window_test / vmx_hlt_with_rvi_test fails with timeout on Bionic'
+  - version: 4\.15\.0.*
+    test-case: vmx_nmi_window_test
+    state: FLAKY
+    comment: 'lp:#1923284 - vmx_host_state_area / vmx_intr_window_test / vmx_nmi_window_test / vmx_hlt_with_rvi_test fails with timeout on Bionic'
+  - version: 4\.15\.0.*
+    test-case: vmx_pending_event_test
+    state: FLAKY
+    comment: 'lp:#1866591 - vmx_pending_event_test failed in ubuntu_kvm_unit_tests'