[18.04 FEAT] zKVM: Add hardware CPU Model - qemu part

The remaining portion of this for Eoan is currently building and migrating via bug 1836066.

The new request here is to backport the combined changes to >=18.04 afterward.

The changes used for qemu 4.0 in Eoan (out of the two referred bugs) are:

(fixup) 86e636951ddc linux-user: fix __NR_semtimedop undeclared error
(optional) b1b9e0dc78a8 update-linux-headers: handle new header file

8cf108c5 linux-headers: add linux/mman.h
d9cb4336159a linux headers: update against Linux 5.2-rc1
eaf6f642abf1 s390x/cpumodel: ignore csske for expansion <--- this might already be in via LP1830704
2ec038836fa0 s390x/cpumodel: Miscellaneous-Instruction-Extensions Facility 3
5dacbe23d23c s390x/cpumodel: msa9 facility
54d65de0b525 s390x/cpumodel: vector enhancements
d220fabf1609 s390x/cpumodel: enhanced sort facility
afc7b8666b62 s390x/cpumodel: add Deflate-conversion facility
caef62430fed s390x/cpumodel: add gen15 defintions
c657e84faee4 s390x/cpumodel: wire up 8561 and 8562 as gen15 machines

(fixup) d05be57d s390: cpumodel: fix description for the new vector facility

For the requested backport we "only" need to consider:
- qemu 2.11
- qemu 3.1

Cosmic (2.12) will be out of support too soon and Xenial (2.5) is not requested.

In terms of "all required information for integration will be provided by IBM" will you provide backports of the set above for 2.11 and 3.1?
If you want to do so branches (git + debian packaging delta as quilt) are available at [1] and [2].

[1]: https://code.launchpad.net/~usd-import-team/ubuntu/+source/qemu/+git/qemu/+ref/ubuntu/disco-devel
[2]: https://code.launchpad.net/~usd-import-team/ubuntu/+source/qemu/+git/qemu/+ref/ubuntu/bionic-devel

------- Comment From <email address hidden> 2019-07-12 03:44 EDT-------
For disco we could either do backports or simply add 3 more patches to make all patches apply cleanly:

The following does apply with git-cherry-pick on top of the disco git that also has all

NEW:
include: update Linux headers to 4.21-rc1/5.0-rc1
(cherry picked from commit da054c646c777bcb487326f46cd5bf9932b05128)

in eoan
linux-headers: add linux/mman.h.
(cherry picked from commit 8cf108c5d159bccfa162a06e6abc35cfa4965781)

in eoan
linux headers: update against Linux 5.2-rc1
(cherry picked from commit d9cb4336159a00bd0d9c81b93f02874ef3626057)

in eoan
linux-user: fix __NR_semtimedop undeclared error
(cherry picked from commit 86e636951ddce2bb3a82fcaad5abc3779ba4f7ab)

NEW
s390x/cpumodel: mepochptff: warn when no mepoch and re-align group init
(cherry picked from commit ddf5d18af3ce3029d5b93222af5a9e8160d4c34b)

NEW
s390x/cpumodel: add z14 GA2 model
(cherry picked from commit f2a7d1577115bda770d619ff5bc45db1f656edc3)

in eoan
s390x/cpumodel: Miscellaneous-Instruction-Extensions Facility 3
(cherry picked from commit 2ec038836fa03103596023e4a1ad7e6eb50ee7c7)

in eoan
s390x/cpumodel: msa9 facility
(cherry picked from commit 5dacbe23d23c7f0395fa0e65ff1698f632846714)

in eoan
s390x/cpumodel: vector enhancements
(cherry picked from commit 54d65de0b525272edfa66eb75c3f67b183f8aff4)

in eoan
s390x/cpumodel: enhanced sort facility
(cherry picked from commit d220fabf16091ca5c26f3313541bdfb7435d6a08)

in eoan
s390x/cpumodel: add Deflate-conversion facility
(cherry picked from commit afc7b8666b62fe72fdbad7ab0c3f206d4c57c1d1)

in eoan
s390x/cpumodel: add gen15 defintions
(cherry picked from commit caef62430fed6e732d3e43d76752d165cf02ad67)

in eoan
s390x/cpumodel: wire up 8561 and 8562 as gen15 machines
(cherry picked from commit c657e84faee48d6ab36665da5a008b8f0649593d)

in eoan
(fixup) d05be57d s390: cpumodel: fix description for the new vector facility

I test build this and it seems to work fine.
Do you want to cherry-pick yourself or do you want to have my patches?

Will look into bionic next.

------- Comment on attachment From <email address hidden> 2019-07-12 04:24 EDT-------

Here is a tgz with patches for bionic.
Instead of adding even more patches here it seems better to just do a backport.

------- Comment From <email address hidden> 2019-07-12 04:51 EDT-------
There is a small bug in the bionic backports. Will attach a new version soon.

------- Comment on attachment From <email address hidden> 2019-07-12 05:30 EDT-------

new version of the bionic patches.

These group descriptions were not part of the first version.

--- a/target/s390x/cpu_features.h
+++ b/target/s390x/cpu_features.h
@@ -81,6 +81,10 @@ typedef enum {
     S390_FEAT_GROUP_MSA_EXT_6,
     S390_FEAT_GROUP_MSA_EXT_7,
     S390_FEAT_GROUP_MSA_EXT_8,
+ S390_FEAT_GROUP_MSA_EXT_9,
+ S390_FEAT_GROUP_MSA_EXT_9_PCKMO,
+ S390_FEAT_GROUP_ENH_SORT,
+ S390_FEAT_GROUP_DEFLATE_CONVERSION,
     S390_FEAT_GROUP_MAX,
 } S390FeatGroup;

and are now part of the new patches.

The three extra commit to pick for Disco should be fine, thanks for identifying them.
Thanks for thr v2 patch set for Bionic.

I have to clear an ongoing set of SRUs first but then we can take a look at adding all of this.

directly picking changes for Disco:
- at first works fine until f2a7d157 "add z14 GA2 model"
  context misses S390_FEAT_GROUP_MULTIPLE_EPOCH_PTFF in target/s390x/gen-features.c
  The define of default_GEN14_GA2 is easily added, but I wonder if the change before being
  ddf5d18a "mepochptff: warn when no mepoch and re-align group init" might also need:
  84176c79 "s390x/cpumodel: default enable mepoch for z14 and later"
  I'll try to build it without, but let me know what you think about that.

- further as we have known already d05be57d needs
  - target/s390x/cpu_features.c -> target/s390x/cpu_features_def.inc.h
  - DEF-FEAT -> FEAT_INIT
  No questions on this one.

The provided bionic-v2 patch set applies as promised and builds together with the disco branch in a PPA [1]. Once built it would be great if you (=IBM) could give those packages a test in regard to the new CPU models on the new HW.
Afterwared I can go on pushing this - once the current SRU cleared the queue- to become the next SRU.

[1]: https://launchpad.net/~paelzer/+archive/ubuntu/bug-1836154-s390x-hwmodels/+packages

------- Comment From <email address hidden> 2019-07-16 08:25 EDT-------
> directly picking changes for Disco:
> - at first works fine until f2a7d157 "add z14 GA2 model"
> context misses S390_FEAT_GROUP_MULTIPLE_EPOCH_PTFF in
> target/s390x/gen-features.c
> The define of default_GEN14_GA2 is easily added, but I wonder if the change
> before being
> ddf5d18a "mepochptff: warn when no mepoch and re-align group init" might
> also need:
> 84176c79 "s390x/cpumodel: default enable mepoch for z14 and later"
> I'll try to build it without, but let me know what you think about that.

Adding these patches would be ok but not strictly necessary as upstream does not
enable mepoch extension for the 3.1 machine.
If you decide to pick those nevertheless, then you could also pick
commit f2a7d157711 ("s390x/cpumodel: add z14 GA2 model")
But as I said. I think those are not strictly necessary.

>
> - further as we have known already d05be57d needs
> - target/s390x/cpu_features.c -> target/s390x/cpu_features_def.inc.h
> - DEF-FEAT -> FEAT_INIT
> No questions on this one.

Yes.

In addition to that there are 2 (or optional 3) bugfixes on top that would be important to have in bionic,disco (and also eaon).

mandatory:
https://github.com/cohuck/qemu/commit/0d4cb295db7503fbac2f5bb3e878a56630231fed
(s390x/cpumodel: also change name of vxbeh)
https://github.com/cohuck/qemu/commit/de6bb08570065eb318849dfd7f316448e51197b5
( s390x/cpumodel: remove esort from the default model)

optional:
https://github.com/cohuck/qemu/commit/5d8866c89817998a3d9c3055d5dc2b5a8e78658a
( s390x/cpumodel: change internal name of vxpdeh to match description )

I expect Cornelia to send a pull request soon so I hope to have final commit IDs soon.

------- Comment From <email address hidden> 2019-07-16 12:42 EDT-------
The fixes have been merged upstream (same commit ids)

optional
5d8866c89817 s390x/cpumodel: change internal name of vxpdeh to match description

mandatory
0d4cb295db75 s390x/cpumodel: also change name of vxbeh
de6bb0857006 s390x/cpumodel: remove esort from the default model

Please add them to the bionic and disco backports, as well as to eoan.

The optional patch does not apply to the old FEAT_INIT style macros, so I kept it out.
The other two apply as expected.

Matching PPA uploads started to build now in [1], please give them a try if they fulfill your needs before I go on and push them to Eoan and then start the SRU.

[1]: https://launchpad.net/~paelzer/+archive/ubuntu/bug-1836154-s390x-hwmodels

------- Comment From <email address hidden> 2019-07-26 03:13 EDT-------
ppa works fine.

Thanks for the check.
I'll upload the fixup to Eoan then ...

Eoan is building as it was tested from the PPA.
Let see how it passes tests to completely reach eoan.

@IBM - would you mind sniff testing the PPA [1] for Disco and Bionic as well then as prep to the SRU activities?

@Team - there are MPs up for review in [2][3].

[1]: https://launchpad.net/~paelzer/+archive/ubuntu/bug-1836154-s390x-hwmodels
[2]: https://code.launchpad.net/~paelzer/ubuntu/+source/qemu/+git/qemu/+merge/370708
[3]: https://code.launchpad.net/~paelzer/ubuntu/+source/qemu/+git/qemu/+merge/370709

This bug was fixed in the package qemu - 1:4.0+dfsg-0ubuntu5

---------------
qemu (1:4.0+dfsg-0ubuntu5) eoan; urgency=medium

  * d/p/ubuntu/lp-1836154-*: further fixups for HW CPU model for newer
    s390x machines (LP: #1836154)

 -- Christian Ehrhardt <email address hidden> Wed, 17 Jul 2019 13:20:42 +0200

The fixup in Eoan migrated waiting on a second pair of eyes to ack my MPs.
Then I just need to decide which of the ongoing changes to include in this times qemu SRU.

------- Comment From <email address hidden> 2019-07-30 11:14 EDT-------
(In reply to comment #21)
> Eoan is building as it was tested from the PPA.
> Let see how it passes tests to completely reach eoan.
>
> @IBM - would you mind sniff testing the PPA [1] for Disco and Bionic as
> well then as prep to the SRU activities?
>

I have tested the ppa for bionic in the scope of
Bug 179024 - launchpad 1836153
See my comments over there.

Ah I didn't realize that already was Bionic (could have been just Eoan).
Thanks - other than waiting for review and a few more fixes to group we are good for the time being then.

We have reviewed and tested the branch individually already.
I now had a test set running over night with the ones applied together that I intend to push in one SRU. All worked fine, uploading to -unapproved for the SRU Team to take a look.

Hello bugproxy, or anyone else affected,

Accepted qemu into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:3.1+dfsg-2ubuntu3.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello bugproxy, or anyone else affected,

Accepted qemu into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:2.11+dfsg-1ubuntu7.18 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

I have ran again some regression tests and they look fine (#1 in the test steps).

@IBM - would you mind for #2 to do some checking of this on new HW, so that we then can mark it verified?

------- Comment From <email address hidden> 2019-08-28 09:45 EDT-------
Disco looks good.

------- Comment From <email address hidden> 2019-08-28 13:16 EDT-------
bionic-proposed looks good (2.11+dfsg-1ubuntu7.18)

Perfect, thanks Christian!

All autopkgtests for the newly accepted qemu (1:2.11+dfsg-1ubuntu7.18) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

ubuntu-image/1.7+18.04ubuntu1 (ppc64el, s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#qemu

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

All autopkgtests for the newly accepted qemu (1:3.1+dfsg-2ubuntu3.4) for disco have finished running.
The following regressions have been reported in tests triggered by the package:

systemd/240-6ubuntu5.3 (i386, ppc64el)
nova/2:19.0.1-0ubuntu2.1 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/disco/update_excuses.html#qemu

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

I was checking and resetting the tests, there were some flaky results none of them related to the actual upload.

FYI - there is one PPC fix part of the overall current ongoing SRU which might delay the release of this change a bit. It seems it might be ok on the qemu side, but lacking a kernel change - never the less this might gate this for a while until resolved on the kernel side or stripped from this upload.

The verification of the Stable Release Update for qemu has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package qemu - 1:2.11+dfsg-1ubuntu7.18

---------------
qemu (1:2.11+dfsg-1ubuntu7.18) bionic; urgency=medium

  * d/p/ubuntu/lp-1832622-*: count cache flush Spectre v2 mitigation for ppc64
    (LP: #1832622)
  * d/p/ubuntu/lp-1840745-*: add amd ssbd / no-ssbd features (LP: #1840745)
  * d/p/ubuntu/lp-1836154-*: add HW CPU model for newer s390x machines
    (LP: #1836154)

 -- Christian Ehrhardt <email address hidden> Thu, 13 Jun 2019 08:08:33 +0200

This bug was fixed in the package qemu - 1:3.1+dfsg-2ubuntu3.4

---------------
qemu (1:3.1+dfsg-2ubuntu3.4) disco; urgency=medium

  * d/p/ubuntu/lp-1832622-*: count cache flush Spectre v2 mitigation for ppc64
    (LP: #1832622)
  * d/p/ubuntu/lp-1836154-*: add HW CPU model for newer s390x machines
    (LP: #1836154)

 -- Christian Ehrhardt <email address hidden> Thu, 13 Jun 2019 08:40:55 +0200

------- Comment From <email address hidden> 2019-09-05 09:41 EDT-------
IBM Bugzilla status -> closed, Fix Released with Bionic