Please update modemmanager in xenial to the 1.6 series

Packages uploaded to https://launchpad.net/~snappy-hwe-team/+archive/ubuntu/stacks-overlay and built for xenial. The versions are the same as for artful, plus additional patches for Dell modems in the modem-manager source package.

verified the packages in #1 on BRM5-DVT1-C2RX with WWAN DW5811 and DW5816. Both of these 2 WWAN cards work well.

BIOS: 0.5.1
IMAGE: X82

@Alfonso, got progress on SRU ?

I am looking now at the number of changes that are included between the versions to determine if this is indeed SRUable. Out of curiosity though - even though I also originally was thinking about backporting 1.6.8, would using 1.6.4 from zesty instead be also feasible? If we go the SRU-way, I now think using 1.6.4 might be easier, as otherwise we'd have to think about updating zesty to 1.4.8 as well (as after upgrading from xenial to zesty a user would get and older package, which is not acceptable).

Another question: did you also by any chance consider using the backports pocket for this purpose? This might be much easier to do than SRUing such a big delta.

I meant 1.6.8 of course, typo in package numbers.

@Lukasz yes, I think that using the version in zesty is perfectly feasible, I can prepare the packages for that. About using the backports pocket, I will let YC or Alex comment on that.

this requirement was came from LP: #1693756 ，which originally need the new stuff which be applied after Yakkety , so I believe so does Zesty.

and from LP: #1693756 comment #12, the Yakkety version looks also be maintained by modem manager maintainer
s ppa : https://launchpad.net/~aleksander-m/+archive/ubuntu/modemmanager-xenial

I'll also install the package from Zesty for a try to check if DW5816 still works fine.

@Lukasz, per https://help.ubuntu.com/community/UbuntuBackports, the Ubuntu Security Team does not update packages in Backports. I hope we use something with Security update.

How did the test with the zesty package go? Does it also look good for your purposes?

verified DW5811 and DW5816 on BRM5-DVT1-C2RX with packages from Zesty, they both work well.

--- a/dpkg-l.log (xenial)
+++ b/dpkg-l.log (install packages from zesty)

-ii libmbim-glib4:amd64 1.12.2-2ubuntu1oem2 amd64 Support library to use the MBIM protocol
-ii libmbim-proxy 1.12.2-2ubuntu1oem2 amd64 Proxy to communicate with MBIM ports
+ii libmbim-glib4:amd64 1.14.0-1 amd64 Support library to use the MBIM protocol
+ii libmbim-proxy 1.14.0-1 amd64 Proxy to communicate with MBIM ports
-ii libmm-glib0:amd64 1.4.12-1ubuntu1oem2 amd64 D-Bus service for managing modems - shared libraries
+ii libmm-glib0:amd64 1.6.4-1 amd64 D-Bus service for managing modems - shared libraries
-ii libqmi-proxy 1.12.6-1oem1 amd64 Proxy to communicate with QMI ports
+ii libqmi-glib5:amd64 1.16.2-1 amd64 Support library to use the Qualcomm MSM Interface (QMI) protocol
+ii libqmi-proxy 1.16.2-1 amd64 Proxy to communicate with QMI ports
-ii modemmanager 1.4.12-1ubuntu1oem2 amd64 D-Bus service for managing modems
+ii modemmanager 1.6.4-1 amd64 D-Bus service for managing modems

@Lukasz, I uploaded the backported packages (zesty->xenial) + patches from the snap in:

https://launchpad.net/~alfonsosanchezbeato/+archive/ubuntu/modem-manager-backport

Patches? Ok, this is something I was not aware of and can potentially be a problem. Where do those patches come from? I don't see them on the zesty version of modemmanager, same for the one we have in bionic. SRUs have rather strict requirements, one of which being that later series cannot revert functionality for packages, enforcing that any change - if applicable - needs to first be present in the development series (and at best any other upgrade path from the given upload).

Why were those changes only in the snap version of modemmanager? Are those unacceptable for general Ubuntu/Debian usage? Are those only valid for xenial? If there are no good reasons for those, we'd have to first get those into bionic (or, at best, Debian even) and only then thing about backporting that into the stable series.

@Lukasz those are patches for Dell modems (plano project, unbuntu core). It makes sense to upstream some of them in fact, and that is something I can try.

But, it would still make sense to backport the zesty packages to xenial, as that has the support needed for DW5816e (lp #1693756). If you prefer, we can go that way and leave out the patches for the moment until I get a response for MM upstream.

The main requirement is that those patches need to be first present in the devel series (so in bionic). I would propose forwarding them upstream and then actually distro-patching them into the current version in bionic. We can then easily pull those into zesty and xenial.

@Lukasz, do you mind that we backport zesty package to xenial first ?

Not sure how much time it takes for those Plano patch to get MM upstream.

The patches do not have to get accepted upstream, the requirement is that they're available in the latest development series - so for bionic.

So, if for this bug to be fixed the plano patches are required, the steps should be:
 * Uploading the patches to modemmanager bionic
 * The patches get backported to zesty modemmanager
 * Xenial gets a zesty backport of modemmanager with the patches

If the patches aren't needed instantly, I can sponsor a quick backport of the current zesty package to xenial if needed (e.g. without any new patches).

@Lukasz please go for sponsoring the zesty->xenial backport, that is enough to get support for the original modem OEM enablement needed. The work in the patches needs some time, so better do this for the moment. Thanks!

Could you try the packages that I prepared in a Bileto PPA?

https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3054

We will need to update this bug to include SRU information and a good rationale for why the backport is needed. Could one of you prepare such a description? [1]

[1] https://wiki.ubuntu.com/StableReleaseUpdates#SRU_Bug_Template

@sil2100, this backport is to fix the following 2 OEM project issues:

https://bugs.launchpad.net/bugs/1693756 -> to better support Sierra DW5815e WWAN module
https://bugs.launchpad.net/ubuntu/+source/modemmanager/+bug/1735134 -> to fix incorrect signal strength readings (by modemmanager) for Sierra DW5818 and DW5819 WWAN modules

I have tried ModemManager 1.6 series mentioned in comment #18, and Sierra MC7455, Sierra MC 7430 and Telit HE910D work fine with this version. The signal strength issue (bug #1735134) is not observed with this version.

verified #18 for DW5816 and DW5811 on BRM4-DVT1-C1H it works fine.

Also filled up the description which SRU needed below.

@Robert, please add information for your new cards as well if they also need these SRU.

[Impact]

 * the new modemmanager packages bring in DW5816 supporting.

 * These modemmanager packages is needed to support new devices.

[Test Case]

 * install modemmanager and it's dependencies from https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3054
libmbim-glib4:amd64 1.14.0-1ubuntu0.16.04.1
libmbim-proxy 1.14.0-1ubuntu0.16.04.1
libmm-glib0:amd64 1.6.4-1ubuntu0.16.04.1
libqmi-glib5:amd64 1.16.2-1ubuntu0.16.04.1
libqmi-proxy 1.16.2-1ubuntu0.16.04.1
modemmanager 1.6.4-1ubuntu0.16.04.1

 * reboot and try WWAN function to see if any regression there.

[Regression Potential]

 * The package comes from Zesty and should not have regression there.

[Other Info]

<alextu> robertliu, https://bugs.launchpad.net/ubuntu/+source/modemmanager/+bug/1725190/comments/21 , please review and add information if there are some I did not write there.
<robertliu> alextu, looks fine, thank you

@Łukasz
Thanks for backporting. #18 is verified passed in #20 and #21 and the needed information for SRU also in #21. Please ping me if anything else is needed.

On the Bileto silo I noticed that the autopkgtests for modemmanager-qt have failed, looks like some ABI-check tests are failing. I need to see if we don't need to re-build modemmanager-qt against the new modemmanager in the silo as an experiment.

Ok, so it seems we might need to pull in modemmanager-qt from zesty as well as the no-change rebuild is still failing. This is starting to turn into a rather big batch of changes that need to get SRUed - which doesn't cope well. It's getting more and more risky.

Just out of curiosity - did anyone look on how much work would be needed to get the new modem support backported to the xenial version of modemmanager?

Hi @Łukasz,
I had tried to pull the modem-manager/libqmi/libmbim of artful, just modify some version number of dependency packages, and built those packages successfully. Sorry, I don't remember if I touched anything about modemmanager-qt and if they work correctly with modemmanager-qt. The backported version works fine with Sierra MC7455/MC7430 and Telit HE910D which I have in my hand.

not experienced modemmanager-qt while backporting qmi over mbim for xenial before. (LP: #1693756)
btw, I also tried just adding zesty ppa in xenial to install modem-manager/libqmi/libmbim, it also works well with DW5811 and DW5816.

And it seems only one package for modemmanager-qt called "modemmanager-qt-dev":
modemmanager-qt-dev - Qt wrapper for ModemManager - devel files

So, we might need to check what the error message for ABI-check tests to understand how is the impaction.

@Łukasz,

per https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3054/+packages,

I can't see "no-change rebuild is still failing" per you mentioned in #24.

Can you share where is the fail log or anything ?

Sadly the test results got wiped out from Bileto for unknown reasons, I have re-run the tests. They should appear here https://bileto.ubuntu.com/#/ticket/3054 under the Automated Test Results.

What about my question about cherry-picking the new modem code to the xenial version of modemmanager? As said, seeing how many packages this SRU involves I'm starting to get worried about if this will get accepted.

@Łukasz, Do you think it's easier for us to SRU LP: #1693756, #17 ?

Hi @Łukasz,
I tried to run the acc test manually, and it didn't show any error.
The environment I used is vagrant + xenial/amd64 image (not fully upgrade all of the packages) + ppa mentioned in #18 + related packages/tools.
May I have your opinion what could be the differences?

I also did full-upgrade, and re-run the test. The test was success as well.

Thanks Robert for testing this! I'm actually re-running the test now with the overlay disabled. It looks like Bileto's britney by default uses the stable-phone-overlay which seems to have a slightly modified Qt5 stack - maybe that's related. Let's see if the run without it will also fail.

Ok, this actually looks better. It seems that the overlay was at fault here indeed. Should we proceed instantly with SRUing the zesty version (it can take time for it to get processed in the SRU queue), or do we want to release LP: #1693756 first? Anyway, looking into if we can just proceed with the backporting tomorrow in the morning.

Hi @Łukasz, per #26, zesty version did fix LP: #1693756. If the difference between SRU zesty version and the backport in LP: #1693756 is something within two weeks, I think we should proceed to SRU zesty version.

@Lukasz, if it's hard for you to pick one, can you just proceed with zesty SRU ?

I have pushed the 3 packages to the xenial UNAPPROVED queue. Now we need to wait for those to get reviewed by someone from the SRU team. It's a big SRU so the reviewers might raise some issues during the review process - let's keep track of any comments from them on this bug.

Hello Alfonso, or anyone else affected,

Accepted libmbim into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libmbim/1.14.0-1ubuntu0.16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Alfonso, or anyone else affected,

Accepted libqmi into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libqmi/1.16.2-1ubuntu0.16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Alfonso, or anyone else affected,

Accepted modemmanager into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/modemmanager/1.6.4-1ubuntu0.16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

I have tried the packages in xenial-proposed for two modems:

Sierra HL8548
ZTE MF626

I was able to connect and everything seemed to be working as expected.

verified
DW5816
DW5811

on BRM2-DVT2-C1R (201709-25745) with image X90 , BIOS 1.0.0, kernel 4.4.0-100

Both wwan cards works well.

u@u-Latitude-5290:~$ dpkg -l | grep "modem\|libmbim\|libqmi"
ii libmbim-glib4:amd64 1.14.0-1ubuntu0.16.04.1 amd64 Support library to use the MBIM protocol
ii libmbim-proxy 1.14.0-1ubuntu0.16.04.1 amd64 Proxy to communicate with MBIM ports
ii libmm-glib0:amd64 1.6.4-1ubuntu0.16.04.1 amd64 D-Bus service for managing modems - shared libraries
ii libqmi-glib5:amd64 1.16.2-1ubuntu0.16.04.1 amd64 Support library to use the Qualcomm MSM Interface (QMI) protocol
ii libqmi-proxy 1.16.2-1ubuntu0.16.04.1 amd64 Proxy to communicate with QMI ports
ii modemmanager 1.6.4-1ubuntu0.16.04.1 amd64 D-Bus service for managing modems

Looking at the autopkgtest failures - the network-manager s390x seems to be failing for every package since over a month. It shouldn't be related to the modemmanager changes. I might hint it in later.

The verification of the Stable Release Update for modemmanager has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package libqmi - 1.16.2-1ubuntu0.16.04.1

---------------
libqmi (1.16.2-1ubuntu0.16.04.1) xenial; urgency=medium

  * Backport to xenial (LP: #1725190).
  * debian/control: bump down debhelper to xenial vesion.

 -- Łukasz 'sil2100' Zemczak <email address hidden> Tue, 28 Nov 2017 12:55:17 +0100

This bug was fixed in the package libmbim - 1.14.0-1ubuntu0.16.04.1

---------------
libmbim (1.14.0-1ubuntu0.16.04.1) xenial; urgency=medium

  * Backport to xenial (LP: #1725190).

 -- Łukasz 'sil2100' Zemczak <email address hidden> Tue, 28 Nov 2017 13:02:56 +0100

This bug was fixed in the package modemmanager - 1.6.4-1ubuntu0.16.04.1

---------------
modemmanager (1.6.4-1ubuntu0.16.04.1) xenial; urgency=medium

  * Backport to xenial (LP: #1725190).
  * debian/control: bump down debhelper to xenial vesion.

 -- Łukasz 'sil2100' Zemczak <email address hidden> Tue, 28 Nov 2017 12:40:02 +0100