Merge ~mfo/ubiquity:automatic_prepare into ubiquity:master

[Mauricio Faria de Oliveira]

This allows the prepare plugin to run in automatic mode (i.e.
with the 'automatic-ubiquity' option / ubiquity --automatic)

Currently the prepare plugin is just not run at all in that case,
but some users could benefit from it now that the ISOs ship with
e.g., nvidia drivers, to install 3rd party drivers automatically
during install, and not consume network bandwidth to download it.

LP: #1895351

[Old Description that introduces a preseed option]

This introduces the 'ubiquity/prepare_automatic' preseed option
to allow the prepare plugin to run in automatic mode (i.e. with
the 'automatic-ubiquity' option / ubiquity --automatic)

Currently the prepare plugin is just not run at all in that case,
but some users could benefit from it now that the ISOs ship with
e.g., nvidia drivers, to install 3rd party drivers automatically
during install, and not consume network bandwidth to download it.

The default value for the option is false, so not to change the
current behavior; otherwise more options that are not preseeded
in existing/working preseed files could be asked, and get stuck
in an automated/unattended install.

And when an user preseeds it to true, then it is expected that
such options are preseeded as needed. Note that the only option
needed as seen / not display the page is 'ubiquity/use_nonfree',
according to existing code.

LP: #1895351

[Mauricio Faria de Oliveira (mfo)]

I have tested this manually by editing files in the installer environment.

I plan to do more 'official' testing with packages and on Gtk and Kde,
but any early opinions/feedback/review on this idea are appreciated.

Thanks!
Mauricio

[Mauricio Faria de Oliveira (mfo)]

Patch tested on both GTK and KDE frontends of ubiquity
with the Groovy Desktop ISO (daily-build of 2020-09-30),
now with proper, patched ubiquity packages.

Test Summary: (details in the next comments.)

If the user does not specify 'ubiquity/prepare_automatic',
the current behavior is maintained; ie, the prepare page
is _not_ included if in automatic mode.

If the user does specify it, only then the prepare page
is included in automatic mode.

If the user does specify it _and_ 'ubiquity/use_nonfree',
the prepare page is included and does not ask questions
(ie, consistent with other pages' behavior.)

The installer on GTK/KDE frontends finishes successfully.
The installer log copy shows the prepare page ran OK.

The ubiquity package built successfully on Groovy/Focal
for all architectures on http://pad.lv/ppa/mfo/lp1895351

[Mauricio Faria de Oliveira (mfo)]

Test on Ubuntu (GTK frontend)
==============

Original:
---

root@ubuntu:~# ubiquity

These pages are shown, in order:
- Welcome
- Keyboard layout
- Updates and other software
- Installation type
- Where are you?
- Who are you?

root@ubuntu:~# ubiquity --automatic

The 'Updates and other software' page is not shown in automatic mode:
- Welcome
- Keyboard layout
- Installation type
- Where are you?
- Who are you?

Patched:
---

root@ubuntu:~# dpkg -i \
 ubiquity_20.10.10+prepauto1_amd64.deb \
 ubiquity-frontend-gtk_20.10.10+prepauto1_amd64.deb \
 ubiquity-ubuntu-artwork_20.10.10+prepauto1_all.deb

1) No option / Default behavior (== current behavior):

root@ubuntu:~# debconf-get ubiquity/prepare_automatic
false

root@ubuntu:~# ubiquity --debug --automatic

Result: There is *NO* 'Updates and other software' (current/default behavior maintained.)

root@ubuntu:~# cat /var/log/installer/debug
...
debconf (developer): <-- GET ubiquity/prepare_automatic
debconf (developer): --> 0 false
Oct 1 16:17:18 ubiquity: prepare: not included in automatic mode
...

2) Option set to 'true' but not 'seen'

root@ubuntu:~# debconf-set ubiquity/prepare_automatic true
root@ubuntu:~# debconf-get ubiquity/prepare_automatic
true

root@ubuntu:~# debconf-get ubiquity/use_nonfree
false

root@ubuntu:~# ubiquity --debug --automatic

Result: now there *IS* 'Updates and other software' page in the automatic mode.

root@ubuntu:~# cat /var/log/installer/debug
...
debconf (developer): <-- GET ubiquity/prepare_automatic
debconf (developer): --> 0 true
...
Oct 1 16:24:20 ubiquity: Starting up '['log-output', '-t', 'ubiquity', '--pass-stdout', '/usr/share/ubiquity/simple-plugins', 'prepare']' for ubi-prepare.Page
...

3) Option set to 'true' and 'seen' (eg, set preseed option in cmdline.)

Result: Again, there *IS* 'Updates and other software', and no questions
are asked on it (skips straight to 'Installation type') as 'seen' flag is set.

$ cat /proc/cmdline
BOOT_IMAGE=/casper/vmlinuz file=/cdrom/preseed/ubuntu.seed maybe-ubiquity quiet splash ubiquity/prepare_automatic=true ubiquity/use_nonfree=true ---

root@ubuntu:~# sudo debconf-get ubiquity/prepare_automatic
true

root@ubuntu:~# sudo debconf-get ubiquity/use_nonfree
true

The system installation completes successfully;
and its installer log shows the prepare page ran OK:

$ sudo grep -i prepare /var/log/installer/debug
Could not translate page (prepare): 'NoneType' object has no attribute 'replace'
debconf (developer): <-- GET ubiquity/prepare_automatic
Could not translate page (prepare): 'NoneType' object has no attribute 'replace'
Oct 1 16:24:20 ubiquity: Starting up '['log-output', '-t', 'ubiquity', '--pass-stdout', '/usr/share/ubiquity/simple-plugins', 'prepare']' for ubi-prepare.Page
Oct 1 16:24:23 ubiquity: ['log-output', '-t', 'ubiquity', '--pass-stdout', '/usr/share/ubiquity/simple-plugins', 'prepare'] exited with code 0

[Mauricio Faria de Oliveira (mfo)]

Test on Kubuntu (KDE frontend)
===============

Original:
---

root@kubuntu:~# ubiquity

These items are listed in the side bar:
- language
- keyboard
- wireless
- software
- disk setup
- timezone
- user info
- install

root@kubuntu:~# ubiquity --automatic

The 'wireless' and 'software' items are not listed in the side bar in automatic mode:
- language
- keyboard
- disk setup
- timezone
- user info
- install

Patched:
---

root@kubuntu:~# dpkg -i \
  ubiquity_20.10.10+prepauto1_amd64.deb \
  ubiquity-frontend-kde_20.10.10+prepauto1_all.deb \
  ubiquity-ubuntu-artwork_20.10.10+prepauto1_all.deb

1) No option / Default behavior (== current behavior):

root@kubuntu:~# debconf-get ubiquity/prepare_automatic
false

root@kubuntu:~# ubiquity --debug --automatic

Result: same list in the side bar (no 'wireless', no 'software')

root@kubuntu:~# cat /var/log/installer/debug
...
debconf (developer): <-- GET ubiquity/prepare_automatic
debconf (developer): --> 0 false
Oct 1 15:45:12 ubiquity: prepare: not included in automatic mode
...

2) Option set to 'true' but not 'seen'

root@kubuntu:~# debconf-set ubiquity/prepare_automatic true
root@kubuntu:~# debconf-get ubiquity/prepare_automatic
true

root@kubuntu:~# ubiquity --debug --automatic

Result: Changes in the list in the side bar! (now there IS 'software'!)

root@kubuntu:~# cat /var/log/installer/debug
...
debconf (developer): <-- GET ubiquity/prepare_automatic
debconf (developer): --> 0 true
...
Oct 1 15:46:42 ubiquity: Starting up '['log-output', '-t', 'ubiquity', '--pass-stdout', '/usr/share/ubiquity/simple-plugins', 'prepare']' for ubi-prepare.Page
...

3) Option set to 'true' and 'seen' (eg, set preseed option in cmdline.)

Result: Again, list now has 'software', and no questions are asked there
(skips straight to the 'disk layout' page) as the 'seen' flag is set.

...

The system installation completes successfully;
and its installer log shows the prepare page ran OK:

$ sudo grep prepare /var/log/installer/debug
Oct 1 15:58:11 ubiquity: Starting up '['log-output', '-t', 'ubiquity', '--pass-stdout', '/usr/share/ubiquity/simple-plugins', 'prepare']' for ubi-prepare.Page
Oct 1 15:58:17 ubiquity: ['log-output', '-t', 'ubiquity', '--pass-stdout', '/usr/share/ubiquity/simple-plugins', 'prepare'] exited with code 0

[Mauricio Faria de Oliveira (mfo)]

Rebased merge request.

[Jean-Baptiste Lallement (jibel)]

Thanks for this patch.

I find it redundant and non obvious to have to explicitly set prepare_automatic to run the prepare step in automatic mode. As a user I expect that all the steps will run in automatic mode. If for some reason we didn't want this step to be executed in automatic mode, this reason is apparently gone given the current content of this page.

Did you try to just remove the test on is_automatic in __init__ of PageGtk and PageKde? That would indeed change the behaviour of this plugin but it would then be aligned on others and it would then be possible to seed use_nonfree and execute simple-plugins at the end of this step.

review: Needs Information

[Mauricio Faria de Oliveira (mfo)]

Thanks for reviewing.

I guess this was only briefly mentioned in the bug, sorry, but this would target/land back on Focal, so the behavior couldn't change there.

Indeed the parameter usage is both redundant and non-obvious, but is the compromise solution for a stable, LTS release that I could find.

I guess we can go without it for the new release, you agree?

Yes, the way you suggested has been tested, but not used for the reason above. But if we could/should go with different patches for devel/stable release, that can change :)

I see that probably the reason why prepare is not run in automatic mode is gone, but I have no means or more extensive installer experience to propose that by myself.

If you/team suggest so, I'm happy to propose a simpler change for devel, but would ask for this change to be reviewed as for the Focal release, for 20.04.2.

Thanks!

[Mauricio Faria de Oliveira (mfo)]

Hi Jean-Baptiste,

Happy new year!

Just following up on this, in case you have a chance to review my previous comment.

Thanks!

[Mauricio Faria de Oliveira (mfo)]

Just for the record,

Discussed this w/ Jean-Baptiste today:
- Laney should have a look/review.
- I suggested that we set different defaults for the new option in devel/stable releases.
  It's true on Hirsute (so user doesn't need to specify it to get the new functionality, and can disable in case of issues)
  It's false on Focal (so user must specify it for the behavior to change.)

[Iain Lane (laney)]

Hey, thanks.

Should we go with something like this for focal, and then jibel's suggestion for ≥ hirsute? That would be another MP which drops the check totally. Hopefully if we do that soon then it's enough time to get it tested properly before release.

what do you think about that?

review: Needs Information

[Mauricio Faria de Oliveira (mfo)]

Hi Laney,

Thanks for reviewing.

Yes, that's an option, and I'd be happy to do it -- but thought of this point yesterday:

The idea to still provide the option on H+ (default to true/enable), so that we _can_ turn it off if needed, seemed to be a good thing.

Specifically thinking of the OEM team, as they use something else to install the 3rd party drivers, and I wondered whether now having the installer go and do it might conflict w/ something on their tooling and give them additional debug/work.

So it seemed that at least they (and possibly other users who already developed similar solutions for automated deployment of the 3rd party drivers) could possibly benefit from turning the option off, at least for a while.

Does that make sense?
Otherwise, happy to re-submit the patch as you suggested for H+, and later SRU this version to F.

Thanks,
Mauricio

[Iain Lane (laney)]

It's a fair point. I think: prepare the MR to drop the option, and we can ask someone from that team to review it and then if they like it we can go ahead.

[Mauricio Faria de Oliveira (mfo)]

MR updated to a version w/out the preseed option; will ask OEM Desktop team for review.

[Mauricio Faria de Oliveira (mfo)]

Test on Ubuntu (GTK frontend)
==============

This is on Hirsute Desktop daily ISO,
with the new patch (no preseed option.)

Original:
---

root@ubuntu:~# ubiquity

These pages are shown, in order:
- Welcome
- Keyboard layout
- Updates and other software
- Installation type
- Where are you?
- Who are you?

root@ubuntu:~# ubiquity --automatic

The 'Updates and other software' page is NOT shown in automatic mode:
- Welcome
- Keyboard layout
- Installation type
- Where are you?
- Who are you?

Patched:
---

# ls -1 *.deb
ubiquity_21.04.4+lp1895351.1_amd64.deb
ubiquity-frontend-gtk_21.04.4+lp1895351.1_amd64.deb
ubiquity-ubuntu-artwork_21.04.4+lp1895351.1_all.deb

# dpkg -i *.deb

1) Option 'ubiquity/use_nonfree' not specified (default)

root@ubuntu:~# ubiquity --automatic

Result: now there *IS* 'Updates and other software' page in the automatic mode.

2) Option 'ubiquity/use_nonfree' set to true

Result: Again, there *IS* 'Updates and other software', and no questions
are asked on it (skips straight to 'Installation type') as 'seen' flag is set.

root@ubuntu:~# debconf-set ubiquity/use_nonfree true
root@ubuntu:~# debconf-get ubiquity/use_nonfree
true

root@ubuntu:~# ubiquity --automatic

root@ubuntu:~# cat /var/log/installer/debug
...
Jan 13 22:31:07 ubiquity: Starting up '['log-output', '-t', 'ubiquity', '--pass-
stdout', '/usr/share/ubiquity/simple-plugins', 'prepare']' for ubi-prepare.Page
Jan 13 22:31:07 ubiquity: Watching for question patterns ubiquity/use_nonfree
Jan 13 22:31:07 debconf (filter): <-- INPUT high ubiquity/use_nonfree
Jan 13 22:31:07 debconf (filter): --> 30 question skipped
...

[Mauricio Faria de Oliveira (mfo)]

Test on Kubuntu (KDE frontend)
===============

This is on *Groovy* Kubuntu ISO,
(*Hirsute* Kubuntu daily ISO does NOT load ubiquity)
with the new patch (no preseed option.)

Original:
---

root@kubuntu:~# ubiquity

These items are listed in the side bar:
- language
- keyboard
- wireless
- software
- disk setup
- timezone
- user info
- install

root@kubuntu:~# ubiquity --automatic

The 'wireless' and 'software' items are not listed in the side bar in automatic mode:
- language
- keyboard
- disk setup
- timezone
- user info
- install

Patched:
---

# ls -1 *.deb
ubiquity_20.10.13.1+lp1895351.1_amd64.deb
ubiquity-frontend-kde_20.10.13.1+lp1895351.1_all.deb
ubiquity-ubuntu-artwork_20.10.13.1+lp1895351.1_all.deb

# sudo dpkg -i *.deb

1) Option 'ubiquity/use_nonfree' not specified (default)

root@kubuntu:~# ubiquity --automatic

Result: Changes in the list in the side bar! (now there IS 'software'!)

2) Option 'ubiquity/use_nonfree' set to true

root@kubuntu:~# debconf-set ubiquity/use_nonfree true
root@kubuntu:~# debconf-get ubiquity/use_nonfree
true

root@kubuntu:~# ubiquity --automatic

Result: Again, list now has 'software', and no questions are asked there
(skips straight to the 'disk setup' page) as the 'seen' flag is set.

root@kubuntu:~# cat /var/log/installer/debug
...
Jan 13 22:42:40 ubiquity: Starting up '['log-output', '-t', 'ubiquity', '--p
ass-stdout', '/usr/share/ubiquity/simple-plugins', 'prepare']' for ubi-prepa
re.Page
Jan 13 22:42:40 ubiquity: Watching for question patterns ubiquity/use_nonfre
e
Jan 13 22:42:40 debconf (filter): <-- INPUT high ubiquity/use_nonfree
Jan 13 22:42:40 debconf (filter): --> 30 question skipped
...

[Mauricio Faria de Oliveira (mfo)]

The OEM enablement team is apparently OK with the changes, but will need a few days to verify their tooling for any impacts.

They have recovery functionality that adds plugin(s) to ubiquity, and that is what needs testing w/ these changes.

I have clarified that the impact of the new version (w/out the option to gate the page from showing up or not) is just the page showing up in automatic mode now, and that can be skipped by preseeding 'ubiquity/use_nonfree', in their case probably to 'false', so it doesn't install any 3rd party drivers as they may likely be doing that in a later step already.

cheers,
Mauricio

[Mauricio Faria de Oliveira (mfo)]

Hi Iain,

We have positive test results and feedback from the OEM team:

From Alex Tu, Lead Software Engineer (OEM Enablement), today:

"""
So far, I don't see the impact of OEM image.
...
Given the test result and Ubiquity is only working in installation time that will always be well verified before we ship the OEM image.
I believe you can feel safe to merge the change.
"""

[Jean-Baptiste Lallement (jibel)]

Thank you for your work. Tested successfully on Hirsute.

review: Approve

[Mauricio Faria de Oliveira (mfo)]

Thanks for reviewing!

This is the MR for the focal backport w/ the preseed option: #396500 [1]

[1] https://code.launchpad.net/~mfo/ubiquity/+git/ubiquity/+merge/396500

diff --git a/debian/changelog b/debian/changelog
index 8683857..cc0e603 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,10 @@
+ubiquity (21.04.5) UNRELEASED; urgency=medium
+
+  * ubiquity/plugins/ubi-prepare.py: let the prepare page run in
+    automatic mode to install 3rd party drivers. (LP: #1895351)
+
+ -- Mauricio Faria de Oliveira <[e-mail]>  Wed, 13 Jan 2021 16:12:50 -0300
+
 ubiquity (21.04.4) hirsute; urgency=medium
 
   [ Jean-Baptiste Lallement ]
diff --git a/ubiquity/plugins/ubi-prepare.py b/ubiquity/plugins/ubi-prepare.py
index 8339803..2cacd7f 100644
--- a/ubiquity/plugins/ubi-prepare.py
+++ b/ubiquity/plugins/ubi-prepare.py
@@ -84,9 +84,6 @@ class PageGtk(PreparePageBase):
     restricted_package_name = 'ubuntu-restricted-addons'
 
     def __init__(self, controller, *args, **kwargs):
-        if self.is_automatic:
-            self.page = None
-            return
         self.controller = controller
         from ubiquity.gtkwidgets import Builder
         builder = Builder()
@@ -317,9 +314,6 @@ class PageKde(PreparePageBase):
 
     def __init__(self, controller, *args, **kwargs):
         from ubiquity.qtwidgets import StateBox
-        if self.is_automatic:
-            self.page = None
-            return
         self.controller = controller
         try:
             from PyQt5 import uic