Merge ~enr0n/ubuntu-release-upgrader:ubuntu/main into ubuntu-release-upgrader:ubuntu/main

[Nick Rosbrook]

Fix several bugs in ubuntu-release-upgrader. This extends/replaces https://code.launchpad.net/~gunnarhj/ubuntu-release-upgrader/+git/ubuntu-release-upgrader/+merge/451815.

To ease review, I purposely left out the pre-build.sh stuff and debian/changelog, because I will do that when I upload.

[Gunnar Hjalmarsson (gunnarhj)]

Hi Nick,

I have only had a brief look at it so far, but can't help wondering:

When I struggled with the other MP, I found that setting DBUS_SESSION_BUS_ADDRESS and using subprocess instead of Gio.Settings were needed to make it work. Can you tell what the 'secret' is, or was i simply mistaken as regards those details?

[Nick Rosbrook (enr0n)]

DBUS_SESSION_BUS_ADDRESS is set in do-release-upgrade, so it is set correctly in this spot when the upgrade is run "for real". When you test manually, you are entering at a different place (i.e. do-release-upgrade is not executed directly), so you need to set in manually, e.g.

sudo env DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus ./mantic

[Gunnar Hjalmarsson (gunnarhj)]

Thanks for that explanation. As regards Gio I see that you used "settings.sync()" which I didn't. Maybe that's it. Or not, see below.

I have a couple of observations on the part of the code I understand. :/

The partition() method does not safely split the value into font family and size. It ought to work for e.g. 'Ubuntu 11', since the family is one single word, but it didn't work for me when testing. I had set 'Ubuntu 14' and it switched to 'Sans 11'.

=> This makes we wonder if the fetching of the user value is at all successful.

(And it would safely fail if the user value was e.g. 'Liberation Sans 12'.)

Another thing is that you use "settings.get_string" to grab the user font. That should give you a string irrespective of whether there actually is a specific user value or not. If not it gives you the system default ('Ubuntu 11' in standard Ubuntu).

In the equivalent variant of my code I used "settings.get_user_value" instead, which gives you None if there is no specific user value, which means that you can reset it with "gsettings reset".

With your way, the user dconf database will always get a font-name value even if there was no such value when the upgrade started. Maybe subtle and not so important...

review: Needs Fixing

[Nick Rosbrook (enr0n)]

> Thanks for that explanation. As regards Gio I see that you used
> "settings.sync()" which I didn't. Maybe that's it. Or not, see below.
>
> I have a couple of observations on the part of the code I understand. :/
>
> The partition() method does not safely split the value into font family and
> size. It ought to work for e.g. 'Ubuntu 11', since the family is one single
> word, but it didn't work for me when testing. I had set 'Ubuntu 14' and it
> switched to 'Sans 11'.

Ah yes, this should be changed. I will test the font size issue, but obviously that's not the intention.
>
> => This makes we wonder if the fetching of the user value is at all
> successful.
>
> (And it would safely fail if the user value was e.g. 'Liberation Sans 12'.)
>
> Another thing is that you use "settings.get_string" to grab the user font.
> That should give you a string irrespective of whether there actually is a
> specific user value or not. If not it gives you the system default ('Ubuntu
> 11' in standard Ubuntu).
>
> In the equivalent variant of my code I used "settings.get_user_value" instead,
> which gives you None if there is no specific user value, which means that you
> can reset it with "gsettings reset".
>
> With your way, the user dconf database will always get a font-name value even
> if there was no such value when the upgrade started. Maybe subtle and not so
> important...
In the general case, get_string('font-name') should give the same as get_user_value('font-name').get_string(), and if not I think getting the system default is the right thing to do.

[Gunnar Hjalmarsson (gunnarhj)]

On 2023-09-28 15:42, Nick Rosbrook wrote:
> In the general case, get_string('font-name') should give the same as
> get_user_value('font-name').get_string(), and if not I think getting
> the system default is the right thing to do.

Well, I'd say that the general case is that the user did not change the desktop font, so my belief is that they differ more often than not.

But this is a detail I'm not going to argue about more.

[Nick Rosbrook (enr0n)]

Alright, yeah I was missing something with the Gio.Settings approach, so shelling out to gsettings seems the way to go. I did this with `systemd-run --user -M $user@.host ...` to simplify the permissions/user env. That way, there is no need to change our running env or EUID -- just need to get the original UID via SUDO_UID or PKEXEC_UID.

[Gunnar Hjalmarsson (gunnarhj)]

TMTOWTDI. :) Looking forward to next variant.

[Nick Rosbrook (enr0n)]

> TMTOWTDI. :) Looking forward to next variant.

It was already pushed.

[Gunnar Hjalmarsson (gunnarhj)]

This looks good to me now. Didn't know that the rpartition() method existed, but it fits well in this context. And it did The Right Thing for me when testing.

Thanks for your work, and especially the idea to postpone the font resetting until after the reboot!

review: Approve

[Nick Rosbrook (enr0n)]

> This looks good to me now. Didn't know that the rpartition() method existed,
> but it fits well in this context. And it did The Right Thing for me when
> testing.
>
> Thanks for your work, and especially the idea to postpone the font resetting
> until after the reboot!

Glad it works well for you now. Thanks for all of your help on this!

[Brian Murray (brian-murray)]

This looks great to me, my only suggestion would be putting 'Sans' in a variable in case we change our minds some day.

review: Needs Fixing

[Brian Murray (brian-murray)]

review: Approve

[Nick Rosbrook (enr0n)]

> This looks great to me, my only suggestion would be putting 'Sans' in a
> variable in case we change our minds some day.

Ack, will do before I merge. Thanks!

[Gunnar Hjalmarsson (gunnarhj)]

On 2023-09-29 17:18, Brian Murray wrote:
> ... in case we change our minds some day.

FTR I have anticipated that we may want to change our minds in next cycle:

https://git.launchpad.net/~ubuntu-core-dev/ubuntu-seeds/+git/platform/commit/?id=67c4e53c

i.e. for the case the fonts-noto source package gets updated with latest upstream.

diff --git a/DistUpgrade/DistUpgradeController.py b/DistUpgrade/DistUpgradeController.py
index 5e4e125..bb30eb2 100644
--- a/DistUpgrade/DistUpgradeController.py
+++ b/DistUpgrade/DistUpgradeController.py
@@ -644,6 +644,15 @@ class DistUpgradeController(object):
                ):
                 continue
 
+            # subiquity was inadvertently configuring installed systems to use
+            # $cc.archive.ubuntu.com as the mirror for the security pocket,
+            # instead of security.ubuntu.com. (LP: #2036679)
+            if (
+                entry.dist == f'{self.fromDist}-security' and
+                'archive.ubuntu.com/ubuntu' in entry.uri
+            ):
+                entry.uri = self.security_source_uri
+
             new_list.append(entry)
         self.sources.list = new_list
 
diff --git a/DistUpgrade/DistUpgradeQuirks.py b/DistUpgrade/DistUpgradeQuirks.py
index 0bd1ef9..0626694 100644
--- a/DistUpgrade/DistUpgradeQuirks.py
+++ b/DistUpgrade/DistUpgradeQuirks.py
@@ -25,9 +25,11 @@ import distro_info
 import glob
 import logging
 import os
+import pwd
 import re
 import hashlib
 import subprocess
+import pathlib
 from subprocess import PIPE, Popen
 
 from .utils import get_arch
@@ -55,6 +57,7 @@ class DistUpgradeQuirks(object):
         self._snap_list = None
         self._from_version = None
         self._to_version = None
+        self._did_change_font = False
 
     # the quirk function have the name:
     #  $Name (e.g. PostUpgrade)
@@ -168,6 +171,7 @@ class DistUpgradeQuirks(object):
         self._killKBluetooth()
         self._pokeScreensaver()
         self._stopDocvertConverter()
+        self._set_generic_font()
 
     # individual quirks handler that run *right before* the dist-upgrade
     # is calculated in the cache
@@ -737,7 +741,9 @@ class DistUpgradeQuirks(object):
         # _calculateSnapSizeRequirements call.
         for snap, snap_object in self._snap_list.items():
             command = snap_object['command']
-            if command == 'refresh':
+            if command == 'switch':
+                # TODO: This status should be updated, but the time of
+                # this change to snap switch is post-translation freeze.
                 self._view.updateStatus(_("refreshing snap %s" % snap))
                 popenargs = ["snap", command,
                              "--channel", snap_object['channel'], snap]
@@ -1194,13 +1200,13 @@ class DistUpgradeQuirks(object):
             self._view.processEvents()
             if re.search("^installed: ", snap_info[0], re.MULTILINE):
                 logging.debug("Snap %s is installed" % snap)
-                # its not tracking the release channel so don't refresh
+                # its not tracking the release channel so don't switch
                 if not re.search(r"^tracking:.*%s" % from_channel,
                                  snap_info[0], re.MULTILINE):
                     logging.debug("Snap %s is not tracking the release channel"
                                   % snap)
                     continue
-                snap_object['command'] = 'refresh'
+                snap_object['command'] = 'switch'
             else:
                 # Do not replace packages not installed
                 cache = self.controller.cache
@@ -1257,7 +1263,7 @@ class DistUpgradeQuirks(object):
                             logging.debug("Snap %s is being used by %s. "
                                           "Switching it to stable track"
                                           % (snap, plug_snap))
-                            snap_object['command'] = 'refresh'
+                            snap_object['command'] = 'switch'
                             snap_object['channel'] = 'stable'
                             break
 
@@ -1447,3 +1453,104 @@ class DistUpgradeQuirks(object):
         except IOError as exc:
             logging.error("unable to write new boot config to %s: %s; %s",
                           boot_config_filename, exc, failure_action)
+
+    def _set_generic_font(self):
+        """ Due to changes to the Ubuntu font we enable a generic font
+            (in practice DejaVu or Noto) during the upgrade.
+            See https://launchpad.net/bugs/2034986
+        """
+        temp_font = 'Sans'
+
+        if self._did_change_font:
+            return
+
+        try:
+            uid = int(os.getenv('SUDO_UID', os.getenv('PKEXEC_UID')))
+            pwuid = pwd.getpwuid(uid)
+        except ValueError:
+            logging.debug(
+                'Cannot determine non-root UID, will not change font'
+            )
+            return
+
+        r = subprocess.run(
+            ['systemd-run', '--user', '-M', f'{pwuid.pw_name}@.host',
+             '--wait', '--pipe', '-q', '--',
+             '/usr/bin/gsettings', 'get',
+             'org.gnome.desktop.interface', 'font-name'],
+            stdout=subprocess.PIPE,
+            encoding='utf-8',
+        )
+
+        (font, _, size) = r.stdout.strip('\'\n').rpartition(' ')
+        font = font or 'Ubuntu'
+        try:
+            int(size)
+        except ValueError:
+            size = '11'
+
+        logging.debug(f'Setting generic font {temp_font} {size} during the '
+                      f'upgrade. Original font is {font} {size}.')
+
+        r = subprocess.run([
+            'systemd-run', '--user', '-M', f'{pwuid.pw_name}@.host',
+            '--wait', '--pipe', '-q', '--',
+            '/usr/bin/gsettings', 'set', 'org.gnome.desktop.interface',
+            'font-name', f'"{temp_font} {size}"'
+        ])
+        if r.returncode != 0:
+            logging.debug(f'Failed to change font to {temp_font} {size}')
+            return
+
+        self._did_change_font = True
+
+        # Touch a file to indiate that the font should be restored on the next
+        # boot.
+        need_font_restore_file = os.path.join(
+            pwuid.pw_dir, '.config/upgrade-need-font-restore'
+        )
+        os.makedirs(os.path.dirname(need_font_restore_file), exist_ok=True)
+        pathlib.Path(need_font_restore_file).touch(mode=0o666)
+        os.chown(need_font_restore_file, pwuid.pw_uid, pwuid.pw_gid)
+
+        # If we set the font back to normal before a reboot, the font will
+        # still get all messed up. To allow normal usage whether the user
+        # reboots immediately or not, create a service that will run only if a
+        # ~/.config/upgrade-need-font-restore exists, and then remove that file
+        # in ExecStart. This has the effect of creating a one-time service on
+        # the next boot.
+        unit_file = '/usr/lib/systemd/user/upgrade-restore-font.service'
+        os.makedirs(os.path.dirname(unit_file), exist_ok=True)
+
+        with open(unit_file, 'w') as f:
+            f.write(
+                '# Auto-generated by ubuntu-release-upgrader\n'
+                '[Unit]\n'
+                'Description=Restore font after upgrade\n'
+                'After=graphical-session.target dconf.service\n'
+                'ConditionPathExists=%h/.config/upgrade-need-font-restore\n'
+                '\n'
+                '[Service]\n'
+                'Type=oneshot\n'
+                'ExecStart=/usr/bin/gsettings set '
+                f'org.gnome.desktop.interface font-name \'{font} {size}\'\n'
+                'ExecStart=/usr/bin/rm -f '
+                '%h/.config/upgrade-need-font-restore\n'
+                '\n'
+                '[Install]\n'
+                'WantedBy=graphical-session.target\n'
+            )
+
+        subprocess.run([
+            'systemctl',
+            '--user', '-M', f'{pwuid.pw_name}@.host',
+            'daemon-reload'
+        ])
+        r = subprocess.run([
+            'systemctl',
+            '--user', '-M', f'{pwuid.pw_name}@.host',
+            'enable', os.path.basename(unit_file)
+        ])
+        if r.returncode != 0:
+            logging.debug(f'Failed to enable {os.path.basename(unit_file)}. '
+                          'Font will not be restored on reboot')
diff --git a/DistUpgrade/DistUpgradeVersion.py b/DistUpgrade/DistUpgradeVersion.py
index 71fb1a7..2b0dece 100644
--- a/DistUpgrade/DistUpgradeVersion.py
+++ b/DistUpgrade/DistUpgradeVersion.py
@@ -1 +1 @@
-VERSION = '23.10.6'
+VERSION = '23.10.7'
diff --git a/data/mirrors.cfg b/data/mirrors.cfg
index 2c54596..59bebe7 100644
--- a/data/mirrors.cfg
+++ b/data/mirrors.cfg
@@ -213,7 +213,6 @@ http://mirror.aminidc.com/ubuntu/
 http://ubuntu.man.lodz.pl/ubuntu/
 http://ubuntu.dts.mg/ubuntu/
 http://ubuntu.mirror.true.nl/ubuntu/
-http://mirror.wiru.co.za/ubuntu/
 http://mirrors.up.pt/ubuntu/
 http://www.club.cc.cmu.edu/pub/ubuntu/
 http://archive.ubuntu.mirror.ba/ubuntu/
@@ -278,7 +277,6 @@ http://ftp.belnet.be/ubuntu/
 http://repo.miserver.it.umich.edu/ubuntu/
 http://no.mirrors.blix.com/ubuntu/
 http://ftp.jaist.ac.jp/pub/Linux/ubuntu/
-http://mirrors.asnet.am/ubuntu/
 http://mirrors.liquidweb.com/ubuntu/
 http://mirrors.dotsrc.org/ubuntu/
 http://mirrors.melbourne.co.uk/ubuntu/
@@ -290,7 +288,6 @@ http://mirror.docker.ru/ubuntu/
 http://mirror.pit.teraswitch.com/ubuntu/
 http://mirrors.sohu.com/ubuntu/
 http://mirror.metrocast.net/ubuntu/
-http://reflector.westga.edu/repos/Ubuntu/archive/
 http://hk.mirrors.thegigabit.com/ubuntu/
 http://mirror.eu-fr.kamatera.com/ubuntu/
 http://mirror.nwlab.tk/ubuntu/
@@ -434,7 +431,6 @@ https://repo.extreme-ix.org/ubuntu/
 https://mirror.coganng.com/ubuntu-ports/
 https://mirror.bjtu.edu.cn/ubuntu/
 https://ftp.tc.edu.tw/Linux/ubuntu/
-https://mirror.onlinehosting.com.tr/ubuntu/
 http://mirror.cepatcloud.id/ubuntu/
 https://mirror.dkm.cz/ubuntu/
 http://ossmirror.mycloud.services/os/linux/ubuntu/
@@ -452,7 +448,6 @@ https://mirror.lyrahosting.com/ubuntuarchive/
 https://ftp.sh.cvut.cz/ubuntu/
 https://mirror.pulsant.com/sites/ubuntu-archive/
 https://mirror.netcologne.de/ubuntu/
-http://mirror.beon.co.id/ubuntu/
 https://mirrors.nipa.cloud/ubuntu/
 https://ubuntu.lafibre.info/ubuntu/
 https://mirror.hostart.az/Ubuntu/
@@ -492,7 +487,6 @@ https://mirrors.nav.ro/ubuntu/
 https://mirror.webworld.ie/ubuntu/
 https://mirror.scaleuptech.com/ubuntu/
 https://mirrors.nxtgen.com/ubuntu-mirror/ubuntu/
-http://mirror.soonkeat.sg/ubuntu/
 https://it1.mirror.vhosting-it.com/ubuntu/
 https://it2.mirror.vhosting-it.com/ubuntu/
 https://ubuntu.hi.no/archive/
@@ -564,7 +558,6 @@ https://mirror.servaxnet.com/ubuntu/
 http://mirror.vietnix.vn/ubuntu/
 https://ubuntu.hostiran.ir/ubuntuarchive/
 http://repo.ugm.ac.id/ubuntu/
-https://mirrors.zju.edu.cn/ubuntu/
 http://mirror.espol.edu.ec/ubuntu/
 http://ubuntu.mirror.iweb.ca/
 https://labs.eif.urjc.es/mirror/ubuntu/
@@ -575,9 +568,6 @@ http://mirror.timeweb.ru/ubuntu/
 http://ubuntu.byteiran.com/ubuntu/
 https://ubuntu.ccns.ncku.edu.tw/ubuntu/
 https://mirror.repository.id/ubuntu/
-https://mirror.papua.go.id/ubuntu/
-https://mirror.faizuladib.com/ubuntu/
-https://mirror.gi.co.id/ubuntu/
 https://mirrors.jlu.edu.cn/ubuntu/
 http://mirror.unesp.br/ubuntu/
 http://ubuntu.org.ua/ubuntu/
@@ -634,3 +624,11 @@ https://repo.jing.rocks/ubuntu/
 https://mirror.citrahost.com/ubuntu/
 https://mirror.siwoo.org/ubuntu/
 https://mirror.gsl.icu/ubuntu/
+http://linuxmirrors.ir/pub/ubuntu/
+https://mirror.neftm.ru/ubuntu/
+https://mirror.timkevin.us/ubuntu/
+http://mirror.lzu.edu.cn/ubuntu/
+https://ftp.kaist.ac.kr/ubuntu/
+https://mirrors.xtom.de/ubuntu/
+https://mirror.netzwerge.de/ubuntu/
+https://mirrors.ocf.berkeley.edu/ubuntu-ports/
diff --git a/debian/changelog b/debian/changelog
index c97e3e8..6ee8764 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,15 @@
+ubuntu-release-upgrader (1:23.10.7) mantic; urgency=medium
+
+  * DistUpgradeQuirks: Use generic font temporarily at upgrade
+    (LP: #2034986)
+  * DistUpgradeQuirks: Switch snap channels instead of refresh
+    (LP: #2036765)
+  * DistUpgradeController: Ensure security archive is used for security pocket
+    (LP: #2036679)
+  * Run pre-build.sh: updating mirrors and demotions.
+
+ -- Nick Rosbrook <[e-mail]>  Fri, 29 Sep 2023 10:36:00 -0400
+
 ubuntu-release-upgrader (1:23.10.6) mantic; urgency=medium
 
   * DistUpgrade: do not migrate to deb822 by default (LP: #2036288)
diff --git a/utils/demoted.cfg b/utils/demoted.cfg
index 4093061..fd3c64f 100644
--- a/utils/demoted.cfg
+++ b/utils/demoted.cfg
@@ -43,9 +43,13 @@ libgcc-12-dev-s390x-cross
 libgfortran-12-dev
 libmlir-15
 libmlir-15-dev
+libpcre16-3
+libpcre3
+libpcre3-dbg
+libpcre3-dev
+libpcre32-3
+libpcrecpp0v5
 libpolly-15-dev
-libpoppler126
-libstd-rust-dev
 libstdc++-12-dev
 libstdc++-12-dev-arm64-cross
 libstdc++-12-dev-armhf-cross
@@ -65,7 +69,6 @@ python3-renderpm
 python3-reportlab
 python3-reportlab-accel
 rhythmbox-dev
-rustc
 ubuntu-image
 ubuntu-wallpapers-lunar
 util-linux-extra
diff --git a/utils/demoted.cfg.jammy b/utils/demoted.cfg.jammy
index a66dd4c..a2c0175 100644
--- a/utils/demoted.cfg.jammy
+++ b/utils/demoted.cfg.jammy
@@ -105,6 +105,12 @@ libmoox-struct-perl
 libnamespace-autoclean-perl
 libnet-ip-perl
 libobject-id-perl
+libpcre16-3
+libpcre3
+libpcre3-dbg
+libpcre3-dev
+libpcre32-3
+libpcrecpp0v5
 libpfm4
 libpfm4-dev
 libpulsedsp
@@ -118,7 +124,6 @@ libsoxr-lsr0
 libsoxr0
 libspeexdsp-dev
 libspeexdsp1
-libstd-rust-dev
 libstdc++-11-dev-arm64-cross
 libstdc++-11-dev-armhf-cross
 libstdc++-11-dev-ppc64el-cross
@@ -173,6 +178,5 @@ python3-reportlab-accel
 python3-rfc3339
 python3-sphinx-rtd-theme
 rhythmbox-dev
-rustc
 ubuntu-image
 ubuntu-wallpapers-jammy