regiond.conf "debug_http: true" causes image downloads from regiond to fail with 500 error code

[Impact]

Image sync from region-controller to rack controller will fail. In the Images tab of the Browser Interface, the message: Loading
Waiting for rack controller(s) to sync is displayed beside the image name and errors similar to those below are found in logs:

Logs in regiond.log:
maasserver: [error] ################################ Exception: 'str' object has no attribute 'decode' ################################
2022-09-16 20:25:41 maasserver: [error] Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/maasserver/utils/views.py", line 243, in handle_uncaught_exception
    raise exc from exc.__cause__
  File "/usr/lib/python3/dist-packages/maasserver/utils/views.py", line 309, in get_response
    response = django_get_response(request)
  File "/usr/lib/python3/dist-packages/django/core/handlers/base.py", line 75, in get_response
    response = self._middleware_chain(request)
  File "/usr/lib/python3/dist-packages/maasserver/prometheus/middleware.py", line 52, in __call__
    response = self.get_response(request)
  File "/usr/lib/python3/dist-packages/maasserver/middleware.py", line 336, in __call__
    decoded_content = content.decode("utf-8")
AttributeError: 'str' object has no attribute 'decode'

regiond: [info] 10.55.0.12 GET /MAAS/images-stream/grub-efi-signed/amd64/generic/uefi/20210819.0/grub2-signed.tar.xz HTTP/1.1 --> 500 INTERNAL_SERVER_ERROR (referrer: -; agent: python-simplestreams/0.1)

provisioningserver.rpc.common: [debug] [RPC <- received] AmpBox({b'_ask': b'72', b'_command': b'SendEvent', b'description': b'Unable to import boot images: Invalid sha256 Checksum at http://10.55.0.15/MAAS/images-stream/grub-efi-signed/amd64/generic/uefi/20210819.0/grub2-signed.tar.xz. Found 834c0eacb1a19526f715f9947bd47904b18ad8c733b0762e690edf6143e10561. Expected 9d4a3a826ed55c46412613d2f7caf3185da4d6b18f35225f4f6a5b86b2bccfe3. read 38 bytes expected 375316 bytes. (size 38 expected 375316)', b'system_id': b'snwq6y', b'type_name': b'RACK_IMPORT_ERROR'})

Logs in rackd.log:
twisted.internet.defer: [critical] Unhandled error in Deferred:
twisted.internet.defer: [critical]
        Traceback (most recent call last):
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 500, in errback
            self._startRunCallbacks(fail)
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 567, in _startRunCallbacks
            self._runCallbacks()
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 653, in _runCallbacks
            current.result = callback(current.result, *args, **kw)
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1442, in gotResult
            _inlineCallbacks(r, g, deferred)
        --- <exception caught here> ---
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
            result = result.throwExceptionIntoGenerator(g)
          File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
            return g.throw(self.type, self.value, self.tb)
          File "/usr/lib/python3/dist-packages/provisioningserver/rpc/boot_images.py", line 159, in _import_boot_images
            yield deferToThread(_run_import, sources, maas_url, **proxies)
          File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 250, in inContext
            result = inContext.theWork()
          File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 266, in <lambda>
            inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
          File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext
            return self.currentContext().callWithContext(ctx, func, *args, **kw)
          File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext
            return func(*args,**kw)
          File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 217, in wrapper
            result = func(*args, **kwargs)
          File "/usr/lib/python3/dist-packages/provisioningserver/rpc/boot_images.py", line 123, in _run_import
            imported = boot_resources.import_images(sources)
          File "/usr/lib/python3/dist-packages/provisioningserver/import_images/boot_resources.py", line 193, in import_images
            sources, storage, product_mapping
          File "/usr/lib/python3/dist-packages/provisioningserver/import_images/download_resources.py", line 365, in download_all_boot_resources
            keyring_file=source.get("keyring"),
          File "/usr/lib/python3/dist-packages/provisioningserver/import_images/download_resources.py", line 307, in download_boot_resources
            writer.sync(reader, rpath)
          File "/usr/lib/python3/dist-packages/simplestreams/mirrors/__init__.py", line 91, in sync
            return self.sync_index(reader, path, data, content)
          File "/usr/lib/python3/dist-packages/simplestreams/mirrors/__init__.py", line 254, in sync_index
            self.sync(reader, path=epath)
          File "/usr/lib/python3/dist-packages/simplestreams/mirrors/__init__.py", line 89, in sync
            return self.sync_products(reader, path, data, content)
          File "/usr/lib/python3/dist-packages/simplestreams/mirrors/__init__.py", line 341, in sync_products
            self.insert_item(item, src, target, pgree, ipath_cs)
          File "/usr/lib/python3/dist-packages/provisioningserver/import_images/download_resources.py", line 235, in insert_item
            self.store, filename, tag, checksums, size, contentsource
          File "/usr/lib/python3/dist-packages/provisioningserver/import_images/download_resources.py", line 116, in extract_archive_tar
            store.insert(tag, content_source, checksums, mutable=False, size=size)
          File "/usr/lib/python3/dist-packages/simplestreams/objectstores/__init__.py", line 179, in insert
            raise checksum_util.invalid_checksum_for_reader(reader)
        simplestreams.checksum_util.InvalidChecksum: Invalid sha256 Checksum at http://10.55.0.15/MAAS/images-stream/grub-efi-signed/amd64/generic/uefi/20210819.0/grub2-signed.tar.xz. Found 834c0eacb1a19526f715f9947bd47904b18ad8c733b0762e690edf6143e10561. Expected 9d4a3a826ed55c46412613d2f7caf3185da4d6b18f35225f4f6a5b86b2bccfe3. read 38 bytes expected 375316 bytes. (size 38 expected 375316)

[Platforms]

debs in Bionic
debs in Focal
Snap

[Reproducer]

In any MAAS installation, edit regiond.conf and add:

debug_http: true

Then restart maas-regiond. In a deb install:

/etc/maas/regiond.conf
systemctl restart maas-regiond

Try to curl an image:

curl -vq http://localhost:5240/MAAS/images-stream/grub-efi-signed/amd64/generic/uefi/20210819.0/grub2-signed.tar.xz

...
< HTTP/1.1 500 Internal Server Error
...
'str' object has no attribute 'decode'

Or add a new image to the list.

[Workaround]

Remove the configuration http_debug from regiond.conf file and restart maas-regiond.