Merge ~r00ta/maas:lp-2058273-sync-bootresources-deterministic into maas:3.5

[Jacopo Rota]

fix: lp-2058273. Make the sync-bootresources workflow deterministic

The temporal workflows must be deterministic. This MP aims to use the random generator provided by the temporal sdk to randomize the source list for the images in order to balance the load on the available regions.

4010fa6... by Jacopo Rota on 2024-03-18

fix: lp-2058273. Make the sync-bootresources workflow deterministic

[MAAS Lander (maas-lander)]

UNIT TESTS
-b lp-2058273-sync-bootresources-deterministic lp:~r00ta/maas/+git/maas into -b 3.5 lp:~maas-committers/maas

STATUS: SUCCESS
COMMIT: a0a79dbf6c9b9a5184bd61c68c80c3eb086f9c8a

review: Approve

Unmerged commits

4010fa6... by Jacopo Rota on 2024-03-18

fix: lp-2058273. Make the sync-bootresources workflow deterministic

81a9976... by Jacopo Rota on 2024-03-18

fix: lp-2058037. Ensure get-bootresourcefile-endpoints activity fails if the region is not exposing its ip yet.

3b58876... by Anton Troyanov on 2024-03-18

fix: /run/maas should be owned by maas

`/run` is usually a tmpfs mount, and it is not persistent across reboots, so
`/run/maas` should not be created using .dirs, .install or .postinst scripts.

MAAS has various services that store their directories and files under
`/run/maas`, but because some services are run as root (e.g. dhcp), this leads
to ownership issues when `mkdir -p` is used (e.g. DHCP will create
`/run/maas/dhcp`)

In order to solve ownership issue, this commit changes DHCP service directory
to `/run/maas-dhcp` while keeping `/run/maas` only for files related to MAAS
itself (not external 3rd party services)

Resolves LP:2056222
Resolves LP:2056225

b7f2542... by Jones Ogolo <email address hidden> on 2024-03-17

Update initial SQL

6466b0c... by Jones Ogolo <email address hidden> on 2024-03-15

Update maas-ui to 84ff1641a
feat: update image sync (tooltip) information (#5352)

07a613f... by Peter Makowski on 2024-03-15

Update maas-ui to 2384eb408
fix: cannot stop importing images MAASENG-2864 (#5351)

d312185... by Peter Makowski on 2024-03-15

Update maas-ui to 064adb558
fix: cannot delete image while downloading MAASENG-2863 (#5350)

Fixes: lp#232057935

5d5a0aa... by Jones Ogolo <email address hidden> on 2024-03-15

Update maas-ui to 7d88fb2f4
feat: move non-partition collapsible forms to side panel (#5349)

7896226... by Anton Troyanov on 2024-03-15

chore: reflect workflow type in wfid

069e184... by Alexsander de Souza on 2024-03-15

fix(bootresource): lp-2057979. Avoid workflow id clash.

ID must be unique within a namespace

diff --git a/src/maasserver/workflow/bootresource.py b/src/maasserver/workflow/bootresource.py
index 4097f7e..559d03b 100644
--- a/src/maasserver/workflow/bootresource.py
+++ b/src/maasserver/workflow/bootresource.py
@@ -2,14 +2,13 @@ import asyncio
 from asyncio import gather
 from dataclasses import dataclass, field, replace
 from datetime import datetime, timedelta
-import random
 from typing import Coroutine, Sequence
 
 from aiohttp.client_exceptions import ClientError
 from temporalio import activity, workflow
 from temporalio.common import RetryPolicy, WorkflowIDReusePolicy
 from temporalio.exceptions import ApplicationError
-from temporalio.workflow import ActivityCancellationType
+from temporalio.workflow import ActivityCancellationType, random
 
 from maasserver.utils.bootresource import (
     get_bootresource_store_path,
@@ -298,9 +297,12 @@ class SyncBootResourcesWorkflow:
                 for reg in sources
                 for ep in endpoints[reg]
             ]
+            # In order to balance the workload on the regions we randomize the order of the source_list. We use the random
+            # generator provided by Temporal to keep the workflow execution deterministic.
+            generator = random()
             new_res = replace(
                 res,
-                source_list=random.sample(eps, min(len(eps), MAX_SOURCES)),
+                source_list=generator.sample(eps, min(len(eps), MAX_SOURCES)),
             )
             for region in missing:
                 sync_jobs.append(_schedule(new_res, region))