Merge ~pguimaraes/maas:db-fail-lp-1843493 into maas:master

[Pedro Guimarães]

* forms/pods.py ComposeMachineForPodsForm: reducing scope of except catch on compose method.
* only PodProblem exceptions should be accepted since they represent a node that cannot overcommit to receive that specific VM. Other exceptions should be raised

Closes-bug: #1843493

[WIP]

forms/pods.py ComposeMachineForPodsForm: reducing scope of except catch on compose method.
Only PodProblem exceptions should be accepted since they represent a node that cannot overcommit to receive that specific VM. Other exceptions should be raised.

[MAAS Lander (maas-lander)]

UNIT TESTS
-b db-fail-lp-1843493 lp:~pguimaraes/maas/+git/maas into -b master lp:~maas-committers/maas

STATUS: FAILED
LOG: http://maas-ci-jenkins.internal:8080/job/maas/job/branch-tester/6379/console
COMMIT: 3541b062f6af81f90c9f8b0f24b30046c36e6f07

review: Needs Fixing

[Blake Rouse (blake-rouse)]

Tests will need to be fixed to handle those specific exception types. Also I think this might still have an issue, of causing MAAS to try to compose another machine on the retry leaving the machine that was composed during the serialization error still inside libvirt (which would be very bad).

review: Needs Fixing

[Pedro Guimarães (pguimaraes)]

Hey blake-rouse, thanks for the review.
Well, I am essentially seeing two kinds of errors: PodProblem-related (which is expected, nothing wrong here); and OperationalError (check: https://bugs.launchpad.net/maas/+bug/1843493/comments/2)
The later is a database issue, actually, which I think comes before libvirt setup, since libvirt is configured once I fire an RPC to provisioningserver, right?

I don't think I am near the solution however, I'm still seeing DB conflicts, even if I set my thread to sleep for several seconds (which is recommended approach as per postgres docs).

I feel we're missing a "with transaction.atomic():" somewhere to make all other pod allocations to hold. However, whenever I add this to compose method, it breaks with TransactionManagementError.

[Blake Rouse (blake-rouse)]

Pedro,

Thanks for digging deep into this, I have done an MP(https://code.launchpad.net/~blake-rouse/maas/+git/maas/+merge/372623) that I think will solve the issue. Could you test it out for me and let me know if this solves it?

Thanks,
Blake

[Blake Rouse (blake-rouse)]

With this MP:

https://code.launchpad.net/~blake-rouse/maas/+git/maas/+merge/372623

review: Disapprove

Unmerged commits

3541b06... by Pedro Guimarães on 2019-09-11

forms/pods.py ComposeMachineForPodsForm: reducing scope of except catch on compose method.
Only PodProblem exceptions should be accepted since they represent a node that cannot overcommit to receive that specific VM. Other exceptions should be raised.

diff --git a/src/maasserver/forms/pods.py b/src/maasserver/forms/pods.py
index 1a68329..96ebb1b 100644
--- a/src/maasserver/forms/pods.py
+++ b/src/maasserver/forms/pods.py
@@ -81,6 +81,7 @@ from provisioningserver.utils.twisted import asynchronous
 from twisted.internet.defer import inlineCallbacks
 from twisted.python.threadable import isInIOThread
 
+from maasserver.exceptions import PodProblem
 
 log = LegacyLogger()
 
@@ -720,7 +721,7 @@ class ComposeMachineForPodsForm(forms.Form):
                 return form.compose(
                     skip_commissioning=True,
                     creation_type=NODE_CREATION_TYPE.DYNAMIC)
-            except Exception:
+            except PodProblem:
                 continue
         # Second, try to compose a machine from commitable pods
         for form in commit_forms:
@@ -728,7 +729,7 @@ class ComposeMachineForPodsForm(forms.Form):
                 return form.compose(
                     skip_commissioning=True,
                     creation_type=NODE_CREATION_TYPE.DYNAMIC)
-            except Exception:
+            except PodProblem:
                 continue
         # No machine found.
         return None