Create KVM fails in MAAS 3.0 Beta with a project error

Upon attempting to debug, discovered that I had apparently uninstalled LXD at some point, so this amounts to an unhandled exception (notifying the user that LXD is not present, unreachable, etc.). I would triage this as a low priority if I could triage my own bugs, and I will also mark this as "doc" for troubleshooting information that should be added various places.

Installing and initialising LXD does not solve the problem, noticing that the terminal number in the error message increments each time I try:

Error:(b'UNHANDLED', 'Unknown Error [wintermute:pid=90065:cmd=DiscoverPodProjects:ask=c7]')

About to try rebooting, which sometimes resolves issues with a newly-installed LXD.

Installing and initialising LXD does not solve the problem, noticing that the terminal number in the error message increments each time I try:

Error:(b'UNHANDLED', 'Unknown Error [wintermute:pid=90065:cmd=DiscoverPodProjects:ask=c7]')

Tried rebooting, that just resets the error "ask" count somewhat:

Error:(b'UNHANDLED', 'Unknown Error [wintermute:pid=3123:cmd=DiscoverPodProjects:ask=4]')

If I'm doing something wrong, it's not documented, and it's different from previous MAAS versions. Leaving this config in this state until someone can take a look.

Bill, do you have more than one rack/region in that maas?

I tried to reproduce the issue on a clean container with maas from 3.0/beta, adding a local LXD works fine for me and creates the project.

no, it was a clean install directly on my laptop using the installation guide straight down the middle, and a clean install of lxd. i've left my laptop in this state, is there debugging info i can get for you?

Attaching regiond.log

It seems this is due to authentication issues (but the error is not propagated correctly)

2021-04-06 11:34:14 provisioningserver.rpc.common: [critical] Unhandled failure dispatching AMP command. This is probably a bug. Please ensure that this error is handled within application code or declared in the signature of the b'DiscoverPodProjects' command. [wintermute:pid=134670:cmd=DiscoverPodProjects:ask=7]
 Traceback (most recent call last):
   File "/usr/lib/python3.8/asyncio/events.py", line 81, in _run
     self._context.run(self._callback, *self._args)
   File "/snap/maas/12731/usr/lib/python3/dist-packages/twisted/internet/asyncioreactor.py", line 290, in run
     f(*args, **kwargs)
   File "/snap/maas/12731/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 501, in errback
     self._startRunCallbacks(fail)
   File "/snap/maas/12731/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 568, in _startRunCallbacks
     self._runCallbacks()
 --- <exception caught here> ---
   File "/snap/maas/12731/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 654, in _runCallbacks
     current.result = callback(current.result, *args, **kw)
   File "/snap/maas/12731/usr/lib/python3/dist-packages/twisted/protocols/amp.py", line 1171, in checkKnownErrors
     key = error.trap(*command.allErrors)
   File "/snap/maas/12731/usr/lib/python3/dist-packages/twisted/python/failure.py", line 439, in trap
     self.raiseException()
   File "/snap/maas/12731/usr/lib/python3/dist-packages/twisted/python/failure.py", line 467, in raiseException
     raise self.value.with_traceback(self.tb)
   File "/snap/maas/12731/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 250, in inContext
     result = inContext.theWork()
   File "/snap/maas/12731/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 266, in <lambda>
     inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
   File "/snap/maas/12731/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext
     return self.currentContext().callWithContext(ctx, func, *args, **kw)
   File "/snap/maas/12731/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext
     return func(*args,**kw)
   File "/snap/maas/12731/lib/python3.8/site-packages/provisioningserver/utils/twisted.py", line 202, in wrapper
     result = func(*args, **kwargs)
   File "/snap/maas/12731/lib/python3.8/site-packages/provisioningserver/drivers/pod/lxd.py", line 239, in discover_projects
     client = self._get_client(pod_id, context)
   File "/snap/maas/12731/lib/python3.8/site-packages/provisioningserver/drivers/pod/lxd.py", line 760, in _get_client
     client.authenticate(password)
   File "/snap/maas/12731/usr/lib/python3/dist-packages/pylxd/client.py", line 447, in authenticate
     self.certificates.create(password, cert)
   File "/snap/maas/12731/usr/lib/python3/dist-packages/pylxd/models/certificate.py", line 62, in create
     client.api.certificates.post(json=data)
   File "/snap/maas/12731/usr/lib/python3/dist-packages/pylxd/client.py", line 208, in post
     self._assert_response(response, allowed_status_codes=(200...

I can confirm this is probably an auth issue; reinstalled LXD, very careful with my trust password, it works fine creating a KVM as long as I choose the default project. Separate bug on what happens if you type in a project name that doesn't already exist in the KVM creation page.