Downloading images failed. provisioningserver.utils.fs.NotAvailable: /run/lock/maas.XXXX=.lock

Is it possible that maas-clusterd was restarted while syncing images?

Did you try deleting the lock file and restarting maas-clusterd?

$ printf '%s\n' $(echo 'L2V0Yy9tYWFzL2NsdXN0ZXJkLmNvbmY=' | base64 -d)
/etc/maas/clusterd.conf

Looks like the lock is for the config file, by the way.

It is possible, as we just did MAAS upgrades. I suppose an image sync could have been underway.

Looks like we have a hard-coded timeout for that lock of 5 seconds.

Can you edit /usr/lib/python2.7/dist-pacakges/provisioningserver/config.py - on line ~549 there should be a line:

            with RunLock(path).wait(timeout=5.0):

Change that to something like 60 seconds to see if that fixes it:

            with RunLock(path).wait(timeout=60.0):

FWIW, the net effect of this, is that I seem to have a half-ready environment. Trusty deployments ultimately fail with:
"Could not find kernel image: ubuntu/amd64/hwe-t/trusty/daily/boot-kernel"

Note, the file doesn't exist: /run/lock/maas.L2V0Yy9tYWFzL2NsdXN0ZXJkLmNvbmY=.lock

I've increased that timeout to 60.0 and restarted svcs, still get the lock error.

Manually confirmed that the maas user is able to touch a file in that dir:
http://paste.ubuntu.com/13833792/

After removing the > 5GB Windows image per request, and setting the timeout to 60, a different trace is appearing (still with that lock file not existing):

http://paste.ubuntu.com/13833338/

2015-12-08 20:10:34+0000 [-] Unhandled error in Deferred:
2015-12-08 20:10:34+0000 [-] Unhandled Error
        Traceback (most recent call last):
          File "/usr/lib/python2.7/threading.py", line 783, in __bootstrap
            self.__bootstrap_inner()
          File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
            self.run()
          File "/usr/lib/python2.7/threading.py", line 763, in run
            self.__target(*self.__args, **self.__kwargs)
        --- <exception caught here> ---
          File "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 191, in _worker
            result = context.call(ctx, function, *args, **kwargs)
          File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
            return self.currentContext().callWithContext(ctx, func, *args, **kw)
          File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
            return func(*args,**kw)
          File "/usr/lib/python2.7/dist-packages/provisioningserver/utils/twisted.py", line 200, in wrapper
            return func(*args, **kwargs)
          File "/usr/lib/python2.7/dist-packages/provisioningserver/rpc/boot_images.py", line 118, in _run_import
            reload_boot_images()
          File "/usr/lib/python2.7/dist-packages/provisioningserver/rpc/boot_images.py", line 57, in reload_boot_images
            CACHED_BOOT_IMAGES = tftppath.list_boot_images(config.tftp_root)
          File "/usr/lib/python2.7/dist-packages/provisioningserver/boot/tftppath.py", line 266, in list_boot_images
            extract_image_params(path, metadata) for path in paths))
          File "/usr/lib/python2.7/dist-packages/provisioningserver/boot/tftppath.py", line 266, in <genexpr>
            extract_image_params(path, metadata) for path in paths))
          File "/usr/lib/python2.7/dist-packages/provisioningserver/boot/tftppath.py", line 180, in extract_image_params
            arch, subarch, release, label)
          File "/usr/lib/python2.7/dist-packages/provisioningserver/drivers/osystem/custom.py", line 58, in get_xinstall_parameters
            with ClusterConfiguration.open() as config:
          File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
            return self.gen.next()
          File "/usr/lib/python2.7/dist-packages/provisioningserver/config.py", line 674, in open
            with cls.backend.open(filepath) as store:
          File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
            return self.gen.next()
          File "/usr/lib/python2.7/dist-packages/provisioningserver/config.py", line 549, in open
            with RunLock(path).wait(timeout=60.0):
          File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
            return self.gen.next()
          File "/usr/lib/python2.7/dist-packages/provisioningserver/utils/fs.py", line 441, in wait
            raise self.NotAvailable(self.fslock.name...

Although I am still getting the lock file trace periodically, my Trusty boot image appears to have made it in! I can now deploy Trusty. Please let me know if there is something we can do to help reveal the cause of the trace. Thank you.

It occurs to me that we should be able to open the configuration for reading without taking the lock.

> It occurs to me that we should be able to open the configuration for
> reading without taking the lock.

In thinking about this I have stumbled on what might be the underlying
cause: there is no ordering when taking locks.

A thread will try to take the lock and then sleep for a short time if it
can't. It'll keep doing this for a few seconds after which it will give
up.

Each time the thread wakes up and finds another thread has the lock,
it's not necessarily the same thread or even the same process that's
holding the lock. It could be a different thread every time, even a
thread that has queue-barged for the lock.

Reads vastly outnumber writes, so adding an open-for-read-only option
that does not require a lock (which is safe because the configuration
file is updated atomically) would work around the problem I describe.
Concurrent writers would still be exposed to queue-barging, but so
infrequent are writes that the problem might never manifest.