HTTP 500 error on block-storage-relation-changed

Charles, thanks for this bug and attaching the log made it easier.

We currently have 10 retries over a total of ~50 seconds while waiting for the volume to transition to "available" status reported by nova volume-list.

Unfortunately, we don't currently retry on spurious 500 errors from nova attach-volume command because I expected that to work when a node is already reporting "available" status. It sounds like this is a frequent occurrence for you, so I'll pull together a branch that specifically retries a configurable number of times for nova commands on 500 errors.

So far it seems better, I'd like to do some more testing. I've deployed 3 times so far and here were the results:
1 - Everything worked, volume was mounted
2 - Volume was mounted, but so was a second one that it created. Still doesn't seem to see the first all the time.
3 - 2014-08-25 20:47:54 ERROR juju-log block-storage:26: Error: Multiple volumes are associated with ci-airline-ts-postgres/0 nova volume. Cannot get_volume_id. This time it saw the two previous volumes and failed completely. Not the HTTP500 error though, this was the other problem I described where we have to remove the volumes before deploying.

So I've done a few more iterations now without deleting the volume and seen it work more often than not. So while there may still be some corner cases, this *is* an overall improvement I think based on my previous experience.

Paul, thanks for the deploy on this.

I think the condition of attempting to create a second volume when one with a label postgresql/0 is already present is a separate bug in how we process nova describe-volumes. I'll try to hit this issue on my side today, but it shouldn't be related to this branch. Generally storage tries to leave around the created volume even when the unit dies, so that the storage data isn't lost. Then that same volume gets remounted when a unit of the same name name is restarted/recreated. It looks like something in the nova describe-volumes didn't find a matching volume label for that unit so it tried to create a new one. I'll try reproducing that case with some additional debug logs to see if I can resolve the problem.