Failed to create bucket (reason: S3ResponseError: S3ResponseError: 404 Not Found

Failed to create bucket (attempt #1) 'lin-v4clogframe.s3.amazonaws.com' failed (reason: S3ResponseError: S3ResponseError: 404 Not Found

The reason appears to be that a change in Boto broke the missing bucket check. That check is in BotoBackend.put from duplicity/backends/_boto_single.py:

    def put(self, source_path, remote_filename=None):
        ...
                try:
                    self.bucket = self.conn.get_bucket(self.bucket_name, validate=True)
                except Exception, e:
                    if "NoSuchBucket" in str(e):
                        if globals.s3_european_buckets:
                            self.bucket = self.conn.create_bucket(self.bucket_name,
                                                                  location=Location.EU)
                        else:
                            self.bucket = self.conn.create_bucket(self.bucket_name)
                    else:
                        raise e

Note that this relies on a string match against the exception message to decide whether the bucket exists or not.

Now Boto (at least python-boto-2.25.0-2.el6.noarch from EPEL 6) implements get_bucket using a HEAD request instead of a GET (boto.s3.connections.S3Connection.get_bucket, from boto/s3/connection.py):

    def get_bucket(self, bucket_name, validate=True, headers=None):
        """
        Retrieves a bucket by name.

        If the bucket does not exist, an ``S3ResponseError`` will be raised. If
        you are unsure if the bucket exists or not, you can use the
        ``S3Connection.lookup`` method, which will either return a valid bucket
        or ``None``.
        ...
        If the default ``validate=True`` is passed, a request is made to the
        service to ensure the bucket exists. Prior to Boto v2.25.0, this fetched
        a list of keys (but with a max limit set to ``0``, always returning an empty
        list) in the bucket (& included better error messages), at an
        increased expense. As of Boto v2.25.0, this now performs a HEAD request
        (less expensive but worse error messages).

        If you were relying on parsing the error message before, you should call
        something like::

            bucket = conn.get_bucket('<bucket_name>', validate=False)
            bucket.get_all_keys(maxkeys=0)
        ...
        if validate:
            return self.head_bucket(bucket_name, headers=headers)
        else:
            return self.bucket_class(self, bucket_name)

So as they say, we need to stop parsing the exception message and either do a lookup() first, or catch S3ResponseError, or set validate=False and see whether get_all_keys() throws an exception.