Merge ~paride/simplestreams:apply-black into simplestreams:master

[Paride Legovini]

- Apply black and isort
- Add black and isort pre-commit hooks
- Run (read-only) pre-commit in tox environment
- Run tox in lpci

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:43978e994d50150f5e7b2107bc310ec57ed2efd2
https://jenkins.canonical.com/server-team/job/simplestreams-ci/11/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-amd64/11/
    FAILURE: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-ppc64el/11/
    FAILURE: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-s390x/11/

Click here to trigger a rebuild:
https://jenkins.canonical.com/server-team/job/simplestreams-ci/11//rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:43978e994d50150f5e7b2107bc310ec57ed2efd2
https://jenkins.canonical.com/server-team/job/simplestreams-ci/12/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-amd64/12/
    FAILURE: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-ppc64el/12/
    FAILURE: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-s390x/12/

Click here to trigger a rebuild:
https://jenkins.canonical.com/server-team/job/simplestreams-ci/12//rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:43978e994d50150f5e7b2107bc310ec57ed2efd2
https://jenkins.canonical.com/server-team/job/simplestreams-ci/13/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-amd64/13/
    FAILURE: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-ppc64el/13/
    FAILURE: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-s390x/13/

Click here to trigger a rebuild:
https://jenkins.canonical.com/server-team/job/simplestreams-ci/13//rebuild

review: Needs Fixing (continuous-integration)

[Paride Legovini (paride)]

CI failures fixed by:

https://code.launchpad.net/~dviererbe/simplestreams/+git/simplestreams/+merge/439831

246f30e... by Paride Legovini on 2023-06-04

lpci: don't run the pre-commit environment

Apparently the lpci workers do not have access to github.com,
so pre-commit can't download the hooks.

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:40311284d9201ed49826ef9e21076aeee1ae5ef5
https://jenkins.canonical.com/server-team/job/simplestreams-ci/14/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-amd64/14/
    FAILURE: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-ppc64el/14/
    FAILURE: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-s390x/14/

Click here to trigger a rebuild:
https://jenkins.canonical.com/server-team/job/simplestreams-ci/14//rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:7833c12c86ed3b188392b89547bd4e4852c8c7a8
https://jenkins.canonical.com/server-team/job/simplestreams-ci/15/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-amd64/15/
    FAILURE: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-ppc64el/15/
    FAILURE: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-s390x/15/

Click here to trigger a rebuild:
https://jenkins.canonical.com/server-team/job/simplestreams-ci/15//rebuild

review: Needs Fixing (continuous-integration)

274817d... by Paride Legovini on 2023-06-04

install-deps: install pkg-config on non-amd64 archs

a4d7a3f... by Paride Legovini on 2023-06-04

lpci: expand the test matrix

Cover: (amd64, arm64, ppc64el, s390x) * (focal, jammy, devel)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:a4d7a3f8132723acf73ff528ff25639e4f58a15c
https://jenkins.canonical.com/server-team/job/simplestreams-ci/16/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-amd64/16/
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-arm64/16/
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-ppc64el/16/
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-s390x/16/

Click here to trigger a rebuild:
https://jenkins.canonical.com/server-team/job/simplestreams-ci/16//rebuild

review: Approve (continuous-integration)

39c2e39... by Paride Legovini on 2023-06-04

lpci: add test dependency: python3-dev

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:39c2e398be3a21b3c7e1362ef6e0ebebfcbaae01
https://jenkins.canonical.com/server-team/job/simplestreams-ci/17/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-amd64/17/
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-arm64/17/
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-ppc64el/17/
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-s390x/17/

Click here to trigger a rebuild:
https://jenkins.canonical.com/server-team/job/simplestreams-ci/17//rebuild

review: Approve (continuous-integration)

19608c6... by Paride Legovini on 2023-06-06

mm

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:fe4262017aff844e207948589dd2cabf882f0d3c
https://jenkins.canonical.com/server-team/job/simplestreams-ci/18/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-amd64/18/
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-arm64/18/
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-ppc64el/18/
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-s390x/18/

Click here to trigger a rebuild:
https://jenkins.canonical.com/server-team/job/simplestreams-ci/18//rebuild

review: Needs Fixing (continuous-integration)

b405e84... by Paride Legovini on 2023-06-06

tox: pass the lowercase *_proxy variables

Workaround for https://github.com/tox-dev/tox/pull/2378/.

363d3b4... by Paride Legovini on 2023-06-06

tox: skip_install in the pre_commit environment

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:37a21b666ed679809f3faa755da34c07b95cbe0b
https://jenkins.canonical.com/server-team/job/simplestreams-ci/19/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-amd64/19/
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-arm64/19/
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-ppc64el/19/
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-s390x/19/

Click here to trigger a rebuild:
https://jenkins.canonical.com/server-team/job/simplestreams-ci/19//rebuild

review: Approve (continuous-integration)

75441dd... by Paride Legovini on 2023-06-06

ci: deal with pre-commit modifying files

We don't need to jump through hoops to avoid pre-commit modifying files
during the tox run: if that happens, the CI run will fail anyway.

5449069... by Paride Legovini on 2023-06-06

prefer bin

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:19608c61d5714b135df6d631771b2ffb67b2f7e0
https://jenkins.canonical.com/server-team/job/simplestreams-ci/20/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-amd64/20/
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-arm64/20/
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-ppc64el/20/
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-s390x/20/

Click here to trigger a rebuild:
https://jenkins.canonical.com/server-team/job/simplestreams-ci/20//rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:6feb6ce9622a6486774a90919f67a098b4355dab
https://jenkins.canonical.com/server-team/job/simplestreams-ci/21/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-amd64/21/
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-arm64/21/
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-ppc64el/21/
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-s390x/21/

Click here to trigger a rebuild:
https://jenkins.canonical.com/server-team/job/simplestreams-ci/21//rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:6bbfb4fbe8515bb1acae934cf0e8d2a555187ed0
https://jenkins.canonical.com/server-team/job/simplestreams-ci/22/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-amd64/22/
    FAILURE: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-arm64/22/
    FAILURE: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-ppc64el/22/
    FAILURE: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-s390x/22/

Click here to trigger a rebuild:
https://jenkins.canonical.com/server-team/job/simplestreams-ci/22//rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:dec84e7e729aee2c8474ef87015f678227f7ea50
https://jenkins.canonical.com/server-team/job/simplestreams-ci/23/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-amd64/23/
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-arm64/23/
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-ppc64el/23/
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-s390x/23/

Click here to trigger a rebuild:
https://jenkins.canonical.com/server-team/job/simplestreams-ci/23//rebuild

review: Approve (continuous-integration)

4c88164... by Paride Legovini on 2023-06-06

use-install-deps

The tool requires sudo to install dependencies.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:f79823a18f7cd7671ea6b80915b3955b4965ab96
https://jenkins.canonical.com/server-team/job/simplestreams-ci/24/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-amd64/24/
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-arm64/24/
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-ppc64el/24/
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-s390x/24/

Click here to trigger a rebuild:
https://jenkins.canonical.com/server-team/job/simplestreams-ci/24//rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:5f19b7bdc807559d9230447ff032623e3dad5534
https://jenkins.canonical.com/server-team/job/simplestreams-ci/25/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-amd64/25/
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-arm64/25/
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-ppc64el/25/
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-s390x/25/

Click here to trigger a rebuild:
https://jenkins.canonical.com/server-team/job/simplestreams-ci/25//rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:0a7d8c46717d30ba88177a5dd21cbed51f76811c
https://jenkins.canonical.com/server-team/job/simplestreams-ci/26/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-amd64/26/
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-arm64/26/
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-ppc64el/26/
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-s390x/26/

Click here to trigger a rebuild:
https://jenkins.canonical.com/server-team/job/simplestreams-ci/26//rebuild

review: Approve (continuous-integration)

[Adam Collard (adam-collard)]

I wouldn't mix a large reformatting like this (applying black across the codebase) with anything else.

I suggest landing the blacken commit, then separately isort, then separately again tox / lp-ci changes.

You'll want to ignore the black commits for e.g. git blame; but should see tox and lp-ci config in their own right

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:5f1da7595cc456df2a26df4148ee0ed26c545d9a
https://jenkins.canonical.com/server-team/job/simplestreams-ci/27/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-amd64/27/
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-arm64/27/
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-ppc64el/27/
    SUCCESS: https://jenkins.canonical.com/server-team/job/simplestreams-ci/nodes=metal-s390x/27/

Click here to trigger a rebuild:
https://jenkins.canonical.com/server-team/job/simplestreams-ci/27//rebuild

review: Approve (continuous-integration)

[Paride Legovini (paride)]

Hi Adam, thanks for chiming in. Yeah I started this with the idea of quickly fixing CI (currently failing in master), then I started experimenting with integrating pre-commit, tox and lpci and the changeset grew.

I'll definitely split it up and resubmit it in smaller chunks.

[Paride Legovini (paride)]

CI is stuck (running for 17 hours). I'll do a no-op force push to retrigger.

6da3cc1... by Paride Legovini on 2023-06-08

ci: append 127.0.0.1 to no_proxy

[Paride Legovini (paride)]

CI passed! Next steps:

- Cleanup git history
- Split the this MP is smaller one.

I'll get back to this after 2023-06-13.

[Adam Collard (adam-collard)]

@Paride - don't forget about this one!

[Paride Legovini (paride)]

heh, knowing that other people are waiting for it will help. thanks!

Unmerged commits

6da3cc1... by Paride Legovini on 2023-06-08

ci: append 127.0.0.1 to no_proxy

	lint:0 (build)
	unit-jammy:0 (build)
	unit-devel:0 (build)
	lint:0 (build)
	unit-jammy:0 (build)
	unit-devel:0 (build)
	lint:0 (build)
	unit-jammy:0 (build)
	unit-devel:0 (build)
	lint:0 (build)
	unit-jammy:0 (build)
	unit-devel:0 (build)

1 → 12 of 12 results

First • Previous • Next • Last

4c88164... by Paride Legovini on 2023-06-06

use-install-deps

The tool requires sudo to install dependencies.

	lint:0 (build)
	unit-jammy:0 (build)
	unit-devel:0 (build)
	lint:0 (build)
	unit-jammy:0 (build)
	unit-devel:0 (build)
	lint:0 (build)
	unit-jammy:0 (build)
	unit-devel:0 (build)
	lint:0 (build)
	unit-jammy:0 (build)
	unit-devel:0 (build)

1 → 12 of 12 results

First • Previous • Next • Last

5449069... by Paride Legovini on 2023-06-06

prefer bin

19608c6... by Paride Legovini on 2023-06-06

mm

	lint:0 (build)
	unit-jammy:0 (build)
	unit-devel:0 (build)
	lint:0 (build)
	unit-jammy:0 (build)
	unit-devel:0 (build)
	lint:0 (build)
	unit-jammy:0 (build)
	unit-devel:0 (build)
	lint:0 (build)
	unit-jammy:0 (build)
	unit-devel:0 (build)

1 → 12 of 12 results

First • Previous • Next • Last

75441dd... by Paride Legovini on 2023-06-06

ci: deal with pre-commit modifying files

We don't need to jump through hoops to avoid pre-commit modifying files
during the tox run: if that happens, the CI run will fail anyway.

363d3b4... by Paride Legovini on 2023-06-06

tox: skip_install in the pre_commit environment

b405e84... by Paride Legovini on 2023-06-06

tox: pass the lowercase *_proxy variables

Workaround for https://github.com/tox-dev/tox/pull/2378/.

39c2e39... by Paride Legovini on 2023-06-04

lpci: add test dependency: python3-dev

	linters:0 (build)
	unit-focal:0 (build)
	unit-jammy:0 (build)
	unit-devel:0 (build)
	linters:0 (build)
	unit-focal:0 (build)
	unit-jammy:0 (build)
	unit-devel:0 (build)
	linters:0 (build)
	unit-focal:0 (build)
	unit-jammy:0 (build)
	unit-devel:0 (build)
	linters:0 (build)
	unit-focal:0 (build)
	unit-jammy:0 (build)
	unit-devel:0 (build)
	linters:0 (build)
	unit-focal:0 (build)
	unit-jammy:0 (build)
	unit-devel:0 (build)
	linters:0 (build)
	unit-focal:0 (build)
	unit-jammy:0 (build)
	unit-devel:0 (build)
	linters:0 (build)
	unit-focal:0 (build)
	unit-jammy:0 (build)
	unit-devel:0 (build)
	linters:0 (build)
	unit-focal:0 (build)
	unit-jammy:0 (build)
	unit-devel:0 (build)

1 → 32 of 32 results

First • Previous • Next • Last

a4d7a3f... by Paride Legovini on 2023-06-04

lpci: expand the test matrix

Cover: (amd64, arm64, ppc64el, s390x) * (focal, jammy, devel)

	linters:0 (build)
	unit-focal:0 (build)
	unit-jammy:0 (build)
	unit-devel:0 (build)
	linters:0 (build)
	unit-focal:0 (build)
	unit-jammy:0 (build)
	unit-devel:0 (build)
	linters:0 (build)
	unit-focal:0 (build)
	unit-jammy:0 (build)
	unit-devel:0 (build)
	linters:0 (build)
	unit-focal:0 (build)
	unit-jammy:0 (build)
	unit-devel:0 (build)
	linters:0 (build)
	unit-focal:0 (build)
	unit-jammy:0 (build)
	unit-devel:0 (build)
	linters:0 (build)
	unit-focal:0 (build)
	unit-jammy:0 (build)
	unit-devel:0 (build)
	linters:0 (build)
	unit-focal:0 (build)
	unit-jammy:0 (build)
	unit-devel:0 (build)
	linters:0 (build)
	unit-focal:0 (build)
	unit-jammy:0 (build)
	unit-devel:0 (build)

1 → 32 of 32 results

First • Previous • Next • Last

274817d... by Paride Legovini on 2023-06-04

install-deps: install pkg-config on non-amd64 archs

diff --git a/.git-blame-ignore-revs b/.git-blame-ignore-revs
new file mode 100644
index 0000000..da1123d
--- /dev/null
+++ b/.git-blame-ignore-revs
@@ -0,0 +1,4 @@
+# Automatically apply to git blame with `git config blame.ignorerevsfile .git-blame-ignore-revs`
+
+# Apply black and isort formatting
+0d4060f7fa2de1e5b8c8b263100b1ed3a2c479bb
diff --git a/.launchpad.yaml b/.launchpad.yaml
new file mode 100644
index 0000000..20a3afb
--- /dev/null
+++ b/.launchpad.yaml
@@ -0,0 +1,38 @@
+pipeline:
+  - lint
+  - unit-jammy
+  - unit-devel
+
+jobs:
+  lint:
+    series: jammy
+    architectures:
+      - amd64
+    packages:
+      - git
+      - tox
+    run: tox -e flake8,pre-commit
+  unit-jammy:
+    series: jammy
+    architectures:
+      - amd64
+      - arm64
+      - ppc64el
+      - s390x
+    packages:
+      - sudo
+    run: |
+      tools/install-deps tox
+      no_proxy="${no_proxy:+$no_proxy,}127.0.0.1" tox -e py3
+  unit-devel:
+    series: devel
+    architectures:
+      - amd64
+      - arm64
+      - ppc64el
+      - s390x
+    packages:
+      - sudo
+    run: |
+      tools/install-deps tox
+      no_proxy="${no_proxy:+$no_proxy,}127.0.0.1" tox -e py3
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
new file mode 100644
index 0000000..0d08ba7
--- /dev/null
+++ b/.pre-commit-config.yaml
@@ -0,0 +1,15 @@
+# Hooks called in the "manual" stage are meant to be read-only. We'll trigger
+# them from tox, and we don't want the tox pre-commit environment to modify
+# code, as this may interfere with other environments.
+#
+# To update the pinned versions run: pre-commit autoupdate
+
+repos:
+  - repo: https://github.com/ambv/black
+    rev: 23.3.0
+    hooks:
+    - id: black
+  - repo: https://github.com/pycqa/isort
+    rev: 5.12.0
+    hooks:
+    - id: isort
diff --git a/bin/json2streams b/bin/json2streams
index bca6a7b..7dfd873 100755
--- a/bin/json2streams
+++ b/bin/json2streams
@@ -2,8 +2,8 @@
 #   Copyright (C) 2013, 2015 Canonical Ltd.
 
 import sys
-from simplestreams.json2streams import main
 
+from simplestreams.json2streams import main
 
-if __name__ == '__main__':
+if __name__ == "__main__":
     sys.exit(main())
diff --git a/bin/sstream-mirror b/bin/sstream-mirror
index 08b71a5..fbe86da 100755
--- a/bin/sstream-mirror
+++ b/bin/sstream-mirror
@@ -18,11 +18,7 @@
 import argparse
 import sys
 
-from simplestreams import filters
-from simplestreams import log
-from simplestreams import mirrors
-from simplestreams import objectstores
-from simplestreams import util
+from simplestreams import filters, log, mirrors, objectstores, util
 
 
 class DotProgress(object):
@@ -39,9 +35,10 @@ class DotProgress(object):
             self.curpath = path
             status = ""
             if self.expected:
-                status = (" %02s%%" %
-                          (int(self.bytes_read * 100 / self.expected)))
-            sys.stderr.write('=> %s [%s]%s\n' % (path, total, status))
+                status = " %02s%%" % (
+                    int(self.bytes_read * 100 / self.expected)
+                )
+            sys.stderr.write("=> %s [%s]%s\n" % (path, total, status))
 
         if cur == total:
             sys.stderr.write("\n")
@@ -52,7 +49,7 @@ class DotProgress(object):
         toprint = int(cur * self.columns / total) - self.printed
         if toprint <= 0:
             return
-        sys.stderr.write('.' * toprint)
+        sys.stderr.write("." * toprint)
         sys.stderr.flush()
         self.printed += toprint
 
@@ -60,81 +57,135 @@ class DotProgress(object):
 def main():
     parser = argparse.ArgumentParser()
 
-    parser.add_argument('--keep', action='store_true', default=False,
-                        help='keep items in target up to MAX items '
-                             'even after they have fallen out of the source')
-    parser.add_argument('--max', type=int, default=None,
-                        help='store at most MAX items in the target')
-    parser.add_argument('--path', default=None,
-                        help='sync from index or products file in mirror')
-    parser.add_argument('--no-item-download', action='store_true',
-                        default=False,
-                        help='do not download items with a "path"')
-    parser.add_argument('--dry-run', action='store_true', default=False,
-                        help='only report what would be done')
-    parser.add_argument('--progress', action='store_true', default=False,
-                        help='show progress for downloading files')
-    parser.add_argument('--mirror', action='append', default=[],
-                        dest="mirrors",
-                        help='additional mirrors to find referenced files')
-
-    parser.add_argument('--verbose', '-v', action='count', default=0)
-    parser.add_argument('--log-file', default=sys.stderr,
-                        type=argparse.FileType('w'))
-
-    parser.add_argument('--keyring', action='store', default=None,
-                        help='keyring to be specified to gpg via --keyring')
-    parser.add_argument('--no-verify', '-U', action='store_false',
-                        dest='verify', default=True,
-                        help="do not gpg check signed json files")
-    parser.add_argument('--no-checksumming-reader', action='store_false',
-                        dest='checksumming_reader', default=True,
-                        help=("do not call 'insert_item' with a reader"
-                              " that does checksumming."))
-
-    parser.add_argument('source_mirror')
-    parser.add_argument('output_d')
-    parser.add_argument('filters', nargs='*', default=[])
+    parser.add_argument(
+        "--keep",
+        action="store_true",
+        default=False,
+        help="keep items in target up to MAX items "
+        "even after they have fallen out of the source",
+    )
+    parser.add_argument(
+        "--max",
+        type=int,
+        default=None,
+        help="store at most MAX items in the target",
+    )
+    parser.add_argument(
+        "--path",
+        default=None,
+        help="sync from index or products file in mirror",
+    )
+    parser.add_argument(
+        "--no-item-download",
+        action="store_true",
+        default=False,
+        help='do not download items with a "path"',
+    )
+    parser.add_argument(
+        "--dry-run",
+        action="store_true",
+        default=False,
+        help="only report what would be done",
+    )
+    parser.add_argument(
+        "--progress",
+        action="store_true",
+        default=False,
+        help="show progress for downloading files",
+    )
+    parser.add_argument(
+        "--mirror",
+        action="append",
+        default=[],
+        dest="mirrors",
+        help="additional mirrors to find referenced files",
+    )
+
+    parser.add_argument("--verbose", "-v", action="count", default=0)
+    parser.add_argument(
+        "--log-file", default=sys.stderr, type=argparse.FileType("w")
+    )
+
+    parser.add_argument(
+        "--keyring",
+        action="store",
+        default=None,
+        help="keyring to be specified to gpg via --keyring",
+    )
+    parser.add_argument(
+        "--no-verify",
+        "-U",
+        action="store_false",
+        dest="verify",
+        default=True,
+        help="do not gpg check signed json files",
+    )
+    parser.add_argument(
+        "--no-checksumming-reader",
+        action="store_false",
+        dest="checksumming_reader",
+        default=True,
+        help=(
+            "do not call 'insert_item' with a reader"
+            " that does checksumming."
+        ),
+    )
+
+    parser.add_argument("source_mirror")
+    parser.add_argument("output_d")
+    parser.add_argument("filters", nargs="*", default=[])
 
     args = parser.parse_args()
 
-    (mirror_url, initial_path) = util.path_from_mirror_url(args.source_mirror,
-                                                           args.path)
+    (mirror_url, initial_path) = util.path_from_mirror_url(
+        args.source_mirror, args.path
+    )
 
     def policy(content, path):
-        if initial_path.endswith('sjson'):
-            return util.read_signed(content,
-                                    keyring=args.keyring,
-                                    checked=args.verify)
+        if initial_path.endswith("sjson"):
+            return util.read_signed(
+                content, keyring=args.keyring, checked=args.verify
+            )
         else:
             return content
 
     filter_list = filters.get_filters(args.filters)
-    mirror_config = {'max_items': args.max, 'keep_items': args.keep,
-                     'filters': filter_list,
-                     'item_download': not args.no_item_download,
-                     'checksumming_reader': args.checksumming_reader}
+    mirror_config = {
+        "max_items": args.max,
+        "keep_items": args.keep,
+        "filters": filter_list,
+        "item_download": not args.no_item_download,
+        "checksumming_reader": args.checksumming_reader,
+    }
 
     level = (log.ERROR, log.INFO, log.DEBUG)[min(args.verbose, 2)]
     log.basicConfig(stream=args.log_file, level=level)
 
-    smirror = mirrors.UrlMirrorReader(mirror_url, mirrors=args.mirrors,
-                                      policy=policy)
+    smirror = mirrors.UrlMirrorReader(
+        mirror_url, mirrors=args.mirrors, policy=policy
+    )
     tstore = objectstores.FileStore(args.output_d)
 
-    drmirror = mirrors.DryRunMirrorWriter(config=mirror_config,
-                                          objectstore=tstore)
+    drmirror = mirrors.DryRunMirrorWriter(
+        config=mirror_config, objectstore=tstore
+    )
     drmirror.sync(smirror, initial_path)
 
     def print_diff(char, items):
         for pedigree, path, size in items:
             fmt = "{char} {pedigree} {path} {size} Mb\n"
             size = int(size / (1024 * 1024))
-            sys.stderr.write(fmt.format(
-                char=char, pedigree=' '.join(pedigree), path=path, size=size))
-
-    print_diff('+', drmirror.downloading)
-    print_diff('-', drmirror.removing)
+            sys.stderr.write(
+                fmt.format(
+                    char=char,
+                    pedigree=" ".join(pedigree),
+                    path=path,
+                    size=size,
+                )
+            )
+
+    print_diff("+", drmirror.downloading)
+    print_diff("-", drmirror.removing)
     sys.stderr.write("%d Mb change\n" % (drmirror.size / (1024 * 1024)))
 
     if args.dry_run:
@@ -147,13 +198,14 @@ def main():
 
     tstore = objectstores.FileStore(args.output_d, complete_callback=callback)
 
-    tmirror = mirrors.ObjectFilterMirror(config=mirror_config,
-                                         objectstore=tstore)
+    tmirror = mirrors.ObjectFilterMirror(
+        config=mirror_config, objectstore=tstore
+    )
 
     tmirror.sync(smirror, initial_path)
 
 
-if __name__ == '__main__':
+if __name__ == "__main__":
     main()
 
 # vi: ts=4 expandtab syntax=python
diff --git a/bin/sstream-mirror-glance b/bin/sstream-mirror-glance
index 5907137..953a2d3 100755
--- a/bin/sstream-mirror-glance
+++ b/bin/sstream-mirror-glance
@@ -23,15 +23,11 @@ import argparse
 import os.path
 import sys
 
-from simplestreams import objectstores
-from simplestreams.objectstores import swift
-from simplestreams import log
-from simplestreams import mirrors
-from simplestreams import openstack
-from simplestreams import util
+from simplestreams import log, mirrors, objectstores, openstack, util
 from simplestreams.mirrors import glance
+from simplestreams.objectstores import swift
 
-DEFAULT_FILTERS = ['ftype~(disk1.img|disk.img)', 'arch~(x86_64|amd64|i386)']
+DEFAULT_FILTERS = ["ftype~(disk1.img|disk.img)", "arch~(x86_64|amd64|i386)"]
 DEFAULT_KEYRING = "/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg"
 
 
@@ -44,94 +40,172 @@ class StdoutProgressAggregator(util.ProgressAggregator):
         super(StdoutProgressAggregator, self).__init__(remaining_items)
 
     def emit(self, progress):
-        size = float(progress['size'])
-        written = float(progress['written'])
-        print("%.2f %s (%d of %d images) - %.2f" %
-              (written / size, progress['name'],
-               self.total_image_count - len(self.remaining_items) + 1,
-               self.total_image_count,
-               float(self.total_written) / self.total_size))
+        size = float(progress["size"])
+        written = float(progress["written"])
+        print(
+            "%.2f %s (%d of %d images) - %.2f"
+            % (
+                written / size,
+                progress["name"],
+                self.total_image_count - len(self.remaining_items) + 1,
+                self.total_image_count,
+                float(self.total_written) / self.total_size,
+            )
+        )
 
 
 def main():
     parser = argparse.ArgumentParser()
 
-    parser.add_argument('--keep', action='store_true', default=False,
-                        help='keep items in target up to MAX items '
-                             'even after they have fallen out of the source')
-    parser.add_argument('--max', type=int, default=None,
-                        help='store at most MAX items in the target')
-
-    parser.add_argument('--region', action='append', default=None,
-                        dest='regions',
-                        help='operate on specified region '
-                             '[useable multiple times]')
-
-    parser.add_argument('--mirror', action='append', default=[],
-                        dest="mirrors",
-                        help='additional mirrors to find referenced files')
-    parser.add_argument('--path', default=None,
-                        help='sync from index or products file in mirror')
-    parser.add_argument('--output-dir', metavar="DIR", default=False,
-                        help='write image data to storage in dir')
-    parser.add_argument('--output-swift', metavar="prefix", default=False,
-                        help='write image data to swift under prefix')
-
-    parser.add_argument('--name-prefix', metavar="prefix", default=None,
-                        help='prefix for each published image name')
-    parser.add_argument('--cloud-name', metavar="name", default=None,
-                        required=True, help='unique name for this cloud')
-    parser.add_argument('--modify-hook', metavar="cmd", default=None,
-                        required=False,
-                        help='invoke cmd on each image prior to upload')
-    parser.add_argument('--content-id', metavar="name", default=None,
-                        required=True,
-                        help='content-id to use for published data.'
-                             '  may contain "%%(region)s"')
-
-    parser.add_argument('--progress', action='store_true', default=False,
-                        help='display per-item download progress')
-    parser.add_argument('--verbose', '-v', action='count', default=0)
-    parser.add_argument('--log-file', default=sys.stderr,
-                        type=argparse.FileType('w'))
-
-    parser.add_argument('--keyring', action='store', default=DEFAULT_KEYRING,
-                        help='The keyring for gpg --keyring')
-
-    parser.add_argument('source_mirror')
-    parser.add_argument('item_filters', nargs='*', default=DEFAULT_FILTERS,
-                        help="Filter expression for mirrored items. "
-                        "Multiple filter arguments can be specified"
-                        "and will be combined with logical AND. "
-                        "Expressions are key[!]=literal_string "
-                        "or key[!]~regexp.")
-
-    parser.add_argument('--hypervisor-mapping', action='store_true',
-                        default=False,
-                        help="Set hypervisor_type attribute on stored images "
-                        "and the virt attribute in the associated stream "
-                        "data. This is useful in OpenStack Clouds which use "
-                        "multiple hypervisor types with in a single region.")
-
-    parser.add_argument('--custom-property', action='append', default=[],
-                        dest="custom_properties",
-                        help='additional properties to add to glance'
-                        ' image metadata (key=value format).')
-
-    parser.add_argument('--visibility', action='store', default='public',
-                        choices=('public', 'private', 'community', 'shared'),
-                        help='Visibility to apply to stored images.')
-
-    parser.add_argument('--image-import-conversion', action='store_true',
-                        default=False,
-                        help="Enable conversion of images to raw format using "
-                        "image import option in Glance.")
-
-    parser.add_argument('--set-latest-property', action='store_true',
-                        default=False,
-                        help="Set 'latest=true' property to latest synced "
-                        "os_version/architecture image metadata and remove "
-                        "latest property from the old images.")
+    parser.add_argument(
+        "--keep",
+        action="store_true",
+        default=False,
+        help="keep items in target up to MAX items "
+        "even after they have fallen out of the source",
+    )
+    parser.add_argument(
+        "--max",
+        type=int,
+        default=None,
+        help="store at most MAX items in the target",
+    )
+
+    parser.add_argument(
+        "--region",
+        action="append",
+        default=None,
+        dest="regions",
+        help="operate on specified region " "[useable multiple times]",
+    )
+
+    parser.add_argument(
+        "--mirror",
+        action="append",
+        default=[],
+        dest="mirrors",
+        help="additional mirrors to find referenced files",
+    )
+    parser.add_argument(
+        "--path",
+        default=None,
+        help="sync from index or products file in mirror",
+    )
+    parser.add_argument(
+        "--output-dir",
+        metavar="DIR",
+        default=False,
+        help="write image data to storage in dir",
+    )
+    parser.add_argument(
+        "--output-swift",
+        metavar="prefix",
+        default=False,
+        help="write image data to swift under prefix",
+    )
+
+    parser.add_argument(
+        "--name-prefix",
+        metavar="prefix",
+        default=None,
+        help="prefix for each published image name",
+    )
+    parser.add_argument(
+        "--cloud-name",
+        metavar="name",
+        default=None,
+        required=True,
+        help="unique name for this cloud",
+    )
+    parser.add_argument(
+        "--modify-hook",
+        metavar="cmd",
+        default=None,
+        required=False,
+        help="invoke cmd on each image prior to upload",
+    )
+    parser.add_argument(
+        "--content-id",
+        metavar="name",
+        default=None,
+        required=True,
+        help="content-id to use for published data."
+        '  may contain "%%(region)s"',
+    )
+
+    parser.add_argument(
+        "--progress",
+        action="store_true",
+        default=False,
+        help="display per-item download progress",
+    )
+    parser.add_argument("--verbose", "-v", action="count", default=0)
+    parser.add_argument(
+        "--log-file", default=sys.stderr, type=argparse.FileType("w")
+    )
+
+    parser.add_argument(
+        "--keyring",
+        action="store",
+        default=DEFAULT_KEYRING,
+        help="The keyring for gpg --keyring",
+    )
+
+    parser.add_argument("source_mirror")
+    parser.add_argument(
+        "item_filters",
+        nargs="*",
+        default=DEFAULT_FILTERS,
+        help="Filter expression for mirrored items. "
+        "Multiple filter arguments can be specified"
+        "and will be combined with logical AND. "
+        "Expressions are key[!]=literal_string "
+        "or key[!]~regexp.",
+    )
+
+    parser.add_argument(
+        "--hypervisor-mapping",
+        action="store_true",
+        default=False,
+        help="Set hypervisor_type attribute on stored images "
+        "and the virt attribute in the associated stream "
+        "data. This is useful in OpenStack Clouds which use "
+        "multiple hypervisor types with in a single region.",
+    )
+
+    parser.add_argument(
+        "--custom-property",
+        action="append",
+        default=[],
+        dest="custom_properties",
+        help="additional properties to add to glance"
+        " image metadata (key=value format).",
+    )
+
+    parser.add_argument(
+        "--visibility",
+        action="store",
+        default="public",
+        choices=("public", "private", "community", "shared"),
+        help="Visibility to apply to stored images.",
+    )
+
+    parser.add_argument(
+        "--image-import-conversion",
+        action="store_true",
+        default=False,
+        help="Enable conversion of images to raw format using "
+        "image import option in Glance.",
+    )
+
+    parser.add_argument(
+        "--set-latest-property",
+        action="store_true",
+        default=False,
+        help="Set 'latest=true' property to latest synced "
+        "os_version/architecture image metadata and remove "
+        "latest property from the old images.",
+    )
 
     args = parser.parse_args()
 
@@ -139,27 +213,32 @@ def main():
     if args.modify_hook:
         modify_hook = args.modify_hook.split()
 
-    mirror_config = {'max_items': args.max, 'keep_items': args.keep,
-                     'cloud_name': args.cloud_name,
-                     'modify_hook': modify_hook,
-                     'item_filters': args.item_filters,
-                     'hypervisor_mapping': args.hypervisor_mapping,
-                     'custom_properties': args.custom_properties,
-                     'visibility': args.visibility,
-                     'image_import_conversion': args.image_import_conversion,
-                     'set_latest_property': args.set_latest_property}
-
-    (mirror_url, args.path) = util.path_from_mirror_url(args.source_mirror,
-                                                        args.path)
+    mirror_config = {
+        "max_items": args.max,
+        "keep_items": args.keep,
+        "cloud_name": args.cloud_name,
+        "modify_hook": modify_hook,
+        "item_filters": args.item_filters,
+        "hypervisor_mapping": args.hypervisor_mapping,
+        "custom_properties": args.custom_properties,
+        "visibility": args.visibility,
+        "image_import_conversion": args.image_import_conversion,
+        "set_latest_property": args.set_latest_property,
+    }
+
+    (mirror_url, args.path) = util.path_from_mirror_url(
+        args.source_mirror, args.path
+    )
 
     def policy(content, path):  # pylint: disable=W0613
-        if args.path.endswith('sjson'):
+        if args.path.endswith("sjson"):
             return util.read_signed(content, keyring=args.keyring)
         else:
             return content
 
-    smirror = mirrors.UrlMirrorReader(mirror_url, mirrors=args.mirrors,
-                                      policy=policy)
+    smirror = mirrors.UrlMirrorReader(
+        mirror_url, mirrors=args.mirrors, policy=policy
+    )
     if args.output_dir and args.output_swift:
         error("--output-dir and --output-swift are mutually exclusive\n")
         sys.exit(1)
@@ -169,7 +248,7 @@ def main():
 
     regions = args.regions
     if regions is None:
-        regions = openstack.get_regions(services=['image'])
+        regions = openstack.get_regions(services=["image"])
 
     for region in regions:
         if args.output_dir:
@@ -181,25 +260,29 @@ def main():
             sys.stderr.write("not writing data anywhere\n")
             tstore = None
 
-        mirror_config['content_id'] = args.content_id % {'region': region}
+        mirror_config["content_id"] = args.content_id % {"region": region}
 
         if args.progress:
-            drmirror = glance.ItemInfoDryRunMirror(config=mirror_config,
-                                                   objectstore=tstore)
+            drmirror = glance.ItemInfoDryRunMirror(
+                config=mirror_config, objectstore=tstore
+            )
             drmirror.sync(smirror, args.path)
             p = StdoutProgressAggregator(drmirror.items)
             progress_callback = p.progress_callback
         else:
             progress_callback = None
 
-        tmirror = glance.GlanceMirror(config=mirror_config,
-                                      objectstore=tstore, region=region,
-                                      name_prefix=args.name_prefix,
-                                      progress_callback=progress_callback)
+        tmirror = glance.GlanceMirror(
+            config=mirror_config,
+            objectstore=tstore,
+            region=region,
+            name_prefix=args.name_prefix,
+            progress_callback=progress_callback,
+        )
         tmirror.sync(smirror, args.path)
 
 
-if __name__ == '__main__':
+if __name__ == "__main__":
     main()
 
 # vi: ts=4 expandtab syntax=python
diff --git a/bin/sstream-query b/bin/sstream-query
index 6534317..3752cac 100755
--- a/bin/sstream-query
+++ b/bin/sstream-query
@@ -16,11 +16,6 @@
 #   You should have received a copy of the GNU Affero General Public License
 #   along with Simplestreams.  If not, see <http://www.gnu.org/licenses/>.
 
-from simplestreams import filters
-from simplestreams import mirrors
-from simplestreams import log
-from simplestreams import util
-
 import argparse
 import errno
 import json
@@ -28,6 +23,8 @@ import pprint
 import signal
 import sys
 
+from simplestreams import filters, log, mirrors, util
+
 FORMAT_PRETTY = "PRETTY"
 FORMAT_JSON = "JSON"
 DEFAULT_KEYRING = "/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg"
@@ -43,15 +40,15 @@ class FilterMirror(mirrors.BasicMirrorWriter):
         if config is None:
             config = {}
         self.config = config
-        self.filters = config.get('filters', [])
-        outfmt = config.get('output_format')
+        self.filters = config.get("filters", [])
+        outfmt = config.get("output_format")
         if not outfmt:
             outfmt = "%s"
         self.output_format = outfmt
         self.json_entries = []
 
     def load_products(self, path=None, content_id=None):
-        return {'content_id': content_id, 'products': {}}
+        return {"content_id": content_id, "products": {}}
 
     def filter_item(self, data, src, target, pedigree):
         return filters.filter_item(self.filters, data, src, pedigree)
@@ -61,8 +58,8 @@ class FilterMirror(mirrors.BasicMirrorWriter):
         # data is src['products'][ped[0]]['versions'][ped[1]]['items'][ped[2]]
         # contentsource is a ContentSource if 'path' exists in data or None
         data = util.products_exdata(src, pedigree)
-        if 'path' in data:
-            data.update({'item_url': contentsource.url})
+        if "path" in data:
+            data.update({"item_url": contentsource.url})
 
         if self.output_format == FORMAT_PRETTY:
             pprint.pprint(data)
@@ -79,39 +76,71 @@ class FilterMirror(mirrors.BasicMirrorWriter):
 def main():
     parser = argparse.ArgumentParser()
 
-    parser.add_argument('--max', type=int, default=None, dest='max_items',
-                        help='store at most MAX items in the target')
+    parser.add_argument(
+        "--max",
+        type=int,
+        default=None,
+        dest="max_items",
+        help="store at most MAX items in the target",
+    )
 
-    parser.add_argument('--path', default=None,
-                        help='sync from index or products file in mirror')
+    parser.add_argument(
+        "--path",
+        default=None,
+        help="sync from index or products file in mirror",
+    )
 
     fmt_group = parser.add_mutually_exclusive_group()
-    fmt_group.add_argument('--output-format', '-o', action='store',
-                           dest='output_format', default=None,
-                           help="specify output format per python str.format")
-    fmt_group.add_argument('--pretty', action='store_const',
-                           const=FORMAT_PRETTY, dest='output_format',
-                           help="pretty print output")
-    fmt_group.add_argument('--json', action='store_const',
-                           const=FORMAT_JSON, dest='output_format',
-                           help="output in JSON as a list of dicts.")
-    parser.add_argument('--verbose', '-v', action='count', default=0)
-    parser.add_argument('--log-file', default=sys.stderr,
-                        type=argparse.FileType('w'))
-
-    parser.add_argument('--keyring', action='store', default=DEFAULT_KEYRING,
-                        help='keyring to be specified to gpg via --keyring')
-    parser.add_argument('--no-verify', '-U', action='store_false',
-                        dest='verify', default=True,
-                        help="do not gpg check signed json files")
-
-    parser.add_argument('mirror_url')
-    parser.add_argument('filters', nargs='*', default=[])
+    fmt_group.add_argument(
+        "--output-format",
+        "-o",
+        action="store",
+        dest="output_format",
+        default=None,
+        help="specify output format per python str.format",
+    )
+    fmt_group.add_argument(
+        "--pretty",
+        action="store_const",
+        const=FORMAT_PRETTY,
+        dest="output_format",
+        help="pretty print output",
+    )
+    fmt_group.add_argument(
+        "--json",
+        action="store_const",
+        const=FORMAT_JSON,
+        dest="output_format",
+        help="output in JSON as a list of dicts.",
+    )
+    parser.add_argument("--verbose", "-v", action="count", default=0)
+    parser.add_argument(
+        "--log-file", default=sys.stderr, type=argparse.FileType("w")
+    )
+
+    parser.add_argument(
+        "--keyring",
+        action="store",
+        default=DEFAULT_KEYRING,
+        help="keyring to be specified to gpg via --keyring",
+    )
+    parser.add_argument(
+        "--no-verify",
+        "-U",
+        action="store_false",
+        dest="verify",
+        default=True,
+        help="do not gpg check signed json files",
+    )
+
+    parser.add_argument("mirror_url")
+    parser.add_argument("filters", nargs="*", default=[])
 
     cmdargs = parser.parse_args()
 
-    (mirror_url, path) = util.path_from_mirror_url(cmdargs.mirror_url,
-                                                   cmdargs.path)
+    (mirror_url, path) = util.path_from_mirror_url(
+        cmdargs.mirror_url, cmdargs.path
+    )
 
     level = (log.ERROR, log.INFO, log.DEBUG)[min(cmdargs.verbose, 2)]
     log.basicConfig(stream=cmdargs.log_file, level=level)
@@ -119,33 +148,41 @@ def main():
     initial_path = path
 
     def policy(content, path):
-        if initial_path.endswith('sjson'):
-            return util.read_signed(content,
-                                    keyring=cmdargs.keyring,
-                                    checked=cmdargs.verify)
+        if initial_path.endswith("sjson"):
+            return util.read_signed(
+                content, keyring=cmdargs.keyring, checked=cmdargs.verify
+            )
         else:
             return content
 
     smirror = mirrors.UrlMirrorReader(mirror_url, policy=policy)
 
     filter_list = filters.get_filters(cmdargs.filters)
-    cfg = {'max_items': cmdargs.max_items,
-           'filters': filter_list,
-           'output_format': cmdargs.output_format}
+    cfg = {
+        "max_items": cmdargs.max_items,
+        "filters": filter_list,
+        "output_format": cmdargs.output_format,
+    }
 
     tmirror = FilterMirror(config=cfg)
     try:
         tmirror.sync(smirror, path)
         if tmirror.output_format == FORMAT_JSON:
-            print(json.dumps(tmirror.json_entries, indent=2, sort_keys=True,
-                             separators=(',', ': ')))
+            print(
+                json.dumps(
+                    tmirror.json_entries,
+                    indent=2,
+                    sort_keys=True,
+                    separators=(",", ": "),
+                )
+            )
     except IOError as e:
         if e.errno == errno.EPIPE:
             sys.exit(0x80 | signal.SIGPIPE)
         raise
 
 
-if __name__ == '__main__':
+if __name__ == "__main__":
     main()
 
 # vi: ts=4 expandtab syntax=python
diff --git a/bin/sstream-sync b/bin/sstream-sync
index d10b90f..4082400 100755
--- a/bin/sstream-sync
+++ b/bin/sstream-sync
@@ -16,18 +16,17 @@
 #   You should have received a copy of the GNU Affero General Public License
 #   along with Simplestreams.  If not, see <http://www.gnu.org/licenses/>.
 
-from simplestreams import mirrors
-from simplestreams.mirrors import command_hook
-from simplestreams import log
-from simplestreams import util
-
 import argparse
 import errno
 import os
 import signal
 import sys
+
 import yaml
 
+from simplestreams import log, mirrors, util
+from simplestreams.mirrors import command_hook
+
 
 def which(program):
     def is_exe(fpath):
@@ -55,51 +54,90 @@ def main():
     parser = argparse.ArgumentParser()
     defhook = command_hook.DEFAULT_HOOK_NAME
 
-    hooks = [("--hook-%s" % hook.replace("_", "-"), hook, False)
-             for hook in command_hook.HOOK_NAMES]
-    hooks.append(('--hook', defhook, False,))
-
-    parser.add_argument('--config', '-c',
-                        help='read config file',
-                        type=argparse.FileType('rb'))
-
-    for (argname, cfgname, _required) in hooks:
+    hooks = [
+        ("--hook-%s" % hook.replace("_", "-"), hook, False)
+        for hook in command_hook.HOOK_NAMES
+    ]
+    hooks.append(
+        (
+            "--hook",
+            defhook,
+            False,
+        )
+    )
+
+    parser.add_argument(
+        "--config", "-c", help="read config file", type=argparse.FileType("rb")
+    )
+
+    for argname, cfgname, _required in hooks:
         parser.add_argument(argname, dest=cfgname, required=False)
 
-    parser.add_argument('--keep', action='store_true', default=False,
-                        dest='keep_items',
-                        help='keep items in target up to MAX items '
-                             'even after they have fallen out of the source')
-    parser.add_argument('--max', type=int, default=None, dest='max_items',
-                        help='store at most MAX items in the target')
-    parser.add_argument('--item-skip-download', action='store_true',
-                        default=False,
-                        help='Do not download items that are to be inserted.')
-    parser.add_argument('--delete', action='store_true', default=False,
-                        dest='delete_filtered_items',
-                        help='remove filtered items from the target')
-    parser.add_argument('--path', default=None,
-                        help='sync from index or products file in mirror')
-
-    parser.add_argument('--verbose', '-v', action='count', default=0)
-    parser.add_argument('--log-file', default=sys.stderr,
-                        type=argparse.FileType('w'))
-
-    parser.add_argument('--keyring', action='store', default=None,
-                        help='keyring to be specified to gpg via --keyring')
-    parser.add_argument('--no-verify', '-U', action='store_false',
-                        dest='verify', default=True,
-                        help="do not gpg check signed json files")
-
-    parser.add_argument('mirror_url')
+    parser.add_argument(
+        "--keep",
+        action="store_true",
+        default=False,
+        dest="keep_items",
+        help="keep items in target up to MAX items "
+        "even after they have fallen out of the source",
+    )
+    parser.add_argument(
+        "--max",
+        type=int,
+        default=None,
+        dest="max_items",
+        help="store at most MAX items in the target",
+    )
+    parser.add_argument(
+        "--item-skip-download",
+        action="store_true",
+        default=False,
+        help="Do not download items that are to be inserted.",
+    )
+    parser.add_argument(
+        "--delete",
+        action="store_true",
+        default=False,
+        dest="delete_filtered_items",
+        help="remove filtered items from the target",
+    )
+    parser.add_argument(
+        "--path",
+        default=None,
+        help="sync from index or products file in mirror",
+    )
+
+    parser.add_argument("--verbose", "-v", action="count", default=0)
+    parser.add_argument(
+        "--log-file", default=sys.stderr, type=argparse.FileType("w")
+    )
+
+    parser.add_argument(
+        "--keyring",
+        action="store",
+        default=None,
+        help="keyring to be specified to gpg via --keyring",
+    )
+    parser.add_argument(
+        "--no-verify",
+        "-U",
+        action="store_false",
+        dest="verify",
+        default=True,
+        help="do not gpg check signed json files",
+    )
+
+    parser.add_argument("mirror_url")
     cmdargs = parser.parse_args()
 
-    known_cfg = [('--item-skip-download', 'item_skip_download', False),
-                 ('--max', 'max_items', False),
-                 ('--keep', 'keep_items', False),
-                 ('--delete', 'delete_filtered_items', False),
-                 ('mirror_url', 'mirror_url', True),
-                 ('--path', 'path', True)]
+    known_cfg = [
+        ("--item-skip-download", "item_skip_download", False),
+        ("--max", "max_items", False),
+        ("--keep", "keep_items", False),
+        ("--delete", "delete_filtered_items", False),
+        ("mirror_url", "mirror_url", True),
+        ("--path", "path", True),
+    ]
     known_cfg.extend(hooks)
 
     cfg = {}
@@ -116,31 +154,41 @@ def main():
     missing = []
     fallback = cfg.get(defhook, getattr(cmdargs, defhook, None))
 
-    for (argname, cfgname, _required) in known_cfg:
+    for argname, cfgname, _required in known_cfg:
         val = getattr(cmdargs, cfgname)
         if val is not None:
             cfg[cfgname] = val
         if val == "":
             cfg[cfgname] = None
 
-        if ((cfgname in command_hook.HOOK_NAMES or cfgname == defhook) and
-                cfg.get(cfgname) is not None):
+        if (
+            cfgname in command_hook.HOOK_NAMES or cfgname == defhook
+        ) and cfg.get(cfgname) is not None:
             if which(cfg[cfgname]) is None:
                 msg = "invalid input for %s. '%s' is not executable\n"
                 sys.stderr.write(msg % (argname, val))
                 sys.exit(1)
 
-        if (cfgname in command_hook.REQUIRED_FIELDS and
-                cfg.get(cfgname) is None and not fallback):
-            missing.append((argname, cfgname,))
+        if (
+            cfgname in command_hook.REQUIRED_FIELDS
+            and cfg.get(cfgname) is None
+            and not fallback
+        ):
+            missing.append(
+                (
+                    argname,
+                    cfgname,
+                )
+            )
 
     pfm = util.path_from_mirror_url
-    (cfg['mirror_url'], cfg['path']) = pfm(cfg['mirror_url'], cfg.get('path'))
+    (cfg["mirror_url"], cfg["path"]) = pfm(cfg["mirror_url"], cfg.get("path"))
 
     if missing:
-        sys.stderr.write("must provide input for (--hook/%s for default):\n"
-                         % defhook)
-        for (flag, cfg) in missing:
+        sys.stderr.write(
+            "must provide input for (--hook/%s for default):\n" % defhook
+        )
+        for flag, cfg in missing:
             sys.stderr.write("  cmdline '%s' or cfgname '%s'\n" % (flag, cfg))
         sys.exit(1)
 
@@ -148,24 +196,24 @@ def main():
     log.basicConfig(stream=cmdargs.log_file, level=level)
 
     def policy(content, path):
-        if cfg['path'].endswith('sjson'):
-            return util.read_signed(content,
-                                    keyring=cmdargs.keyring,
-                                    checked=cmdargs.verify)
+        if cfg["path"].endswith("sjson"):
+            return util.read_signed(
+                content, keyring=cmdargs.keyring, checked=cmdargs.verify
+            )
         else:
             return content
 
-    smirror = mirrors.UrlMirrorReader(cfg['mirror_url'], policy=policy)
+    smirror = mirrors.UrlMirrorReader(cfg["mirror_url"], policy=policy)
     tmirror = command_hook.CommandHookMirror(config=cfg)
     try:
-        tmirror.sync(smirror, cfg['path'])
+        tmirror.sync(smirror, cfg["path"])
     except IOError as e:
         if e.errno == errno.EPIPE:
             sys.exit(0x80 | signal.SIGPIPE)
         raise
 
 
-if __name__ == '__main__':
+if __name__ == "__main__":
     main()
 
 # vi: ts=4 expandtab syntax=python
diff --git a/pyproject.toml b/pyproject.toml
new file mode 100644
index 0000000..d84cc51
--- /dev/null
+++ b/pyproject.toml
@@ -0,0 +1,6 @@
+[tool.black]
+line-length = 79
+
+[tool.isort]
+profile = "black"
+line_length = 79
diff --git a/setup.py b/setup.py
index 6b5a29b..301721b 100644
--- a/setup.py
+++ b/setup.py
@@ -1,8 +1,9 @@
-from setuptools import setup
-from glob import glob
 import os
+from glob import glob
+
+from setuptools import setup
 
-VERSION = '0.1.0'
+VERSION = "0.1.0"
 
 
 def is_f(p):
@@ -11,18 +12,20 @@ def is_f(p):
 
 setup(
     name="python-simplestreams",
-    description='Library and tools for using Simple Streams data',
+    description="Library and tools for using Simple Streams data",
     version=VERSION,
-    author='Scott Moser',
-    author_email='[e-mail]',
+    author="Scott Moser",
+    author_email="[e-mail]",
     license="AGPL",
-    url='http://launchpad.net/simplestreams/',
-    packages=['simplestreams', 'simplestreams.mirrors',
-              'simplestreams.objectstores'],
-    scripts=glob('bin/*'),
+    url="http://launchpad.net/simplestreams/",
+    packages=[
+        "simplestreams",
+        "simplestreams.mirrors",
+        "simplestreams.objectstores",
+    ],
+    scripts=glob("bin/*"),
     data_files=[
-        ('lib/simplestreams', glob('tools/hook-*')),
-        ('share/doc/simplestreams',
-         [f for f in glob('doc/*') if is_f(f)]),
-    ]
+        ("lib/simplestreams", glob("tools/hook-*")),
+        ("share/doc/simplestreams", [f for f in glob("doc/*") if is_f(f)]),
+    ],
 )
diff --git a/simplestreams/checksum_util.py b/simplestreams/checksum_util.py
index dc695e3..bc98161 100644
--- a/simplestreams/checksum_util.py
+++ b/simplestreams/checksum_util.py
@@ -20,7 +20,7 @@ import hashlib
 CHECKSUMS = ("md5", "sha256", "sha512")
 
 try:
-    ALGORITHMS = list(getattr(hashlib, 'algorithms'))
+    ALGORITHMS = list(getattr(hashlib, "algorithms"))
 except AttributeError:
     ALGORITHMS = list(hashlib.algorithms_available)
 
@@ -56,11 +56,13 @@ class checksummer(object):
         return self._hasher.hexdigest()
 
     def check(self):
-        return (self.expected is None or self.expected == self.hexdigest())
+        return self.expected is None or self.expected == self.hexdigest()
 
     def __str__(self):
-        return ("checksummer (algorithm=%s expected=%s)" %
-                (self.algorithm, self.expected))
+        return "checksummer (algorithm=%s expected=%s)" % (
+            self.algorithm,
+            self.expected,
+        )
 
 
 def item_checksums(item):
@@ -69,14 +71,16 @@ def item_checksums(item):
 
 class SafeCheckSummer(checksummer):
     """SafeCheckSummer raises ValueError if checksums are not provided."""
+
     def __init__(self, checksums, allowed=None):
         if allowed is None:
             allowed = CHECKSUMS
         super(SafeCheckSummer, self).__init__(checksums)
         if self.algorithm not in allowed:
             raise ValueError(
-                "provided checksums (%s) did not include any allowed (%s)" %
-                (checksums, allowed))
+                "provided checksums (%s) did not include any allowed (%s)"
+                % (checksums, allowed)
+            )
 
 
 class InvalidChecksum(ValueError):
@@ -93,18 +97,31 @@ class InvalidChecksum(ValueError):
         if not isinstance(self.expected_size, int):
             msg = "Invalid size '%s' at %s." % (self.expected_size, self.path)
         else:
-            msg = ("Invalid %s Checksum at %s. Found %s. Expected %s. "
-                   "read %s bytes expected %s bytes." %
-                   (self.cksum.algorithm, self.path,
-                    self.cksum.hexdigest(), self.cksum.expected,
-                    self.size, self.expected_size))
+            msg = (
+                "Invalid %s Checksum at %s. Found %s. Expected %s. "
+                "read %s bytes expected %s bytes."
+                % (
+                    self.cksum.algorithm,
+                    self.path,
+                    self.cksum.hexdigest(),
+                    self.cksum.expected,
+                    self.size,
+                    self.expected_size,
+                )
+            )
             if self.size:
-                msg += (" (size %s expected %s)" %
-                        (self.size, self.expected_size))
+                msg += " (size %s expected %s)" % (
+                    self.size,
+                    self.expected_size,
+                )
         return msg
 
 
 def invalid_checksum_for_reader(reader, msg=None):
-    return InvalidChecksum(path=reader.url, cksum=reader.checksummer,
-                           size=reader.bytes_read, expected_size=reader.size,
-                           msg=msg)
+    return InvalidChecksum(
+        path=reader.url,
+        cksum=reader.checksummer,
+        size=reader.bytes_read,
+        expected_size=reader.size,
+        msg=msg,
+    )
diff --git a/simplestreams/contentsource.py b/simplestreams/contentsource.py
index ce45097..e5c6c98 100644
--- a/simplestreams/contentsource.py
+++ b/simplestreams/contentsource.py
@@ -23,12 +23,13 @@ import sys
 from . import checksum_util
 
 if sys.version_info > (3, 0):
+    import urllib.error as urllib_error
     import urllib.parse as urlparse
     import urllib.request as urllib_request
-    import urllib.error as urllib_error
 else:
-    import urlparse
     import urllib2 as urllib_request
+    import urlparse
+
     urllib_error = urllib_request
 
 READ_BUFFER_SIZE = 1024 * 10
@@ -38,12 +39,14 @@ try:
     # We try to use requests because we can do gzip encoding with it.
     # however requests < 1.1 didn't have 'stream' argument to 'get'
     # making it completely unsuitable for downloading large files.
-    import requests
     from distutils.version import LooseVersion
+
     import pkg_resources
-    _REQ = pkg_resources.get_distribution('requests')
+    import requests
+
+    _REQ = pkg_resources.get_distribution("requests")
     _REQ_VER = LooseVersion(_REQ.version)
-    if _REQ_VER < LooseVersion('1.1'):
+    if _REQ_VER < LooseVersion("1.1"):
         raise ImportError("Requests version < 1.1, not suitable for usage.")
     URL_READER_CLASSNAME = "RequestsUrlReader"
 except ImportError:
@@ -63,8 +66,8 @@ class ContentSource(object):
         raise NotImplementedError()
 
     def set_start_pos(self, offset):
-        """ Implemented if the ContentSource supports seeking within content.
-        Used to resume failed transfers. """
+        """Implemented if the ContentSource supports seeking within content.
+        Used to resume failed transfers."""
 
         class SetStartPosNotImplementedError(NotImplementedError):
             pass
@@ -122,8 +125,9 @@ class UrlContentSource(ContentSource):
                 if e.errno != errno.ENOENT:
                     raise
                 continue
-        myerr = IOError("Unable to open %s. mirrors=%s" %
-                        (self.input_url, self.mirrors))
+        myerr = IOError(
+            "Unable to open %s. mirrors=%s" % (self.input_url, self.mirrors)
+        )
         myerr.errno = errno.ENOENT
         raise myerr
 
@@ -181,7 +185,7 @@ class IteratorContentSource(ContentSource):
             raise exc
 
     def is_enoent(self, exc):
-        return (isinstance(exc, IOError) and exc.errno == errno.ENOENT)
+        return isinstance(exc, IOError) and exc.errno == errno.ENOENT
 
     def read(self, size=None):
         self.open()
@@ -189,7 +193,7 @@ class IteratorContentSource(ContentSource):
         if self.consumed:
             return bytes()
 
-        if (size is None or size < 0):
+        if size is None or size < 0:
             # read everything
             ret = self.leftover
             self.leftover = bytes()
@@ -227,7 +231,7 @@ class IteratorContentSource(ContentSource):
 class MemoryContentSource(FdContentSource):
     def __init__(self, url=None, content=""):
         if isinstance(content, str):
-            content = content.encode('utf-8')
+            content = content.encode("utf-8")
         fd = io.BytesIO(content)
         if url is None:
             url = "MemoryContentSource://undefined"
@@ -265,8 +269,10 @@ class ChecksummingContentSource(ContentSource):
 
     def _set_checksummer(self, checksummer):
         if checksummer.algorithm not in checksum_util.CHECKSUMS:
-            raise ValueError("algorithm %s is not valid (%s)" %
-                             (checksummer.algorithm, checksum_util.CHECKSUMS))
+            raise ValueError(
+                "algorithm %s is not valid (%s)"
+                % (checksummer.algorithm, checksum_util.CHECKSUMS)
+            )
         self.checksummer = checksummer
 
     def check(self):
@@ -322,7 +328,6 @@ class FileReader(UrlReader):
 
 
 class Urllib2UrlReader(UrlReader):
-
     timeout = TIMEOUT
 
     def __init__(self, url, offset=None, user_agent=None):
@@ -339,9 +344,9 @@ class Urllib2UrlReader(UrlReader):
         try:
             req = urllib_request.Request(url)
             if user_agent is not None:
-                req.add_header('User-Agent', user_agent)
+                req.add_header("User-Agent", user_agent)
             if offset is not None:
-                req.add_header('Range', 'bytes=%d-' % offset)
+                req.add_header("Range", "bytes=%d-" % offset)
             self.req = opener(req, timeout=self.timeout)
         except urllib_error.HTTPError as e:
             if e.code == 404:
@@ -373,8 +378,10 @@ class RequestsUrlReader(UrlReader):
 
     def __init__(self, url, buflen=None, offset=None, user_agent=None):
         if requests is None:
-            raise ImportError("Attempt to use RequestsUrlReader "
-                              "without suitable requests library.")
+            raise ImportError(
+                "Attempt to use RequestsUrlReader "
+                "without suitable requests library."
+            )
         self.url = url
         (url, user, password) = parse_url_auth(url)
         if user is None:
@@ -384,15 +391,15 @@ class RequestsUrlReader(UrlReader):
 
         headers = {}
         if user_agent is not None:
-            headers['User-Agent'] = user_agent
+            headers["User-Agent"] = user_agent
         if offset is not None:
-            headers['Range'] = 'bytes=%d-' % offset
+            headers["Range"] = "bytes=%d-" % offset
         if headers == {}:
             headers = None
 
         # requests version less than 2.4.1 takes an optional
         # float for timeout. There is no separate read timeout
-        if _REQ_VER < LooseVersion('2.4.1'):
+        if _REQ_VER < LooseVersion("2.4.1"):
             self.timeout = TIMEOUT
 
         self.req = requests.get(
@@ -405,13 +412,13 @@ class RequestsUrlReader(UrlReader):
         self.leftover = bytes()
         self.consumed = False
 
-        if (self.req.status_code == requests.codes.NOT_FOUND):
+        if self.req.status_code == requests.codes.NOT_FOUND:
             myerr = IOError("Unable to open %s" % url)
             myerr.errno = errno.ENOENT
             raise myerr
 
-        ce = self.req.headers.get('content-encoding', '').lower()
-        if 'gzip' in ce or 'deflate' in ce:
+        ce = self.req.headers.get("content-encoding", "").lower()
+        if "gzip" in ce or "deflate" in ce:
             self._read = self.read_compressed
         else:
             self._read = self.read_raw
@@ -426,7 +433,7 @@ class RequestsUrlReader(UrlReader):
         if self.consumed:
             return bytes()
 
-        if (size is None or size < 0):
+        if size is None or size < 0:
             # read everything
             ret = self.leftover
             self.leftover = bytes()
diff --git a/simplestreams/filters.py b/simplestreams/filters.py
index 3818949..330a475 100644
--- a/simplestreams/filters.py
+++ b/simplestreams/filters.py
@@ -15,10 +15,10 @@
 #   You should have received a copy of the GNU Affero General Public License
 #   along with Simplestreams.  If not, see <http://www.gnu.org/licenses/>.
 
-from simplestreams import util
-
 import re
 
+from simplestreams import util
+
 
 class ItemFilter(object):
     def __init__(self, content, noneval=""):
@@ -37,7 +37,7 @@ class ItemFilter(object):
         else:
             raise ValueError("Bad parsing of %s" % content)
 
-        self.negator = (op[0] != "!")
+        self.negator = op[0] != "!"
         self.op = op
         self.key = key
         self.value = val
@@ -45,15 +45,19 @@ class ItemFilter(object):
         self.noneval = noneval
 
     def __str__(self):
-        return "%s %s %s [none=%s]" % (self.key, self.op,
-                                       self.value, self.noneval)
+        return "%s %s %s [none=%s]" % (
+            self.key,
+            self.op,
+            self.value,
+            self.noneval,
+        )
 
     def __repr__(self):
         return self.__str__()
 
     def matches(self, item):
         val = str(item.get(self.key, self.noneval))
-        return (self.negator == bool(self._matches(val)))
+        return self.negator == bool(self._matches(val))
 
 
 def get_filters(filters, noneval=""):
diff --git a/simplestreams/generate_simplestreams.py b/simplestreams/generate_simplestreams.py
index 9b7b919..329a91d 100644
--- a/simplestreams/generate_simplestreams.py
+++ b/simplestreams/generate_simplestreams.py
@@ -13,54 +13,58 @@
 #   or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU Affero General Public
 #   License for more details.
 #
-from collections import namedtuple
-from copy import deepcopy
 import json
 import os
 import sys
+from collections import namedtuple
+from copy import deepcopy
 
 from simplestreams import util
 
-
-Item = namedtuple('Item', ['content_id', 'product_name', 'version_name',
-                           'item_name', 'data'])
+Item = namedtuple(
+    "Item", ["content_id", "product_name", "version_name", "item_name", "data"]
+)
 
 
 def items2content_trees(itemslist, exdata):
     # input is a list with each item having:
     #   (content_id, product_name, version_name, item_name, {data})
     ctrees = {}
-    for (content_id, prodname, vername, itemname, data) in itemslist:
+    for content_id, prodname, vername, itemname, data in itemslist:
         if content_id not in ctrees:
-            ctrees[content_id] = {'content_id': content_id,
-                                  'format': 'products:1.0', 'products': {}}
+            ctrees[content_id] = {
+                "content_id": content_id,
+                "format": "products:1.0",
+                "products": {},
+            }
             ctrees[content_id].update(exdata)
 
         ctree = ctrees[content_id]
-        if prodname not in ctree['products']:
-            ctree['products'][prodname] = {'versions': {}}
+        if prodname not in ctree["products"]:
+            ctree["products"][prodname] = {"versions": {}}
 
-        prodtree = ctree['products'][prodname]
-        if vername not in prodtree['versions']:
-            prodtree['versions'][vername] = {'items': {}}
+        prodtree = ctree["products"][prodname]
+        if vername not in prodtree["versions"]:
+            prodtree["versions"][vername] = {"items": {}}
 
-        vertree = prodtree['versions'][vername]
+        vertree = prodtree["versions"][vername]
 
-        if itemname in vertree['items']:
-            raise ValueError("%s: already existed" %
-                             str([content_id, prodname, vername, itemname]))
+        if itemname in vertree["items"]:
+            raise ValueError(
+                "%s: already existed"
+                % str([content_id, prodname, vername, itemname])
+            )
 
-        vertree['items'][itemname] = data
+        vertree["items"][itemname] = data
     return ctrees
 
 
 class FileNamer:
-
-    streamdir = 'streams/v1'
+    streamdir = "streams/v1"
 
     @classmethod
     def get_index_path(cls):
-        return "%s/%s" % (cls.streamdir, 'index.json')
+        return "%s/%s" % (cls.streamdir, "index.json")
 
     @classmethod
     def get_content_path(cls, content_id):
@@ -68,16 +72,16 @@ class FileNamer:
 
 
 def generate_index(trees, updated, namer):
-    index = {"index": {}, 'format': 'index:1.0', 'updated': updated}
-    not_copied_up = ['content_id']
+    index = {"index": {}, "format": "index:1.0", "updated": updated}
+    not_copied_up = ["content_id"]
     for content_id, content in trees.items():
-        index['index'][content_id] = {
-            'products': sorted(list(content['products'].keys())),
-            'path': namer.get_content_path(content_id),
+        index["index"][content_id] = {
+            "products": sorted(list(content["products"].keys())),
+            "path": namer.get_content_path(content_id),
         }
         for k in util.stringitems(content):
             if k not in not_copied_up:
-                index['index'][content_id][k] = content[k]
+                index["index"][content_id][k] = content[k]
     return index
 
 
@@ -85,20 +89,29 @@ def write_streams(out_d, trees, updated, namer=None, condense=True):
     if namer is None:
         namer = FileNamer
     index = generate_index(trees, updated, namer)
-    to_write = [(namer.get_index_path(), index,)]
+    to_write = [
+        (
+            namer.get_index_path(),
+            index,
+        )
+    ]
     # Don't let products_condense modify the input
     trees = deepcopy(trees)
     for content_id in trees:
         if condense:
-            util.products_condense(trees[content_id],
-                                   sticky=[
-                                       'path', 'sha256', 'md5',
-                                       'size', 'mirrors'
-                                   ])
+            util.products_condense(
+                trees[content_id],
+                sticky=["path", "sha256", "md5", "size", "mirrors"],
+            )
         content = trees[content_id]
-        to_write.append((index['index'][content_id]['path'], content,))
+        to_write.append(
+            (
+                index["index"][content_id]["path"],
+                content,
+            )
+        )
     out_filenames = []
-    for (outfile, data) in to_write:
+    for outfile, data in to_write:
         filef = os.path.join(out_d, outfile)
         util.mkdir_p(os.path.dirname(filef))
         json_dump(data, filef)
@@ -108,6 +121,8 @@ def write_streams(out_d, trees, updated, namer=None, condense=True):
 
 def json_dump(data, filename):
     with open(filename, "w") as fp:
-        sys.stderr.write(u"writing %s\n" % filename)
-        fp.write(json.dumps(data, indent=2, sort_keys=True,
-                 separators=(',', ': ')) + "\n")
+        sys.stderr.write("writing %s\n" % filename)
+        fp.write(
+            json.dumps(data, indent=2, sort_keys=True, separators=(",", ": "))
+            + "\n"
+        )
diff --git a/simplestreams/json2streams.py b/simplestreams/json2streams.py
index e9749f6..20b09e7 100755
--- a/simplestreams/json2streams.py
+++ b/simplestreams/json2streams.py
@@ -1,43 +1,41 @@
 #!/usr/bin/env python3
 #   Copyright (C) 2013, 2015 Canonical Ltd.
 
-from argparse import ArgumentParser
 import json
 import os
 import sys
+from argparse import ArgumentParser
 
 from simplestreams import util
-
 from simplestreams.generate_simplestreams import (
     FileNamer,
     Item,
     items2content_trees,
     json_dump,
     write_streams,
-    )
+)
 
 
 class JujuFileNamer(FileNamer):
-
     @classmethod
     def get_index_path(cls):
-        return "%s/%s" % (cls.streamdir, 'index2.json')
+        return "%s/%s" % (cls.streamdir, "index2.json")
 
     @classmethod
     def get_content_path(cls, content_id):
-        return "%s/%s.json" % (cls.streamdir, content_id.replace(':', '-'))
+        return "%s/%s.json" % (cls.streamdir, content_id.replace(":", "-"))
 
 
 def dict_to_item(item_dict):
     """Convert a dict into an Item, mutating input."""
-    item_dict.pop('item_url', None)
-    size = item_dict.get('size')
+    item_dict.pop("item_url", None)
+    size = item_dict.get("size")
     if size is not None:
-        item_dict['size'] = int(size)
-    content_id = item_dict.pop('content_id')
-    product_name = item_dict.pop('product_name')
-    version_name = item_dict.pop('version_name')
-    item_name = item_dict.pop('item_name')
+        item_dict["size"] = int(size)
+    content_id = item_dict.pop("content_id")
+    product_name = item_dict.pop("product_name")
+    version_name = item_dict.pop("version_name")
+    item_name = item_dict.pop("item_name")
     return Item(content_id, product_name, version_name, item_name, item_dict)
 
 
@@ -51,9 +49,11 @@ def write_release_index(out_d):
     in_path = os.path.join(out_d, JujuFileNamer.get_index_path())
     with open(in_path) as in_file:
         full_index = json.load(in_file)
-    full_index['index'] = dict(
-        (k, v) for k, v in list(full_index['index'].items())
-        if k == 'com.ubuntu.juju:released:tools')
+    full_index["index"] = dict(
+        (k, v)
+        for k, v in list(full_index["index"].items())
+        if k == "com.ubuntu.juju:released:tools"
+    )
     out_path = os.path.join(out_d, FileNamer.get_index_path())
     json_dump(full_index, out_path)
     return out_path
@@ -70,7 +70,7 @@ def filenames_to_streams(filenames, updated, out_d, juju_format=False):
     for items_file in filenames:
         items.extend(read_items_file(items_file))
 
-    data = {'updated': updated, 'datatype': 'content-download'}
+    data = {"updated": updated, "datatype": "content-download"}
     trees = items2content_trees(items, data)
     if juju_format:
         write = write_juju_streams
@@ -88,23 +88,31 @@ def write_juju_streams(out_d, trees, updated):
 def parse_args(argv=None):
     parser = ArgumentParser()
     parser.add_argument(
-        'items_file', metavar='items-file', help='File to read items from',
-        nargs='+')
+        "items_file",
+        metavar="items-file",
+        help="File to read items from",
+        nargs="+",
+    )
     parser.add_argument(
-        'out_d', metavar='output-dir',
-        help='The directory to write stream files to.')
+        "out_d",
+        metavar="output-dir",
+        help="The directory to write stream files to.",
+    )
     parser.add_argument(
-        '--juju-format', action='store_true',
-        help='Write stream files in juju format.')
+        "--juju-format",
+        action="store_true",
+        help="Write stream files in juju format.",
+    )
     return parser.parse_args(argv)
 
 
 def main():
     args = parse_args()
     updated = util.timestamp()
-    filenames_to_streams(args.items_file, updated, args.out_d,
-                         args.juju_format)
+    filenames_to_streams(
+        args.items_file, updated, args.out_d, args.juju_format
+    )
 
 
-if __name__ == '__main__':
+if __name__ == "__main__":
     sys.exit(main())
diff --git a/simplestreams/log.py b/simplestreams/log.py
index 061103a..824f248 100644
--- a/simplestreams/log.py
+++ b/simplestreams/log.py
@@ -33,18 +33,22 @@ class NullHandler(logging.Handler):
 
 def basicConfig(**kwargs):
     # basically like logging.basicConfig but only output for our logger
-    if kwargs.get('filename'):
-        handler = logging.FileHandler(filename=kwargs['filename'],
-                                      mode=kwargs.get('filemode', 'a'))
-    elif kwargs.get('stream'):
-        handler = logging.StreamHandler(stream=kwargs['stream'])
+    if kwargs.get("filename"):
+        handler = logging.FileHandler(
+            filename=kwargs["filename"], mode=kwargs.get("filemode", "a")
+        )
+    elif kwargs.get("stream"):
+        handler = logging.StreamHandler(stream=kwargs["stream"])
     else:
         handler = NullHandler()
 
-    level = kwargs.get('level', NOTSET)
+    level = kwargs.get("level", NOTSET)
 
-    handler.setFormatter(logging.Formatter(fmt=kwargs.get('format'),
-                                           datefmt=kwargs.get('datefmt')))
+    handler.setFormatter(
+        logging.Formatter(
+            fmt=kwargs.get("format"), datefmt=kwargs.get("datefmt")
+        )
+    )
     handler.setLevel(level)
 
     logging.getLogger().setLevel(level)
@@ -56,7 +60,7 @@ def basicConfig(**kwargs):
     logger.addHandler(handler)
 
 
-def _getLogger(name='sstreams'):
+def _getLogger(name="sstreams"):
     return logging.getLogger(name)
 
 
diff --git a/simplestreams/mirrors/__init__.py b/simplestreams/mirrors/__init__.py
index 4a9593a..b3374f8 100644
--- a/simplestreams/mirrors/__init__.py
+++ b/simplestreams/mirrors/__init__.py
@@ -18,10 +18,10 @@ import errno
 import io
 import json
 
+import simplestreams.contentsource as cs
 import simplestreams.filters as filters
 import simplestreams.util as util
 from simplestreams import checksum_util
-import simplestreams.contentsource as cs
 from simplestreams.log import LOG
 
 DEFAULT_USER_AGENT = "python-simplestreams/0.1"
@@ -29,8 +29,8 @@ DEFAULT_USER_AGENT = "python-simplestreams/0.1"
 
 class MirrorReader(object):
     def __init__(self, policy=util.policy_read_signed):
-        """ policy should be a function which returns the extracted payload or
-        raises an exception if the policy is violated. """
+        """policy should be a function which returns the extracted payload or
+        raises an exception if the policy is violated."""
         self.policy = policy
 
     def load_products(self, path):
@@ -39,7 +39,7 @@ class MirrorReader(object):
 
     def read_json(self, path):
         with self.source(path) as source:
-            raw = source.read().decode('utf-8')
+            raw = source.read().decode("utf-8")
         return raw, self.policy(content=raw, path=path)
 
     def source(self, path):
@@ -164,8 +164,13 @@ class MirrorWriter(object):
 
 
 class UrlMirrorReader(MirrorReader):
-    def __init__(self, prefix, mirrors=None, policy=util.policy_read_signed,
-                 user_agent=DEFAULT_USER_AGENT):
+    def __init__(
+        self,
+        prefix,
+        mirrors=None,
+        policy=util.policy_read_signed,
+        user_agent=DEFAULT_USER_AGENT,
+    ):
         super(UrlMirrorReader, self).__init__(policy=policy)
         self._cs = cs.UrlContentSource
         if mirrors is None:
@@ -184,13 +189,18 @@ class UrlMirrorReader(MirrorReader):
 
             def url_reader_factory(*args, **kwargs):
                 return cs.URL_READER(
-                    *args, user_agent=self.user_agent, **kwargs)
+                    *args, user_agent=self.user_agent, **kwargs
+                )
+
         else:
             url_reader_factory = None
 
         if self._trailing_slash_checked:
-            return self._cs(self.prefix + path, mirrors=mirrors,
-                            url_reader=url_reader_factory)
+            return self._cs(
+                self.prefix + path,
+                mirrors=mirrors,
+                url_reader=url_reader_factory,
+            )
 
         # A little hack to fix up the user's path. It's fairly common to
         # specify URLs without a trailing slash, so we try to do that here as
@@ -198,22 +208,31 @@ class UrlMirrorReader(MirrorReader):
         # returned is not yet open (LP: #1237658)
         self._trailing_slash_checked = True
         try:
-            with self._cs(self.prefix + path, mirrors=None,
-                          url_reader=url_reader_factory) as csource:
+            with self._cs(
+                self.prefix + path, mirrors=None, url_reader=url_reader_factory
+            ) as csource:
                 csource.read(1024)
         except Exception as e:
             if isinstance(e, IOError) and (e.errno == errno.ENOENT):
-                LOG.warning("got ENOENT for (%s, %s), trying with trailing /",
-                            self.prefix, path)
-                self.prefix = self.prefix + '/'
+                LOG.warning(
+                    "got ENOENT for (%s, %s), trying with trailing /",
+                    self.prefix,
+                    path,
+                )
+                self.prefix = self.prefix + "/"
             else:
                 # this raised exception, but it was sneaky to do it
                 # so just ignore it.
-                LOG.debug("trailing / check on (%s, %s) resulted in %s",
-                          self.prefix, path, e)
+                LOG.debug(
+                    "trailing / check on (%s, %s) resulted in %s",
+                    self.prefix,
+                    path,
+                    e,
+                )
 
-        return self._cs(self.prefix + path, mirrors=mirrors,
-                        url_reader=url_reader_factory)
+        return self._cs(
+            self.prefix + path, mirrors=mirrors, url_reader=url_reader_factory
+        )
 
 
 class ObjectStoreMirrorReader(MirrorReader):
@@ -231,7 +250,7 @@ class BasicMirrorWriter(MirrorWriter):
         if config is None:
             config = {}
         self.config = config
-        self.checksumming_reader = self.config.get('checksumming_reader', True)
+        self.checksumming_reader = self.config.get("checksumming_reader", True)
 
     def load_products(self, path=None, content_id=None):
         super(BasicMirrorWriter, self).load_products(path, content_id)
@@ -243,14 +262,14 @@ class BasicMirrorWriter(MirrorWriter):
 
         check_tree_paths(src)
 
-        itree = src.get('index')
+        itree = src.get("index")
         for content_id, index_entry in itree.items():
             if not self.filter_index_entry(index_entry, src, (content_id,)):
                 continue
-            epath = index_entry.get('path', None)
+            epath = index_entry.get("path", None)
             mycs = None
             if epath:
-                if index_entry.get('format') in ("index:1.0", "products:1.0"):
+                if index_entry.get("format") in ("index:1.0", "products:1.0"):
                     self.sync(reader, path=epath)
                 mycs = reader.source(epath)
 
@@ -265,36 +284,37 @@ class BasicMirrorWriter(MirrorWriter):
 
         check_tree_paths(src)
 
-        content_id = src['content_id']
+        content_id = src["content_id"]
         target = self.load_products(path, content_id)
         if not target:
             target = util.stringitems(src)
 
         util.expand_tree(target)
 
-        stree = src.get('products', {})
-        if 'products' not in target:
-            target['products'] = {}
+        stree = src.get("products", {})
+        if "products" not in target:
+            target["products"] = {}
 
-        tproducts = target['products']
+        tproducts = target["products"]
 
         filtered_products = []
         prodname = None
 
         # Apply filters to items before filtering versions
         for prodname, product in list(stree.items()):
-
-            for vername, version in list(product.get('versions', {}).items()):
-                for itemname, item in list(version.get('items', {}).items()):
+            for vername, version in list(product.get("versions", {}).items()):
+                for itemname, item in list(version.get("items", {}).items()):
                     pgree = (prodname, vername, itemname)
                     if not self.filter_item(item, src, target, pgree):
                         LOG.debug("Filtered out item: %s/%s", itemname, item)
-                        del stree[prodname]['versions'][vername]['items'][
-                            itemname]
-                        if not stree[prodname]['versions'][vername].get(
-                                'items', {}):
-                            del stree[prodname]['versions'][vername]
-                        if not stree[prodname].get('versions', {}):
+                        del stree[prodname]["versions"][vername]["items"][
+                            itemname
+                        ]
+                        if not stree[prodname]["versions"][vername].get(
+                            "items", {}
+                        ):
+                            del stree[prodname]["versions"][vername]
+                        if not stree[prodname].get("versions", {}):
                             del stree[prodname]
 
         for prodname, product in stree.items():
@@ -305,50 +325,62 @@ class BasicMirrorWriter(MirrorWriter):
             if prodname not in tproducts:
                 tproducts[prodname] = util.stringitems(product)
             tproduct = tproducts[prodname]
-            if 'versions' not in tproduct:
-                tproduct['versions'] = {}
+            if "versions" not in tproduct:
+                tproduct["versions"] = {}
 
             src_filtered_items = []
 
             def _filter(itemkey):
-                ret = self.filter_version(product['versions'][itemkey],
-                                          src, target, (prodname, itemkey))
+                ret = self.filter_version(
+                    product["versions"][itemkey],
+                    src,
+                    target,
+                    (prodname, itemkey),
+                )
                 if not ret:
                     src_filtered_items.append(itemkey)
                 return ret
 
             (to_add, to_remove) = util.resolve_work(
-                src=list(product.get('versions', {}).keys()),
-                target=list(tproduct.get('versions', {}).keys()),
-                maxnum=self.config.get('max_items'),
-                keep=self.config.get('keep_items'), itemfilter=_filter)
-
-            LOG.info("%s/%s: to_add=%s to_remove=%s", content_id, prodname,
-                     to_add, to_remove)
-
-            tversions = tproduct['versions']
+                src=list(product.get("versions", {}).keys()),
+                target=list(tproduct.get("versions", {}).keys()),
+                maxnum=self.config.get("max_items"),
+                keep=self.config.get("keep_items"),
+                itemfilter=_filter,
+            )
+
+            LOG.info(
+                "%s/%s: to_add=%s to_remove=%s",
+                content_id,
+                prodname,
+                to_add,
+                to_remove,
+            )
+
+            tversions = tproduct["versions"]
             skipped_versions = []
             for vername in to_add:
-                version = product['versions'][vername]
+                version = product["versions"][vername]
 
                 if vername not in tversions:
                     tversions[vername] = util.stringitems(version)
 
                 added_items = []
-                for itemname, item in version.get('items', {}).items():
+                for itemname, item in version.get("items", {}).items():
                     pgree = (prodname, vername, itemname)
 
                     added_items.append(itemname)
 
-                    ipath = item.get('path', None)
+                    ipath = item.get("path", None)
                     ipath_cs = None
                     if ipath and reader:
                         if self.checksumming_reader:
                             flat = util.products_exdata(src, pgree)
                             ipath_cs = cs.ChecksummingContentSource(
                                 csrc=reader.source(ipath),
-                                size=flat.get('size'),
-                                checksums=checksum_util.item_checksums(flat))
+                                size=flat.get("size"),
+                                checksums=checksum_util.item_checksums(flat),
+                            )
                         else:
                             ipath_cs = reader.source(ipath)
 
@@ -356,28 +388,38 @@ class BasicMirrorWriter(MirrorWriter):
 
                 if len(added_items):
                     # do not insert versions that had all items filtered
-                    self.insert_version(version, src, target,
-                                        (prodname, vername))
+                    self.insert_version(
+                        version, src, target, (prodname, vername)
+                    )
                 else:
                     skipped_versions.append(vername)
 
             for vername in skipped_versions:
-                if vername in tproduct['versions']:
-                    del tproduct['versions'][vername]
+                if vername in tproduct["versions"]:
+                    del tproduct["versions"][vername]
 
-            if self.config.get('delete_filtered_items', False):
-                tkeys = tproduct.get('versions', {}).keys()
+            if self.config.get("delete_filtered_items", False):
+                tkeys = tproduct.get("versions", {}).keys()
                 for v in src_filtered_items:
                     if v not in to_remove and v in tkeys:
                         to_remove.append(v)
-                LOG.info("After deletions %s/%s: to_add=%s to_remove=%s",
-                         content_id, prodname, to_add, to_remove)
+                LOG.info(
+                    "After deletions %s/%s: to_add=%s to_remove=%s",
+                    content_id,
+                    prodname,
+                    to_add,
+                    to_remove,
+                )
 
             for vername in to_remove:
                 tversion = tversions[vername]
-                for itemname in list(tversion.get('items', {}).keys()):
-                    self.remove_item(tversion['items'][itemname], src, target,
-                                     (prodname, vername, itemname))
+                for itemname in list(tversion.get("items", {}).keys()):
+                    self.remove_item(
+                        tversion["items"][itemname],
+                        src,
+                        target,
+                        (prodname, vername, itemname),
+                    )
 
                 self.remove_version(tversion, src, target, (prodname, vername))
                 del tversions[vername]
@@ -389,12 +431,14 @@ class BasicMirrorWriter(MirrorWriter):
         # that could accidentally delete a lot.
         #
         del_products = []
-        if self.config.get('delete_products', False):
-            del_products.extend([p for p in list(tproducts.keys())
-                                 if p not in stree])
-        if self.config.get('delete_filtered_products', False):
-            del_products.extend([p for p in filtered_products
-                                 if p not in stree])
+        if self.config.get("delete_products", False):
+            del_products.extend(
+                [p for p in list(tproducts.keys()) if p not in stree]
+            )
+        if self.config.get("delete_filtered_products", False):
+            del_products.extend(
+                [p for p in filtered_products if p not in stree]
+            )
 
         for prodname in del_products:
             # FIXME: we remove a product here, but unless that acts
@@ -421,7 +465,7 @@ class ObjectStoreMirrorWriter(BasicMirrorWriter):
         try:
             with self.source(self._reference_count_data_path()) as source:
                 raw = source.read()
-            return json.load(io.StringIO(raw.decode('utf-8')))
+            return json.load(io.StringIO(raw.decode("utf-8")))
         except IOError as e:
             if e.errno == errno.ENOENT:
                 return {}
@@ -432,7 +476,7 @@ class ObjectStoreMirrorWriter(BasicMirrorWriter):
         self.store.insert(self._reference_count_data_path(), source)
 
     def _build_rc_id(self, src, pedigree):
-        return '/'.join([src['content_id']] + list(pedigree))
+        return "/".join([src["content_id"]] + list(pedigree))
 
     def _inc_rc(self, path, src, pedigree):
         rc = self._load_rc_dict()
@@ -482,25 +526,30 @@ class ObjectStoreMirrorWriter(BasicMirrorWriter):
 
     def insert_item(self, data, src, target, pedigree, contentsource):
         util.products_set(target, data, pedigree)
-        if 'path' not in data:
+        if "path" not in data:
             return
-        if not self.config.get('item_download', True):
+        if not self.config.get("item_download", True):
             return
-        LOG.debug("inserting %s to %s", contentsource.url, data['path'])
-        self.store.insert(data['path'], contentsource,
-                          checksums=checksum_util.item_checksums(data),
-                          mutable=False, size=data.get('size'))
-        self._inc_rc(data['path'], src, pedigree)
+        LOG.debug("inserting %s to %s", contentsource.url, data["path"])
+        self.store.insert(
+            data["path"],
+            contentsource,
+            checksums=checksum_util.item_checksums(data),
+            mutable=False,
+            size=data.get("size"),
+        )
+        self._inc_rc(data["path"], src, pedigree)
 
     def insert_index_entry(self, data, src, pedigree, contentsource):
-        epath = data.get('path', None)
+        epath = data.get("path", None)
         if not epath:
             return
-        self.store.insert(epath, contentsource,
-                          checksums=checksum_util.item_checksums(data))
+        self.store.insert(
+            epath, contentsource, checksums=checksum_util.item_checksums(data)
+        )
 
     def insert_products(self, path, target, content):
-        dpath = self.products_data_path(target['content_id'])
+        dpath = self.products_data_path(target["content_id"])
         self.store.insert_content(dpath, util.dump_data(target))
         if not path:
             return
@@ -517,16 +566,16 @@ class ObjectStoreMirrorWriter(BasicMirrorWriter):
 
     def remove_item(self, data, src, target, pedigree):
         util.products_del(target, pedigree)
-        if 'path' not in data:
+        if "path" not in data:
             return
-        if self._dec_rc(data['path'], src, pedigree):
-            self.store.remove(data['path'])
+        if self._dec_rc(data["path"], src, pedigree):
+            self.store.remove(data["path"])
 
 
 class ObjectFilterMirror(ObjectStoreMirrorWriter):
     def __init__(self, *args, **kwargs):
         super(ObjectFilterMirror, self).__init__(*args, **kwargs)
-        self.filters = self.config.get('filters', [])
+        self.filters = self.config.get("filters", [])
 
     def filter_item(self, data, src, target, pedigree):
         return filters.filter_item(self.filters, data, src, pedigree)
@@ -552,15 +601,15 @@ class DryRunMirrorWriter(ObjectFilterMirror):
 
     def insert_item(self, data, src, target, pedigree, contentsource):
         data = util.products_exdata(src, pedigree)
-        if 'size' in data and 'path' in data:
+        if "size" in data and "path" in data:
             self.downloading.append(
-                (pedigree, data['path'], int(data['size'])))
+                (pedigree, data["path"], int(data["size"]))
+            )
 
     def remove_item(self, data, src, target, pedigree):
         data = util.products_exdata(src, pedigree)
-        if 'size' in data and 'path' in data:
-            self.removing.append(
-                (pedigree, data['path'], int(data['size'])))
+        if "size" in data and "path" in data:
+            self.removing.append((pedigree, data["path"], int(data["size"])))
 
     @property
     def size(self):
@@ -573,27 +622,31 @@ def _get_data_content(path, data, content, reader):
     if content is None and path:
         _, content = reader.read(path)
         if isinstance(content, bytes):
-            content = content.decode('utf-8')
+            content = content.decode("utf-8")
 
     if data is None and content:
         data = util.load_content(content)
 
     if not data:
-        raise ValueError("Data could not be loaded. "
-                         "Path or content is required")
+        raise ValueError(
+            "Data could not be loaded. " "Path or content is required"
+        )
     return (data, content)
 
 
 def check_tree_paths(tree, fmt=None):
     if fmt is None:
-        fmt = tree.get('format')
+        fmt = tree.get("format")
     if fmt == "products:1.0":
+
         def check_path(item, tree, pedigree):
-            util.assert_safe_path(item.get('path'))
+            util.assert_safe_path(item.get("path"))
+
         util.walk_products(tree, cb_item=check_path)
     elif fmt == "index:1.0":
-        index = tree.get('index')
+        index = tree.get("index")
         for content_id in index:
-            util.assert_safe_path(index[content_id].get('path'))
+            util.assert_safe_path(index[content_id].get("path"))
+
 
 # vi: ts=4 expandtab
diff --git a/simplestreams/mirrors/command_hook.py b/simplestreams/mirrors/command_hook.py
index ab70691..de42623 100644
--- a/simplestreams/mirrors/command_hook.py
+++ b/simplestreams/mirrors/command_hook.py
@@ -15,15 +15,15 @@
 #   You should have received a copy of the GNU Affero General Public License
 #   along with Simplestreams.  If not, see <http://www.gnu.org/licenses/>.
 
-import simplestreams.mirrors as mirrors
-import simplestreams.util as util
-
-import os
 import errno
+import os
 import signal
 import subprocess
 import tempfile
 
+import simplestreams.mirrors as mirrors
+import simplestreams.util as util
+
 REQUIRED_FIELDS = ("load_products",)
 HOOK_NAMES = (
     "filter_index_entry",
@@ -92,6 +92,7 @@ class CommandHookMirror(mirrors.BasicMirrorWriter):
       If the configuration setting 'item_skip_download' is set to True, then
       'path_url' will be set instead to a url where the item can be found.
     """
+
     def __init__(self, config):
         if isinstance(config, str):
             config = util.load_content(config)
@@ -100,32 +101,34 @@ class CommandHookMirror(mirrors.BasicMirrorWriter):
         super(CommandHookMirror, self).__init__(config=config)
 
     def load_products(self, path=None, content_id=None):
-        (_rc, output) = self.call_hook('load_products',
-                                       data={'content_id': content_id},
-                                       capture=True)
+        (_rc, output) = self.call_hook(
+            "load_products", data={"content_id": content_id}, capture=True
+        )
         fmt = self.config.get("product_load_output_format", "serial_list")
 
-        loaded = load_product_output(output=output, content_id=content_id,
-                                     fmt=fmt)
+        loaded = load_product_output(
+            output=output, content_id=content_id, fmt=fmt
+        )
         return loaded
 
     def filter_index_entry(self, data, src, pedigree):
         mdata = util.stringitems(src)
-        mdata['content_id'] = pedigree[0]
+        mdata["content_id"] = pedigree[0]
         mdata.update(util.stringitems(data))
 
-        (ret, _output) = self.call_hook('filter_index_entry', data=mdata,
-                                        rcs=[0, 1])
+        (ret, _output) = self.call_hook(
+            "filter_index_entry", data=mdata, rcs=[0, 1]
+        )
         return ret == 0
 
     def filter_product(self, data, src, target, pedigree):
-        return self._call_filter('filter_product', src, pedigree)
+        return self._call_filter("filter_product", src, pedigree)
 
     def filter_version(self, data, src, target, pedigree):
-        return self._call_filter('filter_version', src, pedigree)
+        return self._call_filter("filter_version", src, pedigree)
 
     def filter_item(self, data, src, target, pedigree):
-        return self._call_filter('filter_item', src, pedigree)
+        return self._call_filter("filter_item", src, pedigree)
 
     def _call_filter(self, name, src, pedigree):
         data = util.products_exdata(src, pedigree)
@@ -133,20 +136,27 @@ class CommandHookMirror(mirrors.BasicMirrorWriter):
         return ret == 0
 
     def insert_index(self, path, src, content):
-        return self.call_hook('insert_index', data=src, content=content,
-                              extra={'path': path})
+        return self.call_hook(
+            "insert_index", data=src, content=content, extra={"path": path}
+        )
 
     def insert_products(self, path, target, content):
-        return self.call_hook('insert_products', data=target,
-                              content=content, extra={'path': path})
+        return self.call_hook(
+            "insert_products",
+            data=target,
+            content=content,
+            extra={"path": path},
+        )
 
     def insert_product(self, data, src, target, pedigree):
-        return self.call_hook('insert_product',
-                              data=util.products_exdata(src, pedigree))
+        return self.call_hook(
+            "insert_product", data=util.products_exdata(src, pedigree)
+        )
 
     def insert_version(self, data, src, target, pedigree):
-        return self.call_hook('insert_version',
-                              data=util.products_exdata(src, pedigree))
+        return self.call_hook(
+            "insert_version", data=util.products_exdata(src, pedigree)
+        )
 
     def insert_item(self, data, src, target, pedigree, contentsource):
         mdata = util.products_exdata(src, pedigree)
@@ -154,43 +164,47 @@ class CommandHookMirror(mirrors.BasicMirrorWriter):
         tmp_path = None
         tmp_del = None
         extra = {}
-        if 'path' in data:
-            extra.update({'item_url': contentsource.url})
-            if not self.config.get('item_skip_download', False):
+        if "path" in data:
+            extra.update({"item_url": contentsource.url})
+            if not self.config.get("item_skip_download", False):
                 try:
                     (tmp_path, tmp_del) = util.get_local_copy(contentsource)
-                    extra['path_local'] = tmp_path
+                    extra["path_local"] = tmp_path
                 finally:
                     contentsource.close()
 
         try:
-            ret = self.call_hook('insert_item', data=mdata, extra=extra)
+            ret = self.call_hook("insert_item", data=mdata, extra=extra)
         finally:
             if tmp_del and os.path.exists(tmp_path):
                 os.unlink(tmp_path)
         return ret
 
     def remove_product(self, data, src, target, pedigree):
-        return self.call_hook('remove_product',
-                              data=util.products_exdata(src, pedigree))
+        return self.call_hook(
+            "remove_product", data=util.products_exdata(src, pedigree)
+        )
 
     def remove_version(self, data, src, target, pedigree):
-        return self.call_hook('remove_version',
-                              data=util.products_exdata(src, pedigree))
+        return self.call_hook(
+            "remove_version", data=util.products_exdata(src, pedigree)
+        )
 
     def remove_item(self, data, src, target, pedigree):
-        return self.call_hook('remove_item',
-                              data=util.products_exdata(target, pedigree))
+        return self.call_hook(
+            "remove_item", data=util.products_exdata(target, pedigree)
+        )
 
-    def call_hook(self, hookname, data, capture=False, rcs=None, extra=None,
-                  content=None):
+    def call_hook(
+        self, hookname, data, capture=False, rcs=None, extra=None, content=None
+    ):
         command = self.config.get(hookname, self.config.get(DEFAULT_HOOK_NAME))
         if not command:
             # return successful execution with no output
-            return (0, '')
+            return (0, "")
 
         if isinstance(command, str):
-            command = ['sh', '-c', command]
+            command = ["sh", "-c", command]
 
         fdata = util.stringitems(data)
 
@@ -200,16 +214,20 @@ class CommandHookMirror(mirrors.BasicMirrorWriter):
             tfile = os.fdopen(tfd, "w")
             tfile.write(content)
             tfile.close()
-            fdata['content_file_path'] = content_file
+            fdata["content_file_path"] = content_file
 
         if extra:
             fdata.update(extra)
-        fdata['HOOK'] = hookname
+        fdata["HOOK"] = hookname
 
         try:
-            return call_hook(command=command, data=fdata,
-                             unset=self.config.get('unset_value', None),
-                             capture=capture, rcs=rcs)
+            return call_hook(
+                command=command,
+                data=fdata,
+                unset=self.config.get("unset_value", None),
+                capture=capture,
+                rcs=rcs,
+            )
         finally:
             if content_file:
                 os.unlink(content_file)
@@ -219,7 +237,7 @@ def call_hook(command, data, unset=None, capture=False, rcs=None):
     env = os.environ.copy()
     data = data.copy()
 
-    data[ENV_FIELDS_NAME] = ' '.join([k for k in data if k != ENV_HOOK_NAME])
+    data[ENV_FIELDS_NAME] = " ".join([k for k in data if k != ENV_HOOK_NAME])
 
     mcommand = render(command, data, unset=unset)
 
@@ -257,12 +275,12 @@ def load_product_output(output, content_id, fmt="serial_list"):
 
     if fmt == "serial_list":
         # "line" format just is a list of serials that are present
-        working = {'content_id': content_id, 'products': {}}
+        working = {"content_id": content_id, "products": {}}
         for line in output.splitlines():
             (product_id, version) = line.split(None, 1)
-            if product_id not in working['products']:
-                working['products'][product_id] = {'versions': {}}
-            working['products'][product_id]['versions'][version] = {}
+            if product_id not in working["products"]:
+                working["products"][product_id] = {"versions": {}}
+            working["products"][product_id]["versions"][version] = {}
         return working
 
     elif fmt == "json":
@@ -293,10 +311,11 @@ def run_command(cmd, env=None, capture=False, rcs=None):
         raise subprocess.CalledProcessError(rc, cmd)
 
     if out is None:
-        out = ''
+        out = ""
     elif isinstance(out, bytes):
-        out = out.decode('utf-8')
+        out = out.decode("utf-8")
 
     return (rc, out)
 
+
 # vi: ts=4 expandtab syntax=python
diff --git a/simplestreams/mirrors/glance.py b/simplestreams/mirrors/glance.py
index b96f8eb..22e46e9 100644
--- a/simplestreams/mirrors/glance.py
+++ b/simplestreams/mirrors/glance.py
@@ -15,42 +15,47 @@
 #   You should have received a copy of the GNU Affero General Public License
 #   along with Simplestreams.  If not, see <http://www.gnu.org/licenses/>.
 
-import simplestreams.filters as filters
-import simplestreams.mirrors as mirrors
-import simplestreams.util as util
-from simplestreams import checksum_util
-import simplestreams.openstack as openstack
-from simplestreams.log import LOG
-
-import copy
 import collections
+import copy
 import errno
-import glanceclient
 import json
 import os
 import re
 
+import glanceclient
+
+import simplestreams.filters as filters
+import simplestreams.mirrors as mirrors
+import simplestreams.openstack as openstack
+import simplestreams.util as util
+from simplestreams import checksum_util
+from simplestreams.log import LOG
+
 
-def get_glanceclient(version='1', **kwargs):
+def get_glanceclient(version="1", **kwargs):
     # newer versions of the glanceclient will do this 'strip_version' for
     # us, but older versions do not.
-    kwargs['endpoint'] = _strip_version(kwargs['endpoint'])
-    pt = ('endpoint', 'token', 'insecure', 'cacert')
+    kwargs["endpoint"] = _strip_version(kwargs["endpoint"])
+    pt = ("endpoint", "token", "insecure", "cacert")
     kskw = {k: kwargs.get(k) for k in pt if k in kwargs}
-    if kwargs.get('session'):
-        sess = kwargs.get('session')
+    if kwargs.get("session"):
+        sess = kwargs.get("session")
         return glanceclient.Client(version, session=sess)
     else:
         return glanceclient.Client(version, **kskw)
 
 
 def empty_iid_products(content_id):
-    return {'content_id': content_id, 'products': {},
-            'datatype': 'image-ids', 'format': 'products:1.0'}
+    return {
+        "content_id": content_id,
+        "products": {},
+        "datatype": "image-ids",
+        "format": "products:1.0",
+    }
 
 
 def canonicalize_arch(arch):
-    '''Canonicalize Ubuntu archs for use in OpenStack'''
+    """Canonicalize Ubuntu archs for use in OpenStack"""
     newarch = arch.lower()
     if newarch == "amd64":
         newarch = "x86_64"
@@ -68,21 +73,21 @@ def canonicalize_arch(arch):
 
 
 LXC_FTYPES = {
-    'root.tar.gz': 'root-tar',
-    'root.tar.xz': 'root-tar',
-    'squashfs': 'squashfs',
+    "root.tar.gz": "root-tar",
+    "root.tar.xz": "root-tar",
+    "squashfs": "squashfs",
 }
 
 QEMU_FTYPES = {
-    'disk.img': 'qcow2',
-    'disk1.img': 'qcow2',
+    "disk.img": "qcow2",
+    "disk1.img": "qcow2",
 }
 
 
 def disk_format(ftype):
-    '''Canonicalize disk formats for use in OpenStack.
+    """Canonicalize disk formats for use in OpenStack.
     Input ftype is a 'ftype' from a simplestream feed.
-    Return value is the appropriate 'disk_format' for glance.'''
+    Return value is the appropriate 'disk_format' for glance."""
     newftype = ftype.lower()
     if newftype in LXC_FTYPES:
         return LXC_FTYPES[newftype]
@@ -92,22 +97,22 @@ def disk_format(ftype):
 
 
 def hypervisor_type(ftype):
-    '''Determine hypervisor type based on image format'''
+    """Determine hypervisor type based on image format"""
     newftype = ftype.lower()
     if newftype in LXC_FTYPES:
-        return 'lxc'
+        return "lxc"
     if newftype in QEMU_FTYPES:
-        return 'qemu'
+        return "qemu"
     return None
 
 
 def virt_type(hypervisor_type):
-    '''Map underlying hypervisor types into high level virt types'''
+    """Map underlying hypervisor types into high level virt types"""
     newhtype = hypervisor_type.lower()
-    if newhtype == 'qemu':
-        return 'kvm'
-    if newhtype == 'lxc':
-        return 'lxd'
+    if newhtype == "qemu":
+        return "kvm"
+    if newhtype == "lxc":
+        return "lxd"
     return None
 
 
@@ -120,20 +125,29 @@ class GlanceMirror(mirrors.BasicMirrorWriter):
     `client` argument is used for testing to override openstack module:
     allows dependency injection of fake "openstack" module.
     """
-    def __init__(self, config, objectstore=None, region=None,
-                 name_prefix=None, progress_callback=None,
-                 client=None):
+
+    def __init__(
+        self,
+        config,
+        objectstore=None,
+        region=None,
+        name_prefix=None,
+        progress_callback=None,
+        client=None,
+    ):
         super(GlanceMirror, self).__init__(config=config)
 
-        self.item_filters = self.config.get('item_filters', [])
+        self.item_filters = self.config.get("item_filters", [])
         if len(self.item_filters) == 0:
-            self.item_filters = ['ftype~(disk1.img|disk.img)',
-                                 'arch~(x86_64|amd64|i386)']
+            self.item_filters = [
+                "ftype~(disk1.img|disk.img)",
+                "arch~(x86_64|amd64|i386)",
+            ]
         self.item_filters = filters.get_filters(self.item_filters)
 
-        self.index_filters = self.config.get('index_filters', [])
+        self.index_filters = self.config.get("index_filters", [])
         if len(self.index_filters) == 0:
-            self.index_filters = ['datatype=image-downloads']
+            self.index_filters = ["datatype=image-downloads"]
         self.index_filters = filters.get_filters(self.index_filters)
 
         self.loaded_content = {}
@@ -146,21 +160,28 @@ class GlanceMirror(mirrors.BasicMirrorWriter):
 
         self.name_prefix = name_prefix or ""
         if region is not None:
-            self.keystone_creds['region_name'] = region
+            self.keystone_creds["region_name"] = region
 
         self.progress_callback = progress_callback
 
         conn_info = client.get_service_conn_info(
-            'image', **self.keystone_creds)
-        self.glance_api_version = conn_info['glance_version']
-        self.gclient = get_glanceclient(version=self.glance_api_version,
-                                        **conn_info)
-        self.tenant_id = conn_info['tenant_id']
-
-        self.region = self.keystone_creds.get('region_name', 'nullregion')
-        self.cloudname = config.get("cloud_name", 'nullcloud')
-        self.crsn = '-'.join((self.cloudname, self.region,))
-        self.auth_url = self.keystone_creds['auth_url']
+            "image", **self.keystone_creds
+        )
+        self.glance_api_version = conn_info["glance_version"]
+        self.gclient = get_glanceclient(
+            version=self.glance_api_version, **conn_info
+        )
+        self.tenant_id = conn_info["tenant_id"]
+
+        self.region = self.keystone_creds.get("region_name", "nullregion")
+        self.cloudname = config.get("cloud_name", "nullcloud")
+        self.crsn = "-".join(
+            (
+                self.cloudname,
+                self.region,
+            )
+        )
+        self.auth_url = self.keystone_creds["auth_url"]
 
         self.content_id = config.get("content_id")
         self.modify_hook = config.get("modify_hook")
@@ -170,7 +191,7 @@ class GlanceMirror(mirrors.BasicMirrorWriter):
             raise TypeError("content_id is required")
 
         self.custom_properties = collections.OrderedDict(
-            prop.split('=') for prop in config.get("custom_properties", [])
+            prop.split("=") for prop in config.get("custom_properties", [])
         )
         self.visibility = config.get("visibility", "public")
 
@@ -208,82 +229,100 @@ class GlanceMirror(mirrors.BasicMirrorWriter):
         for image in images:
             if self.glance_api_version == "1":
                 image = image.to_dict()
-                props = image['properties']
+                props = image["properties"]
             else:
                 props = copy.deepcopy(image)
 
-            if image['owner'] != self.tenant_id:
+            if image["owner"] != self.tenant_id:
                 continue
 
-            if props.get('content_id') != my_cid:
+            if props.get("content_id") != my_cid:
                 continue
 
-            if image.get('status') != "active":
-                LOG.warning("Ignoring inactive image %s with status '%s'" % (
-                    image['id'], image.get('status')))
+            if image.get("status") != "active":
+                LOG.warning(
+                    "Ignoring inactive image %s with status '%s'"
+                    % (image["id"], image.get("status"))
+                )
                 continue
 
-            source_content_id = props.get('source_content_id')
+            source_content_id = props.get("source_content_id")
 
-            product = props.get('product_name')
-            version = props.get('version_name')
-            item = props.get('item_name')
+            product = props.get("product_name")
+            version = props.get("version_name")
+            item = props.get("item_name")
             if not (version and product and item and source_content_id):
-                LOG.warning("%s missing required fields" % image['id'])
+                LOG.warning("%s missing required fields" % image["id"])
                 continue
 
             # get data from the datastore for this item, if it exists
             # and then update that with glance data (just in case different)
             try:
-                item_data = util.products_exdata(store_t,
-                                                 (product, version, item,),
-                                                 include_top=False,
-                                                 insert_fieldnames=False)
+                item_data = util.products_exdata(
+                    store_t,
+                    (
+                        product,
+                        version,
+                        item,
+                    ),
+                    include_top=False,
+                    insert_fieldnames=False,
+                )
             except KeyError:
                 item_data = {}
 
             # If original simplestreams-metadata is stored on the image,
             # use that as well.
-            if 'simplestreams_metadata' in props:
+            if "simplestreams_metadata" in props:
                 simplestreams_metadata = json.loads(
-                    props.get('simplestreams_metadata'))
+                    props.get("simplestreams_metadata")
+                )
             else:
                 simplestreams_metadata = {}
             item_data.update(simplestreams_metadata)
 
-            item_data.update({'name': image['name'], 'id': image['id']})
-            if 'owner_id' not in item_data:
-                item_data['owner_id'] = self.tenant_id
-
-            util.products_set(glance_t, item_data,
-                              (product, version, item,))
+            item_data.update({"name": image["name"], "id": image["id"]})
+            if "owner_id" not in item_data:
+                item_data["owner_id"] = self.tenant_id
+
+            util.products_set(
+                glance_t,
+                item_data,
+                (
+                    product,
+                    version,
+                    item,
+                ),
+            )
 
-        for product in glance_t['products']:
-            glance_t['products'][product]['region'] = self.region
-            glance_t['products'][product]['endpoint'] = self.auth_url
+        for product in glance_t["products"]:
+            glance_t["products"][product]["region"] = self.region
+            glance_t["products"][product]["endpoint"] = self.auth_url
 
         return glance_t
 
     def filter_item(self, data, src, target, pedigree):
         return filters.filter_item(self.item_filters, data, src, pedigree)
 
-    def create_glance_properties(self, content_id, source_content_id,
-                                 image_metadata, hypervisor_mapping):
+    def create_glance_properties(
+        self, content_id, source_content_id, image_metadata, hypervisor_mapping
+    ):
         """
         Construct extra properties to store in Glance for an image.
 
         Based on source image metadata.
         """
         properties = {
-            'content_id': content_id,
-            'source_content_id': source_content_id,
+            "content_id": content_id,
+            "source_content_id": source_content_id,
         }
         # An iterator of properties to carry over: if a property needs
         # renaming, uses a tuple (old name, new name).
-        carry_over_simple = (
-            'product_name', 'version_name', 'item_name')
+        carry_over_simple = ("product_name", "version_name", "item_name")
         carry_over = carry_over_simple + (
-            ('os', 'os_distro'), ('version', 'os_version'))
+            ("os", "os_distro"),
+            ("version", "os_version"),
+        )
         for carry_over_property in carry_over:
             if isinstance(carry_over_property, tuple):
                 name_old, name_new = carry_over_property
@@ -291,33 +330,41 @@ class GlanceMirror(mirrors.BasicMirrorWriter):
                 name_old = name_new = carry_over_property
             properties[name_new] = image_metadata.get(name_old)
 
-        if 'arch' in image_metadata:
-            properties['architecture'] = canonicalize_arch(
-                image_metadata['arch'])
+        if "arch" in image_metadata:
+            properties["architecture"] = canonicalize_arch(
+                image_metadata["arch"]
+            )
 
-        if hypervisor_mapping and 'ftype' in image_metadata:
-            _hypervisor_type = hypervisor_type(image_metadata['ftype'])
+        if hypervisor_mapping and "ftype" in image_metadata:
+            _hypervisor_type = hypervisor_type(image_metadata["ftype"])
             if _hypervisor_type:
-                properties['hypervisor_type'] = _hypervisor_type
+                properties["hypervisor_type"] = _hypervisor_type
 
         properties.update(self.custom_properties)
 
         if self.set_latest_property:
-            properties['latest'] = "true"
+            properties["latest"] = "true"
 
         # Store flattened metadata for a source image along with the
         # image in 'simplestreams_metadata' property.
         simplestreams_metadata = image_metadata.copy()
-        drop_keys = carry_over_simple + ('path',)
+        drop_keys = carry_over_simple + ("path",)
         for remove_key in drop_keys:
             if remove_key in simplestreams_metadata:
                 del simplestreams_metadata[remove_key]
-        properties['simplestreams_metadata'] = json.dumps(
-            simplestreams_metadata, sort_keys=True)
+        properties["simplestreams_metadata"] = json.dumps(
+            simplestreams_metadata, sort_keys=True
+        )
         return properties
 
-    def prepare_glance_arguments(self, full_image_name, image_metadata,
-                                 image_md5_hash, image_size, image_properties):
+    def prepare_glance_arguments(
+        self,
+        full_image_name,
+        image_metadata,
+        image_md5_hash,
+        image_size,
+        image_properties,
+    ):
         """
         Prepare arguments to pass into Glance image creation method.
 
@@ -334,37 +381,39 @@ class GlanceMirror(mirrors.BasicMirrorWriter):
         GlanceClient.images.create().
         """
         create_kwargs = {
-            'name': full_image_name,
-            'container_format': 'bare',
-            'is_public': self.visibility == 'public',
-            'properties': image_properties,
+            "name": full_image_name,
+            "container_format": "bare",
+            "is_public": self.visibility == "public",
+            "properties": image_properties,
         }
 
         # In v2 is_public=True is visibility='public'
         if self.glance_api_version == "2":
-            del create_kwargs['is_public']
-            create_kwargs['visibility'] = self.visibility
+            del create_kwargs["is_public"]
+            create_kwargs["visibility"] = self.visibility
 
         # v2 automatically calculates size and checksum
         if self.glance_api_version == "1":
-            if 'size' in image_metadata:
-                create_kwargs['size'] = int(image_metadata.get('size'))
-            if 'md5' in image_metadata:
-                create_kwargs['checksum'] = image_metadata.get('md5')
+            if "size" in image_metadata:
+                create_kwargs["size"] = int(image_metadata.get("size"))
+            if "md5" in image_metadata:
+                create_kwargs["checksum"] = image_metadata.get("md5")
             if image_md5_hash and image_size:
-                create_kwargs.update({
-                    'checksum': image_md5_hash,
-                    'size': image_size,
-                })
+                create_kwargs.update(
+                    {
+                        "checksum": image_md5_hash,
+                        "size": image_size,
+                    }
+                )
 
         if self.image_import_conversion:
-            create_kwargs['disk_format'] = 'raw'
-        elif 'ftype' in image_metadata:
-            create_kwargs['disk_format'] = (
-                disk_format(image_metadata['ftype']) or 'qcow2'
+            create_kwargs["disk_format"] = "raw"
+        elif "ftype" in image_metadata:
+            create_kwargs["disk_format"] = (
+                disk_format(image_metadata["ftype"]) or "qcow2"
             )
         else:
-            create_kwargs['disk_format'] = 'qcow2'
+            create_kwargs["disk_format"] = "qcow2"
 
         return create_kwargs
 
@@ -378,37 +427,51 @@ class GlanceMirror(mirrors.BasicMirrorWriter):
         Returns a tuple of
            (str(local-image-path), int(image-size), str(image-md5-hash)).
         """
-        image_name = image_stream_data.get('pubname')
-        image_size = image_stream_data.get('size')
+        image_name = image_stream_data.get("pubname")
+        image_size = image_stream_data.get("size")
 
         if self.progress_callback:
+
             def progress_wrapper(written):
                 self.progress_callback(
-                    dict(status="Downloading", name=image_name,
-                         size=None if image_size is None else int(image_size),
-                         written=written))
+                    dict(
+                        status="Downloading",
+                        name=image_name,
+                        size=None if image_size is None else int(image_size),
+                        written=written,
+                    )
+                )
+
         else:
+
             def progress_wrapper(written):
                 pass
 
         try:
             tmp_path, _ = util.get_local_copy(
-                contentsource, progress_callback=progress_wrapper)
+                contentsource, progress_callback=progress_wrapper
+            )
 
             if self.modify_hook:
                 (new_size, new_md5) = call_hook(
-                    item=image_stream_data, path=tmp_path,
-                    cmd=self.modify_hook)
+                    item=image_stream_data, path=tmp_path, cmd=self.modify_hook
+                )
             else:
                 new_size = os.path.getsize(tmp_path)
-                new_md5 = image_stream_data.get('md5')
+                new_md5 = image_stream_data.get("md5")
         finally:
             contentsource.close()
 
         return tmp_path, new_size, new_md5
 
-    def adapt_source_entry(self, source_entry, hypervisor_mapping, image_name,
-                           image_md5_hash, image_size):
+    def adapt_source_entry(
+        self,
+        source_entry,
+        hypervisor_mapping,
+        image_name,
+        image_md5_hash,
+        image_size,
+    ):
         """
         Adapts the source simplestreams dict `source_entry` for use in the
         generated local simplestreams index.
@@ -416,26 +479,30 @@ class GlanceMirror(mirrors.BasicMirrorWriter):
         output_entry = source_entry.copy()
 
         # Drop attributes not needed for the simplestreams index itself.
-        for property_name in ('path', 'product_name', 'version_name',
-                              'item_name'):
+        for property_name in (
+            "path",
+            "product_name",
+            "version_name",
+            "item_name",
+        ):
             if property_name in output_entry:
                 del output_entry[property_name]
 
-        if hypervisor_mapping and 'ftype' in output_entry:
-            _hypervisor_type = hypervisor_type(output_entry['ftype'])
+        if hypervisor_mapping and "ftype" in output_entry:
+            _hypervisor_type = hypervisor_type(output_entry["ftype"])
             if _hypervisor_type:
                 _virt_type = virt_type(_hypervisor_type)
                 if _virt_type:
-                    output_entry['virt'] = _virt_type
+                    output_entry["virt"] = _virt_type
 
-        output_entry['region'] = self.region
-        output_entry['endpoint'] = self.auth_url
-        output_entry['owner_id'] = self.tenant_id
+        output_entry["region"] = self.region
+        output_entry["endpoint"] = self.auth_url
+        output_entry["owner_id"] = self.tenant_id
 
-        output_entry['name'] = image_name
+        output_entry["name"] = image_name
         if image_md5_hash and image_size:
-            output_entry['md5'] = image_md5_hash
-            output_entry['size'] = str(image_size)
+            output_entry["md5"] = image_md5_hash
+            output_entry["size"] = str(image_size)
 
         return output_entry
 
@@ -459,58 +526,74 @@ class GlanceMirror(mirrors.BasicMirrorWriter):
         #   (product-name, version-name, image-type)
         # from the tuple `pedigree` in the source simplestreams index.
         flattened_img_data = util.products_exdata(
-            src, pedigree, include_top=False)
+            src, pedigree, include_top=False
+        )
 
         tmp_path = None
 
         full_image_name = "{}{}".format(
             self.name_prefix,
-            flattened_img_data.get('pubname', flattened_img_data.get('name')))
-        if not full_image_name.endswith(flattened_img_data['item_name']):
-            full_image_name += "-{}".format(flattened_img_data['item_name'])
+            flattened_img_data.get("pubname", flattened_img_data.get("name")),
+        )
+        if not full_image_name.endswith(flattened_img_data["item_name"]):
+            full_image_name += "-{}".format(flattened_img_data["item_name"])
 
         # Download images locally into a temporary file.
         tmp_path, new_size, new_md5 = self.download_image(
-            contentsource, flattened_img_data)
+            contentsource, flattened_img_data
+        )
 
-        hypervisor_mapping = self.config.get('hypervisor_mapping', False)
+        hypervisor_mapping = self.config.get("hypervisor_mapping", False)
 
         glance_props = self.create_glance_properties(
-            target['content_id'], src['content_id'], flattened_img_data,
-            hypervisor_mapping)
+            target["content_id"],
+            src["content_id"],
+            flattened_img_data,
+            hypervisor_mapping,
+        )
         LOG.debug("glance properties %s", glance_props)
         create_kwargs = self.prepare_glance_arguments(
-            full_image_name, flattened_img_data, new_md5, new_size,
-            glance_props)
+            full_image_name,
+            flattened_img_data,
+            new_md5,
+            new_size,
+            glance_props,
+        )
 
         target_sstream_item = self.adapt_source_entry(
-            flattened_img_data, hypervisor_mapping, full_image_name, new_md5,
-            new_size)
+            flattened_img_data,
+            hypervisor_mapping,
+            full_image_name,
+            new_md5,
+            new_size,
+        )
 
         try:
             if self.glance_api_version == "1":
                 # Set data as string if v1
-                create_kwargs['data'] = open(tmp_path, 'rb')
+                create_kwargs["data"] = open(tmp_path, "rb")
             else:
                 # Keep properties for v2 update call
-                _properties = create_kwargs['properties']
-                del create_kwargs['properties']
+                _properties = create_kwargs["properties"]
+                del create_kwargs["properties"]
 
             LOG.debug("glance create_kwargs %s", create_kwargs)
             glance_image = self.gclient.images.create(**create_kwargs)
-            target_sstream_item['id'] = glance_image.id
+            target_sstream_item["id"] = glance_image.id
 
             if self.glance_api_version == "2":
                 if self.image_import_conversion:
                     # Stage the image before starting import
-                    self.gclient.images.stage(glance_image.id,
-                                              open(tmp_path, 'rb'))
+                    self.gclient.images.stage(
+                        glance_image.id, open(tmp_path, "rb")
+                    )
                     # Import the Glance image
                     self.gclient.images.image_import(glance_image.id)
                 else:
                     # Upload for v2
-                    self.gclient.images.upload(glance_image.id,
-                                               open(tmp_path, 'rb'))
+                    self.gclient.images.upload(
+                        glance_image.id, open(tmp_path, "rb")
+                    )
                 # Update properties for v2
                 self.gclient.images.update(glance_image.id, **_properties)
 
@@ -527,15 +610,19 @@ class GlanceMirror(mirrors.BasicMirrorWriter):
             # self.load_products() instead
             if self.set_latest_property:
                 # Search all images with the same target attributtes
-                _filter_properties = {'filters': {
-                    'latest': 'true',
-                    'os_version': glance_props['os_version'],
-                    'architecture': glance_props['architecture']}}
+                _filter_properties = {
+                    "filters": {
+                        "latest": "true",
+                        "os_version": glance_props["os_version"],
+                        "architecture": glance_props["architecture"],
+                    }
+                }
                 images = self.gclient.images.list(**_filter_properties)
                 for image in images:
                     if image.id != glance_image.id:
-                        self.gclient.images.update(image.id,
-                                                   remove_props=['latest'])
+                        self.gclient.images.update(
+                            image.id, remove_props=["latest"]
+                        )
 
         finally:
             if tmp_path and os.path.exists(tmp_path):
@@ -562,10 +649,10 @@ class GlanceMirror(mirrors.BasicMirrorWriter):
         found = self.gclient.images.get(image_id)
         if found.size == size and found.checksum == checksum:
             return
-        msg = (
-            ("Invalid glance image: %s. " % image_id) +
-            ("Expected size=%s md5=%s. Found size=%s md5=%s." %
-             (size, checksum, found.size, found.checksum)))
+        msg = ("Invalid glance image: %s. " % image_id) + (
+            "Expected size=%s md5=%s. Found size=%s md5=%s."
+            % (size, checksum, found.size, found.checksum)
+        )
         if delete:
             LOG.warning("Deleting image %s: %s", image_id, msg)
             self.gclient.images.delete(image_id)
@@ -587,13 +674,15 @@ class GlanceMirror(mirrors.BasicMirrorWriter):
         if version_name not in self.inserts[product_name]:
             self.inserts[product_name][version_name] = {}
 
-        if 'ftype' in data:
-            ftype = data['ftype']
+        if "ftype" in data:
+            ftype = data["ftype"]
         else:
             flat = util.products_exdata(src, pedigree, include_top=False)
-            ftype = flat.get('ftype')
+            ftype = flat.get("ftype")
         self.inserts[product_name][version_name][item_name] = (
-            ftype, (data, src, target, pedigree, contentsource))
+            ftype,
+            (data, src, target, pedigree, contentsource),
+        )
 
     def insert_version(self, data, src, target, pedigree):
         """Upload all images for this version into glance
@@ -605,14 +694,20 @@ class GlanceMirror(mirrors.BasicMirrorWriter):
         product_name, version_name = pedigree
         inserts = self.inserts.get(product_name, {}).get(version_name, [])
 
-        rtar_names = [f for f in inserts
-                      if inserts[f][0] in ('root.tar.gz', 'root.tar.xz')]
+        rtar_names = [
+            f
+            for f in inserts
+            if inserts[f][0] in ("root.tar.gz", "root.tar.xz")
+        ]
 
         for _iname, (ftype, iargs) in inserts.items():
             if ftype == "squashfs" and rtar_names:
-                LOG.info("[%s] Skipping ftype 'squashfs' image in preference"
-                         "for root tarball type in %s",
-                         '/'.join(pedigree), rtar_names)
+                LOG.info(
+                    "[%s] Skipping ftype 'squashfs' image in preference"
+                    "for root tarball type in %s",
+                    "/".join(pedigree),
+                    rtar_names,
+                )
                 continue
             self._insert_item(*iargs)
 
@@ -622,9 +717,9 @@ class GlanceMirror(mirrors.BasicMirrorWriter):
 
     def remove_item(self, data, src, target, pedigree):
         util.products_del(target, pedigree)
-        if 'id' in data:
-            print("removing %s: %s" % (data['id'], data['name']))
-            self.gclient.images.delete(data['id'])
+        if "id" in data:
+            print("removing %s: %s" % (data["id"], data["name"]))
+            self.gclient.images.delete(data["id"])
 
     def filter_index_entry(self, data, src, pedigree):
         return filters.filter_dict(self.index_filters, data)
@@ -637,18 +732,18 @@ class GlanceMirror(mirrors.BasicMirrorWriter):
         util.products_prune(tree, preserve_empty_products=True)
 
         # stop these items from copying up when we call condense
-        sticky = ['ftype', 'md5', 'sha256', 'size', 'name', 'id']
+        sticky = ["ftype", "md5", "sha256", "size", "name", "id"]
 
         # LP: #1329805. Juju expects these on the item.
-        if self.config.get('sticky_endpoint_region', True):
-            sticky += ['endpoint', 'region']
+        if self.config.get("sticky_endpoint_region", True):
+            sticky += ["endpoint", "region"]
 
         util.products_condense(tree, sticky=sticky)
 
         tsnow = util.timestamp()
-        tree['updated'] = tsnow
+        tree["updated"] = tsnow
 
-        dpath = self._cidpath(tree['content_id'])
+        dpath = self._cidpath(tree["content_id"])
         LOG.info("writing data: %s", dpath)
         self.store.insert_content(dpath, util.dump_data(tree))
 
@@ -659,17 +754,20 @@ class GlanceMirror(mirrors.BasicMirrorWriter):
         except IOError as exc:
             if exc.errno != errno.ENOENT:
                 raise
-            index = {"index": {}, 'format': 'index:1.0',
-                     'updated': util.timestamp()}
-
-        index['index'][tree['content_id']] = {
-            'updated': tsnow,
-            'datatype': 'image-ids',
-            'clouds': [{'region': self.region, 'endpoint': self.auth_url}],
-            'cloudname': self.cloudname,
-            'path': dpath,
-            'products': list(tree['products'].keys()),
-            'format': tree['format'],
+            index = {
+                "index": {},
+                "format": "index:1.0",
+                "updated": util.timestamp(),
+            }
+
+        index["index"][tree["content_id"]] = {
+            "updated": tsnow,
+            "datatype": "image-ids",
+            "clouds": [{"region": self.region, "endpoint": self.auth_url}],
+            "cloudname": self.cloudname,
+            "path": dpath,
+            "products": list(tree["products"].keys()),
+            "format": tree["format"],
         }
         LOG.info("writing data: %s", ipath)
         self.store.insert_content(ipath, util.dump_data(index))
@@ -694,13 +792,13 @@ class ItemInfoDryRunMirror(GlanceMirror):
 
     def insert_item(self, data, src, target, pedigree, contentsource):
         data = util.products_exdata(src, pedigree)
-        if 'size' in data and 'path' in data and 'pubname' in data:
-            self.items[data['pubname']] = int(data['size'])
+        if "size" in data and "path" in data and "pubname" in data:
+            self.items[data["pubname"]] = int(data["size"])
 
 
 def _checksum_file(fobj, read_size=util.READ_SIZE, checksums=None):
     if checksums is None:
-        checksums = {'md5': None}
+        checksums = {"md5": None}
     cksum = checksum_util.checksummer(checksums=checksums)
     while True:
         buf = fobj.read(read_size)
@@ -713,13 +811,13 @@ def _checksum_file(fobj, read_size=util.READ_SIZE, checksums=None):
 def call_hook(item, path, cmd):
     env = os.environ.copy()
     env.update(item)
-    env['IMAGE_PATH'] = path
-    env['FIELDS'] = ' '.join(item.keys()) + ' IMAGE_PATH'
+    env["IMAGE_PATH"] = path
+    env["FIELDS"] = " ".join(item.keys()) + " IMAGE_PATH"
 
     util.subp(cmd, env=env, capture=False)
 
     with open(path, "rb") as fp:
-        md5 = _checksum_file(fp, checksums={'md5': None})
+        md5 = _checksum_file(fp, checksums={"md5": None})
 
     return (os.path.getsize(path), md5)
 
@@ -728,12 +826,13 @@ def _strip_version(endpoint):
     """Strip a version from the last component of an endpoint if present"""
 
     # Get rid of trailing '/' if present
-    if endpoint.endswith('/'):
+    if endpoint.endswith("/"):
         endpoint = endpoint[:-1]
-    url_bits = endpoint.split('/')
+    url_bits = endpoint.split("/")
     # regex to match 'v1' or 'v2.0' etc
-    if re.match(r'v\d+\.?\d*', url_bits[-1]):
-        endpoint = '/'.join(url_bits[:-1])
+    if re.match(r"v\d+\.?\d*", url_bits[-1]):
+        endpoint = "/".join(url_bits[:-1])
     return endpoint
 
+
 # vi: ts=4 expandtab syntax=python
diff --git a/simplestreams/objectstores/__init__.py b/simplestreams/objectstores/__init__.py
index f118a92..b9a6bbe 100644
--- a/simplestreams/objectstores/__init__.py
+++ b/simplestreams/objectstores/__init__.py
@@ -35,9 +35,13 @@ class ObjectStore(object):
 
     def insert_content(self, path, content, checksums=None, mutable=True):
         if not isinstance(content, bytes):
-            content = content.encode('utf-8')
-        self.insert(path=path, reader=cs.MemoryContentSource(content=content),
-                    checksums=checksums, mutable=mutable)
+            content = content.encode("utf-8")
+        self.insert(
+            path=path,
+            reader=cs.MemoryContentSource(content=content),
+            checksums=checksums,
+            mutable=mutable,
+        )
 
     def remove(self, path):
         # remove path from store
@@ -48,9 +52,12 @@ class ObjectStore(object):
         raise NotImplementedError()
 
     def exists_with_checksum(self, path, checksums=None):
-        return has_valid_checksum(path=path, reader=self.source,
-                                  checksums=checksums,
-                                  read_size=self.read_size)
+        return has_valid_checksum(
+            path=path,
+            reader=self.source,
+            checksums=checksums,
+            read_size=self.read_size,
+        )
 
 
 class MemoryObjectStore(ObjectStore):
@@ -73,35 +80,43 @@ class MemoryObjectStore(ObjectStore):
             url = "%s://%s" % (self.__class__, path)
             return cs.MemoryContentSource(content=self.data[path], url=url)
         except KeyError:
-            raise IOError(errno.ENOENT, '%s not found' % path)
+            raise IOError(errno.ENOENT, "%s not found" % path)
 
 
 class FileStore(ObjectStore):
-
     def __init__(self, prefix, complete_callback=None):
-        """ complete_callback is called periodically to notify users when a
+        """complete_callback is called periodically to notify users when a
         file is being inserted. It takes three arguments: the path that is
         inserted, the number of bytes downloaded, and the number of total
-        bytes. """
+        bytes."""
         self.prefix = prefix
         self.complete_callback = complete_callback
 
-    def insert(self, path, reader, checksums=None, mutable=True, size=None,
-               sparse=False):
-
+    def insert(
+        self,
+        path,
+        reader,
+        checksums=None,
+        mutable=True,
+        size=None,
+        sparse=False,
+    ):
         wpath = self._fullpath(path)
         if os.path.isfile(wpath):
             if not mutable:
                 # if the file exists, and not mutable, return
                 return
-            if has_valid_checksum(path=path, reader=self.source,
-                                  checksums=checksums,
-                                  read_size=self.read_size):
+            if has_valid_checksum(
+                path=path,
+                reader=self.source,
+                checksums=checksums,
+                read_size=self.read_size,
+            ):
                 return
 
         zeros = None
         if sparse is True:
-            zeros = '\0' * self.read_size
+            zeros = "\0" * self.read_size
 
         cksum = checksum_util.checksummer(checksums)
         out_d = os.path.dirname(wpath)
@@ -110,8 +125,9 @@ class FileStore(ObjectStore):
         util.mkdir_p(out_d)
         orig_part_size = 0
         reader_does_checksum = (
-            isinstance(reader, cs.ChecksummingContentSource) and
-            cksum.algorithm == reader.algorithm)
+            isinstance(reader, cs.ChecksummingContentSource)
+            and cksum.algorithm == reader.algorithm
+        )
 
         if os.path.exists(partfile):
             try:
@@ -121,8 +137,12 @@ class FileStore(ObjectStore):
                 else:
                     reader.set_start_pos(orig_part_size)
 
-                LOG.debug("resuming partial (%s) download of '%s' from '%s'",
-                          orig_part_size, path, partfile)
+                LOG.debug(
+                    "resuming partial (%s) download of '%s' from '%s'",
+                    orig_part_size,
+                    path,
+                    partfile,
+                )
                 with open(partfile, "rb") as fp:
                     while True:
                         buf = fp.read(self.read_size)
@@ -136,15 +156,17 @@ class FileStore(ObjectStore):
                 os.unlink(partfile)
 
         with open(partfile, "ab") as wfp:
-
             while True:
                 try:
                     buf = reader.read(self.read_size)
                 except checksum_util.InvalidChecksum:
                     break
                 buflen = len(buf)
-                if (buflen != self.read_size and zeros is not None and
-                        zeros[0:buflen] == buf):
+                if (
+                    buflen != self.read_size
+                    and zeros is not None
+                    and zeros[0:buflen] == buf
+                ):
                     wfp.seek(wfp.tell() + buflen)
                 elif buf == zeros:
                     wfp.seek(wfp.tell() + buflen)
@@ -209,8 +231,9 @@ class FileStore(ObjectStore):
         return os.path.join(self.prefix, path)
 
 
-def has_valid_checksum(path, reader, checksums=None,
-                       read_size=READ_BUFFER_SIZE):
+def has_valid_checksum(
+    path, reader, checksums=None, read_size=READ_BUFFER_SIZE
+):
     if checksums is None:
         return False
     try:
diff --git a/simplestreams/objectstores/s3.py b/simplestreams/objectstores/s3.py
index f1e9602..b07507f 100644
--- a/simplestreams/objectstores/s3.py
+++ b/simplestreams/objectstores/s3.py
@@ -15,19 +15,19 @@
 #   You should have received a copy of the GNU Affero General Public License
 #   along with Simplestreams.  If not, see <http://www.gnu.org/licenses/>.
 
+import errno
+import tempfile
+from contextlib import closing
+
 import boto.exception
 import boto.s3
 import boto.s3.connection
-from contextlib import closing
-import errno
-import tempfile
 
-import simplestreams.objectstores as objectstores
 import simplestreams.contentsource as cs
+import simplestreams.objectstores as objectstores
 
 
 class S3ObjectStore(objectstores.ObjectStore):
-
     _bucket = None
     _connection = None
 
@@ -92,8 +92,8 @@ class S3ObjectStore(objectstores.ObjectStore):
         if key is None:
             return False
 
-        if 'md5' in checksums:
-            return checksums['md5'] == key.etag.replace('"', "")
+        if "md5" in checksums:
+            return checksums["md5"] == key.etag.replace('"', "")
 
         return False
 
diff --git a/simplestreams/objectstores/swift.py b/simplestreams/objectstores/swift.py
index f2c0d5b..d33fa3b 100644
--- a/simplestreams/objectstores/swift.py
+++ b/simplestreams/objectstores/swift.py
@@ -15,30 +15,30 @@
 #   You should have received a copy of the GNU Affero General Public License
 #   along with Simplestreams.  If not, see <http://www.gnu.org/licenses/>.
 
-import simplestreams.objectstores as objectstores
-import simplestreams.contentsource as cs
-import simplestreams.openstack as openstack
-
 import errno
 import hashlib
-from swiftclient import Connection, ClientException
+
+from swiftclient import ClientException, Connection
+
+import simplestreams.contentsource as cs
+import simplestreams.objectstores as objectstores
+import simplestreams.openstack as openstack
 
 
 def get_swiftclient(**kwargs):
     # nmap has entries that need name changes from a 'get_service_conn_info'
     # to a swift Connection name.
     # pt has names that pass straight through
-    nmap = {'endpoint': 'preauthurl', 'token': 'preauthtoken'}
-    pt = ('insecure', 'cacert')
+    nmap = {"endpoint": "preauthurl", "token": "preauthtoken"}
+    pt = ("insecure", "cacert")
 
     connargs = {v: kwargs.get(k) for k, v in nmap.items() if k in kwargs}
     connargs.update({k: kwargs.get(k) for k in pt if k in kwargs})
-    if kwargs.get('session'):
-        sess = kwargs.get('session')
+    if kwargs.get("session"):
+        sess = kwargs.get("session")
         try:
             # If session is available try it
-            return Connection(session=sess,
-                              cacert=kwargs.get('cacert'))
+            return Connection(session=sess, cacert=kwargs.get("cacert"))
         except TypeError:
             # The edge case where session is availble but swiftclient is
             # < 3.3.0. Use the old style method for Connection.
@@ -52,7 +52,6 @@ class SwiftContentSource(cs.IteratorContentSource):
 
 
 class SwiftObjectStore(objectstores.ObjectStore):
-
     def __init__(self, prefix, region=None):
         # expect 'swift://bucket/path_prefix'
         self.prefix = prefix
@@ -67,35 +66,41 @@ class SwiftObjectStore(objectstores.ObjectStore):
 
         self.keystone_creds = openstack.load_keystone_creds()
         if region is not None:
-            self.keystone_creds['region_name'] = region
+            self.keystone_creds["region_name"] = region
 
-        conn_info = openstack.get_service_conn_info('object-store',
-                                                    **self.keystone_creds)
+        conn_info = openstack.get_service_conn_info(
+            "object-store", **self.keystone_creds
+        )
         self.swiftclient = get_swiftclient(**conn_info)
 
         # http://docs.openstack.org/developer/swift/misc.html#acls
-        self.swiftclient.put_container(self.container,
-                                       headers={'X-Container-Read':
-                                                '.r:*,.rlistings'})
+        self.swiftclient.put_container(
+            self.container, headers={"X-Container-Read": ".r:*,.rlistings"}
+        )
 
     def insert(self, path, reader, checksums=None, mutable=True, size=None):
         # store content from reader.read() into path, expecting result checksum
-        self._insert(path=path, contents=reader, checksums=checksums,
-                     mutable=mutable)
+        self._insert(
+            path=path, contents=reader, checksums=checksums, mutable=mutable
+        )
 
     def insert_content(self, path, content, checksums=None, mutable=True):
-        self._insert(path=path, contents=content, checksums=checksums,
-                     mutable=mutable)
+        self._insert(
+            path=path, contents=content, checksums=checksums, mutable=mutable
+        )
 
     def remove(self, path):
-        self.swiftclient.delete_object(container=self.container,
-                                       obj=self.path_prefix + path)
+        self.swiftclient.delete_object(
+            container=self.container, obj=self.path_prefix + path
+        )
 
     def source(self, path):
         def itgen():
             (_headers, iterator) = self.swiftclient.get_object(
-                container=self.container, obj=self.path_prefix + path,
-                resp_chunk_size=self.read_size)
+                container=self.container,
+                obj=self.path_prefix + path,
+                resp_chunk_size=self.read_size,
+            )
             return iterator
 
         return SwiftContentSource(itgen=itgen, url=self.prefix + path)
@@ -105,8 +110,9 @@ class SwiftObjectStore(objectstores.ObjectStore):
 
     def _head_path(self, path):
         try:
-            headers = self.swiftclient.head_object(container=self.container,
-                                                   obj=self.path_prefix + path)
+            headers = self.swiftclient.head_object(
+                container=self.container, obj=self.path_prefix + path
+            )
         except Exception as exc:
             if is_enoent(exc):
                 return {}
@@ -122,19 +128,22 @@ class SwiftObjectStore(objectstores.ObjectStore):
             if headers_match_checksums(headers, checksums):
                 return
 
-        insargs = {'container': self.container, 'obj': self.path_prefix + path,
-                   'contents': contents}
+        insargs = {
+            "container": self.container,
+            "obj": self.path_prefix + path,
+            "contents": contents,
+        }
 
         if size is not None and isinstance(contents, str):
             size = len(contents)
 
         if size is not None:
-            insargs['content_length'] = size
+            insargs["content_length"] = size
 
-        if checksums and checksums.get('md5'):
-            insargs['etag'] = checksums.get('md5')
+        if checksums and checksums.get("md5"):
+            insargs["etag"] = checksums.get("md5")
         elif isinstance(contents, str):
-            insargs['etag'] = hashlib.md5(contents).hexdigest()
+            insargs["etag"] = hashlib.md5(contents).hexdigest()
 
         self.swiftclient.put_object(**insargs)
 
@@ -142,13 +151,15 @@ class SwiftObjectStore(objectstores.ObjectStore):
 def headers_match_checksums(headers, checksums):
     if not (headers and checksums):
         return False
-    if ('md5' in checksums and headers.get('etag') == checksums.get('md5')):
+    if "md5" in checksums and headers.get("etag") == checksums.get("md5"):
         return True
     return False
 
 
 def is_enoent(exc):
-    return ((isinstance(exc, IOError) and exc.errno == errno.ENOENT) or
-            (isinstance(exc, ClientException) and exc.http_status == 404))
+    return (isinstance(exc, IOError) and exc.errno == errno.ENOENT) or (
+        isinstance(exc, ClientException) and exc.http_status == 404
+    )
+
 
 # vi: ts=4 expandtab
diff --git a/simplestreams/openstack.py b/simplestreams/openstack.py
index ebf63c4..48101e7 100644
--- a/simplestreams/openstack.py
+++ b/simplestreams/openstack.py
@@ -20,9 +20,11 @@ import os
 
 from keystoneclient.v2_0 import client as ksclient_v2
 from keystoneclient.v3 import client as ksclient_v3
+
 try:
     from keystoneauth1 import session
-    from keystoneauth1.identity import (v2, v3)
+    from keystoneauth1.identity import v2, v3
+
     _LEGACY_CLIENTS = False
 except ImportError:
     # 14.04 level packages do not have this.
@@ -31,43 +33,88 @@ except ImportError:
 
 
 OS_ENV_VARS = (
-    'OS_AUTH_TOKEN', 'OS_AUTH_URL', 'OS_CACERT', 'OS_IMAGE_API_VERSION',
-    'OS_IMAGE_URL', 'OS_PASSWORD', 'OS_REGION_NAME', 'OS_STORAGE_URL',
-    'OS_TENANT_ID', 'OS_TENANT_NAME', 'OS_USERNAME', 'OS_INSECURE',
-    'OS_USER_DOMAIN_NAME', 'OS_PROJECT_DOMAIN_NAME',
-    'OS_USER_DOMAIN_ID', 'OS_PROJECT_DOMAIN_ID', 'OS_PROJECT_NAME',
-    'OS_PROJECT_ID'
+    "OS_AUTH_TOKEN",
+    "OS_AUTH_URL",
+    "OS_CACERT",
+    "OS_IMAGE_API_VERSION",
+    "OS_IMAGE_URL",
+    "OS_PASSWORD",
+    "OS_REGION_NAME",
+    "OS_STORAGE_URL",
+    "OS_TENANT_ID",
+    "OS_TENANT_NAME",
+    "OS_USERNAME",
+    "OS_INSECURE",
+    "OS_USER_DOMAIN_NAME",
+    "OS_PROJECT_DOMAIN_NAME",
+    "OS_USER_DOMAIN_ID",
+    "OS_PROJECT_DOMAIN_ID",
+    "OS_PROJECT_NAME",
+    "OS_PROJECT_ID",
 )
 
 
 # only used for legacy client connection
-PT_V2 = ('username', 'password', 'tenant_id', 'tenant_name', 'auth_url',
-         'cacert', 'insecure', )
+PT_V2 = (
+    "username",
+    "password",
+    "tenant_id",
+    "tenant_name",
+    "auth_url",
+    "cacert",
+    "insecure",
+)
 
 # annoyingly the 'insecure' option in the old client constructor is now called
 # the 'verify' option in the session.Session() constructor
-PASSWORD_V2 = ('auth_url', 'username', 'password', 'user_id', 'trust_id',
-               'tenant_id', 'tenant_name', 'reauthenticate')
-PASSWORD_V3 = ('auth_url', 'password', 'username',
-               'user_id', 'user_domain_id', 'user_domain_name',
-               'trust_id', 'system_scope',
-               'domain_id', 'domain_name',
-               'project_id', 'project_name',
-               'project_domain_id', 'project_domain_name',
-               'reauthenticate')
-SESSION_ARGS = ('cert', 'timeout', 'verify', 'original_ip', 'redirect',
-                'addition_headers', 'app_name', 'app_version',
-                'additional_user_agent',
-                'discovery_cache', 'split_loggers', 'collect_timing')
-
-
-Settings = collections.namedtuple('Settings', 'mod ident arg_set')
-KS_VERSION_RESOLVER = {2: Settings(mod=ksclient_v2,
-                                   ident=v2,
-                                   arg_set=PASSWORD_V2),
-                       3: Settings(mod=ksclient_v3,
-                                   ident=v3,
-                                   arg_set=PASSWORD_V3)}
+PASSWORD_V2 = (
+    "auth_url",
+    "username",
+    "password",
+    "user_id",
+    "trust_id",
+    "tenant_id",
+    "tenant_name",
+    "reauthenticate",
+)
+PASSWORD_V3 = (
+    "auth_url",
+    "password",
+    "username",
+    "user_id",
+    "user_domain_id",
+    "user_domain_name",
+    "trust_id",
+    "system_scope",
+    "domain_id",
+    "domain_name",
+    "project_id",
+    "project_name",
+    "project_domain_id",
+    "project_domain_name",
+    "reauthenticate",
+)
+SESSION_ARGS = (
+    "cert",
+    "timeout",
+    "verify",
+    "original_ip",
+    "redirect",
+    "addition_headers",
+    "app_name",
+    "app_version",
+    "additional_user_agent",
+    "discovery_cache",
+    "split_loggers",
+    "collect_timing",
+)
+
+
+Settings = collections.namedtuple("Settings", "mod ident arg_set")
+KS_VERSION_RESOLVER = {
+    2: Settings(mod=ksclient_v2, ident=v2, arg_set=PASSWORD_V2),
+    3: Settings(mod=ksclient_v3, ident=v3, arg_set=PASSWORD_V3),
+}
 
 
 def load_keystone_creds(**kwargs):
@@ -87,32 +134,38 @@ def load_keystone_creds(**kwargs):
             # take off 'os_'
             ret[short] = os.environ[name]
 
-    if 'insecure' in ret:
-        if isinstance(ret['insecure'], str):
-            ret['insecure'] = (ret['insecure'].lower() not in
-                               ("", "0", "no", "off", 'false'))
+    if "insecure" in ret:
+        if isinstance(ret["insecure"], str):
+            ret["insecure"] = ret["insecure"].lower() not in (
+                "",
+                "0",
+                "no",
+                "off",
+                "false",
+            )
         else:
-            ret['insecure'] = bool(ret['insecure'])
+            ret["insecure"] = bool(ret["insecure"])
 
     # verify is the key that is used by requests, and thus the Session object.
     # i.e. verify is either False or a certificate path or file.
-    if not ret.get('insecure', False) and 'cacert' in ret:
-        ret['verify'] = ret['cacert']
+    if not ret.get("insecure", False) and "cacert" in ret:
+        ret["verify"] = ret["cacert"]
 
     missing = []
-    for req in ('username', 'auth_url'):
+    for req in ("username", "auth_url"):
         if not ret.get(req, None):
             missing.append(req)
 
-    if not (ret.get('auth_token') or ret.get('password')):
+    if not (ret.get("auth_token") or ret.get("password")):
         missing.append("(auth_token or password)")
 
-    api_version = get_ks_api_version(ret.get('auth_url', '')) or 2
-    if (api_version == 2 and
-            not (ret.get('tenant_id') or ret.get('tenant_name'))):
+    api_version = get_ks_api_version(ret.get("auth_url", "")) or 2
+    if api_version == 2 and not (
+        ret.get("tenant_id") or ret.get("tenant_name")
+    ):
         missing.append("(tenant_id or tenant_name)")
     if api_version == 3:
-        for k in ('user_domain_name', 'project_domain_name', 'project_name'):
+        for k in ("user_domain_name", "project_domain_name", "project_name"):
             if not ret.get(k, None):
                 missing.append(k)
 
@@ -124,8 +177,8 @@ def load_keystone_creds(**kwargs):
 
 def get_regions(client=None, services=None, kscreds=None):
     # if kscreds had 'region_name', then return that
-    if kscreds and kscreds.get('region_name'):
-        return [kscreds.get('region_name')]
+    if kscreds and kscreds.get("region_name"):
+        return [kscreds.get("region_name")]
 
     if client is None:
         creds = kscreds
@@ -139,7 +192,7 @@ def get_regions(client=None, services=None, kscreds=None):
     regions = set()
     for service in services:
         for r in endpoints.get(service, {}):
-            regions.add(r['region'])
+            regions.add(r["region"])
 
     return list(regions)
 
@@ -153,15 +206,15 @@ def get_ks_api_version(auth_url=None, env=None):
     if env is None:
         env = os.environ
 
-    if env.get('OS_IDENTITY_API_VERSION'):
-        return int(env['OS_IDENTITY_API_VERSION'])
+    if env.get("OS_IDENTITY_API_VERSION"):
+        return int(env["OS_IDENTITY_API_VERSION"])
 
     if auth_url is None:
         auth_url = ""
 
-    if auth_url.endswith('/v3'):
+    if auth_url.endswith("/v3"):
         return 3
-    elif auth_url.endswith('/v2.0'):
+    elif auth_url.endswith("/v2.0"):
         return 2
     # Return None if we can't determine the keystone version
     return None
@@ -178,20 +231,20 @@ def get_ksclient(**kwargs):
     if _LEGACY_CLIENTS:
         return _legacy_ksclient(**kwargs)
 
-    api_version = get_ks_api_version(kwargs.get('auth_url', '')) or 2
+    api_version = get_ks_api_version(kwargs.get("auth_url", "")) or 2
     arg_set = KS_VERSION_RESOLVER[api_version].arg_set
     # Filter/select the args for the api version from the kwargs dictionary
     kskw = {k: v for k, v in kwargs.items() if k in arg_set}
     auth = KS_VERSION_RESOLVER[api_version].ident.Password(**kskw)
     authkw = {k: v for k, v in kwargs.items() if k in SESSION_ARGS}
-    authkw['auth'] = auth
+    authkw["auth"] = auth
     sess = session.Session(**authkw)
     client = KS_VERSION_RESOLVER[api_version].mod.Client(session=sess)
     client.auth_ref = auth.get_access(sess)
     return client
 
 
-def get_service_conn_info(service='image', client=None, **kwargs):
+def get_service_conn_info(service="image", client=None, **kwargs):
     # return a dict with token, insecure, cacert, endpoint
     if not client:
         client = get_ksclient(**kwargs)
@@ -199,16 +252,23 @@ def get_service_conn_info(service='image', client=None, **kwargs):
     endpoint = _get_endpoint(client, service, **kwargs)
     # Session client does not have tenant_id set at client.tenant_id
     # If client.tenant_id not set use method to get it
-    tenant_id = (client.tenant_id or client.get_project_id(client.session) or
-                 client.auth.client.get_project_id())
-    info = {'token': client.auth_token, 'insecure': kwargs.get('insecure'),
-            'cacert': kwargs.get('cacert'), 'endpoint': endpoint,
-            'tenant_id': tenant_id}
+    tenant_id = (
+        client.tenant_id
+        or client.get_project_id(client.session)
+        or client.auth.client.get_project_id()
+    )
+    info = {
+        "token": client.auth_token,
+        "insecure": kwargs.get("insecure"),
+        "cacert": kwargs.get("cacert"),
+        "endpoint": endpoint,
+        "tenant_id": tenant_id,
+    }
     if not _LEGACY_CLIENTS:
-        info['session'] = client.session
-        info['glance_version'] = '2'
+        info["session"] = client.session
+        info["glance_version"] = "2"
     else:
-        info['glance_version'] = '1'
+        info["glance_version"] = "1"
 
     return info
 
@@ -216,12 +276,12 @@ def get_service_conn_info(service='image', client=None, **kwargs):
 def _get_endpoint(client, service, **kwargs):
     """Get an endpoint using the provided keystone client."""
     endpoint_kwargs = {
-        'service_type': service,
-        'interface': kwargs.get('endpoint_type') or 'publicURL',
-        'region_name': kwargs.get('region_name'),
+        "service_type": service,
+        "interface": kwargs.get("endpoint_type") or "publicURL",
+        "region_name": kwargs.get("region_name"),
     }
     if _LEGACY_CLIENTS:
-        del endpoint_kwargs['interface']
+        del endpoint_kwargs["interface"]
 
     endpoint = client.service_catalog.url_for(**endpoint_kwargs)
     return endpoint
diff --git a/simplestreams/util.py b/simplestreams/util.py
index 9866893..ebfe741 100644
--- a/simplestreams/util.py
+++ b/simplestreams/util.py
@@ -16,15 +16,15 @@
 #   along with Simplestreams.  If not, see <http://www.gnu.org/licenses/>.
 
 import errno
+import json
 import os
 import re
 import subprocess
 import tempfile
 import time
-import json
 
-import simplestreams.contentsource as cs
 import simplestreams.checksum_util as checksum_util
+import simplestreams.contentsource as cs
 from simplestreams.log import LOG
 
 ALIASNAME = "_aliases"
@@ -35,7 +35,7 @@ PGP_SIGNATURE_FOOTER = "-----END PGP SIGNATURE-----"
 
 _UNSET = object()
 
-READ_SIZE = (1024 * 10)
+READ_SIZE = 1024 * 10
 
 PRODUCTS_TREE_DATA = (
     ("products", "product_name"),
@@ -84,7 +84,7 @@ def products_exdata(tree, pedigree, include_top=True, insert_fieldnames=True):
     if include_top and tree:
         exdata.update(stringitems(tree))
     clevel = tree
-    for (n, key) in enumerate(pedigree):
+    for n, key in enumerate(pedigree):
         dictname, fieldname = harchy[n]
         clevel = clevel.get(dictname, {}).get(key, {})
         exdata.update(stringitems(clevel))
@@ -131,48 +131,54 @@ def products_del(tree, pedigree):
 
 
 def products_prune(tree, preserve_empty_products=False):
-    for prodname in list(tree.get('products', {}).keys()):
-        keys = list(tree['products'][prodname].get('versions', {}).keys())
+    for prodname in list(tree.get("products", {}).keys()):
+        keys = list(tree["products"][prodname].get("versions", {}).keys())
         for vername in keys:
-            vtree = tree['products'][prodname]['versions'][vername]
-            for itemname in list(vtree.get('items', {}).keys()):
-                if not vtree['items'][itemname]:
-                    del vtree['items'][itemname]
-
-            if 'items' not in vtree or not vtree['items']:
-                del tree['products'][prodname]['versions'][vername]
-
-        if ('versions' not in tree['products'][prodname] or
-                not tree['products'][prodname]['versions']):
-            del tree['products'][prodname]
-
-    if (not preserve_empty_products and 'products' in tree and
-            not tree['products']):
-        del tree['products']
-
-
-def walk_products(tree, cb_product=None, cb_version=None, cb_item=None,
-                  ret_finished=_UNSET):
+            vtree = tree["products"][prodname]["versions"][vername]
+            for itemname in list(vtree.get("items", {}).keys()):
+                if not vtree["items"][itemname]:
+                    del vtree["items"][itemname]
+
+            if "items" not in vtree or not vtree["items"]:
+                del tree["products"][prodname]["versions"][vername]
+
+        if (
+            "versions" not in tree["products"][prodname]
+            or not tree["products"][prodname]["versions"]
+        ):
+            del tree["products"][prodname]
+
+    if (
+        not preserve_empty_products
+        and "products" in tree
+        and not tree["products"]
+    ):
+        del tree["products"]
+
+
+def walk_products(
+    tree, cb_product=None, cb_version=None, cb_item=None, ret_finished=_UNSET
+):
     # walk a product tree. callbacks are called with (item, tree, (pedigree))
-    for prodname, proddata in tree['products'].items():
+    for prodname, proddata in tree["products"].items():
         if cb_product:
             ret = cb_product(proddata, tree, (prodname,))
             if ret_finished != _UNSET and ret == ret_finished:
                 return
 
-        if (not cb_version and not cb_item) or 'versions' not in proddata:
+        if (not cb_version and not cb_item) or "versions" not in proddata:
             continue
 
-        for vername, verdata in proddata['versions'].items():
+        for vername, verdata in proddata["versions"].items():
             if cb_version:
                 ret = cb_version(verdata, tree, (prodname, vername))
                 if ret_finished != _UNSET and ret == ret_finished:
                     return
 
-            if not cb_item or 'items' not in verdata:
+            if not cb_item or "items" not in verdata:
                 continue
 
-            for itemname, itemdata in verdata['items'].items():
+            for itemname, itemdata in verdata["items"].items():
                 ret = cb_item(itemdata, tree, (prodname, vername, itemname))
                 if ret_finished != _UNSET and ret == ret_finished:
                     return
@@ -205,8 +211,9 @@ def expand_data(data, refs=None, delete=False):
             expand_data(item, refs)
 
 
-def resolve_work(src, target, maxnum=None, keep=False, itemfilter=None,
-                 sort_reverse=True):
+def resolve_work(
+    src, target, maxnum=None, keep=False, itemfilter=None, sort_reverse=True
+):
     # if more than maxnum items are in src, only the most recent maxnum will be
     # stored in target.  If keep is true, then the most recent maxnum items
     # will be kept in target even if they are no longer in src.
@@ -241,8 +248,9 @@ def resolve_work(src, target, maxnum=None, keep=False, itemfilter=None,
         while len(remove) and (maxnum > (after_add - len(remove))):
             remove.pop(0)
 
-    mtarget = sorted([f for f in target + add if f not in remove],
-                     reverse=reverse)
+    mtarget = sorted(
+        [f for f in target + add if f not in remove], reverse=reverse
+    )
     if maxnum is not None and len(mtarget) > maxnum:
         for item in mtarget[maxnum:]:
             if item in target:
@@ -263,12 +271,12 @@ def has_gpgv():
     if _HAS_GPGV is not None:
         return _HAS_GPGV
 
-    if which('gpgv'):
+    if which("gpgv"):
         try:
             env = os.environ.copy()
-            env['LANG'] = 'C'
+            env["LANG"] = "C"
             out, err = subp(["gpgv", "--help"], capture=True, env=env)
-            _HAS_GPGV = 'gnupg' in out.lower() or 'gnupg' in err.lower()
+            _HAS_GPGV = "gnupg" in out.lower() or "gnupg" in err.lower()
         except subprocess.CalledProcessError:
             _HAS_GPGV = False
     else:
@@ -292,11 +300,13 @@ def read_signed(content, keyring=None, checked=True):
             try:
                 subp(cmd, data=content)
             except subprocess.CalledProcessError as e:
-                LOG.debug("failed: %s\n out=%s\n err=%s" %
-                          (' '.join(cmd), e.output[0], e.output[1]))
+                LOG.debug(
+                    "failed: %s\n out=%s\n err=%s"
+                    % (" ".join(cmd), e.output[0], e.output[1])
+                )
                 raise e
 
-        ret = {'body': [], 'signature': [], 'garbage': []}
+        ret = {"body": [], "signature": [], "garbage": []}
         lines = content.splitlines()
         i = 0
         for i in range(0, len(lines)):
@@ -320,21 +330,22 @@ def read_signed(content, keyring=None, checked=True):
             else:
                 ret[mode].append(lines[i])
 
-        ret['body'].append('')  # need empty line at end
-        return "\n".join(ret['body'])
+        ret["body"].append("")  # need empty line at end
+        return "\n".join(ret["body"])
     else:
         raise SignatureMissingException("No signature found!")
 
 
 def load_content(content):
     if isinstance(content, bytes):
-        content = content.decode('utf-8')
+        content = content.decode("utf-8")
     return json.loads(content)
 
 
 def dump_data(data):
-    return json.dumps(data, indent=1, sort_keys=True,
-                      separators=(',', ': ')).encode('utf-8')
+    return json.dumps(
+        data, indent=1, sort_keys=True, separators=(",", ": ")
+    ).encode("utf-8")
 
 
 def timestamp(ts=None):
@@ -375,24 +386,26 @@ def move_dups(src, target, sticky=None):
     target.update(updates)
 
 
-def products_condense(ptree, sticky=None, top='versions'):
+def products_condense(ptree, sticky=None, top="versions"):
     # walk a products tree, copying up item keys as far as they'll go
     # only move items to a sibling of the 'top'.
 
-    if top not in ('versions', 'products'):
-        raise ValueError("'top' must be one of: %s" %
-                         ','.join(PRODUCTS_TREE_HIERARCHY))
+    if top not in ("versions", "products"):
+        raise ValueError(
+            "'top' must be one of: %s" % ",".join(PRODUCTS_TREE_HIERARCHY)
+        )
 
     def call_move_dups(cur, _tree, pedigree):
-        (_mtype, stname) = (("product", "versions"),
-                            ("version", "items"))[len(pedigree) - 1]
+        (_mtype, stname) = (("product", "versions"), ("version", "items"))[
+            len(pedigree) - 1
+        ]
         move_dups(cur.get(stname, {}), cur, sticky=sticky)
 
     walk_products(ptree, cb_version=call_move_dups)
     walk_products(ptree, cb_product=call_move_dups)
-    if top == 'versions':
+    if top == "versions":
         return
-    move_dups(ptree['products'], ptree)
+    move_dups(ptree["products"], ptree)
 
 
 def assert_safe_path(path):
@@ -449,16 +462,22 @@ def subp(args, data=None, capture=True, shell=False, env=None):
     else:
         stdout, stderr = (subprocess.PIPE, subprocess.PIPE)
 
-    sp = subprocess.Popen(args, stdout=stdout, stderr=stderr,
-                          stdin=subprocess.PIPE, shell=shell, env=env)
+    sp = subprocess.Popen(
+        args,
+        stdout=stdout,
+        stderr=stderr,
+        stdin=subprocess.PIPE,
+        shell=shell,
+        env=env,
+    )
     if isinstance(data, str):
-        data = data.encode('utf-8')
+        data = data.encode("utf-8")
 
     (out, err) = sp.communicate(data)
     if isinstance(out, bytes):
-        out = out.decode('utf-8')
+        out = out.decode("utf-8")
     if isinstance(err, bytes):
-        err = err.decode('utf-8')
+        err = err.decode("utf-8")
 
     rc = sp.returncode
     if rc != 0:
@@ -468,22 +487,22 @@ def subp(args, data=None, capture=True, shell=False, env=None):
 
 
 def get_sign_cmd(path, output=None, inline=False):
-    cmd = ['gpg']
-    defkey = os.environ.get('SS_GPG_DEFAULT_KEY')
+    cmd = ["gpg"]
+    defkey = os.environ.get("SS_GPG_DEFAULT_KEY")
     if defkey:
-        cmd.extend(['--default-key', defkey])
+        cmd.extend(["--default-key", defkey])
 
-    batch = os.environ.get('SS_GPG_BATCH', "1").lower()
+    batch = os.environ.get("SS_GPG_BATCH", "1").lower()
     if batch not in ("0", "false"):
-        cmd.append('--batch')
+        cmd.append("--batch")
 
     if output:
-        cmd.extend(['--output', output])
+        cmd.extend(["--output", output])
 
     if inline:
-        cmd.append('--clearsign')
+        cmd.append("--clearsign")
     else:
-        cmd.extend(['--armor', '--detach-sign'])
+        cmd.extend(["--armor", "--detach-sign"])
 
     cmd.extend([path])
     return cmd
@@ -498,17 +517,17 @@ def make_signed_content_paths(content):
     if data.get("format") != "index:1.0":
         return (False, None)
 
-    for content_ent in list(data.get('index', {}).values()):
-        path = content_ent.get('path')
+    for content_ent in list(data.get("index", {}).values()):
+        path = content_ent.get("path")
         if path.endswith(".json"):
-            content_ent['path'] = signed_fname(path, inline=True)
+            content_ent["path"] = signed_fname(path, inline=True)
 
     return (True, json.dumps(data, indent=1))
 
 
 def signed_fname(fname, inline=True):
     if inline:
-        sfname = fname[0:-len(".json")] + ".sjson"
+        sfname = fname[0 : -len(".json")] + ".sjson"
     else:
         sfname = fname + ".gpg"
 
@@ -555,8 +574,10 @@ def sign_file(fname, inline=True, outfile=None):
 
 def sign_content(content, outfile="-", inline=True):
     rm_f_file(outfile, skip=["-"])
-    return subp(args=get_sign_cmd(path="-", output=outfile, inline=inline),
-                data=content)[0]
+    return subp(
+        args=get_sign_cmd(path="-", output=outfile, inline=inline),
+        data=content,
+    )[0]
 
 
 def path_from_mirror_url(mirror, path):
@@ -566,8 +587,8 @@ def path_from_mirror_url(mirror, path):
     path_regex = "streams/v1/.*[.](sjson|json)$"
     result = re.search(path_regex, mirror)
     if result:
-        path = mirror[result.start():]
-        mirror = mirror[:result.start()]
+        path = mirror[result.start() :]
+        mirror = mirror[: result.start()]
     else:
         path = "streams/v1/index.sjson"
 
@@ -589,21 +610,21 @@ class ProgressAggregator(object):
         self.total_written = 0
 
     def progress_callback(self, progress):
-        if self.current_file != progress['name']:
+        if self.current_file != progress["name"]:
             if self.remaining_items and self.current_file is not None:
                 del self.remaining_items[self.current_file]
-            self.current_file = progress['name']
+            self.current_file = progress["name"]
             self.last_emitted = 0
             self.current_written = 0
 
-        size = float(progress['size'])
-        written = float(progress['written'])
+        size = float(progress["size"])
+        written = float(progress["written"])
         self.current_written += written
         self.total_written += written
         interval = self.current_written - self.last_emitted
         if interval > size / 100:
             self.last_emitted = self.current_written
-            progress['written'] = self.current_written
+            progress["written"] = self.current_written
             self.emit(progress)
 
     def emit(self, progress):
diff --git a/tests/httpserver.py b/tests/httpserver.py
index e88eb56..10bde9a 100644
--- a/tests/httpserver.py
+++ b/tests/httpserver.py
@@ -1,50 +1,61 @@
 #!/usr/bin/env python
 import os
 import sys
+
 if sys.version_info.major == 2:
-    from SimpleHTTPServer import SimpleHTTPRequestHandler
     from BaseHTTPServer import HTTPServer
+    from SimpleHTTPServer import SimpleHTTPRequestHandler
 else:
-    from http.server import SimpleHTTPRequestHandler
-    from http.server import HTTPServer
+    from http.server import HTTPServer, SimpleHTTPRequestHandler
 
 
 class LoggingHTTPRequestHandler(SimpleHTTPRequestHandler):
-    def log_request(self, code='-', size='-'):
+    def log_request(self, code="-", size="-"):
         """
         Log an accepted request along with user-agent string.
         """
 
         user_agent = self.headers.get("user-agent")
-        self.log_message('"%s" %s %s (%s)',
-                         self.requestline, str(code), str(size), user_agent)
+        self.log_message(
+            '"%s" %s %s (%s)',
+            self.requestline,
+            str(code),
+            str(size),
+            user_agent,
+        )
 
 
-def run(address, port,
-        HandlerClass=LoggingHTTPRequestHandler, ServerClass=HTTPServer):
+def run(
+    address,
+    port,
+    HandlerClass=LoggingHTTPRequestHandler,
+    ServerClass=HTTPServer,
+):
     try:
         server = ServerClass((address, port), HandlerClass)
         address, port = server.socket.getsockname()
-        sys.stderr.write("Serving HTTP: %s %s %s\n" %
-                         (address, port, os.getcwd()))
+        sys.stderr.write(
+            "Serving HTTP: %s %s %s\n" % (address, port, os.getcwd())
+        )
         server.serve_forever()
     except KeyboardInterrupt:
         server.socket.close()
 
 
-if __name__ == '__main__':
+if __name__ == "__main__":
     import sys
+
     if len(sys.argv) == 3:
         # 2 args: address and port
         address = sys.argv[1]
         port = int(sys.argv[2])
     elif len(sys.argv) == 2:
         # 1 arg: port
-        address = '0.0.0.0'
+        address = "0.0.0.0"
         port = int(sys.argv[1])
     elif len(sys.argv) == 1:
         # no args random port (port=0)
-        address = '0.0.0.0'
+        address = "0.0.0.0"
         port = 0
     else:
         sys.stderr.write("Expect [address] [port]\n")
diff --git a/tests/testutil.py b/tests/testutil.py
index 2c89e3a..2816366 100644
--- a/tests/testutil.py
+++ b/tests/testutil.py
@@ -1,10 +1,10 @@
 import os
-from simplestreams import objectstores
-from simplestreams import mirrors
 
+from simplestreams import mirrors, objectstores
 
-EXAMPLES_DIR = os.path.abspath(os.path.join(os.path.dirname(__file__),
-                                            "..", "examples"))
+EXAMPLES_DIR = os.path.abspath(
+    os.path.join(os.path.dirname(__file__), "..", "examples")
+)
 
 
 def get_mirror_reader(name, docdir=None, signed=False):
@@ -20,4 +20,5 @@ def get_mirror_reader(name, docdir=None, signed=False):
     kwargs = {} if signed else {"policy": policy}
     return mirrors.ObjectStoreMirrorReader(sstore, **kwargs)
 
+
 # vi: ts=4 expandtab syntax=python
diff --git a/tests/unittests/test_badmirrors.py b/tests/unittests/test_badmirrors.py
index 6c5546f..d640e28 100644
--- a/tests/unittests/test_badmirrors.py
+++ b/tests/unittests/test_badmirrors.py
@@ -1,11 +1,12 @@
 from unittest import TestCase
-from tests.testutil import get_mirror_reader
+
+from simplestreams import checksum_util, mirrors, util
 from simplestreams.mirrors import (
-    ObjectStoreMirrorWriter, ObjectStoreMirrorReader)
+    ObjectStoreMirrorReader,
+    ObjectStoreMirrorWriter,
+)
 from simplestreams.objectstores import MemoryObjectStore
-from simplestreams import util
-from simplestreams import checksum_util
-from simplestreams import mirrors
+from tests.testutil import get_mirror_reader
 
 
 class TestBadDataSources(TestCase):
@@ -19,7 +20,8 @@ class TestBadDataSources(TestCase):
     def setUp(self):
         self.src = self.get_clean_src(self.example, path=self.dlpath)
         self.target = ObjectStoreMirrorWriter(
-            config={}, objectstore=MemoryObjectStore())
+            config={}, objectstore=MemoryObjectStore()
+        )
 
     def get_clean_src(self, exname, path):
         good_src = get_mirror_reader(exname)
@@ -34,7 +36,8 @@ class TestBadDataSources(TestCase):
                 del objectstore.data[k]
 
         return ObjectStoreMirrorReader(
-            objectstore=objectstore, policy=lambda content, path: content)
+            objectstore=objectstore, policy=lambda content, path: content
+        )
 
     def test_sanity_valid(self):
         # verify that the tests are fine on expected pass
@@ -43,50 +46,77 @@ class TestBadDataSources(TestCase):
 
     def test_larger_size_causes_bad_checksum(self):
         def size_plus_1(item):
-            item['size'] = int(item['size']) + 1
+            item["size"] = int(item["size"]) + 1
             return item
 
         _moditem(self.src, self.dlpath, self.pedigree, size_plus_1)
-        self.assertRaises(checksum_util.InvalidChecksum,
-                          self.target.sync, self.src, self.dlpath)
+        self.assertRaises(
+            checksum_util.InvalidChecksum,
+            self.target.sync,
+            self.src,
+            self.dlpath,
+        )
 
     def test_smaller_size_causes_bad_checksum(self):
         def size_minus_1(item):
-            item['size'] = int(item['size']) - 1
+            item["size"] = int(item["size"]) - 1
             return item
+
         _moditem(self.src, self.dlpath, self.pedigree, size_minus_1)
-        self.assertRaises(checksum_util.InvalidChecksum,
-                          self.target.sync, self.src, self.dlpath)
+        self.assertRaises(
+            checksum_util.InvalidChecksum,
+            self.target.sync,
+            self.src,
+            self.dlpath,
+        )
 
     def test_too_much_content_causes_bad_checksum(self):
         self.src.objectstore.data[self.item_path] += b"extra"
-        self.assertRaises(checksum_util.InvalidChecksum,
-                          self.target.sync, self.src, self.dlpath)
+        self.assertRaises(
+            checksum_util.InvalidChecksum,
+            self.target.sync,
+            self.src,
+            self.dlpath,
+        )
 
     def test_too_little_content_causes_bad_checksum(self):
         orig = self.src.objectstore.data[self.item_path]
         self.src.objectstore.data[self.item_path] = orig[0:-1]
-        self.assertRaises(checksum_util.InvalidChecksum,
-                          self.target.sync, self.src, self.dlpath)
+        self.assertRaises(
+            checksum_util.InvalidChecksum,
+            self.target.sync,
+            self.src,
+            self.dlpath,
+        )
 
     def test_busted_checksum_causes_bad_checksum(self):
         def break_checksum(item):
             chars = "0123456789abcdef"
-            orig = item['sha256']
-            item['sha256'] = ''.join(
-                [chars[(chars.find(c) + 1) % len(chars)] for c in orig])
+            orig = item["sha256"]
+            item["sha256"] = "".join(
+                [chars[(chars.find(c) + 1) % len(chars)] for c in orig]
+            )
             return item
 
         _moditem(self.src, self.dlpath, self.pedigree, break_checksum)
-        self.assertRaises(checksum_util.InvalidChecksum,
-                          self.target.sync, self.src, self.dlpath)
+        self.assertRaises(
+            checksum_util.InvalidChecksum,
+            self.target.sync,
+            self.src,
+            self.dlpath,
+        )
 
     def test_changed_content_causes_bad_checksum(self):
         # correct size but different content should raise bad checksum
-        self.src.objectstore.data[self.item_path] = ''.join(
-            ["x" for c in self.src.objectstore.data[self.item_path]])
-        self.assertRaises(checksum_util.InvalidChecksum,
-                          self.target.sync, self.src, self.dlpath)
+        self.src.objectstore.data[self.item_path] = "".join(
+            ["x" for c in self.src.objectstore.data[self.item_path]]
+        )
+        self.assertRaises(
+            checksum_util.InvalidChecksum,
+            self.target.sync,
+            self.src,
+            self.dlpath,
+        )
 
     def test_no_checksums_cause_bad_checksum(self):
         def del_checksums(item):
@@ -96,32 +126,41 @@ class TestBadDataSources(TestCase):
 
         _moditem(self.src, self.dlpath, self.pedigree, del_checksums)
         with _patched_missing_sum("fail"):
-            self.assertRaises(checksum_util.InvalidChecksum,
-                              self.target.sync, self.src, self.dlpath)
+            self.assertRaises(
+                checksum_util.InvalidChecksum,
+                self.target.sync,
+                self.src,
+                self.dlpath,
+            )
 
     def test_missing_size_causes_bad_checksum(self):
         def del_size(item):
-            del item['size']
+            del item["size"]
             return item
 
         _moditem(self.src, self.dlpath, self.pedigree, del_size)
         with _patched_missing_sum("fail"):
-            self.assertRaises(checksum_util.InvalidChecksum,
-                              self.target.sync, self.src, self.dlpath)
+            self.assertRaises(
+                checksum_util.InvalidChecksum,
+                self.target.sync,
+                self.src,
+                self.dlpath,
+            )
 
 
 class _patched_missing_sum(object):
     """This patches the legacy mode for missing checksum info so
     that it behaves like the new code path.  Thus we can make
     the test run correctly"""
+
     def __init__(self, mode="fail"):
         self.mode = mode
 
     def __enter__(self):
-        self.modmcb = getattr(mirrors, '_missing_cksum_behavior', {})
+        self.modmcb = getattr(mirrors, "_missing_cksum_behavior", {})
         self.orig = self.modmcb.copy()
         if self.modmcb:
-            self.modmcb['mode'] = self.mode
+            self.modmcb["mode"] = self.mode
         return self
 
     def __exit__(self, type, value, traceback):
diff --git a/tests/unittests/test_command_hook_mirror.py b/tests/unittests/test_command_hook_mirror.py
index 6a72749..e718c9e 100644
--- a/tests/unittests/test_command_hook_mirror.py
+++ b/tests/unittests/test_command_hook_mirror.py
@@ -1,4 +1,5 @@
 from unittest import TestCase
+
 import simplestreams.mirrors.command_hook as chm
 from tests.testutil import get_mirror_reader
 
@@ -13,12 +14,11 @@ class TestCommandHookMirror(TestCase):
         self.assertRaises(TypeError, chm.CommandHookMirror, {})
 
     def test_init_with_load_products_works(self):
-        chm.CommandHookMirror({'load_products': 'true'})
+        chm.CommandHookMirror({"load_products": "true"})
 
     def test_stream_load_empty(self):
-
         src = get_mirror_reader("foocloud")
-        target = chm.CommandHookMirror({'load_products': ['true']})
+        target = chm.CommandHookMirror({"load_products": ["true"]})
         oruncmd = chm.run_command
 
         try:
@@ -30,14 +30,16 @@ class TestCommandHookMirror(TestCase):
 
         # the 'load_products' should be called once for each content
         # in the stream.
-        self.assertEqual(self._run_commands, [['true'], ['true']])
+        self.assertEqual(self._run_commands, [["true"], ["true"]])
 
     def test_stream_insert_product(self):
-
         src = get_mirror_reader("foocloud")
         target = chm.CommandHookMirror(
-            {'load_products': ['load-products'],
-             'insert_products': ['insert-products']})
+            {
+                "load_products": ["load-products"],
+                "insert_products": ["insert-products"],
+            }
+        )
         oruncmd = chm.run_command
 
         try:
@@ -49,15 +51,18 @@ class TestCommandHookMirror(TestCase):
 
         # the 'load_products' should be called once for each content
         # in the stream. same for 'insert-products'
-        self.assertEqual(len([f for f in self._run_commands
-                             if f == ['load-products']]), 2)
-        self.assertEqual(len([f for f in self._run_commands
-                             if f == ['insert-products']]), 2)
+        self.assertEqual(
+            len([f for f in self._run_commands if f == ["load-products"]]), 2
+        )
+        self.assertEqual(
+            len([f for f in self._run_commands if f == ["insert-products"]]), 2
+        )
 
     def _run_command(self, cmd, env=None, capture=False, rcs=None):
         self._run_commands.append(cmd)
         rc = 0
-        output = ''
+        output = ""
         return (rc, output)
 
+
 # vi: ts=4 expandtab syntax=python
diff --git a/tests/unittests/test_contentsource.py b/tests/unittests/test_contentsource.py
index ef838a2..2b88a15 100644
--- a/tests/unittests/test_contentsource.py
+++ b/tests/unittests/test_contentsource.py
@@ -2,14 +2,14 @@ import os
 import shutil
 import sys
 import tempfile
-
-from os.path import join, dirname
-from simplestreams import objectstores
-from simplestreams import contentsource
-from subprocess import Popen, PIPE, STDOUT
+from os.path import dirname, join
+from subprocess import PIPE, STDOUT, Popen
 from unittest import TestCase, skipIf
+
 import pytest
 
+from simplestreams import contentsource, objectstores
+
 
 class RandomPortServer(object):
     def __init__(self, path):
@@ -22,14 +22,15 @@ class RandomPortServer(object):
         if self.port and self.process:
             return
         testserver_path = join(
-            dirname(__file__), "..", "..", "tests", "httpserver.py")
-        pre = b'Serving HTTP:'
+            dirname(__file__), "..", "..", "tests", "httpserver.py"
+        )
+        pre = b"Serving HTTP:"
 
-        cmd = [sys.executable, '-u', testserver_path, "0"]
+        cmd = [sys.executable, "-u", testserver_path, "0"]
         p = Popen(cmd, cwd=self.path, stdout=PIPE, stderr=STDOUT)
         line = p.stdout.readline()  # pylint: disable=E1101
         if line.startswith(pre):
-            data = line[len(pre):].strip()
+            data = line[len(pre) :].strip()
             addr, port_str, cwd = data.decode().split(" ", 2)
             self.port = int(port_str)
             self.addr = addr
@@ -39,8 +40,9 @@ class RandomPortServer(object):
         else:
             p.kill()
             raise RuntimeError(
-                "Failed to start server in %s with %s. pid=%s. got: %s" %
-                (self.path, cmd, self.process, line))
+                "Failed to start server in %s with %s. pid=%s. got: %s"
+                % (self.path, cmd, self.process, line)
+            )
 
     def read_output(self):
         return str(self.process.stdout.readline())
@@ -63,13 +65,17 @@ class RandomPortServer(object):
         if self.process:
             pid = self.process.pid
 
-        return ("RandomPortServer(port=%s, addr=%s, process=%s, path=%s)" %
-                (self.port, self.addr, pid, self.path))
+        return "RandomPortServer(port=%s, addr=%s, process=%s, path=%s)" % (
+            self.port,
+            self.addr,
+            pid,
+            self.path,
+        )
 
     def url_for(self, fpath=""):
         if self.port is None:
             raise ValueError("No port available")
-        return 'http://127.0.0.1:%d/' % self.port + fpath
+        return "http://127.0.0.1:%d/" % self.port + fpath
 
 
 class BaseDirUsingTestCase(TestCase):
@@ -100,7 +106,8 @@ class BaseDirUsingTestCase(TestCase):
 
     def getcs(self, path, url_reader=None, rel=None):
         return contentsource.UrlContentSource(
-            self.url_for(path, rel=rel), url_reader=url_reader)
+            self.url_for(path, rel=rel), url_reader=url_reader
+        )
 
     def path_for(self, fpath, rel=None):
         # return full path to fpath.
@@ -120,8 +127,9 @@ class BaseDirUsingTestCase(TestCase):
 
         if not fullpath.startswith(self.tmpd + os.path.sep):
             raise ValueError(
-                "%s is not a valid path.  Not under tmpdir: %s" %
-                (fpath, self.tmpd))
+                "%s is not a valid path.  Not under tmpdir: %s"
+                % (fpath, self.tmpd)
+            )
 
         return fullpath
 
@@ -133,17 +141,18 @@ class BaseDirUsingTestCase(TestCase):
         if not self.server:
             raise ValueError("No server available, but proto == http")
         return self.server.url_for(
-            self.path_for(fpath=fpath, rel=rel)[len(self.tmpd)+1:])
+            self.path_for(fpath=fpath, rel=rel)[len(self.tmpd) + 1 :]
+        )
 
 
 class TestUrlContentSource(BaseDirUsingTestCase):
     http = True
-    fpath = 'foo'
-    fdata = b'hello world\n'
+    fpath = "foo"
+    fdata = b"hello world\n"
 
     def setUp(self):
         super(TestUrlContentSource, self).setUp()
-        with open(join(self.test_d, self.fpath), 'wb') as f:
+        with open(join(self.test_d, self.fpath), "wb") as f:
             f.write(self.fdata)
 
     def test_default_url_read_handles_None(self):
@@ -171,8 +180,10 @@ class TestUrlContentSource(BaseDirUsingTestCase):
 
     @skipIf(contentsource.requests is None, "requests not available")
     def test_requests_default_timeout(self):
-        self.assertEqual(contentsource.RequestsUrlReader.timeout,
-                         (contentsource.TIMEOUT, None))
+        self.assertEqual(
+            contentsource.RequestsUrlReader.timeout,
+            (contentsource.TIMEOUT, None),
+        )
 
     @skipIf(contentsource.requests is None, "requests not available")
     def test_requests_url_read_handles_None(self):