Merge ~tonysimpson/snapstore-client:snapstore-library-integration into snapstore-client:master

Proposed by Tony Simpson on 2019-08-07
Status: Work in progress
Proposed branch: ~tonysimpson/snapstore-client:snapstore-library-integration
Merge into: snapstore-client:master
Diff against target: 3582 lines (+1951/-1049)
19 files modified
Makefile (+32/-12)
dev/null (+0/-185)
requirements.txt (+1/-0)
snap/plugins/_python/__init__.py (+19/-0)
snap/plugins/_python/_pip.py (+575/-0)
snap/plugins/_python/_python_finder.py (+104/-0)
snap/plugins/_python/_sitecustomize.py (+124/-0)
snap/plugins/_python/errors.py (+79/-0)
snap/plugins/python_ols.py (+499/-0)
snap/snapcraft.yaml (+15/-8)
snapstore (+1/-126)
snapstore_client/__main__.py (+136/-0)
snapstore_client/logic/login.py (+36/-59)
snapstore_client/logic/overrides.py (+79/-71)
snapstore_client/logic/push.py (+0/-5)
snapstore_client/logic/tests/test_login.py (+103/-265)
snapstore_client/logic/tests/test_overrides.py (+106/-313)
snapstore_client/logic/tests/utils.py (+42/-0)
snapstore_client/utils.py (+0/-5)
Reviewer Review Type Date Requested Status
Simon Davy 2019-08-07 Approve on 2019-08-09
Review via email: mp+371039@code.launchpad.net

Commit message

Integrate snapstore_library auth and overrides

Description of the change

Integrates the snapstore-library authentication and override

Tests have been simplified because snapstore-library has test coverage. Updated tests just cover functionality in CLI code.

snap/plugins/* should be ignore as it is just a copy of code from lp:snapstore-snap

There is a snapstore-library branch in lp:~siab/+git/siab-dependencies that can be used for testing.

CI integration has not been considered yet - a setup-jenkaas would need to be added to the Makefile and would need to pull siab-dependencies

snapstore-library review here https://code.launchpad.net/~tonysimpson/snapstore-library/+git/snapstore-library/+merge/371038

To post a comment you must log in.
Simon Davy (bloodearnest) wrote :

Ok, so this is a big MP to review, which is not great

It might have have been better to split it into 2 or 3, 1 just adding the python-ols snapcraft plugin, one for overrides and one for authentication?

That said, I think the tests are much improved, and the use of the library makes this all much nicer.

review: Approve

Unmerged commits

9038b0a... by Tony Simpson on 2019-08-07

Removed dead code

e980150... by Tony Simpson on 2019-08-06

Make build process more like snap-store-proxy

Borrow approach from snapstore-snap (snap-store-proxy).
snap/plugins/* is a copy of plugin from snap-store-proxy.

54341a0... by Tony Simpson on 2019-08-02

Integrate snapstore_library auth and overrides

* Test have been rewritten, still requires a few more I think
* Dead code has not been removed yet
* Has not been user tested

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/Makefile b/Makefile
2index d8f1bd4..2266b98 100644
3--- a/Makefile
4+++ b/Makefile
5@@ -1,15 +1,22 @@
6-# In a global tmp dir for now so snapcraft doesn't make copies (see
7-# https://forum.snapcraft.io/t/customizable-ignores/230/12).
8-TMPDIR = /tmp/snapstore-client.tmp
9-
10+FILE_DEPS = Makefile snap/snapcraft.yaml snapstore snapstore_client/**/*.py dependencies/*.*
11 SERVICE_PACKAGE = snapstore_client
12-ENV = $(TMPDIR)/env
13+ENV = env
14 PYTHON3 = $(ENV)/bin/python3
15 PIP = $(PYTHON3) -m pip
16 FLAKE8 = $(ENV)/bin/flake8
17
18 DEPENDENCY_REPO ?= lp:~siab/+git/siab-dependencies
19-SNAPSTORE_DEPENDENCY_DIR ?= $(TMPDIR)/dependencies
20+SNAPSTORE_DEPENDENCY_DIR = dependencies
21+
22+SNAPCRAFT ?= snapcraft
23+VIRT := $(shell systemd-detect-virt)
24+ifeq ($(VIRT),lxc)
25+ SNAPCRAFT_FLAGS ?= --destructive-mode
26+else
27+ SNAPCRAFT_FLAGS ?= --use-lxd
28+endif
29+SNAPCRAFT_BUILD_ENVIRONMENT_CHANNEL_SNAPCRAFT ?= edge
30+export PATH := $(PATH):/snap/bin
31
32
33 $(SNAPSTORE_DEPENDENCY_DIR):
34@@ -26,8 +33,13 @@ $(ENV)/dev: $(ENV)/prod
35
36 bootstrap: $(ENV)/prod
37
38-snap:
39- snapcraft cleanbuild
40+snap-store-proxy-client.snap: $(SNAPSTORE_DEPENDENCY_DIR) $(FILE_DEPS)
41+ SNAPCRAFT_BUILD_ENVIRONMENT_CHANNEL_SNAPCRAFT=${SNAPCRAFT_BUILD_ENVIRONMENT_CHANNEL_SNAPCRAFT} SNAPCRAFT_ENABLE_ERROR_REPORTING=0 $(SNAPCRAFT) snap $(SNAPCRAFT_FLAGS) --output snap-store-proxy-client.snap
42+
43+snap: snap-store-proxy-client.snap
44+
45+install: snap
46+ sudo snap install snap-store-proxy-client.snap --dangerous
47
48 test: $(ENV)/dev
49 $(PYTHON3) -m unittest $(TESTS) 2>&1
50@@ -49,11 +61,19 @@ coverage: $(ENV)/dev
51 $(PYTHON3) -m coverage html
52
53 clean:
54- rm -rf $(TMPDIR)
55+ifneq (,$(findstring --destructive-mode, $(SNAPCRAFT_FLAGS)))
56+ rm -fr parts
57+ rm -fr stage
58+else
59+ $(SNAPCRAFT) clean $(SNAPCRAFT_FLAGS)
60+endif
61+ rm -rf env
62 rm -rf dist docs/build
63 rm -rf .coverage htmlcov
64- find -name '__pycache__' -print0 | xargs -0 rm -rf
65- find -name '*.~*' -delete
66+ snapcraft clean
67+
68+fullclean: clean
69+ rm -rf dependencies
70
71
72-.PHONY: bootstrap test lint coverage clean snap docs
73+.PHONY: bootstrap test lint coverage clean fullclean snap install docs
74diff --git a/requirements.txt b/requirements.txt
75index 09ae15a..19a551f 100644
76--- a/requirements.txt
77+++ b/requirements.txt
78@@ -2,3 +2,4 @@ pymacaroons
79 pysha3
80 pyxdg
81 requests
82+snapstore_library
83\ No newline at end of file
84diff --git a/snap/plugins/_python/__init__.py b/snap/plugins/_python/__init__.py
85new file mode 100644
86index 0000000..6d6ad33
87--- /dev/null
88+++ b/snap/plugins/_python/__init__.py
89@@ -0,0 +1,19 @@
90+# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
91+#
92+# Copyright (C) 2017 Canonical Ltd
93+#
94+# This program is free software: you can redistribute it and/or modify
95+# it under the terms of the GNU General Public License version 3 as
96+# published by the Free Software Foundation.
97+#
98+# This program is distributed in the hope that it will be useful,
99+# but WITHOUT ANY WARRANTY; without even the implied warranty of
100+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
101+# GNU General Public License for more details.
102+#
103+# You should have received a copy of the GNU General Public License
104+# along with this program. If not, see <http://www.gnu.org/licenses/>.
105+
106+from ._pip import Pip # noqa
107+from ._python_finder import get_python_command # noqa
108+from ._sitecustomize import generate_sitecustomize # noqa
109diff --git a/snap/plugins/_python/_pip.py b/snap/plugins/_python/_pip.py
110new file mode 100644
111index 0000000..9acf3ba
112--- /dev/null
113+++ b/snap/plugins/_python/_pip.py
114@@ -0,0 +1,575 @@
115+# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
116+#
117+# Copyright (C) 2016-2019 Canonical Ltd
118+#
119+# This program is free software: you can redistribute it and/or modify
120+# it under the terms of the GNU General Public License version 3 as
121+# published by the Free Software Foundation.
122+#
123+# This program is distributed in the hope that it will be useful,
124+# but WITHOUT ANY WARRANTY; without even the implied warranty of
125+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
126+# GNU General Public License for more details.
127+#
128+# You should have received a copy of the GNU General Public License
129+# along with this program. If not, see <http://www.gnu.org/licenses/>.
130+
131+import collections
132+import contextlib
133+import json
134+import logging
135+import os
136+import re
137+import shutil
138+import stat
139+import subprocess
140+import sys
141+import tempfile
142+from typing import List, Optional, Sequence, Set
143+
144+import snapcraft
145+from snapcraft import file_utils
146+from snapcraft.internal import mangling
147+from ._python_finder import get_python_command, get_python_headers, get_python_home
148+from . import errors
149+
150+logger = logging.getLogger(__name__)
151+
152+
153+def _process_common_args(
154+ *, constraints: Optional[Set[str]] = None, process_dependency_links: bool = False
155+) -> List[str]:
156+ args = []
157+ if constraints:
158+ for constraint in constraints:
159+ args.extend(["--constraint", constraint])
160+
161+ if process_dependency_links:
162+ args.append("--process-dependency-links")
163+
164+ return args
165+
166+
167+def _process_package_args(
168+ *, packages: Sequence[str], requirements: Sequence[str], setup_py_dir: str
169+) -> List[str]:
170+ args = []
171+ if requirements:
172+ for requirement in requirements:
173+ args.extend(["--requirement", requirement])
174+
175+ if packages:
176+ args.extend(packages)
177+
178+ if setup_py_dir:
179+ args.append(".")
180+
181+ return args
182+
183+
184+def _replicate_owner_mode(path):
185+ # Don't bother with a path that doesn't exist or is a symlink. The target
186+ # of the symlink will either be updated anyway, or we won't have permission
187+ # to change it.
188+ if not os.path.exists(path) or os.path.islink(path):
189+ return
190+
191+ file_mode = os.stat(path).st_mode
192+
193+ # We at least need to write to it to fix shebangs later
194+ new_mode = file_mode | stat.S_IWUSR
195+
196+ # If the owner can execute it, so should everyone else.
197+ if file_mode & stat.S_IXUSR:
198+ new_mode |= stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH
199+
200+ # If the owner can read it, so should everyone else
201+ if file_mode & stat.S_IRUSR:
202+ new_mode |= stat.S_IRUSR | stat.S_IRGRP | stat.S_IROTH
203+
204+ os.chmod(path, new_mode)
205+
206+
207+def _fix_permissions(path):
208+ for root, dirs, files in os.walk(path):
209+ for filename in files:
210+ _replicate_owner_mode(os.path.join(root, filename))
211+ for dirname in dirs:
212+ _replicate_owner_mode(os.path.join(root, dirname))
213+
214+
215+def _expand_vars(text, project_dir):
216+ text = text.replace('$SNAPCRAFT_PROJECT_DIR', project_dir)
217+ text = text.replace('${SNAPCRAFT_PROJECT_DIR}', project_dir)
218+ return text
219+
220+
221+class Pip:
222+ """Wrapper for pip abstracting the args necessary for use in a part.
223+
224+ This class takes care of fetching pip, setuptools, and wheel, and then
225+ simply shells out to pip with the magical arguments necessary to install
226+ packages into a part.
227+
228+ Of particular importance: packages must be downloaded (via download())
229+ before they can be installed or have wheels built.
230+ """
231+
232+ def __init__(self, *, python_major_version, part_dir, install_dir, stage_dir, project_dir):
233+ """Initialize pip.
234+
235+ You must call setup() before you can actually use pip.
236+
237+ :param str python_major_version: The python major version to find (2 or
238+ 3)
239+ :param str part_dir: Path to the part's working area
240+ :param str install_dir: Path to the part's install area
241+ :param str stage_dir: Path to the staging area
242+ :param str project_dir: Path to the project area
243+
244+ :raises MissingPythonCommandError: If no python could be found in the
245+ staging or part's install area.
246+ """
247+ self._python_major_version = python_major_version
248+ self._install_dir = install_dir
249+ self._stage_dir = stage_dir
250+ self._project_dir = project_dir
251+
252+ self._python_package_dir = os.path.join(part_dir, "python-packages")
253+ os.makedirs(self._python_package_dir, exist_ok=True)
254+
255+ self.__python_command = None # type:str
256+ self.__python_home = None # type: str
257+
258+ @property
259+ def _python_command(self):
260+ """Lazily determine the python command required."""
261+ if not self.__python_command:
262+ self.__python_command = get_python_command(
263+ self._python_major_version,
264+ stage_dir=self._stage_dir,
265+ install_dir=self._install_dir,
266+ )
267+ return self.__python_command
268+
269+ @property
270+ def _python_home(self):
271+ """Lazily determine the correct python home."""
272+ if not self.__python_home:
273+ self.__python_home = get_python_home(
274+ self._python_major_version,
275+ stage_dir=self._stage_dir,
276+ install_dir=self._install_dir,
277+ )
278+ return self.__python_home
279+
280+ def setup(
281+ self,
282+ no_index: bool = False,
283+ find_links: Optional[Sequence[str]] = None
284+ ):
285+ """Install pip and dependencies.
286+
287+ Check to see if pip has already been installed. If not, fetch pip,
288+ setuptools, and wheel, and install them so they can be used.
289+ """
290+
291+ self._ensure_pip_installed(
292+ no_index=no_index,
293+ find_links=find_links
294+ )
295+ self._ensure_wheel_installed(
296+ no_index=no_index,
297+ find_links=find_links
298+ )
299+ self._ensure_setuptools_installed(
300+ no_index=no_index,
301+ find_links=find_links
302+ )
303+
304+ def is_setup(self):
305+ """Return true if this class has already been setup."""
306+
307+ return (
308+ self._is_pip_installed()
309+ and self._is_wheel_installed()
310+ and self._is_setuptools_installed()
311+ )
312+
313+ def _ensure_pip_installed(
314+ self,
315+ no_index: bool = False,
316+ find_links: Optional[Sequence[str]] = None
317+ ):
318+ # Check to see if we have our own pip. If not, we need to use the pip
319+ # on the host (installed via build-packages) to grab our own.
320+ if not self._is_pip_installed():
321+ logger.info("Fetching and installing pip...")
322+
323+ real_python_home = self.__python_home
324+
325+ # Make sure we're using pip from the host. Wrapping this operation
326+ # in a try/finally to make sure we revert away from the host's
327+ # python at the end.
328+ try:
329+ self.__python_home = os.path.join(os.path.sep, "usr")
330+
331+ # Using the host's pip, install our own pip
332+ self.download(
333+ {"pip"},
334+ no_index=no_index,
335+ find_links=find_links
336+ )
337+ self.install({"pip"}, ignore_installed=True)
338+ finally:
339+ # Now that we have our own pip, reset the python home
340+ self.__python_home = real_python_home
341+
342+ def _ensure_wheel_installed(
343+ self,
344+ no_index: bool = False,
345+ find_links: Optional[Sequence[str]] = None
346+ ):
347+ if not self._is_wheel_installed():
348+ logger.info("Fetching and installing wheel...")
349+ self.download(
350+ {"wheel"},
351+ no_index=no_index,
352+ find_links=find_links
353+ )
354+ self.install({"wheel"}, ignore_installed=True)
355+
356+ def _ensure_setuptools_installed(
357+ self,
358+ no_index: bool = False,
359+ find_links: Optional[Sequence[str]] = None
360+ ):
361+ if not self._is_setuptools_installed():
362+ logger.info("Fetching and installing setuptools...")
363+ self.download(
364+ {"setuptools"},
365+ no_index=no_index,
366+ find_links=find_links
367+ )
368+ self.install({"setuptools"}, ignore_installed=True)
369+
370+ def _is_pip_installed(self):
371+ try:
372+ # We're expecting an error here at least once complaining about
373+ # pip not being installed. In order to verify that the error is the
374+ # one we think it is, we need to process the stderr. So we'll
375+ # redirect it to stdout. If it's not the error we expect, something
376+ # is wrong, so re-raise it.
377+ #
378+ # Using _run_output here so stdout doesn't get printed to the
379+ # terminal.
380+ self._run_output([], stderr=subprocess.STDOUT)
381+ except subprocess.CalledProcessError as e:
382+ output = e.output.decode(sys.getfilesystemencoding()).strip()
383+ if "no module named pip" in output.lower():
384+ return False
385+ else:
386+ raise e
387+ return True
388+
389+ def _is_wheel_installed(self):
390+ return "wheel" in self.list()
391+
392+ def _is_setuptools_installed(self):
393+ return "setuptools" in self.list()
394+
395+ def download(
396+ self,
397+ packages,
398+ *,
399+ setup_py_dir: Optional[str] = None,
400+ constraints: Optional[Set[str]] = None,
401+ requirements: Optional[Sequence[str]] = None,
402+ process_dependency_links: bool = False,
403+ no_index: bool = False,
404+ find_links: Optional[Sequence[str]] = None
405+ ):
406+ """Download packages into cache, but do not install them.
407+
408+ :param iterable packages: Packages to download from index.
409+ :param str setup_py_dir: Directory containing setup.py.
410+ :param iterable constraints: Collection of paths to constraints files.
411+ :param iterable requirements: Collection of paths to requirements
412+ files.
413+ :param boolean process_dependency_links: Enable the processing of
414+ dependency links.
415+ :param boolean no_index: Don't hit PyPI
416+ :param iterable find_links: Locations of additional repositories
417+ """
418+ package_args = _process_package_args(
419+ packages=packages, requirements=requirements, setup_py_dir=setup_py_dir
420+ )
421+
422+ if not package_args:
423+ return # No operation was requested
424+
425+ args = _process_common_args(
426+ process_dependency_links=process_dependency_links, constraints=constraints
427+ )
428+ if no_index:
429+ args.append('--no-index')
430+ if find_links:
431+ for fl_item in find_links:
432+ args.append('--find-links')
433+ args.append(_expand_vars(fl_item, self._project_dir))
434+
435+ # Using pip with a few special parameters:
436+ #
437+ # --disable-pip-version-check: Don't whine if pip is out-of-date with
438+ # the version on pypi.
439+ # --dest: Download packages into the directory we've set aside for it.
440+ #
441+ # For cwd, setup_py_dir will be the actual directory we need to be in
442+ # or None.
443+ self._run(
444+ [
445+ "download",
446+ "--disable-pip-version-check",
447+ "--dest",
448+ self._python_package_dir,
449+ ]
450+ + args
451+ + package_args,
452+ cwd=setup_py_dir,
453+ )
454+
455+ def install(
456+ self,
457+ packages,
458+ *,
459+ setup_py_dir: Optional[str] = None,
460+ constraints: Optional[Set[str]] = None,
461+ requirements: Optional[Sequence[str]] = None,
462+ process_dependency_links: bool = False,
463+ upgrade: bool = False,
464+ install_deps: bool = True,
465+ ignore_installed: bool = False
466+ ):
467+ """Install packages from cache.
468+
469+ The packages should have already been downloaded via `download()`.
470+
471+ :param iterable packages: Packages to install from cache.
472+ :param str setup_py_dir: Directory containing setup.py.
473+ :param iterable constraints: Collection of paths to constraints files.
474+ :param iterable requirements: Collection of paths to requirements
475+ files.
476+ :param boolean process_dependency_links: Enable the processing of
477+ dependency links.
478+ :param boolean upgrade: Recursively upgrade packages.
479+ :param boolean install_deps: Install package dependencies.
480+ :param boolean ignore_installed: Reinstall packages if they're already
481+ installed
482+ """
483+ package_args = _process_package_args(
484+ packages=packages, requirements=requirements, setup_py_dir=setup_py_dir
485+ )
486+
487+ if not package_args:
488+ return # No operation was requested
489+
490+ args = _process_common_args(
491+ process_dependency_links=process_dependency_links, constraints=constraints
492+ )
493+
494+ if upgrade:
495+ args.append("--upgrade")
496+
497+ if not install_deps:
498+ args.append("--no-deps")
499+
500+ if ignore_installed:
501+ args.append("--ignore-installed")
502+
503+ # Using pip with a few special parameters:
504+ #
505+ # --user: Install packages to PYTHONUSERBASE, which we've pointed to
506+ # the installdir.
507+ # --no-index: Don't hit pypi, assume the packages are already
508+ # downloaded (i.e. by using `self.download()`)
509+ # --find-links: Provide the directory into which the packages should
510+ # have already been fetched
511+ #
512+ # For cwd, setup_py_dir will be the actual directory we need to be in
513+ # or None.
514+ self._run(
515+ [
516+ "install",
517+ "--user",
518+ "--no-index",
519+ "--find-links",
520+ self._python_package_dir,
521+ ]
522+ + args
523+ + package_args,
524+ cwd=setup_py_dir,
525+ )
526+
527+ # Installing with --user results in a directory with 700 permissions.
528+ # We need it a bit more open than that, so open it up.
529+ _fix_permissions(self._install_dir)
530+
531+ # Fix all shebangs to use the in-snap python.
532+ mangling.rewrite_python_shebangs(self._install_dir)
533+
534+ def wheel(
535+ self,
536+ packages,
537+ *,
538+ setup_py_dir: Optional[str] = None,
539+ constraints: Optional[Set[str]] = None,
540+ requirements: Optional[Sequence[str]] = None,
541+ process_dependency_links: bool = False
542+ ):
543+ """Build wheels of packages in the cache.
544+
545+ The packages should have already been downloaded via `download()`.
546+
547+ :param iterable packages: Packages in cache for which to build wheels.
548+ :param str setup_py_dir: Directory containing setup.py.
549+ :param iterable constraints: Collection of paths to constraints files.
550+ :param iterable requirements: Collection of paths to requirements
551+ files.
552+ :param boolean process_dependency_links: Enable the processing of
553+ dependency links.
554+
555+ :return: List of paths to each wheel that was built.
556+ :rtype: list
557+ """
558+ package_args = _process_package_args(
559+ packages=packages, requirements=requirements, setup_py_dir=setup_py_dir
560+ )
561+
562+ if not package_args:
563+ return [] # No operation was requested
564+
565+ args = _process_common_args(
566+ process_dependency_links=process_dependency_links, constraints=constraints
567+ )
568+
569+ wheels = [] # type: List[str]
570+ with tempfile.TemporaryDirectory() as temp_dir:
571+
572+ # Using pip with a few special parameters:
573+ #
574+ # --no-index: Don't hit pypi, assume the packages are already
575+ # downloaded (i.e. by using `self.download()`)
576+ # --find-links: Provide the directory into which the packages
577+ # should have already been fetched
578+ # --wheel-dir: Build wheels into a temporary working area rather
579+ # rather than cwd. We'll copy them over. FIXME: We can
580+ # probably get away just building them in the package
581+ # dir. Try that once this refactor has been validated.
582+ self._run(
583+ [
584+ "wheel",
585+ "--no-index",
586+ "--find-links",
587+ self._python_package_dir,
588+ "--wheel-dir",
589+ temp_dir,
590+ ]
591+ + args
592+ + package_args,
593+ cwd=setup_py_dir,
594+ )
595+ wheels = os.listdir(temp_dir)
596+ for wheel in wheels:
597+ file_utils.link_or_copy(
598+ os.path.join(temp_dir, wheel),
599+ os.path.join(self._python_package_dir, wheel),
600+ )
601+
602+ return [os.path.join(self._python_package_dir, wheel) for wheel in wheels]
603+
604+ def list(self, *, user=False):
605+ """Determine which packages have been installed.
606+
607+ :param boolean user: Whether or not to limit results to user base.
608+
609+ :return: Dict of installed python packages and their versions
610+ :rtype: dict
611+ """
612+ command = ["list"]
613+ if user:
614+ command.append("--user")
615+
616+ packages = collections.OrderedDict()
617+ try:
618+ output = self._run_output(command + ["--format=json"])
619+ json_output = json.loads(output, object_pairs_hook=collections.OrderedDict)
620+ except subprocess.CalledProcessError:
621+ # --format requires a newer pip, so fall back to legacy output
622+ output = self._run_output(command)
623+ json_output = [] # type: List[Dict[str, str]]
624+ version_regex = re.compile(r"\((.+)\)")
625+ for line in output.splitlines():
626+ line = line.split()
627+ m = version_regex.search(line[1])
628+ if not m:
629+ raise errors.PipListInvalidLegacyFormatError(output)
630+ json_output.append({"name": line[0], "version": m.group(1)})
631+ except json.decoder.JSONDecodeError as e:
632+ raise errors.PipListInvalidJsonError(output) from e
633+
634+ for package in json_output:
635+ if "name" not in package:
636+ raise errors.PipListMissingFieldError("name", output)
637+ if "version" not in package:
638+ raise errors.PipListMissingFieldError("version", output)
639+ packages[package["name"]] = package["version"]
640+ return packages
641+
642+ def clean_packages(self):
643+ """Remove the package cache."""
644+ with contextlib.suppress(FileNotFoundError):
645+ shutil.rmtree(self._python_package_dir)
646+
647+ def env(self):
648+ """The environment used by pip.
649+
650+ This function is only useful if you happen to need to call into pip's
651+ environment without using the API otherwise made available here (e.g.
652+ calling the setup.py directly instead of with pip).
653+
654+ :return: Dict of the environment necessary to use the pip contained
655+ here.
656+ :rtype: dict
657+ """
658+ env = os.environ.copy()
659+ env["PYTHONUSERBASE"] = self._install_dir
660+ env["PYTHONHOME"] = self._python_home
661+
662+ env["PATH"] = "{}:{}".format(
663+ os.path.join(self._install_dir, "usr", "bin"), os.path.expandvars("$PATH")
664+ )
665+
666+ headers = get_python_headers(
667+ self._python_major_version, stage_dir=self._stage_dir
668+ )
669+ if headers:
670+ current_cppflags = env.get("CPPFLAGS", "")
671+ env["CPPFLAGS"] = "-I{}".format(headers)
672+ if current_cppflags:
673+ env["CPPFLAGS"] = "{} {}".format(env["CPPFLAGS"], current_cppflags)
674+
675+ return env
676+
677+ def _run(self, args, runner=None, **kwargs):
678+ env = self.env()
679+
680+ # Using None as the default value instead of common.run so we can mock
681+ # common.run.
682+ if runner is None:
683+ runner = snapcraft.internal.common.run
684+ return runner(
685+ [self._python_command, "-m", "pip"] + list(args), env=env, **kwargs
686+ )
687+
688+ def _run_output(self, args, **kwargs):
689+ return self._run(args, runner=snapcraft.internal.common.run_output, **kwargs)
690diff --git a/snap/plugins/_python/_python_finder.py b/snap/plugins/_python/_python_finder.py
691new file mode 100644
692index 0000000..c677cd6
693--- /dev/null
694+++ b/snap/plugins/_python/_python_finder.py
695@@ -0,0 +1,104 @@
696+# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
697+#
698+# Copyright (C) 2017 Canonical Ltd
699+#
700+# This program is free software: you can redistribute it and/or modify
701+# it under the terms of the GNU General Public License version 3 as
702+# published by the Free Software Foundation.
703+#
704+# This program is distributed in the hope that it will be useful,
705+# but WITHOUT ANY WARRANTY; without even the implied warranty of
706+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
707+# GNU General Public License for more details.
708+#
709+# You should have received a copy of the GNU General Public License
710+# along with this program. If not, see <http://www.gnu.org/licenses/>.
711+
712+import os
713+import glob
714+
715+from . import errors
716+
717+
718+def get_python_command(python_major_version, *, stage_dir, install_dir):
719+ """Find the python command to use, preferring staged over the part.
720+
721+ We prefer the staged python as opposed to the in-part python in order to
722+ support one part that supplies python, with another part built `after` it
723+ wanting to use its python.
724+
725+ :param str python_major_version: The python major version to find (2 or 3)
726+ :param str stage_dir: Path to the staging area
727+ :param str install_dir: Path to the part's install area
728+
729+ :return: Path to the python command that was found
730+ :rtype: str
731+
732+ :raises MissingPythonCommandError: If no python could be found in the
733+ staging or part's install area.
734+ """
735+ python_command_name = "python{}".format(python_major_version)
736+ python_command = os.path.join("usr", "bin", python_command_name)
737+ staged_python = os.path.join(stage_dir, python_command)
738+ part_python = os.path.join(install_dir, python_command)
739+
740+ if os.path.exists(staged_python):
741+ return staged_python
742+ elif os.path.exists(part_python):
743+ return part_python
744+ else:
745+ raise errors.MissingPythonCommandError(
746+ python_command_name, [stage_dir, install_dir]
747+ )
748+
749+
750+def get_python_headers(python_major_version, *, stage_dir):
751+ """Find the python headers to use, if any, preferring staged over the host.
752+
753+ We want to make sure we use the headers from the staging area if available,
754+ or we may end up building for an older python version than the one we
755+ actually want to use.
756+
757+ :param str python_major_version: The python version to find (2 or 3)
758+ :param str stage_dir: Path to the staging area
759+
760+ :return: Path to the python headers that were found ('' if none)
761+ :rtype: str
762+ """
763+ python_command_name = "python{}".format(python_major_version)
764+ base_match = os.path.join("usr", "include", "{}*".format(python_command_name))
765+ staged_python = glob.glob(os.path.join(stage_dir, base_match))
766+ host_python = glob.glob(os.path.join(os.path.sep, base_match))
767+
768+ if staged_python:
769+ return staged_python[0]
770+ elif host_python:
771+ return host_python[0]
772+ else:
773+ return ""
774+
775+
776+def get_python_home(python_major_version, *, stage_dir, install_dir):
777+ """Find the correct PYTHONHOME, preferring staged over the part.
778+
779+ We prefer the staged python as opposed to the in-part python in order to
780+ support one part that supplies python, with another part built `after` it
781+ wanting to use its python.
782+
783+ :param str python_major_version: The python major version to find (2 or 3)
784+ :param str stage_dir: Path to the staging area
785+ :param str install_dir: Path to the part's install area
786+
787+ :return: Path to the PYTHONHOME that was found
788+ :rtype: str
789+
790+ :raises MissingPythonCommandError: If no python could be found in the
791+ staging or part's install area.
792+ """
793+ python_command = get_python_command(
794+ python_major_version, stage_dir=stage_dir, install_dir=install_dir
795+ )
796+ if python_command.startswith(stage_dir):
797+ return os.path.join(stage_dir, "usr")
798+ else:
799+ return os.path.join(install_dir, "usr")
800diff --git a/snap/plugins/_python/_sitecustomize.py b/snap/plugins/_python/_sitecustomize.py
801new file mode 100644
802index 0000000..aed89ea
803--- /dev/null
804+++ b/snap/plugins/_python/_sitecustomize.py
805@@ -0,0 +1,124 @@
806+# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
807+#
808+# Copyright (C) 2017 Canonical Ltd
809+#
810+# This program is free software: you can redistribute it and/or modify
811+# it under the terms of the GNU General Public License version 3 as
812+# published by the Free Software Foundation.
813+#
814+# This program is distributed in the hope that it will be useful,
815+# but WITHOUT ANY WARRANTY; without even the implied warranty of
816+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
817+# GNU General Public License for more details.
818+#
819+# You should have received a copy of the GNU General Public License
820+# along with this program. If not, see <http://www.gnu.org/licenses/>.
821+
822+import contextlib
823+import glob
824+import os
825+from textwrap import dedent
826+
827+from ._python_finder import get_python_command
828+from . import errors
829+
830+_SITECUSTOMIZE_TEMPLATE = dedent(
831+ """\
832+ import site
833+ import os
834+
835+ snap_dir = os.getenv("SNAP")
836+ snapcraft_stage_dir = os.getenv("SNAPCRAFT_STAGE")
837+ snapcraft_part_install = os.getenv("SNAPCRAFT_PART_INSTALL")
838+
839+ for d in (snap_dir, snapcraft_stage_dir, snapcraft_part_install):
840+ if d:
841+ site_dir = os.path.join(d, "{site_dir}")
842+ site.addsitedir(site_dir)
843+
844+ if snap_dir:
845+ site.ENABLE_USER_SITE = False"""
846+)
847+
848+
849+def _get_user_site_dir(python_major_version, *, install_dir):
850+ path_glob = os.path.join(
851+ install_dir, "lib", "python{}*".format(python_major_version), "site-packages"
852+ )
853+ user_site_dirs = glob.glob(path_glob)
854+ if not user_site_dirs:
855+ raise errors.MissingUserSitePackagesError(path_glob)
856+
857+ return user_site_dirs[0][len(install_dir) + 1 :]
858+
859+
860+def _get_sitecustomize_path(python_major_version, *, stage_dir, install_dir):
861+ # Use the part's install_dir unless there's a python in the staging area
862+ base_dir = install_dir
863+ with contextlib.suppress(errors.MissingPythonCommandError):
864+ python_command = get_python_command(
865+ python_major_version, stage_dir=stage_dir, install_dir=install_dir
866+ )
867+ if python_command.startswith(stage_dir):
868+ base_dir = stage_dir
869+
870+ site_py_glob = os.path.join(
871+ base_dir, "usr", "lib", "python{}*".format(python_major_version), "site.py"
872+ )
873+ python_sites = glob.glob(site_py_glob)
874+ if not python_sites:
875+ raise errors.MissingSitePyError(site_py_glob)
876+
877+ python_site_dir = os.path.dirname(python_sites[0])
878+
879+ return os.path.join(
880+ install_dir, python_site_dir[len(base_dir) + 1 :], "sitecustomize.py"
881+ )
882+
883+
884+def generate_sitecustomize(python_major_version, *, stage_dir, install_dir):
885+ """Generate a sitecustomize.py to look in staging, part install, and snap.
886+
887+ This is done by checking the values of the environment variables $SNAP,
888+ $SNAPCRAFT_STAGE, and $SNAPCRAFT_PART_INSTALL. As a result, the same
889+ sitecustomize.py works to find packages at both build- and run-time.
890+
891+ :param str python_major_version: The python major version to use (2 or 3)
892+ :param str stage_dir: Path to the staging area
893+ :param str install_dir: Path to the part's install area
894+
895+ :raises MissingUserSitePackagesError: If no user site packages are found in
896+ install_dir/lib/pythonX*
897+ :raises MissingSitePyError: If no site.py can be found in either the
898+ staging area or the part install area.
899+ """
900+ sitecustomize_path = _get_sitecustomize_path(
901+ python_major_version, stage_dir=stage_dir, install_dir=install_dir
902+ )
903+ os.makedirs(os.path.dirname(sitecustomize_path), exist_ok=True)
904+
905+ # There may very well already be a sitecustomize.py already there. If so,
906+ # get rid of it. Is may be a symlink to another sitecustomize.py, in which
907+ # case, we'll get rid of that one as well.
908+ if os.path.islink(sitecustomize_path):
909+ target_path = os.path.realpath(sitecustomize_path)
910+
911+ # Only remove the target if it's contained within the install directory
912+ if target_path.startswith(os.path.abspath(install_dir) + os.sep):
913+ with contextlib.suppress(FileNotFoundError):
914+ os.remove(target_path)
915+
916+ with contextlib.suppress(FileNotFoundError):
917+ os.remove(sitecustomize_path)
918+
919+ # Create our sitecustomize. Python from the archives already has one
920+ # which is distro-specific and not needed here, so we truncate it if it's
921+ # already there.
922+ with open(sitecustomize_path, "w") as f:
923+ f.write(
924+ _SITECUSTOMIZE_TEMPLATE.format(
925+ site_dir=_get_user_site_dir(
926+ python_major_version, install_dir=install_dir
927+ )
928+ )
929+ )
930diff --git a/snap/plugins/_python/errors.py b/snap/plugins/_python/errors.py
931new file mode 100644
932index 0000000..87fbcfb
933--- /dev/null
934+++ b/snap/plugins/_python/errors.py
935@@ -0,0 +1,79 @@
936+# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
937+#
938+# Copyright (C) 2017 Canonical Ltd
939+#
940+# This program is free software: you can redistribute it and/or modify
941+# it under the terms of the GNU General Public License version 3 as
942+# published by the Free Software Foundation.
943+#
944+# This program is distributed in the hope that it will be useful,
945+# but WITHOUT ANY WARRANTY; without even the implied warranty of
946+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
947+# GNU General Public License for more details.
948+#
949+# You should have received a copy of the GNU General Public License
950+# along with this program. If not, see <http://www.gnu.org/licenses/>.
951+
952+import snapcraft.internal.errors
953+import snapcraft.formatting_utils
954+
955+
956+class PythonPluginError(snapcraft.internal.errors.SnapcraftError):
957+ pass
958+
959+
960+class MissingPythonCommandError(PythonPluginError):
961+
962+ fmt = "Unable to find {python_version}, searched: {search_paths}"
963+
964+ def __init__(self, python_version, search_paths):
965+ super().__init__(
966+ python_version=python_version,
967+ search_paths=snapcraft.formatting_utils.combine_paths(
968+ search_paths, "", ":"
969+ ),
970+ )
971+
972+
973+class MissingUserSitePackagesError(PythonPluginError):
974+
975+ fmt = "Unable to find user site packages: {site_dir_glob}"
976+
977+ def __init__(self, site_dir_glob):
978+ super().__init__(site_dir_glob=site_dir_glob)
979+
980+
981+class MissingSitePyError(PythonPluginError):
982+
983+ fmt = "Unable to find site.py: {site_py_glob}"
984+
985+ def __init__(self, site_py_glob):
986+ super().__init__(site_py_glob=site_py_glob)
987+
988+
989+class PipListInvalidLegacyFormatError(PythonPluginError):
990+
991+ fmt = (
992+ "Failed to parse Python package list: "
993+ "The returned output is not in the expected format:\n"
994+ "{output}"
995+ )
996+
997+ def __init__(self, output):
998+ super().__init__(output=output)
999+
1000+
1001+class PipListInvalidJsonError(PythonPluginError):
1002+
1003+ fmt = "Pip packages output isn't valid json: {json!r}"
1004+
1005+ def __init__(self, json):
1006+ super().__init__(json=json)
1007+
1008+
1009+class PipListMissingFieldError(PythonPluginError):
1010+
1011+ fmt = "Pip packages json missing {field!r} field: {json!r}"
1012+
1013+ def __init__(self, field, json):
1014+ super().__init__(field=field, json=json)
1015diff --git a/snap/plugins/python_ols.py b/snap/plugins/python_ols.py
1016new file mode 100644
1017index 0000000..95598d1
1018--- /dev/null
1019+++ b/snap/plugins/python_ols.py
1020@@ -0,0 +1,499 @@
1021+# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
1022+#
1023+# Copyright (C) 2016-2018 Canonical Ltd
1024+#
1025+# This program is free software: you can redistribute it and/or modify
1026+# it under the terms of the GNU General Public License version 3 as
1027+# published by the Free Software Foundation.
1028+#
1029+# This program is distributed in the hope that it will be useful,
1030+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1031+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1032+# GNU General Public License for more details.
1033+#
1034+# You should have received a copy of the GNU General Public License
1035+# along with this program. If not, see <http://www.gnu.org/licenses/>.
1036+
1037+"""The python plugin can be used for python 2 or 3 based parts.
1038+
1039+It can be used for python projects where you would want to do:
1040+
1041+ - import python modules with a requirements.txt
1042+ - build a python project that has a setup.py
1043+ - install packages straight from pip
1044+
1045+This plugin uses the common plugin keywords as well as those for "sources".
1046+For more information check the 'plugins' topic for the former and the
1047+'sources' topic for the latter.
1048+
1049+Additionally, this plugin uses the following plugin-specific keywords:
1050+
1051+ - requirements:
1052+ (list of strings)
1053+ List of paths to requirements files.
1054+ - constraints:
1055+ (list of strings)
1056+ List of paths to constraint files.
1057+ - process-dependency-links:
1058+ (bool; default: false)
1059+ Enable the processing of dependency links in pip, which allow one
1060+ project to provide places to look for another project
1061+ - python-packages:
1062+ (list)
1063+ A list of dependencies to get from PyPI
1064+ - python-version:
1065+ (string; default: python3)
1066+ The python version to use. Valid options are: python2 and python3
1067+
1068+If the plugin finds a python interpreter with a basename that matches
1069+`python-version` in the <stage> directory on the following fixed path:
1070+`<stage-dir>/usr/bin/<python-interpreter>` then this interpreter would
1071+be preferred instead and no interpreter would be brought in through
1072+`stage-packages` mechanisms.
1073+"""
1074+
1075+import collections
1076+import contextlib
1077+import os
1078+import re
1079+from shutil import which
1080+from textwrap import dedent
1081+from typing import List, Set
1082+
1083+import requests
1084+
1085+import snapcraft
1086+from snapcraft.common import isurl
1087+from snapcraft.internal import errors, mangling
1088+from snapcraft.internal.errors import SnapcraftPluginCommandError
1089+import _python
1090+
1091+
1092+class UnsupportedPythonVersionError(snapcraft.internal.errors.SnapcraftError):
1093+
1094+ fmt = "Unsupported python version: {python_version!r}"
1095+
1096+
1097+class SnapcraftPluginPythonFileMissing(snapcraft.internal.errors.SnapcraftError):
1098+
1099+ fmt = (
1100+ "Failed to find the referred {plugin_property} file at the given "
1101+ "path: {plugin_property_value!r}.\n"
1102+ "Check the property and ensure the file exists."
1103+ )
1104+
1105+ def __init__(self, *, plugin_property, plugin_property_value):
1106+ super().__init__(
1107+ plugin_property=plugin_property, plugin_property_value=plugin_property_value
1108+ )
1109+
1110+
1111+class PythonPlugin(snapcraft.BasePlugin):
1112+ @classmethod
1113+ def schema(cls):
1114+ schema = super().schema()
1115+ schema["properties"]["requirements"] = {
1116+ "type": "array",
1117+ "minitems": 1,
1118+ "uniqueItems": True,
1119+ "items": {"type": "string"},
1120+ "default": [],
1121+ }
1122+ schema["properties"]["constraints"] = {
1123+ "type": "array",
1124+ "minitems": 1,
1125+ "uniqueItems": True,
1126+ "items": {"type": "string"},
1127+ "default": [],
1128+ }
1129+ schema["properties"]["python-packages"] = {
1130+ "type": "array",
1131+ "minitems": 1,
1132+ "uniqueItems": True,
1133+ "items": {"type": "string"},
1134+ "default": [],
1135+ }
1136+ schema["properties"]["process-dependency-links"] = {
1137+ "type": "boolean",
1138+ "default": False,
1139+ }
1140+ schema["properties"]["python-version"] = {
1141+ "type": "string",
1142+ "default": "python3",
1143+ "enum": ["python2", "python3"],
1144+ }
1145+ schema["properties"]["pip-no-index"] = {
1146+ "type": "boolean",
1147+ "default": False,
1148+ }
1149+ schema["properties"]["pip-find-links"] = {
1150+ "type": "array",
1151+ "items": {"type": "string"},
1152+ "default": [],
1153+ }
1154+ schema["anyOf"] = [{"required": ["source"]}, {"required": ["python-packages"]}]
1155+
1156+ return schema
1157+
1158+ @classmethod
1159+ def get_pull_properties(cls):
1160+ # Inform Snapcraft of the properties associated with pulling. If these
1161+ # change in the YAML Snapcraft will consider the pull step dirty.
1162+ return [
1163+ "requirements",
1164+ "constraints",
1165+ "python-packages",
1166+ "process-dependency-links",
1167+ "python-version",
1168+ "pip-no-index",
1169+ "pip-find-links",
1170+ ]
1171+
1172+ @property
1173+ def plugin_stage_packages(self):
1174+ if self.options.python_version == "python2":
1175+ python_base = "python"
1176+ elif self.options.python_version == "python3":
1177+ python_base = "python3"
1178+
1179+ if self.project.info.base in ("core", "core16", "core18"):
1180+ stage_packages = [python_base]
1181+ else:
1182+ stage_packages = []
1183+
1184+ if self.project.info.base == "core18" and python_base == "python3":
1185+ stage_packages.append("{}-distutils".format(python_base))
1186+
1187+ return stage_packages
1188+
1189+ # ignore mypy error: Read-only property cannot override read-write property
1190+ @property # type: ignore
1191+ def stage_packages(self):
1192+ try:
1193+ _python.get_python_command(
1194+ self._python_major_version,
1195+ stage_dir=self.project.stage_dir,
1196+ install_dir=self.installdir,
1197+ )
1198+ except _python.errors.MissingPythonCommandError:
1199+ return super().stage_packages + self.plugin_stage_packages
1200+ else:
1201+ return super().stage_packages
1202+
1203+ @property
1204+ def _pip(self):
1205+ if not self.__pip:
1206+ self.__pip = _python.Pip(
1207+ python_major_version=self._python_major_version,
1208+ part_dir=self.partdir,
1209+ install_dir=self.installdir,
1210+ stage_dir=self.project.stage_dir,
1211+ project_dir=self.project._project_dir,
1212+ )
1213+ return self.__pip
1214+
1215+ def __init__(self, name, options, project):
1216+ super().__init__(name, options, project)
1217+
1218+ self._setup_base_tools(project.info.base)
1219+
1220+ self._manifest = collections.OrderedDict()
1221+
1222+ # Pip requires only the major version of python rather than the command
1223+ # name like our option requires.
1224+ match = re.match(r"python(?P<major_version>\d).*", self.options.python_version)
1225+ if not match:
1226+ raise UnsupportedPythonVersionError(
1227+ python_version=self.options.python_version
1228+ )
1229+
1230+ self._python_major_version = match.group("major_version")
1231+ self.__pip = None
1232+
1233+ def _setup_base_tools(self, base):
1234+ # NOTE: stage-packages are lazily loaded.
1235+ if base in ("core", "core16", "core18"):
1236+ if self.options.python_version == "python3":
1237+ self.build_packages.extend(
1238+ [
1239+ "python3-dev",
1240+ "python3-pip",
1241+ "python3-pkg-resources",
1242+ "python3-setuptools",
1243+ ]
1244+ )
1245+ elif self.options.python_version == "python2":
1246+ self.build_packages.extend(
1247+ [
1248+ "python-dev",
1249+ "python-pip",
1250+ "python-pkg-resources",
1251+ "python-setuptools",
1252+ ]
1253+ )
1254+ else:
1255+ raise errors.PluginBaseError(part_name=self.name, base=base)
1256+
1257+ def pull(self):
1258+ super().pull()
1259+
1260+ self._pip.setup(
1261+ no_index=self.options.pip_no_index,
1262+ find_links=self.options.pip_find_links
1263+ )
1264+
1265+ with simple_env_bzr(os.path.join(self.installdir, "bin")):
1266+ # Download this project, using its setup.py if present. This will
1267+ # also download any python-packages requested.
1268+ self._download_project()
1269+
1270+ def clean_pull(self):
1271+ super().clean_pull()
1272+ self._pip.clean_packages()
1273+
1274+ def build(self):
1275+ super().build()
1276+
1277+ with simple_env_bzr(os.path.join(self.installdir, "bin")):
1278+ # Install the packages that have already been downloaded
1279+ installed_pipy_packages = self._install_project()
1280+
1281+ requirements = self._get_list_of_packages_from_property(
1282+ self.options.requirements
1283+ )
1284+ if requirements:
1285+ self._manifest["requirements-contents"] = requirements
1286+
1287+ constraints = self._get_list_of_packages_from_property(self.options.constraints)
1288+ if constraints:
1289+ self._manifest["constraints-contents"] = constraints
1290+
1291+ self._manifest["python-packages"] = [
1292+ "{}={}".format(name, installed_pipy_packages[name])
1293+ for name in installed_pipy_packages
1294+ ]
1295+
1296+ _python.generate_sitecustomize(
1297+ self._python_major_version,
1298+ stage_dir=self.project.stage_dir,
1299+ install_dir=self.installdir,
1300+ )
1301+
1302+ def _find_file(self, *, filename: str) -> str:
1303+ # source-subdir defaults to ''
1304+ for basepath in [self.builddir, self.sourcedir]:
1305+ if basepath == self.sourcedir:
1306+ # This is overwritten in the base plugin
1307+ # TODO add consistency
1308+ source_subdir = self.options.source_subdir
1309+ else:
1310+ source_subdir = ""
1311+ filepath = os.path.join(basepath, source_subdir, filename)
1312+ if os.path.exists(filepath):
1313+ return filepath
1314+
1315+ return None
1316+
1317+ def _get_setup_py_dir(self):
1318+ setup_py_dir = None
1319+ setup_py_path = self._find_file(filename="setup.py")
1320+ if setup_py_path:
1321+ setup_py_dir = os.path.dirname(setup_py_path)
1322+
1323+ return setup_py_dir
1324+
1325+ def _get_list_of_packages_from_property(self, property_list: Set[str]) -> List[str]:
1326+ """Return a sorted list of all packages found in property."""
1327+ package_list = list() # type: List[str]
1328+ for entry in property_list:
1329+ contents = self._get_file_contents(entry)
1330+ package_list.extend(contents.splitlines())
1331+ return package_list
1332+
1333+ def _get_normalized_property_set(
1334+ self, property_name, property_list: List[str]
1335+ ) -> Set[str]:
1336+ """Return a normalized set from a requirements or constraints list."""
1337+ normalized = set() # type: Set[str]
1338+ for entry in property_list:
1339+ if isurl(entry):
1340+ normalized.add(entry)
1341+ else:
1342+ entry_file = self._find_file(filename=entry)
1343+ if not entry_file:
1344+ raise SnapcraftPluginPythonFileMissing(
1345+ plugin_property=property_name, plugin_property_value=entry
1346+ )
1347+ normalized.add(entry_file)
1348+
1349+ return normalized
1350+
1351+ def _install_wheels(self, wheels):
1352+ installed = self._pip.list()
1353+ wheel_names = [os.path.basename(w).split("-")[0] for w in wheels]
1354+ # we want to avoid installing what is already provided in
1355+ # stage-packages
1356+ need_install = [k for k in wheel_names if k not in installed]
1357+ self._pip.install(
1358+ need_install,
1359+ upgrade=True,
1360+ install_deps=False,
1361+ process_dependency_links=self.options.process_dependency_links,
1362+ )
1363+
1364+ def _download_project(self):
1365+ constraints = self._get_normalized_property_set(
1366+ "constraints", self.options.constraints
1367+ )
1368+ requirements = self._get_normalized_property_set(
1369+ "requirements", self.options.requirements
1370+ )
1371+
1372+
1373+ self._pip.download(
1374+ self.options.python_packages,
1375+ setup_py_dir=None,
1376+ constraints=constraints,
1377+ requirements=requirements,
1378+ process_dependency_links=self.options.process_dependency_links,
1379+ no_index=self.options.pip_no_index,
1380+ find_links=self.options.pip_find_links,
1381+ )
1382+
1383+ def _install_project(self):
1384+ setup_py_dir = self._get_setup_py_dir()
1385+ constraints = self._get_normalized_property_set(
1386+ "constraints", self.options.constraints
1387+ )
1388+ requirements = self._get_normalized_property_set(
1389+ "requirements", self.options.requirements
1390+ )
1391+
1392+ # setup.py is handled in a different step as some projects may
1393+ # need to satisfy dependencies for setup.py to be parsed.
1394+ wheels = self._pip.wheel(
1395+ self.options.python_packages,
1396+ setup_py_dir=None,
1397+ constraints=constraints,
1398+ requirements=requirements,
1399+ process_dependency_links=self.options.process_dependency_links,
1400+ )
1401+
1402+ if wheels:
1403+ self._install_wheels(wheels)
1404+
1405+ if setup_py_dir is not None:
1406+ self._pip.download(
1407+ [],
1408+ setup_py_dir=setup_py_dir,
1409+ constraints=constraints,
1410+ requirements=set(),
1411+ process_dependency_links=self.options.process_dependency_links,
1412+ no_index=self.options.pip_no_index,
1413+ find_links=self.options.pip_find_links,
1414+ )
1415+ wheels = self._pip.wheel(
1416+ [],
1417+ setup_py_dir=setup_py_dir,
1418+ constraints=constraints,
1419+ requirements=set(),
1420+ process_dependency_links=self.options.process_dependency_links,
1421+ )
1422+
1423+ if wheels:
1424+ self._install_wheels(wheels)
1425+
1426+ setup_py_path = os.path.join(setup_py_dir, "setup.py")
1427+ if os.path.exists(setup_py_path):
1428+ # pbr and others don't work using `pip install .`
1429+ # LP: #1670852
1430+ # There is also a chance that this setup.py is distutils based
1431+ # in which case we will rely on the `pip install .` ran before
1432+ # this.
1433+ with contextlib.suppress(SnapcraftPluginCommandError):
1434+ self._setup_tools_install(setup_py_path)
1435+
1436+ return self._pip.list()
1437+
1438+ def _setup_tools_install(self, setup_file):
1439+ command = [
1440+ _python.get_python_command(
1441+ self._python_major_version,
1442+ stage_dir=self.project.stage_dir,
1443+ install_dir=self.installdir,
1444+ ),
1445+ os.path.basename(setup_file),
1446+ "--no-user-cfg",
1447+ "install",
1448+ "--single-version-externally-managed",
1449+ "--user",
1450+ "--record",
1451+ "install.txt",
1452+ ]
1453+ self.run(command, env=self._pip.env(), cwd=os.path.dirname(setup_file))
1454+
1455+ # Fix all shebangs to use the in-snap python. The stuff installed from
1456+ # pip has already been fixed, but anything done in this step has not.
1457+ mangling.rewrite_python_shebangs(self.installdir)
1458+
1459+ def _get_file_contents(self, path):
1460+ if isurl(path):
1461+ return requests.get(path).text
1462+ else:
1463+ file_path = os.path.join(self.sourcedir, path)
1464+ with open(file_path) as _file:
1465+ return _file.read()
1466+
1467+ def get_manifest(self):
1468+ return self._manifest
1469+
1470+ def snap_fileset(self):
1471+ fileset = super().snap_fileset()
1472+ fileset.append("-bin/pip")
1473+ fileset.append("-bin/pip2")
1474+ fileset.append("-bin/pip3")
1475+ fileset.append("-bin/pip2.7")
1476+ fileset.append("-bin/pip3.*")
1477+ fileset.append("-bin/easy_install*")
1478+ fileset.append("-bin/wheel")
1479+ # The RECORD files include hashes useful when uninstalling packages.
1480+ # In the snap they will cause conflicts when more than one part uses
1481+ # the python plugin.
1482+ fileset.append("-lib/python*/site-packages/*/RECORD")
1483+ return fileset
1484+
1485+
1486+@contextlib.contextmanager
1487+def simple_env_bzr(bin_dir):
1488+ """Create an appropriate environment to run bzr.
1489+
1490+ The python plugin sets up PYTHONUSERBASE and PYTHONHOME which
1491+ conflicts with bzr when using python3 as those two environment
1492+ variables will make bzr look for modules in the wrong location.
1493+ """
1494+ os.makedirs(bin_dir, exist_ok=True)
1495+ bzr_bin = os.path.join(bin_dir, "bzr")
1496+ real_bzr_bin = which("bzr")
1497+ if real_bzr_bin:
1498+ exec_line = 'exec {} "$@"'.format(real_bzr_bin)
1499+ else:
1500+ exec_line = "echo bzr needs to be in PATH; exit 1"
1501+ with open(bzr_bin, "w") as f:
1502+ f.write(
1503+ dedent(
1504+ """#!/bin/sh
1505+ unset PYTHONUSERBASE
1506+ unset PYTHONHOME
1507+ {}
1508+ """.format(
1509+ exec_line
1510+ )
1511+ )
1512+ )
1513+ os.chmod(bzr_bin, 0o777)
1514+ try:
1515+ yield
1516+ finally:
1517+ os.remove(bzr_bin)
1518+ if not os.listdir(bin_dir):
1519+ os.rmdir(bin_dir)
1520diff --git a/snap/snapcraft.yaml b/snap/snapcraft.yaml
1521index f234ee1..613affa 100644
1522--- a/snap/snapcraft.yaml
1523+++ b/snap/snapcraft.yaml
1524@@ -1,10 +1,10 @@
1525+
1526 name: snap-store-proxy-client
1527-version: 1.0
1528+version: "1.0"
1529 summary: Canonical snap store proxy administration client.
1530 description: |
1531 The Canonical snapstore client is used to manage a snap store proxy.
1532-
1533-# Defaults, but snapcraft prints ugly yellow messages without them.
1534+base: core
1535 confinement: strict
1536 grade: stable
1537
1538@@ -16,17 +16,24 @@ apps:
1539
1540 parts:
1541 deps:
1542- plugin: python
1543- requirements: ./requirements.txt
1544+ plugin: python-ols
1545+ source: .
1546+ pip-find-links:
1547+ - $SNAPCRAFT_PROJECT_DIR/dependencies
1548+ pip-no-index: true
1549+ requirements:
1550+ - ./requirements.txt
1551 stage-packages:
1552 - libsodium18
1553 source:
1554 plugin: dump
1555 source: .
1556+ override-build: |
1557+ #TODO: Would be a lot simpler to make snapstore_client normal
1558+ # Python package
1559+ cp -a $SNAPCRAFT_PROJECT_DIR/snapstore $SNAPCRAFT_PART_INSTALL
1560+ cp -a $SNAPCRAFT_PROJECT_DIR/snapstore_client $SNAPCRAFT_PART_INSTALL
1561 stage:
1562 - ./snapstore
1563 - ./snapstore_client/
1564- override-prime: |
1565- snapcraftctl prime
1566- /snap/core/current/usr/bin/python3 -m compileall -q -j0 snapstore_client
1567
1568diff --git a/snapstore b/snapstore
1569index cd67b71..4550e58 100755
1570--- a/snapstore
1571+++ b/snapstore
1572@@ -1,132 +1,7 @@
1573 #!/usr/bin/env python3
1574-# -*- coding: utf-8 -*-
1575-# Copyright 2017 Canonical Ltd. This software is licensed under the
1576-# GNU General Public License version 3 (see the file LICENSE).
1577-
1578-"""CLI utility to manage a store-in-a-box installation."""
1579-
1580-import argparse
1581-import logging
1582-import os
1583 import sys
1584
1585-from snapstore_client.cli import configure_logging
1586-from snapstore_client.logic.login import login
1587-from snapstore_client.logic.overrides import (
1588- delete_override,
1589- list_overrides,
1590- override,
1591-)
1592-from snapstore_client.logic.push import push_snap
1593-
1594-
1595-DEFAULT_SSO_URL = 'https://login.ubuntu.com/'
1596-
1597-
1598-def main():
1599- try:
1600- configure_logging()
1601- args = parse_args()
1602- return args.func(args) or 0
1603- except KeyboardInterrupt:
1604- # use default logger as we can't guarantee that configuration
1605- # has completed.
1606- logging.error("Operation cancelled")
1607- return 1
1608-
1609-
1610-def parse_args():
1611- parser = argparse.ArgumentParser()
1612- subparsers = parser.add_subparsers(help='sub-command help')
1613-
1614- login_parser = subparsers.add_parser(
1615- 'login', help='Sign into a store.',
1616- formatter_class=argparse.ArgumentDefaultsHelpFormatter)
1617- login_parser.add_argument('store_url', help='Store URL')
1618- login_parser.add_argument('email', help='Ubuntu One SSO email', nargs='?')
1619- login_parser.add_argument('--sso-url', help='Ubuntu One SSO URL',
1620- default=DEFAULT_SSO_URL)
1621- login_parser.add_argument('--offline', help="Use offline mode interaction",
1622- action='store_true')
1623- login_parser.set_defaults(func=login)
1624-
1625- list_overrides_parser = subparsers.add_parser(
1626- 'list-overrides', help='List channel map overrides.',
1627- )
1628- list_overrides_parser.add_argument(
1629- '--series', default='16',
1630- help='The series within which to list overrides.')
1631- list_overrides_parser.add_argument(
1632- 'snap_name',
1633- help='The name of the snap whose channel map should be listed.')
1634- list_overrides_parser.add_argument(
1635- '--password',
1636- help='Password for interacting with an offline proxy',
1637- default=os.environ.get('SNAP_PROXY_PASSWORD')
1638- )
1639- list_overrides_parser.set_defaults(func=list_overrides)
1640-
1641- override_parser = subparsers.add_parser(
1642- 'override', help='Set channel map overrides.',
1643- )
1644- override_parser.add_argument(
1645- '--series', default='16',
1646- help='The series within which to set overrides.')
1647- override_parser.add_argument(
1648- 'snap_name',
1649- help='The name of the snap whose channel map should be modified.')
1650- override_parser.add_argument(
1651- 'channel_map_entries', nargs='+', metavar='channel_map_entry',
1652- help='A channel map override, in the form <channel>=<revision>.')
1653- override_parser.add_argument(
1654- '--password',
1655- help='Password for interacting with an offline proxy',
1656- default=os.environ.get('SNAP_PROXY_PASSWORD')
1657- )
1658- override_parser.set_defaults(func=override)
1659-
1660- delete_override_parser = subparsers.add_parser(
1661- 'delete-override', help='Delete channel map overrides.',
1662- )
1663- delete_override_parser.add_argument(
1664- '--series', default='16',
1665- help='The series within which to delete overrides.')
1666- delete_override_parser.add_argument(
1667- 'snap_name',
1668- help='The name of the snap whose channel map should be modified.')
1669- delete_override_parser.add_argument(
1670- 'channels', nargs='+', metavar='channel',
1671- help='A channel whose overrides should be deleted.')
1672- delete_override_parser.add_argument(
1673- '--password',
1674- help='Password for interacting with an offline proxy',
1675- default=os.environ.get('SNAP_PROXY_PASSWORD')
1676- )
1677- delete_override_parser.set_defaults(func=delete_override)
1678-
1679- push_snap_parser = subparsers.add_parser(
1680- 'push-snap', help='push a snap to an offline proxy')
1681- push_snap_parser.add_argument(
1682- 'snap_tar_file',
1683- help='The .tar.gz file of a bundled downloaded snap')
1684- push_snap_parser.add_argument(
1685- '--push-channel-map',
1686- action='store_true',
1687- help="Force push of the channel map,"
1688- " removing any existing overrides")
1689- push_snap_parser.add_argument(
1690- '--password',
1691- help='Password for interacting with an offline proxy',
1692- default=os.environ.get('SNAP_PROXY_PASSWORD')
1693- )
1694- push_snap_parser.set_defaults(func=push_snap)
1695-
1696- if len(sys.argv) == 1:
1697- # Display help if no arguments are provided.
1698- parser.print_help()
1699- sys.exit(1)
1700-
1701- return parser.parse_args()
1702+from snapstore_client.__main__ import main
1703
1704
1705 if __name__ == '__main__':
1706diff --git a/snapstore_client/__main__.py b/snapstore_client/__main__.py
1707new file mode 100644
1708index 0000000..be6029a
1709--- /dev/null
1710+++ b/snapstore_client/__main__.py
1711@@ -0,0 +1,136 @@
1712+# Copyright 2017 Canonical Ltd. This software is licensed under the
1713+# GNU General Public License version 3 (see the file LICENSE).
1714+
1715+"""CLI utility to manage a store-in-a-box installation."""
1716+
1717+import argparse
1718+import logging
1719+import os
1720+import sys
1721+
1722+from snapstore_client.cli import configure_logging
1723+from snapstore_client.logic.login import login
1724+from snapstore_client.logic.overrides import (
1725+ delete_override,
1726+ list_overrides,
1727+ override,
1728+)
1729+from snapstore_client.logic.push import push_snap
1730+
1731+
1732+logger = logging.getLogger(__name__)
1733+
1734+
1735+DEFAULT_SSO_URL = 'https://login.ubuntu.com/'
1736+
1737+
1738+def main():
1739+ try:
1740+ configure_logging()
1741+ args = parse_args()
1742+ return args.func(args) or 0
1743+ except Exception as e:
1744+ logging.error("%s: %s", type(e).__name__, e)
1745+ except KeyboardInterrupt:
1746+ # use default logger as we can't guarantee that configuration
1747+ # has completed.
1748+ logging.error("Operation cancelled")
1749+ return 1
1750+
1751+
1752+def parse_args():
1753+ parser = argparse.ArgumentParser()
1754+ subparsers = parser.add_subparsers(help='sub-command help')
1755+
1756+ login_parser = subparsers.add_parser(
1757+ 'login', help='Sign into a store.',
1758+ formatter_class=argparse.ArgumentDefaultsHelpFormatter)
1759+ login_parser.add_argument('store_url', help='Store URL')
1760+ login_parser.add_argument('email', help='Ubuntu One SSO email', nargs='?')
1761+ login_parser.add_argument('--sso-url', help='Ubuntu One SSO URL',
1762+ default=DEFAULT_SSO_URL)
1763+ login_parser.add_argument('--offline', help="Use offline mode interaction",
1764+ action='store_true')
1765+ login_parser.set_defaults(func=login)
1766+
1767+ list_overrides_parser = subparsers.add_parser(
1768+ 'list-overrides', help='List channel map overrides.',
1769+ )
1770+ list_overrides_parser.add_argument(
1771+ '--series', default='16',
1772+ help='The series within which to list overrides.')
1773+ list_overrides_parser.add_argument(
1774+ 'snap_name',
1775+ help='The name of the snap whose channel map should be listed.')
1776+ list_overrides_parser.add_argument(
1777+ '--password',
1778+ help='Password for interacting with an offline proxy',
1779+ default=os.environ.get('SNAP_PROXY_PASSWORD')
1780+ )
1781+ list_overrides_parser.set_defaults(func=list_overrides)
1782+
1783+ override_parser = subparsers.add_parser(
1784+ 'override', help='Set channel map overrides.',
1785+ )
1786+ override_parser.add_argument(
1787+ '--series', default='16',
1788+ help='The series within which to set overrides.')
1789+ override_parser.add_argument(
1790+ 'snap_name',
1791+ help='The name of the snap whose channel map should be modified.')
1792+ override_parser.add_argument(
1793+ 'channel_map_entries', nargs='+', metavar='channel_map_entry',
1794+ help='A channel map override, in the form <channel>=<revision>.')
1795+ override_parser.add_argument(
1796+ '--password',
1797+ help='Password for interacting with an offline proxy',
1798+ default=os.environ.get('SNAP_PROXY_PASSWORD')
1799+ )
1800+ override_parser.set_defaults(func=override)
1801+
1802+ delete_override_parser = subparsers.add_parser(
1803+ 'delete-override', help='Delete channel map overrides.',
1804+ )
1805+ delete_override_parser.add_argument(
1806+ '--series', default='16',
1807+ help='The series within which to delete overrides.')
1808+ delete_override_parser.add_argument(
1809+ 'snap_name',
1810+ help='The name of the snap whose channel map should be modified.')
1811+ delete_override_parser.add_argument(
1812+ 'channels', nargs='+', metavar='channel',
1813+ help='A channel whose overrides should be deleted.')
1814+ delete_override_parser.add_argument(
1815+ '--password',
1816+ help='Password for interacting with an offline proxy',
1817+ default=os.environ.get('SNAP_PROXY_PASSWORD')
1818+ )
1819+ delete_override_parser.set_defaults(func=delete_override)
1820+
1821+ push_snap_parser = subparsers.add_parser(
1822+ 'push-snap', help='push a snap to an offline proxy')
1823+ push_snap_parser.add_argument(
1824+ 'snap_tar_file',
1825+ help='The .tar.gz file of a bundled downloaded snap')
1826+ push_snap_parser.add_argument(
1827+ '--push-channel-map',
1828+ action='store_true',
1829+ help="Force push of the channel map,"
1830+ " removing any existing overrides")
1831+ push_snap_parser.add_argument(
1832+ '--password',
1833+ help='Password for interacting with an offline proxy',
1834+ default=os.environ.get('SNAP_PROXY_PASSWORD')
1835+ )
1836+ push_snap_parser.set_defaults(func=push_snap)
1837+
1838+ if len(sys.argv) == 1:
1839+ # Display help if no arguments are provided.
1840+ parser.print_help()
1841+ sys.exit(1)
1842+
1843+ return parser.parse_args()
1844+
1845+
1846+if __name__ == '__main__':
1847+ sys.exit(main())
1848diff --git a/snapstore_client/exceptions.py b/snapstore_client/exceptions.py
1849deleted file mode 100644
1850index 061b5fe..0000000
1851--- a/snapstore_client/exceptions.py
1852+++ /dev/null
1853@@ -1,72 +0,0 @@
1854-# Copyright 2017 Canonical Ltd. This software is licensed under the
1855-# GNU General Public License version 3 (see the file LICENSE).
1856-
1857-"""Client exceptions."""
1858-
1859-
1860-class ClientError(Exception):
1861- """The base class for all user-visible exceptions."""
1862-
1863- @property
1864- def fmt(self):
1865- raise NotImplementedError
1866-
1867- def __init__(self, **kwargs):
1868- for key, value in kwargs.items():
1869- setattr(self, key, value)
1870-
1871- def __str__(self):
1872- return self.fmt.format(**self.__dict__)
1873-
1874-
1875-class InvalidCredentials(ClientError):
1876-
1877- fmt = 'Invalid credentials: {message}.'
1878-
1879- def __init__(self, message):
1880- super().__init__(message=message)
1881-
1882-
1883-class InvalidStoreURL(ClientError):
1884-
1885- fmt = 'Invalid store url: {message}.'
1886-
1887- def __init__(self, message):
1888- super().__init__(message=message)
1889-
1890-
1891-class StoreCommunicationError(ClientError):
1892-
1893- fmt = 'Connection error with the store using: {message}.'
1894-
1895- def __init__(self, message):
1896- super().__init__(message=message)
1897-
1898-
1899-class StoreMacaroonSSOMismatch(ClientError):
1900-
1901- fmt = 'Root macaroon does not refer to expected SSO host: {sso_host}.'
1902-
1903- def __init__(self, sso_host):
1904- super().__init__(sso_host=sso_host)
1905-
1906-
1907-class StoreAuthenticationError(ClientError):
1908-
1909- # No terminating full stop because the message from SSO sometimes
1910- # (though not always!) includes one.
1911- fmt = 'Authentication error: {message}'
1912-
1913- def __init__(self, message, extra=None):
1914- super().__init__(message=message, extra=extra)
1915-
1916-
1917-class StoreTwoFactorAuthenticationRequired(StoreAuthenticationError):
1918-
1919- def __init__(self):
1920- super().__init__('Two-factor authentication required.')
1921-
1922-
1923-class StoreMacaroonNeedsRefresh(ClientError):
1924-
1925- fmt = 'Authentication macaroon needs to be refreshed.'
1926diff --git a/snapstore_client/logic/login.py b/snapstore_client/logic/login.py
1927index 9550988..1eb92ef 100644
1928--- a/snapstore_client/logic/login.py
1929+++ b/snapstore_client/logic/login.py
1930@@ -2,86 +2,63 @@
1931
1932 import getpass
1933 import logging
1934-from urllib.parse import urlparse
1935+from urllib.parse import urljoin
1936
1937-from pymacaroons import Macaroon
1938+import requests
1939
1940-from snapstore_client import (
1941- config,
1942- exceptions,
1943- webservices as ws,
1944-)
1945+from snapstore_client import config
1946+from snapstore_library import exceptions, authentication
1947
1948
1949 logger = logging.getLogger(__name__)
1950
1951
1952-def _extract_caveat_id(sso_url, root_macaroon):
1953- macaroon = Macaroon.deserialize(root_macaroon)
1954- sso_host = urlparse(sso_url).netloc
1955- for caveat in macaroon.caveats:
1956- if caveat.location == sso_host:
1957- return caveat.caveat_id
1958- else:
1959- raise exceptions.StoreMacaroonSSOMismatch(sso_host)
1960+def set_offline(args):
1961+ store_name = "default"
1962+ cfg = config.Config()
1963+ store = cfg.store_section(store_name)
1964+ store.set("gw_url", args.store_url)
1965+ cfg.save()
1966+ logger.info("Configured %s as the %s store", args.store_url, store_name)
1967+ return 0
1968
1969
1970 def login(args):
1971- # TODO: validate these before using to avoid ugly errors.
1972- gw_url = args.store_url
1973- sso_url = args.sso_url
1974-
1975 if args.offline:
1976- cfg = config.Config()
1977- store = cfg.store_section('default')
1978- store.set('gw_url', gw_url)
1979- cfg.save()
1980- return
1981+ return set_offline(args)
1982
1983- logger.info('Enter your Ubuntu One SSO credentials.')
1984+ logger.info("Enter your Ubuntu One SSO credentials.")
1985 email = args.email
1986 if not email:
1987- email = input('Email: ')
1988- password = getpass.getpass('Password: ')
1989+ email = input("Email: ")
1990+ password = getpass.getpass("Password: ")
1991
1992+ session = requests.Session()
1993+ auth_url = urljoin(args.store_url, "/v2/auth/issue-store-admin")
1994+ sso_url = urljoin(args.sso_url, "/api/v2/tokens/discharge")
1995 try:
1996- root = ws.issue_store_admin(gw_url)
1997- except exceptions.ClientError as e:
1998- logger.error(str(e))
1999- return
2000- caveat_id = _extract_caveat_id(sso_url, root)
2001- try:
2002- try:
2003- unbound_discharge = ws.get_sso_discharge(
2004- sso_url, email, password, caveat_id)
2005- logger.info('Login successful')
2006- except exceptions.StoreTwoFactorAuthenticationRequired:
2007- one_time_password = input('Second-factor auth: ')
2008- unbound_discharge = ws.get_sso_discharge(
2009- sso_url, email, password, caveat_id,
2010- one_time_password=one_time_password)
2011- logger.info('Login successful')
2012- except exceptions.StoreAuthenticationError as e:
2013- logger.error('Login failed.')
2014- logger.error('%s', e)
2015- if e.extra:
2016- for key, value in e.extra.items():
2017- if isinstance(value, list):
2018- value = ' '.join(value)
2019- logger.error('%s: %s', key, value)
2020- return 1
2021+ root_macaroon, discharge_macaroon = authentication.login(
2022+ session, auth_url, sso_url, email, password
2023+ )
2024+ except exceptions.TwoFactorAuthenticationRequired:
2025+ one_time_password = input("Second-factor auth: ")
2026+ root_macaroon, discharge_macaroon = authentication.login(
2027+ session, auth_url, sso_url, email, password, otp=one_time_password
2028+ )
2029+ logger.info("Login successful")
2030
2031 cfg = config.Config()
2032 # For now, the store is always called "default". In the future we may want
2033 # to support multiple stores by allowing the user to provide a nice name
2034 # for a store at login that can be used to select the store for later
2035 # operations.
2036- store = cfg.store_section('default')
2037- store.set('gw_url', gw_url)
2038- store.set('sso_url', sso_url)
2039- store.set('root', root)
2040- store.set('unbound_discharge', unbound_discharge)
2041- store.set('email', email)
2042+ store_name = "default"
2043+ store = cfg.store_section(store_name)
2044+ store.set("gw_url", args.store_url)
2045+ store.set("sso_url", args.sso_url)
2046+ store.set("root", root_macaroon.serialize())
2047+ store.set("unbound_discharge", discharge_macaroon.serialize())
2048+ store.set("email", email)
2049 cfg.save()
2050-
2051+ logger.info("Configured %s as the %s store", args.store_url, store_name)
2052 return 0
2053diff --git a/snapstore_client/logic/overrides.py b/snapstore_client/logic/overrides.py
2054index d3c8356..aa6d490 100644
2055--- a/snapstore_client/logic/overrides.py
2056+++ b/snapstore_client/logic/overrides.py
2057@@ -1,51 +1,73 @@
2058 # Copyright 2017 Canonical Ltd.
2059
2060 import logging
2061+from urllib.parse import urljoin
2062
2063-from requests.exceptions import HTTPError
2064+import requests
2065
2066-from snapstore_client import (
2067- config,
2068- exceptions,
2069- webservices as ws,
2070-)
2071+from snapstore_client import config
2072 from snapstore_client.presentation_helpers import (
2073 channel_map_string_to_tuple,
2074 override_to_string,
2075 )
2076 from snapstore_client.utils import (
2077 _check_default_store,
2078- _log_authorized_error,
2079- _log_credentials_error,
2080 )
2081+from snapstore_library import authentication, overrides
2082
2083
2084 logger = logging.getLogger(__name__)
2085
2086
2087+# Used by the offline store
2088+HARDCODED_ADMIN = "admin"
2089+
2090+
2091+def _get_macaroon_auth(session, store):
2092+ refresh_url = urljoin(store.get("sso_url"), "/api/v2/tokens/refresh")
2093+ refresh_handler = authentication.MacaroonRefreshHandler(
2094+ session, refresh_url
2095+ )
2096+
2097+ def discharge_updater(discharge):
2098+ store.set("unbound_discharge", discharge)
2099+
2100+ return authentication.HTTPMacaroonAuth(
2101+ authentication.deserialize(store.get("root")),
2102+ authentication.deserialize(store.get("unbound_discharge")),
2103+ refresh_handler=refresh_handler,
2104+ dischage_update_handler=discharge_updater,
2105+ )
2106+
2107+
2108+def _get_auth(session, args, store):
2109+ if args.password:
2110+ auth = requests.auth.HTTPBasicAuth(HARDCODED_ADMIN, args.password)
2111+ else:
2112+ auth = _get_macaroon_auth(session, store)
2113+ return auth
2114+
2115+
2116+def _print_overrides(overrides_response):
2117+ for override in overrides_response["overrides"]:
2118+ logger.info(override_to_string(override))
2119+
2120+
2121 def list_overrides(args):
2122 cfg = config.Config()
2123 store = _check_default_store(cfg)
2124 if not store:
2125 return 1
2126
2127- try:
2128- if args.password:
2129- response = ws.get_overrides(
2130- store, args.snap_name,
2131- series=args.series, password=args.password)
2132- else:
2133- response = ws.refresh_if_necessary(
2134- store, ws.get_overrides,
2135- store, args.snap_name, series=args.series)
2136- except exceptions.InvalidCredentials as e:
2137- _log_credentials_error(e)
2138- return 1
2139- except HTTPError:
2140- _log_authorized_error()
2141- return 1
2142- for override in response['overrides']:
2143- logger.info(override_to_string(override))
2144+ session = requests.Session()
2145+ auth = _get_auth(session, args, store)
2146+ url = urljoin(
2147+ store.get("gw_url"), "/v2/metadata/overrides/{}".format(args.snap_name)
2148+ )
2149+ overrides_response = overrides.get_overrides(
2150+ session, url, args.snap_name, auth=auth
2151+ )
2152+ _print_overrides(overrides_response)
2153
2154
2155 def override(args):
2156@@ -54,31 +76,24 @@ def override(args):
2157 if not store:
2158 return 1
2159
2160- overrides = []
2161+ overrides_data = []
2162 for channel_map_entry in args.channel_map_entries:
2163 channel, revision = channel_map_string_to_tuple(channel_map_entry)
2164- overrides.append({
2165- 'snap_name': args.snap_name,
2166- 'revision': revision,
2167- 'channel': channel,
2168- 'series': args.series,
2169- })
2170- try:
2171- if args.password:
2172- response = ws.set_overrides(
2173- store, overrides, password=args.password)
2174- else:
2175- response = ws.refresh_if_necessary(
2176- store, ws.set_overrides,
2177- store, overrides)
2178- except exceptions.InvalidCredentials as e:
2179- _log_credentials_error(e)
2180- return 1
2181- except HTTPError:
2182- _log_authorized_error()
2183- return 1
2184- for override in response['overrides']:
2185- logger.info(override_to_string(override))
2186+ overrides_data.append(
2187+ {
2188+ "snap_name": args.snap_name,
2189+ "revision": revision,
2190+ "channel": channel,
2191+ "series": args.series,
2192+ }
2193+ )
2194+ session = requests.Session()
2195+ auth = _get_auth(session, args, store)
2196+ url = urljoin(store.get("gw_url"), "/v2/metadata/overrides")
2197+ overrides_response = overrides.set_overrides(
2198+ session, url, overrides_data, auth=auth
2199+ )
2200+ _print_overrides(overrides_response)
2201
2202
2203 def delete_override(args):
2204@@ -87,27 +102,20 @@ def delete_override(args):
2205 if not store:
2206 return 1
2207
2208- overrides = []
2209+ overrides_data = []
2210 for channel in args.channels:
2211- overrides.append({
2212- 'snap_name': args.snap_name,
2213- 'revision': None,
2214- 'channel': channel,
2215- 'series': args.series,
2216- })
2217- try:
2218- if args.password:
2219- response = ws.set_overrides(
2220- store, overrides, password=args.password)
2221- else:
2222- response = ws.refresh_if_necessary(
2223- store, ws.set_overrides,
2224- store, overrides)
2225- except exceptions.InvalidCredentials as e:
2226- _log_credentials_error(e)
2227- return 1
2228- except HTTPError:
2229- _log_authorized_error()
2230- return 1
2231- for override in response['overrides']:
2232- logger.info(override_to_string(override))
2233+ overrides_data.append(
2234+ {
2235+ "snap_name": args.snap_name,
2236+ "revision": None,
2237+ "channel": channel,
2238+ "series": args.series,
2239+ }
2240+ )
2241+ session = requests.Session()
2242+ auth = _get_auth(session, args, store)
2243+ url = urljoin(store.get("gw_url"), "/v2/metadata/overrides")
2244+ overrides_response = overrides.set_overrides(
2245+ session, url, overrides_data, auth=auth
2246+ )
2247+ _print_overrides(overrides_response)
2248diff --git a/snapstore_client/logic/push.py b/snapstore_client/logic/push.py
2249index abd06e8..be5fcd4 100644
2250--- a/snapstore_client/logic/push.py
2251+++ b/snapstore_client/logic/push.py
2252@@ -10,12 +10,10 @@ from requests.exceptions import HTTPError
2253
2254 from snapstore_client import (
2255 config,
2256- exceptions,
2257 )
2258 from snapstore_client.utils import (
2259 _check_default_store,
2260 _log_authorized_error,
2261- _log_credentials_error,
2262 )
2263
2264 logger = logging.getLogger(__name__)
2265@@ -298,9 +296,6 @@ def push_snap(args):
2266 logger.error(
2267 "This command only works with a supplied password"
2268 " and a proxy in offline mode")
2269- except exceptions.InvalidCredentials as e:
2270- _log_credentials_error(e)
2271- return 1
2272 except HTTPError:
2273 _log_authorized_error()
2274 return 1
2275diff --git a/snapstore_client/logic/tests/test_login.py b/snapstore_client/logic/tests/test_login.py
2276index d7854cc..f2fce6e 100644
2277--- a/snapstore_client/logic/tests/test_login.py
2278+++ b/snapstore_client/logic/tests/test_login.py
2279@@ -1,286 +1,124 @@
2280 # Copyright 2017 Canonical Ltd.
2281+from testtools.assertions import assert_that
2282+from testtools.matchers import Equals
2283+import pymacaroons
2284
2285-import json
2286-from unittest import mock
2287-from urllib.parse import urljoin, urlparse
2288+from snapstore_client.logic.tests.utils import RunMainTestCase
2289+from snapstore_library import exceptions
2290
2291-import fixtures
2292-from pymacaroons import Macaroon
2293-import responses
2294-from testtools import TestCase
2295-from testtools.matchers import (
2296- ContainsDict,
2297- Equals,
2298- MatchesDict,
2299-)
2300
2301-from snapstore_client import (
2302- config,
2303- exceptions,
2304-)
2305-from snapstore_client.logic.login import login
2306-from snapstore_client.tests import factory
2307-
2308-
2309-class LoginTests(TestCase):
2310-
2311- def setUp(self):
2312- super().setUp()
2313- self.default_gw_url = 'http://store.local/'
2314- self.default_sso_url = 'https://login.staging.ubuntu.com/'
2315- self.logger = self.useFixture(fixtures.FakeLogger())
2316- self.config_path = self.useFixture(fixtures.TempDir()).path
2317- self.useFixture(fixtures.MonkeyPatch(
2318- 'xdg.BaseDirectory.xdg_config_home', self.config_path))
2319- self.useFixture(fixtures.MonkeyPatch(
2320- 'xdg.BaseDirectory.xdg_config_dirs', [self.config_path]))
2321- self.mock_input = self.useFixture(fixtures.MockPatch(
2322- 'builtins.input')).mock
2323- self.mock_getpass = self.useFixture(fixtures.MockPatch(
2324- 'getpass.getpass')).mock
2325-
2326- def make_responses_callback(self, response_templates):
2327- full_responses = []
2328- for response in response_templates:
2329- status = response.get('status', 200)
2330- content_type = 'text/plain'
2331- if 'json' in response:
2332- content_type = 'application/json'
2333- body = json.dumps(response['json'])
2334- else:
2335- body = response.get('body')
2336- full_responses.append(
2337- (status, {'Content-Type': content_type}, body))
2338- iter_responses = iter(full_responses)
2339- return lambda request: next(iter_responses)
2340-
2341- def make_args(self, store_url=None,
2342- sso_url=None, email=None, offline=False):
2343- return factory.Args(
2344- store_url=store_url or self.default_gw_url,
2345- sso_url=sso_url or self.default_sso_url,
2346- email=email,
2347- offline=offline,
2348+class LoginTests(RunMainTestCase):
2349+ def test_offline_login(self):
2350+ self.mock_with_side_effect(
2351+ "snapstore_library.authentication.login",
2352+ AssertionError("should not be called"),
2353 )
2354
2355- def add_issue_store_admin_response(self, *response_templates, gw_url=None):
2356- gw_url = gw_url or self.default_gw_url
2357- issue_store_admin_url = urljoin(gw_url, '/v2/auth/issue-store-admin')
2358- responses.add_callback(
2359- 'POST', issue_store_admin_url,
2360- self.make_responses_callback(response_templates))
2361-
2362- def add_get_sso_discharge_response(self, *response_templates,
2363- sso_url=None):
2364- sso_url = sso_url or self.default_sso_url
2365- discharge_url = urljoin(sso_url, '/api/v2/tokens/discharge')
2366- responses.add_callback(
2367- 'POST', discharge_url,
2368- self.make_responses_callback(response_templates))
2369-
2370- def make_root_macaroon(self, sso_url=None):
2371- macaroon = Macaroon()
2372- sso_url = sso_url or self.default_sso_url
2373- sso_host = urlparse(sso_url).netloc
2374- macaroon.add_third_party_caveat(sso_host, 'key', 'payload')
2375- return macaroon.serialize()
2376+ exit_status = self.run_main(
2377+ "login", "http://store.local/", "--offline"
2378+ )
2379
2380- @responses.activate
2381- def test_login_sso_mismatch(self):
2382- self.mock_input.return_value = 'user@example.org'
2383- self.mock_getpass.return_value = 'secret'
2384- macaroon = Macaroon()
2385- macaroon.add_third_party_caveat('another.example.com', '', '')
2386- self.add_issue_store_admin_response(
2387- {'status': 200, 'json': {'macaroon': macaroon.serialize()}})
2388- self.add_get_sso_discharge_response({'status': 401})
2389- self.assertRaises(
2390- exceptions.StoreMacaroonSSOMismatch, login, self.make_args())
2391+ assert_that(exit_status, Equals(0))
2392+ assert_that(
2393+ self.logger.output,
2394+ Equals("Configured http://store.local/ as the default store\n"),
2395+ )
2396
2397- @responses.activate
2398- def test_login_sso_bad_email(self):
2399- self.mock_input.return_value = ''
2400- self.mock_getpass.return_value = ''
2401- self.add_issue_store_admin_response(
2402- {'status': 200, 'json': {'macaroon': self.make_root_macaroon()}})
2403- auth_error = {
2404- 'message': 'Invalid request data',
2405- 'extra': {'email': ['Enter a valid email address.']},
2406- }
2407- self.add_get_sso_discharge_response(
2408- {'status': 401, 'json': {'error_list': [auth_error]}})
2409- self.assertEqual(1, login(self.make_args()))
2410- self.assertEqual(
2411- 'Enter your Ubuntu One SSO credentials.\n'
2412- 'Login failed.\n'
2413- 'Authentication error: Invalid request data\n'
2414- 'email: Enter a valid email address.\n',
2415- self.logger.output)
2416+ def test_login(self):
2417+ root_macaroon = pymacaroons.Macaroon(location="test1")
2418+ discharge_macaroon = pymacaroons.Macaroon(location="test2")
2419+ self.mock_with_return(
2420+ "snapstore_library.authentication.login",
2421+ (root_macaroon, discharge_macaroon),
2422+ )
2423+ self.mock_with_return("getpass.getpass", "secret")
2424
2425- @responses.activate
2426- def test_login_sso_unauthorized(self):
2427- self.mock_input.return_value = 'user@example.org'
2428- self.mock_getpass.return_value = 'secret'
2429- self.add_issue_store_admin_response(
2430- {'status': 200, 'json': {'macaroon': self.make_root_macaroon()}})
2431- auth_error = {'message': 'Provided email/password is not correct.'}
2432- self.add_get_sso_discharge_response(
2433- {'status': 401, 'json': {'error_list': [auth_error]}})
2434- self.assertEqual(1, login(self.make_args()))
2435- self.assertEqual(
2436- 'Enter your Ubuntu One SSO credentials.\n'
2437- 'Login failed.\n'
2438- 'Authentication error: Provided email/password is not correct.\n',
2439- self.logger.output)
2440+ exit_status = self.run_main(
2441+ "login", "http://store.local/", "test@example.com"
2442+ )
2443
2444- @responses.activate
2445- def test_login_twofactor_required(self):
2446- self.mock_input.side_effect = ('user@example.org', '123456')
2447- self.mock_getpass.return_value = 'secret'
2448- root = self.make_root_macaroon()
2449- self.add_issue_store_admin_response(
2450- {'status': 200, 'json': {'macaroon': root}})
2451- self.add_get_sso_discharge_response(
2452- {'status': 401,
2453- 'json': {'error_list': [{'code': 'twofactor-required'}]}},
2454- {'status': 200, 'json': {'discharge_macaroon': 'dummy'}})
2455- self.assertEqual(0, login(self.make_args()))
2456+ assert_that(exit_status, Equals(0))
2457+ assert_that(
2458+ self.logger.output,
2459+ Equals(
2460+ "Enter your Ubuntu One SSO credentials.\n"
2461+ "Login successful\n"
2462+ "Configured http://store.local/ as the default store\n"
2463+ ),
2464+ )
2465
2466- self.assertIn(
2467- 'Enter your Ubuntu One SSO credentials.', self.logger.output)
2468- self.mock_input.assert_has_calls([
2469- mock.call('Email: '), mock.call('Second-factor auth: ')])
2470- self.mock_getpass.assert_called_once_with('Password: ')
2471- self.assertEqual(3, len(responses.calls))
2472- self.assertEqual({
2473- 'email': 'user@example.org',
2474- 'password': 'secret',
2475- 'caveat_id': 'payload',
2476- }, json.loads(responses.calls[1].request.body.decode()))
2477- self.assertEqual({
2478- 'email': 'user@example.org',
2479- 'password': 'secret',
2480- 'caveat_id': 'payload',
2481- 'otp': '123456',
2482- }, json.loads(responses.calls[2].request.body.decode()))
2483- self.assertThat(config.Config().parser, ContainsDict({
2484- 'store:default': MatchesDict({
2485- 'gw_url': Equals(self.default_gw_url),
2486- 'sso_url': Equals(self.default_sso_url),
2487- 'root': Equals(root),
2488- 'unbound_discharge': Equals('dummy'),
2489- 'email': Equals('user@example.org'),
2490- }),
2491- }))
2492+ def test_login_two_factor_required(self):
2493+ root_macaroon = pymacaroons.Macaroon(location="test1")
2494+ discharge_macaroon = pymacaroons.Macaroon(location="test2")
2495
2496- @responses.activate
2497- def test_login_twofactor_not_required(self):
2498- self.mock_input.return_value = 'user@example.org'
2499- self.mock_getpass.return_value = 'secret'
2500- root = self.make_root_macaroon()
2501- self.add_issue_store_admin_response(
2502- {'status': 200, 'json': {'macaroon': root}})
2503- self.add_get_sso_discharge_response(
2504- {'status': 200, 'json': {'discharge_macaroon': 'dummy'}})
2505- login(self.make_args())
2506+ def _login(session, auth_url, sso_url, email, password, otp=None):
2507+ if otp is None:
2508+ raise exceptions.TwoFactorAuthenticationRequired()
2509+ assert_that(otp, Equals("123456"))
2510+ return root_macaroon, discharge_macaroon
2511
2512- self.assertIn(
2513- 'Enter your Ubuntu One SSO credentials.', self.logger.output)
2514- self.mock_input.assert_called_once_with('Email: ')
2515- self.mock_getpass.assert_called_once_with('Password: ')
2516- self.assertEqual(2, len(responses.calls))
2517- self.assertEqual({
2518- 'email': 'user@example.org',
2519- 'password': 'secret',
2520- 'caveat_id': 'payload',
2521- }, json.loads(responses.calls[1].request.body.decode()))
2522- self.assertThat(config.Config().parser, ContainsDict({
2523- 'store:default': MatchesDict({
2524- 'gw_url': Equals(self.default_gw_url),
2525- 'sso_url': Equals(self.default_sso_url),
2526- 'root': Equals(root),
2527- 'unbound_discharge': Equals('dummy'),
2528- 'email': Equals('user@example.org'),
2529- }),
2530- }))
2531+ self.mock_with_side_effect(
2532+ "snapstore_library.authentication.login", _login
2533+ )
2534+ self.mock_with_return("getpass.getpass", "secret")
2535+ self.mock_with_return("builtins.input", "123456")
2536
2537- @responses.activate
2538- def test_login_with_email(self):
2539- self.mock_input.side_effect = Exception("shouldn't be called")
2540- self.mock_getpass.return_value = 'secret'
2541- root = self.make_root_macaroon()
2542- self.add_issue_store_admin_response(
2543- {'status': 200, 'json': {'macaroon': root}})
2544- self.add_get_sso_discharge_response(
2545- {'status': 200, 'json': {'discharge_macaroon': 'dummy'}})
2546- login(self.make_args(email='user@example.org'))
2547+ exit_status = self.run_main(
2548+ "login", "http://store.local/", "test@example.com"
2549+ )
2550
2551- self.assertIn(
2552- 'Enter your Ubuntu One SSO credentials.', self.logger.output)
2553- self.mock_input.assert_not_called()
2554- self.mock_getpass.assert_called_once_with('Password: ')
2555- self.assertEqual(2, len(responses.calls))
2556- self.assertEqual({
2557- 'email': 'user@example.org',
2558- 'password': 'secret',
2559- 'caveat_id': 'payload',
2560- }, json.loads(responses.calls[1].request.body.decode()))
2561- self.assertThat(config.Config().parser, ContainsDict({
2562- 'store:default': MatchesDict({
2563- 'gw_url': Equals(self.default_gw_url),
2564- 'sso_url': Equals(self.default_sso_url),
2565- 'root': Equals(root),
2566- 'unbound_discharge': Equals('dummy'),
2567- 'email': Equals('user@example.org'),
2568- }),
2569- }))
2570+ assert_that(exit_status, Equals(0))
2571+ assert_that(
2572+ self.logger.output,
2573+ Equals(
2574+ "Enter your Ubuntu One SSO credentials.\n"
2575+ "Login successful\n"
2576+ "Configured http://store.local/ as the default store\n"
2577+ ),
2578+ )
2579
2580- @responses.activate
2581- def test_store_url(self):
2582- gw_url = 'http://otherstore.local:1234/'
2583+ def test_login_no_email_arg(self):
2584+ root_macaroon = pymacaroons.Macaroon(location="test1")
2585+ discharge_macaroon = pymacaroons.Macaroon(location="test2")
2586
2587- self.mock_input.return_value = 'user@example.org'
2588- self.mock_getpass.return_value = 'secret'
2589- root = self.make_root_macaroon()
2590- self.add_issue_store_admin_response(
2591- {'status': 200, 'json': {'macaroon': root}}, gw_url=gw_url)
2592- self.add_get_sso_discharge_response(
2593- {'status': 200, 'json': {'discharge_macaroon': 'dummy'}})
2594- login(self.make_args(store_url=gw_url))
2595+ def _login(session, auth_url, sso_url, email, password, otp=None):
2596+ assert_that(email, Equals("test@example.com"))
2597+ return root_macaroon, discharge_macaroon
2598
2599- self.assertEqual(2, len(responses.calls))
2600- self.assertEqual(responses.calls[0].request.url[:len(gw_url)], gw_url)
2601- self.assertTrue(
2602- responses.calls[1].request.url.startswith(self.default_sso_url))
2603- self.assertThat(config.Config().parser, ContainsDict({
2604- 'store:default': ContainsDict({
2605- 'gw_url': Equals(gw_url),
2606- 'sso_url': Equals(self.default_sso_url),
2607- }),
2608- }))
2609+ self.mock_with_side_effect(
2610+ "snapstore_library.authentication.login", _login
2611+ )
2612+ self.mock_with_return("builtins.input", "test@example.com")
2613+ self.mock_with_return("getpass.getpass", "secret")
2614+
2615+ exit_status = self.run_main("login", "http://store.local/")
2616+
2617+ assert_that(exit_status, Equals(0))
2618+ assert_that(
2619+ self.logger.output,
2620+ Equals(
2621+ "Enter your Ubuntu One SSO credentials.\n"
2622+ "Login successful\n"
2623+ "Configured http://store.local/ as the default store\n"
2624+ ),
2625+ )
2626
2627- @responses.activate
2628- def test_sso_url(self):
2629- sso_url = 'https://othersso.local:1234/'
2630+ def test_login_unhandled_error(self):
2631+ self.mock_with_side_effect(
2632+ "snapstore_library.authentication.login",
2633+ Exception("Some error happened"),
2634+ )
2635+ self.mock_with_return("getpass.getpass", "secret")
2636
2637- self.mock_input.return_value = 'user@example.org'
2638- self.mock_getpass.return_value = 'secret'
2639- root = self.make_root_macaroon(sso_url=sso_url)
2640- self.add_issue_store_admin_response(
2641- {'status': 200, 'json': {'macaroon': root}})
2642- self.add_get_sso_discharge_response(
2643- {'status': 200, 'json': {'discharge_macaroon': 'dummy'}},
2644- sso_url=sso_url)
2645- login(self.make_args(sso_url=sso_url))
2646+ exit_status = self.run_main(
2647+ "login", "http://store.local/", "test@example.com"
2648+ )
2649
2650- self.assertEqual(2, len(responses.calls))
2651- self.assertTrue(
2652- responses.calls[0].request.url.startswith(self.default_gw_url))
2653- self.assertEqual(
2654- responses.calls[1].request.url[:len(sso_url)], sso_url)
2655- self.assertThat(config.Config().parser, ContainsDict({
2656- 'store:default': ContainsDict({
2657- 'gw_url': Equals(self.default_gw_url),
2658- 'sso_url': Equals(sso_url),
2659- }),
2660- }))
2661+ assert_that(exit_status, Equals(1))
2662+ assert_that(
2663+ self.logger.output,
2664+ Equals(
2665+ "Enter your Ubuntu One SSO credentials.\n"
2666+ "Exception: Some error happened\n"
2667+ ),
2668+ )
2669diff --git a/snapstore_client/logic/tests/test_overrides.py b/snapstore_client/logic/tests/test_overrides.py
2670index 05e32fc..5d53b48 100644
2671--- a/snapstore_client/logic/tests/test_overrides.py
2672+++ b/snapstore_client/logic/tests/test_overrides.py
2673@@ -1,314 +1,107 @@
2674 # Copyright 2017 Canonical Ltd.
2675-
2676-import json
2677-from urllib.parse import urljoin
2678-
2679-import fixtures
2680-import responses
2681-from testtools import TestCase
2682-
2683-from snapstore_client import config
2684-from snapstore_client.logic.overrides import (
2685- delete_override,
2686- list_overrides,
2687- override,
2688-)
2689-from snapstore_client.tests import (
2690- factory,
2691- testfixtures,
2692-)
2693-
2694-
2695-class OverridesTests(TestCase):
2696-
2697- def test_list_overrides_no_store_config(self):
2698- self.useFixture(testfixtures.ConfigFixture(empty=True))
2699- logger = self.useFixture(fixtures.FakeLogger())
2700- rc = list_overrides(factory.Args(snap_name='some-snap', series='16'))
2701- self.assertEqual(rc, 1)
2702- self.assertEqual(
2703- logger.output,
2704- 'No store configuration found. '
2705- 'Have you run "snap-store-proxy-client login"?\n')
2706-
2707- @responses.activate
2708- def test_list_overrides_online(self):
2709- self.useFixture(testfixtures.ConfigFixture())
2710- logger = self.useFixture(fixtures.FakeLogger())
2711- snap_id = factory.generate_snap_id()
2712- overrides = [
2713- factory.SnapDeviceGateway.Override(
2714- snap_id=snap_id, snap_name='mysnap'),
2715- factory.SnapDeviceGateway.Override(
2716- snap_id=snap_id, snap_name='mysnap', revision=3,
2717- upstream_revision=4, channel='foo/stable',
2718- architecture='i386'),
2719- ]
2720- # XXX cjwatson 2017-06-26: Use acceptable-generated doubles once
2721- # they exist.
2722- overrides_url = urljoin(
2723- config.Config().store_section('default').get('gw_url'),
2724- '/v2/metadata/overrides/mysnap')
2725- responses.add(
2726- 'GET', overrides_url, status=200, json={'overrides': overrides})
2727-
2728- list_overrides(
2729- factory.Args(snap_name='mysnap', series='16', password=False))
2730- self.assertEqual(
2731- 'mysnap stable amd64 1 (upstream 2)\n'
2732- 'mysnap foo/stable i386 3 (upstream 4)\n',
2733- logger.output)
2734- # We shouldn't have Basic Authorization headers, but Macaroon
2735- self.assertNotIn(
2736- 'Basic',
2737- responses.calls[0].request.headers['Authorization'])
2738-
2739- @responses.activate
2740- def test_list_overrides_offline(self):
2741- self.useFixture(testfixtures.ConfigFixture())
2742- logger = self.useFixture(fixtures.FakeLogger())
2743- snap_id = factory.generate_snap_id()
2744- overrides = [
2745- factory.SnapDeviceGateway.Override(
2746- snap_id=snap_id, snap_name='mysnap'),
2747- factory.SnapDeviceGateway.Override(
2748- snap_id=snap_id, snap_name='mysnap', revision=3,
2749- upstream_revision=4, channel='foo/stable',
2750- architecture='i386'),
2751- ]
2752- # XXX cjwatson 2017-06-26: Use acceptable-generated doubles once
2753- # they exist.
2754- overrides_url = urljoin(
2755- config.Config().store_section('default').get('gw_url'),
2756- '/v2/metadata/overrides/mysnap')
2757- responses.add(
2758- 'GET', overrides_url, status=200, json={'overrides': overrides})
2759-
2760- list_overrides(
2761- factory.Args(snap_name='mysnap', series='16', password='test'))
2762- self.assertEqual(
2763- 'mysnap stable amd64 1 (upstream 2)\n'
2764- 'mysnap foo/stable i386 3 (upstream 4)\n',
2765- logger.output)
2766- self.assertEqual(
2767- 'Basic YWRtaW46dGVzdA==',
2768- responses.calls[0].request.headers['Authorization'])
2769-
2770- def test_override_no_store_config(self):
2771- self.useFixture(testfixtures.ConfigFixture(empty=True))
2772- logger = self.useFixture(fixtures.FakeLogger())
2773- rc = override(factory.Args(
2774- snap_name='some-snap', channel_map_entries=['stable=1'],
2775- series='16',
2776- password=False))
2777- self.assertEqual(rc, 1)
2778- self.assertEqual(
2779- logger.output,
2780- 'No store configuration found. '
2781- 'Have you run "snap-store-proxy-client login"?\n')
2782-
2783- @responses.activate
2784- def test_override_online(self):
2785- self.useFixture(testfixtures.ConfigFixture())
2786- logger = self.useFixture(fixtures.FakeLogger())
2787- snap_id = factory.generate_snap_id()
2788- overrides = [
2789- factory.SnapDeviceGateway.Override(
2790- snap_id=snap_id, snap_name='mysnap'),
2791- factory.SnapDeviceGateway.Override(
2792- snap_id=snap_id, snap_name='mysnap', revision=3,
2793- upstream_revision=4, channel='foo/stable',
2794- architecture='i386'),
2795- ]
2796- # XXX cjwatson 2017-06-26: Use acceptable-generated doubles once
2797- # they exist.
2798- overrides_url = urljoin(
2799- config.Config().store_section('default').get('gw_url'),
2800- '/v2/metadata/overrides')
2801- responses.add(
2802- 'POST', overrides_url, status=200, json={'overrides': overrides})
2803-
2804- override(factory.Args(
2805- snap_name='mysnap',
2806- channel_map_entries=['stable=1', 'foo/stable=3'],
2807- series='16',
2808- password=False))
2809- self.assertEqual([
2810- {
2811- 'snap_name': 'mysnap',
2812- 'revision': 1,
2813- 'channel': 'stable',
2814- 'series': '16',
2815- },
2816- {
2817- 'snap_name': 'mysnap',
2818- 'revision': 3,
2819- 'channel': 'foo/stable',
2820- 'series': '16',
2821- },
2822- ], json.loads(responses.calls[0].request.body.decode()))
2823- self.assertEqual(
2824- 'mysnap stable amd64 1 (upstream 2)\n'
2825- 'mysnap foo/stable i386 3 (upstream 4)\n',
2826- logger.output)
2827- # We shouldn't have Basic Authorization headers, but Macaroon
2828- self.assertNotIn(
2829- 'Basic',
2830- responses.calls[0].request.headers['Authorization'])
2831-
2832- @responses.activate
2833- def test_override_offline(self):
2834- self.useFixture(testfixtures.ConfigFixture())
2835- logger = self.useFixture(fixtures.FakeLogger())
2836- snap_id = factory.generate_snap_id()
2837- overrides = [
2838- factory.SnapDeviceGateway.Override(
2839- snap_id=snap_id, snap_name='mysnap'),
2840- factory.SnapDeviceGateway.Override(
2841- snap_id=snap_id, snap_name='mysnap', revision=3,
2842- upstream_revision=4, channel='foo/stable',
2843- architecture='i386'),
2844- ]
2845- # XXX cjwatson 2017-06-26: Use acceptable-generated doubles once
2846- # they exist.
2847- overrides_url = urljoin(
2848- config.Config().store_section('default').get('gw_url'),
2849- '/v2/metadata/overrides')
2850- responses.add(
2851- 'POST', overrides_url, status=200, json={'overrides': overrides})
2852-
2853- override(factory.Args(
2854- snap_name='mysnap',
2855- channel_map_entries=['stable=1', 'foo/stable=3'],
2856- series='16',
2857- password='test'))
2858- self.assertEqual([
2859- {
2860- 'snap_name': 'mysnap',
2861- 'revision': 1,
2862- 'channel': 'stable',
2863- 'series': '16',
2864- },
2865- {
2866- 'snap_name': 'mysnap',
2867- 'revision': 3,
2868- 'channel': 'foo/stable',
2869- 'series': '16',
2870- },
2871- ], json.loads(responses.calls[0].request.body.decode()))
2872- self.assertEqual(
2873- 'mysnap stable amd64 1 (upstream 2)\n'
2874- 'mysnap foo/stable i386 3 (upstream 4)\n',
2875- logger.output)
2876- self.assertEqual(
2877- 'Basic YWRtaW46dGVzdA==',
2878- responses.calls[0].request.headers['Authorization'])
2879-
2880- def test_delete_override_no_store_config(self):
2881- self.useFixture(testfixtures.ConfigFixture(empty=True))
2882- logger = self.useFixture(fixtures.FakeLogger())
2883- rc = delete_override(factory.Args(
2884- snap_name='some-snap', channels=['stable'],
2885- series='16', password=False))
2886- self.assertEqual(rc, 1)
2887- self.assertEqual(
2888- logger.output,
2889- 'No store configuration found. '
2890- 'Have you run "snap-store-proxy-client login"?\n')
2891-
2892- @responses.activate
2893- def test_delete_override_online(self):
2894- self.useFixture(testfixtures.ConfigFixture())
2895- logger = self.useFixture(fixtures.FakeLogger())
2896- snap_id = factory.generate_snap_id()
2897- overrides = [
2898- factory.SnapDeviceGateway.Override(
2899- snap_id=snap_id, snap_name='mysnap', revision=None),
2900- factory.SnapDeviceGateway.Override(
2901- snap_id=snap_id, snap_name='mysnap', revision=None,
2902- upstream_revision=4, channel='foo/stable',
2903- architecture='i386'),
2904- ]
2905- # XXX cjwatson 2017-06-26: Use acceptable-generated doubles once
2906- # they exist.
2907- overrides_url = urljoin(
2908- config.Config().store_section('default').get('gw_url'),
2909- '/v2/metadata/overrides')
2910- responses.add(
2911- 'POST', overrides_url, status=200, json={'overrides': overrides})
2912-
2913- delete_override(factory.Args(
2914- snap_name='mysnap',
2915- channels=['stable', 'foo/stable'],
2916- series='16',
2917- password=False))
2918- self.assertEqual([
2919- {
2920- 'snap_name': 'mysnap',
2921- 'revision': None,
2922- 'channel': 'stable',
2923- 'series': '16',
2924- },
2925- {
2926- 'snap_name': 'mysnap',
2927- 'revision': None,
2928- 'channel': 'foo/stable',
2929- 'series': '16',
2930- },
2931- ], json.loads(responses.calls[0].request.body.decode()))
2932- self.assertEqual(
2933- 'mysnap stable amd64 is tracking upstream (revision 2)\n'
2934- 'mysnap foo/stable i386 is tracking upstream (revision 4)\n',
2935- logger.output)
2936- # We shouldn't have Basic Authorization headers, but Macaroon
2937- self.assertNotIn(
2938- 'Basic',
2939- responses.calls[0].request.headers['Authorization'])
2940-
2941- @responses.activate
2942- def test_delete_override_offline(self):
2943- self.useFixture(testfixtures.ConfigFixture())
2944- logger = self.useFixture(fixtures.FakeLogger())
2945- snap_id = factory.generate_snap_id()
2946- overrides = [
2947- factory.SnapDeviceGateway.Override(
2948- snap_id=snap_id, snap_name='mysnap', revision=None),
2949- factory.SnapDeviceGateway.Override(
2950- snap_id=snap_id, snap_name='mysnap', revision=None,
2951- upstream_revision=4, channel='foo/stable',
2952- architecture='i386'),
2953- ]
2954- # XXX cjwatson 2017-06-26: Use acceptable-generated doubles once
2955- # they exist.
2956- overrides_url = urljoin(
2957- config.Config().store_section('default').get('gw_url'),
2958- '/v2/metadata/overrides')
2959- responses.add(
2960- 'POST', overrides_url, status=200, json={'overrides': overrides})
2961-
2962- delete_override(factory.Args(
2963- snap_name='mysnap',
2964- channels=['stable', 'foo/stable'],
2965- series='16',
2966- password='test'))
2967- self.assertEqual([
2968- {
2969- 'snap_name': 'mysnap',
2970- 'revision': None,
2971- 'channel': 'stable',
2972- 'series': '16',
2973- },
2974- {
2975- 'snap_name': 'mysnap',
2976- 'revision': None,
2977- 'channel': 'foo/stable',
2978- 'series': '16',
2979- },
2980- ], json.loads(responses.calls[0].request.body.decode()))
2981- self.assertEqual(
2982- 'mysnap stable amd64 is tracking upstream (revision 2)\n'
2983- 'mysnap foo/stable i386 is tracking upstream (revision 4)\n',
2984- logger.output)
2985- self.assertEqual(
2986- 'Basic YWRtaW46dGVzdA==',
2987- responses.calls[0].request.headers['Authorization'])
2988+from testtools.assertions import assert_that
2989+from testtools.matchers import Equals
2990+
2991+from snapstore_client.logic.tests.utils import RunMainTestCase
2992+
2993+
2994+class OverridesTests(RunMainTestCase):
2995+ def test_list_overrides(self):
2996+ overrides = {
2997+ "overrides": [
2998+ {
2999+ "snap_id": "x" * 32,
3000+ "snap_name": "mysnap",
3001+ "revision": 1,
3002+ "upstream_revision": 2,
3003+ "channel": "stable",
3004+ "architecture": "amd64",
3005+ "series": 16,
3006+ },
3007+ {
3008+ "snap_id": "x" * 32,
3009+ "snap_name": "mysnap",
3010+ "revision": 3,
3011+ "upstream_revision": 5,
3012+ "channel": "foo/stable",
3013+ "architecture": "i386",
3014+ "series": 16,
3015+ },
3016+ ]
3017+ }
3018+ self.mock_with_return(
3019+ "snapstore_library.overrides.get_overrides", overrides
3020+ )
3021+
3022+ self.run_main_preconfigured("list-overrides", "mysnap")
3023+
3024+ assert_that(
3025+ self.logger.output,
3026+ Equals(
3027+ "mysnap stable amd64 1 (upstream 2)\n"
3028+ "mysnap foo/stable i386 3 (upstream 5)\n"
3029+ ),
3030+ )
3031+
3032+ def test_overrides(self):
3033+ overrides = {
3034+ "overrides": [
3035+ {
3036+ "snap_id": "x" * 32,
3037+ "snap_name": "mysnap",
3038+ "revision": 1,
3039+ "upstream_revision": 6,
3040+ "channel": "stable",
3041+ "architecture": "amd64",
3042+ "series": 16,
3043+ },
3044+ {
3045+ "snap_id": "x" * 32,
3046+ "snap_name": "mysnap",
3047+ "revision": 3,
3048+ "upstream_revision": 7,
3049+ "channel": "foo/stable",
3050+ "architecture": "i386",
3051+ "series": 16,
3052+ },
3053+ ]
3054+ }
3055+ self.mock_with_return(
3056+ "snapstore_library.overrides.set_overrides", overrides
3057+ )
3058+
3059+ self.run_main_preconfigured(
3060+ "override", "mysnap", "foo/stable=3", "stable=1"
3061+ )
3062+
3063+ assert_that(
3064+ self.logger.output,
3065+ Equals(
3066+ "mysnap stable amd64 1 (upstream 6)\n"
3067+ "mysnap foo/stable i386 3 (upstream 7)\n"
3068+ ),
3069+ )
3070+
3071+ def test_delete_overrides(self):
3072+ overrides = {
3073+ "overrides": [
3074+ {
3075+ "snap_id": "x" * 32,
3076+ "snap_name": "mysnap",
3077+ "revision": 1,
3078+ "upstream_revision": 6,
3079+ "channel": "stable",
3080+ "architecture": "amd64",
3081+ "series": 16,
3082+ }
3083+ ]
3084+ }
3085+ self.mock_with_return(
3086+ "snapstore_library.overrides.set_overrides", overrides
3087+ )
3088+
3089+ self.run_main_preconfigured("delete-override", "mysnap", "foo/stable")
3090+
3091+ assert_that(
3092+ self.logger.output, Equals("mysnap stable amd64 1 (upstream 6)\n")
3093+ )
3094diff --git a/snapstore_client/logic/tests/utils.py b/snapstore_client/logic/tests/utils.py
3095new file mode 100644
3096index 0000000..fd2d596
3097--- /dev/null
3098+++ b/snapstore_client/logic/tests/utils.py
3099@@ -0,0 +1,42 @@
3100+import sys
3101+
3102+import testtools
3103+import fixtures
3104+
3105+from snapstore_client.__main__ import main
3106+from snapstore_client.tests import testfixtures
3107+
3108+
3109+class RunMainTestCase(testtools.TestCase):
3110+ def mock(self, function):
3111+ return self.useFixture(fixtures.MockPatch(function)).mock
3112+
3113+ def mock_with_return(self, function, rtn):
3114+ mock = self.mock(function)
3115+ mock.return_value = rtn
3116+ return mock
3117+
3118+ def mock_with_side_effect(self, function, side_effect):
3119+ mock = self.mock(function)
3120+ mock.side_effect = side_effect
3121+ return mock
3122+
3123+ def _run_main(self, *args, empty_config=True):
3124+ self.useFixture(testfixtures.ConfigFixture(empty=empty_config))
3125+ self.mock("snapstore_client.__main__.configure_logging")
3126+ self.logger = self.useFixture(fixtures.FakeLogger())
3127+
3128+ old_argv = sys.argv
3129+ try:
3130+ sys.argv = ["snap-store-proxy-client"] + list(args)
3131+ return main()
3132+ except SystemExit as se:
3133+ raise AssertionError("Exited with {}".format(se.code))
3134+ finally:
3135+ sys.argv = old_argv
3136+
3137+ def run_main(self, *args):
3138+ return self._run_main(*args, empty_config=True)
3139+
3140+ def run_main_preconfigured(self, *args):
3141+ return self._run_main(*args, empty_config=False)
3142diff --git a/snapstore_client/tests/test_webservices.py b/snapstore_client/tests/test_webservices.py
3143deleted file mode 100644
3144index c9c743c..0000000
3145--- a/snapstore_client/tests/test_webservices.py
3146+++ /dev/null
3147@@ -1,228 +0,0 @@
3148-# Copyright 2017 Canonical Ltd.
3149-
3150-import json
3151-import sys
3152-from urllib.parse import urljoin
3153-
3154-import fixtures
3155-from requests.exceptions import HTTPError
3156-import responses
3157-from testtools import TestCase
3158-
3159-from snapstore_client import (
3160- config,
3161- exceptions,
3162- webservices,
3163-)
3164-from snapstore_client.tests import (
3165- factory,
3166- matchers,
3167- testfixtures,
3168-)
3169-
3170-if sys.version < '3.6':
3171- import sha3 # noqa
3172-
3173-
3174-class WebservicesTests(TestCase):
3175-
3176- def setUp(self):
3177- super().setUp()
3178- self.config = self.useFixture(testfixtures.ConfigFixture())
3179-
3180- @responses.activate
3181- def test_issue_store_admin_success(self):
3182- gw_url = 'http://store.local/'
3183- issue_store_admin_url = urljoin(gw_url, '/v2/auth/issue-store-admin')
3184- responses.add(
3185- 'POST', issue_store_admin_url, status=200,
3186- json={'macaroon': 'dummy'})
3187-
3188- self.assertEqual('dummy', webservices.issue_store_admin(gw_url))
3189-
3190- @responses.activate
3191- def test_issue_store_admin_error(self):
3192- logger = self.useFixture(fixtures.FakeLogger())
3193- gw_url = 'http://store.local/'
3194- issue_store_admin_url = urljoin(gw_url, '/v2/auth/issue-store-admin')
3195- responses.add(
3196- 'POST', issue_store_admin_url, status=400,
3197- json=factory.APIError.single('Something went wrong').to_dict())
3198-
3199- self.assertRaises(
3200- HTTPError, webservices.issue_store_admin, gw_url)
3201- self.assertEqual(
3202- 'Failed to issue store_admin macaroon:\nSomething went wrong\n',
3203- logger.output)
3204-
3205- @responses.activate
3206- def test_get_sso_discharge_success(self):
3207- sso_url = 'http://sso.local/'
3208- discharge_url = urljoin(sso_url, '/api/v2/tokens/discharge')
3209- responses.add(
3210- 'POST', discharge_url, status=200,
3211- json={'discharge_macaroon': 'dummy'})
3212-
3213- self.assertEqual(
3214- 'dummy',
3215- webservices.get_sso_discharge(
3216- sso_url, 'user@example.org', 'secret', 'caveat'))
3217- request = responses.calls[0].request
3218- self.assertEqual('application/json', request.headers['Content-Type'])
3219- self.assertEqual({
3220- 'email': 'user@example.org',
3221- 'password': 'secret',
3222- 'caveat_id': 'caveat',
3223- }, json.loads(request.body.decode()))
3224-
3225- @responses.activate
3226- def test_get_sso_discharge_success_with_otp(self):
3227- sso_url = 'http://sso.local/'
3228- discharge_url = urljoin(sso_url, '/api/v2/tokens/discharge')
3229- responses.add(
3230- 'POST', discharge_url, status=200,
3231- json={'discharge_macaroon': 'dummy'})
3232-
3233- self.assertEqual(
3234- 'dummy',
3235- webservices.get_sso_discharge(
3236- sso_url, 'user@example.org', 'secret', 'caveat',
3237- one_time_password='123456'))
3238- request = responses.calls[0].request
3239- self.assertEqual('application/json', request.headers['Content-Type'])
3240- self.assertEqual({
3241- 'email': 'user@example.org',
3242- 'password': 'secret',
3243- 'caveat_id': 'caveat',
3244- 'otp': '123456',
3245- }, json.loads(request.body.decode()))
3246-
3247- @responses.activate
3248- def test_get_sso_discharge_twofactor_required(self):
3249- sso_url = 'http://sso.local/'
3250- discharge_url = urljoin(sso_url, '/api/v2/tokens/discharge')
3251- responses.add(
3252- 'POST', discharge_url, status=401,
3253- json={'error_list': [{'code': 'twofactor-required'}]})
3254-
3255- self.assertRaises(
3256- exceptions.StoreTwoFactorAuthenticationRequired,
3257- webservices.get_sso_discharge,
3258- sso_url, 'user@example.org', 'secret', 'caveat')
3259-
3260- @responses.activate
3261- def test_get_sso_discharge_structured_error(self):
3262- sso_url = 'http://sso.local/'
3263- discharge_url = urljoin(sso_url, '/api/v2/tokens/discharge')
3264- responses.add(
3265- 'POST', discharge_url, status=400,
3266- json={'error_list': [{'code': 'invalid-request',
3267- 'message': 'Something went wrong'}]})
3268-
3269- e = self.assertRaises(
3270- exceptions.StoreAuthenticationError, webservices.get_sso_discharge,
3271- sso_url, 'user@example.org', 'secret', 'caveat')
3272- self.assertEqual('Something went wrong', e.message)
3273-
3274- @responses.activate
3275- def test_get_sso_discharge_unstructured_error(self):
3276- logger = self.useFixture(fixtures.FakeLogger())
3277- sso_url = 'http://sso.local/'
3278- discharge_url = urljoin(sso_url, '/api/v2/tokens/discharge')
3279- responses.add(
3280- 'POST', discharge_url, status=503, body='Try again later.')
3281-
3282- self.assertRaises(
3283- HTTPError, webservices.get_sso_discharge,
3284- sso_url, 'user@example.org', 'secret', 'caveat')
3285- self.assertEqual(
3286- 'Failed to get SSO discharge:\n'
3287- '====================\n'
3288- 'Try again later.\n'
3289- '====================\n',
3290- logger.output)
3291-
3292- @responses.activate
3293- def test_get_overrides_success(self):
3294- logger = self.useFixture(fixtures.FakeLogger())
3295- overrides = [factory.SnapDeviceGateway.Override()]
3296- # XXX cjwatson 2017-06-26: Use acceptable-generated double once it
3297- # exists.
3298- store = config.Config().store_section('default')
3299- overrides_url = urljoin(
3300- store.get('gw_url'), '/v2/metadata/overrides/mysnap')
3301- responses.add('GET', overrides_url, status=200, json=overrides)
3302-
3303- self.assertEqual(overrides, webservices.get_overrides(
3304- store, 'mysnap'))
3305- request = responses.calls[0].request
3306- self.assertThat(
3307- request.headers['Authorization'],
3308- matchers.MacaroonHeaderVerifies(self.config.key))
3309- self.assertNotIn('Failed to get overrides:', logger.output)
3310-
3311- @responses.activate
3312- def test_get_overrides_error(self):
3313- logger = self.useFixture(fixtures.FakeLogger())
3314- store = config.Config().store_section('default')
3315- overrides_url = urljoin(
3316- store.get('gw_url'), '/v2/metadata/overrides/mysnap')
3317- responses.add(
3318- 'GET', overrides_url, status=400,
3319- json=factory.APIError.single('Something went wrong').to_dict())
3320-
3321- self.assertRaises(
3322- HTTPError, webservices.get_overrides, store, 'mysnap')
3323- self.assertEqual(
3324- 'Failed to get overrides:\nSomething went wrong\n', logger.output)
3325-
3326- @responses.activate
3327- def test_set_overrides_success(self):
3328- logger = self.useFixture(fixtures.FakeLogger())
3329- override = factory.SnapDeviceGateway.Override()
3330- # XXX cjwatson 2017-06-26: Use acceptable-generated double once it
3331- # exists.
3332- store = config.Config().store_section('default')
3333- overrides_url = urljoin(store.get('gw_url'), '/v2/metadata/overrides')
3334- responses.add('POST', overrides_url, status=200, json=[override])
3335-
3336- self.assertEqual([override], webservices.set_overrides(
3337- store, [{
3338- 'snap_name': override['snap_name'],
3339- 'revision': override['revision'],
3340- 'channel': override['channel'],
3341- 'series': override['series'],
3342- }]))
3343- request = responses.calls[0].request
3344- self.assertThat(
3345- request.headers['Authorization'],
3346- matchers.MacaroonHeaderVerifies(self.config.key))
3347- self.assertEqual([{
3348- 'snap_name': override['snap_name'],
3349- 'revision': override['revision'],
3350- 'channel': override['channel'],
3351- 'series': override['series'],
3352- }], json.loads(request.body.decode()))
3353- self.assertNotIn('Failed to set override:', logger.output)
3354-
3355- @responses.activate
3356- def test_set_overrides_error(self):
3357- logger = self.useFixture(fixtures.FakeLogger())
3358- override = factory.SnapDeviceGateway.Override()
3359- # XXX cjwatson 2017-06-26: Use acceptable-generated double once it
3360- # exists.
3361- store = config.Config().store_section('default')
3362- overrides_url = urljoin(store.get('gw_url'), '/v2/metadata/overrides')
3363- responses.add(
3364- 'POST', overrides_url, status=400,
3365- json=factory.APIError.single('Something went wrong').to_dict())
3366-
3367- self.assertRaises(HTTPError, lambda: webservices.set_overrides(
3368- store, {
3369- 'snap_name': override['snap_name'],
3370- 'revision': override['revision'],
3371- 'channel': override['channel'],
3372- 'series': override['series'],
3373- }))
3374- self.assertEqual(
3375- 'Failed to set override:\nSomething went wrong\n', logger.output)
3376diff --git a/snapstore_client/utils.py b/snapstore_client/utils.py
3377index 461a1c3..8b10606 100644
3378--- a/snapstore_client/utils.py
3379+++ b/snapstore_client/utils.py
3380@@ -3,11 +3,6 @@ import logging
3381 logger = logging.getLogger(__name__)
3382
3383
3384-def _log_credentials_error(e):
3385- logger.error('%s', e)
3386- logger.error('Try to "snap-store-proxy-client login" again.')
3387-
3388-
3389 def _log_authorized_error():
3390 logger.error(("Perhaps you have not been registered as an "
3391 "admin with the proxy."))
3392diff --git a/snapstore_client/webservices.py b/snapstore_client/webservices.py
3393deleted file mode 100644
3394index 3ebf149..0000000
3395--- a/snapstore_client/webservices.py
3396+++ /dev/null
3397@@ -1,185 +0,0 @@
3398-# Copyright 2017 Canonical Ltd. This software is licensed under the
3399-# GNU General Public License version 3 (see the file LICENSE).
3400-
3401-import base64
3402-import json
3403-import logging
3404-import urllib.parse
3405-
3406-from pymacaroons import Macaroon
3407-import requests
3408-
3409-from snapstore_client import exceptions
3410-
3411-
3412-logger = logging.getLogger(__name__)
3413-
3414-
3415-def issue_store_admin(gw_url):
3416- """Ask the store to issue a store_admin macaroon."""
3417- issue_store_admin_url = urllib.parse.urljoin(
3418- gw_url, '/v2/auth/issue-store-admin')
3419- try:
3420- resp = requests.post(issue_store_admin_url)
3421- except requests.exceptions.ConnectionError:
3422- raise exceptions.StoreCommunicationError(gw_url)
3423- except requests.exceptions.RequestException:
3424- raise exceptions.InvalidStoreURL(gw_url)
3425- if resp.status_code == 404:
3426- _print_error_message('issue store_admin macaroon', resp)
3427- raise exceptions.InvalidStoreURL(gw_url)
3428- elif resp.status_code != 200:
3429- _print_error_message('issue store_admin macaroon', resp)
3430- resp.raise_for_status()
3431- return resp.json()['macaroon']
3432-
3433-
3434-def get_sso_discharge(sso_url, email, password, caveat_id,
3435- one_time_password=None):
3436- discharge_url = urllib.parse.urljoin(
3437- sso_url, '/api/v2/tokens/discharge')
3438- data = {'email': email, 'password': password, 'caveat_id': caveat_id}
3439- if one_time_password is not None:
3440- data['otp'] = one_time_password
3441- resp = requests.post(
3442- discharge_url, headers={'Accept': 'application/json'}, json=data)
3443- if not resp.ok:
3444- try:
3445- error_list = resp.json().get('error_list', [])
3446- if resp.status_code == 401:
3447- for error in error_list:
3448- if error.get('code') == 'twofactor-required':
3449- raise exceptions.StoreTwoFactorAuthenticationRequired()
3450- if error_list:
3451- # Only bother about the first error.
3452- error = error_list[0]
3453- raise exceptions.StoreAuthenticationError(
3454- error['message'], extra=error.get('extra'))
3455- except json.JSONDecodeError:
3456- pass
3457- _print_error_message('get SSO discharge', resp)
3458- resp.raise_for_status()
3459- return resp.json()['discharge_macaroon']
3460-
3461-
3462-def refresh_sso_discharge(store, unbound_discharge_raw):
3463- refresh_url = urllib.parse.urljoin(
3464- store.get('sso_url'), '/api/v2/tokens/refresh')
3465- data = {'discharge_macaroon': unbound_discharge_raw}
3466- resp = requests.post(
3467- refresh_url, headers={'Accept': 'application/json'}, json=data)
3468- if not resp.ok:
3469- _print_error_message('refresh SSO discharge', resp)
3470- resp.raise_for_status()
3471- return resp.json()['discharge_macaroon']
3472-
3473-
3474-def _deserialize_macaroon(name, value):
3475- if value is None:
3476- raise exceptions.InvalidCredentials('no {} macaroon'.format(name))
3477- try:
3478- return Macaroon.deserialize(value)
3479- except Exception:
3480- raise exceptions.InvalidCredentials(
3481- 'failed to deserialize {} macaroon'.format(name))
3482-
3483-
3484-def _get_macaroon_auth(store):
3485- """Return an Authorization header containing store macaroons."""
3486- root_raw = store.get('root')
3487- root = _deserialize_macaroon('root', root_raw)
3488- unbound_discharge_raw = store.get('unbound_discharge')
3489- unbound_discharge = _deserialize_macaroon(
3490- 'unbound discharge', unbound_discharge_raw)
3491- bound_discharge = root.prepare_for_request(unbound_discharge)
3492- bound_discharge_raw = bound_discharge.serialize()
3493- return 'Macaroon root="{}", discharge="{}"'.format(
3494- root_raw, bound_discharge_raw)
3495-
3496-
3497-def _get_basic_auth(password):
3498- """Build the basic auth for interacting with an offline proxy"""
3499- # XXX twom 2019-03-15 Hardcoded username, awaiting user management
3500- username = 'admin'
3501- credentials = '{}:{}'.format(username, password)
3502- try:
3503- encoded_credentials = base64.b64encode(credentials.encode('UTF-8'))
3504- except UnicodeEncodeError:
3505- logger.error('Unable to encode password to UTF-8')
3506- raise
3507- return 'Basic {}'.format(encoded_credentials.decode())
3508-
3509-
3510-def _raise_needs_refresh(response):
3511- if (response.status_code == 401 and
3512- response.headers.get('WWW-Authenticate') == (
3513- 'Macaroon needs_refresh=1')):
3514- raise exceptions.StoreMacaroonNeedsRefresh()
3515-
3516-
3517-def refresh_if_necessary(store, func, *args, **kwargs):
3518- """Make a request, refreshing macaroons if necessary."""
3519- try:
3520- return func(*args, **kwargs)
3521- except exceptions.StoreMacaroonNeedsRefresh:
3522- unbound_discharge = refresh_sso_discharge(
3523- store.get('sso_url'), store.get('unbound_discharge'))
3524- store.set('unbound_discharge', unbound_discharge)
3525- store.save()
3526- return func(*args, **kwargs)
3527-
3528-
3529-def get_overrides(store, snap_name, series='16', password=None):
3530- """Get all overrides for a snap."""
3531- overrides_url = urllib.parse.urljoin(
3532- store.get('gw_url'),
3533- '/v2/metadata/overrides/{}'.format(urllib.parse.quote_plus(snap_name)))
3534- headers = {
3535- 'X-Ubuntu-Series': series,
3536- }
3537- if password:
3538- headers['Authorization'] = _get_basic_auth(password)
3539- else:
3540- headers['Authorization'] = _get_macaroon_auth(store)
3541- resp = requests.get(overrides_url, headers=headers)
3542- _raise_needs_refresh(resp)
3543- if resp.status_code != 200:
3544- _print_error_message('get overrides', resp)
3545- resp.raise_for_status()
3546- return resp.json()
3547-
3548-
3549-def set_overrides(store, overrides, password=None):
3550- """Add or remove channel map overrides for a snap."""
3551- overrides_url = urllib.parse.urljoin(
3552- store.get('gw_url'), '/v2/metadata/overrides')
3553- if password:
3554- headers = {'Authorization': _get_basic_auth(password)}
3555- else:
3556- headers = {'Authorization': _get_macaroon_auth(store)}
3557- resp = requests.post(overrides_url, headers=headers, json=overrides)
3558- _raise_needs_refresh(resp)
3559- if resp.status_code != 200:
3560- _print_error_message('set override', resp)
3561- resp.raise_for_status()
3562- return resp.json()
3563-
3564-
3565-def _print_error_message(action, response):
3566- """Print failure messages from other services in a standard way."""
3567- logger.error("Failed to %s:", action)
3568- if response.status_code == 500:
3569- logger.error("Server sent 500 response.")
3570- elif response.status_code == 404:
3571- logger.error("Server sent 404 response")
3572- else:
3573- try:
3574- json_document = response.json()
3575- error_list = json_document.get(
3576- 'error-list', json_document.get('error_list', []))
3577- for error in error_list:
3578- logger.error(error['message'])
3579- except json.JSONDecodeError:
3580- logger.error('=' * 20)
3581- logger.error(response.content.decode('UTF-8', errors='replace'))
3582- logger.error('=' * 20)

Subscribers

People subscribed via source and target branches

to all changes: