Merge lp:~mew/charm-helpers/fetchallthethings into lp:charm-helpers
- fetchallthethings
- Merge into devel
Status: | Merged |
---|---|
Merged at revision: | 48 |
Proposed branch: | lp:~mew/charm-helpers/fetchallthethings |
Merge into: | lp:charm-helpers |
Diff against target: |
150 lines (+41/-15) 4 files modified
charmhelpers/fetch/archiveurl.py (+12/-7) charmhelpers/payload/archive.py (+4/-3) tests/fetch/test_archiveurl.py (+17/-2) tests/payload/test_archive.py (+8/-3) |
To merge this branch: | bzr merge lp:~mew/charm-helpers/fetchallthethings |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Liam Young (community) | Approve | ||
Matthew Wedgwood (community) | Needs Resubmitting | ||
Review via email:
|
Commit message
Extensible support for retrieving software from external sources. Includes support for tar.gz, tar.bz2, and zip archives via http, https, or ftp.
Description of the change
This branch adds a fetch system to charm-helpers, allowing authors to easily support retrieval of external code from a number of different sources. In turn, they can allow their users to decide which sources they trust and how external code is retrieved.
Only one source handler (for tar.gz, tar.bz2, and zip archives via http, https, or ftp) is implemented in this branch. The BaseFetchHandler class allows additional handlers to be implemented easily.
As a peripheral change, 'make lint' no longer checks for line lengths in code.
- 26. By Adam Gandelman
-
[james-page] Import fixes for charmhelpers.
contrib. hahelpers. - 27. By James Page
-
[stub] Ensure that hookenv.relations also returns data that has been set by the local unit.
- 28. By Matthew Wedgwood
-
[michael.nelson] Add declarative machine state support using saltstack
- 29. By Matthew Wedgwood
-
[trivial] lint error cleanups
- 30. By Matthew Wedgwood
-
[michael.nelson] Don't use salt-minion (which installs and runs salt-minion daemon).
- 31. By Matthew Wedgwood
-
[james-page] Various refactoring + additions to hookenv
1) cached decorator
Allows caching of function + args to avoid repeated calls out the cli
to juju commands.2) Normalizing missing unit/config/
relation data to None relation_
get('missing' ) == None
config('missing') == None
unit_get('missing') == None3) Normalized use of the Serializable object a bit
And added a unit test to cover missing attribute lookups
Question: maybe this should actually return 'None' rather than throwing
an exception inline with 2).4) Tweaks some tests to be closer to juju behaviour with json
5) Tidied misc pep8/pylint warnings.
- 32. By Matthew Wedgwood
-
[james-page] Refactoring of service control code in host helper
1) Use 'service' command for all service control
Detecting upstart and init.d configuration files is overkill; this is
exactly what the 'service' command is design todo and it also deals with
saucy onwards where init.d and upstart configuration with the same
name might be installed.'service' will always do the right thing
2) Added restart and reload helpers
reload detects an error (say the service is not running) and will fallback
to restart if so.This is inline with the openstack charm helpers code.
- 33. By James Page
-
[stub] Misc fixes around use of Serializable including:
1) Make Serializable pickleable
2) Ensure string results are usable strings
- 34. By James Page
-
[gandelman-a]
Fixup broken test to ensure that services are only restarted once.
host.core.
restart_ on_change( ): Ensure restarts happen as described in map. - 35. By James Page
-
[gandelman-a]
This adds some small linux-specific utility helpers useful when working with block devices and storage. They are mostly derived from the Openstack helpers but renamed and tests added.
- 36. By Liam Young
-
[jamespage]
Redux of configure_source code in fetch helper1) Fixed up a number of bugs and issues in the original code
2) Implement correct cloud: handling
3) Added tests for everything.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Matthew Wedgwood (mew) wrote : | # |
I've fixed all of these issues, except (4). The command argument "dest_file" refers to the downloaded file, not the destination for extraction. I've renamed that to "dld_file" to make that a bit clearer.
I've also rearranged a bit, moving the archive handling code to the payload.archive module. fetch.archive.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Liam Young (gnuoy) wrote : | # |
Theres a text conflict in charmhelpers/
- 37. By Matthew Wedgwood
-
merge from trunk
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Matthew Wedgwood (mew) wrote : | # |
Conflict is resolved.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Liam Young (gnuoy) wrote : | # |
It looks like ArchiveUrlFetch
$ ls /tmp/testfetch/pay*
/tmp/testfetch/
$ python
Python 2.7.4 (default, Apr 19 2013, 18:32:33)
[GCC 4.7.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import charmhelpers.
>>> fh = charmhelpers.
>>> fh.can_
False
>>> fh.can_
True
>>> quit()
Also, I think ArchiveUrlFetch
- 38. By Matthew Wedgwood
-
Add support for uncompressed tar archives
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Liam Young (gnuoy) wrote : | # |
Deploying a tar file seems to fail:
root@liam-
root@liam-
Python 2.7.3 (default, Apr 10 2013, 05:46:21)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import charmhelpers.
>>> fh = charmhelpers.
>>> fh.can_
True
>>> fh.install(
Unit name is not defined
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "charmhelpers/
return extract(dld_file)
File "charmhelpers/
get_
File "charmhelpers/
archive = tarfile.
File "/usr/lib/
return func(name, "r", fileobj, **kwargs)
File "/usr/lib/
fileobj = bltn_open(name, mode + "b")
IOError: [Errno 21] Is a directory: '/var/lib/
>>>
- 39. By Matthew Wedgwood
-
fix stupid confusing bugs in downloads and archive extraction. update tests to catch them
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Matthew Wedgwood (mew) wrote : | # |
OK, I've fixed the bugs, updated the tests, and taken if for a spin. Sorry for crummy MP.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Liam Young (gnuoy) wrote : | # |
Sorry to be a pita but it seems counter intuitive that the archives and fetched directory are both used during the process and the code creates the archives dir but fails if the fetched directory hasn't been precreated. Could the fetched directory be created on demand as well ?
- 40. By Matthew Wedgwood
-
Create 'fetched' directory if it does not exist
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Matthew Wedgwood (mew) wrote : | # |
Makes total sense. Thanks for the attention to detail.
Preview Diff
1 | === modified file 'charmhelpers/fetch/archiveurl.py' | |||
2 | --- charmhelpers/fetch/archiveurl.py 2013-06-26 22:14:29 +0000 | |||
3 | +++ charmhelpers/fetch/archiveurl.py 2013-07-10 18:59:30 +0000 | |||
4 | @@ -8,6 +8,7 @@ | |||
5 | 8 | get_archive_handler, | 8 | get_archive_handler, |
6 | 9 | extract, | 9 | extract, |
7 | 10 | ) | 10 | ) |
8 | 11 | from charmhelpers.core.host import mkdir | ||
9 | 11 | 12 | ||
10 | 12 | 13 | ||
11 | 13 | class ArchiveUrlFetchHandler(BaseFetchHandler): | 14 | class ArchiveUrlFetchHandler(BaseFetchHandler): |
12 | @@ -24,20 +25,24 @@ | |||
13 | 24 | # propogate all exceptions | 25 | # propogate all exceptions |
14 | 25 | # URLError, OSError, etc | 26 | # URLError, OSError, etc |
15 | 26 | response = urllib2.urlopen(source) | 27 | response = urllib2.urlopen(source) |
18 | 27 | with open(dest, 'w') as dest_file: | 28 | try: |
19 | 28 | dest_file.write(response.read()) | 29 | with open(dest, 'w') as dest_file: |
20 | 30 | dest_file.write(response.read()) | ||
21 | 31 | except Exception as e: | ||
22 | 32 | if os.path.isfile(dest): | ||
23 | 33 | os.unlink(dest) | ||
24 | 34 | raise e | ||
25 | 29 | 35 | ||
26 | 30 | def install(self, source): | 36 | def install(self, source): |
27 | 31 | url_parts = self.parse_url(source) | 37 | url_parts = self.parse_url(source) |
28 | 32 | dest_dir = os.path.join(os.environ.get('CHARM_DIR'), 'fetched') | 38 | dest_dir = os.path.join(os.environ.get('CHARM_DIR'), 'fetched') |
29 | 39 | if not os.path.exists(dest_dir): | ||
30 | 40 | mkdir(dest_dir, perms=0755) | ||
31 | 33 | dld_file = os.path.join(dest_dir, os.path.basename(url_parts.path)) | 41 | dld_file = os.path.join(dest_dir, os.path.basename(url_parts.path)) |
32 | 34 | try: | 42 | try: |
33 | 35 | self.download(source, dld_file) | 43 | self.download(source, dld_file) |
34 | 36 | except urllib2.URLError as e: | 44 | except urllib2.URLError as e: |
36 | 37 | return UnhandledSource(e.reason) | 45 | raise UnhandledSource(e.reason) |
37 | 38 | except OSError as e: | 46 | except OSError as e: |
42 | 39 | return UnhandledSource(e.strerror) | 47 | raise UnhandledSource(e.strerror) |
39 | 40 | finally: | ||
40 | 41 | if os.path.isfile(dld_file): | ||
41 | 42 | os.unlink(dld_file) | ||
43 | 43 | return extract(dld_file) | 48 | return extract(dld_file) |
44 | 44 | 49 | ||
45 | === modified file 'charmhelpers/payload/archive.py' | |||
46 | --- charmhelpers/payload/archive.py 2013-06-26 22:14:29 +0000 | |||
47 | +++ charmhelpers/payload/archive.py 2013-07-10 18:59:30 +0000 | |||
48 | @@ -19,7 +19,7 @@ | |||
49 | 19 | return extract_zipfile | 19 | return extract_zipfile |
50 | 20 | else: | 20 | else: |
51 | 21 | # look at the file name | 21 | # look at the file name |
53 | 22 | for ext in ('.tar.gz', '.tgz', 'tar.bz2', '.tbz2', '.tbz'): | 22 | for ext in ('.tar', '.tar.gz', '.tgz', 'tar.bz2', '.tbz2', '.tbz'): |
54 | 23 | if archive_name.endswith(ext): | 23 | if archive_name.endswith(ext): |
55 | 24 | return extract_tarfile | 24 | return extract_tarfile |
56 | 25 | for ext in ('.zip', '.jar'): | 25 | for ext in ('.zip', '.jar'): |
57 | @@ -28,7 +28,8 @@ | |||
58 | 28 | 28 | ||
59 | 29 | 29 | ||
60 | 30 | def archive_dest_default(archive_name): | 30 | def archive_dest_default(archive_name): |
62 | 31 | return os.path.join(hookenv.charm_dir(), "archives", archive_name) | 31 | archive_file = os.path.basename(archive_name) |
63 | 32 | return os.path.join(hookenv.charm_dir(), "archives", archive_file) | ||
64 | 32 | 33 | ||
65 | 33 | 34 | ||
66 | 34 | def extract(archive_name, destpath=None): | 35 | def extract(archive_name, destpath=None): |
67 | @@ -38,7 +39,7 @@ | |||
68 | 38 | destpath = archive_dest_default(archive_name) | 39 | destpath = archive_dest_default(archive_name) |
69 | 39 | if not os.path.isdir(destpath): | 40 | if not os.path.isdir(destpath): |
70 | 40 | host.mkdir(destpath) | 41 | host.mkdir(destpath) |
72 | 41 | get_archive_handler(archive_name)(archive_name, destpath) | 42 | handler(archive_name, destpath) |
73 | 42 | return destpath | 43 | return destpath |
74 | 43 | else: | 44 | else: |
75 | 44 | raise ArchiveError("No handler for archive") | 45 | raise ArchiveError("No handler for archive") |
76 | 45 | 46 | ||
77 | === modified file 'tests/fetch/test_archiveurl.py' | |||
78 | --- tests/fetch/test_archiveurl.py 2013-06-26 22:14:29 +0000 | |||
79 | +++ tests/fetch/test_archiveurl.py 2013-07-10 18:59:30 +0000 | |||
80 | @@ -6,7 +6,11 @@ | |||
81 | 6 | patch, | 6 | patch, |
82 | 7 | mock_open, | 7 | mock_open, |
83 | 8 | ) | 8 | ) |
85 | 9 | from charmhelpers.fetch import archiveurl | 9 | from charmhelpers.fetch import ( |
86 | 10 | archiveurl, | ||
87 | 11 | UnhandledSource, | ||
88 | 12 | ) | ||
89 | 13 | import urllib2 | ||
90 | 10 | 14 | ||
91 | 11 | 15 | ||
92 | 12 | class ArchiveUrlFetchHandlerTest(TestCase): | 16 | class ArchiveUrlFetchHandlerTest(TestCase): |
93 | @@ -59,8 +63,9 @@ | |||
94 | 59 | _open.assert_called_once_with("foo", 'w') | 63 | _open.assert_called_once_with("foo", 'w') |
95 | 60 | _open().write.assert_called_with("bar") | 64 | _open().write.assert_called_with("bar") |
96 | 61 | 65 | ||
97 | 66 | @patch('charmhelpers.fetch.archiveurl.mkdir') | ||
98 | 62 | @patch('charmhelpers.fetch.archiveurl.extract') | 67 | @patch('charmhelpers.fetch.archiveurl.extract') |
100 | 63 | def test_installs(self, _extract): | 68 | def test_installs(self, _extract, _mkdir): |
101 | 64 | self.fh.download = MagicMock() | 69 | self.fh.download = MagicMock() |
102 | 65 | 70 | ||
103 | 66 | for url in self.valid_urls: | 71 | for url in self.valid_urls: |
104 | @@ -72,3 +77,13 @@ | |||
105 | 72 | self.fh.download.assert_called_with(url, dest) | 77 | self.fh.download.assert_called_with(url, dest) |
106 | 73 | _extract.assert_called_with(dest) | 78 | _extract.assert_called_with(dest) |
107 | 74 | self.assertEqual(where, dest) | 79 | self.assertEqual(where, dest) |
108 | 80 | |||
109 | 81 | url = "http://www.example.com/archive.tar.gz" | ||
110 | 82 | |||
111 | 83 | self.fh.download.side_effect = urllib2.URLError('fail') | ||
112 | 84 | with patch.dict('os.environ', {'CHARM_DIR': 'foo'}): | ||
113 | 85 | self.assertRaises(UnhandledSource, self.fh.install, url) | ||
114 | 86 | |||
115 | 87 | self.fh.download.side_effect = OSError('fail') | ||
116 | 88 | with patch.dict('os.environ', {'CHARM_DIR': 'foo'}): | ||
117 | 89 | self.assertRaises(UnhandledSource, self.fh.install, url) | ||
118 | 75 | 90 | ||
119 | === modified file 'tests/payload/test_archive.py' | |||
120 | --- tests/payload/test_archive.py 2013-06-27 15:59:39 +0000 | |||
121 | +++ tests/payload/test_archive.py 2013-07-10 18:59:30 +0000 | |||
122 | @@ -33,13 +33,15 @@ | |||
123 | 33 | zip_archive_handler = archive.extract_zipfile | 33 | zip_archive_handler = archive.extract_zipfile |
124 | 34 | _isfile.return_value = False | 34 | _isfile.return_value = False |
125 | 35 | 35 | ||
127 | 36 | for ext in ('tar.gz', 'tgz', 'tar.bz2', 'tbz2', 'tbz'): | 36 | for ext in ('tar', 'tar.gz', 'tgz', 'tar.bz2', 'tbz2', 'tbz'): |
128 | 37 | handler = archive.get_archive_handler("somefile.{}".format(ext)) | 37 | handler = archive.get_archive_handler("somefile.{}".format(ext)) |
130 | 38 | self.assertEqual(handler, tar_archive_handler) | 38 | msg = "handler for extension: {}".format(ext) |
131 | 39 | self.assertEqual(handler, tar_archive_handler, msg) | ||
132 | 39 | 40 | ||
133 | 40 | for ext in ('zip', 'jar'): | 41 | for ext in ('zip', 'jar'): |
134 | 41 | handler = archive.get_archive_handler("somefile.{}".format(ext)) | 42 | handler = archive.get_archive_handler("somefile.{}".format(ext)) |
136 | 42 | self.assertEqual(handler, zip_archive_handler) | 43 | msg = "handler for extension {}".format(ext) |
137 | 44 | self.assertEqual(handler, zip_archive_handler, msg) | ||
138 | 43 | 45 | ||
139 | 44 | @patch('zipfile.is_zipfile') | 46 | @patch('zipfile.is_zipfile') |
140 | 45 | @patch('tarfile.is_tarfile') | 47 | @patch('tarfile.is_tarfile') |
141 | @@ -66,6 +68,9 @@ | |||
142 | 66 | thedir = archive.archive_dest_default("baz") | 68 | thedir = archive.archive_dest_default("baz") |
143 | 67 | self.assertEqual(thedir, os.path.join("foo", "archives", "baz")) | 69 | self.assertEqual(thedir, os.path.join("foo", "archives", "baz")) |
144 | 68 | 70 | ||
145 | 71 | thedir = archive.archive_dest_default("baz/qux") | ||
146 | 72 | self.assertEqual(thedir, os.path.join("foo", "archives", "qux")) | ||
147 | 73 | |||
148 | 69 | def test_extracts_tarfile(self): | 74 | def test_extracts_tarfile(self): |
149 | 70 | destdir = mkdtemp() | 75 | destdir = mkdtemp() |
150 | 71 | self.addCleanup(rmtree, destdir) | 76 | self.addCleanup(rmtree, destdir) |
I had a few issues when giving this a test run:
1) s/os.path. environ. get/os. environ. get/ class(archive_ name) extract_ all(destpath) class.open( archive_ name) extractall( destpath) url_parts. path, dest_file) not extract(dest_file)
2) install_remote expects a directory to be passed back by handler.install but the extract* classes don't return that in the archive class
3) For a tar ball the open method needs to be called on the tar ball and the extract all method is named differently i.e. This works for zip but not tar:
archive = archive_
archive.
This works for tar but not zip:
archive = archive_
archive.
4) In the archive plugin the source archive needs to be passed when calling extract, i.e. it should be extract(