Merge lp:~ricardokirkner/locolander/requirements into lp:locolander
- requirements
- Merge into trunk
Status: | Merged |
---|---|
Approved by: | Ricardo Kirkner |
Approved revision: | 15 |
Merged at revision: | 10 |
Proposed branch: | lp:~ricardokirkner/locolander/requirements |
Merge into: | lp:locolander |
Diff against target: |
507 lines (+210/-80) 6 files modified
.locolander.yml (+26/-0) fabfile.py (+26/-0) ns2df.py (+69/-31) requirements.txt (+16/-2) run_tests.sh (+1/-1) tests/test_ns2df.py (+72/-46) |
To merge this branch: | bzr merge lp:~ricardokirkner/locolander/requirements |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Facundo Batista | Approve | ||
Review via email: mp+172214@code.launchpad.net |
Commit message
pre-requisites to make locolander able to land itself
- updated requirements
- updated locolander config
- updated ns2df.py script to create Dockerfile from locolander config
Description of the change
- 12. By Ricardo Kirkner
-
fixed test script path
- 13. By Ricardo Kirkner
-
improvements to ns2df.py
- allow using a requirements file
- use locolander base images
- install dependencies using a single command for improved caching
- allow ns2df.py to be called from the commandline to create a Dockerfile
- build Dockerfile so that it includes the command to run tests
- build Dockerfile so that it times out the test run - 14. By Ricardo Kirkner
-
include ns2df tests in test script
- 15. By Ricardo Kirkner
-
added fabfile for bootstrapping and running tests
Ricardo Kirkner (ricardokirkner-f) wrote : | # |
I did tell you that, but I have changed my mind. The reason being that
while each new command gives us a different cached state (which was the
original reason for having it one per line), I now think that the overload
of having to invoke pip for each dependency is just too high. Also having a
single line for all dependencies should make rebuilding the image from
cache much faster than having to iterate over each intermediate state (plus
avoiding unnecessary wasted disk space by not-really reusable containers).
Hope this makes sense.
On Wed, Jul 10, 2013 at 8:15 PM, Facundo Batista <email address hidden>wrote:
> Review: Needs Information
>
> I see you're changing back to put all apt-get installations in the same
> line... you told me that we should put them all in different lines because
> of docker image caching... did that change?
>
> The rest looks ok!
> --
>
> https:/
> Your team LocoLanderos is subscribed to branch lp:locolander.
>
Preview Diff
1 | === added file '.locolander.yml' | |||
2 | --- .locolander.yml 1970-01-01 00:00:00 +0000 | |||
3 | +++ .locolander.yml 2013-07-09 18:45:42 +0000 | |||
4 | @@ -0,0 +1,26 @@ | |||
5 | 1 | precise: | ||
6 | 2 | pip: | ||
7 | 3 | distribute: 0.6.34 | ||
8 | 4 | Django: 1.5.1 | ||
9 | 5 | github3.py: 0.7.0 | ||
10 | 6 | httplib2: 0.8.0 | ||
11 | 7 | keyring: 1.5.0 | ||
12 | 8 | launchpadlib: 1.10.2 | ||
13 | 9 | lazr.authentication: 0.1.2 | ||
14 | 10 | lazr.restfulclient: 0.13.3 | ||
15 | 11 | lazr.uri: 1.0.3 | ||
16 | 12 | mock: 1.0.1 | ||
17 | 13 | oauth: 1.0.1 | ||
18 | 14 | pep8: 1.4.5 | ||
19 | 15 | pyflakes: 0.7.2 | ||
20 | 16 | PyYAML: 3.10.0 | ||
21 | 17 | requests: 1.2.3 | ||
22 | 18 | simplejson: 3.3.0 | ||
23 | 19 | South: 0.8.1 | ||
24 | 20 | testresources: 0.2.7 | ||
25 | 21 | wadllib: 1.3.2 | ||
26 | 22 | wsgi-intercept: 0.5.1 | ||
27 | 23 | zope.interface: 4.0.5 | ||
28 | 24 | |||
29 | 25 | metadata: | ||
30 | 26 | test_script: ./run_tests.sh | ||
31 | 0 | 27 | ||
32 | === added file 'fabfile.py' | |||
33 | --- fabfile.py 1970-01-01 00:00:00 +0000 | |||
34 | +++ fabfile.py 2013-07-09 18:45:42 +0000 | |||
35 | @@ -0,0 +1,26 @@ | |||
36 | 1 | from fabric.api import local | ||
37 | 2 | from fabric.context_managers import prefix | ||
38 | 3 | |||
39 | 4 | |||
40 | 5 | def virtualenv(func): | ||
41 | 6 | def wrapped(*args): | ||
42 | 7 | with prefix('. .env/bin/activate'): | ||
43 | 8 | return func(*args) | ||
44 | 9 | return wrapped | ||
45 | 10 | |||
46 | 11 | |||
47 | 12 | def bootstrap(): | ||
48 | 13 | local('virtualenv -p /usr/bin/python2 .env') | ||
49 | 14 | local('.env/bin/pip install -r requirements.txt') | ||
50 | 15 | |||
51 | 16 | |||
52 | 17 | @virtualenv | ||
53 | 18 | def manage(*args): | ||
54 | 19 | cmd = ['python locolander/manage.py'] | ||
55 | 20 | cmd.extend(args) | ||
56 | 21 | local(' '.join(cmd)) | ||
57 | 22 | |||
58 | 23 | |||
59 | 24 | @virtualenv | ||
60 | 25 | def test(): | ||
61 | 26 | local('./run_tests.sh') | ||
62 | 0 | 27 | ||
63 | === modified file 'ns2df.py' | |||
64 | --- ns2df.py 2013-06-22 23:29:05 +0000 | |||
65 | +++ ns2df.py 2013-07-09 18:45:42 +0000 | |||
66 | @@ -14,7 +14,7 @@ | |||
67 | 14 | 14 | ||
68 | 15 | will be converted to | 15 | will be converted to |
69 | 16 | 16 | ||
71 | 17 | from ubuntu:precise | 17 | from locolander:precise |
72 | 18 | run apt-get -q -y install apache2=3.3-4 | 18 | run apt-get -q -y install apache2=3.3-4 |
73 | 19 | run apt-get -q -y install bzr | 19 | run apt-get -q -y install bzr |
74 | 20 | run pip install --download=/tmp/pipcache --no-install foobar | 20 | run pip install --download=/tmp/pipcache --no-install foobar |
75 | @@ -24,14 +24,30 @@ | |||
76 | 24 | and also the system will use "./test" as script | 24 | and also the system will use "./test" as script |
77 | 25 | """ | 25 | """ |
78 | 26 | 26 | ||
79 | 27 | import os.path | ||
80 | 28 | |||
81 | 27 | import yaml | 29 | import yaml |
82 | 28 | 30 | ||
83 | 29 | # the conversion between our nice base names and those that are | 31 | # the conversion between our nice base names and those that are |
84 | 30 | # needed by docker | 32 | # needed by docker |
85 | 31 | BASE_TRANSLATIONS = { | 33 | BASE_TRANSLATIONS = { |
87 | 32 | 'precise': 'ubuntu:precise', | 34 | 'precise': 'locolander:precise', |
88 | 33 | } | 35 | } |
89 | 34 | 36 | ||
90 | 37 | |||
91 | 38 | def _get_pip_packages(items): | ||
92 | 39 | if isinstance(items, basestring): | ||
93 | 40 | return items | ||
94 | 41 | packages = [] | ||
95 | 42 | for name, ver in sorted(items.items()): | ||
96 | 43 | if ver is None: | ||
97 | 44 | p = name | ||
98 | 45 | else: | ||
99 | 46 | p = "%s==%s" % (name, ver) | ||
100 | 47 | packages.append(p) | ||
101 | 48 | return ' '.join(packages) | ||
102 | 49 | |||
103 | 50 | |||
104 | 35 | def _get_base(config, **params): | 51 | def _get_base(config, **params): |
105 | 36 | """Process the system base stuff.""" | 52 | """Process the system base stuff.""" |
106 | 37 | # support only one base for now | 53 | # support only one base for now |
107 | @@ -41,6 +57,13 @@ | |||
108 | 41 | return ["from " + BASE_TRANSLATIONS[base]] | 57 | return ["from " + BASE_TRANSLATIONS[base]] |
109 | 42 | 58 | ||
110 | 43 | 59 | ||
111 | 60 | def _get_code(config, **params): | ||
112 | 61 | return ['run mkdir -p {target_path}'.format(**params), | ||
113 | 62 | 'add . {target_path}'.format(**params), | ||
114 | 63 | 'run chown -R locolander.locolander {target_path}'.format( | ||
115 | 64 | **params)] | ||
116 | 65 | |||
117 | 66 | |||
118 | 44 | def _get_depends_prev(config, **params): | 67 | def _get_depends_prev(config, **params): |
119 | 45 | """All the dependencies before securing machine.""" | 68 | """All the dependencies before securing machine.""" |
120 | 46 | # support only one base for now | 69 | # support only one base for now |
121 | @@ -50,26 +73,26 @@ | |||
122 | 50 | # apt | 73 | # apt |
123 | 51 | items = config.get('apt', []) | 74 | items = config.get('apt', []) |
124 | 52 | if items: | 75 | if items: |
125 | 76 | packages = [] | ||
126 | 53 | for name, ver in sorted(items.items()): | 77 | for name, ver in sorted(items.items()): |
127 | 54 | if ver is None: | 78 | if ver is None: |
128 | 55 | p = name | 79 | p = name |
129 | 56 | else: | 80 | else: |
130 | 57 | p = "%s=%s" % (name, ver) | 81 | p = "%s=%s" % (name, ver) |
133 | 58 | dep = "run apt-get -q -y install " + p | 82 | packages.append(p) |
134 | 59 | dependencies.append(dep) | 83 | params['packages'] = ' '.join(packages) |
135 | 84 | dependencies.append('run apt-get -q -y install {packages}'.format(**params)) | ||
136 | 60 | 85 | ||
137 | 61 | # pip | 86 | # pip |
138 | 62 | items = config.get('pip', []) | 87 | items = config.get('pip', []) |
139 | 63 | pip_cache_dir = params["pip_cache_dir"] | ||
140 | 64 | if items: | 88 | if items: |
149 | 65 | for name, ver in sorted(items.items()): | 89 | packages = _get_pip_packages(items) |
150 | 66 | if ver is None: | 90 | pip_cache_dir = params['pip_cache_dir'] |
151 | 67 | p = name | 91 | dependencies.append( |
152 | 68 | else: | 92 | 'run cd {target_path} && pip install --download={pip_cache_dir} ' |
153 | 69 | p = "%s==%s" % (name, ver) | 93 | '--no-install {packages}'.format( |
154 | 70 | dep = "run pip install --download=%s --no-install %s" % ( | 94 | pip_cache_dir=pip_cache_dir, packages=packages, |
155 | 71 | pip_cache_dir, p) | 95 | target_path=params['target_path'])) |
148 | 72 | dependencies.append(dep) | ||
156 | 73 | 96 | ||
157 | 74 | return dependencies | 97 | return dependencies |
158 | 75 | 98 | ||
159 | @@ -85,33 +108,39 @@ | |||
160 | 85 | 108 | ||
161 | 86 | # pip | 109 | # pip |
162 | 87 | items = config.get('pip', []) | 110 | items = config.get('pip', []) |
163 | 88 | pip_cache_dir = params["pip_cache_dir"] | ||
164 | 89 | if items: | 111 | if items: |
173 | 90 | for name, ver in sorted(items.items()): | 112 | packages = _get_pip_packages(items) |
174 | 91 | if ver is None: | 113 | pip_cache_dir = params['pip_cache_dir'] |
175 | 92 | p = name | 114 | dependencies.append( |
176 | 93 | else: | 115 | 'run cd {target_path} && pip install ' |
177 | 94 | p = "%s==%s" % (name, ver) | 116 | '--find-links=file://{pip_cache_dir} --no-index ' |
178 | 95 | dep = "run pip install --find-links=file://%s --no-index %s" % ( | 117 | '{packages}'.format(pip_cache_dir=pip_cache_dir, |
179 | 96 | pip_cache_dir, p) | 118 | packages=packages, |
180 | 97 | dependencies.append(dep) | 119 | target_path=params['target_path'])) |
181 | 98 | 120 | ||
182 | 99 | return dependencies | 121 | return dependencies |
183 | 100 | 122 | ||
184 | 101 | 123 | ||
185 | 102 | def _get_rest(config, **params): | 124 | def _get_rest(config, **params): |
186 | 103 | """The final stuff.""" | 125 | """The final stuff.""" |
191 | 104 | return [] | 126 | return ['cmd cd {target_path} && timeout 300 {test_cmd}'.format(**params)] |
192 | 105 | 127 | ||
193 | 106 | 128 | ||
194 | 107 | def parse(config_text, pip_cache_dir): | 129 | def parse(config_text, pip_cache_dir, project): |
195 | 108 | """Convert Nessita Syntax and return the text for Docker.""" | 130 | """Convert Nessita Syntax and return the text for Docker.""" |
196 | 109 | config = yaml.load(config_text) | 131 | config = yaml.load(config_text) |
197 | 110 | metadata = config.pop("metadata") | 132 | metadata = config.pop("metadata") |
198 | 111 | 133 | ||
199 | 134 | params = { | ||
200 | 135 | 'test_cmd': metadata['test_script'], | ||
201 | 136 | 'target_path': os.path.join('/srv/locolander', project), | ||
202 | 137 | 'pip_cache_dir': pip_cache_dir, | ||
203 | 138 | } | ||
204 | 139 | |||
205 | 112 | # the order for calling these functions is VERY important | 140 | # the order for calling these functions is VERY important |
206 | 113 | functions = [ | 141 | functions = [ |
207 | 114 | _get_base, | 142 | _get_base, |
208 | 143 | _get_code, | ||
209 | 115 | _get_depends_prev, | 144 | _get_depends_prev, |
210 | 116 | _get_secure, | 145 | _get_secure, |
211 | 117 | _get_depends_post, | 146 | _get_depends_post, |
212 | @@ -119,11 +148,20 @@ | |||
213 | 119 | ] | 148 | ] |
214 | 120 | data = [] | 149 | data = [] |
215 | 121 | for func in functions: | 150 | for func in functions: |
217 | 122 | items = func(config, pip_cache_dir=pip_cache_dir) | 151 | items = func(config, **params) |
218 | 123 | data.extend(items) | 152 | data.extend(items) |
219 | 124 | docker = "\n".join(data) | 153 | docker = "\n".join(data) |
220 | 125 | 154 | ||
225 | 126 | # get stuff from the metadata | 155 | return docker |
226 | 127 | script = metadata["test_script"] | 156 | |
227 | 128 | 157 | ||
228 | 129 | return script, docker | 158 | if __name__ == '__main__': |
229 | 159 | import sys | ||
230 | 160 | |||
231 | 161 | project = sys.argv[1] | ||
232 | 162 | infile, outfile = sys.argv[2:] | ||
233 | 163 | |||
234 | 164 | config = open(infile, 'r').read() | ||
235 | 165 | data = parse(config, '/var/cache/locolander', project) | ||
236 | 166 | with open(outfile, 'w') as dockerfile: | ||
237 | 167 | dockerfile.write(data + '\n') | ||
238 | 130 | 168 | ||
239 | === modified file 'requirements.txt' | |||
240 | --- requirements.txt 2013-06-22 19:53:22 +0000 | |||
241 | +++ requirements.txt 2013-07-09 18:45:42 +0000 | |||
242 | @@ -1,7 +1,21 @@ | |||
243 | 1 | distribute==0.6.34 | ||
244 | 1 | Django==1.5.1 | 2 | Django==1.5.1 |
245 | 3 | github3.py==0.7.0 | ||
246 | 4 | httplib2==0.8 | ||
247 | 5 | keyring==1.5 | ||
248 | 6 | launchpadlib==1.10.2 | ||
249 | 7 | lazr.authentication==0.1.2 | ||
250 | 8 | lazr.restfulclient==0.13.3 | ||
251 | 9 | lazr.uri==1.0.3 | ||
252 | 2 | mock==1.0.1 | 10 | mock==1.0.1 |
253 | 11 | oauth==1.0.1 | ||
254 | 3 | pep8==1.4.5 | 12 | pep8==1.4.5 |
255 | 4 | pyflakes==0.7.2 | 13 | pyflakes==0.7.2 |
256 | 14 | PyYAML==3.10 | ||
257 | 15 | requests==1.2.3 | ||
258 | 16 | simplejson==3.3.0 | ||
259 | 5 | South==0.8.1 | 17 | South==0.8.1 |
262 | 6 | github3.py==0.7.0 | 18 | testresources==0.2.7 |
263 | 7 | launchpadlib==1.10.2 | 19 | wadllib==1.3.2 |
264 | 20 | wsgi-intercept==0.5.1 | ||
265 | 21 | zope.interface==4.0.5 | ||
266 | 8 | 22 | ||
267 | === modified file 'run_tests.sh' | |||
268 | --- run_tests.sh 2013-06-22 20:51:04 +0000 | |||
269 | +++ run_tests.sh 2013-07-09 18:45:42 +0000 | |||
270 | @@ -1,7 +1,7 @@ | |||
271 | 1 | #! /bin/bash | 1 | #! /bin/bash |
272 | 2 | 2 | ||
273 | 3 | set -e | 3 | set -e |
274 | 4 | set -x | ||
275 | 5 | 4 | ||
276 | 6 | python locolander/manage.py test locolanderweb | 5 | python locolander/manage.py test locolanderweb |
277 | 6 | python -m unittest discover | ||
278 | 7 | PYTHONPATH=locolander python -m unittest discover -s locolander/repos/ | 7 | PYTHONPATH=locolander python -m unittest discover -s locolander/repos/ |
279 | 8 | 8 | ||
280 | === modified file 'tests/test_ns2df.py' | |||
281 | --- tests/test_ns2df.py 2013-06-22 23:29:05 +0000 | |||
282 | +++ tests/test_ns2df.py 2013-07-09 18:45:42 +0000 | |||
283 | @@ -7,96 +7,120 @@ | |||
284 | 7 | class DependenciesTestCase(unittest.TestCase): | 7 | class DependenciesTestCase(unittest.TestCase): |
285 | 8 | """Tests for the dependencies transformer.""" | 8 | """Tests for the dependencies transformer.""" |
286 | 9 | 9 | ||
287 | 10 | def setUp(self): | ||
288 | 11 | super(DependenciesTestCase, self).setUp() | ||
289 | 12 | self.params = { | ||
290 | 13 | 'pip_cache_dir': '/tmp/pipcache', | ||
291 | 14 | 'target_path': '/srv/locolander/project', | ||
292 | 15 | } | ||
293 | 16 | |||
294 | 10 | def test_no_dependencies(self): | 17 | def test_no_dependencies(self): |
295 | 11 | config = dict(somebase={}) | 18 | config = dict(somebase={}) |
297 | 12 | res = ns2df._get_depends_prev(config, pip_cache_dir="/tmp/pipcache") | 19 | res = ns2df._get_depends_prev(config, **self.params) |
298 | 13 | self.assertEqual(res, []) | 20 | self.assertEqual(res, []) |
300 | 14 | res = ns2df._get_depends_post(config, pip_cache_dir="/tmp/pipcache") | 21 | res = ns2df._get_depends_post(config, **self.params) |
301 | 15 | self.assertEqual(res, []) | 22 | self.assertEqual(res, []) |
302 | 16 | 23 | ||
303 | 17 | def test_apt_one(self): | 24 | def test_apt_one(self): |
304 | 18 | config = dict(somebase={ | 25 | config = dict(somebase={ |
305 | 19 | 'apt': {'bzr': None} | 26 | 'apt': {'bzr': None} |
306 | 20 | }) | 27 | }) |
308 | 21 | res = ns2df._get_depends_prev(config, pip_cache_dir="/tmp/pipcache") | 28 | res = ns2df._get_depends_prev(config, **self.params) |
309 | 22 | self.assertEqual(res, ["run apt-get -q -y install bzr"]) | 29 | self.assertEqual(res, ["run apt-get -q -y install bzr"]) |
310 | 23 | 30 | ||
311 | 24 | def test_apt_several(self): | 31 | def test_apt_several(self): |
312 | 25 | config = dict(somebase={ | 32 | config = dict(somebase={ |
313 | 26 | 'apt': {'bzr': None, 'apache2': None} | 33 | 'apt': {'bzr': None, 'apache2': None} |
314 | 27 | }) | 34 | }) |
316 | 28 | res = ns2df._get_depends_prev(config, pip_cache_dir="/tmp/pipcache") | 35 | res = ns2df._get_depends_prev(config, **self.params) |
317 | 29 | self.assertEqual(res, [ | 36 | self.assertEqual(res, [ |
320 | 30 | "run apt-get -q -y install apache2", | 37 | "run apt-get -q -y install apache2 bzr", |
319 | 31 | "run apt-get -q -y install bzr", | ||
321 | 32 | ]) | 38 | ]) |
322 | 33 | 39 | ||
323 | 34 | def test_apt_versions(self): | 40 | def test_apt_versions(self): |
324 | 35 | config = dict(somebase={ | 41 | config = dict(somebase={ |
325 | 36 | 'apt': {'bzr': None, 'apache2': '3.3-4'} | 42 | 'apt': {'bzr': None, 'apache2': '3.3-4'} |
326 | 37 | }) | 43 | }) |
328 | 38 | res = ns2df._get_depends_prev(config, pip_cache_dir="/tmp/pipcache") | 44 | res = ns2df._get_depends_prev(config, **self.params) |
329 | 39 | self.assertEqual(res, [ | 45 | self.assertEqual(res, [ |
332 | 40 | "run apt-get -q -y install apache2=3.3-4", | 46 | "run apt-get -q -y install apache2=3.3-4 bzr", |
331 | 41 | "run apt-get -q -y install bzr", | ||
333 | 42 | ]) | 47 | ]) |
334 | 43 | 48 | ||
335 | 44 | def test_pip_prev_one(self): | 49 | def test_pip_prev_one(self): |
336 | 45 | config = dict(somebase={ | 50 | config = dict(somebase={ |
337 | 46 | 'pip': {'foo': None} | 51 | 'pip': {'foo': None} |
338 | 47 | }) | 52 | }) |
340 | 48 | res = ns2df._get_depends_prev(config, pip_cache_dir="/tmp/pipcache") | 53 | res = ns2df._get_depends_prev(config, **self.params) |
341 | 49 | self.assertEqual(res, [ | 54 | self.assertEqual(res, [ |
343 | 50 | "run pip install --download=/tmp/pipcache --no-install foo", | 55 | "run cd /srv/locolander/project && pip install --download=/tmp/pipcache --no-install foo", |
344 | 51 | ]) | 56 | ]) |
345 | 52 | 57 | ||
346 | 53 | def test_pip_prev_several(self): | 58 | def test_pip_prev_several(self): |
347 | 54 | config = dict(somebase={ | 59 | config = dict(somebase={ |
348 | 55 | 'pip': {'foo': None, 'bar': None} | 60 | 'pip': {'foo': None, 'bar': None} |
349 | 56 | }) | 61 | }) |
351 | 57 | res = ns2df._get_depends_prev(config, pip_cache_dir="/tmp/pipcache") | 62 | res = ns2df._get_depends_prev(config, **self.params) |
352 | 58 | self.assertEqual(res, [ | 63 | self.assertEqual(res, [ |
355 | 59 | "run pip install --download=/tmp/pipcache --no-install bar", | 64 | "run cd /srv/locolander/project && pip install --download=/tmp/pipcache --no-install bar foo", |
354 | 60 | "run pip install --download=/tmp/pipcache --no-install foo", | ||
356 | 61 | ]) | 65 | ]) |
357 | 62 | 66 | ||
358 | 63 | def test_pip_prev_versions(self): | 67 | def test_pip_prev_versions(self): |
359 | 64 | config = dict(somebase={ | 68 | config = dict(somebase={ |
360 | 65 | 'pip': {'foo': None, 'bar': '2.1'} | 69 | 'pip': {'foo': None, 'bar': '2.1'} |
361 | 66 | }) | 70 | }) |
366 | 67 | res = ns2df._get_depends_prev(config, pip_cache_dir="/tmp/pipcache") | 71 | res = ns2df._get_depends_prev(config, **self.params) |
367 | 68 | self.assertEqual(res, [ | 72 | self.assertEqual(res, [ |
368 | 69 | "run pip install --download=/tmp/pipcache --no-install bar==2.1", | 73 | ("run cd /srv/locolander/project && pip install --download=/tmp/pipcache --no-install " |
369 | 70 | "run pip install --download=/tmp/pipcache --no-install foo", | 74 | "bar==2.1 foo"), |
370 | 75 | ]) | ||
371 | 76 | |||
372 | 77 | def test_pip_prev_requirements(self): | ||
373 | 78 | config = dict(somebase={ | ||
374 | 79 | 'pip': '-r requirements.txt' | ||
375 | 80 | }) | ||
376 | 81 | res = ns2df._get_depends_prev(config, **self.params) | ||
377 | 82 | self.assertEqual(res, [ | ||
378 | 83 | ("run cd /srv/locolander/project && pip install --download=/tmp/pipcache --no-install " | ||
379 | 84 | "-r requirements.txt"), | ||
380 | 71 | ]) | 85 | ]) |
381 | 72 | 86 | ||
382 | 73 | def test_pip_post_one(self): | 87 | def test_pip_post_one(self): |
383 | 74 | config = dict(somebase={ | 88 | config = dict(somebase={ |
384 | 75 | 'pip': {'foo': None} | 89 | 'pip': {'foo': None} |
385 | 76 | }) | 90 | }) |
387 | 77 | res = ns2df._get_depends_post(config, pip_cache_dir="/tmp/pipcache") | 91 | res = ns2df._get_depends_post(config, **self.params) |
388 | 78 | self.assertEqual(res, [ | 92 | self.assertEqual(res, [ |
390 | 79 | "run pip install --find-links=file:///tmp/pipcache --no-index foo" | 93 | "run cd /srv/locolander/project && pip install --find-links=file:///tmp/pipcache --no-index foo" |
391 | 80 | ]) | 94 | ]) |
392 | 81 | 95 | ||
393 | 82 | def test_pip_post_several(self): | 96 | def test_pip_post_several(self): |
394 | 83 | config = dict(somebase={ | 97 | config = dict(somebase={ |
395 | 84 | 'pip': {'foo': None, 'bar': None} | 98 | 'pip': {'foo': None, 'bar': None} |
396 | 85 | }) | 99 | }) |
398 | 86 | res = ns2df._get_depends_post(config, pip_cache_dir="/tmp/pipcache") | 100 | res = ns2df._get_depends_post(config, **self.params) |
399 | 87 | self.assertEqual(res, [ | 101 | self.assertEqual(res, [ |
402 | 88 | "run pip install --find-links=file:///tmp/pipcache --no-index bar", | 102 | ("run cd /srv/locolander/project && pip install --find-links=file:///tmp/pipcache --no-index " |
403 | 89 | "run pip install --find-links=file:///tmp/pipcache --no-index foo", | 103 | "bar foo"), |
404 | 90 | ]) | 104 | ]) |
405 | 91 | 105 | ||
406 | 92 | def test_pip_post_versions(self): | 106 | def test_pip_post_versions(self): |
407 | 93 | config = dict(somebase={ | 107 | config = dict(somebase={ |
408 | 94 | 'pip': {'foo': None, 'bar': '2.1'} | 108 | 'pip': {'foo': None, 'bar': '2.1'} |
409 | 95 | }) | 109 | }) |
414 | 96 | res = ns2df._get_depends_post(config, pip_cache_dir="/tmcache") | 110 | res = ns2df._get_depends_post(config, **self.params) |
415 | 97 | self.assertEqual(res, [ | 111 | self.assertEqual(res, [ |
416 | 98 | "run pip install --find-links=file:///tmcache --no-index bar==2.1", | 112 | ("run cd /srv/locolander/project && pip install --find-links=file:///tmp/pipcache --no-index " |
417 | 99 | "run pip install --find-links=file:///tmcache --no-index foo", | 113 | "bar==2.1 foo"), |
418 | 114 | ]) | ||
419 | 115 | |||
420 | 116 | def test_pip_post_requirements(self): | ||
421 | 117 | config = dict(somebase={ | ||
422 | 118 | 'pip': '-r requirements.txt' | ||
423 | 119 | }) | ||
424 | 120 | res = ns2df._get_depends_post(config, **self.params) | ||
425 | 121 | self.assertEqual(res, [ | ||
426 | 122 | ("run cd /srv/locolander/project && pip install --find-links=file:///tmp/pipcache --no-index " | ||
427 | 123 | "-r requirements.txt"), | ||
428 | 100 | ]) | 124 | ]) |
429 | 101 | 125 | ||
430 | 102 | def test_mixed_prev(self): | 126 | def test_mixed_prev(self): |
431 | @@ -104,12 +128,10 @@ | |||
432 | 104 | 'apt': {'bzr': None, 'apache2': '3.3-4'}, | 128 | 'apt': {'bzr': None, 'apache2': '3.3-4'}, |
433 | 105 | 'pip': {'foo': None, 'bar': None}, | 129 | 'pip': {'foo': None, 'bar': None}, |
434 | 106 | }) | 130 | }) |
436 | 107 | res = ns2df._get_depends_prev(config, pip_cache_dir="/tmp/pipcache") | 131 | res = ns2df._get_depends_prev(config, **self.params) |
437 | 108 | self.assertEqual(res, [ | 132 | self.assertEqual(res, [ |
442 | 109 | "run apt-get -q -y install apache2=3.3-4", | 133 | "run apt-get -q -y install apache2=3.3-4 bzr", |
443 | 110 | "run apt-get -q -y install bzr", | 134 | "run cd /srv/locolander/project && pip install --download=/tmp/pipcache --no-install bar foo", |
440 | 111 | "run pip install --download=/tmp/pipcache --no-install bar", | ||
441 | 112 | "run pip install --download=/tmp/pipcache --no-install foo", | ||
444 | 113 | ]) | 135 | ]) |
445 | 114 | 136 | ||
446 | 115 | def test_mixed_post(self): | 137 | def test_mixed_post(self): |
447 | @@ -117,10 +139,10 @@ | |||
448 | 117 | 'apt': {'bzr': None, 'apache2': '3.3-4'}, | 139 | 'apt': {'bzr': None, 'apache2': '3.3-4'}, |
449 | 118 | 'pip': {'foo': None, 'bar': None}, | 140 | 'pip': {'foo': None, 'bar': None}, |
450 | 119 | }) | 141 | }) |
452 | 120 | res = ns2df._get_depends_post(config, pip_cache_dir="/tmp/pipcache") | 142 | res = ns2df._get_depends_post(config, **self.params) |
453 | 121 | self.assertEqual(res, [ | 143 | self.assertEqual(res, [ |
456 | 122 | "run pip install --find-links=file:///tmp/pipcache --no-index bar", | 144 | ("run cd /srv/locolander/project && pip install --find-links=file:///tmp/pipcache --no-index " |
457 | 123 | "run pip install --find-links=file:///tmp/pipcache --no-index foo", | 145 | "bar foo"), |
458 | 124 | ]) | 146 | ]) |
459 | 125 | 147 | ||
460 | 126 | 148 | ||
461 | @@ -130,7 +152,7 @@ | |||
462 | 130 | def test_one_base(self): | 152 | def test_one_base(self): |
463 | 131 | config = dict(precise={}) | 153 | config = dict(precise={}) |
464 | 132 | res = ns2df._get_base(config) | 154 | res = ns2df._get_base(config) |
466 | 133 | self.assertEqual(res, ["from ubuntu:precise"]) | 155 | self.assertEqual(res, ["from locolander:precise"]) |
467 | 134 | 156 | ||
468 | 135 | def test_several_bases(self): | 157 | def test_several_bases(self): |
469 | 136 | config = dict(base1={}, base2={}) | 158 | config = dict(base1={}, base2={}) |
470 | @@ -141,8 +163,10 @@ | |||
471 | 141 | self.assertEqual(res, ["run ip link set dev eth0 down"]) | 163 | self.assertEqual(res, ["run ip link set dev eth0 down"]) |
472 | 142 | 164 | ||
473 | 143 | def test_rest(self): | 165 | def test_rest(self): |
476 | 144 | res = ns2df._get_rest({}) | 166 | res = ns2df._get_rest({}, |
477 | 145 | self.assertEqual(res, []) | 167 | target_path='/srv/locolander', |
478 | 168 | test_cmd='./run_tests.sh') | ||
479 | 169 | self.assertEqual(res, ["cmd cd /srv/locolander && timeout 300 ./run_tests.sh"]) | ||
480 | 146 | 170 | ||
481 | 147 | def test_all_mixed(self): | 171 | def test_all_mixed(self): |
482 | 148 | config_text = """ | 172 | config_text = """ |
483 | @@ -155,14 +179,16 @@ | |||
484 | 155 | metadata: | 179 | metadata: |
485 | 156 | test_script: foo | 180 | test_script: foo |
486 | 157 | """ | 181 | """ |
489 | 158 | script, docker = ns2df.parse(config_text, "/tmp/pipcache") | 182 | docker = ns2df.parse(config_text, "/tmp/pipcache", 'project') |
488 | 159 | self.assertEqual(script, "foo") | ||
490 | 160 | self.assertEqual(docker, | 183 | self.assertEqual(docker, |
495 | 161 | "from ubuntu:precise\n" | 184 | "from locolander:precise\n" |
496 | 162 | "run apt-get -q -y install apache2=3.3-4\n" | 185 | "run mkdir -p /srv/locolander/project\n" |
497 | 163 | "run apt-get -q -y install bzr\n" | 186 | "add . /srv/locolander/project\n" |
498 | 164 | "run pip install --download=/tmp/pipcache --no-install foobar\n" | 187 | "run chown -R locolander.locolander /srv/locolander/project\n" |
499 | 188 | "run apt-get -q -y install apache2=3.3-4 bzr\n" | ||
500 | 189 | "run cd /srv/locolander/project && pip install --download=/tmp/pipcache --no-install foobar\n" | ||
501 | 165 | "run ip link set dev eth0 down\n" | 190 | "run ip link set dev eth0 down\n" |
504 | 166 | "run pip install --find-links=file:///tmp/pipcache " | 191 | "run cd /srv/locolander/project && pip install --find-links=file:///tmp/pipcache " |
505 | 167 | "--no-index foobar" | 192 | "--no-index foobar\n" |
506 | 193 | "cmd cd /srv/locolander/project && timeout 300 foo" | ||
507 | 168 | ) | 194 | ) |
I see you're changing back to put all apt-get installations in the same line... you told me that we should put them all in different lines because of docker image caching... did that change?
The rest looks ok!