Merge ~artivis/lpci:feature/ros-plugins into lpci:main
- Git
- lp:~artivis/lpci
- feature/ros-plugins
- Merge into main
Status: | Needs review |
---|---|
Proposed branch: | ~artivis/lpci:feature/ros-plugins |
Merge into: | lpci:main |
Diff against target: |
1087 lines (+1041/-2) 2 files modified
lpcraft/plugin/tests/test_plugins.py (+765/-2) lpcraft/plugins/plugins.py (+276/-0) |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Jürgen Gmach | Approve | ||
Review via email: mp+435371@code.launchpad.net |
Commit message
Description of the change
This implements three new plugins for building and testing ROS & ROS 2 projects, namely,
- CatkinPlugin - based on the 'catkin' build tool
- CatkinToolsPlugin - based on the 'catkin-tools' build tool
- ColconPlugin - based on the 'colcon' build tool
These plugins all share a common base class - RosBasePlugin - factoring some common code.
The three plugins follow the same principle,
- Install the ROS project's dependencies during the 'run-before' step
- Build the ROS project during the 'run' step
- Build and run the tests during the 'run-after' step (optional)
The later is optional and depends on a plugin config parameter.
Beside some ROS-specific environment variables, the base class define the following env. vars.,
- ROS_PROJECT_BASE - the parent directory of the lpcraft build tree
- ROS_PROJECT_SRC - the directory of the project
- ROS_PROJECT_BUILD - the directory of the plugins build tree
- ROS_PROJECT_INSTALL - the directory of the plugins install tree
They somewhat mimic env. vars. in snapcraft allowing for out-of-source build/install. They also allow the end user to more easily override the run commands not having to figure out the internal paths. They could be replaced in the future by lpcraft-wide env. vars.
jeremie (artivis) wrote : | # |
Hello,
I've already tackled the missed lines in coverage and the CI is happy as far as I can tell.
We are indeed planning on using those plugins in Launchpad CI.
This MP is stand-alone and totally independent of https:/
Looking forward to your review :+1:
Jürgen Gmach (jugmac00) wrote : | # |
Looks good with some minor comments!
As we have no merge bot activated for lpcraft, I think I need to merge the MP.
Just tell me when you think it is ready to be merged.
Note for myself:
- In a follow-up MP we should split the plugins and the tests into submodules.
- In a follow-up MP we should revisit the currently used exceptions.
Preview Diff
1 | diff --git a/lpcraft/plugin/tests/test_plugins.py b/lpcraft/plugin/tests/test_plugins.py |
2 | index 1def73b..f7ba1b8 100644 |
3 | --- a/lpcraft/plugin/tests/test_plugins.py |
4 | +++ b/lpcraft/plugin/tests/test_plugins.py |
5 | @@ -5,7 +5,7 @@ import os |
6 | import subprocess |
7 | from pathlib import Path, PosixPath |
8 | from textwrap import dedent |
9 | -from typing import List, Optional, Union, cast |
10 | +from typing import Any, Dict, List, Optional, Union, cast |
11 | from unittest.mock import ANY, Mock, call, patch |
12 | |
13 | from craft_providers.lxd import launch |
14 | @@ -17,7 +17,7 @@ from lpcraft.commands.tests import CommandBaseTestCase |
15 | from lpcraft.errors import ConfigurationError |
16 | from lpcraft.plugin.manager import get_plugin_manager |
17 | from lpcraft.plugins import register |
18 | -from lpcraft.plugins.plugins import BaseConfig, BasePlugin |
19 | +from lpcraft.plugins.plugins import BaseConfig, BasePlugin, RosBasePlugin |
20 | from lpcraft.providers.tests import makeLXDProvider |
21 | |
22 | |
23 | @@ -1080,3 +1080,766 @@ class TestPlugins(CommandBaseTestCase): |
24 | lpcraft.config.Config.load, |
25 | config_path, |
26 | ) |
27 | + |
28 | + def _get_ros_before_run_call(self, expected_env: Dict[str, str]) -> Any: |
29 | + return call( |
30 | + [ |
31 | + "bash", |
32 | + "--noprofile", |
33 | + "--norc", |
34 | + "-ec", |
35 | + "if [ -f /opt/ros/*/setup.sh ]; then source /opt/ros/*/setup.sh; fi" # noqa:E501 |
36 | + "\nif [ ! -f /etc/ros/rosdep/sources.list.d/20-default.list ]; then\nrosdep init\nfi" # noqa:E501 |
37 | + "\nrosdep update --rosdistro ${ROS_DISTRO}" |
38 | + "\nrosdep install -i -y --rosdistro ${ROS_DISTRO} --from-paths .", # noqa:E501 |
39 | + ], |
40 | + cwd=PosixPath("/build/lpcraft/project"), |
41 | + env=expected_env, |
42 | + stdout=ANY, |
43 | + stderr=ANY, |
44 | + ) |
45 | + |
46 | + def _get_ros_packages_call( |
47 | + self, expected_env: Dict[str, str], packages: List[str] |
48 | + ) -> Any: |
49 | + return call( |
50 | + [ |
51 | + "apt", |
52 | + "install", |
53 | + "-y", |
54 | + "build-essential", |
55 | + "cmake", |
56 | + "g++", |
57 | + "python3-rosdep", |
58 | + ] |
59 | + + packages, |
60 | + cwd=PosixPath("/build/lpcraft/project"), |
61 | + env=expected_env, |
62 | + stdout=ANY, |
63 | + stderr=ANY, |
64 | + ) |
65 | + |
66 | + def _get_ros_expected_env( |
67 | + self, ros_distro: str, ros_python_version: str, ros_version: str |
68 | + ) -> Dict[str, str]: |
69 | + return { |
70 | + "ROS_PROJECT_BASE": "/build/lpcraft", |
71 | + "ROS_PROJECT_SRC": "/build/lpcraft/project", |
72 | + "ROS_PROJECT_BUILD": "/build/lpcraft/build", |
73 | + "ROS_PROJECT_INSTALL": "/build/lpcraft/install", |
74 | + "ROS_DISTRO": ros_distro, |
75 | + "ROS_PYTHON_VERSION": ros_python_version, |
76 | + "ROS_VERSION": ros_version, |
77 | + } |
78 | + |
79 | + def test_fake_ros_plugin_no_version(self): |
80 | + config = dedent( |
81 | + """ |
82 | + pipeline: |
83 | + - build |
84 | + |
85 | + jobs: |
86 | + build: |
87 | + plugin: fake-ros-plugin |
88 | + series: bionic |
89 | + architectures: amd64 |
90 | + """ |
91 | + ) |
92 | + config_path = Path(".launchpad.yaml") |
93 | + config_path.write_text(config) |
94 | + |
95 | + @register(name="fake-ros-plugin") |
96 | + class FakeRosPlugin(RosBasePlugin): |
97 | + pass |
98 | + |
99 | + config_obj = lpcraft.config.Config.load(config_path) |
100 | + job = config_obj.jobs["build"][0] |
101 | + pm = get_plugin_manager(job) |
102 | + plugins = pm.get_plugins() |
103 | + fake_plugin = [ |
104 | + _ for _ in plugins if _.__class__.__name__ == "FakeRosPlugin" |
105 | + ] |
106 | + self.assertEqual(job.plugin, "fake-ros-plugin") |
107 | + self.assertRaises(NotImplementedError, fake_plugin[0]._get_ros_version) |
108 | + |
109 | + def test_fake_ros_plugin_bad_version(self): |
110 | + config = dedent( |
111 | + """ |
112 | + pipeline: |
113 | + - build |
114 | + |
115 | + jobs: |
116 | + build: |
117 | + plugin: fake-ros-plugin |
118 | + series: bionic |
119 | + architectures: amd64 |
120 | + """ |
121 | + ) |
122 | + config_path = Path(".launchpad.yaml") |
123 | + config_path.write_text(config) |
124 | + |
125 | + @register(name="fake-ros-plugin") |
126 | + class FakeRosPlugin(RosBasePlugin): |
127 | + def _get_ros_version(self) -> int: |
128 | + return 3 |
129 | + |
130 | + config_obj = lpcraft.config.Config.load(config_path) |
131 | + job = config_obj.jobs["build"][0] |
132 | + pm = get_plugin_manager(job) |
133 | + plugins = pm.get_plugins() |
134 | + fake_plugin = [ |
135 | + _ for _ in plugins if _.__class__.__name__ == "FakeRosPlugin" |
136 | + ] |
137 | + self.assertEqual(job.plugin, "fake-ros-plugin") |
138 | + self.assertEqual(3, fake_plugin[0]._get_ros_version()) |
139 | + self.assertRaises(Exception, fake_plugin[0]._get_ros_distro) |
140 | + |
141 | + @patch("lpcraft.commands.run.get_provider") |
142 | + @patch("lpcraft.commands.run.get_host_architecture", return_value="amd64") |
143 | + def test_catkin_plugin_bionic( |
144 | + self, mock_get_host_architecture, mock_get_provider |
145 | + ): |
146 | + launcher = Mock(spec=launch) |
147 | + provider = makeLXDProvider(lxd_launcher=launcher) |
148 | + mock_get_provider.return_value = provider |
149 | + execute_run = launcher.return_value.execute_run |
150 | + execute_run.return_value = subprocess.CompletedProcess([], 0) |
151 | + config = dedent( |
152 | + """ |
153 | + pipeline: |
154 | + - build |
155 | + |
156 | + jobs: |
157 | + build: |
158 | + plugin: catkin |
159 | + series: bionic |
160 | + architectures: amd64 |
161 | + """ |
162 | + ) |
163 | + Path(".launchpad.yaml").write_text(config) |
164 | + |
165 | + expected_env = self._get_ros_expected_env("melodic", "2", "1") |
166 | + |
167 | + self.run_command("run") |
168 | + self.assertEqual( |
169 | + [ |
170 | + call( |
171 | + ["apt", "update"], |
172 | + cwd=PosixPath("/build/lpcraft/project"), |
173 | + env=expected_env, |
174 | + stdout=ANY, |
175 | + stderr=ANY, |
176 | + ), |
177 | + self._get_ros_packages_call( |
178 | + expected_env, ["ros-melodic-catkin"] |
179 | + ), |
180 | + self._get_ros_before_run_call(expected_env), |
181 | + call( |
182 | + [ |
183 | + "bash", |
184 | + "--noprofile", |
185 | + "--norc", |
186 | + "-ec", |
187 | + "if [ -f /opt/ros/*/setup.sh ]; then source /opt/ros/*/setup.sh; fi" # noqa:E501 |
188 | + "\ncatkin_make_isolated --source-space ${ROS_PROJECT_SRC} --directory ${ROS_PROJECT_BASE}", # noqa:E501 |
189 | + ], |
190 | + cwd=PosixPath("/build/lpcraft/project"), |
191 | + env=expected_env, |
192 | + stdout=ANY, |
193 | + stderr=ANY, |
194 | + ), |
195 | + ], |
196 | + execute_run.call_args_list, |
197 | + ) |
198 | + |
199 | + @patch("lpcraft.commands.run.get_provider") |
200 | + @patch("lpcraft.commands.run.get_host_architecture", return_value="amd64") |
201 | + def test_catkin_plugin( |
202 | + self, mock_get_host_architecture, mock_get_provider |
203 | + ): |
204 | + launcher = Mock(spec=launch) |
205 | + provider = makeLXDProvider(lxd_launcher=launcher) |
206 | + mock_get_provider.return_value = provider |
207 | + execute_run = launcher.return_value.execute_run |
208 | + execute_run.return_value = subprocess.CompletedProcess([], 0) |
209 | + config = dedent( |
210 | + """ |
211 | + pipeline: |
212 | + - build |
213 | + |
214 | + jobs: |
215 | + build: |
216 | + plugin: catkin |
217 | + series: focal |
218 | + architectures: amd64 |
219 | + """ |
220 | + ) |
221 | + Path(".launchpad.yaml").write_text(config) |
222 | + |
223 | + expected_env = self._get_ros_expected_env("noetic", "3", "1") |
224 | + |
225 | + self.run_command("run") |
226 | + self.assertEqual( |
227 | + [ |
228 | + call( |
229 | + ["apt", "update"], |
230 | + cwd=PosixPath("/build/lpcraft/project"), |
231 | + env=expected_env, |
232 | + stdout=ANY, |
233 | + stderr=ANY, |
234 | + ), |
235 | + self._get_ros_packages_call( |
236 | + expected_env, ["ros-noetic-catkin"] |
237 | + ), |
238 | + self._get_ros_before_run_call(expected_env), |
239 | + call( |
240 | + [ |
241 | + "bash", |
242 | + "--noprofile", |
243 | + "--norc", |
244 | + "-ec", |
245 | + "if [ -f /opt/ros/*/setup.sh ]; then source /opt/ros/*/setup.sh; fi" # noqa:E501 |
246 | + "\ncatkin_make_isolated --source-space ${ROS_PROJECT_SRC} --directory ${ROS_PROJECT_BASE}", # noqa:E501 |
247 | + ], |
248 | + cwd=PosixPath("/build/lpcraft/project"), |
249 | + env=expected_env, |
250 | + stdout=ANY, |
251 | + stderr=ANY, |
252 | + ), |
253 | + ], |
254 | + execute_run.call_args_list, |
255 | + ) |
256 | + |
257 | + @patch("lpcraft.commands.run.get_provider") |
258 | + @patch("lpcraft.commands.run.get_host_architecture", return_value="amd64") |
259 | + def test_catkin_plugin_with_tests( |
260 | + self, mock_get_host_architecture, mock_get_provider |
261 | + ): |
262 | + launcher = Mock(spec=launch) |
263 | + provider = makeLXDProvider(lxd_launcher=launcher) |
264 | + mock_get_provider.return_value = provider |
265 | + execute_run = launcher.return_value.execute_run |
266 | + execute_run.return_value = subprocess.CompletedProcess([], 0) |
267 | + config = dedent( |
268 | + """ |
269 | + pipeline: |
270 | + - build |
271 | + |
272 | + jobs: |
273 | + build: |
274 | + plugin: catkin |
275 | + run-tests: True |
276 | + series: focal |
277 | + architectures: amd64 |
278 | + packages: [file, git] |
279 | + """ |
280 | + ) |
281 | + Path(".launchpad.yaml").write_text(config) |
282 | + |
283 | + expected_env = self._get_ros_expected_env("noetic", "3", "1") |
284 | + |
285 | + self.run_command("run") |
286 | + self.assertEqual( |
287 | + [ |
288 | + call( |
289 | + ["apt", "update"], |
290 | + cwd=PosixPath("/build/lpcraft/project"), |
291 | + env=expected_env, |
292 | + stdout=ANY, |
293 | + stderr=ANY, |
294 | + ), |
295 | + self._get_ros_packages_call( |
296 | + expected_env, ["ros-noetic-catkin", "file", "git"] |
297 | + ), |
298 | + self._get_ros_before_run_call(expected_env), |
299 | + call( |
300 | + [ |
301 | + "bash", |
302 | + "--noprofile", |
303 | + "--norc", |
304 | + "-ec", |
305 | + "if [ -f /opt/ros/*/setup.sh ]; then source /opt/ros/*/setup.sh; fi" # noqa:E501 |
306 | + "\ncatkin_make_isolated --source-space ${ROS_PROJECT_SRC} --directory ${ROS_PROJECT_BASE}", # noqa:E501 |
307 | + ], |
308 | + cwd=PosixPath("/build/lpcraft/project"), |
309 | + env=expected_env, |
310 | + stdout=ANY, |
311 | + stderr=ANY, |
312 | + ), |
313 | + call( |
314 | + [ |
315 | + "bash", |
316 | + "--noprofile", |
317 | + "--norc", |
318 | + "-ec", |
319 | + "\nsource ${ROS_PROJECT_BASE}/devel_isolated/setup.sh" |
320 | + "\ncatkin_make_isolated --source-space ${ROS_PROJECT_SRC} --directory ${ROS_PROJECT_BASE} --force-cmake --catkin-make-args run_tests" # noqa:E501 |
321 | + "\ncatkin_test_results --verbose ${ROS_PROJECT_BASE}/build_isolated/", # noqa:E501 |
322 | + ], |
323 | + cwd=PosixPath("/build/lpcraft/project"), |
324 | + env=expected_env, |
325 | + stdout=ANY, |
326 | + stderr=ANY, |
327 | + ), |
328 | + ], |
329 | + execute_run.call_args_list, |
330 | + ) |
331 | + |
332 | + @patch("lpcraft.commands.run.get_provider") |
333 | + @patch("lpcraft.commands.run.get_host_architecture", return_value="amd64") |
334 | + def test_catkin_plugin_user_command( |
335 | + self, mock_get_host_architecture, mock_get_provider |
336 | + ): |
337 | + launcher = Mock(spec=launch) |
338 | + provider = makeLXDProvider(lxd_launcher=launcher) |
339 | + mock_get_provider.return_value = provider |
340 | + execute_run = launcher.return_value.execute_run |
341 | + execute_run.return_value = subprocess.CompletedProcess([], 0) |
342 | + config = dedent( |
343 | + """ |
344 | + pipeline: |
345 | + - build |
346 | + |
347 | + jobs: |
348 | + build: |
349 | + plugin: catkin |
350 | + run-tests: True |
351 | + series: focal |
352 | + architectures: amd64 |
353 | + packages: [file, git] |
354 | + run-before: echo 'hello' |
355 | + run: echo 'robot' |
356 | + # run-after: "we shouldn't get anything unless uncommented" |
357 | + """ |
358 | + ) |
359 | + Path(".launchpad.yaml").write_text(config) |
360 | + |
361 | + expected_env = self._get_ros_expected_env("noetic", "3", "1") |
362 | + |
363 | + self.run_command("run") |
364 | + self.assertEqual( |
365 | + [ |
366 | + call( |
367 | + ["apt", "update"], |
368 | + cwd=PosixPath("/build/lpcraft/project"), |
369 | + env=expected_env, |
370 | + stdout=ANY, |
371 | + stderr=ANY, |
372 | + ), |
373 | + self._get_ros_packages_call( |
374 | + expected_env, ["ros-noetic-catkin", "file", "git"] |
375 | + ), |
376 | + call( |
377 | + ["bash", "--noprofile", "--norc", "-ec", "echo 'hello'"], |
378 | + cwd=PosixPath("/build/lpcraft/project"), |
379 | + env=expected_env, |
380 | + stdout=ANY, |
381 | + stderr=ANY, |
382 | + ), |
383 | + call( |
384 | + ["bash", "--noprofile", "--norc", "-ec", "echo 'robot'"], |
385 | + cwd=PosixPath("/build/lpcraft/project"), |
386 | + env=expected_env, |
387 | + stdout=ANY, |
388 | + stderr=ANY, |
389 | + ), |
390 | + ], |
391 | + execute_run.call_args_list, |
392 | + ) |
393 | + |
394 | + @patch("lpcraft.commands.run.get_provider") |
395 | + @patch("lpcraft.commands.run.get_host_architecture", return_value="amd64") |
396 | + def test_catkin_tools_plugin( |
397 | + self, mock_get_host_architecture, mock_get_provider |
398 | + ): |
399 | + launcher = Mock(spec=launch) |
400 | + provider = makeLXDProvider(lxd_launcher=launcher) |
401 | + mock_get_provider.return_value = provider |
402 | + execute_run = launcher.return_value.execute_run |
403 | + execute_run.return_value = subprocess.CompletedProcess([], 0) |
404 | + config = dedent( |
405 | + """ |
406 | + pipeline: |
407 | + - build |
408 | + |
409 | + jobs: |
410 | + build: |
411 | + plugin: catkin-tools |
412 | + series: focal |
413 | + architectures: amd64 |
414 | + """ |
415 | + ) |
416 | + Path(".launchpad.yaml").write_text(config) |
417 | + |
418 | + expected_env = self._get_ros_expected_env("noetic", "3", "1") |
419 | + |
420 | + self.run_command("run") |
421 | + self.assertEqual( |
422 | + [ |
423 | + call( |
424 | + ["apt", "update"], |
425 | + cwd=PosixPath("/build/lpcraft/project"), |
426 | + env=expected_env, |
427 | + stdout=ANY, |
428 | + stderr=ANY, |
429 | + ), |
430 | + self._get_ros_packages_call( |
431 | + expected_env, ["python3-catkin-tools"] |
432 | + ), |
433 | + self._get_ros_before_run_call(expected_env), |
434 | + call( |
435 | + [ |
436 | + "bash", |
437 | + "--noprofile", |
438 | + "--norc", |
439 | + "-ec", |
440 | + "if [ -f /opt/ros/*/setup.sh ]; then source /opt/ros/*/setup.sh; fi\n" # noqa:E501 |
441 | + "catkin init --workspace ${ROS_PROJECT_BASE}\n" |
442 | + "catkin profile add --force lpcraft\n" |
443 | + "catkin config --profile lpcraft --install --source-space ${ROS_PROJECT_SRC} --build-space ${ROS_PROJECT_BUILD} --install-space ${ROS_PROJECT_INSTALL}\n" # noqa:E501 |
444 | + "catkin build --no-notify --profile lpcraft", # noqa:E501 |
445 | + ], |
446 | + cwd=PosixPath("/build/lpcraft/project"), |
447 | + env=expected_env, |
448 | + stdout=ANY, |
449 | + stderr=ANY, |
450 | + ), |
451 | + ], |
452 | + execute_run.call_args_list, |
453 | + ) |
454 | + |
455 | + @patch("lpcraft.commands.run.get_provider") |
456 | + @patch("lpcraft.commands.run.get_host_architecture", return_value="amd64") |
457 | + def test_catkin_tools_plugin_with_tests( |
458 | + self, mock_get_host_architecture, mock_get_provider |
459 | + ): |
460 | + launcher = Mock(spec=launch) |
461 | + provider = makeLXDProvider(lxd_launcher=launcher) |
462 | + mock_get_provider.return_value = provider |
463 | + execute_run = launcher.return_value.execute_run |
464 | + execute_run.return_value = subprocess.CompletedProcess([], 0) |
465 | + config = dedent( |
466 | + """ |
467 | + pipeline: |
468 | + - build |
469 | + |
470 | + jobs: |
471 | + build: |
472 | + plugin: catkin-tools |
473 | + run-tests: True |
474 | + series: focal |
475 | + architectures: amd64 |
476 | + packages: [file, git] |
477 | + """ |
478 | + ) |
479 | + Path(".launchpad.yaml").write_text(config) |
480 | + |
481 | + expected_env = self._get_ros_expected_env("noetic", "3", "1") |
482 | + |
483 | + self.run_command("run") |
484 | + self.assertEqual( |
485 | + [ |
486 | + call( |
487 | + ["apt", "update"], |
488 | + cwd=PosixPath("/build/lpcraft/project"), |
489 | + env=expected_env, |
490 | + stdout=ANY, |
491 | + stderr=ANY, |
492 | + ), |
493 | + self._get_ros_packages_call( |
494 | + expected_env, ["python3-catkin-tools", "file", "git"] |
495 | + ), |
496 | + self._get_ros_before_run_call(expected_env), |
497 | + call( |
498 | + [ |
499 | + "bash", |
500 | + "--noprofile", |
501 | + "--norc", |
502 | + "-ec", |
503 | + "if [ -f /opt/ros/*/setup.sh ]; then source /opt/ros/*/setup.sh; fi\n" # noqa:E501 |
504 | + "catkin init --workspace ${ROS_PROJECT_BASE}\n" |
505 | + "catkin profile add --force lpcraft\n" |
506 | + "catkin config --profile lpcraft --install --source-space ${ROS_PROJECT_SRC} --build-space ${ROS_PROJECT_BUILD} --install-space ${ROS_PROJECT_INSTALL}\n" # noqa:E501 |
507 | + "catkin build --no-notify --profile lpcraft", # noqa:E501 |
508 | + ], |
509 | + cwd=PosixPath("/build/lpcraft/project"), |
510 | + env=expected_env, |
511 | + stdout=ANY, |
512 | + stderr=ANY, |
513 | + ), |
514 | + call( |
515 | + [ |
516 | + "bash", |
517 | + "--noprofile", |
518 | + "--norc", |
519 | + "-ec", |
520 | + "\nsource ${ROS_PROJECT_BASE}/devel/setup.sh\n" |
521 | + "catkin test --profile lpcraft --summary", |
522 | + ], |
523 | + cwd=PosixPath("/build/lpcraft/project"), |
524 | + env=expected_env, |
525 | + stdout=ANY, |
526 | + stderr=ANY, |
527 | + ), |
528 | + ], |
529 | + execute_run.call_args_list, |
530 | + ) |
531 | + |
532 | + @patch("lpcraft.commands.run.get_provider") |
533 | + @patch("lpcraft.commands.run.get_host_architecture", return_value="amd64") |
534 | + def test_catkin_tools_plugin_user_command( |
535 | + self, mock_get_host_architecture, mock_get_provider |
536 | + ): |
537 | + launcher = Mock(spec=launch) |
538 | + provider = makeLXDProvider(lxd_launcher=launcher) |
539 | + mock_get_provider.return_value = provider |
540 | + execute_run = launcher.return_value.execute_run |
541 | + execute_run.return_value = subprocess.CompletedProcess([], 0) |
542 | + config = dedent( |
543 | + """ |
544 | + pipeline: |
545 | + - build |
546 | + |
547 | + jobs: |
548 | + build: |
549 | + plugin: catkin-tools |
550 | + run-tests: True |
551 | + series: focal |
552 | + architectures: amd64 |
553 | + packages: [file, git] |
554 | + run-before: echo 'hello' |
555 | + run: echo 'robot' |
556 | + # run-after: "we shouldn't get anything unless uncommented" |
557 | + """ |
558 | + ) |
559 | + Path(".launchpad.yaml").write_text(config) |
560 | + |
561 | + expected_env = self._get_ros_expected_env("noetic", "3", "1") |
562 | + |
563 | + self.run_command("run") |
564 | + self.assertEqual( |
565 | + [ |
566 | + call( |
567 | + ["apt", "update"], |
568 | + cwd=PosixPath("/build/lpcraft/project"), |
569 | + env=expected_env, |
570 | + stdout=ANY, |
571 | + stderr=ANY, |
572 | + ), |
573 | + self._get_ros_packages_call( |
574 | + expected_env, ["python3-catkin-tools", "file", "git"] |
575 | + ), |
576 | + call( |
577 | + ["bash", "--noprofile", "--norc", "-ec", "echo 'hello'"], |
578 | + cwd=PosixPath("/build/lpcraft/project"), |
579 | + env=expected_env, |
580 | + stdout=ANY, |
581 | + stderr=ANY, |
582 | + ), |
583 | + call( |
584 | + ["bash", "--noprofile", "--norc", "-ec", "echo 'robot'"], |
585 | + cwd=PosixPath("/build/lpcraft/project"), |
586 | + env=expected_env, |
587 | + stdout=ANY, |
588 | + stderr=ANY, |
589 | + ), |
590 | + ], |
591 | + execute_run.call_args_list, |
592 | + ) |
593 | + |
594 | + @patch("lpcraft.commands.run.get_provider") |
595 | + @patch("lpcraft.commands.run.get_host_architecture", return_value="amd64") |
596 | + def test_colcon_plugin( |
597 | + self, mock_get_host_architecture, mock_get_provider |
598 | + ): |
599 | + launcher = Mock(spec=launch) |
600 | + provider = makeLXDProvider(lxd_launcher=launcher) |
601 | + mock_get_provider.return_value = provider |
602 | + execute_run = launcher.return_value.execute_run |
603 | + execute_run.return_value = subprocess.CompletedProcess([], 0) |
604 | + config = dedent( |
605 | + """ |
606 | + pipeline: |
607 | + - build |
608 | + |
609 | + jobs: |
610 | + build: |
611 | + plugin: colcon |
612 | + series: focal |
613 | + architectures: amd64 |
614 | + """ |
615 | + ) |
616 | + Path(".launchpad.yaml").write_text(config) |
617 | + |
618 | + expected_env = self._get_ros_expected_env("foxy", "3", "2") |
619 | + |
620 | + self.run_command("run") |
621 | + self.assertEqual( |
622 | + [ |
623 | + call( |
624 | + ["apt", "update"], |
625 | + cwd=PosixPath("/build/lpcraft/project"), |
626 | + env=expected_env, |
627 | + stdout=ANY, |
628 | + stderr=ANY, |
629 | + ), |
630 | + self._get_ros_packages_call( |
631 | + expected_env, ["python3-colcon-common-extensions"] |
632 | + ), |
633 | + self._get_ros_before_run_call(expected_env), |
634 | + call( |
635 | + [ |
636 | + "bash", |
637 | + "--noprofile", |
638 | + "--norc", |
639 | + "-ec", |
640 | + "if [ -f /opt/ros/*/setup.sh ]; then source /opt/ros/*/setup.sh; fi\n" # noqa:E501 |
641 | + "colcon build --base-paths ${ROS_PROJECT_SRC} --build-base ${ROS_PROJECT_BUILD} --install-base ${ROS_PROJECT_INSTALL} --event-handlers console_direct+", # noqa:E501 |
642 | + ], |
643 | + cwd=PosixPath("/build/lpcraft/project"), |
644 | + env=expected_env, |
645 | + stdout=ANY, |
646 | + stderr=ANY, |
647 | + ), |
648 | + ], |
649 | + execute_run.call_args_list, |
650 | + ) |
651 | + |
652 | + @patch("lpcraft.commands.run.get_provider") |
653 | + @patch("lpcraft.commands.run.get_host_architecture", return_value="amd64") |
654 | + def test_colcon_plugin_with_tests( |
655 | + self, mock_get_host_architecture, mock_get_provider |
656 | + ): |
657 | + launcher = Mock(spec=launch) |
658 | + provider = makeLXDProvider(lxd_launcher=launcher) |
659 | + mock_get_provider.return_value = provider |
660 | + execute_run = launcher.return_value.execute_run |
661 | + execute_run.return_value = subprocess.CompletedProcess([], 0) |
662 | + config = dedent( |
663 | + """ |
664 | + pipeline: |
665 | + - build |
666 | + |
667 | + jobs: |
668 | + build: |
669 | + plugin: colcon |
670 | + run-tests: True |
671 | + series: focal |
672 | + architectures: amd64 |
673 | + packages: [file, git] |
674 | + """ |
675 | + ) |
676 | + Path(".launchpad.yaml").write_text(config) |
677 | + |
678 | + expected_env = self._get_ros_expected_env("foxy", "3", "2") |
679 | + |
680 | + self.run_command("run") |
681 | + self.assertEqual( |
682 | + [ |
683 | + call( |
684 | + ["apt", "update"], |
685 | + cwd=PosixPath("/build/lpcraft/project"), |
686 | + env=expected_env, |
687 | + stdout=ANY, |
688 | + stderr=ANY, |
689 | + ), |
690 | + self._get_ros_packages_call( |
691 | + expected_env, |
692 | + ["python3-colcon-common-extensions", "file", "git"], |
693 | + ), |
694 | + self._get_ros_before_run_call(expected_env), |
695 | + call( |
696 | + [ |
697 | + "bash", |
698 | + "--noprofile", |
699 | + "--norc", |
700 | + "-ec", |
701 | + "if [ -f /opt/ros/*/setup.sh ]; then source /opt/ros/*/setup.sh; fi\n" # noqa:E501 |
702 | + "colcon build --base-paths ${ROS_PROJECT_SRC} --build-base ${ROS_PROJECT_BUILD} --install-base ${ROS_PROJECT_INSTALL} --event-handlers console_direct+", # noqa:E501 |
703 | + ], |
704 | + cwd=PosixPath("/build/lpcraft/project"), |
705 | + env=expected_env, |
706 | + stdout=ANY, |
707 | + stderr=ANY, |
708 | + ), |
709 | + call( |
710 | + [ |
711 | + "bash", |
712 | + "--noprofile", |
713 | + "--norc", |
714 | + "-ec", |
715 | + "\nsource ${ROS_PROJECT_INSTALL}/setup.sh\n" |
716 | + "colcon test --base-paths ${ROS_PROJECT_SRC} --build-base ${ROS_PROJECT_BUILD} --install-base ${ROS_PROJECT_INSTALL} --event-handlers console_direct+\n" # noqa:E501 |
717 | + "colcon test-result --all --verbose --test-result-base ${ROS_PROJECT_BUILD}", # noqa:E501 |
718 | + ], |
719 | + cwd=PosixPath("/build/lpcraft/project"), |
720 | + env=expected_env, |
721 | + stdout=ANY, |
722 | + stderr=ANY, |
723 | + ), |
724 | + ], |
725 | + execute_run.call_args_list, |
726 | + ) |
727 | + |
728 | + @patch("lpcraft.commands.run.get_provider") |
729 | + @patch("lpcraft.commands.run.get_host_architecture", return_value="amd64") |
730 | + def test_colcon_plugin_user_command( |
731 | + self, mock_get_host_architecture, mock_get_provider |
732 | + ): |
733 | + launcher = Mock(spec=launch) |
734 | + provider = makeLXDProvider(lxd_launcher=launcher) |
735 | + mock_get_provider.return_value = provider |
736 | + execute_run = launcher.return_value.execute_run |
737 | + execute_run.return_value = subprocess.CompletedProcess([], 0) |
738 | + config = dedent( |
739 | + """ |
740 | + pipeline: |
741 | + - build |
742 | + |
743 | + jobs: |
744 | + build: |
745 | + plugin: colcon |
746 | + run-tests: True |
747 | + series: focal |
748 | + architectures: amd64 |
749 | + packages: [file, git] |
750 | + run-before: echo 'hello' |
751 | + run: echo 'robot' |
752 | + # run-after: "we shouldn't get anything unless uncommented" |
753 | + """ |
754 | + ) |
755 | + Path(".launchpad.yaml").write_text(config) |
756 | + |
757 | + expected_env = self._get_ros_expected_env("foxy", "3", "2") |
758 | + |
759 | + self.run_command("run") |
760 | + self.assertEqual( |
761 | + [ |
762 | + call( |
763 | + ["apt", "update"], |
764 | + cwd=PosixPath("/build/lpcraft/project"), |
765 | + env=expected_env, |
766 | + stdout=ANY, |
767 | + stderr=ANY, |
768 | + ), |
769 | + self._get_ros_packages_call( |
770 | + expected_env, |
771 | + ["python3-colcon-common-extensions", "file", "git"], |
772 | + ), |
773 | + call( |
774 | + ["bash", "--noprofile", "--norc", "-ec", "echo 'hello'"], |
775 | + cwd=PosixPath("/build/lpcraft/project"), |
776 | + env=expected_env, |
777 | + stdout=ANY, |
778 | + stderr=ANY, |
779 | + ), |
780 | + call( |
781 | + ["bash", "--noprofile", "--norc", "-ec", "echo 'robot'"], |
782 | + cwd=PosixPath("/build/lpcraft/project"), |
783 | + env=expected_env, |
784 | + stdout=ANY, |
785 | + stderr=ANY, |
786 | + ), |
787 | + ], |
788 | + execute_run.call_args_list, |
789 | + ) |
790 | diff --git a/lpcraft/plugins/plugins.py b/lpcraft/plugins/plugins.py |
791 | index d4464ee..2a6e41e 100644 |
792 | --- a/lpcraft/plugins/plugins.py |
793 | +++ b/lpcraft/plugins/plugins.py |
794 | @@ -9,6 +9,9 @@ __all__ = [ |
795 | "MiniCondaPlugin", |
796 | "CondaBuildPlugin", |
797 | "GolangPlugin", |
798 | + "CatkinPlugin", |
799 | + "CatkinToolsPlugin", |
800 | + "ColconPlugin", |
801 | ] |
802 | |
803 | import textwrap |
804 | @@ -18,6 +21,7 @@ from typing import TYPE_CHECKING, ClassVar, Dict, List, Optional, cast |
805 | import pydantic |
806 | from pydantic import StrictStr |
807 | |
808 | +from lpcraft.env import get_managed_environment_project_path |
809 | from lpcraft.plugin import hookimpl |
810 | from lpcraft.plugins import register |
811 | |
812 | @@ -451,3 +455,275 @@ class GolangPlugin(BasePlugin): |
813 | export PATH=/usr/lib/go-{version}/bin/:$PATH |
814 | {run_command}""" |
815 | ) |
816 | + |
817 | + |
818 | +class RosBasePlugin(BasePlugin): |
819 | + """A base class for ROS-related plugins.""" |
820 | + |
821 | + _MAP_SERIES_TO_ROSDISTRO = { |
822 | + "xenial": "kinetic", |
823 | + "bionic": "melodic", |
824 | + "focal": "noetic", |
825 | + } |
826 | + |
827 | + _MAP_SERIES_TO_ROS2DISTRO = { |
828 | + "bionic": "dashing", |
829 | + "focal": "foxy", |
830 | + "jammy": "humble", |
831 | + } |
832 | + |
833 | + class Config(BaseConfig): |
834 | + run_tests: Optional[bool] |
835 | + |
836 | + def get_plugin_config(self) -> "RosBasePlugin.Config": |
837 | + return cast(RosBasePlugin.Config, self.config.plugin_config) |
838 | + |
839 | + def _get_ros_version(self) -> int: |
840 | + raise NotImplementedError |
841 | + |
842 | + def _get_ros_distro(self) -> str: |
843 | + """Get the ROS distro associated to the Ubuntu series in use.""" |
844 | + if self._get_ros_version() == 1: |
845 | + return self._MAP_SERIES_TO_ROSDISTRO[self.config.series] |
846 | + elif self._get_ros_version() == 2: |
847 | + return self._MAP_SERIES_TO_ROS2DISTRO[self.config.series] |
848 | + else: |
849 | + raise Exception("Unknown ROS version.") |
850 | + |
851 | + def _get_project_path(self) -> Path: |
852 | + """Get the project path.""" |
853 | + return get_managed_environment_project_path() |
854 | + |
855 | + def _get_build_path(self) -> Path: |
856 | + """Get the out-of-source build path.""" |
857 | + return get_managed_environment_project_path().parent / "build" |
858 | + |
859 | + def _get_install_path(self) -> Path: |
860 | + """Get the out-of-source install path.""" |
861 | + return get_managed_environment_project_path().parent / "install" |
862 | + |
863 | + def _get_ros_workspace_activation(self) -> str: |
864 | + """Get the ROS system workspace activation command.""" |
865 | + # There should be only one ROS distro installed at any time |
866 | + # therefore let's make use of the wildcard |
867 | + return "if [ -f /opt/ros/*/setup.sh ]; then source /opt/ros/*/setup.sh; fi" # noqa:E501 |
868 | + |
869 | + @hookimpl # type: ignore |
870 | + def lpcraft_set_environment(self) -> dict[str, str | None]: |
871 | + ros_distro = self._get_ros_distro() |
872 | + python_version = ( |
873 | + "3" if ros_distro not in ["kinetic", "melodic"] else "2" |
874 | + ) |
875 | + return { |
876 | + "ROS_DISTRO": ros_distro, |
877 | + "ROS_PROJECT_BASE": str(self._get_project_path().parent), |
878 | + "ROS_PROJECT_SRC": str(self._get_project_path()), |
879 | + "ROS_PROJECT_BUILD": str(self._get_build_path()), |
880 | + "ROS_PROJECT_INSTALL": str(self._get_install_path()), |
881 | + "ROS_PYTHON_VERSION": python_version, |
882 | + "ROS_VERSION": f"{self._get_ros_version()}", |
883 | + } |
884 | + |
885 | + @hookimpl # type: ignore |
886 | + def lpcraft_install_packages(self) -> list[str]: |
887 | + return ["build-essential", "cmake", "g++", "python3-rosdep"] |
888 | + |
889 | + @hookimpl # type: ignore |
890 | + def lpcraft_execute_before_run(self) -> str: |
891 | + return self._get_ros_workspace_activation() + textwrap.dedent( |
892 | + """ |
893 | + if [ ! -f /etc/ros/rosdep/sources.list.d/20-default.list ]; then |
894 | + rosdep init |
895 | + fi |
896 | + rosdep update --rosdistro ${ROS_DISTRO} |
897 | + rosdep install -i -y --rosdistro ${ROS_DISTRO} --from-paths .""" |
898 | + ) |
899 | + |
900 | + |
901 | +@register(name="catkin") |
902 | +class CatkinPlugin(RosBasePlugin): |
903 | + """Installs ROS dependencies, builds and (optionally) tests with catkin. |
904 | + |
905 | + Usage: |
906 | + In `.launchpad.yaml`, create the following structure. |
907 | + |
908 | + .. code-block:: yaml |
909 | + |
910 | + pipeline: |
911 | + - build |
912 | + |
913 | + jobs: |
914 | + build: |
915 | + plugin: catkin |
916 | + run-tests: True # optional |
917 | + series: focal |
918 | + architectures: amd64 |
919 | + packages: [file, git] |
920 | + package-repositories: |
921 | + - components: [main] |
922 | + formats: [deb] |
923 | + suites: [focal] |
924 | + type: apt |
925 | + url: http://packages.ros.org/ros/ubuntu |
926 | + trusted: True |
927 | + |
928 | + Please note that the ROS repository must be |
929 | + set up in `package-repositories`. |
930 | + """ |
931 | + |
932 | + def _get_ros_version(self) -> int: |
933 | + return 1 |
934 | + |
935 | + @hookimpl # type: ignore |
936 | + def lpcraft_install_packages(self) -> list[str]: |
937 | + # XXX artivis 2022-12-9: mypy is struggling with the super() call |
938 | + packages: list[str] = super().lpcraft_install_packages() + [ |
939 | + # "ros-${ROS_DISTRO}-catkin" |
940 | + f"ros-{self._get_ros_distro()}-catkin" |
941 | + ] |
942 | + return packages |
943 | + |
944 | + @hookimpl # type: ignore |
945 | + def lpcraft_execute_run(self) -> str: |
946 | + return self._get_ros_workspace_activation() + textwrap.dedent( |
947 | + """ |
948 | + catkin_make_isolated --source-space ${ROS_PROJECT_SRC} --directory ${ROS_PROJECT_BASE}""" # noqa:E501 |
949 | + ) |
950 | + |
951 | + @hookimpl # type: ignore |
952 | + def lpcraft_execute_after_run(self) -> str: |
953 | + if not self.get_plugin_config().run_tests or self.config.run: |
954 | + return "" |
955 | + |
956 | + return textwrap.dedent( |
957 | + """ |
958 | + source ${ROS_PROJECT_BASE}/devel_isolated/setup.sh |
959 | + catkin_make_isolated --source-space ${ROS_PROJECT_SRC} --directory ${ROS_PROJECT_BASE} --force-cmake --catkin-make-args run_tests |
960 | + catkin_test_results --verbose ${ROS_PROJECT_BASE}/build_isolated/""" # noqa:E501 |
961 | + ) |
962 | + |
963 | + |
964 | +@register(name="catkin-tools") |
965 | +class CatkinToolsPlugin(RosBasePlugin): |
966 | + """Installs ROS dependencies, builds and (optionally) tests with catkin-tools. |
967 | + |
968 | + Usage: |
969 | + In `.launchpad.yaml`, create the following structure. |
970 | + |
971 | + .. code-block:: yaml |
972 | + |
973 | + pipeline: |
974 | + - build |
975 | + |
976 | + jobs: |
977 | + build: |
978 | + plugin: catkin-tools |
979 | + run-tests: True # optional |
980 | + series: focal |
981 | + architectures: amd64 |
982 | + packages: [file, git] |
983 | + package-repositories: |
984 | + - components: [main] |
985 | + formats: [deb] |
986 | + suites: [focal] |
987 | + type: apt |
988 | + url: http://packages.ros.org/ros/ubuntu |
989 | + trusted: True |
990 | + |
991 | + Please note that the ROS repository must be |
992 | + set up in `package-repositories`. |
993 | + """ |
994 | + |
995 | + def _get_ros_version(self) -> int: |
996 | + return 1 |
997 | + |
998 | + @hookimpl # type: ignore |
999 | + def lpcraft_install_packages(self) -> list[str]: |
1000 | + # XXX artivis 2022-12-9: mypy is struggling with the super() call |
1001 | + packages: list[str] = super().lpcraft_install_packages() + [ |
1002 | + "python3-catkin-tools" |
1003 | + ] |
1004 | + return packages |
1005 | + |
1006 | + @hookimpl # type: ignore |
1007 | + def lpcraft_execute_run(self) -> str: |
1008 | + return self._get_ros_workspace_activation() + textwrap.dedent( |
1009 | + """ |
1010 | + catkin init --workspace ${ROS_PROJECT_BASE} |
1011 | + catkin profile add --force lpcraft |
1012 | + catkin config --profile lpcraft --install --source-space ${ROS_PROJECT_SRC} --build-space ${ROS_PROJECT_BUILD} --install-space ${ROS_PROJECT_INSTALL} |
1013 | + catkin build --no-notify --profile lpcraft""" # noqa:E501 |
1014 | + ) |
1015 | + |
1016 | + @hookimpl # type: ignore |
1017 | + def lpcraft_execute_after_run(self) -> str: |
1018 | + if not self.get_plugin_config().run_tests or self.config.run: |
1019 | + return "" |
1020 | + |
1021 | + return textwrap.dedent( |
1022 | + """ |
1023 | + source ${ROS_PROJECT_BASE}/devel/setup.sh |
1024 | + catkin test --profile lpcraft --summary""" |
1025 | + ) |
1026 | + |
1027 | + |
1028 | +@register(name="colcon") |
1029 | +class ColconPlugin(RosBasePlugin): |
1030 | + """Installs ROS dependencies, builds and (optionally) tests with colcon. |
1031 | + |
1032 | + Usage: |
1033 | + In `.launchpad.yaml`, create the following structure. |
1034 | + |
1035 | + .. code-block:: yaml |
1036 | + |
1037 | + pipeline: |
1038 | + - build |
1039 | + |
1040 | + jobs: |
1041 | + build: |
1042 | + plugin: colcon |
1043 | + run-tests: True # optional |
1044 | + series: focal |
1045 | + architectures: amd64 |
1046 | + packages: [file, git] |
1047 | + package-repositories: |
1048 | + - components: [main] |
1049 | + formats: [deb] |
1050 | + suites: [focal] |
1051 | + type: apt |
1052 | + url: http://repo.ros2.org/ubuntu/main/ |
1053 | + trusted: True |
1054 | + |
1055 | + Please note that the ROS 2 repository must be |
1056 | + set up in `package-repositories`. |
1057 | + """ |
1058 | + |
1059 | + def _get_ros_version(self) -> int: |
1060 | + return 2 |
1061 | + |
1062 | + @hookimpl # type: ignore |
1063 | + def lpcraft_install_packages(self) -> list[str]: |
1064 | + # XXX artivis 2022-12-9: mypy is struggling with the super() call |
1065 | + packages: list[str] = super().lpcraft_install_packages() + [ |
1066 | + "python3-colcon-common-extensions" |
1067 | + ] |
1068 | + return packages |
1069 | + |
1070 | + @hookimpl # type: ignore |
1071 | + def lpcraft_execute_run(self) -> str: |
1072 | + return self._get_ros_workspace_activation() + textwrap.dedent( |
1073 | + """ |
1074 | + colcon build --base-paths ${ROS_PROJECT_SRC} --build-base ${ROS_PROJECT_BUILD} --install-base ${ROS_PROJECT_INSTALL} --event-handlers console_direct+""" # noqa:E501 |
1075 | + ) |
1076 | + |
1077 | + @hookimpl # type: ignore |
1078 | + def lpcraft_execute_after_run(self) -> str: |
1079 | + if not self.get_plugin_config().run_tests or self.config.run: |
1080 | + return "" |
1081 | + |
1082 | + return textwrap.dedent( |
1083 | + """ |
1084 | + source ${ROS_PROJECT_INSTALL}/setup.sh |
1085 | + colcon test --base-paths ${ROS_PROJECT_SRC} --build-base ${ROS_PROJECT_BUILD} --install-base ${ROS_PROJECT_INSTALL} --event-handlers console_direct+ |
1086 | + colcon test-result --all --verbose --test-result-base ${ROS_PROJECT_BUILD}""" # noqa:E501 |
1087 | + ) |
Jeremie, thanks for the merge proposal.
CI currently fails as there are two uncovered lines, see buildlog below (below `unmerged commits`):
``` plugins/ plugins. py 284 2 64 1 99% 482, 491
:: lpcraft/
```
Could you please fix these?
As lpcraft is both the CI runner for Launchpad and a standalone tool, I'd like to know how you are planning to use it. Do you use or plan to use it in Launchpad CI?
I am asking as this would influence the plugin story. For a standalone lpcraft we could easily add setuptools based plugins ( https:/ /pluggy. readthedocs. io/en/stable/ #loading- setuptools- entry-points ), so we could have the plugin in a standalone plugin package. For Launchpad CI this is more elaborate, and currently out of scope.
I am discussing these options, as at least for me ROS is a business domain I have never touched before, and including these plugins means I (we) have to maintain them.
Fortunately, the code looks pretty straightforward. I will discuss this MP at today's standup and will get back to you.
Afterwards, I will also give you a detailed review. At very least you would need to add some descriptive docstrings which explain the used business terminology.
Also, does this MP depend on https:/ /code.launchpad .net/~artivis/ lpcraft/ +git/lpcraft/ +merge/ 435370 or would merging this MP on its own already be helpful?