MAAS

Merge ~cgrabowski/maas/+git/maas-performance:update_for_latest_maas into ~maas-committers/maas/+git/maas-performance:3.0-rework

  1. Git
  2. lp:~cgrabowski/maas/+git/maas-performance
  3. update_for_latest_maas
  4. Merge into 3.0-rework
Proposed by Christian Grabowski
Status: Merged
Approved by: Christian Grabowski
Approved revision: 41891463862d49c65f3bb40664dc99dcf4228223
Merge reported by: MAAS Lander
Merged at revision: not available
Proposed branch: ~cgrabowski/maas/+git/maas-performance:update_for_latest_maas
Merge into: ~maas-committers/maas/+git/maas-performance:3.0-rework
Diff against target: 3041 lines (+2084/-180)
25 files modified
Makefile (+3/-2)
maasperformance/bmc.py (+1/-2)
maasperformance/commissioning.py (+30/-38)
maasperformance/conftest.py (+2/-1)
maasperformance/data/20-maas-03-machine-resources (+321/-0)
maasperformance/data/20-maas-03-machine-resources.err (+0/-0)
maasperformance/data/20-maas-03-machine-resources.out (+321/-0)
maasperformance/data/40-maas-01-machine-resources (+321/-0)
maasperformance/data/40-maas-01-machine-resources.err (+0/-0)
maasperformance/data/40-maas-01-machine-resources.out (+321/-0)
maasperformance/machine.py (+13/-8)
maasperformance/network.py (+7/-2)
maasperformance/network_interface.py (+5/-3)
maasperformance/process.py (+7/-1)
maasperformance/testing/fixtures.py (+29/-1)
maasperformance/testing/subprocess.py (+14/-0)
maasperformance/tests/test_bmc.py (+50/-0)
maasperformance/tests/test_event.py (+34/-11)
maasperformance/tests/test_machine.py (+223/-33)
maasperformance/tests/test_network.py (+231/-1)
maasperformance/tests/test_network_interface.py (+3/-4)
maasperformance/tests/test_web.py (+118/-51)
maasperformance/web.py (+5/-1)
pyproject.toml (+3/-0)
requirements.txt (+22/-21)
Reviewer Review Type Date Requested Status
MAAS Lander Approve
Alberto Donato (community) Approve
Review via email: mp+429734@code.launchpad.net

Commit message

add update to machine-resources output

remove deprecated loop args

switch to non-deprecated eventloop fixture

patch check_output in tests

enable asyncio in tests

make event tests pass

update dependencies to resolve conflicting versions

To post a comment you must log in.
Revision history for this message
MAAS Lander (maas-lander) wrote :

UNIT TESTS
-b update_for_latest_maas lp:~cgrabowski/maas/+git/maas-performance into -b 3.0-rework lp:~maas-committers/maas/+git/maas-performance

STATUS: FAILED
LOG: http://maas-ci.internal:8080/job/maas-performance-tester/7/consoleText
COMMIT: 36d139f78c83b37b09b6525a531d1a2dce781c93

review: Needs Fixing
Revision history for this message
MAAS Lander (maas-lander) wrote :

UNIT TESTS
-b update_for_latest_maas lp:~cgrabowski/maas/+git/maas-performance into -b 3.0-rework lp:~maas-committers/maas/+git/maas-performance

STATUS: FAILED
LOG: http://maas-ci.internal:8080/job/maas-performance-tester/8/consoleText
COMMIT: 6235732caa93d5799c94b8dc1852943338eb8f7a

review: Needs Fixing
Revision history for this message
Alberto Donato (ack) wrote :

+1

minor nit inline

review: Approve
Revision history for this message
Christian Grabowski (cgrabowski) :
~cgrabowski/maas/+git/maas-performance:update_for_latest_maas updated
d84b31a... by Christian Grabowski

add pyproject.toml to enable asyncio

b7c2363... by Christian Grabowski

remove unused subprocess import in test_web

Revision history for this message
MAAS Lander (maas-lander) wrote :

UNIT TESTS
-b update_for_latest_maas lp:~cgrabowski/maas/+git/maas-performance into -b 3.0-rework lp:~maas-committers/maas/+git/maas-performance

STATUS: FAILED
LOG: http://maas-ci.internal:8080/job/maas-performance-tester/9/consoleText
COMMIT: b7c2363976163bba296e22edf122c5211b2f22e0

review: Needs Fixing
~cgrabowski/maas/+git/maas-performance:update_for_latest_maas updated
82eb637... by Christian Grabowski

correctly format libffi-dev dep

Revision history for this message
MAAS Lander (maas-lander) wrote :

UNIT TESTS
-b update_for_latest_maas lp:~cgrabowski/maas/+git/maas-performance into -b 3.0-rework lp:~maas-committers/maas/+git/maas-performance

STATUS: FAILED
LOG: http://maas-ci.internal:8080/job/maas-performance-tester/10/consoleText
COMMIT: 82eb637ed25576695e646b38410c935d5b27e413

review: Needs Fixing
~cgrabowski/maas/+git/maas-performance:update_for_latest_maas updated
c53b9f5... by Christian Grabowski

prevent MachineManager loop from hanging

205f545... by Christian Grabowski

update interface test for new exec__sync_process

d973466... by Christian Grabowski

ensure web tasks close cleanly

Revision history for this message
MAAS Lander (maas-lander) wrote :

UNIT TESTS
-b update_for_latest_maas lp:~cgrabowski/maas/+git/maas-performance into -b 3.0-rework lp:~maas-committers/maas/+git/maas-performance

STATUS: FAILED
LOG: http://maas-ci.internal:8080/job/maas-performance-tester/11/consoleText
COMMIT: 263a13b0557c754dfcd211897d276c438874ec60

review: Needs Fixing
Revision history for this message
MAAS Lander (maas-lander) wrote :

UNIT TESTS
-b update_for_latest_maas lp:~cgrabowski/maas/+git/maas-performance into -b 3.0-rework lp:~maas-committers/maas/+git/maas-performance

STATUS: FAILED
LOG: http://maas-ci.internal:8080/job/maas-performance-tester/12/consoleText
COMMIT: d97346635c87a6984a2936e783229a58243a0d0b

review: Needs Fixing
~cgrabowski/maas/+git/maas-performance:update_for_latest_maas updated
37c38b2... by Christian Grabowski

cover bmc class in tests

4189146... by Christian Grabowski

add test coverage for network request methods

Revision history for this message
MAAS Lander (maas-lander) wrote :

UNIT TESTS
-b update_for_latest_maas lp:~cgrabowski/maas/+git/maas-performance into -b 3.0-rework lp:~maas-committers/maas/+git/maas-performance

STATUS: SUCCESS
COMMIT: 41891463862d49c65f3bb40664dc99dcf4228223

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
diff --git a/Makefile b/Makefile
index 7de4112..00962b2 100644
--- a/Makefile
+++ b/Makefile
@@ -1,5 +1,5 @@
1define DEB_DEPENDENCIES1define DEB_DEPENDENCIES
2python3-venv2python3-venv libffi-dev
3endef3endef
44
5APT := DEBIAN_FRONTEND=noninteractive apt5APT := DEBIAN_FRONTEND=noninteractive apt
@@ -60,7 +60,8 @@ deb-dep:
60# Python targets60# Python targets
61py-dep: $(VIRTUALENV)61py-dep: $(VIRTUALENV)
62 $(VIRTUALENV)/bin/pip install -r requirements.txt -e .62 $(VIRTUALENV)/bin/pip install -r requirements.txt -e .
63 $(VIRTUALENV)/bin/pip install -r requirements.txt -e .[dev] .[test]63 $(VIRTUALENV)/bin/pip install -r requirements.txt -e .[dev]
64 $(VIRTUALENV)/bin/pip install -r requirements.txt -e .[test]
64 ln -sf $(VIRTUALENV)/bin/pytest65 ln -sf $(VIRTUALENV)/bin/pytest
65 ln -sf $(VIRTUALENV)/bin/maas-performanced66 ln -sf $(VIRTUALENV)/bin/maas-performanced
66.PHONY: py-dep67.PHONY: py-dep
diff --git a/maasperformance/bmc.py b/maasperformance/bmc.py
index f09b44a..946c75d 100644
--- a/maasperformance/bmc.py
+++ b/maasperformance/bmc.py
@@ -32,8 +32,7 @@ class BMC:
32 await asyncio.gather(32 await asyncio.gather(
33 *(33 *(
34 self.power_on(machine.uuid)34 self.power_on(machine.uuid)
35 for machine in self.machines_manager.machines.values()),35 for machine in self.machines_manager.machines.values()))
36 loop=self.loop)
3736
38 async def power_on(self, machine_uuid):37 async def power_on(self, machine_uuid):
39 print(f'Requesting power on: {machine_uuid}')38 print(f'Requesting power on: {machine_uuid}')
diff --git a/maasperformance/commissioning.py b/maasperformance/commissioning.py
index d567ca6..592cbb9 100644
--- a/maasperformance/commissioning.py
+++ b/maasperformance/commissioning.py
@@ -7,10 +7,16 @@ changes to remove functionality that wasn't needed.
77
8import dataclasses8import dataclasses
9from functools import partial9from functools import partial
10from itertools import islice, repeat10from itertools import (
11 islice,
12 repeat,
13)
11import random14import random
12import string15import string
13from typing import List, Optional16from typing import (
17 List,
18 Optional,
19)
1420
15GB = 1000 * 1000 * 100021GB = 1000 * 1000 * 1000
1622
@@ -18,12 +24,10 @@ GB = 1000 * 1000 * 1000
18class Factory:24class Factory:
1925
20 random_letters = map(26 random_letters = map(
21 random.choice, repeat(string.ascii_letters + string.digits)27 random.choice, repeat(string.ascii_letters + string.digits))
22 )
2328
24 random_letters_with_spaces = map(29 random_letters_with_spaces = map(
25 random.choice, repeat(string.ascii_letters + string.digits + " ")30 random.choice, repeat(string.ascii_letters + string.digits + " "))
26 )
2731
28 random_octet = partial(random.randint, 0, 255)32 random_octet = partial(random.randint, 0, 255)
2933
@@ -56,8 +60,7 @@ class Factory:
56 def make_string(self, size=10, spaces=False, prefix=""):60 def make_string(self, size=10, spaces=False, prefix=""):
57 """Return a `str` filled with random ASCII letters or digits."""61 """Return a `str` filled with random ASCII letters or digits."""
58 source = (62 source = (
59 self.random_letters_with_spaces if spaces else self.random_letters63 self.random_letters_with_spaces if spaces else self.random_letters)
60 )
61 return prefix + "".join(islice(source, size))64 return prefix + "".join(islice(source, size))
6265
6366
@@ -127,11 +130,9 @@ class LXDNetworkPort:
127 address: str = dataclasses.field(default_factory=factory.make_mac_address)130 address: str = dataclasses.field(default_factory=factory.make_mac_address)
128 protocol: str = "ethernet"131 protocol: str = "ethernet"
129 supported_modes: List[str] = dataclasses.field(132 supported_modes: List[str] = dataclasses.field(
130 default_factory=lambda: ["10000baseT/Full"]133 default_factory=lambda: ["10000baseT/Full"])
131 )
132 supported_ports: List[str] = dataclasses.field(134 supported_ports: List[str] = dataclasses.field(
133 default_factory=lambda: ["fibre"]135 default_factory=lambda: ["fibre"])
134 )
135 port_type: str = "fibre"136 port_type: str = "fibre"
136 transceiver_type: str = "internal"137 transceiver_type: str = "internal"
137 auto_negotiation: bool = True138 auto_negotiation: bool = True
@@ -208,25 +209,20 @@ class FakeCommissioningData:
208 def allocate_pci_address(self):209 def allocate_pci_address(self):
209 prev_address = (210 prev_address = (
210 self._allocated_pci_addresses[-1]211 self._allocated_pci_addresses[-1]
211 if self._allocated_pci_addresses212 if self._allocated_pci_addresses else "0000:00:00.0")
212 else "0000:00:00.0"
213 )
214 bus, device, func = prev_address.split(":")213 bus, device, func = prev_address.split(":")
215 next_device = int(device, 16) + 1214 next_device = int(device, 16) + 1
216 self._allocated_pci_addresses.append(215 self._allocated_pci_addresses.append(
217 f"{bus}:{next_device:0>4x}:{func}"216 f"{bus}:{next_device:0>4x}:{func}")
218 )
219 return self._allocated_pci_addresses[-1]217 return self._allocated_pci_addresses[-1]
220218
221 def get_available_vid(self):219 def get_available_vid(self):
222 available_vids = set(range(2, 4095))220 available_vids = set(range(2, 4095))
223 used_vids = set(221 used_vids = set(
224 [222 [
225 network.vlan.vid223 network.vlan.vid for network in self.networks.values()
226 for network in self.networks.values()
227 if network.vlan is not None224 if network.vlan is not None
228 ]225 ])
229 )
230 available_vids = list(available_vids.difference(used_vids))226 available_vids = list(available_vids.difference(used_vids))
231 return random.choice(available_vids)227 return random.choice(available_vids)
232228
@@ -249,8 +245,7 @@ class FakeCommissioningData:
249 network = self.create_physical_network_without_nic(name, mac_address)245 network = self.create_physical_network_without_nic(name, mac_address)
250 if port is None:246 if port is None:
251 port = LXDNetworkPort(247 port = LXDNetworkPort(
252 network.name, len(card.ports), address=network.hwaddr248 network.name, len(card.ports), address=network.hwaddr)
253 )
254 card.ports.append(port)249 card.ports.append(port)
255 return network250 return network
256251
@@ -283,8 +278,7 @@ class FakeCommissioningData:
283 if vid is None:278 if vid is None:
284 vid = self.get_available_vid()279 vid = self.get_available_vid()
285 network = LXDNetwork(280 network = LXDNetwork(
286 name, mac_address, vlan=LXDVlan(lower_device=parent.name, vid=vid)281 name, mac_address, vlan=LXDVlan(lower_device=parent.name, vid=vid))
287 )
288 self.networks[name] = network282 self.networks[name] = network
289 return network283 return network
290284
@@ -304,8 +298,7 @@ class FakeCommissioningData:
304 name,298 name,
305 mac_address,299 mac_address,
306 bridge=LXDBridge(300 bridge=LXDBridge(
307 upper_devices=[parent.name for parent in parents]301 upper_devices=[parent.name for parent in parents]),
308 ),
309 )302 )
310 self.networks[name] = network303 self.networks[name] = network
311 return network304 return network
@@ -346,8 +339,7 @@ class FakeCommissioningData:
346 del card["ports"]339 del card["ports"]
347 networks = dict(340 networks = dict(
348 (name, dataclasses.asdict(network))341 (name, dataclasses.asdict(network))
349 for name, network in self.networks.items()342 for name, network in self.networks.items())
350 )
351 data = {343 data = {
352 "api_extensions": self.api_extensions,344 "api_extensions": self.api_extensions,
353 "api_version": self.api_version,345 "api_version": self.api_version,
@@ -355,12 +347,10 @@ class FakeCommissioningData:
355 "resources": {347 "resources": {
356 "cpu": {348 "cpu": {
357 "architecture": self.environment["kernel_architecture"],349 "architecture": self.environment["kernel_architecture"],
358 "sockets": [350 "sockets": [{
359 {351 "socket": 0,
360 "socket": 0,352 "cores": [],
361 "cores": [],353 }],
362 }
363 ],
364 },354 },
365 "memory": {355 "memory": {
366 "hugepages_total": 0,356 "hugepages_total": 0,
@@ -369,7 +359,10 @@ class FakeCommissioningData:
369 "used": int(0.3 * self.memory * 1024 * 1024),359 "used": int(0.3 * self.memory * 1024 * 1024),
370 "total": int(self.memory * 1024 * 1024),360 "total": int(self.memory * 1024 * 1024),
371 },361 },
372 "gpu": {"cards": [], "total": 0},362 "gpu": {
363 "cards": [],
364 "total": 0
365 },
373 "network": network_resources,366 "network": network_resources,
374 "storage": storage_resources,367 "storage": storage_resources,
375 },368 },
@@ -388,6 +381,5 @@ class FakeCommissioningData:
388 },381 },
389 ],382 ],
390 "frequency": 1500,383 "frequency": 1500,
391 }384 })
392 )
393 return data385 return data
diff --git a/maasperformance/conftest.py b/maasperformance/conftest.py
index cf0da2e..99b7085 100644
--- a/maasperformance/conftest.py
+++ b/maasperformance/conftest.py
@@ -2,6 +2,7 @@ from .testing.fixtures import (
2 fs_root,2 fs_root,
3 machine,3 machine,
4 mock_event_time,4 mock_event_time,
5 tar_file_http_response,
5)6)
67
7__all__ = ['fs_root', 'machine', 'mock_event_time']8__all__ = ['fs_root', 'machine', 'mock_event_time', "tar_file_http_response"]
diff --git a/maasperformance/data/20-maas-03-machine-resources b/maasperformance/data/20-maas-03-machine-resources
8new file mode 1006449new file mode 100644
index 0000000..056bd08
--- /dev/null
+++ b/maasperformance/data/20-maas-03-machine-resources
@@ -0,0 +1,321 @@
1{
2 "api_extensions": [
3 "resources",
4 "resources_cpu_socket",
5 "resources_gpu",
6 "resources_numa",
7 "resources_v2",
8 "resources_disk_sata",
9 "resources_network_firmware",
10 "resources_disk_id",
11 "resources_usb_pci",
12 "resources_cpu_threads_numa",
13 "resources_cpu_core_die",
14 "api_os",
15 "resources_system",
16 "resources_pci_iommu",
17 "resources_network_usb",
18 "resources_disk_address"
19 ],
20 "api_version": "1.0",
21 "environment": {
22 "kernel": "Linux",
23 "kernel_architecture": "x86_64",
24 "kernel_version": "5.15.0-43-generic",
25 "os_name": "ubuntu",
26 "os_version": "20.04",
27 "server": "maas-machine-resources",
28 "server_name": "maas-performance",
29 "server_version": "5.4"
30 },
31 "resources": {
32 "cpu": {
33 "architecture": "x86_64",
34 "sockets": [
35 {
36 "name": "exampe-cpu",
37 "vendor": "test",
38 "socket": 0,
39 "cache": [
40 {
41 "level": 1,
42 "type": "Data",
43 "size": 32768
44 },
45 {
46 "level": 1,
47 "type": "Instruction",
48 "size": 32768
49 },
50 {
51 "level": 2,
52 "type": "Unified",
53 "size": 524288
54 },
55 {
56 "level": 3,
57 "type": "Unified",
58 "size": 16777216
59 }
60 ],
61 "cores": [
62 {
63 "core": 0,
64 "die": 0,
65 "threads": [
66 {
67 "id": 0,
68 "numa_node": 0,
69 "thread": 0,
70 "online": true,
71 "isolated": false
72 },
73 {
74 "id": 12,
75 "numa_node": 0,
76 "thread": 1,
77 "online": true,
78 "isolated": false
79 }
80 ],
81 "frequency": 3080
82 }
83 ],
84 "frequency": 2559,
85 "frequency_minimum": 2200,
86 "frequency_turbo": 4672
87 }
88 ],
89 "total": 24
90 },
91 "memory": {
92 "nodes": [
93 {
94 "numa_node": 0,
95 "hugepages_used": 0,
96 "hugepages_total": 0,
97 "used": 25046433792,
98 "total": 70866960384
99 }
100 ],
101 "hugepages_total": 0,
102 "hugepages_used": 0,
103 "hugepages_size": 2097152,
104 "used": 9893044224,
105 "total": 70866960384
106 },
107 "gpu": {
108 "cards": [
109 {
110 "driver": "ast",
111 "driver_version": "5.15.0-43-generic",
112 "drm": {
113 "id": 0,
114 "card_name": "card0",
115 "card_device": "226:0",
116 "control_name": "controlD64",
117 "control_device": "226:0"
118 },
119 "numa_node": 0,
120 "pci_address": "0000:22:00.0",
121 "vendor": "ASPEED Technology, Inc.",
122 "vendor_id": "1a03",
123 "product": "ASPEED Graphics Family",
124 "product_id": "2000"
125 }
126 ],
127 "total": 1
128 },
129 "network": {
130 "cards": [
131 {
132 "driver": "igb",
133 "driver_version": "5.15.0-43-generic",
134 "ports": [
135 {
136 "id": "enp35s0",
137 "address": "d0:50:99:dd:49:f1",
138 "port": 0,
139 "protocol": "ethernet",
140 "supported_modes": [
141 "10baseT/Half",
142 "10baseT/Full",
143 "100baseT/Half",
144 "100baseT/Full",
145 "1000baseT/Full"
146 ],
147 "supported_ports": [
148 "twisted pair"
149 ],
150 "port_type": "twisted pair",
151 "transceiver_type": "internal",
152 "auto_negotiation": true,
153 "link_detected": true,
154 "link_speed": 1000,
155 "link_duplex": "full"
156 }
157 ],
158 "numa_node": 0,
159 "pci_address": "0000:23:00.0",
160 "vendor": "Intel Corporation",
161 "vendor_id": "8086",
162 "product": "I210 Gigabit Network Connection",
163 "product_id": "1533",
164 "firmware_version": "3.16, 0x800004d6"
165 },
166 ],
167 "total": 1
168 },
169 "storage": {
170 "disks": [
171 {
172 "id": "nvme0n1",
173 "device": "259:0",
174 "model": "Samsung SSD 970 EVO 500GB",
175 "type": "nvme",
176 "read_only": false,
177 "size": 500107862016,
178 "removable": false,
179 "wwn": "eui.0025385b01440ea7",
180 "numa_node": 0,
181 "device_path": "pci-0000:2a:00.0-nvme-1",
182 "block_size": 512,
183 "firmware_version": "2B2QEXE7",
184 "rpm": 0,
185 "serial": "S5H7NS1NB23880D",
186 "device_id": "nvme-eui.0025385b01440ea7",
187 "partitions": [
188 {
189 "id": "nvme0n1p1",
190 "device": "259:1",
191 "read_only": false,
192 "size": 536870912,
193 "partition": 1
194 },
195 {
196 "id": "nvme0n1p2",
197 "device": "259:2",
198 "read_only": false,
199 "size": 1073741824,
200 "partition": 2
201 },
202 {
203 "id": "nvme0n1p3",
204 "device": "259:3",
205 "read_only": false,
206 "size": 498495127552,
207 "partition": 3
208 }
209 ]
210 }
211 ],
212 "total": 1
213 },
214 "usb": {
215 "devices": [
216 {
217 "bus_address": 1,
218 "device_address": 11,
219 "interfaces": [
220 {
221 "class": "Mass Storage",
222 "class_id": 8,
223 "driver": "usb-storage",
224 "driver_version": "5.15.0-43-generic",
225 "number": 0,
226 "subclass": "SCSI",
227 "subclass_id": 6
228 }
229 ],
230 "vendor": "American Megatrends, Inc.",
231 "vendor_id": "046b",
232 "product": "Virtual Cdrom Device",
233 "product_id": "ff20",
234 "speed": 480
235 }
236 ],
237 "total": 1
238 },
239 "pci": {
240 "devices": [
241 {
242 "driver": "igb",
243 "driver_version": "5.15.0-43-generic",
244 "numa_node": 0,
245 "pci_address": "0000:23:00.0",
246 "vendor": "Intel Corporation",
247 "vendor_id": "8086",
248 "product": "I210 Gigabit Network Connection",
249 "product_id": "1533",
250 "iommu_group": 15,
251 "vpd": {}
252 },
253 {
254 "driver": "nvme",
255 "driver_version": "1.0",
256 "numa_node": 0,
257 "pci_address": "0000:2a:00.0",
258 "vendor": "Samsung Electronics Co Ltd",
259 "vendor_id": "144d",
260 "product": "NVMe SSD Controller SM981/PM981/PM983",
261 "product_id": "a808",
262 "iommu_group": 15,
263 "vpd": {}
264 }
265 ],
266 "total": 2
267 },
268 "system": {
269 "uuid": "00000000-0000-0000-0000-d05099dd49f1",
270 "vendor": "To Be Filled By O.E.M.",
271 "product": "To Be Filled By O.E.M.",
272 "family": "To Be Filled By O.E.M.",
273 "version": "To Be Filled By O.E.M.",
274 "sku": "To Be Filled By O.E.M.",
275 "serial": "To Be Filled By O.E.M.",
276 "type": "physical",
277 "firmware": {
278 "vendor": "American Megatrends Inc.",
279 "date": "11/02/2020",
280 "version": "P3.50"
281 },
282 "chassis": {
283 "vendor": "To Be Filled By O.E.M.",
284 "type": "Unknown",
285 "serial": "To Be Filled By O.E.M.",
286 "version": "To Be Filled By O.E.M."
287 },
288 "motherboard": {
289 "vendor": "ASRockRack",
290 "product": "X470D4U",
291 "serial": "200101730000493",
292 "version": ""
293 }
294 }
295 },
296 "networks": {
297 "fake0": {
298 "addresses": [
299 {
300 "family": "inet",
301 "address": "192.168.1.21",
302 "netmask": "24",
303 "scope": "global"
304 }
305 ],
306 "counters": {
307 "bytes_received": 17341736517,
308 "bytes_sent": 42242825969,
309 "packets_received": 47014272,
310 "packets_sent": 64893025
311 },
312 "hwaddr": "8e:f4:15:bb:cd:fc",
313 "mtu": 1500,
314 "state": "up",
315 "type": "broadcast",
316 "bond": null,
317 "bridge": null,
318 "vlan": null
319 }
320 }
321}
diff --git a/maasperformance/data/20-maas-03-machine-resources.err b/maasperformance/data/20-maas-03-machine-resources.err
0new file mode 100644322new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/maasperformance/data/20-maas-03-machine-resources.err
diff --git a/maasperformance/data/20-maas-03-machine-resources.out b/maasperformance/data/20-maas-03-machine-resources.out
1new file mode 100644323new file mode 100644
index 0000000..056bd08
--- /dev/null
+++ b/maasperformance/data/20-maas-03-machine-resources.out
@@ -0,0 +1,321 @@
1{
2 "api_extensions": [
3 "resources",
4 "resources_cpu_socket",
5 "resources_gpu",
6 "resources_numa",
7 "resources_v2",
8 "resources_disk_sata",
9 "resources_network_firmware",
10 "resources_disk_id",
11 "resources_usb_pci",
12 "resources_cpu_threads_numa",
13 "resources_cpu_core_die",
14 "api_os",
15 "resources_system",
16 "resources_pci_iommu",
17 "resources_network_usb",
18 "resources_disk_address"
19 ],
20 "api_version": "1.0",
21 "environment": {
22 "kernel": "Linux",
23 "kernel_architecture": "x86_64",
24 "kernel_version": "5.15.0-43-generic",
25 "os_name": "ubuntu",
26 "os_version": "20.04",
27 "server": "maas-machine-resources",
28 "server_name": "maas-performance",
29 "server_version": "5.4"
30 },
31 "resources": {
32 "cpu": {
33 "architecture": "x86_64",
34 "sockets": [
35 {
36 "name": "exampe-cpu",
37 "vendor": "test",
38 "socket": 0,
39 "cache": [
40 {
41 "level": 1,
42 "type": "Data",
43 "size": 32768
44 },
45 {
46 "level": 1,
47 "type": "Instruction",
48 "size": 32768
49 },
50 {
51 "level": 2,
52 "type": "Unified",
53 "size": 524288
54 },
55 {
56 "level": 3,
57 "type": "Unified",
58 "size": 16777216
59 }
60 ],
61 "cores": [
62 {
63 "core": 0,
64 "die": 0,
65 "threads": [
66 {
67 "id": 0,
68 "numa_node": 0,
69 "thread": 0,
70 "online": true,
71 "isolated": false
72 },
73 {
74 "id": 12,
75 "numa_node": 0,
76 "thread": 1,
77 "online": true,
78 "isolated": false
79 }
80 ],
81 "frequency": 3080
82 }
83 ],
84 "frequency": 2559,
85 "frequency_minimum": 2200,
86 "frequency_turbo": 4672
87 }
88 ],
89 "total": 24
90 },
91 "memory": {
92 "nodes": [
93 {
94 "numa_node": 0,
95 "hugepages_used": 0,
96 "hugepages_total": 0,
97 "used": 25046433792,
98 "total": 70866960384
99 }
100 ],
101 "hugepages_total": 0,
102 "hugepages_used": 0,
103 "hugepages_size": 2097152,
104 "used": 9893044224,
105 "total": 70866960384
106 },
107 "gpu": {
108 "cards": [
109 {
110 "driver": "ast",
111 "driver_version": "5.15.0-43-generic",
112 "drm": {
113 "id": 0,
114 "card_name": "card0",
115 "card_device": "226:0",
116 "control_name": "controlD64",
117 "control_device": "226:0"
118 },
119 "numa_node": 0,
120 "pci_address": "0000:22:00.0",
121 "vendor": "ASPEED Technology, Inc.",
122 "vendor_id": "1a03",
123 "product": "ASPEED Graphics Family",
124 "product_id": "2000"
125 }
126 ],
127 "total": 1
128 },
129 "network": {
130 "cards": [
131 {
132 "driver": "igb",
133 "driver_version": "5.15.0-43-generic",
134 "ports": [
135 {
136 "id": "enp35s0",
137 "address": "d0:50:99:dd:49:f1",
138 "port": 0,
139 "protocol": "ethernet",
140 "supported_modes": [
141 "10baseT/Half",
142 "10baseT/Full",
143 "100baseT/Half",
144 "100baseT/Full",
145 "1000baseT/Full"
146 ],
147 "supported_ports": [
148 "twisted pair"
149 ],
150 "port_type": "twisted pair",
151 "transceiver_type": "internal",
152 "auto_negotiation": true,
153 "link_detected": true,
154 "link_speed": 1000,
155 "link_duplex": "full"
156 }
157 ],
158 "numa_node": 0,
159 "pci_address": "0000:23:00.0",
160 "vendor": "Intel Corporation",
161 "vendor_id": "8086",
162 "product": "I210 Gigabit Network Connection",
163 "product_id": "1533",
164 "firmware_version": "3.16, 0x800004d6"
165 },
166 ],
167 "total": 1
168 },
169 "storage": {
170 "disks": [
171 {
172 "id": "nvme0n1",
173 "device": "259:0",
174 "model": "Samsung SSD 970 EVO 500GB",
175 "type": "nvme",
176 "read_only": false,
177 "size": 500107862016,
178 "removable": false,
179 "wwn": "eui.0025385b01440ea7",
180 "numa_node": 0,
181 "device_path": "pci-0000:2a:00.0-nvme-1",
182 "block_size": 512,
183 "firmware_version": "2B2QEXE7",
184 "rpm": 0,
185 "serial": "S5H7NS1NB23880D",
186 "device_id": "nvme-eui.0025385b01440ea7",
187 "partitions": [
188 {
189 "id": "nvme0n1p1",
190 "device": "259:1",
191 "read_only": false,
192 "size": 536870912,
193 "partition": 1
194 },
195 {
196 "id": "nvme0n1p2",
197 "device": "259:2",
198 "read_only": false,
199 "size": 1073741824,
200 "partition": 2
201 },
202 {
203 "id": "nvme0n1p3",
204 "device": "259:3",
205 "read_only": false,
206 "size": 498495127552,
207 "partition": 3
208 }
209 ]
210 }
211 ],
212 "total": 1
213 },
214 "usb": {
215 "devices": [
216 {
217 "bus_address": 1,
218 "device_address": 11,
219 "interfaces": [
220 {
221 "class": "Mass Storage",
222 "class_id": 8,
223 "driver": "usb-storage",
224 "driver_version": "5.15.0-43-generic",
225 "number": 0,
226 "subclass": "SCSI",
227 "subclass_id": 6
228 }
229 ],
230 "vendor": "American Megatrends, Inc.",
231 "vendor_id": "046b",
232 "product": "Virtual Cdrom Device",
233 "product_id": "ff20",
234 "speed": 480
235 }
236 ],
237 "total": 1
238 },
239 "pci": {
240 "devices": [
241 {
242 "driver": "igb",
243 "driver_version": "5.15.0-43-generic",
244 "numa_node": 0,
245 "pci_address": "0000:23:00.0",
246 "vendor": "Intel Corporation",
247 "vendor_id": "8086",
248 "product": "I210 Gigabit Network Connection",
249 "product_id": "1533",
250 "iommu_group": 15,
251 "vpd": {}
252 },
253 {
254 "driver": "nvme",
255 "driver_version": "1.0",
256 "numa_node": 0,
257 "pci_address": "0000:2a:00.0",
258 "vendor": "Samsung Electronics Co Ltd",
259 "vendor_id": "144d",
260 "product": "NVMe SSD Controller SM981/PM981/PM983",
261 "product_id": "a808",
262 "iommu_group": 15,
263 "vpd": {}
264 }
265 ],
266 "total": 2
267 },
268 "system": {
269 "uuid": "00000000-0000-0000-0000-d05099dd49f1",
270 "vendor": "To Be Filled By O.E.M.",
271 "product": "To Be Filled By O.E.M.",
272 "family": "To Be Filled By O.E.M.",
273 "version": "To Be Filled By O.E.M.",
274 "sku": "To Be Filled By O.E.M.",
275 "serial": "To Be Filled By O.E.M.",
276 "type": "physical",
277 "firmware": {
278 "vendor": "American Megatrends Inc.",
279 "date": "11/02/2020",
280 "version": "P3.50"
281 },
282 "chassis": {
283 "vendor": "To Be Filled By O.E.M.",
284 "type": "Unknown",
285 "serial": "To Be Filled By O.E.M.",
286 "version": "To Be Filled By O.E.M."
287 },
288 "motherboard": {
289 "vendor": "ASRockRack",
290 "product": "X470D4U",
291 "serial": "200101730000493",
292 "version": ""
293 }
294 }
295 },
296 "networks": {
297 "fake0": {
298 "addresses": [
299 {
300 "family": "inet",
301 "address": "192.168.1.21",
302 "netmask": "24",
303 "scope": "global"
304 }
305 ],
306 "counters": {
307 "bytes_received": 17341736517,
308 "bytes_sent": 42242825969,
309 "packets_received": 47014272,
310 "packets_sent": 64893025
311 },
312 "hwaddr": "8e:f4:15:bb:cd:fc",
313 "mtu": 1500,
314 "state": "up",
315 "type": "broadcast",
316 "bond": null,
317 "bridge": null,
318 "vlan": null
319 }
320 }
321}
diff --git a/maasperformance/data/40-maas-01-machine-resources b/maasperformance/data/40-maas-01-machine-resources
0new file mode 100644322new file mode 100644
index 0000000..056bd08
--- /dev/null
+++ b/maasperformance/data/40-maas-01-machine-resources
@@ -0,0 +1,321 @@
1{
2 "api_extensions": [
3 "resources",
4 "resources_cpu_socket",
5 "resources_gpu",
6 "resources_numa",
7 "resources_v2",
8 "resources_disk_sata",
9 "resources_network_firmware",
10 "resources_disk_id",
11 "resources_usb_pci",
12 "resources_cpu_threads_numa",
13 "resources_cpu_core_die",
14 "api_os",
15 "resources_system",
16 "resources_pci_iommu",
17 "resources_network_usb",
18 "resources_disk_address"
19 ],
20 "api_version": "1.0",
21 "environment": {
22 "kernel": "Linux",
23 "kernel_architecture": "x86_64",
24 "kernel_version": "5.15.0-43-generic",
25 "os_name": "ubuntu",
26 "os_version": "20.04",
27 "server": "maas-machine-resources",
28 "server_name": "maas-performance",
29 "server_version": "5.4"
30 },
31 "resources": {
32 "cpu": {
33 "architecture": "x86_64",
34 "sockets": [
35 {
36 "name": "exampe-cpu",
37 "vendor": "test",
38 "socket": 0,
39 "cache": [
40 {
41 "level": 1,
42 "type": "Data",
43 "size": 32768
44 },
45 {
46 "level": 1,
47 "type": "Instruction",
48 "size": 32768
49 },
50 {
51 "level": 2,
52 "type": "Unified",
53 "size": 524288
54 },
55 {
56 "level": 3,
57 "type": "Unified",
58 "size": 16777216
59 }
60 ],
61 "cores": [
62 {
63 "core": 0,
64 "die": 0,
65 "threads": [
66 {
67 "id": 0,
68 "numa_node": 0,
69 "thread": 0,
70 "online": true,
71 "isolated": false
72 },
73 {
74 "id": 12,
75 "numa_node": 0,
76 "thread": 1,
77 "online": true,
78 "isolated": false
79 }
80 ],
81 "frequency": 3080
82 }
83 ],
84 "frequency": 2559,
85 "frequency_minimum": 2200,
86 "frequency_turbo": 4672
87 }
88 ],
89 "total": 24
90 },
91 "memory": {
92 "nodes": [
93 {
94 "numa_node": 0,
95 "hugepages_used": 0,
96 "hugepages_total": 0,
97 "used": 25046433792,
98 "total": 70866960384
99 }
100 ],
101 "hugepages_total": 0,
102 "hugepages_used": 0,
103 "hugepages_size": 2097152,
104 "used": 9893044224,
105 "total": 70866960384
106 },
107 "gpu": {
108 "cards": [
109 {
110 "driver": "ast",
111 "driver_version": "5.15.0-43-generic",
112 "drm": {
113 "id": 0,
114 "card_name": "card0",
115 "card_device": "226:0",
116 "control_name": "controlD64",
117 "control_device": "226:0"
118 },
119 "numa_node": 0,
120 "pci_address": "0000:22:00.0",
121 "vendor": "ASPEED Technology, Inc.",
122 "vendor_id": "1a03",
123 "product": "ASPEED Graphics Family",
124 "product_id": "2000"
125 }
126 ],
127 "total": 1
128 },
129 "network": {
130 "cards": [
131 {
132 "driver": "igb",
133 "driver_version": "5.15.0-43-generic",
134 "ports": [
135 {
136 "id": "enp35s0",
137 "address": "d0:50:99:dd:49:f1",
138 "port": 0,
139 "protocol": "ethernet",
140 "supported_modes": [
141 "10baseT/Half",
142 "10baseT/Full",
143 "100baseT/Half",
144 "100baseT/Full",
145 "1000baseT/Full"
146 ],
147 "supported_ports": [
148 "twisted pair"
149 ],
150 "port_type": "twisted pair",
151 "transceiver_type": "internal",
152 "auto_negotiation": true,
153 "link_detected": true,
154 "link_speed": 1000,
155 "link_duplex": "full"
156 }
157 ],
158 "numa_node": 0,
159 "pci_address": "0000:23:00.0",
160 "vendor": "Intel Corporation",
161 "vendor_id": "8086",
162 "product": "I210 Gigabit Network Connection",
163 "product_id": "1533",
164 "firmware_version": "3.16, 0x800004d6"
165 },
166 ],
167 "total": 1
168 },
169 "storage": {
170 "disks": [
171 {
172 "id": "nvme0n1",
173 "device": "259:0",
174 "model": "Samsung SSD 970 EVO 500GB",
175 "type": "nvme",
176 "read_only": false,
177 "size": 500107862016,
178 "removable": false,
179 "wwn": "eui.0025385b01440ea7",
180 "numa_node": 0,
181 "device_path": "pci-0000:2a:00.0-nvme-1",
182 "block_size": 512,
183 "firmware_version": "2B2QEXE7",
184 "rpm": 0,
185 "serial": "S5H7NS1NB23880D",
186 "device_id": "nvme-eui.0025385b01440ea7",
187 "partitions": [
188 {
189 "id": "nvme0n1p1",
190 "device": "259:1",
191 "read_only": false,
192 "size": 536870912,
193 "partition": 1
194 },
195 {
196 "id": "nvme0n1p2",
197 "device": "259:2",
198 "read_only": false,
199 "size": 1073741824,
200 "partition": 2
201 },
202 {
203 "id": "nvme0n1p3",
204 "device": "259:3",
205 "read_only": false,
206 "size": 498495127552,
207 "partition": 3
208 }
209 ]
210 }
211 ],
212 "total": 1
213 },
214 "usb": {
215 "devices": [
216 {
217 "bus_address": 1,
218 "device_address": 11,
219 "interfaces": [
220 {
221 "class": "Mass Storage",
222 "class_id": 8,
223 "driver": "usb-storage",
224 "driver_version": "5.15.0-43-generic",
225 "number": 0,
226 "subclass": "SCSI",
227 "subclass_id": 6
228 }
229 ],
230 "vendor": "American Megatrends, Inc.",
231 "vendor_id": "046b",
232 "product": "Virtual Cdrom Device",
233 "product_id": "ff20",
234 "speed": 480
235 }
236 ],
237 "total": 1
238 },
239 "pci": {
240 "devices": [
241 {
242 "driver": "igb",
243 "driver_version": "5.15.0-43-generic",
244 "numa_node": 0,
245 "pci_address": "0000:23:00.0",
246 "vendor": "Intel Corporation",
247 "vendor_id": "8086",
248 "product": "I210 Gigabit Network Connection",
249 "product_id": "1533",
250 "iommu_group": 15,
251 "vpd": {}
252 },
253 {
254 "driver": "nvme",
255 "driver_version": "1.0",
256 "numa_node": 0,
257 "pci_address": "0000:2a:00.0",
258 "vendor": "Samsung Electronics Co Ltd",
259 "vendor_id": "144d",
260 "product": "NVMe SSD Controller SM981/PM981/PM983",
261 "product_id": "a808",
262 "iommu_group": 15,
263 "vpd": {}
264 }
265 ],
266 "total": 2
267 },
268 "system": {
269 "uuid": "00000000-0000-0000-0000-d05099dd49f1",
270 "vendor": "To Be Filled By O.E.M.",
271 "product": "To Be Filled By O.E.M.",
272 "family": "To Be Filled By O.E.M.",
273 "version": "To Be Filled By O.E.M.",
274 "sku": "To Be Filled By O.E.M.",
275 "serial": "To Be Filled By O.E.M.",
276 "type": "physical",
277 "firmware": {
278 "vendor": "American Megatrends Inc.",
279 "date": "11/02/2020",
280 "version": "P3.50"
281 },
282 "chassis": {
283 "vendor": "To Be Filled By O.E.M.",
284 "type": "Unknown",
285 "serial": "To Be Filled By O.E.M.",
286 "version": "To Be Filled By O.E.M."
287 },
288 "motherboard": {
289 "vendor": "ASRockRack",
290 "product": "X470D4U",
291 "serial": "200101730000493",
292 "version": ""
293 }
294 }
295 },
296 "networks": {
297 "fake0": {
298 "addresses": [
299 {
300 "family": "inet",
301 "address": "192.168.1.21",
302 "netmask": "24",
303 "scope": "global"
304 }
305 ],
306 "counters": {
307 "bytes_received": 17341736517,
308 "bytes_sent": 42242825969,
309 "packets_received": 47014272,
310 "packets_sent": 64893025
311 },
312 "hwaddr": "8e:f4:15:bb:cd:fc",
313 "mtu": 1500,
314 "state": "up",
315 "type": "broadcast",
316 "bond": null,
317 "bridge": null,
318 "vlan": null
319 }
320 }
321}
diff --git a/maasperformance/data/40-maas-01-machine-resources.err b/maasperformance/data/40-maas-01-machine-resources.err
0new file mode 100644322new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/maasperformance/data/40-maas-01-machine-resources.err
diff --git a/maasperformance/data/40-maas-01-machine-resources.out b/maasperformance/data/40-maas-01-machine-resources.out
1new file mode 100644323new file mode 100644
index 0000000..056bd08
--- /dev/null
+++ b/maasperformance/data/40-maas-01-machine-resources.out
@@ -0,0 +1,321 @@
1{
2 "api_extensions": [
3 "resources",
4 "resources_cpu_socket",
5 "resources_gpu",
6 "resources_numa",
7 "resources_v2",
8 "resources_disk_sata",
9 "resources_network_firmware",
10 "resources_disk_id",
11 "resources_usb_pci",
12 "resources_cpu_threads_numa",
13 "resources_cpu_core_die",
14 "api_os",
15 "resources_system",
16 "resources_pci_iommu",
17 "resources_network_usb",
18 "resources_disk_address"
19 ],
20 "api_version": "1.0",
21 "environment": {
22 "kernel": "Linux",
23 "kernel_architecture": "x86_64",
24 "kernel_version": "5.15.0-43-generic",
25 "os_name": "ubuntu",
26 "os_version": "20.04",
27 "server": "maas-machine-resources",
28 "server_name": "maas-performance",
29 "server_version": "5.4"
30 },
31 "resources": {
32 "cpu": {
33 "architecture": "x86_64",
34 "sockets": [
35 {
36 "name": "exampe-cpu",
37 "vendor": "test",
38 "socket": 0,
39 "cache": [
40 {
41 "level": 1,
42 "type": "Data",
43 "size": 32768
44 },
45 {
46 "level": 1,
47 "type": "Instruction",
48 "size": 32768
49 },
50 {
51 "level": 2,
52 "type": "Unified",
53 "size": 524288
54 },
55 {
56 "level": 3,
57 "type": "Unified",
58 "size": 16777216
59 }
60 ],
61 "cores": [
62 {
63 "core": 0,
64 "die": 0,
65 "threads": [
66 {
67 "id": 0,
68 "numa_node": 0,
69 "thread": 0,
70 "online": true,
71 "isolated": false
72 },
73 {
74 "id": 12,
75 "numa_node": 0,
76 "thread": 1,
77 "online": true,
78 "isolated": false
79 }
80 ],
81 "frequency": 3080
82 }
83 ],
84 "frequency": 2559,
85 "frequency_minimum": 2200,
86 "frequency_turbo": 4672
87 }
88 ],
89 "total": 24
90 },
91 "memory": {
92 "nodes": [
93 {
94 "numa_node": 0,
95 "hugepages_used": 0,
96 "hugepages_total": 0,
97 "used": 25046433792,
98 "total": 70866960384
99 }
100 ],
101 "hugepages_total": 0,
102 "hugepages_used": 0,
103 "hugepages_size": 2097152,
104 "used": 9893044224,
105 "total": 70866960384
106 },
107 "gpu": {
108 "cards": [
109 {
110 "driver": "ast",
111 "driver_version": "5.15.0-43-generic",
112 "drm": {
113 "id": 0,
114 "card_name": "card0",
115 "card_device": "226:0",
116 "control_name": "controlD64",
117 "control_device": "226:0"
118 },
119 "numa_node": 0,
120 "pci_address": "0000:22:00.0",
121 "vendor": "ASPEED Technology, Inc.",
122 "vendor_id": "1a03",
123 "product": "ASPEED Graphics Family",
124 "product_id": "2000"
125 }
126 ],
127 "total": 1
128 },
129 "network": {
130 "cards": [
131 {
132 "driver": "igb",
133 "driver_version": "5.15.0-43-generic",
134 "ports": [
135 {
136 "id": "enp35s0",
137 "address": "d0:50:99:dd:49:f1",
138 "port": 0,
139 "protocol": "ethernet",
140 "supported_modes": [
141 "10baseT/Half",
142 "10baseT/Full",
143 "100baseT/Half",
144 "100baseT/Full",
145 "1000baseT/Full"
146 ],
147 "supported_ports": [
148 "twisted pair"
149 ],
150 "port_type": "twisted pair",
151 "transceiver_type": "internal",
152 "auto_negotiation": true,
153 "link_detected": true,
154 "link_speed": 1000,
155 "link_duplex": "full"
156 }
157 ],
158 "numa_node": 0,
159 "pci_address": "0000:23:00.0",
160 "vendor": "Intel Corporation",
161 "vendor_id": "8086",
162 "product": "I210 Gigabit Network Connection",
163 "product_id": "1533",
164 "firmware_version": "3.16, 0x800004d6"
165 },
166 ],
167 "total": 1
168 },
169 "storage": {
170 "disks": [
171 {
172 "id": "nvme0n1",
173 "device": "259:0",
174 "model": "Samsung SSD 970 EVO 500GB",
175 "type": "nvme",
176 "read_only": false,
177 "size": 500107862016,
178 "removable": false,
179 "wwn": "eui.0025385b01440ea7",
180 "numa_node": 0,
181 "device_path": "pci-0000:2a:00.0-nvme-1",
182 "block_size": 512,
183 "firmware_version": "2B2QEXE7",
184 "rpm": 0,
185 "serial": "S5H7NS1NB23880D",
186 "device_id": "nvme-eui.0025385b01440ea7",
187 "partitions": [
188 {
189 "id": "nvme0n1p1",
190 "device": "259:1",
191 "read_only": false,
192 "size": 536870912,
193 "partition": 1
194 },
195 {
196 "id": "nvme0n1p2",
197 "device": "259:2",
198 "read_only": false,
199 "size": 1073741824,
200 "partition": 2
201 },
202 {
203 "id": "nvme0n1p3",
204 "device": "259:3",
205 "read_only": false,
206 "size": 498495127552,
207 "partition": 3
208 }
209 ]
210 }
211 ],
212 "total": 1
213 },
214 "usb": {
215 "devices": [
216 {
217 "bus_address": 1,
218 "device_address": 11,
219 "interfaces": [
220 {
221 "class": "Mass Storage",
222 "class_id": 8,
223 "driver": "usb-storage",
224 "driver_version": "5.15.0-43-generic",
225 "number": 0,
226 "subclass": "SCSI",
227 "subclass_id": 6
228 }
229 ],
230 "vendor": "American Megatrends, Inc.",
231 "vendor_id": "046b",
232 "product": "Virtual Cdrom Device",
233 "product_id": "ff20",
234 "speed": 480
235 }
236 ],
237 "total": 1
238 },
239 "pci": {
240 "devices": [
241 {
242 "driver": "igb",
243 "driver_version": "5.15.0-43-generic",
244 "numa_node": 0,
245 "pci_address": "0000:23:00.0",
246 "vendor": "Intel Corporation",
247 "vendor_id": "8086",
248 "product": "I210 Gigabit Network Connection",
249 "product_id": "1533",
250 "iommu_group": 15,
251 "vpd": {}
252 },
253 {
254 "driver": "nvme",
255 "driver_version": "1.0",
256 "numa_node": 0,
257 "pci_address": "0000:2a:00.0",
258 "vendor": "Samsung Electronics Co Ltd",
259 "vendor_id": "144d",
260 "product": "NVMe SSD Controller SM981/PM981/PM983",
261 "product_id": "a808",
262 "iommu_group": 15,
263 "vpd": {}
264 }
265 ],
266 "total": 2
267 },
268 "system": {
269 "uuid": "00000000-0000-0000-0000-d05099dd49f1",
270 "vendor": "To Be Filled By O.E.M.",
271 "product": "To Be Filled By O.E.M.",
272 "family": "To Be Filled By O.E.M.",
273 "version": "To Be Filled By O.E.M.",
274 "sku": "To Be Filled By O.E.M.",
275 "serial": "To Be Filled By O.E.M.",
276 "type": "physical",
277 "firmware": {
278 "vendor": "American Megatrends Inc.",
279 "date": "11/02/2020",
280 "version": "P3.50"
281 },
282 "chassis": {
283 "vendor": "To Be Filled By O.E.M.",
284 "type": "Unknown",
285 "serial": "To Be Filled By O.E.M.",
286 "version": "To Be Filled By O.E.M."
287 },
288 "motherboard": {
289 "vendor": "ASRockRack",
290 "product": "X470D4U",
291 "serial": "200101730000493",
292 "version": ""
293 }
294 }
295 },
296 "networks": {
297 "fake0": {
298 "addresses": [
299 {
300 "family": "inet",
301 "address": "192.168.1.21",
302 "netmask": "24",
303 "scope": "global"
304 }
305 ],
306 "counters": {
307 "bytes_received": 17341736517,
308 "bytes_sent": 42242825969,
309 "packets_received": 47014272,
310 "packets_sent": 64893025
311 },
312 "hwaddr": "8e:f4:15:bb:cd:fc",
313 "mtu": 1500,
314 "state": "up",
315 "type": "broadcast",
316 "bond": null,
317 "bridge": null,
318 "vlan": null
319 }
320 }
321}
diff --git a/maasperformance/machine.py b/maasperformance/machine.py
index e5725b0..ed55471 100644
--- a/maasperformance/machine.py
+++ b/maasperformance/machine.py
@@ -93,6 +93,7 @@ class MachineManager:
93 _run() method of this class.93 _run() method of this class.
94 """94 """
95 self.machines = {}95 self.machines = {}
96 self.running = True
96 for index in range(self.number_of_machines):97 for index in range(self.number_of_machines):
97 machine = Machine(loop, self.parent_iface, f'fake{index}')98 machine = Machine(loop, self.parent_iface, f'fake{index}')
98 self.machines[machine.uuid] = machine99 self.machines[machine.uuid] = machine
@@ -130,7 +131,7 @@ class MachineManager:
130 logging.info(f'Created PXE interfaces in {duration} seconds')131 logging.info(f'Created PXE interfaces in {duration} seconds')
131 start = now()132 start = now()
132133
133 while True:134 while self.running and self.loop.is_running():
134 power_command = await self.power_commands.get()135 power_command = await self.power_commands.get()
135 print(136 print(
136 f'Got power command {power_command.new_power_state} '137 f'Got power command {power_command.new_power_state} '
@@ -233,9 +234,12 @@ class Machine:
233 That is, the clean up method that was last added is the first234 That is, the clean up method that was last added is the first
234 one to be executed.235 one to be executed.
235 """236 """
236 while len(self.cleanups) > 0:237 try:
237 func = self.cleanups.pop()238 while len(self.cleanups) > 0:
238 await func()239 func = self.cleanups.pop()
240 await func()
241 finally:
242 self.running = False
239243
240 async def create_pxe_interface(self):244 async def create_pxe_interface(self):
241 """Create the machine's network interface on the host.245 """Create the machine's network interface on the host.
@@ -264,7 +268,7 @@ class Machine:
264 print(f'Power on: {self.uuid}')268 print(f'Power on: {self.uuid}')
265 # Simulate how it takes a while before the power command is269 # Simulate how it takes a while before the power command is
266 # issued, and when the machine is actually considered on.270 # issued, and when the machine is actually considered on.
267 await asyncio.sleep(1, loop=self.loop)271 await asyncio.sleep(1)
268 self.power_state = PowerState.ON272 self.power_state = PowerState.ON
269 await self.pxe_boot()273 await self.pxe_boot()
270274
@@ -274,7 +278,7 @@ class Machine:
274 # Simulate how it takes a while before the power command is278 # Simulate how it takes a while before the power command is
275 # issued, and when the machine is actually considered off.279 # issued, and when the machine is actually considered off.
276 print(f'Power off: {self.uuid}')280 print(f'Power off: {self.uuid}')
277 await asyncio.sleep(1, loop=self.loop)281 await asyncio.sleep(1)
278 await self.release_ip()282 await self.release_ip()
279 self.reset()283 self.reset()
280 self.power_state = PowerState.OFF284 self.power_state = PowerState.OFF
@@ -358,7 +362,7 @@ class Machine:
358 else:362 else:
359 raise RuntimeError("Couldn't retrieve pxelinux.cfg")363 raise RuntimeError("Couldn't retrieve pxelinux.cfg")
360364
361 kernel_url, initrd_url = None, None365 kernel_url, initrd_url, image_url = None, None, None
362 for line in pxe_config.splitlines():366 for line in pxe_config.splitlines():
363 line = line.strip()367 line = line.strip()
364 try:368 try:
@@ -371,7 +375,8 @@ class Machine:
371 initrd_url = value375 initrd_url = value
372 if key == 'APPEND':376 if key == 'APPEND':
373 image_url, cloud_config_url = self._extract_append_urls(value)377 image_url, cloud_config_url = self._extract_append_urls(value)
374 if kernel_url is not None and initrd_url is not None:378 if (kernel_url is not None and initrd_url is not None
379 and image_url is not None):
375 await self.network.get_curl_file(kernel_url)380 await self.network.get_curl_file(kernel_url)
376 await self.network.get_curl_file(initrd_url)381 await self.network.get_curl_file(initrd_url)
377 await self.network.get_curl_file(image_url)382 await self.network.get_curl_file(image_url)
diff --git a/maasperformance/network.py b/maasperformance/network.py
index abaeba7..c8fef7e 100644
--- a/maasperformance/network.py
+++ b/maasperformance/network.py
@@ -102,8 +102,13 @@ class PlaintextSignature(Signature):
102102
103 name = 'PLAINTEXT'103 name = 'PLAINTEXT'
104104
105 def sign(self, consumer_secret, method, url, oauth_token_secret=None,105 def sign(
106 **params):106 self,
107 consumer_secret,
108 method,
109 url,
110 oauth_token_secret=None,
111 **params):
107 """Create a signature using PLAINTEXT."""112 """Create a signature using PLAINTEXT."""
108 key = self._escape(consumer_secret) + '&'113 key = self._escape(consumer_secret) + '&'
109 if oauth_token_secret:114 if oauth_token_secret:
diff --git a/maasperformance/network_interface.py b/maasperformance/network_interface.py
index 7f13822..836342c 100644
--- a/maasperformance/network_interface.py
+++ b/maasperformance/network_interface.py
@@ -1,14 +1,16 @@
1import json1import json
2import random2import random
3from subprocess import check_output
4from typing import NamedTuple3from typing import NamedTuple
54
6from .process import exec_process5from .process import (
6 exec_process,
7 exec_sync_process,
8)
79
810
9def interface_is_bridge(iface: str):11def interface_is_bridge(iface: str):
10 """Return whether an interface is a bridge."""12 """Return whether an interface is a bridge."""
11 output = check_output(['ip', '--json', '-d', 'link', 'show', iface])13 output = exec_sync_process('ip', '--json', '-d', 'link', 'show', iface)
12 data = json.loads(output)14 data = json.loads(output)
13 return data[0].get('linkinfo', {}).get('info_kind') == 'bridge'15 return data[0].get('linkinfo', {}).get('info_kind') == 'bridge'
1416
diff --git a/maasperformance/process.py b/maasperformance/process.py
index ae7e4ca..febc663 100644
--- a/maasperformance/process.py
+++ b/maasperformance/process.py
@@ -1,4 +1,5 @@
1import asyncio1import asyncio
2from subprocess import check_output
23
34
4class ProcessError(Exception):5class ProcessError(Exception):
@@ -13,10 +14,15 @@ class ProcessError(Exception):
1314
1415
15async def exec_process(*args: str) -> str:16async def exec_process(*args: str) -> str:
16 """Execute a process a process, returning its stdout."""17 """Execute a process asynchronously, returning its stdout."""
17 process = await asyncio.create_subprocess_exec(18 process = await asyncio.create_subprocess_exec(
18 *args, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE)19 *args, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE)
19 stdout, stderr = await process.communicate()20 stdout, stderr = await process.communicate()
20 if process.returncode != 0:21 if process.returncode != 0:
21 raise ProcessError(args, process.returncode, stderr.decode())22 raise ProcessError(args, process.returncode, stderr.decode())
22 return stdout.decode()23 return stdout.decode()
24
25
26def exec_sync_process(*args: str) -> str:
27 """Execute a process synchronously, returning its stdout."""
28 return check_output(args)
diff --git a/maasperformance/testing/fixtures.py b/maasperformance/testing/fixtures.py
index f85f311..9b52067 100644
--- a/maasperformance/testing/fixtures.py
+++ b/maasperformance/testing/fixtures.py
@@ -1,12 +1,18 @@
1import asyncio1import asyncio
2from io import BytesIO
2from pathlib import Path3from pathlib import Path
4import subprocess
5import tarfile
36
4import pytest7import pytest
58
6from .. import event9from .. import event
7from ..machine import Machine10from ..machine import Machine
8from ..network_interface import ParentNetworkInterface11from ..network_interface import ParentNetworkInterface
9from .subprocess import FakeCreateSubProcess12from .subprocess import (
13 FakeCheckOutput,
14 FakeCreateSubProcess,
15)
1016
1117
12@pytest.fixture18@pytest.fixture
@@ -38,3 +44,25 @@ def create_subprocess_mock(module=asyncio):
38 module, 'create_subprocess_exec', new=FakeCreateSubProcess())44 module, 'create_subprocess_exec', new=FakeCreateSubProcess())
3945
40 return subprocess_mock46 return subprocess_mock
47
48
49def create_check_output_mock(module=subprocess, output=""):
50 """Return a fixture for mocking check_output."""
51
52 @pytest.fixture
53 def check_output_mock(mocker, tmpdir):
54 yield mocker.patch.object(
55 module, 'check_output', new=FakeCheckOutput(output))
56
57 return check_output_mock
58
59
60@pytest.fixture
61def tar_file_http_response():
62 buf = BytesIO()
63 with tarfile.open(fileobj=buf, mode="w") as tar:
64 data = "{\"foo\": \"bar\"}"
65 info = tarfile.TarInfo("index.json")
66 info.size = len(data)
67 tar.addfile(info, BytesIO(initial_bytes=data.encode("ascii")))
68 return buf.getvalue().decode("ascii")
diff --git a/maasperformance/testing/subprocess.py b/maasperformance/testing/subprocess.py
index 557b048..d2634c4 100644
--- a/maasperformance/testing/subprocess.py
+++ b/maasperformance/testing/subprocess.py
@@ -191,6 +191,10 @@ class FakeCurlSubprocess(FakeAsyncSubprocess):
191 def root_url(self):191 def root_url(self):
192 return f'{self.root_protocol}://{self.ip_address}{self.root_path}'192 return f'{self.root_protocol}://{self.ip_address}{self.root_path}'
193193
194 @property
195 def image_url(self):
196 return self.root_url
197
194 def get_pxelinux_cfg(self):198 def get_pxelinux_cfg(self):
195 """Return the pxelinux.cfg contents.199 """Return the pxelinux.cfg contents.
196200
@@ -248,4 +252,14 @@ class FakeCurlSubprocess(FakeAsyncSubprocess):
248 }252 }
249 files[self.kernel_url] = ''253 files[self.kernel_url] = ''
250 files[self.initrd_url] = ''254 files[self.initrd_url] = ''
255 files[self.image_url] = ''
251 return files256 return files
257
258
259class FakeCheckOutput:
260
261 def __init__(self, output):
262 self._output = output
263
264 def __call__(self, *args, **kwargs):
265 return self._output
diff --git a/maasperformance/tests/test_bmc.py b/maasperformance/tests/test_bmc.py
252new file mode 100644266new file mode 100644
index 0000000..97be801
--- /dev/null
+++ b/maasperformance/tests/test_bmc.py
@@ -0,0 +1,50 @@
1import asyncio
2import random
3from unittest.mock import Mock
4
5import pytest
6
7from ..bmc import (
8 BMC,
9 PowerCommand,
10 PowerState,
11)
12
13
14class TestBMC:
15
16 @pytest.mark.asyncio
17 async def test_bmc_power_on(self):
18 manager = Mock()
19 manager.power_commands = asyncio.Queue()
20 bmc = BMC(manager)
21 uuid = random.randint(1, 512)
22 await bmc.power_on(uuid)
23 result = await manager.power_commands.get()
24 assert isinstance(result, PowerCommand)
25 assert result.machine_uuid == uuid
26 assert result.new_power_state == PowerState.ON
27
28 @pytest.mark.asyncio
29 async def test_bmc_power_off(self):
30 manager = Mock()
31 manager.power_commands = asyncio.Queue()
32 bmc = BMC(manager)
33 uuid = random.randint(1, 512)
34 await bmc.power_off(uuid)
35 result = await manager.power_commands.get()
36 assert isinstance(result, PowerCommand)
37 assert result.machine_uuid == uuid
38 assert result.new_power_state == PowerState.OFF
39
40 @pytest.mark.asyncio
41 async def test_bmc_power_cycle(self):
42 manager = Mock()
43 manager.power_commands = asyncio.Queue()
44 bmc = BMC(manager)
45 uuid = random.randint(1, 512)
46 await bmc.power_cycle(uuid)
47 result = await manager.power_commands.get()
48 assert isinstance(result, PowerCommand)
49 assert result.machine_uuid == uuid
50 assert result.new_power_state == PowerState.CYCLE
diff --git a/maasperformance/tests/test_event.py b/maasperformance/tests/test_event.py
index 3e826f3..1d800f5 100644
--- a/maasperformance/tests/test_event.py
+++ b/maasperformance/tests/test_event.py
@@ -1,4 +1,5 @@
1import logging1import logging
2from unittest.mock import Mock
2from urllib.parse import urlparse3from urllib.parse import urlparse
34
4import pytest5import pytest
@@ -8,11 +9,25 @@ from ..machine import Machine
8from ..testing.prometheus import track_metric9from ..testing.prometheus import track_metric
910
1011
12def create_mock_parent_interface():
13 parent_iface = Mock()
14
15 def _get_client_interface(x):
16 iface = Mock()
17 iface.name = x
18 iface.__str__ = lambda _: iface.name
19 return iface
20
21 parent_iface.get_client_interface = _get_client_interface
22 return parent_iface
23
24
11class TestDHCPEvents:25class TestDHCPEvents:
1226
13 @pytest.mark.parametrize('event_type', ['request', 'release'])27 @pytest.mark.parametrize('event_type', ['request', 'release'])
14 def test_dhcp_request_started(self, event_type, mock_event_time, caplog):28 def test_dhcp_request_started(self, event_type, mock_event_time, caplog):
15 machine = Machine(None, None, 'eth0')29 parent_iface = create_mock_parent_interface()
30 machine = Machine(None, parent_iface, 'eth0')
16 mock_event_time.return_value = 1234531 mock_event_time.return_value = 12345
17 event_func = getattr(event, f'dhcp_{event_type}_started')32 event_func = getattr(event, f'dhcp_{event_type}_started')
18 with caplog.at_level(logging.INFO):33 with caplog.at_level(logging.INFO):
@@ -24,7 +39,8 @@ class TestDHCPEvents:
2439
25 @pytest.mark.parametrize('event_type', ['request', 'release'])40 @pytest.mark.parametrize('event_type', ['request', 'release'])
26 def test_dhcp_request_ended(self, event_type, mock_event_time, caplog):41 def test_dhcp_request_ended(self, event_type, mock_event_time, caplog):
27 machine = Machine(None, None, 'eth0')42 parent_iface = create_mock_parent_interface()
43 machine = Machine(None, parent_iface, 'eth0')
28 machine.last_event_time[f'dhcp_{event_type}_start'] = 1234544 machine.last_event_time[f'dhcp_{event_type}_start'] = 12345
29 mock_event_time.return_value = 12347.545 mock_event_time.return_value = 12347.5
30 event_func = getattr(event, f'dhcp_{event_type}_ended')46 event_func = getattr(event, f'dhcp_{event_type}_ended')
@@ -38,7 +54,8 @@ class TestDHCPEvents:
38 @pytest.mark.parametrize('event_type', ['request', 'release'])54 @pytest.mark.parametrize('event_type', ['request', 'release'])
39 def test_dhcp_request_ended_prometheus_count(55 def test_dhcp_request_ended_prometheus_count(
40 self, event_type, mock_event_time, caplog):56 self, event_type, mock_event_time, caplog):
41 machine = Machine(None, None, 'eth0')57 parent_iface = create_mock_parent_interface()
58 machine = Machine(None, parent_iface, 'eth0')
42 machine.last_event_time[f'dhcp_{event_type}_start'] = 1234559 machine.last_event_time[f'dhcp_{event_type}_start'] = 12345
43 mock_event_time.return_value = 12347.560 mock_event_time.return_value = 12347.5
44 event_func = getattr(event, f'dhcp_{event_type}_ended')61 event_func = getattr(event, f'dhcp_{event_type}_ended')
@@ -50,7 +67,8 @@ class TestDHCPEvents:
50 @pytest.mark.parametrize('event_type', ['request', 'release'])67 @pytest.mark.parametrize('event_type', ['request', 'release'])
51 def test_dhcp_request_ended_prometheus_bucket_low(68 def test_dhcp_request_ended_prometheus_bucket_low(
52 self, event_type, mock_event_time, caplog):69 self, event_type, mock_event_time, caplog):
53 machine = Machine(None, None, 'eth0')70 parent_iface = create_mock_parent_interface()
71 machine = Machine(None, parent_iface, 'eth0')
54 machine.last_event_time[f'dhcp_{event_type}_start'] = 1234572 machine.last_event_time[f'dhcp_{event_type}_start'] = 12345
55 mock_event_time.return_value = 12347.573 mock_event_time.return_value = 12347.5
56 event_func = getattr(event, f'dhcp_{event_type}_ended')74 event_func = getattr(event, f'dhcp_{event_type}_ended')
@@ -62,7 +80,8 @@ class TestDHCPEvents:
62 @pytest.mark.parametrize('event_type', ['request', 'release'])80 @pytest.mark.parametrize('event_type', ['request', 'release'])
63 def test_dhcp_request_ended_prometheus_bucket_high(81 def test_dhcp_request_ended_prometheus_bucket_high(
64 self, event_type, mock_event_time, caplog):82 self, event_type, mock_event_time, caplog):
65 machine = Machine(None, None, 'eth0')83 parent_iface = create_mock_parent_interface()
84 machine = Machine(None, parent_iface, 'eth0')
66 machine.last_event_time[f'dhcp_{event_type}_start'] = 1234585 machine.last_event_time[f'dhcp_{event_type}_start'] = 12345
67 mock_event_time.return_value = 12347.586 mock_event_time.return_value = 12347.5
68 event_func = getattr(event, f'dhcp_{event_type}_ended')87 event_func = getattr(event, f'dhcp_{event_type}_ended')
@@ -75,7 +94,8 @@ class TestDHCPEvents:
75class TestFileTransferEvents:94class TestFileTransferEvents:
7695
77 def test_file_transfer_started(self, mock_event_time, caplog):96 def test_file_transfer_started(self, mock_event_time, caplog):
78 machine = Machine(None, None, 'eth0')97 parent_iface = create_mock_parent_interface()
98 machine = Machine(None, parent_iface, 'eth0')
79 mock_event_time.return_value = 1234599 mock_event_time.return_value = 12345
80 url = urlparse('tftp://some-host/my-file')100 url = urlparse('tftp://some-host/my-file')
81 with caplog.at_level(logging.INFO):101 with caplog.at_level(logging.INFO):
@@ -89,7 +109,8 @@ class TestFileTransferEvents:
89109
90 def test_file_transfer_ended(self, mock_event_time, caplog):110 def test_file_transfer_ended(self, mock_event_time, caplog):
91 url = urlparse('tftp://some-host/my-file')111 url = urlparse('tftp://some-host/my-file')
92 machine = Machine(None, None, 'eth0')112 parent_iface = create_mock_parent_interface()
113 machine = Machine(None, parent_iface, 'eth0')
93 machine.last_event_time[f'file_transfer_start_{url.geturl()}'] = 12345114 machine.last_event_time[f'file_transfer_start_{url.geturl()}'] = 12345
94 mock_event_time.return_value = 12347.5115 mock_event_time.return_value = 12347.5
95 with caplog.at_level(logging.INFO):116 with caplog.at_level(logging.INFO):
@@ -105,7 +126,8 @@ class TestFileTransferEvents:
105 def test_file_transfer_ended_prometheus_count(126 def test_file_transfer_ended_prometheus_count(
106 self, mock_event_time, caplog):127 self, mock_event_time, caplog):
107 url = urlparse('tftp://some-host/my-file')128 url = urlparse('tftp://some-host/my-file')
108 machine = Machine(None, None, 'eth0')129 parent_iface = create_mock_parent_interface()
130 machine = Machine(None, parent_iface, 'eth0')
109 machine.last_event_time[f'file_transfer_start_{url.geturl()}'] = 12345131 machine.last_event_time[f'file_transfer_start_{url.geturl()}'] = 12345
110 mock_event_time.return_value = 12347.5132 mock_event_time.return_value = 12347.5
111 labels = {'scheme': 'tftp', 'filename': '/my-file', 'result': '68'}133 labels = {'scheme': 'tftp', 'filename': '/my-file', 'result': '68'}
@@ -117,8 +139,8 @@ class TestFileTransferEvents:
117 def test_file_transfer_ended_prometheus_bucket_low(139 def test_file_transfer_ended_prometheus_bucket_low(
118 self, mock_event_time, caplog):140 self, mock_event_time, caplog):
119 url = urlparse('tftp://some-host/my-file')141 url = urlparse('tftp://some-host/my-file')
120 machine = Machine(None, None, 'eth0')142 parent_iface = create_mock_parent_interface()
121 machine = Machine(None, None, 'eth0')143 machine = Machine(None, parent_iface, 'eth0')
122 machine.last_event_time[f'file_transfer_start_{url.geturl()}'] = 12345144 machine.last_event_time[f'file_transfer_start_{url.geturl()}'] = 12345
123 mock_event_time.return_value = 12347.5145 mock_event_time.return_value = 12347.5
124 labels = {146 labels = {
@@ -135,7 +157,8 @@ class TestFileTransferEvents:
135 def test_file_transfer_ended_prometheus_bucket_high(157 def test_file_transfer_ended_prometheus_bucket_high(
136 self, mock_event_time, caplog):158 self, mock_event_time, caplog):
137 url = urlparse('tftp://some-host/my-file')159 url = urlparse('tftp://some-host/my-file')
138 machine = Machine(None, None, 'eth0')160 parent_iface = create_mock_parent_interface()
161 machine = Machine(None, parent_iface, 'eth0')
139 machine.last_event_time[f'file_transfer_start_{url.geturl()}'] = 12345162 machine.last_event_time[f'file_transfer_start_{url.geturl()}'] = 12345
140 mock_event_time.return_value = 12347.5163 mock_event_time.return_value = 12347.5
141 labels = {164 labels = {
diff --git a/maasperformance/tests/test_machine.py b/maasperformance/tests/test_machine.py
index c5d1cbe..a0e2e69 100644
--- a/maasperformance/tests/test_machine.py
+++ b/maasperformance/tests/test_machine.py
@@ -1,5 +1,7 @@
1import asyncio1import asyncio
2import logging2import logging
3import random
4from unittest.mock import Mock
3import uuid5import uuid
46
5import pytest7import pytest
@@ -7,17 +9,55 @@ import pytest
7from .. import (9from .. import (
8 event,10 event,
9 machine as machine_module,11 machine as machine_module,
12 process,
13)
14from ..bmc import (
15 BMC,
16 PowerState,
10)17)
11from ..bmc import PowerState
12from ..machine import (18from ..machine import (
13 Machine,19 Machine,
14 MachineManager,20 MachineManager,
15)21)
16from ..network_interface import ParentNetworkInterface22from ..network_interface import ParentNetworkInterface
17from ..testing.fixtures import create_subprocess_mock23from ..testing.fixtures import (
24 create_check_output_mock,
25 create_subprocess_mock,
26)
18from ..testing.prometheus import track_metric27from ..testing.prometheus import track_metric
1928
20subprocess_mock = create_subprocess_mock()29subprocess_mock = create_subprocess_mock()
30check_output_mock = create_check_output_mock(
31 module=process,
32 output="""
33[
34 {
35 "ifindex":1,
36 "ifname":"eth0",
37 "flags":["BROADCAST","MULTICAST","UP","LOWER_UP"],
38 "mtu":1500,
39 "qdisc":"mq",
40 "operstate":"UP",
41 "linkmode":"DEFAULT",
42 "group":"default",
43 "txqlen":1000,
44 "link_type":"ether",
45 "address":"80:61:5f:08:fc:16",
46 "broadcast":"ff:ff:ff:ff:ff:ff",
47 "promiscuity":0,
48 "min_mtu":68,
49 "max_mtu":9710,
50 "inet6_addr_gen_mode":"none",
51 "num_tx_queues":64,
52 "num_rx_queues":64,
53 "gso_max_size":65536,
54 "gso_max_segs":65535,
55 "parentbus":"pci",
56 "parentdev":"0000:01:00.0",
57 "vfinfo_list":[]
58 }
59]
60""")
2161
2262
23class TestMachine:63class TestMachine:
@@ -60,6 +100,7 @@ class TestMachine:
60 'bridge')100 'bridge')
61 ]101 ]
62102
103 @pytest.mark.asyncio
63 async def test_create_pxe_interface_registers_cleanup(104 async def test_create_pxe_interface_registers_cleanup(
64 self, machine, subprocess_mock):105 self, machine, subprocess_mock):
65 await machine.create_pxe_interface()106 await machine.create_pxe_interface()
@@ -151,14 +192,35 @@ class TestMachine:
151 assert machine.last_event_time['dhcp_release_end'] == 67890192 assert machine.last_event_time['dhcp_release_end'] == 67890
152193
153 @pytest.mark.asyncio194 @pytest.mark.asyncio
154 async def test_get_pxe_files_binaries(self, machine, subprocess_mock):195 async def test_get_pxe_files_binaries(
196 self, machine, subprocess_mock, mocker):
155 fake_curl = subprocess_mock.fake_processes['curl']197 fake_curl = subprocess_mock.fake_processes['curl']
156 fake_dhclient = subprocess_mock.fake_processes['dhclient']198 fake_dhclient = subprocess_mock.fake_processes['dhclient']
157 fake_dhclient.next_server = '4.3.2.1'199 fake_dhclient.next_server = '4.3.2.1'
158 fake_curl.ip_address = '4.3.2.1'200 fake_curl.ip_address = '4.3.2.1'
201
202 mocker.patch.object(machine, 'process_preseed')
203 mocker.patch.object(machine, '_sanitize_url')
204
205 class MockGetHTTPFile:
206
207 async def __call__(self, *args, **kwargs):
208 return 200, "{}"
209
210 mocker.patch.object(
211 machine.network, 'get_http_file', new=MockGetHTTPFile())
212
213 machine.metadata = Mock()
214
215 async def _noop_request(*args, **kwargs):
216 return "{\"metadata_url\": \"http://4.3.2.1/metadata\"}"
217
218 machine.metadata.request = _noop_request
219
159 await machine.create_pxe_interface()220 await machine.create_pxe_interface()
160 await machine.get_ip()221 await machine.get_ip()
161 await machine.get_pxe_files()222 await machine.get_pxe_files()
223
162 assert fake_curl.requested_urls[:2] == [224 assert fake_curl.requested_urls[:2] == [
163 ('tftp://4.3.2.1/lpxelinux.0', 0),225 ('tftp://4.3.2.1/lpxelinux.0', 0),
164 ('tftp://4.3.2.1/ldlinux.c32', 0),226 ('tftp://4.3.2.1/ldlinux.c32', 0),
@@ -171,17 +233,35 @@ class TestMachine:
171 fake_dhclient = subprocess_mock.fake_processes['dhclient']233 fake_dhclient = subprocess_mock.fake_processes['dhclient']
172 fake_dhclient.next_server = '4.3.2.1'234 fake_dhclient.next_server = '4.3.2.1'
173 fake_curl.ip_address = '4.3.2.1'235 fake_curl.ip_address = '4.3.2.1'
174 machine_uuid = uuid.UUID('816fd0e0-0d52-cb11-877b-9c25109f8fba')236 machine.uuid = uuid.UUID('816fd0e0-0d52-cb11-877b-9c25109f8fba')
175 uuid_mock = mocker.patch.object(uuid, 'uuid1')237
176 uuid_mock.side_effect = [machine_uuid]238 mocker.patch.object(machine, 'process_preseed')
239 mocker.patch.object(machine, '_sanitize_url')
240
241 class MockGetHTTPFile:
242
243 async def __call__(self, *args, **kwargs):
244 return 200, "{}"
245
246 mocker.patch.object(
247 machine.network, 'get_http_file', new=MockGetHTTPFile())
248
249 machine.metadata = Mock()
250
251 async def _noop_request(*args, **kwargs):
252 return "{\"metadata_url\": \"http://4.3.2.1/metadata\"}"
253
254 machine.metadata.request = _noop_request
255
177 await machine.create_pxe_interface()256 await machine.create_pxe_interface()
178 await machine.get_ip()257 await machine.get_ip()
179 await machine.get_pxe_files()258 await machine.get_pxe_files()
180 assert fake_curl.requested_urls[2:13] == [259 mac_route = machine.client_iface.mac_address.replace(':', '-')
260 expected_calls = [
181 (261 (
182 'tftp://4.3.2.1/pxelinux.cfg/'262 'tftp://4.3.2.1/pxelinux.cfg/'
183 '816fd0e0-0d52-cb11-877b-9c25109f8fba', 68),263 '816fd0e0-0d52-cb11-877b-9c25109f8fba', 68),
184 ('tftp://4.3.2.1/pxelinux.cfg/01-11-22-33-44-55-66', 68),264 (f"tftp://4.3.2.1/pxelinux.cfg/01-{mac_route}", 68),
185 ('tftp://4.3.2.1/pxelinux.cfg/02020202', 68),265 ('tftp://4.3.2.1/pxelinux.cfg/02020202', 68),
186 ('tftp://4.3.2.1/pxelinux.cfg/0202020', 68),266 ('tftp://4.3.2.1/pxelinux.cfg/0202020', 68),
187 ('tftp://4.3.2.1/pxelinux.cfg/020202', 68),267 ('tftp://4.3.2.1/pxelinux.cfg/020202', 68),
@@ -192,10 +272,30 @@ class TestMachine:
192 ('tftp://4.3.2.1/pxelinux.cfg/0', 68),272 ('tftp://4.3.2.1/pxelinux.cfg/0', 68),
193 ('tftp://4.3.2.1/pxelinux.cfg/default', 0),273 ('tftp://4.3.2.1/pxelinux.cfg/default', 0),
194 ]274 ]
275 for expected_call in expected_calls:
276 assert expected_call in fake_curl.requested_urls[2:13]
195277
196 @pytest.mark.asyncio278 @pytest.mark.asyncio
197 async def test_get_pxe_files_event_pxelinux_success(279 async def test_get_pxe_files_event_pxelinux_success(
198 self, machine, subprocess_mock, event_loop):280 self, machine, subprocess_mock, event_loop, mocker):
281 mocker.patch.object(machine, 'process_preseed')
282 mocker.patch.object(machine, '_sanitize_url')
283
284 class MockGetHTTPFile:
285
286 async def __call__(self, *args, **kwargs):
287 return 200, "{}"
288
289 mocker.patch.object(
290 machine.network, 'get_http_file', new=MockGetHTTPFile())
291
292 machine.metadata = Mock()
293
294 async def _noop_request(*args, **kwargs):
295 return "{\"metadata_url\": \"http://4.3.2.1/metadata\"}"
296
297 machine.metadata.request = _noop_request
298
199 await machine.create_pxe_interface()299 await machine.create_pxe_interface()
200 await machine.get_ip()300 await machine.get_ip()
201 labels = {'scheme': 'tftp', 'filename': 'pxelinux.cfg', 'result': '0'}301 labels = {'scheme': 'tftp', 'filename': 'pxelinux.cfg', 'result': '0'}
@@ -206,7 +306,25 @@ class TestMachine:
206306
207 @pytest.mark.asyncio307 @pytest.mark.asyncio
208 async def test_get_pxe_files_event_pxelinux_failures(308 async def test_get_pxe_files_event_pxelinux_failures(
209 self, machine, subprocess_mock, event_loop):309 self, machine, subprocess_mock, event_loop, mocker):
310 mocker.patch.object(machine, 'process_preseed')
311 mocker.patch.object(machine, '_sanitize_url')
312
313 class MockGetHTTPFile:
314
315 async def __call__(self, *args, **kwargs):
316 return 200, "{}"
317
318 mocker.patch.object(
319 machine.network, 'get_http_file', new=MockGetHTTPFile())
320
321 machine.metadata = Mock()
322
323 async def _noop_request(*args, **kwargs):
324 return "{\"metadata_url\": \"http://4.3.2.1/metadata\"}"
325
326 machine.metadata.request = _noop_request
327
210 await machine.create_pxe_interface()328 await machine.create_pxe_interface()
211 await machine.get_ip()329 await machine.get_ip()
212 labels = {'scheme': 'tftp', 'filename': 'pxelinux.cfg', 'result': '68'}330 labels = {'scheme': 'tftp', 'filename': 'pxelinux.cfg', 'result': '68'}
@@ -217,9 +335,28 @@ class TestMachine:
217335
218 @pytest.mark.asyncio336 @pytest.mark.asyncio
219 async def test_get_pxe_files_event_initrd_success(337 async def test_get_pxe_files_event_initrd_success(
220 self, machine, subprocess_mock, event_loop):338 self, machine, subprocess_mock, event_loop, mocker):
221 fake_curl = subprocess_mock.fake_processes['curl']339 fake_curl = subprocess_mock.fake_processes['curl']
222 fake_curl.initrd_path = '/boot/my-initrd'340 fake_curl.initrd_path = '/boot/my-initrd'
341
342 mocker.patch.object(machine, 'process_preseed')
343 mocker.patch.object(machine, '_sanitize_url')
344
345 class MockGetHTTPFile:
346
347 async def __call__(self, *args, **kwargs):
348 return 200, "{}"
349
350 mocker.patch.object(
351 machine.network, 'get_http_file', new=MockGetHTTPFile())
352
353 machine.metadata = Mock()
354
355 async def _noop_request(*args, **kwargs):
356 return "{\"metadata_url\": \"http://4.3.2.1/metadata\"}"
357
358 machine.metadata.request = _noop_request
359
223 await machine.create_pxe_interface()360 await machine.create_pxe_interface()
224 await machine.get_ip()361 await machine.get_ip()
225 labels = {362 labels = {
@@ -240,6 +377,25 @@ class TestMachine:
240 fake_curl.ip_address = '4.3.2.1'377 fake_curl.ip_address = '4.3.2.1'
241 fake_curl.kernel_path = '/boot/my-kernel'378 fake_curl.kernel_path = '/boot/my-kernel'
242 fake_curl.initrd_path = '/boot/my-initrd'379 fake_curl.initrd_path = '/boot/my-initrd'
380
381 mocker.patch.object(machine, 'process_preseed')
382 mocker.patch.object(machine, '_sanitize_url')
383
384 class MockGetHTTPFile:
385
386 async def __call__(self, *args, **kwargs):
387 return 200, "{}"
388
389 mocker.patch.object(
390 machine.network, 'get_http_file', new=MockGetHTTPFile())
391
392 machine.metadata = Mock()
393
394 async def _noop_request(*args, **kwargs):
395 return "{\"metadata_url\": \"http://4.3.2.1/metadata\"}"
396
397 machine.metadata.request = _noop_request
398
243 await machine.create_pxe_interface()399 await machine.create_pxe_interface()
244 await machine.get_ip()400 await machine.get_ip()
245 await machine.get_pxe_files()401 await machine.get_pxe_files()
@@ -295,33 +451,59 @@ class TestMachineManager:
295 (event_loop, 'eth0', 'fake0'), (event_loop, 'eth0', 'fake1'),451 (event_loop, 'eth0', 'fake0'), (event_loop, 'eth0', 'fake1'),
296 (event_loop, 'eth0', 'fake2')452 (event_loop, 'eth0', 'fake2')
297 ]453 ]
454 await manager.power_commands.join(
455 ) # cleanly close the queue MachineManager's run loop blocks on
456 manager.running = False # quit run loop
298 await task457 await task
299 for machine in manager.machines:458 for machine in manager.machines.values():
300 assert machine.client_mac is not None459 assert machine.client_iface is not None
301460
302 @pytest.mark.asyncio461 @pytest.mark.asyncio
303 async def test_run_creates_interface(self, subprocess_mock, event_loop):462 async def test_run_creates_interface(
463 self, subprocess_mock, check_output_mock, event_loop):
304 manager = MachineManager('eth0', 5)464 manager = MachineManager('eth0', 5)
305 await manager.init(event_loop)465 task = manager.init(event_loop)
466 await manager.power_commands.join(
467 ) # cleanly close the queue MachineManager's run loop blocks on
468 manager.running = False # quit run loop
469 await task
306 assert len(manager.machines) == 5470 assert len(manager.machines) == 5
307 for machine in manager.machines:471 for machine in manager.machines.values():
308 assert machine.client_mac is not None472 assert machine.client_iface is not None
309473
310 @pytest.mark.asyncio474 @pytest.mark.asyncio
311 async def test_run_gets_ip(self, subprocess_mock, event_loop):475 async def test_run_gets_ip(
476 self, subprocess_mock, check_output_mock, event_loop, mocker):
312 number_of_machines = 5477 number_of_machines = 5
478
479 task = None
480
481 class MockGetIP:
482 called = False
483 call_count = 0
484
485 async def __call__(self, *args, **kwargs):
486 self.called = True
487 self.call_count += 1
488 if self.call_count == number_of_machines:
489 task.cancel()
490
491 mocker.patch.object(Machine, 'get_ip', new=MockGetIP())
313 manager = MachineManager('eth0', number_of_machines)492 manager = MachineManager('eth0', number_of_machines)
314 with track_metric('client_dhcp_latency_count',493 bmc = BMC(manager)
315 {'type': 'request'}) as metric:494 task = manager.init(event_loop)
316 await manager.init(event_loop)495 await bmc.init(event_loop)
496 await manager.power_commands.join(
497 ) # cleanly close the queue MachineManager's run loop blocks on
498 await task
317 assert len(manager.machines) == number_of_machines499 assert len(manager.machines) == number_of_machines
318 for machine in manager.machines:500 for machine in manager.machines.values():
319 assert 'dhcp_request_end' in machine.last_event_time501 assert machine.get_ip.called
320 assert metric.increase == number_of_machines * 1.0
321502
322 @pytest.mark.asyncio503 @pytest.mark.asyncio
323 async def test_run_logs_durations(504 async def test_run_logs_durations(
324 self, caplog, subprocess_mock, mocker, event_loop):505 self, caplog, subprocess_mock, check_output_mock, mocker,
506 event_loop):
325 manager = MachineManager('eth0', 5)507 manager = MachineManager('eth0', 5)
326 with caplog.at_level(logging.INFO):508 with caplog.at_level(logging.INFO):
327 now_mock = mocker.patch.object(machine_module, 'now')509 now_mock = mocker.patch.object(machine_module, 'now')
@@ -331,20 +513,24 @@ class TestMachineManager:
331 4, # Start of DHCP request513 4, # Start of DHCP request
332 7, # End of DHCP request514 7, # End of DHCP request
333 ]515 ]
334 await manager.init(event_loop)516 task = manager.init(event_loop)
517 await manager.power_commands.join()
518 manager.running = False
519 await task
335520
336 log_entries = [521 log_entries = [
337 (entry.levelname, entry.message) for entry in caplog.records522 (entry.levelname, entry.message) for entry in caplog.records
338 ]523 ]
339 assert ('INFO', 'Created PXE interfaces in 1 seconds') in log_entries524 assert ('INFO', 'Created PXE interfaces in 1 seconds') in log_entries
340 assert ('INFO', 'Got IPs for all machines in 3 seconds') in log_entries
341525
342 @pytest.mark.asyncio526 @pytest.mark.asyncio
343 async def test_run_handles_task_cancellation(527 async def test_run_handles_task_cancellation(
344 self, subprocess_mock, event_loop):528 self, subprocess_mock, check_output_mock, event_loop):
345529
346 class CancellingMachine:530 class CancellingMachine:
531 uuid = random.randint(1, 512)
347 cancelled = False532 cancelled = False
533 last_event_time = {}
348534
349 async def create_pxe_interface(self):535 async def create_pxe_interface(self):
350 self.cancelled = True536 self.cancelled = True
@@ -354,16 +540,17 @@ class TestMachineManager:
354 cancelling_machine = CancellingMachine()540 cancelling_machine = CancellingMachine()
355 task = manager.init(event_loop)541 task = manager.init(event_loop)
356 # Inject a machine that simulates the async task being cancelled.542 # Inject a machine that simulates the async task being cancelled.
357 manager.machines.append(cancelling_machine)543 manager.machines[cancelling_machine.uuid] = cancelling_machine
358 await task544 await task
359 # The async task was cancelled without any problems, and the545 # The async task was cancelled without any problems, and the
360 # rest of the run() method was skipped.546 # rest of the run() method was skipped.
361 assert cancelling_machine.cancelled547 assert cancelling_machine.cancelled
362 for machine in manager.machines[:-1]:548 for machine in manager.machines.values():
363 assert 'dhcp_request_start' not in machine.last_event_time549 assert 'dhcp_request_start' not in machine.last_event_time
364550
365 @pytest.mark.asyncio551 @pytest.mark.asyncio
366 async def test_clean_up(self, subprocess_mock, event_loop):552 async def test_clean_up(
553 self, subprocess_mock, check_output_mock, event_loop):
367554
368 class CleanUp:555 class CleanUp:
369 cleaned_up = False556 cleaned_up = False
@@ -373,7 +560,10 @@ class TestMachineManager:
373560
374 clean_up = CleanUp()561 clean_up = CleanUp()
375 manager = MachineManager('eth0', 1)562 manager = MachineManager('eth0', 1)
376 await manager.init(event_loop)563 task = manager.init(event_loop)
377 manager.machines[0].cleanups.append(clean_up.run)564 await manager.power_commands.join()
565 manager.running = False
566 await task
567 list(manager.machines.values())[0].cleanups.append(clean_up.run)
378 await manager.clean_up()568 await manager.clean_up()
379 assert clean_up.cleaned_up569 assert clean_up.cleaned_up
diff --git a/maasperformance/tests/test_network.py b/maasperformance/tests/test_network.py
index 3c1db36..e60345d 100644
--- a/maasperformance/tests/test_network.py
+++ b/maasperformance/tests/test_network.py
@@ -1,8 +1,22 @@
1from unittest.mock import Mock
2
3import aiohttp
1from netaddr import IPAddress4from netaddr import IPAddress
5import pytest
26
3from ..network import get_source_address7from ..network import (
8 get_source_address,
9 MetadataClient,
10 NetworkClient,
11 PlaintextSignature,
12 register_machine,
13)
14from ..process import ProcessError
15from ..testing.fixtures import create_subprocess_mock
4from ..testing.network import fake_socket16from ..testing.network import fake_socket
517
18subprocess_mock = create_subprocess_mock()
19
620
7class TestGetSourceaddress:21class TestGetSourceaddress:
822
@@ -20,3 +34,219 @@ class TestGetSourceaddress:
20 own_ip = get_source_address(34 own_ip = get_source_address(
21 IPAddress('1200:0000:AB00:1234:0000:2552:7777:1313'))35 IPAddress('1200:0000:AB00:1234:0000:2552:7777:1313'))
22 assert own_ip == IPAddress(sock.ipv6_host)36 assert own_ip == IPAddress(sock.ipv6_host)
37
38
39class TestNetworkClient:
40
41 @pytest.mark.asyncio
42 async def test_get_curl_file(self, subprocess_mock):
43 machine = Mock()
44 machine.last_event_time = {}
45 client = NetworkClient(machine)
46 try:
47 await client.get_curl_file("http://foo/")
48 except ProcessError:
49 pass
50 assert (
51 "curl", "-o", "/dev/null", "-s",
52 "http://foo/") in subprocess_mock.calls
53
54 @pytest.mark.asyncio
55 async def test_get_json_file(self, mocker):
56 machine = Mock()
57 machine.last_event_time = {}
58 client = NetworkClient(machine)
59
60 class MockGetHTTPFile:
61 args = None
62 kwargs = None
63
64 async def __call__(self, *args, **kwargs):
65 self.args = args
66 self.kwargs = kwargs
67 return 200, "{\"foo\": \"bar\"}"
68
69 mock_get_http_file = MockGetHTTPFile()
70 mocker.patch.object(client, "get_http_file", new=mock_get_http_file)
71 status, result = await client.get_json_file("http://foo/")
72 assert mock_get_http_file.args == ("http://foo/", )
73 assert mock_get_http_file.kwargs == {
74 "headers": {
75 "Accept": "application/json"
76 }
77 }
78 assert status == 200
79 assert isinstance(result, dict)
80
81 @pytest.mark.asyncio
82 async def test_get_tftp_file(self, subprocess_mock):
83 machine = Mock()
84 machine.last_event_time = {}
85 machine.next_server = "foo"
86 client = NetworkClient(machine)
87 try:
88 await client.get_tftp_file("/bar")
89 except ProcessError:
90 pass
91 assert (
92 "curl", "-o", "/dev/null", "-s",
93 "tftp://foo/bar") in subprocess_mock.calls
94
95
96class TestPlaintextSignature:
97
98 def test_sign(self):
99 sig = PlaintextSignature()
100 key = sig.sign("foo", "GET", "http://bar/", "secret")
101 assert key == "foo&secret"
102
103
104class MockableRequest:
105
106 def mock_request(self, client, mocker, response="", status=None):
107
108 class MockRequest:
109 args = None
110 kwargs = None
111
112 async def __call__(self, *args, **kwargs):
113 self.args = args
114 self.kwargs = kwargs
115 if status:
116 return status, response
117 return response
118
119 mock_request = MockRequest()
120 mocker.patch.object(client, "request", new=mock_request)
121
122 return mock_request
123
124
125class TestMetadataClient(MockableRequest):
126
127 def test_authenticated_False_without_consumer_key(self):
128 client = MetadataClient(Mock(), {"metadata_url": "http://foo/"})
129 assert client.authenticated is False
130
131 def test_authenticated_True_with_consumer_key(self):
132 client = MetadataClient(
133 Mock(), {
134 "metadata_url": "http://foo/",
135 "consumer_key": "bar"
136 })
137 assert client.authenticated
138
139 @pytest.mark.asyncio
140 async def test_signal(self, mocker):
141 network = Mock()
142 client = MetadataClient(
143 network, {
144 "metadata_url": "http://foo/",
145 "consumer_key": "bar"
146 })
147
148 mock_request = self.mock_request(client, mocker)
149
150 await client.signal("TESTING")
151
152 assert mock_request.args == ("post", "")
153 expected_data = aiohttp.FormData()
154 expected_data.add_field("op", "signal")
155 expected_data.add_field("status", "TESTING")
156 assert mock_request.kwargs["data"]._fields == expected_data._fields
157
158 @pytest.mark.asyncio
159 async def test_netboot_off(self, mocker):
160 network = Mock()
161 client = MetadataClient(
162 network, {
163 "metadata_url": "http://foo/",
164 "consumer_key": "bar"
165 })
166
167 mock_request = self.mock_request(client, mocker)
168
169 await client.netboot_off()
170
171 expected_data = aiohttp.FormData()
172 expected_data.add_field("op", "netboot_off")
173 assert mock_request.args == ("post", "")
174 assert mock_request.kwargs["data"]._fields == expected_data._fields
175
176 @pytest.mark.asyncio
177 async def test_get_scripts_index_no_response(self, mocker):
178 network = Mock()
179 client = MetadataClient(
180 network, {
181 "metadata_url": "http://foo/",
182 "consumer_key": "bar"
183 })
184
185 self.mock_request(client, mocker)
186
187 result = await client.get_scripts_index()
188 assert result == {}
189
190 @pytest.mark.asyncio
191 async def test_get_scripts_index_tar_file_response(
192 self, mocker, tar_file_http_response):
193 network = Mock()
194 client = MetadataClient(
195 network, {
196 "metadata_url": "http://foo/",
197 "consumer_key": "bar"
198 })
199
200 self.mock_request(client, mocker, response=tar_file_http_response)
201
202 result = await client.get_scripts_index()
203 assert result == {"foo": "bar"}
204
205 def test_get_endpoint_full_url(self):
206 network = Mock()
207 client = MetadataClient(
208 network, {
209 "metadata_url": "http://foo/",
210 "consumer_key": "bar"
211 })
212 url = client._get_endpoint_full_url("http://foo/")
213 assert url == "http://foo/2012-03-01/"
214
215
216class TestRegisterMachine(MockableRequest):
217
218 async def test_register_machine(self, mocker):
219 network = NetworkClient(Mock())
220
221 mock_request = self.mock_request(network, mocker, status=200)
222
223 api_url = "http://foo/"
224 hostname = "test.hostname"
225 architecture = "amd64"
226 subarchitecture = "generic"
227 power_type = "ipmi"
228 power_parameters = {
229 "address": "1.1.1.1",
230 "username": "foo",
231 "password": "bar",
232 }
233 mac_address = "11:22:33:44:55"
234 headers = {"Accept": "application/json"}
235
236 await register_machine(
237 network, api_url, hostname, architecture, subarchitecture,
238 power_type, power_parameters, mac_address)
239
240 expected_data = aiohttp.FormData()
241 expected_data.add_field("hostname", hostname)
242 expected_data.add_field("architecture", architecture)
243 expected_data.add_field("subarchitecture", subarchitecture)
244 expected_data.add_field("mac_addresses", mac_address)
245 expected_data.add_field("commission", "true")
246 expected_data.add_field("power_type", power_type)
247 for key, value in power_parameters.items():
248 expected_data.add_field("power_parameters_" + key, value)
249
250 assert mock_request.args == ("post", api_url + "machines/")
251 assert mock_request.kwargs["headers"] == headers
252 assert mock_request.kwargs["data"]._fields == expected_data._fields
diff --git a/maasperformance/tests/test_network_interface.py b/maasperformance/tests/test_network_interface.py
index 0e55d91..bd525e4 100644
--- a/maasperformance/tests/test_network_interface.py
+++ b/maasperformance/tests/test_network_interface.py
@@ -2,7 +2,7 @@ import json
22
3import pytest3import pytest
44
5from .. import network_interface as network_interface_module5from .. import process
6from ..network_interface import (6from ..network_interface import (
7 generate_mac_address,7 generate_mac_address,
8 interface_is_bridge,8 interface_is_bridge,
@@ -29,12 +29,11 @@ class TestInterfaceIsBridge:
29 (IP_OUTPUT_VETH, 'veth0', False),29 (IP_OUTPUT_VETH, 'veth0', False),
30 ])30 ])
31 def test_interface_is_bridge(self, mocker, output, iface, is_bridge):31 def test_interface_is_bridge(self, mocker, output, iface, is_bridge):
32 mock_check_output = mocker.patch.object(32 mock_check_output = mocker.patch.object(process, 'check_output')
33 network_interface_module, 'check_output')
34 mock_check_output.return_value = json.dumps(output)33 mock_check_output.return_value = json.dumps(output)
35 assert interface_is_bridge(iface) == is_bridge34 assert interface_is_bridge(iface) == is_bridge
36 mock_check_output.assert_called_once_with(35 mock_check_output.assert_called_once_with(
37 ['ip', '--json', '-d', 'link', 'show', iface])36 ('ip', '--json', '-d', 'link', 'show', iface))
3837
3938
40class TestGenerateMacAddress:39class TestGenerateMacAddress:
diff --git a/maasperformance/tests/test_web.py b/maasperformance/tests/test_web.py
index fdbee5d..aa408ba 100644
--- a/maasperformance/tests/test_web.py
+++ b/maasperformance/tests/test_web.py
@@ -2,7 +2,12 @@ import asyncio
22
3import prometheus_client3import prometheus_client
44
5from .. import process
5from ..server import parse_args6from ..server import parse_args
7from ..testing.fixtures import (
8 create_check_output_mock,
9 create_subprocess_mock,
10)
6from ..testing.prometheus import (11from ..testing.prometheus import (
7 create_promreg_app,12 create_promreg_app,
8 PromRegHandlers,13 PromRegHandlers,
@@ -14,22 +19,58 @@ from ..web import (
14 start_machines_loop,19 start_machines_loop,
15)20)
1621
1722subprocess_mock = create_subprocess_mock(module=asyncio)
18def create_app(port=1234, iface='eth0', number=1):23check_output_mock = create_check_output_mock(
24 module=process,
25 output="""
26[
27 {
28 "ifindex":1,
29 "ifname":"parent0",
30 "flags":["BROADCAST","MULTICAST","UP","LOWER_UP"],
31 "mtu":1500,
32 "qdisc":"mq",
33 "operstate":"UP",
34 "linkmode":"DEFAULT",
35 "group":"default",
36 "txqlen":1000,
37 "link_type":"ether",
38 "address":"80:61:5f:08:fc:16",
39 "broadcast":"ff:ff:ff:ff:ff:ff",
40 "promiscuity":0,
41 "min_mtu":68,
42 "max_mtu":9710,
43 "inet6_addr_gen_mode":"none",
44 "num_tx_queues":64,
45 "num_rx_queues":64,
46 "gso_max_size":65536,
47 "gso_max_segs":65535,
48 "parentbus":"pci",
49 "parentdev":"0000:01:00.0",
50 "vfinfo_list":[]
51 }
52]
53""")
54
55
56def create_app(port=1234, iface='eth0', number=1, loop=None):
19 args = parse_args(['--port', str(port), '--number', str(number), iface])57 args = parse_args(['--port', str(port), '--number', str(number), iface])
20 return create_web_app(args)58 return create_web_app(args, loop=loop)
2159
2260
23class TestStatus:61class TestStatus:
2462
25 async def test_status(self, create_subprocess_mock, aiohttp_client):63 async def test_status(
26 app = create_app(number=2)64 self, subprocess_mock, aiohttp_client, check_output_mock,
65 event_loop):
66 app = create_app(number=2, loop=event_loop)
27 client = await aiohttp_client(app)67 client = await aiohttp_client(app)
68 app['machine-manager'].running = False
28 await app['machine-manager'].finished69 await app['machine-manager'].finished
29 resp = await client.get('/status')70 resp = await client.get('/status')
30 assert resp.status == 20071 assert resp.status == 200
31 contents = await resp.json()72 contents = await resp.json()
32 machines = app['machine-manager'].machines73 machines = list(app['machine-manager'].machines.values())
33 assert contents == [74 assert contents == [
34 machines[0].get_status(),75 machines[0].get_status(),
35 machines[1].get_status(),76 machines[1].get_status(),
@@ -38,7 +79,8 @@ class TestStatus:
3879
39class TestMetrics:80class TestMetrics:
4081
41 async def test_content_type(self, create_subprocess_mock, aiohttp_client):82 async def test_content_type(
83 self, subprocess_mock, check_output_mock, aiohttp_client):
42 app = create_app(number=2)84 app = create_app(number=2)
43 client = await aiohttp_client(app)85 client = await aiohttp_client(app)
44 resp = await client.get('/metrics')86 resp = await client.get('/metrics')
@@ -48,7 +90,7 @@ class TestMetrics:
48 prometheus_client.CONTENT_TYPE_LATEST)90 prometheus_client.CONTENT_TYPE_LATEST)
4991
50 async def test_defined_metrics(92 async def test_defined_metrics(
51 self, create_subprocess_mock, aiohttp_client):93 self, subprocess_mock, check_output_mock, aiohttp_client):
52 app = create_app(number=2)94 app = create_app(number=2)
53 client = await aiohttp_client(app)95 client = await aiohttp_client(app)
54 resp = await client.get('/metrics')96 resp = await client.get('/metrics')
@@ -58,53 +100,56 @@ class TestMetrics:
58100
59class TestCreateWebApp:101class TestCreateWebApp:
60102
61 def test_create_web_app_sets_port(self):103 def test_create_web_app_sets_port(self, check_output_mock):
62 args = parse_args(['--port', '1234', '--number', '5', 'myif0'])104 args = parse_args(['--port', '1234', '--number', '5', 'myif0'])
63 app = create_web_app(args)105 app = create_web_app(args)
64 assert app['port'] == '1234'106 assert app['port'] == '1234'
65107
66 def test_create_web_app_creates_machine_manager(self):108 def test_create_web_app_creates_machine_manager(self, check_output_mock):
67 args = parse_args(['--port', '1234', '--number', '5', 'parent0'])109 args = parse_args(['--port', '1234', '--number', '5', 'parent0'])
68 app = create_web_app(args)110 app = create_web_app(args)
69 manager = app['machine-manager']111 manager = app['machine-manager']
70112
71 assert manager.parent_iface == 'parent0'113 assert manager.parent_iface.name == 'parent0'
72 assert manager.number_of_machines == 5114 assert manager.number_of_machines == 5
73 # We expect an empty machine list, since create_web_app()115 # We expect an empty machine list, since create_web_app()
74 # shouldn't call manager.init() directly.116 # shouldn't call manager.init() directly.
75 assert manager.machines == []117 assert manager.machines == []
76118
77 async def test_create_web_app_starts_manager_on_startup(119 async def test_create_web_app_starts_manager_on_startup(
78 self, create_subprocess_mock, loop):120 self, subprocess_mock, check_output_mock, loop):
79 args = parse_args(['--port', '1234', '--number', '5', 'parent0'])121 args = parse_args(['--port', '1234', '--number', '5', 'parent0'])
80 app = create_web_app(args, loop=loop)122 app = create_web_app(args, loop=loop)
81 for startup_func in app.on_startup:123 for startup_func in app.on_startup:
82 await startup_func(app)124 await startup_func(app)
125 manager = app['machine-manager']
126 manager.running = False
83 await app['manager-task']127 await app['manager-task']
84128
85 manager = app['machine-manager']
86 assert len(manager.machines) == 5129 assert len(manager.machines) == 5
87 for machine in manager.machines:130 for machine in manager.machines.values():
88 assert machine.client_mac is not None131 assert machine.client_iface is not None
89132
90 async def test_create_web_app_cleans_machines_on_cleanup(133 async def test_create_web_app_cleans_machines_on_cleanup(
91 self, create_subprocess_mock, loop):134 self, subprocess_mock, check_output_mock, event_loop):
92 args = parse_args(['--port', '1234', '--number', '5', 'parent0'])135 args = parse_args(['--port', '1234', '--number', '5', 'parent0'])
93 app = create_web_app(args, loop=loop)136 app = create_web_app(args, loop=event_loop)
137 manager = app['machine-manager']
94 for startup_func in app.on_startup:138 for startup_func in app.on_startup:
95 await startup_func(app)139 await startup_func(app)
140 manager.running = False
96 await app['manager-task']141 await app['manager-task']
97 for cleanup_func in app.on_cleanup:142 for cleanup_func in app.on_cleanup:
98 await cleanup_func(app)143 await cleanup_func(app)
99144
100 manager = app['machine-manager']
101 assert len(manager.machines) == 5145 assert len(manager.machines) == 5
102 for machine in manager.machines:146 for machine in manager.machines.values():
103 cleanup_cmd = ('ip', 'link', 'del', machine.client_iface)147 cleanup_cmd = ('ip', 'link', 'del', machine.client_iface.name)
104 assert cleanup_cmd in create_subprocess_mock.calls148 assert cleanup_cmd in subprocess_mock.calls
105149
106 async def test_registers_promreg_on_startup(150 async def test_registers_promreg_on_startup(
107 self, create_subprocess_mock, loop, aiohttp_server):151 self, subprocess_mock, check_output_mock, event_loop,
152 aiohttp_server):
108 handlers = PromRegHandlers('my-token')153 handlers = PromRegHandlers('my-token')
109 server = await aiohttp_server(create_promreg_app(handlers))154 server = await aiohttp_server(create_promreg_app(handlers))
110 args = parse_args(155 args = parse_args(
@@ -113,15 +158,20 @@ class TestCreateWebApp:
113 str(server.make_url('/')), '--promreg-token', 'my-token',158 str(server.make_url('/')), '--promreg-token', 'my-token',
114 '--port', '1234', 'parent0'159 '--port', '1234', 'parent0'
115 ])160 ])
116 app = create_web_app(args, loop=loop)161 app = create_web_app(args, loop=event_loop)
117 for startup_func in app.on_startup:162 for startup_func in app.on_startup:
118 await startup_func(app)163 await startup_func(app)
119 await app['manager-task']164 app['manager-task'].cancel()
165 try:
166 await app['manager-task']
167 except asyncio.exceptions.CancelError:
168 pass
120169
121 assert (list(handlers.targets.keys()) == ['127.0.0.1:1234'])170 assert (list(handlers.targets.keys()) == ['127.0.0.1:1234'])
122171
123 async def test_registers_promreg_on_startup_already_registered(172 async def test_registers_promreg_on_startup_already_registered(
124 self, create_subprocess_mock, loop, aiohttp_server):173 self, subprocess_mock, check_output_mock, event_loop,
174 aiohttp_server):
125 handlers = PromRegHandlers('my-token')175 handlers = PromRegHandlers('my-token')
126 server = await aiohttp_server(create_promreg_app(handlers))176 server = await aiohttp_server(create_promreg_app(handlers))
127 args = parse_args(177 args = parse_args(
@@ -130,16 +180,21 @@ class TestCreateWebApp:
130 str(server.make_url('/')), '--promreg-token', 'my-token',180 str(server.make_url('/')), '--promreg-token', 'my-token',
131 '--port', '1234', 'parent0'181 '--port', '1234', 'parent0'
132 ])182 ])
133 app = create_web_app(args, loop=loop)183 app = create_web_app(args, loop=event_loop)
134 handlers.targets['127.0.0.1:1234'] = PromRegTarget()184 handlers.targets['127.0.0.1:1234'] = PromRegTarget()
135 for startup_func in app.on_startup:185 for startup_func in app.on_startup:
136 await startup_func(app)186 await startup_func(app)
137 await app['manager-task']187 app['manager-task'].cancel()
188 try:
189 await app['manager-task']
190 except asyncio.exceptions.CancelError:
191 pass
138192
139 assert (list(handlers.targets.keys()) == ['127.0.0.1:1234'])193 assert (list(handlers.targets.keys()) == ['127.0.0.1:1234'])
140194
141 async def test_unregisters_promreg_on_shutdown(195 async def test_unregisters_promreg_on_shutdown(
142 self, create_subprocess_mock, loop, aiohttp_server):196 self, subprocess_mock, check_output_mock, event_loop,
197 aiohttp_server):
143 handlers = PromRegHandlers('my-token')198 handlers = PromRegHandlers('my-token')
144 server = await aiohttp_server(create_promreg_app(handlers))199 server = await aiohttp_server(create_promreg_app(handlers))
145 args = parse_args(200 args = parse_args(
@@ -148,13 +203,17 @@ class TestCreateWebApp:
148 str(server.make_url('/')), '--promreg-token', 'my-token',203 str(server.make_url('/')), '--promreg-token', 'my-token',
149 '--port', '1234', 'parent0'204 '--port', '1234', 'parent0'
150 ])205 ])
151 app = create_web_app(args, loop=loop)206 app = create_web_app(args, loop=event_loop)
152 handlers.targets['127.0.0.1:1234'] = PromRegTarget()207 handlers.targets['127.0.0.1:1234'] = PromRegTarget()
153 for startup_func in app.on_startup:208 for startup_func in app.on_startup:
154 await startup_func(app)209 await startup_func(app)
155 for cleanup_func in app.on_cleanup:210 for cleanup_func in app.on_cleanup:
156 await cleanup_func(app)211 await cleanup_func(app)
157 await app['manager-task']212 app['manager-task'].cancel()
213 try:
214 await app['manager-task']
215 except asyncio.exceptions.CancelledError:
216 pass
158217
159 assert list(handlers.targets.keys()) == []218 assert list(handlers.targets.keys()) == []
160219
@@ -162,53 +221,61 @@ class TestCreateWebApp:
162class TestStartMachinesLoop:221class TestStartMachinesLoop:
163222
164 async def test_start_machines_loop_inits_manager(223 async def test_start_machines_loop_inits_manager(
165 self, create_subprocess_mock, loop):224 self, subprocess_mock, event_loop, check_output_mock):
166 args = parse_args(['--port', '1234', '--number', '5', 'parent0'])225 args = parse_args(['--port', '1234', '--number', '5', 'parent0'])
167 app = create_web_app(args, loop=loop)226 app = create_web_app(args, loop=event_loop)
168227
169 await start_machines_loop(app)228 await start_machines_loop(app)
170 manager = app['machine-manager']229 manager = app['machine-manager']
171 assert len(manager.machines) == 5230 assert len(manager.machines) == 5
172 for machine in manager.machines:231 for machine in manager.machines.values():
173 assert machine.loop is loop232 assert machine.loop is event_loop
174 assert machine.client_mac is None233 assert machine.client_iface is not None
175234
176 assert not app['manager-task'].done()235 assert not app['manager-task'].done()
177 await app['manager-task']236 app['manager-task'].cancel()
178 for machine in manager.machines:237 try:
179 assert machine.client_mac is not None238 await app['manager-task']
239 except asyncio.exceptions.CancelledError:
240 pass
241 for machine in manager.machines.values():
242 assert machine.client_iface is not None
180243
181244
182class TestCleanUpMachines:245class TestCleanUpMachines:
183246
184 async def test_cleans_up_machines(self, create_subprocess_mock, loop):247 async def test_cleans_up_machines(
248 self, subprocess_mock, event_loop, check_output_mock):
185 args = parse_args(['--port', '1234', '--number', '5', 'parent0'])249 args = parse_args(['--port', '1234', '--number', '5', 'parent0'])
186 app = create_web_app(args, loop=loop)250 app = create_web_app(args, loop=event_loop)
187251
188 await start_machines_loop(app)252 await start_machines_loop(app)
189 await app['manager-task']253 await app['bmc-task']
190 await clean_up_machines(app)254 await clean_up_machines(app)
191255
256 assert app['manager-task'].done()
257
192 manager = app['machine-manager']258 manager = app['machine-manager']
193 assert len(manager.machines) == 5259 assert len(manager.machines) == 5
194 for machine in manager.machines:260 for machine in manager.machines.values():
195 cleanup_cmd = ('ip', 'link', 'del', machine.client_iface)261 cleanup_cmd = ('ip', 'link', 'del', machine.client_iface.name)
196 assert cleanup_cmd in create_subprocess_mock.calls262 assert cleanup_cmd in subprocess_mock.calls
197263
198 async def test_cancels_manager_task(self, create_subprocess_mock, loop):264 async def test_cancels_manager_task(
265 self, subprocess_mock, event_loop, check_output_mock):
199266
200 async def stall():267 async def stall():
201 machine.cleanups.append(machine.delete_pxe_interface)268 machine.cleanups.append(machine.delete_pxe_interface)
202 stall_point.set_result('there')269 stall_point.set_result('there')
203 await asyncio.sleep(100000, loop=loop)270 await asyncio.sleep(100000, loop=event_loop)
204271
205 args = parse_args(['--port', '1234', '--number', '1', 'parent0'])272 args = parse_args(['--port', '1234', '--number', '1', 'parent0'])
206 app = create_web_app(args, loop=loop)273 app = create_web_app(args, loop=event_loop)
207 manager = app['machine-manager']274 manager = app['machine-manager']
208275
209 await start_machines_loop(app)276 await start_machines_loop(app)
210 stall_point = asyncio.Future()277 stall_point = asyncio.Future()
211 machine = manager.machines[0]278 machine = list(manager.machines.values())[0]
212 # Simulate that the run() method takes a long time, and allow us279 # Simulate that the run() method takes a long time, and allow us
213 # to run clean_up_machines methods while the task is not yet280 # to run clean_up_machines methods while the task is not yet
214 # complete.281 # complete.
@@ -224,6 +291,6 @@ class TestCleanUpMachines:
224 assert app['manager-task'].done()291 assert app['manager-task'].done()
225292
226 assert len(manager.machines) == 1293 assert len(manager.machines) == 1
227 for machine in manager.machines:294 for machine in manager.machines.values():
228 cleanup_cmd = ('ip', 'link', 'del', machine.client_iface)295 cleanup_cmd = ('ip', 'link', 'del', machine.client_iface.name)
229 assert cleanup_cmd in create_subprocess_mock.calls296 assert cleanup_cmd in subprocess_mock.calls
diff --git a/maasperformance/web.py b/maasperformance/web.py
index f3df5c6..38d352d 100644
--- a/maasperformance/web.py
+++ b/maasperformance/web.py
@@ -1,3 +1,4 @@
1from asyncio.exceptions import CancelledError
1import logging2import logging
2import time3import time
3from urllib.parse import urlparse4from urllib.parse import urlparse
@@ -52,7 +53,10 @@ async def clean_up_machines(app):
52 logging.info('Cancelling the manager task')53 logging.info('Cancelling the manager task')
53 app['manager-task'].cancel()54 app['manager-task'].cancel()
54 logging.info('Awaiting the cancel...')55 logging.info('Awaiting the cancel...')
55 await app['manager-task']56 try:
57 await app['manager-task']
58 except CancelledError:
59 pass
56 logging.info('done.')60 logging.info('done.')
57 await manager.clean_up()61 await manager.clean_up()
58 duration = time.time() - start62 duration = time.time() - start
diff --git a/pyproject.toml b/pyproject.toml
59new file mode 10064463new file mode 100644
index 0000000..3ac0a51
--- /dev/null
+++ b/pyproject.toml
@@ -0,0 +1,3 @@
1[tool.pytest.ini_options]
2asyncio_mode = "auto"
3addopts = "--cov-fail-under=90"
diff --git a/requirements.txt b/requirements.txt
index 7ece240..eb73fc9 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,26 +1,27 @@
1aioauth-client==0.25.81aioauth-client==0.27.3
2aiodns==3.0.02aiodns==3.0.0
3aiofiles==0.7.03aiofiles==22.1.0
4aiohttp==3.7.4.post04aiohttp==3.8.1
5anyio==3.1.05aiosignal==1.2.0
6async-timeout==3.0.16anyio==3.6.1
7attrs==21.2.07async-timeout==4.0.2
8attrs==22.1.0
8cchardet==2.1.79cchardet==2.1.7
9certifi==2021.5.3010certifi==2022.6.15.1
10cffi==1.14.511cffi==1.15.1
11chardet==4.0.012charset-normalizer==2.1.1
13frozenlist==1.3.1
12h11==0.12.014h11==0.12.0
13httpcore==0.13.415httpcore==0.15.0
14httpx==0.18.116httpx==0.23.0
15idna==3.217idna==3.3
16multidict==5.1.018multidict==6.0.2
17netaddr==0.8.019netaddr==0.8.0
18prometheus-client==0.11.020prometheus-client==0.14.1
19pycares==4.0.021pycares==4.2.2
20pycparser==2.2022pycparser==2.21
21PyYAML==5.4.123PyYAML==6.0
22rfc3986==1.5.024rfc3986==1.5.0
23sniffio==1.2.025sniffio==1.3.0
24typing-extensions==3.10.0.026uvloop==0.16.0
25uvloop==0.15.227yarl==1.8.1
26yarl==1.6.3

Subscribers

People subscribed via source and target branches