Merge ~powersj/cloud-init:docs/analyze into cloud-init:master
- Git
- lp:~powersj/cloud-init
- docs/analyze
- Merge into master
Proposed by
Joshua Powers
Status: | Merged |
---|---|
Approved by: | Chad Smith |
Approved revision: | f2d60b63e8e9702b22342cfc4df6966de5d231fd |
Merge reported by: | Server Team CI bot |
Merged at revision: | not available |
Proposed branch: | ~powersj/cloud-init:docs/analyze |
Merge into: | cloud-init:master |
Diff against target: |
392 lines (+302/-68) 1 file modified
doc/rtd/topics/analyze.rst (+302/-68) |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Chad Smith | Approve | ||
Server Team CI bot | continuous-integration | Approve | |
Review via email: mp+372651@code.launchpad.net |
Commit message
docs: added output examples to analyze.rst
Description of the change
To post a comment you must log in.
Revision history for this message
Server Team CI bot (server-team-bot) wrote : | # |
review:
Approve
(continuous-integration)
Preview Diff
[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1 | diff --git a/doc/rtd/topics/analyze.rst b/doc/rtd/topics/analyze.rst |
2 | index 5cf38bd..709131b 100644 |
3 | --- a/doc/rtd/topics/analyze.rst |
4 | +++ b/doc/rtd/topics/analyze.rst |
5 | @@ -1,84 +1,318 @@ |
6 | -************************* |
7 | -Cloud-init Analyze Module |
8 | -************************* |
9 | - |
10 | -Overview |
11 | -======== |
12 | -The analyze module was added to cloud-init in order to help analyze cloud-init boot time |
13 | -performance. It is loosely based on systemd-analyze where there are 4 main actions: |
14 | -show, blame, dump, and boot. |
15 | - |
16 | -The 'show' action is similar to 'systemd-analyze critical-chain' which prints a list of units, the |
17 | -time they started and how long they took. For cloud-init, we have four stages, and within each stage |
18 | -a number of modules may run depending on configuration. ‘cloudinit-analyze show’ will, for each |
19 | -boot, print this information and a summary total time, per boot. |
20 | - |
21 | -The 'blame' action matches 'systemd-analyze blame' where it prints, in descending order, |
22 | -the units that took the longest to run. This output is highly useful for examining where cloud-init |
23 | -is spending its time during execution. |
24 | - |
25 | -The 'dump' action simply dumps the cloud-init logs that the analyze module is performing |
26 | -the analysis on and returns a list of dictionaries that can be consumed for other reporting needs. |
27 | - |
28 | -The 'boot' action prints out kernel related timestamps that are not included in any of the |
29 | -cloud-init logs. There are three different timestamps that are presented to the user: |
30 | -kernel start, kernel finish boot, and cloud-init start. This was added for additional |
31 | -clarity into the boot process that cloud-init does not have control over, to aid in debugging of |
32 | -performance issues related to cloudinit startup or tracking regression. |
33 | +.. _analyze: |
34 | + |
35 | +Analyze |
36 | +******* |
37 | + |
38 | +The analyze subcommand was added to cloud-init in order to help analyze |
39 | +cloud-init boot time performance. It is loosely based on systemd-analyze where |
40 | +there are four subcommands: |
41 | + |
42 | +- blame |
43 | +- show |
44 | +- dump |
45 | +- boot |
46 | |
47 | Usage |
48 | ===== |
49 | -Using each of the printing formats is as easy as running one of the following bash commands: |
50 | + |
51 | +The analyze command requires one of the four subcommands: |
52 | |
53 | .. code-block:: shell-session |
54 | |
55 | - cloud-init analyze show |
56 | - cloud-init analyze blame |
57 | - cloud-init analyze dump |
58 | - cloud-init analyze boot |
59 | + $ cloud-init analyze blame |
60 | + $ cloud-init analyze show |
61 | + $ cloud-init analyze dump |
62 | + $ cloud-init analyze boot |
63 | + |
64 | +Availability |
65 | +============ |
66 | + |
67 | +The analyze subcommand is generally available across all distributions with the |
68 | +exception of Gentoo and FreeBSD. |
69 | + |
70 | +Subcommands |
71 | +=========== |
72 | + |
73 | +Blame |
74 | +----- |
75 | + |
76 | +The ``blame`` action matches ``systemd-analyze blame`` where it prints, in |
77 | +descending order, the units that took the longest to run. This output is |
78 | +highly useful for examining where cloud-init is spending its time during |
79 | +execution. |
80 | + |
81 | +.. code-block:: shell-session |
82 | + |
83 | + $ cloud-init analyze blame |
84 | + -- Boot Record 01 -- |
85 | + 00.80300s (init-network/config-growpart) |
86 | + 00.64300s (init-network/config-resizefs) |
87 | + 00.62100s (init-network/config-ssh) |
88 | + 00.57300s (modules-config/config-grub-dpkg) |
89 | + 00.40300s (init-local/search-NoCloud) |
90 | + 00.38200s (init-network/config-users-groups) |
91 | + 00.19800s (modules-config/config-apt-configure) |
92 | + 00.03700s (modules-final/config-keys-to-console) |
93 | + 00.02100s (init-network/config-update_etc_hosts) |
94 | + 00.02100s (init-network/check-cache) |
95 | + 00.00800s (modules-final/config-ssh-authkey-fingerprints) |
96 | + 00.00800s (init-network/consume-vendor-data) |
97 | + 00.00600s (modules-config/config-timezone) |
98 | + 00.00500s (modules-final/config-final-message) |
99 | + 00.00400s (init-network/consume-user-data) |
100 | + 00.00400s (init-network/config-mounts) |
101 | + 00.00400s (init-network/config-disk_setup) |
102 | + 00.00400s (init-network/config-bootcmd) |
103 | + 00.00400s (init-network/activate-datasource) |
104 | + 00.00300s (init-network/config-update_hostname) |
105 | + 00.00300s (init-network/config-set_hostname) |
106 | + 00.00200s (modules-final/config-snappy) |
107 | + 00.00200s (init-network/config-rsyslog) |
108 | + 00.00200s (init-network/config-ca-certs) |
109 | + 00.00200s (init-local/check-cache) |
110 | + 00.00100s (modules-final/config-scripts-vendor) |
111 | + 00.00100s (modules-final/config-scripts-per-once) |
112 | + 00.00100s (modules-final/config-salt-minion) |
113 | + 00.00100s (modules-final/config-rightscale_userdata) |
114 | + 00.00100s (modules-final/config-phone-home) |
115 | + 00.00100s (modules-final/config-package-update-upgrade-install) |
116 | + 00.00100s (modules-final/config-fan) |
117 | + 00.00100s (modules-config/config-ubuntu-advantage) |
118 | + 00.00100s (modules-config/config-ssh-import-id) |
119 | + 00.00100s (modules-config/config-snap) |
120 | + 00.00100s (modules-config/config-set-passwords) |
121 | + 00.00100s (modules-config/config-runcmd) |
122 | + 00.00100s (modules-config/config-locale) |
123 | + 00.00100s (modules-config/config-byobu) |
124 | + 00.00100s (modules-config/config-apt-pipelining) |
125 | + 00.00100s (init-network/config-write-files) |
126 | + 00.00100s (init-network/config-seed_random) |
127 | + 00.00100s (init-network/config-migrator) |
128 | + 00.00000s (modules-final/config-ubuntu-drivers) |
129 | + 00.00000s (modules-final/config-scripts-user) |
130 | + 00.00000s (modules-final/config-scripts-per-instance) |
131 | + 00.00000s (modules-final/config-scripts-per-boot) |
132 | + 00.00000s (modules-final/config-puppet) |
133 | + 00.00000s (modules-final/config-power-state-change) |
134 | + 00.00000s (modules-final/config-mcollective) |
135 | + 00.00000s (modules-final/config-lxd) |
136 | + 00.00000s (modules-final/config-landscape) |
137 | + 00.00000s (modules-final/config-chef) |
138 | + 00.00000s (modules-config/config-snap_config) |
139 | + 00.00000s (modules-config/config-ntp) |
140 | + 00.00000s (modules-config/config-emit_upstart) |
141 | + 00.00000s (modules-config/config-disable-ec2-metadata) |
142 | + 00.00000s (init-network/setup-datasource) |
143 | + |
144 | + 1 boot records analyzed |
145 | + |
146 | +Show |
147 | +---- |
148 | + |
149 | +The ``show`` action is similar to ``systemd-analyze critical-chain`` which |
150 | +prints a list of units, the time they started and how long they took. |
151 | +Cloud-init has four stages and within each stage a number of modules may run |
152 | +depending on configuration. ``cloudinit-analyze show`` will, for each boot, |
153 | +print this information and a summary total time, per boot. |
154 | + |
155 | +The following is an abbreviated example of the show output: |
156 | + |
157 | +.. code-block:: shell-session |
158 | + |
159 | + $ cloud-init analyze show |
160 | + -- Boot Record 01 -- |
161 | + The total time elapsed since completing an event is printed after the "@" character. |
162 | + The time the event takes is printed after the "+" character. |
163 | + |
164 | + Starting stage: init-local |
165 | + |``->no cache found @00.01700s +00.00200s |
166 | + |`->found local data from DataSourceNoCloud @00.11000s +00.40300s |
167 | + Finished stage: (init-local) 00.94200 seconds |
168 | + |
169 | + Starting stage: init-network |
170 | + |`->restored from cache with run check: DataSourceNoCloud [seed=/dev/sr0][dsmode=net] @04.79500s +00.02100s |
171 | + |`->setting up datasource @04.88900s +00.00000s |
172 | + |`->reading and applying user-data @04.90100s +00.00400s |
173 | + |`->reading and applying vendor-data @04.90500s +00.00800s |
174 | + |`->activating datasource @04.95200s +00.00400s |
175 | + Finished stage: (init-network) 02.72100 seconds |
176 | + |
177 | + Starting stage: modules-config |
178 | + |`->config-emit_upstart ran successfully @15.43100s +00.00000s |
179 | + |`->config-snap ran successfully @15.43100s +00.00100s |
180 | + ... |
181 | + |`->config-runcmd ran successfully @16.22300s +00.00100s |
182 | + |`->config-byobu ran successfully @16.23400s +00.00100s |
183 | + Finished stage: (modules-config) 00.83500 seconds |
184 | + |
185 | + Starting stage: modules-final |
186 | + |`->config-snappy ran successfully @16.87400s +00.00200s |
187 | + |`->config-package-update-upgrade-install ran successfully @16.87600s +00.00100s |
188 | + ... |
189 | + |`->config-final-message ran successfully @16.93700s +00.00500s |
190 | + |`->config-power-state-change ran successfully @16.94300s +00.00000s |
191 | + Finished stage: (modules-final) 00.10300 seconds |
192 | + |
193 | + Total Time: 4.60100 seconds |
194 | + |
195 | + 1 boot records analyzed |
196 | + |
197 | +If additional boot records are detected then they are printed out from oldest |
198 | +to newest. |
199 | + |
200 | +Dump |
201 | +---- |
202 | + |
203 | +The ``dump`` action simply dumps the cloud-init logs that the analyze module |
204 | +is performing the analysis on and returns a list of dictionaries that can be |
205 | +consumed for other reporting needs. Each element in the list is a boot entry. |
206 | + |
207 | +.. code-block:: shell-session |
208 | + |
209 | + $ cloud-init analyze dump |
210 | + [ |
211 | + { |
212 | + "description": "starting search for local datasources", |
213 | + "event_type": "start", |
214 | + "name": "init-local", |
215 | + "origin": "cloudinit", |
216 | + "timestamp": 1567057578.037 |
217 | + }, |
218 | + { |
219 | + "description": "attempting to read from cache [check]", |
220 | + "event_type": "start", |
221 | + "name": "init-local/check-cache", |
222 | + "origin": "cloudinit", |
223 | + "timestamp": 1567057578.054 |
224 | + }, |
225 | + { |
226 | + "description": "no cache found", |
227 | + "event_type": "finish", |
228 | + "name": "init-local/check-cache", |
229 | + "origin": "cloudinit", |
230 | + "result": "SUCCESS", |
231 | + "timestamp": 1567057578.056 |
232 | + }, |
233 | + { |
234 | + "description": "searching for local data from DataSourceNoCloud", |
235 | + "event_type": "start", |
236 | + "name": "init-local/search-NoCloud", |
237 | + "origin": "cloudinit", |
238 | + "timestamp": 1567057578.147 |
239 | + }, |
240 | + { |
241 | + "description": "found local data from DataSourceNoCloud", |
242 | + "event_type": "finish", |
243 | + "name": "init-local/search-NoCloud", |
244 | + "origin": "cloudinit", |
245 | + "result": "SUCCESS", |
246 | + "timestamp": 1567057578.55 |
247 | + }, |
248 | + { |
249 | + "description": "searching for local datasources", |
250 | + "event_type": "finish", |
251 | + "name": "init-local", |
252 | + "origin": "cloudinit", |
253 | + "result": "SUCCESS", |
254 | + "timestamp": 1567057578.979 |
255 | + }, |
256 | + { |
257 | + "description": "searching for network datasources", |
258 | + "event_type": "start", |
259 | + "name": "init-network", |
260 | + "origin": "cloudinit", |
261 | + "timestamp": 1567057582.814 |
262 | + }, |
263 | + { |
264 | + "description": "attempting to read from cache [trust]", |
265 | + "event_type": "start", |
266 | + "name": "init-network/check-cache", |
267 | + "origin": "cloudinit", |
268 | + "timestamp": 1567057582.832 |
269 | + }, |
270 | + ... |
271 | + { |
272 | + "description": "config-power-state-change ran successfully", |
273 | + "event_type": "finish", |
274 | + "name": "modules-final/config-power-state-change", |
275 | + "origin": "cloudinit", |
276 | + "result": "SUCCESS", |
277 | + "timestamp": 1567057594.98 |
278 | + }, |
279 | + { |
280 | + "description": "running modules for final", |
281 | + "event_type": "finish", |
282 | + "name": "modules-final", |
283 | + "origin": "cloudinit", |
284 | + "result": "SUCCESS", |
285 | + "timestamp": 1567057594.982 |
286 | + } |
287 | + ] |
288 | + |
289 | + |
290 | +Boot |
291 | +---- |
292 | + |
293 | +The ``boot`` action prints out kernel related timestamps that are not included |
294 | +in any of the cloud-init logs. There are three different timestamps that are |
295 | +presented to the user: |
296 | + |
297 | +- kernel start |
298 | +- kernel finish boot |
299 | +- cloud-init start |
300 | + |
301 | +This was added for additional clarity into the boot process that cloud-init |
302 | +does not have control over, to aid in debugging of performance issues related |
303 | +to cloud-init startup, and tracking regression. |
304 | + |
305 | +.. code-block:: shell-session |
306 | + |
307 | + $ cloud-init analyze boot |
308 | + -- Most Recent Boot Record -- |
309 | + Kernel Started at: 2019-08-29 01:35:37.753790 |
310 | + Kernel ended boot at: 2019-08-29 01:35:38.807407 |
311 | + Kernel time to boot (seconds): 1.053617000579834 |
312 | + Cloud-init activated by systemd at: 2019-08-29 01:35:43.992460 |
313 | + Time between Kernel end boot and Cloud-init activation (seconds): 5.185053110122681 |
314 | + Cloud-init start: 2019-08-29 08:35:45.867000 |
315 | + successful |
316 | + |
317 | +Timestamp Gathering |
318 | +^^^^^^^^^^^^^^^^^^^ |
319 | + |
320 | +The following boot related timestamps are gathered on demand when cloud-init |
321 | +analyze boot runs: |
322 | |
323 | -Cloud-init analyze boot Timestamp Gathering |
324 | -=========================================== |
325 | -The following boot related timestamps are gathered on demand when cloud-init analyze boot runs: |
326 | -- Kernel Startup, which is inferred from system uptime |
327 | -- Kernel Finishes Initialization, which is inferred from systemd UserSpaceMonotonicTimestamp property |
328 | -- Cloud-init activation, which is inferred from the property InactiveExitTimestamp of the cloud-init |
329 | -local systemd unit. |
330 | +- Kernel startup gathered from system uptime |
331 | +- Kernel finishes initialization from systemd |
332 | + UserSpaceMonotonicTimestamp property |
333 | +- Cloud-init activation from the property InactiveExitTimestamp of the |
334 | + cloud-init local systemd unit |
335 | |
336 | -In order to gather the necessary timestamps using systemd, running the commands |
337 | +In order to gather the necessary timestamps using systemd, running the |
338 | +commands below will gather the UserspaceTimestamp and InactiveExitTimestamp: |
339 | |
340 | .. code-block:: shell-session |
341 | |
342 | - systemctl show -p UserspaceTimestampMonotonic |
343 | - systemctl show cloud-init-local -p InactiveExitTimestampMonotonic |
344 | + $ systemctl show -p UserspaceTimestampMonotonic |
345 | + UserspaceTimestampMonotonic=989279 |
346 | + $ systemctl show cloud-init-local -p InactiveExitTimestampMonotonic |
347 | + InactiveExitTimestampMonotonic=4493126 |
348 | |
349 | -will gather the UserspaceTimestamp and InactiveExitTimestamp. |
350 | -The UserspaceTimestamp tracks when the init system starts, which is used as an indicator of kernel |
351 | -finishing initialization. The InactiveExitTimestamp tracks when a particular systemd unit transitions |
352 | -from the Inactive to Active state, which can be used to mark the beginning of systemd's activation |
353 | -of cloud-init. |
354 | +The UserspaceTimestamp tracks when the init system starts, which is used as |
355 | +an indicator of kernel finishing initialization. The InactiveExitTimestamp |
356 | +tracks when a particular systemd unit transitions from the Inactive to Active |
357 | +state, which can be used to mark the beginning of systemd's activation of |
358 | +cloud-init. |
359 | |
360 | -Currently this only works for distros that use systemd as the init process. We will be expanding |
361 | -support for other distros in the future and this document will be updated accordingly. |
362 | +Currently this only works for distros that use systemd as the init process. |
363 | +We will be expanding support for other distros in the future and this document |
364 | +will be updated accordingly. |
365 | |
366 | -If systemd is not present on the system, dmesg is used to attempt to find an event that logs the |
367 | -beginning of the init system. However, with this method only the first two timestamps are able to be found; |
368 | -dmesg does not monitor userspace processes, so no cloud-init start timestamps are emitted like when |
369 | +If systemd is not present on the system, dmesg is used to attempt to find an |
370 | +event that logs the beginning of the init system. However, with this method |
371 | +only the first two timestamps are able to be found; dmesg does not monitor |
372 | +userspace processes, so no cloud-init start timestamps are emitted like when |
373 | using systemd. |
374 | |
375 | -List of Cloud-init analyze boot supported distros |
376 | -================================================= |
377 | -- Arch |
378 | -- CentOS |
379 | -- Debian |
380 | -- Fedora |
381 | -- OpenSuSE |
382 | -- Red Hat Enterprise Linux |
383 | -- Ubuntu |
384 | -- SUSE Linux Enterprise Server |
385 | -- CoreOS |
386 | - |
387 | -List of Cloud-init analyze boot unsupported distros |
388 | -=================================================== |
389 | -- FreeBSD |
390 | -- Gentoo |
391 | \ No newline at end of file |
392 | +.. vi: textwidth=79 |
PASSED: Continuous integration, rev:f2d60b63e8e 9702b22342cfc4d f6966de5d231fd /jenkins. ubuntu. com/server/ job/cloud- init-ci/ 1125/
https:/
Executed test runs:
SUCCESS: Checkout
SUCCESS: Unit & Style Tests
SUCCESS: Ubuntu LTS: Build
SUCCESS: Ubuntu LTS: Integration
IN_PROGRESS: Declarative: Post Actions
Click here to trigger a rebuild: /jenkins. ubuntu. com/server/ job/cloud- init-ci/ 1125//rebuild
https:/