Merge ~vtqanh/cloud-init:UpdateReporterForAnalyze into cloud-init:master

[Anh Vo (MSFT)]

analyze: fix poor formatting due to additional datasource events

DataSourceAzure is emitting additional events into the cloud-init
log which causes analyze module to produce somewhat confusing output.
This is due to two issues: 1) DataSourceAzure does not emit the
stage (e.g., init-local) and analyze expects to see it in the event
output. 2) analyze did not correctly process nested stages.
This change saves the stage name into the reporting module so that
other reporter can use it to know which stage it is in and fixes the
analyze module to process multiple levels of nested events.

LP: #1833731

e48f66c... by Azure CloudInit Build on 2019-07-15

remove unnecessary change

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:e48f66cd2128b1c15472e9634345418ae4fd8364
https://jenkins.ubuntu.com/server/job/cloud-init-ci/780/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/780/rebuild

review: Approve (continuous-integration)

[Dan Watkins (oddbloke)]

I have some concerns about using a global for the stage here; it opens us up to easily breaking reporting on Azure. I _think_ it would be better if we could find a way to get the top-level reporter through to the Azure classes so they can use parent=... instead, though I haven't dealt with the reporting code before so my intuition may be off-base here.

Other than that, we'll also need some tests around the stage changes, so that we don't regress this in future.

review: Needs Fixing

[Ryan Harper (raharper)]

Could you update the Author of the commits to the person instead of the azure-ci lp account?

Unmerged commits

e48f66c... by Azure CloudInit Build on 2019-07-15

remove unnecessary change

2e775f0... by Ryan Harper on 2019-06-24

Update analyze to handle nested events within a config module

c4ab7e9... by Azure CloudInit Build on 2019-07-15

adding global_stage to the right class

21a801d... by Azure CloudInit Build on 2019-07-15

first attempt at properly stacking azure events within analyze

diff --git a/cloudinit/analyze/show.py b/cloudinit/analyze/show.py
index 3e778b8..b15cd2c 100644
--- a/cloudinit/analyze/show.py
+++ b/cloudinit/analyze/show.py
@@ -94,6 +94,10 @@ def event_parent(event):
     return None
 
 
+def event_is_stage(event):
+    return '/' not in event_name(event)
+
+
 def event_timestamp(event):
     return float(event.get('timestamp'))
 
@@ -146,7 +150,9 @@ def generate_records(events, blame_sort=False,
             next_evt = None
 
         if event_type(event) == 'start':
-            if event.get('name') in stages_seen:
+            stage_name = event_parent(event)
+            if stage_name == event_name(event) and stage_name in stages_seen:
+                # new boot record
                 records.append(total_time_record(total_time))
                 boot_records.append(records)
                 records = []
@@ -166,19 +172,26 @@ def generate_records(events, blame_sort=False,
                                                               event,
                                                               next_evt)))
             else:
-                # This is a parent event
-                records.append("Starting stage: %s" % event.get('name'))
-                unprocessed.append(event)
-                stages_seen.append(event.get('name'))
-                continue
+                if event_is_stage(event):
+                    records.append("Starting stage: %s" % event.get('name'))
+                    unprocessed.append(event)
+                    stages_seen.append(event.get('name'))
+                else:
+                    # Start of a substage event
+                    records.append(format_record(print_format,
+                                                 event_record(start_time,
+                                                              event,
+                                                              next_evt)))
+
         else:
             prev_evt = unprocessed.pop()
             if event_name(event) == event_name(prev_evt):
-                record = event_record(start_time, prev_evt, event)
-                records.append(format_record("Finished stage: "
-                                             "(%n) %d seconds ",
-                                             record) + "\n")
-                total_time += record.get('delta')
+                if event_is_stage(event):
+                    record = event_record(start_time, prev_evt, event)
+                    records.append(format_record("Finished stage: "
+                                                 "(%n) %d seconds ",
+                                                 record) + "\n")
+                    total_time += record.get('delta')
             else:
                 # not a match, put it back
                 unprocessed.append(prev_evt)
diff --git a/cloudinit/cmd/main.py b/cloudinit/cmd/main.py
index a5446da..bcac69e 100644
--- a/cloudinit/cmd/main.py
+++ b/cloudinit/cmd/main.py
@@ -885,7 +885,7 @@ def main(sysv_args=None):
         report_on = False
 
     args.reporter = events.ReportEventStack(
-        rname, rdesc, reporting_enabled=report_on)
+        rname, rdesc, reporting_enabled=report_on, global_stage=rname)
 
     with args.reporter:
         retval = util.log_time(
diff --git a/cloudinit/reporting/events.py b/cloudinit/reporting/events.py
index e5dfab3..a750ba3 100644
--- a/cloudinit/reporting/events.py
+++ b/cloudinit/reporting/events.py
@@ -28,6 +28,7 @@ class _nameset(set):
 
 
 status = _nameset(("SUCCESS", "WARN", "FAIL"))
+reporting_stage = None
 
 
 class ReportingEvent(object):
@@ -153,7 +154,7 @@ class ReportEventStack(object):
     """
     def __init__(self, name, description, message=None, parent=None,
                  reporting_enabled=None, result_on_exception=status.FAIL,
-                 post_files=None):
+                 post_files=None, global_stage=None):
         self.parent = parent
         self.name = name
         self.description = description
@@ -178,6 +179,10 @@ class ReportEventStack(object):
             self.fullname = self.name
         self.children = {}
 
+        global reporting_stage
+        if global_stage is not None:
+            reporting_stage = global_stage
+
     def __repr__(self):
         return ("ReportEventStack(%s, %s, reporting_enabled=%s)" %
                 (self.name, self.description, self.reporting_enabled))
diff --git a/cloudinit/sources/helpers/azure.py b/cloudinit/sources/helpers/azure.py
index 82c4c8c..b862d58 100755
--- a/cloudinit/sources/helpers/azure.py
+++ b/cloudinit/sources/helpers/azure.py
@@ -25,7 +25,7 @@ LOG = logging.getLogger(__name__)
 DEFAULT_WIRESERVER_ENDPOINT = "a8:3f:81:10"
 
 azure_ds_reporter = events.ReportEventStack(
-    name="azure-ds",
+    name="%s/azure-ds" % events.reporting_stage,
     description="initialize reporter for azure ds",
     reporting_enabled=True)