Merge lp:~sinzui/juju-release-tools/validate-streams into lp:juju-release-tools

[Curtis Hovey]

This branch adds a script that can compare the product items in two streams files and verify that the expected changes exist, that there are no missing or extra items, and that the common items are the same. The script ensures that non-numeric version cannot be in the proposed and release json

Nothing uses this script at this time.

The command line is purpose, new-version old-json, new-json. I compared the release to the proposed json like this to verify that 1.20.10 is the only change between them
    ./juju-release-tools/validate_streams.py proposed 1.20.10 old/com.ubuntu.juju\:released\:tools.json new/com.ubuntu.juju\:released\:tools.json

Comparing old with itself, but expecting 1.20.10 change is an error because the tools are missing
    ./juju-release-tools/validate_streams.py proposed 1.20.10 old/com.ubuntu.juju\:released\:tools.json old/com.ubuntu.juju\:released\:tools.json

I will use this script in assemble-streams in the future.

[Aaron Bentley (abentley)]

I am finding it very hard to understand check_expected_tools. I don't know why it will accept version and then basically ignore it if retracted is set. It seems like retracted is really a mode, not a parameter. I would like to see a test like this:

old_tools = make_tools_data('trusty', 'amd64', ['1.20.7', '1.20.9'])
new_tools = make_tools_data('trusty', 'amd64', ['1.20.7', '1.20.8'])
check_expected_tools(old_tools, new_tools, '1.20.8', '1.20.9')

Notice that 1.20.8 has been added and 1.20.9 has been removed. I don't know what the expected behaviour is. I think the actual behaviour is that it will complain that 1.20.8 is unexpected.

review: Needs Information

[Curtis Hovey (sinzui)]

Hi Aaron. Thank you for the review. your question has raised a scenario I had not considered.

Retract is like a mode. We are removing a set instead of adding one. The version being published is likely to be the previous version. The version is always passed because Jerf will always be installing that version to create the streams. I assumed that the version is the previous good version.

I don't think your example is real. I cannot think of retracting a later version, and publishing an earlier non-existent version, but...

I can image waking up and find an email from Ian that says there is a security issue with version n, his team have prepared version q. We need to retract earlier n and since something is ready, publish later q. I would certainly want to do one publication, not two to achieve this. I am certain this test would fail because there are two expected differences.

old_tools = make_tools_data('trusty', 'amd64', ['1.20.7', '1.20.8'])
new_tools = make_tools_data('trusty', 'amd64', ['1.20.7', '1.20.9'])
check_expected_tools(old_tools, new_tools, '1.20.9', '1.20.8')

I am going to change the rules to accommodate this case. This is tricky I think the rule is that when doing a retractions, we need to ask if the version being published is new and need to be included in the expected differences.

[lp:~sinzui/juju-release-tools/validate-streams]

103. By Curtis Hovey on 2014-10-17

Added can be done because we want to be clear about when we add a new version
and retract an old version.

104. By Curtis Hovey on 2014-10-17

Clearly state the content is correct when using verbose.

[Aaron Bentley (abentley)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 14-10-17 01:58 PM, Curtis Hovey wrote:
> I am going to change the rules to accommodate this case. This is
> tricky I think the rule is that when doing a retractions, we need
> to ask if the version being published is new and need to be
> included in the expected differences.

I don't think it's actually that tricky. I think it's something like

added = dict(t for t in new_tools.items() if t[0] not in old_tools)
wrongly_added = dict(t for t in added.items()
                     if t[1][version] != new_version)
removed = dict(t for t in old_tools.items() if t[0] not in new_tools)
if retracted is None:
    wrongly_removed = removed
else:
    wrongly_removed = dict(t for t in added.items()
                           if t[1][version] != retracted)

Aaron
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBAgAGBQJUQWVfAAoJEK84cMOcf+9hk5QIAIk9rRS3YnrPjwSrhsRHB8n9
FXgZPXTi/Z61AbG3nrbESbpHE8R8FVtM5dQL7chVbhUSi52oFl2CVVS4wBTfX+/r
TYViz/bJQOTuAAgOwtRIMZ/ylHSdgjgVcbI0AaOguP5KevuERWxR3mXH6pusdX5+
xeaYtCTkkgK2jfcBNdPW54o97T/MnCh57iRKOFmgWCm5QMktizGkyPAvgpCBwhmx
vaPwjqnFjZUIuN3zWLFL7Gn6YS9QHWWbDAvjKp2XqyoDAuKVk3x+QH5RhFkuQlfB
luf56rgHhj/YoCbkBrj4dB6gTcv2pmFW1mHdontpKJvn2bgcOXK5tKIpYVqQBuw=
=Sd2G
-----END PGP SIGNATURE-----

[lp:~sinzui/juju-release-tools/validate-streams]

105. By Curtis Hovey on 2014-10-17

added is optional. use --added to clearly state a version was added.

[Curtis Hovey (sinzui)]

I have a subtle change to the script. Instead of passing the args that might be passed to assemble-streams, the script now expects the caller to clearly state the expected differences.

I changed the script command line to clearly indicate when a version is added:
   --added 1.20.9 --retracted 1.20.8

Which also renamed version => added

I have changed added to be optional so that the function does not need to guess about expected differences
    check_expected_tools(old_tools, new_tools, added=None, retracted=None)

The assemble-streams script will need intelligence to know when something is added and retracted. This might not be tricky since it is configured with the reacted version and there is a check if any new tools were created.

Note I made a change to verbose. I used this script for the 1.20.10 release, and I had to check the return code because --verbose didn't clearly tell me that everything was fine. I really wanted a clear message that the content was correct.

[lp:~sinzui/juju-release-tools/validate-streams]

106. By Curtis Hovey on 2014-10-17

Extracted temp_dir() to tests.utils

[Curtis Hovey (sinzui)]

Oh, and I am sure you noted that temp_dir() could be extracted to a utils.py. I have done this to keep the code DRY.

[Aaron Bentley (abentley)]

Okay, I think I'm getting the hang of this code, and I'm pretty sure it is not correct. There are two ways check_expected_tools can be tricked out of warning that "added" is missing.

1. Do a successful retract. This means that expected_differences will be non-empty from the retract clause, and so the added clause cannot set missing_errors.

2. Have other missing versions. This means that the "added" clause will set missing_errors, but the "missing" clause will overwrite its value. There will be an error, but it won't mention "added".

Other comments below.

review: Needs Fixing

[lp:~sinzui/juju-release-tools/validate-streams]

107. By Curtis Hovey on 2014-10-20

Merged tip and resolved conflicts.

108. By Curtis Hovey on 2014-10-20

Read the file directly into the stream.

109. By Curtis Hovey on 2014-10-20

Do not overwrite missing_errors.

[Curtis Hovey (sinzui)]

I have replied to the comments and pushed up some changes. I have another change I need to make, at least a test, and maybe a code change to account for streams.canonical.com's behaviour.

[Aaron Bentley (abentley)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 14-10-20 05:55 PM, Curtis Hovey wrote:
> The "versions" keys are arbitrary names :(. juju metadata will
> name it something like "20141020". We only get 1 key in versions,
> but the specification allows many. We don't care if there is 1 or
> many of these versions we only want to get the the next level,
> "items". The goal is to say all items on old are the same as the
> items in new, except we know that we sometimes add items and we
> have one removed items, once published without items, and on a few
> occasions change items.
>
> In Xpath, this is products/versions/*/items. I could change the
> test for items to be if isinstance(version, dict) and "items" in
> versions:

Okay, the code is fine as it stands, then.

>> + # needs a version to install to make streams, even when
>> it + # intends to remove something. + for n, t in
>> old_expected.items(): + if t['version'] == retracted:
>> + expected_differences.update([(n, t)])
>
> The goal was to state the expected differences to create a common
> set of old and new tools that must match in name a content.

Deleting entries from old_expected is perfectly sane. But why are
updating expected_differences? You don't use it anywhere, except the
"added" clause. And in the "added" clause, it can only cause a bug,
by falsely preventing the "missing.append(added)".

> So expected_differences was just a way to remove what we know to
> be different to prove nothing else is changed.

No, it doesn't do anything like that. All it does is prevent
"missing.append(added)", so that this version is not considered missing.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQEcBAEBAgAGBQJURnGpAAoJEK84cMOcf+9hwDkIAJmJOGytVsStF5De6MijLpV5
4G6YNYWhzp+ShM38KIhT39CZ83pJV4DY7yO3ajIi03/359j1rufRug4CxukxPGwq
ByQO+/SuyvKZYpHFRVqmpsCe1kaKX4WC8gInLm2JLgFoMr/zXDWAL9jnZ8TMmEa7
V0a9T/jNit/3q9t4hMEoaNohaAaWj3vxbNWdm/gasmtApMiqtLltHD1SNEm8czau
py1qmtidjNd4R4q+TTjROmo5uJqOR0jPBOe1MoYdzzUY4F/1QgqWlCEo6TEVTseU
rSuaU1aVG0ks63Et+S8/GcBuqGOjT5DIjHDIt1LYVPa4xZYuBkHuP6loJmEyOZ4=
=GrUS
-----END PGP SIGNATURE-----

[lp:~sinzui/juju-release-tools/validate-streams]

110. By Curtis Hovey on 2014-10-21

Merged tip, Removed IGNORE.

111. By Curtis Hovey on 2014-10-21

Renamed retracted => removed.

112. By Curtis Hovey on 2014-10-21

expected_differences is not needed.

113. By Curtis Hovey on 2014-10-22

Make code easier to maintain, though a little redudant, by creating smaller
functions with clear purpose.

114. By Curtis Hovey on 2014-10-22

Revise docstrings.

[Curtis Hovey (sinzui)]

My latest changes follow your suggestion to simplify the purpose of the functions to make the code easier to maintain. I split the rules to check for expected versions from rules to find unexpected version.

I also changed all the checks to always return a list to make the contact simple and it is easy for the callee to extend the list of errors.

[Aaron Bentley (abentley)]

This looks clear and correct. Thanks!

review: Approve

=== modified file 'tests/test_make_release_notes.py'
--- tests/test_make_release_notes.py	2014-10-20 18:43:54 +0000
+++ tests/test_make_release_notes.py	2014-10-22 21:03:40 +0000
@@ -3,7 +3,7 @@
 from StringIO import StringIO
 from unittest import TestCase
 
-
+from utils import temp_dir
 from make_release_notes import (
     DEVEL,
     get_bugs,

=== added file 'tests/test_validate_streams.py'
--- tests/test_validate_streams.py	1970-01-01 00:00:00 +0000
+++ tests/test_validate_streams.py	2014-10-22 21:03:40 +0000
@@ -0,0 +1,234 @@
+from mock import patch
+import json
+from unittest import TestCase
+
+from utils import temp_dir
+from validate_streams import (
+    check_devel_not_stable,
+    check_expected_changes,
+    check_expected_unchanged,
+    check_agents_content,
+    compare_agents,
+    find_agents,
+    parse_args,
+)
+
+
+def make_tool_data(version='1.20.7', release='trusty', arch='amd64'):
+    name = '{}-{}-{}'.format(version, release, arch)
+    tool = {
+        "release": "{}".format(release),
+        "version": "{}".format(version),
+        "arch": "{}".format(arch),
+        "size": 8234578,
+        "path": "releases/juju-{}.tgz".format(name),
+        "ftype": "tar.gz",
+        "sha256": "valid_sum"
+    }
+    return name, tool
+
+
+def make_agents_data(release='trusty', arch='amd64', versions=['1.20.7']):
+    return dict(make_tool_data(v, release, arch) for v in versions)
+
+
+def make_product_data(release='trusty', arch='amd64', versions=['1.20.7']):
+    name = 'com.ubuntu.juju:{}:{}'.format(release, arch)
+    items = make_agents_data(release, arch, versions)
+    product = {
+        "version": "{}".format(versions[0]),
+        "arch": "{}".format(arch),
+        "versions": {
+            "20140919": {
+                "items": items
+            }
+        }
+    }
+    return name, product
+
+
+def make_products_data(versions):
+    products = {}
+    for release, arch in (('trusty', 'amd64'), ('trusty', 'i386')):
+        name, product = make_product_data(release, arch, versions)
+        products[name] = product
+    stream = {
+        "products": products,
+        "updated": "Fri, 19 Sep 2014 13:25:28 -0400",
+        "format": "products:1.0",
+        "content_id": "com.ubuntu.juju:released:agents"
+    }
+    return stream
+
+
+class ValidateStreams(TestCase):
+
+    def test_parge_args(self):
+        # The purpose, release, old json and new json are required.
+        required = ['--added', '1.20.9', 'proposed', 'old/json', 'new/json']
+        args = parse_args(required)
+        self.assertEqual('proposed', args.purpose)
+        self.assertEqual('old/json', args.old_json)
+        self.assertEqual('new/json', args.new_json)
+        self.assertEqual('1.20.9', args.added)
+        # A bad release version can be removed.
+        args = parse_args(['--removed', 'bad'] + required)
+        self.assertEqual('bad', args.removed)
+
+    def test_find_agents(self):
+        products = make_products_data(['1.20.7', '1.20.8'])
+        with temp_dir() as wd:
+            file_path = '{}/json'.format(wd)
+            with open(file_path, 'w') as f:
+                f.write(json.dumps(products))
+            agents = find_agents(file_path)
+        expected = [
+            '1.20.7-trusty-i386', '1.20.7-trusty-amd64',
+            '1.20.8-trusty-amd64', '1.20.8-trusty-i386']
+        self.assertEqual(expected, agents.keys())
+
+    def test_check_devel_not_stable(self):
+        # devel agents cannot ever got to proposed and release.
+        old_agents = make_agents_data('trusty', 'amd64', ['1.20.7', '1.20.8'])
+        new_agents = make_agents_data(
+            'trusty', 'amd64', ['1.20.7', '1.20.8', '1.21-alpha1'])
+        # Devel versions can go to testing
+        message = check_devel_not_stable(old_agents, new_agents, 'testing')
+        self.assertEqual([], message)
+        # Devel versions can go to devel
+        message = check_devel_not_stable(old_agents, new_agents, 'devel')
+        self.assertEqual([], message)
+        # Devel versions cannot be proposed.
+        message = check_devel_not_stable(old_agents, new_agents, 'proposed')
+        self.assertEqual(
+            ["Devel versions in proposed stream: "
+             "['1.21-alpha1-trusty-amd64']"],
+            message)
+        # Devel versions cannot be release.
+        message = check_devel_not_stable(old_agents, new_agents, 'release')
+        self.assertEqual(
+            ["Devel versions in release stream: ['1.21-alpha1-trusty-amd64']"],
+            message)
+
+    def test_check_agents_content(self):
+        old_agents = make_agents_data('trusty', 'amd64', ['1.20.7', '1.20.8'])
+        new_agents = make_agents_data(
+            'trusty', 'amd64', ['1.20.7', '1.20.8'])
+        new_agents['1.20.7-trusty-amd64']['sha256'] = 'bad_sum'
+        message = check_agents_content(old_agents, new_agents)
+        self.assertEqual(
+            (['Tool 1.20.7-trusty-amd64 sha256 changed from '
+              'valid_sum to bad_sum']),
+            message)
+
+    def test_compare_agents_identical(self):
+        old_agents = make_agents_data('trusty', 'amd64', ['1.20.7', '1.20.8'])
+        new_agents = make_agents_data('trusty', 'amd64', ['1.20.7', '1.20.8'])
+        message = compare_agents(
+            old_agents, new_agents, 'proposed', added=None, removed=None)
+        self.assertIs(None, message)
+
+    def test_check_expected_changes_with_no_changes(self):
+        new_agents = make_agents_data('trusty', 'amd64', ['1.20.7', '1.20.8'])
+        errors = check_expected_changes(new_agents, added=None, removed=None)
+        self.assertEqual([], errors)
+
+    def test_check_expected_changes_with_changes(self):
+        new_agents = make_agents_data('trusty', 'amd64', ['1.20.7', '1.20.9'])
+        errors = check_expected_changes(
+            new_agents, added='1.20.9', removed='1.20.8')
+        self.assertEqual([], errors)
+
+    def test_check_expected_changes_with_found_errors(self):
+        new_agents = make_agents_data('trusty', 'amd64', ['1.20.7', '1.20.8'])
+        errors = check_expected_changes(
+            new_agents, added=None, removed='1.20.8')
+        self.assertEqual(
+            ["1.20.8 agents were not removed: ['1.20.8-trusty-amd64']"],
+            errors)
+
+    def test_check_expected_changes_with_missing_errors(self):
+        new_agents = make_agents_data('trusty', 'amd64', ['1.20.7'])
+        errors = check_expected_changes(
+            new_agents, added='1.20.8', removed=None)
+        self.assertEqual(['1.20.8 agents were not added'], errors)
+
+    def test_check_expected_unchanged_without_changes(self):
+        old_agents = make_agents_data('trusty', 'amd64', ['1.20.7', '1.20.8'])
+        new_agents = make_agents_data('trusty', 'amd64', ['1.20.7', '1.20.8'])
+        errors = check_expected_unchanged(
+            old_agents, new_agents, added=None, removed=None)
+        self.assertEqual([], errors)
+
+    def test_check_expected_unchanged_without_changes_and_added_removed(self):
+        old_agents = make_agents_data('trusty', 'amd64', ['1.20.7', '1.20.8'])
+        new_agents = make_agents_data('trusty', 'amd64', ['1.20.7', '1.20.9'])
+        errors = check_expected_unchanged(
+            old_agents, new_agents, added='1.20.9', removed='1.20.8')
+        self.assertEqual([], errors)
+
+    def test_check_expected_unchanged_with_missing_errors(self):
+        old_agents = make_agents_data('trusty', 'amd64', ['1.20.7', '1.20.8'])
+        new_agents = make_agents_data('trusty', 'amd64', ['1.20.7', '1.20.9'])
+        errors = check_expected_unchanged(
+            old_agents, new_agents, added='1.20.9', removed=None)
+        self.assertEqual(
+            ["These agents are missing: ['1.20.8-trusty-amd64']"],
+            errors)
+
+    def test_check_expected_unchanged_with_found_errors(self):
+        old_agents = make_agents_data('trusty', 'amd64', ['1.20.7'])
+        new_agents = make_agents_data('trusty', 'amd64', ['1.20.7', '1.20.9'])
+        errors = check_expected_unchanged(
+            old_agents, new_agents, added=None, removed=None)
+        self.assertEqual(
+            ["These unknown agents were found: ['1.20.9-trusty-amd64']"],
+            errors)
+
+    def test_compare_agents_changed_tool(self):
+        old_agents = make_agents_data('trusty', 'amd64', ['1.20.7', '1.20.8'])
+        new_agents = make_agents_data(
+            'trusty', 'amd64', ['1.20.7', '1.20.8', '1.20.9'])
+        new_agents['1.20.7-trusty-amd64']['sha256'] = 'bad_sum'
+        errors = compare_agents(
+            old_agents, new_agents, 'proposed', '1.20.9', removed=None)
+        self.assertEqual(
+            ['Tool 1.20.7-trusty-amd64 sha256 changed from '
+             'valid_sum to bad_sum'],
+            errors)
+
+    def test_compare_agents_called_check_expected_agents_called(self):
+        old_agents = make_agents_data('trusty', 'amd64', ['1.20.7'])
+        new_agents = make_agents_data('trusty', 'amd64', ['1.20.7', '1.20.8'])
+        with patch("validate_streams.check_expected_changes",
+                   return_value=(['foo'])) as cec_mock:
+            with patch("validate_streams.check_expected_unchanged",
+                       return_value=(['bar'])) as ceu_mock:
+                errors = compare_agents(
+                    old_agents, new_agents, 'proposed', '1.20.9', removed=None)
+                cec_mock.assert_called_with(new_agents, '1.20.9', None)
+                ceu_mock.assert_called_with(
+                    old_agents, new_agents, '1.20.9', None)
+        self.assertEqual(['foo', 'bar'], errors)
+
+    def test_compare_agents_added_devel_version(self):
+        # devel agents cannot ever got to proposed and release.
+        old_agents = make_agents_data('trusty', 'amd64', ['1.20.7', '1.20.8'])
+        new_agents = make_agents_data(
+            'trusty', 'amd64', ['1.20.7', '1.20.8', '1.21-alpha1'])
+        # Devel versions can go to devel
+        message = compare_agents(
+            old_agents, new_agents, 'devel', '1.21-alpha1', removed=None)
+        self.assertIs(None, message)
+        # Devel versions cannot be proposed.
+        errors = compare_agents(
+            old_agents, new_agents, 'proposed', '1.21-alpha1', removed=None)
+        expected = (
+            "Devel versions in proposed stream: ['1.21-alpha1-trusty-amd64']")
+        self.assertEqual([expected], errors)
+        # Devel versions cannot be release.
+        errors = compare_agents(
+            old_agents, new_agents, 'release', '1.21-alpha1', removed=None)
+        expected = (
+            "Devel versions in release stream: ['1.21-alpha1-trusty-amd64']")
+        self.assertEqual([expected], errors)

=== added file 'validate_streams.py'
--- validate_streams.py	1970-01-01 00:00:00 +0000
+++ validate_streams.py	2014-10-22 21:03:40 +0000
@@ -0,0 +1,203 @@
+#!/usr/bin/python
+
+from __future__ import print_function
+
+from argparse import ArgumentParser
+import json
+import re
+import sys
+import traceback
+
+
+RELEASE = 'release'
+PROPOSED = 'proposed'
+DEVEL = 'devel'
+TESTING = 'testing'
+PURPOSES = (RELEASE, PROPOSED, DEVEL, TESTING)
+
+
+def find_agents(file_path):
+    with open(file_path) as f:
+        stream = json.load(f)
+    agents = {}
+    for name, product in stream['products'].items():
+        versions = product['versions']
+        for version in versions.values():
+            if isinstance(version, dict):
+                items = version['items']
+                agents.update(items)
+    return agents
+
+
+def check_devel_not_stable(old_agents, new_agents, purpose):
+    """Return a list of errors if the version can be included in the stream.
+
+    Devel versions cannot be proposed or release because the letters in
+    the version break older jujus.
+
+    :param old_agents: the dict of all the products/versions/*/items
+                       in the old json.
+    :param new_agents: the dict of all the products/versions/*/items
+                       in the new json.
+    :param purpose: either release, proposed, devel, or testing.
+    :return: a list of errors, which will be empty when there are none.
+    """
+    if purpose in (TESTING, DEVEL):
+        return []
+    stable_pattern = re.compile(r'\d+\.\d+\.\d+-*')
+    devel_versions = [
+        v for v in new_agents.keys() if not stable_pattern.match(v)]
+    errors = []
+    if devel_versions:
+        errors.append(
+            'Devel versions in {} stream: {}'.format(purpose, devel_versions))
+    return errors
+
+
+def check_expected_changes(new_agents, added=None, removed=None):
+    """Return an list of errors if the expected changes are not present.
+
+    :param new_agents: the dict of all the products/versions/*/items
+                       in the new json.
+    :param added: the version added to the new json, eg '1.20.9'.
+    :param removed: the version removed from the new json, eg '1.20.8'.
+    :return: a list of errors, which will be empty when there are none.
+    """
+    found = []
+    seen = False
+    for n, t in new_agents.items():
+        if removed and t['version'] == removed:
+            found.append(n)
+        elif added and t['version'] == added:
+            seen = True
+    errors = []
+    if added and not seen:
+        errors.append('{} agents were not added'.format(added))
+    if found:
+        errors.append('{} agents were not removed: {}'.format(removed, found))
+    return errors
+
+
+def check_expected_unchanged(old_agents, new_agents, added=None, removed=None):
+    """Return a list of errors if the expected unchanged versions do not match.
+
+    :param old_agents: the dict of all the products/versions/*/items
+                       in the old json.
+    :param new_agents: the dict of all the products/versions/*/items
+                       in the new json.
+    :param added: the version added to the new json, eg '1.20.9'.
+    :param removed: the version removed from the new json, eg '1.20.8'.
+    :return: a list of errors, which will be empty when there are none.
+    """
+    old_versions = set(k for (k, v) in old_agents.items()
+                       if v['version'] != removed)
+    new_versions = set(k for (k, v) in new_agents.items()
+                       if v['version'] != added)
+    missing_errors = old_versions - new_versions
+    errors = []
+    if missing_errors:
+        missing_errors = list(missing_errors)
+        errors.append('These agents are missing: {}'.format(missing_errors))
+    found_errors = new_versions - old_versions
+    if found_errors:
+        found_errors = list(found_errors)
+        errors.append('These unknown agents were found: {}'.format(
+            found_errors))
+    return errors
+
+
+def check_agents_content(old_agents, new_agents):
+    """Return a list of error messages if agents content changes.
+
+    Are the old versions identical to the new versions?
+    We care about change values, not new keys in the new tool.
+
+    :param old_agents: the dict of all the products/versions/*/items
+                       in the old json.
+    :param new_agents: the dict of all the products/versions/*/items
+                       in the new json.
+    :return: a list of errors, which will be empty when there are none.
+    """
+    if not new_agents:
+        return None
+    errors = []
+    for name, old_tool in old_agents.items():
+        try:
+            new_tool = new_agents[name]
+        except KeyError:
+            # This is a missing version case reported by check_expected_agents.
+            continue
+        for old_key, old_val in old_tool.items():
+            new_val = new_tool[old_key]
+            if old_val != new_val:
+                errors.append(
+                    'Tool {} {} changed from {} to {}'.format(
+                        name, old_key, old_val, new_val))
+    return errors
+
+
+def compare_agents(old_agents, new_agents, purpose, added=None, removed=None):
+    """Return a list of error messages from all the validation checks.
+
+    :param old_agents: the dict of all the products/versions/*/items
+                       in the old json.
+    :param new_agents: the dict of all the products/versions/*/items
+                       in the new json.
+    :return: a list of errors, which will be empty when there are none.
+    """
+    errors = []
+    errors.extend(
+        check_devel_not_stable(old_agents, new_agents, purpose))
+    errors.extend(
+        check_expected_changes(new_agents, added, removed))
+    errors.extend(
+        check_expected_unchanged(old_agents, new_agents, added, removed))
+    errors.extend(
+        check_agents_content(old_agents, new_agents))
+    return errors or None
+
+
+def parse_args(args=None):
+    """Return the argument parser for this program."""
+    parser = ArgumentParser("Compare old and new stream data.")
+    parser.add_argument(
+        '-v', '--verbose', action="store_true", default=False,
+        help='Increse verbosity.')
+    parser.add_argument(
+        '-r', '--removed', default=None, help='The release version removed')
+    parser.add_argument(
+        '-a', '--added', default=None, help="The release version added")
+    parser.add_argument('purpose', help="<{}>".format(' | '.join(PURPOSES)))
+    parser.add_argument('old_json', help="The old simple streams data file")
+    parser.add_argument('new_json', help="The new simple streams data file")
+    return parser.parse_args(args)
+
+
+def main(argv):
+    """Verify that the new json has all the expected changes.
+
+    An exit code of 1 will have a list of strings explaining the problems.
+    An exit code of 0 is a pass and the explanation is None.
+    """
+    args = parse_args(argv[1:])
+    try:
+        old_agents = find_agents(args.old_json)
+        new_agents = find_agents(args.new_json)
+        errors = compare_agents(
+            old_agents, new_agents, args.purpose, args.version,
+            retracted=args.retracted)
+        if errors:
+            print('\n'.join(errors))
+            return 1
+    except Exception as e:
+        print(e)
+        if args.verbose:
+            traceback.print_tb(sys.exc_info()[2])
+        return 2
+    if args.verbose:
+        print("All changes are correct.")
+    return 0
+
+
+if __name__ == '__main__':
+    sys.exit(main(sys.argv))