Merge lp:~abentley/charmworld/related-charms into lp:~juju-jitsu/charmworld/trunk

[Aaron Bentley]

Implement related charms in API1.

This branch adds related charms to the data returned by the charms, charms/interesting and charm endpoints in API 1.

The data includes the charms and their weight or 'score' in ElasticSearch, so that they can be sorted externally. Score includes consideration of whether they are official charms.

The data also include icon information. That can be removed as part of "Remove charm icon SVGs from API response"

ElasticSearch.related_charms() is added to return the charms related to an interface. API._charm_results provides a convenient way of retriving charms related to a list of charms, and formatting the charms at the same time.

All the endpoints are updated to use _charm_results. For the charms/interesting endpoint, this means ElasticSearch.related_charms is called three times, when technically it needs to be called only once. But optimizing this doesn't seem necessary right now.

Most of the line count comes from reorganizing the tests. I'm not especially happy with the triple-inheritance this provoked, but I'll leave it for now.

[Curtis Hovey (sinzui)]

Hi Aaron.

It seems possible (because a test does not show it is not) that a charm could be listed among it's related charms. Eg apache requires and provides http, and I think its popularity would guarantee it to be listed among the other related charms. Except that is is not an "other" charm. Are we concerned about this?

review: Needs Information (code)

[Curtis Hovey (sinzui)]

We spoke on IRC and we know there are many similar apache charms that could match. This is not ideal. Excluding exact match on name seems sensible. We will explore this issue in a follow up branch. This branch is fine to land.

review: Approve (code)

[Charmworld Lander (charmworld-lander)]

The attempt to merge lp:~abentley/charmworld/related-charms into lp:charmworld failed. Below is the output from the failed tests.

/home/charmworld-merger/bin/update-revno: line 3: /var/lib/jenkins/.canonistack/novarc: Permission denied

=== modified file 'charmworld/search.py'
--- charmworld/search.py	2013-05-03 19:10:32 +0000
+++ charmworld/search.py	2013-05-16 15:59:26 +0000
@@ -57,6 +57,13 @@
         ]
     }
 
+    valid_charms_clauses = [
+        {'missing': {'field': 'store_data.errors'}},
+        # 'missing' ignores objects/dicts, but 'not': {'exists'}
+        # considers them.
+        {'not': {'exists': {'field': 'error'}}}
+    ]
+
     def __init__(self, client, index_name):
         self._client = client
         self.index_name = index_name
@@ -173,12 +180,7 @@
             ]
         }
         if valid_charms_only:
-            and_clause['and'].extend(
-                [{'missing': {'field': 'store_data.errors'}},
-                 # 'missing' ignores objects/dicts, but 'not': {'exists'}
-                 # considers them.
-                 {'not': {'exists': {'field': 'error'}}}]
-            )
+            and_clause['and'].extend(cls.valid_charms_clauses)
         if type_ is not None:
             typeclauses = []
             for value in type_:
@@ -301,6 +303,32 @@
             'matches_human_readable_estimate': '%d' % len(charms),
         }
 
+    def related_charms(self, requires, provides):
+        and_clauses = [{'or': [
+            {'terms': {'i_provides': provides}},
+            {'terms': {'i_requires': requires}},
+        ]}]
+        and_clauses.extend(self.valid_charms_clauses)
+        dsl = {
+            'filtered': {
+                'filter': {
+                    'and': and_clauses
+                }
+            }
+        }
+        dsl = {'query': self._get_official_boost(dsl)}
+        hits = self._unlimited_search(dsl, None)
+        result_p = {}
+        result_r = {}
+        for hit in hits['hits']['hits']:
+            charm = hit['_source']
+            payload = {'data': charm, 'weight': hit['_score']}
+            for interface in provides.intersection(charm['i_provides']):
+                result_p.setdefault(interface, []).append(payload)
+            for interface in requires.intersection(charm['i_requires']):
+                result_r.setdefault(interface, []).append(payload)
+        return result_r, result_p,
+
     def get_aliasable_client(self, name=None):
         """Return a client that can be aliased to this one.
 

=== renamed file 'charmworld/test_models.py' => 'charmworld/tests/test_models.py'
--- charmworld/test_models.py	2013-05-03 15:47:38 +0000
+++ charmworld/tests/test_models.py	2013-05-16 15:59:26 +0000
@@ -193,7 +193,7 @@
         'bname': 'trunk',
     }
     charm_path = join(
-        dirname(__file__),
+        dirname(dirname(__file__)),
         'testing/data/sample_charm'
     )
 

=== modified file 'charmworld/tests/test_search.py'
--- charmworld/tests/test_search.py	2013-05-03 19:04:04 +0000
+++ charmworld/tests/test_search.py	2013-05-16 15:59:26 +0000
@@ -456,6 +456,60 @@
         with self.assertRaises(IndexMissing):
             client.put_mapping()
 
+    def test_related_charms(self):
+        client = self.index_client
+        charm = factory.get_charm_json(provides={'abc': {'interface': 'def'}})
+        client.index_charm(charm)
+        requires, provides = client.related_charms(
+            set(charm['i_provides']), set(charm['i_requires']))
+        self.assertEqual({}, provides)
+        self.assertEqual({}, requires)
+        requires, provides = client.related_charms(set(['def']), set(['def']))
+        self.assertEqual(['def'], provides.keys())
+        self.assertEqual(charm, provides['def'][0]['data'])
+        self.assertIs(float, type(provides['def'][0]['weight']))
+        self.assertEqual(2, len(provides['def'][0]))
+        self.assertEqual({}, requires)
+        charm2 = factory.get_charm_json(
+            provides={
+                'abc': {'interface': 'def'},
+                'mno': {'interface': 'pqr'}},
+            requires={'ghi': {'interface': 'jkl'}})
+        client.index_charm(charm2)
+        requires, provides = client.related_charms(set(['jkl']), set())
+        self.assertEqual({}, provides)
+        self.assertEqual(['jkl'], requires.keys())
+        self.assertEqual(charm2, requires['jkl'][0]['data'])
+        requires, provides = client.related_charms(set(['jkl']), set(['def']))
+        self.assertEqual(['def'], provides.keys())
+        self.assertItemsEqual([charm, charm2],
+                              [payload['data'] for payload in provides['def']])
+        self.assertEqual([charm2],
+                         [payload['data'] for payload in requires['jkl']])
+        requires, provides = client.related_charms(set(),
+                                                   set(['def', 'pqr', 'xyz']))
+        self.assertItemsEqual(['def', 'pqr'], provides.keys())
+        self.assertEqual([charm2], [payload['data']
+                         for payload in provides['pqr']])
+        self.assertItemsEqual([charm, charm2],
+                              [payload['data'] for payload in provides['def']])
+
+    def test_related_charms_official_boost(self):
+        """An official version of a charm rates 10x higher."""
+        charm = factory.get_charm_json(
+            provides={'foo': {'interface': 'telnet'}})
+        official_id = charm['_id']
+        unofficial_id = official_id + '-1'
+        client = self.index_client
+        client.index_charm(charm)
+        charm['owner'] = 'jrandom'
+        charm['_id'] = unofficial_id
+        client.index_charm(charm)
+        requires, provides = client.related_charms(set(), set(['telnet']))
+        by_id = dict((result['data']['_id'], result['weight'])
+                     for result in provides['telnet'])
+        self.assertAlmostEqual(by_id[official_id], by_id[unofficial_id] * 10)
+
 
 def put_mapping(client, properties):
     client._client.put_mapping(

=== modified file 'charmworld/views/api.py'
--- charmworld/views/api.py	2013-05-03 19:02:38 +0000
+++ charmworld/views/api.py	2013-05-16 15:59:26 +0000
@@ -128,8 +128,40 @@
         }
 
     @classmethod
-    def _charm_result(cls, charm):
-        return {'charm': cls._format_charm(charm)}
+    def _charm_result(cls, charm, requires, provides):
+        charm_provides = set(requires).intersection(charm['i_provides'])
+        charm_requires = set(provides).intersection(charm['i_requires'])
+        related = {
+            'requires': dict((key, requires[key]) for key in charm_provides),
+            'provides': dict((key, provides[key]) for key in charm_requires),
+        }
+        return {
+            'charm': cls._format_charm(charm),
+            'metadata': {'related': related}
+        }
+
+    def _related_charms(self, charms):
+        charms_provide = set()
+        charms_require = set()
+        for charm in charms:
+            charms_provide.update(charm['i_provides'])
+            charms_require.update(charm['i_requires'])
+        # Swap provides and requires, so we get the charms which *require*
+        # what these charms provide, and the charms that *provide* what these
+        # charms require.
+        r_requires, r_provides = self.request.index_client.related_charms(
+            charms_provide, charms_require)
+        f_requires = {}
+        for key, value in r_requires.items():
+            f_requires[key] = [
+                self._format_related(payload['data'], payload['weight'])
+                for payload in value]
+        f_provides = {}
+        for key, value in r_provides.items():
+            f_provides[key] = [
+                self._format_related(payload['data'], payload['weight'])
+                for payload in value]
+        return f_requires, f_provides
 
     @classmethod
     def _format_charm(cls, charm):
@@ -189,6 +221,21 @@
             output['icon'] = charm['icon']
         return output
 
+    @classmethod
+    def _format_related(cls, charm, weight):
+        result = {
+            'id': cls._get_api_id(charm),
+            'name': charm['name'],
+            'categories': charm.get('categories', []),
+            'weight': weight,
+        }
+        if 'icon' in charm:
+            result['icon'] = {
+                'body': charm['icon'],
+                'link': '/api/1/' + result['id'] + '/icon'
+            }
+        return result
+
     @staticmethod
     def _extract_charm_id(path):
         """Extract the charm_id from a path.
@@ -254,7 +301,7 @@
             return json_response(
                 404, {'type': 'no_such_charm', 'charm_id': charm_id})
         if trailing is None:
-            return self._charm_result(charm)
+            return self._charm_results([charm])[0]
         elif trailing.startswith('file/'):
             return self._charm_file(charm, trailing)
         elif trailing == ('qa'):
@@ -329,7 +376,12 @@
             return json_response(406, {
                                  'type': 'negative_value',
                                  'parameter': 'limit'})
-        return {'result': [self._charm_result(result) for result in results]}
+        return {'result': self._charm_results(results)}
+
+    def _charm_results(self, charms):
+        requires, provides = self._related_charms(charms)
+        return [self._charm_result(charm, requires, provides)
+                for charm in charms]
 
     def _interesting_charms(self):
         """Generate a JSON structure of interesting charms.
@@ -338,15 +390,14 @@
         """
         popular = self.request.index_client.api_search(
             sort='downloaded', limit=10)
-        new = [self._charm_result(charm) for charm in
-               self.request.index_client.api_search(sort='new', limit=10)]
+        new = self.request.index_client.api_search(sort='new', limit=10)
         featured = self.request.index_client.api_search(
             filters={'is_featured': [True]})
         return {
             'result': {
-                'new': new,
-                'featured': [self._charm_result(charm) for charm in featured],
-                'popular': [self._charm_result(charm) for charm in popular],
+                'new': self._charm_results(new),
+                'featured': self._charm_results(featured),
+                'popular': self._charm_results(popular),
             }
         }
 
@@ -360,9 +411,12 @@
     """
 
     @classmethod
-    def _charm_result(cls, charm):
+    def _charm_result(cls, charm, requires, provides):
         return cls._format_charm(charm)
 
+    def _charm_results(self, charms):
+        return [self._charm_result(charm, {}, {}) for charm in charms]
+
     def _charms(self, limit=None, name=None, series=None, owner=None,
                 provides=None, requires=None, type_=None, provider=None,
                 scope=None, category=None, categories=None, text=None):

=== modified file 'charmworld/views/tests/test_api.py'
--- charmworld/views/tests/test_api.py	2013-05-01 19:20:02 +0000
+++ charmworld/views/tests/test_api.py	2013-05-16 15:59:26 +0000
@@ -1,6 +1,8 @@
 # Copyright 2012, 2013 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
+__metaclass__ = type
+
 from datetime import datetime
 import hashlib
 
@@ -11,6 +13,7 @@
     get_address,
     make_store_url,
 )
+from charmworld.models import CharmSource
 from charmworld.views.api import (
     API0,
     API1,
@@ -70,12 +73,10 @@
         return self.api_class(request)()
 
 
-class TestAPI1Charms(APITestBase):
-
-    api_class = API1
+class TestAPICharms:
 
     def setUp(self):
-        super(ViewTestBase, self).setUp()
+        super(TestAPICharms, self).setUp()
         self.use_index_client()
 
     def makeCharm(self, *args, **kwargs):
@@ -92,179 +93,6 @@
             ("Access-Control-Allow-Headers", "X-Requested-With"),
             resp.headerlist)
 
-    def test_charms(self):
-        """Charms endpoint produces expected result."""
-        one_changes = [factory.makeChange(created=datetime(1999, 12, 31),
-                                          revno=37, message='Hello')]
-        one_id, one = self.makeCharm(name='name1', changes=one_changes,
-                                     date_created=datetime(1997, 10, 29,
-                                                           12, 10, 9))
-        self.index_client.index_charm(one)
-        maintainer2 = '[e-mail] (J. Random Hacker)'
-        provides2 = {'cookie-factory': {
-            'interface': 'imap',
-        }}
-        requires2 = {'postal-service': {
-            'interface': 'envelope',
-        }}
-        options2 = {
-            'foobar': {
-                'type': 'str',
-                'default': 'baz',
-                'description': 'quxx',
-            },
-        }
-        files2 = {
-            'readme': {
-                'filename': 'README.md',
-                'subdir': '',
-            },
-            'config-changed': {
-                'filename': 'config-changed',
-                'subdir': 'hooks',
-            },
-        }
-        two_changes = [factory.makeChange(created=datetime(1998, 11, 30),
-                                          revno=32, message='Hi',
-                                          authors=['Baz <[e-mail]>'])]
-        two_id, two = self.makeCharm(owner='jrandom', revision=3,
-                                     series='warty', summary='Charm two',
-                                     name='name2', revno=13,
-                                     maintainer=maintainer2,
-                                     commit_message='message2',
-                                     provides=provides2,
-                                     requires=requires2, options=options2,
-                                     files=files2, subordinate=True,
-                                     changes=two_changes,
-                                     date_created=datetime(1996, 9, 28,
-                                                           11, 9, 8))
-        response = self.get_response('charms')
-        self.assertEqual(200, response.status_code)
-        self.assertEqual('application/json', response.content_type)
-        result = response.json_body['result']
-        self.assertEqual(2, len(result))
-        one_url = make_store_url(1, get_address(one, short=True))
-        self.assertEqual({
-            'charm': {
-                'owner': 'charmers',
-                'maintainer': {
-                    'name': 'John Doe',
-                    'email': '[e-mail]',
-                },
-                'id': one_url[3:],
-                'date_created': '1997-10-29T12:10:09',
-                'name': 'name1',
-                'url': one_url,
-                'description': one['description'],
-                'revision': 1,
-                'summary': "Remote terminal classroom over https",
-                'distro_series': 'precise',
-                'is_approved': True,
-                'rating_numerator': 0,
-                'rating_denominator': 0,
-                'code_source': {
-                    'type': 'bzr',
-                    'location':
-                        'lp:~charmers/charms/precise/name1/trunk',
-                    'revision': '12',
-                    'last_log': 'maintainer',
-                    'bugs_link':
-                        'https://bugs.launchpad.net/charms/'
-                        '+source/name1',
-                    'revisions': [{
-                        'revno': 37,
-                        'message': 'Hello',
-                        'date': '1999-12-31T00:00:00Z',
-                        'authors': [
-                            {
-                                'name': 'Foo',
-                                'email': '[e-mail]'
-                            },
-                            {
-                                'name': 'Bar',
-                                'email': '[e-mail]'
-                            },
-                        ],
-                    }],
-                },
-                'relations': {
-                    'provides': {
-                        'website': {
-                            'interface': 'https',
-                        },
-                    },
-                    'requires': {
-                        'database': {
-                            'interface': 'mongodb',
-                        },
-                    },
-                },
-                'options': {
-                    'script-interval': {
-                        'type': 'int',
-                        'default': 5,
-                        'description':
-                        'The interval between script runs.'
-                    },
-                },
-                'files': ['README', 'hooks/install'],
-                'tested_providers': {},
-                'downloads_in_past_30_days': 0,
-                'is_subordinate': False,
-            }}, result[0])
-        two_url = make_store_url(1, get_address(two, short=False))
-        self.assertEqual({
-            'charm': {
-                'owner': 'jrandom',
-                'maintainer': {
-                    'name': 'J. Random Hacker',
-                    'email': '[e-mail]',
-                },
-                'id': two_url[3:],
-                'date_created': '1996-09-28T11:09:08',
-                'name': 'name2',
-                'url': two_url,
-                'description': two['description'],
-                'revision': 3,
-                'summary': 'Charm two',
-                'distro_series': 'warty',
-                'is_approved': False,
-                'rating_numerator': 0,
-                'rating_denominator': 0,
-                'code_source': {
-                    'type': 'bzr',
-                    'location':
-                        'lp:~jrandom/charms/warty/name2/trunk',
-                    'revision': '13',
-                    'revisions': [{
-                        'revno': 32,
-                        'message': 'Hi',
-                        'date': '1998-11-30T00:00:00Z',
-                        'authors': [
-                            {
-                                'name': 'Baz',
-                                'email': '[e-mail]'
-                            },
-                        ],
-                    }],
-                    'last_log': 'message2',
-                    'bugs_link':
-                    'https://bugs.launchpad.net/charms/+source/name2'
-                },
-                'relations': {
-                    'provides': provides2,
-                    'requires': requires2,
-
-                },
-                'options': options2,
-                'files': ['README.md', 'hooks/config-changed'],
-                'tested_providers': {},
-                'downloads_in_past_30_days': 0,
-                'is_subordinate': True,
-            }}, result[1])
-        self.assertNotEqual(one_id, two_id)
-        self.assertNotEqual(one_url, two_url)
-
     def get_charms(self, *args, **kwargs):
         return self.get_response('charms', *args, **kwargs).json_body['result']
 
@@ -285,7 +113,7 @@
         self.index_client.index_charm(four)
         result = self.get_charms(**{filter_str: 'common'})
         self.assertEqual(2, len(result))
-        res = self.api_class._charm_result
+        res = lambda x: self.api_class._charm_result(x, {}, {})
         self.assertItemsEqual([res(one), res(two)], result)
         result = self.get_charms(**{filter_str: 'different'})
         self.assertItemsEqual([res(three)], result)
@@ -332,8 +160,9 @@
         promulgated), 'community' otherwise.  'environment' causes no charms
         to be returned.
         """
-        charmers = self.api_class._charm_result(self.makeCharm()[1])
-        steve = self.api_class._charm_result(self.makeCharm(owner='steve')[1])
+        charmers = self.api_class._charm_result(self.makeCharm()[1], {}, {})
+        steve = self.api_class._charm_result(self.makeCharm(owner='steve')[1],
+                                             {}, {})
         result = self.get_charms(type_='community')
         self.assertEqual([steve], [charm for charm in result])
         result = self.get_charms(type_='approved')
@@ -352,9 +181,9 @@
     def test_charms_type_filter_considers_promulgation(self):
         """Promulgation controls type."""
         reviewed = self.api_class._charm_result(
-            self.makeCharm(promulgated=True)[1])
+            self.makeCharm(promulgated=True)[1], {}, {})
         unreviewed = self.api_class._charm_result(
-            self.makeCharm(promulgated=False)[1])
+            self.makeCharm(promulgated=False)[1], {}, {})
         result = self.get_charms(type_='community')
         self.assertEqual([unreviewed], [charm for charm in result])
         result = self.get_charms(type_='approved')
@@ -379,7 +208,7 @@
         self.assertEqual(200, response.status_code)
         charms = response.json_body['result']
         self.assertEqual(1, len(charms))
-        self.assertEqual(self.api_class._charm_result(abc_charm),
+        self.assertEqual(self.api_class._charm_result(abc_charm, {}, {}),
                          charms[0])
 
     def test_charms_text_search_respects_filters(self):
@@ -388,7 +217,7 @@
         self.makeCharm(summary='abc def', owner='steve')
         self.makeCharm(owner='steve')
         result = self.get_charms(type_=['approved'], text='def')
-        self.assertEqual([self.api_class._charm_result(official)],
+        self.assertEqual([self.api_class._charm_result(official, {}, {})],
                          [charm for charm in result])
 
     def test_charms_limit(self):
@@ -411,10 +240,6 @@
                          'parameter': 'limit',
                          }, response.json_body)
 
-    @staticmethod
-    def get_name(charm):
-        return charm['charm']['name']
-
     def test_charms_interesting(self):
         """A json dump of interesting charms is available."""
         for num in range(15):
@@ -437,19 +262,21 @@
         self.assertIn('featured', data['result'])
         self.assertEqual(['pop%d' % num for num in range(14, 4, -1)],
                          [self.get_name(r) for r in data['result']['popular']])
-        self.assertEqual(self.api_class._charm_result(last_popular_charm),
+        self.assertEqual(self.api_class._charm_result(last_popular_charm, {},
+                                                      {}),
                          data['result']['popular'][0])
         self.assertEqual(['new%d' % num for num in range(14, 4, -1)],
                          [self.get_name(r) for r in data['result']['new']])
-        self.assertEqual(self.api_class._charm_result(last_new_charm),
+        self.assertEqual(self.api_class._charm_result(last_new_charm, {}, {}),
                          data['result']['new'][0])
         self.assertItemsEqual(
             ['featured%d' % num for num in range(14, -1, -1)],
             [self.get_name(r) for r in data['result']['featured']])
         featured14, = [charm for charm in data['result']['featured'] if
                        self.get_name(charm) == 'featured14']
-        self.assertEqual(self.api_class._charm_result(last_featured_charm),
-                         featured14)
+        self.assertEqual(
+            self.api_class._charm_result(last_featured_charm, {}, {}),
+            featured14)
 
     def test_charms_accepts_categories(self):
         """Categories are accepted (but ignored)."""
@@ -457,14 +284,308 @@
         response = self.get_response('charms', categories='asdf')
         self.assertEqual(200, response.status_code)
 
+
+class API1Mixin:
+
+    api_class = API1
+
+    @staticmethod
+    def weightless(charm):
+        """Remove weight from charm because we cannot predict its value."""
+        charm = dict(charm)
+        del charm['weight']
+        return charm
+
+    @classmethod
+    def weightless_relation(cls, relation):
+        new_relation = {}
+        for interface, charms in relation.items():
+            charms = [cls.weightless(charm) for charm in charms]
+            new_relation[interface] = charms
+        return new_relation
+
+    @classmethod
+    def related_weightless(cls, charm):
+        return cls.weightless(cls.api_class._format_related(charm, 0))
+
+
+class TestAPI1Charms(TestAPICharms, APITestBase, API1Mixin):
+
+    def setUp(self):
+        super(TestAPI1Charms, self).setUp()
+
+    @staticmethod
+    def get_name(charm):
+        return charm['charm']['name']
+
+    def test_charms(self):
+        """Charms endpoint produces expected result."""
+        one_changes = [factory.makeChange(created=datetime(1999, 12, 31),
+                                          revno=37, message='Hello')]
+        one_id, one = self.makeCharm(name='name1', changes=one_changes,
+                                     date_created=datetime(1997, 10, 29,
+                                                           12, 10, 9))
+        self.index_client.index_charm(one)
+        maintainer2 = '[e-mail] (J. Random Hacker)'
+        provides2 = {'cookie-factory': {
+            'interface': 'imap',
+        }}
+        requires2 = {'postal-service': {
+            'interface': 'envelope',
+        }}
+        options2 = {
+            'foobar': {
+                'type': 'str',
+                'default': 'baz',
+                'description': 'quxx',
+            },
+        }
+        files2 = {
+            'readme': {
+                'filename': 'README.md',
+                'subdir': '',
+            },
+            'config-changed': {
+                'filename': 'config-changed',
+                'subdir': 'hooks',
+            },
+        }
+        two_changes = [factory.makeChange(created=datetime(1998, 11, 30),
+                                          revno=32, message='Hi',
+                                          authors=['Baz <[e-mail]>'])]
+        two_id, two = self.makeCharm(owner='jrandom', revision=3,
+                                     series='warty', summary='Charm two',
+                                     name='name2', revno=13,
+                                     maintainer=maintainer2,
+                                     commit_message='message2',
+                                     provides=provides2,
+                                     requires=requires2, options=options2,
+                                     files=files2, subordinate=True,
+                                     changes=two_changes,
+                                     date_created=datetime(1996, 9, 28,
+                                                           11, 9, 8))
+        response = self.get_response('charms')
+        self.assertEqual(200, response.status_code)
+        self.assertEqual('application/json', response.content_type)
+        result = response.json_body['result']
+        self.assertEqual(2, len(result))
+        one_url = make_store_url(1, get_address(one, short=True))
+        self.assertEqual({
+            'charm': {
+                'owner': 'charmers',
+                'maintainer': {
+                    'name': 'John Doe',
+                    'email': '[e-mail]',
+                },
+                'id': one_url[3:],
+                'date_created': '1997-10-29T12:10:09',
+                'name': 'name1',
+                'url': one_url,
+                'description': one['description'],
+                'revision': 1,
+                'summary': "Remote terminal classroom over https",
+                'distro_series': 'precise',
+                'is_approved': True,
+                'rating_numerator': 0,
+                'rating_denominator': 0,
+                'code_source': {
+                    'type': 'bzr',
+                    'location':
+                        'lp:~charmers/charms/precise/name1/trunk',
+                    'revision': '12',
+                    'last_log': 'maintainer',
+                    'bugs_link':
+                        'https://bugs.launchpad.net/charms/'
+                        '+source/name1',
+                    'revisions': [{
+                        'revno': 37,
+                        'message': 'Hello',
+                        'date': '1999-12-31T00:00:00Z',
+                        'authors': [
+                            {
+                                'name': 'Foo',
+                                'email': '[e-mail]'
+                            },
+                            {
+                                'name': 'Bar',
+                                'email': '[e-mail]'
+                            },
+                        ],
+                    }],
+                },
+                'relations': {
+                    'provides': {
+                        'website': {
+                            'interface': 'https',
+                        },
+                    },
+                    'requires': {
+                        'database': {
+                            'interface': 'mongodb',
+                        },
+                    },
+                },
+                'options': {
+                    'script-interval': {
+                        'type': 'int',
+                        'default': 5,
+                        'description':
+                        'The interval between script runs.'
+                    },
+                },
+                'files': ['README', 'hooks/install'],
+                'tested_providers': {},
+                'downloads_in_past_30_days': 0,
+                'is_subordinate': False,
+            },
+            'metadata': {
+                'related': {
+                    'provides': {},
+                    'requires': {},
+                }
+            }
+        }, result[0])
+        two_url = make_store_url(1, get_address(two, short=False))
+        self.assertEqual({
+            'charm': {
+                'owner': 'jrandom',
+                'maintainer': {
+                    'name': 'J. Random Hacker',
+                    'email': '[e-mail]',
+                },
+                'id': two_url[3:],
+                'date_created': '1996-09-28T11:09:08',
+                'name': 'name2',
+                'url': two_url,
+                'description': two['description'],
+                'revision': 3,
+                'summary': 'Charm two',
+                'distro_series': 'warty',
+                'is_approved': False,
+                'rating_numerator': 0,
+                'rating_denominator': 0,
+                'code_source': {
+                    'type': 'bzr',
+                    'location':
+                        'lp:~jrandom/charms/warty/name2/trunk',
+                    'revision': '13',
+                    'revisions': [{
+                        'revno': 32,
+                        'message': 'Hi',
+                        'date': '1998-11-30T00:00:00Z',
+                        'authors': [
+                            {
+                                'name': 'Baz',
+                                'email': '[e-mail]'
+                            },
+                        ],
+                    }],
+                    'last_log': 'message2',
+                    'bugs_link':
+                    'https://bugs.launchpad.net/charms/+source/name2'
+                },
+                'relations': {
+                    'provides': provides2,
+                    'requires': requires2,
+
+                },
+                'options': options2,
+                'files': ['README.md', 'hooks/config-changed'],
+                'tested_providers': {},
+                'downloads_in_past_30_days': 0,
+                'is_subordinate': True,
+            },
+            'metadata': {
+                'related': {
+                    'provides': {},
+                    'requires': {},
+                }
+            }
+        }, result[1])
+        self.assertNotEqual(one_id, two_id)
+        self.assertNotEqual(one_url, two_url)
+
     def test_charms_denies_category(self):
         """Category is not accepted."""
         self.index_client.wait_for_startup()
         response = self.get_response('charms', category='asdf')
         self.assertEqual(406, response.status_code)
 
-
-class TestAPI0Charms(TestAPI1Charms):
+    def test_charms_interesting_includes_related(self):
+        older = datetime.utcfromtimestamp(0)
+        newer = datetime.utcfromtimestamp(7200)
+        self.makeCharm(
+            date_created=newer, name='new',
+            provides={'asdf': {'interface': 'new-p'}},
+            requires={'asdf': {'interface': 'new-r'}},
+        )[1]
+        requires_newest = self.makeCharm(
+            date_created=older, name='req-new', requires={
+                'asdf': {'interface': 'new-p'}
+            })[1]
+        provides_newest = self.makeCharm(
+            date_created=older, name='provides-new', provides={
+                'asdf': {'interface': 'new-r'}
+            })[1]
+        self.makeCharm(
+            date_created=older, name='pop',
+            provides={'asdf': {'interface': 'pop-p'}},
+            requires={'asdf': {'interface': 'pop-r'}},
+            downloads_in_past_30_days=10,
+        )[1]
+        requires_popular = self.makeCharm(
+            date_created=older, name='req-pop', requires={
+                'asdf': {'interface': 'pop-p'}
+            })[1]
+        provides_popular = self.makeCharm(
+            date_created=older, name='provides-pop', provides={
+                'asdf': {'interface': 'pop-r'}
+            })[1]
+        self.makeCharm(
+            date_created=older, name='feat',
+            provides={'asdf': {'interface': 'feat-p'}},
+            requires={'asdf': {'interface': 'feat-r'}},
+            is_featured=True,
+        )[1]
+        requires_featured = self.makeCharm(
+            date_created=older, name='req-feat', requires={
+                'asdf': {'interface': 'feat-p'}
+            })[1]
+        provides_featured = self.makeCharm(
+            date_created=older, name='provides-feat', provides={
+                'asdf': {'interface': 'feat-r'}
+            })[1]
+        data = self.get_response('charms', remainder='/interesting').json_body
+        new_related = data['result']['new'][0]['metadata']['related']
+        self.assertEqual(
+            self.weightless_relation(new_related['requires']),
+            {'new-p': [self.related_weightless(requires_newest)]}
+        )
+        self.assertEqual(
+            self.weightless_relation(new_related['provides']),
+            {'new-r': [self.related_weightless(provides_newest)]}
+        )
+        pop_related = data['result']['popular'][0]['metadata']['related']
+        self.assertEqual(
+            self.weightless_relation(pop_related['requires']),
+            {'pop-p': [self.related_weightless(requires_popular)]}
+        )
+        self.assertEqual(
+            self.weightless_relation(pop_related['provides']),
+            {'pop-r': [self.related_weightless(provides_popular)]}
+        )
+        feat_related = data['result']['featured'][0]['metadata']['related']
+        self.assertEqual(
+            self.weightless_relation(feat_related['requires']),
+            {'feat-p': [self.related_weightless(requires_featured)]}
+        )
+        self.assertEqual(
+            self.weightless_relation(feat_related['provides']),
+            {'feat-r': [self.related_weightless(provides_featured)]}
+        )
+
+
+class TestAPI0Charms(TestAPICharms, APITestBase):
 
     api_class = API0
 
@@ -647,19 +768,14 @@
         response = self.get_response('charms', category='asdf')
         self.assertEqual(200, response.status_code)
 
-    def test_charms_denies_category(self):
-        pass
-
-
-class TestAPI1(APITestBase):
-
-    api_class = API1
+
+class TestAPI:
 
     def make_charm(self, *args, **kwargs):
         """Make a charm formatted for api."""
         charm = factory.makeCharm(self.db, *args, **kwargs)[1]
         return (self.api_class._get_api_id(charm),
-                self.api_class._charm_result(charm))
+                self.api_class._charm_result(charm, {}, {}))
 
     def test_unknown_endpoint(self):
         """Unknown endpoints produce a 404."""
@@ -807,6 +923,7 @@
 
     def test_charm(self):
         """Retrieving a charm works."""
+        self.use_index_client()
         charmid, charm = self.make_charm()
         response = self.get_response(
             'charm', remainder=charmid)
@@ -823,6 +940,7 @@
 
     def test_charm_wrong_revision(self):
         """Retrieving a charm works even if revision is wrong."""
+        self.use_index_client()
         charm = self.make_charm()[1]
         stale_charm_id = self.get_id(charm) + '37'
         response = self.get_response('charm', remainder=stale_charm_id)
@@ -1003,10 +1121,114 @@
         self.assertEqual(0, response.content_length)
 
 
-class TestAPI0(TestAPI1):
+class TestAPI1(APITestBase, TestAPI, API1Mixin):
+
+    def test_charm_result_includes_charm_and_metadata(self):
+        charm = factory.get_charm_json(promulgated=False)
+        result = self.api_class._charm_result(charm, {}, {})
+        self.assertIn('metadata', result)
+        self.assertIn('charm', result)
+
+    def test_format_related(self):
+        charm = factory.get_charm_json()
+        formatted = self.api_class._format_related(charm, 0)
+        self.assertItemsEqual(['id', 'name', 'categories', 'weight'],
+                              formatted.keys())
+        self.assertEqual(self.api_class._get_api_id(charm), formatted['id'])
+        self.assertEqual(charm['name'], formatted['name'])
+        self.assertEqual(charm['categories'], formatted['categories'])
+
+    def test_format_related_no_categories(self):
+        charm = factory.get_charm_json()
+        del charm['categories']
+        formatted = self.api_class._format_related(charm, 0)
+        self.assertEqual([], formatted['categories'])
+
+    def test_format_related_icon(self):
+        charm = factory.get_charm_json()
+        charm['icon'] = 'hello_world'
+        formatted = self.api_class._format_related(charm, 0)
+        self.assertEqual(charm['icon'], formatted['icon']['body'])
+        self.assertEqual('/api/1/' + formatted['id'] + '/icon',
+                         formatted['icon']['link'])
+
+    def test_charm_result_includes_related(self):
+        charm = factory.get_charm_json(promulgated=False, provides={
+            'cookie-factory': {
+            'interface': 'imap',
+            }
+        }, requires={
+            'cookie-factory': {
+            'interface': 'ssl',
+            }
+        })
+        imap_user = factory.get_charm_json(requires={
+            'foo': {'interface': 'imap'}
+        })
+        ssl_user = factory.get_charm_json(requires={
+            'bar': {'interface': 'ssl'}
+        })
+        ssl_provider = factory.get_charm_json(provides={
+            'bar': {'interface': 'ssl'}
+        })
+        imap_provider = factory.get_charm_json(provides={
+            'bar': {'interface': 'imap'}
+        })
+        requires = {
+            'imap': [self.api_class._format_related(imap_user, 0)],
+            'ssl': [self.api_class._format_related(ssl_user, 0)],
+        }
+        provides = {
+            'ssl': [self.api_class._format_related(ssl_provider, 0)],
+            'imap': [self.api_class._format_related(imap_provider, 0)],
+        }
+        result = self.api_class._charm_result(charm, requires, provides)
+        self.assertEqual([
+            self.api_class._format_related(imap_user, 0)],
+            result['metadata']['related']['requires']['imap'])
+        self.assertEqual([
+            self.api_class._format_related(ssl_provider, 0)],
+            result['metadata']['related']['provides']['ssl'])
+
+    def test_charm_includes_related(self):
+        self.use_index_client()
+        source = CharmSource.from_request(self)
+        main_charm = factory.get_charm_json(
+            requires={'foo': {'interface': 'imap'}},
+            provides={'foo': {'interface': 'smtp'}},
+        )
+        source.save(main_charm)
+        provides_charm = factory.get_charm_json(
+            provides={'foo': {'interface': 'imap'}},
+        )
+        source.save(provides_charm)
+        requires_charm = factory.get_charm_json(
+            requires={'foo': {'interface': 'smtp'}},
+        )
+        source.save(requires_charm)
+        charm_id = self.api_class._get_api_id(main_charm)
+        response = self.get_response('charm', remainder=charm_id).json_body
+        related = response['metadata']['related']
+        self.assertEqual(
+            self.weightless_relation(related['requires']),
+            {'smtp': [self.related_weightless(requires_charm)]}
+        )
+        self.assertEqual(
+            self.weightless_relation(related['provides']),
+            {'imap': [self.related_weightless(provides_charm)]}
+        )
+
+
+class TestAPI0(APITestBase, TestAPI):
 
     api_class = API0
 
     @staticmethod
     def get_id(charm):
         return charm['id']
+
+    def test_charm_result_excludes_charm_and_metadata(self):
+        charm = factory.get_charm_json(promulgated=False)
+        result = self.api_class._charm_result(charm, {}, {})
+        self.assertNotIn('metadata', result)
+        self.assertNotIn('charm', result)