pydoctor

Merge lp:~adiroiban/pydoctor/1318325-intersphinx into lp:~mwhudson/pydoctor/dev

  1. 1318325-intersphinx
  2. Merge into dev
Proposed by Adi Roiban
Status: Merged
Merge reported by: Michael Hudson-Doyle
Merged at revision: not available
Proposed branch: lp:~adiroiban/pydoctor/1318325-intersphinx
Merge into: lp:~mwhudson/pydoctor/dev
Diff against target: 994 lines (+839/-34) (has conflicts)
8 files modified
README.txt (+68/-30)
pydoctor/driver.py (+11/-0)
pydoctor/epydoc2stan.py (+13/-0)
pydoctor/model.py (+23/-4)
pydoctor/sphinx.py (+187/-0)
pydoctor/test/test_epydoc2stan.py (+52/-0)
pydoctor/test/test_model.py (+79/-0)
pydoctor/test/test_sphinx.py (+406/-0)
Text conflict in README.txt
Conflict adding file pydoctor/sphinx.py.  Moved existing file to pydoctor/sphinx.py.moved.
Conflict adding file pydoctor/test/test_sphinx.py.  Moved existing file to pydoctor/test/test_sphinx.py.moved.
To merge this branch: bzr merge lp:~adiroiban/pydoctor/1318325-intersphinx
Reviewer Review Type Date Requested Status
Michael Hudson-Doyle Pending
Review via email: mp+219128@code.launchpad.net

Description of the change

Problem description
-------------------

pydoctor has an hardcoded links for stdlibs but it would be nice to be able
to link to sphinx apidocs using Sphinx inventory files

Changes
-------

Added '--intersphinx' configuration option.

It assumes that a remote object inventory contains indexed for a single root package.
This is why sphinx:http://domain.tld/latest/objects.inv will using 'http://domain.tld/latest/objects.inv' for any link starting with 'sphinx'

System was updated to allow initialization with options and to download and parse requested objects inventory.

Intersphinx inventories are stored using a simple dict and they provide a getLink method.

epydoc2stan linker was updated to look for intersphinx links when previous lookups fail.

How to test
-----------

Check that README documentation make sense... not sure where to put this
info.

I have also done a simple manual test using sphinx object.inv from rtd.org

Set one of the following markups

def getparser():
    """
    Return parser for command line options.

    L{pretty <sphinx.addnodes.desc_optional>}
    """

or

def getparser():
    """
    Return parser for command line options.

    L{sphinx.addnodes.desc_optional}
    """

build pydoctor using

pydoctor --intersphinx=sphinx:http://sphinx.readthedocs.org/en/latest/objects.inv pydoctor

I tried to write as much test as possible but this is a weekend hack and there might be gaps in coverage.. or errors in code.

Please check that changes make sense.
Let me know what needs to be changes or if you have better idea for how
this could be implemented.

I don't have to much experience with pydoctor usage and how people expect
to link to 3rd party.

Thanks!

To post a comment you must log in.
Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :
Download full text (23.5 KiB)

Adi Roiban <email address hidden> writes:

> Adi Roiban has proposed merging lp:~adiroiban/pydoctor/1318325-intersphinx into lp:pydoctor.
>
> Requested reviews:
> Michael Hudson-Doyle (mwhudson)
> Related bugs:
> Bug #1318325 in pydoctor: "Support linking to interlinks objects.inv"
> https://bugs.launchpad.net/pydoctor/+bug/1318325
>
> For more details, see:
> https://code.launchpad.net/~adiroiban/pydoctor/1318325-intersphinx/+merge/219128
>
> Problem description
> -------------------
>
> pydoctor has an hardcoded links for stdlibs but it would be nice to be able
> to link to sphinx apidocs using Sphinx inventory files

Heh, more amazingness!

>
> Changes
> -------
>
> Added '--intersphinx' configuration option.
>
> It assumes that a remote object inventory contains indexed for a single root package.
> This is why sphinx:http://domain.tld/latest/objects.inv will using 'http://domain.tld/latest/objects.inv' for any link starting with 'sphinx'
>
> System was updated to allow initialization with options and to download and parse requested objects inventory.
>
> Intersphinx inventories are stored using a simple dict and they provide a getLink method.
>
> epydoc2stan linker was updated to look for intersphinx links when previous lookups fail.

I guess in the fullness of time it would be nice to:

 a) support caching the objects.inv so that doc generation doesn't have
    to download it each time.
 b) allow the project being documented to define which objects.inv to use

neither of these feel at all urgent though.

> How to test
> -----------
>
> Check that README documentation make sense... not sure where to put this
> info.
>
> I have also done a simple manual test using sphinx object.inv from rtd.org
>
> Set one of the following markups
>
> def getparser():
> """
> Return parser for command line options.
>
> L{pretty <sphinx.addnodes.desc_optional>}
> """
>
> or
>
> def getparser():
> """
> Return parser for command line options.
>
> L{sphinx.addnodes.desc_optional}
> """
>
> build pydoctor using
>
> pydoctor --intersphinx=sphinx:http://sphinx.readthedocs.org/en/latest/objects.inv pydoctor
>
> I tried to write as much test as possible but this is a weekend hack and there might be gaps in coverage.. or errors in code.
>
> Please check that changes make sense.
> Let me know what needs to be changes or if you have better idea for how
> this could be implemented.
>
> I don't have to much experience with pydoctor usage and how people expect
> to link to 3rd party.
>
> Thanks!
>
> --
> https://code.launchpad.net/~adiroiban/pydoctor/1318325-intersphinx/+merge/219128
> You are requested to review the proposed merge of lp:~adiroiban/pydoctor/1318325-intersphinx into lp:pydoctor.
> === modified file 'README.txt'
> --- README.txt 2013-05-29 03:28:59 +0000
> +++ README.txt 2014-05-12 01:08:05 +0000
> @@ -14,3 +14,20 @@
> The default HTML generator requires Twisted.
>
> There are some more notes in the doc/ subdirectory.
> +
> +
> +Sphinx Integration
> +------------------
> +
> +It can link to external API documentation using Sphinx objects inventory using
> +the following cumulative configuration option::
> +
> + --in...

Revision history for this message
Adi Roiban (adiroiban) wrote :

a) For cache part, I prefer to do it in a separate ticket to keep this diff small.

b) I don't know what do you want from this ... can you please add more details?

Thanks!

Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

Adi Roiban <email address hidden> writes:

> a) For cache part, I prefer to do it in a separate ticket to keep this diff small.

+1. I also wonder about automatically appending objects.inv to the
supplied URL but well, that can wait as well.

> b) I don't know what do you want from this ... can you please add more details?

Well, all I mean is that it would be nice if Twisted could somehow say
that you should use the objects.inv from the stdlib, pyasn1 and
pycrypto, or whatever, rather than everyone who generates documentation
having to remember to pass them each time.

Having looked at intersphinx's documentation a bit, I think that it
should work a bit differently from this branch. In particular, I don't
think that naming the objects.inv really makes sense for pydoctor -- I
think we should just take a list of them and consult them in order.

But I tested it and it works, so thanks!

Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

I pushed a change to lp:~mwhudson/pydoctor/1318325-intersphinx to search through all inventories. (it still makes you name them)

It generates a lot more invalid refs than trunk but afaict trunk is generating broken links for all of them...

Revision history for this message
Adi Roiban (adiroiban) wrote :

Appending objects.inv could be added later :) ... this is a tool for developers and I assume all command arguments are only written once and then integrated into the build system.

I prefer to have explicit settings and know what URL is used.

-------

b) ... ok but then at least for stdlib we also need to configure if this is py2 or py3 ... maybe based on current python version.

I would leave this for a new ticket so that people start using the current code and then send some feedback.

Also, I see that pydoctor has a hack to load configuration options from a file... one workaround is to provide a sample

From my point of view intersphinx works in a similar way, as intershpinx does not arbitrary search all configured objects.inv

intersphinx_mapping = {
 'py2': ('http://docs.python.org/2.7', None),
 'py3': ('http://docs.python.org/3.2', None),
}

Adapted from documentation

A link like :ref:`comparison manual <py2:os.path>` will link to the label “os.path” in the doc set “py2”, if it exists.

To have a similar markup as intersphinx, we should need to use L{py2:os.path} instead of L{os.path}

I agree that for pydoctor this does not make sense and is ok to search whole list.

In this case --intersphinx http://url/to/objects.inv should be enough.

---

Maybe we can merge this branch is is and then have a new ticket to update code to look in all objects.inv and a new ticket to update stdlib links .... and see what to do with py2 vs py3 for stdlib.
..
Thanks!

Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

Adi Roiban <email address hidden> writes:

> Appending objects.inv could be added later :) ... this is a tool for developers and I assume all command arguments are only written once and then integrated into the build system.
>
> I prefer to have explicit settings and know what URL is used.

Fair enough.

> -------
>
> b) ... ok but then at least for stdlib we also need to configure if this is py2 or py3 ... maybe based on current python version.

Well, pydoctor is python 2 only currently. I guess you could use it to
document something written in the intersection of py2 and py3, but I'd
be very surprised if it ran under py3...

> I would leave this for a new ticket so that people start using the current code and then send some feedback.
>
> Also, I see that pydoctor has a hack to load configuration options from a file... one workaround is to provide a sample
>
>>From my point of view intersphinx works in a similar way, as intershpinx does not arbitrary search all configured objects.inv
>
> intersphinx_mapping = {
> 'py2': ('http://docs.python.org/2.7', None),
> 'py3': ('http://docs.python.org/3.2', None),
> }
>
> Adapted from documentation
>
> A link like :ref:`comparison manual <py2:os.path>` will link to the label “os.path” in the doc set “py2”, if it exists.
>
> To have a similar markup as intersphinx, we should need to use L{py2:os.path} instead of L{os.path}
>
> I agree that for pydoctor this does not make sense and is ok to search whole list.
>
> In this case --intersphinx http://url/to/objects.inv should be enough.

Heh, I think you just went through the same thought process I did. If
we supported things like :ref: links it would make sense -- but we
don't, it's more like the :py:class: stuff.

> ---
>
> Maybe we can merge this branch is is and then have a new ticket to update code to look in all objects.inv and a new ticket to update stdlib links .... and see what to do with py2 vs py3 for stdlib.

Yeah, that seems fair. I think I might just check the objects.inv for
python (maybe both 2 and 3) into pydoctor and refer to them by
default... but yeah, let's get something simple in first.

Cheers,
mwh

lp:~adiroiban/pydoctor/1318325-intersphinx updated
616. By Adi Roiban <email address hidden>

Merge master.

617. By Adi Roiban <email address hidden>

Update with a single db for all intersphinx.

Revision history for this message
Adi Roiban (adiroiban) wrote :

I have updated the code so that intersphinx only requires a full url to objects.inv files.
The lookup for intersphinx is done from a single database.

If this is ok, I can create a new ticket/branch to automatically populate the inventory with python2.7 object.inv ... but I find it of a lower priority...

The next step would be to cache downloaded intersphinx files.

Please check the latest code.

Revision history for this message
Adi Roiban (adiroiban) wrote :

Hm.. looks like the branch has conflicts.

I have merged master branch into this and fix conflicts with git-bzr ... but looks like they were not applied to bzr :( just use code from latest branch as it is merged with master.

Thanks!

lp:~adiroiban/pydoctor/1318325-intersphinx updated
618. By Adi Roiban <email address hidden>

Add tests.

Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

Well fiiiiiiiiiiiiiinally I sorted out the conflicts and merged this. Sorry for letting it sit for so very long.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'README.txt'
--- README.txt 2014-05-11 11:09:03 +0000
+++ README.txt 2014-05-30 08:40:38 +0000
@@ -14,33 +14,71 @@
14The default HTML generator requires Twisted.14The default HTML generator requires Twisted.
1515
16There are some more notes in the doc/ subdirectory.16There are some more notes in the doc/ subdirectory.
1717<<<<<<< TREE
1818
19Sphinx Integration19
20------------------20Sphinx Integration
2121------------------
22HTML generator will also generate a Sphinx objects inventory using the22
23following mapping:23HTML generator will also generate a Sphinx objects inventory using the
2424following mapping:
25* packages, modules -> py:mod:25
26* classes -> py:class:26* packages, modules -> py:mod:
27* functions -> py:func:27* classes -> py:class:
28* methods -> py:meth:28* functions -> py:func:
29* attributes -> py:attr:29* methods -> py:meth:
3030* attributes -> py:attr:
31Configure Sphinx intersphinx extension:31
3232Configure Sphinx intersphinx extension:
33 intersphinx_mapping = {33
34 'pydoctor': ('http://domain.tld/api', None),34 intersphinx_mapping = {
35 }35 'pydoctor': ('http://domain.tld/api', None),
3636 }
37Use external references::37
3838Use external references::
39 :py:func:`External API <pydoctor:pydoctor.model.Documentable.reparent>`39
4040 :py:func:`External API <pydoctor:pydoctor.model.Documentable.reparent>`
41 :py:mod:`pydoctor:pydoctor`41
42 :py:mod:`pydoctor:pydoctor.model`42 :py:mod:`pydoctor:pydoctor`
43 :py:func:`pydoctor:pydoctor.driver.getparser`43 :py:mod:`pydoctor:pydoctor.model`
44 :py:class:`pydoctor:pydoctor.model.Documentable`44 :py:func:`pydoctor:pydoctor.driver.getparser`
45 :py:meth:`pydoctor:pydoctor.model.Documentable.reparent`45 :py:class:`pydoctor:pydoctor.model.Documentable`
46 :py:attr:`pydoctor:pydoctor.model.Documentable.kind`46 :py:meth:`pydoctor:pydoctor.model.Documentable.reparent`
47 :py:attr:`pydoctor:pydoctor.model.Documentable.kind`
48=======
49
50
51Sphinx Integration
52------------------
53
54HTML generator will also generate a Sphinx objects inventory using the
55following mapping:
56
57* packages, modules -> py:mod:
58* classes -> py:class:
59* functions -> py:func:
60* methods -> py:meth:
61* attributes -> py:attr:
62
63Configure Sphinx intersphinx extension:
64
65 intersphinx_mapping = {
66 'pydoctor': ('http://domain.tld/api', None),
67 }
68
69Use external references::
70
71 :py:func:`External API <pydoctor:pydoctor.model.Documentable.reparent>`
72
73 :py:mod:`pydoctor:pydoctor`
74 :py:mod:`pydoctor:pydoctor.model`
75 :py:func:`pydoctor:pydoctor.driver.getparser`
76 :py:class:`pydoctor:pydoctor.model.Documentable`
77 :py:meth:`pydoctor:pydoctor.model.Documentable.reparent`
78 :py:attr:`pydoctor:pydoctor.model.Documentable.kind`
79
80It can link to external API documentation using Sphinx objects inventory using
81the following cumulative configuration option::
82
83 --intersphinx=http://sphinx.org/objects.inv
84>>>>>>> MERGE-SOURCE
4785
=== modified file 'pydoctor/driver.py'
--- pydoctor/driver.py 2014-05-13 08:46:30 +0000
+++ pydoctor/driver.py 2014-05-30 08:40:38 +0000
@@ -186,6 +186,14 @@
186 parser.add_option(186 parser.add_option(
187 '--introspect-c-modules', default=False, action='store_true',187 '--introspect-c-modules', default=False, action='store_true',
188 help=("Import and introspect any C modules found."))188 help=("Import and introspect any C modules found."))
189
190 parser.add_option(
191 '--intersphinx', action='append', dest='intersphinx',
192 metavar='URL_TO_OBJECTS.INV', default=[],
193 help=(
194 "Use Sphinx objects inventory to generate links to external"
195 "documetation. Can be repeated."))
196
189 return parser197 return parser
190198
191def readConfigFile(options):199def readConfigFile(options):
@@ -252,7 +260,10 @@
252 else:260 else:
253 system = systemclass()261 system = systemclass()
254262
263 # Once pickle support is removed, always instantiate System with
264 # options and make fetchIntersphinxInventories private in __init__.
255 system.options = options265 system.options = options
266 system.fetchIntersphinxInventories()
256267
257 system.urlprefix = ''268 system.urlprefix = ''
258 if options.moresystems:269 if options.moresystems:
259270
=== modified file 'pydoctor/epydoc2stan.py'
--- pydoctor/epydoc2stan.py 2014-01-06 20:37:07 +0000
+++ pydoctor/epydoc2stan.py 2014-05-30 08:40:38 +0000
@@ -113,6 +113,14 @@
113 thresh=-1)113 thresh=-1)
114 return None114 return None
115115
116 def look_for_intersphinx(self, name):
117 """
118 Return link for `name` based on intersphinx inventory.
119
120 Return None if link is not found.
121 """
122 return self.obj.system.intersphinx.getLink(name)
123
116 def translate_identifier_xref(self, fullID, prettyID):124 def translate_identifier_xref(self, fullID, prettyID):
117 """Figure out what ``L{fullID}`` should link to.125 """Figure out what ``L{fullID}`` should link to.
118126
@@ -163,6 +171,11 @@
163 self.obj.system.objectsOfType(model.Package)))171 self.obj.system.objectsOfType(model.Package)))
164 if target is not None:172 if target is not None:
165 return self._objLink(target, prettyID)173 return self._objLink(target, prettyID)
174
175 target = self.look_for_intersphinx(fullID)
176 if target:
177 return '<a href="%s"><code>%s</code></a>'%(target, prettyID)
178
166 self.obj.system.msg(179 self.obj.system.msg(
167 "translate_identifier_xref", "%s:%s invalid ref to %s" % (180 "translate_identifier_xref", "%s:%s invalid ref to %s" % (
168 self.obj.fullName(), self.obj.linenumber, fullID),181 self.obj.fullName(), self.obj.linenumber, fullID),
169182
=== modified file 'pydoctor/model.py'
--- pydoctor/model.py 2014-05-13 08:46:30 +0000
+++ pydoctor/model.py 2014-05-30 08:40:38 +0000
@@ -14,6 +14,8 @@
14import types14import types
15import __builtin__15import __builtin__
1616
17from pydoctor.sphinx import SphinxInventory
18
17# originally when I started to write pydoctor I had this idea of a big19# originally when I started to write pydoctor I had this idea of a big
18# tree of Documentables arranged in an almost arbitrary tree.20# tree of Documentables arranged in an almost arbitrary tree.
19#21#
@@ -347,7 +349,7 @@
347 #defaultBuilder = astbuilder.ASTBuilder349 #defaultBuilder = astbuilder.ASTBuilder
348 sourcebase = None350 sourcebase = None
349351
350 def __init__(self):352 def __init__(self, options=None):
351 self.allobjects = {}353 self.allobjects = {}
352 self.orderedallobjects = []354 self.orderedallobjects = []
353 self.rootobjects = []355 self.rootobjects = []
@@ -356,9 +358,14 @@
356 self.moresystems = []358 self.moresystems = []
357 self.subsystems = []359 self.subsystems = []
358 self.urlprefix = ''360 self.urlprefix = ''
359 from pydoctor.driver import parse_args361
360 self.options, _ = parse_args([])362 if options:
361 self.options.verbosity = 3363 self.options = options
364 else:
365 from pydoctor.driver import parse_args
366 self.options, _ = parse_args([])
367 self.options.verbosity = 3
368
362 self.abbrevmapping = {}369 self.abbrevmapping = {}
363 self.projectname = 'my project'370 self.projectname = 'my project'
364 self.epytextproblems = [] # fullNames of objects that failed to epytext properly371 self.epytextproblems = [] # fullNames of objects that failed to epytext properly
@@ -369,6 +376,10 @@
369 self.module_count = 0376 self.module_count = 0
370 self.processing_modules = []377 self.processing_modules = []
371 self.buildtime = datetime.datetime.now()378 self.buildtime = datetime.datetime.now()
379 # Once pickle support is removed, System should be
380 # initialized with project name so that we can reuse intersphinx instace for
381 # object.inv generation.
382 self.intersphinx = SphinxInventory(logger=self.msg, project_name=self.projectname)
372383
373 def verbosity(self, section=None):384 def verbosity(self, section=None):
374 if isinstance(section, str):385 if isinstance(section, str):
@@ -702,3 +713,11 @@
702 while self.unprocessed_modules:713 while self.unprocessed_modules:
703 mod = iter(self.unprocessed_modules).next()714 mod = iter(self.unprocessed_modules).next()
704 self.processModule(mod)715 self.processModule(mod)
716
717
718 def fetchIntersphinxInventories(self):
719 """
720 Download and parse intersphinx inventories based on configuration.
721 """
722 for url in self.options.intersphinx:
723 self.intersphinx.update(url)
705724
=== added file 'pydoctor/sphinx.py'
--- pydoctor/sphinx.py 1970-01-01 00:00:00 +0000
+++ pydoctor/sphinx.py 2014-05-30 08:40:38 +0000
@@ -0,0 +1,187 @@
1"""
2Support for Sphinx compatibility.
3"""
4from __future__ import absolute_import
5
6import os
7import urllib2
8import zlib
9
10
11class SphinxInventory(object):
12 """
13 Sphinx inventory handler.
14 """
15
16 version = (2, 0)
17
18 def __init__(self, logger, project_name):
19 self.project_name = project_name
20 self.msg = logger
21 self._links = {}
22
23 def generate(self, subjects, basepath):
24 """
25 Generate Sphinx objects inventory version 2 at `basepath`/objects.inv.
26 """
27 path = os.path.join(basepath, 'objects.inv')
28 self.msg('sphinx', 'Generating objects inventory at %s' % (path,))
29
30 with self._openFileForWriting(path) as target:
31 target.write(self._generateHeader())
32 content = self._generateContent(subjects)
33 target.write(zlib.compress(content))
34
35 def _openFileForWriting(self, path):
36 """
37 Helper for testing.
38 """
39 return open(path, 'w')
40
41 def _generateHeader(self):
42 """
43 Return header for project with name.
44 """
45 version = [str(part) for part in self.version]
46 return """# Sphinx inventory version 2
47# Project: %s
48# Version: %s
49# The rest of this file is compressed with zlib.
50""" % (self.project_name, '.'.join(version))
51
52 def _generateContent(self, subjects):
53 """
54 Write inventory for all `subjects`.
55 """
56 content = []
57 for obj in subjects:
58 if not obj.isVisible:
59 continue
60 content.append(self._generateLine(obj))
61 content.append(self._generateContent(obj.orderedcontents))
62
63 return ''.join(content)
64
65 def _generateLine(self, obj):
66 """
67 Return inventory line for object.
68
69 name domain_name:type priority URL display_name
70
71 Domain name is always: py
72 Priority is always: -1
73 Display name is always: -
74 """
75 # Avoid circular import.
76 from pydoctor import model
77
78 full_name = obj.fullName()
79
80 if obj.documentation_location == model.DocLocation.OWN_PAGE:
81 url = obj.fullName() + '.html'
82 else:
83 url = obj.parent.fullName() + '.html#' + obj.name
84
85 display = '-'
86 if isinstance(obj, (model.Package, model.Module)):
87 domainname = 'module'
88 elif isinstance(obj, model.Class):
89 domainname = 'class'
90 elif isinstance(obj, model.Function):
91 if obj.kind == 'Function':
92 domainname = 'function'
93 else:
94 domainname = 'method'
95 elif isinstance(obj, model.Attribute):
96 domainname = 'attribute'
97 else:
98 domainname = 'obj'
99 self.msg('sphinx', "Unknown type %r for %s." % (type(obj), full_name,))
100
101 return '%s py:%s -1 %s %s\n' % (full_name, domainname, url, display)
102
103 def update(self, url):
104 """
105 Update inventory from URL.
106 """
107 parts = url.rsplit('/', 1)
108 if len(parts) != 2:
109 self.msg(
110 'sphinx', 'Failed to get remote base url for %s' % (url,))
111 return
112
113 base_url = parts[0]
114
115 data = self._getURL(url)
116
117 if not data:
118 self.msg(
119 'sphinx', 'Failed to get object inventory from %s' % (url, ))
120 return
121
122 payload = self._getPayload(base_url, data)
123 self._links.update(self._parseInventory(base_url, payload))
124
125 def _getURL(self, url):
126 """
127 Get content of URL.
128
129 This is a helper for testing.
130 """
131 try:
132 response = urllib2.urlopen(url)
133 return response.read()
134 except:
135 return None
136
137 def _getPayload(self, base_url, data):
138 """
139 Parse inventory and return clear text payload without comments.
140 """
141 payload = ''
142 while True:
143 parts = data.split('\n', 1)
144 if len(parts) != 2:
145 payload = data
146 break
147 if not parts[0].startswith('#'):
148 payload = data
149 break
150 data = parts[1]
151 try:
152 return zlib.decompress(payload)
153 except:
154 self.msg(
155 'sphinx',
156 'Failed to uncompress inventory from %s' % (base_url,))
157 return ''
158
159 def _parseInventory(self, base_url, payload):
160 """
161 Parse clear text payload and return a dict with module to link mapping.
162 """
163 result = {}
164 for line in payload.splitlines():
165 parts = line.split(' ', 4)
166 if len(parts) != 5:
167 self.msg(
168 'sphinx',
169 'Failed to parse line "%s" for %s' % (line, base_url),
170 )
171 continue
172 result[parts[0]] = (base_url, parts[3])
173 return result
174
175 def getLink(self, name):
176 """
177 Return link for `name` or None if no link is found.
178 """
179 base_url, relative_link = self._links.get(name, (None, None))
180 if not relative_link:
181 return None
182
183 # For links ending with $, replace it with full name.
184 if relative_link.endswith('$'):
185 relative_link = relative_link[:-1] + name
186
187 return '%s/%s' % (base_url, relative_link)
0188
=== renamed file 'pydoctor/sphinx.py' => 'pydoctor/sphinx.py.moved'
=== modified file 'pydoctor/test/test_epydoc2stan.py'
--- pydoctor/test/test_epydoc2stan.py 2013-05-23 02:03:27 +0000
+++ pydoctor/test/test_epydoc2stan.py 2014-05-30 08:40:38 +0000
@@ -1,6 +1,9 @@
1from pydoctor import epydoc2stan1from pydoctor import epydoc2stan
2from pydoctor import model
3from pydoctor.sphinx import SphinxInventory
2from pydoctor.test.test_astbuilder import fromText4from pydoctor.test.test_astbuilder import fromText
35
6
4def test_multiple_types():7def test_multiple_types():
5 mod = fromText('''8 mod = fromText('''
6 def f(a):9 def f(a):
@@ -64,3 +67,52 @@
64 assert u'Lorem Ipsum' == get_summary('single_line_summary')67 assert u'Lorem Ipsum' == get_summary('single_line_summary')
65 assert u'Foo Bar Baz' == get_summary('three_lines_summary')68 assert u'Foo Bar Baz' == get_summary('three_lines_summary')
66 assert u'No summary' == get_summary('no_summary')69 assert u'No summary' == get_summary('no_summary')
70
71
72def test_EpydocLinker_look_for_intersphinx_no_link():
73 """
74 Return None if inventory had no link for our markup.
75 """
76 system = model.System()
77 target = model.Module(system, 'ignore-name', 'ignore-docstring')
78 sut = epydoc2stan._EpydocLinker(target)
79
80 result = sut.look_for_intersphinx('base.module')
81
82 assert None is result
83
84
85def test_EpydocLinker_look_for_intersphinx_hit():
86 """
87 Return the link from inventory based on first package name.
88 """
89 system = model.System()
90 inventory = SphinxInventory(system.msg, 'some-project')
91 inventory._links['base.module.other'] = ('http://tm.tld', 'some.html')
92 system.intersphinx = inventory
93 target = model.Module(system, 'ignore-name', 'ignore-docstring')
94 sut = epydoc2stan._EpydocLinker(target)
95
96 result = sut.look_for_intersphinx('base.module.other')
97
98 assert 'http://tm.tld/some.html' == result
99
100
101def test_EpydocLinker_translate_identifier_xref_intersphinx():
102 """
103 Return the link from inventory.
104 """
105 system = model.System()
106 inventory = SphinxInventory(system.msg, 'some-project')
107 inventory._links['base.module.other'] = ('http://tm.tld', 'some.html')
108 system.intersphinx = inventory
109 target = model.Module(system, 'ignore-name', 'ignore-docstring')
110 sut = epydoc2stan._EpydocLinker(target)
111
112 result = sut.translate_identifier_xref(
113 'base.module.other', 'base.module.pretty')
114
115 expected = (
116 '<a href="http://tm.tld/some.html"><code>base.module.pretty</code></a>'
117 )
118 assert expected == result
67119
=== modified file 'pydoctor/test/test_model.py'
--- pydoctor/test/test_model.py 2008-03-16 19:09:50 +0000
+++ pydoctor/test/test_model.py 2014-05-30 08:40:38 +0000
@@ -1,4 +1,11 @@
1"""
2Unit tests for model.
3"""
1from pydoctor import model4from pydoctor import model
5from pydoctor.driver import parse_args
6from pydoctor.sphinx import SphinxInventory
7import zlib
8
29
3class FakeOptions(object):10class FakeOptions(object):
4 """11 """
@@ -41,3 +48,75 @@
4148
42 expected = viewSourceBase + moduleRelativePart49 expected = viewSourceBase + moduleRelativePart
43 assert mod.sourceHref == expected50 assert mod.sourceHref == expected
51
52
53def test_initialization_default():
54 """
55 When initialized without options, will use default options and default
56 verbosity.
57 """
58 sut = model.System()
59
60 assert None is sut.options.projectname
61 assert 3 == sut.options.verbosity
62
63
64def test_initialization_options():
65 """
66 Can be initialized with options.
67 """
68 options = object()
69
70 sut = model.System(options=options)
71
72 assert options is sut.options
73
74
75def test_fetchIntersphinxInventories_empty():
76 """
77 Convert option to empty dict.
78 """
79 options, _ = parse_args([])
80 options.intersphinx = []
81 sut = model.System(options=options)
82
83 sut.fetchIntersphinxInventories()
84
85 # Use internal state since I don't know how else to
86 # check for SphinxInventory state.
87 assert {} == sut.intersphinx._links
88
89
90def test_fetchIntersphinxInventories_content():
91 """
92 Download and parse intersphinx inventories for each configured
93 intersphix.
94 """
95 options, _ = parse_args([])
96 options.intersphinx = [
97 'http://sphinx/objects.inv',
98 'file:///twisted/index.inv',
99 ]
100 url_content = {
101 'http://sphinx/objects.inv': zlib.compress(
102 'sphinx.module py:module -1 sp.html -'),
103 'file:///twisted/index.inv': zlib.compress(
104 'twisted.package py:module -1 tm.html -'),
105 }
106 sut = model.System(options=options)
107 log = []
108 sut.msg = lambda part, msg: log.append((part, msg))
109 # Patch url getter to avoid touching the network.
110 sut.intersphinx._getURL = lambda url: url_content[url]
111
112 sut.fetchIntersphinxInventories()
113
114 assert [] == log
115 assert (
116 'http://sphinx/sp.html' ==
117 sut.intersphinx.getLink('sphinx.module')
118 )
119 assert (
120 'file:///twisted/tm.html' ==
121 sut.intersphinx.getLink('twisted.package')
122 )
44123
=== added file 'pydoctor/test/test_sphinx.py'
--- pydoctor/test/test_sphinx.py 1970-01-01 00:00:00 +0000
+++ pydoctor/test/test_sphinx.py 2014-05-30 08:40:38 +0000
@@ -0,0 +1,406 @@
1"""
2Tests for Sphinx integration.
3"""
4from contextlib import closing
5from StringIO import StringIO
6import zlib
7
8from pydoctor import model
9from pydoctor.sphinx import SphinxInventory
10
11
12class PersistentStringIO(StringIO):
13 """
14 A custom stringIO which keeps content after file is closed.
15 """
16 def close(self):
17 """
18 Close, but keep the memory buffer and seek position.
19 """
20 if not self.closed:
21 self.closed = True
22
23 def getvalue(self):
24 """
25 Retrieve the entire contents of the "file" at any time even after
26 the StringIO object's close() method is called.
27 """
28 if self.buflist:
29 self.buf += ''.join(self.buflist)
30 self.buflist = []
31 return self.buf
32
33
34def test_initialization():
35 """
36 Is initialized with logger and project name.
37 """
38 logger = object()
39 name = object()
40
41 sut = SphinxInventory(logger=logger, project_name=name)
42
43 assert logger is sut.msg
44 assert name is sut.project_name
45
46
47def test_generate_empty_functional():
48 """
49 Functional test for index generation of empty API.
50
51 Header is plain text while content is compressed.
52 """
53 project_name = 'some-name'
54 log = []
55 logger = lambda part, message: log.append((part, message))
56 sut = SphinxInventory(logger=logger, project_name=project_name)
57 output = PersistentStringIO()
58 sut._openFileForWriting = lambda path: closing(output)
59
60 sut.generate(subjects=[], basepath='base-path')
61
62 expected_log = [(
63 'sphinx',
64 'Generating objects inventory at base-path/objects.inv'
65 )]
66 assert expected_log == log
67
68 expected_ouput = """# Sphinx inventory version 2
69# Project: some-name
70# Version: 2.0
71# The rest of this file is compressed with zlib.
72x\x9c\x03\x00\x00\x00\x00\x01"""
73 assert expected_ouput == output.getvalue()
74
75
76def make_SphinxInventory():
77 """
78 Return a SphinxInventory.
79 """
80 return SphinxInventory(logger=object(), project_name='project_name')
81
82
83def test_generateContent():
84 """
85 Return a string with inventory for all targeted objects, recursive.
86 """
87 sut = make_SphinxInventory()
88 system = model.System()
89 root1 = model.Package(system, 'package1', 'docstring1')
90 root2 = model.Package(system, 'package2', 'docstring2')
91 child1 = model.Package(system, 'child1', 'docstring3', parent=root2)
92 system.addObject(child1)
93 subjects = [root1, root2]
94
95 result = sut._generateContent(subjects)
96
97 expected_result = (
98 'package1 py:module -1 package1.html -\n'
99 'package2 py:module -1 package2.html -\n'
100 'package2.child1 py:module -1 package2.child1.html -\n'
101 )
102 assert expected_result == result
103
104
105def test_generateLine_package():
106 """
107 Check inventory for package.
108 """
109 sut = make_SphinxInventory()
110
111 result = sut._generateLine(
112 model.Package('ignore-system', 'package1', 'ignore-docstring'))
113
114 assert 'package1 py:module -1 package1.html -\n' == result
115
116
117def test_generateLine_module():
118 """
119 Check inventory for module.
120 """
121 sut = make_SphinxInventory()
122
123 result = sut._generateLine(
124 model.Module('ignore-system', 'module1', 'ignore-docstring'))
125
126 assert 'module1 py:module -1 module1.html -\n' == result
127
128
129def test_generateLine_class():
130 """
131 Check inventory for class.
132 """
133 sut = make_SphinxInventory()
134
135 result = sut._generateLine(
136 model.Class('ignore-system', 'class1', 'ignore-docstring'))
137
138 assert 'class1 py:class -1 class1.html -\n' == result
139
140
141def test_generateLine_function():
142 """
143 Check inventory for function.
144
145 Functions are inside a module.
146 """
147 sut = make_SphinxInventory()
148 parent = model.Module('ignore-system', 'module1', 'docstring')
149
150 result = sut._generateLine(
151 model.Function('ignore-system', 'func1', 'ignore-docstring', parent))
152
153 assert 'module1.func1 py:function -1 module1.html#func1 -\n' == result
154
155
156def test_generateLine_method():
157 """
158 Check inventory for method.
159
160 Methods are functions inside a class.
161 """
162 sut = make_SphinxInventory()
163 parent = model.Class('ignore-system', 'class1', 'docstring')
164
165 result = sut._generateLine(
166 model.Function('ignore-system', 'meth1', 'ignore-docstring', parent))
167
168 assert 'class1.meth1 py:method -1 class1.html#meth1 -\n' == result
169
170
171def test_generateLine_attribute():
172 """
173 Check inventory for attributes.
174 """
175 sut = make_SphinxInventory()
176 parent = model.Class('ignore-system', 'class1', 'docstring')
177
178 result = sut._generateLine(
179 model.Attribute('ignore-system', 'attr1', 'ignore-docstring', parent))
180
181 assert 'class1.attr1 py:attribute -1 class1.html#attr1 -\n' == result
182
183
184class UnknownType(model.Documentable):
185 """
186 Documentable type to help with testing.
187 """
188
189
190def test_generateLine_unknown():
191 """
192 When object type is uknown a message is logged and is handled as
193 generic object.
194 """
195 log = []
196 sut = make_SphinxInventory()
197 sut.msg = lambda part, message: log.append((part, message))
198
199 result = sut._generateLine(
200 UnknownType('ignore-system', 'unknown1', 'ignore-docstring'))
201
202 assert 'unknown1 py:obj -1 unknown1.html -\n' == result
203
204
205def make_SphinxInventory():
206 """
207 Return a SphinxInventory.
208 """
209 return SphinxInventory(logger=object(), project_name='project_name')
210
211
212def make_SphinxInventoryWithLog():
213 """
214 Return a SphinxInventory with patched log.
215 """
216 inventory = make_SphinxInventory()
217 log = []
218 inventory.msg = lambda part, msg: log.append((part, msg))
219 return (inventory, log)
220
221
222def test_getPayload_empty():
223 """
224 Return empty string.
225 """
226 sut = make_SphinxInventory()
227 content = """# Sphinx inventory version 2
228# Project: some-name
229# Version: 2.0
230# The rest of this file is compressed with zlib.
231x\x9c\x03\x00\x00\x00\x00\x01"""
232
233 result = sut._getPayload('http://base.ignore', content)
234
235 assert '' == result
236
237
238def test_getPayload_content():
239 """
240 Return content as string.
241 """
242 payload = 'first_line\nsecond line'
243 sut = make_SphinxInventory()
244 content = """# Ignored line
245# Project: some-name
246# Version: 2.0
247# commented line.
248%s""" % (zlib.compress(payload),)
249
250 result = sut._getPayload('http://base.ignore', content)
251
252 assert payload == result
253
254
255def test_getPayload_invalid():
256 """
257 Return empty string and log an error when failing to uncompress data.
258 """
259 sut, log = make_SphinxInventoryWithLog()
260 base_url = 'http://tm.tld'
261 content = """# Project: some-name
262# Version: 2.0
263not-valid-zlib-content"""
264
265 result = sut._getPayload(base_url, content)
266
267 assert '' == result
268 assert [(
269 'sphinx', 'Failed to uncompress inventory from http://tm.tld',
270 )] == log
271
272
273def test_getLink_not_found():
274 """
275 Return None if link does not exists.
276 """
277 sut = make_SphinxInventory()
278
279 assert None is sut.getLink('no.such.name')
280
281
282def test_getLink_found():
283 """
284 Return the link from internal state.
285 """
286 sut = make_SphinxInventory()
287 sut._links['some.name'] = ('http://base.tld', 'some/url.php')
288
289 assert 'http://base.tld/some/url.php' == sut.getLink('some.name')
290
291
292def test_getLink_self_anchor():
293 """
294 Return the link with anchor as target name when link end with $.
295 """
296 sut = make_SphinxInventory()
297 sut._links['some.name'] = ('http://base.tld', 'some/url.php#$')
298
299 assert 'http://base.tld/some/url.php#some.name' == sut.getLink('some.name')
300
301
302def test_update_functional():
303 """
304 Functional test for updating from an empty inventory.
305 """
306 payload = (
307 'some.module1 py:module -1 module1.html -\n'
308 'other.module2 py:module 0 module2.html Other description\n'
309 )
310 sut = make_SphinxInventory()
311 # Patch URL loader to avoid hitting the system.
312 content = """# Sphinx inventory version 2
313# Project: some-name
314# Version: 2.0
315# The rest of this file is compressed with zlib.
316%s""" % (zlib.compress(payload),)
317 sut._getURL = lambda _: content
318
319 sut.update('http://some.url/api/objects.inv')
320
321 assert 'http://some.url/api/module1.html' == sut.getLink('some.module1')
322 assert 'http://some.url/api/module2.html' == sut.getLink('other.module2')
323
324
325def test_update_bad_url():
326 """
327 Log an error when failing to get base url from url.
328 """
329 sut, log = make_SphinxInventoryWithLog()
330
331 sut.update('really.bad.url')
332
333 assert sut._links == {}
334 expected_log = [(
335 'sphinx', 'Failed to get remote base url for really.bad.url'
336 )]
337 assert expected_log == log
338
339
340def test_update_fail():
341 """
342 Log an error when failing to get content from url.
343 """
344 sut, log = make_SphinxInventoryWithLog()
345 sut._getURL = lambda _: None
346
347 sut.update('http://some.tld/o.inv')
348
349 assert sut._links == {}
350 expected_log = [(
351 'sphinx', 'Failed to get object inventory from http://some.tld/o.inv'
352 )]
353 assert expected_log == log
354
355
356def test_parseInventory_empty():
357 """
358 Return empty dict for empty input.
359 """
360 sut = make_SphinxInventory()
361
362 result = sut._parseInventory('http://base.tld', '')
363
364 assert {} == result
365
366
367def test_parseInventory_single_line():
368 """
369 Return a dict with a single member.
370 """
371 sut = make_SphinxInventory()
372
373 result = sut._parseInventory(
374 'http://base.tld', 'some.attr py:attr -1 some.html De scription')
375
376 assert {'some.attr': ('http://base.tld', 'some.html')} == result
377
378
379def test_parseInventory_invalid_lines():
380 """
381 Skip line and log an error.
382 """
383 sut, log = make_SphinxInventoryWithLog()
384 base_url = 'http://tm.tld'
385 content = (
386 'good.attr py:attribute -1 some.html -\n'
387 'bad.attr bad format\n'
388 'very.bad\n'
389 '\n'
390 'good.again py:module 0 again.html -\n'
391 )
392
393 result = sut._parseInventory(base_url, content)
394
395 assert {
396 'good.attr': (base_url, 'some.html'),
397 'good.again': (base_url, 'again.html'),
398 } == result
399 assert [
400 (
401 'sphinx',
402 'Failed to parse line "bad.attr bad format" for http://tm.tld'
403 ),
404 ('sphinx', 'Failed to parse line "very.bad" for http://tm.tld'),
405 ('sphinx', 'Failed to parse line "" for http://tm.tld'),
406 ] == log
0407
=== renamed file 'pydoctor/test/test_sphinx.py' => 'pydoctor/test/test_sphinx.py.moved'

Subscribers

People subscribed via source and target branches

to all changes:
to status/vote changes: