Merge lp:~mszamot-gmail/beautifulsoup/beautifulsoup into lp:beautifulsoup

[Marcin Szamotulski]

Make Tag.descendants a proper generator. Make possible to resume iteration at a given node, which is important when one replaces nodes.

Make Tag.descendants a proper generator. With this patch one can send back where iteration should resume. This important if one replaces nodes with Tag.replace_with() method. See the doc string for an example.

[lp:~mszamot-gmail/beautifulsoup/beautifulsoup]

305. By Marcin Szamotulski on 2013-06-01

Make descendants a proper generator. This makes possible to resume iteration
at a given node, which is important when one replaces nodes.

[Leonard Richardson (leonardr)]

Quick questions:

1. Can you write some tests for this? Turning your docstring example into a test would be a good start.
2. Do you want to do this for the other generators as well?

[lp:~mszamot-gmail/beautifulsoup/beautifulsoup]

306. By Marcin Szamotulski on 2013-06-01

Add a test for Tag.descendants generator

[Marcin Szamotulski (mszamot-gmail)]

On 11:08 Sat 01 Jun , Leonard Richardson wrote:
> Quick questions:
>
> 1. Can you write some tests for this? Turning your docstring example into a test would be a good start.
> 2. Do you want to do this for the other generators as well?
> --
> https://code.launchpad.net/~mszamot-gmail/beautifulsoup/beautifulsoup/+merge/166933
> You are the owner of lp:~mszamot-gmail/beautifulsoup/beautifulsoup.

Yes I will include some tests and I will add this for other generators
as well. Do you agree with returning iter([]) instead of None? It
makes a little bit easier to write an iteration loop, though iter([])
does not have send attribute. I don't know however how to build an
empty generator besides:

    def iterator_func()
 raise StopIteration
 yield None
    empty_gen = iterator_func() # is an empty generator with both next and send attribute

Best regards,
Marcin Szamotulski

[lp:~mszamot-gmail/beautifulsoup/beautifulsoup]

307. By Marcin Szamotulski on 2013-06-01

Added proper generators and tests: Tag.next_elements, Tag.previous_elements,
Tag.next_siblings, Tag.previous_siblings, Tag.parents.

Unmerged revisions

307. By Marcin Szamotulski on 2013-06-01

Added proper generators and tests: Tag.next_elements, Tag.previous_elements,
Tag.next_siblings, Tag.previous_siblings, Tag.parents.

306. By Marcin Szamotulski on 2013-06-01

Add a test for Tag.descendants generator

305. By Marcin Szamotulski on 2013-06-01

Make descendants a proper generator. This makes possible to resume iteration
at a given node, which is important when one replaces nodes.

=== modified file 'bs4/element.py'
--- bs4/element.py	2013-05-31 13:44:37 +0000
+++ bs4/element.py	2013-06-01 21:29:26 +0000
@@ -509,35 +509,45 @@
     def next_elements(self):
         i = self.next_element
         while i is not None:
-            yield i
+            n = (yield i)
+            if n is not None:
+                i = n
             i = i.next_element
 
     @property
     def next_siblings(self):
         i = self.next_sibling
         while i is not None:
-            yield i
+            n = (yield i)
+            if n is not None:
+                i = n
             i = i.next_sibling
 
     @property
     def previous_elements(self):
         i = self.previous_element
         while i is not None:
-            yield i
+            n = (yield i)
+            if n is not None:
+                i = n
             i = i.previous_element
 
     @property
     def previous_siblings(self):
         i = self.previous_sibling
         while i is not None:
-            yield i
+            n = (yield i)
+            if n is not None:
+                i = n
             i = i.previous_sibling
 
     @property
     def parents(self):
         i = self.parent
         while i is not None:
-            yield i
+            n = (yield i)
+            if n is not None:
+                i = n
             i = i.parent
 
     # Methods for supporting CSS selectors.
@@ -810,25 +820,36 @@
 
         By default, yields only NavigableString and CData objects. So
         no comments, processing instructions, etc.
+
+        Note: strip=True returns unicode instances rather than NavigableStrings.
         """
-        for descendant in self.descendants:
-            if (
-                (types is None and not isinstance(descendant, NavigableString))
-                or
-                (types is not None and type(descendant) not in types)):
-                continue
-            if strip:
-                descendant = descendant.strip()
-                if len(descendant) == 0:
+        generator = self.descendants
+        if not hasattr(generator, 'send'):
+            raise StopIteration
+        descendant = generator.send(None)
+        while True:
+            try:
+                if (
+                    (types is None and not isinstance(descendant, NavigableString))
+                    or
+                    (types is not None and type(descendant) not in types)):
+                    descendant = generator.send(None)
                     continue
-            yield descendant
+                if strip:
+                    descendant = descendant.strip()
+                    if len(descendant) == 0:
+                        descendant = generator.send(None)
+                        continue
+                new = (yield descendant)
+                descendant = generator.send(new)
+            except StopIteration:
+                break
 
     strings = property(_all_strings)
 
     @property
     def stripped_strings(self):
-        for string in self._all_strings(True):
-            yield string
+        return self._all_strings(True)
 
     def get_text(self, separator=u"", strip=False,
                  types=(NavigableString, CData)):
@@ -1176,13 +1197,47 @@
 
     @property
     def descendants(self):
+        """
+        Generator over all descendants.  You can also use it to substitute
+        nodes.
+
+        If you replace nodes (with :meth:`self.replace_tag`) then you should
+        send the new node back to the generator so it knows how to go further.
+
+        .. python::
+
+            soup = BeautifulSoup(html)
+
+            generator = soup.descendants
+            tag = generator.send(None)
+            while True:
+                ntag = soup.new_tag('div')
+                if hasattr(tag, 'name'):
+                    ntag.append('here was: %s' % tag.name)
+                last = ntag._last_descendant()
+                try:
+                    if not isinstance(tag, bs4.NavigableString):
+                        tag.replace_with(ntag)
+                        # resume at the end of the ntag (this guards against
+                        # cirucral loops)
+                        tag = generator.send(last)
+                    else:
+                        tag = generator.send(None)
+                except StopIteration:
+                    break
+        """
+
         if not len(self.contents):
-            return
-        stopNode = self._last_descendant().next_element
-        current = self.contents[0]
-        while current is not stopNode:
-            yield current
-            current = current.next_element
+            return iter([])
+        def generator():
+            stopNode = self._last_descendant().next_element
+            current = self.contents[0]
+            while current is not stopNode:
+                new = (yield current)
+                if new is not None:
+                    current = new
+                current = current.next_element
+        return generator()
 
     # CSS selector code
 

=== modified file 'bs4/tests/test_soup.py'
--- bs4/tests/test_soup.py	2013-05-20 18:59:32 +0000
+++ bs4/tests/test_soup.py	2013-06-01 21:29:26 +0000
@@ -13,6 +13,7 @@
     ContentMetaAttributeValue,
     SoupStrainer,
     NamespacedAttribute,
+    NavigableString,
     )
 import bs4.dammit
 from bs4.dammit import EntitySubstitution, UnicodeDammit
@@ -381,3 +382,146 @@
         self.assertEqual("text/html; charset=euc-jp", value)
         self.assertEqual("text/html; charset=euc-jp", value.original_value)
         self.assertEqual("text/html; charset=utf8", value.encode("utf8"))
+
+
+class TestSoupGenerators(SoupTest):
+    """
+    Todo: add tests for:
+        Tag.next_elements
+        Tag.next_siblings
+        Tag.previous_elements
+        Tag.previous_siblings
+        Tag.parents
+    """
+
+    def setUp(self):
+
+        self.markup = ("level0<div>level1<span>level2<b>level3</b>"
+                       "level2'</span><span>level1'</span></div>level0'")
+        self.MARKUP = ("<it>LEVEL0</it><div><it>LEVEL1</it><span><it>LEVEL2</it><b>"
+                       "<it>LEVEL3</it></b><it>LEVEL2'</it></span><span>"
+                       "<it>LEVEL1'</it></span></div><it>LEVEL0'</it>")
+        self.MARKUP_next_elements = ("level0<div><it>LEVEL1</it><span>"
+                                     "<it>LEVEL2</it><b><it>LEVEL3</it></b>"
+                                     "<it>LEVEL2'</it></span><span>"
+                                     "<it>LEVEL1'</it></span></div><it>LEVEL0'</it>")
+        self.MARKUP_previous_elements = ("<it>LEVEL0</it><div><it>LEVEL1</it>"
+                                         "<span><it>LEVEL2</it><b><it>LEVEL3</it>"
+                                         "</b><it>LEVEL2'</it></span><span>"
+                                         "<it>LEVEL1'</it></span></div>level0'")
+
+        self.markup_siblings = ("<div><span>a</span><span>b<span>c</span>"
+                                "<span>d</span></span><span>e</span>"
+                                "<span>f</span></div>")
+        self.MARKUP_next_siblings = ("<div><span>a</span><it>B</it><it>E</it>"
+                                     "<it>F</it></div>")
+        self.MARKUP_previous_siblings = ("<div><it>A</it><it>B</it><it>E</it>"
+                                         "<span>f</span></div>")
+
+        self.markup_parents = ("<div>level0<div>level1<div>level2"
+                               "<div class=\"inner\">level3</div>level2</div>"
+                               "level1</div>level0</div>")
+        self.MARKUP_parents = ("<SPAN>level0<SPAN>level1<SPAN>level2"
+                               "<div class=\"inner\">level3</div>level2</SPAN>"
+                               "level1</SPAN>level0</SPAN>")
+    def _run(self, generator, soup):
+
+        tag = generator.send(None)
+        while True:
+            try:
+                if isinstance(tag, NavigableString):
+                    new_tag = soup.new_tag('it')
+                    new_string = soup.new_string(unicode(tag).upper())
+                    new_tag.append(new_string)
+                    tag.replace_with(new_tag)
+                    last = new_tag._last_descendant()
+                    tag = generator.send(last)
+                else:
+                    tag = generator.send(None)
+            except StopIteration:
+                break
+
+    def _run2(self, generator, soup):
+
+        tag = generator.send(None)
+        while True:
+            try:
+                if hasattr(tag, 'name') and tag.name == 'span':
+                    new_tag = soup.new_tag('it')
+                    new_string = soup.new_string(
+                        unicode(tag.contents[0]).upper()
+                    )
+                    new_tag.append(new_string)
+                    tag.replace_with(new_tag)
+                    tag = generator.send(new_tag)
+                else:
+                    tag = generator.send(None)
+            except StopIteration:
+                break
+
+    def test_descendants_generator(self):
+
+        soup = self.soup(self.markup)
+        generator = soup.descendants
+        self._run(generator, soup)
+        self.assertEquals(self.MARKUP, unicode(soup))
+
+    def test_strings_generator(self):
+
+        soup = self.soup(self.markup)
+        generator = soup.strings
+        self._run(generator, soup)
+        self.assertEquals(self.MARKUP, unicode(soup))
+
+    def test_next_elements(self):
+
+        soup = self.soup(self.markup)
+        el = soup.contents[0].next_element
+        generator = el.next_elements
+        self._run(generator, soup)
+        self.assertEquals(self.MARKUP_next_elements, unicode(soup))
+
+    def test_previous_elements(self):
+
+        soup = self.soup(self.markup)
+        el = soup._last_descendant()
+        generator = el.previous_elements
+        self._run(generator, soup)
+        self.assertEquals(self.MARKUP_previous_elements, unicode(soup))
+
+    def test_next_siblings(self):
+
+        soup = self.soup(self.markup_siblings)
+        el = soup.contents[0].next_element
+        generator = el.next_siblings
+        self._run2(generator, soup)
+        self.assertEquals(self.MARKUP_next_siblings, unicode(soup))
+
+    def test_next_siblings(self):
+
+        soup = self.soup(self.markup_siblings)
+        el = soup._last_descendant().previous_element
+        generator = el.previous_siblings
+        self._run2(generator, soup)
+        self.assertEquals(self.MARKUP_previous_siblings, unicode(soup))
+
+    def test_parents(self):
+
+        soup = self.soup(self.markup_parents)
+        div = soup.select('div.inner')[0]
+        generator = div.parents
+
+        tag = generator.send(None)
+        while True:
+            try:
+                new_tag = soup.new_tag('SPAN')
+                while len(list(tag.children)):
+                    new_tag.append(tag.contents[0])
+                tag.replace_with(new_tag)
+                tag = generator.send(new_tag)
+                if tag.parent is None:
+                    # one cannot replace if the parent is None
+                    raise StopIteration
+            except StopIteration:
+                break
+        self.assertEquals(self.MARKUP_parents, unicode(soup))