Merge lp:~stevanr/linaro-license-protection/production-integration-tests into lp:~linaro-automation/linaro-license-protection/trunk

[Stevan Radaković]

Added doctest production tests for both snapshots.linaro.org and releases.linaro.org.
New helper class for directory/file navigation.
Changes to __init to include doctest.

[Данило Шеган (danilo)]

So much nicer, thanks for working on this.

Now, considering what is mostly tested, I'd suggest you add a get_content_title() method which extracts the title tag (considering you are already using BeautifulSoup, that should be easy), so the output would be nicer and even more clear, for example:

    Browsing into the android/~linaro-android/*snowball* works without asking for any license acceptance:

        >>> browser.browse_to_relative("android/")
        >>> browser.get_content_title()
        u'Index of /android'
        >>> browser.browse_to_relative("~linaro-android")
        >>> browser.get_content_title()
        u'Index of /android/~linaro-android'
        >>> browser.browse_to_relative("snowball")
        >>> browser.get_content_title()
        u'Index of /android/~linaro-android/...snowball...'

As you can see, I am also suggesting joining a few steps behind a single narrative. I'd suggest you do the same for the target/product/* step as well.

I like all the other improvements (DoctestProductionBrowser(host), get_license() call etc).

It would still be nicer to print out the headers one-per-line instead of as a dict:

        >>> print browser.get_header_when_redirected()
        Content-Length: ...
        Location: http://snapshots.../boot.tar.bz2
        Content-Type: application/x-bzip2
        ...

Also note that there is no guarantee in what order headers will be returned, and this test might easily break, so I suggest get_headers_when_redirected sorts them by name before returning a string with one header per line.

[Данило Шеган (danilo)]

Uhm, make those browse_to_relative browse_to_next, mea culpa. :)

[Данило Шеган (danilo)]

Also, I think we are missing one step. We need to confirm there is an 'accept' link in the license, and we want to simulate clicking it. I am not sure how best to do that, but I am sure you can figure something out (we can simply check if the URL there is absolute or relative, and then browse to it).

[James Tunnicliffe (dooferlad)]

On 16 May 2012 11:34, Данило Шеган <email address hidden> wrote:
> Also, I think we are missing one step.  We need to confirm there is an
> 'accept' link in the license, and we want to simulate clicking it.  I am
> not sure how best to do that, but I am sure you can figure something out
> (we can simply check if the URL there is absolute or relative, and then
> browse to it).

Finding and clicking the accept link is exactly what the download
script does :-)

--
James Tunnicliffe

[Данило Шеган (danilo)]

Right, but this is about proving that this works from a users' perspective, not that we've got code which can get through the click-through.

[James Tunnicliffe (dooferlad)]

True, though my point was that the script emulates a user by finding
the link and following it. It may be useful in this case because it
would prove that the link exists and that clicking on it allows the
user to download the file they are expecting.

[Данило Шеган (danilo)]

Right, understood. However, since this is supposed to be a user-readable (and repeatable) test plan, I'd rather if all the steps are clear from the actual doctest.

[Stevan Radaković (stevanr)]

> True, though my point was that the script emulates a user by finding
> the link and following it. It may be useful in this case because it
> would prove that the link exists and that clicking on it allows the
> user to download the file they are expecting.

Imho, Danilo has the point, but James' script allows me to test this with get_protected_file method. It emulates a click to the accept license link and then gets the file requested. I'll use it in my browser class and that's it.

[Данило Шеган (danilo)]

This looks good. It would be nice to test for 404 errors ending up on the same URL, but definitely not for this branch. At this time, this is more than good enough.

We'll probably want to decouple these tests from the automated tests since they need to run at different times (most of them before landing, these tests after deployment).

review: Approve

[lp:~stevanr/linaro-license-protection/production-integration-tests]

81. By Stevan Radaković on 2012-05-16

PEP8 changes.

82. By Stevan Radaković on 2012-05-16

Add new failing test for link in platform directory.

=== added directory 'docs'
=== added file 'docs/releases.txt'
--- docs/releases.txt	1970-01-01 00:00:00 +0000
+++ docs/releases.txt	2012-05-16 16:49:24 +0000
@@ -0,0 +1,144 @@
+Test releases.linaro.org production server
+===========================================
+
+Navigate to the regular ST-E license-protected file and initiate download
+-------------------------------------------------------------------------
+
+Import class we will use for this test and init browser object.
+
+    >>> from testing.doctest_production_browser import DoctestProductionBrowser
+    >>> browser = DoctestProductionBrowser("http://releases.linaro.org/")
+
+Visiting homepage and check for title.
+
+    >>> print browser.get_content_title()
+    Index of /
+
+Browsing into the latest/android/leb-snowball should work without any
+license popping out.
+
+    >>> browser.browse_to_relative("latest/")
+    >>> print browser.get_content_title()
+    Index of /latest
+    >>> browser.browse_to_relative("android/")
+    >>> print browser.get_content_title()
+    Index of /latest/android
+    >>> browser.browse_to_relative("leb-snowball/")
+    >>> print browser.get_content_title()
+    Index of /latest/android/leb-snowball
+
+Mock the boot.tar.bz2 file download and check the license.
+Check if the ST-E license is encountered.
+
+    >>> browser.browse_to_relative("boot.tar.bz2")
+    >>> print browser.get_license_text()
+    This Agreement is a legal...ST-Ericsson...GOVERNING LAW AND JURISDICTION...
+    ...
+
+Now, emulate clicking on the Accept Licence link which redirects us to the
+download file. Check if the headers of the requested file are in order.
+
+    >>> print browser.accept_license_get_header()
+    Accept-Ranges:...
+    Content-Type: application/x-bzip2...
+    Location: http://releases...snowball...boot.tar.bz2...
+    ...
+
+Now, emulate clicking on the Decline Licence link which redirects us to the
+decline page.
+
+    >>> print browser.decline_license()
+    License has not been accepted
+
+
+Navigate to the regular Samsung license-protected file and initiate download
+----------------------------------------------------------------------------
+
+Browsing back into the /latest/android/leb-origen. It should work
+without any license popping out.
+
+    >>> browser.browse_to_absolute("latest/")
+    >>> print browser.get_content_title()
+    Index of /latest
+    >>> browser.browse_to_relative("android/")
+    >>> print browser.get_content_title()
+    Index of /latest/android
+    >>> browser.browse_to_relative("leb-origen/")
+    >>> print browser.get_content_title()
+    Index of /latest/android/leb-origen
+
+Mock the boot.tar.bz2 file download and check the license.
+Check if the Samsung license is encountered.
+
+    >>> browser.browse_to_relative("boot.tar.bz2")
+    >>> print browser.get_license_text()
+    IMPORTANT...SAMSUNG ELECTRONICS...Entire Agreement...
+    ...
+
+Now, emulate clicking on the Accept Licence link which redirects us to the
+download file. Check if the headers of the requested file are in order.
+
+    >>> print browser.accept_license_get_header()
+    Accept-Ranges:...
+    Content-Type: application/x-bzip2...
+    Location: http://releases...origen...boot.tar.bz2...
+    ...
+
+Now, emulate clicking on the Decline Licence link which redirects us to the
+decline page.
+
+    >>> print browser.decline_license()
+    License has not been accepted
+
+
+Navigate to the non-license-protected file and initiate download
+----------------------------------------------------------------
+
+Browsing back into the latest/android/leb-panda. It should work
+without any license popping out.
+
+    >>> browser.browse_to_absolute("latest/")
+    >>> print browser.get_content_title()
+    Index of /latest
+    >>> browser.browse_to_relative("android/")
+    >>> print browser.get_content_title()
+    Index of /latest/android
+    >>> browser.browse_to_relative("leb-panda/")
+    >>> print browser.get_content_title()
+    Index of /latest/android/leb-panda
+
+Mock the boot.tar.bz2 file download. There should not be any
+license encountered.
+
+    >>> browser.browse_to_relative("boot.tar.bz2")
+    >>> print browser.get_unprotected_file_header()
+    Accept-Ranges:...
+    Content-Type: application/x-bzip2...
+    ...
+
+
+Try accessing the leb-snowball link in platform latest android dir
+------------------------------------------------------------------
+
+Browsing back into the platform/latest/android/latest. It should work
+without any license popping out.
+
+    >>> browser.browse_to_absolute("platform/")
+    >>> print browser.get_content_title()
+    Index of /platform
+    >>> browser.browse_to_relative("latest/")
+    >>> print browser.get_content_title()
+    Index of /platform/latest
+    >>> browser.browse_to_relative("android/")
+    >>> print browser.get_content_title()
+    Index of /platform/latest/android
+    >>> browser.browse_to_relative("latest/")
+    >>> print browser.get_content_title()
+    Index of /platform/latest/android/latest
+
+
+Now try opening the leb-snowball link.
+
+    >>> browser.browse_to_relative("leb-snowball/")
+    >>> print browser.get_content_title()
+    Index of /platform/latest/android/latest/leb-snowball

=== added file 'docs/snapshots.txt'
--- docs/snapshots.txt	1970-01-01 00:00:00 +0000
+++ docs/snapshots.txt	2012-05-16 16:49:24 +0000
@@ -0,0 +1,168 @@
+Test snapshots.linaro.org production server
+===========================================
+
+Navigate to the regular ST-E license-protected file and initiate download
+-------------------------------------------------------------------------
+
+Import class we will use for this test and init browser object.
+
+    >>> from testing.doctest_production_browser import DoctestProductionBrowser
+    >>> browser = DoctestProductionBrowser("http://snapshots.linaro.org/")
+
+Visiting homepage and check for title.
+
+    >>> print browser.get_content_title()
+    Index of /
+
+Browsing into the android/~linaro-android/*snowball* should work without any
+license popping out.
+
+    >>> browser.browse_to_relative("android/")
+    >>> print browser.get_content_title()
+    Index of /android
+    >>> browser.browse_to_relative("~linaro-android/")
+    >>> print browser.get_content_title()
+    Index of /android/~linaro-android
+    >>> browser.browse_to_next("snowball")
+    >>> print browser.get_content_title()
+    Index of /android/~linaro-android/...snowball...
+
+Go to build number page. We don't know which are the build numbers so we
+will visit the first directory link available. Next, go to target, then product
+then snowball links, respectively.
+
+    >>> browser.browse_to_next("")
+    >>> print browser.get_content_title()
+    Index of /android/~linaro-android/...snowball...
+    >>> browser.browse_to_relative("target/")
+    >>> print browser.get_content_title()
+    Index of /android/~linaro-android/...snowball...target
+    >>> browser.browse_to_relative("product/")
+    >>> print browser.get_content_title()
+    Index of /android/~linaro-android/...snowball...target...product...
+    >>> browser.browse_to_relative("snowball/")
+    >>> print browser.get_content_title()
+    Index of /android/~linaro-android/...snowball...product...snowball...
+
+Finally, mock the boot.tar.bz2 file download and check the license.
+Check if the ST-E license is encountered.
+
+    >>> browser.browse_to_relative("boot.tar.bz2")
+    >>> print browser.get_license_text()
+    This Agreement is a legal...ST-Ericsson...GOVERNING LAW AND JURISDICTION...
+    ...
+
+Now, emulate clicking on the Accept Licence link which redirects us to the
+download file. Check if the headers of the requested file are in order.
+
+    >>> print browser.accept_license_get_header()
+    Accept-Ranges:...
+    Content-Type: application/x-bzip2...
+    Location: http://snapshots...snowball...boot.tar.bz2...
+    ...
+
+Now, emulate clicking on the Decline Licence link which redirects us to the
+decline page.
+
+    >>> print browser.decline_license()
+    License has not been accepted
+
+
+Navigate to the regular Samsung license-protected file and initiate download
+----------------------------------------------------------------------------
+
+Browsing back into the android/~linaro-android/*origen*. It should work
+without any license popping out.
+
+    >>> browser.browse_to_absolute("android/")
+    >>> print browser.get_content_title()
+    Index of /android
+    >>> browser.browse_to_relative("~linaro-android/")
+    >>> print browser.get_content_title()
+    Index of /android/~linaro-android
+    >>> browser.browse_to_next("origen")
+    >>> print browser.get_content_title()
+    Index of /android/~linaro-android/...origen...
+
+Go to build number page. We don't know which are the build numbers so we
+will visit the first directory link available. Next, go to target, then product
+then origen links, respectively.
+
+    >>> browser.browse_to_next("")
+    >>> print browser.get_content_title()
+    Index of /android/~linaro-android/...origen...
+    >>> browser.browse_to_relative("target/")
+    >>> print browser.get_content_title()
+    Index of /android/~linaro-android/...origen...target
+    >>> browser.browse_to_relative("product/")
+    >>> print browser.get_content_title()
+    Index of /android/~linaro-android/...origen...target...product...
+    >>> browser.browse_to_relative("origen/")
+    >>> print browser.get_content_title()
+    Index of /android/~linaro-android/...origen...product...origen...
+
+Finally, mock the boot.tar.bz2 file download and check the license.
+Check if the Samsung license is encountered.
+
+    >>> browser.browse_to_relative("boot.tar.bz2")
+    >>> print browser.get_license_text()
+    IMPORTANT...SAMSUNG ELECTRONICS...Entire Agreement...
+    ...
+
+Now, emulate clicking on the Accept Licence link which redirects us to the
+download file. Check if the headers of the requested file are in order.
+
+    >>> print browser.accept_license_get_header()
+    Accept-Ranges:...
+    Content-Type: application/x-bzip2...
+    Location: http://snapshots...origen...boot.tar.bz2...
+    ...
+
+Now, emulate clicking on the Decline Licence link which redirects us to the
+decline page.
+
+    >>> print browser.decline_license()
+    License has not been accepted
+
+
+Navigate to the non-license-protected file and initiate download
+----------------------------------------------------------------
+
+Browsing back into the android/~linaro-android/*panda*. It should work
+without any license popping out.
+
+    >>> browser.browse_to_absolute("android/")
+    >>> print browser.get_content_title()
+    Index of /android
+    >>> browser.browse_to_relative("~linaro-android/")
+    >>> print browser.get_content_title()
+    Index of /android/~linaro-android
+    >>> browser.browse_to_next("panda")
+    >>> print browser.get_content_title()
+    Index of /android/~linaro-android/...panda...
+
+Go to build number page. We don't know which are the build numbers so we
+will visit the first directory link available. Next, go to target, then product
+then pandaboard links, respectively.
+
+    >>> browser.browse_to_next("")
+    >>> print browser.get_content_title()
+    Index of /android/~linaro-android/...panda...
+    >>> browser.browse_to_relative("target/")
+    >>> print browser.get_content_title()
+    Index of /android/~linaro-android/...panda...target
+    >>> browser.browse_to_relative("product/")
+    >>> print browser.get_content_title()
+    Index of /android/~linaro-android/...panda...target...product...
+    >>> browser.browse_to_next("panda")
+    >>> print browser.get_content_title()
+    Index of /android/~linaro-android/...panda...product...panda...
+
+Finally, mock the boot.tar.bz2 file download. There should not be any
+license encountered.
+
+    >>> browser.browse_to_relative("boot.tar.bz2")
+    >>> print browser.get_unprotected_file_header()
+    Accept-Ranges:...
+    Content-Type: application/x-bzip2...
+    ...

=== modified file 'testing/__init__.py'
--- testing/__init__.py	2012-05-11 14:02:26 +0000
+++ testing/__init__.py	2012-05-16 16:49:24 +0000
@@ -1,9 +1,11 @@
 import os
 import unittest
+import doctest
 
 from testing.test_click_through_license import *
 from testing.test_publish_to_snapshots import *
 
+
 def test_suite():
     module_names = [
         'testing.test_click_through_license.TestLicense',
@@ -12,5 +14,9 @@
         ]
     loader = unittest.TestLoader()
     suite = loader.loadTestsFromNames(module_names)
+    for filename in os.listdir("docs/"):
+        suite.addTest(doctest.DocFileSuite(
+            'docs/' + filename, module_relative=False,
+            optionflags=doctest.ELLIPSIS)
+            )
     return suite
-

=== added file 'testing/doctest_production_browser.py'
--- testing/doctest_production_browser.py	1970-01-01 00:00:00 +0000
+++ testing/doctest_production_browser.py	2012-05-16 16:49:24 +0000
@@ -0,0 +1,172 @@
+from BeautifulSoup import BeautifulSoup
+
+from license_protected_file_downloader import LicenseProtectedFileFetcher
+
+
+class EmptyDirectoryException(Exception):
+    ''' Directory at the current URL is empty. '''
+
+
+class NoLicenseException(Exception):
+    ''' No license protecting the file. '''
+
+
+class UnexpectedLicenseException(Exception):
+    ''' License protecting non-licensed the file. '''
+
+
+class DoctestProductionBrowser():
+    """Doctest production testing browser class."""
+
+    def __init__(self, host_address):
+        self.host_address = host_address
+        self.current_url = host_address
+        self.fetcher = LicenseProtectedFileFetcher()
+
+    def is_dir(self, link):
+        """Check if the link is a directory."""
+        return link[-1] == "/"
+
+    def get_header(self):
+        """Get header from the current url."""
+        return self.parse_header(self.fetcher.get_headers(self.current_url))
+
+    def get_license_text(self):
+        """Get license from the current URL if it redirects to license."""
+        license = self.fetcher.get_or_return_license(self.current_url)
+        if license[0]:
+            return license[0]
+        else:
+            raise NoLicenseException("License expected here.")
+
+    def get_unprotected_file_header(self):
+        """Get headers from unprotected file."""
+        page = self.fetcher.get_or_return_license(self.current_url)
+        # Check if license with accept and decline links is returned.
+        if len(page) == 3:
+            raise UnexpectedLicenseException("License not expected here.")
+        else:
+            return self.parse_header(self.fetcher.header)
+
+    def get_content(self):
+        """Get contents from the current url."""
+        return self.fetcher.get(self.current_url)
+
+    def get_content_title(self):
+        """Get content title from the current url."""
+        return self.get_title(self.fetcher.get(self.current_url))
+
+    def get_header_when_redirected(self):
+        """Get header when the client is redirected to the license."""
+        self.fetcher.get(self.current_url)
+        return self.parse_header(self.fetcher.header)
+
+    def accept_license_get_header(self):
+        """Accept license and get header of the file it redirects to."""
+        license = self.fetcher.get_or_return_license(self.current_url)
+        # Second element in result is the accept link.
+        if license[1]:
+            self.fetcher.get_protected_file(license[1], self.current_url)
+            return self.parse_header(self.fetcher.header)
+        else:
+            raise NoLicenseException("License expected here.")
+
+    def decline_license(self):
+        """Decline license. Return title of the page."""
+        return self.get_title(
+            self.fetcher.get(self.current_url, accept_license=False)
+            )
+
+    def parse_header(self, header):
+        """Formats headers from dict form to the multi-line string."""
+        header_str = ""
+        for key in sorted(header.iterkeys()):
+            header_str += "%s: %s\n" % (key, header[key])
+        return header_str
+
+    def get_title(self, html):
+        soup = BeautifulSoup(html)
+        titles_all = soup.findAll('title')
+        if len(titles_all) > 0:
+            return titles_all[0].contents[0]
+        else:
+            return ""
+
+    def browse_to_relative(self, path):
+        """Change current url relatively."""
+        self.current_url += path
+
+    def browse_to_absolute(self, path):
+        """Change current url to specified path."""
+        self.current_url = self.host_address + path
+
+    def browse_to_next(self, condition):
+        """Browse to next dir/build file that matches condition.
+
+        Set the current URL to to match the condition among the
+        links in the current page with priority to build files.
+        If there's no match, set link to build file if present.
+        Otherwise, set link to first directory present.
+        """
+        links = self.find_links(self.get_content())
+        link = self.find_link_with_condition(links, condition)
+        if not link:
+            # No link matching condition, get first build in list.
+            link = self.find_build_tar_bz2(links)
+        if not link:
+            # Still no link, just get first dir in list.
+            link = self.find_directory(links)
+        if not link:
+            # We found page with no directories nor builds.
+            raise EmptyDirectoryException("Directory is empty.")
+
+        self.browse_to_relative(link)
+
+    def find_links(self, html):
+        """Return list of links on the page with special conditions.
+
+        Return all links below the "Parent directory" link.
+        Return whole list if there is no such link.
+        """
+        soup = BeautifulSoup(html)
+        links_all = soup.findAll('a')
+        had_parent = False
+        links = []
+        for link in links_all:
+            if had_parent:
+                links.append(link.get("href"))
+            if link.contents[0] == "Parent Directory":
+                had_parent = True
+
+        if had_parent:
+            return links
+        else:
+            return [each.get('href') for each in links_all]
+
+    def find_link_with_condition(self, links, condition):
+        """Finds a link which satisfies the condition.
+
+        Condition is actually to contain the string from the list.
+        Build files (which end in .tar.bz2) have the priority.
+        """
+        for link in links:
+            if condition in link and link[-7:] == "tar.bz2":
+                return link
+        for link in links:
+            if condition in link:
+                return link
+        return None
+
+    def find_directory(self, links):
+        """Finds a directory among list of links."""
+        for link in links:
+            if self.is_dir(link):
+                return link
+        return None
+
+    def find_build_tar_bz2(self, links):
+        """Finds a file list of links which ends in tar.bz2."""
+        for link in links:
+            if link[-7:] == "tar.bz2":
+                return link
+        return None

=== modified file 'testing/license_protected_file_downloader.py'
--- testing/license_protected_file_downloader.py	2012-05-11 08:32:52 +0000
+++ testing/license_protected_file_downloader.py	2012-05-16 16:49:24 +0000
@@ -8,6 +8,7 @@
 import html2text
 from BeautifulSoup import BeautifulSoup
 
+
 class LicenseProtectedFileFetcher:
     """Fetch a file from the web that may be protected by a license redirect
 
@@ -124,7 +125,8 @@
 
         return self.body
 
-    def get(self, url, file_name=None, ignore_license=False, accept_license=True):
+    def get(self, url, file_name=None, ignore_license=False,
+            accept_license=True):
         """Fetch the requested URL, accepting licenses
 
         Fetches the file at url. If a redirect is encountered, it is
@@ -241,7 +243,7 @@
 
         # Only buffer first 1MB of body. This should be plenty for anything
         # we wish to parse internally.
-        if len(self.body) < 1024*1024*1024:
+        if len(self.body) < 1024 * 1024 * 1024:
             # XXX Would be nice to stop keeping the file in RAM at all and
             # passing large buffers around. Perhaps only keep in RAM if
             # file_name == None? (used for getting directory listings
@@ -260,6 +262,7 @@
         """Wrapper to close curl - this will allow curl to write out cookies"""
         self.curl.close()
 
+
 def main():
     """Download file specified on command line"""
     parser = argparse.ArgumentParser(description="Download a file, accepting "