Merge lp:~vila/bzr/integration into lp:bzr

[Vincent Ladeuil]

Merge 2.7 into trunk including fixes for bugs #1622039, #1644003, #1579093 and #1645017.

Merge 2.7 into trunk including fixes for bugs #1644003, #1579093 and #1645017.

[Vincent Ladeuil (vila)]

Re-sync trunk with 2.7.

review: Approve

[bzr PQM (bzr-pqm)]

The attempt to merge lp:~vila/bzr/integration into lp:bzr failed. Below is the output from the failed tests.

python tools/rst2html.py --link-stylesheet --footnote-references=superscript --halt=warning --stylesheet=../../default.css doc/en/tutorials/tutorial.txt "doc/en/tutorials/tutorial.html"
python tools/rst2html.py --link-stylesheet --footnote-references=superscript --halt=warning --stylesheet=../../default.css doc/en/tutorials/using_bazaar_with_launchpad.txt "doc/en/tutorials/using_bazaar_with_launchpad.html"
python tools/rst2html.py --link-stylesheet --footnote-references=superscript --halt=warning --stylesheet=../../default.css doc/en/tutorials/centralized_workflow.txt "doc/en/tutorials/centralized_workflow.html"
python tools/rst2html.py --link-stylesheet --footnote-references=superscript --halt=warning --stylesheet=../../default.css doc/ru/tutorials/centralized_workflow.txt "doc/ru/tutorials/centralized_workflow.html"
python tools/rst2html.py --link-stylesheet --footnote-references=superscript --halt=warning --stylesheet=../../default.css doc/ru/tutorials/using_bazaar_with_launchpad.txt "doc/ru/tutorials/using_bazaar_with_launchpad.html"
python tools/rst2html.py --link-stylesheet --footnote-references=superscript --halt=warning --stylesheet=../../default.css doc/ru/tutorials/tutorial.txt "doc/ru/tutorials/tutorial.html"
python tools/rst2html.py --link-stylesheet --footnote-references=superscript --halt=warning --stylesheet=../../default.css doc/ja/tutorials/tutorial.txt "doc/ja/tutorials/tutorial.html"
python tools/rst2html.py --link-stylesheet --footnote-references=superscript --halt=warning --stylesheet=../../default.css doc/ja/tutorials/using_bazaar_with_launchpad.txt "doc/ja/tutorials/using_bazaar_with_launchpad.html"
python tools/rst2html.py --link-stylesheet --footnote-references=superscript --halt=warning --stylesheet=../../default.css doc/ja/tutorials/centralized_workflow.txt "doc/ja/tutorials/centralized_workflow.html"
python tools/rst2html.py --link-stylesheet --footnote-references=superscript --halt=warning --stylesheet=../../default.css doc/en/mini-tutorial/index.txt "doc/en/mini-tutorial/index.html"
python tools/rst2html.py --link-stylesheet --footnote-references=superscript --halt=warning --stylesheet=../../default.css doc/ja/mini-tutorial/index.txt "doc/ja/mini-tutorial/index.html"
python tools/rst2html.py --link-stylesheet --footnote-references=superscript --halt=warning --stylesheet=../../default.css doc/ru/mini-tutorial/index.txt "doc/ru/mini-tutorial/index.html"
python tools/rst2html.py --link-stylesheet --footnote-references=superscript --halt=warning --stylesheet=../../default.css doc/es/mini-tutorial/index.txt "doc/es/mini-tutorial/index.html"
python tools/rst2html.py --link-stylesheet --footnote-references=superscript --halt=warning --stylesheet=../../default.css doc/en/user-guide/index-plain.txt doc/en/user-guide/index-plain.html
python tools/rst2html.py --link-stylesheet --footnote-references=superscript --halt=warning --stylesheet=../../default.css doc/ja/user-guide/index-plain.txt "doc/ja/user-guide/index-plain.html"
python tools/rst2html.py --link-stylesheet --footnote-references=superscript --halt=warning ...

[Vincent Ladeuil (vila)]

15:59:23 O: ======================================================================
15:59:23 O: ERROR: bzrlib.tests.per_transport.TransportTests.test_copy_to(SFTPTransport,SFTPHomeDirServer)
15:59:23 O: ----------------------------------------------------------------------
15:59:23 O: Traceback (most recent call last):
15:59:23 O: testtools.testresult.real._StringException: lost connection during test 'bzrlib.tests.per_transport.TransportTests.test_copy_to(SFTPTransport,SFTPHomeDirServer)'

Progress: on more debian reported test failure :-}

=== modified file '.bzrignore'
--- .bzrignore	2012-08-07 06:32:51 +0000
+++ .bzrignore	2017-01-17 15:27:25 +0000
@@ -7,6 +7,7 @@
 CHANGELOG
 # generated documents
 bzr.1
+MANIFEST
 ./doc/*.html
 ./doc/*/_build/
 ./doc/*/Makefile

=== modified file 'MANIFEST.in'
--- MANIFEST.in	2010-01-13 16:23:07 +0000
+++ MANIFEST.in	2017-01-17 15:27:25 +0000
@@ -1,3 +1,12 @@
-include bzr README setup.py
+include bzr README setup.py COPYING.txt
+# FIXME: Not needed, remove after 2.7.0 -- vila 2016-02-07
+include BRANCH.TODO bzr.ico INSTALL Makefile MANIFEST.in NEWS profile_imports.py README_BDIST_RPM .rsyncexclude .testr.conf TODO tools/bzr_epydoc tools/packaging/lp-upload-release tools/subunit-sum bzrlib/plugins/news_merge/README bzrlib/plugins/po_merge/README bzrlib/tests/ssl_certs/ca.key bzrlib/tests/ssl_certs/server.csr bzrlib/plugins/grep/.bzrignore bzrlib/plugins/grep/NEWS
+# bzr export still create some empty dirs that need to be removed
+# bzrlib/plugins/weave_fmt/tests/ bzrlib/store/revision/ doc/ja/_templates/ man1/ man1
+recursive-include tools/win32 *
 recursive-include bzrlib *.py *.pyx *.pxd *.txt *.c *.h
 recursive-include tools *.py *.sh
+recursive-include apport *
+recursive-include contrib *
+recursive-include doc *
+recursive-include po *

=== modified file 'Makefile'
--- Makefile	2016-04-03 16:32:31 +0000
+++ Makefile	2017-01-17 15:27:25 +0000
@@ -1,4 +1,4 @@
-# Copyright (C) 2005-2012, 2016 Canonical Ltd
+# Copyright (C) 2005-2012, 2016, 2017 Canonical Ltd
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -454,9 +454,6 @@
 .PHONY: dist check-dist-tarball
 
 # build a distribution source tarball
-#
-# this method of copying the pyrex generated files is a bit ugly; it would be
-# nicer to generate it from distutils.
 dist: 
 	version=`./bzr version --short` && \
 	echo Building distribution of bzr $$version && \
@@ -465,10 +462,8 @@
 	tarball=$$PWD/../bzr-$$version.tar.gz && \
 	$(MAKE) clean && \
 	$(MAKE) && \
-	bzr export $$expdir && \
-	cp bzrlib/*.c bzrlib/*.h $$expdir/bzrlib/. && \
-	tar cfz $$tarball -C $$expbasedir bzr-$$version && \
-	gpg --detach-sign $$tarball && \
+	$(PYTHON) setup.py sdist -d $$PWD/.. && \
+	gpg --detach-sign --armor $$tarball && \
 	rm -rf $$expbasedir
 
 # run all tests in a previously built tarball

=== modified file 'bzr'
--- bzr	2016-02-01 19:26:41 +0000
+++ bzr	2017-01-17 15:27:25 +0000
@@ -1,6 +1,6 @@
 #! /usr/bin/env python
 
-# Copyright (C) 2005-2013, 2016 Canonical Ltd
+# Copyright (C) 2005-2013, 2016, 2017 Canonical Ltd
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

=== modified file 'bzrlib/__init__.py'
--- bzrlib/__init__.py	2016-02-01 19:26:41 +0000
+++ bzrlib/__init__.py	2017-01-17 15:27:25 +0000
@@ -1,4 +1,4 @@
-# Copyright (C) 2005-2013, 2016 Canonical Ltd
+# Copyright (C) 2005-2013, 2016, 2017 Canonical Ltd
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

=== modified file 'bzrlib/lazy_regex.py'
--- bzrlib/lazy_regex.py	2011-12-19 13:23:58 +0000
+++ bzrlib/lazy_regex.py	2017-01-17 15:27:25 +0000
@@ -1,4 +1,4 @@
-# Copyright (C) 2006 Canonical Ltd
+# Copyright (C) 2006, 2008-2011, 2017 Canonical Ltd
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -131,3 +131,13 @@
     raise AssertionError(
         "re.compile has already been overridden as lazy_compile, but this would" \
         " cause infinite recursion")
+
+
+# re.finditer get confused if it receives a LazyRegex
+if getattr(re, 'finditer', None is not None):
+    def finditer_public(pattern, string, flags=0):
+        if isinstance(pattern, LazyRegex):
+            return pattern.finditer(string)
+        else:
+            return _real_re_compile(pattern, flags).finditer(string)
+re.finditer = finditer_public

=== modified file 'bzrlib/tests/test_diff.py'
--- bzrlib/tests/test_diff.py	2016-02-01 18:06:32 +0000
+++ bzrlib/tests/test_diff.py	2017-01-17 15:27:25 +0000
@@ -1,4 +1,4 @@
-# Copyright (C) 2005-2012, 2014, 2016 Canonical Ltd
+# Copyright (C) 2005-2012, 2014, 2016, 2017 Canonical Ltd
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -313,8 +313,6 @@
         pipe = subprocess.Popen(cmd, stdout=subprocess.PIPE,
                                      stdin=subprocess.PIPE)
         out, err = pipe.communicate()
-        # Diff returns '2' on Binary files.
-        self.assertEqual(2, pipe.returncode)
         # We should output whatever diff tells us, plus a trailing newline
         self.assertEqual(out.splitlines(True) + ['\n'], lines)
 

=== modified file 'bzrlib/tests/test_http.py'
--- bzrlib/tests/test_http.py	2016-02-01 18:06:32 +0000
+++ bzrlib/tests/test_http.py	2017-01-17 15:27:25 +0000
@@ -1,4 +1,4 @@
-# Copyright (C) 2005-2012, 2015, 2016 Canonical Ltd
+# Copyright (C) 2005-2012, 2015, 2016, 2017 Canonical Ltd
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -260,6 +260,16 @@
         self.assertEqual('basic', scheme)
         self.assertEqual('realm="Thou should not pass"', remainder)
 
+    def test_build_basic_header_with_long_creds(self):
+        handler = _urllib2_wrappers.BasicAuthHandler()
+        user = 'user' * 10  # length 40
+        password = 'password' * 5  # length 40
+        header = handler.build_auth_header(
+            dict(user=user, password=password), None)
+        # https://bugs.launchpad.net/bzr/+bug/1606203 was caused by incorrectly
+        # creating a header value with an embedded '\n'
+        self.assertFalse('\n' in header)
+
     def test_basic_extract_realm(self):
         scheme, remainder = self.parse_header(
             'Basic realm="Thou should not pass"',
@@ -1273,6 +1283,8 @@
             self.no_proxy_host = self.server_host_port
         # The secondary server is the proxy
         self.proxy_url = self.get_secondary_url()
+        if self._testing_pycurl():
+            self.proxy_url = self.proxy_url.replace('+pycurl', '')
 
     def _testing_pycurl(self):
         # TODO: This is duplicated for lots of the classes in this file
@@ -1853,7 +1865,10 @@
                                   ])
 
     def get_user_transport(self, user, password):
-        self.overrideEnv('all_proxy', self.get_user_url(user, password))
+        proxy_url = self.get_user_url(user, password)
+        if self._testing_pycurl():
+            proxy_url = proxy_url.replace('+pycurl', '')
+        self.overrideEnv('all_proxy', proxy_url)
         return TestAuth.get_user_transport(self, user, password)
 
     def test_empty_pass(self):

=== modified file 'bzrlib/transport/http/_pycurl.py'
--- bzrlib/transport/http/_pycurl.py	2011-12-19 13:23:58 +0000
+++ bzrlib/transport/http/_pycurl.py	2017-01-17 15:27:25 +0000
@@ -1,4 +1,4 @@
-# Copyright (C) 2006-2010 Canonical Ltd
+# Copyright (C) 2006-2011, 2017 Canonical Ltd
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -85,6 +85,9 @@
     """
     return pycurl.__dict__.get(symbol, default)
 
+# Yes, weird but returned on weird http error (invalid status line)
+CURLE_FTP_WEIRD_SERVER_REPLY = _get_pycurl_errcode(
+    'E_FTP_WEIRD_SERVER_REPLY', 8)
 CURLE_COULDNT_CONNECT = _get_pycurl_errcode('E_COULDNT_CONNECT', 7)
 CURLE_COULDNT_RESOLVE_HOST = _get_pycurl_errcode('E_COULDNT_RESOLVE_HOST', 6)
 CURLE_COULDNT_RESOLVE_PROXY = _get_pycurl_errcode('E_COULDNT_RESOLVE_PROXY', 5)
@@ -397,6 +400,7 @@
             if e[0] in (CURLE_COULDNT_RESOLVE_HOST,
                         CURLE_COULDNT_RESOLVE_PROXY,
                         CURLE_COULDNT_CONNECT,
+                        CURLE_FTP_WEIRD_SERVER_REPLY,
                         CURLE_GOT_NOTHING,
                         CURLE_SSL_CACERT,
                         CURLE_SSL_CACERT_BADFILE,

=== modified file 'bzrlib/transport/http/_urllib2_wrappers.py'
--- bzrlib/transport/http/_urllib2_wrappers.py	2016-01-31 12:55:31 +0000
+++ bzrlib/transport/http/_urllib2_wrappers.py	2017-01-17 15:27:25 +0000
@@ -1,4 +1,4 @@
-# Copyright (C) 2006-2013, 2016 Canonical Ltd
+# Copyright (C) 2006-2013, 2016, 2017 Canonical Ltd
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -48,6 +48,7 @@
 # actual code more or less do that, tests should be written to
 # ensure that.
 
+import base64
 import errno
 import httplib
 import os
@@ -1491,7 +1492,7 @@
 
     def build_auth_header(self, auth, request):
         raw = '%s:%s' % (auth['user'], auth['password'])
-        auth_header = 'Basic ' + raw.encode('base64').strip()
+        auth_header = 'Basic ' + base64.b64encode(raw)
         return auth_header
 
     def extract_realm(self, header_value):

=== modified file 'bzrlib/transport/sftp.py'
--- bzrlib/transport/sftp.py	2016-01-21 17:48:07 +0000
+++ bzrlib/transport/sftp.py	2017-01-17 15:27:25 +0000
@@ -1,4 +1,4 @@
-# Copyright (C) 2005-2011, 2016 Canonical Ltd
+# Copyright (C) 2005-2011, 2016, 2017 Canonical Ltd
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -410,8 +410,9 @@
         try:
             path = self._remote_path(relpath)
             f = self._get_sftp().file(path, mode='rb')
+            size = f.stat().st_size
             if self._do_prefetch and (getattr(f, 'prefetch', None) is not None):
-                f.prefetch()
+                f.prefetch(size)
             return f
         except (IOError, paramiko.SSHException), e:
             self._translate_io_exception(e, path, ': error retrieving',

=== modified file 'doc/developers/releasing.txt'
--- doc/developers/releasing.txt	2013-08-05 12:16:25 +0000
+++ doc/developers/releasing.txt	2017-01-17 15:27:25 +0000
@@ -513,7 +513,7 @@
 ----------------------------
 
 #. Post to the ``[e-mail]`` and
-   ``[e-mail]`` lists, saying that the source has
+   ``[e-mail]`` lists, saying that the source has
    been frozen. Be extra clear that this is only a *source* release targeted
    at packagers and installer builders (see
    <https://bugs.launchpad.net/launchpad/+bug/645084>).  This is the cue for
@@ -551,12 +551,12 @@
    <http://doc.bazaar.canonical.com>.  It should be automatically build when
    the branch is created, by a cron script ``update-bzr-docs`` on
    ``escudero``. When the first release is created in a new series, a branch
-   needs to be created on zhongshan::
+   needs to be created on ``escudero``::
 
-     ssh zhongshan.canonical.com
+     ssh escudero.canonical.com
      sudo -u bzr-web -s
      cd /srv/doc.bazaar.canonical.com/
-     bzr branch http://bazaar.launchpad.net/~bzr-pqm/bzr/2.6 bzr/bzr.2.6
+     bzr branch lp:bzr/2.7 bzr/bzr.2.7
 
    And the ``bzr/bin/update-bzr-docs`` script needs to refer to it.
 
@@ -640,17 +640,48 @@
 #. Update `<http://en.wikipedia.org/wiki/Bazaar_(software)>`_ -- this should
    be done for the stable and beta releases.
 
-#. Update the python package index: <http://pypi.python.org/pypi/bzr> - best
-   done by running ::
-
-       python setup.py register
+#. Update the python package index: <http://pypi.python.org/pypi/bzr>
+
+   From the tarball created and tested earlier ::
+
+     twine upload -s ../bzr-2.7.0.tar.gz
 
    Remember to check the results afterward -- this should be done for
    stable releases but not for beta releases nor SRUs.
 
-   To be able to register the release you must create an account on
-   <http://pypi.python.org/pypi> and have one of the existing owners of
-   the project add you to the group.
+   The ``twine`` command is provided by the ``twine`` package on Debian and
+   Ubuntu.
+
+   If in doubt, you can test with <https://testpypi.python.org/pypi> where
+   you can register yourself, register the bzr project and upload (and then
+   remove the upload and delete the project so a new release manager can
+   enjoy the same facilities). If the project already exists, have one of
+   the existing owners of the project add you to the group.
+
+   To use the pypi test site, you need to record your credentials for an
+   existing user (the registration process is manual and starts at
+   <https://testpypi.python.org/pypi?%3Aaction=register_form>.
+
+   Once registered, you can record your crdentials in your ``~/pypirc`` file
+   with ::
+
+     [testpypi]
+     username:<login on testpypi>
+     password:<password on testpypi>
+     repository = https://testpypi.python.org/pypi
+
+   Registering the bzr project if doesn't exist is achieved with::
+
+     python setup.py -r https://testpypi.python.org/pypi register
+
+  Uploading is done with::
+
+    twine upload -r testpypi -s ../bzr-2.7.0.tar.gz
+
+   To be able to upload the release you must create an account on
+   <http://pypi.python.org/pypi> and have one of the existing owners of the
+   project add you to the group and update your ``[pypi]`` section in your
+   ``~/pypirc`` file.
 
 
 Merging the released code back to trunk

=== modified file 'doc/en/release-notes/bzr-2.7.txt'
--- doc/en/release-notes/bzr-2.7.txt	2016-02-01 18:09:33 +0000
+++ doc/en/release-notes/bzr-2.7.txt	2017-01-17 15:27:25 +0000
@@ -5,6 +5,65 @@
 .. toctree::
    :maxdepth: 1
 
+bzr 2.7.1
+#########
+
+:2.7.1: NOT RELEASED YET
+
+
+External Compatibility Breaks
+*****************************
+
+None.
+
+New Features
+************
+
+None.
+
+Improvements
+************
+
+None.
+
+
+Bug Fixes
+*********
+
+* Complete monkey-patching of re.finditer or LazyRegexps leak.
+  (Vincent Ladeuil, #1644003)
+
+* Cope with paramiko making argument to SFTPFile.prefetch() mandatory.
+  (Jelmer Vernooij, #1579093)
+
+Documentation
+*************
+
+* Clarify pypi uploads for releases. (Vincent Ladeuil, #1542903)
+
+API Changes
+***********
+
+None.
+
+Internals
+*********
+
+None.
+
+Changed Behaviour
+*****************
+
+None.
+
+Testing
+*******
+
+* Recent versions of pycurl started being stricter about the proxy url,
+  using 'http+pycurl:' is not allowed anymore and should just be 'http:'
+  (Vincent Ladeuil, #1645017).
+
+
 bzr 2.7.0
 #########
 

=== modified file 'doc/en/release-notes/bzr-2.8.txt'
--- doc/en/release-notes/bzr-2.8.txt	2016-04-03 16:32:31 +0000
+++ doc/en/release-notes/bzr-2.8.txt	2017-01-17 15:27:25 +0000
@@ -56,10 +56,9 @@
 Testing
 *******
 
-.. Fixes and changes that are only relevant to bzr's test framework and 
-   suite.  This can include new facilities for writing tests, fixes to 
-   spurious test failures and changes to the way things should be tested.
-
+* POSIX mandates diff to exit with status 1 (was 2 previously), bzr didn't
+  rely on that feature but one of the tests did.
+  (Vincent Ladeuil, #1622039)
 
 ..
    vim: tw=74 ft=rst ff=unix