Merge lp:~gary/zc.buildout/betafix6 into lp:zc.buildout

[Gary Poster]

By default, Buildout and the bootstrap script now prefer final versions of Buildout, recipes, and extensions. This was done to try and prevent problems such as those that happened the last time I tried to make a zc.buildout release. See the CHANGES file updates (which also include some notes I should have included in an earlier revision) for how it works; also not that betafix7 has a help document for users, so if you think the mechanism is fine but you want the users to have some better docs, please look at that branch. The logic for figuring out final versions in bootstrap was ripped from easy_install. The bootstrap test had to do some new tricks to exercise the changes. These new tricks revealed that, once the new version of zc.buildout was released, the bootstrap test would have failed. I believe this is now fixed as a side effect of the main effort.

I factored out some bool parsing behavior into a method on the Options object, because I was adding yet another bool and wanted to not copy and paste the code again. Note that I change the behavior of get_bool in betafix7, so it might be better to comment on its approach on that branch.

[Francis J. Lacoste (flacoste)]

Hi Gary,

I don't understand the need for both an option to bootstrap and one for
buildout. What happen if we only use the bootstrap.py option?

Is it that buildout/setuptools will upgrade itself whenever the buildout is
used? Or only the recipes? I think some clarifications of why one would use
the various options is needed to make this clearer.

Otherwise, everything looks good.

review: Needs Information (code)

[Francis J. Lacoste (flacoste)]

OK, after discussion on IRC it does seem that this limitation is what is best acceptable trade-off. There is already duplication of options already (download-cache, eggs) etc.

If it's easy to detect that buildout would be downgraded (boostrap used --accept, but the option is missing in buildout), then I'd suggest adding an error asking the user to either rerun bootstrap without the option or change the buildout.cfg variable.

Otherwise (or in addition) change the documentation of the option to make it clear that one should set the buildout.cfg option when using the bootstrap option otherwise things will get confused.

review: Approve (code)

[lp:~gary/zc.buildout/betafix6]

564. By Gary Poster on 2010-08-07

clarify that --accept-early-release must be accompanied with prefer-final-build-system= false.

[Gary Poster (gary)]

Hey Francis. What do you think of this approach to addressing your concerns?

http://pastebin.ubuntu.com/479699/

Essentially, you only have to use --accept-early-release when calling the bootstrap. This flag is remembered thanks to code scribbled in the buildout script. When you rerun bootstrap without that argument it goes back to the usual behavior.

This is much nicer usability, at the expense of a somewhat odd pattern; however, I think it is defensible since what we are trying to control extends from bootstrapping through normal use.

If you think this is reasonable, I'll need to update CHANGES and make sure the rest of the tests still pass.

[Francis J. Lacoste (flacoste)]

I like that!

[Gary Poster (gary)]

Cool. The patch got on the big side (526 lines) so I made it into another branch based on betafix7: https://code.edge.launchpad.net/~gary/zc.buildout/betafix8/+merge/33077 .

Unmerged revisions

564. By Gary Poster on 2010-08-07

clarify that --accept-early-release must be accompanied with prefer-final-build-system= false.

563. By Gary Poster on 2010-08-06

merge from trunk/previous branches

562. By Gary Poster on 2010-08-06

By default, Buildout and the bootstrap script now prefer final versions of Buildout, recipes, and extensions.

561. By Gary Poster on 2010-08-06

get tests passing for Python 2.7. Also includes Lennart Regebro changes to use the standard library doctest.

560. By Gary Poster on 2010-08-06

small Python 2.6 bugfix and Distribute test clean-ups

559. By Gary Poster on 2010-06-16

fix tests for changes

558. By Gary Poster on 2010-06-15

mimic standard site.py behavior for inclusion of .pth files

557. By Gary Poster on 2010-06-15

eliminate spurious warning if you are using distribute.

556. By Gary Poster on 2010-06-15

add files so releases can be made from a non-SVN checkout

555. By Gary Poster on 2010-06-15

fix virtualenv interaction by identfying broken virtualenv characteristic and reverting to previous behavior in that case.

=== modified file 'CHANGES.txt'
--- CHANGES.txt	2010-08-07 00:12:52 +0000
+++ CHANGES.txt	2010-08-07 00:12:52 +0000
@@ -1,19 +1,52 @@
 Change History
 **************
 
+1.5.0b3 (unreleased)
+====================
+
+New features:
+
+- zc.buildout supports Python 2.7.
+
+- By default, Buildout and the bootstrap script now prefer final versions of
+  Buildout, recipes, and extensions.  This can be changed by setting
+  ``prefer-final-build-system = false`` in your configuration's
+  [buildout] section, and by using the --accept-early-release flag when
+  calling bootstrap.  This will hopefully allow beta releases to be more
+  easily and safely made in the future.  Note that dependencies of your
+  software do not have this behavior: use the pre-existing switch
+  ``prefer-final = true`` to get this behavior.
+
+Bugs fixed:
+
+- You can now again use virtualenv with zc.buildout.  The new features to let
+  buildout be used with a system Python are disabled in this configuration,
+  and the previous script generation behavior (1.4.3) is used, even if
+  the new function ``zc.buildout.easy_install.sitepackage_safe_scripts``
+  is used.
+
+1.5.0b2 (2010-04-29)
+====================
+
+This was a re-release of 1.4.3 in order to keep 1.5.0b1 release from hurting
+workflows that combined virtualenv with zc.buildout.
+
 1.5.0b1 (2010-04-29)
 ====================
 
 New Features:
 
-- zc.buildout supports Python 2.7.
-
 - Added buildout:socket-timout option so that socket timeout can be configured
   both from command line and from config files. (gotcha)
 
 - Buildout can be safely used with a system Python (or any Python with code
-  in site-packages), as long as you use the new z3c.recipe.scripts
-  recipe to generate scripts and interpreters, rather than zc.recipe.egg.
+  in site-packages), as long as you use (1) A fresh checkout, (2) the
+  new bootstrap.py, and (3) recipes that use the new
+  ``zc.buildout.easy_install.sitepackage_safe_scripts`` function to generate
+  scripts and interpreters.  Many recipes will need to be updated to use
+  this new function.  The scripts and interpreters generated by
+  ``zc.recipe.egg`` will continue to use the older function, not safe
+  with system Pythons.  Use the ``z3c.recipe.scripts`` as a replacement.
 
   zc.recipe.egg is still a fully supported, and simpler, way of
   generating scripts and interpreters if you are using a "clean" Python,

=== modified file 'src/zc/buildout/bootstrap.txt'
--- src/zc/buildout/bootstrap.txt	2010-08-07 00:12:52 +0000
+++ src/zc/buildout/bootstrap.txt	2010-08-07 00:12:52 +0000
@@ -47,6 +47,87 @@
     X...
     d  zc.buildout-...egg
 
+The buildout script it has generated is a new-style script, using a
+customized site.py.
+
+    >>> buildout_script = join(sample_buildout, 'bin', 'buildout')
+    >>> if sys.platform.startswith('win'):
+    ...     buildout_script += '-script.py'
+    >>> print open(buildout_script).read() # doctest: +ELLIPSIS
+    #...
+    <BLANKLINE>
+    import sys
+    sys.path[0:0] = [
+        '/sample/parts/buildout',
+        ]
+    <BLANKLINE>
+    <BLANKLINE>
+    import os
+    path = sys.path[0]
+    if os.environ.get('PYTHONPATH'):
+        path = os.pathsep.join([path, os.environ['PYTHONPATH']])
+    os.environ['PYTHONPATH'] = path
+    import site # imports custom buildout-generated site.py
+    <BLANKLINE>
+    import zc.buildout.buildout
+    <BLANKLINE>
+    if __name__ == '__main__':
+        zc.buildout.buildout.main()
+    <BLANKLINE>
+
+The bootstrap process prefers final versions of zc.buildout, so it has
+selected the (generated-locally) 99.99 egg rather than the also-available
+100.0b1 egg.  We can see that in the buildout script's site.py.
+
+    >>> buildout_site_py = join(
+    ...     sample_buildout, 'parts', 'buildout', 'site.py')
+    >>> print open(buildout_site_py).read() # doctest: +ELLIPSIS
+    "...
+        buildout_paths = [
+            '/sample/eggs/setuptools-...egg',
+            '/sample/eggs/zc.buildout-99.99-pyN.N.egg'
+            ]
+    ...
+
+If you want to accept early releases of zc.buildout, you either need to
+specify an explicit version (using --version here and specifying the
+version in the buildout configuration file using the
+``buildout-version`` option or the ``versions`` option) or specify that you
+accept early releases.
+
+You accept early releases by using ``--accept-early-release`` on the
+bootstrap script and specifying ``prefer-final-build-system = false`` in the
+buildout configuration file.  You must do both.
+
+Here's an example.
+
+    >>> write('buildout.cfg',
+    ... '''
+    ... [buildout]
+    ... parts =
+    ... prefer-final-build-system = false
+    ... ''')
+    >>> ignored = system(
+    ...     zc.buildout.easy_install._safe_arg(sys.executable)+' '+
+    ...     'bootstrap.py --accept-early-release')
+    >>> print open(buildout_site_py).read() # doctest: +ELLIPSIS
+    "...
+        buildout_paths = [
+            '/sample/eggs/setuptools-...egg',
+            '/sample/eggs/zc.buildout-100.0b1-pyN.N.egg'
+            ]
+    ...
+
+Notice we are now using zc.buildout 100.0b1, a non-final release.
+
+Now we'll go back to the default of preferring final versions.
+
+    >>> write('buildout.cfg',
+    ... '''
+    ... [buildout]
+    ... parts =
+    ... ''')
+
 Now we will try the `--version` option, which lets you define a version for
 `zc.buildout`. If not provided, bootstrap will look for the latest one.
 
@@ -71,11 +152,9 @@
     <BLANKLINE>
     X
 
-Let's make sure the generated `buildout` script uses it::
+Versions older than 1.5.0 put their egg dependencies in the ``buildout`` script.
+Let's make sure it was generated as we expect::
 
-    >>> buildout_script = join(sample_buildout, 'bin', 'buildout')
-    >>> if sys.platform.startswith('win'):
-    ...     buildout_script += '-script.py'
     >>> print open(buildout_script).read() # doctest: +ELLIPSIS
     #...
     <BLANKLINE>
@@ -102,7 +181,7 @@
     <BLANKLINE>
     X
 
-Let's make sure the generated `buildout` script uses it::
+Let's make sure the generated ``buildout`` script uses it::
 
     >>> print open(buildout_script).read() # doctest: +ELLIPSIS
     #...
@@ -119,7 +198,7 @@
         zc.buildout.buildout.main()
     <BLANKLINE>
 
-`zc.buildout` now can also run with `Distribute` with the `--distribute`
+``zc.buildout`` now can also run with `Distribute` with the `--distribute`
 option::
 
     >>> print 'X'; print system(
@@ -131,22 +210,14 @@
     Generated script '/sample/bin/buildout'...
     X
 
-Let's make sure the generated `buildout` script uses it::
-
-    >>> print open(buildout_script).read() # doctest: +ELLIPSIS
-    #...
-    <BLANKLINE>
-    import sys
-    sys.path[0:0] = [
-      '/sample/eggs/distribute-...egg',
-      '/sample/eggs/zc.buildout-...egg',
-      ]
-    <BLANKLINE>
-    import zc.buildout.buildout
-    <BLANKLINE>
-    if __name__ == '__main__':
-        zc.buildout.buildout.main()
-    <BLANKLINE>
+Let's make sure the generated ``site.py`` uses it::
+    >>> print open(buildout_site_py).read() # doctest: +ELLIPSIS
+    "...
+        buildout_paths = [
+            '/sample/eggs/distribute-...egg',
+            '/sample/eggs/zc.buildout-99.99-pyN.N.egg'
+            ]
+    ...
 
 Make sure both options can be used together::
 
@@ -160,8 +231,8 @@
     Generated script '/sample/bin/buildout'...
     X
 
-Let's make sure the generated `buildout` script uses ``Distribute`` *and*
-``zc.buildout-1.2.1``::
+Let's make sure the old-style generated ``buildout`` script uses
+``Distribute`` *and* ``zc.buildout-1.2.1``::
 
     >>> print open(buildout_script).read() # doctest: +ELLIPSIS
     #...
@@ -258,5 +329,13 @@
       --eggs=EGGS           Specify a directory for storing eggs.  Defaults to a
                             temporary directory that is deleted when the bootstrap
                             script completes.
+      --accept-early-release
+                            Normally, if you do not specify a --version, the
+                            bootstrap script gets the newest *final* versions of
+                            zc.buildout for you.  If you use this flag, and also
+                            set ``prefer-final-build-system= false`` in the
+                            [buildout] section of your configuration file,
+                            bootstrap will get the newest releases even if they
+                            are alphas or betas.
       -c CONFIG_FILE        Specify the path to the buildout configuration file to
                             be used.

=== modified file 'src/zc/buildout/buildout.py'
--- src/zc/buildout/buildout.py	2010-08-07 00:12:52 +0000
+++ src/zc/buildout/buildout.py	2010-08-07 00:12:52 +0000
@@ -131,6 +131,7 @@
     'offline': 'false',
     'parts-directory': 'parts',
     'prefer-final': 'false',
+    'prefer-final-build-system': 'true',
     'python': 'buildout',
     'relative-paths': 'false',
     'socket-timeout': '',
@@ -234,6 +235,8 @@
         self._logger = logging.getLogger('zc.buildout')
         self.offline = (buildout_section['offline'] == 'true')
         self.newest = (buildout_section['newest'] == 'true')
+        self.prefer_final_build_system = (
+            buildout_section['prefer-final-build-system'] == 'true')
 
         ##################################################################
         ## WARNING!!!
@@ -267,42 +270,26 @@
 
         self._setup_logging()
 
-        offline = options['offline']
-        if offline not in ('true', 'false'):
-            self._error('Invalid value for offline option: %s', offline)
-        self.offline = (offline == 'true')
-
-        if self.offline:
-            newest = options['newest'] = 'false'
-        else:
-            newest = options['newest']
-            if newest not in ('true', 'false'):
-                self._error('Invalid value for newest option: %s', newest)
-        self.newest = (newest == 'true')
-
         versions = options.get('versions')
         if versions:
             zc.buildout.easy_install.default_versions(dict(self[versions]))
 
-        prefer_final = options['prefer-final']
-        if prefer_final not in ('true', 'false'):
-            self._error('Invalid value for prefer-final option: %s',
-                        prefer_final)
-        zc.buildout.easy_install.prefer_final(prefer_final=='true')
 
-        use_dependency_links = options['use-dependency-links']
-        if use_dependency_links not in ('true', 'false'):
-            self._error('Invalid value for use-dependency-links option: %s',
-                        use_dependency_links)
+        self.offline = options.get_bool('offline')
+        if self.offline:
+            options['newest'] = 'false'
+        self.newest = options.get_bool('newest')
+        zc.buildout.easy_install.prefer_final(
+            options.get_bool('prefer-final'))
+        self.prefer_final_build_system = options.get_bool(
+            'prefer-final-build-system')
         zc.buildout.easy_install.use_dependency_links(
-            use_dependency_links == 'true')
-
-        allow_picked_versions = options['allow-picked-versions']
-        if allow_picked_versions not in ('true', 'false'):
-            self._error('Invalid value for allow-picked-versions option: %s',
-                        allow_picked_versions)
+            options.get_bool('use-dependency-links'))
         zc.buildout.easy_install.allow_picked_versions(
-            allow_picked_versions == 'true')
+            options.get_bool('allow-picked-versions'))
+        zc.buildout.easy_install.install_from_cache(
+            options.get_bool('install-from-cache'))
+        zc.buildout.easy_install.always_unzip(options.get_bool('unzip'))
 
         download_cache = options.get('download-cache')
         if download_cache:
@@ -319,19 +306,6 @@
 
             zc.buildout.easy_install.download_cache(download_cache)
 
-        install_from_cache = options['install-from-cache']
-        if install_from_cache not in ('true', 'false'):
-            self._error('Invalid value for install-from-cache option: %s',
-                        install_from_cache)
-        zc.buildout.easy_install.install_from_cache(
-            install_from_cache=='true')
-
-        always_unzip = options['unzip']
-        if always_unzip not in ('true', 'false'):
-            self._error('Invalid value for unzip option: %s',
-                        always_unzip)
-        zc.buildout.easy_install.always_unzip(always_unzip=='true')
-
         # "Use" each of the defaults so they aren't reported as unused options.
         for name in _buildout_default_options:
             options[name]
@@ -364,6 +338,7 @@
                 [options['develop-eggs-directory'],
                  options['eggs-directory']],
                 include_site_packages=_sys_executable_has_broken_dash_S,
+                prefer_final = self.prefer_final_build_system,
                 )
         else:
             ws = zc.buildout.easy_install.install(
@@ -375,6 +350,7 @@
                 newest=self.newest,
                 allow_hosts=self._allow_hosts,
                 include_site_packages=_sys_executable_has_broken_dash_S,
+                prefer_final = self.prefer_final_build_system,
                 )
 
         # Now copy buildout and setuptools eggs, and record destination eggs:
@@ -410,6 +386,8 @@
         else:
             assert relative_paths == 'false'
             relative_paths = ''
+        # Ideally the (possibly) new version of buildout would get a
+        # chance to write the script.  Not sure how to do that.
         zc.buildout.easy_install.sitepackage_safe_scripts(
             options['bin-directory'], ws, options['executable'], partsdir,
             reqs=['zc.buildout'], relative_paths=relative_paths,
@@ -427,7 +405,7 @@
         # for eggs:
         sys.path.insert(0, self['buildout']['develop-eggs-directory'])
 
-        # Check for updates. This could cause the process to be rstarted
+        # Check for updates. This could cause the process to be restarted.
         self._maybe_upgrade()
 
         # load installed data
@@ -480,7 +458,7 @@
         # compute new part recipe signatures
         self._compute_part_signatures(install_parts)
 
-        # uninstall parts that are no-longer used or who's configs
+        # uninstall parts that are no-longer used or whose configs
         # have changed
         for part in reversed(installed_parts):
             if part in install_parts:
@@ -626,11 +604,11 @@
         f.close()
 
     def _uninstall_part(self, part, installed_part_options):
-        # ununstall part
+        # uninstall part
         __doing__ = 'Uninstalling %s.', part
         self._logger.info(*__doing__)
 
-        # run uinstall recipe
+        # run uninstall recipe
         recipe, entry = _recipe(installed_part_options[part])
         try:
             uninstaller = _install_and_load(
@@ -859,7 +837,8 @@
             index = options.get('index'),
             path = [options['develop-eggs-directory']],
             allow_hosts = self._allow_hosts,
-            include_site_packages=_sys_executable_has_broken_dash_S
+            include_site_packages=_sys_executable_has_broken_dash_S,
+            prefer_final=self.prefer_final_build_system,
             )
 
         upgraded = []
@@ -913,6 +892,8 @@
             # fast for Python to know to regenerate the .pyc/.pyo files.
             shutil.rmtree(partsdir)
         os.mkdir(partsdir)
+        # Ideally the new version of buildout would get a chance to write the
+        # script.  Not sure how to do that.
         zc.buildout.easy_install.sitepackage_safe_scripts(
             options['bin-directory'], ws, sys.executable, partsdir,
             reqs=['zc.buildout'],
@@ -957,7 +938,8 @@
                 links = self['buildout'].get('find-links', '').split(),
                 index = self['buildout'].get('index'),
                 newest=self.newest, allow_hosts=self._allow_hosts,
-                include_site_packages=_sys_executable_has_broken_dash_S)
+                include_site_packages=_sys_executable_has_broken_dash_S,
+                prefer_final=self.prefer_final_build_system)
 
             # Clear cache because extensions might now let us read pages we
             # couldn't read before.
@@ -988,6 +970,7 @@
         setup = os.path.abspath(setup)
 
         fd, tsetup = tempfile.mkstemp()
+        exe = zc.buildout.easy_install._safe_arg(sys.executable)
         try:
             os.write(fd, zc.buildout.easy_install.runsetup_template % dict(
                 setuptools=pkg_resources_loc,
@@ -1001,14 +984,10 @@
                 for a in args:
                     arg_list.append(zc.buildout.easy_install._safe_arg(a))
 
-                subprocess.Popen(
-                    [zc.buildout.easy_install._safe_arg(sys.executable)]
-                    + list(tsetup)
-                    + arg_list
-                    ).wait()
+                subprocess.Popen([exe] + list(tsetup) + arg_list).wait()
 
             else:
-                os.spawnl(os.P_WAIT, sys.executable, zc.buildout.easy_install._safe_arg (sys.executable), tsetup,
+                os.spawnl(os.P_WAIT, sys.executable, exe, tsetup,
                         *[zc.buildout.easy_install._safe_arg(a)
                             for a in args])
         finally:
@@ -1075,7 +1054,8 @@
                 working_set=pkg_resources.working_set,
                 newest=buildout.newest,
                 allow_hosts=buildout._allow_hosts,
-                include_site_packages=_sys_executable_has_broken_dash_S)
+                include_site_packages=_sys_executable_has_broken_dash_S,
+                prefer_final=buildout.prefer_final_build_system)
 
         __doing__ = 'Loading %s recipe entry %s:%s.', group, spec, entry
         return pkg_resources.load_entry_point(
@@ -1297,6 +1277,31 @@
                 self.name)
         return self._created
 
+    def get_bool(self, name, default=None, on_error=None):
+        """Given a name, return a boolean value for that name.
+
+        ``default``, if given, should be 'true', 'false', or None.  None
+        is the default, and means that there is no default for the
+        value: the call should raise a MissingOption error if the name
+        is not present.
+
+        ``on_error``, if given, should be a callable that takes the name and
+        the found value.
+        """
+        if default is None:
+            value = self[name]
+        else:
+            value = self.get(name, default=default)
+        if value not in ('true', 'false'):
+            if on_error is None:
+                raise zc.buildout.UserError(
+                    'Invalid value for %s option: %s' % (name, value))
+            else:
+                on_error(name, value)
+        else:
+            return value == 'true'
+
+
 _spacey_nl = re.compile('[ \t\r\f\v]*\n[ \t\r\f\v\n]*'
                         '|'
                         '^[ \t\r\f\v]+'

=== modified file 'src/zc/buildout/buildout.txt'
--- src/zc/buildout/buildout.txt	2010-06-23 12:07:30 +0000
+++ src/zc/buildout/buildout.txt	2010-08-07 00:12:52 +0000
@@ -765,6 +765,8 @@
         DEFAULT_VALUE
     prefer-final= false
         DEFAULT_VALUE
+    prefer-final-build-system= true
+        DEFAULT_VALUE
     python= buildout
         DEFAULT_VALUE
     relative-paths= false
@@ -2244,6 +2246,7 @@
     parts =
     parts-directory = /sample-buildout/parts
     prefer-final = false
+    prefer-final-build-system = true
     python = buildout
     relative-paths = false
     socket-timeout =
@@ -2484,25 +2487,34 @@
 Preferring Final Releases
 -------------------------
 
-Currently, when searching for new releases, the newest available
-release is used.  This isn't usually ideal, as you may get a
-development release or alpha releases not ready to be widely used.
-You can request that final releases be preferred using the prefer
-final option in the buildout section::
+Currently, when searching for new releases of your project's
+dependencies, the newest available release is used.  This isn't usually
+ideal, as you may get a development release or alpha releases not ready
+to be widely used. You can request that final releases be preferred
+using the ``prefer-final`` option in the buildout section::
 
   [buildout]
   ...
   prefer-final = true
 
-When the prefer-final option is set to true, then when searching for
+When the ``prefer-final`` option is set to true, then when searching for
 new releases, final releases are preferred.  If there are final
 releases that satisfy distribution requirements, then those releases
-are used even if newer non-final releases are available.  The buildout
-prefer-final option can be used to override this behavior.
-
-In buildout version 2, final releases will be preferred by default.
-You will then need to use a false value for prefer-final to get the
-newest releases.
+are used even if newer non-final releases are available.
+
+A separate option controls the behavior of the build system itself.
+When buildout looks for recipes, extensions, and for updates to itself,
+it does prefer final releases by default, as of the 1.5.0 release.  The
+``prefer-final-build-system`` option will let you override this behavior.
+
+  [buildout]
+  ...
+  prefer-final-build-system = false
+
+In buildout version 2, all final releases will be preferred by
+default--that is ``prefer-final`` will also default to 'true'. You will
+then need to use a 'false' value for ``prefer-final`` to get the newest
+releases, like with ``prefer-final-build-system``.
 
 Finding distributions
 ---------------------

=== modified file 'src/zc/buildout/easy_install.py'
--- src/zc/buildout/easy_install.py	2010-08-07 00:12:52 +0000
+++ src/zc/buildout/easy_install.py	2010-08-07 00:12:52 +0000
@@ -322,7 +322,8 @@
                  use_dependency_links=None,
                  allow_hosts=('*',),
                  include_site_packages=None,
-                 allowed_eggs_from_site_packages=None
+                 allowed_eggs_from_site_packages=None,
+                 prefer_final=None,
                  ):
         self._dest = dest
         self._allow_hosts = allow_hosts
@@ -336,6 +337,8 @@
 
         if use_dependency_links is not None:
             self._use_dependency_links = use_dependency_links
+        if prefer_final is not None:
+            self._prefer_final = prefer_final
         self._links = links = list(_fix_file_links(links))
         if self._download_cache and (self._download_cache not in links):
             links.insert(0, self._download_cache)
@@ -1060,13 +1063,14 @@
             executable=sys.executable, always_unzip=None,
             path=None, working_set=None, newest=True, versions=None,
             use_dependency_links=None, allow_hosts=('*',),
-            include_site_packages=None, allowed_eggs_from_site_packages=None):
-    installer = Installer(dest, links, index, executable, always_unzip, path,
-                          newest, versions, use_dependency_links,
-                          allow_hosts=allow_hosts,
-                          include_site_packages=include_site_packages,
-                          allowed_eggs_from_site_packages=
-                            allowed_eggs_from_site_packages)
+            include_site_packages=None, allowed_eggs_from_site_packages=None,
+            prefer_final=None):
+    installer = Installer(
+        dest, links, index, executable, always_unzip, path, newest,
+        versions, use_dependency_links, allow_hosts=allow_hosts,
+        include_site_packages=include_site_packages,
+        allowed_eggs_from_site_packages=allowed_eggs_from_site_packages,
+        prefer_final=prefer_final)
     return installer.install(specs, working_set)
 
 
@@ -1075,11 +1079,11 @@
           executable=sys.executable,
           path=None, newest=True, versions=None, allow_hosts=('*',),
           include_site_packages=None, allowed_eggs_from_site_packages=None):
-    installer = Installer(dest, links, index, executable, True, path, newest,
-                          versions, allow_hosts=allow_hosts,
-                          include_site_packages=include_site_packages,
-                          allowed_eggs_from_site_packages=
-                            allowed_eggs_from_site_packages)
+    installer = Installer(
+        dest, links, index, executable, True, path, newest, versions,
+        allow_hosts=allow_hosts,
+        include_site_packages=include_site_packages,
+        allowed_eggs_from_site_packages=allowed_eggs_from_site_packages)
     return installer.build(spec, build_ext)
 
 
@@ -1175,11 +1179,12 @@
         [f() for f in undo]
 
 def working_set(specs, executable, path, include_site_packages=None,
-                allowed_eggs_from_site_packages=None):
+                allowed_eggs_from_site_packages=None, prefer_final=None):
     return install(
         specs, None, executable=executable, path=path,
         include_site_packages=include_site_packages,
-        allowed_eggs_from_site_packages=allowed_eggs_from_site_packages)
+        allowed_eggs_from_site_packages=allowed_eggs_from_site_packages,
+        prefer_final=prefer_final)
 
 ############################################################################
 # Script generation functions

=== modified file 'src/zc/buildout/tests.py'
--- src/zc/buildout/tests.py	2010-08-07 00:12:52 +0000
+++ src/zc/buildout/tests.py	2010-08-07 00:12:52 +0000
@@ -3126,6 +3126,105 @@
 
     """
 
+def buildout_prefer_final_build_system_option():
+    """
+The prefer-final-build-system buildout option can be used for overriding
+the default preference for final distributions for recipes, buildout
+extensions, and buildout itself.
+
+Set up.  This creates sdists for demorecipe 1.0 and 1.1b1, and for
+demoextension 1.0 and 1.1b1.
+
+    >>> create_sample_recipe_sdists(sample_eggs)
+    >>> create_sample_extension_sdists(sample_eggs)
+
+The default is prefer-final-build-system = true:
+
+    >>> write('buildout.cfg',
+    ... '''
+    ... [buildout]
+    ... parts = demo
+    ... find-links = %(link_server)s
+    ... extensions = demoextension
+    ...
+    ... [demo]
+    ... recipe = demorecipe
+    ... ''' % globals())
+
+    >>> print system(buildout+' -v'), # doctest: +ELLIPSIS
+    Installing ...
+    Picked: demoextension = 1.0
+    ...
+    Picked: demorecipe = 1.0
+    ...
+
+Here we see that the final versions of demorecipe and demoextension were used.
+
+We get the same behavior if we explicitly state that
+prefer-final-build-system = true.
+
+    >>> write('buildout.cfg',
+    ... '''
+    ... [buildout]
+    ... parts = demo
+    ... find-links = %(link_server)s
+    ... extensions = demoextension
+    ... prefer-final-build-system = true
+    ...
+    ... [demo]
+    ... recipe = demorecipe
+    ... ''' % globals())
+
+    >>> print system(buildout+' -v'), # doctest: +ELLIPSIS
+    Installing ...
+    Picked: demoextension = 1.0
+    ...
+    Picked: demorecipe = 1.0
+    ...
+
+If we specify prefer-final-build-system = false, we'll get the newest
+distributions in the build system:
+
+    >>> write('buildout.cfg',
+    ... '''
+    ... [buildout]
+    ... parts = demo
+    ... find-links = %(link_server)s
+    ... extensions = demoextension
+    ... prefer-final-build-system = false
+    ...
+    ... [demo]
+    ... recipe = demorecipe
+    ... ''' % globals())
+
+    >>> print system(buildout+' -v'), # doctest: +ELLIPSIS
+    Installing ...
+    Picked: demoextension = 1.1b1
+    ...
+    Picked: demorecipe = 1.1b1
+    ...
+
+We get an error if we specify anything but true or false:
+
+    >>> write('buildout.cfg',
+    ... '''
+    ... [buildout]
+    ... parts = demo
+    ... find-links = %(link_server)s
+    ... extensions = demoextension
+    ... prefer-final-build-system = no
+    ...
+    ... [demo]
+    ... recipe = demorecipe
+    ... ''' % globals())
+
+    >>> print system(buildout+' -v'), # doctest: +ELLIPSIS
+    While:
+      Initializing.
+    Error: Invalid value for prefer-final-build-system option: no
+
+    """
+
 def develop_with_modules():
     """
 Distribution setup scripts can import modules in the distribution directory:
@@ -3492,6 +3591,68 @@
         finally:
             shutil.rmtree(tmp)
 
+def create_sample_extension_sdists(dest):
+    from zc.buildout.testing import write, mkdir
+    name = 'demoextension'
+    for version in ('1.0', '1.1b1'):
+        tmp = tempfile.mkdtemp()
+        try:
+            write(tmp, 'README.txt', '')
+            write(tmp, name + '.py',
+                  "def ext(buildout):\n"
+                  "    pass\n"
+                  "def unload(buildout):\n"
+                  "    pass\n"
+                  % locals())
+            write(tmp, 'setup.py',
+                  "from setuptools import setup\n"
+                  "setup(\n"
+                  "    name = %(name)r,\n"
+                  "    py_modules = [%(name)r],\n"
+                  "    entry_points = {\n"
+                  "       'zc.buildout.extension': "
+                              "['ext = %(name)s:ext'],\n"
+                  "       'zc.buildout.unloadextension': "
+                              "['ext = %(name)s:unload'],\n"
+                  "       },\n"
+                  "    zip_safe=True, version=%(version)r,\n"
+                  "    author='bob', url='bob', author_email='bob')\n"
+                  % locals())
+            zc.buildout.testing.sdist(tmp, dest)
+        finally:
+            shutil.rmtree(tmp)
+
+def create_sample_recipe_sdists(dest):
+    from zc.buildout.testing import write, mkdir
+    name = 'demorecipe'
+    for version in ('1.0', '1.1b1'):
+        tmp = tempfile.mkdtemp()
+        try:
+            write(tmp, 'README.txt', '')
+            write(tmp, name + '.py',
+                  "import logging, os, zc.buildout\n"
+                  "class Demorecipe:\n"
+                  "    def __init__(self, buildout, name, options):\n"
+                  "        self.name, self.options = name, options\n"
+                  "    def install(self):\n"
+                  "        return ()\n"
+                  "    def update(self):\n"
+                  "        pass\n"
+                  % locals())
+            write(tmp, 'setup.py',
+                  "from setuptools import setup\n"
+                  "setup(\n"
+                  "    name = %(name)r,\n"
+                  "    py_modules = [%(name)r],\n"
+                  "    entry_points = {'zc.buildout': "
+                                       "['default = %(name)s:Demorecipe']},\n"
+                  "    zip_safe=True, version=%(version)r,\n"
+                  "    author='bob', url='bob', author_email='bob')\n"
+                  % locals())
+            zc.buildout.testing.sdist(tmp, dest)
+        finally:
+            shutil.rmtree(tmp)
+
 def _write_eggrecipedemoneeded(tmp, minor_version, suffix=''):
     from zc.buildout.testing import write
     write(tmp, 'README.txt', '')
@@ -3639,37 +3800,33 @@
 
 egg_parse = re.compile('([0-9a-zA-Z_.]+)-([0-9a-zA-Z_.]+)-py(\d[.]\d).egg$'
                        ).match
-def makeNewRelease(project, ws, dest):
+def makeNewRelease(project, ws, dest, version='99.99'):
     dist = ws.find(pkg_resources.Requirement.parse(project))
     eggname, oldver, pyver = egg_parse(
         os.path.basename(dist.location)
         ).groups()
-    dest = os.path.join(dest, "%s-99.99-py%s.egg" % (eggname, pyver))
+    dest = os.path.join(dest, "%s-%s-py%s.egg" % (eggname, version, pyver))
     if os.path.isfile(dist.location):
         shutil.copy(dist.location, dest)
         zip = zipfile.ZipFile(dest, 'a')
         zip.writestr(
             'EGG-INFO/PKG-INFO',
             zip.read('EGG-INFO/PKG-INFO').replace("Version: %s" % oldver,
-                                                  "Version: 99.99")
+                                                  "Version: %s" % version)
             )
         zip.close()
     else:
         shutil.copytree(dist.location, dest)
         info_path = os.path.join(dest, 'EGG-INFO', 'PKG-INFO')
         info = open(info_path).read().replace("Version: %s" % oldver,
-                                              "Version: 99.99")
+                                              "Version: %s" % version)
         open(info_path, 'w').write(info)
 
-
-def updateSetup(test):
-    zc.buildout.testing.buildoutSetUp(test)
-    new_releases = test.globs['tmpdir']('new_releases')
-    test.globs['new_releases'] = new_releases
+def getWorkingSetWithBuildoutEgg(test):
     sample_buildout = test.globs['sample_buildout']
     eggs = os.path.join(sample_buildout, 'eggs')
 
-    # If the zc.buildout dist is a develo dist, convert it to a
+    # If the zc.buildout dist is a develop dist, convert it to a
     # regular egg in the sample buildout
     req = pkg_resources.Requirement.parse('zc.buildout')
     dist = pkg_resources.working_set.find(req)
@@ -3699,9 +3856,16 @@
             os.path.join(sample_buildout, 'bin'))
     else:
         ws = pkg_resources.working_set
+    return ws
 
+def updateSetup(test):
+    zc.buildout.testing.buildoutSetUp(test)
+    new_releases = test.globs['tmpdir']('new_releases')
+    test.globs['new_releases'] = new_releases
+    ws = getWorkingSetWithBuildoutEgg(test)
     # now let's make the new releases
     makeNewRelease('zc.buildout', ws, new_releases)
+    makeNewRelease('zc.buildout', ws, new_releases, '100.0b1')
     os.mkdir(os.path.join(new_releases, 'zc.buildout'))
     if zc.buildout.easy_install.is_distribute:
         makeNewRelease('distribute', ws, new_releases)
@@ -3710,6 +3874,13 @@
         makeNewRelease('setuptools', ws, new_releases)
         os.mkdir(os.path.join(new_releases, 'setuptools'))
 
+def bootstrapSetup(test):
+    easy_install_SetUp(test)
+    sample_eggs = test.globs['sample_eggs']
+    ws = getWorkingSetWithBuildoutEgg(test)
+    makeNewRelease('zc.buildout', ws, sample_eggs)
+    makeNewRelease('zc.buildout', ws, sample_eggs, '100.0b1')
+    os.environ['bootstrap-testing-find-links'] = test.globs['link_server']
 
 normalize_bang = (
     re.compile(re.escape('#!'+
@@ -3952,12 +4123,13 @@
     if os.path.exists(bootstrap_py):
         test_suite.append(doctest.DocFileSuite(
             'bootstrap.txt',
-            setUp=easy_install_SetUp,
+            setUp=bootstrapSetup,
             tearDown=zc.buildout.testing.buildoutTearDown,
             checker=renormalizing.RENormalizing([
                 zc.buildout.testing.normalize_path,
                 zc.buildout.testing.normalize_endings,
                 zc.buildout.testing.normalize_script,
+                zc.buildout.testing.normalize_egg_py,
                 normalize_bang,
                 (re.compile('Downloading.*setuptools.*egg\n'), ''),
                 (re.compile('options:'), 'Options:'),

=== modified file 'src/zc/buildout/update.txt'
--- src/zc/buildout/update.txt	1970-01-01 00:00:00 +0000
+++ src/zc/buildout/update.txt	2010-08-07 00:12:52 +0000
@@ -3,13 +3,14 @@
 
 When a buildout is run, one of the first steps performed is to check
 for updates to either zc.buildout or setuptools.  To demonstrate this,
-we've creates some "new releases" of buildout and setuptools in a
+we've created some "new releases" of buildout and setuptools in a
 new_releases folder:
 
     >>> ls(new_releases)
     d  setuptools
     -  setuptools-99.99-py2.4.egg
     d  zc.buildout
+    -  zc.buildout-100.0b1-pyN.N.egg
     -  zc.buildout-99.99-py2.4.egg
 
 Let's update the sample buildout.cfg to look in this area:
@@ -78,6 +79,11 @@
     zc.buildout 99.99
     setuptools 99.99
 
+Notice that, even though we have a newer beta version of zc.buildout
+available, the final "99.99" was selected.  If you want to get non-final
+versions, specify a specific version in your buildout's versions section,
+or use the ``prefer-final-build-system = false`` discussed below.
+
 Our buildout script's site.py has been updated to use the new eggs:
 
     >>> cat(sample_buildout, 'parts', 'buildout', 'site.py')
@@ -162,7 +168,7 @@
     setuptools 0.6
 
 We also won't upgrade if the buildout script being run isn't in the
-buildouts bin directory.  To see this we'll create a new buildout
+buildout's bin directory.  To see this we'll create a new buildout
 directory:
 
     >>> sample_buildout2 = tmpdir('sample_buildout2')
@@ -187,3 +193,33 @@
     Not upgrading because not running a local buildout command.
 
     >>> ls('bin')
+
+Notice that, as mentioned above, the ``prefer-final-build-system =
+false`` means that newer non-final versions of these dependencies are
+preferred.
+
+    >>> cd(sample_buildout)
+    >>> write(sample_buildout, 'buildout.cfg',
+    ... """
+    ... [buildout]
+    ... find-links = %(new_releases)s
+    ... index = %(new_releases)s
+    ... parts = show-versions
+    ... develop = showversions
+    ... prefer-final-build-system = false
+    ...
+    ... [show-versions]
+    ... recipe = showversions
+    ... """ % dict(new_releases=new_releases))
+
+    >>> print system(buildout),
+    Getting distribution for 'zc.buildout'.
+    Got zc.buildout 100.0b1.
+    Upgraded:
+      zc.buildout version 100.0b1,
+      setuptools version 99.99;
+    restarting.
+    Develop: '/sample-buildout/showversions'
+    Updating show-versions.
+    zc.buildout 100.0b1
+    setuptools 99.99