Merge lp:~gz/bzr/support_OO_flag_installer into lp:bzr

[Martin Packman]

Make the installer on windows strip non-plugin docstrings.

Followup to the branch getting bzrlib to run correctly when docstrings are stripped. This alters the setup script to build the windows standalone installer that way, but should leave plugins untouched as they can be adapted later.

I'm not set up to properly test this here. If someone who builds windows installers can do a before and after, to see if this a) breaks anything and b) helps at all, that would be great.

The library.zip with the 2.1.1 installer is 17.2MB, I'm interested in how much of a saving can be got.

[Alexander Belchenko (bialix)]

So test this changes one needs to run following commands:

make run
python setup.py py2exe

(You need to remove `build` directory before you run py2exe because it keeps a cache of py/pyo files).

So, testing bzr.dev revno 5218 and this branch we have:

bzr.dev:
library.zip = 14 MB
and entire exe folder (just bzrlib w/o tbzr and other plugins) = 19 MB

This branch (with trunk changes merged):
library.zip = 11 MB
and entire exe folder = 17 MB

The numbers from the FAR file manager, so they are not very accurate. So this branch saves about 2-3 MB.

review: Approve

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Alexander Belchenko wrote:
> Review: Approve
> So test this changes one needs to run following commands:
>
> make run
> python setup.py py2exe
>
> (You need to remove `build` directory before you run py2exe because it keeps a cache of py/pyo files).
>
> So, testing bzr.dev revno 5218 and this branch we have:
>
> bzr.dev:
> library.zip = 14 MB
> and entire exe folder (just bzrlib w/o tbzr and other plugins) = 19 MB
>
> This branch (with trunk changes merged):
> library.zip = 11 MB
> and entire exe folder = 17 MB
>
> The numbers from the FAR file manager, so they are not very accurate. So this branch saves about 2-3 MB.
>

Can you run 7zip to generate the LZMA compressed version of both the
pre- and post- patch versions? That will give us a view of how much the
download size will change.

I'm also curious where this will actually help, I would guess it would
be a reasonable cold-start improvement. Though if we really wanted a
change, we would set DEFLATE on the .zip file. Though that would
actually slow down *hot* performance, which isn't great. (w/ hot
performance, everything is already in mem, so adding decompression is
just overhead.)

Anyway, someone did the work, as long as it doesn't break stuff, it is
probably reasonable to use it. I guess few people try to introspect
objects from the all-in-one installer. (I use help(my_branch.foo) all
the time, but from the python interpreter.)

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkvoKJsACgkQJdeBCYSNAAN8UQCfaQwo/2lYaB2WgTJ/BarUDssG
cQwAoM18qRZE2UBPg4JErT8+TEFty51N
=OtOv
-----END PGP SIGNATURE-----

[Alexander Belchenko (bialix)]

Some numbers re (cold) startup time:

The command is: timeit bzr.exe --no-plugins --no-aliases info

For -OO: 2.453 sec
For bzr.dev: 2.656 sec

So the difference is really small (~200 ms).

[Martin Pool (mbp)]

On 11 May 2010 10:26, Alexander Belchenko <email address hidden> wrote:
> Some numbers re (cold) startup time:
>
> The command is: timeit bzr.exe --no-plugins --no-aliases info
>
> For -OO: 2.453 sec
> For bzr.dev: 2.656 sec
>
> So the difference is really small (~200 ms).

Well, it's 7.7% faster. Not a huge amount but they add up.

--
Martin <http://launchpad.net/~mbp/>

[Alexander Belchenko (bialix)]

About compressed size difference (the sizes in bytes):

5 074 449 bzr-2.2.0dev1-setup-dev.exe

4 542 024 bzr-2.2.0dev1-setup-OO.exe

Difference is about 500 KB.

[John A Meinel (jameinel)]

A point of clarification. If I run "python -O ..." then it does *not* load modules named .pyc. As such, won't this break the compiled bzr, since some plugins will get .pyc, but the rest of bzr will be running as "-OO"? Or am I misunderstanding the optimization levels?

Anyway, this seems fine for a 2.2 series, I'd certainly like to have an installer built and someone running it to make sure things are working ok.

review: Needs Information

[Martin Packman (gz)]

Okay, some more details are warranted.

The py2exe.optimize param has three possible values, 0 meaning __debug__ is True and pyc files are used, 1 meaning __debug__ is False and pyo files are used, and 2 meaning the same as 1 and docstrings are stripped as well.

The current Bazaar installer has ./lib which contains binary extensions and library.zip which contains pyo files for the standard library, bzrlib, and other packages. It also has ./plugins which contains subfolders for each plugin, with both py and pyo files. With this branch merged, all of that will still be the case.

There's one possible gotcha, if people are using the all-in-one installer, and wanting to use extra, unbundled plugins, they'll need to install with the right params to get -O style files or they'll be runtime-compiled without docstrings.

It'd be great if Ian or someone could build a whole installer after this has landed so it can get some field testing.

[Robert Collins (lifeless)]

Martin[gz] - please pick a date for when we'll change the rules on plugins, so that we can tell plugin authors clearly, now. And add a NEWS item to assist in that communication.

Other than that I think you've addressed jam's info request and should land this. You do have landing rights, don't you ?

review: Needs Fixing

[Martin Packman (gz)]

> Martin[gz] - please pick a date for when we'll change the rules on plugins, so
> that we can tell plugin authors clearly, now. And add a NEWS item to assist in
> that communication.

Well, the main branch for this had documentation for plugin authors and news summarising the change. I agree picking a date and clearly communicating it is a good idea, but wouldn't that be better done on the mailing list or something? Will add more news saying the windows all-in-one installer has been switched over, but really I want to make sure this change can actually stick before making a big fuss.

> Other than that I think you've addressed jam's info request and should land
> this. You do have landing rights, don't you ?

Not that I know of.

[lp:~gz/bzr/support_OO_flag_installer]

5197. By Martin Packman on 2010-05-25

NEWS entry for change to all-in-one Windows installer

[Robert Collins (lifeless)]

I don't mind where it is done, but I think we should have it to land the change. What you've written is sufficient now, but please do discuss on the list - and then come back and update the NEWS entry.

review: Approve

[Robert Collins (lifeless)]

sent to pqm by email

=== modified file 'NEWS'
--- NEWS	2010-05-21 07:17:13 +0000
+++ NEWS	2010-05-25 18:09:32 +0000
@@ -170,6 +170,12 @@
   be interpreted as a boolean.
   (Vincent Ladeuil)
 
+* The all-in-one Windows installer will now be built with docstrings stripped
+  from the library zip, reducing the size and slightly improving cold startup
+  time. Bundled plugins are unchanged for the moment, but if adding other new
+  plugins to an all-in-one installation, ensure they are compiled and
+  installed with -O1 or help may not work. (Martin [gz])
+
 Documentation
 *************
 

=== modified file 'setup.py'
--- setup.py	2010-04-19 21:04:31 +0000
+++ setup.py	2010-05-25 18:09:32 +0000
@@ -532,17 +532,15 @@
             install_data.run(self)
 
             py2exe = self.distribution.get_command_obj('py2exe', False)
-            optimize = py2exe.optimize
+            # GZ 2010-04-19: Setup has py2exe.optimize as 2, but give plugins
+            #                time before living with docstring stripping
+            optimize = 1
             compile_names = [f for f in self.outfiles if f.endswith('.py')]
             byte_compile(compile_names,
                          optimize=optimize,
                          force=self.force, prefix=self.install_dir,
                          dry_run=self.dry_run)
-            if optimize:
-                suffix = 'o'
-            else:
-                suffix = 'c'
-            self.outfiles.extend([f + suffix for f in compile_names])
+            self.outfiles.extend([f + 'o' for f in compile_names])
     # end of class install_data_with_bytecompile
 
     target = py2exe.build_exe.Target(script = "bzr",
@@ -690,7 +688,7 @@
                                "excludes": excludes,
                                "dll_excludes": dll_excludes,
                                "dist_dir": "win32_bzr.exe",
-                               "optimize": 1,
+                               "optimize": 2,
                               },
                    }