Merge lp:~mterry/quickly/overwrite into lp:quickly

[Michael Terry]

A long while back, for natty, we decided to change the project_root layout to more clearly delineate what was quickly-owned code and what was user-modifiable code.

Specifically, quickly owns bin/, parts of setup.py, and python_lib/.

The intention was to make it easier for us as maintainers, because we didn't need to support user modifications to those files. It is a real problem. Drive-by contributions rarely consider upgrade.py. Even among maintainers, we often forget it or do it wrong.

This branch finally makes the dream a reality. Any quickly-owned code is automatically updated to the latest template version as needed. It's careful to only do it if a new template version is available.

This will let us make more changes and more complicated changes easier. More velocity!

Note that a user that wants to work-around quickly's code can still do so! They get a section in setup.py that they own. And they have a single entry point in python/ where they can do what they want.

Note that we still have to be careful for any changes we make to python/. But at least this gives us a lot more flexibility than before.

[Tony Byrne (tony-badwolf)]

My unease at overwriting "quickly-owned" files remains, what does a user do when he finds a "quickly-owned" file needs fixing?

Scenario: A user finds a bug in a "quickly-owned" file in his project and fixes it in his project. Later on he quickly upgrades and breaks his project.
Mitigation 1: quickly upgrade bails if the user has made any changes in a "quickly-owned" file.
   (might need to introduce quickly upgrade --force ?)
Mitigation 2: When a user changes a "quickly-owned" file warn him of the later overwrite and suggests he submit a bug report.
Mitigation 3: Write a warning do not edit comment into all "quickly-owned" files.

I assume quickly upgrade calls (or should call) bzr commit. Does quickly upgrade bail out if there are any un-commited changes in source tree?

Noticing changes to a "quickly-owned" file eluded me for a while but how about "quickly create" writing a hash of the "quickly-owned" files into .quickly? Then quickly upgrade could compare current files with the hash.

More attention is needed for bug fixes affecting "quickly-owned" files. A practical example has shown up recently.

Before revision 645 "finally merge gtk3 branch" you could do either of these
cd /path_to/foo
quickly run

cd /path_to/foo
./bin/foo

Afterwards .bin/foo no longer works. This affects users who want to use an IDE like pydev. A fix has been tested by Dražen Lučanin and found to work.
https://bugs.launchpad.net/quickly/+bug/976817/comments/9
https://bugs.launchpad.net/quickly/+bug/976817/comments/10

As thing stand he can no longer upgrade without difficulty.

[Tony Byrne (tony-badwolf)]

A better overwrite algorithm

Use bzr to extract version 1 of project to a temp directory - this contains only files created by quickly
compare file with current version
upgrade can overwrite if no changes

If file changed by user he probably has a good reason so we need a stategy

Write a "Can't upgrade because ..." message
Offer a automatic patch submission to quickly team to look at changes

Or

bzr commit -m 'pre upgrade'
upgrade --force overwrite
bzr commit -m 'post upgrade'
bzr diff

This point is one for the quickly talk list

[Didier Roche-Tolomelli (didrocks)]

ok, I agree (see the hangouts on Quickly reboot) that overwriting and owning those files makes totally sense.

I however have some questions/spotted some errors (I guess) in that branch:

So, those files are licensed, I think:
1. they shouldn't
2. this introduces some unecessary diff and unecessary check we need to do. Like this code:

590 + # First, don't write anything if we don't need to
591 + if os.path.exists(filename):
592 + with file(filename, 'r') as ftest:
593 + if ftest.read() == contents:
594 + return

Would never be executed right? Everytime the boiler plate files will be different?
So maybe just putting those files in the public domain and removing the #BEGIN/END license stenzas.
(but in that case, we need to do something to upgade existing projects).

also, the python_nameconfig.py file will always be different as we bump it with the versionning, but we will maybe want to add some more configuration later on?
Maybe we should have a Config.py in private_lib, which define some kind of class, and python_nameconfig.py in the python/ directory, which inherit from it. That way, we can still upgrade Config.py, add new attributes, and let the user override them?

(Basically, that's how I see private_lib/: only super class which defines and automate some code, and then the lib/ itself inheriting from those).

What do you think?

review: Needs Information

[Michael Terry (mterry)]

You're right that the 'should we copy over this file' check is a little pointless for files that are licensed. But it's not really a problem if we copy it over and license it over again. It will end up the same file and won't show any differences in 'bzr diff'.

I could change the license bits of the quickly-owned bits to say public domain and nothing else [you mention needing to do something to upgrade existing projects -- but nope! that's what this branch would do. :)]. But it doesn't seem worth the change just to avoid a bit of file copying during the new-version-upgrade step.

The python_nameconfig.py file is a good point. I can fix it much the same way that I fixed the desktop file in the ARB branch: by modifying that file after we install it in debian/tmp. That way the original file always stays pristine, and we don't have duplicate copies of the version number in the code (setup.py and the config.py file). Will fix soon.

[Michael Terry (mterry)]

You're right that the 'should we copy over this file' check is a little pointless for files that are licensed. But it's not really a problem if we copy it over and license it over again. It will end up the same file and won't show any differences in 'bzr diff'.

I could change the license bits of the quickly-owned bits to say public domain and nothing else [you mention needing to do something to upgrade existing projects -- but nope! that's what this branch would do. :)]. But it doesn't seem worth the change just to avoid a bit of file copying during the new-version-upgrade step.

The python_nameconfig.py file is a good point. I can fix it much the same way that I fixed the desktop file in the ARB branch: by modifying that file after we install it in debian/tmp. That way the original file always stays pristine, and we don't have duplicate copies of the version number in the code (setup.py and the config.py file). Will fix soon.

That will leave VERSION as the version when running under quickly run. But I think that's OK and actually kind of like it (the idea that doesn't have a version until you release it with one).

[Michael Terry (mterry)]

Whoops, sorry for double messages. I've fixed it. Please re-review.

[Michael Terry (mterry)]

I just realized that didrocks is on vacation. Well, I wanted to get this in today for busy-during-FF-week reasons, so I'm going to push this in.

Didrocks approved the basic idea and most of the code. The remaining issues can be dealt with as bug fixes after FF.

I'll leave this merge open for further discussion into a real signoff happens.

[Didier Roche-Tolomelli (didrocks)]

resetted to "merge", really happy with the branch. I have a stupid idea about how to easy dealing to not update the copyright in this file, https://code.launchpad.net/~didrocks/quickly/dont-licence-private-lib/+merge/122842

Thanks for the quick fixes! Looking good :)

=== modified file 'data/templates/ubuntu-application/create.py'
--- data/templates/ubuntu-application/create.py	2012-02-16 07:24:06 +0000
+++ data/templates/ubuntu-application/create.py	2012-08-17 20:23:19 +0000
@@ -64,40 +64,7 @@
 
 os.chdir(project_name)
 
-# get origin path
-pathname = templatetools.get_template_path_from_project()
-abs_path_project_root = os.path.join(pathname, 'project_root')
-
-python_name = templatetools.python_name(project_name)
-sentence_name, camel_case_name = templatetools.conventional_names(project_name)
-substitutions = (("project_name",project_name),
-            ("camel_case_name",camel_case_name),
-            ("python_name",python_name),
-            ("sentence_name",sentence_name),)
-
-
-for root, dirs, files in os.walk(abs_path_project_root):
-    try:
-        relative_dir = root.split('project_root/')[1]
-    except:
-        relative_dir = ""
-    # python dir should be replace by python_name (project "pythonified" name)
-    if relative_dir.startswith('python'):
-        relative_dir = relative_dir.replace('python', python_name)
-
-    for directory in dirs:
-        if directory.startswith('python'):
-            directory = directory.replace('python', python_name)
-        os.mkdir(os.path.join(relative_dir, directory))
-    for filename in files:
-        templatetools.file_from_template(root, filename, relative_dir, substitutions)
-
-# set the mode to executable for executable file 
-exec_file = os.path.join('bin', project_name)
-try:
-    os.chmod(exec_file, 0755)
-except:
-    pass
+templatetools.copy_dirs_from_template()
 
 # add it to revision control
 print _("Creating bzr repository and committing")
@@ -120,6 +87,7 @@
         env['XDG_DATA_DIRS'] = datadir
 
 # run the new application if X display
+exec_file = os.path.join('bin', project_name)
 if templatetools.is_X_display() and os.path.isfile(exec_file):
     print _("Launching your newly created project!")
     subprocess.call(['./' + project_name], cwd='bin/', env=env)

=== modified file 'data/templates/ubuntu-application/project_root/bin/project_name' (properties changed: -x to +x)
--- data/templates/ubuntu-application/project_root/bin/project_name	2012-05-25 17:48:13 +0000
+++ data/templates/ubuntu-application/project_root/bin/project_name	2012-08-17 20:23:19 +0000
@@ -4,6 +4,8 @@
 # This file is in the public domain
 ### END LICENSE
 
+### DO NOT EDIT THIS FILE ###
+
 import sys
 import os
 

=== modified file 'data/templates/ubuntu-application/project_root/python_lib/AboutDialog.py'
--- data/templates/ubuntu-application/project_root/python_lib/AboutDialog.py	2011-06-08 19:20:41 +0000
+++ data/templates/ubuntu-application/project_root/python_lib/AboutDialog.py	2012-08-17 20:23:19 +0000
@@ -3,6 +3,8 @@
 # This file is in the public domain
 ### END LICENSE
 
+### DO NOT EDIT THIS FILE ###
+
 from gi.repository import Gtk # pylint: disable=E0611
 
 from . helpers import get_builder

=== modified file 'data/templates/ubuntu-application/project_root/python_lib/Builder.py'
--- data/templates/ubuntu-application/project_root/python_lib/Builder.py	2011-12-15 19:49:20 +0000
+++ data/templates/ubuntu-application/project_root/python_lib/Builder.py	2012-08-17 20:23:19 +0000
@@ -3,6 +3,8 @@
 # This file is in the public domain
 ### END LICENSE
 
+### DO NOT EDIT THIS FILE ###
+
 '''Enhances builder connections, provides object to access glade objects'''
 
 from gi.repository import GObject, Gtk # pylint: disable=E0611

=== modified file 'data/templates/ubuntu-application/project_root/python_lib/PreferencesDialog.py'
--- data/templates/ubuntu-application/project_root/python_lib/PreferencesDialog.py	2011-11-07 21:55:54 +0000
+++ data/templates/ubuntu-application/project_root/python_lib/PreferencesDialog.py	2012-08-17 20:23:19 +0000
@@ -3,6 +3,8 @@
 # This file is in the public domain
 ### END LICENSE
 
+### DO NOT EDIT THIS FILE ###
+
 """this dialog adjusts values in gsettings
 """
 

=== modified file 'data/templates/ubuntu-application/project_root/python_lib/Window.py'
--- data/templates/ubuntu-application/project_root/python_lib/Window.py	2012-08-17 20:23:19 +0000
+++ data/templates/ubuntu-application/project_root/python_lib/Window.py	2012-08-17 20:23:19 +0000
@@ -3,6 +3,8 @@
 # This file is in the public domain
 ### END LICENSE
 
+### DO NOT EDIT THIS FILE ###
+
 from gi.repository import Gio, Gtk # pylint: disable=E0611
 import logging
 logger = logging.getLogger('python_name_lib')

=== modified file 'data/templates/ubuntu-application/project_root/python_lib/__init__.py'
--- data/templates/ubuntu-application/project_root/python_lib/__init__.py	2011-11-07 21:55:54 +0000
+++ data/templates/ubuntu-application/project_root/python_lib/__init__.py	2012-08-17 20:23:19 +0000
@@ -3,6 +3,8 @@
 # This file is in the public domain
 ### END LICENSE
 
+### DO NOT EDIT THIS FILE ###
+
 '''facade - makes python_name_lib package easy to refactor
 
 while keeping its api constant'''

=== modified file 'data/templates/ubuntu-application/project_root/python_lib/helpers.py'
--- data/templates/ubuntu-application/project_root/python_lib/helpers.py	2011-11-08 02:27:24 +0000
+++ data/templates/ubuntu-application/project_root/python_lib/helpers.py	2012-08-17 20:23:19 +0000
@@ -3,6 +3,8 @@
 # This file is in the public domain
 ### END LICENSE
 
+### DO NOT EDIT THIS FILE ###
+
 """Helpers for an Ubuntu application."""
 import logging
 import os

=== modified file 'data/templates/ubuntu-application/project_root/python_lib/python_nameconfig.py'
--- data/templates/ubuntu-application/project_root/python_lib/python_nameconfig.py	2011-03-31 15:19:55 +0000
+++ data/templates/ubuntu-application/project_root/python_lib/python_nameconfig.py	2012-08-17 20:23:19 +0000
@@ -3,10 +3,7 @@
 # This file is in the public domain
 ### END LICENSE
 
-# THIS IS camel_case_name CONFIGURATION FILE
-# YOU CAN PUT THERE SOME GLOBAL VALUE
-# Do not touch unless you know what you're doing.
-# you're warned :)
+### DO NOT EDIT THIS FILE ###
 
 __all__ = [
     'project_path_not_found',

=== modified file 'data/templates/ubuntu-application/project_root/setup.py'
--- data/templates/ubuntu-application/project_root/setup.py	2011-03-31 15:19:55 +0000
+++ data/templates/ubuntu-application/project_root/setup.py	2012-08-17 20:23:19 +0000
@@ -16,12 +16,13 @@
     sys.exit(1)
 assert DistUtilsExtra.auto.__version__ >= '2.18', 'needs DistUtilsExtra.auto >= 2.18'
 
-def update_config(values = {}):
+def update_config(libdir, values = {}):
 
+    filename = os.path.join(libdir, 'python_name_lib/python_nameconfig.py')
     oldvalues = {}
     try:
-        fin = file('python_name_lib/python_nameconfig.py', 'r')
-        fout = file(fin.name + '.new', 'w')
+        fin = file(filename, 'r')
+        fout = file(filename + '.new', 'w')
 
         for line in fin:
             fields = line.split(' = ') # Separate variable from value
@@ -35,38 +36,42 @@
         fin.close()
         os.rename(fout.name, fin.name)
     except (OSError, IOError), e:
-        print ("ERROR: Can't find python_name_lib/python_nameconfig.py")
+        print ("ERROR: Can't find %s" % filename)
         sys.exit(1)
     return oldvalues
 
 
-def update_desktop_file(datadir):
+def update_desktop_file(installdatadir, prefix):
 
+    filename = os.path.join(installdatadir, 'share', 'applications',
+                            'project_name.desktop')
     try:
-        fin = file('project_name.desktop.in', 'r')
-        fout = file(fin.name + '.new', 'w')
+        fin = file(filename, 'r')
+        fout = file(filename + '.new', 'w')
 
         for line in fin:            
             if 'Icon=' in line:
-                line = "Icon=%s\n" % (datadir + 'media/project_name.svg')
+                line = "Icon=%s\n" % os.path.join(prefix, 'share',
+                                                  'project_name', 'media',
+                                                  'project_name.svg')
             fout.write(line)
         fout.flush()
         fout.close()
         fin.close()
         os.rename(fout.name, fin.name)
     except (OSError, IOError), e:
-        print ("ERROR: Can't find project_name.desktop.in")
+        print ("ERROR: Can't find %s" % filename)
         sys.exit(1)
 
 
 class InstallAndUpdateDataDirectory(DistUtilsExtra.auto.install_auto):
     def run(self):
+        DistUtilsExtra.auto.install_auto.run(self)
+
         values = {'__python_name_data_directory__': "'%s'" % (self.prefix + '/share/project_name/'),
                   '__version__': "'%s'" % self.distribution.get_version()}
-        previous_values = update_config(values)
-        update_desktop_file(self.prefix + '/share/project_name/')
-        DistUtilsExtra.auto.install_auto.run(self)
-        update_config(previous_values)
+        update_config(self.install_lib, values)
+        update_desktop_file(self.install_data, self.prefix)
 
 
         

=== modified file 'data/templates/ubuntu-application/test/apport/apport.sh'
--- data/templates/ubuntu-application/test/apport/apport.sh	2012-08-17 20:23:19 +0000
+++ data/templates/ubuntu-application/test/apport/apport.sh	2012-08-17 20:23:19 +0000
@@ -242,6 +242,10 @@
 # Committed revision 4.
 
 quickly upgrade 0.3
+# Note: This is the first time you have run Quickly since it has been updated.
+# Quickly will now upgrade its files (bin/*, test_project_lib/*, and setup.py).
+# But first it will save your project.  View Quickly's changes by running:
+# bzr diff
 
 bzr status
 
@@ -263,13 +267,17 @@
 # deleted etc/apport/crashdb.conf.d
 # missing etc/apport/crashdb.conf.d/test-project-crashdb.conf
 # deleted etc/apport/crashdb.conf.d/test-project-crashdb.conf
-# Committed revision 5.
+# Committed revision 6.
 
 bzr status
 
 quickly upgrade 0.3
 # Creating new apport crashdb configuration
 # Creating new apport hooks
+# Note: This is the first time you have run Quickly since it has been updated.
+# Quickly will now upgrade its files (bin/*, test_project_lib/*, and setup.py).
+# But first it will save your project.  View Quickly's changes by running:
+# bzr diff
 
 cat etc/apport/crashdb.conf.d/test-project-crashdb.conf
 # ### BEGIN LICENSE
@@ -307,6 +315,10 @@
 quickly upgrade 0.3
 # Creating new apport crashdb configuration
 # Creating new apport hooks
+# Note: This is the first time you have run Quickly since it has been updated.
+# Quickly will now upgrade its files (bin/*, test_project_lib/*, and setup.py).
+# But first it will save your project.  View Quickly's changes by running:
+# bzr diff
 
 cp "$TEST_SCRIPT_DIR/TestProjectWindow.ui.no_gtk-about" ./data/ui/TestProjectWindow.ui
 
@@ -318,18 +330,37 @@
 
 bzr commit -m "Committing after removing all lpi integration"
 # Committing to: /tmp/test-project/
+# missing apport
+# deleted apport
+# missing etc
+# deleted etc
+# missing apport/source_test-project.py
+# deleted apport/source_test-project.py
 # modified data/ui/TestProjectWindow.ui
-# Committed revision 6.
+# missing etc/apport
+# deleted etc/apport
+# missing etc/apport/crashdb.conf.d
+# deleted etc/apport/crashdb.conf.d
+# missing etc/apport/crashdb.conf.d/test-project-crashdb.conf
+# deleted etc/apport/crashdb.conf.d/test-project-crashdb.conf
+# Committed revision 9.
 
 quickly upgrade 0.3
 # Creating new apport crashdb configuration
 # Creating new apport hooks
+# Note: This is the first time you have run Quickly since it has been updated.
+# Quickly will now upgrade its files (bin/*, test_project_lib/*, and setup.py).
+# But first it will save your project.  View Quickly's changes by running:
+# bzr diff
 
-bzr status
-# unknown:
-#   apport/
-#   etc/
+ls -dF apport etc
+# apport/
+# etc/
 
 rm ./data/ui/TestProjectWindow.ui
 
 quickly upgrade 0.3
+# Note: This is the first time you have run Quickly since it has been updated.
+# Quickly will now upgrade its files (bin/*, test_project_lib/*, and setup.py).
+# But first it will save your project.  View Quickly's changes by running:
+# bzr diff

=== modified file 'data/templates/ubuntu-application/test/gtk3upgrade.sh'
--- data/templates/ubuntu-application/test/gtk3upgrade.sh	2011-12-15 21:32:15 +0000
+++ data/templates/ubuntu-application/test/gtk3upgrade.sh	2012-08-17 20:23:19 +0000
@@ -34,6 +34,10 @@
 # version = 11.10
 
 quickly upgrade
+# Note: This is the first time you have run Quickly since it has been updated.
+# Quickly will now upgrade its files (bin/*, test_project_lib/*, and setup.py).
+# But first it will save your project.  View Quickly's changes by running:
+# bzr diff
 
 grep '11\.10' .quickly
 

=== added file 'data/templates/ubuntu-application/test/overwrite.sh'
--- data/templates/ubuntu-application/test/overwrite.sh	1970-01-01 00:00:00 +0000
+++ data/templates/ubuntu-application/test/overwrite.sh	2012-08-17 20:23:19 +0000
@@ -0,0 +1,85 @@
+#!/bin/sh
+
+cd /tmp
+
+rm -rf test-project
+
+quickly create ubuntu-application test-project
+# Creating bzr repository and committing
+# Congrats, your new project is setup! cd /tmp/test-project/ to start hacking.
+# Creating project directory test-project
+
+cd test-project
+
+sed -i 's/import os/import os #test-blarg/' setup.py
+
+(echo "#test-blarg" >> setup.py)
+
+(echo "#test-blarg" >> bin/test-project)
+
+(echo "#test-blarg" >> test_project/TestProjectWindow.py)
+
+(echo "#test-blarg" >> test_project_lib/Window.py)
+
+rm test_project_lib/Builder.py
+
+cp -a test_project_lib/helpers.py test_project_lib/helpers.py.test-bak
+
+grep -R "#test-blarg" .
+# ./test_project_lib/Window.py:#test-blarg
+# ./setup.py:import os #test-blarg
+# ./setup.py:#test-blarg
+# ./bin/test-project:#test-blarg
+# ./test_project/TestProjectWindow.py:#test-blarg
+
+quickly upgrade 11.04
+# Note: This is the first time you have run Quickly since it has been updated.
+# Quickly will now upgrade its files (bin/*, test_project_lib/*, and setup.py).
+# But first it will save your project.  View Quickly's changes by running:
+# bzr diff
+
+(test "$(ls --full-time test_project_lib/helpers.py)" = "$(ls --full-time test_project_lib/helpers.py.test-bak | sed 's/\.test-bak//')" && echo "Same")
+# Same
+
+ls test_project_lib/Builder.py
+# test_project_lib/Builder.py
+
+grep -R "#test-blarg" .
+# ./setup.py:#test-blarg
+# ./test_project/TestProjectWindow.py:#test-blarg
+
+grep python_name setup.py test_project_lib/Builder.py
+
+## A run to see if we futz with setup.py when we don't need to
+
+cp -a setup.py setup.py.test-bak
+
+rm test_project_lib/Builder.py
+
+## We also make sure that we correctly keep the wrapper executable
+
+rm bin/test-project
+
+quickly upgrade 11.04
+# Note: This is the first time you have run Quickly since it has been updated.
+# Quickly will now upgrade its files (bin/*, test_project_lib/*, and setup.py).
+# But first it will save your project.  View Quickly's changes by running:
+# bzr diff
+
+(test "$(ls --full-time setup.py)" = "$(ls --full-time setup.py.test-bak | sed 's/\.test-bak//')" && echo "Same")
+# Same
+
+ls test_project_lib/Builder.py
+# test_project_lib/Builder.py
+
+ls -F bin/test-project
+# bin/test-project*
+
+## A run to see if we change anything when version hasn't changed
+
+rm test_project_lib/Builder.py
+
+quickly upgrade
+
+ls test_project_lib/Builder.py
+# ls: cannot access test_project_lib/Builder.py: No such file or directory

=== modified file 'data/templates/ubuntu-application/test/package.sh'
--- data/templates/ubuntu-application/test/package.sh	2012-05-25 18:45:02 +0000
+++ data/templates/ubuntu-application/test/package.sh	2012-08-17 20:23:19 +0000
@@ -21,18 +21,9 @@
 # Ubuntu package has been successfully created in ../test-project_0.1_all.deb
 
 bzr status
-# modified:
-#   test-project.desktop.in
 # unknown:
 #   po/
 
-bzr diff --diff-options="-e"
-# === modified file 'test-project.desktop.in'
-# 6c
-# Icon=/usr/share/test-project/media/test-project.svg
-# .
-# 
-
 ls debian
 # changelog
 # compat

=== modified file 'data/templates/ubuntu-application/upgrade.py'
--- data/templates/ubuntu-application/upgrade.py	2012-08-17 20:23:19 +0000
+++ data/templates/ubuntu-application/upgrade.py	2012-08-17 20:23:19 +0000
@@ -25,6 +25,7 @@
 
 from internal import quicklyutils
 from quickly import commands, configurationhandler, templatetools
+import license
 
 import gettext
 from gettext import gettext as _
@@ -113,9 +114,9 @@
         pass
     # update config file to add __license__
     try:
-        license = quicklyutils.get_setup_value('license')
+        setup_license = quicklyutils.get_setup_value('license')
     except quicklyutils.cant_deal_with_setup_value:
-        license = ''
+        setup_license = ''
     try:
         skip = 0
         config_file = '%s/%sconfig.py' % (python_name, python_name)
@@ -134,7 +135,7 @@
                 continue
             if fields[0] == '__%s_data_directory__' % python_name:
                 fout.write(line)
-                line = "__license__ = '%s'\n" % license
+                line = "__license__ = '%s'\n" % setup_license
             if "get_data_file(*path_segments):" in line:
                 data_file_function_found = True
                 skip = 9
@@ -173,15 +174,6 @@
     if lp_project_name is not None:
         internal.apportutils.update_apport(project_name, lp_project_name, lp_project_name)
 
-    # new dialog file needs helpers.py
-    if not os.path.isfile('%s/helpers.py' % python_name) and os.path.isdir(python_name):
-        source_dir = os.path.join(os.path.dirname(__file__), 'project_root',
-                                  'python')
-        templatetools.file_from_template(source_dir, 
-                                        "helpers.py", 
-                                        python_name, 
-                                        substitutions)
-
 if project_version < '0.4.3':
     ## update dependencies format
     if 'dependencies' in configurationhandler.project_config \
@@ -196,49 +188,6 @@
     # Use full modelines for all python files
     sedline = "sed -i 's/-\*- coding: utf-8 -\*-/-*- Mode: Python; coding: utf-8; indent-tabs-mode: nil; tab-width: 4 -*-/'"
     os.system("find . -name '*.py' -exec %s {} \;" % sedline)
-    os.system("%s bin/%s" % (sedline, project_name))
-
-### 11.03 update
-if project_version < '11.03':
-    # support /opt installation
-    content_to_update = '''python_path = []
-if os.path.abspath(__file__).startswith('/opt'):
-    syspath = sys.path[:] # copy to avoid infinite loop in pending objects
-    for path in syspath:
-        opt_path = path.replace('/usr', '/opt/extras.ubuntu.com/%(project_name)s')
-        python_path.insert(0, opt_path)
-        sys.path.insert(0, opt_path)
-if (os.path.exists(os.path.join(PROJECT_ROOT_DIRECTORY, '%(python_name)s'))
-    and PROJECT_ROOT_DIRECTORY not in sys.path):
-    python_path.insert(0, PROJECT_ROOT_DIRECTORY)
-    sys.path.insert(0, PROJECT_ROOT_DIRECTORY)
-if python_path:
-    os.putenv('PYTHONPATH', "%%s:%%s" %% (os.getenv('PYTHONPATH', ''), ':'.join(python_path))) # for subprocesses''' % {'python_name' : python_name, 'project_name' : project_name}
-
-    # projects based on ~/quickly-projects/template-name have no project_version
-    # so only upgrade if really necessary     
-    with open("./bin/%s" % project_name) as project_bin:
-        contents =  project_bin.read()
-    if not content_to_update in contents:
-        try:
-            templatetools.update_file_content("./bin/%s" % project_name,
-                    'if (os.path.exists(os.path.join(PROJECT_ROOT_DIRECTORY',
-                    "    os.putenv('PYTHONPATH', PROJECT_ROOT_DIRECTORY) # for subprocesses",
-                    content_to_update)
-        except templatetools.CantUpdateFile, e:
-            print _("WARNING: can't update your project to support /opt. This doesn't matter if you don't plan to submit your project to the application review board. Cause is: %s" % e)
-
-### 11.09 update (but only through 11.10; later versions don't want this change)
-if project_version < '11.09' and template_version <= '11.10':
-    filename = './%s_lib/Builder.py' % python_name
-    try:
-        with open(filename) as fileobj:
-            contents = fileobj.read()
-        contents = contents.replace('from gi.repository import GObject', 'import gobject')
-        contents = contents.replace('GObject.', 'gobject.')
-        templatetools.set_file_contents(filename, contents)
-    except IOError:
-        pass
 
 ### EPOCH CHANGE
 ### This is where we upgraded the default projects to GTK3, PyGI, and GSettings.
@@ -250,5 +199,24 @@
 'quickly upgrade' to get rid of this message.""")
     sys.exit(0)
 
+# Overwrite quickly-owned files as necessary
+if project_version < template_version:
+    print _(
+"""Note: This is the first time you have run Quickly since it has been updated.
+Quickly will now upgrade its files (bin/*, %s_lib/*, and setup.py).
+But first it will save your project.  View Quickly's changes by running:
+bzr diff""" % python_name)
+    subprocess.call(["bzr", "add", "-q"])
+    subprocess.call(["bzr", "commit", "--unchanged", "-q",
+                     "-m", "Pre-upgrade checkpoint"])
+    templatetools.copy_dirs_from_template(dirs = ['bin', 'python_lib'])
+    templatetools.copy_setup_py_from_template()
+    try:
+        # License new files as needed
+        license.licensing()
+    except license.LicenceError as e:
+        pass  # Don't worry about it, user may not have set it up yet
+    subprocess.call(["bzr", "add", "-q"])  # bzr diff will show new files
+
 templatetools.update_version_in_project_file(template_version, 'ubuntu-application')
 sys.exit(0)

=== modified file 'data/templates/ubuntu-cli/commandsconfig'
--- data/templates/ubuntu-cli/commandsconfig	2011-03-02 22:59:32 +0000
+++ data/templates/ubuntu-cli/commandsconfig	2012-08-17 20:23:19 +0000
@@ -7,4 +7,4 @@
 #COMMANDS_FOLLOWED_BY_COMMAND = 
 
 [ubuntu-application]
-IMPORT=configure;create;debug;edit;license;package;release;run;save;share;test
+IMPORT=configure;create;debug;edit;license;package;release;run;save;share;test;upgrade

=== added directory 'data/templates/ubuntu-cli/test'
=== added file 'data/templates/ubuntu-cli/test/overwrite.sh'
--- data/templates/ubuntu-cli/test/overwrite.sh	1970-01-01 00:00:00 +0000
+++ data/templates/ubuntu-cli/test/overwrite.sh	2012-08-17 20:23:19 +0000
@@ -0,0 +1,36 @@
+#!/bin/sh
+
+cd /tmp
+
+rm -rf test-project
+
+quickly create ubuntu-cli test-project
+# Creating bzr repository and committing
+# Congrats, your new project is setup! cd /tmp/test-project/ to start hacking.
+# Creating project directory test-project
+
+cd test-project
+
+sed -i 's/import os/import os #test-blarg/' setup.py
+
+(echo "#test-blarg" >> setup.py)
+
+(echo "#test-blarg" >> bin/test-project)
+
+(echo "#test-blarg" >> test_project/__init__.py)
+
+grep -R "#test-blarg" .
+# ./setup.py:import os #test-blarg
+# ./setup.py:#test-blarg
+# ./bin/test-project:#test-blarg
+# ./test_project/__init__.py:#test-blarg
+
+quickly upgrade 11.04
+# Note: This is the first time you have run Quickly since it has been updated.
+# Quickly will now upgrade its files (bin/*, test_project_lib/*, and setup.py).
+# But first it will save your project.  View Quickly's changes by running:
+# bzr diff
+
+grep -R "#test-blarg" .
+# ./setup.py:#test-blarg
+# ./test_project/__init__.py:#test-blarg

=== modified file 'data/templates/ubuntu-flash-game/create.py'
--- data/templates/ubuntu-flash-game/create.py	2011-07-22 10:47:34 +0000
+++ data/templates/ubuntu-flash-game/create.py	2012-08-17 20:23:19 +0000
@@ -80,13 +80,6 @@
 
 os.chdir(project_name)
 
-# get origin path
-pathname = templatetools.get_template_path_from_project()
-abs_path_project_root = os.path.join(pathname, 'project_root')
-
-python_name = templatetools.python_name(project_name)
-sentence_name, camel_case_name = templatetools.conventional_names(project_name)
-
 # Calculate the SWF's dimensions
 try:
     width, height = SWF.dimensions(swf)
@@ -98,30 +91,9 @@
     width, height = (640, 480)
 
 
-substitutions = (("project_name",project_name),
-            ("camel_case_name",camel_case_name),
-            ("python_name",python_name),
-            ("sentence_name",sentence_name),
-            ("swf_height",str(height)),
-            ("swf_width",str(width)),
-            )
-
-
-for root, dirs, files in os.walk(abs_path_project_root):
-    try:
-        relative_dir = root.split('project_root/')[1]
-    except:
-        relative_dir = ""
-    # python dir should be replace by python_name (project "pythonified" name)
-    if relative_dir.startswith('python'):
-        relative_dir = relative_dir.replace('python', python_name)
-
-    for directory in dirs:
-        if directory == 'python':
-            directory = python_name
-        os.mkdir(os.path.join(relative_dir, directory))
-    for filename in files:
-        templatetools.file_from_template(root, filename, relative_dir, substitutions)
+substitutions = [("swf_height",str(height)),
+                 ("swf_width",str(width))]
+templatetools.copy_dirs_from_template(extra_substitutions = substitutions)
 
 # set the mode to executable for executable file 
 exec_file = os.path.join('bin', project_name)

=== modified file 'quickly/templatetools.py'
--- quickly/templatetools.py	2011-06-06 10:14:40 +0000
+++ quickly/templatetools.py	2012-08-17 20:23:19 +0000
@@ -94,6 +94,13 @@
     os.chmod(dest_file_name, mode)
 
 def set_file_contents(filename, contents):
+    # First, don't write anything if we don't need to
+    if os.path.exists(filename):
+        with file(filename, 'r') as ftest:
+            if ftest.read() == contents:
+                return
+
+    # OK, we have novel content, let's write it out
     new_filename = filename + '.new'
     fout = file(new_filename, 'w')
     fout.write(contents)
@@ -111,33 +118,29 @@
     try:
         filename = os.path.abspath(filename)
         ftarget_file_name = file(filename, 'r')
-        ftarget_file_name_out = file(ftarget_file_name.name + '.new', 'w')
+        new_content = ''
         for line in ftarget_file_name:
             # seek if we have to add something
             if start_marker in line:
                 skip_until_end_found = True
                 marker_found = True
-                ftarget_file_name_out.write(replacing_content)
+                new_content = new_content + replacing_content
 
             if end_marker in line:
                 skip_until_end_found = False
 
             if not skip_until_end_found:
-                ftarget_file_name_out.write(line)
+                new_content = new_content + line
 
         ftarget_file_name.close()
-        ftarget_file_name_out.close()
 
         if skip_until_end_found: # that means we didn't find the end_tag, don't copy the file
-            os.remove(ftarget_file_name_out.name)
             raise CantUpdateFile(_("%s was not found in the file %s.") % (end_marker, ftarget_file_name.name))
 
         if not marker_found:
-             os.remove(ftarget_file_name_out.name)
              raise CantUpdateFile(_("%s was not found in the file %s.") % (start_marker, ftarget_file_name.name))
 
-        apply_file_rights(ftarget_file_name.name, ftarget_file_name_out.name)
-        os.rename(ftarget_file_name_out.name, ftarget_file_name.name)
+        set_file_contents(ftarget_file_name.name, new_content)
 
     except (OSError, IOError), e:
         msg = _("%s file was not found or can't update it") % ftarget_file_name
@@ -261,7 +264,7 @@
     camel_case_name = get_camel_case_name(name)
     return sentence_name, camel_case_name
 
-def file_from_template(template_dir, template_file, target_dir, substitutions=[], rename = True):
+def file_from_template(template_dir, template_file, target_dir, substitutions=[], rename = True, overwrite = False):
 
     if not os.path.isfile(os.path.join(template_dir, template_file)):
         return
@@ -278,9 +281,86 @@
         file_contents = file_contents.replace(pattern,sub)
 
     target_path = os.path.join(target_dir, target_file)
-    if os.path.exists(target_path):
+    if os.path.exists(target_path) and not overwrite:
         print _("Failed to add file to project\n cannot add: %s - this file already exists." % target_path)
-        sys.exit(4)        
+        sys.exit(4)
 
     set_file_contents(target_path, file_contents)
+    apply_file_rights(os.path.join(template_dir, template_file), target_path)
     fin.close()
+
+def copy_dirs_from_template(dirs = ["."], extra_substitutions = []):
+    if not configurationhandler.project_config:
+        configurationhandler.loadConfig()
+    project_name = configurationhandler.project_config['project']
+
+    pathname = get_template_path_from_project()
+    abs_path_project_root = os.path.join(pathname, 'project_root')
+
+    py_name = python_name(project_name)
+    sentence_name, camel_case_name = conventional_names(project_name)
+    substitutions = [("project_name",project_name),
+                     ("camel_case_name",camel_case_name),
+                     ("python_name",py_name),
+                     ("sentence_name",sentence_name),] + extra_substitutions
+
+    for top_dir in dirs:
+        full_top_dir = os.path.join(abs_path_project_root, top_dir)
+        for root, subdirs, files in os.walk(full_top_dir):
+            try:
+                relative_dir = root.split('project_root/')[1]
+            except:
+                relative_dir = ""
+            # python dir should be replace by py_name (project "pythonified"
+            # name)
+            if os.path.basename(relative_dir).startswith('python'):
+                relative_dir = relative_dir.replace('python', py_name)
+
+            for directory in subdirs:
+                if os.path.basename(directory).startswith('python'):
+                    directory = directory.replace('python', py_name)
+                if not os.path.exists(os.path.join(relative_dir, directory)):
+                    os.mkdir(os.path.join(relative_dir, directory))
+            for filename in files:
+                file_from_template(root, filename, relative_dir,
+                                   substitutions, overwrite = True)
+
+def copy_setup_py_from_template(extra_substitutions = []):
+    if not configurationhandler.project_config:
+        configurationhandler.loadConfig()
+    project_name = configurationhandler.project_config['project']
+
+    pathname = get_template_path_from_project()
+    abs_path_project_root = os.path.join(pathname, 'project_root')
+
+    py_name = python_name(project_name)
+    sentence_name, camel_case_name = conventional_names(project_name)
+    substitutions = [("project_name",project_name),
+                     ("camel_case_name",camel_case_name),
+                     ("python_name",py_name),
+                     ("sentence_name",sentence_name),] + extra_substitutions
+
+    template_setup = os.path.join(abs_path_project_root, 'setup.py')
+    project_setup = 'setup.py'
+
+    # Grab quickly-owned bits in our original and copy them to user's version
+    start_marker = "###################### DO NOT TOUCH THIS (HEAD TO THE SECOND PART) ######################"
+    end_marker = "##################################################################################"
+    try:
+        with file(template_setup, 'r') as template_fd:
+            template_contents = ''
+            for line in template_fd.readlines():
+                if line == end_marker + '\n':
+                    break
+                elif template_contents:
+                    template_contents = template_contents + line
+                elif line == start_marker + '\n':
+                    template_contents = line
+        for s in substitutions:
+            pattern, sub = s
+            template_contents = template_contents.replace(pattern, sub)
+        update_file_content(project_setup, start_marker, end_marker, template_contents)
+    except Exception as e:
+        print(_("Failed to update setup.py:\n%s" % e))
+        sys.exit(4)
+

=== modified file 'quickly/test/completion.sh'
--- quickly/test/completion.sh	2012-04-19 14:30:00 +0000
+++ quickly/test/completion.sh	2012-08-17 20:23:19 +0000
@@ -130,7 +130,7 @@
 # add commands configure debug design edit getstarted help license package quickly release run save share submitubuntu test tutorial upgrade
 
 quickly shell-completion quickly -t ubuntu-cli foo
-# commands configure debug edit getstarted help license package quickly release run save share test
+# commands configure debug edit getstarted help license package quickly release run save share test upgrade
 
 quickly shell-completion quickly add foo
 # dialog help-guide help-topic indicator

=== modified file 'quickly/test/help.sh'
--- quickly/test/help.sh	2012-04-19 14:30:00 +0000
+++ quickly/test/help.sh	2012-08-17 20:23:19 +0000
@@ -84,7 +84,7 @@
 HOME=/ quickly help ubuntu-cli
 # ERROR: No command provided to help command.
 # Usage: quickly help [template] <command>
-# Candidate commands are: commands, configure, create, debug, edit, getstarted, help, license, package, quickly, release, run, save, share, test
+# Candidate commands are: commands, configure, create, debug, edit, getstarted, help, license, package, quickly, release, run, save, share, test, upgrade
 
 HOME=/ quickly help foobar
 # ERROR: No foobar command found in ubuntu-application template.