Merge ~smoser/cloud-init:feature/reproducible-builds into cloud-init:master

[Scott Moser]

setup.py: Do not include rendered files in SOURCES.txt

cloud-init renders template files during its run of setup.py.
Those rendered files were in a temp dir and were making their
way into the SOURCES.txt file.

We do not want to include those, so here we explicitly prune them out.

The issue of SOURCES.txt containing tmp files was reproducible with:
  $ rm -Rf cloud_init.egg-info
  $ git clean --force
  $ rm -Rf ../root.d; python3 setup.py install --root=../root.d
  $ grep ^tmp cloud_init.egg-info/SOURCES.txt

The SOURCES.txt under ../root.d/ would have tmp files.

[Scott Moser (smoser)]

originally from
https://code.launchpad.net/~rjschwei/cloud-init/+git/cloud-init/+merge/334337

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:0f5d42898e423f89c9a13fa5a846926c2a719999
https://jenkins.ubuntu.com/server/job/cloud-init-ci/602/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/602/rebuild

review: Approve (continuous-integration)

[Ryan Harper (raharper)]

I reproduced the issue with your script, applied the patch from this branch and it's fixed.

review: Approve

diff --git a/setup.py b/setup.py
index bf697d7..bc3f52a 100755
--- a/setup.py
+++ b/setup.py
@@ -18,11 +18,14 @@ import tempfile
 
 import setuptools
 from setuptools.command.install import install
+from setuptools.command.egg_info import egg_info
 
 from distutils.errors import DistutilsArgError
 
 import subprocess
 
+RENDERED_TMPD_PREFIX = "RENDERED_TEMPD"
+
 
 def is_f(p):
     return os.path.isfile(p)
@@ -107,7 +110,7 @@ def render_tmpl(template):
         return template
 
     topdir = os.path.dirname(sys.argv[0])
-    tmpd = tempfile.mkdtemp(dir=topdir)
+    tmpd = tempfile.mkdtemp(dir=topdir, prefix=RENDERED_TMPD_PREFIX)
     atexit.register(shutil.rmtree, tmpd)
     bname = os.path.basename(template).rstrip(tmpl_ext)
     fpath = os.path.join(tmpd, bname)
@@ -156,6 +159,25 @@ elif os.path.isfile('/etc/redhat-release'):
     USR_LIB_EXEC = "usr/libexec"
 
 
+class MyEggInfo(egg_info):
+    """This makes sure to not include the rendered files in SOURCES.txt."""
+
+    def find_sources(self):
+        ret = egg_info.find_sources(self)
+        # update the self.filelist.
+        self.filelist.exclude_pattern(RENDERED_TMPD_PREFIX + ".*",
+                                      is_regex=True)
+        # but since mfname is already written we have to update it also.
+        mfname = os.path.join(self.egg_info, "SOURCES.txt")
+        if os.path.exists(mfname):
+            with open(mfname) as fp:
+                files = [f for f in fp
+                         if not f.startswith(RENDERED_TMPD_PREFIX)]
+            with open(mfname, "w") as fp:
+                fp.write(''.join(files))
+        return ret
+
+
 # TODO: Is there a better way to do this??
 class InitsysInstallData(install):
     init_system = None
@@ -229,6 +251,7 @@ if os.uname()[0] != 'FreeBSD':
 # adding on the right init system configuration files
 cmdclass = {
     'install': InitsysInstallData,
+    'egg_info': MyEggInfo,
 }
 
 requirements = read_requires()