Merge ~rjschwei/cloud-init:reproducibleBuild into cloud-init:master

[Robert Schweikert]

Make the package build for distributions reproducible, [1] [2]

[1] https://reproducible-builds.org/
[2] http://bugzilla.opensuse.org/show_bug.cgi?id=1069635

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:3292754b6aa52b43150edf3dbe28a470da44d7f1
https://jenkins.ubuntu.com/server/job/cloud-init-ci/551/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/551/rebuild

review: Approve (continuous-integration)

[Scott Moser (smoser)]

What does "reproducible" mean in this context.
I fully understand the value of of the output of a build being identical if all inputs are identical.

The code here uses a tmpdir and at least attempts to remove that temporary directory
at exit, the output of the build *should* still be the same.

Is the 'atext.register(shutil.rmtree, tmpd)' not working correctly?

review: Needs Information

[Ryan Harper (raharper)]

I suspect the setup.py generates the SOURCES.txt before running atexit stuff.
Maybe there is an ignore setting for SOURCES.txt ?

[Robert Schweikert (rjschwei)]

Yes, it is related to SOURCES.txt.

I did not investigate how atext.register and generation of SOURCES.txt relate.

[Scott Moser (smoser)]

looking at it, found this was about all i could do.
http://paste.ubuntu.com/26091173/

i really do not want to include the rendered files in the SOURCES.txt, which
is what Robert's suggestion does. They're *not* SOURCE files.

[Robert Schweikert (rjschwei)]

I am not attached to the proposed changes of simply making the directory name for the rendered files predictable. Excluding them from SOURCES.txt is probably the more appropriate approach.

[Scott Moser (smoser)]

Robert,
Can you test that that works for you?
The other thing i'd check is that it builds in centos6 (ie, older python).
I can do the centos6 check. But i'd lke to know that my patch actually
works for you.
then i'll submit a MP myself.
thanks.

On Tue, Dec 5, 2017 at 8:46 AM, Robert Schweikert <email address hidden> wrote:

> I am not attached to the proposed changes of simply making the directory
> name for the rendered files predictable. Excluding them from SOURCES.txt is
> probably the more appropriate approach.
>
>
> --
> https://code.launchpad.net/~rjschwei/cloud-init/+git/
> cloud-init/+merge/334337
> You are reviewing the proposed merge of ~rjschwei/cloud-init:reproducibleBuild
> into cloud-init:master.
>

[Robert Schweikert (rjschwei)]

> Robert,
> Can you test that that works for you?

I added the patch to our package build and asked the person working on reproducible builds for feedback.

[Scott Moser (smoser)]

https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/334991

I think that will work.

I'm going to mark *this* merge proposal as rejected, and just follow over there.

Unmerged commits

3292754... by Robert Schweikert on 2017-11-27

- Render the config into a predictable directory name. This makes the
  package build for distributions reproducible. Theer is no need from
  a security perspective to have a non-predictable name for the
  template rendering

diff --git a/setup.py b/setup.py
index bf697d7..d2d50a4 100755
--- a/setup.py
+++ b/setup.py
@@ -107,7 +107,11 @@ def render_tmpl(template):
         return template
 
     topdir = os.path.dirname(sys.argv[0])
-    tmpd = tempfile.mkdtemp(dir=topdir)
+    tmpd = os.path.join(topdir + 'cloud-init-render.tmp')
+    try:
+        os.mkdir(tmpd)
+    except OSError:
+        tmpd = tempfile.mkdtemp(dir=topdir)
     atexit.register(shutil.rmtree, tmpd)
     bname = os.path.basename(template).rstrip(tmpl_ext)
     fpath = os.path.join(tmpd, bname)