Merge lp:~wesley-wiedenmeier/curtin/without-environment into lp:~curtin-dev/curtin/trunk

[Wesley Wiedenmeier]

Remove global state and dependance on os.environ from curtin for simplification.
Branch isn't ready for merge yet, still need to debug. Everything should be in place though.

[lp:~wesley-wiedenmeier/curtin/without-environment]

451. By Wesley Wiedenmeier on 2016-08-19

In workingdir.WorkingDir, fix attrs used for env

452. By Wesley Wiedenmeier on 2016-08-19

Handle non existant config files from workingdir in config.load_command_config

453. By Wesley Wiedenmeier on 2016-08-19

In workingdir.WorkingDir, don't try to clean up dirs that haven't been created

454. By Wesley Wiedenmeier on 2016-08-19

Merge from trunk

455. By Wesley Wiedenmeier on 2016-08-19

In commands.extract, fix typo in test for missing target in args

456. By Wesley Wiedenmeier on 2016-08-19

Don't export environment during install

457. By Wesley Wiedenmeier on 2016-08-19

Update commands.hook to use target from commandline rather than environment

458. By Wesley Wiedenmeier on 2016-08-20

In commands.block_meta and commands.curt_hooks, use WorkingDir.mdadm_conf and
WorkingDir.crypttab rather than getting paths via WorkingDir.fstab

459. By Wesley Wiedenmeier on 2016-08-20

Add mdadm_conf and crypttab to state files in WorkingDir

460. By Wesley Wiedenmeier on 2016-08-20

Add util.has_content to test if a file is non empty and use it in curthooks
instead of os.path.exists as WorkingDir will cause the files to exist

461. By Wesley Wiedenmeier on 2016-08-20

Fixed use of WorkingDir.mdadm_config in commands.block_meta

[Wesley Wiedenmeier (wesley-wiedenmeier)]

I think this should be pretty much ready to go. I've run through several vmtests on it and it looks like everything works. It might still be good to run the full suite though, I haven't done that yet.

This merge ensures that all curtin subcommands support the flags '--target', '--scratch', '--state', and '--report-stack-prefix'. If these flags are already present in the COMMAND_ARGS for the subcommand, then it will not modify the arguments. If the flags are not present yet it will add them and mark them as hidden.

Instead of putting information about curtin's current state during an installation into the system environemnt, which is hard to mimic in tests and can be slightly inconsistant depending on how the system is set up, the flags for target, state, and scratch will always be set when running any curtin subcommands as part of the commands for an installation stage.

If an installation stage includes commands that are not curtin subcommands, then the command will be run normally.

The files under the state dir have paths defined centrally in curtin.workingdir, so that no path manipulation needs to be done to get their paths. curtin.workingdir also handles the creation and removal of temporary directories and the logic to unmount everything under the target directory so that it can be removed at the end of installation has been cleaned up and moved there.

This is a bit simpler than passing information in the system environment was too, because it does not require the subcommands to look for information about what they should be doing in multiple places, everything they need to run is in args.

[Wesley Wiedenmeier (wesley-wiedenmeier)]

There aren't any unittests for curtin.workingdir yet, I'll mark this needs review once I have those.

[lp:~wesley-wiedenmeier/curtin/without-environment]

462. By Wesley Wiedenmeier on 2016-08-21

Started unittests for curtin.workingdir

463. By Wesley Wiedenmeier on 2016-08-22

Added unittests for curtin.workingdir

Unmerged revisions

463. By Wesley Wiedenmeier on 2016-08-22

Added unittests for curtin.workingdir

462. By Wesley Wiedenmeier on 2016-08-21

Started unittests for curtin.workingdir

461. By Wesley Wiedenmeier on 2016-08-20

Fixed use of WorkingDir.mdadm_config in commands.block_meta

460. By Wesley Wiedenmeier on 2016-08-20

Add util.has_content to test if a file is non empty and use it in curthooks
instead of os.path.exists as WorkingDir will cause the files to exist

459. By Wesley Wiedenmeier on 2016-08-20

Add mdadm_conf and crypttab to state files in WorkingDir

458. By Wesley Wiedenmeier on 2016-08-20

In commands.block_meta and commands.curt_hooks, use WorkingDir.mdadm_conf and
WorkingDir.crypttab rather than getting paths via WorkingDir.fstab

457. By Wesley Wiedenmeier on 2016-08-19

Update commands.hook to use target from commandline rather than environment

456. By Wesley Wiedenmeier on 2016-08-19

Don't export environment during install

455. By Wesley Wiedenmeier on 2016-08-19

In commands.extract, fix typo in test for missing target in args

454. By Wesley Wiedenmeier on 2016-08-19

Merge from trunk

=== modified file 'curtin/commands/__init__.py'
--- curtin/commands/__init__.py	2013-08-26 14:09:15 +0000
+++ curtin/commands/__init__.py	2016-08-22 05:07:32 +0000
@@ -15,6 +15,10 @@
 #   You should have received a copy of the GNU Affero General Public License
 #   along with Curtin.  If not, see <http://www.gnu.org/licenses/>.
 
+import argparse
+
+HIDDEN_OPTS = ('--target', '--state', '--scratch', '--report-stack-prefix')
+
 
 def populate_one_subcmd(parser, options_dict, handler):
     for ent in options_dict:
@@ -22,6 +26,10 @@
         if not isinstance(args, (list, tuple)):
             args = (args,)
         parser.add_argument(*args, **ent[1])
+    for optname in HIDDEN_OPTS:
+        if optname not in parser._option_string_actions.keys():
+            parser.add_argument(optname, help=argparse.SUPPRESS,
+                                action='store', default=None)
     parser.set_defaults(func=handler)
 
 # vi: ts=4 expandtab syntax=python

=== modified file 'curtin/commands/apply_net.py'
--- curtin/commands/apply_net.py	2016-07-13 07:50:49 +0000
+++ curtin/commands/apply_net.py	2016-08-22 05:07:32 +0000
@@ -15,12 +15,11 @@
 #   You should have received a copy of the GNU Affero General Public License
 #   along with Curtin.  If not, see <http://www.gnu.org/licenses/>.
 
-import os
 import sys
 
 from .. import log
 import curtin.net as net
-import curtin.util as util
+from curtin.workingdir import WorkingDir
 from . import populate_one_subcmd
 
 
@@ -49,33 +48,28 @@
 def apply_net_main(args):
     #  curtin apply_net [--net-state=/config/netstate.yml] [--target=/]
     #                   [--net-config=/config/maas_net.yml]
-    state = util.load_command_environment()
-
     log.basicConfig(stream=args.log_file, verbosity=1)
 
-    if args.target is not None:
-        state['target'] = args.target
-
-    if args.net_state is not None:
-        state['network_state'] = args.net_state
-
-    if args.net_config is not None:
-        state['network_config'] = args.net_config
-
-    if state['target'] is None:
+    if args.target is None:
         sys.stderr.write("Unable to find target.  "
                          "Use --target or set TARGET_MOUNT_POINT\n")
         sys.exit(2)
 
-    if not state['network_config'] and not state['network_state']:
+    if args.state and (not args.network_config and not args.network_state):
+        wd = WorkingDir(target=args.target, state=args.state,
+                        scratch=args.scratch)
+        args.network_state = wd.network_state
+        args.network_config = wd.network_config
+
+    if not args.network_config and not args.network_state:
         sys.stderr.write("Must provide at least config or state\n")
         sys.exit(2)
 
     LOG.info('Applying network configuration')
     try:
-        apply_net(target=state['target'],
-                  network_state=state['network_state'],
-                  network_config=state['network_config'])
+        apply_net(target=args.target,
+                  network_state=args.network_state,
+                  network_config=args.network_config)
     except Exception:
         LOG.exception('failed to apply network config')
 
@@ -85,20 +79,14 @@
 
 CMD_ARGUMENTS = (
     ((('-s', '--net-state'),
-     {'help': ('file to read containing network state. '
-               'defaults to env["OUTPUT_NETWORK_STATE"]'),
-      'metavar': 'NETSTATE', 'action': 'store',
-      'default': os.environ.get('OUTPUT_NETWORK_STATE')}),
+     {'help': ('file to read containing network state.'),
+      'metavar': 'NETSTATE', 'action': 'store', 'default': None}),
      (('-t', '--target'),
-      {'help': ('target filesystem root to add swap file to. '
-                'default is env["TARGET_MOUNT_POINT"]'),
-       'metavar': 'TARGET', 'action': 'store',
-       'default': os.environ.get('TARGET_MOUNT_POINT')}),
+      {'help': ('target filesystem to write data too.'),
+       'metavar': 'TARGET', 'action': 'store', 'default': None}),
      (('-c', '--net-config'),
-      {'help': ('file to read containing curtin network config.'
-                'defaults to env["OUTPUT_NETWORK_CONFIG"]'),
-       'metavar': 'NETCONFIG', 'action': 'store',
-       'default': os.environ.get('OUTPUT_NETWORK_CONFIG')})))
+      {'help': ('file to read containing curtin network config.'),
+       'metavar': 'NETCONFIG', 'action': 'store', 'default': None})))
 
 
 def POPULATE_SUBCMD(parser):

=== modified file 'curtin/commands/apt_config.py'
--- curtin/commands/apt_config.py	2016-08-18 14:53:38 +0000
+++ curtin/commands/apt_config.py	2016-08-22 05:07:32 +0000
@@ -552,25 +552,18 @@
         """
     cfg = config.load_command_config(args, {})
 
-    if args.target is not None:
-        target = args.target
-    else:
-        state = util.load_command_environment()
-        target = state['target']
-
-    if target is None:
-        sys.stderr.write("Unable to find target.  "
-                         "Use --target or set TARGET_MOUNT_POINT\n")
+    if args.target is None:
+        sys.stderr.write("Unable to find target. Use --target.\n")
         sys.exit(2)
 
     apt_cfg = cfg.get("apt")
     # if no apt config section is available, do nothing
     if apt_cfg is not None:
         LOG.debug("Handling apt to target %s with config %s",
-                  target, apt_cfg)
+                  args.target, apt_cfg)
         try:
-            with util.ChrootableTarget(target, sys_resolvconf=True):
-                handle_apt(apt_cfg, target)
+            with util.ChrootableTarget(args.target, sys_resolvconf=True):
+                handle_apt(apt_cfg, args.target)
         except (RuntimeError, TypeError, ValueError, IOError):
             LOG.exception("Failed to configure apt features '%s'", apt_cfg)
             sys.exit(1)
@@ -651,9 +644,8 @@
        'metavar': 'FILE', 'type': argparse.FileType("rb"),
        'dest': 'cfgopts', 'default': []}),
      (('-t', '--target'),
-      {'help': 'chroot to target. default is env[TARGET_MOUNT_POINT]',
-       'action': 'store', 'metavar': 'TARGET',
-       'default': os.environ.get('TARGET_MOUNT_POINT')}),)
+      {'help': 'chroot to target.', 'action': 'store',
+       'metavar': 'TARGET', 'default': None}),)
 )
 
 

=== modified file 'curtin/commands/block_meta.py'
--- curtin/commands/block_meta.py	2016-08-02 07:59:23 +0000
+++ curtin/commands/block_meta.py	2016-08-22 05:07:32 +0000
@@ -16,7 +16,7 @@
 #   along with Curtin.  If not, see <http://www.gnu.org/licenses/>.
 
 from collections import OrderedDict
-from curtin import (block, config, util)
+from curtin import (block, config, util, workingdir)
 from curtin.block import mdadm
 from curtin.log import LOG
 from curtin.block import mkfs
@@ -57,14 +57,15 @@
 
 def block_meta(args):
     # main entry point for the block-meta command.
-    state = util.load_command_environment()
-    cfg = config.load_command_config(args, state)
-    if args.mode == CUSTOM or cfg.get("storage") is not None:
-        meta_custom(args)
-    elif args.mode in (SIMPLE, SIMPLE_BOOT):
-        meta_simple(args)
-    else:
-        raise NotImplementedError("mode=%s is not implemented" % args.mode)
+    cfg = config.load_command_config(args)
+    with workingdir.WorkingDir(state=args.state, target=args.target,
+                               scratch=args.scratch) as wd:
+        if args.mode == CUSTOM or cfg.get("storage") is not None:
+            meta_custom(args, wd)
+        elif args.mode in (SIMPLE, SIMPLE_BOOT):
+            meta_simple(args, wd)
+        else:
+            raise NotImplementedError("mode=%s is not implemented" % args.mode)
 
 
 def logtime(msg, func, *args, **kwargs):
@@ -275,9 +276,7 @@
     return ''.join(c if c in valid else '-' for c in dname)
 
 
-def make_dname(volume, storage_config):
-    state = util.load_command_environment()
-    rules_dir = os.path.join(state['scratch'], "rules.d")
+def make_dname(volume, storage_config, rules_dir):
     vol = storage_config.get(volume)
     path = get_path_to_storage_volume(volume, storage_config)
     ptuuid = None
@@ -427,7 +426,7 @@
     return volume_path
 
 
-def disk_handler(info, storage_config):
+def disk_handler(info, storage_config, wd):
     ptable = info.get('ptable')
 
     disk = get_path_to_storage_volume(info.get('id'), storage_config)
@@ -497,7 +496,7 @@
 
     # Make the name if needed
     if info.get('name'):
-        make_dname(info.get('id'), storage_config)
+        make_dname(info.get('id'), storage_config, wd.rules_dir)
 
 
 def getnumberoflogicaldisks(device, storage_config):
@@ -525,7 +524,7 @@
     return last_partnum
 
 
-def partition_handler(info, storage_config):
+def partition_handler(info, storage_config, wd):
     device = info.get('device')
     size = info.get('size')
     flag = info.get('flag')
@@ -673,10 +672,10 @@
             mode=info.get('wipe'))
     # Make the name if needed
     if storage_config.get(device).get('name') and partition_type != 'extended':
-        make_dname(info.get('id'), storage_config)
-
-
-def format_handler(info, storage_config):
+        make_dname(info.get('id'), storage_config, wd.rules_dir)
+
+
+def format_handler(info, storage_config, wd):
     volume = info.get('volume')
     if not volume:
         raise ValueError("volume must be specified for partition '%s'" %
@@ -695,8 +694,7 @@
     mkfs.mkfs_from_config(volume_path, info)
 
 
-def mount_handler(info, storage_config):
-    state = util.load_command_environment()
+def mount_handler(info, storage_config, wd):
     path = info.get('path')
     filesystem = storage_config.get(info.get('device'))
     if not path and filesystem.get('fstype') != "swap":
@@ -711,7 +709,7 @@
         # Figure out what point should be
         while len(path) > 0 and path[0] == "/":
             path = path[1:]
-        mount_point = os.path.join(state['target'], path)
+        mount_point = os.path.join(wd.target, path)
 
         # Create mount point if does not exist
         util.ensure_dir(mount_point)
@@ -720,8 +718,8 @@
         util.subp(['mount', volume_path, mount_point])
 
     # Add volume to fstab
-    if state['fstab']:
-        with open(state['fstab'], "a") as fp:
+    if wd.fstab:
+        with open(wd.fstab, "a") as fp:
             if volume.get('type') in ["raid", "bcache",
                                       "disk", "lvm_partition"]:
                 location = get_path_to_storage_volume(volume.get('id'),
@@ -749,7 +747,7 @@
         LOG.info("fstab not in environment, so not writing")
 
 
-def lvm_volgroup_handler(info, storage_config):
+def lvm_volgroup_handler(info, storage_config, wd):
     devices = info.get('devices')
     device_paths = []
     name = info.get('name')
@@ -790,7 +788,7 @@
         util.subp(cmd)
 
 
-def lvm_partition_handler(info, storage_config):
+def lvm_partition_handler(info, storage_config, wd):
     volgroup = storage_config.get(info.get('volgroup')).get('name')
     name = info.get('name')
     if not volgroup:
@@ -832,11 +830,10 @@
         raise ValueError("Partition tables on top of lvm logical volumes is \
                          not supported")
 
-    make_dname(info.get('id'), storage_config)
-
-
-def dm_crypt_handler(info, storage_config):
-    state = util.load_command_environment()
+    make_dname(info.get('id'), storage_config, wd.rules_dir)
+
+
+def dm_crypt_handler(info, storage_config, wd):
     volume = info.get('volume')
     key = info.get('key')
     keysize = info.get('keysize')
@@ -874,22 +871,21 @@
 
     os.remove(tmp_keyfile)
 
-    # A crypttab will be created in the same directory as the fstab in the
-    # configuration. This will then be copied onto the system later
-    if state['fstab']:
-        crypt_tab_location = os.path.join(os.path.split(state['fstab'])[0],
-                                          "crypttab")
-        uuid = block.get_volume_uuid(volume_path)
-        with open(crypt_tab_location, "a") as fp:
-            fp.write("%s UUID=%s none luks\n" % (dm_name, uuid))
-    else:
-        LOG.info("fstab configuration is not present in environment, so \
-            cannot locate an appropriate directory to write crypttab in \
-            so not writing crypttab")
-
-
-def raid_handler(info, storage_config):
-    state = util.load_command_environment()
+    # the WorkingDir instance will ensure that there is a state dir to write
+    # to, even if this is being used outside of install and it will be removed
+    # immediately
+    mdadm_scan_data = mdadm.mdadm_detail_scan()
+    util.write_file(wd.crypttab, mdadm_scan_data)
+
+    # the WorkingDir instance will ensure that there is a state dir to write
+    # to, even if this is being used outside of install and it will be removed
+    # immediately
+    uuid = block.get_volume_uuid(volume_path)
+    util.write_file(wd.crypttab,
+                    "{} UUID={} none luks\n".format(dm_name, uuid))
+
+
+def raid_handler(info, storage_config, wd):
     devices = info.get('devices')
     raidlevel = info.get('raidlevel')
     spare_devices = info.get('spare_devices')
@@ -939,22 +935,13 @@
                        info.get('mdname', ''))
 
     # Make dname rule for this dev
-    make_dname(info.get('id'), storage_config)
+    make_dname(info.get('id'), storage_config, wd.rules_dir)
 
-    # A mdadm.conf will be created in the same directory as the fstab in the
-    # configuration. This will then be copied onto the installed system later.
-    # The file must also be written onto the running system to enable it to run
-    # mdadm --assemble and continue installation
-    if state['fstab']:
-        mdadm_location = os.path.join(os.path.split(state['fstab'])[0],
-                                      "mdadm.conf")
-        mdadm_scan_data = mdadm.mdadm_detail_scan()
-        with open(mdadm_location, "w") as fp:
-            fp.write(mdadm_scan_data)
-    else:
-        LOG.info("fstab configuration is not present in the environment, so \
-            cannot locate an appropriate directory to write mdadm.conf in, \
-            so not writing mdadm.conf")
+    # the WorkingDir instance will ensure that there is a state dir to write
+    # to, even if this is being used outside of install and it will be removed
+    # immediately
+    mdadm_scan_data = mdadm.mdadm_detail_scan()
+    util.write_file(wd.mdadm_config, mdadm_scan_data)
 
     # If ptable is specified, call disk_handler on this mdadm device to create
     # the table
@@ -962,7 +949,7 @@
         disk_handler(info, storage_config)
 
 
-def bcache_handler(info, storage_config):
+def bcache_handler(info, storage_config, wd):
     backing_device = get_path_to_storage_volume(info.get('backing_device'),
                                                 storage_config)
     cache_device = get_path_to_storage_volume(info.get('cache_device'),
@@ -1099,7 +1086,7 @@
 
     if info.get('name'):
         # Make dname rule for this dev
-        make_dname(info.get('id'), storage_config)
+        make_dname(info.get('id'), storage_config, wd.rules_dir)
 
     if info.get('ptable'):
         raise ValueError("Partition tables on top of lvm logical volumes is \
@@ -1123,7 +1110,7 @@
     return OrderedDict((d["id"], d) for (i, d) in enumerate(scfg))
 
 
-def meta_custom(args):
+def meta_custom(args, wd):
     """Does custom partitioning based on the layout provided in the config
     file. Section with the name storage contains information on which
     partitions on which disks to create. It also contains information about
@@ -1142,13 +1129,10 @@
         'bcache': bcache_handler
     }
 
-    state = util.load_command_environment()
-    cfg = config.load_command_config(args, state)
-
+    cfg = config.load_command_config(args)
     storage_config_dict = extract_storage_ordered_dict(cfg)
-
-    # set up reportstack
-    stack_prefix = state.get('report_stack_prefix', '')
+    stack_prefix = (args.report_stack_prefix
+                    if args.report_stack_prefix else '')
 
     for item_id, command in storage_config_dict.items():
         handler = command_handlers.get(command['type'])
@@ -1156,10 +1140,10 @@
             raise ValueError("unknown command type '%s'" % command['type'])
         with events.ReportEventStack(
                 name=stack_prefix, reporting_enabled=True, level="INFO",
-                description="configuring %s: %s" % (command['type'],
-                                                    command['id'])):
+                description="configuring %s: %s" % (
+                    command['type'], command['id'])):
             try:
-                handler(command, storage_config_dict)
+                handler(command, storage_config_dict, wd)
             except Exception as error:
                 LOG.error("An error occured handling '%s': %s - %s" %
                           (item_id, type(error).__name__, error))
@@ -1168,20 +1152,14 @@
     return 0
 
 
-def meta_simple(args):
+def meta_simple(args, wd):
     """Creates a root partition. If args.mode == SIMPLE_BOOT, it will also
     create a separate /boot partition.
     """
-    state = util.load_command_environment()
-
-    cfg = config.load_command_config(args, state)
-
-    if args.target is not None:
-        state['target'] = args.target
-
-    if state['target'] is None:
-        sys.stderr.write("Unable to find target.  "
-                         "Use --target or set TARGET_MOUNT_POINT\n")
+    cfg = config.load_command_config(args)
+
+    if wd.target is None:
+        sys.stderr.write("Unable to find target. Use --target\n")
         sys.exit(2)
 
     devices = args.devices
@@ -1250,7 +1228,7 @@
         # we have at least one dd-able image
         # we will only take the first one
         rootdev = write_image_to_disk(dd_images[0], devname)
-        util.subp(['mount', rootdev, state['target']])
+        util.subp(['mount', rootdev, wd.target])
         return 0
 
     # helper partition will forcibly set up partition there
@@ -1303,11 +1281,11 @@
     # mkfs for root partition first and mount
     cmd = ['mkfs.%s' % args.fstype, '-q', '-L', 'cloudimg-rootfs', rootdev]
     logtime(' '.join(cmd), util.subp, cmd)
-    util.subp(['mount', rootdev, state['target']])
+    util.subp(['mount', rootdev, wd.target])
 
     if bootpt['enabled']:
-        # create 'boot' directory in state['target']
-        boot_dir = os.path.join(state['target'], 'boot')
+        # create 'boot' directory in target
+        boot_dir = os.path.join(wd.target, 'boot')
         util.subp(['mkdir', boot_dir])
         # mkfs for boot partition and mount
         cmd = ['mkfs.%s' % bootpt['fstype'],
@@ -1316,12 +1294,12 @@
         util.subp(['mount', bootdev, boot_dir])
 
     if ptfmt == "uefi":
-        uefi_dir = os.path.join(state['target'], 'boot', 'efi')
+        uefi_dir = os.path.join(target, 'boot', 'efi')
         util.ensure_dir(uefi_dir)
         util.subp(['mount', uefi_dev, uefi_dir])
 
-    if state['fstab']:
-        with open(state['fstab'], "w") as fp:
+    if wd.fstab:
+        with open(wd.fstab, "w") as fp:
             if bootpt['enabled']:
                 fp.write("LABEL=%s /boot %s defaults 0 0\n" %
                          (bootpt['label'], bootpt['fstype']))

=== modified file 'curtin/commands/curthooks.py'
--- curtin/commands/curthooks.py	2016-07-29 17:19:20 +0000
+++ curtin/commands/curthooks.py	2016-08-22 05:07:32 +0000
@@ -31,16 +31,17 @@
 from curtin import net
 from curtin.reporter import events
 from curtin.commands import apt_config
+from curtin.workingdir import WorkingDir
 
 from . import populate_one_subcmd
 
 CMD_ARGUMENTS = (
     ((('-t', '--target'),
-      {'help': 'operate on target. default is env[TARGET_MOUNT_POINT]',
-       'action': 'store', 'metavar': 'TARGET', 'default': None}),
+      {'help': 'operate on target.', 'action': 'store',
+       'metavar': 'TARGET', 'default': None}),
      (('-c', '--config'),
-      {'help': 'operate on config. default is env[CONFIG]',
-       'action': 'store', 'metavar': 'CONFIG', 'default': None}),
+      {'help': 'operate on config.', 'action': 'store',
+       'metavar': 'CONFIG', 'default': None}),
      )
 )
 
@@ -386,7 +387,7 @@
 
 def copy_fstab(fstab, target):
     if not fstab:
-        LOG.warn("fstab variable not in state, not copying fstab")
+        LOG.warn("no fstab provided to copy, not copying fstab")
         return
 
     shutil.copy(fstab, os.path.sep.join([target, 'etc/fstab']))
@@ -410,10 +411,7 @@
                 'etc/mdadm/mdadm.conf']))
 
 
-def apply_networking(target, state):
-    netstate = state.get('network_state')
-    netconf = state.get('network_config')
-    interfaces = state.get('interfaces')
+def apply_networking(target, network_state, network_config, interfaces):
 
     def is_valid_src(infile):
         with open(infile, 'r') as fp:
@@ -423,12 +421,14 @@
         return False
 
     ns = None
-    if is_valid_src(netstate):
+    if is_valid_src(network_state):
         LOG.debug("applying network_state")
-        ns = net.network_state.from_state_file(netstate)
-    elif is_valid_src(netconf):
+        ns = net.network_state.from_state_file(network_state)
+    elif is_valid_src(network_config):
         LOG.debug("applying network_config")
-        ns = net.parse_net_config(netconf)
+        ns = net.parse_net_config(network_config)
+    else:
+        LOG.debug("no available network_state or network_config, none applied")
 
     if ns is not None:
         net.render_network_state(target=target, network_state=ns)
@@ -593,7 +593,7 @@
     update_initramfs(target, all_kernels=True)
 
 
-def install_missing_packages(cfg, target):
+def install_missing_packages(cfg, target, stack_prefix):
     ''' describe which operation types will require specific packages
 
     'custom_config_key': {
@@ -642,10 +642,8 @@
                 needed_packages.append(pkg)
 
     if needed_packages:
-        state = util.load_command_environment()
         with events.ReportEventStack(
-                name=state.get('report_stack_prefix'),
-                reporting_enabled=True, level="INFO",
+                name=stack_prefix, reporting_enabled=True, level="INFO",
                 description="Installing packages on target system: " +
                 str(needed_packages)):
             util.install_packages(needed_packages, target=target)
@@ -674,95 +672,87 @@
 
 
 def curthooks(args):
-    state = util.load_command_environment()
-
-    if args.target is not None:
-        target = args.target
-    else:
-        target = state['target']
-
-    if target is None:
-        sys.stderr.write("Unable to find target.  "
-                         "Use --target or set TARGET_MOUNT_POINT\n")
+    """
+    configure installed system
+    """
+    wd = WorkingDir(target=args.target, state=args.state, scratch=args.scratch)
+
+    if wd.target is None:
+        sys.stderr.write("Unable to find target. Use --target \n")
         sys.exit(2)
 
+    stack_prefix = (args.report_stack_prefix
+                    if hasattr(args, 'report_stack_prefix') else '')
+
     # if network-config hook exists in target,
     # we do not run the builtin
-    if util.run_hook_if_exists(target, 'curtin-hooks'):
+    if util.run_hook_if_exists(wd.target, 'curtin-hooks'):
         sys.exit(0)
 
-    cfg = config.load_command_config(args, state)
-    stack_prefix = state.get('report_stack_prefix', '')
+    cfg = config.load_command_config(args)
 
     with events.ReportEventStack(
             name=stack_prefix, reporting_enabled=True, level="INFO",
             description="writing config files and configuring apt"):
-        write_files(cfg, target)
-        do_apt_config(cfg, target)
-        disable_overlayroot(cfg, target)
+        write_files(cfg, wd.target)
+        do_apt_config(cfg, wd.target)
+        disable_overlayroot(cfg, wd.target)
 
     # packages may be needed prior to installing kernel
-    install_missing_packages(cfg, target)
+    install_missing_packages(cfg, wd.target, stack_prefix)
 
     # If a mdadm.conf file was created by block_meta than it needs to be copied
     # onto the target system
-    mdadm_location = os.path.join(os.path.split(state['fstab'])[0],
-                                  "mdadm.conf")
-    if os.path.exists(mdadm_location):
-        copy_mdadm_conf(mdadm_location, target)
+    if util.has_content(wd.mdadm_config):
+        copy_mdadm_conf(wd.mdadm_config, wd.target)
         # as per https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/964052
         # reconfigure mdadm
         util.subp(['dpkg-reconfigure', '--frontend=noninteractive', 'mdadm'],
-                  data=None, target=target)
+                  data=None, target=wd.target)
+
+    # copy crypttab if any was written. this is done before install_kernel so
+    # that the initramfs will be built by when the kernel is installed
+    if util.has_content(wd.crypttab):
+        copy_crypttab(wd.crypttab, wd.target)
 
     with events.ReportEventStack(
             name=stack_prefix, reporting_enabled=True, level="INFO",
             description="installing kernel"):
-        setup_zipl(cfg, target)
-        install_kernel(cfg, target)
-        run_zipl(cfg, target)
+        setup_zipl(cfg, wd.target)
+        install_kernel(cfg, wd.target)
+        run_zipl(cfg, wd.target)
 
-        restore_dist_interfaces(cfg, target)
+        restore_dist_interfaces(cfg, wd.target)
 
     with events.ReportEventStack(
             name=stack_prefix, reporting_enabled=True, level="INFO",
             description="setting up swap"):
-        add_swap(cfg, target, state.get('fstab'))
+        add_swap(cfg, wd.target, wd.fstab)
 
     with events.ReportEventStack(
             name=stack_prefix, reporting_enabled=True, level="INFO",
             description="apply networking"):
-        apply_networking(target, state)
+        apply_networking(wd.target, wd.network_state,
+                         wd.network_config, wd.interfaces)
 
     with events.ReportEventStack(
             name=stack_prefix, reporting_enabled=True, level="INFO",
             description="writing etc/fstab"):
-        copy_fstab(state.get('fstab'), target)
+        copy_fstab(wd.fstab, wd.target)
 
     with events.ReportEventStack(
             name=stack_prefix, reporting_enabled=True, level="INFO",
             description="configuring multipath"):
-        detect_and_handle_multipath(cfg, target)
+        detect_and_handle_multipath(cfg, wd.target)
 
     with events.ReportEventStack(
             name=stack_prefix, reporting_enabled=True, level="INFO",
             description="updating packages on target system"):
-        system_upgrade(cfg, target)
-
-    # If a crypttab file was created by block_meta than it needs to be copied
-    # onto the target system, and update_initramfs() needs to be run, so that
-    # the cryptsetup hooks are properly configured on the installed system and
-    # it will be able to open encrypted volumes at boot.
-    crypttab_location = os.path.join(os.path.split(state['fstab'])[0],
-                                     "crypttab")
-    if os.path.exists(crypttab_location):
-        copy_crypttab(crypttab_location, target)
-        update_initramfs(target)
+        system_upgrade(cfg, wd.target)
 
     # If udev dname rules were created, copy them to target
-    udev_rules_d = os.path.join(state['scratch'], "rules.d")
-    if os.path.isdir(udev_rules_d):
-        copy_dname_rules(udev_rules_d, target)
+    if os.path.isdir(wd.rules_dir):
+        copy_dname_rules(wd.rules_dir, wd.target)
 
     # As a rule, ARMv7 systems don't use grub. This may change some
     # day, but for now, assume no. They do require the initramfs
@@ -772,9 +762,9 @@
     if (machine.startswith('armv7') or
             machine.startswith('s390x') or
             machine.startswith('aarch64') and not util.is_uefi_bootable()):
-        update_initramfs(target)
+        update_initramfs(wd.target)
     else:
-        setup_grub(cfg, target)
+        setup_grub(cfg, wd.target)
 
     sys.exit(0)
 

=== modified file 'curtin/commands/extract.py'
--- curtin/commands/extract.py	2016-07-13 07:50:49 +0000
+++ curtin/commands/extract.py	2016-08-22 05:07:32 +0000
@@ -27,9 +27,8 @@
 
 CMD_ARGUMENTS = (
     ((('-t', '--target'),
-      {'help': ('target directory to extract to (root) '
-                '[default TARGET_MOUNT_POINT]'),
-       'action': 'store', 'default': os.environ.get('TARGET_MOUNT_POINT')}),
+      {'help': ('target directory to extract to (root)'),
+       'action': 'store', 'default': None}),
      (('sources',),
       {'help': 'the sources to install [default read from CONFIG]',
        'nargs': '*'}),
@@ -83,11 +82,9 @@
     if not args.target:
         raise ValueError("Target must be defined or set in environment")
 
-    state = curtin.util.load_command_environment()
-    cfg = curtin.config.load_command_config(args, state)
+    cfg = curtin.config.load_command_config(args)
 
     sources = args.sources
-    target = args.target
     if not sources:
         if not cfg.get('sources'):
             raise ValueError("'sources' must be on cmdline or in config")
@@ -96,8 +93,9 @@
     if isinstance(sources, dict):
         sources = [sources[k] for k in sorted(sources.keys())]
 
-    LOG.debug("Installing sources: %s to target at %s" % (sources, target))
-    stack_prefix = state.get('report_stack_prefix', '')
+    LOG.debug("Installing sources: %s to target at %s" %
+              (sources, args.target))
+    stack_prefix = args.report_stack_prefix if args.report_stack_prefix else ''
 
     for source in sources:
         with events.ReportEventStack(
@@ -107,16 +105,16 @@
             if source['type'].startswith('dd-'):
                 continue
             if source['uri'].startswith("cp://"):
-                copy_to_target(source['uri'], target)
+                copy_to_target(source['uri'], args.target)
             elif os.path.isfile(source['uri']):
-                extract_root_tgz_file(source['uri'], target)
+                extract_root_tgz_file(source['uri'], args.target)
             elif source['uri'].startswith("file://"):
                 extract_root_tgz_file(
                     source['uri'][len("file://"):],
-                    target)
+                    args.target)
             elif (source['uri'].startswith("http://") or
                   source['uri'].startswith("https://")):
-                extract_root_tgz_url(source['uri'], target)
+                extract_root_tgz_url(source['uri'], args.target)
             else:
                 raise TypeError(
                     "do not know how to extract '%s'" %

=== modified file 'curtin/commands/hook.py'
--- curtin/commands/hook.py	2016-07-13 07:50:49 +0000
+++ curtin/commands/hook.py	2016-08-22 05:07:32 +0000
@@ -15,7 +15,6 @@
 #   You should have received a copy of the GNU Affero General Public License
 #   along with Curtin.  If not, see <http://www.gnu.org/licenses/>.
 
-import os
 import sys
 
 import curtin.config
@@ -25,17 +24,16 @@
 from . import populate_one_subcmd
 
 CMD_ARGUMENTS = (
-    ((('target',),
-      {'help': 'finalize the provided directory [default TARGET_MOUNT_POINT]',
-       'action': 'store', 'default': os.environ.get('TARGET_MOUNT_POINT'),
-       'nargs': '?'}),
+    ((('-t', '--target'),
+      {'help': 'finalize the provided directory',
+       'action': 'store', 'default': None, 'nargs': '?'}),
      )
 )
 
 
 def hook(args):
     if not args.target:
-        raise ValueError("Target must be provided or set in environment")
+        raise ValueError("Target must be provided")
 
     LOG.debug("Finalizing %s" % args.target)
     curtin.util.run_hook_if_exists(args.target, "finalize")

=== modified file 'curtin/commands/in_target.py'
--- curtin/commands/in_target.py	2016-07-13 07:50:49 +0000
+++ curtin/commands/in_target.py	2016-08-22 05:07:32 +0000
@@ -34,9 +34,8 @@
       {'help': 'capture/swallow output of command',
        'action': 'store_true', 'default': False}),
      (('-t', '--target'),
-      {'help': 'chroot to target. default is env[TARGET_MOUNT_POINT]',
-       'action': 'store', 'metavar': 'TARGET',
-       'default': os.environ.get('TARGET_MOUNT_POINT')}),
+      {'help': 'chroot to target.', 'action': 'store',
+       'metavar': 'TARGET', 'default': None}),
      ('command_args',
       {'help': 'run a command chrooted in the target', 'nargs': '*'}),
      )
@@ -56,26 +55,20 @@
 
 
 def in_target_main(args):
-    if args.target is not None:
-        target = args.target
-    else:
-        state = util.load_command_environment()
-        target = state['target']
-
     if args.target is None:
-        sys.stderr.write("Unable to find target.  "
-                         "Use --target or set TARGET_MOUNT_POINT\n")
+        sys.stderr.write("Unable to find target. Use --target.\n")
         sys.exit(2)
 
-    if os.path.abspath(target) == "/":
+    if os.path.abspath(args.target) == "/":
         cmd = args.command_args
     else:
-        cmd = ['chroot', target] + args.command_args
+        cmd = ['chroot', args.target] + args.command_args
 
-    if target == "/" and args.allow_daemons:
+    if args.target == "/" and args.allow_daemons:
         ret = run_command(cmd, args.interactive, capture=args.capture)
     else:
-        with util.ChrootableTarget(target, allow_daemons=args.allow_daemons):
+        with util.ChrootableTarget(
+                args.target, allow_daemons=args.allow_daemons):
             ret = run_command(cmd, args.interactive)
 
     sys.exit(ret)

=== modified file 'curtin/commands/install.py'
--- curtin/commands/install.py	2016-07-13 07:57:01 +0000
+++ curtin/commands/install.py	2016-08-22 05:07:32 +0000
@@ -16,21 +16,18 @@
 #   along with Curtin.  If not, see <http://www.gnu.org/licenses/>.
 
 import argparse
-import json
 import os
 import re
 import shlex
-import shutil
 import subprocess
 import sys
-import tempfile
 
-from curtin import block
 from curtin import config
 from curtin import util
 from curtin.log import LOG
 from curtin.reporter.legacy import load_reporter
 from curtin.reporter import events
+from curtin.workingdir import WorkingDir
 from . import populate_one_subcmd
 
 INSTALL_LOG = "/var/log/curtin/install.log"
@@ -83,54 +80,13 @@
         pass
 
 
-class WorkingDir(object):
-    def __init__(self, config):
-        top_d = tempfile.mkdtemp()
-        state_d = os.path.join(top_d, 'state')
-        target_d = os.path.join(top_d, 'target')
-        scratch_d = os.path.join(top_d, 'scratch')
-        for p in (state_d, target_d, scratch_d):
-            os.mkdir(p)
-
-        netconf_f = os.path.join(state_d, 'network_config')
-        netstate_f = os.path.join(state_d, 'network_state')
-        interfaces_f = os.path.join(state_d, 'interfaces')
-        config_f = os.path.join(state_d, 'config')
-        fstab_f = os.path.join(state_d, 'fstab')
-
-        with open(config_f, "w") as fp:
-            json.dump(config, fp)
-
-        # just touch these files to make sure they exist
-        for f in (interfaces_f, config_f, fstab_f, netconf_f, netstate_f):
-            with open(f, "ab") as fp:
-                pass
-
-        self.scratch = scratch_d
-        self.target = target_d
-        self.top = top_d
-        self.interfaces = interfaces_f
-        self.netconf = netconf_f
-        self.netstate = netstate_f
-        self.fstab = fstab_f
-        self.config = config
-        self.config_file = config_f
-
-    def env(self):
-        return ({'WORKING_DIR': self.scratch, 'OUTPUT_FSTAB': self.fstab,
-                 'OUTPUT_INTERFACES': self.interfaces,
-                 'OUTPUT_NETWORK_CONFIG': self.netconf,
-                 'OUTPUT_NETWORK_STATE': self.netstate,
-                 'TARGET_MOUNT_POINT': self.target,
-                 'CONFIG': self.config_file})
-
-
 class Stage(object):
 
-    def __init__(self, name, commands, env, reportstack=None, logfile=None):
+    def __init__(self, name, commands, workingdir,
+                 reportstack=None, logfile=None):
         self.name = name
         self.commands = commands
-        self.env = env
+        self.workingdir = workingdir
         if logfile is None:
             logfile = INSTALL_LOG
         self.install_log = self._open_install_log(logfile)
@@ -179,35 +135,42 @@
                 name=cmdname, description="running '%s'" % ' '.join(cmd),
                 parent=self.reportstack, level="DEBUG")
 
-            env = self.env.copy()
+            env = self.workingdir.env
             env['CURTIN_REPORTSTACK'] = cur_res.fullname
 
+            extra_args = ['='.join(arg) for arg in {
+                '--target': self.workingdir.target,
+                '--scratch': self.workingdir.scratch,
+                '--state': self.workingdir.state,
+                '--report-stack-prefix': cur_res.fullname
+            }.items()]
             shell = not isinstance(cmd, list)
-            with util.LogTimer(LOG.debug, cmdname):
-                with cur_res:
-                    try:
-                        sp = subprocess.Popen(
-                            cmd, stdout=subprocess.PIPE,
-                            stderr=subprocess.STDOUT,
-                            env=env, shell=shell)
-                    except OSError as e:
-                        LOG.warn("%s command failed", cmdname)
-                        raise util.ProcessExecutionError(cmd=cmd, reason=e)
-
-                    output = b""
-                    while True:
-                        data = sp.stdout.read(1)
-                        if not data and sp.poll() is not None:
-                            break
-                        self.write(data)
-                        output += data
-
-                    rc = sp.returncode
-                    if rc != 0:
-                        LOG.warn("%s command failed", cmdname)
-                        raise util.ProcessExecutionError(
-                            stdout=output, stderr="",
-                            exit_code=rc, cmd=cmd)
+            if cmd.startswith('curtin') if shell else cmd[0] == 'curtin':
+                cmd += ' '.join(extra_args) if shell else extra_args
+
+            with util.LogTimer(LOG.debug, cmdname), cur_res:
+                try:
+                    sp = subprocess.Popen(cmd, stdout=subprocess.PIPE,
+                                          stderr=subprocess.STDOUT,
+                                          shell=shell)
+                except OSError as e:
+                    LOG.warn("%s command failed", cmdname)
+                    raise util.ProcessExecutionError(cmd=cmd, reason=e)
+
+                output = b""
+                while True:
+                    data = sp.stdout.read(1)
+                    if not data and sp.poll() is not None:
+                        break
+                    self.write(data)
+                    output += data
+
+                rc = sp.returncode
+                if rc != 0:
+                    LOG.warn("%s command failed", cmdname)
+                    raise util.ProcessExecutionError(
+                        stdout=output, stderr="",
+                        exit_code=rc, cmd=cmd)
 
 
 def apply_power_state(pstate):
@@ -375,60 +338,53 @@
     post_files = cfg.get('post_files', [logfile])
     legacy_reporter = load_reporter(cfg)
     legacy_reporter.files = post_files
-
     args.reportstack.post_files = post_files
-    try:
-        dd_images = util.get_dd_images(cfg.get('sources', {}))
-        if len(dd_images) > 1:
-            raise ValueError("You may not use more then one disk image")
-
-        workingd = WorkingDir(cfg)
-        LOG.debug(workingd.env())
-        env = os.environ.copy()
-        env.update(workingd.env())
-
-        for name in cfg.get('stages'):
-            desc = STAGE_DESCRIPTIONS.get(name, "stage %s" % name)
-            reportstack = events.ReportEventStack(
-                "stage-%s" % name, description=desc,
-                parent=args.reportstack)
-            env['CURTIN_REPORTSTACK'] = reportstack.fullname
-
-            with reportstack:
-                commands_name = '%s_commands' % name
-                with util.LogTimer(LOG.debug, 'stage_%s' % name):
-                    stage = Stage(name, cfg.get(commands_name, {}), env,
-                                  reportstack=reportstack, logfile=logfile)
-                    stage.run()
-
-        if apply_kexec(cfg.get('kexec'), workingd.target):
-            cfg['power_state'] = {'mode': 'reboot', 'delay': 'now',
-                                  'message': "'rebooting with kexec'"}
-
-        writeline(logfile, INSTALL_PASS_MSG)
-        out = sys.stdout
-        msg = "%s\n" % INSTALL_PASS_MSG
-        if hasattr(out, 'buffer'):
-            out = out.buffer
-            msg = msg.encode()
-        out.write(msg)
-        out.flush()
-        legacy_reporter.report_success()
-    except Exception as e:
-        exp_msg = "Installation failed with exception: %s" % e
-        writeline(logfile, exp_msg)
-        LOG.error(exp_msg)
-        legacy_reporter.report_failure(exp_msg)
-        raise e
-    finally:
-        for d in ('sys', 'dev', 'proc'):
-            util.do_umount(os.path.join(workingd.target, d))
-        mounted = block.get_mountpoints()
-        mounted.sort(key=lambda x: -1 * x.count("/"))
-        for d in filter(lambda x: workingd.target in x, mounted):
-            util.do_umount(d)
-        util.do_umount(workingd.target)
-        shutil.rmtree(workingd.top)
+
+    # create tmp dir
+    with WorkingDir(config=cfg) as workingd:
+        try:
+            dd_images = util.get_dd_images(cfg.get('sources', {}))
+            if len(dd_images) > 1:
+                raise ValueError("You may not use more then one disk image")
+
+            LOG.debug(workingd.env)
+            env = os.environ.copy()
+            env.update(workingd.env)
+
+            for name in cfg.get('stages'):
+                desc = STAGE_DESCRIPTIONS.get(name, "stage %s" % name)
+                reportstack = events.ReportEventStack(
+                    "stage-%s" % name, description=desc,
+                    parent=args.reportstack)
+                env['CURTIN_REPORTSTACK'] = reportstack.fullname
+
+                with reportstack:
+                    commands_name = '%s_commands' % name
+                    with util.LogTimer(LOG.debug, 'stage_%s' % name):
+                        stage = Stage(name, cfg.get(commands_name, {}),
+                                      workingd, reportstack=reportstack,
+                                      logfile=logfile)
+                        stage.run()
+
+            if apply_kexec(cfg.get('kexec'), workingd.target):
+                cfg['power_state'] = {'mode': 'reboot', 'delay': 'now',
+                                      'message': "'rebooting with kexec'"}
+
+            writeline(logfile, INSTALL_PASS_MSG)
+            out = sys.stdout
+            msg = "%s\n" % INSTALL_PASS_MSG
+            if hasattr(out, 'buffer'):
+                out = out.buffer
+                msg = msg.encode()
+            out.write(msg)
+            out.flush()
+            legacy_reporter.report_success()
+        except Exception as e:
+            exp_msg = "Installation failed with exception: %s" % e
+            writeline(logfile, exp_msg)
+            LOG.error(exp_msg)
+            legacy_reporter.report_failure(exp_msg)
+            raise e
 
     apply_power_state(cfg.get('power_state'))
 

=== modified file 'curtin/commands/main.py'
--- curtin/commands/main.py	2016-07-26 13:43:15 +0000
+++ curtin/commands/main.py	2016-08-22 05:07:32 +0000
@@ -157,7 +157,7 @@
             elif flag in ('--set'):
                 config.merge_cmdarg(cfg, val)
     else:
-        cfg = config.load_command_config(args, util.load_command_environment())
+        cfg = config.load_command_config(args)
 
     args.config = cfg
 
@@ -196,13 +196,14 @@
     # set up the reportstack
     update_configuration(cfg.get('reporting', {}))
 
-    stack_prefix = (os.environ.get("CURTIN_REPORTSTACK", "") +
-                    "/cmd-%s" % args.subcmd)
-    if stack_prefix.startswith("/"):
-        stack_prefix = stack_prefix[1:]
-    os.environ["CURTIN_REPORTSTACK"] = stack_prefix
+    if not args.report_stack_prefix:
+        args.report_stack_prefix = ''
+    args.report_stack_prefix = (args.report_stack_prefix +
+                                '/cmd-{}'.format(args.subcmd))
+    if args.report_stack_prefix.startswith("/"):
+        args.report_stack_prefix = args.report_stack_prefix[1:]
     args.reportstack = events.ReportEventStack(
-        name=stack_prefix, reporting_enabled=True, level="DEBUG",
+        name=args.report_stack_prefix, reporting_enabled=True, level="DEBUG",
         description="curtin command %s" % args.subcmd)
 
     try:

=== modified file 'curtin/commands/net_meta.py'
--- curtin/commands/net_meta.py	2016-07-13 07:50:49 +0000
+++ curtin/commands/net_meta.py	2016-08-22 05:07:32 +0000
@@ -21,6 +21,7 @@
 
 from curtin import net
 from curtin.log import LOG
+from curtin.workingdir import WorkingDir
 import curtin.util as util
 import curtin.config as config
 
@@ -71,8 +72,7 @@
 
 
 def interfaces_custom(args):
-    state = util.load_command_environment()
-    cfg = config.load_command_config(args, state)
+    cfg = config.load_command_config(args)
 
     network_config = cfg.get('network', [])
     if not network_config:
@@ -93,8 +93,7 @@
     if util.run_hook_if_exists(args.target, 'network-config'):
         sys.exit(0)
 
-    state = util.load_command_environment()
-    cfg = config.load_command_config(args, state)
+    cfg = config.load_command_config(args)
     if cfg.get("network") is not None:
         args.mode = "custom"
 
@@ -124,7 +123,9 @@
 
     LOG.debug("net-meta mode is '%s'.  devices=%s", args.mode, devices)
 
-    output_network_config = os.environ.get("OUTPUT_NETWORK_CONFIG", "")
+    wd = WorkingDir(state=args.state)
+    output_network_config = wd.network_config
+
     if args.mode == "copy":
         if not args.target:
             raise argparse.ArgumentTypeError("mode 'copy' requires --target")
@@ -167,13 +168,11 @@
       {'help': 'which devices to operate on', 'action': 'append',
        'metavar': 'DEVICE', 'type': network_device}),
      (('-o', '--output'),
-      {'help': 'file to write to. defaults to env["OUTPUT_INTERFACES"] or "-"',
-       'metavar': 'IFILE', 'action': 'store',
-       'default': os.environ.get('OUTPUT_INTERFACES', "-")}),
+      {'help': 'file to write to.', 'metavar': 'IFILE',
+       'action': 'store', 'default': None}),
      (('-t', '--target'),
-      {'help': 'operate on target. default is env[TARGET_MOUNT_POINT]',
-       'action': 'store', 'metavar': 'TARGET',
-       'default': os.environ.get('TARGET_MOUNT_POINT')}),
+      {'help': 'operate on target.', 'action': 'store',
+       'metavar': 'TARGET', 'default': None}),
      ('mode', {'help': 'meta-mode to use',
                'choices': ['dhcp', 'copy', 'auto', 'custom']})
      )

=== modified file 'curtin/commands/swap.py'
--- curtin/commands/swap.py	2016-07-13 07:50:49 +0000
+++ curtin/commands/swap.py	2016-08-22 05:07:32 +0000
@@ -15,28 +15,23 @@
 #   You should have received a copy of the GNU Affero General Public License
 #   along with Curtin.  If not, see <http://www.gnu.org/licenses/>.
 
-import os
 import sys
 
 import curtin.swap as swap
 import curtin.util as util
+import curtin.workingdir as workingdir
 
 from . import populate_one_subcmd
 
 
 def swap_main(args):
     #  curtin swap [--size=4G] [--target=/] [--fstab=/etc/fstab] [swap]
-    state = util.load_command_environment()
-
-    if args.target is not None:
-        state['target'] = args.target
-
-    if args.fstab is not None:
-        state['fstab'] = args.fstab
-
-    if state['target'] is None:
-        sys.stderr.write("Unable to find target.  "
-                         "Use --target or set TARGET_MOUNT_POINT\n")
+    if not args.fstab and args.state:
+        wd = workingdir.WorkingDir(state=args.state)
+        args.fstab = wd.fstab
+
+    if args.target is None:
+        sys.stderr.write("Unable to find target. Use --target\n")
         sys.exit(2)
 
     size = args.size
@@ -53,7 +48,7 @@
     if args.maxsize is not None:
         args.maxsize = util.human2bytes(args.maxsize)
 
-    swap.setup_swapfile(target=state['target'], fstab=state['fstab'],
+    swap.setup_swapfile(target=args.target, fstab=args.fstab,
                         swapfile=args.swapfile, size=size,
                         maxsize=args.maxsize)
     sys.exit(2)
@@ -61,14 +56,11 @@
 
 CMD_ARGUMENTS = (
     ((('-f', '--fstab'),
-      {'help': 'file to write to. defaults to env["OUTPUT_FSTAB"]',
-       'metavar': 'FSTAB', 'action': 'store',
-       'default': os.environ.get('OUTPUT_FSTAB')}),
+      {'help': 'file to write to.', 'metavar': 'FSTAB', 'action': 'store',
+       'default': None}),
      (('-t', '--target'),
-      {'help': ('target filesystem root to add swap file to. '
-                'default is env[TARGET_MOUNT_POINT]'),
-       'action': 'store', 'metavar': 'TARGET',
-       'default': os.environ.get('TARGET_MOUNT_POINT')}),
+      {'help': ('target filesystem root to add swap file to.'),
+       'action': 'store', 'metavar': 'TARGET', 'default': None}),
      (('-s', '--size'),
       {'help': 'size of swap file (eg: 1G, 1500M, 1024K, 100000. def: "auto")',
                'default': None, 'action': 'store'}),

=== modified file 'curtin/commands/system_install.py'
--- curtin/commands/system_install.py	2016-07-13 07:50:49 +0000
+++ curtin/commands/system_install.py	2016-08-22 05:07:32 +0000
@@ -15,7 +15,6 @@
 #   You should have received a copy of the GNU Affero General Public License
 #   along with Curtin.  If not, see <http://www.gnu.org/licenses/>.
 
-import os
 import sys
 
 import curtin.util as util
@@ -46,10 +45,8 @@
       {'help': ('do not disable running of daemons during upgrade.'),
        'action': 'store_true', 'default': False}),
      (('-t', '--target'),
-      {'help': ('target root to upgrade. '
-                'default is env[TARGET_MOUNT_POINT]'),
-       'action': 'store', 'metavar': 'TARGET',
-       'default': os.environ.get('TARGET_MOUNT_POINT')}),
+      {'help': ('target root to upgrade.'),
+       'action': 'store', 'metavar': 'TARGET', 'default': None}),
      ('packages',
       {'help': 'the list of packages to install',
        'metavar': 'PACKAGES', 'action': 'store', 'nargs': '+'}),

=== modified file 'curtin/commands/system_upgrade.py'
--- curtin/commands/system_upgrade.py	2016-07-13 07:50:49 +0000
+++ curtin/commands/system_upgrade.py	2016-08-22 05:07:32 +0000
@@ -15,7 +15,6 @@
 #   You should have received a copy of the GNU Affero General Public License
 #   along with Curtin.  If not, see <http://www.gnu.org/licenses/>.
 
-import os
 import sys
 
 import curtin.util as util
@@ -45,10 +44,8 @@
       {'help': ('do not disable running of daemons during upgrade.'),
        'action': 'store_true', 'default': False}),
      (('-t', '--target'),
-      {'help': ('target root to upgrade. '
-                'default is env[TARGET_MOUNT_POINT]'),
-       'action': 'store', 'metavar': 'TARGET',
-       'default': os.environ.get('TARGET_MOUNT_POINT')}),
+      {'help': ('target root to upgrade.'),
+       'action': 'store', 'metavar': 'TARGET', 'default': None}),
      )
 )
 

=== modified file 'curtin/config.py'
--- curtin/config.py	2016-03-11 15:14:46 +0000
+++ curtin/config.py	2016-08-22 05:07:32 +0000
@@ -15,8 +15,11 @@
 #   You should have received a copy of the GNU Affero General Public License
 #   along with Curtin.  If not, see <http://www.gnu.org/licenses/>.
 
+import os
+import json
 import yaml
-import json
+
+from curtin import workingdir
 
 ARCHIVE_HEADER = "#curtin-config-archive"
 ARCHIVE_TYPE = "text/curtin-config-archive"
@@ -119,17 +122,14 @@
         return load_config_archive(content)
 
 
-def load_command_config(args, state):
+def load_command_config(args):
+    cfg = {}
     if hasattr(args, 'config') and args.config:
-        return args.config
-    else:
-        # state 'config' points to a file with fully rendered config
-        cfg_file = state.get('config')
-
-    if not cfg_file:
-        cfg = {}
-    else:
-        cfg = load_config(cfg_file)
+        cfg = args.config
+    elif hasattr(args, 'state') and args.state is not None:
+        wd = workingdir.WorkingDir(state=args.state)
+        if os.path.exists(wd.config):
+            cfg = load_config(wd.config)
     return cfg
 
 

=== modified file 'curtin/util.py'
--- curtin/util.py	2016-07-29 17:19:20 +0000
+++ curtin/util.py	2016-08-22 05:07:32 +0000
@@ -254,42 +254,6 @@
                      (self.msg, time.time() - self.start))
 
 
-def is_mounted(target, src=None, opts=None):
-    # return whether or not src is mounted on target
-    mounts = ""
-    with open("/proc/mounts", "r") as fp:
-        mounts = fp.read()
-
-    for line in mounts.splitlines():
-        if line.split()[1] == os.path.abspath(target):
-            return True
-    return False
-
-
-def do_mount(src, target, opts=None):
-    # mount src at target with opts and return True
-    # if already mounted, return False
-    if opts is None:
-        opts = []
-    if isinstance(opts, str):
-        opts = [opts]
-
-    if is_mounted(target, src, opts):
-        return False
-
-    ensure_dir(target)
-    cmd = ['mount'] + opts + [src, target]
-    subp(cmd)
-    return True
-
-
-def do_umount(mountpoint):
-    if not is_mounted(mountpoint):
-        return False
-    subp(['umount', mountpoint])
-    return True
-
-
 def ensure_dir(path, mode=None):
     try:
         os.makedirs(path)
@@ -323,6 +287,13 @@
             raise e
 
 
+def has_content(path):
+    """
+    check if file exists and is not empty
+    """
+    return os.path.exists(path) and os.stat(path).st_size > 0
+
+
 def disable_daemons_in_root(target):
     contents = "\n".join(
         ['#!/bin/sh',
@@ -355,6 +326,49 @@
     return True
 
 
+def is_mounted(target, src=None, opts=None):
+    """
+    return whether or not src is mounted on target
+    """
+    mounts = ""
+    with open("/proc/mounts", "r") as fp:
+        mounts = fp.read()
+
+    for line in mounts.splitlines():
+        if line.split()[1] == os.path.abspath(target):
+            return True
+    return False
+
+
+def do_mount(src, target, opts=None):
+    """
+    mount src at target with opts and return True
+    if already mounted, return False
+    """
+    if opts is None:
+        opts = []
+    if isinstance(opts, str):
+        opts = [opts]
+
+    if is_mounted(target, src, opts):
+        return False
+
+    ensure_dir(target)
+    cmd = ['mount'] + opts + [src, target]
+    subp(cmd)
+    return True
+
+
+def do_umount(mountpoint):
+    """
+    unmount specified mountpoint
+    """
+    if not is_mounted(mountpoint):
+        return False
+    subp(['umount', mountpoint])
+    return True
+
+
 class ChrootableTarget(object):
     def __init__(self, target, allow_daemons=False, sys_resolvconf=True):
         if target is None:

=== added file 'curtin/workingdir.py'
--- curtin/workingdir.py	1970-01-01 00:00:00 +0000
+++ curtin/workingdir.py	2016-08-22 05:07:32 +0000
@@ -0,0 +1,106 @@
+#   Copyright (C) 2016 Canonical Ltd.
+#
+#   Author: Wesley Wiedenmeier <[e-mail]>
+#
+#   Curtin is free software: you can redistribute it and/or modify it under
+#   the terms of the GNU Affero General Public License as published by the
+#   Free Software Foundation, either version 3 of the License, or (at your
+#   option) any later version.
+#
+#   Curtin is distributed in the hope that it will be useful, but WITHOUT ANY
+#   WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+#   FOR A PARTICULAR PURPOSE.  See the GNU Affero General Public License for
+#   more details.
+#
+#   You should have received a copy of the GNU Affero General Public License
+#   along with Curtin.  If not, see <http://www.gnu.org/licenses/>.
+
+# this has to be in a seperate module from util because curtin.block does a
+# relative import from util before the name 'block' exists in sys.modules. this
+# is permitted in python3 but not python2.7
+
+from curtin import (block, util)
+from curtin.log import LOG
+
+import os
+import tempfile
+import shutil
+
+
+def umount_all(basedir):
+    """
+    unmount all mountpoints under base dir
+    """
+    mounts = sorted((m for m in block.get_mountpoints() if basedir in m),
+                    key=lambda x: -1 * x.count(os.path.sep))
+    LOG.debug('removing mounts: %s', mounts)
+    for mount in mounts:
+        util.do_umount(mount)
+
+
+class WorkingDir(object):
+    def __init__(self, target=None, state=None, scratch=None, config=None):
+        self._config_dict = config
+
+        # base tmpdir will not be used if state target and scratch specified
+        self.base = os.path.join(tempfile.gettempdir(),
+                                 next(tempfile._get_candidate_names()))
+        self.target = target if target else os.path.join(self.base, 'target')
+        self.state = state if state else os.path.join(self.base, 'state')
+        self.scratch = (scratch if scratch else
+                        os.path.join(self.base, 'scratch'))
+        self.rules_dir = os.path.join(self.scratch, 'rules.d')
+
+        # only dirs which are not passed in should be removed on exit
+        self._dirs_to_clean = []
+        if not state:
+            self._dirs_to_clean.append(self.state)
+        if not scratch:
+            self._dirs_to_clean.append(self.scratch)
+        if not target:
+            self._dirs_to_clean.append(self.target)
+        self._dirs_to_clean.append(self.base)
+
+        # set state files
+        self._state_files = ('fstab', 'config', 'interfaces',
+                             'network_state', 'network_config',
+                             'mdadm_config', 'crypttab')
+        for state_file in self._state_files:
+            setattr(self, state_file, os.path.join(self.state, state_file))
+
+    def __enter__(self):
+        # create subdirs, basedir is not created if not needed
+        for path in (self.target, self.state, self.scratch, self.rules_dir):
+            util.ensure_dir(path)
+
+        # touch state files to make sure they exist
+        for state_file in self._state_files:
+            with open(getattr(self, state_file), 'ab'):
+                pass
+
+        # dump config
+        if self._config_dict is not None:
+            util.write_file(
+                self.config, util.json_dumps(self._config_dict), omode='wb')
+
+        return self
+
+    def __exit__(self, etype, value, trace):
+        for path in [p for p in self._dirs_to_clean if os.path.exists(p)]:
+            umount_all(path)
+            shutil.rmtree(path)
+
+        # raise all encountered errors
+        return False
+
+    @property
+    def env(self):
+        return {
+            'WORKING_DIR': self.scratch,
+            'OUTPUT_FSTAB': self.fstab,
+            'OUTPUT_INTERFACES': self.interfaces,
+            'OUTPUT_NETWORK_CONFIG': self.network_config,
+            'OUTPUT_NETWORK_STATE': self.network_state,
+            'TARGET_MOUNT_POINT': self.target,
+            'CONFIG': self.config
+        }

=== modified file 'tests/unittests/test_make_dname.py'
--- tests/unittests/test_make_dname.py	2016-07-13 17:07:11 +0000
+++ tests/unittests/test_make_dname.py	2016-08-22 05:07:32 +0000
@@ -8,7 +8,6 @@
 
 
 class TestMakeDname(TestCase):
-    state = {'scratch': '/tmp/null'}
     rules_d = '/tmp/null/rules.d'
     rule_file = '/tmp/null/rules.d/{}.rules'
     storage_config = {
@@ -56,7 +55,6 @@
         disk_ptuuid = str(uuid.uuid1())
         mock_util.subp.side_effect = self._make_mock_subp_blkid(
             disk_ptuuid, self.disk_blkid)
-        mock_util.load_command_environment.return_value = self.state
         rule_identifiers = [
             ('DEVTYPE', 'disk'),
             ('ID_PART_TABLE_UUID', disk_ptuuid)
@@ -64,7 +62,7 @@
 
         # simple run
         res_dname = 'main_disk'
-        block_meta.make_dname('disk1', self.storage_config)
+        block_meta.make_dname('disk1', self.storage_config, self.rules_d)
         mock_util.ensure_dir.assert_called_with(self.rules_d)
         self.assertTrue(mock_log.debug.called)
         self.assertFalse(mock_log.warning.called)
@@ -74,7 +72,7 @@
 
         # run invalid dname
         res_dname = 'in_valid-name----------disk'
-        block_meta.make_dname('disk2', self.storage_config)
+        block_meta.make_dname('disk2', self.storage_config, self.rules_d)
         self.assertTrue(mock_log.warning.called)
         mock_util.write_file.assert_called_with(
             self.rule_file.format(res_dname),
@@ -86,17 +84,16 @@
     def test_make_dname_failures(self, mock_util, mock_get_path, mock_log):
         mock_util.subp.side_effect = self._make_mock_subp_blkid(
             '', self.trusty_blkid)
-        mock_util.load_command_environment.return_value = self.state
 
         warning_msg = "Can't find a uuid for volume: {}. Skipping dname."
 
         # disk with no PT_UUID
-        block_meta.make_dname('disk1', self.storage_config)
+        block_meta.make_dname('disk1', self.storage_config, self.rules_d)
         mock_log.warning.assert_called_with(warning_msg.format('disk1'))
         self.assertFalse(mock_util.write_file.called)
 
         # partition with no PART_UUID
-        block_meta.make_dname('disk1p1', self.storage_config)
+        block_meta.make_dname('disk1p1', self.storage_config, self.rules_d)
         mock_log.warning.assert_called_with(warning_msg.format('disk1p1'))
         self.assertFalse(mock_util.write_file.called)
 
@@ -107,7 +104,6 @@
         part_uuid = str(uuid.uuid1())
         mock_util.subp.side_effect = self._make_mock_subp_blkid(
             part_uuid, self.part_blkid)
-        mock_util.load_command_environment.return_value = self.state
 
         rule_identifiers = [
             ('DEVTYPE', 'partition'),
@@ -116,7 +112,7 @@
 
         # simple run
         res_dname = 'main_disk-part1'
-        block_meta.make_dname('disk1p1', self.storage_config)
+        block_meta.make_dname('disk1p1', self.storage_config, self.rules_d)
         mock_util.ensure_dir.assert_called_with(self.rules_d)
         self.assertTrue(mock_log.debug.called)
         self.assertFalse(mock_log.warning.called)
@@ -126,7 +122,7 @@
 
         # run invalid dname
         res_dname = 'in_valid-name----------disk-part1'
-        block_meta.make_dname('disk2p1', self.storage_config)
+        block_meta.make_dname('disk2p1', self.storage_config, self.rules_d)
         self.assertTrue(mock_log.warning.called)
         mock_util.write_file.assert_called_with(
             self.rule_file.format(res_dname),
@@ -140,12 +136,11 @@
                              mock_mdadm):
         md_uuid = str(uuid.uuid1())
         mock_mdadm.mdadm_query_detail.return_value = {'MD_UUID': md_uuid}
-        mock_util.load_command_environment.return_value = self.state
         rule_identifiers = [('MD_UUID', md_uuid)]
 
         # simple
         res_dname = 'mdadm_name'
-        block_meta.make_dname('md_id', self.storage_config)
+        block_meta.make_dname('md_id', self.storage_config, self.rules_d)
         self.assertTrue(mock_log.debug.called)
         self.assertFalse(mock_log.warning.called)
         mock_util.write_file.assert_called_with(
@@ -154,7 +149,7 @@
 
         # invalid name
         res_dname = 'mdadm-name'
-        block_meta.make_dname('md_id2', self.storage_config)
+        block_meta.make_dname('md_id2', self.storage_config, self.rules_d)
         self.assertTrue(mock_log.warning.called)
         mock_util.write_file.assert_called_with(
             self.rule_file.format(res_dname),
@@ -165,12 +160,10 @@
     @mock.patch('curtin.commands.block_meta.util')
     def test_make_dname_lvm_partition(self, mock_util, mock_get_path,
                                       mock_log):
-        mock_util.load_command_environment.return_value = self.state
-
         # simple
         res_dname = 'vg1-lpartition1'
         rule_identifiers = [('DM_NAME', res_dname)]
-        block_meta.make_dname('lpart_id', self.storage_config)
+        block_meta.make_dname('lpart_id', self.storage_config, self.rules_d)
         self.assertTrue(mock_log.debug.called)
         self.assertFalse(mock_log.warning.called)
         mock_util.write_file.assert_called_with(
@@ -180,7 +173,7 @@
         # with invalid name
         res_dname = 'vg1-lvm-part-2'
         rule_identifiers = [('DM_NAME', 'vg1-lvm part/2')]
-        block_meta.make_dname('lpart2_id', self.storage_config)
+        block_meta.make_dname('lpart2_id', self.storage_config, self.rules_d)
         self.assertTrue(mock_log.warning.called)
         mock_util.write_file.assert_called_with(
             self.rule_file.format(res_dname),

=== added file 'tests/unittests/test_workingdir.py'
--- tests/unittests/test_workingdir.py	1970-01-01 00:00:00 +0000
+++ tests/unittests/test_workingdir.py	2016-08-22 05:07:32 +0000
@@ -0,0 +1,95 @@
+from unittest import TestCase
+from curtin import workingdir
+import mock
+import os
+
+
+class TestWorkingDir(TestCase):
+    base_tmp = '/tmp'
+    tmp_name = 'asdf'
+    mock_cfg = {'testkey': 'value'}
+
+    @mock.patch('curtin.workingdir.tempfile')
+    def test_attributes(self, mock_tempfile):
+        mock_tempfile.gettempdir.return_value = self.base_tmp
+        mock_tempfile._get_candidate_names.return_value = (
+            (name for name in (self.tmp_name,)))
+        wd = workingdir.WorkingDir()
+        self.assertIsNone(wd._config_dict)
+        self.assertTrue(mock_tempfile.gettempdir.called)
+        self.assertTrue(mock_tempfile._get_candidate_names.called)
+        for name in ('target', 'state', 'scratch'):
+            self.assertEqual(getattr(wd, name), os.path.sep.join(
+                (self.base_tmp, self.tmp_name, name)))
+        self.assertEqual(wd.base, os.path.sep.join((self.base_tmp,
+                                                    self.tmp_name)))
+        for name in ('fstab', 'config', 'interfaces', 'network_state',
+                     'network_config', 'mdadm_config', 'crypttab'):
+            self.assertEqual(getattr(wd, name), os.path.sep.join(
+                (self.base_tmp, self.tmp_name, 'state', name)))
+
+    @mock.patch('curtin.workingdir.tempfile')
+    @mock.patch('curtin.workingdir.open')
+    @mock.patch('curtin.workingdir.util')
+    def test_enter_creates_files(self, mock_util, mock_open, mock_tempfile):
+        """
+        ensure that tmp directories are created and state files are touched
+        on __enter__ when using WorkingDir as a context manager
+        """
+        mock_tempfile.gettempdir.return_value = self.base_tmp
+        mock_tempfile._get_candidate_names.return_value = (
+            (name for name in (self.tmp_name,)))
+        mock_util.json_dumps.side_effect = lambda x: str(x)
+        with workingdir.WorkingDir(config=self.mock_cfg) as wd:
+            self.assertIsInstance(wd, workingdir.WorkingDir)
+            self.assertEqual(wd._config_dict, self.mock_cfg)
+            self.assertEqual(wd.base, os.path.sep.join((self.base_tmp,
+                                                        self.tmp_name)))
+        for (count, dir_name) in enumerate(('target', 'state', 'scratch',
+                                            'scratch/rules.d')):
+            self.assertEqual(mock_util.ensure_dir.call_args_list[count],
+                             mock.call(os.path.sep.join(
+                                 (self.base_tmp, self.tmp_name, dir_name))))
+        for (count, target_file) in enumerate(
+                ('fstab', 'config', 'interfaces', 'network_state',
+                 'network_config', 'mdadm_config', 'crypttab')):
+            self.assertEqual(
+                mock_open.call_args_list[count],
+                mock.call(os.path.sep.join((self.base_tmp, self.tmp_name,
+                                            'state', target_file)),
+                          'ab'))
+        mock_util.json_dumps.assert_called_with(self.mock_cfg)
+        mock_util.write_file.assert_called_once_with(
+            os.path.sep.join(
+                (self.base_tmp, self.tmp_name, 'state', 'config')),
+            str(self.mock_cfg), omode='wb')
+
+    @mock.patch('curtin.workingdir.tempfile')
+    @mock.patch('curtin.workingdir.open')
+    @mock.patch('curtin.workingdir.util')
+    @mock.patch('curtin.workingdir.os')
+    @mock.patch('curtin.workingdir.umount_all')
+    @mock.patch('curtin.workingdir.shutil')
+    def test_exit_removes_unspecified(self, mock_shutil, mock_umount_all,
+                                      mock_os, mock_util, mock_open,
+                                      mock_tempfile):
+        """
+        test that workingdir exit removes only dirs that were not specified
+        """
+        mock_tempfile.gettempdir.return_value = self.base_tmp
+        mock_tempfile._get_candidate_names.return_value = (
+            (name for name in (self.tmp_name,)))
+        target_dir = '/tmp/target'
+        mock_os.path.exists.return_value = True
+        mock_os.path.join.side_effect = os.path.join
+        with workingdir.WorkingDir(target=target_dir):
+            pass
+        for (count, dir_name) in enumerate(('state', 'scratch', '')):
+            path = os.path.sep.join((self.base_tmp, self.tmp_name, dir_name))
+            path = path.rstrip('/')
+            self.assertEqual(mock_os.path.exists.call_args_list[count],
+                             mock.call(path))
+            self.assertEqual(mock_umount_all.call_args_list[count],
+                             mock.call(path))
+            self.assertEqual(mock_shutil.rmtree.call_args_list[count],
+                             mock.call(path))