Merge lp:~sergiusens/snapcraft/filesets into lp:~snappy-dev/snapcraft/core

[Sergio Schvezov]

Filesets based filtering for stage and snap

I'm bit torn on this one.

Right now, the plugin can add additional filters to the snap stage, but that is not in spec. I'm not sure we want this in the part declaration, but that breaks bad when we have plugins that depend on other plugins to run providing no means to configure them easily.

I can try and hack this into the plugin yaml files (at the top level with options and build-deps) but that is one layer above.

I'm also hesitant to provide a 'stage_fileset' method if we don't go down the yaml path.

[Michael Vogt (mvo)]

Thanks, this looks very nice, some inline comments but mostly idle remarks, fine to land IMO. I have no good answer for your questions though. I agree that it feels better to keep a way to filter in the plugin and I think thats ok for part -> stage. There was some controversy about stage -> final (snap).

review: Approve

[Sergio Schvezov (sergiusens)]

@mvo, thanks for the review!

[lp:~sergiusens/snapcraft/filesets]

177. By Sergio Schvezov on 2015-09-11

escape char

178. By Sergio Schvezov on 2015-09-11

not so aggresive filtering for plugins or everything is included

[Sergio Schvezov (sergiusens)]

fixes from review

review: Approve

=== modified file 'examples/gopaste/snapcraft.yaml'
--- examples/gopaste/snapcraft.yaml	2015-09-08 11:48:30 +0000
+++ examples/gopaste/snapcraft.yaml	2015-09-11 13:39:05 +0000
@@ -4,7 +4,7 @@
 services:
   - name: gopaste
     description: "gopaste"
-    start: bin/gopaste
+    start: bin/gopasted
 summary: Simple pasting tool
 description: Runs a service that allows you to paste to and share.
 icon: icon.png

=== modified file 'plugins/ant-project.yaml'
--- plugins/ant-project.yaml	2015-09-02 21:27:17 +0000
+++ plugins/ant-project.yaml	2015-09-11 13:39:05 +0000
@@ -9,5 +9,7 @@
   source-tag:
   source-branch:
   stage-packages:
+  snap:
+  stage:
 build-tools:
   - ant

=== modified file 'plugins/autotools-project.yaml'
--- plugins/autotools-project.yaml	2015-09-02 15:10:58 +0000
+++ plugins/autotools-project.yaml	2015-09-11 13:39:05 +0000
@@ -9,3 +9,5 @@
   configflags:
     required: false
   stage-packages:
+  snap:
+  stage:

=== modified file 'plugins/cmake-project.yaml'
--- plugins/cmake-project.yaml	2015-09-02 15:10:58 +0000
+++ plugins/cmake-project.yaml	2015-09-11 13:39:05 +0000
@@ -1,11 +1,13 @@
 module: cmake_project
 build-tools: [cmake]
 options:
-    source:
-        required: true
-    source-type:
-    source-tag:
-    source-branch:
-    configflags:
-        required: false
-    stage-packages:
+  source:
+    required: true
+  source-type:
+  source-tag:
+  source-branch:
+  configflags:
+    required: false
+  stage-packages:
+  snap:
+  stage:

=== modified file 'plugins/copy.yaml'
--- plugins/copy.yaml	2015-09-02 15:10:58 +0000
+++ plugins/copy.yaml	2015-09-11 13:39:05 +0000
@@ -2,3 +2,5 @@
   files:
     required: true
   stage-packages:
+  snap:
+  stage:

=== modified file 'plugins/go-project.yaml'
--- plugins/go-project.yaml	2015-09-02 15:10:58 +0000
+++ plugins/go-project.yaml	2015-09-11 13:39:05 +0000
@@ -7,3 +7,5 @@
   configflags:
     required: false
   stage-packages:
+  snap:
+  stage:

=== modified file 'plugins/make-project.yaml'
--- plugins/make-project.yaml	2015-09-02 15:10:58 +0000
+++ plugins/make-project.yaml	2015-09-11 13:39:05 +0000
@@ -1,9 +1,11 @@
 module: make_project
 build-tools: [make]
 options:
-    source:
-        required: true
-    source-type:
-    source-tag:
-    source-branch:
-    stage-packages:
+  source:
+    required: true
+  source-type:
+  source-tag:
+  source-branch:
+  stage-packages:
+  snap:
+  stage:

=== modified file 'plugins/maven-project.yaml'
--- plugins/maven-project.yaml	2015-09-02 15:10:58 +0000
+++ plugins/maven-project.yaml	2015-09-11 13:39:05 +0000
@@ -10,3 +10,5 @@
   source-tag:
   source-branch:
   stage-packages:
+  snap:
+  stage:

=== modified file 'plugins/python2-project.yaml'
--- plugins/python2-project.yaml	2015-09-02 15:10:58 +0000
+++ plugins/python2-project.yaml	2015-09-11 13:39:05 +0000
@@ -8,3 +8,5 @@
   source-tag:
   source-branch:
   stage-packages:
+  snap:
+  stage:

=== modified file 'plugins/python3-project.yaml'
--- plugins/python3-project.yaml	2015-09-02 15:10:58 +0000
+++ plugins/python3-project.yaml	2015-09-11 13:39:05 +0000
@@ -8,3 +8,5 @@
   source-tag:
   source-branch:
   stage-packages:
+  snap:
+  stage:

=== modified file 'plugins/tar-content.yaml'
--- plugins/tar-content.yaml	2015-09-02 15:10:58 +0000
+++ plugins/tar-content.yaml	2015-09-11 13:39:05 +0000
@@ -3,3 +3,5 @@
   source:
     required: true
   stage-packages:
+  snap:
+  stage:

=== modified file 'snapcraft/__init__.py'
--- snapcraft/__init__.py	2015-09-07 19:25:14 +0000
+++ snapcraft/__init__.py	2015-09-11 13:39:05 +0000
@@ -48,12 +48,12 @@
     def build(self):
         return True
 
-    def snap_files(self):
-        """Returns two iteratables of globs:
-            - the first is the set of files/dirs to include
-            - the second is the set of files/dirs to exclude
-           For example: (['bin', 'lib'], ['lib/*.a'])"""
-        return (['*'], [])
+    def snap_fileset(self):
+        """Returns one iteratables of globs specific to the plugin:
+            - includes can be just listed
+            - excludes must be preceded by -
+           For example: (['bin', 'lib', '-include'])"""
+        return ([])
 
     def env(self, root):
         return []

=== modified file 'snapcraft/plugin.py'
--- snapcraft/plugin.py	2015-09-04 20:54:34 +0000
+++ snapcraft/plugin.py	2015-09-11 13:39:05 +0000
@@ -206,8 +206,10 @@
             return True
 
         self.notify_stage("Staging")
-        common.run(['cp', '-arT', self.installdir, self.stagedir])
+        fileset = getattr(self.code.options, 'stage', ['*']) or ['*']
+        _migrate_files(fileset, self.installdir, self.stagedir)
         self.mark_done('stage')
+
         return True
 
     def snap(self, force=False):
@@ -215,39 +217,15 @@
             return True
         self.makedirs()
 
-        if self.code and hasattr(self.code, 'snap_files'):
-            self.notify_stage("Snapping")
-
-            includes, excludes = getattr(self.code, 'snap_files')()
-            snapDirs, snap_files = self.collect_snap_files(includes, excludes)
-
-            if snapDirs:
-                common.run(['mkdir', '-p'] + list(snapDirs), cwd=self.stagedir)
-            if snap_files:
-                common.run(['cp', '-a', '--parent'] + list(snap_files) + [self.snapdir], cwd=self.stagedir)
-
-            self.mark_done('snap')
+        self.notify_stage("Snapping")
+        plugin_fileset = self.code.snap_fileset()
+        fileset = getattr(self.code.options, 'snap', ['*']) or ['*']
+        fileset.extend(plugin_fileset)
+        _migrate_files(fileset, self.installdir, self.snapdir)
+        self.mark_done('snap')
+
         return True
 
-    def collect_snap_files(self, includes, excludes):
-        # validate
-        _validate_relative_paths(includes + excludes)
-
-        source_files = _generate_source_set(self.installdir)
-        include_files = _generate_include_set(self.stagedir, includes)
-        exclude_files, exclude_dirs = _generate_exclude_set(self.stagedir, excludes)
-
-        # And chop files, including whole trees if any dirs are mentioned
-        snap_files = (include_files & source_files) - exclude_files
-        for exclude_dir in exclude_dirs:
-            snap_files = set([x for x in snap_files if not x.startswith(exclude_dir + '/')])
-
-        # Separate dirs from files
-        snap_dirs = set([x for x in snap_files if os.path.isdir(os.path.join(self.stagedir, x)) and not os.path.islink(os.path.join(self.stagedir, x))])
-        snap_files = snap_files - snap_dirs
-
-        return snap_dirs, snap_files
-
     def env(self, root):
         if self.code and hasattr(self.code, 'env'):
             return getattr(self.code, 'env')(root)
@@ -262,44 +240,75 @@
     return part
 
 
-def _generate_source_set(installdir):
-    source_files = set()
-    for root, dirs, files in os.walk(installdir):
-        source_files |= set([os.path.join(root, d) for d in dirs])
-        source_files |= set([os.path.join(root, f) for f in files])
-    source_files = set([os.path.relpath(x, installdir) for x in source_files])
-
-    return source_files
-
-
-def _generate_include_set(stagedir, includes):
+def _migrate_files(fileset, srcdir, dstdir):
+    # validate
+    includes, excludes = _get_file_list(fileset)
+
+    include_files = _generate_include_set(srcdir, includes)
+    exclude_files, exclude_dirs = _generate_exclude_set(srcdir, excludes)
+
+    # And chop files, including whole trees if any dirs are mentioned
+    snap_files = include_files - exclude_files
+    for exclude_dir in exclude_dirs:
+        snap_files = set([x for x in snap_files if not x.startswith(exclude_dir + '/')])
+
+    # Separate dirs from files
+    snap_dirs = set([x for x in snap_files if os.path.isdir(os.path.join(srcdir, x)) and not os.path.islink(os.path.join(srcdir, x))])
+    snap_files = snap_files - snap_dirs
+
+    if snap_dirs:
+        common.run(['mkdir', '-p'] + list(snap_dirs), cwd=dstdir)
+    if snap_files:
+        common.run(['cp', '-a', '--parent'] + list(snap_files) + [dstdir], cwd=srcdir)
+
+
+def _get_file_list(stage_set):
+    includes = []
+    excludes = []
+
+    for item in stage_set:
+        if item.startswith('-'):
+            excludes.append(item[1:])
+        elif item.startswith('\\'):
+            includes.append(item[1:])
+        else:
+            includes.append(item)
+
+    _validate_relative_paths(includes + excludes)
+
+    includes = includes or ['*']
+
+    return includes, excludes
+
+
+def _generate_include_set(directory, includes):
     include_files = set()
     for include in includes:
-        matches = glob.glob(os.path.join(stagedir, include))
+        matches = glob.glob(os.path.join(directory, include))
         include_files |= set(matches)
 
     include_dirs = [x for x in include_files if os.path.isdir(x)]
-    include_files = set([os.path.relpath(x, stagedir) for x in include_files])
+    include_files = set([os.path.relpath(x, directory) for x in include_files])
 
     # Expand includeFiles, so that an exclude like '*/*.so' will still match
     # files from an include like 'lib'
     for include_dir in include_dirs:
         for root, dirs, files in os.walk(include_dir):
-            include_files |= set([os.path.relpath(os.path.join(root, d), stagedir) for d in dirs])
-            include_files |= set([os.path.relpath(os.path.join(root, f), stagedir) for f in files])
+            include_files |= set([os.path.relpath(os.path.join(root, d), directory) for d in dirs])
+            include_files |= set([os.path.relpath(os.path.join(root, f), directory) for f in files])
 
     return include_files
 
 
-def _generate_exclude_set(stagedir, excludes):
+def _generate_exclude_set(directory, excludes):
     exclude_files = set()
 
     for exclude in excludes:
-        matches = glob.glob(os.path.join(stagedir, exclude))
+        matches = glob.glob(os.path.join(directory, exclude))
         exclude_files |= set(matches)
 
-    exclude_dirs = [os.path.relpath(x, stagedir) for x in exclude_files if os.path.isdir(x)]
-    exclude_files = set([os.path.relpath(x, stagedir) for x in exclude_files])
+    exclude_dirs = [os.path.relpath(x, directory) for x in exclude_files if os.path.isdir(x)]
+    exclude_files = set([os.path.relpath(x, directory) for x in exclude_files])
 
     return exclude_files, exclude_dirs
 

=== modified file 'snapcraft/plugins/go.py'
--- snapcraft/plugins/go.py	2015-08-14 20:04:30 +0000
+++ snapcraft/plugins/go.py	2015-09-11 13:39:05 +0000
@@ -19,14 +19,8 @@
 
 class GoPlugin(snapcraft.BasePlugin):
 
-    def __init__(self, name, options):
-        super().__init__(name, options)
-
     def env(self, root):
         # usr/lib/go/bin on newer Ubuntus, usr/bin on trusty
         return [
             "GOPATH={}/go".format(root),
         ]
-
-    def snap_files(self):
-        return ([], [])  # Go statically links everything, no need for runtime

=== modified file 'snapcraft/plugins/jdk.py'
--- snapcraft/plugins/jdk.py	2015-09-07 19:25:14 +0000
+++ snapcraft/plugins/jdk.py	2015-09-11 13:39:05 +0000
@@ -28,8 +28,10 @@
                 'PATH=%s/usr/lib/jvm/default-java/bin:'
                 '%s/usr/lib/jvm/default-java/jre/bin:$PATH' % (root, root)]
 
-    def snap_files(self):
+    def snap_fileset(self):
         # Cut out jdk bits (jre bits are in default-java/jre)
-        return (['*'], ['usr/lib/jvm/default-java/bin',
-                        'usr/lib/jvm/default-java/include',
-                        'usr/lib/jvm/default-java/lib'])
+        return (['-usr/lib/jvm/default-java/bin',
+                 '-usr/lib/jvm/default-java/include',
+                 '-usr/lib/jvm/default-java/lib',
+                 '-usr/share/doc',
+                 ])

=== modified file 'snapcraft/plugins/qml.py'
--- snapcraft/plugins/qml.py	2015-09-07 19:25:14 +0000
+++ snapcraft/plugins/qml.py	2015-09-11 13:39:05 +0000
@@ -57,10 +57,10 @@
         "qml-module-ubuntu-onlineaccounts-client",
     ]
 
-    def snap_files(self):
-        include, exclude = ['*'], []
-        include.append('./etc/xdg/qtchooser/snappy-qt5.conf')
-        return (include, exclude)
+    def snap_fileset(self):
+        return ['*',
+                'etc/xdg/qtchooser/snappy-qt5.conf',
+                ]
 
     def build_qt_config(self):
         arch = snapcraft.common.get_arch_triplet()

=== modified file 'snapcraft/tests/test_plugin.py'
--- snapcraft/tests/test_plugin.py	2015-08-05 14:38:33 +0000
+++ snapcraft/tests/test_plugin.py	2015-09-11 13:39:05 +0000
@@ -65,53 +65,140 @@
         p.pull()
         self.assertFalse(p.code.pull.called)
 
-    def test_collect_snap_files(self):
-        p = get_test_plugin()
-
-        tmpdirObject = tempfile.TemporaryDirectory()
-        self.addCleanup(tmpdirObject.cleanup)
-        tmpdir = tmpdirObject.name
-
-        p.installdir = tmpdir + '/install'
-        os.makedirs(tmpdir + '/install/1/1a/1b')
-        os.makedirs(tmpdir + '/install/2/2a')
-        os.makedirs(tmpdir + '/install/3')
-        open(tmpdir + '/install/a', mode='w').close()
-        open(tmpdir + '/install/b', mode='w').close()
-        open(tmpdir + '/install/1/a', mode='w').close()
-        open(tmpdir + '/install/3/a', mode='w').close()
-
-        p.stagedir = tmpdir + '/stage'
-        os.makedirs(tmpdir + '/stage/1/1a/1b')
-        os.makedirs(tmpdir + '/stage/2/2a')
-        os.makedirs(tmpdir + '/stage/2/2b')
-        os.makedirs(tmpdir + '/stage/3')
-        open(tmpdir + '/stage/a', mode='w').close()
-        open(tmpdir + '/stage/b', mode='w').close()
-        open(tmpdir + '/stage/c', mode='w').close()
-        open(tmpdir + '/stage/1/a', mode='w').close()
-        open(tmpdir + '/stage/2/2b/a', mode='w').close()
-        open(tmpdir + '/stage/3/a', mode='w').close()
-
-        self.assertEqual(p.collect_snap_files([], []), (set(), set()))
-
-        self.assertEqual(p.collect_snap_files(['*'], []), (
-            set(['1', '1/1a', '1/1a/1b', '2', '2/2a', '3']),
-            set(['a', 'b', '1/a', '3/a'])))
-
-        self.assertEqual(p.collect_snap_files(['*'], ['1']), (
-            set(['2', '2/2a', '3']),
-            set(['a', 'b', '3/a'])))
-
-        self.assertEqual(p.collect_snap_files(['a'], ['*']), (set(), set()))
-
-        self.assertEqual(p.collect_snap_files(['*'], ['*/*']), (
-            set(['1', '2', '3']),
-            set(['a', 'b'])))
-
-        self.assertEqual(p.collect_snap_files(['1', '2'], ['*/a']), (
-            set(['1', '1/1a', '1/1a/1b', '2', '2/2a']),
-            set()))
+    def test_fileset_include_excludes(self):
+        stage_set = [
+            '-etc',
+            'opt/something',
+            '-usr/lib/*.a',
+            'usr/bin',
+            '\-everything',
+            r'\\a',
+        ]
+
+        include, exclude = plugin._get_file_list(stage_set)
+
+        self.assertEqual(include, ['opt/something', 'usr/bin', '-everything', r'\a'])
+        self.assertEqual(exclude, ['etc', 'usr/lib/*.a'])
+
+    def test_fileset_only_includes(self):
+        stage_set = [
+            'opt/something',
+            'usr/bin',
+        ]
+
+        include, exclude = plugin._get_file_list(stage_set)
+
+        self.assertEqual(include, ['opt/something', 'usr/bin'])
+        self.assertEqual(exclude, [])
+
+    def test_fileset_only_excludes(self):
+        stage_set = [
+            '-etc',
+            '-usr/lib/*.a',
+        ]
+
+        include, exclude = plugin._get_file_list(stage_set)
+
+        self.assertEqual(include, ['*'])
+        self.assertEqual(exclude, ['etc', 'usr/lib/*.a'])
+
+    def test_migrate_snap_files(self):
+        filesets = {
+            'nothing': {
+                'fileset': ['-*'],
+                'result': []
+            },
+            'all': {
+                'fileset': ['*'],
+                'result': [
+                    'stage/1',
+                    'stage/1/1a/1b',
+                    'stage/1/1a',
+                    'stage/1/a',
+                    'stage/2',
+                    'stage/2/2a',
+                    'stage/3',
+                    'stage/3/a',
+                    'stage/a',
+                    'stage/b',
+                ],
+            },
+            'no1': {
+                'fileset': ['-1'],
+                'result': [
+                    'stage/2',
+                    'stage/2/2a',
+                    'stage/3',
+                    'stage/3/a',
+                    'stage/a',
+                    'stage/b',
+                ],
+            },
+            'onlya': {
+                'fileset': ['a'],
+                'result': [
+                    'stage/a',
+                ],
+            },
+            'onlybase': {
+                'fileset': ['*', '-*/*'],
+                'result': [
+                    'stage/a',
+                    'stage/b',
+                    'stage/1',
+                    'stage/2',
+                    'stage/3',
+                ],
+            },
+            'nostara': {
+                'fileset': ['-*/a'],
+                'result': [
+                    'stage/1',
+                    'stage/1/1a/1b',
+                    'stage/1/1a',
+                    'stage/2',
+                    'stage/2/2a',
+                    'stage/3',
+                    'stage/a',
+                    'stage/b',
+                ],
+            },
+        }
+
+        for key in filesets:
+            with self.subTest(key=key):
+                tmpdirObject = tempfile.TemporaryDirectory()
+                self.addCleanup(tmpdirObject.cleanup)
+                tmpdir = tmpdirObject.name
+
+                srcdir = tmpdir + '/install'
+                os.makedirs(tmpdir + '/install/1/1a/1b')
+                os.makedirs(tmpdir + '/install/2/2a')
+                os.makedirs(tmpdir + '/install/3')
+                open(tmpdir + '/install/a', mode='w').close()
+                open(tmpdir + '/install/b', mode='w').close()
+                open(tmpdir + '/install/1/a', mode='w').close()
+                open(tmpdir + '/install/3/a', mode='w').close()
+
+                dstdir = tmpdir + '/stage'
+                os.makedirs(dstdir)
+
+                plugin._migrate_files(filesets[key]['fileset'], srcdir, dstdir)
+
+                expected = []
+                for item in filesets[key]['result']:
+                    expected.append(os.path.join(tmpdir, item))
+                expected.sort()
+
+                result = []
+                for root, subdirs, files in os.walk(dstdir):
+                    for item in files:
+                        result.append(os.path.join(root, item))
+                    for item in subdirs:
+                        result.append(os.path.join(root, item))
+                result.sort()
+
+                self.assertEqual(expected, result)
 
     def test_notify_stage_must_log_information(self):
         fake_logger = fixtures.FakeLogger(level=logging.INFO)
@@ -145,18 +232,16 @@
             plugin.PluginHandler(
                 "mock", "mock-part", {}, load_config=False, load_code=True)
 
-    def test_collect_snap_files_with_absolute_includes_must_raise_error(self):
-        p = get_test_plugin()
+    def test_filesets_includes_without_relative_paths(self):
         with self.assertRaises(plugin.PluginError) as raised:
-            p.collect_snap_files(includes=['rel', '/abs/include'], excludes=[])
+            plugin._get_file_list(['rel', '/abs/include'])
 
         self.assertEqual(
             "path '/abs/include' must be relative", str(raised.exception))
 
-    def test_collect_snap_files_with_absolute_excludes_must_raise_error(self):
-        p = get_test_plugin()
+    def test_filesets_exlcudes_without_relative_paths(self):
         with self.assertRaises(plugin.PluginError) as raised:
-            p.collect_snap_files(includes=[], excludes=['rel', '/abs/exclude'])
+            plugin._get_file_list(['rel', '-/abs/exclude'])
 
         self.assertEqual(
             "path '/abs/exclude' must be relative", str(raised.exception))

=== modified file 'snapcraft/tests/test_yaml.py'
--- snapcraft/tests/test_yaml.py	2015-09-08 11:48:30 +0000
+++ snapcraft/tests/test_yaml.py	2015-09-11 13:39:05 +0000
@@ -61,8 +61,9 @@
     stage-packages: [fswebcam]
 """)
         snapcraft.yaml.Config()
-        mock_loadPlugin.assert_called_with("part1", "go", {
-            "stage-packages": ["fswebcam"],
+        mock_loadPlugin.assert_called_with('part1', 'go', {
+            'stage-packages': ['fswebcam'],
+            'stage': [], 'snap': [],
         })
 
     def test_config_raises_on_missing_snapcraft_yaml(self):
@@ -211,6 +212,40 @@
             'found character \'\\t\' that cannot start any token '
             'on line 2 of snapcraft.yaml')
 
+    @unittest.mock.patch('snapcraft.yaml.Config.load_plugin')
+    def test_config_expands_filesets(self, mock_loadPlugin):
+        self.make_snapcraft_yaml("""name: test
+version: "1"
+vendor: me <[e-mail]>
+summary: test
+description: test
+icon: my-icon.png
+
+parts:
+  part1:
+    type: go
+    stage-packages: [fswebcam]
+    filesets:
+      wget:
+        - /usr/lib/wget.so
+        - /usr/bin/wget
+      build-wget:
+        - /usr/lib/wget.a
+    stage:
+      - $wget
+      - $build-wget
+    snap:
+      - $wget
+      - /usr/share/my-icon.png
+""")
+        snapcraft.yaml.Config()
+
+        mock_loadPlugin.assert_called_with('part1', 'go', {
+            'snap': ['/usr/lib/wget.so', '/usr/bin/wget', '/usr/share/my-icon.png'],
+            'stage-packages': ['fswebcam'],
+            'stage': ['/usr/lib/wget.so', '/usr/bin/wget', '/usr/lib/wget.a'],
+        })
+
 
 class TestValidation(TestCase):
 
@@ -354,3 +389,39 @@
 
         expected_message = '\'my-icon.png\' is not a \'icon-path\''
         self.assertEqual(raised.exception.message, expected_message, msg=self.data)
+
+
+class TestFilesets(TestCase):
+
+    def setUp(self):
+        super().setUp()
+
+        self.properties = {
+            'filesets': {
+                '1': ['1', '2', '3'],
+                '2': [],
+            }
+        }
+
+    def test_expand_var(self):
+        self.properties['stage'] = ['$1']
+
+        fs = snapcraft.yaml._expand_filesets_for('stage', self.properties)
+        self.assertEqual(fs, ['1', '2', '3'])
+
+    def test_no_expansion(self):
+        self.properties['stage'] = ['1']
+
+        fs = snapcraft.yaml._expand_filesets_for('stage', self.properties)
+        self.assertEqual(fs, ['1'])
+
+    def test_invalid_expansion(self):
+        self.properties['stage'] = ['$3']
+
+        with self.assertRaises(snapcraft.yaml.SnapcraftLogicError) as raised:
+            snapcraft.yaml._expand_filesets_for('stage', self.properties)
+
+        self.assertEqual(
+            raised.exception.message,
+            '\'$3\' referred to in the \'stage\' fileset but it is not '
+            'in filesets')

=== modified file 'snapcraft/yaml.py'
--- snapcraft/yaml.py	2015-09-08 11:48:30 +0000
+++ snapcraft/yaml.py	2015-09-11 13:39:05 +0000
@@ -86,7 +86,11 @@
                 after_requests[part_name] = properties["after"]
                 del properties["after"]
 
-            # TODO: support 'filter' or 'blacklist' field to filter what gets put in snap/
+            properties['stage'] = _expand_filesets_for('stage', properties)
+            properties['snap'] = _expand_filesets_for('snap', properties)
+
+            if 'filesets' in properties:
+                del properties['filesets']
 
             self.load_plugin(part_name, plugin_name, properties)
 
@@ -224,3 +228,22 @@
         raise SnapcraftSchemaError(
             '{} on line {} of {}'.format(
                 e.problem, e.problem_mark.line, yaml_file))
+
+
+def _expand_filesets_for(stage, properties):
+    filesets = properties.get('filesets', {})
+    fileset_for_stage = properties.get(stage, {})
+    new_stage_set = []
+
+    for item in fileset_for_stage:
+        if item.startswith('$'):
+            try:
+                new_stage_set.extend(filesets[item[1:]])
+            except KeyError:
+                raise SnapcraftLogicError(
+                    '\'{}\' referred to in the \'{}\' fileset but it is not '
+                    'in filesets'.format(item, stage))
+        else:
+            new_stage_set.append(item)
+
+    return new_stage_set