systemctl link request failed for service FOO: Unit name FOO is not valid.

The wily slave has these packages.

$ apt-cache policy dbus
dbus:
  Installed: 1.8.12-1ubuntu5
  Candidate: 1.8.12-1ubuntu5
  Version table:
 *** 1.8.12-1ubuntu5 0
        500 http://az3.clouds.archive.ubuntu.com/ubuntu/ wily/main amd64 Packages
        100 /var/lib/dpkg/status

$ apt-cache policy systemd
systemd:
  Installed: 223-1ubuntu1
  Candidate: 223-1ubuntu1
  Version table:
 *** 223-1ubuntu1 0
        500 http://az3.clouds.archive.ubuntu.com/ubuntu/ wily/main amd64 Packages
        100 /var/lib/dpkg/status

$ apt-cache policy juju-core
juju-core:
  Installed: 1.24.3-0ubuntu1~15.10.1~juju1
  Candidate: 1.24.3-0ubuntu1~15.10.1~juju1
  Version table:
 *** 1.24.3-0ubuntu1~15.10.1~juju1 0
        500 http://ppa.launchpad.net/juju/stable/ubuntu/ wily/main amd64 Packages
        100 /var/lib/dpkg/status
     1.22.6-0ubuntu1 0
        500 http://az3.clouds.archive.ubuntu.com/ubuntu/ wily/universe amd64 Packages

This appears to be a regression in systemd. I extracted the unit file we're trying to link and manually linked it through systemctl and it succeeded with 222-2ubuntu1, but failed with 223-1ubuntu1:

ubuntu@cherylj-wily4:~$ apt-cache policy systemd
systemd:
  Installed: 222-2ubuntu1
  Candidate: 223-1ubuntu1
  Version table:
     223-1ubuntu1 0
        500 http://nova.clouds.archive.ubuntu.com/ubuntu/ wily/main amd64 Packages
 *** 222-2ubuntu1 0
        100 /var/lib/dpkg/status

root@cherylj-wily4:~$ sudo systemctl link /var/lib/juju/init/juju-db-ubuntu-local/juju-db-ubuntu-local.service
Created symlink from /etc/systemd/system/juju-db-ubuntu-local.service to /var/lib/juju/init/juju-db-ubuntu-local/juju-db-ubuntu-local.service.

ubuntu@cherylj-wily4:~$ sudo systemctl disable /var/lib/juju/init/juju-db-ubuntu-local/juju-db-ubuntu-local.service
Removed symlink /etc/systemd/system/juju-db-ubuntu-local.service.

ubuntu@cherylj-wily4:~$ sudo apt-get upgrade
...

ubuntu@cherylj-wily4:~$ apt-cache policy systemd
systemd:
  Installed: 223-1ubuntu1
  Candidate: 223-1ubuntu1
  Version table:
 *** 223-1ubuntu1 0
        500 http://nova.clouds.archive.ubuntu.com/ubuntu/ wily/main amd64 Packages
        100 /var/lib/dpkg/status

ubuntu@cherylj-wily4:~$ sudo systemctl link /var/lib/juju/init/juju-db-ubuntu-local/juju-db-ubuntu-local.service
Failed to execute operation: Unit name /var/lib/juju/init/juju-db-ubuntu-local/juju-db-ubuntu-local.service is not valid.

This bug blocks bug 1481556

Confirmed, and forwarded to https://github.com/systemd/systemd/issues/879 .

I created an autopkgtest: http://paste.ubuntu.com/12009633/

Bisecting now.

Bisected to https://github.com/systemd/systemd/commit/4938696301a914ec26bcfc60bb99a1e9624e3789

Quick workaround: Use "systemctl --global link foo.service" to forego the SELinux check.

Fixed in packaging git.

This bug was fixed in the package systemd - 224-1ubuntu3

---------------
systemd (224-1ubuntu3) wily; urgency=medium

  * Skip systemd-fsckd autopkgtest if /run/initramfs/fsck-root exists, i. e.
    the initramfs already ran fsck.
  * Fix broken ACL in tmpfiles.d/systemd.conf. (Closes: #794645, LP: #1480552)
  * Add debian/tests/unit-config: Test "systemctl link"; reproduces LP#1480310.
  * Add a hack to unbreak "systemctl link". (LP: #1480310)

 -- Martin Pitt <email address hidden> Wed, 12 Aug 2015 15:21:19 +0200

We are still seeing this issue after getting this fix and rebooting the machine:

jenkins@wily-slave:~$ apt-cache policy systemd
systemd:
  Installed: 224-1ubuntu3
  Candidate: 224-1ubuntu3
  Version table:
 *** 224-1ubuntu3 0
        500 http://archive.ubuntu.com/ubuntu/ wily/main amd64 Packages
        100 /var/lib/dpkg/status

2015-08-14 13:19:28 ERROR juju.service.systemd service.go:452 failed to install service "juju-db-jenkins-local-deploy-wily-amd64": dbus enable request failed for service "juju-db-jenkins-local-deploy-wily-amd64": Unit name /var/lib/juju/init/juju-db-jenkins-local-deploy-wily-amd64/juju-db-jenkins-local-deploy-wily-amd64.service is not valid.
2015-08-14 13:19:28 ERROR juju.cmd supercommand.go:429 dbus enable request failed for service "juju-db-jenkins-local-deploy-wily-amd64": Unit name /var/lib/juju/init/juju-db-jenkins-local-deploy-wily-amd64/juju-db-jenkins-local-deploy-wily-amd64.service is not valid.

The duplicate bug confirms what we are seeing in CI
wily + 1.24.5: Failed to execute operation: Unit name /etc/systemd/system/juju-clean-shutdown.service is not valid

FYI duplicate was @ https://bugs.launchpad.net/juju-core/+bug/1488554. Thanks for pointing it here.

On Jul 28, uosci was able to deploy Wily @ Juju 1.24.2. Granted, that was with older Wily images and perhaps older systemd, so it may or may not be helpful information at this point.

Here is that successful amulet test output from the trusty/ubuntu charm, which exercises all currently-supported Ubuntu releases (+ current dev Ubuntu release):
charm_amulet_test/5429: http://paste.ubuntu.com/12193551/

I think it may be of interest to begin cycling the trusty/ubuntu charm's amulet test in Juju CI as an easy way to do a really basic and quick functional check. ie. grab lp:charms/trusty/ubuntu, then make functional_test.

The tests are maintained and validated as Ubuntu releases come and go. Note that tests are only added and enabled for Ubuntu dev versions after confirming that they pass for that version.
http://bazaar.launchpad.net/~charmers/charms/trusty/ubuntu/trunk/files/head:/tests/

The Ubuntu charm is, and is intended to be, a minimal hookless nothing. That makes it attractive for inexpensive general validation of test automation, environments, tooling and such (which is uosci's specific interest in it). That could also make it attractive for testing Ubuntu releases vs. Juju releases.

Ah, this indeed does not only affect the "link" command but others too. I'm just confused -- why do you call "link" *and* "enable" with an absolute path? You can just do the latter, that implies "link". Or you explicitly "link" and then usually call "enable" on a name only, not a path.

Either way, https://github.com/systemd/systemd/pull/1044 should fix this in a more generic fashion, I'm testing this now.

This bug was fixed in the package systemd - 224-2ubuntu2

---------------
systemd (224-2ubuntu2) wily; urgency=medium

  * Cherry-pick FTBFS (with pkg-create-dbgsym) fix from Debian master branch.

systemd (224-2ubuntu1) wily; urgency=medium

  * Merge with Debian unstable. Remaining Ubuntu changes:
    - Hack to support system-image read-only /etc, and modify files in
      /etc/writable/ instead.
    - Keep our much simpler udev maintainer scripts (all platforms must
      support udev, no debconf).
    - initramfs init-bottom: If LVM is installed, settle udev,
      otherwise we get missing LV symlinks. Workaround for LP #1185394.
    - Add debian/udev.lvm2.init: Dummy SysV init script to satisfy insserv
      dependencies to "lvm2" which is handled with udev rules in Ubuntu.
    - Add debian/udev.lvm2.service to avoid running the dummy lvm2 init
      script.
    - Provide shutdown fallback for upstart. (LP: #1370329)
    - debian/extra/ifup@.service: Additionally run for "auto" class. We don't
      really support "allow-hotplug" in Ubuntu at the moment, so we need to
      deal with "auto" devices appearing after "/etc/init.d/networking start"
      already ran. (LP: #1374521)
    - ifup@.service: Drop dependency on networking.service (i. e.
      /etc/init.d/networking), and merely ensure that /run/network exists.
      This avoids unnecessary dependencies/waiting during boot and dependency
      cycles if hooks wait for other interfaces to come up (like ifenslave
      with bonding interfaces). (LP: #1414544)
    - Add Get-RTC-is-in-local-time-setting-from-etc-default-rc.patch: In
      Ubuntu we currently keep the setting whether the RTC is in local or UTC
      time in /etc/default/rcS "UTC=yes|no", instead of /etc/adjtime.
      (LP: #1377258)
    - Put session scopes into all cgroup controllers. This makes unprivileged
      user LXC containers work under systemd. (LP: #1346734)
    - systemctl: Don't forward telinit u to upstart. This works around
      upstart's Restart() always reexec'ing /sbin/init on Restart(), even if
      that changes to point to systemd during the upgrade. This avoids running
      systemd during a dist-upgrade. (LP: #1430479)
    - Drop hwdb-update dependency from udev-trigger.service, which got
      introduced in v219-stable. This causes udev and plymouth to start too
      late and isn't really needed in Ubuntu yet as we don't support stateless
      systems yet and handle hwdb.bin updates through dpkg triggers. This can
      be dropped again with initramfs-tools 0.117.
    - Lower Breaks: to plymouth version which has the udev inotify fix in
      Ubuntu.
    - Lower apparmor Breaks: to the Ubuntu version that dropped $remote_fs.
    - Change systemd-sysv's conflicts to upstart-sysv. (LP: #1422681)

    Upgrade fixes, keep until 16.04 LTS release:
    - systemd Conflicts/Replaces/Provides systemd-services.
    - Remove obsolete systemd-logind upstart job.
    - Clean up obsolete /etc/udev/rules.d/README.
    - systemd.postinst: Migrate mountall specific fstab options to standard
      util-linux "nofail" option.

  * Don't build new systemd-journal-remote package and drop libmicrohttpd-dev.
    This...

Juju CI confirms this is fixed:
    http://reports.vapour.ws/releases/3016/job/local-deploy-wily-amd64/attempt/297

Also confirmed via the Ubuntu charm amulet test in uosci, that Wily is deployable again. Thanks, all!

http://paste.ubuntu.com/12238490/