Can't disable a backports pocket when deploying Ubuntu 14.04

Hi Tom,

Can you please attach logs?

1. /var/log/maas/*.log
2. /var/log/maas/rsyslog/<machine-name>/<date>/messages
3. console log for the said machine.

$ dpkg -l | grep maas

ii maas 2.1.3+bzr5573-0ubuntu1~16.04.1 all "Metal as a Service" is a physical cloud and IPAM
ii maas-cli 2.1.3+bzr5573-0ubuntu1~16.04.1 all MAAS client and command-line interface
ii maas-common 2.1.3+bzr5573-0ubuntu1~16.04.1 all MAAS server common files
ii maas-dhcp 2.1.3+bzr5573-0ubuntu1~16.04.1 all MAAS DHCP server
ii maas-dns 2.1.3+bzr5573-0ubuntu1~16.04.1 all MAAS DNS server
ii maas-enlist 0.4+bzr38-0ubuntu2 amd64 MAAS enlistment tool
ii maas-proxy 2.1.3+bzr5573-0ubuntu1~16.04.1 all MAAS Caching Proxy
ii maas-rack-controller 2.1.3+bzr5573-0ubuntu1~16.04.1 all Rack Controller for MAAS
ii maas-region-api 2.1.3+bzr5573-0ubuntu1~16.04.1 all Region controller API service for MAAS
ii maas-region-controller 2.1.3+bzr5573-0ubuntu1~16.04.1 all Region Controller for MAAS
ii python3-django-maas 2.1.3+bzr5573-0ubuntu1~16.04.1 all MAAS server Django web framework (Python 3)
ii python3-maas-client 2.1.3+bzr5573-0ubuntu1~16.04.1 all MAAS python API client (Python 3)
ii python3-maas-provisioningserver 2.1.3+bzr5573-0ubuntu1~16.04.1 all MAAS server provisioning libraries (Python 3)

1. /var/log/maas/*.log
  I attached the logs in the zip file
2. /var/log/maas/rsyslog/<machine-name>/<date>/messages
  No file was generated in this location
3. console log for the said machine.
  How do you propose I get a log of the system's console?

Hi Tom,

Could you also please attach:

maas <user> machine get-curtin-config <system_id>

@Tom,

For (3), you need to watch the console of your server and provide screenshots or a log (if you watch it via ipmiconsole or ipmitool or similar). You can also set kernel parameters in MAAS if you need to enable the console: https://docs.ubuntu.com/maas/2.1/en/installconfig-kernel

Attached are the results of:

maas <user> machine get-curtin-config <system_id>

It is rather hard to get a screenshot of the exact point it errors out since it immediately reboots. But attached is the closest I got.

So after banging my head a while trying to find out why this wasn't working for you, I realized that you were deploying Ubuntu 14.04.

The ability to disable_pockets was introduced in MAAS 2.1, and will only work when deploying Xenial+. This is because MAAS depends on cloud-init to do such configuration, and the cloud-init version in Trusty does not support this feature.

I'll mark this bug as low from the MAAS perspective to ensure we add a proper notification of this being the case, but I'll also add a wishlist bug for Trusty in cloud-init.

So why does Trusty deploy with an ephemeral image of Trusty? For example when I deploy centos it uses Xenial to perform the deploy and everything works fine. It seems like this would be an easy way to insure that all features are available on all or at least more versions.

This is a very large issue for enterprises that do not have access to the general internet and only want to mirror the necessary repos. Is there anyway to increase priority allowing pockets to be disabled on all versions of Ubuntu. It would also be very helpful to have more control that it currently has. For example multiverse should not be needed but for some reason is forced.

Do you know why it is trying to pull down source from the repos? I don't see why the ephemeral image should ever compile anything so this seems like another useless dependency.

@tpmullan, could you configure your mirror to mirror backports, as well?

Yeah we can configure it to mirror back-ports. Just not multiverse, and we would prefer not to host source either.

@Tom,

All third party OS' are deployed with the latest LTS ephemeral image or the one that is default for commissioning.

There are multiple reasons why we always deploy a Ubuntu based OS' with the same ephemeral image. Examples:

1. If you deploy 'Trusty' with Xenial ephemeral on a hardware for which Trusty's default kernel does not include support for, then the deployment process will succeed, but the deployed machine won't work.

2. Similarly, if you deploy 'Zesty' with a Xenial ephemeral image, on a machine for which Xenial doesn't have drivers for, then the deployment process will fail altogether.

That said, I did a bit more digging and I need to correct my previous statement. MAAS/cloud-init/curtin do not support configuring 'components' nor 'sources' for the Ubuntu mirrors:

 * components = main, restricted, universe, multiverse
 * source = deb-src

Configuring pockets for the Ubuntu mirrors is supported though.

 * pockets = *-updates, *-security, *-backports

Changing this as a wishlist.

For the second part of my last comment, I've filed:
https://bugs.launchpad.net/maas/+bug/1664384

Okay. For my use case this bug will be irreverent if the bug you created is fixed and ideally works for all versions of Ubuntu. But if it works just for Xenial and onward that would be enough.

@Tom,

Based on the MAAS screenshot you provided you are getting a 403 error code, which is unexpected (404 would make more sense).
Can you please provide a listing of the directory structure of the mirror?

$ find /tmp/ -type d -printf '%M %g %u %p\n'

I am getting a 404 now. I can take another screenshot if you want. That screenshot was not completely applicable to this specific issue, I found that I concurrently had another issue when I was trying to report it. I was getting a 403 from the maas server since for some reason it did not add my subnet to the internal proxy. I need to figure out why maas is not setting the values for that subnet like it should. I would also like the ability to manually specify values for that config. It is helpful to set a proxy peer and have a no proxy settings. Currently I am just stomping over the file with chef and not letting maas update it, and could just set up another server. But while that works it is not the best of solutions.

Tom,

1. Check that /etc/maas/rackd.conf does not point to 'localhost'. Instead, make sure that it points to an IP address of the Region Controller that the machines can access to (you'll need to restart maas-rackd: sudo service maas-rackd restart).

2. Once you do that, attempt to deploy a machine and when in 'Failed deployment' state, please attach the output of:

maas <user> node-results read system_id=<machine-system-id> result_type=1 | grep data | cut -d"\"" -f4 | base64 --decode

Actually, If you could:

1. Deploy trusty, attach log as above once 'Failed Deployment' or 'Deployed' (maas <user> node-results read system_id=<machine-system-id> result_type=1 | grep data | cut -d"\"" -f4 | base64 --decode)

2. Do the same when deploying Xenial.

Andres,

1. It had localhost, and I changed it. I don't see why this would effect anything though. Could you explain why this wouldn't just be used for the rack to communicate with the region?

2. I ran that command after the deploy failed and received an empty array so the "| grep data | cut -d"\"" -f4 | base64 --decode" part returns nothing.

It does not seem to report back properly when it fails to install packages with apt

Also there does not seem to be any problem when I mirror everything from for xenial and trusty. So for my use case https://bugs.launchpad.net/maas/+bug/1664384 is really more applicable. It does not sound like it will never work for trusty, but that is about it.

Tom, Andres is exploring because we're trying to pull apart two issues here.

First, MAAS should definitely be able to deploy Ubuntu even when pointed at a local mirror that has chosen to not sync specified components and pockets. If that is failing it is a serious bug. However, the data you've provided in this bug hasn't provided us with a definitive answer on this one -- there might be some odd interaction with the proxy, for instance.

If this works, then you can immediately address your use case by just using a correctly configured mirror. You don't need to set any option in MAAS-- it should just work, regardless of what is in the configured sources.list in the ephemeral environment or deployed node.

Second, as Andres is pointing out, the option to restrict certain pockets (i.e. backports) in the MAAS UI should work with Xenial (but not Trusty). This would lead MAAS to configuring its sources.list appropriately, which would mean even if the mirror was complete no packages would be requested. That facility however doesn't support excluding components (i.e. multiverse) today.

It's still unclear to me if you've isolated the problem to the first or second problem above, but that would be super helpful.

Christian,

I agree the original screenshot that I supplied has an additional error with the proxy that I have resolved. And I think I have conclusively narrowed it down to the mirror. When I have the multiverse folder on the mirror it works when I don't it fails. I supplied Michael Iatrou with more logs directly, and would prefer not to post them publicly if possible. That said if you need them and can't get them from Michael I can try and mask sensitive information and post them.

Since this bug is about Trusty, marking as Wont Fix for Trusty version