MAAS

MAAS sometimes get in a state where machine tags aren't updated from their definitions

Bug #1845351 reported by Björn Tillenius
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Fix Released
High
Björn Tillenius
MAAS 2.7.0b1
2.3
Fix Committed
Medium
Björn Tillenius
MAAS 2.3.7
2.4
Fix Committed
High
Björn Tillenius
MAAS 2.4.3
2.6
Fix Released
High
Björn Tillenius
MAAS 2.6.2

Bug Description

Now and then we get a failure in CI where it adds a machine tag with defintion=true(), which should apply to all machines.

I stopped the CI when the tags test failed and debugged what's going in. regiond sends an RPC request, EvaluateTag, to rackd. rackd then uses the MAAS API to get the node details, but gets a 503 from the node GET request.

https://pastebin.ubuntu.com/p/BmyJ6Dh3nR/

There are nothing logged in regiond .log about the 503, but trying to add more tags result in the same error. The API itself is working fine, though, since I can use it to add the tags.

Another odd thing is that restarting maas-rackd makes the problem go away. After restarting maas-rackd I couldn't reproduce it, so I couldn't add any extra debug logging in the code.

Related branches

~bjornt/maas:bug-1845351-2.3
Björn Tillenius: Approve
Diff: 193 lines (+96/-11)
4 files modified
src/apiclient/maas_client.py (+11/-1)
src/apiclient/tests/test_maas_client.py (+67/-8)
src/provisioningserver/rpc/tags.py (+7/-2)
src/provisioningserver/rpc/tests/test_tags.py (+11/-0)
~bjornt/maas:bug-1845351-2.4
MAAS Lander: Needs Fixing
Björn Tillenius: Approve
Diff: 193 lines (+96/-11)
4 files modified
src/apiclient/maas_client.py (+11/-1)
src/apiclient/tests/test_maas_client.py (+67/-8)
src/provisioningserver/rpc/tags.py (+7/-2)
src/provisioningserver/rpc/tests/test_tags.py (+11/-0)
~bjornt/maas:bug-1845351-2.6
Björn Tillenius: Approve
MAAS Lander: Pending (unittests) requested
Diff: 243 lines (+112/-21)
4 files modified
src/apiclient/maas_client.py (+11/-1)
src/apiclient/tests/test_maas_client.py (+74/-16)
src/provisioningserver/rpc/tags.py (+6/-2)
src/provisioningserver/rpc/tests/test_tags.py (+21/-2)
~bjornt/maas:bug-1845351-update-tags-no-proxy
MAAS Lander: Needs Fixing
Alberto Donato (community): Approve
Diff: 231 lines (+98/-15)
4 files modified
src/apiclient/maas_client.py (+11/-1)
src/apiclient/tests/test_maas_client.py (+67/-13)
src/provisioningserver/rpc/tags.py (+3/-1)
src/provisioningserver/rpc/tests/test_tags.py (+17/-0)
Björn Tillenius (bjornt)
Changed in maas:
status: New → Triaged
importance: Undecided → High
assignee: nobody → Björn Tillenius (bjornt)
milestone: none → 2.7.0alpha1
Revision history for this message
Björn Tillenius (bjornt) wrote :

The 2.4 tests also fail with this failure now and then.

Revision history for this message
Björn Tillenius (bjornt) wrote :

BTW, I marked this as High, since it breaks the CI runs. From the looks of it, it doesn't seem to affect many users, since it's an old bug and no one has reported it before.

Revision history for this message
Björn Tillenius (bjornt) wrote :

There are two reasons why this bug is happening. The first is that something is changing the proxy environment variables. When rackd starts up, it has http_proxy and no_proxy defined and updating the tags work in our CI environment, since the region IP is in no_proxy. But then at some point, at least the no_proxy variable is gone. From the looks of it http_proxy was also gone, but somehow the API call still went through the proxy, that doesn't allow access to the region.

But still, the best fix is still not to use any proxies for the API communication. It's documented that rackd needs TCP access to the region on port 5240. If we for some reason want to support proxy for rack-region communication, we need to add explicit support for it.

Björn Tillenius (bjornt)
Changed in maas:
status: Triaged → In Progress
MAAS Lander (maas-lander)
Changed in maas:
status: In Progress → Fix Committed
Revision history for this message
Björn Tillenius (bjornt) wrote :

I'm targetting this to 2.3 as well, since it affects systemtests, which fail for the 2.3 branch due to this bug.

Adam Collard (adam-collard)
Changed in maas:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.