Merge ~mitchburton/ubuntu/+source/landscape-client:ubuntu/lunar-devel into ubuntu/+source/landscape-client:ubuntu/lunar-devel
- Git
- lp:~mitchburton/ubuntu/+source/landscape-client
- ubuntu/lunar-devel
- Merge into ubuntu/lunar-devel
Status: | Merged |
---|---|
Merge reported by: | Robie Basak |
Merged at revision: | e14dec00a3918bec7c87b50d271fec1cb95bd4a9 |
Proposed branch: | ~mitchburton/ubuntu/+source/landscape-client:ubuntu/lunar-devel |
Merge into: | ubuntu/+source/landscape-client:ubuntu/lunar-devel |
Diff against target: |
4985 lines (+1840/-569) 89 files modified
.github/workflows/ci.yml (+34/-0) Makefile (+6/-8) Makefile.packaging (+1/-1) README (+58/-15) debian/changelog (+29/-0) debian/control (+1/-1) debian/landscape-client.logrotate (+2/-1) debian/landscape-client.postrm (+3/-0) debian/patches/series (+0/-7) debian/rules (+1/-1) dev/null (+0/-25) example.conf (+8/-0) landscape/__init__.py (+1/-1) landscape/client/broker/exchange.py (+57/-9) landscape/client/broker/registration.py (+18/-13) landscape/client/broker/server.py (+9/-0) landscape/client/broker/service.py (+2/-1) landscape/client/broker/tests/badprivate.ssl (+25/-13) landscape/client/broker/tests/badpublic.ssl (+19/-14) landscape/client/broker/tests/helpers.py (+10/-8) landscape/client/broker/tests/private.ssl (+25/-13) landscape/client/broker/tests/public.ssl (+19/-14) landscape/client/broker/tests/test_exchange.py (+184/-0) landscape/client/broker/tests/test_registration.py (+46/-17) landscape/client/broker/tests/test_server.py (+19/-0) landscape/client/broker/tests/test_store.py (+1/-1) landscape/client/broker/tests/test_transport.py (+1/-1) landscape/client/broker/transport.py (+1/-1) landscape/client/configuration.py (+43/-2) landscape/client/lockfile.py (+52/-0) landscape/client/manager/aptsources.py (+6/-8) landscape/client/manager/config.py (+8/-0) landscape/client/manager/keystonetoken.py (+1/-1) landscape/client/manager/scriptexecution.py (+3/-5) landscape/client/manager/service.py (+2/-1) landscape/client/manager/tests/test_aptsources.py (+10/-198) landscape/client/manager/tests/test_processkiller.py (+1/-1) landscape/client/manager/tests/test_scriptexecution.py (+13/-13) landscape/client/monitor/__init__.py (+1/-1) landscape/client/monitor/computertags.py (+29/-0) landscape/client/monitor/config.py (+1/-1) landscape/client/monitor/processorinfo.py (+42/-2) landscape/client/monitor/service.py (+2/-1) landscape/client/monitor/tests/test_computertags.py (+64/-0) landscape/client/monitor/tests/test_ubuntuproinfo.py (+47/-0) landscape/client/monitor/ubuntuproinfo.py (+47/-0) landscape/client/package/changer.py (+2/-2) landscape/client/package/reporter.py (+5/-0) landscape/client/package/tests/test_changer.py (+3/-3) landscape/client/package/tests/test_releaseupgrader.py (+2/-2) landscape/client/package/tests/test_reporter.py (+26/-0) landscape/client/package/tests/test_taskhandler.py (+4/-1) landscape/client/reactor.py (+3/-0) landscape/client/tests/test_amp.py (+89/-1) landscape/client/tests/test_configuration.py (+73/-2) landscape/client/tests/test_lockfile.py (+21/-0) landscape/client/user/provider.py (+1/-1) landscape/lib/apt/package/facade.py (+19/-6) landscape/lib/apt/package/skeleton.py (+1/-2) landscape/lib/apt/package/testing.py (+5/-5) landscape/lib/apt/package/tests/test_facade.py (+67/-0) landscape/lib/apt/package/tests/test_skeleton.py (+10/-2) landscape/lib/backoff.py (+53/-0) landscape/lib/base64.py (+8/-0) landscape/lib/bpickle.py (+1/-1) landscape/lib/compat.py (+5/-1) landscape/lib/config.py (+1/-1) landscape/lib/disk.py (+2/-2) landscape/lib/gpg.py (+17/-5) landscape/lib/lsb_release.py (+48/-15) landscape/lib/message.py (+14/-4) landscape/lib/network.py (+99/-66) landscape/lib/schema.py (+18/-4) landscape/lib/testing.py (+1/-1) landscape/lib/tests/test_backoff.py (+37/-0) landscape/lib/tests/test_config.py (+15/-6) landscape/lib/tests/test_gpg.py (+43/-4) landscape/lib/tests/test_lsb_release.py (+45/-4) landscape/lib/tests/test_network.py (+93/-12) landscape/lib/tests/test_schema.py (+2/-2) landscape/lib/tests/test_vm_info.py (+15/-3) landscape/lib/user.py (+1/-1) landscape/lib/vm_info.py (+2/-3) landscape/message_schemas/server_bound.py (+14/-4) landscape/message_schemas/test_message.py (+9/-1) landscape/sysinfo/load.py (+2/-1) landscape/sysinfo/network.py (+2/-1) man/landscape-config.1 (+7/-1) man/landscape-config.txt (+3/-0) |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Andreas Hasenack | Approve | ||
Ubuntu Sponsors | Pending | ||
git-ubuntu import | Pending | ||
Review via email: mp+437634@code.launchpad.net |
Commit message
Description of the change
Mitch Burton (mitchburton) wrote : | # |
Andreas Hasenack (ahasenack) wrote : | # |
Looking at this now.
Andreas Hasenack (ahasenack) wrote : | # |
This branch doesn't build:
$ dquilt push -a
Applying patch 0001-Handle-
patching file landscape/
Hunk #1 FAILED at 248.
1 out of 1 hunk FAILED -- rejects in file landscape/
Patch 0001-Handle-
I grabbed the source package from your ppa:
$ dget https:/
(...)
And that source package doesn't have a debian/patches directory.
You have a lot of patches still being applied from debian/
$ cat debian/
0001-Handle-
0002-lp1870087-
py3.9.patch
0003-clean-
replace-
1962539_
lp1903776-
Andreas Hasenack (ahasenack) wrote : | # |
If I stop applying all the patches from this branch (debian/
--- landscape-
+++ landscape-
@@ -10,7 +10,7 @@ from landscape.
# List of filesystem types authorized when generating disk use statistics.
STABLE_FILESYSTEMS = frozenset(
["ext", "ext2", "ext3", "ext4", "reiserfs", "ntfs", "msdos", "dos", "vfat",
- "xfs", "hpfs", "jfs", "ufs", "hfs", "hfsplus", "simfs", "drvfs", "lxfs"])
+ "xfs", "hpfs", "jfs", "ufs", "hfs", "hfsplus", "simfs"])
That is a diff between the orig tarball, and the branch contents. They should be identical.
Andreas Hasenack (ahasenack) wrote : | # |
While that is sorted out, I'll try to check the other changes, starting with debian/* (the packaging)
Mitch Burton (mitchburton) wrote : | # |
Thanks. Looks like I missed a commit when sorting out the changes from the last version (adding to the STABLE_FILESYSTEMS set). I'm currently going through the patches to make sure they were applied upstream and sorting out the ones that were.
Andreas Hasenack (ahasenack) : | # |
Andreas Hasenack (ahasenack) : | # |
Andreas Hasenack (ahasenack) wrote : | # |
Is this commit closing bug #1878957?
commit a8759e3f79b13f1
Author: Simon Poirier <email address hidden>
Date: Fri Jun 5 11:52:32 2020 -0400
Preserve auto flag on package changer operations. (#85)
address issues with package not autoremoved (LP: #1878957).
Maybe mention it in d/changelog?
I don´t recall the landscape client case/exception 100%, but keep in mind every bug mentioned in d/changelog might need to have an SRU template (test case, regression analysis, etc). Or how have you been dealing with that in past SRUs?
Mitch Burton (mitchburton) : | # |
Andreas Hasenack (ahasenack) wrote : | # |
Here is another case:
commit 35a35f3b59aed70
Author: Simon Poirier <email address hidden>
Date: Tue Aug 11 18:42:24 2020 -0400
decode registration error codes as str (#87)
fixes (LP: #1889464)
This bug at least does not have an ubuntu task, just a landscape-client (project) one. I suppose maybe back then LC bugs were tracked in LP instead of github?
Andreas Hasenack (ahasenack) wrote : | # |
This one is still "in progress" in LP (in the landscape-client project, not ubuntu packages), and I didn't find it mentioned in any previous d/changelog entry. Checking the code it's still an open issue in the current lunar package, which this branch is fixing:
commit 77a8b93e082e3df
Author: Simon Poirier <email address hidden>
Date: Mon Nov 16 09:48:09 2020 -0500
Add RHEV to vm_info map. (LP: #1884116) (#86)
Are these all old bugs from before the move to github? Or is the project also tracking bugs in lp? I'm unsure now if they all should be listed in d/changelog, as there are going to be many, looks like.
Kevin Nasto (silverdrake11) : | # |
Andreas Hasenack (ahasenack) wrote : | # |
Another one still "in progress" in LP:
commit 4e518be77cdd1f7
Author: Simon Poirier <email address hidden>
Date: Fri Sep 24 15:35:19 2021 -0400
Translate legacy py27 messages to py3 messages in broker server. (#94)
This should address issues with upgrades never reporting back
when migrating to python3. (LP: #1943291)
Mitch Burton (mitchburton) wrote : | # |
Right. There was a transition from launchpad to github, but bugs should still be solely tracked in launchpad. I'll add the missing ones to d/changelog (and update the bug report statuses).
Past SRUs have been a bit of a mixed bag, with some older ones I've seen covering multiple bugs in a single template.
The landscape client case/exception, at least as detailed at the start of https:/
Mitch Burton (mitchburton) : | # |
Andreas Hasenack (ahasenack) : | # |
Mitch Burton (mitchburton) : | # |
Andreas Hasenack (ahasenack) : | # |
Andreas Hasenack (ahasenack) : | # |
Mitch Burton (mitchburton) : | # |
Andreas Hasenack (ahasenack) : | # |
Andreas Hasenack (ahasenack) : | # |
Andreas Hasenack (ahasenack) wrote : | # |
I made the d/control debhelper changes and sponsored.
Things to discuss/do after this upload, before the SRU potentially:
- revisit debhelper for the SRU, and even lunar
- handle the case where ua is not installed, or not executable. We shouldn't spam the LS server every 15min with an error in that case
- gpg key handling: use a more specific filename for the keyfiles, and restrict the globbing in postrm accordingly
- e14dec0... by Mitch Burton
-
Set debhelper-compat version to 12 for lunar.
Robie Basak (racb) wrote : | # |
Seb asked me to mark this as Merged as it was uploaded in https:/
Preview Diff
1 | diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml |
2 | new file mode 100644 |
3 | index 0000000..54fd142 |
4 | --- /dev/null |
5 | +++ b/.github/workflows/ci.yml |
6 | @@ -0,0 +1,34 @@ |
7 | +name: ci |
8 | +on: [pull_request, workflow_dispatch] |
9 | +jobs: |
10 | + check3: |
11 | + runs-on: ${{ matrix.os }} |
12 | + strategy: |
13 | + matrix: |
14 | + os: ["ubuntu-22.04", "ubuntu-20.04"] |
15 | + steps: |
16 | + - uses: actions/checkout@v2 |
17 | + - run: | |
18 | + make depends |
19 | + # -common seems a catch-22, but this is just a shortcut to |
20 | + # initialize user and dirs, some used through tests. |
21 | + sudo apt-get -y install landscape-common |
22 | + - run: make check3 TRIAL=/usr/bin/trial3 |
23 | + lint: |
24 | + runs-on: ubuntu-latest |
25 | + steps: |
26 | + - uses: actions/checkout@v2 |
27 | + - run: make depends |
28 | + - run: make lint |
29 | + coverage: |
30 | + runs-on: ubuntu-latest |
31 | + steps: |
32 | + - uses: actions/checkout@v2 |
33 | + - run: | |
34 | + make depends |
35 | + # -common seems a catch-22, but this is just a shortcut to |
36 | + # initialize user and dirs, some used through tests. |
37 | + sudo apt-get -y install landscape-common |
38 | + - run: make coverage TRIAL=/usr/bin/trial3 |
39 | + - name: upload |
40 | + uses: codecov/codecov-action@v1 |
41 | diff --git a/.travis.yml b/.travis.yml |
42 | deleted file mode 100644 |
43 | index b0fed89..0000000 |
44 | --- a/.travis.yml |
45 | +++ /dev/null |
46 | @@ -1,71 +0,0 @@ |
47 | -dist: trusty |
48 | -sudo: true |
49 | -language: python |
50 | -matrix: |
51 | - include: |
52 | - - env: TARGET=check2 IMAGE=ubuntu:xenial |
53 | - script: |
54 | - - sg lxd -c 'lxc exec testcontainer -- sh -c "sudo apt-get update && sudo apt-get -y install make python-distutils-extra python-mock python-twisted python-apt python-twisted-core python-pycurl python-netifaces"'; |
55 | - # creates user and dirs, some used through tests |
56 | - - sg lxd -c 'lxc exec testcontainer -- sh -c "sudo apt-get -y install landscape-common"'; |
57 | - - sg lxd -c 'lxc exec testcontainer -- sh -c "sudo chown -R ubuntu:ubuntu /target"'; |
58 | - - sg lxd -c 'lxc exec testcontainer -- sudo -i -u ubuntu sh -c "cd /target; make ${TARGET}" ubuntu'; |
59 | - - env: TARGET=check3 IMAGE=ubuntu:bionic |
60 | - script: |
61 | - # bionic needs cgroupv2 not on this kernel, so nudge the system a bit |
62 | - - sg lxd -c 'lxc exec testcontainer -- sh -c "dhclient eth0; cloud-init init"'; |
63 | - - sg lxd -c 'lxc exec testcontainer -- sh -c "sudo apt-get update && sudo apt-get -y install make python3-distutils-extra python3-mock python3-twisted python3-apt python-twisted-core python3-pycurl python3-netifaces"'; |
64 | - # creates user and dirs, some used through tests |
65 | - - sg lxd -c 'lxc exec testcontainer -- sh -c "sudo apt-get -y install landscape-common"'; |
66 | - - sg lxd -c 'lxc exec testcontainer -- sh -c "sudo chown -R ubuntu:ubuntu /target"'; |
67 | - - sg lxd -c 'lxc exec testcontainer -- sudo -i -u ubuntu sh -c "cd /target; make ${TARGET}" ubuntu'; |
68 | - - env: TARGET=check3 IMAGE=ubuntu-daily:focal |
69 | - script: |
70 | - # bionic needs cgroupv2 not on this kernel, so nudge the system a bit |
71 | - - sg lxd -c 'lxc exec testcontainer -- sh -c "dhclient eth0; cloud-init init"'; |
72 | - - sg lxd -c 'lxc exec testcontainer -- sh -c "sudo apt-get update && sudo apt-get -y install make python3-distutils-extra python3-mock python3-twisted python3-apt python-twisted-core python3-pycurl python3-netifaces net-tools"'; |
73 | - # creates user and dirs, some used through tests |
74 | - - sg lxd -c 'lxc exec testcontainer -- sh -c "sudo apt-get -y install landscape-common"'; |
75 | - - sg lxd -c 'lxc exec testcontainer -- sh -c "sudo chown -R ubuntu:ubuntu /target"'; |
76 | - - sg lxd -c 'lxc exec testcontainer -- sudo -i -u ubuntu sh -c "cd /target; make ${TARGET}" ubuntu'; |
77 | - - python: 3.6 |
78 | - env: TARGET=lint |
79 | - install: |
80 | - - pip install flake8 |
81 | - - python: 3.4 |
82 | - env: TARGET=coverage |
83 | - before_script: |
84 | - - python3 -m pip install -U coverage |
85 | - - python3 -m pip install -U codecov |
86 | - install: |
87 | - - pip install flake8 |
88 | - # These match "make depends" |
89 | - - pip install twisted==16.4.0 mock==1.3.0 configobj==5.0.6 pycurl netifaces==0.10.4 |
90 | - - pip install http://launchpad.net/python-distutils-extra/trunk/2.39/+download/python-distutils-extra-2.39.tar.gz |
91 | - # build & install python-apt |
92 | - - make -f Makefile.travis pipinstallpythonapt |
93 | - script: |
94 | - - make $TARGET |
95 | - after_success: |
96 | - - codecov |
97 | -env: |
98 | - global: |
99 | - - TRIAL_ARGS=-j4 |
100 | -before_script: |
101 | - - if [ ! -z ${IMAGE} ]; then |
102 | - sudo apt-get -y -t trusty-backports install lxd; |
103 | - sudo lxd init --auto; |
104 | - sudo usermod -a -G lxd travis; |
105 | - sudo sed -e 's/LXD_IPV4_ADDR=""/LXD_IPV4_ADDR="10.0.8.1"/' -e 's/LXD_IPV4_NETMASK=""/LXD_IPV4_NETMASK="255.255.255.0"/' -e 's:LXD_IPV4_NETWORK="":LXD_IPV4_NETWORK="10.0.8.0/24":' -e 's/LXD_IPV4_DHCP_RANGE=""/LXD_IPV4_DHCP_RANGE="10.0.8.2,10.0.8.254"/' -e 's/LXD_IPV4_DHCP_MAX=""/LXD_IPV4_DHCP_MAX="250"/' -i /etc/default/lxd-bridge; |
106 | - sudo /etc/init.d/lxd restart; |
107 | - fi |
108 | - - if [ ! -z ${IMAGE} ]; then |
109 | - sg lxd -c 'lxc launch ${IMAGE} testcontainer -c security.privileged=true'; |
110 | - echo "waiting for nic"; |
111 | - for t in 1 2 5 8; do sg lxd -c 'lxc list' | grep 10.0.8 && break; sleep ${t}; done; |
112 | - sg lxd -c 'lxc config device add testcontainer srcdir disk path=/target source=${PWD}'; |
113 | - sg lxd -c 'lxc config device add testcontainer srcdir disk path=/target source=${PWD}'; |
114 | - sg lxd -c 'lxc exec testcontainer -- sh -c "echo 1 | sudo tee /proc/sys/net/ipv6/conf/all/disable_ipv6"'; |
115 | - fi |
116 | -script: |
117 | - - make $TARGET PYTHON=python$TRAVIS_PYTHON_VERSION; |
118 | diff --git a/Makefile b/Makefile |
119 | index fb49996..fe183ec 100644 |
120 | --- a/Makefile |
121 | +++ b/Makefile |
122 | @@ -2,7 +2,7 @@ PYDOCTOR ?= pydoctor |
123 | TXT2MAN ?= txt2man |
124 | PYTHON2 ?= python2 |
125 | PYTHON3 ?= python3 |
126 | -TRIAL ?= $(shell which trial) |
127 | +TRIAL ?= -m twisted.trial |
128 | TRIAL_ARGS ?= |
129 | |
130 | .PHONY: help |
131 | @@ -10,16 +10,16 @@ help: ## Print help about available targets |
132 | @grep -h -E '^[a-zA-Z0-9_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-20s\033[0m %s\n", $$1, $$2}' |
133 | |
134 | .PHONY: depends |
135 | -depends: depends2 depends3 ## Install py2 and py3 dependencies. |
136 | - sudo apt -y install python3-flake8 python3-coverage |
137 | +depends: depends3 ## py2 is deprecated |
138 | + sudo apt-get -y install python3-flake8 python3-coverage |
139 | |
140 | .PHONY: depends2 |
141 | depends2: |
142 | - sudo apt -y install python-twisted-core python-distutils-extra python-mock python-configobj python-netifaces |
143 | + sudo apt-get -y install python-twisted-core python-distutils-extra python-mock python-configobj python-netifaces python-pycurl |
144 | |
145 | .PHONY: depends3 |
146 | depends3: |
147 | - sudo apt -y install python3-twisted python3-distutils-extra python3-mock python3-configobj python3-netifaces |
148 | + sudo apt-get -y install python3-twisted python3-distutils-extra python3-mock python3-configobj python3-netifaces python3-pycurl |
149 | |
150 | all: build |
151 | |
152 | @@ -52,9 +52,7 @@ check3: build3 |
153 | .PHONY: coverage |
154 | coverage: |
155 | PYTHONPATH=$(PYTHONPATH):$(CURDIR) LC_ALL=C $(PYTHON3) -m coverage run $(TRIAL) --unclean-warnings landscape |
156 | - |
157 | -.PHONY: ci-check |
158 | -ci-check: depends build check ## Install dependencies and run all the tests. |
159 | + PYTHONPATH=$(PYTHONPATH):$(CURDIR) LC_ALL=C $(PYTHON3) -m coverage xml |
160 | |
161 | .PHONY: lint |
162 | lint: |
163 | diff --git a/Makefile.packaging b/Makefile.packaging |
164 | index 068eb5f..5a43ce7 100644 |
165 | --- a/Makefile.packaging |
166 | +++ b/Makefile.packaging |
167 | @@ -3,7 +3,7 @@ UBUNTU_RELEASE := $(shell lsb_release -cs) |
168 | # Use := here, not =, it's really important, otherwise UPSTREAM_VERSION |
169 | # will be updated behind your back with the current result of that |
170 | # command everytime it is mentioned/used. |
171 | -UPSTREAM_VERSION := $(shell python -c "from landscape import UPSTREAM_VERSION; print(UPSTREAM_VERSION)") |
172 | +UPSTREAM_VERSION := $(shell python3 -c "from landscape import UPSTREAM_VERSION; print(UPSTREAM_VERSION)") |
173 | CHANGELOG_VERSION := $(shell dpkg-parsechangelog | grep ^Version | cut -f 2 -d " " | cut -f 1 -d '-') |
174 | GIT_HASH := $(shell git rev-parse --short HEAD) |
175 | # We simulate a git "revno" for the sake of sortability. |
176 | diff --git a/Makefile.travis b/Makefile.travis |
177 | deleted file mode 100644 |
178 | index b532241..0000000 |
179 | --- a/Makefile.travis |
180 | +++ /dev/null |
181 | @@ -1,40 +0,0 @@ |
182 | -UBUNTU_RELEASE := $(shell lsb_release -cs) |
183 | - |
184 | -# We'd use the packaged version, but travis runs python in virtualenv, |
185 | -# thus this pip workaround. |
186 | -.PHONY: pipinstallpythonapt |
187 | -pipinstallpythonapt: pipinstallpythonapt_deps |
188 | - $(MAKE) -f $(lastword $(MAKEFILE_LIST)) pipinstallpythonapt_default || \ |
189 | - $(MAKE) -f $(lastword $(MAKEFILE_LIST)) pipinstallpythonapt_src_$(UBUNTU_RELEASE) |
190 | - |
191 | -.PHONY: pipinstallpythonapt_deps |
192 | -pipinstallpythonapt_deps: |
193 | - pip install pyopenssl |
194 | - pip install service_identity |
195 | - sudo apt-get update |
196 | - sudo apt-get -y build-dep python-apt python3-apt |
197 | - sudo apt-get -y install libapt-pkg-dev |
198 | - |
199 | -.PHONY: pipinstallpythonapt_default |
200 | -pipinstallpythonapt_default: |
201 | - # See: https://code.launchpad.net/ubuntu/+source/python-apt |
202 | - bzr branch lp:ubuntu/$(UBUNTU_RELEASE)/python-apt /tmp/python-apt |
203 | - pip install /tmp/python-apt |
204 | - |
205 | -.PHONY: pipinstallpythonapt_src |
206 | -pipinstallpythonapt_src: |
207 | - wget -O /tmp/python-apt_$(PYAPT_VER).tar.xz https://launchpad.net/ubuntu/+archive/primary/+files/python-apt_$(PYAPT_VER).tar.xz |
208 | - # pip2.7 does not support .xz |
209 | - xz -dfk /tmp/python-apt_$(PYAPT_VER).tar.xz |
210 | - pip install /tmp/python-apt_$(PYAPT_VER).tar |
211 | - |
212 | -.PHONY: pipinstallpythonapt_src_xenial |
213 | -pipinstallpythonapt_src_xenial: PYAPT_VER = 1.1.0~beta1build1 |
214 | -pipinstallpythonapt_src_xenial: pipinstallpythonapt_src |
215 | - |
216 | -# travis-ci nodes use a backported version of apt incompatible with |
217 | -# the version of python-apt on trusty. |
218 | -# This version matches the libapt-pkg on travis nodes so it compiles. |
219 | -.PHONY: pipinstallpythonapt_src_trusty |
220 | -pipinstallpythonapt_src_trusty: PYAPT_VER = 1.1.0~beta1build1 |
221 | -pipinstallpythonapt_src_trusty: pipinstallpythonapt_src |
222 | diff --git a/README b/README |
223 | index 8e5e4b1..06be0e1 100644 |
224 | --- a/README |
225 | +++ b/README |
226 | @@ -1,45 +1,88 @@ |
227 | -[![Build Status](https://travis-ci.org/CanonicalLtd/landscape-client.svg?branch=master)](https://travis-ci.org/CanonicalLtd/landscape-client) |
228 | +[![Build Status](https://github.com/CanonicalLtd/landscape-client/actions/workflows/ci.yml/badge.svg)](https://github.com/CanonicalLtd/landscape-client/actions/workflows/ci.yml) |
229 | [![codecov](https://codecov.io/gh/CanonicalLtd/landscape-client/branch/master/graph/badge.svg)](https://codecov.io/gh/CanonicalLtd/landscape-client) |
230 | |
231 | -== Non-root mode == |
232 | +## Installation Instructions |
233 | |
234 | -The Landscape Client generally runs as a combination of the 'root' and |
235 | -'landscape' users. It is possible to disable the administrative features of |
236 | -Landscape and run only the monitoring parts of it without using the 'root' |
237 | +Add our beta PPA to get the latest updates to the landscape-client package |
238 | + |
239 | +#### Add repo to an Ubuntu series |
240 | +``` |
241 | +sudo add-apt-repository ppa:landscape/self-hosted-beta |
242 | +``` |
243 | + |
244 | +#### Add repo to a Debian based series that is not Ubuntu (experimental) |
245 | + |
246 | +``` |
247 | +# 1. Install our signing key |
248 | +gpg --keyserver keyserver.ubuntu.com --recv-keys 6e85a86e4652b4e6 |
249 | +gpg --export 6e85a86e4652b4e6 | sudo tee -a /usr/share/keyrings/landscape-client-keyring.gpg > /dev/null |
250 | + |
251 | +# 2. Add repository |
252 | +echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/landscape-client-keyring.gpg] https://ppa.launchpadcontent.net/landscape/self-hosted-beta/ubuntu focal main" | sudo tee -a /etc/apt/sources.list.d/landscape-client.list |
253 | +``` |
254 | + |
255 | +#### Install the package |
256 | +``` |
257 | +sudo apt update && sudo apt install landscape-client |
258 | +``` |
259 | + |
260 | +## Non-root mode |
261 | + |
262 | +The Landscape Client generally runs as a combination of the `root` and |
263 | +`landscape` users. It is possible to disable the administrative features of |
264 | +Landscape and run only the monitoring parts of it without using the `root` |
265 | user at all. |
266 | |
267 | If you wish to use the Landscape Client in this way, it's recommended that you |
268 | perform these steps immediately after installing the landscape-client package. |
269 | |
270 | -Edit /etc/default/landscape-client and add the following lines: |
271 | +Edit `/etc/default/landscape-client` and add the following lines: |
272 | |
273 | - RUN=1 |
274 | - DAEMON_USER=landscape |
275 | +``` |
276 | +RUN=1 |
277 | +DAEMON_USER=landscape |
278 | +``` |
279 | |
280 | -Edit /etc/landscape/client.conf and add the following line: |
281 | +Edit `/etc/landscape/client.conf` and add the following line: |
282 | +``` |
283 | +monitor_only = true |
284 | +``` |
285 | |
286 | - monitor_only = true |
287 | +## Running |
288 | |
289 | -Now you can run 'sudo landscape-config' as usual to complete the configuration |
290 | -of your client and register with the Landscape service. |
291 | +Now you can complete the configuration of your client and register with the |
292 | +Landscape service. There are two ways to do this: |
293 | |
294 | +1. `sudo landscape-config` and answer interactive prompts to finalize your configuration |
295 | +2. `sudo landscape-config --account-name standalone --url https://<server>/message-system --ping-url http://<server>/ping` if registering to a self-hosted Landscape instance. Replace `<server>` with the hostname of your self-hosted Landscape instance. |
296 | |
297 | -== Developing == |
298 | +## Developing |
299 | |
300 | To run the full test suite, run the following command: |
301 | |
302 | +``` |
303 | make check |
304 | +``` |
305 | |
306 | When you want to test the landscape client manually without management |
307 | features, you can simply run: |
308 | |
309 | +``` |
310 | $ ./scripts/landscape-client |
311 | +``` |
312 | |
313 | -This defaults to the 'landscape-client.conf' configuration file. |
314 | +This defaults to the `landscape-client.conf` configuration file. |
315 | |
316 | When you want to test management features manually, you'll need to run as root. |
317 | -There's a configuration file 'root-client.conf' which specifies use of the |
318 | +There's a configuration file `root-client.conf` which specifies use of the |
319 | system bus. |
320 | |
321 | +``` |
322 | $ sudo ./scripts/landscape-client -c root-client.conf |
323 | +``` |
324 | |
325 | +Before opening a PR, make sure to run the full testsuite and lint |
326 | +``` |
327 | +make check3 |
328 | +make lint |
329 | +``` |
330 | diff --git a/debian/changelog b/debian/changelog |
331 | index fcf9dca..4ba984f 100644 |
332 | --- a/debian/changelog |
333 | +++ b/debian/changelog |
334 | @@ -1,3 +1,32 @@ |
335 | +landscape-client (23.02-0ubuntu1) lunar; urgency=medium |
336 | + |
337 | + * New upstream release 23.02: |
338 | + - Preventing the generation of large messages and logs that can overwhelm |
339 | + Landscape Server (LP: #1995775) |
340 | + - Improved MOTD slowdown on machines with many tap network interfaces |
341 | + (LP: #2006396) |
342 | + - No longer using deprecated apt-key when storing trusted GPG keys |
343 | + (LP: #1973202) |
344 | + - Fixed issue recognising Parallels VMs as Virtual Machine clients |
345 | + (LP: #1827909) |
346 | + - Fixes for incorrect logfile rotation config (LP: #1968189) |
347 | + - Client-side backoff handling to moderate traffic to Landscape Server |
348 | + during high load (LP: #1947399) |
349 | + - Avoid sending empty messages when catching up to expected next message |
350 | + (LP: #1917540) |
351 | + - --is-registered CLI option to quickly check if client is registered |
352 | + (LP: #1912516) |
353 | + - Can now report Ubuntu Pro attachment information if the version of |
354 | + Landscape Server it is registered to supports this (LP: #2006401) |
355 | + - Packages installed as dependencies as part of package profiles are now |
356 | + appropriately autoremovable (LP: #1878957) |
357 | + - Registration timeouts give an error instead of timing out (LP: #1889464) |
358 | + - RHEV hypervisor VMs are now recognized as virtual machines (LP: #1884116) |
359 | + - Doing a Landscape-driven release upgrade from a release running python 2 |
360 | + to one running python 3 no longer hangs forever (LP: #1943291) |
361 | + |
362 | + -- Mitch Burton <mitch.burton@canonical.com> Wed, 08 Feb 2023 10:23:31 -0800 |
363 | + |
364 | landscape-client (19.12-0ubuntu13) jammy; urgency=medium |
365 | |
366 | * d/landscape-sysinfo.wrapper, d/landscape-common.postrm: avoid too |
367 | diff --git a/debian/compat b/debian/compat |
368 | deleted file mode 100644 |
369 | index 48082f7..0000000 |
370 | --- a/debian/compat |
371 | +++ /dev/null |
372 | @@ -1 +0,0 @@ |
373 | -12 |
374 | diff --git a/debian/control b/debian/control |
375 | index e6178c8..3c8af12 100644 |
376 | --- a/debian/control |
377 | +++ b/debian/control |
378 | @@ -3,7 +3,7 @@ Section: admin |
379 | Priority: optional |
380 | Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com> |
381 | XSBC-Original-Maintainer: Landscape Team <landscape-team@canonical.com> |
382 | -Build-Depends: debhelper (>= 12), po-debconf, libdistro-info-perl, |
383 | +Build-Depends: debhelper (= 12), po-debconf, libdistro-info-perl, |
384 | dh-python, python3-dev, python3-distutils-extra, |
385 | lsb-release, gawk, net-tools, |
386 | python3-apt, python3-twisted, python3-configobj |
387 | diff --git a/debian/landscape-client.logrotate b/debian/landscape-client.logrotate |
388 | index c91ebf0..09f86e2 100644 |
389 | --- a/debian/landscape-client.logrotate |
390 | +++ b/debian/landscape-client.logrotate |
391 | @@ -5,8 +5,9 @@ |
392 | notifempty |
393 | compress |
394 | nocreate |
395 | + sharedscripts |
396 | postrotate |
397 | - [ -f /var/run/landscape/landscape-client.pid ] && kill -s USR1 `cat /var/run/landscape/landscape-client.pid` > /dev/null 2>&1 || : |
398 | + systemctl kill --signal=USR1 --kill-who=main landscape-client > /dev/null 2>&1 || : |
399 | endscript |
400 | } |
401 | |
402 | diff --git a/debian/landscape-client.postrm b/debian/landscape-client.postrm |
403 | index a436e39..10f79b8 100644 |
404 | --- a/debian/landscape-client.postrm |
405 | +++ b/debian/landscape-client.postrm |
406 | @@ -18,6 +18,7 @@ case "$1" in |
407 | purge) |
408 | LOG_DIR=/var/log/landscape |
409 | DATA_DIR=/var/lib/landscape |
410 | + GPG_DIR=/etc/apt/trusted.gpg.d |
411 | |
412 | rm -f "/etc/default/landscape-client" |
413 | |
414 | @@ -28,6 +29,8 @@ case "$1" in |
415 | rm -f "${LOG_DIR}/package-reporter.log"* |
416 | rm -f "${LOG_DIR}/package-changer.log"* |
417 | |
418 | + rm -f "${GPG_DIR}/landscape-server"*.asc |
419 | + |
420 | rm -rf "${DATA_DIR}/client" |
421 | rm -rf "${DATA_DIR}/.gnupg" |
422 | |
423 | diff --git a/debian/patches/0001-Handle-EINVAL-error-of-SIOCETHTOOL-ioctl.patch b/debian/patches/0001-Handle-EINVAL-error-of-SIOCETHTOOL-ioctl.patch |
424 | deleted file mode 100644 |
425 | index 49e8b44..0000000 |
426 | --- a/debian/patches/0001-Handle-EINVAL-error-of-SIOCETHTOOL-ioctl.patch |
427 | +++ /dev/null |
428 | @@ -1,28 +0,0 @@ |
429 | -From 97f74e0354d60b78a75fd7ed96c3f7fad6cd35ef Mon Sep 17 00:00:00 2001 |
430 | -From: Balint Reczey <balint.reczey@canonical.com> |
431 | -Date: Sat, 7 Dec 2019 10:11:09 +0100 |
432 | -Subject: [PATCH] Handle EINVAL error of SIOCETHTOOL ioctl |
433 | - |
434 | -The error occurs on WSL (1) and causes an error showing up in MOTD. |
435 | - |
436 | -LP: #1855522 |
437 | ---- |
438 | - landscape/lib/network.py | 4 ++-- |
439 | - 1 file changed, 2 insertions(+), 2 deletions(-) |
440 | - |
441 | ---- a/landscape/lib/network.py |
442 | -+++ b/landscape/lib/network.py |
443 | -@@ -248,11 +248,11 @@ |
444 | - fcntl.ioctl(sock, SIOCETHTOOL, packed) # Status ioctl() call |
445 | - res = status_cmd.tostring() |
446 | - speed, duplex = struct.unpack("12xHB28x", res) |
447 | -- except IOError as e: |
448 | -+ except (IOError, OSError) as e: |
449 | - if e.errno == errno.EPERM: |
450 | - logging.warn("Could not determine network interface speed, " |
451 | - "operation not permitted.") |
452 | -- elif e.errno != errno.EOPNOTSUPP: |
453 | -+ elif e.errno != errno.EOPNOTSUPP and e.errno != errno.EINVAL: |
454 | - raise e |
455 | - speed = -1 |
456 | - duplex = False |
457 | diff --git a/debian/patches/0002-lp1870087-stale-locks.patch b/debian/patches/0002-lp1870087-stale-locks.patch |
458 | deleted file mode 100644 |
459 | index f57b5b6..0000000 |
460 | --- a/debian/patches/0002-lp1870087-stale-locks.patch |
461 | +++ /dev/null |
462 | @@ -1,210 +0,0 @@ |
463 | -Description: Clean up stale lock files |
464 | - Rework lockfile to look for process names. |
465 | -Author: Simon Poirier <simon.poirier@canonical.com> |
466 | -Origin: upstream, https://github.com/CanonicalLtd/landscape-client/commit/2450eacd331097431122b9861613b8a03e5f74d9 |
467 | -Bug-Ubuntu: https://bugs.launchpad.net/bugs/1870087 |
468 | - |
469 | -Index: landscape-client-19.12/landscape/client/lockfile.py |
470 | -=================================================================== |
471 | ---- /dev/null |
472 | -+++ landscape-client-19.12/landscape/client/lockfile.py |
473 | -@@ -0,0 +1,52 @@ |
474 | -+import os |
475 | -+ |
476 | -+from twisted.python import lockfile |
477 | -+ |
478 | -+ |
479 | -+def patch_lockfile(): |
480 | -+ if lockfile.FilesystemLock is PatchedFilesystemLock: |
481 | -+ return |
482 | -+ lockfile.FilesystemLock = PatchedFilesystemLock |
483 | -+ |
484 | -+ |
485 | -+class PatchedFilesystemLock(lockfile.FilesystemLock): |
486 | -+ """ |
487 | -+ Patched Twisted's FilesystemLock.lock to handle PermissionError |
488 | -+ when trying to lock. |
489 | -+ """ |
490 | -+ |
491 | -+ def lock(self): |
492 | -+ # XXX Twisted assumes PIDs don't get reused, which is incorrect. |
493 | -+ # As such, we pre-check that any existing lock file isn't |
494 | -+ # associated to a live process, and that any associated |
495 | -+ # process is from landscape. Otherwise, clean up the lock file, |
496 | -+ # considering it to be locked to a recycled PID. |
497 | -+ # |
498 | -+ # Although looking for the process name may seem fragile, it's the |
499 | -+ # most acurate info we have since: |
500 | -+ # * some process run as root, so the UID is not a reference |
501 | -+ # * process may not be spawned by systemd, so cgroups are not reliable |
502 | -+ # * python executable is not a reference |
503 | -+ clean = True |
504 | -+ try: |
505 | -+ pid = os.readlink(self.name) |
506 | -+ ps_name = get_process_name(int(pid)) |
507 | -+ if not ps_name.startswith("landscape"): |
508 | -+ os.remove(self.name) |
509 | -+ clean = False |
510 | -+ except Exception: |
511 | -+ # We can't figure the lock state, let FilesystemLock figure it |
512 | -+ # out normally. |
513 | -+ pass |
514 | -+ |
515 | -+ result = super(PatchedFilesystemLock, self).lock() |
516 | -+ self.clean = self.clean and clean |
517 | -+ return result |
518 | -+ |
519 | -+ |
520 | -+def get_process_name(pid): |
521 | -+ """Return a process name from a pid.""" |
522 | -+ stat_path = "/proc/{}/stat".format(pid) |
523 | -+ with open(stat_path) as stat_file: |
524 | -+ stat = stat_file.read() |
525 | -+ return stat.partition("(")[2].rpartition(")")[0] |
526 | -Index: landscape-client-19.12/landscape/client/reactor.py |
527 | -=================================================================== |
528 | ---- landscape-client-19.12.orig/landscape/client/reactor.py |
529 | -+++ landscape-client-19.12/landscape/client/reactor.py |
530 | -@@ -2,6 +2,9 @@ |
531 | - Extend the regular Twisted reactor with event-handling features. |
532 | - """ |
533 | - from landscape.lib.reactor import EventHandlingReactor |
534 | -+from landscape.client.lockfile import patch_lockfile |
535 | -+ |
536 | -+patch_lockfile() |
537 | - |
538 | - |
539 | - class LandscapeReactor(EventHandlingReactor): |
540 | -Index: landscape-client-19.12/landscape/client/tests/test_amp.py |
541 | -=================================================================== |
542 | ---- landscape-client-19.12.orig/landscape/client/tests/test_amp.py |
543 | -+++ landscape-client-19.12/landscape/client/tests/test_amp.py |
544 | -@@ -1,9 +1,17 @@ |
545 | --from twisted.internet.error import ConnectError |
546 | -+import os |
547 | -+import errno |
548 | -+import subprocess |
549 | -+import textwrap |
550 | -+ |
551 | -+import mock |
552 | -+ |
553 | -+from twisted.internet.error import ConnectError, CannotListenError |
554 | - from twisted.internet.task import Clock |
555 | - |
556 | - from landscape.client.tests.helpers import LandscapeTest |
557 | - from landscape.client.deployment import Configuration |
558 | - from landscape.client.amp import ComponentPublisher, ComponentConnector, remote |
559 | -+from landscape.client.reactor import LandscapeReactor |
560 | - from landscape.lib.amp import MethodCallError |
561 | - from landscape.lib.testing import FakeReactor |
562 | - |
563 | -@@ -159,3 +167,83 @@ class ComponentConnectorTest(LandscapeTe |
564 | - effectively a no-op. |
565 | - """ |
566 | - self.connector.disconnect() |
567 | -+ |
568 | -+ @mock.patch("twisted.python.lockfile.kill") |
569 | -+ def test_stale_locks_with_dead_pid(self, mock_kill): |
570 | -+ """Publisher starts with stale lock.""" |
571 | -+ mock_kill.side_effect = [ |
572 | -+ OSError(errno.ESRCH, "No such process")] |
573 | -+ sock_path = os.path.join(self.config.sockets_path, u"test.sock") |
574 | -+ lock_path = u"{}.lock".format(sock_path) |
575 | -+ # fake a PID which does not exist |
576 | -+ os.symlink("-1", lock_path) |
577 | -+ |
578 | -+ component = TestComponent() |
579 | -+ # Test the actual Unix reactor implementation. Fakes won't do. |
580 | -+ reactor = LandscapeReactor() |
581 | -+ publisher = ComponentPublisher(component, reactor, self.config) |
582 | -+ |
583 | -+ # Shouldn't raise the exception. |
584 | -+ publisher.start() |
585 | -+ |
586 | -+ # ensure stale lock was replaced |
587 | -+ self.assertNotEqual("-1", os.readlink(lock_path)) |
588 | -+ mock_kill.assert_called_with(-1, 0) |
589 | -+ |
590 | -+ publisher.stop() |
591 | -+ reactor._cleanup() |
592 | -+ |
593 | -+ @mock.patch("twisted.python.lockfile.kill") |
594 | -+ def test_stale_locks_recycled_pid(self, mock_kill): |
595 | -+ """Publisher starts with stale lock pointing to recycled process.""" |
596 | -+ mock_kill.side_effect = [ |
597 | -+ OSError(errno.EPERM, "Operation not permitted")] |
598 | -+ sock_path = os.path.join(self.config.sockets_path, u"test.sock") |
599 | -+ lock_path = u"{}.lock".format(sock_path) |
600 | -+ # fake a PID recycled by a known process which isn't landscape (init) |
601 | -+ os.symlink("1", lock_path) |
602 | -+ |
603 | -+ component = TestComponent() |
604 | -+ # Test the actual Unix reactor implementation. Fakes won't do. |
605 | -+ reactor = LandscapeReactor() |
606 | -+ publisher = ComponentPublisher(component, reactor, self.config) |
607 | -+ |
608 | -+ # Shouldn't raise the exception. |
609 | -+ publisher.start() |
610 | -+ |
611 | -+ # ensure stale lock was replaced |
612 | -+ self.assertNotEqual("1", os.readlink(lock_path)) |
613 | -+ mock_kill.assert_not_called() |
614 | -+ self.assertFalse(publisher._port.lockFile.clean) |
615 | -+ |
616 | -+ publisher.stop() |
617 | -+ reactor._cleanup() |
618 | -+ |
619 | -+ @mock.patch("twisted.python.lockfile.kill") |
620 | -+ def test_with_valid_lock(self, mock_kill): |
621 | -+ """Publisher raises lock error if a valid lock is held.""" |
622 | -+ sock_path = os.path.join(self.config.sockets_path, u"test.sock") |
623 | -+ lock_path = u"{}.lock".format(sock_path) |
624 | -+ # fake a landscape process |
625 | -+ app = self.makeFile(textwrap.dedent("""\ |
626 | -+ #!/usr/bin/python3 |
627 | -+ import time |
628 | -+ time.sleep(10) |
629 | -+ """), basename="landscape-manager") |
630 | -+ os.chmod(app, 0o755) |
631 | -+ call = subprocess.Popen([app]) |
632 | -+ self.addCleanup(call.terminate) |
633 | -+ os.symlink(str(call.pid), lock_path) |
634 | -+ |
635 | -+ component = TestComponent() |
636 | -+ # Test the actual Unix reactor implementation. Fakes won't do. |
637 | -+ reactor = LandscapeReactor() |
638 | -+ publisher = ComponentPublisher(component, reactor, self.config) |
639 | -+ |
640 | -+ with self.assertRaises(CannotListenError): |
641 | -+ publisher.start() |
642 | -+ |
643 | -+ # ensure lock was not replaced |
644 | -+ self.assertEqual(str(call.pid), os.readlink(lock_path)) |
645 | -+ mock_kill.assert_called_with(call.pid, 0) |
646 | -+ reactor._cleanup() |
647 | -Index: landscape-client-19.12/landscape/client/tests/test_lockfile.py |
648 | -=================================================================== |
649 | ---- /dev/null |
650 | -+++ landscape-client-19.12/landscape/client/tests/test_lockfile.py |
651 | -@@ -0,0 +1,21 @@ |
652 | -+import os |
653 | -+import subprocess |
654 | -+import textwrap |
655 | -+ |
656 | -+from landscape.client import lockfile |
657 | -+from landscape.client.tests.helpers import LandscapeTest |
658 | -+ |
659 | -+ |
660 | -+class LockFileTest(LandscapeTest): |
661 | -+ |
662 | -+ def test_read_process_name(self): |
663 | -+ app = self.makeFile(textwrap.dedent("""\ |
664 | -+ #!/usr/bin/python3 |
665 | -+ import time |
666 | -+ time.sleep(10) |
667 | -+ """), basename="my_fancy_app") |
668 | -+ os.chmod(app, 0o755) |
669 | -+ call = subprocess.Popen([app]) |
670 | -+ self.addCleanup(call.terminate) |
671 | -+ proc_name = lockfile.get_process_name(call.pid) |
672 | -+ self.assertEqual("my_fancy_app", proc_name) |
673 | diff --git a/debian/patches/0003-clean-publisher-shutdown.patch b/debian/patches/0003-clean-publisher-shutdown.patch |
674 | deleted file mode 100644 |
675 | index 3c7bc1e..0000000 |
676 | --- a/debian/patches/0003-clean-publisher-shutdown.patch |
677 | +++ /dev/null |
678 | @@ -1,49 +0,0 @@ |
679 | -Description: Return missing deferred on service shutdown. |
680 | - This should allow services to let their publisher shut down and clean |
681 | - after itself, thus avoiding stale locks and sockets. |
682 | -Author: Simon Poirier <simon.poirier@canonical.com> |
683 | -Origin: upstream, https://github.com/CanonicalLtd/landscape-client/commit/8a617a1aafbbe9d972410707563710c3cbedd8e7 |
684 | -Bug-Ubuntu: https://bugs.launchpad.net/bugs/1870087 |
685 | ---- a/landscape/client/broker/service.py |
686 | -+++ b/landscape/client/broker/service.py |
687 | -@@ -82,10 +82,11 @@ |
688 | - |
689 | - def stopService(self): |
690 | - """Stop the broker.""" |
691 | -- self.publisher.stop() |
692 | -+ deferred = self.publisher.stop() |
693 | - self.exchanger.stop() |
694 | - self.pinger.stop() |
695 | - super(BrokerService, self).stopService() |
696 | -+ return deferred |
697 | - |
698 | - |
699 | - def run(args): |
700 | ---- a/landscape/client/manager/service.py |
701 | -+++ b/landscape/client/manager/service.py |
702 | -@@ -54,8 +54,9 @@ |
703 | - def stopService(self): |
704 | - """Stop the manager and close the connection with the broker.""" |
705 | - self.connector.disconnect() |
706 | -- self.publisher.stop() |
707 | -+ deferred = self.publisher.stop() |
708 | - super(ManagerService, self).stopService() |
709 | -+ return deferred |
710 | - |
711 | - |
712 | - def run(args): |
713 | ---- a/landscape/client/monitor/service.py |
714 | -+++ b/landscape/client/monitor/service.py |
715 | -@@ -56,10 +56,11 @@ |
716 | - The monitor is flushed to ensure that things like persist databases |
717 | - get saved to disk. |
718 | - """ |
719 | -- self.publisher.stop() |
720 | -+ deferred = self.publisher.stop() |
721 | - self.monitor.flush() |
722 | - self.connector.disconnect() |
723 | - super(MonitorService, self).stopService() |
724 | -+ return deferred |
725 | - |
726 | - |
727 | - def run(args): |
728 | diff --git a/debian/patches/1962539_twisted_py3.patch b/debian/patches/1962539_twisted_py3.patch |
729 | deleted file mode 100644 |
730 | index 7593ffd..0000000 |
731 | --- a/debian/patches/1962539_twisted_py3.patch |
732 | +++ /dev/null |
733 | @@ -1,156 +0,0 @@ |
734 | -Description: Fix obsolete imports on jammy. |
735 | - * replace deprecated twisted _PY3 |
736 | - * replace legacy types which are gone (unicode, long) |
737 | -Author: Simon Poirier <simon.poirier@canonical.com> |
738 | -Origin: backport, https://github.com/CanonicalLtd/landscape-client/commit/80d3da45ca8773d3eee262d4de5dc6e6b8d99121 |
739 | -Bug-Ubuntu: https://bugs.launchpad.net/bugs/1962539 |
740 | -Last-Update: 2022-03-03 |
741 | - |
742 | ---- a/landscape/client/broker/exchange.py |
743 | -+++ b/landscape/client/broker/exchange.py |
744 | -@@ -347,7 +347,7 @@ |
745 | - from landscape.lib.hashlib import md5 |
746 | - |
747 | - from twisted.internet.defer import Deferred, succeed |
748 | --from twisted.python.compat import _PY3 |
749 | -+from landscape.lib.compat import _PY3 |
750 | - |
751 | - from landscape.lib.fetch import HTTPCodeError, PyCurlError |
752 | - from landscape.lib.format import format_delta |
753 | ---- a/landscape/client/broker/tests/test_registration.py |
754 | -+++ b/landscape/client/broker/tests/test_registration.py |
755 | -@@ -3,7 +3,7 @@ |
756 | - import socket |
757 | - import mock |
758 | - |
759 | --from twisted.python.compat import _PY3 |
760 | -+from landscape.lib.compat import _PY3 |
761 | - |
762 | - from landscape.client.broker.registration import RegistrationError, Identity |
763 | - from landscape.client.tests.helpers import LandscapeTest |
764 | ---- a/landscape/client/broker/transport.py |
765 | -+++ b/landscape/client/broker/transport.py |
766 | -@@ -6,7 +6,7 @@ |
767 | - |
768 | - import pycurl |
769 | - |
770 | --from twisted.python.compat import unicode, _PY3 |
771 | -+from landscape.lib.compat import unicode, _PY3 |
772 | - |
773 | - from landscape.lib import bpickle |
774 | - from landscape.lib.fetch import fetch |
775 | ---- a/landscape/client/manager/keystonetoken.py |
776 | -+++ b/landscape/client/manager/keystonetoken.py |
777 | -@@ -1,7 +1,7 @@ |
778 | - import os |
779 | - import logging |
780 | - |
781 | --from twisted.python.compat import _PY3 |
782 | -+from landscape.lib.compat import _PY3 |
783 | - |
784 | - from landscape.lib.compat import ConfigParser, NoOptionError |
785 | - from landscape.client.monitor.plugin import DataWatcher |
786 | ---- a/landscape/client/user/provider.py |
787 | -+++ b/landscape/client/user/provider.py |
788 | -@@ -4,7 +4,7 @@ |
789 | - import logging |
790 | - import subprocess |
791 | - |
792 | --from twisted.python.compat import _PY3 |
793 | -+from landscape.lib.compat import _PY3 |
794 | - |
795 | - |
796 | - class UserManagementError(Exception): |
797 | ---- a/landscape/lib/apt/package/skeleton.py |
798 | -+++ b/landscape/lib/apt/package/skeleton.py |
799 | -@@ -1,9 +1,8 @@ |
800 | -+from landscape.lib.compat import unicode, _PY3 |
801 | - from landscape.lib.hashlib import sha1 |
802 | - |
803 | - import apt_pkg |
804 | - |
805 | --from twisted.python.compat import unicode, _PY3 |
806 | -- |
807 | - |
808 | - PACKAGE = 1 << 0 |
809 | - PROVIDES = 1 << 1 |
810 | ---- a/landscape/lib/bpickle.py |
811 | -+++ b/landscape/lib/bpickle.py |
812 | -@@ -32,7 +32,7 @@ |
813 | - wire compatible and behave the same way (bugs notwithstanding). |
814 | - """ |
815 | - |
816 | --from twisted.python.compat import _PY3 |
817 | -+from landscape.lib.compat import long, _PY3 |
818 | - |
819 | - dumps_table = {} |
820 | - loads_table = {} |
821 | ---- a/landscape/lib/compat.py |
822 | -+++ b/landscape/lib/compat.py |
823 | -@@ -1,6 +1,6 @@ |
824 | - # flake8: noqa |
825 | - |
826 | --from twisted.python.compat import _PY3 |
827 | -+_PY3 = str != bytes |
828 | - |
829 | - |
830 | - if _PY3: |
831 | -@@ -13,6 +13,8 @@ |
832 | - from io import StringIO |
833 | - stringio = cstringio = StringIO |
834 | - from builtins import input |
835 | -+ unicode = str |
836 | -+ long = int |
837 | - |
838 | - else: |
839 | - import cPickle |
840 | -@@ -24,3 +26,5 @@ |
841 | - stringio = StringIO |
842 | - from cStringIO import StringIO as cstringio |
843 | - input = raw_input |
844 | -+ long = long |
845 | -+ unicode = unicode |
846 | ---- a/landscape/lib/disk.py |
847 | -+++ b/landscape/lib/disk.py |
848 | -@@ -4,7 +4,7 @@ |
849 | - import re |
850 | - import codecs |
851 | - |
852 | --from twisted.python.compat import _PY3 |
853 | -+from landscape.lib.compat import _PY3 |
854 | - |
855 | - |
856 | - # List of filesystem types authorized when generating disk use statistics. |
857 | ---- a/landscape/lib/network.py |
858 | -+++ b/landscape/lib/network.py |
859 | -@@ -11,7 +11,7 @@ |
860 | - import logging |
861 | - |
862 | - import netifaces |
863 | --from twisted.python.compat import long, _PY3 |
864 | -+from landscape.lib.compat import long, _PY3 |
865 | - |
866 | - __all__ = ["get_active_device_info", "get_network_traffic"] |
867 | - |
868 | ---- a/landscape/lib/testing.py |
869 | -+++ b/landscape/lib/testing.py |
870 | -@@ -15,7 +15,7 @@ |
871 | - from logging import Handler, ERROR, Formatter |
872 | - from twisted.trial.unittest import TestCase |
873 | - from twisted.python.compat import StringType as basestring |
874 | --from twisted.python.compat import _PY3 |
875 | -+from landscape.lib.compat import _PY3 |
876 | - from twisted.python.failure import Failure |
877 | - from twisted.internet.defer import Deferred |
878 | - from twisted.internet.error import ConnectError |
879 | ---- a/landscape/lib/user.py |
880 | -+++ b/landscape/lib/user.py |
881 | -@@ -1,7 +1,7 @@ |
882 | - import os.path |
883 | - import pwd |
884 | - |
885 | --from twisted.python.compat import _PY3 |
886 | -+from landscape.lib.compat import _PY3 |
887 | - |
888 | - from landscape.lib.encoding import encode_if_needed |
889 | - |
890 | diff --git a/debian/patches/lp1903776-release-upgrade.patch b/debian/patches/lp1903776-release-upgrade.patch |
891 | deleted file mode 100644 |
892 | index bd3a154..0000000 |
893 | --- a/debian/patches/lp1903776-release-upgrade.patch |
894 | +++ /dev/null |
895 | @@ -1,141 +0,0 @@ |
896 | -Description: Use /etc/apt/trusted.gpg.d for validating upgrade-tool signature. |
897 | -Author: Simon Poirier <simon.poirier@canonical.com> |
898 | -Origin: upstream, https://github.com/CanonicalLtd/landscape-client/commit/bcbe04db6ca45c7a0afac171e85a9c88c19253ae |
899 | -Bug-Ubuntu: https://bugs.launchpad.net/bugs/1903776 |
900 | - |
901 | -diff --git a/landscape/lib/gpg.py b/landscape/lib/gpg.py |
902 | -index 66400d18..58218c54 100644 |
903 | ---- a/landscape/lib/gpg.py |
904 | -+++ b/landscape/lib/gpg.py |
905 | -@@ -1,6 +1,9 @@ |
906 | -+import itertools |
907 | - import shutil |
908 | - import tempfile |
909 | - |
910 | -+from glob import glob |
911 | -+ |
912 | - from twisted.internet.utils import getProcessOutputAndValue |
913 | - |
914 | - |
915 | -@@ -8,14 +11,15 @@ class InvalidGPGSignature(Exception): |
916 | - """Raised when the gpg signature for a given file is invalid.""" |
917 | - |
918 | - |
919 | --def gpg_verify(filename, signature, gpg="/usr/bin/gpg"): |
920 | -+def gpg_verify(filename, signature, gpg="/usr/bin/gpg", apt_dir="/etc/apt"): |
921 | - """Verify the GPG signature of a file. |
922 | - |
923 | - @param filename: Path to the file to verify the signature against. |
924 | - @param signature: Path to signature to use. |
925 | - @param gpg: Optionally, path to the GPG binary to use. |
926 | -+ @param apt_dir: Optionally, path to apt trusted keyring. |
927 | - @return: a C{Deferred} resulting in C{True} if the signature is |
928 | -- valid, C{False} otherwise. |
929 | -+ valid, C{False} otherwise. |
930 | - """ |
931 | - |
932 | - def remove_gpg_home(ignored): |
933 | -@@ -32,9 +36,17 @@ def check_gpg_exit_code(args): |
934 | - "code='%d')" % (gpg, out, err, code)) |
935 | - |
936 | - gpg_home = tempfile.mkdtemp() |
937 | -- args = ("--no-options", "--homedir", gpg_home, "--no-default-keyring", |
938 | -- "--ignore-time-conflict", "--keyring", "/etc/apt/trusted.gpg", |
939 | -- "--verify", signature, filename) |
940 | -+ keyrings = tuple(itertools.chain(*[ |
941 | -+ ("--keyring", keyring) |
942 | -+ for keyring in sorted( |
943 | -+ glob("{}/trusted.gpg".format(apt_dir)) + |
944 | -+ glob("{}/trusted.gpg.d/*.gpg".format(apt_dir)) |
945 | -+ ) |
946 | -+ ])) |
947 | -+ args = ( |
948 | -+ "--no-options", "--homedir", gpg_home, "--no-default-keyring", |
949 | -+ "--ignore-time-conflict" |
950 | -+ ) + keyrings + ("--verify", signature, filename) |
951 | - |
952 | - result = getProcessOutputAndValue(gpg, args=args) |
953 | - result.addBoth(remove_gpg_home) |
954 | -diff --git a/landscape/lib/tests/test_gpg.py b/landscape/lib/tests/test_gpg.py |
955 | -index e6165a26..4c84e008 100644 |
956 | ---- a/landscape/lib/tests/test_gpg.py |
957 | -+++ b/landscape/lib/tests/test_gpg.py |
958 | -@@ -1,9 +1,10 @@ |
959 | - import mock |
960 | - import os |
961 | -+import textwrap |
962 | - import unittest |
963 | - |
964 | - from twisted.internet import reactor |
965 | --from twisted.internet.defer import Deferred |
966 | -+from twisted.internet.defer import Deferred, inlineCallbacks |
967 | - |
968 | - from landscape.lib import testing |
969 | - from landscape.lib.gpg import gpg_verify |
970 | -@@ -16,6 +17,8 @@ def test_gpg_verify(self): |
971 | - L{gpg_verify} runs the given gpg binary and returns C{True} if the |
972 | - provided signature is valid. |
973 | - """ |
974 | -+ aptdir = self.makeDir() |
975 | -+ os.mknod("{}/trusted.gpg".format(aptdir)) |
976 | - gpg_options = self.makeFile() |
977 | - gpg = self.makeFile("#!/bin/sh\n" |
978 | - "touch $3/trustdb.gpg\n" |
979 | -@@ -27,14 +30,15 @@ def test_gpg_verify(self): |
980 | - @mock.patch("tempfile.mkdtemp") |
981 | - def do_test(mkdtemp_mock): |
982 | - mkdtemp_mock.return_value = gpg_home |
983 | -- result = gpg_verify("/some/file", "/some/signature", gpg=gpg) |
984 | -+ result = gpg_verify( |
985 | -+ "/some/file", "/some/signature", gpg=gpg, apt_dir=aptdir) |
986 | - |
987 | - def check_result(ignored): |
988 | - self.assertEqual( |
989 | - open(gpg_options).read(), |
990 | - "--no-options --homedir %s --no-default-keyring " |
991 | -- "--ignore-time-conflict --keyring /etc/apt/trusted.gpg " |
992 | -- "--verify /some/signature /some/file" % gpg_home) |
993 | -+ "--ignore-time-conflict --keyring %s/trusted.gpg " |
994 | -+ "--verify /some/signature /some/file" % (gpg_home, aptdir)) |
995 | - self.assertFalse(os.path.exists(gpg_home)) |
996 | - |
997 | - result.addCallback(check_result) |
998 | -@@ -70,3 +74,38 @@ def check_failure(failure): |
999 | - |
1000 | - reactor.callWhenRunning(do_test) |
1001 | - return deferred |
1002 | -+ |
1003 | -+ @inlineCallbacks |
1004 | -+ def test_gpg_verify_trusted_dir(self): |
1005 | -+ """ |
1006 | -+ gpg_verify uses keys from the trusted.gpg.d if such a folder exists. |
1007 | -+ """ |
1008 | -+ apt_dir = self.makeDir() |
1009 | -+ os.mkdir("{}/trusted.gpg.d".format(apt_dir)) |
1010 | -+ os.mknod("{}/trusted.gpg.d/foo.gpg".format(apt_dir)) |
1011 | -+ os.mknod("{}/trusted.gpg.d/baz.gpg".format(apt_dir)) |
1012 | -+ os.mknod("{}/trusted.gpg.d/bad.gpg~".format(apt_dir)) |
1013 | -+ |
1014 | -+ gpg_call = self.makeFile() |
1015 | -+ fake_gpg = self.makeFile(textwrap.dedent("""\ |
1016 | -+ #!/bin/sh |
1017 | -+ touch $3/trustdb.gpg |
1018 | -+ echo -n $@ > {} |
1019 | -+ """).format(gpg_call)) |
1020 | -+ os.chmod(fake_gpg, 0o755) |
1021 | -+ gpg_home = self.makeDir() |
1022 | -+ |
1023 | -+ with mock.patch("tempfile.mkdtemp", return_value=gpg_home): |
1024 | -+ yield gpg_verify( |
1025 | -+ "/some/file", "/some/signature", gpg=fake_gpg, apt_dir=apt_dir) |
1026 | -+ |
1027 | -+ expected = ( |
1028 | -+ "--no-options --homedir {gpg_home} --no-default-keyring " |
1029 | -+ "--ignore-time-conflict " |
1030 | -+ "--keyring {apt_dir}/trusted.gpg.d/baz.gpg " |
1031 | -+ "--keyring {apt_dir}/trusted.gpg.d/foo.gpg " |
1032 | -+ "--verify /some/signature /some/file" |
1033 | -+ ).format(gpg_home=gpg_home, apt_dir=apt_dir) |
1034 | -+ with open(gpg_call) as call: |
1035 | -+ self.assertEqual(expected, call.read()) |
1036 | -+ self.assertFalse(os.path.exists(gpg_home)) |
1037 | diff --git a/debian/patches/py3.9.patch b/debian/patches/py3.9.patch |
1038 | deleted file mode 100644 |
1039 | index 4c70c7d..0000000 |
1040 | --- a/debian/patches/py3.9.patch |
1041 | +++ /dev/null |
1042 | @@ -1,65 +0,0 @@ |
1043 | -Description: Switch from depreated base64.decodestring to |
1044 | - base64.decodebytes to fix FTBFS with python3.9. |
1045 | -Author: Dimitri John Ledkov <xnox@ubuntu.com> |
1046 | - |
1047 | - |
1048 | ---- landscape-client-19.12.orig/landscape/client/configuration.py |
1049 | -+++ landscape-client-19.12/landscape/client/configuration.py |
1050 | -@@ -513,7 +513,7 @@ def decode_base64_ssl_public_certificate |
1051 | - # WARNING: ssl_public_certificate is misnamed, it's not the key of the |
1052 | - # certificate, but the actual certificate itself. |
1053 | - if config.ssl_public_key and config.ssl_public_key.startswith("base64:"): |
1054 | -- decoded_cert = base64.decodestring( |
1055 | -+ decoded_cert = base64.decodebytes( |
1056 | - config.ssl_public_key[7:].encode("ascii")) |
1057 | - config.ssl_public_key = store_public_key_data( |
1058 | - config, decoded_cert) |
1059 | ---- landscape-client-19.12.orig/landscape/client/package/changer.py |
1060 | -+++ landscape-client-19.12/landscape/client/package/changer.py |
1061 | -@@ -173,7 +173,7 @@ class PackageChanger(PackageTaskHandler) |
1062 | - hash_ids = {} |
1063 | - for hash, id, deb in binaries: |
1064 | - create_binary_file(os.path.join(binaries_path, "%d.deb" % id), |
1065 | -- base64.decodestring(deb)) |
1066 | -+ base64.decodebytes(deb)) |
1067 | - hash_ids[hash] = id |
1068 | - self._store.set_hash_ids(hash_ids) |
1069 | - self._facade.add_channel_deb_dir(binaries_path) |
1070 | ---- landscape-client-19.12.orig/landscape/client/package/tests/test_changer.py |
1071 | -+++ landscape-client-19.12/landscape/client/package/tests/test_changer.py |
1072 | -@@ -849,9 +849,9 @@ class AptPackageChangerTest(LandscapeTes |
1073 | - |
1074 | - binaries_path = self.config.binaries_path |
1075 | - self.assertFileContent(os.path.join(binaries_path, "111.deb"), |
1076 | -- base64.decodestring(PKGDEB1)) |
1077 | -+ base64.decodebytes(PKGDEB1)) |
1078 | - self.assertFileContent(os.path.join(binaries_path, "222.deb"), |
1079 | -- base64.decodestring(PKGDEB2)) |
1080 | -+ base64.decodebytes(PKGDEB2)) |
1081 | - self.assertEqual( |
1082 | - self.facade.get_channels(), |
1083 | - self.get_binaries_channels(binaries_path)) |
1084 | ---- landscape-client-19.12.orig/landscape/lib/apt/package/testing.py |
1085 | -+++ landscape-client-19.12/landscape/lib/apt/package/testing.py |
1086 | -@@ -322,9 +322,9 @@ PKGDEB_OR_RELATIONS = ( |
1087 | - ).encode("ascii") |
1088 | - |
1089 | - |
1090 | --HASH1 = base64.decodestring(b"/ezv4AefpJJ8DuYFSq4RiEHJYP4=") |
1091 | --HASH2 = base64.decodestring(b"glP4DwWOfMULm0AkRXYsH/exehc=") |
1092 | --HASH3 = base64.decodestring(b"NJM05mj86veaSInYxxqL1wahods=") |
1093 | -+HASH1 = base64.decodebytes(b"/ezv4AefpJJ8DuYFSq4RiEHJYP4=") |
1094 | -+HASH2 = base64.decodebytes(b"glP4DwWOfMULm0AkRXYsH/exehc=") |
1095 | -+HASH3 = base64.decodebytes(b"NJM05mj86veaSInYxxqL1wahods=") |
1096 | - HASH_MINIMAL = b"6\xce\x8f\x1bM\x82MWZ\x1a\xffjAc(\xdb(\xa1\x0eG" |
1097 | - HASH_SIMPLE_RELATIONS = ( |
1098 | - b"'#\xab&k\xe6\xf5E\xcfB\x9b\xceO7\xe6\xec\xa9\xddY\xaa") |
1099 | -@@ -339,7 +339,7 @@ HASH_OR_RELATIONS = ( |
1100 | - def create_deb(target_dir, pkg_name, pkg_data): |
1101 | - """Create a Debian package in the specified C{target_dir}.""" |
1102 | - path = os.path.join(target_dir, pkg_name) |
1103 | -- data = base64.decodestring(pkg_data) |
1104 | -+ data = base64.decodebytes(pkg_data) |
1105 | - create_binary_file(path, data) |
1106 | - |
1107 | - |
1108 | diff --git a/debian/patches/replace-tostring.patch b/debian/patches/replace-tostring.patch |
1109 | deleted file mode 100644 |
1110 | index 834eb53..0000000 |
1111 | --- a/debian/patches/replace-tostring.patch |
1112 | +++ /dev/null |
1113 | @@ -1,25 +0,0 @@ |
1114 | -Index: landscape-client-19.12/landscape/lib/network.py |
1115 | -=================================================================== |
1116 | ---- landscape-client-19.12.orig/landscape/lib/network.py |
1117 | -+++ landscape-client-19.12/landscape/lib/network.py |
1118 | -@@ -11,7 +11,7 @@ import errno |
1119 | - import logging |
1120 | - |
1121 | - import netifaces |
1122 | --from twisted.python.compat import long |
1123 | -+from twisted.python.compat import long, _PY3 |
1124 | - |
1125 | - __all__ = ["get_active_device_info", "get_network_traffic"] |
1126 | - |
1127 | -@@ -246,7 +246,10 @@ def get_network_interface_speed(sock, in |
1128 | - speed = -1 |
1129 | - try: |
1130 | - fcntl.ioctl(sock, SIOCETHTOOL, packed) # Status ioctl() call |
1131 | -- res = status_cmd.tostring() |
1132 | -+ if _PY3: |
1133 | -+ res = status_cmd.tobytes() |
1134 | -+ else: |
1135 | -+ res = status_cmd.tostring() |
1136 | - speed, duplex = struct.unpack("12xHB28x", res) |
1137 | - except (IOError, OSError) as e: |
1138 | - if e.errno == errno.EPERM: |
1139 | diff --git a/debian/patches/series b/debian/patches/series |
1140 | index dcad3a3..e69de29 100644 |
1141 | --- a/debian/patches/series |
1142 | +++ b/debian/patches/series |
1143 | @@ -1,7 +0,0 @@ |
1144 | -0001-Handle-EINVAL-error-of-SIOCETHTOOL-ioctl.patch |
1145 | -0002-lp1870087-stale-locks.patch |
1146 | -py3.9.patch |
1147 | -0003-clean-publisher-shutdown.patch |
1148 | -replace-tostring.patch |
1149 | -1962539_twisted_py3.patch |
1150 | -lp1903776-release-upgrade.patch |
1151 | diff --git a/debian/rules b/debian/rules |
1152 | index 37cce8e..ad30f5c 100755 |
1153 | --- a/debian/rules |
1154 | +++ b/debian/rules |
1155 | @@ -23,4 +23,4 @@ override_dh_auto_install: |
1156 | install -D -o root -g root -m 755 apt-update/apt-update $(CURDIR)/$(root_dir)$(LIBDIR)/apt-update |
1157 | |
1158 | override_dh_installsystemd: |
1159 | - dh_installsystemd --no-enable --no-start |
1160 | + dh_installsystemd --no-enable --no-start --no-stop-on-upgrade |
1161 | diff --git a/example.conf b/example.conf |
1162 | index 3965d29..4b7fd94 100644 |
1163 | --- a/example.conf |
1164 | +++ b/example.conf |
1165 | @@ -174,3 +174,11 @@ manager_plugins = ALL |
1166 | # |
1167 | # By default, all usernames are allowed. |
1168 | script_users = ALL |
1169 | + |
1170 | + |
1171 | +# The maximum script output length transmitted to landscape |
1172 | +# Output over this limit is truncated |
1173 | +# |
1174 | +# The default is 512kB |
1175 | +# 2MB is allowed in this example |
1176 | +#script_output_limit=2048 |
1177 | diff --git a/landscape/__init__.py b/landscape/__init__.py |
1178 | index d76ad6c..23f8f79 100644 |
1179 | --- a/landscape/__init__.py |
1180 | +++ b/landscape/__init__.py |
1181 | @@ -1,5 +1,5 @@ |
1182 | DEBIAN_REVISION = "" |
1183 | -UPSTREAM_VERSION = "18.01" |
1184 | +UPSTREAM_VERSION = "23.02" |
1185 | VERSION = "%s%s" % (UPSTREAM_VERSION, DEBIAN_REVISION) |
1186 | |
1187 | # The minimum server API version that all Landscape servers are known to speak |
1188 | diff --git a/landscape/client/broker/exchange.py b/landscape/client/broker/exchange.py |
1189 | index ee6d0e9..924cb9a 100644 |
1190 | --- a/landscape/client/broker/exchange.py |
1191 | +++ b/landscape/client/broker/exchange.py |
1192 | @@ -347,11 +347,12 @@ import logging |
1193 | from landscape.lib.hashlib import md5 |
1194 | |
1195 | from twisted.internet.defer import Deferred, succeed |
1196 | -from twisted.python.compat import _PY3 |
1197 | +from landscape.lib.compat import _PY3 |
1198 | |
1199 | +from landscape.lib.backoff import ExponentialBackoff |
1200 | from landscape.lib.fetch import HTTPCodeError, PyCurlError |
1201 | from landscape.lib.format import format_delta |
1202 | -from landscape.lib.message import got_next_expected, ANCIENT |
1203 | +from landscape.lib.message import got_next_expected, RESYNC |
1204 | from landscape.lib.versioning import is_version_higher, sort_versions |
1205 | |
1206 | from landscape import DEFAULT_SERVER_API, SERVER_API, CLIENT_API |
1207 | @@ -397,6 +398,7 @@ class MessageExchange(object): |
1208 | self._exchange_interval = config.exchange_interval |
1209 | self._urgent_exchange_interval = config.urgent_exchange_interval |
1210 | self._max_messages = max_messages |
1211 | + self._max_log_text_bytes = 100000 # 100KB |
1212 | self._notification_id = None |
1213 | self._exchange_id = None |
1214 | self._exchanging = False |
1215 | @@ -406,6 +408,7 @@ class MessageExchange(object): |
1216 | self._message_handlers = {} |
1217 | self._exchange_store = exchange_store |
1218 | self._stopped = False |
1219 | + self._backoff_counter = ExponentialBackoff(300, 7200) # 5 to 120 min |
1220 | |
1221 | self.register_message("accepted-types", self._handle_accepted_types) |
1222 | self.register_message("resynchronize", self._handle_resynchronize) |
1223 | @@ -436,6 +439,20 @@ class MessageExchange(object): |
1224 | |
1225 | return result |
1226 | |
1227 | + def truncate_message_field(self, field, message): |
1228 | + """ |
1229 | + Truncates message field value based on length |
1230 | + """ |
1231 | + if field in message: |
1232 | + value = message[field] |
1233 | + if isinstance(value, str): |
1234 | + # Note this is an approximation based on 1 byte chars |
1235 | + max_bytes = self._max_log_text_bytes |
1236 | + if len(value) > max_bytes: |
1237 | + value = value[:max_bytes] |
1238 | + value += '...MESSAGE TRUNCATED DUE TO SIZE' |
1239 | + message[field] = value |
1240 | + |
1241 | def send(self, message, urgent=False): |
1242 | """Include a message to be sent in an exchange. |
1243 | |
1244 | @@ -451,6 +468,10 @@ class MessageExchange(object): |
1245 | % message.get('operation-id')) |
1246 | return None |
1247 | |
1248 | + # These fields sometimes have really long output we need to trim |
1249 | + self.truncate_message_field('err', message) |
1250 | + self.truncate_message_field('result-text', message) |
1251 | + |
1252 | if "timestamp" not in message: |
1253 | message["timestamp"] = int(self._reactor.time()) |
1254 | message_id = self._message_store.add(message) |
1255 | @@ -567,6 +588,7 @@ class MessageExchange(object): |
1256 | self._urgent_exchange = False |
1257 | self._handle_result(payload, result) |
1258 | self._message_store.record_success(int(self._reactor.time())) |
1259 | + self._backoff_counter.decrease() |
1260 | else: |
1261 | self._reactor.fire("exchange-failed") |
1262 | logging.info("Message exchange failed.") |
1263 | @@ -586,6 +608,20 @@ class MessageExchange(object): |
1264 | self.exchange() |
1265 | return |
1266 | |
1267 | + if isinstance(error, HTTPCodeError): |
1268 | + if error.http_code == 429 or (500 <= error.http_code <= 599): |
1269 | + # We add an exponentially increasing delay ("backoff") if |
1270 | + # the server is overloaded to decrease load. We assume that |
1271 | + # any server backend error is deserving backoff, including |
1272 | + # 429 which is sent from the server on purpose to trigger |
1273 | + # the backoff. Whether the server is down or overloaded |
1274 | + # (503), has a server bug crashing the response (500), |
1275 | + # backing off should have no ill effect and help the |
1276 | + # service to recover. Client-configuration related errors |
1277 | + # (501, 505) are probably fine to throttle too as a higher |
1278 | + # rate won't help resolve them |
1279 | + self._backoff_counter.increase() |
1280 | + |
1281 | ssl_error = False |
1282 | if isinstance(error, PyCurlError) and error.error_code == 60: |
1283 | # The error returned is an SSL error, most likely the server |
1284 | @@ -642,6 +678,11 @@ class MessageExchange(object): |
1285 | interval = self._config.urgent_exchange_interval |
1286 | else: |
1287 | interval = self._config.exchange_interval |
1288 | + backoff_delay = self._backoff_counter.get_random_delay() |
1289 | + if backoff_delay: |
1290 | + logging.warning("Server is busy. Backing off client for {} " |
1291 | + "seconds".format(backoff_delay)) |
1292 | + interval += backoff_delay |
1293 | |
1294 | if self._notification_id is not None: |
1295 | self._reactor.cancel_call(self._notification_id) |
1296 | @@ -746,9 +787,9 @@ class MessageExchange(object): |
1297 | next_expected += len(payload["messages"]) |
1298 | |
1299 | message_store_state = got_next_expected(message_store, next_expected) |
1300 | - if message_store_state == ANCIENT: |
1301 | - # The server has probably lost some data we sent it. The |
1302 | - # slate has been wiped clean (by got_next_expected), now |
1303 | + if message_store_state == RESYNC: |
1304 | + # The server has probably lost some data we sent it. Or next |
1305 | + # expected is too high so the sequences are out of sync. Now |
1306 | # let's fire an event to tell all the plugins that they |
1307 | # ought to generate new messages so the server gets some |
1308 | # up-to-date data. |
1309 | @@ -800,8 +841,7 @@ class MessageExchange(object): |
1310 | # actually expect it to be a string. Some unit tests set it to |
1311 | # a regular string (since there is no difference between strings |
1312 | # and bytes in Python 2), so we check the type before converting. |
1313 | - if _PY3 and isinstance(message["type"], bytes): |
1314 | - message["type"] = message["type"].decode("ascii") |
1315 | + message["type"] = maybe_bytes(message["type"]) |
1316 | self.handle_message(message) |
1317 | sequence += 1 |
1318 | message_store.set_server_sequence(sequence) |
1319 | @@ -843,8 +883,9 @@ class MessageExchange(object): |
1320 | |
1321 | self._reactor.fire("message", message) |
1322 | # This has plan interference! but whatever. |
1323 | - if message["type"] in self._message_handlers: |
1324 | - for handler in self._message_handlers[message["type"]]: |
1325 | + message_type = maybe_bytes(message["type"]) |
1326 | + if message_type in self._message_handlers: |
1327 | + for handler in self._message_handlers[message_type]: |
1328 | handler(message) |
1329 | |
1330 | def register_client_accepted_message_type(self, type): |
1331 | @@ -866,3 +907,10 @@ def get_accepted_types_diff(old_types, new_types): |
1332 | diff.extend(["%s" % type for type in stable_types]) |
1333 | diff.extend(["-%s" % type for type in removed_types]) |
1334 | return " ".join(diff) |
1335 | + |
1336 | + |
1337 | +def maybe_bytes(thing): |
1338 | + """Return a py3 ascii string from maybe py2 bytes.""" |
1339 | + if _PY3 and isinstance(thing, bytes): |
1340 | + return thing.decode("ascii") |
1341 | + return thing |
1342 | diff --git a/landscape/client/broker/registration.py b/landscape/client/broker/registration.py |
1343 | index 2caef90..864ce99 100644 |
1344 | --- a/landscape/client/broker/registration.py |
1345 | +++ b/landscape/client/broker/registration.py |
1346 | @@ -8,10 +8,13 @@ the machinery in this module will notice that we have no identification |
1347 | credentials yet and that the server accepts registration messages, so it |
1348 | will craft an appropriate one and send it out. |
1349 | """ |
1350 | +import json |
1351 | import logging |
1352 | |
1353 | from twisted.internet.defer import Deferred |
1354 | |
1355 | +from landscape.client.broker.exchange import maybe_bytes |
1356 | +from landscape.client.monitor.ubuntuproinfo import get_ubuntu_pro_info |
1357 | from landscape.lib.juju import get_juju_info |
1358 | from landscape.lib.tag import is_valid_tag_list |
1359 | from landscape.lib.network import get_fqdn |
1360 | @@ -106,6 +109,7 @@ class RegistrationHandler(object): |
1361 | self._should_register = None |
1362 | self._fetch_async = fetch_async |
1363 | self._juju_data = None |
1364 | + self._clone_secure_id = None |
1365 | |
1366 | def should_register(self): |
1367 | id = self._identity |
1368 | @@ -191,6 +195,13 @@ class RegistrationHandler(object): |
1369 | "container-info": get_container_info(), |
1370 | "vm-info": get_vm_info()} |
1371 | |
1372 | + if self._clone_secure_id: |
1373 | + # We use the secure id here because the registration is encrypted |
1374 | + # and the insecure id has been already exposed to unencrypted |
1375 | + # http from the ping server. In addition it's more straightforward |
1376 | + # to get the computer from the server through it than the insecure |
1377 | + message["clone_secure_id"] = self._clone_secure_id |
1378 | + |
1379 | if group: |
1380 | message["access_group"] = group |
1381 | |
1382 | @@ -212,6 +223,8 @@ class RegistrationHandler(object): |
1383 | with_tags = "and tags %s " % tags if tags else "" |
1384 | with_group = "in access group '%s' " % group if group else "" |
1385 | |
1386 | + message["ubuntu_pro_info"] = json.dumps(get_ubuntu_pro_info()) |
1387 | + |
1388 | logging.info( |
1389 | u"Queueing message to register with account %r %s%s" |
1390 | "%s a password." % ( |
1391 | @@ -239,8 +252,9 @@ class RegistrationHandler(object): |
1392 | self._reactor.fire("resynchronize-clients") |
1393 | |
1394 | def _handle_registration(self, message): |
1395 | - if message["info"] in ("unknown-account", "max-pending-computers"): |
1396 | - self._reactor.fire("registration-failed", reason=message["info"]) |
1397 | + message_info = maybe_bytes(message["info"]) |
1398 | + if message_info in ("unknown-account", "max-pending-computers"): |
1399 | + self._reactor.fire("registration-failed", reason=message_info) |
1400 | |
1401 | def _handle_unknown_id(self, message): |
1402 | id = self._identity |
1403 | @@ -248,18 +262,9 @@ class RegistrationHandler(object): |
1404 | if clone is None: |
1405 | logging.info("Client has unknown secure-id for account %s." |
1406 | % id.account_name) |
1407 | - else: |
1408 | + else: # Save the secure id as the clone, and clear it so it's renewed |
1409 | logging.info("Client is clone of computer %s" % clone) |
1410 | - # Set a new computer title so when a registration request will be |
1411 | - # made, the pending computer UI will indicate that this is a clone |
1412 | - # of another computer. There's no need to persist the changes since |
1413 | - # a new registration will be requested immediately. |
1414 | - if clone == self._config.computer_title: |
1415 | - title = "%s (clone)" % self._config.computer_title |
1416 | - else: |
1417 | - title = "%s (clone of %s)" % (self._config.computer_title, |
1418 | - clone) |
1419 | - self._config.computer_title = title |
1420 | + self._clone_secure_id = id.secure_id |
1421 | id.secure_id = None |
1422 | id.insecure_id = None |
1423 | |
1424 | diff --git a/landscape/client/broker/server.py b/landscape/client/broker/server.py |
1425 | index 2805ea5..1f8d708 100644 |
1426 | --- a/landscape/client/broker/server.py |
1427 | +++ b/landscape/client/broker/server.py |
1428 | @@ -46,6 +46,7 @@ Diagram:: |
1429 | import logging |
1430 | |
1431 | from twisted.internet.defer import Deferred |
1432 | +from landscape.lib.compat import _PY3 |
1433 | |
1434 | from landscape.lib.twisted_util import gather_results |
1435 | from landscape.client.amp import remote |
1436 | @@ -191,6 +192,14 @@ class BrokerServer(object): |
1437 | during the next regularly scheduled exchange. |
1438 | @return: The message identifier created when queuing C{message}. |
1439 | """ |
1440 | + if b"type" in message and _PY3: |
1441 | + # XXX We are getting called by a python2 process. |
1442 | + # This occurs in the the specific case of a landscape-driven |
1443 | + # release upgrade to bionic, where the upgrading process is still |
1444 | + # running under python2.7. Therefore we do backward-compatible |
1445 | + # translation of byte keys in the sent message |
1446 | + message = {k.decode("ascii"): v for k, v in message.items()} |
1447 | + message["type"] = message["type"].decode("ascii") |
1448 | if isinstance(session_id, bool) and message["type"] in ( |
1449 | "operation-result", "change-packages-result"): |
1450 | # XXX This means we're performing a Landscape-driven upgrade and |
1451 | diff --git a/landscape/client/broker/service.py b/landscape/client/broker/service.py |
1452 | index 6e89414..7e36149 100644 |
1453 | --- a/landscape/client/broker/service.py |
1454 | +++ b/landscape/client/broker/service.py |
1455 | @@ -82,10 +82,11 @@ class BrokerService(LandscapeService): |
1456 | |
1457 | def stopService(self): |
1458 | """Stop the broker.""" |
1459 | - self.publisher.stop() |
1460 | + deferred = self.publisher.stop() |
1461 | self.exchanger.stop() |
1462 | self.pinger.stop() |
1463 | super(BrokerService, self).stopService() |
1464 | + return deferred |
1465 | |
1466 | |
1467 | def run(args): |
1468 | diff --git a/landscape/client/broker/tests/badprivate.ssl b/landscape/client/broker/tests/badprivate.ssl |
1469 | index db67ad4..d5e868c 100644 |
1470 | --- a/landscape/client/broker/tests/badprivate.ssl |
1471 | +++ b/landscape/client/broker/tests/badprivate.ssl |
1472 | @@ -1,15 +1,27 @@ |
1473 | -----BEGIN RSA PRIVATE KEY----- |
1474 | -MIICXgIBAAKBgQDGYFWP2Ine2OFIPjX+Tu+S403KW63EWq/I1DYXiezLoUpYPed3 |
1475 | -0tAkAXH1gOwQZbARFlUn0LgvXDSpuQLvgKQZwP/e1D8SvZUZ6nexW+aYlPE9kjd1 |
1476 | -dhK1xpe1h5y09AjCz02xxzcFzrJrJ47uU7vV+FGArE8FFh3hO+dz0/PmZQIDAQAB |
1477 | -AoGBAKfv+983yJfgcO9QwzLULlrilQNfk36r6y6QAG7y84T7uU10spSs4kno80mL |
1478 | -58yF2YTNrC91scdePrMEDikldUVcCqtPYcZKHyw5+4aGaDDO244tznexOQnQcNIe |
1479 | -2BbLFuh+jmJpoFIY/H7EsLQQzn6+6dGPnYGBQfiyitWfAXRNAkEA/ShQkYCRAHgq |
1480 | -g6WBIYsw/ISQydhiMiKrL2ZUXERT+pWU9MoSdMskgyMi3S7wzwJQXkrHA36q8QkL |
1481 | -+H8n5K+f5wJBAMiajfEtv0wRW0awX40qJtuqW3cSKeGHBH9mMObcRJd5OcK6giC/ |
1482 | -Cc5st/ZcuE/8i4r44DfeC+cwY6QdIqI8rdMCQQCKuq78LWJIyZEyt12+ThK4LsVR |
1483 | -d1zIcKsyvHb6YQ9MQPBx/NKEYlZN7tFKOFEKgBAevAe3aJCwqe5/bN8luQB9AkEA |
1484 | -uQVD8bR+AgzoIPS/zJWaLXSc09/e3PIJBfAdHnD+mq7mxWH8b3OD+e5wZjvyi2Ok |
1485 | -2NLfCug0FlGdNVrh/Lz2nQJATdcNvHNzJcWOHe05lo+xAqkjz73FWGpPNrdXRigG |
1486 | -YnjIsZVy4k48xIxPhT2rC44yo1iPEP5EnHCE2bLyUlTAYA== |
1487 | +MIIEowIBAAKCAQEA1f55iafJwi9EEL1jp8lX89SFVsordyhAJzQP+rRElFIcLSXU |
1488 | +2qC4whUErO3h3nKJXkIx4bHaQSn+/UKawr/kSVJI2QT3vimI/JTD5srxZJUrWEgc |
1489 | +slQNihx27jsRfpVeFK2BV2UpBYg65D6UhWyxNmoenBszBRRi11QnOF2VVX0BQLjO |
1490 | +lOP5mysSnq+VGEIEhViVICBKVb3dYle/LTh9uBH4oBBl4RvmzRZ/WUXqVLao8ryr |
1491 | +hciYDmCEpJH6tE4b/aEQNxou7nZqPJ91fkKZh4SmlhzXMFsjpQALZNYAxYZY6Kr4 |
1492 | +9Z+Xw8ckf3qqYfi8KovjCec7gewhCdZNpBxxOQIDAQABAoIBAHvT5DpOmEZAmY9i |
1493 | +OB9oN/fFO18sX5h09yJ4UuLMm36EQP+zC4dzR1YvWWRDxta0yl57yWeDRfs9NOsS |
1494 | +NoGJDq2K6tKBuGYWnMkjwHR1bNe6JbnRCKH8V1VbAUr7bTUlc6pdeCG9TM6BtSpM |
1495 | +OB849Ra6s3m7l3tR/5wAey13oak0PHM19c2Iv3Sniydt9toWyvuVbXWzOuupKMah |
1496 | ++eFkD0wpEgigL7gS4ToYXMqWn2LNHYsUSXKyuhtR0B/Ow6fwwb2KYxG5a5/hFI+E |
1497 | +URxFS148R/BtywUmxbzycTvZbx8XY7/lZjAlwYRCsYEF2EmmRbRGrHWOIuoHBMJu |
1498 | +49OMFu0CgYEA9P/EgmPeuIhrk3WFKnUTvHHTiW/Knv5xWRpkV+pXyGh4l+1F9lrj |
1499 | +WxORh+ws/6IpZevVVkUQxsd/Q4Uda7LlocS/YNSFC+LV471Y854+Wonbun8JBd+9 |
1500 | +GAFtG6o2XOaezp8zf8oDVqnfIx2qpySll3aNY7JrNZngtaE2bKz3sRMCgYEA35pO |
1501 | +47eTLxgIoFwr/M5cCT1nZ2FZUTBET9grrBgE5KTjtNbFAYJh7mP1lhILI58q7Lan |
1502 | +k9m5YoqAGsoKrilA715ks6vBQYWhlsg6p+rCvBTIYp4kxZwhTad5fFqXyhokwUZY |
1503 | +0ZJMtxYxK0xIZIEkuyzdzLiuYrKupFAB+3p+6gMCgYARQoeMjA6fv3ScsdXM1Oys |
1504 | +BPTbJNYId3JyzYouK2M9yiZcxal9HpAP1YQWKExPQhRais+/wSPabSmJDzKwaK0G |
1505 | +xX6aCr7IxJU+8xL2LrrD1Bx3ugVftZBzxX3zSf2Ec/bSJaMSKKAtldATgD6Kgels |
1506 | +jzyMvoARCaMsCIx2AYV9owKBgQCdd6UA9wHfE3TXwbF0mrr0Ats0Ubk91Nj2xcyT |
1507 | +qGKhxoFZlDovAuwGnzyPT+uqTWhERamkFJtaiyEGPKzi08iYCgivA1DY3MvcTOwJ |
1508 | +3uj+3T/1O1u4EmjdsAh9C6uDt3+U4P6hr/74nNdJn7IHnW8JpeIZTyH3/c/BhVqw |
1509 | +CCcikwKBgD7Kbl1AaK+6/+oDYunLOzxfDXkxkiXRDeBpvgOIW1G1F+bNJyO800oW |
1510 | +eZDtGU01Ze8gjNUbbYX39q4rCixtBHuJEwAH8J3grvtaHYpwjfuYgnhaXM1MHJo2 |
1511 | +kxfrQTDg/v3NjaccxCESeV1jjxz5hS2GrvqOvZrwyVtgi/r71cLJ |
1512 | -----END RSA PRIVATE KEY----- |
1513 | diff --git a/landscape/client/broker/tests/badpublic.ssl b/landscape/client/broker/tests/badpublic.ssl |
1514 | index 0a9b87d..9451fcd 100644 |
1515 | --- a/landscape/client/broker/tests/badpublic.ssl |
1516 | +++ b/landscape/client/broker/tests/badpublic.ssl |
1517 | @@ -1,5 +1,5 @@ |
1518 | -----BEGIN CERTIFICATE----- |
1519 | -MIIDzjCCAzegAwIBAgIJANqT3vXxSVFjMA0GCSqGSIb3DQEBBQUAMIGhMQswCQYD |
1520 | +MIIE0zCCA7ugAwIBAgIJANqT3vXxSVFjMA0GCSqGSIb3DQEBCwUAMIGhMQswCQYD |
1521 | VQQGEwJCUjEPMA0GA1UECBMGUGFyYW5hMREwDwYDVQQHEwhDdXJpdGliYTEhMB8G |
1522 | A1UEChMYRmFrZSBMYW5kc2NhcGUgKFRlc3RpbmcpMREwDwYDVQQLEwhTZWN1cml0 |
1523 | eTESMBAGA1UEAxMJbG9jYWxob3N0MSQwIgYJKoZIhvcNAQkBFhVhbmRyZWFzQGNh |
1524 | @@ -7,17 +7,22 @@ bm9uaWNhbC5jb20wHhcNMDkwMTA5MTUyNTAwWhcNMTkwMTA3MTUyNTAwWjCBoTEL |
1525 | MAkGA1UEBhMCQlIxDzANBgNVBAgTBlBhcmFuYTERMA8GA1UEBxMIQ3VyaXRpYmEx |
1526 | ITAfBgNVBAoTGEZha2UgTGFuZHNjYXBlIChUZXN0aW5nKTERMA8GA1UECxMIU2Vj |
1527 | dXJpdHkxEjAQBgNVBAMTCWxvY2FsaG9zdDEkMCIGCSqGSIb3DQEJARYVYW5kcmVh |
1528 | -c0BjYW5vbmljYWwuY29tMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDGYFWP |
1529 | -2Ine2OFIPjX+Tu+S403KW63EWq/I1DYXiezLoUpYPed30tAkAXH1gOwQZbARFlUn |
1530 | -0LgvXDSpuQLvgKQZwP/e1D8SvZUZ6nexW+aYlPE9kjd1dhK1xpe1h5y09AjCz02x |
1531 | -xzcFzrJrJ47uU7vV+FGArE8FFh3hO+dz0/PmZQIDAQABo4IBCjCCAQYwHQYDVR0O |
1532 | -BBYEFF4A8+YHCLAt19OtWTjIjBKzLUokMIHWBgNVHSMEgc4wgcuAFF4A8+YHCLAt |
1533 | -19OtWTjIjBKzLUokoYGnpIGkMIGhMQswCQYDVQQGEwJCUjEPMA0GA1UECBMGUGFy |
1534 | -YW5hMREwDwYDVQQHEwhDdXJpdGliYTEhMB8GA1UEChMYRmFrZSBMYW5kc2NhcGUg |
1535 | -KFRlc3RpbmcpMREwDwYDVQQLEwhTZWN1cml0eTESMBAGA1UEAxMJbG9jYWxob3N0 |
1536 | -MSQwIgYJKoZIhvcNAQkBFhVhbmRyZWFzQGNhbm9uaWNhbC5jb22CCQDak9718UlR |
1537 | -YzAMBgNVHRMEBTADAQH/MA0GCSqGSIb3DQEBBQUAA4GBABszkA3CCzt+nTOX+A7/ |
1538 | -I98DvI0W1Ss0J+Tq+diLr+kw6Z5ZTj5hrIS/x6XhVHjpim4724UBXA0Sels4JXbw |
1539 | -hhJovuncExce316gAol/9eEzTffZ9mt1jZQy9LL7IAENiobnsj2F65zNaJzXp5UC |
1540 | -rE/h/xIxz9rAmXtVOWHqZLcw |
1541 | +c0BjYW5vbmljYWwuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA |
1542 | +1f55iafJwi9EEL1jp8lX89SFVsordyhAJzQP+rRElFIcLSXU2qC4whUErO3h3nKJ |
1543 | +XkIx4bHaQSn+/UKawr/kSVJI2QT3vimI/JTD5srxZJUrWEgcslQNihx27jsRfpVe |
1544 | +FK2BV2UpBYg65D6UhWyxNmoenBszBRRi11QnOF2VVX0BQLjOlOP5mysSnq+VGEIE |
1545 | +hViVICBKVb3dYle/LTh9uBH4oBBl4RvmzRZ/WUXqVLao8ryrhciYDmCEpJH6tE4b |
1546 | +/aEQNxou7nZqPJ91fkKZh4SmlhzXMFsjpQALZNYAxYZY6Kr49Z+Xw8ckf3qqYfi8 |
1547 | +KovjCec7gewhCdZNpBxxOQIDAQABo4IBCjCCAQYwHQYDVR0OBBYEFF4A8+YHCLAt |
1548 | +19OtWTjIjBKzLUokMIHWBgNVHSMEgc4wgcuAFF4A8+YHCLAt19OtWTjIjBKzLUok |
1549 | +oYGnpIGkMIGhMQswCQYDVQQGEwJCUjEPMA0GA1UECBMGUGFyYW5hMREwDwYDVQQH |
1550 | +EwhDdXJpdGliYTEhMB8GA1UEChMYRmFrZSBMYW5kc2NhcGUgKFRlc3RpbmcpMREw |
1551 | +DwYDVQQLEwhTZWN1cml0eTESMBAGA1UEAxMJbG9jYWxob3N0MSQwIgYJKoZIhvcN |
1552 | +AQkBFhVhbmRyZWFzQGNhbm9uaWNhbC5jb22CCQDak9718UlRYzAMBgNVHRMEBTAD |
1553 | +AQH/MA0GCSqGSIb3DQEBCwUAA4IBAQBGwsH3kzRp8hQMlhNBFjzGCfycS8VT/AIr |
1554 | +VIORXzitEEovBqnPM5VT48sx7y7L7e9bmtBU9jv2aqk/PtRgTIl7tFcI5XeVzlsF |
1555 | +HB3A5Y7ytJqp80bIZvKt8gEq2qaBmyIK9VIHBO56Yxb7lQhkWWZmVBeCBbOut6HI |
1556 | +oBC8IX1hck/K3DjQh8gZnuJz7Ut1Pvb7uIGkfHWb6fGuYDcpZXj3uhRAKzRIZgdE |
1557 | +xs28Om1dKO6CDeTVkT2OSPfj21EJ1hNXrPQfXlInbyjYgR72gvqN+kbj9sc1dUr5 |
1558 | +l0McTP+hrRFpRN/rUrqtOWtHFW+INXj/HAuK+i4MlwkQXgPOTmoy |
1559 | -----END CERTIFICATE----- |
1560 | diff --git a/landscape/client/broker/tests/helpers.py b/landscape/client/broker/tests/helpers.py |
1561 | index fc01020..367a03f 100644 |
1562 | --- a/landscape/client/broker/tests/helpers.py |
1563 | +++ b/landscape/client/broker/tests/helpers.py |
1564 | @@ -39,14 +39,16 @@ class BrokerConfigurationHelper(object): |
1565 | log_dir = test_case.makeDir() |
1566 | test_case.config_filename = os.path.join(test_case.makeDir(), |
1567 | "client.conf") |
1568 | - open(test_case.config_filename, "w").write( |
1569 | - "[client]\n" |
1570 | - "url = http://localhost:91919\n" |
1571 | - "computer_title = Some Computer\n" |
1572 | - "account_name = some_account\n" |
1573 | - "ping_url = http://localhost:91910\n" |
1574 | - "data_path = %s\n" |
1575 | - "log_dir = %s\n" % (data_path, log_dir)) |
1576 | + |
1577 | + with open(test_case.config_filename, "w") as fh: |
1578 | + fh.write( |
1579 | + "[client]\n" |
1580 | + "url = http://localhost:91919\n" |
1581 | + "computer_title = Some Computer\n" |
1582 | + "account_name = some_account\n" |
1583 | + "ping_url = http://localhost:91910\n" |
1584 | + "data_path = %s\n" |
1585 | + "log_dir = %s\n" % (data_path, log_dir)) |
1586 | |
1587 | bootstrap_list.bootstrap(data_path=data_path, log_dir=log_dir) |
1588 | |
1589 | diff --git a/landscape/client/broker/tests/private.ssl b/landscape/client/broker/tests/private.ssl |
1590 | index 1484432..c103d59 100644 |
1591 | --- a/landscape/client/broker/tests/private.ssl |
1592 | +++ b/landscape/client/broker/tests/private.ssl |
1593 | @@ -1,15 +1,27 @@ |
1594 | -----BEGIN RSA PRIVATE KEY----- |
1595 | -MIICWwIBAAKBgQDX2VNEDZHtl5nimNocshar8pBmjqiGn9olCR2LcKifuJY4bFTg |
1596 | -qib+Rr3v2DwDTbOMaquRSxFgwLJLCug3WclsGrYSPIsFCx+k3XhqM61JXEwrKuIp |
1597 | -Js893XHkeg3SEFua/oVfDxNfJttoHW3FbsnDx5964kYwGExjJcH73GInUQIDAQAB |
1598 | -AoGASiM9NEys6Lx/gJMbp2uL2fdwnak2PTc+iCX/XduOL34JKswawyfuSLwnlO/i |
1599 | -fQf9OaeR0k/EYkUNeDUA2bIfOj6wWS8tamnX4fxL7A20y5VyqMMah8mcerZgtPdS |
1600 | -7ZtYCbeijWSKpHgjALc2Hym7R68WZI+IHe0DQkcW6WxOMFkCQQD2jqHZn/Qtd62u |
1601 | -mWVwIx6G7+Po5vzd86KyWWftdUtVCY9DmiX1rmWXbJhLnmaKCLkmHxyBvw7Biarr |
1602 | -ZnCAafebAkEA4B2dSpLi7bAzjCa7JBtzV9kr1FVZOl2vA+9BqTAjCQu0b9VDEm8V |
1603 | -x0061Z8rN7Og3ECGtKH/r3/4RnHUPpwJgwJAdyZQkvHYt4xJc8IPolRmcUFGu4u9 |
1604 | -Eammq1fHgJqZcBvxjvLUe1jvIXFKW+jNltFGYGTSiuUAxYi4/49+uJ/9FwJAGBB1 |
1605 | -/DTrcvQxhMH/5C+iYfNqtmD3tMGscjK1jTIjAOyl0kBG9GrDHuRXBesSW+fIxP2U |
1606 | -uT6P0std4EqGrLZaewJAHT0n/3tXnsPjj+BMlC4ZkRKgPJ4I7zTU1XSlLY5zbMoV |
1607 | -NvtHLlq7ttiarsH95xyge69uV1/zJVj/IiS71YY9PQ== |
1608 | +MIIEogIBAAKCAQEAw1Cf8FvbgP/FBGyCPXrTw23qa1KqmDkEu3q/yqmVpBlHh93K |
1609 | +UZu0T0WT7UJykEwOJqY+5G3Xaw53gWx7Lvz5nsI+X8Womd9NSqUkCiVq/TxeuWVm |
1610 | +07+pohQi8OJMmdfflx2eDX+3xciHdB4+A7baefyGut1vGr1wTYA3pu3/agqNIcAo |
1611 | +2F7LE13vnxgFzXCW0StdQeeT8KapOSJTZ8vxxkEKmXTixN7rpACP6GpD4jLTHZ/L |
1612 | +YUxjfHJdRGBF6TH/4NaA5CnS2yJpGidBmeK8Jid/S54P7HXrcJTrC0xwdYFfUTEI |
1613 | +25mBSqbEfGNqKhkYeJT/Pvy5TGdqAVSr4OjpEwIDAQABAoIBABw/4hI6xwHefJmK |
1614 | +NEA+Lrjagghp2YDQ5m1TcMAYTSuB+IWfP68UDT1V+/JaJQXX6kgOzZPuizTRz9kp |
1615 | +XpvKPTSINctWZG91C9HbFt5c0R+1hqHcF8ZSt29Y6EDdCmVKAu3xe7XKHkN+IJFb |
1616 | ++m5BGVKBgt8uPe6pLcAX5nS/gazNfyQ9s9zypMkCVSFoCOI+KkzqBCYPR/dqJUxP |
1617 | +hnxDt78ndUJL9QIDIGu7aCr9UwQCwSiRTB3+fDt1iPk1RWl+zmqvHUscPdyY1oMe |
1618 | +dvPNbF7Ea1nGjxzVG6+vyRHC1dQYSaqU8Ri26p0MBXGHCl8mq8tqV5ra30Q87ynV |
1619 | +wNu2TvkCgYEA6UCUBoetPoT3ncp/QXOs2obPQJ9W7VRMIFpkiZNGs8Zivp06CY3e |
1620 | +z7FqimDROKsFaYgDwPk5sOpzotnTA1/LsuoHK49Nux7Zn9RQak++6NrSozxvJ1P/ |
1621 | +ArpKkoTZNBbMPDghmib9gKUENjG1+6CkO5Z4tr0l9NyOBLz5j7oW2o8CgYEA1lzl |
1622 | +X2olxIY2nxPlmCHwjXVD90FT9xmcAe4xtU96aXv8v+f+6V2rwSBr8oGI8oyNjZN9 |
1623 | +nLkT9fRVhGi1bf9VZYb8fWB8PtGumZRZcfqYDyMD/0xbT6vEwaNtPtqNf24t8jf3 |
1624 | +ijU+Yx2jIwE7cAtOKcUg0eMGRPzSF8NRgPTaWz0CgYBwXE1yP9VysnbdqfhXPTPd |
1625 | +KOeZh6hGNz9crm6T30BFxaE3lWGpzI+ymRJrimv+0lOPHJhCU0w5Lxd5MVj23SSx |
1626 | +EQ9XKncVVq0a0xnRvIyIezDQtYIN/eZwF/FoV1qSPxEvSRLWwUWIvPUkbhnuFtpG |
1627 | +YhvQW5l3NO+s1KObWtc7fQKBgG4dIBJIU4hFLU/AB8ODQ69WmogrfbdD53iyY8Rw |
1628 | +REBlWWs3ACHeZTj6r5jN44w8mQYtymu0QsWoMjmnE/OiIrrZgV/iLVCTo23u35eG |
1629 | +E5BK+2WsUod1g8e4bIjJ+b+I2H9BMp5DRX3inod/vYmLtSYNxhMq3HCZsk5Uncxx |
1630 | +eq09AoGAfAX7tLtZDxpvyzEEoNdtt3zf9OUYDb74vH0CLAwahDE6ZdK/BVoc+JJB |
1631 | +Cm74/54z1nj7ubLNgMFuJbcgX/xthnuHcPSNH9FnPP//twYiuK1Zy67Q8aqFFxj4 |
1632 | +41H4idM4yLHJX1wjJDbufGHS/CjGg10Iy+gfFzqKqhkRhrdD3aI= |
1633 | -----END RSA PRIVATE KEY----- |
1634 | diff --git a/landscape/client/broker/tests/public.ssl b/landscape/client/broker/tests/public.ssl |
1635 | index 432b24d..ebd1682 100644 |
1636 | --- a/landscape/client/broker/tests/public.ssl |
1637 | +++ b/landscape/client/broker/tests/public.ssl |
1638 | @@ -1,22 +1,27 @@ |
1639 | -----BEGIN CERTIFICATE----- |
1640 | -MIIDnDCCAwWgAwIBAgIJAMjc7CvbvQHcMA0GCSqGSIb3DQEBCwUAMIGRMQswCQYD |
1641 | +MIIEoTCCA4mgAwIBAgIJAMjc7CvbvQHcMA0GCSqGSIb3DQEBCwUAMIGRMQswCQYD |
1642 | VQQGEwJCUjEPMA0GA1UECBMGUGFyYW5hMREwDwYDVQQHEwhDdXJpdGliYTESMBAG |
1643 | A1UEChMJTGFuZHNjYXBlMRAwDgYDVQQLEwdUZXN0aW5nMRIwEAYDVQQDEwlsb2Nh |
1644 | bGhvc3QxJDAiBgkqhkiG9w0BCQEWFWFuZHJlYXNAY2Fub25pY2FsLmNvbTAeFw0x |
1645 | OTAxMzEyMjI4NTJaFw0yOTAxMjgyMjI4NTJaMIGRMQswCQYDVQQGEwJCUjEPMA0G |
1646 | A1UECBMGUGFyYW5hMREwDwYDVQQHEwhDdXJpdGliYTESMBAGA1UEChMJTGFuZHNj |
1647 | YXBlMRAwDgYDVQQLEwdUZXN0aW5nMRIwEAYDVQQDEwlsb2NhbGhvc3QxJDAiBgkq |
1648 | -hkiG9w0BCQEWFWFuZHJlYXNAY2Fub25pY2FsLmNvbTCBnzANBgkqhkiG9w0BAQEF |
1649 | -AAOBjQAwgYkCgYEA19lTRA2R7ZeZ4pjaHLIWq/KQZo6ohp/aJQkdi3Con7iWOGxU |
1650 | -4Kom/ka979g8A02zjGqrkUsRYMCySwroN1nJbBq2EjyLBQsfpN14ajOtSVxMKyri |
1651 | -KSbPPd1x5HoN0hBbmv6FXw8TXybbaB1txW7Jw8efeuJGMBhMYyXB+9xiJ1ECAwEA |
1652 | -AaOB+TCB9jAdBgNVHQ4EFgQU3eUz2XxK1J/oavkn/hAvYfGOZM0wgcYGA1UdIwSB |
1653 | -vjCBu4AU3eUz2XxK1J/oavkn/hAvYfGOZM2hgZekgZQwgZExCzAJBgNVBAYTAkJS |
1654 | -MQ8wDQYDVQQIEwZQYXJhbmExETAPBgNVBAcTCEN1cml0aWJhMRIwEAYDVQQKEwlM |
1655 | -YW5kc2NhcGUxEDAOBgNVBAsTB1Rlc3RpbmcxEjAQBgNVBAMTCWxvY2FsaG9zdDEk |
1656 | -MCIGCSqGSIb3DQEJARYVYW5kcmVhc0BjYW5vbmljYWwuY29tggkAyNzsK9u9Adww |
1657 | -DAYDVR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOBgQCUq4lOtq19Q/NQmSdSQvpB |
1658 | -GWj3NKkpsH6sDlzjfGTVDqbL6buSUq4fTmXfrx5ce/Y+APhfAINAwIL/zjCSV3zI |
1659 | -EdhMG3BqMBmrQ60YfN7Z3drqfFzlg2yEVd/nJwrppjAI58KJgamN12WZ0eQTV8FR |
1660 | -co8+OnqJEOPM8cMg7VEbtQ== |
1661 | +hkiG9w0BCQEWFWFuZHJlYXNAY2Fub25pY2FsLmNvbTCCASIwDQYJKoZIhvcNAQEB |
1662 | +BQADggEPADCCAQoCggEBAMNQn/Bb24D/xQRsgj1608Nt6mtSqpg5BLt6v8qplaQZ |
1663 | +R4fdylGbtE9Fk+1CcpBMDiamPuRt12sOd4Fsey78+Z7CPl/FqJnfTUqlJAolav08 |
1664 | +XrllZtO/qaIUIvDiTJnX35cdng1/t8XIh3QePgO22nn8hrrdbxq9cE2AN6bt/2oK |
1665 | +jSHAKNheyxNd758YBc1wltErXUHnk/CmqTkiU2fL8cZBCpl04sTe66QAj+hqQ+Iy |
1666 | +0x2fy2FMY3xyXURgRekx/+DWgOQp0tsiaRonQZnivCYnf0ueD+x163CU6wtMcHWB |
1667 | +X1ExCNuZgUqmxHxjaioZGHiU/z78uUxnagFUq+Do6RMCAwEAAaOB+TCB9jAdBgNV |
1668 | +HQ4EFgQU3eUz2XxK1J/oavkn/hAvYfGOZM0wgcYGA1UdIwSBvjCBu4AU3eUz2XxK |
1669 | +1J/oavkn/hAvYfGOZM2hgZekgZQwgZExCzAJBgNVBAYTAkJSMQ8wDQYDVQQIEwZQ |
1670 | +YXJhbmExETAPBgNVBAcTCEN1cml0aWJhMRIwEAYDVQQKEwlMYW5kc2NhcGUxEDAO |
1671 | +BgNVBAsTB1Rlc3RpbmcxEjAQBgNVBAMTCWxvY2FsaG9zdDEkMCIGCSqGSIb3DQEJ |
1672 | +ARYVYW5kcmVhc0BjYW5vbmljYWwuY29tggkAyNzsK9u9AdwwDAYDVR0TBAUwAwEB |
1673 | +/zANBgkqhkiG9w0BAQsFAAOCAQEAnHA/iAGqfjEVlB5PJbYkKNasaflJJnT34BUZ |
1674 | +QdArozYCbC3dsysu+BkKEQ74FbDb/ZV0QhtV2/qYRlPsauWIiWU3gYk9wVjJdaBk |
1675 | ++MEViKOLV8CYe7ZA1OFNRQjfWUAiuRJdSsxnvOHcqcxrJT/hFmFz1OWyqcwfHBZY |
1676 | +4+8A7Byuy7x0O3qOWT2gaXpFL9neqtla1nEa1egJTZLrMF8Pi3vIpFthEgQ+yXoW |
1677 | +kBdWiYuMIry7YtAqBb4WgursCRftOCkPRYh3MVryqCDsVtOfGW2RO+YZnQgUpeTX |
1678 | +a6kHrB1hh+p8Az9pPWQVPb4w/jn4L2kEjTNpUxYWVB8Sc8GtGA== |
1679 | -----END CERTIFICATE----- |
1680 | diff --git a/landscape/client/broker/tests/test_exchange.py b/landscape/client/broker/tests/test_exchange.py |
1681 | index 20ee434..3736bd4 100644 |
1682 | --- a/landscape/client/broker/tests/test_exchange.py |
1683 | +++ b/landscape/client/broker/tests/test_exchange.py |
1684 | @@ -145,6 +145,50 @@ class MessageExchangeTest(LandscapeTest): |
1685 | self.mstore.add_pending_offset(1) |
1686 | self.assertFalse(self.mstore.is_pending(message_id)) |
1687 | |
1688 | + def test_send_big_message_trimmed_err(self): |
1689 | + """ |
1690 | + When package reporter sends error, message is trimmed if too long |
1691 | + """ |
1692 | + self.mstore.set_accepted_types(["package-reporter-result"]) |
1693 | + self.exchanger._max_log_text_bytes = 5 |
1694 | + self.exchanger.send({"type": "package-reporter-result", "err": "E"*10, |
1695 | + "code": 0}) |
1696 | + self.exchanger.exchange() |
1697 | + self.assertEqual(len(self.transport.payloads), 1) |
1698 | + messages = self.transport.payloads[0]["messages"] |
1699 | + self.assertIn('TRUNCATED', messages[0]['err']) |
1700 | + self.assertIn('EEEEE', messages[0]['err']) |
1701 | + self.assertNotIn('EEEEEE', messages[0]['err']) |
1702 | + |
1703 | + def test_send_big_message_trimmed_result(self): |
1704 | + """ |
1705 | + When an activity sends result log, message is trimmed if too long |
1706 | + """ |
1707 | + self.mstore.set_accepted_types(["operation-result"]) |
1708 | + self.exchanger._max_log_text_bytes = 5 |
1709 | + self.exchanger.send({"type": "operation-result", "result-text": "E"*10, |
1710 | + "code": 0, "status": 0, "operation-id": 0}) |
1711 | + self.exchanger.exchange() |
1712 | + self.assertEqual(len(self.transport.payloads), 1) |
1713 | + messages = self.transport.payloads[0]["messages"] |
1714 | + self.assertIn('TRUNCATED', messages[0]['result-text']) |
1715 | + self.assertIn('EEEEE', messages[0]['result-text']) |
1716 | + self.assertNotIn('EEEEEE', messages[0]['result-text']) |
1717 | + |
1718 | + def test_send_small_message_not_trimmed(self): |
1719 | + """ |
1720 | + If message is below length, nothing should happen |
1721 | + """ |
1722 | + self.mstore.set_accepted_types(["package-reporter-result"]) |
1723 | + self.exchanger._max_log_text_bytes = 4 |
1724 | + self.exchanger.send({"type": "package-reporter-result", "err": "E"*4, |
1725 | + "code": 0}) |
1726 | + self.exchanger.exchange() |
1727 | + self.assertEqual(len(self.transport.payloads), 1) |
1728 | + messages = self.transport.payloads[0]["messages"] |
1729 | + self.assertNotIn('TRUNCATED', messages[0]['err']) |
1730 | + self.assertIn('EEEE', messages[0]['err']) |
1731 | + |
1732 | def test_wb_include_accepted_types(self): |
1733 | """ |
1734 | Every payload from the client needs to specify an ID which |
1735 | @@ -324,6 +368,88 @@ class MessageExchangeTest(LandscapeTest): |
1736 | self.assertEqual(payload["sequence"], 1) |
1737 | self.assertEqual(payload["next-expected-sequence"], 0) |
1738 | |
1739 | + @mock.patch("landscape.client.broker.store.MessageStore" |
1740 | + ".delete_old_messages") |
1741 | + def test_pending_offset_when_next_expected_too_high(self, |
1742 | + mock_rm_all_messages): |
1743 | + ''' |
1744 | + When next expected sequence received from server is too high, then the |
1745 | + pending offset should reset to zero. This will cause the client to |
1746 | + resend the pending messages. |
1747 | + ''' |
1748 | + |
1749 | + self.mstore.set_accepted_types(["data"]) |
1750 | + self.mstore.add({"type": "data", "data": 0}) |
1751 | + self.mstore.add({"type": "data", "data": 1}) |
1752 | + |
1753 | + self.exchanger.exchange() |
1754 | + |
1755 | + self.assertEqual(self.mstore.get_pending_offset(), 2) |
1756 | + |
1757 | + self.mstore.add({"type": "data", "data": 2}) |
1758 | + |
1759 | + self.transport.next_expected_sequence = 100 |
1760 | + |
1761 | + # Confirm pending offset is reset so that messages are sent again |
1762 | + self.exchanger.exchange() |
1763 | + self.assertEqual(self.mstore.get_pending_offset(), 0) |
1764 | + |
1765 | + # This function flushes the queue, otherwise an offset of 0 will resend |
1766 | + # previous messages that were already successful |
1767 | + self.assertTrue(mock_rm_all_messages.called) |
1768 | + |
1769 | + def test_payloads_when_next_expected_too_high(self): |
1770 | + ''' |
1771 | + When next expected sequence received from server is too high, then the |
1772 | + current messages should get sent again since we don't have confirmation |
1773 | + that the server received it. Also previous messages should not get |
1774 | + repeated. |
1775 | + ''' |
1776 | + |
1777 | + self.mstore.set_accepted_types(["data"]) |
1778 | + |
1779 | + message0 = {"type": "data", "data": 0} |
1780 | + self.mstore.add(message0) |
1781 | + self.exchanger.exchange() |
1782 | + |
1783 | + message1 = {"type": "data", "data": 1} |
1784 | + message2 = {"type": "data", "data": 2} |
1785 | + self.mstore.add(message1) |
1786 | + self.mstore.add(message2) |
1787 | + |
1788 | + self.transport.next_expected_sequence = 100 |
1789 | + self.exchanger.exchange() # Resync |
1790 | + self.exchanger.exchange() # Resend |
1791 | + |
1792 | + # Confirm messages is not empty which was the original bug |
1793 | + last_messages = self.transport.payloads[-1]["messages"] |
1794 | + self.assertTrue(last_messages) |
1795 | + |
1796 | + # Confirm earlier messages are not resent |
1797 | + self.assertNotIn(message0["data"], |
1798 | + [m["data"] for m in last_messages]) |
1799 | + |
1800 | + # Confirm contents of payload |
1801 | + self.assertEqual([message1, message2], last_messages) |
1802 | + |
1803 | + def test_resync_when_next_expected_too_high(self): |
1804 | + ''' |
1805 | + When next expected sequence received from the server is too high, then |
1806 | + a resynchronize should happen |
1807 | + ''' |
1808 | + |
1809 | + self.mstore.set_accepted_types(["empty", "resynchronize"]) |
1810 | + self.mstore.add({"type": "empty"}) |
1811 | + self.exchanger.exchange() |
1812 | + |
1813 | + self.transport.next_expected_sequence = 100 |
1814 | + |
1815 | + self.reactor.call_on("resynchronize-clients", lambda scope=None: None) |
1816 | + |
1817 | + self.exchanger.exchange() |
1818 | + self.assertMessage(self.mstore.get_pending_messages()[-1], |
1819 | + {"type": "resynchronize"}) |
1820 | + |
1821 | def test_start_with_urgent_exchange(self): |
1822 | """ |
1823 | Immediately after registration, an urgent exchange should be scheduled. |
1824 | @@ -1088,6 +1214,64 @@ class MessageExchangeTest(LandscapeTest): |
1825 | self.exchanger.exchange() |
1826 | self.assertEqual(b"3.2", self.mstore.get_server_api()) |
1827 | |
1828 | + def test_500_backoff(self): |
1829 | + """ |
1830 | + If we get a server error then the exponential backoff is triggered |
1831 | + """ |
1832 | + self.config.urgent_exchange_interval = 10 |
1833 | + self.exchanger._backoff_counter._start_delay = 300 |
1834 | + self.exchanger._backoff_counter._max_delay = 1000 |
1835 | + self.transport.responses.append(HTTPCodeError(503, "")) |
1836 | + self.exchanger.schedule_exchange(urgent=True) |
1837 | + self.reactor.advance(50) |
1838 | + self.assertEqual(len(self.transport.payloads), 1) |
1839 | + self.reactor.advance(400) |
1840 | + self.assertEqual(len(self.transport.payloads), 2) |
1841 | + |
1842 | + def test_429_backoff(self): |
1843 | + """ |
1844 | + HTTP error 429 should also trigger backoff |
1845 | + """ |
1846 | + self.config.urgent_exchange_interval = 10 |
1847 | + self.exchanger._backoff_counter._start_delay = 300 |
1848 | + self.exchanger._backoff_counter._max_delay = 1000 |
1849 | + self.transport.responses.append(HTTPCodeError(429, "")) |
1850 | + self.exchanger.schedule_exchange(urgent=True) |
1851 | + self.reactor.advance(50) |
1852 | + self.assertEqual(len(self.transport.payloads), 1) |
1853 | + |
1854 | + def test_backoff_reset_after_success(self): |
1855 | + """ |
1856 | + If we get a success after a 500 error then backoff should be zero |
1857 | + """ |
1858 | + self.config.urgent_exchange_interval = 10 |
1859 | + self.exchanger._backoff_counter._start_delay = 300 |
1860 | + self.exchanger._backoff_counter._max_delay = 1000 |
1861 | + self.transport.responses.append(HTTPCodeError(500, "")) |
1862 | + self.exchanger.schedule_exchange(urgent=True) |
1863 | + self.reactor.advance(50) |
1864 | + |
1865 | + # Confirm it's not zero after the error |
1866 | + self.assertTrue(self.exchanger._backoff_counter.get_random_delay()) |
1867 | + |
1868 | + server_message = [{"type": "type-R", "whatever": 5678}] |
1869 | + self.transport.responses.append(server_message) |
1870 | + self.exchanger.schedule_exchange(urgent=True) |
1871 | + self.reactor.advance(500) |
1872 | + |
1873 | + # Confirm it is zero after the success |
1874 | + self.assertFalse(self.exchanger._backoff_counter.get_random_delay()) |
1875 | + |
1876 | + def test_400_no_backoff(self): |
1877 | + """ |
1878 | + If we get a 400 error then the backoff should not be triggered |
1879 | + """ |
1880 | + self.config.urgent_exchange_interval = 10 |
1881 | + self.transport.responses.append(HTTPCodeError(400, "")) |
1882 | + self.exchanger.schedule_exchange(urgent=True) |
1883 | + self.reactor.advance(20) |
1884 | + self.assertEqual(len(self.transport.payloads), 2) |
1885 | + |
1886 | |
1887 | class AcceptedTypesMessageExchangeTest(LandscapeTest): |
1888 | |
1889 | diff --git a/landscape/client/broker/tests/test_registration.py b/landscape/client/broker/tests/test_registration.py |
1890 | index 8ae8574..8ca0cb4 100644 |
1891 | --- a/landscape/client/broker/tests/test_registration.py |
1892 | +++ b/landscape/client/broker/tests/test_registration.py |
1893 | @@ -3,7 +3,7 @@ import logging |
1894 | import socket |
1895 | import mock |
1896 | |
1897 | -from twisted.python.compat import _PY3 |
1898 | +from landscape.lib.compat import _PY3 |
1899 | |
1900 | from landscape.client.broker.registration import RegistrationError, Identity |
1901 | from landscape.client.tests.helpers import LandscapeTest |
1902 | @@ -90,7 +90,7 @@ class RegistrationHandlerTest(RegistrationHandlerTestBase): |
1903 | and insecure ids even if no requests were sent. |
1904 | """ |
1905 | self.exchanger.handle_message( |
1906 | - {"type": "set-id", "id": "abc", "insecure-id": "def"}) |
1907 | + {"type": b"set-id", "id": b"abc", "insecure-id": b"def"}) |
1908 | self.assertEqual(self.identity.secure_id, "abc") |
1909 | self.assertEqual(self.identity.insecure_id, "def") |
1910 | |
1911 | @@ -101,29 +101,58 @@ class RegistrationHandlerTest(RegistrationHandlerTestBase): |
1912 | """ |
1913 | reactor_fire_mock = self.reactor.fire = mock.Mock() |
1914 | self.exchanger.handle_message( |
1915 | - {"type": "set-id", "id": "abc", "insecure-id": "def"}) |
1916 | + {"type": b"set-id", "id": b"abc", "insecure-id": b"def"}) |
1917 | reactor_fire_mock.assert_any_call("registration-done") |
1918 | |
1919 | def test_unknown_id(self): |
1920 | self.identity.secure_id = "old_id" |
1921 | self.identity.insecure_id = "old_id" |
1922 | self.mstore.set_accepted_types(["register"]) |
1923 | - self.exchanger.handle_message({"type": "unknown-id"}) |
1924 | + self.exchanger.handle_message({"type": b"unknown-id"}) |
1925 | self.assertEqual(self.identity.secure_id, None) |
1926 | self.assertEqual(self.identity.insecure_id, None) |
1927 | |
1928 | def test_unknown_id_with_clone(self): |
1929 | """ |
1930 | If the server reports us that we are a clone of another computer, then |
1931 | - set our computer's title accordingly. |
1932 | + make sure we handle it |
1933 | """ |
1934 | self.config.computer_title = "Wu" |
1935 | self.mstore.set_accepted_types(["register"]) |
1936 | - self.exchanger.handle_message({"type": "unknown-id", "clone-of": "Wu"}) |
1937 | - self.assertEqual("Wu (clone)", self.config.computer_title) |
1938 | + self.exchanger.handle_message( |
1939 | + {"type": b"unknown-id", "clone-of": "Wu"}) |
1940 | self.assertIn("Client is clone of computer Wu", |
1941 | self.logfile.getvalue()) |
1942 | |
1943 | + def test_clone_secure_id_saved(self): |
1944 | + """ |
1945 | + Make sure that secure id is saved when theres a clone and existing |
1946 | + value is cleared out |
1947 | + """ |
1948 | + secure_id = "foo" |
1949 | + self.identity.secure_id = secure_id |
1950 | + self.config.computer_title = "Wu" |
1951 | + self.mstore.set_accepted_types(["register"]) |
1952 | + self.exchanger.handle_message( |
1953 | + {"type": b"unknown-id", "clone-of": "Wu"}) |
1954 | + self.assertEqual(self.handler._clone_secure_id, secure_id) |
1955 | + self.assertIsNone(self.identity.secure_id) |
1956 | + |
1957 | + def test_clone_id_in_message(self): |
1958 | + """ |
1959 | + Make sure that the clone id is present in the registration message |
1960 | + """ |
1961 | + secure_id = "foo" |
1962 | + self.identity.secure_id = secure_id |
1963 | + self.config.computer_title = "Wu" |
1964 | + self.mstore.set_accepted_types(["register"]) |
1965 | + self.mstore.set_server_api(b"3.3") # Note this is only for later api |
1966 | + self.exchanger.handle_message( |
1967 | + {"type": b"unknown-id", "clone-of": "Wu"}) |
1968 | + self.reactor.fire("pre-exchange") |
1969 | + messages = self.mstore.get_pending_messages() |
1970 | + self.assertEqual(messages[0]["clone_secure_id"], secure_id) |
1971 | + |
1972 | def test_should_register(self): |
1973 | self.mstore.set_accepted_types(["register"]) |
1974 | self.config.computer_title = "Computer Title" |
1975 | @@ -366,7 +395,7 @@ class RegistrationHandlerTest(RegistrationHandlerTestBase): |
1976 | """ |
1977 | reactor_fire_mock = self.reactor.fire = mock.Mock() |
1978 | self.exchanger.handle_message( |
1979 | - {"type": "registration", "info": "unknown-account"}) |
1980 | + {"type": b"registration", "info": b"unknown-account"}) |
1981 | reactor_fire_mock.assert_called_with( |
1982 | "registration-failed", reason="unknown-account") |
1983 | |
1984 | @@ -378,7 +407,7 @@ class RegistrationHandlerTest(RegistrationHandlerTestBase): |
1985 | """ |
1986 | reactor_fire_mock = self.reactor.fire = mock.Mock() |
1987 | self.exchanger.handle_message( |
1988 | - {"type": "registration", "info": "max-pending-computers"}) |
1989 | + {"type": b"registration", "info": b"max-pending-computers"}) |
1990 | reactor_fire_mock.assert_called_with( |
1991 | "registration-failed", reason="max-pending-computers") |
1992 | |
1993 | @@ -389,7 +418,7 @@ class RegistrationHandlerTest(RegistrationHandlerTestBase): |
1994 | """ |
1995 | reactor_fire_mock = self.reactor.fire = mock.Mock() |
1996 | self.exchanger.handle_message( |
1997 | - {"type": "registration", "info": "blah-blah"}) |
1998 | + {"type": b"registration", "info": b"blah-blah"}) |
1999 | for name, args, kwargs in reactor_fire_mock.mock_calls: |
2000 | self.assertNotEquals("registration-failed", args[0]) |
2001 | |
2002 | @@ -420,13 +449,13 @@ class RegistrationHandlerTest(RegistrationHandlerTestBase): |
2003 | |
2004 | # This should somehow callback the deferred. |
2005 | self.exchanger.handle_message( |
2006 | - {"type": "set-id", "id": "abc", "insecure-id": "def"}) |
2007 | + {"type": b"set-id", "id": b"abc", "insecure-id": b"def"}) |
2008 | |
2009 | self.assertEqual(calls, [1]) |
2010 | |
2011 | # Doing it again to ensure that the deferred isn't called twice. |
2012 | self.exchanger.handle_message( |
2013 | - {"type": "set-id", "id": "abc", "insecure-id": "def"}) |
2014 | + {"type": b"set-id", "id": b"abc", "insecure-id": b"def"}) |
2015 | |
2016 | self.assertEqual(calls, [1]) |
2017 | |
2018 | @@ -448,7 +477,7 @@ class RegistrationHandlerTest(RegistrationHandlerTestBase): |
2019 | |
2020 | # This should somehow callback the deferred. |
2021 | self.exchanger.handle_message( |
2022 | - {"type": "set-id", "id": "abc", "insecure-id": "def"}) |
2023 | + {"type": b"set-id", "id": b"abc", "insecure-id": b"def"}) |
2024 | |
2025 | self.assertEqual(results, [None]) |
2026 | |
2027 | @@ -473,13 +502,13 @@ class RegistrationHandlerTest(RegistrationHandlerTestBase): |
2028 | |
2029 | # This should somehow callback the deferred. |
2030 | self.exchanger.handle_message( |
2031 | - {"type": "registration", "info": "unknown-account"}) |
2032 | + {"type": b"registration", "info": b"unknown-account"}) |
2033 | |
2034 | self.assertEqual(calls, [True]) |
2035 | |
2036 | # Doing it again to ensure that the deferred isn't called twice. |
2037 | self.exchanger.handle_message( |
2038 | - {"type": "registration", "info": "unknown-account"}) |
2039 | + {"type": b"registration", "info": b"unknown-account"}) |
2040 | |
2041 | self.assertEqual(calls, [True]) |
2042 | |
2043 | @@ -505,13 +534,13 @@ class RegistrationHandlerTest(RegistrationHandlerTestBase): |
2044 | d.addErrback(add_call) |
2045 | |
2046 | self.exchanger.handle_message( |
2047 | - {"type": "registration", "info": "max-pending-computers"}) |
2048 | + {"type": b"registration", "info": b"max-pending-computers"}) |
2049 | |
2050 | self.assertEqual(calls, [True]) |
2051 | |
2052 | # Doing it again to ensure that the deferred isn't called twice. |
2053 | self.exchanger.handle_message( |
2054 | - {"type": "registration", "info": "max-pending-computers"}) |
2055 | + {"type": b"registration", "info": b"max-pending-computers"}) |
2056 | |
2057 | self.assertEqual(calls, [True]) |
2058 | |
2059 | diff --git a/landscape/client/broker/tests/test_server.py b/landscape/client/broker/tests/test_server.py |
2060 | index 5059e48..99fa65a 100644 |
2061 | --- a/landscape/client/broker/tests/test_server.py |
2062 | +++ b/landscape/client/broker/tests/test_server.py |
2063 | @@ -130,6 +130,25 @@ class BrokerServerTest(LandscapeTest): |
2064 | self.assertMessages(self.mstore.get_pending_messages(), [message]) |
2065 | self.assertTrue(self.exchanger.is_urgent()) |
2066 | |
2067 | + def test_send_message_from_py27_upgrader(self): |
2068 | + """ |
2069 | + If we receive release-upgrade results from a py27 release upgrader, |
2070 | + it gets translated to a py3-compatible message. |
2071 | + """ |
2072 | + legacy_message = { |
2073 | + b"type": b"change-packages-result", |
2074 | + b"operation-id": 99, |
2075 | + b"result-code": 123} |
2076 | + self.mstore.set_accepted_types(["change-packages-result"]) |
2077 | + self.broker.send_message(legacy_message, True) |
2078 | + expected = [{ |
2079 | + "type": "change-packages-result", |
2080 | + "operation-id": 99, |
2081 | + "result-code": 123 |
2082 | + }] |
2083 | + self.assertMessages(self.mstore.get_pending_messages(), expected) |
2084 | + self.assertTrue(self.exchanger.is_urgent()) |
2085 | + |
2086 | def test_is_pending(self): |
2087 | """ |
2088 | The L{BrokerServer.is_pending} method indicates if a message with |
2089 | diff --git a/landscape/client/broker/tests/test_store.py b/landscape/client/broker/tests/test_store.py |
2090 | index efa64d4..208c75c 100644 |
2091 | --- a/landscape/client/broker/tests/test_store.py |
2092 | +++ b/landscape/client/broker/tests/test_store.py |
2093 | @@ -149,7 +149,7 @@ class MessageStoreTest(LandscapeTest): |
2094 | for i in range(10): |
2095 | self.store.add(dict(type="data", data=intToBytes(i))) |
2096 | il = [m["data"] for m in self.store.get_pending_messages(5)] |
2097 | - self.assertEqual(il, [intToBytes(i) for i in[0, 1, 2, 3, 4]]) |
2098 | + self.assertEqual(il, [intToBytes(i) for i in [0, 1, 2, 3, 4]]) |
2099 | |
2100 | def test_offset(self): |
2101 | self.store.set_pending_offset(5) |
2102 | diff --git a/landscape/client/broker/tests/test_transport.py b/landscape/client/broker/tests/test_transport.py |
2103 | index 537c36d..87106a1 100644 |
2104 | --- a/landscape/client/broker/tests/test_transport.py |
2105 | +++ b/landscape/client/broker/tests/test_transport.py |
2106 | @@ -16,7 +16,7 @@ from twisted.internet.threads import deferToThread |
2107 | |
2108 | |
2109 | def sibpath(path): |
2110 | - return os.path.join(os.path.dirname(__file__), path) |
2111 | + return os.path.abspath(os.path.join(os.path.dirname(__file__), path)) |
2112 | |
2113 | |
2114 | PRIVKEY = sibpath("private.ssl") |
2115 | diff --git a/landscape/client/broker/transport.py b/landscape/client/broker/transport.py |
2116 | index ed080cd..27b70fb 100644 |
2117 | --- a/landscape/client/broker/transport.py |
2118 | +++ b/landscape/client/broker/transport.py |
2119 | @@ -6,7 +6,7 @@ import uuid |
2120 | |
2121 | import pycurl |
2122 | |
2123 | -from twisted.python.compat import unicode, _PY3 |
2124 | +from landscape.lib.compat import unicode, _PY3 |
2125 | |
2126 | from landscape.lib import bpickle |
2127 | from landscape.lib.fetch import fetch |
2128 | diff --git a/landscape/client/configuration.py b/landscape/client/configuration.py |
2129 | index e08a865..e06dd26 100644 |
2130 | --- a/landscape/client/configuration.py |
2131 | +++ b/landscape/client/configuration.py |
2132 | @@ -7,14 +7,15 @@ for the C{landscape-config} script. |
2133 | from __future__ import print_function |
2134 | |
2135 | from functools import partial |
2136 | -import base64 |
2137 | import getpass |
2138 | import io |
2139 | import os |
2140 | import pwd |
2141 | import sys |
2142 | +import textwrap |
2143 | |
2144 | from landscape.lib.compat import input |
2145 | +from landscape.lib import base64 |
2146 | |
2147 | from landscape.lib.tag import is_valid_tag |
2148 | |
2149 | @@ -33,6 +34,9 @@ from landscape.client.broker.registration import Identity |
2150 | from landscape.client.broker.service import BrokerService |
2151 | |
2152 | |
2153 | +EXIT_NOT_REGISTERED = 5 |
2154 | + |
2155 | + |
2156 | class ConfigurationError(Exception): |
2157 | """Raised when required configuration values are missing.""" |
2158 | |
2159 | @@ -192,6 +196,11 @@ class LandscapeSetupConfiguration(BrokerConfiguration): |
2160 | parser.add_option("--init", action="store_true", default=False, |
2161 | help="Set up the client directories structure " |
2162 | "and exit.") |
2163 | + parser.add_option("--is-registered", action="store_true", |
2164 | + help="Exit with code 0 (success) if client is " |
2165 | + "registered else returns {}. Displays " |
2166 | + "registration info." |
2167 | + .format(EXIT_NOT_REGISTERED)) |
2168 | return parser |
2169 | |
2170 | |
2171 | @@ -513,7 +522,7 @@ def decode_base64_ssl_public_certificate(config): |
2172 | # WARNING: ssl_public_certificate is misnamed, it's not the key of the |
2173 | # certificate, but the actual certificate itself. |
2174 | if config.ssl_public_key and config.ssl_public_key.startswith("base64:"): |
2175 | - decoded_cert = base64.decodestring( |
2176 | + decoded_cert = base64.decodebytes( |
2177 | config.ssl_public_key[7:].encode("ascii")) |
2178 | config.ssl_public_key = store_public_key_data( |
2179 | config, decoded_cert) |
2180 | @@ -759,6 +768,26 @@ def is_registered(config): |
2181 | return bool(identity.secure_id) |
2182 | |
2183 | |
2184 | +def registration_info_text(config, registration_status): |
2185 | + ''' |
2186 | + A simple output displaying whether the client is registered or not, the |
2187 | + account name, and config and data paths |
2188 | + ''' |
2189 | + |
2190 | + config_path = os.path.abspath(config._config_filename) |
2191 | + |
2192 | + text = textwrap.dedent(""" |
2193 | + Registered: {} |
2194 | + Config Path: {} |
2195 | + Data Path {}""" |
2196 | + .format(registration_status, config_path, |
2197 | + config.data_path)) |
2198 | + if registration_status: |
2199 | + text += '\nAccount Name: {}'.format(config.account_name) |
2200 | + |
2201 | + return text |
2202 | + |
2203 | + |
2204 | def main(args, print=print): |
2205 | """Interact with the user and the server to set up client configuration.""" |
2206 | |
2207 | @@ -769,6 +798,18 @@ def main(args, print=print): |
2208 | print_text(str(error), error=True) |
2209 | sys.exit(1) |
2210 | |
2211 | + if config.is_registered: |
2212 | + |
2213 | + registration_status = is_registered(config) |
2214 | + |
2215 | + info_text = registration_info_text(config, registration_status) |
2216 | + print(info_text) |
2217 | + |
2218 | + if registration_status: |
2219 | + sys.exit(0) |
2220 | + else: |
2221 | + sys.exit(EXIT_NOT_REGISTERED) |
2222 | + |
2223 | if os.getuid() != 0: |
2224 | sys.exit("landscape-config must be run as root.") |
2225 | |
2226 | diff --git a/landscape/client/lockfile.py b/landscape/client/lockfile.py |
2227 | new file mode 100644 |
2228 | index 0000000..aa0f916 |
2229 | --- /dev/null |
2230 | +++ b/landscape/client/lockfile.py |
2231 | @@ -0,0 +1,52 @@ |
2232 | +import os |
2233 | + |
2234 | +from twisted.python import lockfile |
2235 | + |
2236 | + |
2237 | +def patch_lockfile(): |
2238 | + if lockfile.FilesystemLock is PatchedFilesystemLock: |
2239 | + return |
2240 | + lockfile.FilesystemLock = PatchedFilesystemLock |
2241 | + |
2242 | + |
2243 | +class PatchedFilesystemLock(lockfile.FilesystemLock): |
2244 | + """ |
2245 | + Patched Twisted's FilesystemLock.lock to handle PermissionError |
2246 | + when trying to lock. |
2247 | + """ |
2248 | + |
2249 | + def lock(self): |
2250 | + # XXX Twisted assumes PIDs don't get reused, which is incorrect. |
2251 | + # As such, we pre-check that any existing lock file isn't |
2252 | + # associated to a live process, and that any associated |
2253 | + # process is from landscape. Otherwise, clean up the lock file, |
2254 | + # considering it to be locked to a recycled PID. |
2255 | + # |
2256 | + # Although looking for the process name may seem fragile, it's the |
2257 | + # most acurate info we have since: |
2258 | + # * some process run as root, so the UID is not a reference |
2259 | + # * process may not be spawned by systemd, so cgroups are not reliable |
2260 | + # * python executable is not a reference |
2261 | + clean = True |
2262 | + try: |
2263 | + pid = os.readlink(self.name) |
2264 | + ps_name = get_process_name(int(pid)) |
2265 | + if not ps_name.startswith("landscape"): |
2266 | + os.remove(self.name) |
2267 | + clean = False |
2268 | + except Exception: |
2269 | + # We can't figure the lock state, let FilesystemLock figure it |
2270 | + # out normally. |
2271 | + pass |
2272 | + |
2273 | + result = super(PatchedFilesystemLock, self).lock() |
2274 | + self.clean = self.clean and clean |
2275 | + return result |
2276 | + |
2277 | + |
2278 | +def get_process_name(pid): |
2279 | + """Return a process name from a pid.""" |
2280 | + stat_path = "/proc/{}/stat".format(pid) |
2281 | + with open(stat_path) as stat_file: |
2282 | + stat = stat_file.read() |
2283 | + return stat.partition("(")[2].rpartition(")")[0] |
2284 | diff --git a/landscape/client/manager/aptsources.py b/landscape/client/manager/aptsources.py |
2285 | index 649569b..866c6ba 100644 |
2286 | --- a/landscape/client/manager/aptsources.py |
2287 | +++ b/landscape/client/manager/aptsources.py |
2288 | @@ -4,6 +4,7 @@ import pwd |
2289 | import grp |
2290 | import shutil |
2291 | import tempfile |
2292 | +import uuid |
2293 | |
2294 | from twisted.internet.defer import succeed |
2295 | |
2296 | @@ -22,6 +23,7 @@ class AptSources(ManagerPlugin): |
2297 | |
2298 | SOURCES_LIST = "/etc/apt/sources.list" |
2299 | SOURCES_LIST_D = "/etc/apt/sources.list.d" |
2300 | + TRUSTED_GPG_D = "/etc/apt/trusted.gpg.d" |
2301 | |
2302 | def register(self, registry): |
2303 | super(AptSources, self).register(registry) |
2304 | @@ -83,16 +85,12 @@ class AptSources(ManagerPlugin): |
2305 | "-----END PGP PUBLIC KEY BLOCK-----"]} |
2306 | """ |
2307 | deferred = succeed(None) |
2308 | + prefix = 'landscape-server-' |
2309 | for key in message["gpg-keys"]: |
2310 | - fd, path = tempfile.mkstemp() |
2311 | - os.close(fd) |
2312 | - with open(path, "w") as key_file: |
2313 | + filename = prefix + str(uuid.uuid4()) + '.asc' |
2314 | + key_path = os.path.join(self.TRUSTED_GPG_D, filename) |
2315 | + with open(key_path, "w") as key_file: |
2316 | key_file.write(key) |
2317 | - deferred.addCallback( |
2318 | - lambda ignore, path=path: |
2319 | - self._run_process("/usr/bin/apt-key", ["add", path])) |
2320 | - deferred.addCallback(self._handle_process_error) |
2321 | - deferred.addBoth(self._remove_and_continue, path) |
2322 | deferred.addErrback(self._handle_process_failure) |
2323 | deferred.addCallback(self._handle_sources, message["sources"]) |
2324 | return self.call_with_operation_result(message, lambda: deferred) |
2325 | diff --git a/landscape/client/manager/config.py b/landscape/client/manager/config.py |
2326 | index 1461767..bfc0a5d 100644 |
2327 | --- a/landscape/client/manager/config.py |
2328 | +++ b/landscape/client/manager/config.py |
2329 | @@ -30,6 +30,14 @@ class ManagerConfiguration(Configuration): |
2330 | help="Comma-delimited list of usernames that scripts" |
2331 | " may be run as. Default is to allow all " |
2332 | "users.") |
2333 | + parser.add_option("--script-output-limit", |
2334 | + metavar="SCRIPT_OUTPUT_LIMIT", |
2335 | + type="int", default=512, |
2336 | + help="Maximum allowed output size that scripts" |
2337 | + " can send. " |
2338 | + "Script output will be truncated at that limit." |
2339 | + " Default is 512 (kB)") |
2340 | + |
2341 | return parser |
2342 | |
2343 | @property |
2344 | diff --git a/landscape/client/manager/keystonetoken.py b/landscape/client/manager/keystonetoken.py |
2345 | index f79d4bc..687f5c7 100644 |
2346 | --- a/landscape/client/manager/keystonetoken.py |
2347 | +++ b/landscape/client/manager/keystonetoken.py |
2348 | @@ -1,7 +1,7 @@ |
2349 | import os |
2350 | import logging |
2351 | |
2352 | -from twisted.python.compat import _PY3 |
2353 | +from landscape.lib.compat import _PY3 |
2354 | |
2355 | from landscape.lib.compat import ConfigParser, NoOptionError |
2356 | from landscape.client.monitor.plugin import DataWatcher |
2357 | diff --git a/landscape/client/manager/scriptexecution.py b/landscape/client/manager/scriptexecution.py |
2358 | index 4481e17..b20b58a 100644 |
2359 | --- a/landscape/client/manager/scriptexecution.py |
2360 | +++ b/landscape/client/manager/scriptexecution.py |
2361 | @@ -115,7 +115,8 @@ class ScriptRunnerMixin(object): |
2362 | } |
2363 | |
2364 | pp = ProcessAccumulationProtocol( |
2365 | - self.registry.reactor, self.size_limit, self.truncation_indicator) |
2366 | + self.registry.reactor, self.registry.config.script_output_limit, |
2367 | + self.truncation_indicator) |
2368 | args = (filename,) |
2369 | self.process_factory.spawnProcess( |
2370 | pp, filename, args=args, uid=uid, gid=gid, path=path, env=env) |
2371 | @@ -127,11 +128,8 @@ class ScriptRunnerMixin(object): |
2372 | class ScriptExecutionPlugin(ManagerPlugin, ScriptRunnerMixin): |
2373 | """A plugin which allows execution of arbitrary shell scripts. |
2374 | |
2375 | - @ivar size_limit: The number of bytes at which to truncate process output. |
2376 | """ |
2377 | |
2378 | - size_limit = 500000 |
2379 | - |
2380 | def register(self, registry): |
2381 | super(ScriptExecutionPlugin, self).register(registry) |
2382 | registry.register_message( |
2383 | @@ -316,7 +314,7 @@ class ProcessAccumulationProtocol(ProcessProtocol): |
2384 | self._size = 0 |
2385 | self.result_deferred = Deferred() |
2386 | self._cancelled = False |
2387 | - self.size_limit = size_limit |
2388 | + self.size_limit = size_limit * 1024 |
2389 | self._truncation_indicator = truncation_indicator.encode("utf-8") |
2390 | self._truncation_offset = len(self._truncation_indicator) |
2391 | self._truncated_size_limit = self.size_limit - self._truncation_offset |
2392 | diff --git a/landscape/client/manager/service.py b/landscape/client/manager/service.py |
2393 | index b95f4eb..ef543d6 100644 |
2394 | --- a/landscape/client/manager/service.py |
2395 | +++ b/landscape/client/manager/service.py |
2396 | @@ -54,8 +54,9 @@ class ManagerService(LandscapeService): |
2397 | def stopService(self): |
2398 | """Stop the manager and close the connection with the broker.""" |
2399 | self.connector.disconnect() |
2400 | - self.publisher.stop() |
2401 | + deferred = self.publisher.stop() |
2402 | super(ManagerService, self).stopService() |
2403 | + return deferred |
2404 | |
2405 | |
2406 | def run(args): |
2407 | diff --git a/landscape/client/manager/tests/test_aptsources.py b/landscape/client/manager/tests/test_aptsources.py |
2408 | index b63d4db..a4d2cd8 100644 |
2409 | --- a/landscape/client/manager/tests/test_aptsources.py |
2410 | +++ b/landscape/client/manager/tests/test_aptsources.py |
2411 | @@ -7,7 +7,6 @@ from twisted.internet.defer import Deferred, succeed |
2412 | from landscape.client.manager.aptsources import AptSources |
2413 | from landscape.client.manager.plugin import SUCCEEDED, FAILED |
2414 | |
2415 | -from landscape.lib.twisted_util import gather_results, SignalError |
2416 | from landscape.client.tests.helpers import LandscapeTest, ManagerHelper |
2417 | from landscape.client.package.reporter import find_reporter_command |
2418 | |
2419 | @@ -272,210 +271,23 @@ class AptSourcesTests(LandscapeTest): |
2420 | C{AptSources} runs a process with apt-key for every keys in the |
2421 | message. |
2422 | """ |
2423 | - deferred = Deferred() |
2424 | - |
2425 | - def _run_process(command, args, env={}, path=None, uid=None, gid=None): |
2426 | - self.assertEqual("/usr/bin/apt-key", command) |
2427 | - self.assertEqual("add", args[0]) |
2428 | - filename = args[1] |
2429 | - with open(filename) as file: |
2430 | - result = file.read() |
2431 | - self.assertEqual("Some key content", result) |
2432 | - deferred.callback(("ok", "", 0)) |
2433 | - return deferred |
2434 | - |
2435 | - self.sourceslist._run_process = _run_process |
2436 | - |
2437 | - self.manager.dispatch_message( |
2438 | - {"type": "apt-sources-replace", "sources": [], |
2439 | - "gpg-keys": ["Some key content"], "operation-id": 1}) |
2440 | - |
2441 | - return deferred |
2442 | - |
2443 | - def test_import_delete_temporary_files(self): |
2444 | - """ |
2445 | - The files created to be imported by C{apt-key} are removed after the |
2446 | - import. |
2447 | - """ |
2448 | - deferred = Deferred() |
2449 | - filenames = [] |
2450 | - |
2451 | - def _run_process(command, args, env={}, path=None, uid=None, gid=None): |
2452 | - if not filenames: |
2453 | - filenames.append(args[1]) |
2454 | - deferred.callback(("ok", "", 0)) |
2455 | - return deferred |
2456 | - |
2457 | - self.sourceslist._run_process = _run_process |
2458 | - |
2459 | - self.manager.dispatch_message( |
2460 | - {"type": "apt-sources-replace", "sources": [], |
2461 | - "gpg-keys": ["Some key content"], "operation-id": 1}) |
2462 | - |
2463 | - self.assertFalse(os.path.exists(filenames[0])) |
2464 | - |
2465 | - return deferred |
2466 | - |
2467 | - def test_failed_import_delete_temporary_files(self): |
2468 | - """ |
2469 | - The files created to be imported by C{apt-key} are removed after the |
2470 | - import, even if there is a failure. |
2471 | - """ |
2472 | - deferred = Deferred() |
2473 | - filenames = [] |
2474 | - |
2475 | - def _run_process(command, args, env={}, path=None, uid=None, gid=None): |
2476 | - filenames.append(args[1]) |
2477 | - deferred.callback(("error", "", 1)) |
2478 | - return deferred |
2479 | |
2480 | - self.sourceslist._run_process = _run_process |
2481 | + self.sourceslist.TRUSTED_GPG_D = self.makeDir() |
2482 | |
2483 | + gpg_keys = ["key1", "key2"] |
2484 | self.manager.dispatch_message( |
2485 | {"type": "apt-sources-replace", "sources": [], |
2486 | - "gpg-keys": ["Some key content"], "operation-id": 1}) |
2487 | - |
2488 | - self.assertFalse(os.path.exists(filenames[0])) |
2489 | - |
2490 | - return deferred |
2491 | - |
2492 | - def test_failed_import_reported(self): |
2493 | - """ |
2494 | - If the C{apt-key} command failed for some reasons, the output of the |
2495 | - command is reported and the activity fails. |
2496 | - """ |
2497 | - deferred = Deferred() |
2498 | - |
2499 | - def _run_process(command, args, env={}, path=None, uid=None, gid=None): |
2500 | - deferred.callback(("nok", "some error", 1)) |
2501 | - return deferred |
2502 | - |
2503 | - self.sourceslist._run_process = _run_process |
2504 | - |
2505 | - self.manager.dispatch_message( |
2506 | - {"type": "apt-sources-replace", "sources": [], "gpg-keys": ["key"], |
2507 | + "gpg-keys": gpg_keys, |
2508 | "operation-id": 1}) |
2509 | |
2510 | - service = self.broker_service |
2511 | - msg = "ProcessError: nok\nsome error" |
2512 | - self.assertMessages(service.message_store.get_pending_messages(), |
2513 | - [{"type": "operation-result", |
2514 | - "result-text": msg, "status": FAILED, |
2515 | - "operation-id": 1}]) |
2516 | - return deferred |
2517 | + keys = [] |
2518 | + gpg_dirpath = self.sourceslist.TRUSTED_GPG_D |
2519 | + for filename in os.listdir(gpg_dirpath): |
2520 | + filepath = os.path.join(gpg_dirpath, filename) |
2521 | + with open(filepath, 'r') as fh: |
2522 | + keys.append(fh.read()) |
2523 | |
2524 | - def test_signaled_import_reported(self): |
2525 | - """ |
2526 | - If the C{apt-key} fails with a signal, the output of the command is |
2527 | - reported and the activity fails. |
2528 | - """ |
2529 | - deferred = Deferred() |
2530 | - |
2531 | - def _run_process(command, args, env={}, path=None, uid=None, gid=None): |
2532 | - deferred.errback(SignalError("nok", "some error", 1)) |
2533 | - return deferred |
2534 | - |
2535 | - self.sourceslist._run_process = _run_process |
2536 | - |
2537 | - self.manager.dispatch_message( |
2538 | - {"type": "apt-sources-replace", "sources": [], "gpg-keys": ["key"], |
2539 | - "operation-id": 1}) |
2540 | - |
2541 | - service = self.broker_service |
2542 | - msg = "ProcessError: nok\nsome error" |
2543 | - self.assertMessages(service.message_store.get_pending_messages(), |
2544 | - [{"type": "operation-result", |
2545 | - "result-text": msg, "status": FAILED, |
2546 | - "operation-id": 1}]) |
2547 | - return deferred |
2548 | - |
2549 | - def test_failed_import_no_changes(self): |
2550 | - """ |
2551 | - If the C{apt-key} command failed for some reasons, the current |
2552 | - repositories aren't changed. |
2553 | - """ |
2554 | - deferred = Deferred() |
2555 | - |
2556 | - def _run_process(command, args, env={}, path=None, uid=None, gid=None): |
2557 | - deferred.callback(("nok", "some error", 1)) |
2558 | - return deferred |
2559 | - |
2560 | - self.sourceslist._run_process = _run_process |
2561 | - |
2562 | - with open(self.sourceslist.SOURCES_LIST, "w") as sources: |
2563 | - sources.write("oki\n\ndoki\n#comment\n") |
2564 | - |
2565 | - self.manager.dispatch_message( |
2566 | - {"type": "apt-sources-replace", "sources": [], "gpg-keys": ["key"], |
2567 | - "operation-id": 1}) |
2568 | - |
2569 | - with open(self.sourceslist.SOURCES_LIST) as sources_list: |
2570 | - result = sources_list.read() |
2571 | - |
2572 | - self.assertEqual("oki\n\ndoki\n#comment\n", result) |
2573 | - |
2574 | - return deferred |
2575 | - |
2576 | - def test_multiple_import_sequential(self): |
2577 | - """ |
2578 | - If multiple keys are specified, the imports run sequentially, not in |
2579 | - parallel. |
2580 | - """ |
2581 | - deferred1 = Deferred() |
2582 | - deferred2 = Deferred() |
2583 | - deferreds = [deferred1, deferred2] |
2584 | - |
2585 | - def _run_process(command, args, env={}, path=None, uid=None, gid=None): |
2586 | - if not deferreds: |
2587 | - return succeed(None) |
2588 | - return deferreds.pop(0) |
2589 | - |
2590 | - self.sourceslist._run_process = _run_process |
2591 | - |
2592 | - self.manager.dispatch_message( |
2593 | - {"type": "apt-sources-replace", "sources": [], |
2594 | - "gpg-keys": ["key1", "key2"], "operation-id": 1}) |
2595 | - |
2596 | - self.assertEqual(1, len(deferreds)) |
2597 | - deferred1.callback(("ok", "", 0)) |
2598 | - |
2599 | - self.assertEqual(0, len(deferreds)) |
2600 | - deferred2.callback(("ok", "", 0)) |
2601 | - |
2602 | - service = self.broker_service |
2603 | - self.assertMessages(service.message_store.get_pending_messages(), |
2604 | - [{"type": "operation-result", |
2605 | - "status": SUCCEEDED, "operation-id": 1}]) |
2606 | - return gather_results(deferreds) |
2607 | - |
2608 | - def test_multiple_import_failure(self): |
2609 | - """ |
2610 | - If multiple keys are specified, and that the first one fails, the error |
2611 | - is correctly reported. |
2612 | - """ |
2613 | - deferred1 = Deferred() |
2614 | - deferred2 = Deferred() |
2615 | - deferreds = [deferred1, deferred2] |
2616 | - |
2617 | - def _run_process(command, args, env={}, path=None, uid=None, gid=None): |
2618 | - return deferreds.pop(0) |
2619 | - |
2620 | - self.sourceslist._run_process = _run_process |
2621 | - |
2622 | - self.manager.dispatch_message( |
2623 | - {"type": "apt-sources-replace", "sources": [], |
2624 | - "gpg-keys": ["key1", "key2"], "operation-id": 1}) |
2625 | - |
2626 | - deferred1.callback(("error", "", 1)) |
2627 | - deferred2.callback(("error", "", 1)) |
2628 | - |
2629 | - msg = "ProcessError: error\n" |
2630 | - service = self.broker_service |
2631 | - self.assertMessages(service.message_store.get_pending_messages(), |
2632 | - [{"type": "operation-result", |
2633 | - "result-text": msg, "status": FAILED, |
2634 | - "operation-id": 1}]) |
2635 | - return gather_results(deferreds) |
2636 | + self.assertCountEqual(keys, gpg_keys) |
2637 | |
2638 | def test_run_reporter(self): |
2639 | """ |
2640 | diff --git a/landscape/client/manager/tests/test_processkiller.py b/landscape/client/manager/tests/test_processkiller.py |
2641 | index 75ef833..ae45b0a 100644 |
2642 | --- a/landscape/client/manager/tests/test_processkiller.py |
2643 | +++ b/landscape/client/manager/tests/test_processkiller.py |
2644 | @@ -14,7 +14,7 @@ from landscape.client.manager.processkiller import ( |
2645 | |
2646 | |
2647 | def get_active_process(): |
2648 | - return subprocess.Popen(["python", "-c", "raw_input()"], |
2649 | + return subprocess.Popen(["python3", "-c", "input()"], |
2650 | stdin=subprocess.PIPE, |
2651 | stdout=subprocess.PIPE, |
2652 | stderr=subprocess.PIPE) |
2653 | diff --git a/landscape/client/manager/tests/test_scriptexecution.py b/landscape/client/manager/tests/test_scriptexecution.py |
2654 | index 2f21949..db3ef12 100644 |
2655 | --- a/landscape/client/manager/tests/test_scriptexecution.py |
2656 | +++ b/landscape/client/manager/tests/test_scriptexecution.py |
2657 | @@ -70,7 +70,7 @@ class RunScriptTests(LandscapeTest): |
2658 | |
2659 | def test_other_interpreter(self): |
2660 | """Non-shell interpreters can be specified.""" |
2661 | - result = self.plugin.run_script("/usr/bin/python", "print 'hi'") |
2662 | + result = self.plugin.run_script("/usr/bin/python3", "print('hi')") |
2663 | result.addCallback(self.assertEqual, "hi\n") |
2664 | return result |
2665 | |
2666 | @@ -492,18 +492,18 @@ class RunScriptTests(LandscapeTest): |
2667 | """Data returned from the command is limited.""" |
2668 | factory = StubProcessFactory() |
2669 | self.plugin.process_factory = factory |
2670 | - self.plugin.size_limit = 100 |
2671 | + self.manager.config.script_output_limit = 1 |
2672 | result = self.plugin.run_script("/bin/sh", "") |
2673 | |
2674 | # Ultimately we assert that the resulting output is limited to |
2675 | - # 100 bytes and indicates its truncation. |
2676 | + # 1024 bytes and indicates its truncation. |
2677 | result.addCallback(self.assertEqual, |
2678 | - ("x" * 79) + "\n**OUTPUT TRUNCATED**") |
2679 | + ("x" * (1024 - 21)) + "\n**OUTPUT TRUNCATED**") |
2680 | |
2681 | protocol = factory.spawns[0][0] |
2682 | |
2683 | - # Push 200 bytes of output, so we trigger truncation. |
2684 | - protocol.childDataReceived(1, b"x" * 200) |
2685 | + # Push 2kB of output, so we trigger truncation. |
2686 | + protocol.childDataReceived(1, b"x" * (2*1024)) |
2687 | |
2688 | for fd in (0, 1, 2): |
2689 | protocol.childConnectionLost(fd) |
2690 | @@ -515,19 +515,19 @@ class RunScriptTests(LandscapeTest): |
2691 | """After truncation, no further output is recorded.""" |
2692 | factory = StubProcessFactory() |
2693 | self.plugin.process_factory = factory |
2694 | - self.plugin.size_limit = 100 |
2695 | + self.manager.config.script_output_limit = 1 |
2696 | result = self.plugin.run_script("/bin/sh", "") |
2697 | |
2698 | # Ultimately we assert that the resulting output is limited to |
2699 | - # 100 bytes and indicates its truncation. |
2700 | + # 1024 bytes and indicates its truncation. |
2701 | result.addCallback(self.assertEqual, |
2702 | - ("x" * 79) + "\n**OUTPUT TRUNCATED**") |
2703 | + ("x" * (1024 - 21)) + "\n**OUTPUT TRUNCATED**") |
2704 | protocol = factory.spawns[0][0] |
2705 | |
2706 | - # Push 200 bytes of output, so we trigger truncation. |
2707 | - protocol.childDataReceived(1, b"x" * 200) |
2708 | - # Push 200 bytes more |
2709 | - protocol.childDataReceived(1, b"x" * 200) |
2710 | + # Push 1024 bytes of output, so we trigger truncation. |
2711 | + protocol.childDataReceived(1, b"x" * 1024) |
2712 | + # Push 1024 bytes more |
2713 | + protocol.childDataReceived(1, b"x" * 1024) |
2714 | |
2715 | for fd in (0, 1, 2): |
2716 | protocol.childConnectionLost(fd) |
2717 | diff --git a/landscape/client/monitor/__init__.py b/landscape/client/monitor/__init__.py |
2718 | index d8678a3..bb6b536 100644 |
2719 | --- a/landscape/client/monitor/__init__.py |
2720 | +++ b/landscape/client/monitor/__init__.py |
2721 | @@ -1,4 +1,4 @@ |
2722 | """ |
2723 | The monitor extracts data about the local machine and sends it in |
2724 | -messages to the Landcsape server via the broker. |
2725 | +messages to the Landscape server via the broker. |
2726 | """ |
2727 | diff --git a/landscape/client/monitor/computertags.py b/landscape/client/monitor/computertags.py |
2728 | new file mode 100644 |
2729 | index 0000000..1429367 |
2730 | --- /dev/null |
2731 | +++ b/landscape/client/monitor/computertags.py |
2732 | @@ -0,0 +1,29 @@ |
2733 | +import logging |
2734 | +import sys |
2735 | + |
2736 | +from landscape.client.broker.config import BrokerConfiguration |
2737 | +from landscape.client.monitor.plugin import DataWatcher |
2738 | +from landscape.lib.tag import is_valid_tag_list |
2739 | + |
2740 | + |
2741 | +class ComputerTags(DataWatcher): |
2742 | + """Plugin watches config file for changes in computer tags""" |
2743 | + |
2744 | + persist_name = "computer-tags" |
2745 | + message_type = "computer-tags" |
2746 | + message_key = "tags" |
2747 | + run_interval = 3600 # Every hour only when data changed |
2748 | + run_immediately = True |
2749 | + |
2750 | + def __init__(self, args=sys.argv): |
2751 | + super(ComputerTags, self).__init__() |
2752 | + self.args = args # Defined to specify args in unit tests |
2753 | + |
2754 | + def get_data(self): |
2755 | + config = BrokerConfiguration() |
2756 | + config.load(self.args) # Load the default or specified config |
2757 | + tags = config.tags |
2758 | + if not is_valid_tag_list(tags): |
2759 | + tags = None |
2760 | + logging.warning("Invalid tags provided for computer-tags message.") |
2761 | + return tags |
2762 | diff --git a/landscape/client/monitor/config.py b/landscape/client/monitor/config.py |
2763 | index 2525157..19f6aa4 100644 |
2764 | --- a/landscape/client/monitor/config.py |
2765 | +++ b/landscape/client/monitor/config.py |
2766 | @@ -6,7 +6,7 @@ ALL_PLUGINS = ["ActiveProcessInfo", "ComputerInfo", |
2767 | "Temperature", "PackageMonitor", "UserMonitor", |
2768 | "RebootRequired", "AptPreferences", "NetworkActivity", |
2769 | "NetworkDevice", "UpdateManager", "CPUUsage", "SwiftUsage", |
2770 | - "CephUsage"] |
2771 | + "CephUsage", "ComputerTags", "UbuntuProInfo"] |
2772 | |
2773 | |
2774 | class MonitorConfiguration(Configuration): |
2775 | diff --git a/landscape/client/monitor/processorinfo.py b/landscape/client/monitor/processorinfo.py |
2776 | index c129707..aaa15e3 100644 |
2777 | --- a/landscape/client/monitor/processorinfo.py |
2778 | +++ b/landscape/client/monitor/processorinfo.py |
2779 | @@ -87,7 +87,8 @@ class ProcessorInfo(MonitorPlugin): |
2780 | dirty = True |
2781 | |
2782 | if dirty: |
2783 | - logging.info("Queueing message with updated processor info.") |
2784 | + logging.info("Queueing updated processor info. Contents:") |
2785 | + logging.info(message) |
2786 | self.registry.broker.send_message( |
2787 | message, self._session_id, urgent=urgent) |
2788 | |
2789 | @@ -313,8 +314,47 @@ class S390XMessageFactory: |
2790 | return processors |
2791 | |
2792 | |
2793 | +class RISCVMessageFactory: |
2794 | + """Factory for risc-v-based processors provides processor information. |
2795 | + |
2796 | + @param source_filename: The file name of the data source. |
2797 | + """ |
2798 | + |
2799 | + def __init__(self, source_filename): |
2800 | + self._source_filename = source_filename |
2801 | + |
2802 | + def create_message(self): |
2803 | + """Returns a list containing information about each processor.""" |
2804 | + processors = [] |
2805 | + file = open(self._source_filename) |
2806 | + logging.info("Entered RISCVMessageFactory") |
2807 | + |
2808 | + try: |
2809 | + current = None |
2810 | + |
2811 | + for line in file: |
2812 | + parts = line.split(":", 1) |
2813 | + key = parts[0].strip() |
2814 | + |
2815 | + if key == "processor": |
2816 | + current = {"processor-id": int(parts[1].strip())} |
2817 | + processors.append(current) |
2818 | + elif key == "isa": |
2819 | + current["vendor"] = parts[1].strip() |
2820 | + elif key == "uarch": |
2821 | + current["model"] = parts[1].strip() |
2822 | + |
2823 | + finally: |
2824 | + file.close() |
2825 | + |
2826 | + logging.info("RISC-V processor info collected:") |
2827 | + logging.info(processors) |
2828 | + return processors |
2829 | + |
2830 | + |
2831 | message_factories = [("(arm*|aarch64)", ARMMessageFactory), |
2832 | ("ppc(64)?", PowerPCMessageFactory), |
2833 | ("sparc[64]", SparcMessageFactory), |
2834 | ("i[3-7]86|x86_64", X86MessageFactory), |
2835 | - ("s390x", S390XMessageFactory)] |
2836 | + ("s390x", S390XMessageFactory), |
2837 | + ("riscv64", RISCVMessageFactory)] |
2838 | diff --git a/landscape/client/monitor/service.py b/landscape/client/monitor/service.py |
2839 | index 77d6f20..6b9c63a 100644 |
2840 | --- a/landscape/client/monitor/service.py |
2841 | +++ b/landscape/client/monitor/service.py |
2842 | @@ -56,10 +56,11 @@ class MonitorService(LandscapeService): |
2843 | The monitor is flushed to ensure that things like persist databases |
2844 | get saved to disk. |
2845 | """ |
2846 | - self.publisher.stop() |
2847 | + deferred = self.publisher.stop() |
2848 | self.monitor.flush() |
2849 | self.connector.disconnect() |
2850 | super(MonitorService, self).stopService() |
2851 | + return deferred |
2852 | |
2853 | |
2854 | def run(args): |
2855 | diff --git a/landscape/client/monitor/tests/test_computertags.py b/landscape/client/monitor/tests/test_computertags.py |
2856 | new file mode 100644 |
2857 | index 0000000..7f52943 |
2858 | --- /dev/null |
2859 | +++ b/landscape/client/monitor/tests/test_computertags.py |
2860 | @@ -0,0 +1,64 @@ |
2861 | +from landscape.client.monitor.computertags import ComputerTags |
2862 | +from landscape.client.tests.helpers import MonitorHelper, LandscapeTest |
2863 | + |
2864 | + |
2865 | +class ComputerTagsTest(LandscapeTest): |
2866 | + |
2867 | + helpers = [MonitorHelper] |
2868 | + |
2869 | + def setUp(self): |
2870 | + super(ComputerTagsTest, self).setUp() |
2871 | + test_sys_args = ['hello.py'] |
2872 | + self.plugin = ComputerTags(args=test_sys_args) |
2873 | + self.monitor.add(self.plugin) |
2874 | + |
2875 | + def test_tags_are_read(self): |
2876 | + """ |
2877 | + Tags are read from the default config path |
2878 | + """ |
2879 | + tags = 'check,linode,profile-test' |
2880 | + file_text = "[client]\ntags = {}".format(tags) |
2881 | + config_filename = self.config.default_config_filenames[0] |
2882 | + self.makeFile(file_text, path=config_filename) |
2883 | + self.assertEqual(self.plugin.get_data(), tags) |
2884 | + |
2885 | + def test_tags_are_read_from_args_path(self): |
2886 | + """ |
2887 | + Tags are read from path specified in command line args |
2888 | + """ |
2889 | + tags = 'check,linode,profile-test' |
2890 | + file_text = "[client]\ntags = {}".format(tags) |
2891 | + filename = self.makeFile(file_text) |
2892 | + test_sys_args = ['hello.py', '--config', filename] |
2893 | + self.plugin.args = test_sys_args |
2894 | + self.assertEqual(self.plugin.get_data(), tags) |
2895 | + |
2896 | + def test_tags_message_sent(self): |
2897 | + """ |
2898 | + Tags message is sent correctly |
2899 | + """ |
2900 | + tags = 'check,linode,profile-test' |
2901 | + file_text = "[client]\ntags = {}".format(tags) |
2902 | + config_filename = self.config.default_config_filenames[0] |
2903 | + self.makeFile(file_text, path=config_filename) |
2904 | + |
2905 | + self.mstore.set_accepted_types(["computer-tags"]) |
2906 | + self.plugin.exchange() |
2907 | + messages = self.mstore.get_pending_messages() |
2908 | + self.assertEqual(messages[0]['tags'], tags) |
2909 | + |
2910 | + def test_invalid_tags(self): |
2911 | + """ |
2912 | + If invalid tag detected then message contents should be None |
2913 | + """ |
2914 | + tags = 'check,lin ode' |
2915 | + file_text = "[client]\ntags = {}".format(tags) |
2916 | + config_filename = self.config.default_config_filenames[0] |
2917 | + self.makeFile(file_text, path=config_filename) |
2918 | + self.assertEqual(self.plugin.get_data(), None) |
2919 | + |
2920 | + def test_empty_tags(self): |
2921 | + """ |
2922 | + Makes sure no errors when tags section is empty |
2923 | + """ |
2924 | + self.assertEqual(self.plugin.get_data(), None) |
2925 | diff --git a/landscape/client/monitor/tests/test_ubuntuproinfo.py b/landscape/client/monitor/tests/test_ubuntuproinfo.py |
2926 | new file mode 100644 |
2927 | index 0000000..ece255c |
2928 | --- /dev/null |
2929 | +++ b/landscape/client/monitor/tests/test_ubuntuproinfo.py |
2930 | @@ -0,0 +1,47 @@ |
2931 | +from unittest import mock |
2932 | + |
2933 | +from landscape.client.monitor.ubuntuproinfo import UbuntuProInfo |
2934 | +from landscape.client.tests.helpers import LandscapeTest, MonitorHelper |
2935 | + |
2936 | + |
2937 | +class UbuntuProInfoTest(LandscapeTest): |
2938 | + """Ubuntu Pro info plugin tests.""" |
2939 | + |
2940 | + helpers = [MonitorHelper] |
2941 | + |
2942 | + def setUp(self): |
2943 | + super(UbuntuProInfoTest, self).setUp() |
2944 | + self.mstore.set_accepted_types(["ubuntu-pro-info"]) |
2945 | + |
2946 | + def test_ubuntu_pro_info(self): |
2947 | + """Tests calling `ua status`.""" |
2948 | + plugin = UbuntuProInfo() |
2949 | + self.monitor.add(plugin) |
2950 | + |
2951 | + with mock.patch("subprocess.run") as run_mock: |
2952 | + run_mock.return_value = mock.Mock( |
2953 | + stdout="\"This is a test\"", |
2954 | + ) |
2955 | + plugin.exchange() |
2956 | + |
2957 | + messages = self.mstore.get_pending_messages() |
2958 | + run_mock.assert_called_once() |
2959 | + self.assertTrue(len(messages) > 0) |
2960 | + self.assertTrue("ubuntu-pro-info" in messages[0]) |
2961 | + self.assertEqual(messages[0]["ubuntu-pro-info"], |
2962 | + "\"This is a test\"") |
2963 | + |
2964 | + def test_ubuntu_pro_info_no_ua(self): |
2965 | + """Tests calling `ua status` when it is not installed.""" |
2966 | + plugin = UbuntuProInfo() |
2967 | + self.monitor.add(plugin) |
2968 | + |
2969 | + with mock.patch("subprocess.run") as run_mock: |
2970 | + run_mock.side_effect = FileNotFoundError() |
2971 | + plugin.exchange() |
2972 | + |
2973 | + messages = self.mstore.get_pending_messages() |
2974 | + run_mock.assert_called_once() |
2975 | + self.assertTrue(len(messages) > 0) |
2976 | + self.assertTrue("ubuntu-pro-info" in messages[0]) |
2977 | + self.assertIn("errors", messages[0]["ubuntu-pro-info"]) |
2978 | diff --git a/landscape/client/monitor/ubuntuproinfo.py b/landscape/client/monitor/ubuntuproinfo.py |
2979 | new file mode 100644 |
2980 | index 0000000..4f78999 |
2981 | --- /dev/null |
2982 | +++ b/landscape/client/monitor/ubuntuproinfo.py |
2983 | @@ -0,0 +1,47 @@ |
2984 | +import json |
2985 | +import subprocess |
2986 | + |
2987 | +from landscape.client.monitor.plugin import DataWatcher |
2988 | + |
2989 | + |
2990 | +class UbuntuProInfo(DataWatcher): |
2991 | + """ |
2992 | + Plugin that captures and reports Ubuntu Pro registration |
2993 | + information. |
2994 | + |
2995 | + We use the `ua` CLI with output formatted as JSON. This is sent |
2996 | + as-is and parsed by Landscape Server because the JSON content is |
2997 | + considered "Experimental" and we don't want to have to change in |
2998 | + both Client and Server in the event that the format changes. |
2999 | + """ |
3000 | + |
3001 | + run_interval = 900 # 15 minutes |
3002 | + message_type = "ubuntu-pro-info" |
3003 | + message_key = message_type |
3004 | + persist_name = message_type |
3005 | + scope = "ubuntu-pro" |
3006 | + run_immediately = True |
3007 | + |
3008 | + def get_data(self): |
3009 | + ubuntu_pro_info = get_ubuntu_pro_info() |
3010 | + |
3011 | + return json.dumps(ubuntu_pro_info, separators=(",", ":")) |
3012 | + |
3013 | + |
3014 | +def get_ubuntu_pro_info(): |
3015 | + try: |
3016 | + completed_process = subprocess.run( |
3017 | + ["ua", "status", "--format", "json"], |
3018 | + encoding="utf8", stdout=subprocess.PIPE) |
3019 | + except FileNotFoundError: |
3020 | + return { |
3021 | + "errors": [{ |
3022 | + "message": "ubuntu-advantage-tools not found.", |
3023 | + "message_code": "tools-error", |
3024 | + "service": None, |
3025 | + "type": "system", |
3026 | + }], |
3027 | + "result": "failure", |
3028 | + } |
3029 | + else: |
3030 | + return json.loads(completed_process.stdout) |
3031 | diff --git a/landscape/client/package/changer.py b/landscape/client/package/changer.py |
3032 | index d4042ae..31253d2 100644 |
3033 | --- a/landscape/client/package/changer.py |
3034 | +++ b/landscape/client/package/changer.py |
3035 | @@ -1,5 +1,4 @@ |
3036 | import logging |
3037 | -import base64 |
3038 | import time |
3039 | import os |
3040 | import pwd |
3041 | @@ -14,6 +13,7 @@ from landscape.constants import ( |
3042 | UNKNOWN_PACKAGE_DATA_TIMEOUT) |
3043 | |
3044 | from landscape.lib.config import get_bindir |
3045 | +from landscape.lib import base64 |
3046 | from landscape.lib.fs import create_binary_file |
3047 | from landscape.lib.log import log_failure |
3048 | from landscape.client.package.reporter import find_reporter_command |
3049 | @@ -173,7 +173,7 @@ class PackageChanger(PackageTaskHandler): |
3050 | hash_ids = {} |
3051 | for hash, id, deb in binaries: |
3052 | create_binary_file(os.path.join(binaries_path, "%d.deb" % id), |
3053 | - base64.decodestring(deb)) |
3054 | + base64.decodebytes(deb)) |
3055 | hash_ids[hash] = id |
3056 | self._store.set_hash_ids(hash_ids) |
3057 | self._facade.add_channel_deb_dir(binaries_path) |
3058 | diff --git a/landscape/client/package/reporter.py b/landscape/client/package/reporter.py |
3059 | index 39b67df..0ac4381 100644 |
3060 | --- a/landscape/client/package/reporter.py |
3061 | +++ b/landscape/client/package/reporter.py |
3062 | @@ -3,6 +3,7 @@ try: |
3063 | except ImportError: |
3064 | import urllib.parse as urlparse |
3065 | |
3066 | +import locale |
3067 | import logging |
3068 | import time |
3069 | import os |
3070 | @@ -866,6 +867,10 @@ class FakeReporter(PackageReporter): |
3071 | |
3072 | |
3073 | def main(args): |
3074 | + # Force UTF-8 encoding only for the reporter, thus allowing libapt-pkg to |
3075 | + # return unmangled descriptions. |
3076 | + locale.setlocale(locale.LC_CTYPE, ("C", "UTF-8")) |
3077 | + |
3078 | if "FAKE_GLOBAL_PACKAGE_STORE" in os.environ: |
3079 | return run_task_handler(FakeGlobalReporter, args) |
3080 | elif "FAKE_PACKAGE_STORE" in os.environ: |
3081 | diff --git a/landscape/client/package/tests/test_changer.py b/landscape/client/package/tests/test_changer.py |
3082 | index db97d94..fe2c858 100644 |
3083 | --- a/landscape/client/package/tests/test_changer.py |
3084 | +++ b/landscape/client/package/tests/test_changer.py |
3085 | @@ -1,5 +1,4 @@ |
3086 | # -*- encoding: utf-8 -*- |
3087 | -import base64 |
3088 | import time |
3089 | import sys |
3090 | import os |
3091 | @@ -16,6 +15,7 @@ from landscape.lib.apt.package.facade import ( |
3092 | from landscape.lib.apt.package.testing import ( |
3093 | HASH1, HASH2, HASH3, PKGDEB1, PKGDEB2, |
3094 | AptFacadeHelper, SimpleRepositoryHelper) |
3095 | +from landscape.lib import base64 |
3096 | from landscape.lib.fs import create_text_file, read_text_file, touch_file |
3097 | from landscape.lib.testing import StubProcessFactory, FakeReactor |
3098 | from landscape.client.package.changer import ( |
3099 | @@ -849,9 +849,9 @@ class AptPackageChangerTest(LandscapeTest): |
3100 | |
3101 | binaries_path = self.config.binaries_path |
3102 | self.assertFileContent(os.path.join(binaries_path, "111.deb"), |
3103 | - base64.decodestring(PKGDEB1)) |
3104 | + base64.decodebytes(PKGDEB1)) |
3105 | self.assertFileContent(os.path.join(binaries_path, "222.deb"), |
3106 | - base64.decodestring(PKGDEB2)) |
3107 | + base64.decodebytes(PKGDEB2)) |
3108 | self.assertEqual( |
3109 | self.facade.get_channels(), |
3110 | self.get_binaries_channels(binaries_path)) |
3111 | diff --git a/landscape/client/package/tests/test_releaseupgrader.py b/landscape/client/package/tests/test_releaseupgrader.py |
3112 | index 13d03b3..9844a8a 100644 |
3113 | --- a/landscape/client/package/tests/test_releaseupgrader.py |
3114 | +++ b/landscape/client/package/tests/test_releaseupgrader.py |
3115 | @@ -433,12 +433,12 @@ class ReleaseUpgraderTest(LandscapeTest): |
3116 | upgrade_tool_filename = os.path.join(upgrade_tool_directory, "karmic") |
3117 | child_pid_filename = self.makeFile() |
3118 | fd = open(upgrade_tool_filename, "w") |
3119 | - fd.write("#!/usr/bin/env python\n" |
3120 | + fd.write("#!/usr/bin/env python3\n" |
3121 | "import os\n" |
3122 | "import time\n" |
3123 | "import sys\n" |
3124 | "if __name__ == '__main__':\n" |
3125 | - " print 'First parent'\n" |
3126 | + " print('First parent')\n" |
3127 | " pid = os.fork()\n" |
3128 | " if pid > 0:\n" |
3129 | " time.sleep(0.5)\n" |
3130 | diff --git a/landscape/client/package/tests/test_reporter.py b/landscape/client/package/tests/test_reporter.py |
3131 | index 6500aff..c4c4d0c 100644 |
3132 | --- a/landscape/client/package/tests/test_reporter.py |
3133 | +++ b/landscape/client/package/tests/test_reporter.py |
3134 | @@ -1,3 +1,4 @@ |
3135 | +import locale |
3136 | import sys |
3137 | import os |
3138 | import time |
3139 | @@ -1231,6 +1232,31 @@ class PackageReporterAptTest(LandscapeTest): |
3140 | self.assertEqual("RESULT", main(["ARGS"])) |
3141 | m.assert_called_once_with(PackageReporter, ["ARGS"]) |
3142 | |
3143 | + def test_main_resets_locale(self): |
3144 | + """ |
3145 | + Reporter entry point should reset encoding to utf-8, as libapt-pkg |
3146 | + encodes description with system encoding and python-apt decodes |
3147 | + them as utf-8 (LP: #1827857). |
3148 | + """ |
3149 | + self._add_package_to_deb_dir( |
3150 | + self.repository_dir, "gosa", description=u"GOsa\u00B2") |
3151 | + self.facade.reload_channels() |
3152 | + |
3153 | + # Set the only non-utf8 locale which we're sure exists. |
3154 | + # It behaves slightly differently than the bug, but fails on the |
3155 | + # same condition. |
3156 | + locale.setlocale(locale.LC_CTYPE, (None, None)) |
3157 | + self.addCleanup(locale.resetlocale) |
3158 | + |
3159 | + with mock.patch("landscape.client.package.reporter.run_task_handler"): |
3160 | + main([]) |
3161 | + |
3162 | + # With the actual package, the failure will occur looking up the |
3163 | + # description translation. |
3164 | + pkg = self.facade.get_packages_by_name("gosa")[0] |
3165 | + skel = self.facade.get_package_skeleton(pkg, with_info=True) |
3166 | + self.assertEqual(u"GOsa\u00B2", skel.description) |
3167 | + |
3168 | def test_find_reporter_command_with_bindir(self): |
3169 | self.config.bindir = "/spam/eggs" |
3170 | command = find_reporter_command(self.config) |
3171 | diff --git a/landscape/client/package/tests/test_taskhandler.py b/landscape/client/package/tests/test_taskhandler.py |
3172 | index 2a37bd4..a6e998f 100644 |
3173 | --- a/landscape/client/package/tests/test_taskhandler.py |
3174 | +++ b/landscape/client/package/tests/test_taskhandler.py |
3175 | @@ -1,4 +1,5 @@ |
3176 | import os |
3177 | +from subprocess import CalledProcessError |
3178 | |
3179 | from mock import patch, Mock, ANY |
3180 | |
3181 | @@ -103,7 +104,9 @@ class PackageTaskHandlerTest(LandscapeTest): |
3182 | self.handler.lsb_release_filename = self.makeFile() |
3183 | |
3184 | # Go! |
3185 | - result = self.handler.use_hash_id_db() |
3186 | + with patch("landscape.lib.lsb_release.check_output") as co_mock: |
3187 | + co_mock.side_effect = CalledProcessError(127, "") |
3188 | + result = self.handler.use_hash_id_db() |
3189 | |
3190 | # The failure should be properly logged |
3191 | logging_mock.assert_called_with( |
3192 | diff --git a/landscape/client/reactor.py b/landscape/client/reactor.py |
3193 | index 501572e..df71d6c 100644 |
3194 | --- a/landscape/client/reactor.py |
3195 | +++ b/landscape/client/reactor.py |
3196 | @@ -2,6 +2,9 @@ |
3197 | Extend the regular Twisted reactor with event-handling features. |
3198 | """ |
3199 | from landscape.lib.reactor import EventHandlingReactor |
3200 | +from landscape.client.lockfile import patch_lockfile |
3201 | + |
3202 | +patch_lockfile() |
3203 | |
3204 | |
3205 | class LandscapeReactor(EventHandlingReactor): |
3206 | diff --git a/landscape/client/tests/test_amp.py b/landscape/client/tests/test_amp.py |
3207 | index 04fb278..0960a38 100644 |
3208 | --- a/landscape/client/tests/test_amp.py |
3209 | +++ b/landscape/client/tests/test_amp.py |
3210 | @@ -1,9 +1,17 @@ |
3211 | -from twisted.internet.error import ConnectError |
3212 | +import os |
3213 | +import errno |
3214 | +import subprocess |
3215 | +import textwrap |
3216 | + |
3217 | +import mock |
3218 | + |
3219 | +from twisted.internet.error import ConnectError, CannotListenError |
3220 | from twisted.internet.task import Clock |
3221 | |
3222 | from landscape.client.tests.helpers import LandscapeTest |
3223 | from landscape.client.deployment import Configuration |
3224 | from landscape.client.amp import ComponentPublisher, ComponentConnector, remote |
3225 | +from landscape.client.reactor import LandscapeReactor |
3226 | from landscape.lib.amp import MethodCallError |
3227 | from landscape.lib.testing import FakeReactor |
3228 | |
3229 | @@ -159,3 +167,83 @@ class ComponentConnectorTest(LandscapeTest): |
3230 | effectively a no-op. |
3231 | """ |
3232 | self.connector.disconnect() |
3233 | + |
3234 | + @mock.patch("twisted.python.lockfile.kill") |
3235 | + def test_stale_locks_with_dead_pid(self, mock_kill): |
3236 | + """Publisher starts with stale lock.""" |
3237 | + mock_kill.side_effect = [ |
3238 | + OSError(errno.ESRCH, "No such process")] |
3239 | + sock_path = os.path.join(self.config.sockets_path, u"test.sock") |
3240 | + lock_path = u"{}.lock".format(sock_path) |
3241 | + # fake a PID which does not exist |
3242 | + os.symlink("-1", lock_path) |
3243 | + |
3244 | + component = TestComponent() |
3245 | + # Test the actual Unix reactor implementation. Fakes won't do. |
3246 | + reactor = LandscapeReactor() |
3247 | + publisher = ComponentPublisher(component, reactor, self.config) |
3248 | + |
3249 | + # Shouldn't raise the exception. |
3250 | + publisher.start() |
3251 | + |
3252 | + # ensure stale lock was replaced |
3253 | + self.assertNotEqual("-1", os.readlink(lock_path)) |
3254 | + mock_kill.assert_called_with(-1, 0) |
3255 | + |
3256 | + publisher.stop() |
3257 | + reactor._cleanup() |
3258 | + |
3259 | + @mock.patch("twisted.python.lockfile.kill") |
3260 | + def test_stale_locks_recycled_pid(self, mock_kill): |
3261 | + """Publisher starts with stale lock pointing to recycled process.""" |
3262 | + mock_kill.side_effect = [ |
3263 | + OSError(errno.EPERM, "Operation not permitted")] |
3264 | + sock_path = os.path.join(self.config.sockets_path, u"test.sock") |
3265 | + lock_path = u"{}.lock".format(sock_path) |
3266 | + # fake a PID recycled by a known process which isn't landscape (init) |
3267 | + os.symlink("1", lock_path) |
3268 | + |
3269 | + component = TestComponent() |
3270 | + # Test the actual Unix reactor implementation. Fakes won't do. |
3271 | + reactor = LandscapeReactor() |
3272 | + publisher = ComponentPublisher(component, reactor, self.config) |
3273 | + |
3274 | + # Shouldn't raise the exception. |
3275 | + publisher.start() |
3276 | + |
3277 | + # ensure stale lock was replaced |
3278 | + self.assertNotEqual("1", os.readlink(lock_path)) |
3279 | + mock_kill.assert_not_called() |
3280 | + self.assertFalse(publisher._port.lockFile.clean) |
3281 | + |
3282 | + publisher.stop() |
3283 | + reactor._cleanup() |
3284 | + |
3285 | + @mock.patch("twisted.python.lockfile.kill") |
3286 | + def test_with_valid_lock(self, mock_kill): |
3287 | + """Publisher raises lock error if a valid lock is held.""" |
3288 | + sock_path = os.path.join(self.config.sockets_path, u"test.sock") |
3289 | + lock_path = u"{}.lock".format(sock_path) |
3290 | + # fake a landscape process |
3291 | + app = self.makeFile(textwrap.dedent("""\ |
3292 | + #!/usr/bin/python3 |
3293 | + import time |
3294 | + time.sleep(10) |
3295 | + """), basename="landscape-manager") |
3296 | + os.chmod(app, 0o755) |
3297 | + call = subprocess.Popen([app]) |
3298 | + self.addCleanup(call.terminate) |
3299 | + os.symlink(str(call.pid), lock_path) |
3300 | + |
3301 | + component = TestComponent() |
3302 | + # Test the actual Unix reactor implementation. Fakes won't do. |
3303 | + reactor = LandscapeReactor() |
3304 | + publisher = ComponentPublisher(component, reactor, self.config) |
3305 | + |
3306 | + with self.assertRaises(CannotListenError): |
3307 | + publisher.start() |
3308 | + |
3309 | + # ensure lock was not replaced |
3310 | + self.assertEqual(str(call.pid), os.readlink(lock_path)) |
3311 | + mock_kill.assert_called_with(call.pid, 0) |
3312 | + reactor._cleanup() |
3313 | diff --git a/landscape/client/tests/test_configuration.py b/landscape/client/tests/test_configuration.py |
3314 | index 36d6d3e..d925d4e 100644 |
3315 | --- a/landscape/client/tests/test_configuration.py |
3316 | +++ b/landscape/client/tests/test_configuration.py |
3317 | @@ -4,6 +4,7 @@ import mock |
3318 | from functools import partial |
3319 | import os |
3320 | import sys |
3321 | +import textwrap |
3322 | import unittest |
3323 | |
3324 | from twisted.internet.defer import succeed, fail, Deferred |
3325 | @@ -11,7 +12,7 @@ from twisted.python.compat import iteritems |
3326 | |
3327 | from landscape.lib.compat import ConfigParser |
3328 | from landscape.lib.compat import StringIO |
3329 | -from landscape.client.broker.registration import RegistrationError |
3330 | +from landscape.client.broker.registration import RegistrationError, Identity |
3331 | from landscape.client.broker.tests.helpers import ( |
3332 | RemoteBrokerHelper, BrokerConfigurationHelper) |
3333 | from landscape.client.configuration import ( |
3334 | @@ -21,7 +22,8 @@ from landscape.client.configuration import ( |
3335 | ImportOptionError, store_public_key_data, |
3336 | bootstrap_tree, got_connection, success, failure, exchange_failure, |
3337 | handle_registration_errors, done, got_error, report_registration_outcome, |
3338 | - determine_exit_code, is_registered) |
3339 | + determine_exit_code, is_registered, registration_info_text, |
3340 | + EXIT_NOT_REGISTERED) |
3341 | from landscape.lib.amp import MethodCallError |
3342 | from landscape.lib.fetch import HTTPCodeError, PyCurlError |
3343 | from landscape.lib.fs import read_binary_file |
3344 | @@ -2233,3 +2235,72 @@ class IsRegisteredTest(LandscapeTest): |
3345 | self.persist.set("registration.secure-id", "super-secure") |
3346 | self.persist.save() |
3347 | self.assertTrue(is_registered(self.config)) |
3348 | + |
3349 | + |
3350 | +class RegistrationInfoTest(LandscapeTest): |
3351 | + |
3352 | + helpers = [BrokerConfigurationHelper] |
3353 | + |
3354 | + def setUp(self): |
3355 | + super(RegistrationInfoTest, self).setUp() |
3356 | + self.custom_args = ['hello.py'] # Fake python script name |
3357 | + self.account_name = 'world' |
3358 | + self.data_path = self.makeDir() |
3359 | + self.config_text = textwrap.dedent(""" |
3360 | + [client] |
3361 | + computer_title = hello |
3362 | + account_name = {} |
3363 | + data_path = {} |
3364 | + """.format(self.account_name, self.data_path)) |
3365 | + |
3366 | + def test_not_registered(self): |
3367 | + '''False when client is not registered''' |
3368 | + config_filename = self.config.default_config_filenames[0] |
3369 | + self.makeFile(self.config_text, path=config_filename) |
3370 | + self.config.load(self.custom_args) |
3371 | + text = registration_info_text(self.config, False) |
3372 | + self.assertIn('False', text) |
3373 | + self.assertNotIn(self.account_name, text) |
3374 | + |
3375 | + def test_registered(self): |
3376 | + ''' |
3377 | + When client is registered, then the text should display as True and |
3378 | + account name should be present |
3379 | + ''' |
3380 | + config_filename = self.config.default_config_filenames[0] |
3381 | + self.makeFile(self.config_text, path=config_filename) |
3382 | + self.config.load(self.custom_args) |
3383 | + text = registration_info_text(self.config, True) |
3384 | + self.assertIn('True', text) |
3385 | + self.assertIn(self.account_name, text) |
3386 | + |
3387 | + def test_custom_config_path(self): |
3388 | + '''The custom config path should show up in the text''' |
3389 | + custom_path = self.makeFile(self.config_text) |
3390 | + self.custom_args += ['-c', custom_path] |
3391 | + self.config.load(self.custom_args) |
3392 | + text = registration_info_text(self.config, False) |
3393 | + self.assertIn(custom_path, text) |
3394 | + |
3395 | + def test_data_path(self): |
3396 | + '''The config data path should show in the text''' |
3397 | + config_filename = self.config.default_config_filenames[0] |
3398 | + self.makeFile(self.config_text, path=config_filename) |
3399 | + self.config.load(self.custom_args) |
3400 | + text = registration_info_text(self.config, False) |
3401 | + self.assertIn(self.data_path, text) |
3402 | + |
3403 | + def test_registered_exit_code(self): |
3404 | + '''Returns exit code zero when client is registered''' |
3405 | + Identity.secure_id = 'test' # Simulate successful registration |
3406 | + exception = self.assertRaises( |
3407 | + SystemExit, main, ["--is-registered", "--silent"], |
3408 | + print=noop_print) |
3409 | + self.assertEqual(0, exception.code) |
3410 | + |
3411 | + def test_not_registered_exit_code(self): |
3412 | + '''Returns special return code when client is not registered''' |
3413 | + exception = self.assertRaises( |
3414 | + SystemExit, main, ["--is-registered", "--silent"], |
3415 | + print=noop_print) |
3416 | + self.assertEqual(EXIT_NOT_REGISTERED, exception.code) |
3417 | diff --git a/landscape/client/tests/test_lockfile.py b/landscape/client/tests/test_lockfile.py |
3418 | new file mode 100644 |
3419 | index 0000000..3b61c9f |
3420 | --- /dev/null |
3421 | +++ b/landscape/client/tests/test_lockfile.py |
3422 | @@ -0,0 +1,21 @@ |
3423 | +import os |
3424 | +import subprocess |
3425 | +import textwrap |
3426 | + |
3427 | +from landscape.client import lockfile |
3428 | +from landscape.client.tests.helpers import LandscapeTest |
3429 | + |
3430 | + |
3431 | +class LockFileTest(LandscapeTest): |
3432 | + |
3433 | + def test_read_process_name(self): |
3434 | + app = self.makeFile(textwrap.dedent("""\ |
3435 | + #!/usr/bin/python3 |
3436 | + import time |
3437 | + time.sleep(10) |
3438 | + """), basename="my_fancy_app") |
3439 | + os.chmod(app, 0o755) |
3440 | + call = subprocess.Popen([app]) |
3441 | + self.addCleanup(call.terminate) |
3442 | + proc_name = lockfile.get_process_name(call.pid) |
3443 | + self.assertEqual("my_fancy_app", proc_name) |
3444 | diff --git a/landscape/client/user/provider.py b/landscape/client/user/provider.py |
3445 | index 61cd933..6abd5e9 100644 |
3446 | --- a/landscape/client/user/provider.py |
3447 | +++ b/landscape/client/user/provider.py |
3448 | @@ -4,7 +4,7 @@ import csv |
3449 | import logging |
3450 | import subprocess |
3451 | |
3452 | -from twisted.python.compat import _PY3 |
3453 | +from landscape.lib.compat import _PY3 |
3454 | |
3455 | |
3456 | class UserManagementError(Exception): |
3457 | diff --git a/landscape/lib/apt/package/facade.py b/landscape/lib/apt/package/facade.py |
3458 | index 0c9114a..37e265e 100644 |
3459 | --- a/landscape/lib/apt/package/facade.py |
3460 | +++ b/landscape/lib/apt/package/facade.py |
3461 | @@ -164,9 +164,10 @@ class AptFacade(object): |
3462 | self._dpkg_status = os.path.join(dpkg_dir, "status") |
3463 | if not os.path.exists(self._dpkg_status): |
3464 | create_text_file(self._dpkg_status, "") |
3465 | - # Apt will fail if it does not have a keyring. It does not care if |
3466 | - # the keyring is empty. |
3467 | - touch_file(os.path.join(apt_dir, "trusted.gpg")) |
3468 | + # Apt will fail if it does not have a keyring. It does not care if |
3469 | + # the keyring is empty. (Do not create one if dir exists LP: #1973202) |
3470 | + if not os.path.isdir(os.path.join(apt_dir, "trusted.gpg.d")): |
3471 | + touch_file(os.path.join(apt_dir, "trusted.gpg")) |
3472 | |
3473 | def _ensure_sub_dir(self, sub_dir): |
3474 | """Ensure that a dir in the Apt root exists.""" |
3475 | @@ -605,9 +606,12 @@ class AptFacade(object): |
3476 | all_info = ["The following packages have unmet dependencies:"] |
3477 | for package in sorted(broken_packages, key=attrgetter("name")): |
3478 | found_dependency_error = False |
3479 | + # Fetch candidate version from our install list because |
3480 | + # apt-2.1.5 resets broken packages candidate. |
3481 | + candidate = next(v._cand for v in self._version_installs |
3482 | + if v.package == package) |
3483 | for dep_type in ["PreDepends", "Depends", "Conflicts", "Breaks"]: |
3484 | - dependencies = package.candidate._cand.depends_list.get( |
3485 | - dep_type, []) |
3486 | + dependencies = candidate.depends_list.get(dep_type, []) |
3487 | for dependency in dependencies: |
3488 | if self._is_dependency_satisfied(dependency, dep_type): |
3489 | continue |
3490 | @@ -744,9 +748,18 @@ class AptFacade(object): |
3491 | # Set the candidate version, so that the version we want to |
3492 | # install actually is the one getting installed. |
3493 | version.package.candidate = version |
3494 | + |
3495 | + # Flag the package as manual if it's a new install, otherwise |
3496 | + # preserve the auto flag. This should preserve explicitly |
3497 | + # installed packages from auto-removal, while allowing upgrades |
3498 | + # of auto-removable packages. |
3499 | + is_manual = ( |
3500 | + not version.package.installed or |
3501 | + not version.package.is_auto_installed) |
3502 | + |
3503 | # Set auto_fix=False to avoid removing the package we asked to |
3504 | # install when we need to resolve dependencies. |
3505 | - version.package.mark_install(auto_fix=False) |
3506 | + version.package.mark_install(auto_fix=False, from_user=is_manual) |
3507 | self._package_installs.add(version.package) |
3508 | fixer.clear(version.package._pkg) |
3509 | fixer.protect(version.package._pkg) |
3510 | diff --git a/landscape/lib/apt/package/skeleton.py b/landscape/lib/apt/package/skeleton.py |
3511 | index eb9ba25..846ab58 100644 |
3512 | --- a/landscape/lib/apt/package/skeleton.py |
3513 | +++ b/landscape/lib/apt/package/skeleton.py |
3514 | @@ -1,9 +1,8 @@ |
3515 | +from landscape.lib.compat import unicode, _PY3 |
3516 | from landscape.lib.hashlib import sha1 |
3517 | |
3518 | import apt_pkg |
3519 | |
3520 | -from twisted.python.compat import unicode, _PY3 |
3521 | - |
3522 | |
3523 | PACKAGE = 1 << 0 |
3524 | PROVIDES = 1 << 1 |
3525 | diff --git a/landscape/lib/apt/package/testing.py b/landscape/lib/apt/package/testing.py |
3526 | index c3e3213..4e8b551 100644 |
3527 | --- a/landscape/lib/apt/package/testing.py |
3528 | +++ b/landscape/lib/apt/package/testing.py |
3529 | @@ -1,4 +1,3 @@ |
3530 | -import base64 |
3531 | import os |
3532 | import time |
3533 | |
3534 | @@ -8,6 +7,7 @@ import apt_pkg |
3535 | from landscape.lib.apt.package.facade import AptFacade |
3536 | from landscape.lib.fs import append_binary_file |
3537 | from landscape.lib.fs import create_binary_file |
3538 | +from landscape.lib import base64 |
3539 | |
3540 | |
3541 | class AptFacadeHelper(object): |
3542 | @@ -322,9 +322,9 @@ PKGDEB_OR_RELATIONS = ( |
3543 | ).encode("ascii") |
3544 | |
3545 | |
3546 | -HASH1 = base64.decodestring(b"/ezv4AefpJJ8DuYFSq4RiEHJYP4=") |
3547 | -HASH2 = base64.decodestring(b"glP4DwWOfMULm0AkRXYsH/exehc=") |
3548 | -HASH3 = base64.decodestring(b"NJM05mj86veaSInYxxqL1wahods=") |
3549 | +HASH1 = base64.decodebytes(b"/ezv4AefpJJ8DuYFSq4RiEHJYP4=") |
3550 | +HASH2 = base64.decodebytes(b"glP4DwWOfMULm0AkRXYsH/exehc=") |
3551 | +HASH3 = base64.decodebytes(b"NJM05mj86veaSInYxxqL1wahods=") |
3552 | HASH_MINIMAL = b"6\xce\x8f\x1bM\x82MWZ\x1a\xffjAc(\xdb(\xa1\x0eG" |
3553 | HASH_SIMPLE_RELATIONS = ( |
3554 | b"'#\xab&k\xe6\xf5E\xcfB\x9b\xceO7\xe6\xec\xa9\xddY\xaa") |
3555 | @@ -339,7 +339,7 @@ HASH_OR_RELATIONS = ( |
3556 | def create_deb(target_dir, pkg_name, pkg_data): |
3557 | """Create a Debian package in the specified C{target_dir}.""" |
3558 | path = os.path.join(target_dir, pkg_name) |
3559 | - data = base64.decodestring(pkg_data) |
3560 | + data = base64.decodebytes(pkg_data) |
3561 | create_binary_file(path, data) |
3562 | |
3563 | |
3564 | diff --git a/landscape/lib/apt/package/tests/test_facade.py b/landscape/lib/apt/package/tests/test_facade.py |
3565 | index 820f6ec..9ea0654 100644 |
3566 | --- a/landscape/lib/apt/package/tests/test_facade.py |
3567 | +++ b/landscape/lib/apt/package/tests/test_facade.py |
3568 | @@ -1812,6 +1812,7 @@ class AptFacadeTest(testing.HelperTestCase, testing.FSTestCase, |
3569 | self._add_system_package("foo") |
3570 | self.facade.reload_channels() |
3571 | [foo] = self.facade.get_packages_by_name("foo") |
3572 | + self.facade._version_installs.append(foo) |
3573 | self.facade._get_broken_packages = lambda: set([foo.package]) |
3574 | self.assertEqual( |
3575 | ["The following packages have unmet dependencies:", |
3576 | @@ -2601,6 +2602,72 @@ class AptFacadeTest(testing.HelperTestCase, testing.FSTestCase, |
3577 | [bar] = self.facade._cache.get_changes() |
3578 | self.assertTrue(bar.marked_upgrade) |
3579 | |
3580 | + def test_changer_upgrade_keeps_auto(self): |
3581 | + """ |
3582 | + An upgrade request should preserve an existing auto flag on the |
3583 | + upgraded package. |
3584 | + """ |
3585 | + self._add_system_package( |
3586 | + "foo", control_fields={"Depends": "bar"}) |
3587 | + self._add_system_package("bar", version="1.0") |
3588 | + deb_dir = self.makeDir() |
3589 | + self._add_package_to_deb_dir(deb_dir, "bar", version="2.0") |
3590 | + self.facade.add_channel_apt_deb( |
3591 | + "file://%s" % deb_dir, "./", trusted=True) |
3592 | + self.facade.reload_channels() |
3593 | + bar_1, bar_2 = sorted(self.facade.get_packages_by_name("bar")) |
3594 | + bar_1.package.mark_auto() |
3595 | + |
3596 | + self.facade.mark_install(bar_2) |
3597 | + self.facade.mark_remove(bar_1) |
3598 | + self.patch_cache_commit() |
3599 | + self.facade.perform_changes() |
3600 | + [bar] = self.facade._cache.get_changes() |
3601 | + self.assertTrue(bar.marked_upgrade) |
3602 | + self.assertTrue(bar.is_auto_installed) |
3603 | + |
3604 | + def test_changer_upgrade_keeps_manual(self): |
3605 | + """ |
3606 | + An upgrade request should mark a package as manual if the installed |
3607 | + version is manual. |
3608 | + """ |
3609 | + self._add_system_package( |
3610 | + "foo", control_fields={"Depends": "bar"}) |
3611 | + self._add_system_package("bar", version="1.0") |
3612 | + deb_dir = self.makeDir() |
3613 | + self._add_package_to_deb_dir(deb_dir, "bar", version="2.0") |
3614 | + self.facade.add_channel_apt_deb( |
3615 | + "file://%s" % deb_dir, "./", trusted=True) |
3616 | + self.facade.reload_channels() |
3617 | + bar_1, bar_2 = sorted(self.facade.get_packages_by_name("bar")) |
3618 | + |
3619 | + self.facade.mark_install(bar_2) |
3620 | + self.facade.mark_remove(bar_1) |
3621 | + self.patch_cache_commit() |
3622 | + self.facade.perform_changes() |
3623 | + [bar] = self.facade._cache.get_changes() |
3624 | + self.assertTrue(bar.marked_upgrade) |
3625 | + self.assertFalse(bar.is_auto_installed) |
3626 | + |
3627 | + def test_changer_install_sets_manual(self): |
3628 | + """ |
3629 | + An installation request should mark the new package as manually |
3630 | + installed. |
3631 | + """ |
3632 | + deb_dir = self.makeDir() |
3633 | + self._add_package_to_deb_dir(deb_dir, "bar", version="2.0") |
3634 | + self.facade.add_channel_apt_deb( |
3635 | + "file://%s" % deb_dir, "./", trusted=True) |
3636 | + self.facade.reload_channels() |
3637 | + bar_2, = self.facade.get_packages_by_name("bar") |
3638 | + |
3639 | + self.facade.mark_install(bar_2) |
3640 | + self.patch_cache_commit() |
3641 | + self.facade.perform_changes() |
3642 | + [bar] = self.facade._cache.get_changes() |
3643 | + self.assertTrue(bar.marked_upgrade) |
3644 | + self.assertFalse(bar.is_auto_installed) |
3645 | + |
3646 | def test_mark_global_upgrade_held_packages(self): |
3647 | """ |
3648 | If a package that is on hold is marked for upgrade, |
3649 | diff --git a/landscape/lib/apt/package/tests/test_skeleton.py b/landscape/lib/apt/package/tests/test_skeleton.py |
3650 | index 63737b5..a240d10 100644 |
3651 | --- a/landscape/lib/apt/package/tests/test_skeleton.py |
3652 | +++ b/landscape/lib/apt/package/tests/test_skeleton.py |
3653 | @@ -1,3 +1,4 @@ |
3654 | +import locale |
3655 | import unittest |
3656 | |
3657 | from landscape.lib import testing |
3658 | @@ -147,15 +148,22 @@ class SkeletonAptTest(BaseTestCase): |
3659 | def test_build_skeleton_with_unicode_and_non_ascii(self): |
3660 | """ |
3661 | If with_unicode and with_info are passed to build_skeleton_apt, |
3662 | - the description is decoded and non-ascii chars replaced. |
3663 | + the description is decoded. |
3664 | """ |
3665 | + # Py2 used to convert to lossy ascii (thus LC_ in Makefile) |
3666 | + # Py3 doesn't, and python3-apt assumes UTF8 (LP: #1827857). |
3667 | + # If you revisit this test, also check reporter.main(), which |
3668 | + # should set this globally to the reporter process. |
3669 | + locale.setlocale(locale.LC_CTYPE, "C.UTF-8") |
3670 | + self.addCleanup(locale.resetlocale) |
3671 | + |
3672 | self._add_package_to_deb_dir( |
3673 | self.skeleton_repository_dir, "pkg", description=u"T\xe9st") |
3674 | self.facade._cache.update(None) |
3675 | self.facade._cache.open(None) |
3676 | pkg = self.get_package("pkg") |
3677 | skeleton = build_skeleton_apt(pkg, with_unicode=True, with_info=True) |
3678 | - self.assertEqual(u"T?st", skeleton.description) |
3679 | + self.assertEqual(u"T\u00E9st", skeleton.description) |
3680 | |
3681 | def test_build_skeleton_minimal(self): |
3682 | """ |
3683 | diff --git a/landscape/lib/backoff.py b/landscape/lib/backoff.py |
3684 | new file mode 100644 |
3685 | index 0000000..b2c7ac3 |
3686 | --- /dev/null |
3687 | +++ b/landscape/lib/backoff.py |
3688 | @@ -0,0 +1,53 @@ |
3689 | +import random |
3690 | + |
3691 | + |
3692 | +class ExponentialBackoff: |
3693 | + """ |
3694 | + Keeps track of a backoff delay that staggers down and staggers up |
3695 | + exponentially. |
3696 | + """ |
3697 | + |
3698 | + def __init__(self, start_delay, max_delay): |
3699 | + |
3700 | + self._error_count = 0 # A tally of server errors |
3701 | + |
3702 | + self._start_delay = start_delay |
3703 | + self._max_delay = max_delay |
3704 | + |
3705 | + def decrease(self): |
3706 | + """Decreases error count with zero being the lowest""" |
3707 | + self._error_count -= 1 |
3708 | + self._error_count = max(self._error_count, 0) |
3709 | + |
3710 | + def increase(self): |
3711 | + """Increases error count but not higher than gives the max delay""" |
3712 | + if self.get_delay() < self._max_delay: |
3713 | + self._error_count += 1 |
3714 | + |
3715 | + def get_delay(self): |
3716 | + """ |
3717 | + Calculates the delay using formula that gives this chart. In this |
3718 | + specific example start is 5 seconds and max is 60 seconds |
3719 | + Count Delay |
3720 | + 0 0 |
3721 | + 1 5 |
3722 | + 2 10 |
3723 | + 3 20 |
3724 | + 4 40 |
3725 | + 5 60 (max) |
3726 | + """ |
3727 | + if self._error_count: |
3728 | + delay = (2 ** (self._error_count - 1)) * self._start_delay |
3729 | + else: |
3730 | + delay = 0 |
3731 | + return min(int(delay), self._max_delay) |
3732 | + |
3733 | + def get_random_delay(self, stagger_fraction=0.25): |
3734 | + """ |
3735 | + Adds randomness to the specified stagger of the delay. For example |
3736 | + for a delay of 12 and 25% stagger, it works out to 9 + rand(0,3) |
3737 | + """ |
3738 | + delay = self.get_delay() |
3739 | + non_random_part = delay * (1-stagger_fraction) |
3740 | + random_part = delay * stagger_fraction * random.random() |
3741 | + return int(non_random_part + random_part) |
3742 | diff --git a/landscape/lib/base64.py b/landscape/lib/base64.py |
3743 | new file mode 100644 |
3744 | index 0000000..148a2d5 |
3745 | --- /dev/null |
3746 | +++ b/landscape/lib/base64.py |
3747 | @@ -0,0 +1,8 @@ |
3748 | +from __future__ import absolute_import |
3749 | + |
3750 | +from landscape.lib.compat import _PY3 |
3751 | + |
3752 | +if _PY3: |
3753 | + from base64 import decodebytes # noqa |
3754 | +else: |
3755 | + from base64 import decodestring as decodebytes # noqa |
3756 | diff --git a/landscape/lib/bpickle.py b/landscape/lib/bpickle.py |
3757 | index 93d5760..4f0d5ca 100644 |
3758 | --- a/landscape/lib/bpickle.py |
3759 | +++ b/landscape/lib/bpickle.py |
3760 | @@ -32,7 +32,7 @@ This file is modified from the original to work with python3, but should be |
3761 | wire compatible and behave the same way (bugs notwithstanding). |
3762 | """ |
3763 | |
3764 | -from twisted.python.compat import _PY3 |
3765 | +from landscape.lib.compat import long, _PY3 |
3766 | |
3767 | dumps_table = {} |
3768 | loads_table = {} |
3769 | diff --git a/landscape/lib/compat.py b/landscape/lib/compat.py |
3770 | index 115f86a..2452524 100644 |
3771 | --- a/landscape/lib/compat.py |
3772 | +++ b/landscape/lib/compat.py |
3773 | @@ -1,6 +1,6 @@ |
3774 | # flake8: noqa |
3775 | |
3776 | -from twisted.python.compat import _PY3 |
3777 | +_PY3 = str != bytes |
3778 | |
3779 | |
3780 | if _PY3: |
3781 | @@ -13,6 +13,8 @@ if _PY3: |
3782 | from io import StringIO |
3783 | stringio = cstringio = StringIO |
3784 | from builtins import input |
3785 | + unicode = str |
3786 | + long = int |
3787 | |
3788 | else: |
3789 | import cPickle |
3790 | @@ -24,3 +26,5 @@ else: |
3791 | stringio = StringIO |
3792 | from cStringIO import StringIO as cstringio |
3793 | input = raw_input |
3794 | + long = long |
3795 | + unicode = unicode |
3796 | diff --git a/landscape/lib/config.py b/landscape/lib/config.py |
3797 | index 7893649..5880876 100644 |
3798 | --- a/landscape/lib/config.py |
3799 | +++ b/landscape/lib/config.py |
3800 | @@ -228,7 +228,7 @@ class BaseConfiguration(object): |
3801 | raise_errors=False, write_empty_values=True) |
3802 | except ConfigObjError as e: |
3803 | logger = getLogger() |
3804 | - logger.warn(str(e)) |
3805 | + logger.warn("ERROR at {}: {}".format(config_source, str(e))) |
3806 | # Good configuration values are recovered here |
3807 | config_obj = e.config |
3808 | return config_obj |
3809 | diff --git a/landscape/lib/disk.py b/landscape/lib/disk.py |
3810 | index 7bf5884..2a8ad35 100644 |
3811 | --- a/landscape/lib/disk.py |
3812 | +++ b/landscape/lib/disk.py |
3813 | @@ -4,13 +4,13 @@ import os |
3814 | import re |
3815 | import codecs |
3816 | |
3817 | -from twisted.python.compat import _PY3 |
3818 | +from landscape.lib.compat import _PY3 |
3819 | |
3820 | |
3821 | # List of filesystem types authorized when generating disk use statistics. |
3822 | STABLE_FILESYSTEMS = frozenset( |
3823 | ["ext", "ext2", "ext3", "ext4", "reiserfs", "ntfs", "msdos", "dos", "vfat", |
3824 | - "xfs", "hpfs", "jfs", "ufs", "hfs", "hfsplus", "simfs"]) |
3825 | + "xfs", "hpfs", "jfs", "ufs", "hfs", "hfsplus", "simfs", "drvfs", "lxfs"]) |
3826 | |
3827 | |
3828 | EXTRACT_DEVICE = re.compile("([a-z]+)[0-9]*") |
3829 | diff --git a/landscape/lib/gpg.py b/landscape/lib/gpg.py |
3830 | index 66400d1..58218c5 100644 |
3831 | --- a/landscape/lib/gpg.py |
3832 | +++ b/landscape/lib/gpg.py |
3833 | @@ -1,6 +1,9 @@ |
3834 | +import itertools |
3835 | import shutil |
3836 | import tempfile |
3837 | |
3838 | +from glob import glob |
3839 | + |
3840 | from twisted.internet.utils import getProcessOutputAndValue |
3841 | |
3842 | |
3843 | @@ -8,14 +11,15 @@ class InvalidGPGSignature(Exception): |
3844 | """Raised when the gpg signature for a given file is invalid.""" |
3845 | |
3846 | |
3847 | -def gpg_verify(filename, signature, gpg="/usr/bin/gpg"): |
3848 | +def gpg_verify(filename, signature, gpg="/usr/bin/gpg", apt_dir="/etc/apt"): |
3849 | """Verify the GPG signature of a file. |
3850 | |
3851 | @param filename: Path to the file to verify the signature against. |
3852 | @param signature: Path to signature to use. |
3853 | @param gpg: Optionally, path to the GPG binary to use. |
3854 | + @param apt_dir: Optionally, path to apt trusted keyring. |
3855 | @return: a C{Deferred} resulting in C{True} if the signature is |
3856 | - valid, C{False} otherwise. |
3857 | + valid, C{False} otherwise. |
3858 | """ |
3859 | |
3860 | def remove_gpg_home(ignored): |
3861 | @@ -32,9 +36,17 @@ def gpg_verify(filename, signature, gpg="/usr/bin/gpg"): |
3862 | "code='%d')" % (gpg, out, err, code)) |
3863 | |
3864 | gpg_home = tempfile.mkdtemp() |
3865 | - args = ("--no-options", "--homedir", gpg_home, "--no-default-keyring", |
3866 | - "--ignore-time-conflict", "--keyring", "/etc/apt/trusted.gpg", |
3867 | - "--verify", signature, filename) |
3868 | + keyrings = tuple(itertools.chain(*[ |
3869 | + ("--keyring", keyring) |
3870 | + for keyring in sorted( |
3871 | + glob("{}/trusted.gpg".format(apt_dir)) + |
3872 | + glob("{}/trusted.gpg.d/*.gpg".format(apt_dir)) |
3873 | + ) |
3874 | + ])) |
3875 | + args = ( |
3876 | + "--no-options", "--homedir", gpg_home, "--no-default-keyring", |
3877 | + "--ignore-time-conflict" |
3878 | + ) + keyrings + ("--verify", signature, filename) |
3879 | |
3880 | result = getProcessOutputAndValue(gpg, args=args) |
3881 | result.addBoth(remove_gpg_home) |
3882 | diff --git a/landscape/lib/lsb_release.py b/landscape/lib/lsb_release.py |
3883 | index 6e05fd8..cfc3cd3 100644 |
3884 | --- a/landscape/lib/lsb_release.py |
3885 | +++ b/landscape/lib/lsb_release.py |
3886 | @@ -1,26 +1,59 @@ |
3887 | -"""Get information from /etc/lsb_release.""" |
3888 | +"""Get information from /usr/bin/lsb_release.""" |
3889 | +import os |
3890 | +from subprocess import CalledProcessError, check_output |
3891 | |
3892 | -LSB_RELEASE_FILENAME = "/etc/lsb-release" |
3893 | -LSB_RELEASE_INFO_KEYS = {"DISTRIB_ID": "distributor-id", |
3894 | - "DISTRIB_DESCRIPTION": "description", |
3895 | - "DISTRIB_RELEASE": "release", |
3896 | - "DISTRIB_CODENAME": "code-name"} |
3897 | +LSB_RELEASE = "/usr/bin/lsb_release" |
3898 | +LSB_RELEASE_FILENAME = "/etc/lsb_release" |
3899 | +LSB_RELEASE_FILE_KEYS = { |
3900 | + "DISTRIB_ID": "distributor-id", |
3901 | + "DISTRIB_DESCRIPTION": "description", |
3902 | + "DISTRIB_RELEASE": "release", |
3903 | + "DISTRIB_CODENAME": "code-name", |
3904 | +} |
3905 | |
3906 | |
3907 | -def parse_lsb_release(lsb_release_filename): |
3908 | - """Return a C{dict} holding information about the system LSB release. |
3909 | +def parse_lsb_release(lsb_release_filename=None): |
3910 | + """ |
3911 | + Returns a C{dict} holding information about the system LSB release. |
3912 | + Reads from C{lsb_release_filename} if it exists, else calls |
3913 | + C{LSB_RELEASE} |
3914 | + """ |
3915 | + if lsb_release_filename and os.path.exists(lsb_release_filename): |
3916 | + return parse_lsb_release_file(lsb_release_filename) |
3917 | + |
3918 | + with open(os.devnull, 'w') as FNULL: |
3919 | + try: |
3920 | + lsb_info = check_output([LSB_RELEASE, "-as"], stderr=FNULL) |
3921 | + except (CalledProcessError, FileNotFoundError): |
3922 | + # Fall back to reading file, even if it doesn't exist. |
3923 | + return parse_lsb_release_file(lsb_release_filename) |
3924 | + else: |
3925 | + dist, desc, release, code_name, _ = lsb_info.decode().split("\n") |
3926 | + |
3927 | + return { |
3928 | + "distributor-id": dist, |
3929 | + "release": release, |
3930 | + "code-name": code_name, |
3931 | + "description": desc, |
3932 | + } |
3933 | + |
3934 | |
3935 | - @raises: An IOError exception if C{lsb_release_filename} could not be read. |
3936 | +def parse_lsb_release_file(filename): |
3937 | + """ |
3938 | + Returns a C{dict} holding information about the system LSB release |
3939 | + by attempting to parse C{filename}. |
3940 | + |
3941 | + @raises: A FileNotFoundError if C{filename} does not exist. |
3942 | """ |
3943 | - fd = open(lsb_release_filename, "r") |
3944 | info = {} |
3945 | - try: |
3946 | + |
3947 | + with open(filename) as fd: |
3948 | for line in fd: |
3949 | key, value = line.split("=") |
3950 | - if key in LSB_RELEASE_INFO_KEYS: |
3951 | - key = LSB_RELEASE_INFO_KEYS[key.strip()] |
3952 | + |
3953 | + if key in LSB_RELEASE_FILE_KEYS: |
3954 | + key = LSB_RELEASE_FILE_KEYS[key.strip()] |
3955 | value = value.strip().strip('"') |
3956 | info[key] = value |
3957 | - finally: |
3958 | - fd.close() |
3959 | + |
3960 | return info |
3961 | diff --git a/landscape/lib/message.py b/landscape/lib/message.py |
3962 | index ed742af..7ed55da 100644 |
3963 | --- a/landscape/lib/message.py |
3964 | +++ b/landscape/lib/message.py |
3965 | @@ -1,6 +1,6 @@ |
3966 | """Helpers for reliable persistent message queues.""" |
3967 | |
3968 | -ANCIENT = 1 |
3969 | +RESYNC = object() # Used as a flag indicating a resync is needed |
3970 | |
3971 | |
3972 | def got_next_expected(store, next_expected): |
3973 | @@ -10,6 +10,12 @@ def got_next_expected(store, next_expected): |
3974 | wants next; this will do various things based on what *this* side |
3975 | has in its outbound queue store. |
3976 | |
3977 | + 0. The peer expects a sequence number from the server that is too high, the |
3978 | + difference greater than pending messages in the peer. We flush the older |
3979 | + messages in queue b/c the server does not want old or ancient messages, |
3980 | + however we also reset the offset so that the pending messages are |
3981 | + resent. Then we resynchronize by returning RESYNC. See LP: #1917540 |
3982 | + |
3983 | 1. The peer expects a sequence greater than what we last |
3984 | sent. This is the common case and generally it should be |
3985 | expecting last_sent_sequence+len(messages_sent)+1. |
3986 | @@ -25,18 +31,22 @@ def got_next_expected(store, next_expected): |
3987 | from that message. |
3988 | |
3989 | If the next expected sequence from the server refers to a message |
3990 | - older than we have, then L{ANCIENT} will be returned. |
3991 | + older than we have, then L{RESYNC} will be returned. |
3992 | """ |
3993 | ret = None |
3994 | old_sequence = store.get_sequence() |
3995 | - if next_expected > old_sequence: |
3996 | + if (next_expected - old_sequence) > store.count_pending_messages(): |
3997 | + store.delete_old_messages() # Flush queue from previous iteration |
3998 | + pending_offset = 0 # This means current messages will be resent |
3999 | + ret = RESYNC |
4000 | + elif next_expected > old_sequence: |
4001 | store.delete_old_messages() |
4002 | pending_offset = next_expected - old_sequence |
4003 | elif next_expected < (old_sequence - store.get_pending_offset()): |
4004 | # "Ancient": The other side wants messages we don't have, |
4005 | # so let's just reset our counter to what it expects. |
4006 | pending_offset = 0 |
4007 | - ret = ANCIENT |
4008 | + ret = RESYNC |
4009 | else: |
4010 | # No messages transferred, or |
4011 | # "Old": We'll try to send these old messages that the |
4012 | diff --git a/landscape/lib/network.py b/landscape/lib/network.py |
4013 | index 0ee336a..c9801c2 100644 |
4014 | --- a/landscape/lib/network.py |
4015 | +++ b/landscape/lib/network.py |
4016 | @@ -11,7 +11,7 @@ import errno |
4017 | import logging |
4018 | |
4019 | import netifaces |
4020 | -from twisted.python.compat import long |
4021 | +from landscape.lib.compat import long, _PY3 |
4022 | |
4023 | __all__ = ["get_active_device_info", "get_network_traffic"] |
4024 | |
4025 | @@ -35,32 +35,14 @@ def is_up(flags): |
4026 | return flags & 1 |
4027 | |
4028 | |
4029 | -def get_active_interfaces(): |
4030 | - """Generator yields (active network interface name, address data) tuples. |
4031 | +def is_active(ifaddresses): |
4032 | + """Checks if interface address data has an IP address |
4033 | |
4034 | - Address data is formatted exactly like L{netifaces.ifaddresses}, e.g.:: |
4035 | - |
4036 | - ('eth0', { |
4037 | - AF_LINK: [ |
4038 | - {'addr': '...', 'broadcast': '...'}, ], |
4039 | - AF_INET: [ |
4040 | - {'addr': '...', 'broadcast': '...', 'netmask': '...'}, |
4041 | - {'addr': '...', 'broadcast': '...', 'netmask': '...'}, |
4042 | - ...], |
4043 | - AF_INET6: [ |
4044 | - {'addr': '...', 'netmask': '...'}, |
4045 | - {'addr': '...', 'netmask': '...'}, |
4046 | - ...], }) |
4047 | - |
4048 | - Interfaces with no IP address are ignored. |
4049 | + @param ifaddresses: a dict as returned by L{netifaces.ifaddresses} |
4050 | """ |
4051 | - for interface in netifaces.interfaces(): |
4052 | - ifaddresses = netifaces.ifaddresses(interface) |
4053 | - # Skip interfaces with no IPv4 or IPv6 addresses. |
4054 | - inet_addr = ifaddresses.get(netifaces.AF_INET, [{}])[0].get('addr') |
4055 | - inet6_addr = ifaddresses.get(netifaces.AF_INET6, [{}])[0].get('addr') |
4056 | - if inet_addr or inet6_addr: |
4057 | - yield interface, ifaddresses |
4058 | + inet_addr = ifaddresses.get(netifaces.AF_INET, [{}])[0].get('addr') |
4059 | + inet6_addr = ifaddresses.get(netifaces.AF_INET6, [{}])[0].get('addr') |
4060 | + return bool(inet_addr or inet6_addr) |
4061 | |
4062 | |
4063 | def get_ip_addresses(ifaddresses): |
4064 | @@ -69,8 +51,7 @@ def get_ip_addresses(ifaddresses): |
4065 | Returns the same structure as L{ifaddresses}, but filtered to keep |
4066 | IP addresses only. |
4067 | |
4068 | - @param ifaddresses: a dict as returned by L{netifaces.ifaddresses} or |
4069 | - the address data in L{get_active_interfaces}'s output. |
4070 | + @param ifaddresses: a dict as returned by L{netifaces.ifaddresses} |
4071 | """ |
4072 | results = {} |
4073 | if netifaces.AF_INET in ifaddresses: |
4074 | @@ -88,8 +69,7 @@ def get_ip_addresses(ifaddresses): |
4075 | def get_broadcast_address(ifaddresses): |
4076 | """Return the broadcast address associated to an interface. |
4077 | |
4078 | - @param ifaddresses: a dict as returned by L{netifaces.ifaddresses} or |
4079 | - the address data in L{get_active_interfaces}'s output. |
4080 | + @param ifaddresses: a dict as returned by L{netifaces.ifaddresses} |
4081 | """ |
4082 | return ifaddresses[netifaces.AF_INET][0].get('broadcast', '0.0.0.0') |
4083 | |
4084 | @@ -97,8 +77,7 @@ def get_broadcast_address(ifaddresses): |
4085 | def get_netmask(ifaddresses): |
4086 | """Return the network mask associated to an interface. |
4087 | |
4088 | - @param ifaddresses: a dict as returned by L{netifaces.ifaddresses} or |
4089 | - the address data in L{get_active_interfaces}'s output. |
4090 | + @param ifaddresses: a dict as returned by L{netifaces.ifaddresses} |
4091 | """ |
4092 | return ifaddresses[netifaces.AF_INET][0].get('netmask', '') |
4093 | |
4094 | @@ -106,8 +85,7 @@ def get_netmask(ifaddresses): |
4095 | def get_ip_address(ifaddresses): |
4096 | """Return the first IPv4 address associated to the interface. |
4097 | |
4098 | - @param ifaddresses: a dict as returned by L{netifaces.ifaddresses} or |
4099 | - the address data in L{get_active_interfaces}'s output. |
4100 | + @param ifaddresses: a dict as returned by L{netifaces.ifaddresses} |
4101 | """ |
4102 | return ifaddresses[netifaces.AF_INET][0]['addr'] |
4103 | |
4104 | @@ -118,8 +96,7 @@ def get_mac_address(ifaddresses): |
4105 | ie. six colon separated groups of two hexadecimal digits, if available; |
4106 | otherwise an empty string. |
4107 | |
4108 | - @param ifaddresses: a dict as returned by L{netifaces.ifaddresses} or |
4109 | - the address data in L{get_active_interfaces}'s output. |
4110 | + @param ifaddresses: a dict as returned by L{netifaces.ifaddresses} |
4111 | """ |
4112 | if netifaces.AF_LINK in ifaddresses: |
4113 | return ifaddresses[netifaces.AF_LINK][0].get('addr', '') |
4114 | @@ -138,51 +115,104 @@ def get_flags(sock, interface): |
4115 | return struct.unpack("H", data[16:18])[0] |
4116 | |
4117 | |
4118 | -def get_active_device_info(skipped_interfaces=("lo",), |
4119 | - skip_vlan=True, skip_alias=True, extended=False): |
4120 | +def get_default_interfaces(): |
4121 | + """ |
4122 | + Returns a list of interfaces with default routes |
4123 | + """ |
4124 | + default_table = netifaces.gateways()['default'] |
4125 | + interfaces = [gateway[1] for gateway in default_table.values()] |
4126 | + return interfaces |
4127 | + |
4128 | + |
4129 | +def get_filtered_if_info(filters=(), extended=False): |
4130 | """ |
4131 | - Returns a dictionary containing information on each active network |
4132 | - interface present on a machine. |
4133 | + Returns a dictionary containing info on each active network |
4134 | + interface that passes all `filters`. |
4135 | + |
4136 | + A filter is a callable that returns True if the interface should be |
4137 | + skipped. |
4138 | """ |
4139 | results = [] |
4140 | + |
4141 | try: |
4142 | sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, |
4143 | socket.IPPROTO_IP) |
4144 | - for interface, ifaddresses in get_active_interfaces(): |
4145 | - if interface in skipped_interfaces: |
4146 | - continue |
4147 | - if skip_vlan and "." in interface: |
4148 | + |
4149 | + for interface in netifaces.interfaces(): |
4150 | + if any(f(interface) for f in filters): |
4151 | continue |
4152 | - if skip_alias and ":" in interface: |
4153 | + |
4154 | + ifaddresses = netifaces.ifaddresses(interface) |
4155 | + if not is_active(ifaddresses): |
4156 | continue |
4157 | - flags = get_flags(sock, interface.encode()) |
4158 | + |
4159 | + ifencoded = interface.encode() |
4160 | + flags = get_flags(sock, ifencoded) |
4161 | if not is_up(flags): |
4162 | continue |
4163 | - interface_info = {"interface": interface} |
4164 | - interface_info["flags"] = flags |
4165 | - speed, duplex = get_network_interface_speed( |
4166 | - sock, interface.encode()) |
4167 | - interface_info["speed"] = speed |
4168 | - interface_info["duplex"] = duplex |
4169 | + |
4170 | ip_addresses = get_ip_addresses(ifaddresses) |
4171 | + if not extended and netifaces.AF_INET not in ip_addresses: |
4172 | + # Skip interfaces with no IPv4 addr unless extended to |
4173 | + # keep backwards compatibility with single-IPv4 addr |
4174 | + # support. |
4175 | + continue |
4176 | + |
4177 | + ifinfo = {"interface": interface} |
4178 | + ifinfo["flags"] = flags |
4179 | + ifinfo["speed"], ifinfo["duplex"] = get_network_interface_speed( |
4180 | + sock, ifencoded) |
4181 | + |
4182 | if extended: |
4183 | - interface_info["ip_addresses"] = ip_addresses |
4184 | + ifinfo["ip_addresses"] = ip_addresses |
4185 | + |
4186 | if netifaces.AF_INET in ip_addresses: |
4187 | - interface_info["ip_address"] = get_ip_address(ifaddresses) |
4188 | - interface_info["mac_address"] = get_mac_address(ifaddresses) |
4189 | - interface_info["broadcast_address"] = get_broadcast_address( |
4190 | + ifinfo["ip_address"] = get_ip_address(ifaddresses) |
4191 | + ifinfo["mac_address"] = get_mac_address(ifaddresses) |
4192 | + ifinfo["broadcast_address"] = get_broadcast_address( |
4193 | ifaddresses) |
4194 | - interface_info["netmask"] = get_netmask(ifaddresses) |
4195 | - # Skip interfaces with no IPv4 address in non-extended mode |
4196 | - # to keep backwards compatibility with single-IPv4 addr support. |
4197 | - if netifaces.AF_INET in ip_addresses or extended: |
4198 | - results.append(interface_info) |
4199 | + ifinfo["netmask"] = get_netmask(ifaddresses) |
4200 | + |
4201 | + results.append(ifinfo) |
4202 | finally: |
4203 | - del sock |
4204 | + sock.close() |
4205 | |
4206 | return results |
4207 | |
4208 | |
4209 | +def get_active_device_info(skipped_interfaces=("lo",), |
4210 | + skip_vlan=True, skip_alias=True, |
4211 | + extended=False, default_only=False): |
4212 | + def filter_local(interface): |
4213 | + return interface in skipped_interfaces |
4214 | + |
4215 | + def filter_vlan(interface): |
4216 | + return "." in interface |
4217 | + |
4218 | + def filter_alias(interface): |
4219 | + return ":" in interface |
4220 | + |
4221 | + # Get default interfaces here because it could be expensive and |
4222 | + # there's no reason to do it more than once. |
4223 | + default_ifs = get_default_interfaces() |
4224 | + |
4225 | + def filter_default(interface): |
4226 | + return default_only and interface not in default_ifs |
4227 | + |
4228 | + # Tap interfaces can be extremely numerous, slowing us down |
4229 | + # significantly. |
4230 | + def filter_tap(interface): |
4231 | + return interface.startswith("tap") |
4232 | + |
4233 | + return get_filtered_if_info(filters=( |
4234 | + filter_tap, |
4235 | + filter_local, |
4236 | + filter_vlan, |
4237 | + filter_alias, |
4238 | + filter_default, |
4239 | + ), extended=extended) |
4240 | + |
4241 | + |
4242 | def get_network_traffic(source_file="/proc/net/dev"): |
4243 | """ |
4244 | Retrieves an array of information regarding the network activity per |
4245 | @@ -246,13 +276,16 @@ def get_network_interface_speed(sock, interface_name): |
4246 | speed = -1 |
4247 | try: |
4248 | fcntl.ioctl(sock, SIOCETHTOOL, packed) # Status ioctl() call |
4249 | - res = status_cmd.tostring() |
4250 | + if _PY3: |
4251 | + res = status_cmd.tobytes() |
4252 | + else: |
4253 | + res = status_cmd.tostring() |
4254 | speed, duplex = struct.unpack("12xHB28x", res) |
4255 | - except IOError as e: |
4256 | + except (IOError, OSError) as e: |
4257 | if e.errno == errno.EPERM: |
4258 | - logging.warn("Could not determine network interface speed, " |
4259 | - "operation not permitted.") |
4260 | - elif e.errno != errno.EOPNOTSUPP: |
4261 | + logging.warning("Could not determine network interface speed, " |
4262 | + "operation not permitted.") |
4263 | + elif e.errno != errno.EOPNOTSUPP and e.errno != errno.EINVAL: |
4264 | raise e |
4265 | speed = -1 |
4266 | duplex = False |
4267 | diff --git a/landscape/lib/schema.py b/landscape/lib/schema.py |
4268 | index e2d65d0..f9397db 100644 |
4269 | --- a/landscape/lib/schema.py |
4270 | +++ b/landscape/lib/schema.py |
4271 | @@ -13,6 +13,12 @@ class Constant(object): |
4272 | self.value = value |
4273 | |
4274 | def coerce(self, value): |
4275 | + if isinstance(self.value, str) and isinstance(value, bytes): |
4276 | + try: |
4277 | + value = value.decode() |
4278 | + except UnicodeDecodeError: |
4279 | + pass |
4280 | + |
4281 | if value != self.value: |
4282 | raise InvalidError("%r != %r" % (value, self.value)) |
4283 | return value |
4284 | @@ -65,11 +71,19 @@ class Float(object): |
4285 | |
4286 | |
4287 | class Bytes(object): |
4288 | - """A binary string.""" |
4289 | + """A binary string. |
4290 | + |
4291 | + If the value is a Python3 str (unicode), it will be automatically |
4292 | + encoded. |
4293 | + """ |
4294 | def coerce(self, value): |
4295 | - if not isinstance(value, bytes): |
4296 | - raise InvalidError("%r isn't a bytestring" % (value,)) |
4297 | - return value |
4298 | + if isinstance(value, bytes): |
4299 | + return value |
4300 | + |
4301 | + if isinstance(value, str): |
4302 | + return value.encode() |
4303 | + |
4304 | + raise InvalidError("%r isn't a bytestring" % value) |
4305 | |
4306 | |
4307 | class Unicode(object): |
4308 | diff --git a/landscape/lib/testing.py b/landscape/lib/testing.py |
4309 | index c024055..101ca97 100644 |
4310 | --- a/landscape/lib/testing.py |
4311 | +++ b/landscape/lib/testing.py |
4312 | @@ -15,7 +15,7 @@ import unittest |
4313 | from logging import Handler, ERROR, Formatter |
4314 | from twisted.trial.unittest import TestCase |
4315 | from twisted.python.compat import StringType as basestring |
4316 | -from twisted.python.compat import _PY3 |
4317 | +from landscape.lib.compat import _PY3 |
4318 | from twisted.python.failure import Failure |
4319 | from twisted.internet.defer import Deferred |
4320 | from twisted.internet.error import ConnectError |
4321 | diff --git a/landscape/lib/tests/test_backoff.py b/landscape/lib/tests/test_backoff.py |
4322 | new file mode 100644 |
4323 | index 0000000..18de645 |
4324 | --- /dev/null |
4325 | +++ b/landscape/lib/tests/test_backoff.py |
4326 | @@ -0,0 +1,37 @@ |
4327 | +from landscape.lib.backoff import ExponentialBackoff |
4328 | +from landscape.client.tests.helpers import LandscapeTest |
4329 | + |
4330 | + |
4331 | +class TestBackoff(LandscapeTest): |
4332 | + |
4333 | + def test_increase(self): |
4334 | + '''Test the delay values start correctly and double''' |
4335 | + backoff_counter = ExponentialBackoff(5, 10) |
4336 | + backoff_counter.increase() |
4337 | + self.assertEqual(backoff_counter.get_delay(), 5) |
4338 | + backoff_counter.increase() |
4339 | + self.assertEqual(backoff_counter.get_delay(), 10) |
4340 | + |
4341 | + def test_min(self): |
4342 | + '''Test the count and the delay never go below zero''' |
4343 | + backoff_counter = ExponentialBackoff(1, 5) |
4344 | + for _ in range(10): |
4345 | + backoff_counter.decrease() |
4346 | + self.assertEqual(backoff_counter.get_delay(), 0) |
4347 | + self.assertEqual(backoff_counter.get_random_delay(), 0) |
4348 | + self.assertEqual(backoff_counter._error_count, 0) |
4349 | + |
4350 | + def test_max(self): |
4351 | + '''Test the delay never goes above max''' |
4352 | + backoff_counter = ExponentialBackoff(1, 5) |
4353 | + for _ in range(10): |
4354 | + backoff_counter.increase() |
4355 | + self.assertEqual(backoff_counter.get_delay(), 5) |
4356 | + |
4357 | + def test_decreased_when_maxed(self): |
4358 | + '''Test the delay goes down one step when maxed''' |
4359 | + backoff_counter = ExponentialBackoff(1, 5) |
4360 | + for _ in range(10): |
4361 | + backoff_counter.increase() |
4362 | + backoff_counter.decrease() |
4363 | + self.assertTrue(backoff_counter.get_delay() < 5) |
4364 | diff --git a/landscape/lib/tests/test_config.py b/landscape/lib/tests/test_config.py |
4365 | index ee48727..3fbee3f 100644 |
4366 | --- a/landscape/lib/tests/test_config.py |
4367 | +++ b/landscape/lib/tests/test_config.py |
4368 | @@ -338,8 +338,10 @@ class BaseConfigurationTest(ConfigTestCase, HelperTestCase, unittest.TestCase): |
4369 | """)) |
4370 | self.config.load_configuration_file(filename) |
4371 | self.assertEqual(self.config.whatever, "spam") |
4372 | - self.assertIn("WARNING: Duplicate keyword name at line 4.", |
4373 | - self.logfile.getvalue()) |
4374 | + self.assertIn( |
4375 | + f"WARNING: ERROR at {filename}: Duplicate keyword name at line 4.", |
4376 | + self.logfile.getvalue(), |
4377 | + ) |
4378 | |
4379 | def test_duplicate_key(self): |
4380 | """ |
4381 | @@ -354,8 +356,10 @@ class BaseConfigurationTest(ConfigTestCase, HelperTestCase, unittest.TestCase): |
4382 | filename = self.makeFile(config) |
4383 | self.config.load_configuration_file(filename) |
4384 | self.assertEqual(self.config.computer_title, "frog") |
4385 | - self.assertIn("WARNING: Duplicate keyword name at line 4.", |
4386 | - self.logfile.getvalue()) |
4387 | + self.assertIn( |
4388 | + f"WARNING: ERROR at {filename}: Duplicate keyword name at line 4.", |
4389 | + self.logfile.getvalue(), |
4390 | + ) |
4391 | |
4392 | def test_triplicate_key(self): |
4393 | """ |
4394 | @@ -372,8 +376,13 @@ class BaseConfigurationTest(ConfigTestCase, HelperTestCase, unittest.TestCase): |
4395 | self.config.load_configuration_file(filename) |
4396 | self.assertEqual(self.config.computer_title, "frog") |
4397 | logged = self.logfile.getvalue() |
4398 | - self.assertIn("WARNING: Parsing failed with several errors.", |
4399 | - logged) |
4400 | + self.assertIn( |
4401 | + ( |
4402 | + f"WARNING: ERROR at {filename}: Parsing failed with several " |
4403 | + "errors." |
4404 | + ), |
4405 | + logged, |
4406 | + ) |
4407 | self.assertIn("First error at line 4.", logged) |
4408 | |
4409 | def test_load_not_found_default_accept_missing(self): |
4410 | diff --git a/landscape/lib/tests/test_gpg.py b/landscape/lib/tests/test_gpg.py |
4411 | index e6165a2..4c84e00 100644 |
4412 | --- a/landscape/lib/tests/test_gpg.py |
4413 | +++ b/landscape/lib/tests/test_gpg.py |
4414 | @@ -1,9 +1,10 @@ |
4415 | import mock |
4416 | import os |
4417 | +import textwrap |
4418 | import unittest |
4419 | |
4420 | from twisted.internet import reactor |
4421 | -from twisted.internet.defer import Deferred |
4422 | +from twisted.internet.defer import Deferred, inlineCallbacks |
4423 | |
4424 | from landscape.lib import testing |
4425 | from landscape.lib.gpg import gpg_verify |
4426 | @@ -16,6 +17,8 @@ class GPGTest(testing.FSTestCase, testing.TwistedTestCase, unittest.TestCase): |
4427 | L{gpg_verify} runs the given gpg binary and returns C{True} if the |
4428 | provided signature is valid. |
4429 | """ |
4430 | + aptdir = self.makeDir() |
4431 | + os.mknod("{}/trusted.gpg".format(aptdir)) |
4432 | gpg_options = self.makeFile() |
4433 | gpg = self.makeFile("#!/bin/sh\n" |
4434 | "touch $3/trustdb.gpg\n" |
4435 | @@ -27,14 +30,15 @@ class GPGTest(testing.FSTestCase, testing.TwistedTestCase, unittest.TestCase): |
4436 | @mock.patch("tempfile.mkdtemp") |
4437 | def do_test(mkdtemp_mock): |
4438 | mkdtemp_mock.return_value = gpg_home |
4439 | - result = gpg_verify("/some/file", "/some/signature", gpg=gpg) |
4440 | + result = gpg_verify( |
4441 | + "/some/file", "/some/signature", gpg=gpg, apt_dir=aptdir) |
4442 | |
4443 | def check_result(ignored): |
4444 | self.assertEqual( |
4445 | open(gpg_options).read(), |
4446 | "--no-options --homedir %s --no-default-keyring " |
4447 | - "--ignore-time-conflict --keyring /etc/apt/trusted.gpg " |
4448 | - "--verify /some/signature /some/file" % gpg_home) |
4449 | + "--ignore-time-conflict --keyring %s/trusted.gpg " |
4450 | + "--verify /some/signature /some/file" % (gpg_home, aptdir)) |
4451 | self.assertFalse(os.path.exists(gpg_home)) |
4452 | |
4453 | result.addCallback(check_result) |
4454 | @@ -70,3 +74,38 @@ class GPGTest(testing.FSTestCase, testing.TwistedTestCase, unittest.TestCase): |
4455 | |
4456 | reactor.callWhenRunning(do_test) |
4457 | return deferred |
4458 | + |
4459 | + @inlineCallbacks |
4460 | + def test_gpg_verify_trusted_dir(self): |
4461 | + """ |
4462 | + gpg_verify uses keys from the trusted.gpg.d if such a folder exists. |
4463 | + """ |
4464 | + apt_dir = self.makeDir() |
4465 | + os.mkdir("{}/trusted.gpg.d".format(apt_dir)) |
4466 | + os.mknod("{}/trusted.gpg.d/foo.gpg".format(apt_dir)) |
4467 | + os.mknod("{}/trusted.gpg.d/baz.gpg".format(apt_dir)) |
4468 | + os.mknod("{}/trusted.gpg.d/bad.gpg~".format(apt_dir)) |
4469 | + |
4470 | + gpg_call = self.makeFile() |
4471 | + fake_gpg = self.makeFile(textwrap.dedent("""\ |
4472 | + #!/bin/sh |
4473 | + touch $3/trustdb.gpg |
4474 | + echo -n $@ > {} |
4475 | + """).format(gpg_call)) |
4476 | + os.chmod(fake_gpg, 0o755) |
4477 | + gpg_home = self.makeDir() |
4478 | + |
4479 | + with mock.patch("tempfile.mkdtemp", return_value=gpg_home): |
4480 | + yield gpg_verify( |
4481 | + "/some/file", "/some/signature", gpg=fake_gpg, apt_dir=apt_dir) |
4482 | + |
4483 | + expected = ( |
4484 | + "--no-options --homedir {gpg_home} --no-default-keyring " |
4485 | + "--ignore-time-conflict " |
4486 | + "--keyring {apt_dir}/trusted.gpg.d/baz.gpg " |
4487 | + "--keyring {apt_dir}/trusted.gpg.d/foo.gpg " |
4488 | + "--verify /some/signature /some/file" |
4489 | + ).format(gpg_home=gpg_home, apt_dir=apt_dir) |
4490 | + with open(gpg_call) as call: |
4491 | + self.assertEqual(expected, call.read()) |
4492 | + self.assertFalse(os.path.exists(gpg_home)) |
4493 | diff --git a/landscape/lib/tests/test_lsb_release.py b/landscape/lib/tests/test_lsb_release.py |
4494 | index c1a23a6..da1932d 100644 |
4495 | --- a/landscape/lib/tests/test_lsb_release.py |
4496 | +++ b/landscape/lib/tests/test_lsb_release.py |
4497 | @@ -1,4 +1,6 @@ |
4498 | import unittest |
4499 | +from subprocess import CalledProcessError |
4500 | +from unittest import mock |
4501 | |
4502 | from landscape.lib import testing |
4503 | from landscape.lib.lsb_release import parse_lsb_release |
4504 | @@ -7,6 +9,30 @@ from landscape.lib.lsb_release import parse_lsb_release |
4505 | class LsbReleaseTest(testing.FSTestCase, unittest.TestCase): |
4506 | |
4507 | def test_parse_lsb_release(self): |
4508 | + with mock.patch("landscape.lib.lsb_release.check_output") as co_mock: |
4509 | + co_mock.return_value = (b"Ubuntu\nUbuntu 22.04.1 LTS\n22.04\njammy" |
4510 | + b"\n") |
4511 | + lsb_release = parse_lsb_release() |
4512 | + |
4513 | + self.assertEqual(lsb_release, |
4514 | + {"distributor-id": "Ubuntu", |
4515 | + "description": "Ubuntu 22.04.1 LTS", |
4516 | + "release": "22.04", |
4517 | + "code-name": "jammy"}) |
4518 | + |
4519 | + def test_parse_lsb_release_debian(self): |
4520 | + with mock.patch("landscape.lib.lsb_release.check_output") as co_mock: |
4521 | + co_mock.return_value = (b"Debian\nDebian GNU/Linux 11 (bullseye)\n" |
4522 | + b"11\nbullseye\n") |
4523 | + lsb_release = parse_lsb_release() |
4524 | + |
4525 | + self.assertEqual(lsb_release, |
4526 | + {"distributor-id": "Debian", |
4527 | + "description": "Debian GNU/Linux 11 (bullseye)", |
4528 | + "release": "11", |
4529 | + "code-name": "bullseye"}) |
4530 | + |
4531 | + def test_parse_lsb_release_file(self): |
4532 | """ |
4533 | L{parse_lsb_release} returns a C{dict} holding information from |
4534 | the given LSB release file. |
4535 | @@ -17,18 +43,33 @@ class LsbReleaseTest(testing.FSTestCase, unittest.TestCase): |
4536 | "DISTRIB_DESCRIPTION=" |
4537 | "\"Ubuntu 6.06.1 LTS\"\n") |
4538 | |
4539 | - self.assertEqual(parse_lsb_release(lsb_release_filename), |
4540 | + with mock.patch("landscape.lib.lsb_release.check_output") as co_mock: |
4541 | + co_mock.side_effect = CalledProcessError(127, "") |
4542 | + lsb_release = parse_lsb_release(lsb_release_filename) |
4543 | + |
4544 | + self.assertEqual(lsb_release, |
4545 | {"distributor-id": "Ubuntu", |
4546 | "description": "Ubuntu 6.06.1 LTS", |
4547 | "release": "6.06", |
4548 | "code-name": "dapper"}) |
4549 | |
4550 | - def test_parse_lsb_release_with_missing_or_extra_fields(self): |
4551 | + def test_parse_lsb_release_file_with_missing_or_extra_fields(self): |
4552 | """ |
4553 | L{parse_lsb_release} ignores lines not matching the map of |
4554 | known keys, and returns only keys with an actual value. |
4555 | """ |
4556 | lsb_release_filename = self.makeFile("DISTRIB_ID=Ubuntu\n" |
4557 | "FOO=Bar\n") |
4558 | - self.assertEqual(parse_lsb_release(lsb_release_filename), |
4559 | - {"distributor-id": "Ubuntu"}) |
4560 | + |
4561 | + with mock.patch("landscape.lib.lsb_release.check_output") as co_mock: |
4562 | + co_mock.side_effect = CalledProcessError(127, "") |
4563 | + lsb_release = parse_lsb_release(lsb_release_filename) |
4564 | + |
4565 | + self.assertEqual(lsb_release, {"distributor-id": "Ubuntu"}) |
4566 | + |
4567 | + def test_parse_lsb_release_file_not_found(self): |
4568 | + with mock.patch("landscape.lib.lsb_release.check_output") as co_mock: |
4569 | + co_mock.side_effect = CalledProcessError(127, "") |
4570 | + |
4571 | + self.assertRaises(FileNotFoundError, parse_lsb_release, |
4572 | + "TheresNoWayThisFileExists") |
4573 | diff --git a/landscape/lib/tests/test_network.py b/landscape/lib/tests/test_network.py |
4574 | index 0b56971..27d2e20 100644 |
4575 | --- a/landscape/lib/tests/test_network.py |
4576 | +++ b/landscape/lib/tests/test_network.py |
4577 | @@ -1,7 +1,7 @@ |
4578 | import socket |
4579 | import unittest |
4580 | |
4581 | -from mock import patch, ANY, mock_open |
4582 | +from unittest.mock import ANY, DEFAULT, patch, mock_open |
4583 | from netifaces import ( |
4584 | AF_INET, |
4585 | AF_INET6, |
4586 | @@ -14,8 +14,8 @@ from subprocess import Popen, PIPE |
4587 | |
4588 | from landscape.lib import testing |
4589 | from landscape.lib.network import ( |
4590 | - get_network_traffic, get_active_device_info, get_active_interfaces, |
4591 | - get_fqdn, get_network_interface_speed, is_up) |
4592 | + get_active_device_info, get_filtered_if_info, get_fqdn, |
4593 | + get_network_interface_speed, get_network_traffic, is_up) |
4594 | |
4595 | |
4596 | class BaseTestCase(testing.HelperTestCase, unittest.TestCase): |
4597 | @@ -148,33 +148,62 @@ class NetworkInfoTest(BaseTestCase): |
4598 | 'duplex': True, |
4599 | 'ip_addresses': {AF_INET6: [{'addr': '2001::1'}]}}]) |
4600 | |
4601 | + @patch("landscape.lib.network.get_default_interfaces") |
4602 | + @patch("landscape.lib.network.get_network_interface_speed") |
4603 | + @patch("landscape.lib.network.get_flags") |
4604 | + @patch("landscape.lib.network.netifaces.ifaddresses") |
4605 | + @patch("landscape.lib.network.netifaces.interfaces") |
4606 | + def test_default_only_interface( |
4607 | + self, mock_interfaces, mock_ifaddresses, mock_get_flags, |
4608 | + mock_get_network_interface_speed, mock_get_default_interfaces): |
4609 | + default_iface = "test_iface" |
4610 | + mock_get_default_interfaces.return_value = [default_iface] |
4611 | + mock_get_network_interface_speed.return_value = (100, True) |
4612 | + mock_get_flags.return_value = 4163 |
4613 | + mock_interfaces.return_value = [default_iface] |
4614 | + mock_ifaddresses.return_value = {AF_INET: [{"addr": "192.168.0.50"}]} |
4615 | + device_info = get_active_device_info(extended=False, default_only=True) |
4616 | + interfaces = [i["interface"] for i in device_info] |
4617 | + self.assertIn(default_iface, interfaces) |
4618 | + self.assertEqual(len(interfaces), 1) |
4619 | + |
4620 | def test_skip_loopback(self): |
4621 | """The C{lo} interface is not reported by L{get_active_device_info}.""" |
4622 | device_info = get_active_device_info() |
4623 | interfaces = [i["interface"] for i in device_info] |
4624 | self.assertNotIn("lo", interfaces) |
4625 | |
4626 | - @patch("landscape.lib.network.get_active_interfaces") |
4627 | - def test_skip_vlan(self, mock_get_active_interfaces): |
4628 | + @patch("landscape.lib.network.netifaces.interfaces") |
4629 | + def test_skip_vlan(self, mock_interfaces): |
4630 | """VLAN interfaces are not reported by L{get_active_device_info}.""" |
4631 | - mock_get_active_interfaces.side_effect = lambda: ( |
4632 | - list(get_active_interfaces()) + [("eth0.1", {})]) |
4633 | + mock_interfaces.side_effect = lambda: ( |
4634 | + _interfaces() + ["eth0.1"]) |
4635 | device_info = get_active_device_info() |
4636 | - self.assertTrue(mock_get_active_interfaces.called) |
4637 | interfaces = [i["interface"] for i in device_info] |
4638 | self.assertNotIn("eth0.1", interfaces) |
4639 | |
4640 | - @patch("landscape.lib.network.get_active_interfaces") |
4641 | - def test_skip_alias(self, mock_get_active_interfaces): |
4642 | + @patch("landscape.lib.network.netifaces.interfaces") |
4643 | + def test_skip_alias(self, mock_interfaces): |
4644 | """Interface aliases are not reported by L{get_active_device_info}.""" |
4645 | - mock_get_active_interfaces.side_effect = lambda: ( |
4646 | - list(get_active_interfaces()) + [("eth0:foo", {})]) |
4647 | + mock_interfaces.side_effect = lambda: ( |
4648 | + _interfaces() + ["eth0:foo"]) |
4649 | device_info = get_active_device_info() |
4650 | interfaces = [i["interface"] for i in device_info] |
4651 | self.assertNotIn("eth0:foo", interfaces) |
4652 | |
4653 | @patch("landscape.lib.network.netifaces.ifaddresses") |
4654 | @patch("landscape.lib.network.netifaces.interfaces") |
4655 | + def test_no_extra_netifaces_calls(self, mock_interfaces, mock_ifaddresses): |
4656 | + """ |
4657 | + Make sure filtered out interfaces aren't used in netiface calls due to |
4658 | + their impact on sysinfo/login time with a large amount of interfaces. |
4659 | + """ |
4660 | + mock_interfaces.return_value = ["eth0:foo"] |
4661 | + get_active_device_info() |
4662 | + assert not mock_ifaddresses.called |
4663 | + |
4664 | + @patch("landscape.lib.network.netifaces.ifaddresses") |
4665 | + @patch("landscape.lib.network.netifaces.interfaces") |
4666 | def test_skip_iface_with_no_addr(self, mock_interfaces, mock_ifaddresses): |
4667 | mock_interfaces.return_value = _interfaces() + ["test_iface"] |
4668 | mock_ifaddresses.side_effect = lambda iface: ( |
4669 | @@ -283,6 +312,53 @@ class NetworkInfoTest(BaseTestCase): |
4670 | self.assertFalse(is_up(2 + 64 + 4096)) |
4671 | self.assertFalse(is_up(0b11111111111110)) |
4672 | |
4673 | + def test_get_filtered_if_info(self): |
4674 | + def filter_tap(interface): |
4675 | + return interface.startswith("tap") |
4676 | + |
4677 | + with patch.multiple( |
4678 | + "landscape.lib.network", |
4679 | + get_flags=DEFAULT, |
4680 | + get_network_interface_speed=DEFAULT, |
4681 | + netifaces=DEFAULT, |
4682 | + ) as mocks: |
4683 | + mocks["netifaces"].interfaces.return_value = [ |
4684 | + "tap0123", |
4685 | + "test_iface", |
4686 | + "tap4567", |
4687 | + "test_iface2", |
4688 | + ] |
4689 | + mocks["get_flags"].return_value = 4163 |
4690 | + mocks["get_network_interface_speed"].return_value = (100, True) |
4691 | + device_info = get_filtered_if_info(filters=(filter_tap,), |
4692 | + extended=True) |
4693 | + |
4694 | + self.assertEqual(len(device_info), 2) |
4695 | + self.assertTrue(all("tap" not in i["interface"] for i in device_info)) |
4696 | + |
4697 | + def test_get_active_device_info_filtered_taps(self): |
4698 | + """ |
4699 | + Tests that tap network interfaces are filtered out. |
4700 | + """ |
4701 | + with patch.multiple( |
4702 | + "landscape.lib.network", |
4703 | + get_flags=DEFAULT, |
4704 | + get_network_interface_speed=DEFAULT, |
4705 | + netifaces=DEFAULT, |
4706 | + ) as mocks: |
4707 | + mocks["netifaces"].interfaces.return_value = [ |
4708 | + "tap0123", |
4709 | + "test_iface", |
4710 | + "tap4567", |
4711 | + "test_iface2", |
4712 | + ] |
4713 | + mocks["get_flags"].return_value = 4163 |
4714 | + mocks["get_network_interface_speed"].return_value = (100, True) |
4715 | + device_info = get_active_device_info(extended=True) |
4716 | + |
4717 | + self.assertEqual(len(device_info), 2) |
4718 | + self.assertTrue(all("tap" not in i["interface"] for i in device_info)) |
4719 | + |
4720 | |
4721 | # exact output of cat /proc/net/dev snapshot with line continuations for pep8 |
4722 | test_proc_net_dev_output = """\ |
4723 | @@ -370,6 +446,7 @@ class NetworkInterfaceSpeedTest(BaseTestCase): |
4724 | mock_unpack.assert_called_with("12xHB28x", ANY) |
4725 | |
4726 | self.assertEqual((100, False), result) |
4727 | + sock.close() |
4728 | |
4729 | @patch("struct.unpack") |
4730 | @patch("fcntl.ioctl") |
4731 | @@ -389,6 +466,7 @@ class NetworkInterfaceSpeedTest(BaseTestCase): |
4732 | mock_unpack.assert_called_with("12xHB28x", ANY) |
4733 | |
4734 | self.assertEqual((0, False), result) |
4735 | + sock.close() |
4736 | |
4737 | @patch("fcntl.ioctl") |
4738 | def test_get_network_interface_speed_not_supported(self, mock_ioctl): |
4739 | @@ -411,6 +489,7 @@ class NetworkInterfaceSpeedTest(BaseTestCase): |
4740 | mock_ioctl.assert_called_with(ANY, ANY, ANY) |
4741 | |
4742 | self.assertEqual((-1, False), result) |
4743 | + sock.close() |
4744 | |
4745 | @patch("fcntl.ioctl") |
4746 | def test_get_network_interface_speed_not_permitted(self, mock_ioctl): |
4747 | @@ -433,6 +512,7 @@ class NetworkInterfaceSpeedTest(BaseTestCase): |
4748 | mock_ioctl.assert_called_with(ANY, ANY, ANY) |
4749 | |
4750 | self.assertEqual((-1, False), result) |
4751 | + sock.close() |
4752 | |
4753 | @patch("fcntl.ioctl") |
4754 | def test_get_network_interface_speed_other_io_error(self, mock_ioctl): |
4755 | @@ -450,3 +530,4 @@ class NetworkInterfaceSpeedTest(BaseTestCase): |
4756 | mock_ioctl.side_effect = theerror |
4757 | |
4758 | self.assertRaises(IOError, get_network_interface_speed, sock, b"eth0") |
4759 | + sock.close() |
4760 | diff --git a/landscape/lib/tests/test_schema.py b/landscape/lib/tests/test_schema.py |
4761 | index cd6d71f..fce6979 100644 |
4762 | --- a/landscape/lib/tests/test_schema.py |
4763 | +++ b/landscape/lib/tests/test_schema.py |
4764 | @@ -69,8 +69,8 @@ class BasicTypesTest(unittest.TestCase): |
4765 | def test_string(self): |
4766 | self.assertEqual(Bytes().coerce(b"foo"), b"foo") |
4767 | |
4768 | - def test_string_bad_unicode(self): |
4769 | - self.assertRaises(InvalidError, Bytes().coerce, u"foo") |
4770 | + def test_string_unicode(self): |
4771 | + self.assertEqual(Bytes().coerce(u"foo"), b"foo") |
4772 | |
4773 | def test_string_bad_anything(self): |
4774 | self.assertRaises(InvalidError, Bytes().coerce, object()) |
4775 | diff --git a/landscape/lib/tests/test_vm_info.py b/landscape/lib/tests/test_vm_info.py |
4776 | index 68bf6a3..3e46912 100644 |
4777 | --- a/landscape/lib/tests/test_vm_info.py |
4778 | +++ b/landscape/lib/tests/test_vm_info.py |
4779 | @@ -165,9 +165,9 @@ class GetVMInfoTest(BaseTestCase): |
4780 | """ |
4781 | cpuinfo_path = os.path.join(self.proc_path, "cpuinfo") |
4782 | cpuinfo = ( |
4783 | - "platform : Some Machine\n" |
4784 | - "model : Some CPU (emulated by qemu)\n" |
4785 | - "machine : Some Machine (emulated by qemu)\n") |
4786 | + "platform : Some Machine\n" |
4787 | + "model : Some CPU (emulated by qemu)\n" |
4788 | + "machine : Some Machine (emulated by qemu)\n") |
4789 | self.makeFile(path=cpuinfo_path, content=cpuinfo) |
4790 | self.assertEqual(b"kvm", get_vm_info(root_path=self.root_path)) |
4791 | |
4792 | @@ -184,6 +184,18 @@ class GetVMInfoTest(BaseTestCase): |
4793 | self.make_dmi_info("product_name", "KVM") |
4794 | self.assertEqual(b"kvm", get_vm_info(root_path=self.root_path)) |
4795 | |
4796 | + def test_get_vm_info_with_rhev(self): |
4797 | + """get_vm_info returns 'kvm' if running under RHEV Hypervisor.""" |
4798 | + self.make_dmi_info("product_name", "RHEV Hypervisor") |
4799 | + self.make_dmi_info("sys_vendor", "Red Hat") |
4800 | + self.assertEqual(b"kvm", get_vm_info(root_path=self.root_path)) |
4801 | + |
4802 | + def test_get_vm_info_with_parallels(self): |
4803 | + """get_vm_info returns 'kvm' if running under Parallels""" |
4804 | + self.make_dmi_info("product_name", "Parallels Virtual Platform") |
4805 | + self.make_dmi_info("sys_vendor", "Parallels Software International") |
4806 | + self.assertEqual(b"kvm", get_vm_info(root_path=self.root_path)) |
4807 | + |
4808 | |
4809 | class GetContainerInfoTest(BaseTestCase): |
4810 | |
4811 | diff --git a/landscape/lib/user.py b/landscape/lib/user.py |
4812 | index db8a24f..dfac4f9 100644 |
4813 | --- a/landscape/lib/user.py |
4814 | +++ b/landscape/lib/user.py |
4815 | @@ -1,7 +1,7 @@ |
4816 | import os.path |
4817 | import pwd |
4818 | |
4819 | -from twisted.python.compat import _PY3 |
4820 | +from landscape.lib.compat import _PY3 |
4821 | |
4822 | from landscape.lib.encoding import encode_if_needed |
4823 | |
4824 | diff --git a/landscape/lib/vm_info.py b/landscape/lib/vm_info.py |
4825 | index 2ede13b..a5d7ca7 100644 |
4826 | --- a/landscape/lib/vm_info.py |
4827 | +++ b/landscape/lib/vm_info.py |
4828 | @@ -66,9 +66,6 @@ def _get_vm_by_vendor(sys_vendor_path): |
4829 | # We need bytes here as required by the message schema. |
4830 | vendor = read_binary_file(sys_vendor_path, limit=1024).lower() |
4831 | |
4832 | - # 2018-01: AWS and DO are now returning custom sys_vendor names |
4833 | - # instead of qemu. If this becomes a trend, it may be worth also checking |
4834 | - # dmi/id/chassis_vendor which seems to unchanged (bochs). |
4835 | content_vendors_map = ( |
4836 | (b"amazon ec2", b"kvm"), |
4837 | (b"bochs", b"kvm"), |
4838 | @@ -81,6 +78,8 @@ def _get_vm_by_vendor(sys_vendor_path): |
4839 | (b"qemu", b"kvm"), |
4840 | (b"kvm", b"kvm"), |
4841 | (b"vmware", b"vmware"), |
4842 | + (b"rhev", b"kvm"), |
4843 | + (b"parallels", b"kvm") |
4844 | ) |
4845 | for name, vm_type in content_vendors_map: |
4846 | if name in vendor: |
4847 | diff --git a/landscape/message_schemas/server_bound.py b/landscape/message_schemas/server_bound.py |
4848 | index a03d76c..1e8f698 100644 |
4849 | --- a/landscape/message_schemas/server_bound.py |
4850 | +++ b/landscape/message_schemas/server_bound.py |
4851 | @@ -19,7 +19,7 @@ __all__ = [ |
4852 | "NETWORK_DEVICE", "NETWORK_ACTIVITY", |
4853 | "REBOOT_REQUIRED_INFO", "UPDATE_MANAGER_INFO", "CPU_USAGE", |
4854 | "CEPH_USAGE", "SWIFT_USAGE", "SWIFT_DEVICE_INFO", "KEYSTONE_TOKEN", |
4855 | - "JUJU_UNITS_INFO", "CLOUD_METADATA", |
4856 | + "JUJU_UNITS_INFO", "CLOUD_METADATA", "COMPUTER_TAGS", "UBUNTU_PRO_INFO", |
4857 | ] |
4858 | |
4859 | |
4860 | @@ -219,10 +219,13 @@ REGISTER_3_3 = Message( |
4861 | "juju-info": KeyDict({"environment-uuid": Unicode(), |
4862 | "api-addresses": List(Unicode()), |
4863 | "machine-id": Unicode()}), |
4864 | - "access_group": Unicode()}, |
4865 | + "access_group": Unicode(), |
4866 | + "clone_secure_id": Any(Unicode(), Constant(None)), |
4867 | + "ubuntu_pro_info": Unicode()}, |
4868 | api=b"3.3", |
4869 | optional=["registration_password", "hostname", "tags", "vm-info", |
4870 | - "container-info", "access_group", "juju-info"]) |
4871 | + "container-info", "access_group", "juju-info", |
4872 | + "clone_secure_id", "ubuntu_pro_info"]) |
4873 | |
4874 | |
4875 | # XXX The register-provisioned-machine message is obsolete, it's kept around |
4876 | @@ -504,6 +507,13 @@ NETWORK_ACTIVITY = Message( |
4877 | |
4878 | UPDATE_MANAGER_INFO = Message("update-manager-info", {"prompt": Unicode()}) |
4879 | |
4880 | +COMPUTER_TAGS = Message( |
4881 | + "computer-tags", |
4882 | + {"tags": Any(Unicode(), Constant(None))}) |
4883 | + |
4884 | +UBUNTU_PRO_INFO = Message( |
4885 | + "ubuntu-pro-info", |
4886 | + {"ubuntu-pro-info": Unicode()}) |
4887 | |
4888 | message_schemas = ( |
4889 | ACTIVE_PROCESS_INFO, COMPUTER_UPTIME, CLIENT_UPTIME, |
4890 | @@ -518,4 +528,4 @@ message_schemas = ( |
4891 | NETWORK_DEVICE, NETWORK_ACTIVITY, |
4892 | REBOOT_REQUIRED_INFO, UPDATE_MANAGER_INFO, CPU_USAGE, |
4893 | CEPH_USAGE, SWIFT_USAGE, SWIFT_DEVICE_INFO, KEYSTONE_TOKEN, |
4894 | - JUJU_UNITS_INFO, CLOUD_METADATA) |
4895 | + JUJU_UNITS_INFO, CLOUD_METADATA, COMPUTER_TAGS, UBUNTU_PRO_INFO) |
4896 | diff --git a/landscape/message_schemas/test_message.py b/landscape/message_schemas/test_message.py |
4897 | index 0d87eac..7b8c66b 100644 |
4898 | --- a/landscape/message_schemas/test_message.py |
4899 | +++ b/landscape/message_schemas/test_message.py |
4900 | @@ -1,6 +1,6 @@ |
4901 | import unittest |
4902 | |
4903 | -from landscape.lib.schema import Int |
4904 | +from landscape.lib.schema import Constant, Int |
4905 | from landscape.message_schemas.message import Message |
4906 | |
4907 | |
4908 | @@ -13,6 +13,14 @@ class MessageTest(unittest.TestCase): |
4909 | schema.coerce({"type": "foo", "data": 3}), |
4910 | {"type": "foo", "data": 3}) |
4911 | |
4912 | + def test_coerce_bytes_to_str(self): |
4913 | + """ |
4914 | + The L{Constant} schema type recognizes bytestrings that decode to |
4915 | + matching strings. |
4916 | + """ |
4917 | + constant = Constant("register") |
4918 | + self.assertEqual(constant.coerce(b"register"), "register") |
4919 | + |
4920 | def test_timestamp(self): |
4921 | """L{Message} schemas should accept C{timestamp} keys.""" |
4922 | schema = Message("bar", {}) |
4923 | diff --git a/landscape/sysinfo/load.py b/landscape/sysinfo/load.py |
4924 | index 718a929..4843038 100644 |
4925 | --- a/landscape/sysinfo/load.py |
4926 | +++ b/landscape/sysinfo/load.py |
4927 | @@ -9,5 +9,6 @@ class Load(object): |
4928 | self._sysinfo = sysinfo |
4929 | |
4930 | def run(self): |
4931 | - self._sysinfo.add_header("System load", str(os.getloadavg()[0])) |
4932 | + self._sysinfo.add_header( |
4933 | + "System load", str(round(os.getloadavg()[0], 2))) |
4934 | return succeed(None) |
4935 | diff --git a/landscape/sysinfo/network.py b/landscape/sysinfo/network.py |
4936 | index 6df4b7c..420a603 100644 |
4937 | --- a/landscape/sysinfo/network.py |
4938 | +++ b/landscape/sysinfo/network.py |
4939 | @@ -16,7 +16,8 @@ class Network(object): |
4940 | |
4941 | def __init__(self, get_device_info=None): |
4942 | if get_device_info is None: |
4943 | - get_device_info = partial(get_active_device_info, extended=True) |
4944 | + get_device_info = partial(get_active_device_info, |
4945 | + extended=True, default_only=True) |
4946 | self._get_device_info = get_device_info |
4947 | |
4948 | def register(self, sysinfo): |
4949 | diff --git a/man/landscape-config.1 b/man/landscape-config.1 |
4950 | index 7d398ff..eeafefd 100644 |
4951 | --- a/man/landscape-config.1 |
4952 | +++ b/man/landscape-config.1 |
4953 | @@ -1,5 +1,5 @@ |
4954 | .\" Text automatically generated by txt2man |
4955 | -.TH landscape-config 1 "14 February 2019" "" "" |
4956 | +.TH landscape-config 1 "07 February 2022" "" "" |
4957 | .SH NAME |
4958 | \fBlandscape-config \fP- configure the Landscape management client |
4959 | \fB |
4960 | @@ -181,6 +181,12 @@ Stop running clients and disable start at boot. |
4961 | .B |
4962 | \fB--otp\fP=OTP |
4963 | The one-time password (OTP) to use in cloud configuration. |
4964 | +.TP |
4965 | +.B |
4966 | +\fB--is-registered\fP |
4967 | +Exit with code 0 (success) if client |
4968 | +is registered else returns 5. Display |
4969 | +registration info. |
4970 | .SH CLOUD |
4971 | |
4972 | Landscape has some cloud features that become available when the EC2 or |
4973 | diff --git a/man/landscape-config.txt b/man/landscape-config.txt |
4974 | index 5a100bf..2bce364 100644 |
4975 | --- a/man/landscape-config.txt |
4976 | +++ b/man/landscape-config.txt |
4977 | @@ -75,6 +75,9 @@ OPTIONS |
4978 | --silent Run without manual interaction. |
4979 | --disable Stop running clients and disable start at boot. |
4980 | --otp=OTP The one-time password (OTP) to use in cloud configuration. |
4981 | + --is-registered Exit with code 0 (success) if client |
4982 | + is registered else returns 5. Display |
4983 | + registration info. |
4984 | |
4985 | CLOUD |
4986 |
PPA: https:/ /launchpad. net/~mitchburto n/+archive/ ubuntu/ landscape- client- ppa
We have been following the development procedure laid out in https:/ /wiki.ubuntu. com/LandscapeUp dates
Furthermore, the testing procedure results can be seen in /wiki.canonical .com/Landscape/ ClientSRUTests/ 23.02
https:/