Merge lp:~nataliabidart/magicicada-server/clean-deps-for-stable into lp:magicicada-server
- clean-deps-for-stable
- Merge into trunk
Proposed by
Natalia Bidart
Status: | Merged |
---|---|
Approved by: | Natalia Bidart |
Approved revision: | 83 |
Merged at revision: | 82 |
Proposed branch: | lp:~nataliabidart/magicicada-server/clean-deps-for-stable |
Merge into: | lp:magicicada-server |
Diff against target: |
864 lines (+123/-189) 24 files modified
Makefile (+10/-12) README.rst (+1/-1) config-manager.txt (+0/-4) dependencies-devel.txt (+1/-0) dependencies.txt (+0/-2) dev-scripts/cmd_client.py (+1/-1) dev-scripts/deploy_api_server.py (+1/-1) dev-scripts/dev_launcher.py (+1/-1) dev-scripts/supervisor-dev.conf.tpl (+2/-14) magicicada/rpcdb/tests/test_backend.py (+1/-1) magicicada/server/content.py (+3/-3) magicicada/server/integration/run_integtests.py (+1/-6) magicicada/server/server.py (+14/-15) magicicada/server/ssl_proxy.py (+9/-9) magicicada/server/testing/aq_helpers.py (+2/-2) magicicada/server/testing/testcase.py (+4/-4) magicicada/server/tests/test_auth.py (+4/-4) magicicada/server/tests/test_content.py (+13/-13) magicicada/server/tests/test_generations.py (+3/-3) magicicada/server/tests/test_server.py (+5/-5) magicicada/server/tests/test_ssl_proxy.py (+4/-4) magicicada/settings/__init__.py (+43/-2) magicicada/settings/api_server.py (+0/-58) magicicada/settings/ssl_proxy.py (+0/-24) |
To merge this branch: | bzr merge lp:~nataliabidart/magicicada-server/clean-deps-for-stable |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Facundo Batista | Approve | ||
Review via email: mp+322999@code.launchpad.net |
Commit message
- Unified settings into a single file.
- Removed uneeded dependencies.
- Removed every outdated reference to "oauth".
Description of the change
To post a comment you must log in.
- 83. By Natalia Bidart
-
Missing replacement.
Preview Diff
[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1 | === modified file 'Makefile' | |||
2 | --- Makefile 2016-10-01 12:48:16 +0000 | |||
3 | +++ Makefile 2017-04-22 22:01:02 +0000 | |||
4 | @@ -128,14 +128,12 @@ | |||
5 | 128 | version: | 128 | version: |
6 | 129 | bzr version-info --format=python > lib/versioninfo.py || true | 129 | bzr version-info --format=python > lib/versioninfo.py || true |
7 | 130 | 130 | ||
16 | 131 | start: build start-base start-filesync-dummy-group publish-api-port | 131 | start: build start-base start-filesync-server-group publish-api-port |
17 | 132 | 132 | ||
18 | 133 | start-oauth: build start-base start-filesync-oauth-group publish-api-port | 133 | resume: start-base start-filesync-server-group |
19 | 134 | 134 | ||
20 | 135 | resume: start-base start-filesync-oauth-group | 135 | start-heapy: |
21 | 136 | 136 | USE_HEAPY=1 $(MAKE) start | |
14 | 137 | start-oauth-heapy: | ||
15 | 138 | USE_HEAPY=1 $(MAKE) start-oauth | ||
22 | 139 | 137 | ||
23 | 140 | start-base: | 138 | start-base: |
24 | 141 | $(MAKE) start-supervisor && $(MAKE) start-dbus || ( $(MAKE) stop ; exit 1 ) | 139 | $(MAKE) start-supervisor && $(MAKE) start-dbus || ( $(MAKE) stop ; exit 1 ) |
25 | @@ -168,9 +166,9 @@ | |||
26 | 168 | -@dev-scripts/supervisorctl-dev stop $* | 166 | -@dev-scripts/supervisorctl-dev stop $* |
27 | 169 | 167 | ||
28 | 170 | publish-api-port: | 168 | publish-api-port: |
32 | 171 | python -c 'from magicicada import settings; print >> file("tmp/filesyncserver.port", "w"), settings.api_server.TCP_PORT' | 169 | python -c 'from magicicada import settings; print >> file("tmp/filesyncserver.port", "w"), settings.TCP_PORT' |
33 | 172 | python -c 'from magicicada import settings; print >> file("tmp/filesyncserver.port.ssl", "w"), settings.ssl_proxy.PORT' | 170 | python -c 'from magicicada import settings; print >> file("tmp/filesyncserver.port.ssl", "w"), settings.SSL_PORT' |
34 | 173 | python -c 'from magicicada import settings; print >> file("tmp/filesyncserver-status.port", "w"), settings.api_server.STATUS_PORT' | 171 | python -c 'from magicicada import settings; print >> file("tmp/filesyncserver-status.port", "w"), settings.API_STATUS_PORT' |
35 | 174 | 172 | ||
36 | 175 | shell: | 173 | shell: |
37 | 176 | $(DJANGO_MANAGE) shell | 174 | $(DJANGO_MANAGE) shell |
38 | @@ -184,4 +182,4 @@ | |||
39 | 184 | .PHONY: sourcedeps link-sourcedeps build-sourcedeps build-deploy-sourcedeps \ | 182 | .PHONY: sourcedeps link-sourcedeps build-sourcedeps build-deploy-sourcedeps \ |
40 | 185 | build clean version lint test ci-test build-for-deployment \ | 183 | build clean version lint test ci-test build-for-deployment \ |
41 | 186 | clean-sourcedeps tarball start stop publish-api-port start-supervisor \ | 184 | clean-sourcedeps tarball start stop publish-api-port start-supervisor \ |
43 | 187 | stop-supervisor start-dbus stop-dbus start-oauth start-oauth-heapy | 185 | stop-supervisor start-dbus stop-dbus start-heapy |
44 | 188 | 186 | ||
45 | === modified file 'README.rst' | |||
46 | --- README.rst 2017-02-25 15:29:01 +0000 | |||
47 | +++ README.rst 2017-04-22 22:01:02 +0000 | |||
48 | @@ -139,7 +139,7 @@ | |||
49 | 139 | Start the server via command line this way:: | 139 | Start the server via command line this way:: |
50 | 140 | 140 | ||
51 | 141 | cd ~/magicicada/magicicada-server | 141 | cd ~/magicicada/magicicada-server |
53 | 142 | make start-oauth | 142 | make start |
54 | 143 | 143 | ||
55 | 144 | Note that the server will listen on port 21101, so you need to assure that the | 144 | Note that the server will listen on port 21101, so you need to assure that the |
56 | 145 | client could reach it (open the whole it in your firewall config, etc). | 145 | client could reach it (open the whole it in your firewall config, etc). |
57 | 146 | 146 | ||
58 | === modified file 'config-manager.txt' | |||
59 | --- config-manager.txt 2016-12-27 21:57:00 +0000 | |||
60 | +++ config-manager.txt 2017-04-22 22:01:02 +0000 | |||
61 | @@ -24,10 +24,6 @@ | |||
62 | 24 | ./.sourcecode/configglue ~configglue/configglue/trunk;revno=1 | 24 | ./.sourcecode/configglue ~configglue/configglue/trunk;revno=1 |
63 | 25 | ./.sourcecode/dirspec ~ubuntuone-control-tower/dirspec/trunk;revno=14 | 25 | ./.sourcecode/dirspec ~ubuntuone-control-tower/dirspec/trunk;revno=14 |
64 | 26 | ./.sourcecode/django ~ubuntuone-pqm-team/django/stable;revno=17 | 26 | ./.sourcecode/django ~ubuntuone-pqm-team/django/stable;revno=17 |
65 | 27 | ./.sourcecode/mock ~ubuntuone-pqm-team/python-mock/stable;revno=1 | ||
66 | 28 | ./.sourcecode/oauthlib ~ubuntuone-pqm-team/oauthlib/stable;revno=209 | ||
67 | 29 | ./.sourcecode/requests ~ubuntuone-pqm-team/requests/stable;revno=816 | ||
68 | 30 | ./.sourcecode/requests_oauthlib ~ubuntuone-pqm-team/requests-oauthlib/stable;revno=16 | ||
69 | 31 | ./.sourcecode/u1sync ~facundo/u1sync/opensourcing;revno=10 | 27 | ./.sourcecode/u1sync ~facundo/u1sync/opensourcing;revno=10 |
70 | 32 | ./.sourcecode/magicicada-client ~chicharreros/magicicada-client/trunk;revno=1433 | 28 | ./.sourcecode/magicicada-client ~chicharreros/magicicada-client/trunk;revno=1433 |
71 | 33 | ./.sourcecode/magicicada-protocol ~chicharreros/magicicada-protocol/trunk;revno=169 | 29 | ./.sourcecode/magicicada-protocol ~chicharreros/magicicada-protocol/trunk;revno=169 |
72 | 34 | 30 | ||
73 | === modified file 'dependencies-devel.txt' | |||
74 | --- dependencies-devel.txt 2016-09-15 13:46:24 +0000 | |||
75 | +++ dependencies-devel.txt 2017-04-22 22:01:02 +0000 | |||
76 | @@ -4,6 +4,7 @@ | |||
77 | 4 | python-dbus | 4 | python-dbus |
78 | 5 | python-gobject | 5 | python-gobject |
79 | 6 | python-httplib2 | 6 | python-httplib2 |
80 | 7 | python-mock | ||
81 | 7 | python-mocker | 8 | python-mocker |
82 | 8 | python-pyinotify | 9 | python-pyinotify |
83 | 9 | python-testtools | 10 | python-testtools |
84 | 10 | 11 | ||
85 | === modified file 'dependencies.txt' | |||
86 | --- dependencies.txt 2016-02-20 17:42:06 +0000 | |||
87 | +++ dependencies.txt 2017-04-22 22:01:02 +0000 | |||
88 | @@ -15,8 +15,6 @@ | |||
89 | 15 | python-psutil | 15 | python-psutil |
90 | 16 | python-psycopg2 | 16 | python-psycopg2 |
91 | 17 | python-setuptools | 17 | python-setuptools |
92 | 18 | python-testresources | ||
93 | 19 | python-transaction | ||
94 | 20 | python-twisted-web | 18 | python-twisted-web |
95 | 21 | python-tz | 19 | python-tz |
96 | 22 | python-virtualenv | 20 | python-virtualenv |
97 | 23 | 21 | ||
98 | === modified file 'dev-scripts/cmd_client.py' | |||
99 | --- dev-scripts/cmd_client.py 2015-09-17 19:30:41 +0000 | |||
100 | +++ dev-scripts/cmd_client.py 2017-04-22 22:01:02 +0000 | |||
101 | @@ -719,7 +719,7 @@ | |||
102 | 719 | @parse_args(str, str) | 719 | @parse_args(str, str) |
103 | 720 | @require_connection | 720 | @require_connection |
104 | 721 | @show_exception | 721 | @show_exception |
106 | 722 | def do_oauth(self): | 722 | def do_auth(self): |
107 | 723 | """Perform authorisation.""" | 723 | """Perform authorisation.""" |
108 | 724 | self.defer_from_thread(self._auth) | 724 | self.defer_from_thread(self._auth) |
109 | 725 | 725 | ||
110 | 726 | 726 | ||
111 | === modified file 'dev-scripts/deploy_api_server.py' | |||
112 | --- dev-scripts/deploy_api_server.py 2016-05-29 21:20:25 +0000 | |||
113 | +++ dev-scripts/deploy_api_server.py 2017-04-22 22:01:02 +0000 | |||
114 | @@ -55,7 +55,7 @@ | |||
115 | 55 | """Start the server.""" | 55 | """Start the server.""" |
116 | 56 | 56 | ||
117 | 57 | status_port = int( | 57 | status_port = int( |
119 | 58 | os.getenv('API_STATUS_PORT', settings.api_server.STATUS_PORT)) | 58 | os.getenv('API_STATUS_PORT', settings.STATUS_PORT)) |
120 | 59 | service = server.create_service(status_port) | 59 | service = server.create_service(status_port) |
121 | 60 | 60 | ||
122 | 61 | yield service.startService() | 61 | yield service.startService() |
123 | 62 | 62 | ||
124 | === modified file 'dev-scripts/dev_launcher.py' | |||
125 | --- dev-scripts/dev_launcher.py 2015-09-17 19:30:41 +0000 | |||
126 | +++ dev-scripts/dev_launcher.py 2017-04-22 22:01:02 +0000 | |||
127 | @@ -37,7 +37,7 @@ | |||
128 | 37 | "runs PROGNAME with args plus:\n" | 37 | "runs PROGNAME with args plus:\n" |
129 | 38 | " --port with the local server port\n" | 38 | " --port with the local server port\n" |
130 | 39 | " --host localhost\n" | 39 | " --host localhost\n" |
132 | 40 | " --oauth keys for user USERNAME\n") | 40 | " --auth username and password for user USERNAME\n") |
133 | 41 | 41 | ||
134 | 42 | 42 | ||
135 | 43 | def main(args): | 43 | def main(args): |
136 | 44 | 44 | ||
137 | === modified file 'dev-scripts/supervisor-dev.conf.tpl' | |||
138 | --- dev-scripts/supervisor-dev.conf.tpl 2016-06-03 00:35:53 +0000 | |||
139 | +++ dev-scripts/supervisor-dev.conf.tpl 2017-04-22 22:01:02 +0000 | |||
140 | @@ -24,7 +24,7 @@ | |||
141 | 24 | files=services-supervisor.conf workers-supervisor.conf | 24 | files=services-supervisor.conf workers-supervisor.conf |
142 | 25 | 25 | ||
143 | 26 | [eventlistener:heartbeat] | 26 | [eventlistener:heartbeat] |
145 | 27 | command=python %(basepath)s/lib/ubuntuone/supervisor/heartbeat_listener.py --interval=10 --timeout=20 --log_level=DEBUG --log_file=%(tmp_dir)s/heartbeat.log --groups=filesync-dummy,filesync-oauth | 27 | command=python %(basepath)s/lib/ubuntuone/supervisor/heartbeat_listener.py --interval=10 --timeout=20 --log_level=DEBUG --log_file=%(tmp_dir)s/heartbeat.log --groups=filesync-server |
146 | 28 | environment=PYTHONPATH="%(basepath)s:%(basepath)s/lib" | 28 | environment=PYTHONPATH="%(basepath)s:%(basepath)s/lib" |
147 | 29 | events=PROCESS_COMMUNICATION,TICK_5 | 29 | events=PROCESS_COMMUNICATION,TICK_5 |
148 | 30 | buffer_size=42 | 30 | buffer_size=42 |
149 | @@ -36,15 +36,6 @@ | |||
150 | 36 | autostart=false | 36 | autostart=false |
151 | 37 | stopsignal=INT | 37 | stopsignal=INT |
152 | 38 | 38 | ||
153 | 39 | [program:filesync-dummy-auth] | ||
154 | 40 | command=python %(basepath)s/dev-scripts/deploy_api_server.py | ||
155 | 41 | environment=PYTHONPATH="%(basepath)s:%(basepath)s/lib",DJANGO_SETTINGS_MODULE="magicicada.settings",PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp | ||
156 | 42 | stdout_capture_maxbytes=16384 | ||
157 | 43 | autostart=false | ||
158 | 44 | redirect_stderr=true ; send stderr to the log file | ||
159 | 45 | stdout_logfile=%(tmp_dir)s/filesync.log | ||
160 | 46 | stdout_logfile_maxbytes=0 | ||
161 | 47 | |||
162 | 48 | [program:ssl-proxy] | 39 | [program:ssl-proxy] |
163 | 49 | command=/usr/bin/twistd --pidfile %(tmp_dir)s/ssl-proxy.pid -n -y %(basepath)s/magicicada/server/ssl_proxy.tac --reactor=epoll | 40 | command=/usr/bin/twistd --pidfile %(tmp_dir)s/ssl-proxy.pid -n -y %(basepath)s/magicicada/server/ssl_proxy.tac --reactor=epoll |
164 | 50 | environment=PYTHONPATH="%(basepath)s:%(basepath)s/lib",DJANGO_SETTINGS_MODULE="magicicada.settings" | 41 | environment=PYTHONPATH="%(basepath)s:%(basepath)s/lib",DJANGO_SETTINGS_MODULE="magicicada.settings" |
165 | @@ -52,8 +43,5 @@ | |||
166 | 52 | autostart=false | 43 | autostart=false |
167 | 53 | stopsignal=INT | 44 | stopsignal=INT |
168 | 54 | 45 | ||
173 | 55 | [group:filesync-dummy] | 46 | [group:filesync-server] |
170 | 56 | programs=filesync-dummy-auth | ||
171 | 57 | |||
172 | 58 | [group:filesync-oauth] | ||
174 | 59 | programs=filesync,ssl-proxy | 47 | programs=filesync,ssl-proxy |
175 | 60 | 48 | ||
176 | === modified file 'magicicada/rpcdb/tests/test_backend.py' | |||
177 | --- magicicada/rpcdb/tests/test_backend.py 2016-11-07 01:37:46 +0000 | |||
178 | +++ magicicada/rpcdb/tests/test_backend.py 2017-04-22 22:01:02 +0000 | |||
179 | @@ -57,7 +57,7 @@ | |||
180 | 57 | self.assertEqual(result, dict(user_id=self.usr.id)) | 57 | self.assertEqual(result, dict(user_id=self.usr.id)) |
181 | 58 | 58 | ||
182 | 59 | def test_get_user_id_bad_auth(self): | 59 | def test_get_user_id_bad_auth(self): |
184 | 60 | """Bad parameters in the oauth request.""" | 60 | """Bad parameters in the auth request.""" |
185 | 61 | bad_parameters = self.auth_parameters.copy() | 61 | bad_parameters = self.auth_parameters.copy() |
186 | 62 | bad_parameters['password'] = "bad" | 62 | bad_parameters['password'] = "bad" |
187 | 63 | try: | 63 | try: |
188 | 64 | 64 | ||
189 | === modified file 'magicicada/server/content.py' | |||
190 | --- magicicada/server/content.py 2016-11-07 01:37:46 +0000 | |||
191 | +++ magicicada/server/content.py 2017-04-22 22:01:02 +0000 | |||
192 | @@ -190,8 +190,8 @@ | |||
193 | 190 | def add_part(self, chunk_size): | 190 | def add_part(self, chunk_size): |
194 | 191 | """Add a part to an upload job.""" | 191 | """Add a part to an upload job.""" |
195 | 192 | self.unsaved_count += chunk_size | 192 | self.unsaved_count += chunk_size |
198 | 193 | if self.unsaved_count >= settings.api_server.STORAGE_CHUNK_SIZE: | 193 | if self.unsaved_count >= settings.STORAGE_CHUNK_SIZE: |
199 | 194 | self.unsaved_count -= settings.api_server.STORAGE_CHUNK_SIZE | 194 | self.unsaved_count -= settings.STORAGE_CHUNK_SIZE |
200 | 195 | kwargs = dict(user_id=self.user.id, volume_id=self.volume_id, | 195 | kwargs = dict(user_id=self.user.id, volume_id=self.volume_id, |
201 | 196 | uploadjob_id=self.uploadjob_id, | 196 | uploadjob_id=self.uploadjob_id, |
202 | 197 | chunk_size=chunk_size) | 197 | chunk_size=chunk_size) |
203 | @@ -870,7 +870,7 @@ | |||
204 | 870 | if upload is None: | 870 | if upload is None: |
205 | 871 | # no uploadjob found, create a new one: if the file is small just | 871 | # no uploadjob found, create a new one: if the file is small just |
206 | 872 | # get a bo | 872 | # get a bo |
208 | 873 | if deflated_size <= settings.api_server.STORAGE_CHUNK_SIZE: | 873 | if deflated_size <= settings.STORAGE_CHUNK_SIZE: |
209 | 874 | upload = BogusUploadJob() | 874 | upload = BogusUploadJob() |
210 | 875 | else: | 875 | else: |
211 | 876 | try: | 876 | try: |
212 | 877 | 877 | ||
213 | === modified file 'magicicada/server/integration/run_integtests.py' | |||
214 | --- magicicada/server/integration/run_integtests.py 2016-08-30 00:50:31 +0000 | |||
215 | +++ magicicada/server/integration/run_integtests.py 2017-04-22 22:01:02 +0000 | |||
216 | @@ -48,11 +48,6 @@ | |||
217 | 48 | dbus.mainloop.glib.DBusGMainLoop(set_as_default=True) | 48 | dbus.mainloop.glib.DBusGMainLoop(set_as_default=True) |
218 | 49 | 49 | ||
219 | 50 | LIB_DIR = os.path.abspath("lib") | 50 | LIB_DIR = os.path.abspath("lib") |
220 | 51 | |||
221 | 52 | # this should be done manually before: | ||
222 | 53 | DEP_STARTOAUTH = """ | ||
223 | 54 | make start-oauth | ||
224 | 55 | """ | ||
225 | 56 | CLIENT_DIR = os.path.abspath(os.path.join('.sourcecode', 'magicicada-client')) | 51 | CLIENT_DIR = os.path.abspath(os.path.join('.sourcecode', 'magicicada-client')) |
226 | 57 | PROTOCOL_DIR = os.path.abspath( | 52 | PROTOCOL_DIR = os.path.abspath( |
227 | 58 | os.path.join('.sourcecode', 'magicicada-protocol')) | 53 | os.path.join('.sourcecode', 'magicicada-protocol')) |
228 | @@ -101,7 +96,7 @@ | |||
229 | 101 | """ | 96 | """ |
230 | 102 | couchdb_port = os.path.join(TMP_DIR, "couchdb-master0.port") | 97 | couchdb_port = os.path.join(TMP_DIR, "couchdb-master0.port") |
231 | 103 | if not os.path.exists(couchdb_port): | 98 | if not os.path.exists(couchdb_port): |
233 | 104 | print "Not ready! Hint: did you do...?:\n" + DEP_STARTOAUTH | 99 | print "Not ready! Hint: did you run...?:\nmake start" |
234 | 105 | return True | 100 | return True |
235 | 106 | 101 | ||
236 | 107 | log_conf = os.path.join(CLIENT_DIR, "data", "logging.conf") | 102 | log_conf = os.path.join(CLIENT_DIR, "data", "logging.conf") |
237 | 108 | 103 | ||
238 | === modified file 'magicicada/server/server.py' | |||
239 | --- magicicada/server/server.py 2016-11-07 01:37:46 +0000 | |||
240 | +++ magicicada/server/server.py 2017-04-22 22:01:02 +0000 | |||
241 | @@ -285,8 +285,7 @@ | |||
242 | 285 | self.pending_requests = collections.deque() | 285 | self.pending_requests = collections.deque() |
243 | 286 | self.ping_loop = LoopingPing(StorageServer.PING_INTERVAL, | 286 | self.ping_loop = LoopingPing(StorageServer.PING_INTERVAL, |
244 | 287 | StorageServer.PING_TIMEOUT, | 287 | StorageServer.PING_TIMEOUT, |
247 | 288 | settings.api_server.IDLE_TIMEOUT, | 288 | settings.IDLE_TIMEOUT, self) |
246 | 289 | self) | ||
248 | 290 | 289 | ||
249 | 291 | # capabilities that the server is working with | 290 | # capabilities that the server is working with |
250 | 292 | self.working_caps = MIN_CAP | 291 | self.working_caps = MIN_CAP |
251 | @@ -624,7 +623,7 @@ | |||
252 | 624 | 623 | ||
253 | 625 | def __init__(self, protocol, message): | 624 | def __init__(self, protocol, message): |
254 | 626 | """Create the request response.""" | 625 | """Create the request response.""" |
256 | 627 | self.use_protocol_weakref = settings.api_server.PROTOCOL_WEAKREF | 626 | self.use_protocol_weakref = settings.PROTOCOL_WEAKREF |
257 | 628 | self._protocol_ref = None | 627 | self._protocol_ref = None |
258 | 629 | super(BaseRequestResponse, self).__init__(protocol, message) | 628 | super(BaseRequestResponse, self).__init__(protocol, message) |
259 | 630 | 629 | ||
260 | @@ -1841,7 +1840,7 @@ | |||
261 | 1841 | def _get_upload_job(self): | 1840 | def _get_upload_job(self): |
262 | 1842 | """Get the uploadjob.""" | 1841 | """Get the uploadjob.""" |
263 | 1843 | share_id = self.convert_share_id(self.source_message.put_content.share) | 1842 | share_id = self.convert_share_id(self.source_message.put_content.share) |
265 | 1844 | if settings.api_server.MAGIC_UPLOAD_ACTIVE: | 1843 | if settings.MAGIC_UPLOAD_ACTIVE: |
266 | 1845 | magic_hash = self.source_message.put_content.magic_hash or None | 1844 | magic_hash = self.source_message.put_content.magic_hash or None |
267 | 1846 | else: | 1845 | else: |
268 | 1847 | magic_hash = None | 1846 | magic_hash = None |
269 | @@ -1907,7 +1906,7 @@ | |||
270 | 1907 | msg = self.source_message | 1906 | msg = self.source_message |
271 | 1908 | share_id = self.convert_share_id(msg.get_delta.share) | 1907 | share_id = self.convert_share_id(msg.get_delta.share) |
272 | 1909 | from_generation = msg.get_delta.from_generation | 1908 | from_generation = msg.get_delta.from_generation |
274 | 1910 | delta_max_size = settings.api_server.DELTA_MAX_SIZE | 1909 | delta_max_size = settings.DELTA_MAX_SIZE |
275 | 1911 | delta_info = yield self.protocol.user.get_delta( | 1910 | delta_info = yield self.protocol.user.get_delta( |
276 | 1912 | share_id, from_generation, limit=delta_max_size) | 1911 | share_id, from_generation, limit=delta_max_size) |
277 | 1913 | nodes, vol_generation, free_bytes = delta_info | 1912 | nodes, vol_generation, free_bytes = delta_info |
278 | @@ -1942,7 +1941,7 @@ | |||
279 | 1942 | """Build and send the DELTA_INFO for each node.""" | 1941 | """Build and send the DELTA_INFO for each node.""" |
280 | 1943 | count = 0 | 1942 | count = 0 |
281 | 1944 | for node in nodes: | 1943 | for node in nodes: |
283 | 1945 | if count == settings.api_server.MAX_DELTA_INFO: | 1944 | if count == settings.MAX_DELTA_INFO: |
284 | 1946 | count = 0 | 1945 | count = 0 |
285 | 1947 | yield | 1946 | yield |
286 | 1948 | message = protocol_pb2.Message() | 1947 | message = protocol_pb2.Message() |
287 | @@ -1983,7 +1982,7 @@ | |||
288 | 1983 | def _process(self): | 1982 | def _process(self): |
289 | 1984 | """Get all the live nodes and send DeltaInfos.""" | 1983 | """Get all the live nodes and send DeltaInfos.""" |
290 | 1985 | msg = self.source_message | 1984 | msg = self.source_message |
292 | 1986 | limit = settings.api_server.GET_FROM_SCRATCH_LIMIT | 1985 | limit = settings.GET_FROM_SCRATCH_LIMIT |
293 | 1987 | share_id = self.convert_share_id(msg.get_delta.share) | 1986 | share_id = self.convert_share_id(msg.get_delta.share) |
294 | 1988 | # get the first chunk | 1987 | # get the first chunk |
295 | 1989 | delta_info = yield self.protocol.user.get_from_scratch( | 1988 | delta_info = yield self.protocol.user.get_from_scratch( |
296 | @@ -2333,7 +2332,7 @@ | |||
297 | 2333 | @cvar protocol: The class of the server. | 2332 | @cvar protocol: The class of the server. |
298 | 2334 | """ | 2333 | """ |
299 | 2335 | protocol = StorageServer | 2334 | protocol = StorageServer |
301 | 2336 | graceful_shutdown = settings.api_server.GRACEFUL_SHUTDOWN | 2335 | graceful_shutdown = settings.GRACEFUL_SHUTDOWN |
302 | 2337 | 2336 | ||
303 | 2338 | def __init__(self, auth_provider_class=auth.DummyAuthProvider, | 2337 | def __init__(self, auth_provider_class=auth.DummyAuthProvider, |
304 | 2339 | content_class=content.ContentManager, | 2338 | content_class=content.ContentManager, |
305 | @@ -2341,7 +2340,7 @@ | |||
306 | 2341 | """Create a StorageServerFactory.""" | 2340 | """Create a StorageServerFactory.""" |
307 | 2342 | self.auth_provider = auth_provider_class(self) | 2341 | self.auth_provider = auth_provider_class(self) |
308 | 2343 | self.content = content_class(self) | 2342 | self.content = content_class(self) |
310 | 2344 | self.diskstorage = DiskStorage(settings.api_server.STORAGE_BASEDIR) | 2343 | self.diskstorage = DiskStorage(settings.STORAGE_BASEDIR) |
311 | 2345 | 2344 | ||
312 | 2346 | self.metrics = metrics.get_meter('root') | 2345 | self.metrics = metrics.get_meter('root') |
313 | 2347 | self.user_metrics = metrics.get_meter('user') | 2346 | self.user_metrics = metrics.get_meter('user') |
314 | @@ -2377,7 +2376,7 @@ | |||
315 | 2377 | 2376 | ||
316 | 2378 | self.protocols = [] | 2377 | self.protocols = [] |
317 | 2379 | self.reactor = reactor | 2378 | self.reactor = reactor |
319 | 2380 | self.trace_users = set(settings.api_server.TRACE_USERS) | 2379 | self.trace_users = set(settings.TRACE_USERS) |
320 | 2381 | 2380 | ||
321 | 2382 | twisted.python.log.addObserver(self._deferror_handler) | 2381 | twisted.python.log.addObserver(self._deferror_handler) |
322 | 2383 | 2382 | ||
323 | @@ -2631,10 +2630,10 @@ | |||
324 | 2631 | OrderedMultiService.__init__(self) | 2630 | OrderedMultiService.__init__(self) |
325 | 2632 | self.heartbeat_writer = None | 2631 | self.heartbeat_writer = None |
326 | 2633 | if heartbeat_interval is None: | 2632 | if heartbeat_interval is None: |
328 | 2634 | heartbeat_interval = float(settings.api_server.HEARTBEAT_INTERVAL) | 2633 | heartbeat_interval = float(settings.HEARTBEAT_INTERVAL) |
329 | 2635 | self.heartbeat_interval = heartbeat_interval | 2634 | self.heartbeat_interval = heartbeat_interval |
330 | 2636 | self.rpc_dal = None | 2635 | self.rpc_dal = None |
332 | 2637 | self.servername = settings.api_server.SERVERNAME | 2636 | self.servername = settings.API_SERVER_NAME |
333 | 2638 | logger.info('Starting %s', self.servername) | 2637 | logger.info('Starting %s', self.servername) |
334 | 2639 | logger.info( | 2638 | logger.info( |
335 | 2640 | 'protocol buffers implementation: %s', | 2639 | 'protocol buffers implementation: %s', |
336 | @@ -2726,15 +2725,15 @@ | |||
337 | 2726 | logger.debug('activated heapy remote monitor') | 2725 | logger.debug('activated heapy remote monitor') |
338 | 2727 | 2726 | ||
339 | 2728 | # set GC's debug | 2727 | # set GC's debug |
341 | 2729 | if settings.api_server.GC_DEBUG: | 2728 | if settings.GC_DEBUG: |
342 | 2730 | import gc | 2729 | import gc |
343 | 2731 | gc.set_debug(gc.DEBUG_UNCOLLECTABLE) | 2730 | gc.set_debug(gc.DEBUG_UNCOLLECTABLE) |
344 | 2732 | logger.debug('set gc debug on') | 2731 | logger.debug('set gc debug on') |
345 | 2733 | 2732 | ||
346 | 2734 | if status_port is None: | 2733 | if status_port is None: |
348 | 2735 | status_port = settings.api_server.STATUS_PORT | 2734 | status_port = settings.API_STATUS_PORT |
349 | 2736 | 2735 | ||
350 | 2737 | # create the service | 2736 | # create the service |
351 | 2738 | service = StorageServerService( | 2737 | service = StorageServerService( |
353 | 2739 | settings.api_server.TCP_PORT, auth_provider_class, status_port) | 2738 | settings.TCP_PORT, auth_provider_class, status_port) |
354 | 2740 | return service | 2739 | return service |
355 | 2741 | 2740 | ||
356 | === modified file 'magicicada/server/ssl_proxy.py' | |||
357 | --- magicicada/server/ssl_proxy.py 2016-08-08 01:14:41 +0000 | |||
358 | +++ magicicada/server/ssl_proxy.py 2017-04-22 22:01:02 +0000 | |||
359 | @@ -245,7 +245,7 @@ | |||
360 | 245 | self.status_service = create_status_service(self.factory, status_port) | 245 | self.status_service = create_status_service(self.factory, status_port) |
361 | 246 | self.status_service.setServiceParent(self) | 246 | self.status_service.setServiceParent(self) |
362 | 247 | # disable ssl compression | 247 | # disable ssl compression |
364 | 248 | if settings.ssl_proxy.DISABLE_SSL_COMPRESSION: | 248 | if settings.DISABLE_SSL_COMPRESSION: |
365 | 249 | disable_ssl_compression(logger) | 249 | disable_ssl_compression(logger) |
366 | 250 | 250 | ||
367 | 251 | @property | 251 | @property |
368 | @@ -265,7 +265,7 @@ | |||
369 | 265 | # setup stats in the factory | 265 | # setup stats in the factory |
370 | 266 | yield MultiService.startService(self) | 266 | yield MultiService.startService(self) |
371 | 267 | # only start the HeartbeatWriter if the interval is > 0 | 267 | # only start the HeartbeatWriter if the interval is > 0 |
373 | 268 | heartbeat_interval = float(settings.ssl_proxy.HEARTBEAT_INTERVAL) | 268 | heartbeat_interval = float(settings.HEARTBEAT_INTERVAL) |
374 | 269 | if heartbeat_interval > 0: | 269 | if heartbeat_interval > 0: |
375 | 270 | self.heartbeat_writer = stdio.StandardIO( | 270 | self.heartbeat_writer = stdio.StandardIO( |
376 | 271 | supervisor_utils.HeartbeatWriter(heartbeat_interval, logger)) | 271 | supervisor_utils.HeartbeatWriter(heartbeat_interval, logger)) |
377 | @@ -283,9 +283,9 @@ | |||
378 | 283 | 283 | ||
379 | 284 | def create_service(): | 284 | def create_service(): |
380 | 285 | """Create the service instance.""" | 285 | """Create the service instance.""" |
384 | 286 | server_key = settings.api_server.KEY | 286 | server_key = settings.CRT_KEY |
385 | 287 | server_crt = settings.api_server.CRT | 287 | server_crt = settings.CRT |
386 | 288 | server_crt_chain = settings.api_server.CRT_CHAIN | 288 | server_crt_chain = settings.CRT_CHAIN |
387 | 289 | 289 | ||
388 | 290 | ssl_cert = crypto.load_certificate(crypto.FILETYPE_PEM, server_crt) | 290 | ssl_cert = crypto.load_certificate(crypto.FILETYPE_PEM, server_crt) |
389 | 291 | if server_crt_chain: | 291 | if server_crt_chain: |
390 | @@ -296,11 +296,11 @@ | |||
391 | 296 | ssl_key = crypto.load_privatekey(crypto.FILETYPE_PEM, server_key) | 296 | ssl_key = crypto.load_privatekey(crypto.FILETYPE_PEM, server_key) |
392 | 297 | 297 | ||
393 | 298 | ssl_proxy = ProxyService(ssl_cert, ssl_key, ssl_cert_chain, | 298 | ssl_proxy = ProxyService(ssl_cert, ssl_key, ssl_cert_chain, |
395 | 299 | settings.ssl_proxy.PORT, | 299 | settings.SSL_PORT, |
396 | 300 | '127.0.0.1', | 300 | '127.0.0.1', |
400 | 301 | settings.api_server.TCP_PORT, | 301 | settings.TCP_PORT, |
401 | 302 | settings.ssl_proxy.SERVER_NAME, | 302 | settings.SSL_SERVER_NAME, |
402 | 303 | settings.ssl_proxy.STATUS_PORT) | 303 | settings.SSL_STATUS_PORT) |
403 | 304 | return ssl_proxy | 304 | return ssl_proxy |
404 | 305 | 305 | ||
405 | 306 | 306 | ||
406 | 307 | 307 | ||
407 | === modified file 'magicicada/server/testing/aq_helpers.py' | |||
408 | --- magicicada/server/testing/aq_helpers.py 2016-12-27 21:57:00 +0000 | |||
409 | +++ magicicada/server/testing/aq_helpers.py 2017-04-22 22:01:02 +0000 | |||
410 | @@ -332,8 +332,8 @@ | |||
411 | 332 | 0, # port | 332 | 0, # port |
412 | 333 | "localhost", self.port, | 333 | "localhost", self.port, |
413 | 334 | "ssl-proxy-test", 0) | 334 | "ssl-proxy-test", 0) |
416 | 335 | self.patch(settings.ssl_proxy, 'HEARTBEAT_INTERVAL', | 335 | self.patch( |
417 | 336 | self.ssl_proxy_heartbeat_interval) | 336 | settings, 'HEARTBEAT_INTERVAL', self.ssl_proxy_heartbeat_interval) |
418 | 337 | yield self.ssl_service.startService() | 337 | yield self.ssl_service.startService() |
419 | 338 | 338 | ||
420 | 339 | if os.path.exists(self.tmpdir): | 339 | if os.path.exists(self.tmpdir): |
421 | 340 | 340 | ||
422 | === modified file 'magicicada/server/testing/testcase.py' | |||
423 | --- magicicada/server/testing/testcase.py 2016-12-27 21:57:00 +0000 | |||
424 | +++ magicicada/server/testing/testcase.py 2017-04-22 22:01:02 +0000 | |||
425 | @@ -43,9 +43,9 @@ | |||
426 | 43 | from magicicada.testing.testcase import BaseTestCase | 43 | from magicicada.testing.testcase import BaseTestCase |
427 | 44 | 44 | ||
428 | 45 | logger = logging.getLogger(__name__) | 45 | logger = logging.getLogger(__name__) |
432 | 46 | server_key = settings.api_server.KEY | 46 | server_key = settings.CRT_KEY |
433 | 47 | server_crt = settings.api_server.CRT | 47 | server_crt = settings.CRT |
434 | 48 | server_crt_chain = settings.api_server.CRT_CHAIN | 48 | server_crt_chain = settings.CRT_CHAIN |
435 | 49 | 49 | ||
436 | 50 | 50 | ||
437 | 51 | class FakeTimestampChecker(object): | 51 | class FakeTimestampChecker(object): |
438 | @@ -365,7 +365,7 @@ | |||
439 | 365 | dummy_tokens[password] = user.id | 365 | dummy_tokens[password] = user.id |
440 | 366 | 366 | ||
441 | 367 | # tune the config for this tests | 367 | # tune the config for this tests |
443 | 368 | self.patch(settings.api_server, 'STORAGE_CHUNK_SIZE', 1024 * 64) | 368 | self.patch(settings, 'STORAGE_CHUNK_SIZE', 1024 * 64) |
444 | 369 | 369 | ||
445 | 370 | def save_req(self, req, name): | 370 | def save_req(self, req, name): |
446 | 371 | """Save a request for later use.""" | 371 | """Save a request for later use.""" |
447 | 372 | 372 | ||
448 | === modified file 'magicicada/server/tests/test_auth.py' | |||
449 | --- magicicada/server/tests/test_auth.py 2016-06-08 21:04:35 +0000 | |||
450 | +++ magicicada/server/tests/test_auth.py 2017-04-22 22:01:02 +0000 | |||
451 | @@ -96,7 +96,7 @@ | |||
452 | 96 | 96 | ||
453 | 97 | @defer.inlineCallbacks | 97 | @defer.inlineCallbacks |
454 | 98 | def test_authenticate(self): | 98 | def test_authenticate(self): |
456 | 99 | """The OAuth provider succeeds with a valid PLAINTEXT signature.""" | 99 | """The Auth provider succeeds with a valid password.""" |
457 | 100 | user = yield self.provider.authenticate(self.creds, None) | 100 | user = yield self.provider.authenticate(self.creds, None) |
458 | 101 | self.assertEqual(user.id, self.usr0.id) | 101 | self.assertEqual(user.id, self.usr0.id) |
459 | 102 | 102 | ||
460 | @@ -106,7 +106,7 @@ | |||
461 | 106 | 106 | ||
462 | 107 | @defer.inlineCallbacks | 107 | @defer.inlineCallbacks |
463 | 108 | def test_authenticate_failure(self): | 108 | def test_authenticate_failure(self): |
465 | 109 | """The OAuth provider succeeds with an invalid PLAINTEXT signature.""" | 109 | """The Auth provider succeeds with an invalid password.""" |
466 | 110 | auth_parameters = self.creds.copy() | 110 | auth_parameters = self.creds.copy() |
467 | 111 | auth_parameters['password'] = 'invalid' | 111 | auth_parameters['password'] = 'invalid' |
468 | 112 | 112 | ||
469 | @@ -115,7 +115,7 @@ | |||
470 | 115 | 115 | ||
471 | 116 | @defer.inlineCallbacks | 116 | @defer.inlineCallbacks |
472 | 117 | def test_authenticate_no_parameters(self): | 117 | def test_authenticate_no_parameters(self): |
474 | 118 | """The OAuth provider fails with no parameters.""" | 118 | """The Auth provider fails with no parameters.""" |
475 | 119 | user = yield self.provider.authenticate({}, None) | 119 | user = yield self.provider.authenticate({}, None) |
476 | 120 | self.assertEqual(user, None) | 120 | self.assertEqual(user, None) |
477 | 121 | 121 | ||
478 | @@ -248,7 +248,7 @@ | |||
479 | 248 | 248 | ||
480 | 249 | 249 | ||
481 | 250 | class ClientSimpleAuthTests(ClientDummyAuthTests): | 250 | class ClientSimpleAuthTests(ClientDummyAuthTests): |
483 | 251 | """Client authentication tests using the OAuth provider.""" | 251 | """Client authentication tests using the Auth provider.""" |
484 | 252 | 252 | ||
485 | 253 | auth_provider_class = SimpleAuthProvider | 253 | auth_provider_class = SimpleAuthProvider |
486 | 254 | 254 | ||
487 | 255 | 255 | ||
488 | === modified file 'magicicada/server/tests/test_content.py' | |||
489 | --- magicicada/server/tests/test_content.py 2016-11-07 01:37:46 +0000 | |||
490 | +++ magicicada/server/tests/test_content.py 2017-04-22 22:01:02 +0000 | |||
491 | @@ -565,7 +565,7 @@ | |||
492 | 565 | 565 | ||
493 | 566 | def test_putcontent_cancel_middle(self): | 566 | def test_putcontent_cancel_middle(self): |
494 | 567 | """Test putting content to a file and cancelling it in the middle.""" | 567 | """Test putting content to a file and cancelling it in the middle.""" |
496 | 568 | size = int(settings.api_server.STORAGE_CHUNK_SIZE * 1.5) | 568 | size = int(settings.STORAGE_CHUNK_SIZE * 1.5) |
497 | 569 | data = os.urandom(size) | 569 | data = os.urandom(size) |
498 | 570 | deflated_data = zlib.compress(data) | 570 | deflated_data = zlib.compress(data) |
499 | 571 | hash_object = content_hash_factory() | 571 | hash_object = content_hash_factory() |
500 | @@ -1146,7 +1146,7 @@ | |||
501 | 1146 | Test that a PutContent fails and is terminated as soon we get an | 1146 | Test that a PutContent fails and is terminated as soon we get an |
502 | 1147 | error, instead of wait until the full upload is done. | 1147 | error, instead of wait until the full upload is done. |
503 | 1148 | """ | 1148 | """ |
505 | 1149 | chunk_size = settings.api_server.STORAGE_CHUNK_SIZE | 1149 | chunk_size = settings.STORAGE_CHUNK_SIZE |
506 | 1150 | user, content_user = self._get_users(chunk_size ** 2) | 1150 | user, content_user = self._get_users(chunk_size ** 2) |
507 | 1151 | # create the file | 1151 | # create the file |
508 | 1152 | a_file = user.root.make_file(u"A new file") | 1152 | a_file = user.root.make_file(u"A new file") |
509 | @@ -1596,7 +1596,7 @@ | |||
510 | 1596 | self.handler = self.add_memento_handler(server.logger, level=0) | 1596 | self.handler = self.add_memento_handler(server.logger, level=0) |
511 | 1597 | yield super(TestMultipartPutContent, self).setUp() | 1597 | yield super(TestMultipartPutContent, self).setUp() |
512 | 1598 | # override defaults set by TestWithDatabase.setUp. | 1598 | # override defaults set by TestWithDatabase.setUp. |
514 | 1599 | self.patch(settings.api_server, 'STORAGE_CHUNK_SIZE', 1024) | 1599 | self.patch(settings, 'STORAGE_CHUNK_SIZE', 1024) |
515 | 1600 | 1600 | ||
516 | 1601 | def get_data(self, size): | 1601 | def get_data(self, size): |
517 | 1602 | """Return random data of the specified size.""" | 1602 | """Return random data of the specified size.""" |
518 | @@ -1650,7 +1650,7 @@ | |||
519 | 1650 | @defer.inlineCallbacks | 1650 | @defer.inlineCallbacks |
520 | 1651 | def test_resume_putcontent(self): | 1651 | def test_resume_putcontent(self): |
521 | 1652 | """Test that the client can resume a putcontent request.""" | 1652 | """Test that the client can resume a putcontent request.""" |
523 | 1653 | self.patch(settings.api_server, 'STORAGE_CHUNK_SIZE', 1024 * 64) | 1653 | self.patch(settings, 'STORAGE_CHUNK_SIZE', 1024 * 64) |
524 | 1654 | size = 2 * 1024 * 512 | 1654 | size = 2 * 1024 * 512 |
525 | 1655 | StorageUser.objects.filter(id=self.usr0.id).update( | 1655 | StorageUser.objects.filter(id=self.usr0.id).update( |
526 | 1656 | max_storage_bytes=size * 2) | 1656 | max_storage_bytes=size * 2) |
527 | @@ -1776,7 +1776,7 @@ | |||
528 | 1776 | 1776 | ||
529 | 1777 | It receives a new upload_id. | 1777 | It receives a new upload_id. |
530 | 1778 | """ | 1778 | """ |
532 | 1779 | self.patch(settings.api_server, 'STORAGE_CHUNK_SIZE', 1024 * 32) | 1779 | self.patch(settings, 'STORAGE_CHUNK_SIZE', 1024 * 32) |
533 | 1780 | size = 2 * 1024 * 128 | 1780 | size = 2 * 1024 * 128 |
534 | 1781 | StorageUser.objects.filter(id=self.usr0.id).update( | 1781 | StorageUser.objects.filter(id=self.usr0.id).update( |
535 | 1782 | max_storage_bytes=size * 2) | 1782 | max_storage_bytes=size * 2) |
536 | @@ -1848,7 +1848,7 @@ | |||
537 | 1848 | 1848 | ||
538 | 1849 | def test_putcontent_corrupt(self): | 1849 | def test_putcontent_corrupt(self): |
539 | 1850 | """Put content on a file with corrupt data.""" | 1850 | """Put content on a file with corrupt data.""" |
541 | 1851 | self.patch(settings.api_server, 'STORAGE_CHUNK_SIZE', 1024 * 64) | 1851 | self.patch(settings, 'STORAGE_CHUNK_SIZE', 1024 * 64) |
542 | 1852 | size = 2 * 1024 * 512 | 1852 | size = 2 * 1024 * 512 |
543 | 1853 | StorageUser.objects.filter(id=self.usr0.id).update( | 1853 | StorageUser.objects.filter(id=self.usr0.id).update( |
544 | 1854 | max_storage_bytes=size * 2) | 1854 | max_storage_bytes=size * 2) |
545 | @@ -1965,7 +1965,7 @@ | |||
546 | 1965 | Test that a PutContent fails and is terminated as soon we get an | 1965 | Test that a PutContent fails and is terminated as soon we get an |
547 | 1966 | error, instead of wait until the full upload is done. | 1966 | error, instead of wait until the full upload is done. |
548 | 1967 | """ | 1967 | """ |
550 | 1968 | chunk_size = settings.api_server.STORAGE_CHUNK_SIZE | 1968 | chunk_size = settings.STORAGE_CHUNK_SIZE |
551 | 1969 | user = self.make_user(max_storage_bytes=chunk_size ** 2) | 1969 | user = self.make_user(max_storage_bytes=chunk_size ** 2) |
552 | 1970 | content_user = User( | 1970 | content_user = User( |
553 | 1971 | self.service.factory.content, user.id, user.root_volume_id, | 1971 | self.service.factory.content, user.id, user.root_volume_id, |
554 | @@ -2079,11 +2079,11 @@ | |||
555 | 2079 | """Setup the test.""" | 2079 | """Setup the test.""" |
556 | 2080 | yield super(TestChunkedContent, self).setUp() | 2080 | yield super(TestChunkedContent, self).setUp() |
557 | 2081 | # tune the config for this tests | 2081 | # tune the config for this tests |
559 | 2082 | self.patch(settings.api_server, 'STORAGE_CHUNK_SIZE', 1024 * 1024) | 2082 | self.patch(settings, 'STORAGE_CHUNK_SIZE', 1024 * 1024) |
560 | 2083 | 2083 | ||
561 | 2084 | def test_putcontent_chunked(self, put_fail=False, get_fail=False): | 2084 | def test_putcontent_chunked(self, put_fail=False, get_fail=False): |
562 | 2085 | """Checks a chunked putcontent.""" | 2085 | """Checks a chunked putcontent.""" |
564 | 2086 | size = int(settings.api_server.STORAGE_CHUNK_SIZE * 1.5) | 2086 | size = int(settings.STORAGE_CHUNK_SIZE * 1.5) |
565 | 2087 | data = os.urandom(size) | 2087 | data = os.urandom(size) |
566 | 2088 | deflated_data = zlib.compress(data) | 2088 | deflated_data = zlib.compress(data) |
567 | 2089 | hash_object = content_hash_factory() | 2089 | hash_object = content_hash_factory() |
568 | @@ -2162,7 +2162,7 @@ | |||
569 | 2162 | 2162 | ||
570 | 2163 | def test_deferred_add_part_to_uj(self): | 2163 | def test_deferred_add_part_to_uj(self): |
571 | 2164 | """Check that parts are added to upload job only after a limit.""" | 2164 | """Check that parts are added to upload job only after a limit.""" |
573 | 2165 | size = int(settings.api_server.STORAGE_CHUNK_SIZE * 2.5) | 2165 | size = int(settings.STORAGE_CHUNK_SIZE * 2.5) |
574 | 2166 | data = os.urandom(size) | 2166 | data = os.urandom(size) |
575 | 2167 | deflated_data = zlib.compress(data) | 2167 | deflated_data = zlib.compress(data) |
576 | 2168 | hash_object = content_hash_factory() | 2168 | hash_object = content_hash_factory() |
577 | @@ -2344,7 +2344,7 @@ | |||
578 | 2344 | def setUp(self): | 2344 | def setUp(self): |
579 | 2345 | """Setup the test.""" | 2345 | """Setup the test.""" |
580 | 2346 | yield super(TestUploadJob, self).setUp() | 2346 | yield super(TestUploadJob, self).setUp() |
582 | 2347 | self.chunk_size = settings.api_server.STORAGE_CHUNK_SIZE | 2347 | self.chunk_size = settings.STORAGE_CHUNK_SIZE |
583 | 2348 | self.half_size = self.chunk_size / 2 | 2348 | self.half_size = self.chunk_size / 2 |
584 | 2349 | self.double_size = self.chunk_size * 2 | 2349 | self.double_size = self.chunk_size * 2 |
585 | 2350 | self.user = self.make_user(max_storage_bytes=self.chunk_size ** 2) | 2350 | self.user = self.make_user(max_storage_bytes=self.chunk_size ** 2) |
586 | @@ -2673,7 +2673,7 @@ | |||
587 | 2673 | def setUp(self): | 2673 | def setUp(self): |
588 | 2674 | """Setup the test.""" | 2674 | """Setup the test.""" |
589 | 2675 | yield super(TestNode, self).setUp() | 2675 | yield super(TestNode, self).setUp() |
591 | 2676 | self.chunk_size = settings.api_server.STORAGE_CHUNK_SIZE | 2676 | self.chunk_size = settings.STORAGE_CHUNK_SIZE |
592 | 2677 | self.half_size = self.chunk_size / 2 | 2677 | self.half_size = self.chunk_size / 2 |
593 | 2678 | self.double_size = self.chunk_size * 2 | 2678 | self.double_size = self.chunk_size * 2 |
594 | 2679 | self.user = self.make_user(max_storage_bytes=self.chunk_size ** 2) | 2679 | self.user = self.make_user(max_storage_bytes=self.chunk_size ** 2) |
595 | @@ -3073,7 +3073,7 @@ | |||
596 | 3073 | def test_add_part(self): | 3073 | def test_add_part(self): |
597 | 3074 | """Test add_part method.""" | 3074 | """Test add_part method.""" |
598 | 3075 | dbuj = yield self._make_uj() | 3075 | dbuj = yield self._make_uj() |
600 | 3076 | chunk_size = int(settings.api_server.STORAGE_CHUNK_SIZE) + 1 | 3076 | chunk_size = int(settings.STORAGE_CHUNK_SIZE) + 1 |
601 | 3077 | yield dbuj.add_part(chunk_size) | 3077 | yield dbuj.add_part(chunk_size) |
602 | 3078 | 3078 | ||
603 | 3079 | # check it called rpc dal correctly | 3079 | # check it called rpc dal correctly |
604 | 3080 | 3080 | ||
605 | === modified file 'magicicada/server/tests/test_generations.py' | |||
606 | --- magicicada/server/tests/test_generations.py 2016-05-29 20:58:45 +0000 | |||
607 | +++ magicicada/server/tests/test_generations.py 2017-04-22 22:01:02 +0000 | |||
608 | @@ -139,7 +139,7 @@ | |||
609 | 139 | yield client.dummy_authenticate("open sesame") | 139 | yield client.dummy_authenticate("open sesame") |
610 | 140 | # create some nodes to get a delta | 140 | # create some nodes to get a delta |
611 | 141 | limit = 10 | 141 | limit = 10 |
613 | 142 | self.patch(server.settings.api_server, 'DELTA_MAX_SIZE', limit) | 142 | self.patch(server.settings, 'DELTA_MAX_SIZE', limit) |
614 | 143 | for i in range(20): | 143 | for i in range(20): |
615 | 144 | self.usr0.root.make_file(u"name%s" % i) | 144 | self.usr0.root.make_file(u"name%s" % i) |
616 | 145 | req = yield client.get_delta(request.ROOT, 5) | 145 | req = yield client.get_delta(request.ROOT, 5) |
617 | @@ -182,7 +182,7 @@ | |||
618 | 182 | def auth(client): | 182 | def auth(client): |
619 | 183 | yield client.dummy_authenticate("open sesame") | 183 | yield client.dummy_authenticate("open sesame") |
620 | 184 | # create some nodes to get a delta | 184 | # create some nodes to get a delta |
622 | 185 | self.patch(server.settings.api_server, 'MAX_DELTA_INFO', 5) | 185 | self.patch(server.settings, 'MAX_DELTA_INFO', 5) |
623 | 186 | for i in range(20): | 186 | for i in range(20): |
624 | 187 | self.usr0.root.make_file(u"name%s" % i) | 187 | self.usr0.root.make_file(u"name%s" % i) |
625 | 188 | req = yield client.get_delta(request.ROOT, 5) | 188 | req = yield client.get_delta(request.ROOT, 5) |
626 | @@ -248,7 +248,7 @@ | |||
627 | 248 | def auth(client): | 248 | def auth(client): |
628 | 249 | yield client.dummy_authenticate("open sesame") | 249 | yield client.dummy_authenticate("open sesame") |
629 | 250 | # create some nodes to get a delta | 250 | # create some nodes to get a delta |
631 | 251 | self.patch(server.settings.api_server, 'MAX_DELTA_INFO', 5) | 251 | self.patch(server.settings, 'MAX_DELTA_INFO', 5) |
632 | 252 | for i in range(20): | 252 | for i in range(20): |
633 | 253 | self.usr0.root.make_file(u"name%s" % i) | 253 | self.usr0.root.make_file(u"name%s" % i) |
634 | 254 | req = yield client.get_delta(request.ROOT, 5) | 254 | req = yield client.get_delta(request.ROOT, 5) |
635 | 255 | 255 | ||
636 | === modified file 'magicicada/server/tests/test_server.py' | |||
637 | --- magicicada/server/tests/test_server.py 2016-11-07 01:37:46 +0000 | |||
638 | +++ magicicada/server/tests/test_server.py 2017-04-22 22:01:02 +0000 | |||
639 | @@ -349,7 +349,7 @@ | |||
640 | 349 | 349 | ||
641 | 350 | def test_protocol_ref_enabled(self): | 350 | def test_protocol_ref_enabled(self): |
642 | 351 | """Test that protocol weakref is disabled in tests.""" | 351 | """Test that protocol weakref is disabled in tests.""" |
644 | 352 | self.patch(settings.api_server, 'PROTOCOL_WEAKREF', True) | 352 | self.patch(settings, 'PROTOCOL_WEAKREF', True) |
645 | 353 | _server = StorageServer() | 353 | _server = StorageServer() |
646 | 354 | response = StorageServerRequestResponse( | 354 | response = StorageServerRequestResponse( |
647 | 355 | protocol=_server, message=self.make_protocol_message()) | 355 | protocol=_server, message=self.make_protocol_message()) |
648 | @@ -358,7 +358,7 @@ | |||
649 | 358 | 358 | ||
650 | 359 | def test_protocol_ref_disabled(self): | 359 | def test_protocol_ref_disabled(self): |
651 | 360 | """Test that protocol weakref is disabled in tests.""" | 360 | """Test that protocol weakref is disabled in tests.""" |
653 | 361 | self.patch(settings.api_server, 'PROTOCOL_WEAKREF', False) | 361 | self.patch(settings, 'PROTOCOL_WEAKREF', False) |
654 | 362 | _server = StorageServer() | 362 | _server = StorageServer() |
655 | 363 | response = StorageServerRequestResponse( | 363 | response = StorageServerRequestResponse( |
656 | 364 | protocol=_server, message=self.make_protocol_message()) | 364 | protocol=_server, message=self.make_protocol_message()) |
657 | @@ -2340,7 +2340,7 @@ | |||
658 | 2340 | 2340 | ||
659 | 2341 | def test_reset_send_delta_info_counter(self): | 2341 | def test_reset_send_delta_info_counter(self): |
660 | 2342 | """Test that the count is reset on each iteration.""" | 2342 | """Test that the count is reset on each iteration.""" |
662 | 2343 | self.patch(settings.api_server, 'MAX_DELTA_INFO', 5) | 2343 | self.patch(settings, 'MAX_DELTA_INFO', 5) |
663 | 2344 | # create a few fake nodes | 2344 | # create a few fake nodes |
664 | 2345 | nodes = [] | 2345 | nodes = [] |
665 | 2346 | right_now = now() | 2346 | right_now = now() |
666 | @@ -2415,7 +2415,7 @@ | |||
667 | 2415 | @defer.inlineCallbacks | 2415 | @defer.inlineCallbacks |
668 | 2416 | def test_chunked_get_from_scratch(self): | 2416 | def test_chunked_get_from_scratch(self): |
669 | 2417 | """Get the nodes list in chunks.""" | 2417 | """Get the nodes list in chunks.""" |
671 | 2418 | self.patch(settings.api_server, 'GET_FROM_SCRATCH_LIMIT', 5) | 2418 | self.patch(settings, 'GET_FROM_SCRATCH_LIMIT', 5) |
672 | 2419 | # build fake nodes | 2419 | # build fake nodes |
673 | 2420 | nodes = [] | 2420 | nodes = [] |
674 | 2421 | right_now = now() | 2421 | right_now = now() |
675 | @@ -2612,7 +2612,7 @@ | |||
676 | 2612 | def test_trace_users(self): | 2612 | def test_trace_users(self): |
677 | 2613 | """Check trace users are correctly set.""" | 2613 | """Check trace users are correctly set.""" |
678 | 2614 | # set a specific config to test | 2614 | # set a specific config to test |
680 | 2615 | self.patch(settings.api_server, 'TRACE_USERS', ['foo', 'bar', 'baz']) | 2615 | self.patch(settings, 'TRACE_USERS', ['foo', 'bar', 'baz']) |
681 | 2616 | factory = StorageServerFactory() | 2616 | factory = StorageServerFactory() |
682 | 2617 | self.assertEqual(factory.trace_users, set(['foo', 'bar', 'baz'])) | 2617 | self.assertEqual(factory.trace_users, set(['foo', 'bar', 'baz'])) |
683 | 2618 | 2618 | ||
684 | 2619 | 2619 | ||
685 | === modified file 'magicicada/server/tests/test_ssl_proxy.py' | |||
686 | --- magicicada/server/tests/test_ssl_proxy.py 2016-08-14 20:40:09 +0000 | |||
687 | +++ magicicada/server/tests/test_ssl_proxy.py 2017-04-22 22:01:02 +0000 | |||
688 | @@ -54,8 +54,8 @@ | |||
689 | 54 | ssl_proxy.logger, level=logging.DEBUG) | 54 | ssl_proxy.logger, level=logging.DEBUG) |
690 | 55 | self.metrics = MetricReceiver() | 55 | self.metrics = MetricReceiver() |
691 | 56 | self.patch(metrics, 'get_meter', lambda n: self.metrics) | 56 | self.patch(metrics, 'get_meter', lambda n: self.metrics) |
694 | 57 | self.patch(settings.ssl_proxy, 'HEARTBEAT_INTERVAL', | 57 | self.patch( |
695 | 58 | self.ssl_proxy_heartbeat_interval) | 58 | settings, 'HEARTBEAT_INTERVAL', self.ssl_proxy_heartbeat_interval) |
696 | 59 | 59 | ||
697 | 60 | @defer.inlineCallbacks | 60 | @defer.inlineCallbacks |
698 | 61 | def test_start_stop(self): | 61 | def test_start_stop(self): |
699 | @@ -87,8 +87,8 @@ | |||
700 | 87 | # keep metrics in our MetricReceiver | 87 | # keep metrics in our MetricReceiver |
701 | 88 | self.metrics = MetricReceiver() | 88 | self.metrics = MetricReceiver() |
702 | 89 | self.patch(metrics, 'get_meter', lambda n: self.metrics) | 89 | self.patch(metrics, 'get_meter', lambda n: self.metrics) |
705 | 90 | self.patch(settings.ssl_proxy, 'HEARTBEAT_INTERVAL', | 90 | self.patch( |
706 | 91 | self.ssl_proxy_heartbeat_interval) | 91 | settings, 'HEARTBEAT_INTERVAL', self.ssl_proxy_heartbeat_interval) |
707 | 92 | yield self.ssl_service.startService() | 92 | yield self.ssl_service.startService() |
708 | 93 | self.addCleanup(self.ssl_service.stopService) | 93 | self.addCleanup(self.ssl_service.stopService) |
709 | 94 | 94 | ||
710 | 95 | 95 | ||
711 | === modified file 'magicicada/settings/__init__.py' | |||
712 | --- magicicada/settings/__init__.py 2016-12-27 21:57:00 +0000 | |||
713 | +++ magicicada/settings/__init__.py 2017-04-22 22:01:02 +0000 | |||
714 | @@ -35,6 +35,18 @@ | |||
715 | 35 | 35 | ||
716 | 36 | from psycopg2.extensions import ISOLATION_LEVEL_REPEATABLE_READ | 36 | from psycopg2.extensions import ISOLATION_LEVEL_REPEATABLE_READ |
717 | 37 | 37 | ||
718 | 38 | |||
719 | 39 | def get_file_content(folder, filename): | ||
720 | 40 | filepath = os.path.join(folder, filename) | ||
721 | 41 | if not os.path.exists(filepath): | ||
722 | 42 | filepath = os.path.join(folder, 'dev-' + filename) | ||
723 | 43 | |||
724 | 44 | with open(filepath) as f: | ||
725 | 45 | content = f.read() | ||
726 | 46 | |||
727 | 47 | return content | ||
728 | 48 | |||
729 | 49 | |||
730 | 38 | BASE_DIR = os.path.dirname( | 50 | BASE_DIR = os.path.dirname( |
731 | 39 | os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) | 51 | os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) |
732 | 40 | 52 | ||
733 | @@ -216,8 +228,37 @@ | |||
734 | 216 | SYSLOG_FORMAT = ( | 228 | SYSLOG_FORMAT = ( |
735 | 217 | '%(processName)-13s %(levelname)-8s %(name)s[%(process)d]: %(message)s') | 229 | '%(processName)-13s %(levelname)-8s %(name)s[%(process)d]: %(message)s') |
736 | 218 | 230 | ||
739 | 219 | from . import api_server # noqa | 231 | |
740 | 220 | from . import ssl_proxy # noqa | 232 | # Server settings |
741 | 233 | |||
742 | 234 | API_SERVER_NAME = 'filesync-server' | ||
743 | 235 | API_STATUS_PORT = 21102 | ||
744 | 236 | CERTS_FOLDER = os.path.join(BASE_DIR, 'certs') | ||
745 | 237 | # the `crt` key with the content of `cacert.pem` file | ||
746 | 238 | CRT = get_file_content(CERTS_FOLDER, 'cacert.pem') | ||
747 | 239 | CRT_CHAIN = None | ||
748 | 240 | # the `key` key with the content of `privkey.pem` file | ||
749 | 241 | CRT_KEY = get_file_content(CERTS_FOLDER, 'privkey.pem') | ||
750 | 242 | DELTA_MAX_SIZE = 1000 | ||
751 | 243 | DISABLE_SSL_COMPRESSION = True | ||
752 | 244 | GC_DEBUG = False | ||
753 | 245 | GET_FROM_SCRATCH_LIMIT = 2000 | ||
754 | 246 | GRACEFUL_SHUTDOWN = True | ||
755 | 247 | HEARTBEAT_INTERVAL = 5 | ||
756 | 248 | IDLE_TIMEOUT = 7200 | ||
757 | 249 | MAGIC_UPLOAD_ACTIVE = True | ||
758 | 250 | MAX_DELTA_INFO = 20 | ||
759 | 251 | PROTOCOL_WEAKREF = False | ||
760 | 252 | SSL_LOG_FILENAME = 'ssl-proxy.log' | ||
761 | 253 | SSL_PORT = 21101 | ||
762 | 254 | SSL_SERVER_NAME = 'ssl-proxy' | ||
763 | 255 | SSL_STATUS_PORT = 21103 | ||
764 | 256 | STATS_LOG_INTERVAL = 0 | ||
765 | 257 | STORAGE_CHUNK_SIZE = 5242880 | ||
766 | 258 | TCP_PORT = 21100 | ||
767 | 259 | TRACE_USERS = ['test', 'etc'] | ||
768 | 260 | UPLOAD_BUFFER_MAX_SIZE = 10485761 | ||
769 | 261 | STORAGE_BASEDIR = os.path.join(BASE_DIR, 'tmp', 'filestorage') | ||
770 | 221 | 262 | ||
771 | 222 | 263 | ||
772 | 223 | try: | 264 | try: |
773 | 224 | 265 | ||
774 | === removed file 'magicicada/settings/api_server.py' | |||
775 | --- magicicada/settings/api_server.py 2016-05-29 23:43:14 +0000 | |||
776 | +++ magicicada/settings/api_server.py 1970-01-01 00:00:00 +0000 | |||
777 | @@ -1,58 +0,0 @@ | |||
778 | 1 | # Copyright 2008-2015 Canonical | ||
779 | 2 | # Copyright 2015 Chicharreros (https://launchpad.net/~chicharreros) | ||
780 | 3 | # | ||
781 | 4 | # This program is free software: you can redistribute it and/or modify | ||
782 | 5 | # it under the terms of the GNU Affero General Public License as | ||
783 | 6 | # published by the Free Software Foundation, either version 3 of the | ||
784 | 7 | # License, or (at your option) any later version. | ||
785 | 8 | # | ||
786 | 9 | # This program is distributed in the hope that it will be useful, | ||
787 | 10 | # but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
788 | 11 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
789 | 12 | # GNU Affero General Public License for more details. | ||
790 | 13 | # | ||
791 | 14 | # You should have received a copy of the GNU Affero General Public License | ||
792 | 15 | # along with this program. If not, see <http://www.gnu.org/licenses/>. | ||
793 | 16 | # | ||
794 | 17 | # For further info, check http://launchpad.net/filesync-server | ||
795 | 18 | |||
796 | 19 | import os | ||
797 | 20 | from django.conf import settings | ||
798 | 21 | |||
799 | 22 | |||
800 | 23 | def get_file_content(folder, filename): | ||
801 | 24 | filepath = os.path.join(folder, filename) | ||
802 | 25 | if not os.path.exists(filepath): | ||
803 | 26 | filepath = os.path.join(folder, 'dev-' + filename) | ||
804 | 27 | |||
805 | 28 | with open(filepath) as f: | ||
806 | 29 | content = f.read() | ||
807 | 30 | |||
808 | 31 | return content | ||
809 | 32 | |||
810 | 33 | |||
811 | 34 | CERTS_FOLDER = os.path.join(settings.BASE_DIR, 'certs') | ||
812 | 35 | # the `crt` key with the content of `cacert.pem` file | ||
813 | 36 | CRT = get_file_content(CERTS_FOLDER, 'cacert.pem') | ||
814 | 37 | CRT_CHAIN = None | ||
815 | 38 | # the `key` key with the content of `privkey.pem` file | ||
816 | 39 | KEY = get_file_content(CERTS_FOLDER, 'privkey.pem') | ||
817 | 40 | DELTA_MAX_SIZE = 1000 | ||
818 | 41 | GC_DEBUG = True | ||
819 | 42 | GET_FROM_SCRATCH_LIMIT = 2000 | ||
820 | 43 | GRACEFUL_SHUTDOWN = True | ||
821 | 44 | HEARTBEAT_INTERVAL = 5 | ||
822 | 45 | IDLE_TIMEOUT = 7200 | ||
823 | 46 | MAGIC_UPLOAD_ACTIVE = True | ||
824 | 47 | MAX_DELTA_INFO = 20 | ||
825 | 48 | METRICS_NAMESPACE = 'development.filesync.server' | ||
826 | 49 | PROTOCOL_WEAKREF = False | ||
827 | 50 | SERVERNAME = 'filesyncserver-development' | ||
828 | 51 | SLI_METRIC_NAMESPACE = None | ||
829 | 52 | STATS_LOG_INTERVAL = 0 | ||
830 | 53 | STATUS_PORT = 21102 | ||
831 | 54 | STORAGE_CHUNK_SIZE = 5242880 | ||
832 | 55 | TCP_PORT = 21100 | ||
833 | 56 | TRACE_USERS = ['test', 'etc'] | ||
834 | 57 | UPLOAD_BUFFER_MAX_SIZE = 10485761 | ||
835 | 58 | STORAGE_BASEDIR = os.path.join(settings.BASE_DIR, 'tmp', 'filestorage') | ||
836 | 59 | 0 | ||
837 | === removed file 'magicicada/settings/ssl_proxy.py' | |||
838 | --- magicicada/settings/ssl_proxy.py 2016-05-31 22:59:18 +0000 | |||
839 | +++ magicicada/settings/ssl_proxy.py 1970-01-01 00:00:00 +0000 | |||
840 | @@ -1,24 +0,0 @@ | |||
841 | 1 | # Copyright 2008-2015 Canonical | ||
842 | 2 | # Copyright 2015-2016 Chicharreros (https://launchpad.net/~chicharreros) | ||
843 | 3 | # | ||
844 | 4 | # This program is free software: you can redistribute it and/or modify | ||
845 | 5 | # it under the terms of the GNU Affero General Public License as | ||
846 | 6 | # published by the Free Software Foundation, either version 3 of the | ||
847 | 7 | # License, or (at your option) any later version. | ||
848 | 8 | # | ||
849 | 9 | # This program is distributed in the hope that it will be useful, | ||
850 | 10 | # but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
851 | 11 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
852 | 12 | # GNU Affero General Public License for more details. | ||
853 | 13 | # | ||
854 | 14 | # You should have received a copy of the GNU Affero General Public License | ||
855 | 15 | # along with this program. If not, see <http://www.gnu.org/licenses/>. | ||
856 | 16 | # | ||
857 | 17 | # For further info, check http://launchpad.net/filesync-server | ||
858 | 18 | |||
859 | 19 | DISABLE_SSL_COMPRESSION = True | ||
860 | 20 | HEARTBEAT_INTERVAL = 5 | ||
861 | 21 | LOG_FILENAME = 'ssl-proxy.log' | ||
862 | 22 | PORT = 21101 | ||
863 | 23 | SERVER_NAME = 'ssl-proxy-dev' | ||
864 | 24 | STATUS_PORT = 21103 |
Awesome, thanks!