Adding new sites with allocated ports re-shuffled, race between haproxy and nginx reload
Bug #1865945 reported by
Haw Loeung
This bug affects 1 person
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| Content Cache Charm |
Fix Released
|
Critical
|
Haw Loeung | ||
Bug Description
Hi,
When adding new sites, allocated ports for both nginx cache and the backend haproxy stanzas are reshuffled around. This can cause issues with requests hitting the wrong backends and caching that. We should fix this, likely by having haproxy and nginx reload at the same time. Or store a persistent map of sites and allocated ports.
We could also use this opportunity to remove the haproxy backend layer (nginx -> haproxy -> backend) since we now have full metrics from nginx's LUA script.
Related branches
~hloeung/content-cache-charm:nagios-monitoring
- Joel Sing (community): Approve (+1)
- Canonical IS Reviewers: Pending requested
-
Diff: 16 lines (+6/-0)1 file modifiedreactive/content_cache.py (+6/-0)
~hloeung/content-cache-charm:sites-persistent-ports-kv
- Stuart Bishop (community): Approve
- Content Cache Charmers: Pending requested
-
Diff: 387 lines (+311/-9)3 files modifiedreactive/content_cache.py (+78/-9)
tests/unit/files/config_test_sites_map.txt (+127/-0)
tests/unit/test_content_cache.py (+106/-0)
~hloeung/content-cache-charm:sites-persistent-ports
- Content Cache Charmers: Pending requested
-
Diff: 234 lines (+187/-6)3 files modifiedreactive/content_cache.py (+17/-6)
tests/unit/files/config_test_sites_map.txt (+127/-0)
tests/unit/test_content_cache.py (+43/-0)
~hloeung/content-cache-charm:master
- Joel Sing (community): Approve (+1)
- Canonical IS Reviewers: Pending requested
-
Diff: 82 lines (+66/-0)2 files modifiedlib/haproxy.py (+32/-0)
tests/unit/test_haproxy.py (+34/-0)
~hloeung/content-cache-charm:master
- Joel Sing (community): Approve (+1)
- Canonical IS Reviewers: Pending requested
-
Diff: 186 lines (+42/-31)2 files modifiedreactive/content_cache.py (+23/-12)
tests/unit/test_content_cache.py (+19/-19)
| summary: |
- Adding new sites with port allocates shuffling, race between haproxy and - nginx reload + Adding new sites with allocated ports re-shuffled, race between haproxy + and nginx reload |
| Changed in content-cache-charm: | |
| importance: | Undecided → High |
| Changed in content-cache-charm: | |
| status: | New → Triaged |
Revision history for this message
| Haw Loeung (hloeung) wrote | #1 |
Revision history for this message
| Haw Loeung (hloeung) wrote | #2 |
| Changed in content-cache-charm: | |
| importance: | High → Critical |
Revision history for this message
| Joel Sing (jsing) wrote | #3 |
| Changed in content-cache-charm: | |
| assignee: | nobody → Haw Loeung (hloeung) |
| Changed in content-cache-charm: | |
| status: | Triaged → In Progress |
| Changed in content-cache-charm: | |
| status: | In Progress → Fix Committed |
| Changed in content-cache-charm: | |
| status: | Fix Committed → Fix Released |
To post a comment you must log in.
So what's left is to add mapping sites to ports so new sites allocated unused port-pairs and existing sites retains the same set of port-pairs.