lp:~ubuntuone-pqm-team/python-swiftclient/trunk

Created by Sidnei da Silva on 2013-06-12 and last modified on 2018-04-20
Get this branch:
bzr branch lp:~ubuntuone-pqm-team/python-swiftclient/trunk

Branch merges

Related bugs

Related blueprints

Branch information

Owner:
Ubuntu One PQM Team
Project:
python-swiftclient
Status:
Development

Import details

Import Status: Reviewed

This branch is an import of the HEAD branch of the Git repository at git://github.com/openstack/python-swiftclient.git.

The next import is scheduled to run in 3 hours.

Last successful import was 2 hours ago.

Import started 2 hours ago on izar and finished 2 hours ago taking 15 seconds — see the log
Import started 8 hours ago on alnitak and finished 8 hours ago taking 15 seconds — see the log
Import started 14 hours ago on izar and finished 14 hours ago taking 20 seconds — see the log
Import started 20 hours ago on alnitak and finished 20 hours ago taking 20 seconds — see the log
Import started on 2018-04-22 on izar and finished on 2018-04-22 taking 20 seconds — see the log
Import started on 2018-04-21 on alnitak and finished on 2018-04-21 taking 20 seconds — see the log
Import started on 2018-04-21 on alnitak and finished on 2018-04-21 taking 25 seconds — see the log
Import started on 2018-04-21 on izar and finished on 2018-04-21 taking 20 seconds — see the log
Import started on 2018-04-21 on alnitak and finished on 2018-04-21 taking 20 seconds — see the log
Import started on 2018-04-20 on izar and finished on 2018-04-20 taking 20 seconds — see the log

Recent revisions

528. By Tovin Seven on 2018-04-20

Trivial: Update pypi url to new url

Pypi url changed from [1] to [2]

[1] https://pypi.python.org/pypi/<package>
[2] https://pypi.org/project/<package>

Change-Id: Ia406e9c8be6ba672b96e8584ef26f92348c8328b

527. By Zuul <email address hidden> on 2018-04-12

Merge "Remove trailing white space in tox.ini"

526. By Nguyen Hai on 2018-04-06

Remove py34 from envlist in tox.ini

py35 is enough.

Change-Id: Iebd7a6741dd60ed2fb11d1758bfec8e03e30a086

525. By Kota Tsuyuzaki on 2018-03-13

Add force auth retry mode in swiftclient

This patch attemps to add an option to force get_auth call while retrying
an operation even if it gets errors other than 401 Unauthorized.

Why we need this:
The main reason why we need this is current python-swiftclient requests could
never get succeeded under certion situation using third party proxies/load balancers
between the client and swift-proxy server. I think, it would be general situation
of the use case.

Specifically describing nginx case, the nginx can close the socket from the client
when the response code from swift is not 2xx series. In default, nginx can wait the
buffers from the client for a while (default 30s)[1] but after the time past, nginx
will close the socket immediately. Unfortunately, if python-swiftclient has still been
sending the data into the socket, python-swiftclient will get socket error (EPIPE,
BrokenPipe). From the swiftclient perspective, this is absolutely not an auth error,
so current python-swiftclient will continue to retry without re-auth.
However, if the root cause is sort of 401 (i.e. nginx got 401 unauthorized from the
swift-proxy because of token expiration), swiftclient will loop 401 -> EPIPE -> 401...
until it consume the max retry times.

In particlar, less time to live of the token and multipart object upload with large
segments could not get succeeded as below:

Connection Model:

python-swiftclient -> nginx -> swift-proxy -> swift-backend

Case: Try to create slo with large segments and the auth token expired with 1 hour

1. client create a connection to nginx with successful response from swift-proxy and its auth
2. client continue to put large segment objects
   (e.g. 1~5GB for each and the total would 20~30GB, i.e. 20~30 segments)
3. after some of segments uploaded, 1 hour past but client is still trying to
   send remaining segment objects.
4. nginx got 401 from swift-proxy for a request and wait that the connection is closed
   from the client but timeout past because the python-swiftclient is still sending much data
   into the socket before reading the 401 response.
5. client got socket error because nginx closed the connection during sending the buffer.
6. client retries a new connection to nginx without re-auth...

<loop 4-6>

7. finally python-swiftclient failed with socket error (Broken Pipe)

In operational perspective, setting longer timeout for lingering close would be an option but
it's not complete solution because any other proxy/LB may not support the options.

If we actually do THE RIGHT THING in python-swiftclient, we should send expects: 100-continue
header and handle the first response to re-auth correctly.

HOWEVER, the current python's httplib and requests module used by python-swiftclient doesn't
support expects: 100-continue header [2] and the thread proposed a fix [3] is not super active.
And we know the reason we depends on the library is to fix a security issue that existed
in older python-swiftclient [4] so that we should touch around it super carefully.

In the reality, as the hot fix, this patch try to mitigate the unfortunate situation
described above WITHOUT 100-continue fix, just users can force to re-auth when any errors
occurred during the retries that can be accepted in the upstream.

1: http://nginx.org/en/docs/http/ngx_http_core_module.html#lingering_close
2: https://github.com/requests/requests/issues/713
3: https://bugs.python.org/issue1346874
4: https://review.openstack.org/#/c/69187/

Change-Id: I3470b56e3f9cf9cdb8c2fc2a94b2c551927a3440

524. By Zuul <email address hidden> on 2018-03-06

Merge "Add a query_string option to head_object()."

523. By wangqi on 2018-03-05

Update links in README

Change the outdated links to the latest links in README

Change-Id: Ic0adc686592265f2be2acb55f0520c35d1717f76

522. By Zuul <email address hidden> on 2018-01-31

Merge "Update the old http doc links"

521. By OpenStack Release Bot <email address hidden> on 2018-01-29

Update reno for stable/queens

Change-Id: I7f7b21e3dd0d1ef71c159e2b72f9ad9f963b0773

520. By John Dickinson on 2018-01-23

authors/changelog updates for 3.5.0 release

Change-Id: I70b79c0fd6e9adbfdcc799459dc52063c7402be2

519. By Timur Alperovich <email address hidden> on 2018-01-18

Allow for object uploads > 5GB from stdin.

When uploading from standard input, swiftclient should turn the upload
into an SLO in the case of large objects. This patch picks the
threshold as 10MB (and uses that as the default segment size). The
consumers can also supply the --segment-size option to alter that
threshold and the SLO segment size. The patch does buffer one segment
in memory (which is why 10MB default was chosen).

(test is updated)

Change-Id: Ib13e0b687bc85930c29fe9f151cf96bc53b2e594

Branch metadata

Branch format:
Branch format 7
Repository format:
Bazaar repository format 2a (needs bzr 1.16 or later)
This branch contains Public information 
Everyone can see this information.