Merge ~techalchemy/ubuntu-qa-tools:feature/in-memory-cookiedb into ubuntu-qa-tools:master

Proposed by Dan Ryan
Status: Needs review
Proposed branch: ~techalchemy/ubuntu-qa-tools:feature/in-memory-cookiedb
Merge into: ubuntu-qa-tools:master
Diff against target: 121 lines (+49/-35)
1 file modified
common/ (+49/-35)
Reviewer Review Type Date Requested Status
Mike Salvatore Pending
Review via email:

Commit message

Use in-memory sqlite database for cookie storage and cookiejar generation

- Also addresses an issue converting launchpadlib.__version__ to and from integers due to byte/unicode encoding changes which prevented importing on python 3

To post a comment you must log in.
Revision history for this message
Steve Beattie (sbeattie) wrote :

Did you actually try to use this? The sql portion of the change fails for me like so:

  $UCT/scripts/sis-changes --ppa ubuntu --pocket Proposed -r bionic --download ${HOME}/tmp/prepare-kernel-usn-6r8qbh5h/usn-bionic-linux-azure-5.3 linux-azure-5.3
  Traceback (most recent call last):
    File "/home/steve/git/ubuntu-cve-tracker/scripts/sis-changes", line 158, in <module>
      opener = lpl_common.opener_with_cookie(cve_lib.config["plb_authentication"])
    File "/home/steve/git/ubuntu-cve-tracker/scripts/", line 148, in opener_with_cookie
      for line in cookie_db.iterdump():
    File "/usr/lib/python3.7/sqlite3/", line 30, in _iterdump
      schema_res = cu.execute(q)
  sqlite3.OperationalError: database is locked

which is why the shutil copy is performed in the original code.

Also, would prefer two distinct fixes as two distinct commits, please.


Revision history for this message
Steve Beattie (sbeattie) wrote :

That said, the changes around the launchpadlib version check looked fine; I pulled that change out and have merged that into master.


Revision history for this message
Steve Beattie (sbeattie) wrote :

The real solution, by the by, is to kill the remaining users of opener_with_cookie(), and convert them to using the proper auth mechanism from python-launchpadlib, either via or directly.

(The opener_with_cookie() function is a hysterical/historical artifact dating back before python-launchpadlib existed and launchpad supported application auth tokens, and the only way you could interact was via web-scraping.)

Revision history for this message
Dan Ryan (techalchemy) wrote :

First of all thanks for taking the time to try this out, I have no idea how the code actually gets invoked so I wasn't able to verify that there wouldn't be any issues with competitive locks on the db in question. I included the encoding change because while attempting to import the function in question, I found I couldn't. I'll be sure to make more granular commits so that they can be cherry picked as needed.

I anticipated that, if necessary, I'd iterate on this change (e.g. by loading the db in memory instead of accessing it directly). This was a change we discussed (and there is a 'todo' in the code about it), but I am happy to either 'close' this merge request if I can figure out how to do that, or to make the changes you are suggesting to the remaining callers of this function.

Unmerged commits

1d490ec... by Dan Ryan

Use in-memory sqlite database for cookie storage and cookiejar generation

Signed-off-by: Dan Ryan <email address hidden>

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/common/ b/common/
2index de9281f..6b57854 100644
3--- a/common/
4+++ b/common/
5@@ -7,7 +7,7 @@
7 from __future__ import print_function
9-import os, sys, tempfile, time, shutil, launchpadlib
10+import os, sys, tempfile, time, shutil, launchpadlib, io
11 from launchpadlib.launchpad import Launchpad
12 from launchpadlib.credentials import Credentials
13 import launchpadlib.errors
14@@ -20,23 +20,26 @@ except ImportError:
15 # as of 16.04, launchpadlib supports python3
16 # so make code support both python2 and python3
17 if sys.version_info > (3, 0):
18+ string_types = (str,)
19 from http.cookiejar import LoadError, MozillaCookieJar
20+ from io import StringIO
21 from urllib.request import HTTPCookieProcessor, HTTPError, build_opener
22 from urllib.parse import unquote
23 else:
24+ string_types = (basestring,)
25 from cookielib import LoadError, MozillaCookieJar
26+ from cStringIO import StringIO
27 from urllib2 import HTTPCookieProcessor, HTTPError, build_opener
28 from urllib import unquote
30 version_min = [1,5,7]
31-if sys.version_info > (3, 0):
32- if [int(x) for x in launchpadlib.__version__.split('.')] < version_min:
33- raise ValueError("Requires launchpadlib version %s or later (%s in use)" \
34- % (".".join(version_min), launchpadlib.__version__))
36- if [int(x) for x in launchpadlib.__version__.decode().split('.')] < version_min:
37- raise ValueError("Requires launchpadlib version %s or later (%s in use)" \
38- % (".".join(version_min), launchpadlib.__version__))
39+lp_version = launchpadlib.__version__
40+if not isinstance(lp_version, string_types):
41+ lp_version = lp_version.decode()
43+if version_min > list(map(int, lp_version.split('.'))):
44+ raise ValueError("Requires launchpadlib version %s or later (%s in use)" \
45+ % (".".join(map(str, version_min)), lp_version))
47 def connect(use_edge=False, beta=False, version=None, uri=None, bot=None):
49@@ -136,36 +139,47 @@ def opener_with_cookie(cookie_file):
50 old_umask = os.umask(0o077)
52 # Work around Firefox 3.5's dumb sqlite locking problems by copying cookies out:
53- tmp = None
54+ opener = None
55+ cj = MozillaCookieJar()
56 if cookie_file.endswith('.sqlite'):
57- (cookie_path, cookie_name) = os.path.split(cookie_file)
58- with tempfile.NamedTemporaryFile(prefix='cookies-XXXXXX', suffix='.sqlite') as sql_handle:
59- sql =
60- shutil.copyfile(cookie_file, sql)
62- match = ''
63- con = sqlite.connect(sql)
64- cur = con.cursor()
65- cur.execute("select host, path, isSecure, expiry, name, value from moz_cookies where host like ?", [match])
66- ftstr = ["FALSE","TRUE"]
67- tmp = tempfile.NamedTemporaryFile(prefix='cookies-XXXXXX', suffix='.mozilla', mode='w+')
68- cookie_file =
69- tmp.write("# HTTP Cookie File\n")
70- for item in cur.fetchall():
71- str = "%s\t%s\t%s\t%s\t%s\t%s\t%s\n" % ( item[0], \
72- ftstr[item[0].startswith('.')], item[1], \
73- ftstr[item[2]], item[3], item[4], item[5])
74- tmp.write(str)
75- sql = None
76- tmp.flush()
77+ sql_script = StringIO()
78+ with sqlite.connect(cookie_file) as cookie_db:
79+ for line in cookie_db.iterdump():
80+ sql_script.write("{0}\n".format(line))
83- cj = MozillaCookieJar()
84- try:
85- cj.load(cookie_file)
86- except LoadError as e:
87- print("Failed to load cookie from file (%s): %s - continuing anyway..." % (cookie_file, e.strerror))
88+ match = ''
89+ query = "select host, path, isSecure, expiry, name, value from moz_cookies where host like ?"
90+ with tempfile.NamedTemporaryFile(prefix='cookies-XXXXXX', suffix='.mozilla', mode='w+') as cookie_jar:
91+ with sqlite.connect(":memory:") as con:
92+ cur = con.cursor()
93+ try:
94+ cur.executescript(
95+ cur.commit()
96+ except sqlite.Error:
97+ print("Failed loading cookied DB...")
98+ raise
100+ try:
101+ cur.execute(query, [match])
102+ except sqlite.Error:
103+ print("Failed reading database...")
104+ raise
105+ ftstr = ["FALSE","TRUE"]
106+ cookie_jar.write("# HTTP Cookie File\n")
107+ for item in cur.fetchall():
108+ str_ = "%s\t%s\t%s\t%s\t%s\t%s\t%s\n" % ( item[0], \
109+ ftstr[item[0].startswith('.')], item[1], \
110+ ftstr[item[2]], item[3], item[4], item[5])
111+ cookie_jar.write(str_)
113+ cookie_jar.flush()
114+ try:
115+ cj.load(
116+ except LoadError as e:
117+ print("Failed to load cookie from file ({0}): {1!s} - continuing anyway...".format(, e))
118 opener = build_opener(HTTPCookieProcessor(cj))
119- tmp = None
120 os.umask(old_umask)
121 return opener


People subscribed via source and target branches