Merge lp:~cjwatson/launchpad/multiarch-translations into lp:launchpad

Proposed by Colin Watson
Status: Merged
Approved by: Jeroen T. Vermeulen
Approved revision: no longer in the source branch.
Merged at revision: 13909
Proposed branch: lp:~cjwatson/launchpad/multiarch-translations
Merge into: lp:launchpad
Diff against target: 282 lines (+55/-2)
10 files modified
lib/lp/archivepublisher/model/ftparchive.py (+9/-1)
lib/lp/archivepublisher/tests/apt-data/apt.conf (+14/-0)
lib/lp/archivepublisher/tests/apt-data/apt_conf_single_empty_suite_test (+1/-0)
lib/lp/archivepublisher/tests/test_ftparchive.py (+7/-0)
lib/lp/registry/configure.zcml (+2/-1)
lib/lp/registry/interfaces/distroseries.py (+15/-0)
lib/lp/registry/model/distroseries.py (+1/-0)
lib/lp/registry/stories/webservice/xx-distroseries.txt (+1/-0)
lib/lp/soyuz/scripts/initialize_distroseries.py (+3/-0)
lib/lp/soyuz/scripts/tests/test_initialize_distroseries.py (+2/-0)
To merge this branch: bzr merge lp:~cjwatson/launchpad/multiarch-translations
Reviewer Review Type Date Requested Status
Henning Eggers Pending
Stuart Bishop db Pending
Robert Collins db Pending
Review via email: mp+74743@code.launchpad.net

This proposal supersedes a proposal from 2011-07-11.

Commit message

[r=jtv][bug=845475] Set LongDescription "true" in apt.conf for series that want it.

Description of the change

(Resubmitted against devel now that multiarch-translations-schema has been rolled out to the database.)

Summary: https://blueprints.launchpad.net/ubuntu/+spec/foundations-o-multiarch-next-steps requires that some clients start downloading Packages files for multiple architectures on 'apt-get update'. As it stands, this will double users' bandwidth costs and time.

Proposed fix: Move the long descriptions from the Packages files (which are in practice common across architectures) out to the Translation-en file. apt-ftparchive already knows how to do this with a configuration option, and apt-get already knows to fetch the Translation-* files, so all we need is a small change to Launchpad to start doing this for Ubuntu Oneiric and beyond.

Pre-implementation notes: I discussed this with William Grant, and agreed that we could follow much the same path as he took recently with the NotAutomatic flag for backports: add a new column to DistroSeries and make the new behaviour depend on that, so that we don't change older series. He also suggested that I export DistroSeries.include_long_descriptions over the API so that we don't need SQL to change it for Oneiric. William has allocated me a db patch number.

Tests: 'testr run -- -t archivepublisher; testr run -- -t soyuz' should cover it. I adjusted lib/lp/archivepublisher/tests/test_ftparchive.py and its data to ensure that the correct apt.conf text is generated, and lib/lp/soyuz/scripts/tests/test_initialize_distroseries.py to ensure that include_long_descriptions is copied across initialising a new distroseries.

QA: On a dogfood Soyuz instance, use the API to set include_long_descriptions to False on a development distroseries of Ubuntu. Perform a publisher run (either making a fake upload, or using the publish-distro.py -A option to force careful apt generation). Make sure that long descriptions (everything after the first line) are written to Translation-en files in the output dists/ tree rather than to Packages files. Set up a client-side apt instance to talk to this, run 'apt-get update', and make sure that 'apt-cache show some-package-name-in-the-archive' still shows a sensible long description.

lint: Nothing introduced by my branch as far as I can see.

To post a comment you must log in.
Stuart Bishop (stub) wrote : Posted in a previous version of this proposal

DB patch looks fine. Please add a short description of the new column to database/schema/comments.sql ('Split long descriptions out of Packages into Translations-en' is fine per the Interface).

review: Approve (db)
Colin Watson (cjwatson) wrote : Posted in a previous version of this proposal

comments.sql change done. Per a conversation with William, I renamed
the column to include_long_descriptions (as I'd got the sense of the
underlying apt.conf variable wrong, and wanted to keep the boolean
values the same way round), and adjusted the description accordingly; I
hope that doesn't invalidate the DB review.

For William: I checked with Michael Vogt, and apparently apt will fetch
Translation-en files in the LANG=C case, so I don't think we need to
worry about Translation-C files.

I also noticed that apt-ftparchive won't create the i18n directories
itself, so I added a conditional safe_mkdir call to
generateConfigForPocket.

I've asked Michael to prepare an apt backport with sufficient
LongDescription support, as lucid-updates doesn't have everything we
need.

Henning Eggers (henninge) wrote : Posted in a previous version of this proposal

Looks good. Thank you very much. ;-)

review: Approve
Colin Watson (cjwatson) wrote : Posted in a previous version of this proposal

Deployment: There are separate apt and python-apt changes working their way through lucid-proposed at the moment, which should land in a day or two. After that we'll need to get a patched apt plus modified meta-lp-deps into the Launchpad PPA, and then it will be possible to deploy this.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'lib/lp/archivepublisher/model/ftparchive.py'
2--- lib/lp/archivepublisher/model/ftparchive.py 2011-08-03 15:06:39 +0000
3+++ lib/lp/archivepublisher/model/ftparchive.py 2011-09-09 15:15:53 +0000
4@@ -96,6 +96,7 @@
5 Packages::Extensions "%(EXTENSIONS)s";
6 BinCacheDB "packages-%(CACHEINSERT)s$(ARCH).db";
7 Contents " ";
8+ LongDescription "%(LONGDESCRIPTION)s";
9 }
10
11 """
12@@ -802,7 +803,11 @@
13 "EXTENSIONS": ".deb",
14 "CACHEINSERT": "",
15 "DISTS": os.path.basename(self._config.distsroot),
16- "HIDEEXTRA": ""})
17+ "HIDEEXTRA": "",
18+ "LONGDESCRIPTION":
19+ "true" if distroseries.include_long_descriptions
20+ else "false",
21+ })
22
23 if archs and suite in self._di_release_components:
24 for component in self._di_release_components[suite]:
25@@ -817,6 +822,7 @@
26 "CACHEINSERT": "debian-installer-",
27 "DISTS": os.path.basename(self._config.distsroot),
28 "HIDEEXTRA": "// ",
29+ "LONGDESCRIPTION": "true",
30 })
31
32 # XXX: 2006-08-24 kiko: Why do we do this directory creation here?
33@@ -831,5 +837,7 @@
34 for base_path in base_paths:
35 if "debian-installer" not in base_path:
36 safe_mkdir(os.path.join(base_path, "source"))
37+ if not distroseries.include_long_descriptions:
38+ safe_mkdir(os.path.join(base_path, "i18n"))
39 for arch in archs:
40 safe_mkdir(os.path.join(base_path, "binary-"+arch))
41
42=== modified file 'lib/lp/archivepublisher/tests/apt-data/apt.conf'
43--- lib/lp/archivepublisher/tests/apt-data/apt.conf 2010-10-15 05:36:31 +0000
44+++ lib/lp/archivepublisher/tests/apt-data/apt.conf 2011-09-09 15:15:53 +0000
45@@ -34,6 +34,7 @@
46 Packages::Extensions ".deb";
47 BinCacheDB "packages-$(ARCH).db";
48 Contents " ";
49+ LongDescription "true";
50 }
51
52
53@@ -49,6 +50,7 @@
54 Packages::Extensions ".udeb";
55 BinCacheDB "packages-debian-installer-$(ARCH).db";
56 Contents " ";
57+ LongDescription "true";
58 }
59
60
61@@ -64,6 +66,7 @@
62 Packages::Extensions ".udeb";
63 BinCacheDB "packages-debian-installer-$(ARCH).db";
64 Contents " ";
65+ LongDescription "true";
66 }
67
68
69@@ -79,6 +82,7 @@
70 Packages::Extensions ".udeb";
71 BinCacheDB "packages-debian-installer-$(ARCH).db";
72 Contents " ";
73+ LongDescription "true";
74 }
75
76
77@@ -94,6 +98,7 @@
78 Packages::Extensions ".udeb";
79 BinCacheDB "packages-debian-installer-$(ARCH).db";
80 Contents " ";
81+ LongDescription "true";
82 }
83
84
85@@ -109,6 +114,7 @@
86 Packages::Extensions ".deb";
87 BinCacheDB "packages-$(ARCH).db";
88 Contents " ";
89+ LongDescription "true";
90 }
91
92
93@@ -124,6 +130,7 @@
94 Packages::Extensions ".deb";
95 BinCacheDB "packages-$(ARCH).db";
96 Contents " ";
97+ LongDescription "true";
98 }
99
100
101@@ -139,6 +146,7 @@
102 Packages::Extensions ".deb";
103 BinCacheDB "packages-$(ARCH).db";
104 Contents " ";
105+ LongDescription "true";
106 }
107
108
109@@ -154,6 +162,7 @@
110 Packages::Extensions ".deb";
111 BinCacheDB "packages-$(ARCH).db";
112 Contents " ";
113+ LongDescription "true";
114 }
115
116
117@@ -169,6 +178,7 @@
118 Packages::Extensions ".deb";
119 BinCacheDB "packages-$(ARCH).db";
120 Contents " ";
121+ LongDescription "false";
122 }
123
124
125@@ -184,6 +194,7 @@
126 Packages::Extensions ".deb";
127 BinCacheDB "packages-$(ARCH).db";
128 Contents " ";
129+ LongDescription "false";
130 }
131
132
133@@ -199,6 +210,7 @@
134 Packages::Extensions ".deb";
135 BinCacheDB "packages-$(ARCH).db";
136 Contents " ";
137+ LongDescription "false";
138 }
139
140
141@@ -214,6 +226,7 @@
142 Packages::Extensions ".deb";
143 BinCacheDB "packages-$(ARCH).db";
144 Contents " ";
145+ LongDescription "false";
146 }
147
148
149@@ -229,5 +242,6 @@
150 Packages::Extensions ".deb";
151 BinCacheDB "packages-$(ARCH).db";
152 Contents " ";
153+ LongDescription "false";
154 }
155
156
157=== modified file 'lib/lp/archivepublisher/tests/apt-data/apt_conf_single_empty_suite_test'
158--- lib/lp/archivepublisher/tests/apt-data/apt_conf_single_empty_suite_test 2007-02-06 13:56:30 +0000
159+++ lib/lp/archivepublisher/tests/apt-data/apt_conf_single_empty_suite_test 2011-09-09 15:15:53 +0000
160@@ -34,5 +34,6 @@
161 Packages::Extensions ".deb";
162 BinCacheDB "packages-$(ARCH).db";
163 Contents " ";
164+ LongDescription "true";
165 }
166
167
168=== modified file 'lib/lp/archivepublisher/tests/test_ftparchive.py'
169--- lib/lp/archivepublisher/tests/test_ftparchive.py 2011-02-25 11:54:05 +0000
170+++ lib/lp/archivepublisher/tests/test_ftparchive.py 2011-09-09 15:15:53 +0000
171@@ -286,6 +286,13 @@
172 self._addRepositoryFile('main', 'foo', 'foo_1.dsc')
173 self._addRepositoryFile('main', 'foo', 'foo_1_i386.deb')
174
175+ # When include_long_descriptions is set, apt.conf has
176+ # LongDescription "true" for that series.
177+ hoary_test = self._distribution.getSeries('hoary-test')
178+ self.assertTrue(hoary_test.include_long_descriptions)
179+ breezy_autotest = self._distribution.getSeries('breezy-autotest')
180+ breezy_autotest.include_long_descriptions = False
181+
182 # XXX cprov 2007-03-21: Relying on byte-to-byte configuration file
183 # comparing is weak. We should improve this methodology to avoid
184 # wasting time on test failures due to irrelevant format changes.
185
186=== modified file 'lib/lp/registry/configure.zcml'
187--- lib/lp/registry/configure.zcml 2011-08-25 13:10:45 +0000
188+++ lib/lp/registry/configure.zcml 2011-09-09 15:15:53 +0000
189@@ -233,7 +233,8 @@
190 <require
191 permission="launchpad.Edit"
192 set_attributes="displayname title summary description driver
193- backports_not_automatic"/>
194+ backports_not_automatic
195+ include_long_descriptions"/>
196
197 <!-- NB: check with SABDFL before modifying these, there is potential to
198 make a huge mess if these get changed and Soyuz has to republish
199
200=== modified file 'lib/lp/registry/interfaces/distroseries.py'
201--- lib/lp/registry/interfaces/distroseries.py 2011-08-25 06:18:33 +0000
202+++ lib/lp/registry/interfaces/distroseries.py 2011-09-09 15:15:53 +0000
203@@ -360,6 +360,21 @@
204 automatically upgrade within backports, but not into it.
205 """))
206
207+ include_long_descriptions = exported(
208+ Bool(
209+ title=_(
210+ "Include long descriptions in Packages rather than in "
211+ "Translation-en"),
212+ default=True,
213+ required=True,
214+ description=_("""
215+ If True, write long descriptions to the per-architecture
216+ Packages files; if False, write them to a Translation-en
217+ file common across architectures instead. Using a common
218+ file reduces the bandwidth footprint of enabling multiarch
219+ on clients, which requires downloading Packages files for
220+ multiple architectures.""")))
221+
222 def priorReleasedSeries():
223 """Prior series *by date* from the same distribution."""
224
225
226=== modified file 'lib/lp/registry/model/distroseries.py'
227--- lib/lp/registry/model/distroseries.py 2011-08-25 06:18:33 +0000
228+++ lib/lp/registry/model/distroseries.py 2011-09-09 15:15:53 +0000
229@@ -256,6 +256,7 @@
230 notNull=False, default=None)
231 language_pack_full_export_requested = BoolCol(notNull=True, default=False)
232 backports_not_automatic = BoolCol(notNull=True, default=False)
233+ include_long_descriptions = BoolCol(notNull=True, default=True)
234
235 language_packs = SQLMultipleJoin(
236 'LanguagePack', joinColumn='distroseries', orderBy='-date_exported')
237
238=== modified file 'lib/lp/registry/stories/webservice/xx-distroseries.txt'
239--- lib/lp/registry/stories/webservice/xx-distroseries.txt 2011-06-02 09:31:49 +0000
240+++ lib/lp/registry/stories/webservice/xx-distroseries.txt 2011-09-09 15:15:53 +0000
241@@ -74,6 +74,7 @@
242 driver_link: None
243 drivers_collection_link: u'http://.../ubuntu/hoary/drivers'
244 fullseriesname: u'Ubuntu Hoary'
245+ include_long_descriptions: True
246 main_archive_link: u'http://.../ubuntu/+archive/primary'
247 name: u'hoary'
248 official_bug_tags: []
249
250=== modified file 'lib/lp/soyuz/scripts/initialize_distroseries.py'
251--- lib/lp/soyuz/scripts/initialize_distroseries.py 2011-09-06 12:10:33 +0000
252+++ lib/lp/soyuz/scripts/initialize_distroseries.py 2011-09-09 15:15:53 +0000
253@@ -331,6 +331,9 @@
254 self.distroseries.backports_not_automatic = any(
255 parent.backports_not_automatic
256 for parent in self.derivation_parents)
257+ self.distroseries.include_long_descriptions = any(
258+ parent.include_long_descriptions
259+ for parent in self.derivation_parents)
260
261 def _copy_architectures(self):
262 das_filter = ' AND distroseries IN %s ' % (
263
264=== modified file 'lib/lp/soyuz/scripts/tests/test_initialize_distroseries.py'
265--- lib/lp/soyuz/scripts/tests/test_initialize_distroseries.py 2011-09-06 12:10:33 +0000
266+++ lib/lp/soyuz/scripts/tests/test_initialize_distroseries.py 2011-09-09 15:15:53 +0000
267@@ -88,6 +88,7 @@
268 if existing_format_selection is None:
269 spfss_utility.add(parent, format_selection)
270 parent.backports_not_automatic = True
271+ parent.include_long_descriptions = False
272 self._populate_parent(parent, parent_das, packages, pocket)
273 return parent, parent_das
274
275@@ -582,6 +583,7 @@
276 SourcePackageFormat.FORMAT_1_0))
277 # Other configuration bits are copied too.
278 self.assertTrue(child.backports_not_automatic)
279+ self.assertFalse(child.include_long_descriptions)
280
281 def test_initialize(self):
282 # Test a full initialize with no errors.