Merge lp:~cjwatson/launchpad/multiarch-translations into lp:launchpad

Proposed by Colin Watson
Status: Merged
Approved by: Jeroen T. Vermeulen
Approved revision: no longer in the source branch.
Merged at revision: 13909
Proposed branch: lp:~cjwatson/launchpad/multiarch-translations
Merge into: lp:launchpad
Diff against target: 282 lines (+55/-2)
10 files modified
lib/lp/archivepublisher/model/ftparchive.py (+9/-1)
lib/lp/archivepublisher/tests/apt-data/apt.conf (+14/-0)
lib/lp/archivepublisher/tests/apt-data/apt_conf_single_empty_suite_test (+1/-0)
lib/lp/archivepublisher/tests/test_ftparchive.py (+7/-0)
lib/lp/registry/configure.zcml (+2/-1)
lib/lp/registry/interfaces/distroseries.py (+15/-0)
lib/lp/registry/model/distroseries.py (+1/-0)
lib/lp/registry/stories/webservice/xx-distroseries.txt (+1/-0)
lib/lp/soyuz/scripts/initialize_distroseries.py (+3/-0)
lib/lp/soyuz/scripts/tests/test_initialize_distroseries.py (+2/-0)
To merge this branch: bzr merge lp:~cjwatson/launchpad/multiarch-translations
Reviewer Review Type Date Requested Status
Henning Eggers Pending
Stuart Bishop db Pending
Robert Collins db Pending
Review via email: mp+74743@code.launchpad.net

This proposal supersedes a proposal from 2011-07-11.

Commit message

[r=jtv][bug=845475] Set LongDescription "true" in apt.conf for series that want it.

Description of the change

(Resubmitted against devel now that multiarch-translations-schema has been rolled out to the database.)

Summary: https://blueprints.launchpad.net/ubuntu/+spec/foundations-o-multiarch-next-steps requires that some clients start downloading Packages files for multiple architectures on 'apt-get update'. As it stands, this will double users' bandwidth costs and time.

Proposed fix: Move the long descriptions from the Packages files (which are in practice common across architectures) out to the Translation-en file. apt-ftparchive already knows how to do this with a configuration option, and apt-get already knows to fetch the Translation-* files, so all we need is a small change to Launchpad to start doing this for Ubuntu Oneiric and beyond.

Pre-implementation notes: I discussed this with William Grant, and agreed that we could follow much the same path as he took recently with the NotAutomatic flag for backports: add a new column to DistroSeries and make the new behaviour depend on that, so that we don't change older series. He also suggested that I export DistroSeries.include_long_descriptions over the API so that we don't need SQL to change it for Oneiric. William has allocated me a db patch number.

Tests: 'testr run -- -t archivepublisher; testr run -- -t soyuz' should cover it. I adjusted lib/lp/archivepublisher/tests/test_ftparchive.py and its data to ensure that the correct apt.conf text is generated, and lib/lp/soyuz/scripts/tests/test_initialize_distroseries.py to ensure that include_long_descriptions is copied across initialising a new distroseries.

QA: On a dogfood Soyuz instance, use the API to set include_long_descriptions to False on a development distroseries of Ubuntu. Perform a publisher run (either making a fake upload, or using the publish-distro.py -A option to force careful apt generation). Make sure that long descriptions (everything after the first line) are written to Translation-en files in the output dists/ tree rather than to Packages files. Set up a client-side apt instance to talk to this, run 'apt-get update', and make sure that 'apt-cache show some-package-name-in-the-archive' still shows a sensible long description.

lint: Nothing introduced by my branch as far as I can see.

To post a comment you must log in.
Revision history for this message
Stuart Bishop (stub) wrote : Posted in a previous version of this proposal

DB patch looks fine. Please add a short description of the new column to database/schema/comments.sql ('Split long descriptions out of Packages into Translations-en' is fine per the Interface).

review: Approve (db)
Revision history for this message
Colin Watson (cjwatson) wrote : Posted in a previous version of this proposal

comments.sql change done. Per a conversation with William, I renamed
the column to include_long_descriptions (as I'd got the sense of the
underlying apt.conf variable wrong, and wanted to keep the boolean
values the same way round), and adjusted the description accordingly; I
hope that doesn't invalidate the DB review.

For William: I checked with Michael Vogt, and apparently apt will fetch
Translation-en files in the LANG=C case, so I don't think we need to
worry about Translation-C files.

I also noticed that apt-ftparchive won't create the i18n directories
itself, so I added a conditional safe_mkdir call to
generateConfigForPocket.

I've asked Michael to prepare an apt backport with sufficient
LongDescription support, as lucid-updates doesn't have everything we
need.

Revision history for this message
Henning Eggers (henninge) wrote : Posted in a previous version of this proposal

Looks good. Thank you very much. ;-)

review: Approve
Revision history for this message
Colin Watson (cjwatson) wrote : Posted in a previous version of this proposal

Deployment: There are separate apt and python-apt changes working their way through lucid-proposed at the moment, which should land in a day or two. After that we'll need to get a patched apt plus modified meta-lp-deps into the Launchpad PPA, and then it will be possible to deploy this.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'lib/lp/archivepublisher/model/ftparchive.py'
--- lib/lp/archivepublisher/model/ftparchive.py 2011-08-03 15:06:39 +0000
+++ lib/lp/archivepublisher/model/ftparchive.py 2011-09-09 15:15:53 +0000
@@ -96,6 +96,7 @@
96 Packages::Extensions "%(EXTENSIONS)s";96 Packages::Extensions "%(EXTENSIONS)s";
97 BinCacheDB "packages-%(CACHEINSERT)s$(ARCH).db";97 BinCacheDB "packages-%(CACHEINSERT)s$(ARCH).db";
98 Contents " ";98 Contents " ";
99 LongDescription "%(LONGDESCRIPTION)s";
99}100}
100101
101"""102"""
@@ -802,7 +803,11 @@
802 "EXTENSIONS": ".deb",803 "EXTENSIONS": ".deb",
803 "CACHEINSERT": "",804 "CACHEINSERT": "",
804 "DISTS": os.path.basename(self._config.distsroot),805 "DISTS": os.path.basename(self._config.distsroot),
805 "HIDEEXTRA": ""})806 "HIDEEXTRA": "",
807 "LONGDESCRIPTION":
808 "true" if distroseries.include_long_descriptions
809 else "false",
810 })
806811
807 if archs and suite in self._di_release_components:812 if archs and suite in self._di_release_components:
808 for component in self._di_release_components[suite]:813 for component in self._di_release_components[suite]:
@@ -817,6 +822,7 @@
817 "CACHEINSERT": "debian-installer-",822 "CACHEINSERT": "debian-installer-",
818 "DISTS": os.path.basename(self._config.distsroot),823 "DISTS": os.path.basename(self._config.distsroot),
819 "HIDEEXTRA": "// ",824 "HIDEEXTRA": "// ",
825 "LONGDESCRIPTION": "true",
820 })826 })
821827
822 # XXX: 2006-08-24 kiko: Why do we do this directory creation here?828 # XXX: 2006-08-24 kiko: Why do we do this directory creation here?
@@ -831,5 +837,7 @@
831 for base_path in base_paths:837 for base_path in base_paths:
832 if "debian-installer" not in base_path:838 if "debian-installer" not in base_path:
833 safe_mkdir(os.path.join(base_path, "source"))839 safe_mkdir(os.path.join(base_path, "source"))
840 if not distroseries.include_long_descriptions:
841 safe_mkdir(os.path.join(base_path, "i18n"))
834 for arch in archs:842 for arch in archs:
835 safe_mkdir(os.path.join(base_path, "binary-"+arch))843 safe_mkdir(os.path.join(base_path, "binary-"+arch))
836844
=== modified file 'lib/lp/archivepublisher/tests/apt-data/apt.conf'
--- lib/lp/archivepublisher/tests/apt-data/apt.conf 2010-10-15 05:36:31 +0000
+++ lib/lp/archivepublisher/tests/apt-data/apt.conf 2011-09-09 15:15:53 +0000
@@ -34,6 +34,7 @@
34 Packages::Extensions ".deb";34 Packages::Extensions ".deb";
35 BinCacheDB "packages-$(ARCH).db";35 BinCacheDB "packages-$(ARCH).db";
36 Contents " ";36 Contents " ";
37 LongDescription "true";
37}38}
3839
3940
@@ -49,6 +50,7 @@
49 Packages::Extensions ".udeb";50 Packages::Extensions ".udeb";
50 BinCacheDB "packages-debian-installer-$(ARCH).db";51 BinCacheDB "packages-debian-installer-$(ARCH).db";
51 Contents " ";52 Contents " ";
53 LongDescription "true";
52}54}
5355
5456
@@ -64,6 +66,7 @@
64 Packages::Extensions ".udeb";66 Packages::Extensions ".udeb";
65 BinCacheDB "packages-debian-installer-$(ARCH).db";67 BinCacheDB "packages-debian-installer-$(ARCH).db";
66 Contents " ";68 Contents " ";
69 LongDescription "true";
67}70}
6871
6972
@@ -79,6 +82,7 @@
79 Packages::Extensions ".udeb";82 Packages::Extensions ".udeb";
80 BinCacheDB "packages-debian-installer-$(ARCH).db";83 BinCacheDB "packages-debian-installer-$(ARCH).db";
81 Contents " ";84 Contents " ";
85 LongDescription "true";
82}86}
8387
8488
@@ -94,6 +98,7 @@
94 Packages::Extensions ".udeb";98 Packages::Extensions ".udeb";
95 BinCacheDB "packages-debian-installer-$(ARCH).db";99 BinCacheDB "packages-debian-installer-$(ARCH).db";
96 Contents " ";100 Contents " ";
101 LongDescription "true";
97}102}
98103
99104
@@ -109,6 +114,7 @@
109 Packages::Extensions ".deb";114 Packages::Extensions ".deb";
110 BinCacheDB "packages-$(ARCH).db";115 BinCacheDB "packages-$(ARCH).db";
111 Contents " ";116 Contents " ";
117 LongDescription "true";
112}118}
113119
114120
@@ -124,6 +130,7 @@
124 Packages::Extensions ".deb";130 Packages::Extensions ".deb";
125 BinCacheDB "packages-$(ARCH).db";131 BinCacheDB "packages-$(ARCH).db";
126 Contents " ";132 Contents " ";
133 LongDescription "true";
127}134}
128135
129136
@@ -139,6 +146,7 @@
139 Packages::Extensions ".deb";146 Packages::Extensions ".deb";
140 BinCacheDB "packages-$(ARCH).db";147 BinCacheDB "packages-$(ARCH).db";
141 Contents " ";148 Contents " ";
149 LongDescription "true";
142}150}
143151
144152
@@ -154,6 +162,7 @@
154 Packages::Extensions ".deb";162 Packages::Extensions ".deb";
155 BinCacheDB "packages-$(ARCH).db";163 BinCacheDB "packages-$(ARCH).db";
156 Contents " ";164 Contents " ";
165 LongDescription "true";
157}166}
158167
159168
@@ -169,6 +178,7 @@
169 Packages::Extensions ".deb";178 Packages::Extensions ".deb";
170 BinCacheDB "packages-$(ARCH).db";179 BinCacheDB "packages-$(ARCH).db";
171 Contents " ";180 Contents " ";
181 LongDescription "false";
172}182}
173183
174184
@@ -184,6 +194,7 @@
184 Packages::Extensions ".deb";194 Packages::Extensions ".deb";
185 BinCacheDB "packages-$(ARCH).db";195 BinCacheDB "packages-$(ARCH).db";
186 Contents " ";196 Contents " ";
197 LongDescription "false";
187}198}
188199
189200
@@ -199,6 +210,7 @@
199 Packages::Extensions ".deb";210 Packages::Extensions ".deb";
200 BinCacheDB "packages-$(ARCH).db";211 BinCacheDB "packages-$(ARCH).db";
201 Contents " ";212 Contents " ";
213 LongDescription "false";
202}214}
203215
204216
@@ -214,6 +226,7 @@
214 Packages::Extensions ".deb";226 Packages::Extensions ".deb";
215 BinCacheDB "packages-$(ARCH).db";227 BinCacheDB "packages-$(ARCH).db";
216 Contents " ";228 Contents " ";
229 LongDescription "false";
217}230}
218231
219232
@@ -229,5 +242,6 @@
229 Packages::Extensions ".deb";242 Packages::Extensions ".deb";
230 BinCacheDB "packages-$(ARCH).db";243 BinCacheDB "packages-$(ARCH).db";
231 Contents " ";244 Contents " ";
245 LongDescription "false";
232}246}
233247
234248
=== modified file 'lib/lp/archivepublisher/tests/apt-data/apt_conf_single_empty_suite_test'
--- lib/lp/archivepublisher/tests/apt-data/apt_conf_single_empty_suite_test 2007-02-06 13:56:30 +0000
+++ lib/lp/archivepublisher/tests/apt-data/apt_conf_single_empty_suite_test 2011-09-09 15:15:53 +0000
@@ -34,5 +34,6 @@
34 Packages::Extensions ".deb";34 Packages::Extensions ".deb";
35 BinCacheDB "packages-$(ARCH).db";35 BinCacheDB "packages-$(ARCH).db";
36 Contents " ";36 Contents " ";
37 LongDescription "true";
37}38}
3839
3940
=== modified file 'lib/lp/archivepublisher/tests/test_ftparchive.py'
--- lib/lp/archivepublisher/tests/test_ftparchive.py 2011-02-25 11:54:05 +0000
+++ lib/lp/archivepublisher/tests/test_ftparchive.py 2011-09-09 15:15:53 +0000
@@ -286,6 +286,13 @@
286 self._addRepositoryFile('main', 'foo', 'foo_1.dsc')286 self._addRepositoryFile('main', 'foo', 'foo_1.dsc')
287 self._addRepositoryFile('main', 'foo', 'foo_1_i386.deb')287 self._addRepositoryFile('main', 'foo', 'foo_1_i386.deb')
288288
289 # When include_long_descriptions is set, apt.conf has
290 # LongDescription "true" for that series.
291 hoary_test = self._distribution.getSeries('hoary-test')
292 self.assertTrue(hoary_test.include_long_descriptions)
293 breezy_autotest = self._distribution.getSeries('breezy-autotest')
294 breezy_autotest.include_long_descriptions = False
295
289 # XXX cprov 2007-03-21: Relying on byte-to-byte configuration file296 # XXX cprov 2007-03-21: Relying on byte-to-byte configuration file
290 # comparing is weak. We should improve this methodology to avoid297 # comparing is weak. We should improve this methodology to avoid
291 # wasting time on test failures due to irrelevant format changes.298 # wasting time on test failures due to irrelevant format changes.
292299
=== modified file 'lib/lp/registry/configure.zcml'
--- lib/lp/registry/configure.zcml 2011-08-25 13:10:45 +0000
+++ lib/lp/registry/configure.zcml 2011-09-09 15:15:53 +0000
@@ -233,7 +233,8 @@
233 <require233 <require
234 permission="launchpad.Edit"234 permission="launchpad.Edit"
235 set_attributes="displayname title summary description driver235 set_attributes="displayname title summary description driver
236 backports_not_automatic"/>236 backports_not_automatic
237 include_long_descriptions"/>
237238
238 <!-- NB: check with SABDFL before modifying these, there is potential to239 <!-- NB: check with SABDFL before modifying these, there is potential to
239 make a huge mess if these get changed and Soyuz has to republish240 make a huge mess if these get changed and Soyuz has to republish
240241
=== modified file 'lib/lp/registry/interfaces/distroseries.py'
--- lib/lp/registry/interfaces/distroseries.py 2011-08-25 06:18:33 +0000
+++ lib/lp/registry/interfaces/distroseries.py 2011-09-09 15:15:53 +0000
@@ -360,6 +360,21 @@
360 automatically upgrade within backports, but not into it.360 automatically upgrade within backports, but not into it.
361 """))361 """))
362362
363 include_long_descriptions = exported(
364 Bool(
365 title=_(
366 "Include long descriptions in Packages rather than in "
367 "Translation-en"),
368 default=True,
369 required=True,
370 description=_("""
371 If True, write long descriptions to the per-architecture
372 Packages files; if False, write them to a Translation-en
373 file common across architectures instead. Using a common
374 file reduces the bandwidth footprint of enabling multiarch
375 on clients, which requires downloading Packages files for
376 multiple architectures.""")))
377
363 def priorReleasedSeries():378 def priorReleasedSeries():
364 """Prior series *by date* from the same distribution."""379 """Prior series *by date* from the same distribution."""
365380
366381
=== modified file 'lib/lp/registry/model/distroseries.py'
--- lib/lp/registry/model/distroseries.py 2011-08-25 06:18:33 +0000
+++ lib/lp/registry/model/distroseries.py 2011-09-09 15:15:53 +0000
@@ -256,6 +256,7 @@
256 notNull=False, default=None)256 notNull=False, default=None)
257 language_pack_full_export_requested = BoolCol(notNull=True, default=False)257 language_pack_full_export_requested = BoolCol(notNull=True, default=False)
258 backports_not_automatic = BoolCol(notNull=True, default=False)258 backports_not_automatic = BoolCol(notNull=True, default=False)
259 include_long_descriptions = BoolCol(notNull=True, default=True)
259260
260 language_packs = SQLMultipleJoin(261 language_packs = SQLMultipleJoin(
261 'LanguagePack', joinColumn='distroseries', orderBy='-date_exported')262 'LanguagePack', joinColumn='distroseries', orderBy='-date_exported')
262263
=== modified file 'lib/lp/registry/stories/webservice/xx-distroseries.txt'
--- lib/lp/registry/stories/webservice/xx-distroseries.txt 2011-06-02 09:31:49 +0000
+++ lib/lp/registry/stories/webservice/xx-distroseries.txt 2011-09-09 15:15:53 +0000
@@ -74,6 +74,7 @@
74 driver_link: None74 driver_link: None
75 drivers_collection_link: u'http://.../ubuntu/hoary/drivers'75 drivers_collection_link: u'http://.../ubuntu/hoary/drivers'
76 fullseriesname: u'Ubuntu Hoary'76 fullseriesname: u'Ubuntu Hoary'
77 include_long_descriptions: True
77 main_archive_link: u'http://.../ubuntu/+archive/primary'78 main_archive_link: u'http://.../ubuntu/+archive/primary'
78 name: u'hoary'79 name: u'hoary'
79 official_bug_tags: []80 official_bug_tags: []
8081
=== modified file 'lib/lp/soyuz/scripts/initialize_distroseries.py'
--- lib/lp/soyuz/scripts/initialize_distroseries.py 2011-09-06 12:10:33 +0000
+++ lib/lp/soyuz/scripts/initialize_distroseries.py 2011-09-09 15:15:53 +0000
@@ -331,6 +331,9 @@
331 self.distroseries.backports_not_automatic = any(331 self.distroseries.backports_not_automatic = any(
332 parent.backports_not_automatic332 parent.backports_not_automatic
333 for parent in self.derivation_parents)333 for parent in self.derivation_parents)
334 self.distroseries.include_long_descriptions = any(
335 parent.include_long_descriptions
336 for parent in self.derivation_parents)
334337
335 def _copy_architectures(self):338 def _copy_architectures(self):
336 das_filter = ' AND distroseries IN %s ' % (339 das_filter = ' AND distroseries IN %s ' % (
337340
=== modified file 'lib/lp/soyuz/scripts/tests/test_initialize_distroseries.py'
--- lib/lp/soyuz/scripts/tests/test_initialize_distroseries.py 2011-09-06 12:10:33 +0000
+++ lib/lp/soyuz/scripts/tests/test_initialize_distroseries.py 2011-09-09 15:15:53 +0000
@@ -88,6 +88,7 @@
88 if existing_format_selection is None:88 if existing_format_selection is None:
89 spfss_utility.add(parent, format_selection)89 spfss_utility.add(parent, format_selection)
90 parent.backports_not_automatic = True90 parent.backports_not_automatic = True
91 parent.include_long_descriptions = False
91 self._populate_parent(parent, parent_das, packages, pocket)92 self._populate_parent(parent, parent_das, packages, pocket)
92 return parent, parent_das93 return parent, parent_das
9394
@@ -582,6 +583,7 @@
582 SourcePackageFormat.FORMAT_1_0))583 SourcePackageFormat.FORMAT_1_0))
583 # Other configuration bits are copied too.584 # Other configuration bits are copied too.
584 self.assertTrue(child.backports_not_automatic)585 self.assertTrue(child.backports_not_automatic)
586 self.assertFalse(child.include_long_descriptions)
585587
586 def test_initialize(self):588 def test_initialize(self):
587 # Test a full initialize with no errors.589 # Test a full initialize with no errors.