Merge lp:~paelzer/ubuntu-manpage-repository/ubuntu-manpage-repository-update-and-release into lp:ubuntu-manpage-repository

[Christian Ehrhardt ]

[Christian Ehrhardt  (paelzer)]

Tests are still running, but I'd appreciate an upfront review so I can land the fix if the tests are fine

[lp:~paelzer/ubuntu-manpage-repository/ubuntu-manpage-repository-update-and-release]

237. By Christian Ehrhardt  on 2023-11-30

Use dpkg compare-version

Instead of just hoping they are in order use dpkg compare-version
to identify which content needs to be (re-)rendered.

[Paride Legovini (paride)]

This looks all sensible to me, as usual the comments and commit messages are helpful. However I have no reasonable way to test this, so this review doesn't go much beyond the lgtm level.

review: Approve (lgtm)

[Christian Ehrhardt  (paelzer)]

FYI: Still debugging some more things to do it right this time. Will land some more commits before this is fully complete.

[lp:~paelzer/ubuntu-manpage-repository/ubuntu-manpage-repository-update-and-release]

238. By Christian Ehrhardt  on 2023-12-01

rename src -> binpkg as we handle binary packages

Many archive tools handle source packages, but this parses and
iterates binaries. Make it less misleading by fixing the
variable names and log out put to say binpkg instead of calling
them src.

239. By Christian Ehrhardt  on 2023-12-01

Track versions in the right context

The while loop used to be a subshell behind curl + processing.
To properly track versions that were already handled it has to
move into the same context as the rest of the program.

While touching the decision making on which package to process
ensure that each path creates a log entry and make them all
follow the same pattern which eases log reading.

[Christian Ehrhardt  (paelzer)]

Full test of mantic + noble running over the weekend.

That serves as validation and also as time check to compare with the currently stuck production deployment.

[lp:~paelzer/ubuntu-manpage-repository/ubuntu-manpage-repository-update-and-release]

240. By Christian Ehrhardt  on 2023-12-01

Remove debug output

Further unify output text and remove debug-only content.

[Christian Ehrhardt  (paelzer)]

Ok, the new code ran completely - generated the translated pages (see recent issues in RT 157256) and used the updated content (this issue). Also the former issues of getting $rel-updates paths is fixed.

We might need more to optimize performance a bit, but that is for a future MR.

Also approving to reflect the tests being completed

review: Approve

[Christian Ehrhardt  (paelzer)]

Merged to trunk and production branches.
Asked in RT157256 to please redeploy the new code.

=== modified file 'bin/make-manpage-repo.sh'
--- bin/make-manpage-repo.sh	2023-11-27 09:37:26 +0000
+++ bin/make-manpage-repo.sh	2023-12-01 14:25:15 +0000
@@ -135,26 +135,47 @@
 	return 0
 }
 
-
+declare -A pkg_handled
 for dist in $DISTROS; do
-	# In some cases we do not want to know the pocket, as it would show up in
-	# paths, URLs and bug links, so strip it here if present
-	distnopocket=$(echo "${dist}" | cut -d '-' -f 1)
-	mkdir -p "$PUBLIC_HTML_DIR/manpages/$distnopocket/.cache" "$PUBLIC_HTML_DIR/manpages.gz/$distnopocket" || true
-	link_en_locale "$distnopocket"
-	for repo in $REPOS; do
-		for arch in $ARCH; do
-			file=$(get_packages_url "$dist" "$repo" "$arch")
-			echo "INFO: Packages.gz: $file"
-			curl -s "$file" \
-				| gunzip -c \
-				| grep -E "(^Filename: |^SHA1: )" \
-				| awk '{print $2}' \
-				| sed 'N;s/\n/ /' \
-				| sort -u \
-				| while read -r deb sum; do
-					handle_deb "$distnopocket" "$deb" "$sum"
-				done
+	# On one hand in some cases we do not want to know the pocket, as it
+	# would show up in paths, URLs and bug links
+	distnopocket="${dist}"
+	# Yet on the other hand we need to process all pockets. Without -release
+	# we'd miss content that never got an update, without -updates we'd not
+	# pick up changes.
+	# It orders by likely most up-to-date pocket first and only re-renders if
+	# a newer version of the same source package is found later (even single
+	# Packages files can list the same source multiple times).
+	pkg_handled=()
+	for pocket in "-updates" "-security" ""; do
+		mkdir -p "$PUBLIC_HTML_DIR/manpages/$distnopocket/.cache" "$PUBLIC_HTML_DIR/manpages.gz/$distnopocket" || true
+		link_en_locale "$distnopocket"
+		for repo in $REPOS; do
+			for arch in $ARCH; do
+				file=$(get_packages_url "${dist}${pocket}" "$repo" "$arch")
+				echo "INFO: Packages.gz: $file"
+				plist=$(mktemp "/tmp/XXXXXXX.manpages.${dist}${pocket}.$repo.$arch.plist")
+				curl -s "$file" \
+					| gunzip -c \
+					| grep -E "(^Package: |^Version: |^Filename: |^SHA1: )" \
+					| awk '{print $2}' \
+					| sed 'N;N;N;s/\n/ /g' \
+					| sort -u > "${plist}"
+				while read -r binpkg version deb sum; do
+					if dpkg --compare-versions "${version}" gt "${pkg_handled["$binpkg"]}"; then
+						if [[ -n "${pkg_handled[$binpkg]}" ]]; then
+							echo "INFO: binpkg: $binpkg ${version} > ${pkg_handled["$binpkg"]} (processing it again)"
+						else
+							echo "INFO: First encounter of binpkg: $binpkg ${version} (processing)"
+						fi
+						pkg_handled["$binpkg"]="${version}"
+						handle_deb "$distnopocket" "$deb" "$sum"
+					else
+						echo "INFO: binpkg: $binpkg ${version} < ${pkg_handled["$binpkg"]} (not processing)"
+					fi
+				done < "${plist}"
+				rm -f "${plist}"
+			done
 		done
 	done
 done

=== modified file 'bin/manpages-modify-config'
--- bin/manpages-modify-config	2023-11-27 08:35:42 +0000
+++ bin/manpages-modify-config	2023-12-01 14:25:15 +0000
@@ -23,12 +23,7 @@
         continue
     if s.name in series_blacklist:
         continue
-    if s == current_series:
-        # Has no -updates yet
-        active_series.append(s.name)
-    else:
-        # To pick up changes we need to fetch updates package lists
-        active_series.append("%s-updates" % s.name)
+    active_series.append(s.name)
 
 out_str = ''
 with open(config_fn) as f: