Merge lp:ubuntu-manpage-repository into lp:~ya-bo-ng/ubuntu-manpage-repository/add-gtm

[Anthony Dillon]

Add GTM tags to the head and body to enable Google Analytics to track usage

[Christian Ehrhardt  (paelzer)]

Ack to the changes themselve
This is the new version of https://bazaar.launchpad.net/~dpb/ubuntu-manpage-repository/ga-update/revision/220 which everybody believed to have landed in the past (surprise, it did not)

As coordinated via mail, this MR is the wrong way around and not towards the production repo.
I'll integrate it directly after checking some constraints with the release people usually updating it.

[Christian Ehrhardt  (paelzer)]

Merged in https://bazaar.launchpad.net/~ubuntu-manpage-repository-dev/ubuntu-manpage-repository/production/revision/239

Deployment is blocked by https://rt.admin.canonical.com/Ticket/Update.html?id=157256

review: Approve

[lp:ubuntu-manpage-repository]

234. By Christian Ehrhardt  on 2023-11-29

Merge production branch back to trunk

235. By Christian Ehrhardt  on 2023-11-30

config: revert back to just release names

Turns out that we need all pockets anyway and will teach the script
to iterate over them. So the config can be simple release names just
like it always was.

236. By Christian Ehrhardt  on 2023-11-30

Need to consider all pockets

The former change to allo to define release-updates in the config
worked, but only so far to pick up the content of -updates.

Obviously packages that never got an update so far are not listed
at all in the Packages.gz behind -updates and therefore were forgotten.

Fix this by reverting the config back to just release names and on the
other hand extending make-manpage-repo.sh to always look for release,
-updates and -security.

237. By Christian Ehrhardt  on 2023-11-30

Use dpkg compare-version

Instead of just hoping they are in order use dpkg compare-version
to identify which content needs to be (re-)rendered.

238. By Christian Ehrhardt  on 2023-12-01

rename src -> binpkg as we handle binary packages

Many archive tools handle source packages, but this parses and
iterates binaries. Make it less misleading by fixing the
variable names and log out put to say binpkg instead of calling
them src.

239. By Christian Ehrhardt  on 2023-12-01

Track versions in the right context

The while loop used to be a subshell behind curl + processing.
To properly track versions that were already handled it has to
move into the same context as the rest of the program.

While touching the decision making on which package to process
ensure that each path creates a log entry and make them all
follow the same pattern which eases log reading.

240. By Christian Ehrhardt  on 2023-12-01

Remove debug output

Further unify output text and remove debug-only content.

241. By Christian Ehrhardt  on 2023-12-04

Process releases concurrently

Execution is slow and mostly toggling between being CPU bound (conversion)
and network bound (download). Making releases run concurrently allows
to utilize more CPUs at once and the network more continuously.

242. By Christian Ehrhardt  on 2023-12-04

remove trailing whitespace

243. By Christian Ehrhardt  on 2023-12-04

Add missing whitespace in log entry

244. By Christian Ehrhardt  on 2023-12-06

Add timing into to the log entries

Since we are debugging the slow execution on prodstack that might help
to compare and track execution.

Unmerged revisions

244. By Christian Ehrhardt  on 2023-12-06

Add timing into to the log entries

Since we are debugging the slow execution on prodstack that might help
to compare and track execution.

243. By Christian Ehrhardt  on 2023-12-04

Add missing whitespace in log entry

242. By Christian Ehrhardt  on 2023-12-04

remove trailing whitespace

241. By Christian Ehrhardt  on 2023-12-04

Process releases concurrently

Execution is slow and mostly toggling between being CPU bound (conversion)
and network bound (download). Making releases run concurrently allows
to utilize more CPUs at once and the network more continuously.

240. By Christian Ehrhardt  on 2023-12-01

Remove debug output

Further unify output text and remove debug-only content.

239. By Christian Ehrhardt  on 2023-12-01

Track versions in the right context

The while loop used to be a subshell behind curl + processing.
To properly track versions that were already handled it has to
move into the same context as the rest of the program.

While touching the decision making on which package to process
ensure that each path creates a log entry and make them all
follow the same pattern which eases log reading.

238. By Christian Ehrhardt  on 2023-12-01

rename src -> binpkg as we handle binary packages

Many archive tools handle source packages, but this parses and
iterates binaries. Make it less misleading by fixing the
variable names and log out put to say binpkg instead of calling
them src.

237. By Christian Ehrhardt  on 2023-11-30

Use dpkg compare-version

Instead of just hoping they are in order use dpkg compare-version
to identify which content needs to be (re-)rendered.

236. By Christian Ehrhardt  on 2023-11-30

Need to consider all pockets

The former change to allo to define release-updates in the config
worked, but only so far to pick up the content of -updates.

Obviously packages that never got an update so far are not listed
at all in the Packages.gz behind -updates and therefore were forgotten.

Fix this by reverting the config back to just release names and on the
other hand extending make-manpage-repo.sh to always look for release,
-updates and -security.

235. By Christian Ehrhardt  on 2023-11-30

config: revert back to just release names

Turns out that we need all pockets anyway and will teach the script
to iterate over them. So the config can be simple release names just
like it always was.

=== modified file 'bin/fetch-man-pages.sh'
--- bin/fetch-man-pages.sh	2018-06-05 23:34:38 +0000
+++ bin/fetch-man-pages.sh	2023-12-07 08:37:19 +0000
@@ -43,7 +43,7 @@
 NAME_AND_VER=$(echo "$PKG" | sed "s/\.deb$//")
 DEB="$TEMPDIR/$PKG"
 
-echo "INFO: fetching: $PKGURL"
+echo "INFO ($(date '+%H:%M:%S.%N')) - ${DIST}: fetching: $PKGURL"
 curl --silent "$PKGURL" > "$DEB"
 
 DESTDIR="$PUBLIC_HTML_DIR/manpages/$DIST"
@@ -52,13 +52,13 @@
 export W3MMAN_MAN='man --no-hyphenation'
 export MAN_KEEP_FORMATTING=1
 
-echo "INFO: Looking for manpages in [$DEB]"
+echo "INFO ($(date '+%H:%M:%S.%N')) - ${DIST}: Looking for manpages in [$DEB]"
 # The .*man bit is to handle postgres' inane manpage installation
 man=$(dpkg-deb -c "$DEB" | grep -E " \./usr/share.*/man/.*\.[0-9][a-zA-Z0-9\.\-]*\.gz$" | sed -e "s/^.*\.\//\.\//" -e "s/ \-> /\->/")
 
 # Exit immediately if this package does not contain manpages
 if [ -z "$man" ]; then
-	echo "INFO: No manpages: [$DIST] [$PKG]"
+	echo "INFO ($(date '+%H:%M:%S.%N')) - ${DIST}: No manpages: [$DIST] [$PKG]"
 	# Touch the cache file so we don't look again until package updated
 	sha1sum "$DEB" | awk '{ print $1 }' > "$DESTDIR/.cache/$NAME"
 	exit 0
@@ -68,22 +68,22 @@
 
 dpkg-deb -x "$DEB" "$TEMPDIR"
 for i in $man; do
-	#printf "%s\n" "INFO: Considering entry [$i]"
+	#printf "%s\n" "DEBUG: Considering entry [$i]"
 	i=$(printf "%s" "$i" | sed "s/^.*\.\///")
 	if printf "%s" "$i" | grep -qs "\->"; then
 		SYMLINK=1
 		symlink_src_html=$(printf "%s" "$i" | sed -e "s/^.*\->//" -e "s/\.gz$/\.html/")
-		i=$(printf "%s" "$i" | sed "s/\->.*$//") 
-		#printf "%s\n" "INFO: [$i] is a symbolic link"
+		i=$(printf "%s" "$i" | sed "s/\->.*$//")
+		#printf "%s\n" "DEBUG: [$i] is a symbolic link"
 	else
 		SYMLINK=0
 	fi
 	manpage="$TEMPDIR/$i"
 	i=$(printf "%s" "$i" | sed -e "s/usr\/share.*\/man\///i" -e "s/\.gz$//")
-	#printf "%s\n" "INFO: Considering manpage [$i]"
+	#printf "%s\n" "DEBUG: Considering manpage [$i]"
 	# shellcheck disable=SC2166
 	if [ ! -s "$manpage" -o  -z "$i" ] && [ "$SYMLINK" = "0" ]; then
-		#printf "%s\n" "INFO: Skipping empty manpage [$manpage]"
+		#printf "%s\n" "DEBUG: Skipping empty manpage [$manpage]"
 		continue
 	fi
 	out="$DESTDIR"/"$i".html
@@ -91,12 +91,12 @@
 	mkdir -p "$(dirname "$out")" "$outgz" > /dev/null || true
 	if [ "$SYMLINK" = "1" ]; then
 		ln -f -s "$symlink_src_html" "$out"
-		printf "%s\n" "INFO: Created symlink [$out]"
+		printf "%s\n" "INFO ($(date '+%H:%M:%S.%N')) - ${DIST}: Created symlink [$out]"
 	else
 		if LN=$(zcat "$manpage" | head -n1 | grep "^\.so "); then
 			LN=$(printf "%s" "$LN" | sed -e 's/^\.so /\.\.\//' -e 's/\/\.\.\//\//g' -e 's/$/\.html/')
 			ln -f -s "$LN" "$out"
-			printf "INFO: Created symlink [%s]" "$out"
+			printf "INFO ($(date '+%H:%M:%S.%N')) - ${DIST}: Created symlink [%s]" "$out"
                 else
 			BODY=$(COLUMNS=100 /usr/lib/w3m/cgi-bin/w3mman2html.cgi "local=$manpage" | grep -A 1000000 "^<b>" | sed -e '/<\/body>/,+100 d' -e 's:^<b>\(.*\)</b>$:</pre><h4><b>\1</b></h4><pre>:g' -e 's:<a href="file\:///[^?]*?\([^(]*\)(\([^)]*\))">:<a href="../man\2/\1.\2.html">:g')
 			TITLE=$(printf "%s" "$BODY" | head -n2 | tail -n1 | sed "s/<[^>]\+>//g")
@@ -110,7 +110,7 @@
 $BODY
 </pre><!--#include virtual='/below.html' -->" > "$out"
 
-			printf "%s\n" "INFO: Created manpage [$out]"
+			printf "%s\n" "INFO ($(date '+%H:%M:%S.%N')) - ${DIST}: Created manpage [$out]"
 		fi
 	fi
 	mv -f "$manpage" "$outgz"
@@ -121,7 +121,7 @@
 done
 
 # After extracting all manpages, cache the sha1sum, so we don't
-# repeat the downloads 
+# repeat the downloads
 sha1sum "$DEB" | awk '{ print $1 }' > "$DESTDIR/.cache/$NAME"
 
 # In the case of freakish package permissions, fix them on rm failure.

=== modified file 'bin/make-manpage-repo.sh'
--- bin/make-manpage-repo.sh	2018-06-05 23:29:26 +0000
+++ bin/make-manpage-repo.sh	2023-12-07 08:37:19 +0000
@@ -2,23 +2,23 @@
 
 ###############################################################################
 # Copyright (C) 2008 Canonical Ltd.
-# 
+#
 # This code was originally written by Dustin Kirkland <[e-mail]>,
 # based on a framework by Kees Cook <[e-mail]>.
-# 
+#
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
-# 
+#
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
-# 
+#
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
-# 
+#
 # On Debian-based systems, the complete text of the GNU General Public
 # License can be found in /usr/share/common-licenses/GPL-3
 ###############################################################################
@@ -29,7 +29,9 @@
 ARCH=""
 
 # shellcheck disable=SC1090
-. "$DIR/../config"
+config="$DIR/../config"
+echo "INFO: source $config"
+. "$config"
 
 # Establish some locking, to keep multiple updates from running
 mkdir -p "$PUBLIC_HTML_DIR/manpages"
@@ -46,9 +48,9 @@
 FORCE="$1"
 
 get_packages_url() {
-	dist=$1
-	repo=$2
-	arch=$3
+	local dist=$1
+	local repo=$2
+	local arch=$3
 	if [ -e "$DEBDIR/dists/$dist/$repo/binary-$arch/Packages.gz" ]; then
 		echo "file://$DEBDIR/dists/$dist/$repo/binary-$arch/Packages.gz"
 	else
@@ -58,7 +60,7 @@
 
 
 get_deb_url() {
-	deb=$1
+	local deb=$1
 	if [ -e "$DEBDIR/$deb" ]; then
 		echo "file://$DEBDIR/$deb"
 	else
@@ -71,50 +73,51 @@
 	if [ "$FORCE" = "-f" ] || [ "$FORCE" = "--force" ]; then
 		return 0
 	fi
-	deb="$1"
-	sum="$2"
-	name=$(basename "$deb" | awk -F_ '{print $1}')
+	local deb="$1"
+	local sum="$2"
+	local distnopocket="$3"
+	local name=$(basename "$deb" | awk -F_ '{print $1}')
 	existing_sum=$(cat "$PUBLIC_HTML_DIR/manpages/$dist/.cache/$name" 2>/dev/null)
 
 	# Take the first two digits of the existing_sum modulo 28 to
 	# compare to the current day of month.
-	# 
+	#
 	# Reasoning: this will invalidate the cache for everything ~
 	# once per month (days: 1-28)
 	day_mod=$((0x$(echo "$existing_sum" | cut -b 1-2)%27 + 1))
 	if [ "$day_mod" -eq "$(date +%d)" ]; then
-		echo "INFO: date_mod match, regnerating: $deb ($day_mod)"
+		echo "INFO ($(date '+%H:%M:%S.%N')) - ${distnopocket}: date_mod match, regnerating: $deb ($day_mod)"
 		return 0
 	fi
 
 	# Of course, if the sum found in the packages file for this
 	# package does not equal the sum I have on disk, regenerate.
 	if [ "$existing_sum" = "$sum" ]; then
-	        echo "INFO: cksum skip: $deb"
+	        echo "INFO ($(date '+%H:%M:%S.%N')) - ${distnopocket}: cksum skip: $deb"
 		return 1
 	else
-	        echo "INFO: cksum mismatch: $deb"
+	        echo "INFO ($(date '+%H:%M:%S.%N')) - ${distnopocket}: cksum mismatch: $deb"
 		return 0
 	fi
 }
 
 handle_deb() {
-	dist="$1"
-	deb="$2"
-	sum="$3"
-	deburl=$(get_deb_url "$deb")
+	local distnopocket="$1"
+	local deb="$2"
+	local sum="$3"
+	local deburl=$(get_deb_url "$deb")
 	# FIXME: the || true needs to bubble up to a list of things wrong obviously.
 	# shellcheck disable=SC2015
-	is_pkg_cache_invalid "$deb" "$sum" && "$DIR/fetch-man-pages.sh" "$dist" "$deburl" || true
+	is_pkg_cache_invalid "$deb" "$sum" "$distnopocket" && "$DIR/fetch-man-pages.sh" "$distnopocket" "$deburl" || true
 }
 
 link_en_locale() {
-	dist="$1"
-	mkdir -p "$PUBLIC_HTML_DIR/manpages/$dist/en"
+	local distnopocket="$1"
+	mkdir -p "$PUBLIC_HTML_DIR/manpages/$distnopocket/en"
 	for i in $(seq 1 9); do
 		for j in "manpages" "manpages.gz"; do
-			mkdir -p "$PUBLIC_HTML_DIR/$j/$dist/en"
-			dir="$PUBLIC_HTML_DIR/$j/$dist/en/man$i"
+			mkdir -p "$PUBLIC_HTML_DIR/$j/$distnopocket/en"
+			dir="$PUBLIC_HTML_DIR/$j/$distnopocket/en/man$i"
 			if [ -L "$dir" ]; then
 				# link exists: we're good
 				continue
@@ -133,26 +136,61 @@
 	return 0
 }
 
-
+handle_series() {
+	local dist="${1}"
+	# On one hand in some cases we do not want to know the pocket, as it
+	# would show up in paths, URLs and bug links
+	distnopocket="${dist}"
+	# Yet on the other hand we need to process all pockets. Without -release
+	# we'd miss content that never got an update, without -updates we'd not
+	# pick up changes.
+	# It orders by likely most up-to-date pocket first and only re-renders if
+	# a newer version of the same source package is found later (even single
+	# Packages files can list the same source multiple times).
+	declare -A pkg_handled
+	pkg_handled=()
+	for pocket in "-updates" "-security" ""; do
+		mkdir -p "$PUBLIC_HTML_DIR/manpages/$distnopocket/.cache" "$PUBLIC_HTML_DIR/manpages.gz/$distnopocket" || true
+		link_en_locale "$distnopocket"
+		for repo in $REPOS; do
+			for arch in $ARCH; do
+				file=$(get_packages_url "${dist}${pocket}" "$repo" "$arch")
+				echo "INFO ($(date '+%H:%M:%S.%N')) - ${dist}: Packages.gz: $file"
+				plist=$(mktemp "/tmp/XXXXXXX.manpages.${dist}${pocket}.$repo.$arch.plist")
+				curl -s "$file" \
+					| gunzip -c \
+					| grep -E "(^Package: |^Version: |^Filename: |^SHA1: )" \
+					| awk '{print $2}' \
+					| sed 'N;N;N;s/\n/ /g' \
+					| sort -u > "${plist}"
+				while read -r binpkg version deb sum; do
+					if dpkg --compare-versions "${version}" gt "${pkg_handled["$binpkg"]}"; then
+						if [[ -n "${pkg_handled[$binpkg]}" ]]; then
+							echo "INFO ($(date '+%H:%M:%S.%N')) - ${dist}: binpkg: $binpkg ${version} > ${pkg_handled["$binpkg"]} (processing it again)"
+						else
+							echo "INFO ($(date '+%H:%M:%S.%N')) - ${dist}: First encounter of binpkg: $binpkg ${version} (processing)"
+						fi
+						pkg_handled["$binpkg"]="${version}"
+						handle_deb "$distnopocket" "$deb" "$sum"
+					else
+						echo "INFO ($(date '+%H:%M:%S.%N')) - ${dist}: binpkg: $binpkg ${version} < ${pkg_handled["$binpkg"]} (not processing)"
+					fi
+				done < "${plist}"
+				rm -f "${plist}"
+			done
+		done
+	done
+
+}
+
+# Simple parallelization on the level of releases; they do not overlap
+# in regard to directories/files, but doing so help to keep the network
+# connection utilized as one can fetch while the other is converting.
+# Furthermore it avoids that issues, or a lot of new content, in one release
+# (e.g. -dev opened) will make the regular update on the others take ages.
 for dist in $DISTROS; do
-	export dist
-	mkdir -p "$PUBLIC_HTML_DIR/manpages/$dist/.cache" "$PUBLIC_HTML_DIR/manpages.gz/$dist" || true
-	link_en_locale "$dist"
-	for repo in $REPOS; do
-		for arch in $ARCH; do
-			file=$(get_packages_url "$dist" "$repo" "$arch")
-			echo "INFO: Packages.gz: $file"
-			curl -s "$file" \
-				| gunzip -c \
-				| grep -E "(^Filename: |^SHA1: )" \
-				| awk '{print $2}' \
-				| sed 'N;s/\n/ /' \
-				| sort -u \
-				| while read -r deb sum; do
-					handle_deb "$dist" "$deb" "$sum"
-				done
-		done
-	done
+	handle_series "${dist}" &
 done
+wait
 
 "$DIR/make-sitemaps.sh"

=== modified file 'bin/manpages-modify-config'
--- bin/manpages-modify-config	2020-10-21 12:52:23 +0000
+++ bin/manpages-modify-config	2023-12-07 08:37:19 +0000
@@ -13,6 +13,10 @@
 # launchpad = Launchpad.login_anonymously('manpages', launchpadlib.uris.STAGING_SERVICE_ROOT)
 launchpad = Launchpad.login_anonymously('manpages', launchpadlib.uris.LPNET_SERVICE_ROOT)
 
+ubuntu = launchpad.distributions["ubuntu"]
+archive = ubuntu.main_archive
+current_series = ubuntu.current_series
+
 active_series = []
 for s in launchpad.projects['ubuntu'].series:
     if not s.active:

=== modified file 'bin/w3mman-to-html.pl'
--- bin/w3mman-to-html.pl	2009-02-09 21:37:49 +0000
+++ bin/w3mman-to-html.pl	2023-12-07 08:37:19 +0000
@@ -2,25 +2,25 @@
 
 ##############################################################################
 # This is the Ubuntu manpage repository generator and interface.
-# 
+#
 # Copyright (C) 2008 Canonical Ltd.
-# 
+#
 # This code was originally written by Dustin Kirkland <[e-mail]>,
 # based on a framework by Kees Cook <[e-mail]>.
-# 
+#
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
-# 
+#
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
-# 
+#
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
-# 
+#
 # On Debian-based systems, the complete text of the GNU General Public
 # License can be found in /usr/share/common-licenses/GPL-3
 ##############################################################################

=== modified file 'cgi-bin/search.py'
--- cgi-bin/search.py	2021-04-23 15:01:08 +0000
+++ cgi-bin/search.py	2023-12-07 08:37:19 +0000
@@ -114,11 +114,11 @@
     lr = "en"
 
 versions = OrderedDict()
-versions["bionic"] = "18.04 LTS"
 versions["focal"] = "20.04 LTS"
-versions["groovy"] = "20.10"
-versions["hirsute"] = "21.04"
-versions["impish"] = "21.10"
+versions["jammy"] = "22.04 LTS"
+versions["lunar"] = "23.04"
+versions["mantic"] = "23.10"
+versions["noble"] = "24.04 LTS"
 
 distros = versions.keys()
 title_html += ("</div></div><div class='p-strip u-no-padding--top'>"

=== modified file 'examples/config'
--- examples/config	2021-04-23 15:01:08 +0000
+++ examples/config	2023-12-07 08:37:19 +0000
@@ -20,7 +20,8 @@
 SITE="http://manpages.ubuntu.com"
 
 # These form a combinatorial download matrix
-DISTROS="impish hirsute groovy focal bionic"
+# The release names will be automatically modified by bin/manpages-modify-config
+DISTROS="noble mantic lunar jammy focal"
 REPOS="main restricted universe multiverse"
 
 # For now, stick to a single arch

=== modified file 'www/functions.js'
--- www/functions.js	2021-04-23 15:01:08 +0000
+++ www/functions.js	2023-12-07 08:37:19 +0000
@@ -58,11 +58,11 @@
     var navigationContainer = document.getElementById('navigation-container');
     var navigationOutput = "";
     versions = new Array();
-    versions.push({"name":"bionic", "number":"18.04 LTS"});
     versions.push({"name":"focal", "number":"20.04 LTS"});
-    versions.push({"name":"groovy", "number":"20.10"});
-    versions.push({"name":"hirsute", "number":"21.04"});
-    versions.push({"name":"impish", "number":"21.10"});
+    versions.push({"name":"jammy", "number":"22.04 LTS"});
+    versions.push({"name":"lunar", "number":"23.04"});
+    versions.push({"name":"mantic", "number":"23.10"});
+    versions.push({"name":"noble", "number":"24.04 LTS"});
     for (var i = 0; i < versions.length; i++) {
         if (location.href.match("\.html$")) {
             href = location.href;