Merge lp:~paelzer/ubuntu-manpage-repository/ubuntu-manpage-repository-speed-boost into lp:ubuntu-manpage-repository

[Christian Ehrhardt ]

[Christian Ehrhardt  (paelzer)]

Trying to help with fixing and debugging the slow execution in prodstack.
On one hand this will run the releases concurrently allowing it to make more use of the CPUs now given to us.

On the other hand adding timing into to the log will allow comparisons of this vs execution on other platforms where it seems faster.

A full run of noble+mantic is running locally atm, I'll provide a log later.

[Christian Ehrhardt  (paelzer)]

FYI: there is a bit of extra churn as I couldn't ignore the trailing whitespaces anymore :-)

[Paride Legovini (paride)]

Another LGTM level review, meaning that I didn't test any of this, however the changes do look good.

review: Approve (lgtm)

[Christian Ehrhardt  (paelzer)]

It built the same content just fine utilization gone up linearly as expected.

Two full releases in 18h
That formerly was more like 24h on my system being blocked by network.

The prod system is less network limited and should gain even more.

And either way, the timestamped logs help to track things better ...

[Christian Ehrhardt  (paelzer)]

Thereby tests are good as well

review: Approve

=== modified file 'bin/fetch-man-pages.sh'
--- bin/fetch-man-pages.sh	2018-06-05 23:34:38 +0000
+++ bin/fetch-man-pages.sh	2023-12-06 08:10:05 +0000
@@ -43,7 +43,7 @@
 NAME_AND_VER=$(echo "$PKG" | sed "s/\.deb$//")
 DEB="$TEMPDIR/$PKG"
 
-echo "INFO: fetching: $PKGURL"
+echo "INFO ($(date '+%H:%M:%S.%N')) - ${DIST}: fetching: $PKGURL"
 curl --silent "$PKGURL" > "$DEB"
 
 DESTDIR="$PUBLIC_HTML_DIR/manpages/$DIST"
@@ -52,13 +52,13 @@
 export W3MMAN_MAN='man --no-hyphenation'
 export MAN_KEEP_FORMATTING=1
 
-echo "INFO: Looking for manpages in [$DEB]"
+echo "INFO ($(date '+%H:%M:%S.%N')) - ${DIST}: Looking for manpages in [$DEB]"
 # The .*man bit is to handle postgres' inane manpage installation
 man=$(dpkg-deb -c "$DEB" | grep -E " \./usr/share.*/man/.*\.[0-9][a-zA-Z0-9\.\-]*\.gz$" | sed -e "s/^.*\.\//\.\//" -e "s/ \-> /\->/")
 
 # Exit immediately if this package does not contain manpages
 if [ -z "$man" ]; then
-	echo "INFO: No manpages: [$DIST] [$PKG]"
+	echo "INFO ($(date '+%H:%M:%S.%N')) - ${DIST}: No manpages: [$DIST] [$PKG]"
 	# Touch the cache file so we don't look again until package updated
 	sha1sum "$DEB" | awk '{ print $1 }' > "$DESTDIR/.cache/$NAME"
 	exit 0
@@ -68,22 +68,22 @@
 
 dpkg-deb -x "$DEB" "$TEMPDIR"
 for i in $man; do
-	#printf "%s\n" "INFO: Considering entry [$i]"
+	#printf "%s\n" "DEBUG: Considering entry [$i]"
 	i=$(printf "%s" "$i" | sed "s/^.*\.\///")
 	if printf "%s" "$i" | grep -qs "\->"; then
 		SYMLINK=1
 		symlink_src_html=$(printf "%s" "$i" | sed -e "s/^.*\->//" -e "s/\.gz$/\.html/")
-		i=$(printf "%s" "$i" | sed "s/\->.*$//") 
-		#printf "%s\n" "INFO: [$i] is a symbolic link"
+		i=$(printf "%s" "$i" | sed "s/\->.*$//")
+		#printf "%s\n" "DEBUG: [$i] is a symbolic link"
 	else
 		SYMLINK=0
 	fi
 	manpage="$TEMPDIR/$i"
 	i=$(printf "%s" "$i" | sed -e "s/usr\/share.*\/man\///i" -e "s/\.gz$//")
-	#printf "%s\n" "INFO: Considering manpage [$i]"
+	#printf "%s\n" "DEBUG: Considering manpage [$i]"
 	# shellcheck disable=SC2166
 	if [ ! -s "$manpage" -o  -z "$i" ] && [ "$SYMLINK" = "0" ]; then
-		#printf "%s\n" "INFO: Skipping empty manpage [$manpage]"
+		#printf "%s\n" "DEBUG: Skipping empty manpage [$manpage]"
 		continue
 	fi
 	out="$DESTDIR"/"$i".html
@@ -91,12 +91,12 @@
 	mkdir -p "$(dirname "$out")" "$outgz" > /dev/null || true
 	if [ "$SYMLINK" = "1" ]; then
 		ln -f -s "$symlink_src_html" "$out"
-		printf "%s\n" "INFO: Created symlink [$out]"
+		printf "%s\n" "INFO ($(date '+%H:%M:%S.%N')) - ${DIST}: Created symlink [$out]"
 	else
 		if LN=$(zcat "$manpage" | head -n1 | grep "^\.so "); then
 			LN=$(printf "%s" "$LN" | sed -e 's/^\.so /\.\.\//' -e 's/\/\.\.\//\//g' -e 's/$/\.html/')
 			ln -f -s "$LN" "$out"
-			printf "INFO: Created symlink [%s]" "$out"
+			printf "INFO ($(date '+%H:%M:%S.%N')) - ${DIST}: Created symlink [%s]" "$out"
                 else
 			BODY=$(COLUMNS=100 /usr/lib/w3m/cgi-bin/w3mman2html.cgi "local=$manpage" | grep -A 1000000 "^<b>" | sed -e '/<\/body>/,+100 d' -e 's:^<b>\(.*\)</b>$:</pre><h4><b>\1</b></h4><pre>:g' -e 's:<a href="file\:///[^?]*?\([^(]*\)(\([^)]*\))">:<a href="../man\2/\1.\2.html">:g')
 			TITLE=$(printf "%s" "$BODY" | head -n2 | tail -n1 | sed "s/<[^>]\+>//g")
@@ -110,7 +110,7 @@
 $BODY
 </pre><!--#include virtual='/below.html' -->" > "$out"
 
-			printf "%s\n" "INFO: Created manpage [$out]"
+			printf "%s\n" "INFO ($(date '+%H:%M:%S.%N')) - ${DIST}: Created manpage [$out]"
 		fi
 	fi
 	mv -f "$manpage" "$outgz"
@@ -121,7 +121,7 @@
 done
 
 # After extracting all manpages, cache the sha1sum, so we don't
-# repeat the downloads 
+# repeat the downloads
 sha1sum "$DEB" | awk '{ print $1 }' > "$DESTDIR/.cache/$NAME"
 
 # In the case of freakish package permissions, fix them on rm failure.

=== modified file 'bin/make-manpage-repo.sh'
--- bin/make-manpage-repo.sh	2023-12-01 14:24:45 +0000
+++ bin/make-manpage-repo.sh	2023-12-06 08:10:05 +0000
@@ -2,23 +2,23 @@
 
 ###############################################################################
 # Copyright (C) 2008 Canonical Ltd.
-# 
+#
 # This code was originally written by Dustin Kirkland <[e-mail]>,
 # based on a framework by Kees Cook <[e-mail]>.
-# 
+#
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
-# 
+#
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
-# 
+#
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
-# 
+#
 # On Debian-based systems, the complete text of the GNU General Public
 # License can be found in /usr/share/common-licenses/GPL-3
 ###############################################################################
@@ -75,27 +75,28 @@
 	fi
 	local deb="$1"
 	local sum="$2"
+	local distnopocket="$3"
 	local name=$(basename "$deb" | awk -F_ '{print $1}')
 	existing_sum=$(cat "$PUBLIC_HTML_DIR/manpages/$dist/.cache/$name" 2>/dev/null)
 
 	# Take the first two digits of the existing_sum modulo 28 to
 	# compare to the current day of month.
-	# 
+	#
 	# Reasoning: this will invalidate the cache for everything ~
 	# once per month (days: 1-28)
 	day_mod=$((0x$(echo "$existing_sum" | cut -b 1-2)%27 + 1))
 	if [ "$day_mod" -eq "$(date +%d)" ]; then
-		echo "INFO: date_mod match, regnerating: $deb ($day_mod)"
+		echo "INFO ($(date '+%H:%M:%S.%N')) - ${distnopocket}: date_mod match, regnerating: $deb ($day_mod)"
 		return 0
 	fi
 
 	# Of course, if the sum found in the packages file for this
 	# package does not equal the sum I have on disk, regenerate.
 	if [ "$existing_sum" = "$sum" ]; then
-	        echo "INFO: cksum skip: $deb"
+	        echo "INFO ($(date '+%H:%M:%S.%N')) - ${distnopocket}: cksum skip: $deb"
 		return 1
 	else
-	        echo "INFO: cksum mismatch: $deb"
+	        echo "INFO ($(date '+%H:%M:%S.%N')) - ${distnopocket}: cksum mismatch: $deb"
 		return 0
 	fi
 }
@@ -107,7 +108,7 @@
 	local deburl=$(get_deb_url "$deb")
 	# FIXME: the || true needs to bubble up to a list of things wrong obviously.
 	# shellcheck disable=SC2015
-	is_pkg_cache_invalid "$deb" "$sum" && "$DIR/fetch-man-pages.sh" "$distnopocket" "$deburl" || true
+	is_pkg_cache_invalid "$deb" "$sum" "$distnopocket" && "$DIR/fetch-man-pages.sh" "$distnopocket" "$deburl" || true
 }
 
 link_en_locale() {
@@ -135,8 +136,8 @@
 	return 0
 }
 
-declare -A pkg_handled
-for dist in $DISTROS; do
+handle_series() {
+	local dist="${1}"
 	# On one hand in some cases we do not want to know the pocket, as it
 	# would show up in paths, URLs and bug links
 	distnopocket="${dist}"
@@ -146,6 +147,7 @@
 	# It orders by likely most up-to-date pocket first and only re-renders if
 	# a newer version of the same source package is found later (even single
 	# Packages files can list the same source multiple times).
+	declare -A pkg_handled
 	pkg_handled=()
 	for pocket in "-updates" "-security" ""; do
 		mkdir -p "$PUBLIC_HTML_DIR/manpages/$distnopocket/.cache" "$PUBLIC_HTML_DIR/manpages.gz/$distnopocket" || true
@@ -153,7 +155,7 @@
 		for repo in $REPOS; do
 			for arch in $ARCH; do
 				file=$(get_packages_url "${dist}${pocket}" "$repo" "$arch")
-				echo "INFO: Packages.gz: $file"
+				echo "INFO ($(date '+%H:%M:%S.%N')) - ${dist}: Packages.gz: $file"
 				plist=$(mktemp "/tmp/XXXXXXX.manpages.${dist}${pocket}.$repo.$arch.plist")
 				curl -s "$file" \
 					| gunzip -c \
@@ -164,20 +166,31 @@
 				while read -r binpkg version deb sum; do
 					if dpkg --compare-versions "${version}" gt "${pkg_handled["$binpkg"]}"; then
 						if [[ -n "${pkg_handled[$binpkg]}" ]]; then
-							echo "INFO: binpkg: $binpkg ${version} > ${pkg_handled["$binpkg"]} (processing it again)"
+							echo "INFO ($(date '+%H:%M:%S.%N')) - ${dist}: binpkg: $binpkg ${version} > ${pkg_handled["$binpkg"]} (processing it again)"
 						else
-							echo "INFO: First encounter of binpkg: $binpkg ${version} (processing)"
+							echo "INFO ($(date '+%H:%M:%S.%N')) - ${dist}: First encounter of binpkg: $binpkg ${version} (processing)"
 						fi
 						pkg_handled["$binpkg"]="${version}"
 						handle_deb "$distnopocket" "$deb" "$sum"
 					else
-						echo "INFO: binpkg: $binpkg ${version} < ${pkg_handled["$binpkg"]} (not processing)"
+						echo "INFO ($(date '+%H:%M:%S.%N')) - ${dist}: binpkg: $binpkg ${version} < ${pkg_handled["$binpkg"]} (not processing)"
 					fi
 				done < "${plist}"
 				rm -f "${plist}"
 			done
 		done
 	done
+
+}
+
+# Simple parallelization on the level of releases; they do not overlap
+# in regard to directories/files, but doing so help to keep the network
+# connection utilized as one can fetch while the other is converting.
+# Furthermore it avoids that issues, or a lot of new content, in one release
+# (e.g. -dev opened) will make the regular update on the others take ages.
+for dist in $DISTROS; do
+	handle_series "${dist}" &
 done
+wait
 
 "$DIR/make-sitemaps.sh"

=== modified file 'bin/w3mman-to-html.pl'
--- bin/w3mman-to-html.pl	2009-02-09 21:37:49 +0000
+++ bin/w3mman-to-html.pl	2023-12-06 08:10:05 +0000
@@ -2,25 +2,25 @@
 
 ##############################################################################
 # This is the Ubuntu manpage repository generator and interface.
-# 
+#
 # Copyright (C) 2008 Canonical Ltd.
-# 
+#
 # This code was originally written by Dustin Kirkland <[e-mail]>,
 # based on a framework by Kees Cook <[e-mail]>.
-# 
+#
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
-# 
+#
 # This program is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
-# 
+#
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
-# 
+#
 # On Debian-based systems, the complete text of the GNU General Public
 # License can be found in /usr/share/common-licenses/GPL-3
 ##############################################################################