Merge ~alextu/pc-enablement/+git/oem-scripts:rsync_iso into ~oem-solutions-engineers/pc-enablement/+git/oem-scripts:master

[Alex Tu]

[Alex Tu (alextu)]

a test on random usb disk:
https://pastebin.canonical.com/p/5Y4j8CWjzz/

040f634... by Alex Tu on 2021-09-20

rsync_iso.sh: new parameger, --allusb to flash all detected usb.

[Alex Tu (alextu)]

a test result over VPN.
$ rsync_iso.sh --jenkins-url jenkins.canonical.pw --jenkins-job dell-bto-focal-fossa-edge-alloem-init -s
https://pastebin.canonical.com/p/8h8Zbzy3Pc/

It will popup a dialog reminding you in the end.
https://i.imgur.com/v20GtNB.png

[Andy Chi (andch)]

Test pass on local machine.

$ ./rsync_iso.sh ~/Downloads/dell-bto-focal-fossa-pypar-adl-X163-20211027-6.iso -f /media/andch/Dell\ Recovery\ Media/ -s

https://pastebin.canonical.com/p/88vfNxRzHM/

Can we remove set -x?

By the way, tmp.xxxx folder won't be deleted after installation usb is finished.

$ ls
tmp.Oe7dubjaNd

[Alex Tu (alextu)]

sure thing, let me remove the set -x and fix the un-deleted `tmp.xxxx`

7249df2... by Alex Tu on 2021-10-28

removed -x and solidly remove tmp folders.

[Andy Chi (andch)]

Test done.

sent 4,145,771,921 bytes received 42,432 bytes 236,903,677.31 bytes/sec
total size is 4,144,596,677 speedup is 1.00
================================================================
    dell-bto-focal-fossa-tentacool-X169-20211014-12.iso
================================================================
~/oem-scripts/tmp.AAZF4XBUX7 ~/oem-scripts
~/oem-scripts
/usr/bin/zenity
umount: /home/andch/oem-scripts/tmp.ICAjrBvoSh: no mount point specified.
umount: /home/andch/oem-scripts/tmp.AAZF4XBUX7: no mount point specified.

LGTM

review: Approve

56821cb... by Alex Tu on 2021-11-01

fix the warning during umount

[OEM Taipei Bot (oem-taipei-bot)]

[autopkgtest]
autopkgtest-collect-credentials PASS
autopkgtest-oem-scripts-auto PASS
pkg-somerville-meta PASS
pkg-stella-meta PASS
pkg-sutton-meta PASS
bug-bind PASS
get-private-ppa PASS
jq-lp PASS
launchpad-api PASS
lp-bug PASS
oem-meta-packages PASS
pkg-list PASS
review-merge-proposal PASS
run-autopkgtest PASS
setup-apt-dir PASS
mir-bug FAIL non-zero exit status 1
git-url-insteadof-setting FAIL non-zero exit status 128
recovery-from-iso.sh PASS
mir-bug-verification FAIL stderr: E: Unable to locate package oem-somerville-maester-meta

https://oem-share.canonical.com/partners/lyoncore/share/artifacts/oem-scripts-0.99-56821cb-in-linux-container-focal

[Alex Tu (alextu)]

testing for commit 215f181:

# Environment:
2 USB disks be plugged
sda 8:0 1 58.4G 0 disk
└─sda1 8:1 1 58.4G 0 part /media/alextu/C695-6B4E
sdb 8:16 1 14.5G 0 disk
└─sdb1 8:17 1 14.5G 0 part /media/alextu/CEC2-88FB

# verified command
$ ./rsync_iso.sh --jenkins-url jenkins.canonical.pw --jenkins-job dell-bto-focal-fossa-edge-alloem-init --jenkins-job-number 18 --all-usb -s 2>&1 | tee log

https://pastebin.canonical.com/p/kRc2P2x3Rg/

# result, 2 USB disks be made successfully.

[Yao Wei (medicalwei)]

Verified manually that by using the following command:
./rsync_iso.sh --jenkins-url jenkins.canonical.pw --jenkins-job dell-bto-focal-fossa-edge-alloem-init

The external disk I plugged in has been formatted into FAT32 and files from dell-bto-focal-fossa-edge-alloem-init-X186-20220111-27.iso has been copied over.

Other than having too much log messages (suggested by andch to remove set -x), it LGTM.

review: Approve

[Alex Tu (alextu)]

+1, to remove the set -x.

2b0b120... by Alex Tu on 2022-01-21

recovery-from-iso.sh: remove set -x

[OEM Taipei Bot (oem-taipei-bot)]

[autopkgtest]
autopkgtest-collect-credentials PASS
autopkgtest-oem-scripts-auto PASS
pkg-somerville-meta PASS
pkg-stella-meta PASS
pkg-sutton-meta PASS
bug-bind PASS
get-private-ppa PASS
jq-lp PASS
launchpad-api PASS
lp-bug PASS
oem-meta-packages PASS
pkg-list PASS
review-merge-proposal PASS
run-autopkgtest PASS
setup-apt-dir PASS
mir-bug FAIL non-zero exit status 1
git-url-insteadof-setting FAIL non-zero exit status 128
recovery-from-iso.sh PASS
mir-bug-verification PASS

https://oem-share.canonical.com/partners/lyoncore/share/artifacts/oem-scripts/oem-scripts-0.99-2b0b120-in-linux-container-focal

Unmerged commits

2b0b120... by Alex Tu on 2022-01-21

recovery-from-iso.sh: remove set -x

040f634... by Alex Tu on 2021-09-20

rsync_iso.sh: new parameger, --allusb to flash all detected usb.

56821cb... by Alex Tu on 2021-11-01

fix the warning during umount

7249df2... by Alex Tu on 2021-10-28

removed -x and solidly remove tmp folders.

a931e27... by Alex Tu on 2021-08-09

take care the case of multiple partitions in target usb disk

045cd32... by Alex Tu on 2021-07-04

make sure the url of Jenkins is accessable

36a2a31... by Alex Tu on 2021-06-23

a handy script to create bootable usb with OEM image

diff --git a/recovery-from-iso.sh b/recovery-from-iso.sh
index 843d9cc..cb20f90 100755
--- a/recovery-from-iso.sh
+++ b/recovery-from-iso.sh
@@ -1,5 +1,5 @@
 #!/bin/bash
-set -ex
+set -e
 
 jenkins_job_for_iso=""
 jenkins_job_build_no="lastSuccessfulBuild"
diff --git a/rsync_iso.sh b/rsync_iso.sh
new file mode 100755
index 0000000..6ef5f2d
--- /dev/null
+++ b/rsync_iso.sh
@@ -0,0 +1,180 @@
+#!/bin/bash
+set -ex
+CD_PATH="$PWD"
+JENKINS_B_NO="lastSuccessfulBuild"
+
+mnt_folder="$(mktemp -d -p "$PWD")"
+target_folder="$(mktemp -d -p "$PWD")"
+# $1: the target to unumount
+do_unmount() {
+    if [ -n "$1" ] && grep -q "$1" /proc/mounts; then
+        sudo umount "$1"
+    fi
+}
+clear_all() {
+    do_unmount "$mnt_folder"
+    do_unmount "$target_folder"
+    rm -rf "$mnt_folder"
+    rm -rf "$target_folder"
+}
+trap clear_all EXIT
+
+usage() {
+cat << EOF
+usage:
+$0 options --iso [ISO file] | [file.iso] [<options to get iso from jenkins artifacts>]
+$0 options [<options to get iso from jenkins artifacts>] [-s][-f]
+
+** IMPORTANT **
+1. This script only works when only one USB disk plugged and it can be found in file manager,
+2. USB disk will be wipe.
+**
+
+This command will create a bootable usb disk by the ISO file,
+It assume the iso file in the same folder where script be executed.
+options:
+    -h|--help   Print this message
+    -f|--folder The folder you would like to copy to root of target usb disk
+    -i|--iso    The iso file which be used to create a bootable USB disk.
+    -a|--allusb Flash all detected usb disk.
+    -s          No interactive
+
+options to get iso from jenkins artifacts:
+    --jenkins-url           The url of target Jenkins
+    --jenkins-job           The job of target Jenkins
+    --jenkins-job-number    The buile number of target Jenkins job
+
+example:
+    # Download iso from specific Jenkins job and build number then rsync the iso to usb disk without user interactive.
+    $ rsync_iso.sh --jenkins-url jenkins.canonical.pw --jenkins-job pc-stella-cmit-focal-amd64 --jenkins-job-number 1569 -s
+
+    # rsync local iso and dump everything in ~/path-to-the-folder to usb disk with user interactive.
+    $ rsync_iso.sh any-oem-image.iso -f ~/path-to-the-folder
+
+EOF
+exit 1
+}
+
+
+while [ $# -gt 0 ]
+do
+    case "$1" in
+        -h | --help)
+            usage 0
+            exit 0
+            ;;
+        -f |--folder)
+            shift
+            FOLDER=$1;
+            [ -z "${FOLDER##/*}" ] || [ -z "${FOLDER##~/*}" ] || FOLDER="$CD_PATH/$FOLDER"
+            if [ ! -d "$FOLDER" ];then
+                echo "not exists $FOLDER"
+                exit 1
+            fi
+            ;;
+        -i | --iso)
+            shift
+            ISO=$1
+            if [ ! -e "$ISO" ]; then
+                echo "not exists $FOLDER"
+                exit 1
+            fi
+            ;;
+        -s)
+            SILENCE="TRUE"
+            ;;
+        --jenkins-url)
+            shift
+            JENKINS_URL="$1"
+            if ! ping -c 1 -W 1 "$JENKINS_URL"; then
+                echo "[ERROR] Please make sure you can connect to Jenkins: $JENKINS_URL"
+                exit 1
+            fi
+            ;;
+        --jenkins-job)
+            shift
+            JENKINS_JOB="$1"
+            ;;
+        --jenkins-job-number)
+            shift
+            JENKINS_B_NO="$1"
+            ;;
+        -a | --all-usb)
+            FLASH_ALL_USB="true"
+            ;;
+        *)
+            if [ -z "${1##*.iso}" ]; then
+                ISO=$1
+            else
+                usage
+            fi
+       esac
+       shift
+done
+
+if [ -n "$JENKINS_URL" ] && [ -n "$JENKINS_JOB" ]; then
+    img_jenkins_out_url="ftp://$JENKINS_URL/jenkins_host/jobs/$JENKINS_JOB/builds/$JENKINS_B_NO/archive/out"
+    img_name="$(wget -q "$img_jenkins_out_url/" -O - | grep -o 'href=.*iso"' | awk -F/ '{print $NF}' | tr -d \")"
+    rm -f "$img_jenkins_out_url/$img_name".md5sum
+    wget "$img_jenkins_out_url/$img_name".md5sum
+    md5sum -c "$img_name".md5sum || wget "$img_jenkins_out_url"/"$img_name"
+    md5sum -c "$img_name".md5sum || usage
+    ISO="$img_name"
+fi
+
+[ -z "$ISO" ] && usage
+usb_mnt_list=$(mount | grep udisk | awk -F'type|on' '{print $1}' | sort)
+usb_list=$(echo "$usb_mnt_list"| sed 's/[1-9]//' | uniq)
+if [ -z "$usb_list" ]; then
+    echo "can not find any USB be insertted!"
+    usage
+fi
+
+if [ "$(echo "$usb_list" | wc -l)" -gt 1 ] && [ "$FLASH_ALL_USB" != "true" ];then
+    echo "There are more than one usb disk, please make sure only one usb disk connected to your host."
+    echo "$usb_list"
+    exit
+else
+    usb_targets=$(echo "$usb_list" | tr -d "[:blank:]")
+fi
+
+sudo mount -o loop "$ISO" "$mnt_folder"
+
+while read -r disk; do
+    do_unmount "$disk"
+done <<<"$usb_mnt_list"
+
+for usb_target in $usb_targets; do
+    echo "press any key to wipe $usb_target and create bootable usb..... "
+    [ "$SILENCE" == "TRUE" ] || read -r
+    sudo wipefs -a "$usb_target" 2>&1 || (echo "Wipefs failed"; usage)
+    sudo parted -s "$usb_target" mktable msdos || (echo "parted mktable failed."; usage)
+    sudo parted -s "$usb_target" mkpart primary  0% 100% || (echo "parted mkpart failed."; usage)
+    usb_target="${usb_target}1"
+    sleep 2
+    sudo mkfs.vfat -F 32 "$usb_target"
+
+    target_folder="$(mktemp -d -p "$PWD")"
+    sudo mount -t vfat -o gid=$UID,uid=$UID "$usb_target" "$target_folder"
+
+    echo "pless any keye to do ionice -c 3 rsync --delete -avP $mnt_folder/ $target_folder"
+    [ "$SILENCE" == "TRUE" ] || read -r
+    ionice -c 3 rsync --delete -avP "$mnt_folder"/ "$target_folder"
+    iso_name=$(grep "<iso" "$target_folder/bto.xml" | sed 's/<[^>]*>//g')
+    echo ================================================================
+    echo "$iso_name"
+    echo ================================================================
+    pushd "$target_folder" || exit 1
+        if [ -n "$FOLDER" ] && [ -d "$FOLDER" ]; then
+            echo "pless any keye to do ionice -c 3 rsync -avP $FOLDER/ ."
+            [ "$SILENCE" == "TRUE" ] || read -r
+            ionice -c 3 rsync -avP "$FOLDER"/ .
+        fi
+    popd || exit 1
+
+    do_unmount "$target_folder"
+    rm -rf "$target_folder"
+    udisksctl mount -b "$usb_target"
+done
+clear_all
+command -v zenity && zenity --timeout 600 --info --text="$0 $ISO done" &