[Ubuntu 20.04] virt-install fails to detect path after images folder name has changed

Note that the virt-manager package is in the "universe" archive.

in the function _set_url_paths it makes up the defaults:

  # generic
  url_prefix = "current/images"
  ...
  # s390x specific as it can't find "normal" kernel/initrd there
  hvmroot = "%s/generic/" % url_prefix
  kernel_basename = "kernel.%s" % self._debname

That will construct
http://ports.ubuntu.com/ubuntu-ports/dists/bionic/main/installer-s390x/current/images/generic/kernel.ubuntu

Which was the correct old path.
The new path would be the new path.

http://ports.ubuntu.com/ubuntu-ports/dists/focal/main/installer-s390x/current/legacy-images/generic/kernel.ubuntu

But that differs based on the version, if you want to install a bionic guest the old code works.
@Andrew - you might want to ask to keep the old layout instead of changing this (and probably many other tools/scripts)?

Actually the following works pretty well for getting to the old-style installer on s390x.
Instead of a location to the install-dir, just hand it an ISO (can also be a local file).

virt-install --name ubuntu20-guest1 --memory 4096 --vcpus 4 --disk "size=4" --cdrom http://cdimage.ubuntu.com/ubuntu-server/daily/current/focal-server-s390x.iso --network "network=default"

And for the new installer:
virt-install --name ubuntu20-guest1 --memory 4096 --vcpus 4 --disk "size=4" --cdrom http://cdimage.ubuntu.com/ubuntu-server/daily-live/current/focal-live-server-s390x.iso --network "network=default"

P.S. I haven't tried the latter recently, but I know JFH is filing and pushing on related fixes.
Never the less this should solve your issue in case the path as published can't be renamed back.

------- Comment From <email address hidden> 2020-04-15 09:32 EDT-------
The ISO is not equivalent to the install file tree. virt-install can change things like kernel parameters on the fly for a file tree. This is used for kickstart and pre-seed when necessary. virt-install is not able to do the same for iso files.

Agreed @cborntra, if that is your use case then those two ways are not equivalent.
Then for now I only know of the "workaround to specify kernel/initramfs directly" that you are aware of as well.

Let's see what feedback Andrew gets when asking to change the path back.

------- Comment From <email address hidden> 2020-04-15 09:50 EDT-------
Unfortunately this won't work with our preseed files.

ERROR Kernel arguments are only supported with location or kernel installs.

First of all, both examples from comment #3 work (subiquity also installs that way).

I guess it's obvious that the '--location' doesn't work with the directory tree anymore due to the change in the URL from
http://ports.ubuntu.com/ubuntu-ports/dists/focal/main/installer-s390x/current/images/
to:
http://ports.ubuntu.com/ubuntu-ports/dists/focal/main/installer-s390x/current/legacy-images/
(notice that <eoan> http://ports.ubuntu.com/ubuntu-ports/dists/eoan/main/installer-s390x still works).
As far as I can tell the URL change was done by intention and to indicate that the default installer for 20.04 is no longer d-i, but subiquity.

The '--cdrom' is a valid option for many cases, but unfortunately not for all, since it doesn't allow the '-extra-args' argument - '-extra-args' only works in combination with '--location', not with '--cdrom'.

I currently see two options, one is renaming the URL back and the second option is to directly specify kernel and initrd, like:

ubuntu@s1lp11:~$ virt-install --name ubuntu20-guest1 --memory 4096 --vcpus 4 --disk "size=4" --install kernel=http://ports.ubuntu.com/ubuntu-ports/dists/focal/main/installer-s390x/current/legacy-images/generic/kernel.ubuntu,initrd=http://ports.ubuntu.com/ubuntu-ports/dists/focal/main/installer-s390x/current/legacy-images/generic/initrd.ubuntu --network "network=default"

As far as I can tell netboot images/files at 'ports.ubuntu.com' are all d-i only.
For subiquity one needs to grab kernel and initrd for potential netbooting from the focal-live-server-s390x.iso or the iso itself.

Btw. preseed is the non-interactive installation method based on d-i.
With the new subiquity installer the preseed successor is 'autoinstallation' - for more details let me refer to:

Automated Server Installs for 20.04
https://wiki.ubuntu.com/FoundationsTeam/AutomatedServerInstalls

Automated Server Installs Config File Reference
https://wiki.ubuntu.com/FoundationsTeam/AutomatedServerInstalls/ConfigReference

JSON Schema for autoinstall config
https://wiki.ubuntu.com/FoundationsTeam/AutomatedServerInstalls/ConfigSchema

Autoinstall Quick Start
https://wiki.ubuntu.com/FoundationsTeam/AutomatedServerInstalls/QuickStart

Autoinstall Quick Start for s390x
https://wiki.ubuntu.com/FoundationsTeam/AutomatedServerInstalls/QuickStart/s390x

The URL is changed intentionally, to notify all users that these images are deprecated and will not be updated after 20.04.0 GA, and will no longer be published in the future.

You should not be using virt-install, and simply boot the s390x cloud-image in-place, with suitable cloud-config drive attached. This way, you simply directly boot into installed ubuntu server, which executes cloud-init and reconfigures the instance on first boot as desired without performing installation & reboot.

If you want to test the installer, please use the new installer interactively or with auto-install config provided as a cloud-config drive as mentioned on the download page itself http://ports.ubuntu.com/dists/focal/main/installer-s390x/current/legacy-images/

We felt that we had to rename and change URLs, to notify all automated systems that these images are going away.

This communication / notification mechanism seems to be working as intended. Because it broke your current setup.

------- Comment From <email address hidden> 2020-04-15 11:14 EDT-------
If I read the comments correctly, virt-install --location isn't supported anymore for the 20.04 default installer.
It would work however for the d-i installer, if virt-install would employ current/legacy-images as suffix to the URL in the case of distro being focal or higher, potentially with a warning that this method is deprecated.
No good idea to break an existing, well-working solution.
This creates additional churn in our test team, and I'd be surprised, if we're the only ones affected.

Renaming the artefacts was a compromise. the other option was to not publish them at all.

d-i artefacts will not exist for 20.10 release, nor for 20.04.1 release, on any architecture. Thus renamed d-i artefacts is a transitional period of ~3 months until 20.04.1 is released.

cloud-images for s390x with working cloud-init existed since the pre-release of the very first s390x architecture build. And work with xenial, bionic, and focal, and all intermediate releases.

Why is using cloud-images not suitable for your test team which works for all Ubuntu s390x releases, and are pre-built and refreshed for every new kernel SRU? Unlike the d-i based images which are only refreshed every 6 months?

https://cloud-images.ubuntu.com/ are genuinely more up to date, and are quicker to provision than d-i based installs. There are tested/curated release streams with ~2-3 week cadence, and daily streams which are updated ~1-3 days, with all packages up to date from the first boot.

Can we help you move away from virt-instal d-i, to booting cloud images with any configuration applied on first-boot with cloud-init?

Frank pointed out that virt-install uses:

--location http://ports.ubuntu.com/ubuntu-ports/dists/focal/main/installer-s390x

Meaning, it is sensitive to "/images/" changing to "/legacy-images/".

However, instead of changing images to legacy-images, we could instead change "installer-" prefix to "legacy-installer-".

The change of URL is achieved, yet virt-install would then work with like:

--location http://ports.ubuntu.com/ubuntu-ports/dists/focal/main/legacy-installer-s390x

which strikes a better balance of incompatibility & friction.

------- Comment From <email address hidden> 2020-04-16 09:54 EDT-------
(In reply to comment #19)
> Frank pointed out that virt-install uses:
>
> --location
> http://ports.ubuntu.com/ubuntu-ports/dists/focal/main/installer-s390x
>
> Meaning, it is sensitive to "/images/" changing to "/legacy-images/".
>
> However, instead of changing images to legacy-images, we could instead
> change "installer-" prefix to "legacy-installer-".
>
> The change of URL is achieved, yet virt-install would then work with like:
>
> --location
> http://ports.ubuntu.com/ubuntu-ports/dists/focal/main/legacy-installer-s390x
>
> which strikes a better balance of incompatibility & friction.

I like that idea, because it gives users the chance to help themselves, no need to change virt-install.

Unfortunately installer-arch/ is hardcoded portion of the prefix in the internal archive publishing, internal mirroring, and external mirroring, thus I cannot change it to legacy-installer-arch.

Also these will only exist for a short period of time until June, so I don't see the point of investing into redoing all of our archive publishing to support this.

If you use an internal mirror, you can modify it with extra symlinks to symlink legacy-images to images to fix your internal deployment. However, we should work together on moving away your virt-install deployments away from d-i.

------- Comment From <email address hidden> 2020-04-17 03:12 EDT-------
(In reply to comment #21)
> Unfortunately installer-arch/ is hardcoded portion of the prefix in the
> internal archive publishing, internal mirroring, and external mirroring,
> thus I cannot change it to legacy-installer-arch.
>
> Also these will only exist for a short period of time until June, so I don't
> see the point of investing into redoing all of our archive publishing to
> support this.
>
> If you use an internal mirror, you can modify it with extra symlinks to
> symlink legacy-images to images to fix your internal deployment. However, we
> should work together on moving away your virt-install deployments away from
> d-i.

You can install suse,redhat,fedora,debian and old ubuntu guests with virt-install. cockpit uses virt-install under the covers. virt-manager uses virt-install under the covers. For any system that has non-Ubuntu guests as well saying "hey we have something better" is just not helpful.

I think the proper fix is to teach virt-install the new installer. We already have lots of special handling code in virt-install for different versions of different distros.

On Fri, 17 Apr 2020, 08:30 bugproxy, <email address hidden> wrote:

> ------- Comment From <email address hidden> 2020-04-17 03:12 EDT-------
> (In reply to comment #21)
> > Unfortunately installer-arch/ is hardcoded portion of the prefix in the
> > internal archive publishing, internal mirroring, and external mirroring,
> > thus I cannot change it to legacy-installer-arch.
> >
> > Also these will only exist for a short period of time until June, so I
> don't
> > see the point of investing into redoing all of our archive publishing to
> > support this.
> >
> > If you use an internal mirror, you can modify it with extra symlinks to
> > symlink legacy-images to images to fix your internal deployment.
> However, we
> > should work together on moving away your virt-install deployments away
> from
> > d-i.
>
> You can install suse,redhat,fedora,debian and old ubuntu guests with
> virt-install. cockpit uses virt-install under the covers. virt-manager
> uses virt-install under the covers. For any system that has non-Ubuntu
> guests as well saying "hey we have something better" is just not
> helpful.
>
> I think the proper fix is to teach virt-install the new installer. We
> already have lots of special handling code in virt-install for different
> versions of different distros.

Sure, however virt-install currently is not the best way to experience any
release of Ubuntu. That's why MAAS, Canonical Distribution of Openstack,
LXD, Multipass and all public & private clouds that provide Ubuntu consume
and use cloud images, providing a superior experience & configurability
than all other distributions mentioned. The above products cater from
Multipass "give me a quick VM", to manage a "small cloud" / "very bit
cloud".

And virt-install using d-i or iso, instead of qcow2 images (with or without
unpacked kernel) as inputs for any release of Ubuntu, on any architecture
is suboptimal. virt-install should be using cloud images for Ubuntu 12.04
LTS (precise and up). Just like Multipass, lxd-kvm, maas KVM pods,
OpenStack. Many other distributions also provide cloud-init enabled images.
There was GSOC project to get "cloud-init" support but that only landed on
a branch, and hasn't made it into release.

It seems to me that virt-install pre-dates the times of distributions
providing preinstalled cloud-init enabled qcow2 images. Hence so much
automation grew around the suboptimal inputs. Cloud-images is not an
ubuntu-only feature.

Given that the d-i paths changed, and I can't make them compatible enough,
I will reopen the virt-install task to fix path detection in virt-install,
at a wishlist priority. I'll open a separate feature request to support
cloud-images in virt-install & forward it upstream, also at wishlist
priority.

------- Comment From <email address hidden> 2020-04-17 06:55 EDT-------
(In reply to comment #23)
[...]
> Given that the d-i paths changed, and I can't make them compatible enough,
> I will reopen the virt-install task to fix path detection in virt-install,
> at a wishlist priority. I'll open a separate feature request to support
> cloud-images in virt-install & forward it upstream, also at wishlist
> priority.

Yes, both requests make perfect sense. Thank you.

This bug was fixed in the package virt-manager - 1:2.2.1-3ubuntu2

---------------
virt-manager (1:2.2.1-3ubuntu2) focal; urgency=medium

  * Fix legacy-images path for focal. LP: #1872941

 -- Dimitri John Ledkov <email address hidden> Fri, 17 Apr 2020 18:26:25 +0100

------- Comment From <email address hidden> 2020-04-18 02:54 EDT-------
(In reply to comment #25)
> This bug was fixed in the package virt-manager - 1:2.2.1-3ubuntu2
>
> ---------------
> virt-manager (1:2.2.1-3ubuntu2) focal; urgency=medium
>
> * Fix legacy-images path for focal. LP: #1872941
>
> -- Dimitri John Ledkov <email address hidden> Fri, 17 Apr 2020 18:26:25 +0100

Thanks for doing this, seems to work fine.

Any chance to submit this to the upstream project as well?

If the legacy d-i netboot images are not going to be published/supported 20.04.1 onwards, do we have an equivalent replacement that's able to perform the same function?

The mini.iso allows me to boot my 384 Mb ram VPS without a cd-rom drive into the installation environment by just loading the kernel/initrd from grub and complete a minimal setup by downloading only the necessary components from the internet. It also allows me to keep my existing disk layout (LVM volumes hosted in an encrypted luks partition) and install over it.

Are we expecting the new netboot/subiquity setup to be able to do this by the 20.04.1 release? Or are we just going to drop the legacy installer that works well to address these use cases in favor of something that must download and load a 1 Gb ISO image in RAM to work? What about the ability to install over existing LVM volumes within an encrypted partition? Is that planned to be incorporated before d-i is dropped completely?

------- Comment From <email address hidden> 2020-04-20 05:02 EDT-------
I can confirm that virt-manager (1:2.2.1-3ubuntu2) works with http://ports.ubuntu.com/dists/focal/main/installer-s390x again. Also tested it with a preseed.cfg which worked as expected.

But as far as I understood Dimitri this will only be available until June?

------- Comment From <email address hidden> 2020-04-20 06:22 EDT-------
IBM Bugzilla status-> closed, Fix Released with updated package within 20.04

------- Comment From <email address hidden> 2020-04-24 05:09 EDT-------
So now a stupid question:
Obviously the new installer also has kernel, ramdisk parmfile
see
https://ubuntu-on-big-iron.blogspot.com/2020/03/glimpse-at-subiquity.html

why dont you just put THOSE NEW files on the webpage on the installer location?
Then virt-install would continue to work (maybe small fixes) and you would redirect people to the new installer?

I think that's a brilliant question.

I can go on explaining that how we build & publish the new thing, is unlike how we build and publish the old thing, and that it's all a bit non-trivial, but literary that's all irrelevant from the virt-install users' point of view.

We are discussing subiquity plans for the next cycle this week. I'm not sure if we can squeeze this in, or not. But at least I can try.

------- Comment From <email address hidden> 2020-04-27 03:47 EDT-------
Independent from KVM and virt-install, reading again
https://ubuntu-on-big-iron.blogspot.com/2020/03/glimpse-at-subiquity.html
I think this is actually a big step backwards for LPAR.
Do you really have no network configuration script in the installer itself and now you HAVE to specify this via the parmfile?
Until now Ubuntu was the only distribution that could be installed via a public ftpserver via the HMC. If I read the doc right, I now NEED an FTP server myself.
Looks like you just got rid of your big installation advantage over SLES and RHEL.

Hi Christian - there is a separate ticket open for this, it's LP 1854513.
May I suggest to leave a comment there and/or clicking the link "Does this bug affect you?" at the top left? That would be helpful - thank you!

hi, all, while reading this thread I can't really follow which link would work with virt-install.

I've tried all of them but virt-install keep telling there is no distro.

Hi Peter,
for the (s390x) kernel and initrd of d-i (legacy) usage, the URL is still:
http://ports.ubuntu.com/ubuntu-ports/dists/bionic/main/installer-s390x
The problem was that virt-install expected .../current/images as tail and refused to work with .../current/legacy-images, but this is solved now.
(full path/URL:
http://ports.ubuntu.com/ubuntu-ports/dists/focal/main/installer-s390x/current/legacy-images/ )
But you need to have your 'virtinst' package (that contains virt-install) updated to the very latest level (that incl. Dimitri's modification).

The URL is different depending on the architecture:
For example for amd64 the location/URL is:
http://archive.ubuntu.com/ubuntu/dists/bionic/main/installer-amd64
(full path/URL: http://archive.ubuntu.com/ubuntu/dists/bionic/main/installer-amd64/current/images/netboot/ubuntu-installer/amd64/ )

On Mon, 20 Apr 2020 at 10:20, bugproxy <email address hidden> wrote:
>
> ------- Comment From <email address hidden> 2020-04-20 05:02 EDT-------
> I can confirm that virt-manager (1:2.2.1-3ubuntu2) works with http://ports.ubuntu.com/dists/focal/main/installer-s390x again. Also tested it with a preseed.cfg which worked as expected.
>
> But as far as I understood Dimitri this will only be available until
> June?
>

/focal/ aka "release" at
http://ports.ubuntu.com/dists/focal/main/installer-s390x will remain
available forever, just like
http://ports.ubuntu.com/dists/bionic/main/installer-s390x/ , i.e. the
GA release / kernel build, frozen in time.

What is not going to be available are the point releases. I.e.
focal-updates, with newer SRU'ed & hwe kernel.

For bionic, there is bionic-updates at
http://ports.ubuntu.com/dists/bionic-updates/main/installer-s390x/
which has d-i rebuilt against point release & hwe kernels, and is the
preferred location to install from as it boots a newer (and CVE
patched kernel). Such builds will not be created for focal-updates.

--
Regards,

Dimitri.

On Mon, 27 Apr 2020 at 09:05, bugproxy <email address hidden> wrote:
>
> ------- Comment From <email address hidden> 2020-04-27 03:47 EDT-------
> Independent from KVM and virt-install, reading again
> https://ubuntu-on-big-iron.blogspot.com/2020/03/glimpse-at-subiquity.html
> I think this is actually a big step backwards for LPAR.
> Do you really have no network configuration script in the installer itself and now you HAVE to specify this via the parmfile?
> Until now Ubuntu was the only distribution that could be installed via a public ftpserver via the HMC. If I read the doc right, I now NEED an FTP server myself.
> Looks like you just got rid of your big installation advantage over SLES and RHEL.
>

I hope dream and wish, for HMC to offer a selection of .iso, download
them, DD them onto the zfcp drive, and boot the HMC using el-torito
boot record from it, with the correct zfcp zdev-auto-activated. That
way, without doing a thing, the LPAR comes up, initrd automatically
activates the right drive and boots into the installer on the ascii
terminal, with installation happening at blazing fast speeds.
Potentially disconnected from the internet / ubuntu mirrors. Because
.iso is self-contained.

I have asked for HMC to allow storing / launching .iso or cloud-image
.img, but that's not supported at the moment.

Booting over the public internet FTP server, terrifies me. Because
there is no secureboot verification of either kernel or the initrd.
Meaning it's trivial to man-in-the-middle attack, and inject things
into the installer. Can HMC support HTTPS load?

--
Regards,

Dimitri.

Upstream pull request to resolve this for virt-install is not merged https://github.com/virt-manager/virt-manager/pull/101

I guess you want for ftp://cdimage.ubuntu.com/cdimage/releases/20.04/release/ to publish an .ins file that HMC could load, and then be able to bring up network and possibly default to said url to use .iso from there to complete the install.

------- Comment From <email address hidden> 2020-04-28 02:27 EDT-------
> Booting over the public internet FTP server, terrifies me. Because
> there is no secureboot verification of either kernel or the initrd.
> Meaning it's trivial to man-in-the-middle attack, and inject things
> into the installer. Can HMC support HTTPS load?

It does support FTPS (via SSL) and SFTP (via SSH). Not sure how good the certificate verification is.

Can you maybe open an RFE against the HMC to also provide HTTPS?

RFE got created: # 141996

On Tue, 28 Apr 2020 at 07:41, bugproxy <email address hidden> wrote:
>
> ------- Comment From <email address hidden> 2020-04-28 02:27 EDT-------
> > Booting over the public internet FTP server, terrifies me. Because
> > there is no secureboot verification of either kernel or the initrd.
> > Meaning it's trivial to man-in-the-middle attack, and inject things
> > into the installer. Can HMC support HTTPS load?
>
> It does support FTPS (via SSL) and SFTP (via SSH). Not sure how good
> the certificate verification is.
>
> Can you maybe open an RFE against the HMC to also provide HTTPS?
>

Oooh, let me check if we have or can deploy FTPS.

Pure HTTPS is a bit awkward, because there is no directory discovery
like there is over (S)FTP(S). I need to check how UEFI HTTPS boot
works, to see if something "in spirit" similar can be done on HMC.

--
Regards,

Dimitri.

I am working on implementing network config & network .iso discovery

On s390x, when using virt-install, do you really want to configure networking at all or do you want it to simply dhcp, self-find the right iso, download that and attempt booting that?

Or do you still want interactive experience to ask for any chzdev devices to activate, dhcp or static, ip address, gateway (? or just always use default), dns (? or just always use gateway), vlan id, http_proxy, url to iso?

Because it feels like the best experience is no questions asked, just works. Which might be the case with DPM autoconfigured networking devices & qemu autoconfigured devices.

Also virt-install can download & import boot off .iso. Have you tried to point virt-install just at the live-server.iso? On s390x, I expect it to be downloaded, added as a SCSI disk, and el-torito boot to be performed. And if kernel params are needed, virt-install knows how to extract kernels from iso, and it can then do direct kernel boot with kernel params, which iso attached.

------- Comment From <email address hidden> 2020-04-30 02:30 EDT-------
(In reply to comment #43)
> I am working on implementing network config & network .iso discovery
>
> On s390x, when using virt-install, do you really want to configure
> networking at all or do you want it to simply dhcp, self-find the right iso,
> download that and attempt booting that?

I think in a lot of places s390x guest still do have assigned IP addresses, so maybe asking for autoconfig or manual network config is the right thing?
>
> Or do you still want interactive experience to ask for any chzdev devices to
> activate, dhcp or static, ip address, gateway (? or just always use
> default), dns (? or just always use gateway), vlan id, http_proxy, url to
> iso?

Fortunately under KVM all virtio devices are auto activated. And for everything else just do that depending on the first decision.

If you want to have full headleass installation, the user should use kickstart, autoyast, preseed or your new way.

> Because it feels like the best experience is no questions asked, just works.
> Which might be the case with DPM autoconfigured networking devices & qemu
> autoconfigured devices.
>
> Also virt-install can download & import boot off .iso. Have you tried to
> point virt-install just at the live-server.iso? On s390x, I expect it to be
> downloaded, added as a SCSI disk, and el-torito boot to be performed. And if
> kernel params are needed, virt-install knows how to extract kernels from
> iso, and it can then do direct kernel boot with kernel params, which iso
> attached.

Yes that works, but it requires root as the parameter things mounts the ISO.

FYI
Upstream implemented it similar in
  https://github.com/virt-manager/virt-manager/commit/b55b7e94622dd039d00b6e21b5d28d44ab944693
I submitted the one missing bit in
  https://github.com/virt-manager/virt-manager/pull/115

Neither of the above is in a new release yet.

FYI:
It seems that the issue is still present in the currently available focal/universe packages:
virt-manager 1:2.2.1-3ubuntu2
If you run into this issue, like I did, the upstream version like Christian Ehrhardt mentioned above is needed.
See also: https://github.com/virt-manager/virt-manager/issues/171

------- Comment From <email address hidden> 2020-10-20 02:13 EDT-------
@Canonical, should be create a new LP, due to the fact, that this one is Fix Released for furhter tracking ?

I'm not sure what is expected to happen from your side.
The patch is included in virt-manager version 1:2.2.1-3ubuntu2 an newer:
virt-manager (1:2.2.1-3ubuntu2) focal; urgency=medium
  * Fix legacy-images path for focal. LP: #1872941
and we got positive feedback that it works.
And I just retried again:
$ virt-install --name ubu-vm-focal --memory 2048 --vcpus 2 --disk "size=4" --location http://ports.ubuntu.com/ubuntu-ports/dists/focal/main/installer-s390x --network "network=default"
$ virt-install --name ubu-vm-bionic --memory 2048 --vcpus 2 --disk "size=4" --location http://ports.ubuntu.com/ubuntu-ports/dists/bionic/main/installer-s390x --network "network=default"
and both worked for me (the installation started and I didn't got any <python> errors).

@hws also note that we have patched Ubuntu's virt-install & worked with upstream on landing things there too.

If one is using non-Ubuntu virt-install, they need to patch it first, to be usable to deploy focal.

------- Comment From <email address hidden> 2020-10-20 05:11 EDT-------
@Xnox: thx for this addl. information... This might help..