download_mirror.sh fails with 404 errors

Bug #1818911 reported by Scott Little
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Fix Released
High
Scott Little

Bug Description

Title
-----
download_mirror.sh fails with 404 errors

Brief Description
-----------------
The 'yum makecache' step is failing. The issue is seen when the fast mirror plugin for yum is in use (default).

The metadata for a yum repo as scattered across several files. The master file is repomd.xml, and it may list several supporting files, e.g.
   031968ec4da5b39b4fc60461f1bf2ff0a060ca0dd4cba833d0a4410402c4f486-filelists.xml.gz
   15b5cd8d1acadd8c7e28092adabf8ddff5411bddfc0fefffffe0f534cefbb4dc-primary.xml.gz
   25d95604a1d7a5bd99ab7ac9466046157c9c72e0aca7bb7c8a4de5427122c71d-filelists.sqlite.bz2
   7bdd84cb362f467203c4c89cca9358b0fe816c17dba6af5597c2f91901b7d445-other.sqlite.bz2
   d911b4643a441a2d157e8492659eb684b432be73a77d0466b7b514b3995ec3b2-other.xml.gz
   e01e41c9dc82015c809159206c20ac95e557a2ae4965eb8b8c9c55b98bdfc2f4-primary.sqlite.bz2

When fast mirror plugin is in use, each file might download from a separate source. This opens the door for a race condition when a repo update is rolling out across the mirrors.

The error suggests we have already downloaded a repomd.xml, but it is obsolete and references a supporting file that no longer exists.

Severity
--------
Major: This is impacting CENGN nightly build on a sporadic basis.

Steps to Reproduce
------------------
run 'download_mirrors.sh' often enough

Expected Behavior
------------------
Successful download of the rpms and tarballs required to build a StarlingX load.

Actual Behavior
----------------
Failures of download_mirrors.sh with 404 errors beinf reported vs a repodata support file.

Reproducibility
---------------
Reproducible/Intermittent
perhaps 10% as seen by CENGN

System Configuration
--------------------
N/A

Branch/Pull Time/Commit
-----------------------
2019-03-06 1:00am EST

Timestamp/Logs
--------------
From jenkins-STX_download_mirror-198.log
...
step #1: start downloading RPMs/SRPMs from 3rd-party websites...
\cp -r '/etc/yum.repos.d' '/tmp/stx_mirror_LbJ4gb/yum.repos.d'
\cp '/etc/yum.conf' '/tmp/stx_mirror_LbJ4gb/yum.conf'
/tmp/stx_mirror_LbJ4gb
using ./rpms_3rdparties.lst as the download name lists
2019-03-06_1858
Loaded plugins: fastestmirror, ovl
http://mirror.netflash.net/centos/7.6.1810/updates/x86_64/repodata/bc76c4dda8b2de78ef78a3e1855c8c2c6c37b185b1de78eda73b5bcd9f10920f-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found
Trying other mirror.
To address this issue please refer to the below knowledge base article

https://access.redhat.com/articles/1320623

If above article doesn't help to resolve this issue please create a bug on https://bugs.centos.org/

http://mirror.netflash.net/centos/7.6.1810/updates/x86_64/repodata/daeed700252971df6449e4b4a47b28fc73607ec0fdd11b4d5bc23ce600895ae5-filelists.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found
Trying other mirror.
http://mirror.csclub.uwaterloo.ca/centos/7.6.1810/updates/x86_64/repodata/bc76c4dda8b2de78ef78a3e1855c8c2c6c37b185b1de78eda73b5bcd9f10920f-primary.sqlite.bz2: [Errno 14] HTTP Error 404 - Not Found
...
Warning: Not all download steps succeeded. You are likely missing files.
Build step 'Execute shell' marked build as failure

Ghada Khalil (gkhalil)
tags: added: stx.build
Revision history for this message
Ghada Khalil (gkhalil) wrote :

Marking as release gating; affects consistent CENGN builds

Changed in starlingx:
assignee: nobody → Scott Little (slittle1)
importance: Undecided → High
tags: added: stx.2019.05
Changed in starlingx:
status: New → In Progress
Revision history for this message
Scott Little (slittle1) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to stx-tools (master)

Reviewed: https://review.openstack.org/641495
Committed: https://git.openstack.org/cgit/openstack/stx-tools/commit/?id=cf435e6c2a8f8b0be479fbe16a1cc3fa1f752d69
Submitter: Zuul
Branch: master

commit cf435e6c2a8f8b0be479fbe16a1cc3fa1f752d69
Author: Scott Little <email address hidden>
Date: Wed Mar 6 16:47:22 2019 -0500

    download_mirror.sh fails with 404 errors

    The 'yum makecache' step is failing. The issue is seen when the fast
    mirror plugin for yum is in use (default).

    The metadata for a yum repo as scattered across several files. The
    master file is repomd.xml, and it may list several supporting files,
    e.g.
       filelists.xml.gz
       primary.xml.gz
       filelists.sqlite.bz2
       other.sqlite.bz2
       other.xml.gz
       primary.sqlite.bz2

    When fast mirror plugin is in use, each file might download from a
    separate source. This opens the door for a race condition when a repo
    update is rolling out across the mirrors.

    The error suggests we have already downloaded a repomd.xml, but it is
    obsolete and references a supporting file that no longer exists.

    A second possible source of inconsistent data is caching proxies.

    Solution:
    1) Disabling the yum fast cache plugin is undesirable, and there
    is no option forcing metadata iof a given repo to be single sourced.
    So this update adds a limited number of retries for the
    'yum makecache' command.

    So far we've never seen 'yum makecache' fail twice in a row...
    largely because the 'fastest mirror' plugin rarely downloaded from the
    same source twice.

    2) Add 'http_caching=packages' to yum.conf, which asks upstream proxies
    to not cache repodata.

    Change-Id: I3fa3f61e38d868fb14f4175b87c0d323abb57def
    Closes-bug: 1818911
    Signed-off-by: Scott Little <email address hidden>

Changed in starlingx:
status: In Progress → Fix Released
Ken Young (kenyis)
tags: added: stx.2.0
removed: stx.2019.05
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.