Containers in CI updates all packages which takes time and leads to timeouts

Bug #1770355 reported by Sagi (Sergey) Shnaidman
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Sagi (Sergey) Shnaidman

Bug Description

http://logs.openstack.org/75/567275/1/gate/tripleo-ci-centos-7-scenario007-multinode-oooq-container/a0d5161/logs/undercloud/home/zuul/overcloud_prep_containers.log.txt.gz

Instead of updating only built package, containers are updated with all base os packages which takes about 1 hour and all jobs fails with timeout.
Possibly leftover from bug https://bugs.launchpad.net/tripleo/+bug/1770298

Before this containers updated only built packages and dependencies from base-os only, not the whole base-os update:
http://logs.openstack.org/49/564049/2/check/tripleo-ci-centos-7-containers-multinode/4888fb1/logs/undercloud/home/zuul/overcloud_prep_containers.log.txt.gz

tags: added: promotion-blocker
Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

So we need the tripleo containers images update pushed

Changed in tripleo:
importance: Undecided → Critical
tags: added: containers
description: updated
description: updated
Revision history for this message
Javier Peña (jpena-c) wrote :

CentOS 7.5 was released on the evening of May 9th, so there is a lot of packages to update. We need updated container images (not sure if just updating the base image will be enough).

Revision history for this message
Alfredo Moralejo (amoralej) wrote :

IMO, the actual problem is in how overcloud-prep-containers.sh works to update containers with packages built on the fly for gate jobs. As my understanding the goal of this script would be to update containers with the updated code in the review which is being gated. However, as currently implemented, this task is updating the containers with all available updates in the repos enabled in the script, including dependencies and base OS packages. Until now it hasn't been a problem because the centos updates repo is not enabled and base packages don't change after initial CentOS minor release is published. However, yesterday, CentOS 7.5 was released and published into base repo, so the script is trying to update all the base packages.

I think the best solution for this would be that overcloud-prep-containers.sh makes a smarter update selecting only the required packages to be updated, that means, the ones in the temporary repository created by DLRN as part of the oooq job.

Revision history for this message
Sagi (Sergey) Shnaidman (sshnaidm) wrote :

@Alfredo, it was working so until we had problems that built packages required updated packages from base repo: https://bugs.launchpad.net/tripleo/+bug/1747986

Revision history for this message
Alfredo Moralejo (amoralej) wrote :

@Sagi, yes we need to have enabled base and dependencies repo so that they are available when a package update requires it. But my point is that those packages shouldn't be updated unless they are required by the code being tested.

Let me put an example, let's say this job is running in neutron gate

1. dlrn rebuilds neutron as part of the oooq job on the fly, with the tested patch
2. It creates a "temporary repo" with only the rebuilt packages, openstack-neutron-common openstack-neutron-l3, etc...
3. we should do yum -y update openstack-neutron-common openstack-neutron-... instead of yum update -y8
4. if a new dependency has been added to the spec wich is in base repo it will be installed by rpm dependencies resolution

Revision history for this message
Sagi (Sergey) Shnaidman (sshnaidm) wrote :

OK, so we have a few approaches to solve it.
1) Promote new containers based on 7.5 - promotion jobs should pass for it.
2) To update only we need to update:
  a) change to container-check script: https://github.com/imain/container-check/pull/2
  b) change to extras: https://review.openstack.org/#/c/567550/
3) to disable base os repo temporary: https://review.openstack.org/#/c/567526/

Revision history for this message
wes hayutin (weshayutin) wrote :

Let's turn off the container update until we get a promotion

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-common (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/567636

Revision history for this message
wes hayutin (weshayutin) wrote :

Turning off container updates for now https://review.openstack.org/#/c/567580/

wes hayutin (weshayutin)
tags: removed: alert
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-common (master)

Reviewed: https://review.openstack.org/567636
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=3f12a00533e32d894ad18627e6be1163c0dd5ad8
Submitter: Zuul
Branch: master

commit 3f12a00533e32d894ad18627e6be1163c0dd5ad8
Author: Alex Schultz <email address hidden>
Date: Thu May 10 10:18:55 2018 -0600

    Add yum update to base

    In order to handle out of date distro containers, let's add a yum update
    to the kolla base container to ensure that at the end of the base
    container build we'll have the latest base packages installed. This will
    ensure we don't hit out of date dependency bits in later containers. For
    example, we currently only have 7.4 base centos image available from the
    upstream but we're install the base repos in the base build. This would
    mean that we'd be getting 7.5 packages in later containers but might
    possibly have a mix of 7.4 and 7.5 packages installed.

    Change-Id: Ifdfc1b84b7b6a306868260611271fd49bdd594d4
    Related-Bug: #1770355

Revision history for this message
Sagi (Sergey) Shnaidman (sshnaidm) wrote :

We wait for https://pypi.org/project/container-check/#history to be released by @imain and then https://review.openstack.org/567550 (Run containers update only for required packages) to be merged.
Permanent solution will be in tripleo-common, patches are in pipeline: https://review.openstack.org/#/q/I4905c35fcd45ef0fe1845857020c7e1a44c8c3d6

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-common (stable/queens)

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/568270

wes hayutin (weshayutin)
Changed in tripleo:
status: Triaged → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-common (stable/queens)

Reviewed: https://review.openstack.org/568270
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=5bb2fe60e325eabc604a0d8596ec137b36997ba2
Submitter: Zuul
Branch: stable/queens

commit 5bb2fe60e325eabc604a0d8596ec137b36997ba2
Author: Alex Schultz <email address hidden>
Date: Thu May 10 10:18:55 2018 -0600

    Add yum update to base

    In order to handle out of date distro containers, let's add a yum update
    to the kolla base container to ensure that at the end of the base
    container build we'll have the latest base packages installed. This will
    ensure we don't hit out of date dependency bits in later containers. For
    example, we currently only have 7.4 base centos image available from the
    upstream but we're install the base repos in the base build. This would
    mean that we'd be getting 7.5 packages in later containers but might
    possibly have a mix of 7.4 and 7.5 packages installed.

    Change-Id: Ifdfc1b84b7b6a306868260611271fd49bdd594d4
    Related-Bug: #1770355
    (cherry picked from commit 3f12a00533e32d894ad18627e6be1163c0dd5ad8)

tags: added: in-stable-queens
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.