10:25:06  Started by upstream project "integration-distribution-mri-test-titanium" build number 25
10:25:06  originally caused by:
10:25:06   Started by timer
10:25:06  Running as SYSTEM
10:25:06  [EnvInject] - Loading node environment variables.
10:25:06  Building remotely on prd-centos8-robot-2c-8g-18626 (centos8-robot-2c-8g) in workspace /w/workspace/controller-csit-3node-clustering-tell-all-titanium
10:25:08  [ssh-agent] Looking for ssh-agent implementation...
10:25:08  [ssh-agent]   Exec ssh-agent (binary ssh-agent on a remote machine)
10:25:08  $ ssh-agent
10:25:08  SSH_AUTH_SOCK=/tmp/ssh-e8pYDbRZKmPn/agent.5279
10:25:08  SSH_AGENT_PID=5280
10:25:08  [ssh-agent] Started.
10:25:08  Running ssh-add (command line suppressed)
10:25:08  Identity added: /w/workspace/controller-csit-3node-clustering-tell-all-titanium@tmp/private_key_13272480186168674430.key (/w/workspace/controller-csit-3node-clustering-tell-all-titanium@tmp/private_key_13272480186168674430.key)
10:25:08  [ssh-agent] Using credentials jenkins (Release Engineering Jenkins Key)
10:25:08  The recommended git tool is: NONE
10:25:11  using credential opendaylight-jenkins-ssh
10:25:11  Wiping out workspace first.
10:25:11  Cloning the remote Git repository
10:25:11  Cloning repository git://devvexx.opendaylight.org/mirror/integration/test
10:25:11   > git init /w/workspace/controller-csit-3node-clustering-tell-all-titanium/test # timeout=10
10:25:11  Fetching upstream changes from git://devvexx.opendaylight.org/mirror/integration/test
10:25:11   > git --version # timeout=10
10:25:11   > git --version # 'git version 2.43.0'
10:25:11  using GIT_SSH to set credentials Release Engineering Jenkins Key
10:25:11  [INFO] Currently running in a labeled security context
10:25:11  [INFO] Currently SELinux is 'enforcing' on the host
10:25:11   > /usr/bin/chcon --type=ssh_home_t /w/workspace/controller-csit-3node-clustering-tell-all-titanium/test@tmp/jenkins-gitclient-ssh100721136121354808.key
10:25:11  Verifying host key using known hosts file
10:25:11  You're using 'Known hosts file' strategy to verify ssh host keys, but your known_hosts file does not exist, please go to 'Manage Jenkins' -> 'Security' -> 'Git Host Key Verification Configuration' and configure host key verification.
10:25:11   > git fetch --tags --force --progress -- git://devvexx.opendaylight.org/mirror/integration/test +refs/heads/*:refs/remotes/origin/* # timeout=10
10:25:16   > git config remote.origin.url git://devvexx.opendaylight.org/mirror/integration/test # timeout=10
10:25:16   > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
10:25:16   > git config remote.origin.url git://devvexx.opendaylight.org/mirror/integration/test # timeout=10
10:25:16  Fetching upstream changes from git://devvexx.opendaylight.org/mirror/integration/test
10:25:16  using GIT_SSH to set credentials Release Engineering Jenkins Key
10:25:16  [INFO] Currently running in a labeled security context
10:25:16  [INFO] Currently SELinux is 'enforcing' on the host
10:25:16   > /usr/bin/chcon --type=ssh_home_t /w/workspace/controller-csit-3node-clustering-tell-all-titanium/test@tmp/jenkins-gitclient-ssh1673857666362375066.key
10:25:16  Verifying host key using known hosts file
10:25:16  You're using 'Known hosts file' strategy to verify ssh host keys, but your known_hosts file does not exist, please go to 'Manage Jenkins' -> 'Security' -> 'Git Host Key Verification Configuration' and configure host key verification.
10:25:16   > git fetch --tags --force --progress -- git://devvexx.opendaylight.org/mirror/integration/test master # timeout=10
10:25:17   > git rev-parse FETCH_HEAD^{commit} # timeout=10
10:25:17  Checking out Revision 1d2a1a30bcc48e3c8d571eb0d9b6891fe29dfed4 (origin/master)
10:25:17   > git config core.sparsecheckout # timeout=10
10:25:17   > git checkout -f 1d2a1a30bcc48e3c8d571eb0d9b6891fe29dfed4 # timeout=10
10:25:17  Commit message: "Add bgpcep variables for titanium"
10:25:17   > git rev-parse FETCH_HEAD^{commit} # timeout=10
10:25:17   > git rev-list --no-walk d42b2c24fe4fcc7fe3ae0d352311e768f07a15b7 # timeout=10
10:25:18  No emails were triggered.
10:25:18  provisioning config files...
10:25:18  copy managed file [npmrc] to file:/home/jenkins/.npmrc
10:25:18  copy managed file [pipconf] to file:/home/jenkins/.config/pip/pip.conf
10:25:19  copy managed file [clouds-yaml] to file:/home/jenkins/.config/openstack/clouds.yaml
10:25:19  [controller-csit-3node-clustering-tell-all-titanium] $ /bin/bash /tmp/jenkins10148553457177012809.sh
10:25:19  ---> python-tools-install.sh
10:25:19  Setup pyenv:
10:25:19    system
10:25:19  * 3.8.13 (set by /opt/pyenv/version)
10:25:19  * 3.9.13 (set by /opt/pyenv/version)
10:25:19  * 3.10.13 (set by /opt/pyenv/version)
10:25:19  * 3.11.7 (set by /opt/pyenv/version)
10:25:23  lf-activate-venv(): INFO: Creating python3 venv at /tmp/venv-YBX5
10:25:23  lf-activate-venv(): INFO: Save venv in file: /tmp/.os_lf_venv
10:25:26  lf-activate-venv(): INFO: Installing: lftools 
10:25:51  lf-activate-venv(): INFO: Adding /tmp/venv-YBX5/bin to PATH
10:25:51  Generating Requirements File
10:26:10  Python 3.11.7
10:26:11  pip 25.0.1 from /tmp/venv-YBX5/lib/python3.11/site-packages/pip (python 3.11)
10:26:11  appdirs==1.4.4
10:26:11  argcomplete==3.5.3
10:26:11  aspy.yaml==1.3.0
10:26:11  attrs==25.1.0
10:26:11  autopage==0.5.2
10:26:11  beautifulsoup4==4.13.3
10:26:11  boto3==1.36.21
10:26:11  botocore==1.36.21
10:26:11  bs4==0.0.2
10:26:11  cachetools==5.5.1
10:26:11  certifi==2025.1.31
10:26:11  cffi==1.17.1
10:26:11  cfgv==3.4.0
10:26:11  chardet==5.2.0
10:26:11  charset-normalizer==3.4.1
10:26:11  click==8.1.8
10:26:11  cliff==4.8.0
10:26:11  cmd2==2.5.11
10:26:11  cryptography==3.3.2
10:26:11  debtcollector==3.0.0
10:26:11  decorator==5.1.1
10:26:11  defusedxml==0.7.1
10:26:11  Deprecated==1.2.18
10:26:11  distlib==0.3.9
10:26:11  dnspython==2.7.0
10:26:11  docker==4.2.2
10:26:11  dogpile.cache==1.3.4
10:26:11  durationpy==0.9
10:26:11  email_validator==2.2.0
10:26:11  filelock==3.17.0
10:26:11  future==1.0.0
10:26:11  gitdb==4.0.12
10:26:11  GitPython==3.1.44
10:26:11  google-auth==2.38.0
10:26:11  httplib2==0.22.0
10:26:11  identify==2.6.7
10:26:11  idna==3.10
10:26:11  importlib-resources==1.5.0
10:26:11  iso8601==2.1.0
10:26:11  Jinja2==3.1.5
10:26:11  jmespath==1.0.1
10:26:11  jsonpatch==1.33
10:26:11  jsonpointer==3.0.0
10:26:11  jsonschema==4.23.0
10:26:11  jsonschema-specifications==2024.10.1
10:26:11  keystoneauth1==5.9.1
10:26:11  kubernetes==32.0.0
10:26:11  lftools==0.37.10
10:26:11  lxml==5.3.1
10:26:11  MarkupSafe==3.0.2
10:26:11  msgpack==1.1.0
10:26:11  multi_key_dict==2.0.3
10:26:11  munch==4.0.0
10:26:11  netaddr==1.3.0
10:26:11  niet==1.4.2
10:26:11  nodeenv==1.9.1
10:26:11  oauth2client==4.1.3
10:26:11  oauthlib==3.2.2
10:26:11  openstacksdk==4.3.0
10:26:11  os-client-config==2.1.0
10:26:11  os-service-types==1.7.0
10:26:11  osc-lib==3.2.0
10:26:11  oslo.config==9.7.0
10:26:11  oslo.context==5.7.0
10:26:11  oslo.i18n==6.5.0
10:26:11  oslo.log==7.0.0
10:26:11  oslo.serialization==5.6.0
10:26:11  oslo.utils==8.1.0
10:26:11  packaging==24.2
10:26:11  pbr==6.1.1
10:26:11  platformdirs==4.3.6
10:26:11  prettytable==3.14.0
10:26:11  psutil==7.0.0
10:26:11  pyasn1==0.6.1
10:26:11  pyasn1_modules==0.4.1
10:26:11  pycparser==2.22
10:26:11  pygerrit2==2.0.15
10:26:11  PyGithub==2.6.0
10:26:11  PyJWT==2.10.1
10:26:11  PyNaCl==1.5.0
10:26:11  pyparsing==2.4.7
10:26:11  pyperclip==1.9.0
10:26:11  pyrsistent==0.20.0
10:26:11  python-cinderclient==9.6.0
10:26:11  python-dateutil==2.9.0.post0
10:26:11  python-heatclient==4.1.0
10:26:11  python-jenkins==1.8.2
10:26:11  python-keystoneclient==5.5.0
10:26:11  python-magnumclient==4.7.0
10:26:11  python-openstackclient==7.2.1
10:26:11  python-swiftclient==4.6.0
10:26:11  PyYAML==6.0.2
10:26:11  referencing==0.36.2
10:26:11  requests==2.32.3
10:26:11  requests-oauthlib==2.0.0
10:26:11  requestsexceptions==1.4.0
10:26:11  rfc3986==2.0.0
10:26:11  rpds-py==0.22.3
10:26:11  rsa==4.9
10:26:11  ruamel.yaml==0.18.10
10:26:11  ruamel.yaml.clib==0.2.12
10:26:11  s3transfer==0.11.2
10:26:11  simplejson==3.20.1
10:26:11  six==1.17.0
10:26:11  smmap==5.0.2
10:26:11  soupsieve==2.6
10:26:11  stevedore==5.4.0
10:26:11  tabulate==0.9.0
10:26:11  toml==0.10.2
10:26:11  tomlkit==0.13.2
10:26:11  tqdm==4.67.1
10:26:11  typing_extensions==4.12.2
10:26:11  tzdata==2025.1
10:26:11  urllib3==1.26.20
10:26:11  virtualenv==20.29.2
10:26:11  wcwidth==0.2.13
10:26:11  websocket-client==1.8.0
10:26:11  wrapt==1.17.2
10:26:11  xdg==6.0.0
10:26:11  xmltodict==0.14.2
10:26:11  yq==3.4.3
10:26:11  [EnvInject] - Injecting environment variables from a build step.
10:26:11  [EnvInject] - Injecting as environment variables the properties content 
10:26:11  OS_STACK_TEMPLATE=csit-2-instance-type.yaml
10:26:11  OS_CLOUD=vex
10:26:11  OS_STACK_NAME=releng-controller-csit-3node-clustering-tell-all-titanium-25
10:26:11  OS_STACK_TEMPLATE_DIR=openstack-hot
10:26:11  
10:26:11  [EnvInject] - Variables injected successfully.
10:26:11  provisioning config files...
10:26:11  copy managed file [clouds-yaml] to file:/home/jenkins/.config/openstack/clouds.yaml
10:26:11  [controller-csit-3node-clustering-tell-all-titanium] $ /bin/bash /tmp/jenkins1161626474477355448.sh
10:26:11  ---> Create parameters file for OpenStack HOT
10:26:11  OpenStack Heat parameters generated
10:26:11  -----------------------------------
10:26:11  parameters:
10:26:11      vm_0_count: '3'
10:26:11      vm_0_flavor: 'v3-standard-4'
10:26:11      vm_0_image: 'ZZCI - Ubuntu 20.04 - builder - x86_64 - 20250101-160150.998'
10:26:11      vm_1_count: '0'
10:26:11      vm_1_flavor: 'v3-standard-2'
10:26:11      vm_1_image: 'ZZCI - Ubuntu 22.04 - mininet-ovs-217 - x86_64 - 20250201-060151.911'
10:26:11      
10:26:11      job_name: '14749-25'
10:26:11      silo: 'releng'
10:26:11  [controller-csit-3node-clustering-tell-all-titanium] $ /bin/bash -l /tmp/jenkins1319615629330504793.sh
10:26:11  ---> Create HEAT stack
10:26:11  + source /home/jenkins/lf-env.sh
10:26:11  + lf-activate-venv --python python3 'lftools[openstack]' kubernetes niet python-heatclient python-openstackclient python-magnumclient yq
10:26:11  ++ mktemp -d /tmp/venv-XXXX
10:26:11  + lf_venv=/tmp/venv-Iusu
10:26:11  + local venv_file=/tmp/.os_lf_venv
10:26:11  + local python=python3
10:26:11  + local options
10:26:11  + local set_path=true
10:26:11  + local install_args=
10:26:11  ++ getopt -o np:v: -l no-path,system-site-packages,python:,venv-file: -n lf-activate-venv -- --python python3 'lftools[openstack]' kubernetes niet python-heatclient python-openstackclient python-magnumclient yq
10:26:11  + options=' --python '\''python3'\'' -- '\''lftools[openstack]'\'' '\''kubernetes'\'' '\''niet'\'' '\''python-heatclient'\'' '\''python-openstackclient'\'' '\''python-magnumclient'\'' '\''yq'\'''
10:26:11  + eval set -- ' --python '\''python3'\'' -- '\''lftools[openstack]'\'' '\''kubernetes'\'' '\''niet'\'' '\''python-heatclient'\'' '\''python-openstackclient'\'' '\''python-magnumclient'\'' '\''yq'\'''
10:26:11  ++ set -- --python python3 -- 'lftools[openstack]' kubernetes niet python-heatclient python-openstackclient python-magnumclient yq
10:26:11  + true
10:26:11  + case $1 in
10:26:11  + python=python3
10:26:11  + shift 2
10:26:11  + true
10:26:11  + case $1 in
10:26:11  + shift
10:26:11  + break
10:26:11  + case $python in
10:26:11  + local pkg_list=
10:26:11  + [[ -d /opt/pyenv ]]
10:26:11  + echo 'Setup pyenv:'
10:26:11  Setup pyenv:
10:26:11  + export PYENV_ROOT=/opt/pyenv
10:26:11  + PYENV_ROOT=/opt/pyenv
10:26:11  + export PATH=/opt/pyenv/bin:/home/jenkins/.local/bin:/home/jenkins/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin
10:26:11  + PATH=/opt/pyenv/bin:/home/jenkins/.local/bin:/home/jenkins/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin
10:26:11  + pyenv versions
10:26:11    system
10:26:11    3.8.13
10:26:11    3.9.13
10:26:11    3.10.13
10:26:11  * 3.11.7 (set by /w/workspace/controller-csit-3node-clustering-tell-all-titanium/.python-version)
10:26:11  + command -v pyenv
10:26:11  ++ pyenv init - --no-rehash
10:26:11  + eval 'PATH="$(bash --norc -ec '\''IFS=:; paths=($PATH); 
10:26:11  for i in ${!paths[@]}; do 
10:26:11  if [[ ${paths[i]} == "'\'''\''/opt/pyenv/shims'\'''\''" ]]; then unset '\''\'\'''\''paths[i]'\''\'\'''\''; 
10:26:11  fi; done; 
10:26:11  echo "${paths[*]}"'\'')"
10:26:11  export PATH="/opt/pyenv/shims:${PATH}"
10:26:11  export PYENV_SHELL=bash
10:26:11  source '\''/opt/pyenv/libexec/../completions/pyenv.bash'\''
10:26:11  pyenv() {
10:26:11    local command
10:26:11    command="${1:-}"
10:26:11    if [ "$#" -gt 0 ]; then
10:26:11      shift
10:26:11    fi
10:26:11  
10:26:11    case "$command" in
10:26:11    rehash|shell)
10:26:11      eval "$(pyenv "sh-$command" "$@")"
10:26:11      ;;
10:26:11    *)
10:26:11      command pyenv "$command" "$@"
10:26:11      ;;
10:26:11    esac
10:26:11  }'
10:26:11  +++ bash --norc -ec 'IFS=:; paths=($PATH); 
10:26:11  for i in ${!paths[@]}; do 
10:26:11  if [[ ${paths[i]} == "/opt/pyenv/shims" ]]; then unset '\''paths[i]'\''; 
10:26:11  fi; done; 
10:26:11  echo "${paths[*]}"'
10:26:11  ++ PATH=/opt/pyenv/bin:/home/jenkins/.local/bin:/home/jenkins/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin
10:26:11  ++ export PATH=/opt/pyenv/shims:/opt/pyenv/bin:/home/jenkins/.local/bin:/home/jenkins/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin
10:26:11  ++ PATH=/opt/pyenv/shims:/opt/pyenv/bin:/home/jenkins/.local/bin:/home/jenkins/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin
10:26:11  ++ export PYENV_SHELL=bash
10:26:11  ++ PYENV_SHELL=bash
10:26:11  ++ source /opt/pyenv/libexec/../completions/pyenv.bash
10:26:11  +++ complete -F _pyenv pyenv
10:26:11  ++ lf-pyver python3
10:26:11  ++ local py_version_xy=python3
10:26:11  ++ local py_version_xyz=
10:26:11  ++ awk '{ print $1 }'
10:26:11  ++ pyenv versions
10:26:11  ++ local command
10:26:11  ++ command=versions
10:26:11  ++ '[' 1 -gt 0 ']'
10:26:11  ++ shift
10:26:11  ++ case "$command" in
10:26:11  ++ command pyenv versions
10:26:11  ++ pyenv versions
10:26:11  ++ sed 's/^[ *]* //'
10:26:12  ++ grep -E '^[0-9.]*[0-9]$'
10:26:12  ++ [[ ! -s /tmp/.pyenv_versions ]]
10:26:12  +++ grep '^3' /tmp/.pyenv_versions
10:26:12  +++ tail -n 1
10:26:12  +++ sort -V
10:26:12  ++ py_version_xyz=3.11.7
10:26:12  ++ [[ -z 3.11.7 ]]
10:26:12  ++ echo 3.11.7
10:26:12  ++ return 0
10:26:12  + pyenv local 3.11.7
10:26:12  + local command
10:26:12  + command=local
10:26:12  + '[' 2 -gt 0 ']'
10:26:12  + shift
10:26:12  + case "$command" in
10:26:12  + command pyenv local 3.11.7
10:26:12  + pyenv local 3.11.7
10:26:12  + for arg in "$@"
10:26:12  + case $arg in
10:26:12  + pkg_list+='lftools[openstack] '
10:26:12  + for arg in "$@"
10:26:12  + case $arg in
10:26:12  + pkg_list+='kubernetes '
10:26:12  + for arg in "$@"
10:26:12  + case $arg in
10:26:12  + pkg_list+='niet '
10:26:12  + for arg in "$@"
10:26:12  + case $arg in
10:26:12  + pkg_list+='python-heatclient '
10:26:12  + for arg in "$@"
10:26:12  + case $arg in
10:26:12  + pkg_list+='python-openstackclient '
10:26:12  + for arg in "$@"
10:26:12  + case $arg in
10:26:12  + pkg_list+='python-magnumclient '
10:26:12  + for arg in "$@"
10:26:12  + case $arg in
10:26:12  + pkg_list+='yq '
10:26:12  + [[ -f /tmp/.os_lf_venv ]]
10:26:12  ++ cat /tmp/.os_lf_venv
10:26:12  + lf_venv=/tmp/venv-YBX5
10:26:12  + echo 'lf-activate-venv(): INFO: Reuse venv:/tmp/venv-YBX5 from' file:/tmp/.os_lf_venv
10:26:12  lf-activate-venv(): INFO: Reuse venv:/tmp/venv-YBX5 from file:/tmp/.os_lf_venv
10:26:12  + /tmp/venv-YBX5/bin/python3 -m pip install --upgrade --quiet pip virtualenv
10:26:13  + [[ -z lftools[openstack] kubernetes niet python-heatclient python-openstackclient python-magnumclient yq  ]]
10:26:13  + echo 'lf-activate-venv(): INFO: Installing: lftools[openstack] kubernetes niet python-heatclient python-openstackclient python-magnumclient yq '
10:26:13  lf-activate-venv(): INFO: Installing: lftools[openstack] kubernetes niet python-heatclient python-openstackclient python-magnumclient yq 
10:26:13  + /tmp/venv-YBX5/bin/python3 -m pip install --upgrade --quiet --upgrade-strategy eager 'lftools[openstack]' kubernetes niet python-heatclient python-openstackclient python-magnumclient yq
10:26:28  + type python3
10:26:28  + true
10:26:28  + echo 'lf-activate-venv(): INFO: Adding /tmp/venv-YBX5/bin to PATH'
10:26:28  lf-activate-venv(): INFO: Adding /tmp/venv-YBX5/bin to PATH
10:26:28  + PATH=/tmp/venv-YBX5/bin:/opt/pyenv/shims:/opt/pyenv/bin:/home/jenkins/.local/bin:/home/jenkins/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin
10:26:28  + return 0
10:26:28  + openstack --os-cloud vex limits show --absolute
10:26:29  +--------------------------+---------+
10:26:29  | Name                     |   Value |
10:26:29  +--------------------------+---------+
10:26:29  | maxTotalInstances        |      -1 |
10:26:29  | maxTotalCores            |     450 |
10:26:29  | maxTotalRAMSize          | 1000000 |
10:26:29  | maxServerMeta            |     128 |
10:26:29  | maxImageMeta             |     128 |
10:26:29  | maxPersonality           |       5 |
10:26:29  | maxPersonalitySize       |   10240 |
10:26:29  | maxTotalKeypairs         |     100 |
10:26:29  | maxServerGroups          |      10 |
10:26:29  | maxServerGroupMembers    |      10 |
10:26:29  | maxTotalFloatingIps      |      -1 |
10:26:29  | maxSecurityGroups        |      -1 |
10:26:29  | maxSecurityGroupRules    |      -1 |
10:26:29  | totalRAMUsed             |  868352 |
10:26:29  | totalCoresUsed           |     212 |
10:26:29  | totalInstancesUsed       |      64 |
10:26:29  | totalFloatingIpsUsed     |       0 |
10:26:29  | totalSecurityGroupsUsed  |       0 |
10:26:29  | totalServerGroupsUsed    |       0 |
10:26:29  | maxTotalVolumes          |      -1 |
10:26:29  | maxTotalSnapshots        |      10 |
10:26:29  | maxTotalVolumeGigabytes  |    4096 |
10:26:29  | maxTotalBackups          |      10 |
10:26:29  | maxTotalBackupGigabytes  |    1000 |
10:26:29  | totalVolumesUsed         |       3 |
10:26:29  | totalGigabytesUsed       |      60 |
10:26:29  | totalSnapshotsUsed       |       0 |
10:26:29  | totalBackupsUsed         |       0 |
10:26:29  | totalBackupGigabytesUsed |       0 |
10:26:29  +--------------------------+---------+
10:26:29  + pushd /opt/ciman/openstack-hot
10:26:29  /opt/ciman/openstack-hot /w/workspace/controller-csit-3node-clustering-tell-all-titanium
10:26:29  + lftools openstack --os-cloud vex stack create releng-controller-csit-3node-clustering-tell-all-titanium-25 csit-2-instance-type.yaml /w/workspace/controller-csit-3node-clustering-tell-all-titanium/stack-parameters.yaml
10:26:43  Creating stack releng-controller-csit-3node-clustering-tell-all-titanium-25
10:26:43  WARN: Failed to initialize stack. Reason: Resource CREATE failed: Forbidden: resources.vm_0_group.resources[0].resources.instance: Quota exceeded for ram: Requested 16384, but already used 999424 of 1000000 ram (HTTP 403) (Request-ID: req-d9002790-cb2a-474e-b143-cb3fc96ef0de)
10:26:43  TypeError: delete() missing 1 required positional argument: 'force'
10:26:43  Build step 'Execute shell' marked build as failure
10:26:43  $ ssh-agent -k
10:26:43  unset SSH_AUTH_SOCK;
10:26:43  unset SSH_AGENT_PID;
10:26:43  echo Agent pid 5280 killed;
10:26:43  [ssh-agent] Stopped.
10:26:43  Recording plot data
10:26:43  Robot results publisher started...
10:26:43  INFO: Checking test criticality is deprecated and will be dropped in a future release!
10:26:43  -Parsing output xml:
10:26:43  Failed!
10:26:43  hudson.AbortException: No files found in path /w/workspace/controller-csit-3node-clustering-tell-all-titanium with configured filemask: output.xml
10:26:43  	at hudson.plugins.robot.RobotParser$RobotParserCallable.invoke(RobotParser.java:81)
10:26:43  	at hudson.plugins.robot.RobotParser$RobotParserCallable.invoke(RobotParser.java:52)
10:26:43  	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3616)
10:26:43  	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
10:26:43  	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
10:26:43  	at hudson.remoting.Request$2.run(Request.java:377)
10:26:43  	at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
10:26:43  	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
10:26:43  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
10:26:43  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
10:26:43  	at java.base/java.lang.Thread.run(Thread.java:829)
10:26:43  	Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to prd-centos8-robot-2c-8g-18626
10:26:43  		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1787)
10:26:43  		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
10:26:43  		at hudson.remoting.Channel.call(Channel.java:1003)
10:26:43  		at hudson.FilePath.act(FilePath.java:1230)
10:26:43  		at hudson.FilePath.act(FilePath.java:1219)
10:26:43  		at hudson.plugins.robot.RobotParser.parse(RobotParser.java:48)
10:26:43  		at hudson.plugins.robot.RobotPublisher.parse(RobotPublisher.java:246)
10:26:43  		at hudson.plugins.robot.RobotPublisher.perform(RobotPublisher.java:268)
10:26:43  		at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:80)
10:26:43  		at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
10:26:43  		at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:818)
10:26:43  		at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:767)
10:26:43  		at hudson.model.Build$BuildExecution.post2(Build.java:179)
10:26:43  		at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:711)
10:26:43  		at hudson.model.Run.execute(Run.java:1918)
10:26:43  		at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
10:26:43  		at hudson.model.ResourceController.execute(ResourceController.java:101)
10:26:43  		at hudson.model.Executor.run(Executor.java:442)
10:26:43  [PostBuildScript] - [INFO] Executing post build scripts.
10:26:43  [controller-csit-3node-clustering-tell-all-titanium] $ /bin/bash /tmp/jenkins10208409535434859731.sh
10:26:43  Archiving csit artifacts
10:26:43  seq: invalid floating point argument: ‘’
10:26:43  Try 'seq --help' for more information.
10:26:43    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
10:26:43                                   Dload  Upload   Total   Spent    Left  Speed
10:26:43  
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   237    0   237    0     0   6076      0 --:--:-- --:--:-- --:--:--  6236
10:26:43  Archive:  robot-plugin.zip
10:26:43    End-of-central-directory signature not found.  Either this file is not
10:26:43    a zipfile, or it constitutes one disk of a multi-part archive.  In the
10:26:43    latter case the central directory and zipfile comment will be found on
10:26:43    the last disk(s) of this archive.
10:26:43  unzip:  cannot find zipfile directory in one of robot-plugin.zip or
10:26:43          robot-plugin.zip.zip, and cannot find robot-plugin.zip.ZIP, period.
10:26:43  mv: cannot stat '*.log': No such file or directory
10:26:43  mv: cannot stat '*.log.gz': No such file or directory
10:26:43  mv: cannot stat '*.csv': No such file or directory
10:26:43  mv: cannot stat '*.png': No such file or directory
10:26:44  [PostBuildScript] - [INFO] Executing post build scripts.
10:26:44  [controller-csit-3node-clustering-tell-all-titanium] $ /bin/bash /tmp/jenkins18171016394871128937.sh
10:26:44  [PostBuildScript] - [INFO] Executing post build scripts.
10:26:44  [EnvInject] - Injecting environment variables from a build step.
10:26:44  [EnvInject] - Injecting as environment variables the properties content 
10:26:44  OS_CLOUD=vex
10:26:44  OS_STACK_NAME=releng-controller-csit-3node-clustering-tell-all-titanium-25
10:26:44  
10:26:44  [EnvInject] - Variables injected successfully.
10:26:44  provisioning config files...
10:26:44  copy managed file [clouds-yaml] to file:/home/jenkins/.config/openstack/clouds.yaml
10:26:44  [controller-csit-3node-clustering-tell-all-titanium] $ /bin/bash /tmp/jenkins13749055620442473890.sh
10:26:44  ---> openstack-stack-delete.sh
10:26:44  Setup pyenv:
10:26:44    system
10:26:44    3.8.13
10:26:44    3.9.13
10:26:44    3.10.13
10:26:44  * 3.11.7 (set by /w/workspace/controller-csit-3node-clustering-tell-all-titanium/.python-version)
10:26:44  lf-activate-venv(): INFO: Reuse venv:/tmp/venv-YBX5 from file:/tmp/.os_lf_venv
10:26:45  lf-activate-venv(): INFO: Installing: lftools[openstack] kubernetes python-heatclient python-openstackclient 
10:26:56  lf-activate-venv(): INFO: Adding /tmp/venv-YBX5/bin to PATH
10:26:56  INFO: Retrieving stack cost for: releng-controller-csit-3node-clustering-tell-all-titanium-25
10:27:00  openstack.exceptions.BadRequestException: BadRequestException: 400: Client Error for url: https://compute.public.mtl1.vexxhost.net/v2.1/12c36e260d8e4bb2913965203b1b491f/servers/detail?name=, Invalid input for query parameters name. Value: . '' is not a 'regex'
10:27:00  WARNING: Unable to get stack costs, continuing anyway
10:27:12  INFO: Deleting stack releng-controller-csit-3node-clustering-tell-all-titanium-25
10:27:12  Successfully deleted stack releng-controller-csit-3node-clustering-tell-all-titanium-25
10:27:12  [PostBuildScript] - [INFO] Executing post build scripts.
10:27:12  [controller-csit-3node-clustering-tell-all-titanium] $ /bin/bash /tmp/jenkins7987818424206494848.sh
10:27:12  ---> sysstat.sh
10:27:12  [controller-csit-3node-clustering-tell-all-titanium] $ /bin/bash /tmp/jenkins7937120272244318922.sh
10:27:12  ---> package-listing.sh
10:27:12  ++ facter osfamily
10:27:12  ++ tr '[:upper:]' '[:lower:]'
10:27:12  + OS_FAMILY=redhat
10:27:12  + workspace=/w/workspace/controller-csit-3node-clustering-tell-all-titanium
10:27:12  + START_PACKAGES=/tmp/packages_start.txt
10:27:12  + END_PACKAGES=/tmp/packages_end.txt
10:27:12  + DIFF_PACKAGES=/tmp/packages_diff.txt
10:27:12  + PACKAGES=/tmp/packages_start.txt
10:27:12  + '[' /w/workspace/controller-csit-3node-clustering-tell-all-titanium ']'
10:27:12  + PACKAGES=/tmp/packages_end.txt
10:27:12  + case "${OS_FAMILY}" in
10:27:12  + rpm -qa
10:27:12  + sort
10:27:13  + '[' -f /tmp/packages_start.txt ']'
10:27:13  + '[' -f /tmp/packages_end.txt ']'
10:27:13  + diff /tmp/packages_start.txt /tmp/packages_end.txt
10:27:13  + '[' /w/workspace/controller-csit-3node-clustering-tell-all-titanium ']'
10:27:13  + mkdir -p /w/workspace/controller-csit-3node-clustering-tell-all-titanium/archives/
10:27:13  + cp -f /tmp/packages_diff.txt /tmp/packages_end.txt /tmp/packages_start.txt /w/workspace/controller-csit-3node-clustering-tell-all-titanium/archives/
10:27:13  [controller-csit-3node-clustering-tell-all-titanium] $ /bin/bash /tmp/jenkins9856561416432791004.sh
10:27:13  ---> capture-instance-metadata.sh
10:27:13  Setup pyenv:
10:27:13    system
10:27:13    3.8.13
10:27:13    3.9.13
10:27:13    3.10.13
10:27:13  * 3.11.7 (set by /w/workspace/controller-csit-3node-clustering-tell-all-titanium/.python-version)
10:27:14  lf-activate-venv(): INFO: Reuse venv:/tmp/venv-YBX5 from file:/tmp/.os_lf_venv
10:27:14  lf-activate-venv(): INFO: Installing: lftools 
10:27:22  lf-activate-venv(): INFO: Adding /tmp/venv-YBX5/bin to PATH
10:27:22  INFO: Running in OpenStack, capturing instance metadata
10:27:23  [controller-csit-3node-clustering-tell-all-titanium] $ /bin/bash /tmp/jenkins3306302338351224077.sh
10:27:23  provisioning config files...
10:27:23  Could not find credentials [logs] for controller-csit-3node-clustering-tell-all-titanium #25
10:27:23  copy managed file [jenkins-log-archives-settings] to file:/w/workspace/controller-csit-3node-clustering-tell-all-titanium@tmp/config12960196143958755147tmp
10:27:23  Regular expression run condition: Expression=[^.*logs-s3.*], Label=[odl-logs-s3-cloudfront-index]
10:27:23  Run condition [Regular expression match] enabling perform for step [Provide Configuration files]
10:27:23  provisioning config files...
10:27:23  copy managed file [jenkins-s3-log-ship] to file:/home/jenkins/.aws/credentials
10:27:23  [EnvInject] - Injecting environment variables from a build step.
10:27:23  [EnvInject] - Injecting as environment variables the properties content 
10:27:23  SERVER_ID=logs
10:27:23  
10:27:23  [EnvInject] - Variables injected successfully.
10:27:23  [controller-csit-3node-clustering-tell-all-titanium] $ /bin/bash /tmp/jenkins7251208718517682582.sh
10:27:23  ---> create-netrc.sh
10:27:23  WARN: Log server credential not found.
10:27:23  [controller-csit-3node-clustering-tell-all-titanium] $ /bin/bash /tmp/jenkins13505363133853042596.sh
10:27:23  ---> python-tools-install.sh
10:27:23  Setup pyenv:
10:27:24    system
10:27:24    3.8.13
10:27:24    3.9.13
10:27:24    3.10.13
10:27:24  * 3.11.7 (set by /w/workspace/controller-csit-3node-clustering-tell-all-titanium/.python-version)
10:27:24  lf-activate-venv(): INFO: Reuse venv:/tmp/venv-YBX5 from file:/tmp/.os_lf_venv
10:27:25  lf-activate-venv(): INFO: Installing: lftools 
10:27:32  lf-activate-venv(): INFO: Adding /tmp/venv-YBX5/bin to PATH
10:27:32  [controller-csit-3node-clustering-tell-all-titanium] $ /bin/bash /tmp/jenkins4705654392151439309.sh
10:27:32  ---> sudo-logs.sh
10:27:32  Archiving 'sudo' log..
10:27:32  [controller-csit-3node-clustering-tell-all-titanium] $ /bin/bash /tmp/jenkins3255053830271423497.sh
10:27:32  ---> job-cost.sh
10:27:32  Setup pyenv:
10:27:32    system
10:27:32    3.8.13
10:27:32    3.9.13
10:27:32    3.10.13
10:27:32  * 3.11.7 (set by /w/workspace/controller-csit-3node-clustering-tell-all-titanium/.python-version)
10:27:32  lf-activate-venv(): INFO: Reuse venv:/tmp/venv-YBX5 from file:/tmp/.os_lf_venv
10:27:33  lf-activate-venv(): INFO: Installing: zipp==1.1.0 python-openstackclient urllib3~=1.26.15 
10:27:38  lf-activate-venv(): INFO: Adding /tmp/venv-YBX5/bin to PATH
10:27:38  DEBUG: total: 0
10:27:38  INFO: Retrieving Stack Cost...
10:27:39  INFO: Retrieving Pricing Info for: v3-standard-2
10:27:39  INFO: Archiving Costs
10:27:39  [controller-csit-3node-clustering-tell-all-titanium] $ /bin/bash -l /tmp/jenkins2039627767303328667.sh
10:27:39  ---> logs-deploy.sh
10:27:39  Setup pyenv:
10:27:39    system
10:27:39    3.8.13
10:27:39    3.9.13
10:27:39    3.10.13
10:27:39  * 3.11.7 (set by /w/workspace/controller-csit-3node-clustering-tell-all-titanium/.python-version)
10:27:39  lf-activate-venv(): INFO: Reuse venv:/tmp/venv-YBX5 from file:/tmp/.os_lf_venv
10:27:40  lf-activate-venv(): INFO: Installing: lftools 
10:27:48  lf-activate-venv(): INFO: Adding /tmp/venv-YBX5/bin to PATH
10:27:48  WARNING: Nexus logging server not set
10:27:48  INFO: S3 path logs/releng/vex-yul-odl-jenkins-1/controller-csit-3node-clustering-tell-all-titanium/25/
10:27:48  INFO: archiving logs to S3
10:27:49  ---> uname -a:
10:27:49  Linux prd-centos8-robot-2c-8g-18626.novalocal 4.18.0-553.5.1.el8.x86_64 #1 SMP Tue May 21 05:46:01 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
10:27:49  
10:27:49  
10:27:49  ---> lscpu:
10:27:49  Architecture:        x86_64
10:27:49  CPU op-mode(s):      32-bit, 64-bit
10:27:49  Byte Order:          Little Endian
10:27:49  CPU(s):              2
10:27:49  On-line CPU(s) list: 0,1
10:27:49  Thread(s) per core:  1
10:27:49  Core(s) per socket:  1
10:27:49  Socket(s):           2
10:27:49  NUMA node(s):        1
10:27:49  Vendor ID:           AuthenticAMD
10:27:49  CPU family:          23
10:27:49  Model:               49
10:27:49  Model name:          AMD EPYC-Rome Processor
10:27:49  Stepping:            0
10:27:49  CPU MHz:             2799.998
10:27:49  BogoMIPS:            5599.99
10:27:49  Virtualization:      AMD-V
10:27:49  Hypervisor vendor:   KVM
10:27:49  Virtualization type: full
10:27:49  L1d cache:           32K
10:27:49  L1i cache:           32K
10:27:49  L2 cache:            512K
10:27:49  L3 cache:            16384K
10:27:49  NUMA node0 CPU(s):   0,1
10:27:49  Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd ibrs ibpb stibp vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr wbnoinvd arat npt nrip_save umip rdpid arch_capabilities
10:27:49  
10:27:49  
10:27:49  ---> nproc:
10:27:49  2
10:27:49  
10:27:49  
10:27:49  ---> df -h:
10:27:49  Filesystem      Size  Used Avail Use% Mounted on
10:27:49  devtmpfs        3.8G     0  3.8G   0% /dev
10:27:49  tmpfs           3.8G     0  3.8G   0% /dev/shm
10:27:49  tmpfs           3.8G   17M  3.8G   1% /run
10:27:49  tmpfs           3.8G     0  3.8G   0% /sys/fs/cgroup
10:27:49  /dev/vda1        40G  8.3G   32G  21% /
10:27:49  tmpfs           770M     0  770M   0% /run/user/1001
10:27:49  
10:27:49  
10:27:49  ---> free -m:
10:27:49                total        used        free      shared  buff/cache   available
10:27:49  Mem:           7697         564        5069          19        2063        6838
10:27:49  Swap:          1023           0        1023
10:27:49  
10:27:49  
10:27:49  ---> ip addr:
10:27:49  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
10:27:49      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
10:27:49      inet 127.0.0.1/8 scope host lo
10:27:49         valid_lft forever preferred_lft forever
10:27:49      inet6 ::1/128 scope host 
10:27:49         valid_lft forever preferred_lft forever
10:27:49  2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1458 qdisc mq state UP group default qlen 1000
10:27:49      link/ether fa:16:3e:89:2a:88 brd ff:ff:ff:ff:ff:ff
10:27:49      altname enp0s3
10:27:49      altname ens3
10:27:49      inet 10.30.170.66/23 brd 10.30.171.255 scope global dynamic noprefixroute eth0
10:27:49         valid_lft 86192sec preferred_lft 86192sec
10:27:49      inet6 fe80::f816:3eff:fe89:2a88/64 scope link 
10:27:49         valid_lft forever preferred_lft forever
10:27:49  
10:27:49  
10:27:49  ---> sar -b -r -n DEV:
10:27:49  Linux 4.18.0-553.5.1.el8.x86_64 (centos-stream-8-robot-7d7a37eb-bc14-4dd6-9530-dc22c5eae738.noval) 	02/17/2025 	_x86_64_	(2 CPU)
10:27:49  
10:27:49  10:24:16     LINUX RESTART	(2 CPU)
10:27:49  
10:27:49  10:25:01 AM       tps      rtps      wtps   bread/s   bwrtn/s
10:27:49  10:26:01 AM     89.69     13.58     76.10   3760.84   6118.06
10:27:49  10:27:01 AM     89.75      0.95     88.80     88.79  10722.03
10:27:49  Average:        89.72      7.27     82.45   1924.81   8420.05
10:27:49  
10:27:49  10:25:01 AM kbmemfree   kbavail kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty
10:27:49  10:26:01 AM   5378604   7070864   2503820     31.76      2688   1875152    690344      7.73    186668   1997388    166016
10:27:49  10:27:01 AM   5229352   7032440   2653072     33.66      2688   1980684    702584      7.87    224936   2081516       176
10:27:49  Average:      5303978   7051652   2578446     32.71      2688   1927918    696464      7.80    205802   2039452     83096
10:27:49  
10:27:49  10:25:01 AM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s   %ifutil
10:27:49  10:26:01 AM        lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:27:49  10:26:01 AM      eth0    325.31    204.38   1782.39     64.31      0.00      0.00      0.00      0.00
10:27:49  10:27:01 AM        lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:27:49  10:27:01 AM      eth0     63.09     46.66    502.22     12.13      0.00      0.00      0.00      0.00
10:27:49  Average:           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
10:27:49  Average:         eth0    194.20    125.52   1142.30     38.22      0.00      0.00      0.00      0.00
10:27:49  
10:27:49  
10:27:49  ---> sar -P ALL:
10:27:49  Linux 4.18.0-553.5.1.el8.x86_64 (centos-stream-8-robot-7d7a37eb-bc14-4dd6-9530-dc22c5eae738.noval) 	02/17/2025 	_x86_64_	(2 CPU)
10:27:49  
10:27:49  10:24:16     LINUX RESTART	(2 CPU)
10:27:49  
10:27:49  10:25:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
10:27:49  10:26:01 AM     all     37.97      0.00      6.13      1.21      0.10     54.59
10:27:49  10:26:01 AM       0     42.68      0.00      7.35      1.65      0.10     48.21
10:27:49  10:26:01 AM       1     33.27      0.00      4.92      0.77      0.10     60.95
10:27:49  10:27:01 AM     all     28.44      0.00      3.78      2.04      0.09     65.65
10:27:49  10:27:01 AM       0     32.10      0.00      3.83      1.82      0.10     62.15
10:27:49  10:27:01 AM       1     24.77      0.00      3.72      2.27      0.08     69.16
10:27:49  Average:        all     33.20      0.00      4.95      1.63      0.10     60.12
10:27:49  Average:          0     37.39      0.00      5.59      1.74      0.10     55.19
10:27:49  Average:          1     29.02      0.00      4.32      1.52      0.09     65.05
10:27:49  
10:27:49  
10:27:49