01:01:53  Started by upstream project "integration-distribution-test-scandium" build number 198
01:01:53  originally caused by:
01:01:53   Started by upstream project "autorelease-release-scandium-mvn39-openjdk21" build number 204
01:01:53   originally caused by:
01:01:53    Started by timer
01:01:53  Running as SYSTEM
01:01:53  [EnvInject] - Loading node environment variables.
01:01:53  Building remotely on prd-centos8-robot-2c-8g-1856 (centos8-robot-2c-8g) in workspace /w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium
01:01:53  [ssh-agent] Looking for ssh-agent implementation...
01:01:54  [ssh-agent]   Exec ssh-agent (binary ssh-agent on a remote machine)
01:01:54  $ ssh-agent
01:01:54  SSH_AUTH_SOCK=/tmp/ssh-zu8gDNBYF5WA/agent.5281
01:01:54  SSH_AGENT_PID=5282
01:01:54  [ssh-agent] Started.
01:01:54  Running ssh-add (command line suppressed)
01:01:54  Identity added: /w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium@tmp/private_key_11689749455861283052.key (/w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium@tmp/private_key_11689749455861283052.key)
01:01:54  [ssh-agent] Using credentials jenkins (Release Engineering Jenkins Key)
01:01:54  The recommended git tool is: NONE
01:01:56  using credential opendaylight-jenkins-ssh
01:01:56  Wiping out workspace first.
01:01:56  Cloning the remote Git repository
01:01:56  Cloning repository git://devvexx.opendaylight.org/mirror/integration/test
01:01:56   > git init /w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium/test # timeout=10
01:01:56  Fetching upstream changes from git://devvexx.opendaylight.org/mirror/integration/test
01:01:56   > git --version # timeout=10
01:01:56   > git --version # 'git version 2.43.0'
01:01:56  using GIT_SSH to set credentials Release Engineering Jenkins Key
01:01:56  [INFO] Currently running in a labeled security context
01:01:56  [INFO] Currently SELinux is 'enforcing' on the host
01:01:56   > /usr/bin/chcon --type=ssh_home_t /w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium/test@tmp/jenkins-gitclient-ssh1170851672833974381.key
01:01:56  Verifying host key using known hosts file
01:01:56  You're using 'Known hosts file' strategy to verify ssh host keys, but your known_hosts file does not exist, please go to 'Manage Jenkins' -> 'Security' -> 'Git Host Key Verification Configuration' and configure host key verification.
01:01:56   > git fetch --tags --force --progress -- git://devvexx.opendaylight.org/mirror/integration/test +refs/heads/*:refs/remotes/origin/* # timeout=10
01:01:59   > git config remote.origin.url git://devvexx.opendaylight.org/mirror/integration/test # timeout=10
01:01:59   > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
01:01:59   > git config remote.origin.url git://devvexx.opendaylight.org/mirror/integration/test # timeout=10
01:01:59  Fetching upstream changes from git://devvexx.opendaylight.org/mirror/integration/test
01:01:59  using GIT_SSH to set credentials Release Engineering Jenkins Key
01:01:59  [INFO] Currently running in a labeled security context
01:01:59  [INFO] Currently SELinux is 'enforcing' on the host
01:01:59   > /usr/bin/chcon --type=ssh_home_t /w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium/test@tmp/jenkins-gitclient-ssh8922627609665520829.key
01:01:59  Verifying host key using known hosts file
01:01:59  You're using 'Known hosts file' strategy to verify ssh host keys, but your known_hosts file does not exist, please go to 'Manage Jenkins' -> 'Security' -> 'Git Host Key Verification Configuration' and configure host key verification.
01:01:59   > git fetch --tags --force --progress -- git://devvexx.opendaylight.org/mirror/integration/test master # timeout=10
01:01:59   > git rev-parse FETCH_HEAD^{commit} # timeout=10
01:01:59  Checking out Revision 1d2a1a30bcc48e3c8d571eb0d9b6891fe29dfed4 (origin/master)
01:01:59   > git config core.sparsecheckout # timeout=10
01:01:59   > git checkout -f 1d2a1a30bcc48e3c8d571eb0d9b6891fe29dfed4 # timeout=10
01:02:00  Commit message: "Add bgpcep variables for titanium"
01:02:00   > git rev-parse FETCH_HEAD^{commit} # timeout=10
01:02:00   > git rev-list --no-walk d42b2c24fe4fcc7fe3ae0d352311e768f07a15b7 # timeout=10
01:02:00  No emails were triggered.
01:02:00  provisioning config files...
01:02:00  copy managed file [npmrc] to file:/home/jenkins/.npmrc
01:02:00  copy managed file [pipconf] to file:/home/jenkins/.config/pip/pip.conf
01:02:00  copy managed file [clouds-yaml] to file:/home/jenkins/.config/openstack/clouds.yaml
01:02:00  [openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium] $ /bin/bash /tmp/jenkins12596629294701770678.sh
01:02:00  ---> python-tools-install.sh
01:02:00  Setup pyenv:
01:02:00    system
01:02:01  * 3.8.13 (set by /opt/pyenv/version)
01:02:01  * 3.9.13 (set by /opt/pyenv/version)
01:02:01  * 3.10.13 (set by /opt/pyenv/version)
01:02:01  * 3.11.7 (set by /opt/pyenv/version)
01:02:05  lf-activate-venv(): INFO: Creating python3 venv at /tmp/venv-pKtp
01:02:05  lf-activate-venv(): INFO: Save venv in file: /tmp/.os_lf_venv
01:02:08  lf-activate-venv(): INFO: Installing: lftools 
01:02:32  lf-activate-venv(): INFO: Adding /tmp/venv-pKtp/bin to PATH
01:02:32  Generating Requirements File
01:02:52  Python 3.11.7
01:02:52  pip 25.0.1 from /tmp/venv-pKtp/lib/python3.11/site-packages/pip (python 3.11)
01:02:53  appdirs==1.4.4
01:02:53  argcomplete==3.5.3
01:02:53  aspy.yaml==1.3.0
01:02:53  attrs==25.1.0
01:02:53  autopage==0.5.2
01:02:53  beautifulsoup4==4.13.3
01:02:53  boto3==1.36.26
01:02:53  botocore==1.36.26
01:02:53  bs4==0.0.2
01:02:53  cachetools==5.5.2
01:02:53  certifi==2025.1.31
01:02:53  cffi==1.17.1
01:02:53  cfgv==3.4.0
01:02:53  chardet==5.2.0
01:02:53  charset-normalizer==3.4.1
01:02:53  click==8.1.8
01:02:53  cliff==4.8.0
01:02:53  cmd2==2.5.11
01:02:53  cryptography==3.3.2
01:02:53  debtcollector==3.0.0
01:02:53  decorator==5.2.0
01:02:53  defusedxml==0.7.1
01:02:53  Deprecated==1.2.18
01:02:53  distlib==0.3.9
01:02:53  dnspython==2.7.0
01:02:53  docker==4.2.2
01:02:53  dogpile.cache==1.3.4
01:02:53  durationpy==0.9
01:02:53  email_validator==2.2.0
01:02:53  filelock==3.17.0
01:02:53  future==1.0.0
01:02:53  gitdb==4.0.12
01:02:53  GitPython==3.1.44
01:02:53  google-auth==2.38.0
01:02:53  httplib2==0.22.0
01:02:53  identify==2.6.8
01:02:53  idna==3.10
01:02:53  importlib-resources==1.5.0
01:02:53  iso8601==2.1.0
01:02:53  Jinja2==3.1.5
01:02:53  jmespath==1.0.1
01:02:53  jsonpatch==1.33
01:02:53  jsonpointer==3.0.0
01:02:53  jsonschema==4.23.0
01:02:53  jsonschema-specifications==2024.10.1
01:02:53  keystoneauth1==5.10.0
01:02:53  kubernetes==32.0.1
01:02:53  lftools==0.37.10
01:02:53  lxml==5.3.1
01:02:53  MarkupSafe==3.0.2
01:02:53  msgpack==1.1.0
01:02:53  multi_key_dict==2.0.3
01:02:53  munch==4.0.0
01:02:53  netaddr==1.3.0
01:02:53  niet==1.4.2
01:02:53  nodeenv==1.9.1
01:02:53  oauth2client==4.1.3
01:02:53  oauthlib==3.2.2
01:02:53  openstacksdk==4.3.0
01:02:53  os-client-config==2.1.0
01:02:53  os-service-types==1.7.0
01:02:53  osc-lib==3.2.0
01:02:53  oslo.config==9.7.1
01:02:53  oslo.context==5.7.1
01:02:53  oslo.i18n==6.5.1
01:02:53  oslo.log==7.1.0
01:02:53  oslo.serialization==5.7.0
01:02:53  oslo.utils==8.2.0
01:02:53  packaging==24.2
01:02:53  pbr==6.1.1
01:02:53  platformdirs==4.3.6
01:02:53  prettytable==3.14.0
01:02:53  psutil==7.0.0
01:02:53  pyasn1==0.6.1
01:02:53  pyasn1_modules==0.4.1
01:02:53  pycparser==2.22
01:02:53  pygerrit2==2.0.15
01:02:53  PyGithub==2.6.1
01:02:53  PyJWT==2.10.1
01:02:53  PyNaCl==1.5.0
01:02:53  pyparsing==2.4.7
01:02:53  pyperclip==1.9.0
01:02:53  pyrsistent==0.20.0
01:02:53  python-cinderclient==9.6.0
01:02:53  python-dateutil==2.9.0.post0
01:02:53  python-heatclient==4.1.0
01:02:53  python-jenkins==1.8.2
01:02:53  python-keystoneclient==5.5.0
01:02:53  python-magnumclient==4.7.0
01:02:53  python-openstackclient==7.3.1
01:02:53  python-swiftclient==4.6.0
01:02:53  PyYAML==6.0.2
01:02:53  referencing==0.36.2
01:02:53  requests==2.32.3
01:02:53  requests-oauthlib==2.0.0
01:02:53  requestsexceptions==1.4.0
01:02:53  rfc3986==2.0.0
01:02:53  rpds-py==0.23.1
01:02:53  rsa==4.9
01:02:53  ruamel.yaml==0.18.10
01:02:53  ruamel.yaml.clib==0.2.12
01:02:53  s3transfer==0.11.2
01:02:53  simplejson==3.20.1
01:02:53  six==1.17.0
01:02:53  smmap==5.0.2
01:02:53  soupsieve==2.6
01:02:53  stevedore==5.4.1
01:02:53  tabulate==0.9.0
01:02:53  toml==0.10.2
01:02:53  tomlkit==0.13.2
01:02:53  tqdm==4.67.1
01:02:53  typing_extensions==4.12.2
01:02:53  tzdata==2025.1
01:02:53  urllib3==1.26.20
01:02:53  virtualenv==20.29.2
01:02:53  wcwidth==0.2.13
01:02:53  websocket-client==1.8.0
01:02:53  wrapt==1.17.2
01:02:53  xdg==6.0.0
01:02:53  xmltodict==0.14.2
01:02:53  yq==3.4.3
01:02:53  [EnvInject] - Injecting environment variables from a build step.
01:02:53  [EnvInject] - Injecting as environment variables the properties content 
01:02:53  OS_STACK_TEMPLATE=csit-2-instance-type.yaml
01:02:53  OS_CLOUD=vex
01:02:53  OS_STACK_NAME=releng-openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium-196
01:02:53  OS_STACK_TEMPLATE_DIR=openstack-hot
01:02:53  
01:02:53  [EnvInject] - Variables injected successfully.
01:02:53  provisioning config files...
01:02:53  copy managed file [clouds-yaml] to file:/home/jenkins/.config/openstack/clouds.yaml
01:02:53  [openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium] $ /bin/bash /tmp/jenkins14352467968965232620.sh
01:02:53  ---> Create parameters file for OpenStack HOT
01:02:53  OpenStack Heat parameters generated
01:02:53  -----------------------------------
01:02:53  parameters:
01:02:53      vm_0_count: '3'
01:02:53      vm_0_flavor: 'v3-standard-4'
01:02:53      vm_0_image: 'ZZCI - Ubuntu 20.04 - builder - x86_64 - 20250101-160150.998'
01:02:53      vm_1_count: '1'
01:02:53      vm_1_flavor: 'v3-standard-2'
01:02:53      vm_1_image: 'ZZCI - Ubuntu 22.04 - mininet-ovs-217 - x86_64 - 20250201-060151.911'
01:02:53      
01:02:53      job_name: '53193-196'
01:02:53      silo: 'releng'
01:02:53  [openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium] $ /bin/bash -l /tmp/jenkins9254348349351357805.sh
01:02:53  ---> Create HEAT stack
01:02:53  + source /home/jenkins/lf-env.sh
01:02:53  + lf-activate-venv --python python3 'lftools[openstack]' kubernetes niet python-heatclient python-openstackclient python-magnumclient yq
01:02:53  ++ mktemp -d /tmp/venv-XXXX
01:02:53  + lf_venv=/tmp/venv-UMbS
01:02:53  + local venv_file=/tmp/.os_lf_venv
01:02:53  + local python=python3
01:02:53  + local options
01:02:53  + local set_path=true
01:02:53  + local install_args=
01:02:53  ++ getopt -o np:v: -l no-path,system-site-packages,python:,venv-file: -n lf-activate-venv -- --python python3 'lftools[openstack]' kubernetes niet python-heatclient python-openstackclient python-magnumclient yq
01:02:53  + options=' --python '\''python3'\'' -- '\''lftools[openstack]'\'' '\''kubernetes'\'' '\''niet'\'' '\''python-heatclient'\'' '\''python-openstackclient'\'' '\''python-magnumclient'\'' '\''yq'\'''
01:02:53  + eval set -- ' --python '\''python3'\'' -- '\''lftools[openstack]'\'' '\''kubernetes'\'' '\''niet'\'' '\''python-heatclient'\'' '\''python-openstackclient'\'' '\''python-magnumclient'\'' '\''yq'\'''
01:02:53  ++ set -- --python python3 -- 'lftools[openstack]' kubernetes niet python-heatclient python-openstackclient python-magnumclient yq
01:02:53  + true
01:02:53  + case $1 in
01:02:53  + python=python3
01:02:53  + shift 2
01:02:53  + true
01:02:53  + case $1 in
01:02:53  + shift
01:02:53  + break
01:02:53  + case $python in
01:02:53  + local pkg_list=
01:02:53  + [[ -d /opt/pyenv ]]
01:02:53  + echo 'Setup pyenv:'
01:02:53  Setup pyenv:
01:02:53  + export PYENV_ROOT=/opt/pyenv
01:02:53  + PYENV_ROOT=/opt/pyenv
01:02:53  + export PATH=/opt/pyenv/bin:/home/jenkins/.local/bin:/home/jenkins/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin
01:02:53  + PATH=/opt/pyenv/bin:/home/jenkins/.local/bin:/home/jenkins/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin
01:02:53  + pyenv versions
01:02:53    system
01:02:53    3.8.13
01:02:53    3.9.13
01:02:53    3.10.13
01:02:53  * 3.11.7 (set by /w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium/.python-version)
01:02:53  + command -v pyenv
01:02:53  ++ pyenv init - --no-rehash
01:02:54  + eval 'PATH="$(bash --norc -ec '\''IFS=:; paths=($PATH); 
01:02:54  for i in ${!paths[@]}; do 
01:02:54  if [[ ${paths[i]} == "'\'''\''/opt/pyenv/shims'\'''\''" ]]; then unset '\''\'\'''\''paths[i]'\''\'\'''\''; 
01:02:54  fi; done; 
01:02:54  echo "${paths[*]}"'\'')"
01:02:54  export PATH="/opt/pyenv/shims:${PATH}"
01:02:54  export PYENV_SHELL=bash
01:02:54  source '\''/opt/pyenv/libexec/../completions/pyenv.bash'\''
01:02:54  pyenv() {
01:02:54    local command
01:02:54    command="${1:-}"
01:02:54    if [ "$#" -gt 0 ]; then
01:02:54      shift
01:02:54    fi
01:02:54  
01:02:54    case "$command" in
01:02:54    rehash|shell)
01:02:54      eval "$(pyenv "sh-$command" "$@")"
01:02:54      ;;
01:02:54    *)
01:02:54      command pyenv "$command" "$@"
01:02:54      ;;
01:02:54    esac
01:02:54  }'
01:02:54  +++ bash --norc -ec 'IFS=:; paths=($PATH); 
01:02:54  for i in ${!paths[@]}; do 
01:02:54  if [[ ${paths[i]} == "/opt/pyenv/shims" ]]; then unset '\''paths[i]'\''; 
01:02:54  fi; done; 
01:02:54  echo "${paths[*]}"'
01:02:54  ++ PATH=/opt/pyenv/bin:/home/jenkins/.local/bin:/home/jenkins/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin
01:02:54  ++ export PATH=/opt/pyenv/shims:/opt/pyenv/bin:/home/jenkins/.local/bin:/home/jenkins/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin
01:02:54  ++ PATH=/opt/pyenv/shims:/opt/pyenv/bin:/home/jenkins/.local/bin:/home/jenkins/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin
01:02:54  ++ export PYENV_SHELL=bash
01:02:54  ++ PYENV_SHELL=bash
01:02:54  ++ source /opt/pyenv/libexec/../completions/pyenv.bash
01:02:54  +++ complete -F _pyenv pyenv
01:02:54  ++ lf-pyver python3
01:02:54  ++ local py_version_xy=python3
01:02:54  ++ local py_version_xyz=
01:02:54  ++ grep -E '^[0-9.]*[0-9]$'
01:02:54  ++ awk '{ print $1 }'
01:02:54  ++ sed 's/^[ *]* //'
01:02:54  ++ pyenv versions
01:02:54  ++ local command
01:02:54  ++ command=versions
01:02:54  ++ '[' 1 -gt 0 ']'
01:02:54  ++ shift
01:02:54  ++ case "$command" in
01:02:54  ++ command pyenv versions
01:02:54  ++ pyenv versions
01:02:54  ++ [[ ! -s /tmp/.pyenv_versions ]]
01:02:54  +++ grep '^3' /tmp/.pyenv_versions
01:02:54  +++ sort -V
01:02:54  +++ tail -n 1
01:02:54  ++ py_version_xyz=3.11.7
01:02:54  ++ [[ -z 3.11.7 ]]
01:02:54  ++ echo 3.11.7
01:02:54  ++ return 0
01:02:54  + pyenv local 3.11.7
01:02:54  + local command
01:02:54  + command=local
01:02:54  + '[' 2 -gt 0 ']'
01:02:54  + shift
01:02:54  + case "$command" in
01:02:54  + command pyenv local 3.11.7
01:02:54  + pyenv local 3.11.7
01:02:54  + for arg in "$@"
01:02:54  + case $arg in
01:02:54  + pkg_list+='lftools[openstack] '
01:02:54  + for arg in "$@"
01:02:54  + case $arg in
01:02:54  + pkg_list+='kubernetes '
01:02:54  + for arg in "$@"
01:02:54  + case $arg in
01:02:54  + pkg_list+='niet '
01:02:54  + for arg in "$@"
01:02:54  + case $arg in
01:02:54  + pkg_list+='python-heatclient '
01:02:54  + for arg in "$@"
01:02:54  + case $arg in
01:02:54  + pkg_list+='python-openstackclient '
01:02:54  + for arg in "$@"
01:02:54  + case $arg in
01:02:54  + pkg_list+='python-magnumclient '
01:02:54  + for arg in "$@"
01:02:54  + case $arg in
01:02:54  + pkg_list+='yq '
01:02:54  + [[ -f /tmp/.os_lf_venv ]]
01:02:54  ++ cat /tmp/.os_lf_venv
01:02:54  + lf_venv=/tmp/venv-pKtp
01:02:54  + echo 'lf-activate-venv(): INFO: Reuse venv:/tmp/venv-pKtp from' file:/tmp/.os_lf_venv
01:02:54  lf-activate-venv(): INFO: Reuse venv:/tmp/venv-pKtp from file:/tmp/.os_lf_venv
01:02:54  + /tmp/venv-pKtp/bin/python3 -m pip install --upgrade --quiet pip virtualenv
01:02:55  + [[ -z lftools[openstack] kubernetes niet python-heatclient python-openstackclient python-magnumclient yq  ]]
01:02:55  + echo 'lf-activate-venv(): INFO: Installing: lftools[openstack] kubernetes niet python-heatclient python-openstackclient python-magnumclient yq '
01:02:55  lf-activate-venv(): INFO: Installing: lftools[openstack] kubernetes niet python-heatclient python-openstackclient python-magnumclient yq 
01:02:55  + /tmp/venv-pKtp/bin/python3 -m pip install --upgrade --quiet --upgrade-strategy eager 'lftools[openstack]' kubernetes niet python-heatclient python-openstackclient python-magnumclient yq
01:03:09  + type python3
01:03:09  + true
01:03:09  + echo 'lf-activate-venv(): INFO: Adding /tmp/venv-pKtp/bin to PATH'
01:03:09  lf-activate-venv(): INFO: Adding /tmp/venv-pKtp/bin to PATH
01:03:09  + PATH=/tmp/venv-pKtp/bin:/opt/pyenv/shims:/opt/pyenv/bin:/home/jenkins/.local/bin:/home/jenkins/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin
01:03:09  + return 0
01:03:09  + openstack --os-cloud vex limits show --absolute
01:03:10  +--------------------------+---------+
01:03:10  | Name                     |   Value |
01:03:10  +--------------------------+---------+
01:03:10  | maxTotalInstances        |      -1 |
01:03:10  | maxTotalCores            |     450 |
01:03:10  | maxTotalRAMSize          | 1000000 |
01:03:10  | maxServerMeta            |     128 |
01:03:10  | maxImageMeta             |     128 |
01:03:10  | maxPersonality           |       5 |
01:03:10  | maxPersonalitySize       |   10240 |
01:03:10  | maxTotalKeypairs         |     100 |
01:03:10  | maxServerGroups          |      10 |
01:03:10  | maxServerGroupMembers    |      10 |
01:03:10  | maxTotalFloatingIps      |      -1 |
01:03:10  | maxSecurityGroups        |      -1 |
01:03:10  | maxSecurityGroupRules    |      -1 |
01:03:10  | totalRAMUsed             |  843776 |
01:03:10  | totalCoresUsed           |     206 |
01:03:10  | totalInstancesUsed       |      67 |
01:03:10  | totalFloatingIpsUsed     |       0 |
01:03:10  | totalSecurityGroupsUsed  |       0 |
01:03:10  | totalServerGroupsUsed    |       0 |
01:03:10  | maxTotalVolumes          |      -1 |
01:03:10  | maxTotalSnapshots        |      10 |
01:03:10  | maxTotalVolumeGigabytes  |    4096 |
01:03:10  | maxTotalBackups          |      10 |
01:03:10  | maxTotalBackupGigabytes  |    1000 |
01:03:10  | totalVolumesUsed         |       3 |
01:03:10  | totalGigabytesUsed       |      60 |
01:03:10  | totalSnapshotsUsed       |       0 |
01:03:10  | totalBackupsUsed         |       0 |
01:03:10  | totalBackupGigabytesUsed |       0 |
01:03:10  +--------------------------+---------+
01:03:10  + pushd /opt/ciman/openstack-hot
01:03:10  /opt/ciman/openstack-hot /w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium
01:03:10  + lftools openstack --os-cloud vex stack create releng-openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium-196 csit-2-instance-type.yaml /w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium/stack-parameters.yaml
01:03:24  Creating stack releng-openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium-196
01:03:24  WARN: Failed to initialize stack. Reason: Resource CREATE failed: Forbidden: resources.vm_1_group.resources[0].resources.instance: Quota exceeded for ram: Requested 8192, but already used 1007616 of 1000000 ram (HTTP 403) (Request-ID: req-a63aabd7-fe7f-4dea-a987-93074ad4d230)
01:03:24  TypeError: delete() missing 1 required positional argument: 'force'
01:03:24  Build step 'Execute shell' marked build as failure
01:03:25  $ ssh-agent -k
01:03:25  unset SSH_AUTH_SOCK;
01:03:25  unset SSH_AGENT_PID;
01:03:25  echo Agent pid 5282 killed;
01:03:25  [ssh-agent] Stopped.
01:03:25  Recording plot data
01:03:25  Robot results publisher started...
01:03:25  INFO: Checking test criticality is deprecated and will be dropped in a future release!
01:03:25  -Parsing output xml:
01:03:25  Failed!
01:03:25  hudson.AbortException: No files found in path /w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium with configured filemask: output.xml
01:03:25  	at hudson.plugins.robot.RobotParser$RobotParserCallable.invoke(RobotParser.java:81)
01:03:25  	at hudson.plugins.robot.RobotParser$RobotParserCallable.invoke(RobotParser.java:52)
01:03:25  	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3616)
01:03:25  	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
01:03:25  	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
01:03:25  	at hudson.remoting.Request$2.run(Request.java:377)
01:03:25  	at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
01:03:25  	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
01:03:25  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
01:03:25  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
01:03:25  	at java.base/java.lang.Thread.run(Thread.java:829)
01:03:25  	Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to prd-centos8-robot-2c-8g-1856
01:03:25  		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1787)
01:03:25  		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
01:03:25  		at hudson.remoting.Channel.call(Channel.java:1003)
01:03:25  		at hudson.FilePath.act(FilePath.java:1230)
01:03:25  		at hudson.FilePath.act(FilePath.java:1219)
01:03:25  		at hudson.plugins.robot.RobotParser.parse(RobotParser.java:48)
01:03:25  		at hudson.plugins.robot.RobotPublisher.parse(RobotPublisher.java:246)
01:03:25  		at hudson.plugins.robot.RobotPublisher.perform(RobotPublisher.java:268)
01:03:25  		at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:80)
01:03:25  		at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
01:03:25  		at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:818)
01:03:25  		at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:767)
01:03:25  		at hudson.model.Build$BuildExecution.post2(Build.java:179)
01:03:25  		at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:711)
01:03:25  		at hudson.model.Run.execute(Run.java:1918)
01:03:25  		at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
01:03:25  		at hudson.model.ResourceController.execute(ResourceController.java:101)
01:03:25  		at hudson.model.Executor.run(Executor.java:442)
01:03:25  [PostBuildScript] - [INFO] Executing post build scripts.
01:03:25  [openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium] $ /bin/bash /tmp/jenkins15500304892413837207.sh
01:03:25  Archiving csit artifacts
01:03:25  seq: invalid floating point argument: ‘’
01:03:25  Try 'seq --help' for more information.
01:03:25    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
01:03:25                                   Dload  Upload   Total   Spent    Left  Speed
01:03:25  
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   237    0   237    0     0  10304      0 --:--:-- --:--:-- --:--:-- 10304
01:03:25  Archive:  robot-plugin.zip
01:03:25    End-of-central-directory signature not found.  Either this file is not
01:03:25    a zipfile, or it constitutes one disk of a multi-part archive.  In the
01:03:25    latter case the central directory and zipfile comment will be found on
01:03:25    the last disk(s) of this archive.
01:03:25  unzip:  cannot find zipfile directory in one of robot-plugin.zip or
01:03:25          robot-plugin.zip.zip, and cannot find robot-plugin.zip.ZIP, period.
01:03:25  mv: cannot stat '*.log': No such file or directory
01:03:25  mv: cannot stat '*.log.gz': No such file or directory
01:03:25  mv: cannot stat '*.csv': No such file or directory
01:03:25  mv: cannot stat '*.png': No such file or directory
01:03:25  [PostBuildScript] - [INFO] Executing post build scripts.
01:03:25  [openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium] $ /bin/bash /tmp/jenkins5646008244526708730.sh
01:03:25  [PostBuildScript] - [INFO] Executing post build scripts.
01:03:25  [EnvInject] - Injecting environment variables from a build step.
01:03:25  [EnvInject] - Injecting as environment variables the properties content 
01:03:25  OS_CLOUD=vex
01:03:25  OS_STACK_NAME=releng-openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium-196
01:03:25  
01:03:25  [EnvInject] - Variables injected successfully.
01:03:25  provisioning config files...
01:03:26  copy managed file [clouds-yaml] to file:/home/jenkins/.config/openstack/clouds.yaml
01:03:26  [openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium] $ /bin/bash /tmp/jenkins10353342623405189618.sh
01:03:26  ---> openstack-stack-delete.sh
01:03:26  Setup pyenv:
01:03:26    system
01:03:26    3.8.13
01:03:26    3.9.13
01:03:26    3.10.13
01:03:26  * 3.11.7 (set by /w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium/.python-version)
01:03:26  lf-activate-venv(): INFO: Reuse venv:/tmp/venv-pKtp from file:/tmp/.os_lf_venv
01:03:27  lf-activate-venv(): INFO: Installing: lftools[openstack] kubernetes python-heatclient python-openstackclient 
01:03:38  lf-activate-venv(): INFO: Adding /tmp/venv-pKtp/bin to PATH
01:03:38  INFO: Retrieving stack cost for: releng-openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium-196
01:03:41  openstack.exceptions.BadRequestException: BadRequestException: 400: Client Error for url: https://compute.public.mtl1.vexxhost.net/v2.1/12c36e260d8e4bb2913965203b1b491f/servers/detail?name=, Invalid input for query parameters name. Value: . '' is not a 'regex'
01:03:41  WARNING: Unable to get stack costs, continuing anyway
01:03:53  INFO: Deleting stack releng-openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium-196
01:03:53  Successfully deleted stack releng-openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium-196
01:03:53  [PostBuildScript] - [INFO] Executing post build scripts.
01:03:53  [openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium] $ /bin/bash /tmp/jenkins7710337483620344263.sh
01:03:53  ---> sysstat.sh
01:03:54  [openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium] $ /bin/bash /tmp/jenkins990394056710242442.sh
01:03:54  ---> package-listing.sh
01:03:54  ++ facter osfamily
01:03:54  ++ tr '[:upper:]' '[:lower:]'
01:03:54  + OS_FAMILY=redhat
01:03:54  + workspace=/w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium
01:03:54  + START_PACKAGES=/tmp/packages_start.txt
01:03:54  + END_PACKAGES=/tmp/packages_end.txt
01:03:54  + DIFF_PACKAGES=/tmp/packages_diff.txt
01:03:54  + PACKAGES=/tmp/packages_start.txt
01:03:54  + '[' /w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium ']'
01:03:54  + PACKAGES=/tmp/packages_end.txt
01:03:54  + case "${OS_FAMILY}" in
01:03:54  + rpm -qa
01:03:54  + sort
01:03:54  + '[' -f /tmp/packages_start.txt ']'
01:03:54  + '[' -f /tmp/packages_end.txt ']'
01:03:54  + diff /tmp/packages_start.txt /tmp/packages_end.txt
01:03:54  + '[' /w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium ']'
01:03:54  + mkdir -p /w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium/archives/
01:03:54  + cp -f /tmp/packages_diff.txt /tmp/packages_end.txt /tmp/packages_start.txt /w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium/archives/
01:03:54  [openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium] $ /bin/bash /tmp/jenkins5770642173009138655.sh
01:03:54  ---> capture-instance-metadata.sh
01:03:54  Setup pyenv:
01:03:55    system
01:03:55    3.8.13
01:03:55    3.9.13
01:03:55    3.10.13
01:03:55  * 3.11.7 (set by /w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium/.python-version)
01:03:55  lf-activate-venv(): INFO: Reuse venv:/tmp/venv-pKtp from file:/tmp/.os_lf_venv
01:03:56  lf-activate-venv(): INFO: Installing: lftools 
01:04:05  lf-activate-venv(): INFO: Adding /tmp/venv-pKtp/bin to PATH
01:04:05  INFO: Running in OpenStack, capturing instance metadata
01:04:05  [openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium] $ /bin/bash /tmp/jenkins3855575248275789234.sh
01:04:05  provisioning config files...
01:04:06  Could not find credentials [logs] for openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium #196
01:04:06  copy managed file [jenkins-log-archives-settings] to file:/w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium@tmp/config14325134459583030400tmp
01:04:06  Regular expression run condition: Expression=[^.*logs-s3.*], Label=[odl-logs-s3-cloudfront-index]
01:04:06  Run condition [Regular expression match] enabling perform for step [Provide Configuration files]
01:04:06  provisioning config files...
01:04:06  copy managed file [jenkins-s3-log-ship] to file:/home/jenkins/.aws/credentials
01:04:06  [EnvInject] - Injecting environment variables from a build step.
01:04:06  [EnvInject] - Injecting as environment variables the properties content 
01:04:06  SERVER_ID=logs
01:04:06  
01:04:06  [EnvInject] - Variables injected successfully.
01:04:06  [openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium] $ /bin/bash /tmp/jenkins9136884205755824330.sh
01:04:06  ---> create-netrc.sh
01:04:06  WARN: Log server credential not found.
01:04:06  [openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium] $ /bin/bash /tmp/jenkins13277514864187108701.sh
01:04:06  ---> python-tools-install.sh
01:04:06  Setup pyenv:
01:04:06    system
01:04:06    3.8.13
01:04:06    3.9.13
01:04:06    3.10.13
01:04:06  * 3.11.7 (set by /w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium/.python-version)
01:04:07  lf-activate-venv(): INFO: Reuse venv:/tmp/venv-pKtp from file:/tmp/.os_lf_venv
01:04:08  lf-activate-venv(): INFO: Installing: lftools 
01:04:15  lf-activate-venv(): INFO: Adding /tmp/venv-pKtp/bin to PATH
01:04:15  [openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium] $ /bin/bash /tmp/jenkins2142658702673794129.sh
01:04:15  ---> sudo-logs.sh
01:04:15  Archiving 'sudo' log..
01:04:15  [openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium] $ /bin/bash /tmp/jenkins2979417372817164514.sh
01:04:15  ---> job-cost.sh
01:04:15  Setup pyenv:
01:04:15    system
01:04:15    3.8.13
01:04:15    3.9.13
01:04:15    3.10.13
01:04:15  * 3.11.7 (set by /w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium/.python-version)
01:04:16  lf-activate-venv(): INFO: Reuse venv:/tmp/venv-pKtp from file:/tmp/.os_lf_venv
01:04:16  lf-activate-venv(): INFO: Installing: zipp==1.1.0 python-openstackclient urllib3~=1.26.15 
01:04:22  lf-activate-venv(): INFO: Adding /tmp/venv-pKtp/bin to PATH
01:04:22  DEBUG: total: 0
01:04:22  INFO: Retrieving Stack Cost...
01:04:22  INFO: Retrieving Pricing Info for: v3-standard-2
01:04:22  INFO: Archiving Costs
01:04:22  [openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium] $ /bin/bash -l /tmp/jenkins5074789354465152510.sh
01:04:22  ---> logs-deploy.sh
01:04:22  Setup pyenv:
01:04:22    system
01:04:22    3.8.13
01:04:22    3.9.13
01:04:22    3.10.13
01:04:22  * 3.11.7 (set by /w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium/.python-version)
01:04:23  lf-activate-venv(): INFO: Reuse venv:/tmp/venv-pKtp from file:/tmp/.os_lf_venv
01:04:24  lf-activate-venv(): INFO: Installing: lftools 
01:04:31  lf-activate-venv(): INFO: Adding /tmp/venv-pKtp/bin to PATH
01:04:31  WARNING: Nexus logging server not set
01:04:31  INFO: S3 path logs/releng/vex-yul-odl-jenkins-1/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-scandium/196/
01:04:31  INFO: archiving logs to S3
01:04:32  ---> uname -a:
01:04:32  Linux prd-centos8-robot-2c-8g-1856.novalocal 4.18.0-553.5.1.el8.x86_64 #1 SMP Tue May 21 05:46:01 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
01:04:32  
01:04:32  
01:04:32  ---> lscpu:
01:04:32  Architecture:        x86_64
01:04:32  CPU op-mode(s):      32-bit, 64-bit
01:04:32  Byte Order:          Little Endian
01:04:32  CPU(s):              2
01:04:32  On-line CPU(s) list: 0,1
01:04:32  Thread(s) per core:  1
01:04:32  Core(s) per socket:  1
01:04:32  Socket(s):           2
01:04:32  NUMA node(s):        1
01:04:32  Vendor ID:           AuthenticAMD
01:04:32  CPU family:          23
01:04:32  Model:               49
01:04:32  Model name:          AMD EPYC-Rome Processor
01:04:32  Stepping:            0
01:04:32  CPU MHz:             2800.000
01:04:32  BogoMIPS:            5600.00
01:04:32  Virtualization:      AMD-V
01:04:32  Hypervisor vendor:   KVM
01:04:32  Virtualization type: full
01:04:32  L1d cache:           32K
01:04:32  L1i cache:           32K
01:04:32  L2 cache:            512K
01:04:32  L3 cache:            16384K
01:04:32  NUMA node0 CPU(s):   0,1
01:04:32  Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd ibrs ibpb stibp vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr wbnoinvd arat npt nrip_save umip rdpid arch_capabilities
01:04:32  
01:04:32  
01:04:32  ---> nproc:
01:04:32  2
01:04:32  
01:04:32  
01:04:32  ---> df -h:
01:04:32  Filesystem      Size  Used Avail Use% Mounted on
01:04:32  devtmpfs        3.8G     0  3.8G   0% /dev
01:04:32  tmpfs           3.8G     0  3.8G   0% /dev/shm
01:04:32  tmpfs           3.8G   17M  3.8G   1% /run
01:04:32  tmpfs           3.8G     0  3.8G   0% /sys/fs/cgroup
01:04:32  /dev/vda1        40G  8.3G   32G  21% /
01:04:32  tmpfs           770M     0  770M   0% /run/user/1001
01:04:32  
01:04:32  
01:04:32  ---> free -m:
01:04:32                total        used        free      shared  buff/cache   available
01:04:32  Mem:           7697         632        4995          19        2069        6769
01:04:32  Swap:          1023           0        1023
01:04:32  
01:04:32  
01:04:32  ---> ip addr:
01:04:32  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
01:04:32      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
01:04:32      inet 127.0.0.1/8 scope host lo
01:04:32         valid_lft forever preferred_lft forever
01:04:32      inet6 ::1/128 scope host 
01:04:32         valid_lft forever preferred_lft forever
01:04:32  2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1458 qdisc mq state UP group default qlen 1000
01:04:32      link/ether fa:16:3e:f2:9b:99 brd ff:ff:ff:ff:ff:ff
01:04:32      altname enp0s3
01:04:32      altname ens3
01:04:32      inet 10.30.170.36/23 brd 10.30.171.255 scope global dynamic noprefixroute eth0
01:04:32         valid_lft 86200sec preferred_lft 86200sec
01:04:32      inet6 fe80::f816:3eff:fef2:9b99/64 scope link 
01:04:32         valid_lft forever preferred_lft forever
01:04:32  
01:04:32  
01:04:33  ---> sar -b -r -n DEV:
01:04:33  Linux 4.18.0-553.5.1.el8.x86_64 (centos-stream-8-robot-7d7a37eb-bc14-4dd6-9530-dc22c5eae738.noval) 	02/24/2025 	_x86_64_	(2 CPU)
01:04:33  
01:04:33  01:01:09     LINUX RESTART	(2 CPU)
01:04:33  
01:04:33  01:02:01 AM       tps      rtps      wtps   bread/s   bwrtn/s
01:04:33  01:03:01 AM     99.55      3.22     96.33    381.47   8381.32
01:04:33  01:04:01 AM     70.80      0.95     69.86     95.85   8019.15
01:04:33  Average:        85.18      2.08     83.10    238.67   8200.25
01:04:33  
01:04:33  01:02:01 AM kbmemfree   kbavail kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty
01:04:33  01:03:01 AM   5164140   7000000   2718292     34.49      2688   2013836    766772      8.59    189904   2192960    232132
01:04:33  01:04:01 AM   5166056   6975852   2716376     34.46      2688   1987152    772452      8.65    381440   1991084       144
01:04:33  Average:      5165098   6987926   2717334     34.47      2688   2000494    769612      8.62    285672   2092022    116138
01:04:33  
01:04:33  01:02:01 AM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s   %ifutil
01:04:33  01:03:01 AM        lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:04:33  01:03:01 AM      eth0    109.28     88.69   1438.32     19.56      0.00      0.00      0.00      0.00
01:04:33  01:04:01 AM        lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:04:33  01:04:01 AM      eth0     24.51     17.61     19.83      8.68      0.00      0.00      0.00      0.00
01:04:33  Average:           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
01:04:33  Average:         eth0     66.90     53.15    729.13     14.12      0.00      0.00      0.00      0.00
01:04:33  
01:04:33  
01:04:33  ---> sar -P ALL:
01:04:33  Linux 4.18.0-553.5.1.el8.x86_64 (centos-stream-8-robot-7d7a37eb-bc14-4dd6-9530-dc22c5eae738.noval) 	02/24/2025 	_x86_64_	(2 CPU)
01:04:33  
01:04:33  01:01:09     LINUX RESTART	(2 CPU)
01:04:33  
01:04:33  01:02:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
01:04:33  01:03:01 AM     all     39.02      0.00      5.55      1.57      0.11     53.74
01:04:33  01:03:01 AM       0     49.66      0.00      6.38      1.50      0.12     42.35
01:04:33  01:03:01 AM       1     28.39      0.00      4.73      1.65      0.10     65.13
01:04:33  01:04:01 AM     all     21.63      0.00      2.76      1.16      0.08     74.37
01:04:33  01:04:01 AM       0     24.14      0.00      2.54      1.24      0.07     72.02
01:04:33  01:04:01 AM       1     19.13      0.00      2.99      1.08      0.08     76.72
01:04:33  Average:        all     30.34      0.00      4.16      1.37      0.09     64.04
01:04:33  Average:          0     36.92      0.00      4.46      1.37      0.09     57.15
01:04:33  Average:          1     23.76      0.00      3.86      1.37      0.09     70.92
01:04:33  
01:04:33  
01:04:33