07:32:19  Started by timer
07:32:19  Running as SYSTEM
07:32:19  [EnvInject] - Loading node environment variables.
07:32:19  Building remotely on prd-centos8-builder-2c-2g-881 (centos8-builder-2c-2g) in workspace /w/workspace/builder-openstack-cron
07:32:19  [ssh-agent] Looking for ssh-agent implementation...
07:32:19  [ssh-agent]   Exec ssh-agent (binary ssh-agent on a remote machine)
07:32:19  $ ssh-agent
07:32:19  SSH_AUTH_SOCK=/tmp/ssh-CUPtQb3PmcHx/agent.5451
07:32:19  SSH_AGENT_PID=5452
07:32:19  [ssh-agent] Started.
07:32:19  Running ssh-add (command line suppressed)
07:32:19  Identity added: /w/workspace/builder-openstack-cron@tmp/private_key_7528040721462048403.key (/w/workspace/builder-openstack-cron@tmp/private_key_7528040721462048403.key)
07:32:19  [ssh-agent] Using credentials jenkins (jenkins-ssh)
07:32:19  The recommended git tool is: NONE
07:32:22  using credential jenkins-ssh
07:32:22  Wiping out workspace first.
07:32:22  Cloning the remote Git repository
07:32:22  Cloning repository git://devvexx.opendaylight.org/mirror/releng/builder
07:32:22   > git init /w/workspace/builder-openstack-cron # timeout=10
07:32:22  Fetching upstream changes from git://devvexx.opendaylight.org/mirror/releng/builder
07:32:22   > git --version # timeout=10
07:32:22   > git --version # 'git version 2.43.0'
07:32:22  using GIT_SSH to set credentials jenkins-ssh
07:32:22  [INFO] Currently running in a labeled security context
07:32:22  [INFO] Currently SELinux is 'enforcing' on the host
07:32:22   > /usr/bin/chcon --type=ssh_home_t /w/workspace/builder-openstack-cron@tmp/jenkins-gitclient-ssh3093040156911355231.key
07:32:22  Verifying host key using known hosts file
07:32:22  You're using 'Known hosts file' strategy to verify ssh host keys, but your known_hosts file does not exist, please go to 'Manage Jenkins' -> 'Security' -> 'Git Host Key Verification Configuration' and configure host key verification.
07:32:22   > git fetch --tags --force --progress -- git://devvexx.opendaylight.org/mirror/releng/builder +refs/heads/*:refs/remotes/origin/* # timeout=10
07:32:24   > git config remote.origin.url git://devvexx.opendaylight.org/mirror/releng/builder # timeout=10
07:32:24   > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
07:32:25  Avoid second fetch
07:32:25   > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
07:32:25  Checking out Revision ad4454e7152e08644cc555c24c6d93062688ea60 (refs/remotes/origin/master)
07:32:25   > git config core.sparsecheckout # timeout=10
07:32:25   > git checkout -f ad4454e7152e08644cc555c24c6d93062688ea60 # timeout=10
07:32:25  Commit message: "Chore: Add Ubuntu 2004 1C-4G node label"
07:32:25   > git rev-list --no-walk ad4454e7152e08644cc555c24c6d93062688ea60 # timeout=10
07:32:25   > git remote # timeout=10
07:32:25   > git submodule init # timeout=10
07:32:25   > git submodule sync # timeout=10
07:32:25   > git config --get remote.origin.url # timeout=10
07:32:25   > git submodule init # timeout=10
07:32:25   > git config -f .gitmodules --get-regexp ^submodule\.(.+)\.url # timeout=10
07:32:25   > git config --get submodule.global-jjb.url # timeout=10
07:32:25   > git config -f .gitmodules --get submodule.global-jjb.path # timeout=10
07:32:25   > git config --get submodule.packer/common-packer.url # timeout=10
07:32:25   > git config -f .gitmodules --get submodule.packer/common-packer.path # timeout=10
07:32:26   > git submodule update --init --recursive global-jjb # timeout=10
07:32:26   > git submodule update --init --recursive packer/common-packer # timeout=10
07:32:30  Run condition [Boolean condition] enabling prebuild for step [Execute shell]
07:32:30  Run condition [Boolean condition] enabling prebuild for step [Execute shell]
07:32:30  Run condition [Boolean condition] enabling prebuild for step [Execute shell]
07:32:30  Run condition [Boolean condition] enabling prebuild for step [Execute shell]
07:32:30  Run condition [Boolean condition] enabling prebuild for step [Execute shell]
07:32:30  Run condition [Boolean condition] enabling prebuild for step [Execute shell]
07:32:30  Run condition [Boolean condition] enabling prebuild for step [BuilderChain]
07:32:30  No emails were triggered.
07:32:30  provisioning config files...
07:32:30  copy managed file [clouds-yaml] to file:/home/jenkins/.config/openstack/clouds.yaml
07:32:30  copy managed file [npmrc] to file:/home/jenkins/.npmrc
07:32:30  copy managed file [pipconf] to file:/home/jenkins/.config/pip/pip.conf
07:32:30  [builder-openstack-cron] $ /bin/bash /tmp/jenkins9556821005135628235.sh
07:32:30  ---> python-tools-install.sh
07:32:30  Setup pyenv:
07:32:31    system
07:32:31  * 3.8.13 (set by /opt/pyenv/version)
07:32:31  * 3.9.13 (set by /opt/pyenv/version)
07:32:31  * 3.10.13 (set by /opt/pyenv/version)
07:32:31  * 3.11.7 (set by /opt/pyenv/version)
07:32:36  lf-activate-venv(): INFO: Creating python3 venv at /tmp/venv-LUjT
07:32:36  lf-activate-venv(): INFO: Save venv in file: /tmp/.os_lf_venv
07:32:39  lf-activate-venv(): INFO: Installing: lftools 
07:33:11  lf-activate-venv(): INFO: Adding /tmp/venv-LUjT/bin to PATH
07:33:11  Generating Requirements File
07:33:34  Python 3.11.7
07:33:34  pip 24.1.2 from /tmp/venv-LUjT/lib/python3.11/site-packages/pip (python 3.11)
07:33:34  appdirs==1.4.4
07:33:34  argcomplete==3.4.0
07:33:34  aspy.yaml==1.3.0
07:33:34  attrs==23.2.0
07:33:34  autopage==0.5.2
07:33:34  beautifulsoup4==4.12.3
07:33:34  boto3==1.34.145
07:33:34  botocore==1.34.145
07:33:34  bs4==0.0.2
07:33:34  cachetools==5.4.0
07:33:34  certifi==2024.7.4
07:33:34  cffi==1.16.0
07:33:34  cfgv==3.4.0
07:33:34  chardet==5.2.0
07:33:34  charset-normalizer==3.3.2
07:33:34  click==8.1.7
07:33:34  cliff==4.7.0
07:33:34  cmd2==2.4.3
07:33:34  cryptography==3.3.2
07:33:34  debtcollector==3.0.0
07:33:34  decorator==5.1.1
07:33:34  defusedxml==0.7.1
07:33:34  Deprecated==1.2.14
07:33:34  distlib==0.3.8
07:33:34  dnspython==2.6.1
07:33:34  docker==4.2.2
07:33:34  dogpile.cache==1.3.3
07:33:34  email_validator==2.2.0
07:33:34  filelock==3.15.4
07:33:34  future==1.0.0
07:33:34  gitdb==4.0.11
07:33:34  GitPython==3.1.43
07:33:34  google-auth==2.32.0
07:33:34  httplib2==0.22.0
07:33:34  identify==2.6.0
07:33:34  idna==3.7
07:33:34  importlib-resources==1.5.0
07:33:34  iso8601==2.1.0
07:33:34  Jinja2==3.1.4
07:33:34  jmespath==1.0.1
07:33:34  jsonpatch==1.33
07:33:34  jsonpointer==3.0.0
07:33:34  jsonschema==4.23.0
07:33:34  jsonschema-specifications==2023.12.1
07:33:34  keystoneauth1==5.7.0
07:33:34  kubernetes==30.1.0
07:33:34  lftools==0.37.10
07:33:34  lxml==5.2.2
07:33:34  MarkupSafe==2.1.5
07:33:34  msgpack==1.0.8
07:33:34  multi_key_dict==2.0.3
07:33:34  munch==4.0.0
07:33:34  netaddr==1.3.0
07:33:34  netifaces==0.11.0
07:33:34  niet==1.4.2
07:33:34  nodeenv==1.9.1
07:33:34  oauth2client==4.1.3
07:33:34  oauthlib==3.2.2
07:33:34  openstacksdk==3.3.0
07:33:34  os-client-config==2.1.0
07:33:34  os-service-types==1.7.0
07:33:34  osc-lib==3.1.0
07:33:34  oslo.config==9.5.0
07:33:34  oslo.context==5.5.0
07:33:34  oslo.i18n==6.3.0
07:33:34  oslo.log==6.1.1
07:33:34  oslo.serialization==5.4.0
07:33:34  oslo.utils==7.2.0
07:33:34  packaging==24.1
07:33:34  pbr==6.0.0
07:33:34  platformdirs==4.2.2
07:33:34  prettytable==3.10.2
07:33:34  pyasn1==0.6.0
07:33:34  pyasn1_modules==0.4.0
07:33:34  pycparser==2.22
07:33:34  pygerrit2==2.0.15
07:33:34  PyGithub==2.3.0
07:33:34  PyJWT==2.8.0
07:33:34  PyNaCl==1.5.0
07:33:34  pyparsing==2.4.7
07:33:34  pyperclip==1.9.0
07:33:34  pyrsistent==0.20.0
07:33:34  python-cinderclient==9.5.0
07:33:34  python-dateutil==2.9.0.post0
07:33:34  python-heatclient==3.5.0
07:33:34  python-jenkins==1.8.2
07:33:34  python-keystoneclient==5.4.0
07:33:34  python-magnumclient==4.6.0
07:33:34  python-novaclient==18.6.0
07:33:34  python-openstackclient==6.6.0
07:33:34  python-swiftclient==4.6.0
07:33:34  PyYAML==6.0.1
07:33:34  referencing==0.35.1
07:33:34  requests==2.32.3
07:33:34  requests-oauthlib==2.0.0
07:33:34  requestsexceptions==1.4.0
07:33:34  rfc3986==2.0.0
07:33:34  rpds-py==0.19.0
07:33:34  rsa==4.9
07:33:34  ruamel.yaml==0.18.6
07:33:34  ruamel.yaml.clib==0.2.8
07:33:34  s3transfer==0.10.2
07:33:34  simplejson==3.19.2
07:33:34  six==1.16.0
07:33:34  smmap==5.0.1
07:33:34  soupsieve==2.5
07:33:34  stevedore==5.2.0
07:33:34  tabulate==0.9.0
07:33:34  toml==0.10.2
07:33:34  tomlkit==0.13.0
07:33:34  tqdm==4.66.4
07:33:34  typing_extensions==4.12.2
07:33:34  tzdata==2024.1
07:33:34  urllib3==1.26.19
07:33:34  virtualenv==20.26.3
07:33:34  wcwidth==0.2.13
07:33:34  websocket-client==1.8.0
07:33:34  wrapt==1.16.0
07:33:34  xdg==6.0.0
07:33:34  xmltodict==0.13.0
07:33:34  yq==3.4.3
07:33:34  [EnvInject] - Injecting environment variables from a build step.
07:33:34  [EnvInject] - Injecting as environment variables the properties content 
07:33:34  OS_CLOUD=vex
07:33:34  
07:33:34  [EnvInject] - Variables injected successfully.
07:33:34  [Boolean condition] checking [True] against [^(1|y|yes|t|true|on|run)$] (origin token: True)
07:33:34  Run condition [Boolean condition] enabling perform for step [Execute shell]
07:33:34  [builder-openstack-cron] $ /bin/bash -l /tmp/jenkins17469323966883480157.sh
07:33:35  ---> Orphaned k8s clusters
07:33:35  Setup pyenv:
07:33:35    system
07:33:35    3.8.13
07:33:35    3.9.13
07:33:35    3.10.13
07:33:35  * 3.11.7 (set by /w/workspace/builder-openstack-cron/.python-version)
07:33:35  lf-activate-venv(): INFO: Reuse venv:/tmp/venv-LUjT from file:/tmp/.os_lf_venv
07:33:36  lf-activate-venv(): INFO: Installing: kubernetes python-heatclient python-openstackclient python-magnumclient 
07:33:46  ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
07:33:46  lftools 0.37.10 requires urllib3<2.1.0, but you have urllib3 2.2.2 which is incompatible.
07:33:46  lf-activate-venv(): INFO: Adding /tmp/venv-LUjT/bin to PATH
07:33:49  -----> Delete orphaned cluster
07:33:49  [Boolean condition] checking [True] against [^(1|y|yes|t|true|on|run)$] (origin token: True)
07:33:49  Run condition [Boolean condition] enabling perform for step [Execute shell]
07:33:49  [builder-openstack-cron] $ /bin/bash -l /tmp/jenkins18075121467240164807.sh
07:33:49  ---> Orphaned stacks
07:33:49  Setup pyenv:
07:33:49    system
07:33:49    3.8.13
07:33:49    3.9.13
07:33:49    3.10.13
07:33:49  * 3.11.7 (set by /w/workspace/builder-openstack-cron/.python-version)
07:33:49  lf-activate-venv(): INFO: Reuse venv:/tmp/venv-LUjT from file:/tmp/.os_lf_venv
07:33:50  lf-activate-venv(): INFO: Installing: lftools[openstack] kubernetes niet python-heatclient python-openstackclient python-magnumclient yq 
07:34:07  lf-activate-venv(): INFO: Adding /tmp/venv-LUjT/bin to PATH
07:34:07  + mapfile -t OS_COE_CLUSTERS_ID
07:34:07  ++ awk '{print $1}'
07:34:07  ++ openstack --os-cloud vex coe cluster list -f value -c uuid -c name
07:34:07  ++ grep -E '(DELETE_FAILED|UNKNOWN|UNHEALTHY)'
07:34:09  + echo '-----> Active clusters -> stacks'
07:34:09  -----> Active clusters -> stacks
07:34:09  + OS_COE_STACKS=()
07:34:09  + [[ 0 -gt 0 ]]
07:34:09  + mapfile -t OS_STACKS
07:34:09  ++ openstack --os-cloud vex stack list -f value -c 'Stack Name' -c 'Stack Status' --property stack_status=CREATE_COMPLETE --property stack_status=DELETE_FAILED --property stack_status=CREATE_FAILED
07:34:09  ++ awk '{print $1}'
07:34:11  + echo '-----> Active stacks'
07:34:11  -----> Active stacks
07:34:11  + for stack in "${OS_STACKS[@]}"
07:34:11  + echo releng-controller-csit-3node-clustering-tell-all-potassium-53
07:34:11  releng-controller-csit-3node-clustering-tell-all-potassium-53
07:34:11  + echo '-----> Delete orphaned stacks'
07:34:11  -----> Delete orphaned stacks
07:34:11  + for STACK_NAME in "${OS_STACKS[@]}"
07:34:11  + [[ 0 -gt 0 ]]
07:34:11  + stack_in_jenkins releng-controller-csit-3node-clustering-tell-all-potassium-53 https://jenkins.opendaylight.org/releng https://jenkins.opendaylight.org/sandbox
07:34:11  + STACK_NAME=releng-controller-csit-3node-clustering-tell-all-potassium-53
07:34:11  + builds=()
07:34:11  + for jenkins in "${@:2}"
07:34:11  + PARAMS='tree=computer[executors[currentExecutable[url]],'
07:34:11  + PARAMS='tree=computer[executors[currentExecutable[url]],oneOffExecutors[currentExecutable[url]]]'
07:34:11  + PARAMS='tree=computer[executors[currentExecutable[url]],oneOffExecutors[currentExecutable[url]]]&xpath=//url&wrapper=builds'
07:34:11  + JENKINS_URL='https://jenkins.opendaylight.org/releng/computer/api/json?tree=computer[executors[currentExecutable[url]],oneOffExecutors[currentExecutable[url]]]&xpath=//url&wrapper=builds'
07:34:11  ++ curl -s -w '\n\n%{http_code}' --globoff -H Content-Type:application/json 'https://jenkins.opendaylight.org/releng/computer/api/json?tree=computer[executors[currentExecutable[url]],oneOffExecutors[currentExecutable[url]]]&xpath=//url&wrapper=builds'
07:34:11  + resp='{"_class":"hudson.model.ComputerSet","computer":[{"_class":"hudson.model.Hudson$MasterComputer","executors":[],"oneOffExecutors":[]},{"_class":"jenkins.plugins.openstack.compute.JCloudsComputer","executors":[{"currentExecutable":null}],"oneOffExecutors":[]},{"_class":"jenkins.plugins.openstack.compute.JCloudsComputer","executors":[{"currentExecutable":null}],"oneOffExecutors":[]},{"_class":"jenkins.plugins.openstack.compute.JCloudsComputer","executors":[{"currentExecutable":{"_class":"hudson.model.FreeStyleBuild","url":"https://jenkins.opendaylight.org/releng/job/builder-openstack-cron/63662/"}}],"oneOffExecutors":[]},{"_class":"jenkins.plugins.openstack.compute.JCloudsComputer","executors":[{"currentExecutable":{"_class":"hudson.model.FreeStyleBuild","url":"https://jenkins.opendaylight.org/releng/job/controller-csit-3node-clustering-tell-all-potassium/53/"}}],"oneOffExecutors":[]},{"_class":"jenkins.plugins.openstack.compute.JCloudsComputer","executors":[{"currentExecutable":{"_class":"hudson.model.FreeStyleBuild","url":"https://jenkins.opendaylight.org/releng/job/mdsal-maven-merge-master/3396/"}}],"oneOffExecutors":[]}]}
07:34:11  
07:34:11  200'
07:34:11  ++ echo '{"_class":"hudson.model.ComputerSet","computer":[{"_class":"hudson.model.Hudson$MasterComputer","executors":[],"oneOffExecutors":[]},{"_class":"jenkins.plugins.openstack.compute.JCloudsComputer","executors":[{"currentExecutable":null}],"oneOffExecutors":[]},{"_class":"jenkins.plugins.openstack.compute.JCloudsComputer","executors":[{"currentExecutable":null}],"oneOffExecutors":[]},{"_class":"jenkins.plugins.openstack.compute.JCloudsComputer","executors":[{"currentExecutable":{"_class":"hudson.model.FreeStyleBuild","url":"https://jenkins.opendaylight.org/releng/job/builder-openstack-cron/63662/"}}],"oneOffExecutors":[]},{"_class":"jenkins.plugins.openstack.compute.JCloudsComputer","executors":[{"currentExecutable":{"_class":"hudson.model.FreeStyleBuild","url":"https://jenkins.opendaylight.org/releng/job/controller-csit-3node-clustering-tell-all-potassium/53/"}}],"oneOffExecutors":[]},{"_class":"jenkins.plugins.openstack.compute.JCloudsComputer","executors":[{"currentExecutable":{"_class":"hudson.model.FreeStyleBuild","url":"https://jenkins.opendaylight.org/releng/job/mdsal-maven-merge-master/3396/"}}],"oneOffExecutors":[]}]}
07:34:11  
07:34:11  200'
07:34:11  ++ head -n1
07:34:11  + json_data='{"_class":"hudson.model.ComputerSet","computer":[{"_class":"hudson.model.Hudson$MasterComputer","executors":[],"oneOffExecutors":[]},{"_class":"jenkins.plugins.openstack.compute.JCloudsComputer","executors":[{"currentExecutable":null}],"oneOffExecutors":[]},{"_class":"jenkins.plugins.openstack.compute.JCloudsComputer","executors":[{"currentExecutable":null}],"oneOffExecutors":[]},{"_class":"jenkins.plugins.openstack.compute.JCloudsComputer","executors":[{"currentExecutable":{"_class":"hudson.model.FreeStyleBuild","url":"https://jenkins.opendaylight.org/releng/job/builder-openstack-cron/63662/"}}],"oneOffExecutors":[]},{"_class":"jenkins.plugins.openstack.compute.JCloudsComputer","executors":[{"currentExecutable":{"_class":"hudson.model.FreeStyleBuild","url":"https://jenkins.opendaylight.org/releng/job/controller-csit-3node-clustering-tell-all-potassium/53/"}}],"oneOffExecutors":[]},{"_class":"jenkins.plugins.openstack.compute.JCloudsComputer","executors":[{"currentExecutable":{"_class":"hudson.model.FreeStyleBuild","url":"https://jenkins.opendaylight.org/releng/job/mdsal-maven-merge-master/3396/"}}],"oneOffExecutors":[]}]}'
07:34:11  ++ echo '{"_class":"hudson.model.ComputerSet","computer":[{"_class":"hudson.model.Hudson$MasterComputer","executors":[],"oneOffExecutors":[]},{"_class":"jenkins.plugins.openstack.compute.JCloudsComputer","executors":[{"currentExecutable":null}],"oneOffExecutors":[]},{"_class":"jenkins.plugins.openstack.compute.JCloudsComputer","executors":[{"currentExecutable":null}],"oneOffExecutors":[]},{"_class":"jenkins.plugins.openstack.compute.JCloudsComputer","executors":[{"currentExecutable":{"_class":"hudson.model.FreeStyleBuild","url":"https://jenkins.opendaylight.org/releng/job/builder-openstack-cron/63662/"}}],"oneOffExecutors":[]},{"_class":"jenkins.plugins.openstack.compute.JCloudsComputer","executors":[{"currentExecutable":{"_class":"hudson.model.FreeStyleBuild","url":"https://jenkins.opendaylight.org/releng/job/controller-csit-3node-clustering-tell-all-potassium/53/"}}],"oneOffExecutors":[]},{"_class":"jenkins.plugins.openstack.compute.JCloudsComputer","executors":[{"currentExecutable":{"_class":"hudson.model.FreeStyleBuild","url":"https://jenkins.opendaylight.org/releng/job/mdsal-maven-merge-master/3396/"}}],"oneOffExecutors":[]}]}
07:34:11  
07:34:11  200'
07:34:11  ++ awk 'END {print $NF}'
07:34:11  + status=200
07:34:11  + '[' 200 '!=' 200 ']'
07:34:11  + [[ https://jenkins.opendaylight.org/releng == *\j\e\n\k\i\n\s\.*\.\o\r\g ]]
07:34:11  + [[ https://jenkins.opendaylight.org/releng == *\j\e\n\k\i\n\s\.*\.\i\o ]]
07:34:11  ++ echo https://jenkins.opendaylight.org/releng
07:34:11  ++ awk -F/ '{print $NF}'
07:34:11  ++ sed 's/\/*$//'
07:34:11  + silo=releng
07:34:11  + export silo
07:34:11  + builds=(${builds[@]} $(echo "$json_data" |             jq -r '.computer[].executors[].currentExecutable.url'             | grep -v null | awk -F'/' '{print ENVIRON["silo"] "-" $6 "-" $7}'))
07:34:11  ++ grep -v null
07:34:11  ++ echo '{"_class":"hudson.model.ComputerSet","computer":[{"_class":"hudson.model.Hudson$MasterComputer","executors":[],"oneOffExecutors":[]},{"_class":"jenkins.plugins.openstack.compute.JCloudsComputer","executors":[{"currentExecutable":null}],"oneOffExecutors":[]},{"_class":"jenkins.plugins.openstack.compute.JCloudsComputer","executors":[{"currentExecutable":null}],"oneOffExecutors":[]},{"_class":"jenkins.plugins.openstack.compute.JCloudsComputer","executors":[{"currentExecutable":{"_class":"hudson.model.FreeStyleBuild","url":"https://jenkins.opendaylight.org/releng/job/builder-openstack-cron/63662/"}}],"oneOffExecutors":[]},{"_class":"jenkins.plugins.openstack.compute.JCloudsComputer","executors":[{"currentExecutable":{"_class":"hudson.model.FreeStyleBuild","url":"https://jenkins.opendaylight.org/releng/job/controller-csit-3node-clustering-tell-all-potassium/53/"}}],"oneOffExecutors":[]},{"_class":"jenkins.plugins.openstack.compute.JCloudsComputer","executors":[{"currentExecutable":{"_class":"hudson.model.FreeStyleBuild","url":"https://jenkins.opendaylight.org/releng/job/mdsal-maven-merge-master/3396/"}}],"oneOffExecutors":[]}]}'
07:34:11  ++ jq -r '.computer[].executors[].currentExecutable.url'
07:34:11  ++ awk -F/ '{print ENVIRON["silo"] "-" $6 "-" $7}'
07:34:11  + for jenkins in "${@:2}"
07:34:11  + PARAMS='tree=computer[executors[currentExecutable[url]],'
07:34:11  + PARAMS='tree=computer[executors[currentExecutable[url]],oneOffExecutors[currentExecutable[url]]]'
07:34:11  + PARAMS='tree=computer[executors[currentExecutable[url]],oneOffExecutors[currentExecutable[url]]]&xpath=//url&wrapper=builds'
07:34:11  + JENKINS_URL='https://jenkins.opendaylight.org/sandbox/computer/api/json?tree=computer[executors[currentExecutable[url]],oneOffExecutors[currentExecutable[url]]]&xpath=//url&wrapper=builds'
07:34:11  ++ curl -s -w '\n\n%{http_code}' --globoff -H Content-Type:application/json 'https://jenkins.opendaylight.org/sandbox/computer/api/json?tree=computer[executors[currentExecutable[url]],oneOffExecutors[currentExecutable[url]]]&xpath=//url&wrapper=builds'
07:34:11  + resp='{"_class":"hudson.model.ComputerSet","computer":[{"_class":"hudson.model.Hudson$MasterComputer","executors":[],"oneOffExecutors":[]}]}
07:34:11  
07:34:11  200'
07:34:11  ++ echo '{"_class":"hudson.model.ComputerSet","computer":[{"_class":"hudson.model.Hudson$MasterComputer","executors":[],"oneOffExecutors":[]}]}
07:34:11  
07:34:11  200'
07:34:11  ++ head -n1
07:34:11  + json_data='{"_class":"hudson.model.ComputerSet","computer":[{"_class":"hudson.model.Hudson$MasterComputer","executors":[],"oneOffExecutors":[]}]}'
07:34:11  ++ echo '{"_class":"hudson.model.ComputerSet","computer":[{"_class":"hudson.model.Hudson$MasterComputer","executors":[],"oneOffExecutors":[]}]}
07:34:11  
07:34:11  200'
07:34:11  ++ awk 'END {print $NF}'
07:34:11  + status=200
07:34:11  + '[' 200 '!=' 200 ']'
07:34:11  + [[ https://jenkins.opendaylight.org/sandbox == *\j\e\n\k\i\n\s\.*\.\o\r\g ]]
07:34:11  + [[ https://jenkins.opendaylight.org/sandbox == *\j\e\n\k\i\n\s\.*\.\i\o ]]
07:34:11  ++ echo https://jenkins.opendaylight.org/sandbox
07:34:11  ++ awk -F/ '{print $NF}'
07:34:11  ++ sed 's/\/*$//'
07:34:11  + silo=sandbox
07:34:11  + export silo
07:34:11  + builds=(${builds[@]} $(echo "$json_data" |             jq -r '.computer[].executors[].currentExecutable.url'             | grep -v null | awk -F'/' '{print ENVIRON["silo"] "-" $6 "-" $7}'))
07:34:11  ++ echo '{"_class":"hudson.model.ComputerSet","computer":[{"_class":"hudson.model.Hudson$MasterComputer","executors":[],"oneOffExecutors":[]}]}'
07:34:11  ++ jq -r '.computer[].executors[].currentExecutable.url'
07:34:11  ++ awk -F/ '{print ENVIRON["silo"] "-" $6 "-" $7}'
07:34:11  ++ grep -v null
07:34:11  + [[ releng-builder-openstack-cron-63662 releng-controller-csit-3node-clustering-tell-all-potassium-53 releng-mdsal-maven-merge-master-3396 =~ releng-controller-csit-3node-clustering-tell-all-potassium-53 ]]
07:34:11  + return 0
07:34:11  + continue
07:34:11  [Boolean condition] checking [True] against [^(1|y|yes|t|true|on|run)$] (origin token: True)
07:34:11  Run condition [Boolean condition] enabling perform for step [Execute shell]
07:34:11  [builder-openstack-cron] $ /bin/bash -l /tmp/jenkins18230986495498472842.sh
07:34:11  ---> Orphaned servers
07:34:11  Setup pyenv:
07:34:11    system
07:34:11    3.8.13
07:34:11    3.9.13
07:34:11    3.10.13
07:34:11  * 3.11.7 (set by /w/workspace/builder-openstack-cron/.python-version)
07:34:11  lf-activate-venv(): INFO: Reuse venv:/tmp/venv-LUjT from file:/tmp/.os_lf_venv
07:34:12  lf-activate-venv(): INFO: Installing: lftools[openstack] kubernetes niet python-heatclient python-openstackclient python-magnumclient yq 
07:34:24  lf-activate-venv(): INFO: Adding /tmp/venv-LUjT/bin to PATH
07:34:29  -----> Active servers
07:34:29  prd-centos8-builder-2c-2g-881
07:34:29  prd-ubuntu2004-docker-4c-4g-880
07:34:29  prd-centos8-robot-2c-8g-805
07:34:29  prd-centos7-docker-1c-4g-791
07:34:29  prd-centos7-builder-2c-2g-724
07:34:29  prd-ubuntu2004-docker-4c-4g-591
07:34:29  prd-ubuntu2004-docker-2c-2g-588
07:34:29  -----> Delete orphaned servers
07:34:30  Deleting orphaned server: prd-ubuntu2004-docker-4c-4g-591
07:34:33  AttributeError: 'Server' object has no attribute 'created'. Did you mean: 'create'?
07:34:33  Deleting orphaned server: prd-ubuntu2004-docker-2c-2g-588
07:34:35  AttributeError: 'Server' object has no attribute 'created'. Did you mean: 'create'?
07:34:36  Build step 'Conditional step (single)' marked build as failure
07:34:36  $ ssh-agent -k
07:34:36  unset SSH_AUTH_SOCK;
07:34:36  unset SSH_AGENT_PID;
07:34:36  echo Agent pid 5452 killed;
07:34:36  [ssh-agent] Stopped.
07:34:36  [PostBuildScript] - [INFO] Executing post build scripts.
07:34:36  [builder-openstack-cron] $ /bin/bash /tmp/jenkins5581685609691964546.sh
07:34:36  ---> sysstat.sh
07:34:36  [builder-openstack-cron] $ /bin/bash /tmp/jenkins9232417842976469385.sh
07:34:36  ---> package-listing.sh
07:34:36  ++ tr '[:upper:]' '[:lower:]'
07:34:36  ++ facter osfamily
07:34:36  + OS_FAMILY=redhat
07:34:36  + workspace=/w/workspace/builder-openstack-cron
07:34:36  + START_PACKAGES=/tmp/packages_start.txt
07:34:36  + END_PACKAGES=/tmp/packages_end.txt
07:34:36  + DIFF_PACKAGES=/tmp/packages_diff.txt
07:34:36  + PACKAGES=/tmp/packages_start.txt
07:34:36  + '[' /w/workspace/builder-openstack-cron ']'
07:34:36  + PACKAGES=/tmp/packages_end.txt
07:34:36  + case "${OS_FAMILY}" in
07:34:36  + rpm -qa
07:34:36  + sort
07:34:37  + '[' -f /tmp/packages_start.txt ']'
07:34:37  + '[' -f /tmp/packages_end.txt ']'
07:34:37  + diff /tmp/packages_start.txt /tmp/packages_end.txt
07:34:37  + '[' /w/workspace/builder-openstack-cron ']'
07:34:37  + mkdir -p /w/workspace/builder-openstack-cron/archives/
07:34:37  + cp -f /tmp/packages_diff.txt /tmp/packages_end.txt /tmp/packages_start.txt /w/workspace/builder-openstack-cron/archives/
07:34:37  [builder-openstack-cron] $ /bin/bash /tmp/jenkins9857295533008309004.sh
07:34:37  ---> capture-instance-metadata.sh
07:34:37  Setup pyenv:
07:34:37    system
07:34:37    3.8.13
07:34:37    3.9.13
07:34:37    3.10.13
07:34:37  * 3.11.7 (set by /w/workspace/builder-openstack-cron/.python-version)
07:34:37  lf-activate-venv(): INFO: Reuse venv:/tmp/venv-LUjT from file:/tmp/.os_lf_venv
07:34:38  lf-activate-venv(): INFO: Installing: lftools 
07:34:47  lf-activate-venv(): INFO: Adding /tmp/venv-LUjT/bin to PATH
07:34:47  INFO: Running in OpenStack, capturing instance metadata
07:34:47  [builder-openstack-cron] $ /bin/bash /tmp/jenkins2104227037813481416.sh
07:34:47  provisioning config files...
07:34:47  Could not find credentials [logs] for builder-openstack-cron #63662
07:34:47  copy managed file [jenkins-log-archives-settings] to file:/w/workspace/builder-openstack-cron@tmp/config10315533553896651311tmp
07:34:47  Regular expression run condition: Expression=[^.*logs-s3.*], Label=[odl-logs-s3-cloudfront-index]
07:34:47  Run condition [Regular expression match] enabling perform for step [Provide Configuration files]
07:34:47  provisioning config files...
07:34:48  copy managed file [jenkins-s3-log-ship] to file:/home/jenkins/.aws/credentials
07:34:48  [EnvInject] - Injecting environment variables from a build step.
07:34:48  [EnvInject] - Injecting as environment variables the properties content 
07:34:48  SERVER_ID=logs
07:34:48  
07:34:48  [EnvInject] - Variables injected successfully.
07:34:48  [builder-openstack-cron] $ /bin/bash /tmp/jenkins15574488713361344886.sh
07:34:48  ---> create-netrc.sh
07:34:48  WARN: Log server credential not found.
07:34:48  [builder-openstack-cron] $ /bin/bash /tmp/jenkins401582776531623692.sh
07:34:48  ---> python-tools-install.sh
07:34:48  Setup pyenv:
07:34:48    system
07:34:48    3.8.13
07:34:48    3.9.13
07:34:48    3.10.13
07:34:48  * 3.11.7 (set by /w/workspace/builder-openstack-cron/.python-version)
07:34:48  lf-activate-venv(): INFO: Reuse venv:/tmp/venv-LUjT from file:/tmp/.os_lf_venv
07:34:49  lf-activate-venv(): INFO: Installing: lftools 
07:34:57  lf-activate-venv(): INFO: Adding /tmp/venv-LUjT/bin to PATH
07:34:57  [builder-openstack-cron] $ /bin/bash /tmp/jenkins1554680513491553432.sh
07:34:57  ---> sudo-logs.sh
07:34:57  Archiving 'sudo' log..
07:34:58  [builder-openstack-cron] $ /bin/bash /tmp/jenkins11157378228177159043.sh
07:34:58  ---> job-cost.sh
07:34:58  Setup pyenv:
07:34:58    system
07:34:58    3.8.13
07:34:58    3.9.13
07:34:58    3.10.13
07:34:58  * 3.11.7 (set by /w/workspace/builder-openstack-cron/.python-version)
07:34:58  lf-activate-venv(): INFO: Reuse venv:/tmp/venv-LUjT from file:/tmp/.os_lf_venv
07:34:59  lf-activate-venv(): INFO: Installing: zipp==1.1.0 python-openstackclient urllib3~=1.26.15 
07:35:13  lf-activate-venv(): INFO: Adding /tmp/venv-LUjT/bin to PATH
07:35:13  INFO: No Stack...
07:35:14  INFO: Retrieving Pricing Info for: v3-standard-2
07:35:14  INFO: Archiving Costs
07:35:14  [builder-openstack-cron] $ /bin/bash -l /tmp/jenkins302059217301405664.sh
07:35:14  ---> logs-deploy.sh
07:35:14  Setup pyenv:
07:35:14    system
07:35:14    3.8.13
07:35:14    3.9.13
07:35:14    3.10.13
07:35:14  * 3.11.7 (set by /w/workspace/builder-openstack-cron/.python-version)
07:35:15  lf-activate-venv(): INFO: Reuse venv:/tmp/venv-LUjT from file:/tmp/.os_lf_venv
07:35:16  lf-activate-venv(): INFO: Installing: lftools 
07:35:26  lf-activate-venv(): INFO: Adding /tmp/venv-LUjT/bin to PATH
07:35:26  WARNING: Nexus logging server not set
07:35:26  INFO: S3 path logs/releng/vex-yul-odl-jenkins-1/builder-openstack-cron/63662/
07:35:26  INFO: archiving logs to S3
07:35:27  ---> uname -a:
07:35:27  Linux prd-centos8-builder-2c-2g-881.novalocal 4.18.0-553.5.1.el8.x86_64 #1 SMP Tue May 21 05:46:01 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
07:35:27  
07:35:27  
07:35:27  ---> lscpu:
07:35:27  Architecture:        x86_64
07:35:27  CPU op-mode(s):      32-bit, 64-bit
07:35:27  Byte Order:          Little Endian
07:35:27  CPU(s):              2
07:35:27  On-line CPU(s) list: 0,1
07:35:27  Thread(s) per core:  1
07:35:27  Core(s) per socket:  1
07:35:27  Socket(s):           2
07:35:27  NUMA node(s):        1
07:35:27  Vendor ID:           AuthenticAMD
07:35:27  CPU family:          23
07:35:27  Model:               49
07:35:27  Model name:          AMD EPYC-Rome Processor
07:35:27  Stepping:            0
07:35:27  CPU MHz:             2799.998
07:35:27  BogoMIPS:            5599.99
07:35:27  Virtualization:      AMD-V
07:35:27  Hypervisor vendor:   KVM
07:35:27  Virtualization type: full
07:35:27  L1d cache:           32K
07:35:27  L1i cache:           32K
07:35:27  L2 cache:            512K
07:35:27  L3 cache:            16384K
07:35:27  NUMA node0 CPU(s):   0,1
07:35:27  Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd ibrs ibpb stibp vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr wbnoinvd arat npt nrip_save umip rdpid arch_capabilities
07:35:27  
07:35:27  
07:35:27  ---> nproc:
07:35:27  2
07:35:27  
07:35:27  
07:35:27  ---> df -h:
07:35:27  Filesystem      Size  Used Avail Use% Mounted on
07:35:27  devtmpfs        3.8G     0  3.8G   0% /dev
07:35:27  tmpfs           3.8G     0  3.8G   0% /dev/shm
07:35:27  tmpfs           3.8G   17M  3.8G   1% /run
07:35:27  tmpfs           3.8G     0  3.8G   0% /sys/fs/cgroup
07:35:27  /dev/vda1        40G  8.4G   32G  21% /
07:35:27  tmpfs           770M     0  770M   0% /run/user/1001
07:35:27  
07:35:27  
07:35:27  ---> free -m:
07:35:27                total        used        free      shared  buff/cache   available
07:35:27  Mem:           7697         617        4839          19        2241        6785
07:35:27  Swap:          1023           0        1023
07:35:27  
07:35:27  
07:35:27  ---> ip addr:
07:35:27  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
07:35:27      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
07:35:27      inet 127.0.0.1/8 scope host lo
07:35:27         valid_lft forever preferred_lft forever
07:35:27      inet6 ::1/128 scope host 
07:35:27         valid_lft forever preferred_lft forever
07:35:27  2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1458 qdisc mq state UP group default qlen 1000
07:35:27      link/ether fa:16:3e:3e:26:58 brd ff:ff:ff:ff:ff:ff
07:35:27      altname enp0s3
07:35:27      altname ens3
07:35:27      inet 10.30.171.218/23 brd 10.30.171.255 scope global dynamic noprefixroute eth0
07:35:27         valid_lft 86171sec preferred_lft 86171sec
07:35:27      inet6 fe80::f816:3eff:fe3e:2658/64 scope link 
07:35:27         valid_lft forever preferred_lft forever
07:35:27  3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1458 qdisc noqueue state DOWN group default 
07:35:27      link/ether 02:42:f8:0e:81:3f brd ff:ff:ff:ff:ff:ff
07:35:27      inet 10.250.0.254/24 brd 10.250.0.255 scope global docker0
07:35:27         valid_lft forever preferred_lft forever
07:35:27  
07:35:27  
07:35:27  ---> sar -b -r -n DEV:
07:35:27  Linux 4.18.0-553.5.1.el8.x86_64 (centos-stream-8-builder-5e6ee0fb-b6d9-4116-90d4-35c890e2b0f7.nov) 	07/21/2024 	_x86_64_	(2 CPU)
07:35:27  
07:35:27  07:31:33     LINUX RESTART	(2 CPU)
07:35:27  
07:35:27  07:32:01 AM       tps      rtps      wtps   bread/s   bwrtn/s
07:35:27  07:33:01 AM    157.82     30.05    127.77   7437.48  38007.21
07:35:27  07:34:01 AM    101.47      3.57     97.90     72.91   8645.28
07:35:27  07:35:01 AM     41.14      1.17     39.98    134.51   4009.63
07:35:27  Average:       100.15     11.60     88.55   2548.71  16889.26
07:35:27  
07:35:27  07:32:01 AM kbmemfree   kbavail kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty
07:35:27  07:33:01 AM   5160240   6957736   2722184     34.53      2688   1987784   1032952     11.57    231152   2192572    105740
07:35:27  07:34:01 AM   4947496   6951512   2934928     37.23      2688   2181976    980596     10.98    229024   2358732    106324
07:35:27  07:35:01 AM   5008932   6999220   2873492     36.45      2688   2167944    936280     10.48    428876   2097304      1400
07:35:27  Average:      5038889   6969489   2843535     36.07      2688   2112568    983276     11.01    296351   2216203     71155
07:35:27  
07:35:27  07:32:01 AM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s   txcmp/s  rxmcst/s   %ifutil
07:35:27  07:33:01 AM        lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:35:27  07:33:01 AM      eth0    280.96    176.76   1695.46     61.54      0.00      0.00      0.00      0.00
07:35:27  07:33:01 AM   docker0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:35:27  07:34:01 AM        lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:35:27  07:34:01 AM      eth0     59.95     49.23    507.54      8.46      0.00      0.00      0.00      0.00
07:35:27  07:34:01 AM   docker0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:35:27  07:35:01 AM        lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:35:27  07:35:01 AM      eth0     28.38     27.13     31.71     10.83      0.00      0.00      0.00      0.00
07:35:27  07:35:01 AM   docker0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:35:27  Average:           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:35:27  Average:         eth0    123.11     84.38    745.00     26.94      0.00      0.00      0.00      0.00
07:35:27  Average:      docker0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
07:35:27  
07:35:27  
07:35:27  ---> sar -P ALL:
07:35:27  Linux 4.18.0-553.5.1.el8.x86_64 (centos-stream-8-builder-5e6ee0fb-b6d9-4116-90d4-35c890e2b0f7.nov) 	07/21/2024 	_x86_64_	(2 CPU)
07:35:27  
07:35:27  07:31:33     LINUX RESTART	(2 CPU)
07:35:27  
07:35:27  07:32:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
07:35:27  07:33:01 AM     all     38.64      0.00      7.64      6.52      0.09     47.11
07:35:27  07:33:01 AM       0     41.22      0.00      7.69      5.03      0.10     45.95
07:35:27  07:33:01 AM       1     36.05      0.00      7.58      8.01      0.08     48.27
07:35:27  07:34:01 AM     all     34.42      0.00      5.62      3.37      0.10     56.48
07:35:27  07:34:01 AM       0     32.33      0.00      5.63      3.27      0.10     58.67
07:35:27  07:34:01 AM       1     36.51      0.00      5.61      3.48      0.10     54.30
07:35:27  07:35:01 AM     all     34.91      0.00      4.16      1.16      0.10     59.67
07:35:27  07:35:01 AM       0     31.39      0.00      4.22      1.25      0.10     63.04
07:35:27  07:35:01 AM       1     38.43      0.00      4.10      1.07      0.10     56.30
07:35:27  Average:        all     35.99      0.00      5.80      3.68      0.10     54.43
07:35:27  Average:          0     34.98      0.00      5.85      3.18      0.10     55.90
07:35:27  Average:          1     37.00      0.00      5.76      4.18      0.09     52.96
07:35:27  
07:35:27  
07:35:27