19:32:12 Started by timer 19:32:12 Running as SYSTEM 19:32:12 [EnvInject] - Loading node environment variables. 19:32:12 Building remotely on prd-centos8-builder-2c-2g-30409 (centos8-builder-2c-2g) in workspace /w/workspace/builder-openstack-cron 19:32:12 [ssh-agent] Looking for ssh-agent implementation... 19:32:12 [ssh-agent] Exec ssh-agent (binary ssh-agent on a remote machine) 19:32:12 $ ssh-agent 19:32:12 SSH_AUTH_SOCK=/tmp/ssh-EN68u944pJ29/agent.5457 19:32:12 SSH_AGENT_PID=5459 19:32:12 [ssh-agent] Started. 19:32:12 Running ssh-add (command line suppressed) 19:32:12 Identity added: /w/workspace/builder-openstack-cron@tmp/private_key_11445661306231088615.key (/w/workspace/builder-openstack-cron@tmp/private_key_11445661306231088615.key) 19:32:12 [ssh-agent] Using credentials jenkins (jenkins-ssh) 19:32:12 The recommended git tool is: NONE 19:32:15 using credential jenkins-ssh 19:32:15 Wiping out workspace first. 19:32:15 Cloning the remote Git repository 19:32:15 Cloning repository git://devvexx.opendaylight.org/mirror/releng/builder 19:32:15 > git init /w/workspace/builder-openstack-cron # timeout=10 19:32:15 Fetching upstream changes from git://devvexx.opendaylight.org/mirror/releng/builder 19:32:15 > git --version # timeout=10 19:32:15 > git --version # 'git version 2.43.0' 19:32:15 using GIT_SSH to set credentials jenkins-ssh 19:32:15 [INFO] Currently running in a labeled security context 19:32:15 [INFO] Currently SELinux is 'enforcing' on the host 19:32:15 > /usr/bin/chcon --type=ssh_home_t /w/workspace/builder-openstack-cron@tmp/jenkins-gitclient-ssh13939495533835886503.key 19:32:15 Verifying host key using known hosts file 19:32:15 You're using 'Known hosts file' strategy to verify ssh host keys, but your known_hosts file does not exist, please go to 'Manage Jenkins' -> 'Security' -> 'Git Host Key Verification Configuration' and configure host key verification. 19:32:15 > git fetch --tags --force --progress -- git://devvexx.opendaylight.org/mirror/releng/builder +refs/heads/*:refs/remotes/origin/* # timeout=10 19:32:17 > git config remote.origin.url git://devvexx.opendaylight.org/mirror/releng/builder # timeout=10 19:32:17 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 19:32:18 Avoid second fetch 19:32:18 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10 19:32:18 Checking out Revision fdd66ab9c49943e4d5dbff91f859ba7a9f434438 (refs/remotes/origin/master) 19:32:18 > git config core.sparsecheckout # timeout=10 19:32:18 > git checkout -f fdd66ab9c49943e4d5dbff91f859ba7a9f434438 # timeout=10 19:32:18 Commit message: "Fix distribution jobs" 19:32:18 > git rev-list --no-walk fdd66ab9c49943e4d5dbff91f859ba7a9f434438 # timeout=10 19:32:18 > git remote # timeout=10 19:32:18 > git submodule init # timeout=10 19:32:18 > git submodule sync # timeout=10 19:32:18 > git config --get remote.origin.url # timeout=10 19:32:18 > git submodule init # timeout=10 19:32:18 > git config -f .gitmodules --get-regexp ^submodule\.(.+)\.url # timeout=10 19:32:18 > git config --get submodule.global-jjb.url # timeout=10 19:32:18 > git config -f .gitmodules --get submodule.global-jjb.path # timeout=10 19:32:18 > git config --get submodule.packer/common-packer.url # timeout=10 19:32:19 > git config -f .gitmodules --get submodule.packer/common-packer.path # timeout=10 19:32:19 > git submodule update --init --recursive global-jjb # timeout=10 19:32:20 > git submodule update --init --recursive packer/common-packer # timeout=10 19:32:23 Run condition [Boolean condition] enabling prebuild for step [Execute shell] 19:32:23 Run condition [Boolean condition] enabling prebuild for step [Execute shell] 19:32:23 Run condition [Boolean condition] enabling prebuild for step [Execute shell] 19:32:23 Run condition [Boolean condition] enabling prebuild for step [Execute shell] 19:32:23 Run condition [Boolean condition] enabling prebuild for step [Execute shell] 19:32:23 Run condition [Boolean condition] enabling prebuild for step [Execute shell] 19:32:23 Run condition [Boolean condition] enabling prebuild for step [BuilderChain] 19:32:23 No emails were triggered. 19:32:23 provisioning config files... 19:32:24 copy managed file [clouds-yaml] to file:/home/jenkins/.config/openstack/clouds.yaml 19:32:24 copy managed file [npmrc] to file:/home/jenkins/.npmrc 19:32:24 copy managed file [pipconf] to file:/home/jenkins/.config/pip/pip.conf 19:32:24 [builder-openstack-cron] $ /bin/bash /tmp/jenkins11977435753196551713.sh 19:32:24 ---> python-tools-install.sh 19:32:24 Setup pyenv: 19:32:24 system 19:32:24 * 3.8.13 (set by /opt/pyenv/version) 19:32:24 * 3.9.13 (set by /opt/pyenv/version) 19:32:24 * 3.10.13 (set by /opt/pyenv/version) 19:32:24 * 3.11.7 (set by /opt/pyenv/version) 19:32:29 lf-activate-venv(): INFO: Creating python3 venv at /tmp/venv-qtS7 19:32:29 lf-activate-venv(): INFO: Save venv in file: /tmp/.os_lf_venv 19:32:32 lf-activate-venv(): INFO: Installing: lftools 19:33:05 lf-activate-venv(): INFO: Adding /tmp/venv-qtS7/bin to PATH 19:33:05 Generating Requirements File 19:33:28 Python 3.11.7 19:33:28 pip 24.2 from /tmp/venv-qtS7/lib/python3.11/site-packages/pip (python 3.11) 19:33:28 appdirs==1.4.4 19:33:28 argcomplete==3.5.0 19:33:28 aspy.yaml==1.3.0 19:33:28 attrs==24.2.0 19:33:28 autopage==0.5.2 19:33:28 beautifulsoup4==4.12.3 19:33:28 boto3==1.35.29 19:33:28 botocore==1.35.29 19:33:28 bs4==0.0.2 19:33:28 cachetools==5.5.0 19:33:28 certifi==2024.8.30 19:33:28 cffi==1.17.1 19:33:28 cfgv==3.4.0 19:33:28 chardet==5.2.0 19:33:28 charset-normalizer==3.3.2 19:33:28 click==8.1.7 19:33:28 cliff==4.7.0 19:33:28 cmd2==2.4.3 19:33:28 cryptography==3.3.2 19:33:28 debtcollector==3.0.0 19:33:28 decorator==5.1.1 19:33:28 defusedxml==0.7.1 19:33:28 Deprecated==1.2.14 19:33:28 distlib==0.3.8 19:33:28 dnspython==2.6.1 19:33:28 docker==4.2.2 19:33:28 dogpile.cache==1.3.3 19:33:28 durationpy==0.7 19:33:28 email_validator==2.2.0 19:33:28 filelock==3.16.1 19:33:28 future==1.0.0 19:33:28 gitdb==4.0.11 19:33:28 GitPython==3.1.43 19:33:28 google-auth==2.35.0 19:33:28 httplib2==0.22.0 19:33:28 identify==2.6.1 19:33:28 idna==3.10 19:33:28 importlib-resources==1.5.0 19:33:28 iso8601==2.1.0 19:33:28 Jinja2==3.1.4 19:33:28 jmespath==1.0.1 19:33:28 jsonpatch==1.33 19:33:28 jsonpointer==3.0.0 19:33:28 jsonschema==4.23.0 19:33:28 jsonschema-specifications==2023.12.1 19:33:28 keystoneauth1==5.8.0 19:33:28 kubernetes==31.0.0 19:33:28 lftools==0.37.10 19:33:28 lxml==5.3.0 19:33:28 MarkupSafe==2.1.5 19:33:28 msgpack==1.1.0 19:33:28 multi_key_dict==2.0.3 19:33:28 munch==4.0.0 19:33:28 netaddr==1.3.0 19:33:28 netifaces==0.11.0 19:33:28 niet==1.4.2 19:33:28 nodeenv==1.9.1 19:33:28 oauth2client==4.1.3 19:33:28 oauthlib==3.2.2 19:33:28 openstacksdk==4.0.0 19:33:28 os-client-config==2.1.0 19:33:28 os-service-types==1.7.0 19:33:28 osc-lib==3.1.0 19:33:28 oslo.config==9.6.0 19:33:28 oslo.context==5.6.0 19:33:28 oslo.i18n==6.4.0 19:33:28 oslo.log==6.1.2 19:33:28 oslo.serialization==5.5.0 19:33:28 oslo.utils==7.3.0 19:33:28 packaging==24.1 19:33:28 pbr==6.1.0 19:33:28 platformdirs==4.3.6 19:33:28 prettytable==3.11.0 19:33:28 pyasn1==0.6.1 19:33:28 pyasn1_modules==0.4.1 19:33:28 pycparser==2.22 19:33:28 pygerrit2==2.0.15 19:33:28 PyGithub==2.4.0 19:33:28 PyJWT==2.9.0 19:33:28 PyNaCl==1.5.0 19:33:28 pyparsing==2.4.7 19:33:28 pyperclip==1.9.0 19:33:28 pyrsistent==0.20.0 19:33:28 python-cinderclient==9.6.0 19:33:28 python-dateutil==2.9.0.post0 19:33:28 python-heatclient==4.0.0 19:33:28 python-jenkins==1.8.2 19:33:28 python-keystoneclient==5.5.0 19:33:28 python-magnumclient==4.7.0 19:33:28 python-openstackclient==7.1.2 19:33:28 python-swiftclient==4.6.0 19:33:28 PyYAML==6.0.2 19:33:28 referencing==0.35.1 19:33:28 requests==2.32.3 19:33:28 requests-oauthlib==2.0.0 19:33:28 requestsexceptions==1.4.0 19:33:28 rfc3986==2.0.0 19:33:28 rpds-py==0.20.0 19:33:28 rsa==4.9 19:33:28 ruamel.yaml==0.18.6 19:33:28 ruamel.yaml.clib==0.2.8 19:33:28 s3transfer==0.10.2 19:33:28 simplejson==3.19.3 19:33:28 six==1.16.0 19:33:28 smmap==5.0.1 19:33:28 soupsieve==2.6 19:33:28 stevedore==5.3.0 19:33:28 tabulate==0.9.0 19:33:28 toml==0.10.2 19:33:28 tomlkit==0.13.2 19:33:28 tqdm==4.66.5 19:33:28 typing_extensions==4.12.2 19:33:28 tzdata==2024.2 19:33:28 urllib3==1.26.20 19:33:28 virtualenv==20.26.6 19:33:28 wcwidth==0.2.13 19:33:28 websocket-client==1.8.0 19:33:28 wrapt==1.16.0 19:33:28 xdg==6.0.0 19:33:28 xmltodict==0.13.0 19:33:28 yq==3.4.3 19:33:28 [EnvInject] - Injecting environment variables from a build step. 19:33:28 [EnvInject] - Injecting as environment variables the properties content 19:33:28 OS_CLOUD=vex 19:33:28 19:33:28 [EnvInject] - Variables injected successfully. 19:33:28 [Boolean condition] checking [True] against [^(1|y|yes|t|true|on|run)$] (origin token: True) 19:33:28 Run condition [Boolean condition] enabling perform for step [Execute shell] 19:33:28 [builder-openstack-cron] $ /bin/bash -l /tmp/jenkins8767438879056600137.sh 19:33:28 ---> Orphaned k8s clusters 19:33:28 Setup pyenv: 19:33:29 system 19:33:29 3.8.13 19:33:29 3.9.13 19:33:29 3.10.13 19:33:29 * 3.11.7 (set by /w/workspace/builder-openstack-cron/.python-version) 19:33:29 lf-activate-venv(): INFO: Reuse venv:/tmp/venv-qtS7 from file:/tmp/.os_lf_venv 19:33:30 lf-activate-venv(): INFO: Installing: kubernetes python-heatclient python-openstackclient python-magnumclient 19:33:36 ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 19:33:36 lftools 0.37.10 requires urllib3<2.1.0, but you have urllib3 2.2.3 which is incompatible. 19:33:36 lf-activate-venv(): INFO: Adding /tmp/venv-qtS7/bin to PATH 19:33:39 -----> Delete orphaned cluster 19:33:39 [Boolean condition] checking [True] against [^(1|y|yes|t|true|on|run)$] (origin token: True) 19:33:39 Run condition [Boolean condition] enabling perform for step [Execute shell] 19:33:39 [builder-openstack-cron] $ /bin/bash -l /tmp/jenkins9468272423755606007.sh 19:33:39 ---> Orphaned stacks 19:33:39 Setup pyenv: 19:33:39 system 19:33:39 3.8.13 19:33:39 3.9.13 19:33:39 3.10.13 19:33:39 * 3.11.7 (set by /w/workspace/builder-openstack-cron/.python-version) 19:33:39 lf-activate-venv(): INFO: Reuse venv:/tmp/venv-qtS7 from file:/tmp/.os_lf_venv 19:33:40 lf-activate-venv(): INFO: Installing: lftools[openstack] kubernetes niet python-heatclient python-openstackclient python-magnumclient yq 19:33:57 lf-activate-venv(): INFO: Adding /tmp/venv-qtS7/bin to PATH 19:33:57 + mapfile -t OS_COE_CLUSTERS_ID 19:33:57 ++ awk '{print $1}' 19:33:57 ++ openstack --os-cloud vex coe cluster list -f value -c uuid -c name 19:33:57 ++ grep -E '(DELETE_FAILED|UNKNOWN|UNHEALTHY)' 19:34:00 + echo '-----> Active clusters -> stacks' 19:34:00 -----> Active clusters -> stacks 19:34:00 + OS_COE_STACKS=() 19:34:00 + [[ 0 -gt 0 ]] 19:34:00 + mapfile -t OS_STACKS 19:34:00 ++ awk '{print $1}' 19:34:00 ++ openstack --os-cloud vex stack list -f value -c 'Stack Name' -c 'Stack Status' --property stack_status=CREATE_COMPLETE --property stack_status=DELETE_FAILED --property stack_status=CREATE_FAILED 19:34:01 + echo '-----> Active stacks' 19:34:01 -----> Active stacks 19:34:01 + for stack in "${OS_STACKS[@]}" 19:34:01 + echo releng-openflowplugin-csit-1node-perf-bulkomatic-only-potassium-400 19:34:01 releng-openflowplugin-csit-1node-perf-bulkomatic-only-potassium-400 19:34:01 + echo '-----> Delete orphaned stacks' 19:34:01 -----> Delete orphaned stacks 19:34:01 + for STACK_NAME in "${OS_STACKS[@]}" 19:34:01 + [[ 0 -gt 0 ]] 19:34:01 + stack_in_jenkins releng-openflowplugin-csit-1node-perf-bulkomatic-only-potassium-400 https://jenkins.opendaylight.org/releng https://jenkins.opendaylight.org/sandbox 19:34:01 + STACK_NAME=releng-openflowplugin-csit-1node-perf-bulkomatic-only-potassium-400 19:34:01 + builds=() 19:34:01 + for jenkins in "${@:2}" 19:34:01 + PARAMS='tree=computer[executors[currentExecutable[url]],' 19:34:01 + PARAMS='tree=computer[executors[currentExecutable[url]],oneOffExecutors[currentExecutable[url]]]' 19:34:01 + PARAMS='tree=computer[executors[currentExecutable[url]],oneOffExecutors[currentExecutable[url]]]&xpath=//url&wrapper=builds' 19:34:01 + JENKINS_URL='https://jenkins.opendaylight.org/releng/computer/api/json?tree=computer[executors[currentExecutable[url]],oneOffExecutors[currentExecutable[url]]]&xpath=//url&wrapper=builds' 19:34:01 ++ curl -s -w '\n\n%{http_code}' --globoff -H Content-Type:application/json 'https://jenkins.opendaylight.org/releng/computer/api/json?tree=computer[executors[currentExecutable[url]],oneOffExecutors[currentExecutable[url]]]&xpath=//url&wrapper=builds' 19:34:01 + resp='{"_class":"hudson.model.ComputerSet","computer":[{"_class":"hudson.model.Hudson$MasterComputer","executors":[],"oneOffExecutors":[]},{"_class":"jenkins.plugins.openstack.compute.JCloudsComputer","executors":[{"currentExecutable":null}],"oneOffExecutors":[]},{"_class":"jenkins.plugins.openstack.compute.JCloudsComputer","executors":[{"currentExecutable":{"_class":"hudson.model.FreeStyleBuild","url":"https://jenkins.opendaylight.org/releng/job/builder-openstack-cron/65340/"}}],"oneOffExecutors":[]},{"_class":"jenkins.plugins.openstack.compute.JCloudsComputer","executors":[{"currentExecutable":{"_class":"hudson.model.FreeStyleBuild","url":"https://jenkins.opendaylight.org/releng/job/aaa-maven-merge-0.18.x/260/"}}],"oneOffExecutors":[]}]} 19:34:01 19:34:01 200' 19:34:01 ++ echo '{"_class":"hudson.model.ComputerSet","computer":[{"_class":"hudson.model.Hudson$MasterComputer","executors":[],"oneOffExecutors":[]},{"_class":"jenkins.plugins.openstack.compute.JCloudsComputer","executors":[{"currentExecutable":null}],"oneOffExecutors":[]},{"_class":"jenkins.plugins.openstack.compute.JCloudsComputer","executors":[{"currentExecutable":{"_class":"hudson.model.FreeStyleBuild","url":"https://jenkins.opendaylight.org/releng/job/builder-openstack-cron/65340/"}}],"oneOffExecutors":[]},{"_class":"jenkins.plugins.openstack.compute.JCloudsComputer","executors":[{"currentExecutable":{"_class":"hudson.model.FreeStyleBuild","url":"https://jenkins.opendaylight.org/releng/job/aaa-maven-merge-0.18.x/260/"}}],"oneOffExecutors":[]}]} 19:34:01 19:34:01 200' 19:34:01 ++ head -n1 19:34:01 + json_data='{"_class":"hudson.model.ComputerSet","computer":[{"_class":"hudson.model.Hudson$MasterComputer","executors":[],"oneOffExecutors":[]},{"_class":"jenkins.plugins.openstack.compute.JCloudsComputer","executors":[{"currentExecutable":null}],"oneOffExecutors":[]},{"_class":"jenkins.plugins.openstack.compute.JCloudsComputer","executors":[{"currentExecutable":{"_class":"hudson.model.FreeStyleBuild","url":"https://jenkins.opendaylight.org/releng/job/builder-openstack-cron/65340/"}}],"oneOffExecutors":[]},{"_class":"jenkins.plugins.openstack.compute.JCloudsComputer","executors":[{"currentExecutable":{"_class":"hudson.model.FreeStyleBuild","url":"https://jenkins.opendaylight.org/releng/job/aaa-maven-merge-0.18.x/260/"}}],"oneOffExecutors":[]}]}' 19:34:01 ++ echo '{"_class":"hudson.model.ComputerSet","computer":[{"_class":"hudson.model.Hudson$MasterComputer","executors":[],"oneOffExecutors":[]},{"_class":"jenkins.plugins.openstack.compute.JCloudsComputer","executors":[{"currentExecutable":null}],"oneOffExecutors":[]},{"_class":"jenkins.plugins.openstack.compute.JCloudsComputer","executors":[{"currentExecutable":{"_class":"hudson.model.FreeStyleBuild","url":"https://jenkins.opendaylight.org/releng/job/builder-openstack-cron/65340/"}}],"oneOffExecutors":[]},{"_class":"jenkins.plugins.openstack.compute.JCloudsComputer","executors":[{"currentExecutable":{"_class":"hudson.model.FreeStyleBuild","url":"https://jenkins.opendaylight.org/releng/job/aaa-maven-merge-0.18.x/260/"}}],"oneOffExecutors":[]}]} 19:34:01 19:34:01 200' 19:34:01 ++ awk 'END {print $NF}' 19:34:01 + status=200 19:34:01 + '[' 200 '!=' 200 ']' 19:34:01 + [[ https://jenkins.opendaylight.org/releng == *\j\e\n\k\i\n\s\.*\.\o\r\g ]] 19:34:01 + [[ https://jenkins.opendaylight.org/releng == *\j\e\n\k\i\n\s\.*\.\i\o ]] 19:34:01 ++ awk -F/ '{print $NF}' 19:34:01 ++ echo https://jenkins.opendaylight.org/releng 19:34:01 ++ sed 's/\/*$//' 19:34:01 + silo=releng 19:34:01 + export silo 19:34:01 + builds=(${builds[@]} $(echo "$json_data" | jq -r '.computer[].executors[].currentExecutable.url' | grep -v null | awk -F'/' '{print ENVIRON["silo"] "-" $6 "-" $7}')) 19:34:01 ++ jq -r '.computer[].executors[].currentExecutable.url' 19:34:01 ++ grep -v null 19:34:01 ++ echo '{"_class":"hudson.model.ComputerSet","computer":[{"_class":"hudson.model.Hudson$MasterComputer","executors":[],"oneOffExecutors":[]},{"_class":"jenkins.plugins.openstack.compute.JCloudsComputer","executors":[{"currentExecutable":null}],"oneOffExecutors":[]},{"_class":"jenkins.plugins.openstack.compute.JCloudsComputer","executors":[{"currentExecutable":{"_class":"hudson.model.FreeStyleBuild","url":"https://jenkins.opendaylight.org/releng/job/builder-openstack-cron/65340/"}}],"oneOffExecutors":[]},{"_class":"jenkins.plugins.openstack.compute.JCloudsComputer","executors":[{"currentExecutable":{"_class":"hudson.model.FreeStyleBuild","url":"https://jenkins.opendaylight.org/releng/job/aaa-maven-merge-0.18.x/260/"}}],"oneOffExecutors":[]}]}' 19:34:01 ++ awk -F/ '{print ENVIRON["silo"] "-" $6 "-" $7}' 19:34:01 + for jenkins in "${@:2}" 19:34:01 + PARAMS='tree=computer[executors[currentExecutable[url]],' 19:34:01 + PARAMS='tree=computer[executors[currentExecutable[url]],oneOffExecutors[currentExecutable[url]]]' 19:34:01 + PARAMS='tree=computer[executors[currentExecutable[url]],oneOffExecutors[currentExecutable[url]]]&xpath=//url&wrapper=builds' 19:34:01 + JENKINS_URL='https://jenkins.opendaylight.org/sandbox/computer/api/json?tree=computer[executors[currentExecutable[url]],oneOffExecutors[currentExecutable[url]]]&xpath=//url&wrapper=builds' 19:34:01 ++ curl -s -w '\n\n%{http_code}' --globoff -H Content-Type:application/json 'https://jenkins.opendaylight.org/sandbox/computer/api/json?tree=computer[executors[currentExecutable[url]],oneOffExecutors[currentExecutable[url]]]&xpath=//url&wrapper=builds' 19:34:01 + resp='{"_class":"hudson.model.ComputerSet","computer":[{"_class":"hudson.model.Hudson$MasterComputer","executors":[],"oneOffExecutors":[]}]} 19:34:01 19:34:01 200' 19:34:01 ++ echo '{"_class":"hudson.model.ComputerSet","computer":[{"_class":"hudson.model.Hudson$MasterComputer","executors":[],"oneOffExecutors":[]}]} 19:34:01 19:34:01 200' 19:34:01 ++ head -n1 19:34:01 + json_data='{"_class":"hudson.model.ComputerSet","computer":[{"_class":"hudson.model.Hudson$MasterComputer","executors":[],"oneOffExecutors":[]}]}' 19:34:01 ++ echo '{"_class":"hudson.model.ComputerSet","computer":[{"_class":"hudson.model.Hudson$MasterComputer","executors":[],"oneOffExecutors":[]}]} 19:34:01 19:34:01 200' 19:34:01 ++ awk 'END {print $NF}' 19:34:01 + status=200 19:34:01 + '[' 200 '!=' 200 ']' 19:34:01 + [[ https://jenkins.opendaylight.org/sandbox == *\j\e\n\k\i\n\s\.*\.\o\r\g ]] 19:34:01 + [[ https://jenkins.opendaylight.org/sandbox == *\j\e\n\k\i\n\s\.*\.\i\o ]] 19:34:01 ++ echo https://jenkins.opendaylight.org/sandbox 19:34:01 ++ sed 's/\/*$//' 19:34:01 ++ awk -F/ '{print $NF}' 19:34:01 + silo=sandbox 19:34:01 + export silo 19:34:01 + builds=(${builds[@]} $(echo "$json_data" | jq -r '.computer[].executors[].currentExecutable.url' | grep -v null | awk -F'/' '{print ENVIRON["silo"] "-" $6 "-" $7}')) 19:34:01 ++ echo '{"_class":"hudson.model.ComputerSet","computer":[{"_class":"hudson.model.Hudson$MasterComputer","executors":[],"oneOffExecutors":[]}]}' 19:34:01 ++ awk -F/ '{print ENVIRON["silo"] "-" $6 "-" $7}' 19:34:01 ++ jq -r '.computer[].executors[].currentExecutable.url' 19:34:01 ++ grep -v null 19:34:01 + [[ releng-builder-openstack-cron-65340 releng-aaa-maven-merge-0.18.x-260 =~ releng-openflowplugin-csit-1node-perf-bulkomatic-only-potassium-400 ]] 19:34:01 + return 1 19:34:01 + echo 'Deleting orphaned stack: releng-openflowplugin-csit-1node-perf-bulkomatic-only-potassium-400' 19:34:01 Deleting orphaned stack: releng-openflowplugin-csit-1node-perf-bulkomatic-only-potassium-400 19:34:01 + lftools openstack --os-cloud vex stack delete --force releng-openflowplugin-csit-1node-perf-bulkomatic-only-potassium-400 19:42:23 Build timed out (after 10 minutes). Marking the build as failed. 19:42:24 /tmp/jenkins9468272423755606007.sh: line 129: 7096 Terminated lftools openstack --os-cloud "${os_cloud}" stack delete --force "${STACK_NAME}" 19:42:24 $ ssh-agent -k 19:42:24 unset SSH_AUTH_SOCK; 19:42:24 unset SSH_AGENT_PID; 19:42:24 echo Agent pid 5459 killed; 19:42:24 [ssh-agent] Stopped. 19:42:24 Build was aborted 19:42:24 [PostBuildScript] - [INFO] Executing post build scripts. 19:42:24 [builder-openstack-cron] $ /bin/bash /tmp/jenkins3441044419972613145.sh 19:42:24 ---> sysstat.sh 19:42:24 [builder-openstack-cron] $ /bin/bash /tmp/jenkins1327029314317201928.sh 19:42:24 ---> package-listing.sh 19:42:24 ++ facter osfamily 19:42:24 ++ tr '[:upper:]' '[:lower:]' 19:42:24 + OS_FAMILY=redhat 19:42:24 + workspace=/w/workspace/builder-openstack-cron 19:42:24 + START_PACKAGES=/tmp/packages_start.txt 19:42:24 + END_PACKAGES=/tmp/packages_end.txt 19:42:24 + DIFF_PACKAGES=/tmp/packages_diff.txt 19:42:24 + PACKAGES=/tmp/packages_start.txt 19:42:24 + '[' /w/workspace/builder-openstack-cron ']' 19:42:24 + PACKAGES=/tmp/packages_end.txt 19:42:24 + case "${OS_FAMILY}" in 19:42:24 + rpm -qa 19:42:24 + sort 19:42:25 + '[' -f /tmp/packages_start.txt ']' 19:42:25 + '[' -f /tmp/packages_end.txt ']' 19:42:25 + diff /tmp/packages_start.txt /tmp/packages_end.txt 19:42:25 + '[' /w/workspace/builder-openstack-cron ']' 19:42:25 + mkdir -p /w/workspace/builder-openstack-cron/archives/ 19:42:25 + cp -f /tmp/packages_diff.txt /tmp/packages_end.txt /tmp/packages_start.txt /w/workspace/builder-openstack-cron/archives/ 19:42:25 [builder-openstack-cron] $ /bin/bash /tmp/jenkins18247291528115287013.sh 19:42:25 ---> capture-instance-metadata.sh 19:42:25 Setup pyenv: 19:42:25 system 19:42:25 3.8.13 19:42:25 3.9.13 19:42:25 3.10.13 19:42:25 * 3.11.7 (set by /w/workspace/builder-openstack-cron/.python-version) 19:42:25 lf-activate-venv(): INFO: Reuse venv:/tmp/venv-qtS7 from file:/tmp/.os_lf_venv 19:42:26 lf-activate-venv(): INFO: Installing: lftools 19:42:34 lf-activate-venv(): INFO: Adding /tmp/venv-qtS7/bin to PATH 19:42:34 INFO: Running in OpenStack, capturing instance metadata 19:42:35 [builder-openstack-cron] $ /bin/bash /tmp/jenkins6891383186427736940.sh 19:42:35 provisioning config files... 19:42:35 Could not find credentials [logs] for builder-openstack-cron #65340 19:42:35 copy managed file [jenkins-log-archives-settings] to file:/w/workspace/builder-openstack-cron@tmp/config12173494651678059901tmp 19:42:35 Regular expression run condition: Expression=[^.*logs-s3.*], Label=[odl-logs-s3-cloudfront-index] 19:42:35 Run condition [Regular expression match] enabling perform for step [Provide Configuration files] 19:42:35 provisioning config files... 19:42:36 copy managed file [jenkins-s3-log-ship] to file:/home/jenkins/.aws/credentials 19:42:36 [EnvInject] - Injecting environment variables from a build step. 19:42:36 [EnvInject] - Injecting as environment variables the properties content 19:42:36 SERVER_ID=logs 19:42:36 19:42:36 [EnvInject] - Variables injected successfully. 19:42:36 [builder-openstack-cron] $ /bin/bash /tmp/jenkins14712918089800173883.sh 19:42:36 ---> create-netrc.sh 19:42:36 WARN: Log server credential not found. 19:42:36 [builder-openstack-cron] $ /bin/bash /tmp/jenkins14928466869501279166.sh 19:42:36 ---> python-tools-install.sh 19:42:36 Setup pyenv: 19:42:36 system 19:42:36 3.8.13 19:42:36 3.9.13 19:42:36 3.10.13 19:42:36 * 3.11.7 (set by /w/workspace/builder-openstack-cron/.python-version) 19:42:36 lf-activate-venv(): INFO: Reuse venv:/tmp/venv-qtS7 from file:/tmp/.os_lf_venv 19:42:37 lf-activate-venv(): INFO: Installing: lftools 19:42:47 lf-activate-venv(): INFO: Adding /tmp/venv-qtS7/bin to PATH 19:42:47 [builder-openstack-cron] $ /bin/bash /tmp/jenkins1444287481116380218.sh 19:42:47 ---> sudo-logs.sh 19:42:47 Archiving 'sudo' log.. 19:42:48 [builder-openstack-cron] $ /bin/bash /tmp/jenkins4513542629266216938.sh 19:42:48 ---> job-cost.sh 19:42:48 Setup pyenv: 19:42:48 system 19:42:48 3.8.13 19:42:48 3.9.13 19:42:48 3.10.13 19:42:48 * 3.11.7 (set by /w/workspace/builder-openstack-cron/.python-version) 19:42:48 lf-activate-venv(): INFO: Reuse venv:/tmp/venv-qtS7 from file:/tmp/.os_lf_venv 19:42:49 lf-activate-venv(): INFO: Installing: zipp==1.1.0 python-openstackclient urllib3~=1.26.15 19:42:54 lf-activate-venv(): INFO: Adding /tmp/venv-qtS7/bin to PATH 19:42:54 INFO: No Stack... 19:42:54 INFO: Retrieving Pricing Info for: v3-standard-2 19:42:55 INFO: Archiving Costs 19:42:55 [builder-openstack-cron] $ /bin/bash -l /tmp/jenkins12307091040093900387.sh 19:42:55 ---> logs-deploy.sh 19:42:55 Setup pyenv: 19:42:55 system 19:42:55 3.8.13 19:42:55 3.9.13 19:42:55 3.10.13 19:42:55 * 3.11.7 (set by /w/workspace/builder-openstack-cron/.python-version) 19:42:55 lf-activate-venv(): INFO: Reuse venv:/tmp/venv-qtS7 from file:/tmp/.os_lf_venv 19:42:56 lf-activate-venv(): INFO: Installing: lftools 19:43:04 lf-activate-venv(): INFO: Adding /tmp/venv-qtS7/bin to PATH 19:43:04 WARNING: Nexus logging server not set 19:43:04 INFO: S3 path logs/releng/vex-yul-odl-jenkins-1/builder-openstack-cron/65340/ 19:43:04 INFO: archiving logs to S3 19:43:06 ---> uname -a: 19:43:06 Linux prd-centos8-builder-2c-2g-30409.novalocal 4.18.0-553.5.1.el8.x86_64 #1 SMP Tue May 21 05:46:01 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 19:43:06 19:43:06 19:43:06 ---> lscpu: 19:43:06 Architecture: x86_64 19:43:06 CPU op-mode(s): 32-bit, 64-bit 19:43:06 Byte Order: Little Endian 19:43:06 CPU(s): 2 19:43:06 On-line CPU(s) list: 0,1 19:43:06 Thread(s) per core: 1 19:43:06 Core(s) per socket: 1 19:43:06 Socket(s): 2 19:43:06 NUMA node(s): 1 19:43:06 Vendor ID: AuthenticAMD 19:43:06 CPU family: 23 19:43:06 Model: 49 19:43:06 Model name: AMD EPYC-Rome Processor 19:43:06 Stepping: 0 19:43:06 CPU MHz: 2799.998 19:43:06 BogoMIPS: 5599.99 19:43:06 Virtualization: AMD-V 19:43:06 Hypervisor vendor: KVM 19:43:06 Virtualization type: full 19:43:06 L1d cache: 32K 19:43:06 L1i cache: 32K 19:43:06 L2 cache: 512K 19:43:06 L3 cache: 16384K 19:43:06 NUMA node0 CPU(s): 0,1 19:43:06 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd ibrs ibpb stibp vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr wbnoinvd arat npt nrip_save umip rdpid arch_capabilities 19:43:06 19:43:06 19:43:06 ---> nproc: 19:43:06 2 19:43:06 19:43:06 19:43:06 ---> df -h: 19:43:06 Filesystem Size Used Avail Use% Mounted on 19:43:06 devtmpfs 3.8G 0 3.8G 0% /dev 19:43:06 tmpfs 3.8G 0 3.8G 0% /dev/shm 19:43:06 tmpfs 3.8G 17M 3.8G 1% /run 19:43:06 tmpfs 3.8G 0 3.8G 0% /sys/fs/cgroup 19:43:06 /dev/vda1 40G 8.4G 32G 21% / 19:43:06 tmpfs 770M 0 770M 0% /run/user/1001 19:43:06 19:43:06 19:43:06 ---> free -m: 19:43:06 total used free shared buff/cache available 19:43:06 Mem: 7697 675 4768 19 2253 6727 19:43:06 Swap: 1023 0 1023 19:43:06 19:43:06 19:43:06 ---> ip addr: 19:43:06 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 19:43:06 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 19:43:06 inet 127.0.0.1/8 scope host lo 19:43:06 valid_lft forever preferred_lft forever 19:43:06 inet6 ::1/128 scope host 19:43:06 valid_lft forever preferred_lft forever 19:43:06 2: eth0: mtu 1458 qdisc mq state UP group default qlen 1000 19:43:06 link/ether fa:16:3e:79:88:65 brd ff:ff:ff:ff:ff:ff 19:43:06 altname enp0s3 19:43:06 altname ens3 19:43:06 inet 10.30.170.65/23 brd 10.30.171.255 scope global dynamic noprefixroute eth0 19:43:06 valid_lft 85712sec preferred_lft 85712sec 19:43:06 inet6 fe80::f816:3eff:fe79:8865/64 scope link 19:43:06 valid_lft forever preferred_lft forever 19:43:06 3: docker0: mtu 1458 qdisc noqueue state DOWN group default 19:43:06 link/ether 02:42:62:16:5a:66 brd ff:ff:ff:ff:ff:ff 19:43:06 inet 10.250.0.254/24 brd 10.250.0.255 scope global docker0 19:43:06 valid_lft forever preferred_lft forever 19:43:06 19:43:06 19:43:06 ---> sar -b -r -n DEV: 19:43:06 Linux 4.18.0-553.5.1.el8.x86_64 (centos-stream-8-builder-5e6ee0fb-b6d9-4116-90d4-35c890e2b0f7.nov) 09/29/2024 _x86_64_ (2 CPU) 19:43:06 19:43:06 19:31:33 LINUX RESTART (2 CPU) 19:43:06 19:43:06 07:32:01 PM tps rtps wtps bread/s bwrtn/s 19:43:06 07:33:01 PM 159.89 33.57 126.32 7529.44 15796.60 19:43:06 07:34:01 PM 107.18 0.25 106.93 18.53 10716.43 19:43:06 07:35:01 PM 16.23 0.65 15.58 69.98 1407.81 19:43:06 07:36:01 PM 2.93 0.00 2.93 0.00 84.52 19:43:06 07:37:01 PM 0.17 0.00 0.17 0.00 1.72 19:43:06 07:38:01 PM 0.38 0.00 0.38 0.00 6.70 19:43:06 07:39:01 PM 0.12 0.00 0.12 0.00 0.90 19:43:06 07:40:01 PM 0.10 0.00 0.10 0.00 0.80 19:43:06 07:41:01 PM 0.27 0.00 0.27 0.00 2.82 19:43:06 07:42:01 PM 0.22 0.00 0.22 0.00 2.92 19:43:06 07:43:01 PM 13.71 0.50 13.21 63.71 237.77 19:43:06 Average: 27.38 3.18 24.20 698.48 2569.19 19:43:06 19:43:06 07:32:01 PM kbmemfree kbavail kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty 19:43:06 07:33:01 PM 5095744 6947540 2786680 35.35 2688 2039412 1060712 11.88 243504 2236528 143880 19:43:06 07:34:01 PM 4965352 6962392 2917072 37.01 2688 2174192 992468 11.11 250272 2312760 30080 19:43:06 07:35:01 PM 4939192 6937728 2943232 37.34 2688 2176248 1033644 11.57 252404 2340528 4 19:43:06 07:36:01 PM 4937832 6936368 2944592 37.36 2688 2176248 1033644 11.57 252404 2341728 4 19:43:06 07:37:01 PM 4932552 6931080 2949872 37.42 2688 2176252 1033644 11.57 252404 2347116 4 19:43:06 07:38:01 PM 4933424 6931952 2949000 37.41 2688 2176252 1033644 11.57 252404 2346416 4 19:43:06 07:39:01 PM 4933536 6932064 2948888 37.41 2688 2176252 1033644 11.57 252404 2346356 4 19:43:06 07:40:01 PM 4933312 6931844 2949112 37.41 2688 2176256 1028884 11.52 252404 2346484 4 19:43:06 07:41:01 PM 4933568 6932112 2948856 37.41 2688 2176256 1028884 11.52 252404 2346400 4 19:43:06 07:42:01 PM 4933632 6932180 2948792 37.41 2688 2176260 1028884 11.52 252404 2346404 4 19:43:06 07:43:01 PM 4900236 6905108 2982188 37.83 2688 2181888 1052308 11.78 409804 2221724 16700 19:43:06 Average: 4948944 6934579 2933480 37.22 2688 2164138 1032760 11.56 265710 2321131 17336 19:43:06 19:43:06 07:32:01 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil 19:43:06 07:33:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 19:43:06 07:33:01 PM eth0 292.30 177.03 1692.46 60.56 0.00 0.00 0.00 0.00 19:43:06 07:33:01 PM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 19:43:06 07:34:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 19:43:06 07:34:01 PM eth0 64.91 55.42 616.72 9.45 0.00 0.00 0.00 0.00 19:43:06 07:34:01 PM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 19:43:06 07:35:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 19:43:06 07:35:01 PM eth0 2.92 2.42 1.44 0.69 0.00 0.00 0.00 0.00 19:43:06 07:35:01 PM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 19:43:06 07:36:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 19:43:06 07:36:01 PM eth0 0.62 0.45 0.47 0.15 0.00 0.00 0.00 0.00 19:43:06 07:36:01 PM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 19:43:06 07:37:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 19:43:06 07:37:01 PM eth0 0.47 0.40 0.34 0.15 0.00 0.00 0.00 0.00 19:43:06 07:37:01 PM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 19:43:06 07:38:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 19:43:06 07:38:01 PM eth0 0.67 0.48 0.47 0.21 0.00 0.00 0.00 0.00 19:43:06 07:38:01 PM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 19:43:06 07:39:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 19:43:06 07:39:01 PM eth0 0.45 0.38 0.34 0.14 0.00 0.00 0.00 0.00 19:43:06 07:39:01 PM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 19:43:06 07:40:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 19:43:06 07:40:01 PM eth0 0.43 0.38 0.33 0.14 0.00 0.00 0.00 0.00 19:43:06 07:40:01 PM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 19:43:06 07:41:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 19:43:06 07:41:01 PM eth0 0.47 0.40 0.34 0.15 0.00 0.00 0.00 0.00 19:43:06 07:41:01 PM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 19:43:06 07:42:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 19:43:06 07:42:01 PM eth0 0.47 0.38 0.34 0.14 0.00 0.00 0.00 0.00 19:43:06 07:42:01 PM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 19:43:06 07:43:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 19:43:06 07:43:01 PM eth0 18.76 16.86 15.46 12.27 0.00 0.00 0.00 0.00 19:43:06 07:43:01 PM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 19:43:06 Average: lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 19:43:06 Average: eth0 34.77 23.15 211.73 7.64 0.00 0.00 0.00 0.00 19:43:06 Average: docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 19:43:06 19:43:06 19:43:06 ---> sar -P ALL: 19:43:06 Linux 4.18.0-553.5.1.el8.x86_64 (centos-stream-8-builder-5e6ee0fb-b6d9-4116-90d4-35c890e2b0f7.nov) 09/29/2024 _x86_64_ (2 CPU) 19:43:06 19:43:06 19:31:33 LINUX RESTART (2 CPU) 19:43:06 19:43:06 07:32:01 PM CPU %user %nice %system %iowait %steal %idle 19:43:06 07:33:01 PM all 41.19 0.00 7.87 5.26 0.09 45.58 19:43:06 07:33:01 PM 0 48.79 0.00 6.63 4.59 0.08 39.90 19:43:06 07:33:01 PM 1 33.56 0.00 9.12 5.94 0.10 51.28 19:43:06 07:34:01 PM all 34.45 0.00 5.16 3.15 0.09 57.15 19:43:06 07:34:01 PM 0 35.10 0.00 5.57 3.20 0.08 56.04 19:43:06 07:34:01 PM 1 33.80 0.00 4.75 3.10 0.10 58.25 19:43:06 07:35:01 PM all 1.22 0.00 0.39 0.40 0.03 97.96 19:43:06 07:35:01 PM 0 2.19 0.00 0.48 0.40 0.02 96.91 19:43:06 07:35:01 PM 1 0.25 0.00 0.30 0.40 0.03 99.01 19:43:06 07:36:01 PM all 0.38 0.00 0.06 0.03 0.04 99.50 19:43:06 07:36:01 PM 0 0.27 0.00 0.07 0.00 0.05 99.62 19:43:06 07:36:01 PM 1 0.48 0.00 0.05 0.05 0.03 99.38 19:43:06 07:37:01 PM all 0.25 0.00 0.09 0.00 0.03 99.63 19:43:06 07:37:01 PM 0 0.30 0.00 0.13 0.00 0.02 99.55 19:43:06 07:37:01 PM 1 0.20 0.00 0.05 0.00 0.03 99.72 19:43:06 07:38:01 PM all 0.39 0.00 0.08 0.00 0.03 99.49 19:43:06 07:38:01 PM 0 0.68 0.00 0.08 0.00 0.02 99.22 19:43:06 07:38:01 PM 1 0.10 0.00 0.08 0.00 0.05 99.77 19:43:06 07:39:01 PM all 0.33 0.00 0.07 0.00 0.03 99.57 19:43:06 07:39:01 PM 0 0.32 0.00 0.05 0.00 0.02 99.62 19:43:06 07:39:01 PM 1 0.33 0.00 0.08 0.00 0.05 99.53 19:43:06 07:40:01 PM all 0.36 0.00 0.06 0.00 0.03 99.55 19:43:06 07:40:01 PM 0 0.17 0.00 0.05 0.00 0.02 99.77 19:43:06 07:40:01 PM 1 0.55 0.00 0.07 0.00 0.05 99.33 19:43:06 07:41:01 PM all 0.37 0.00 0.08 0.00 0.03 99.52 19:43:06 07:41:01 PM 0 0.18 0.00 0.08 0.00 0.03 99.70 19:43:06 07:41:01 PM 1 0.55 0.00 0.08 0.00 0.02 99.35 19:43:06 07:42:01 PM all 0.24 0.00 0.07 0.00 0.03 99.66 19:43:06 07:42:01 PM 0 0.22 0.00 0.08 0.00 0.03 99.67 19:43:06 07:42:01 PM 1 0.27 0.00 0.05 0.00 0.03 99.65 19:43:06 07:43:01 PM all 23.44 0.00 2.81 0.10 0.07 73.58 19:43:06 07:43:01 PM 0 34.70 0.00 3.16 0.13 0.07 61.95 19:43:06 07:43:01 PM 1 12.18 0.00 2.46 0.07 0.07 85.23 19:43:06 19:43:06 Average: CPU %user %nice %system %iowait %steal %idle 19:43:06 Average: all 9.32 0.00 1.52 0.81 0.05 88.30 19:43:06 Average: 0 11.17 0.00 1.49 0.76 0.04 86.54 19:43:06 Average: 1 7.48 0.00 1.55 0.87 0.05 90.05 19:43:06 19:43:06 19:43:06