Started by upstream project "integration-distribution-mri-test-vanadium" build number 25 originally caused by: Started by timer Running as SYSTEM [EnvInject] - Loading node environment variables. Building remotely on prd-centos8-robot-2c-8g-1588 (centos8-robot-2c-8g) in workspace /w/workspace/controller-csit-3node-clustering-ask-all-vanadium [ssh-agent] Looking for ssh-agent implementation... [ssh-agent] Exec ssh-agent (binary ssh-agent on a remote machine) $ ssh-agent SSH_AUTH_SOCK=/tmp/ssh-qkToD4qGOoF9/agent.5286 SSH_AGENT_PID=5287 [ssh-agent] Started. Running ssh-add (command line suppressed) Identity added: /w/workspace/controller-csit-3node-clustering-ask-all-vanadium@tmp/private_key_17526360159873214317.key (/w/workspace/controller-csit-3node-clustering-ask-all-vanadium@tmp/private_key_17526360159873214317.key) [ssh-agent] Using credentials jenkins (Release Engineering Jenkins Key) The recommended git tool is: NONE using credential opendaylight-jenkins-ssh Wiping out workspace first. Cloning the remote Git repository Cloning repository git://devvexx.opendaylight.org/mirror/integration/test > git init /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test # timeout=10 Fetching upstream changes from git://devvexx.opendaylight.org/mirror/integration/test > git --version # timeout=10 > git --version # 'git version 2.43.0' using GIT_SSH to set credentials Release Engineering Jenkins Key [INFO] Currently running in a labeled security context [INFO] Currently SELinux is 'enforcing' on the host > /usr/bin/chcon --type=ssh_home_t /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test@tmp/jenkins-gitclient-ssh12750793538826220384.key Verifying host key using known hosts file You're using 'Known hosts file' strategy to verify ssh host keys, but your known_hosts file does not exist, please go to 'Manage Jenkins' -> 'Security' -> 'Git Host Key Verification Configuration' and configure host key verification. > git fetch --tags --force --progress -- git://devvexx.opendaylight.org/mirror/integration/test +refs/heads/*:refs/remotes/origin/* # timeout=10 > git config remote.origin.url git://devvexx.opendaylight.org/mirror/integration/test # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git config remote.origin.url git://devvexx.opendaylight.org/mirror/integration/test # timeout=10 Fetching upstream changes from git://devvexx.opendaylight.org/mirror/integration/test using GIT_SSH to set credentials Release Engineering Jenkins Key [INFO] Currently running in a labeled security context [INFO] Currently SELinux is 'enforcing' on the host > /usr/bin/chcon --type=ssh_home_t /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test@tmp/jenkins-gitclient-ssh5370154541234616868.key Verifying host key using known hosts file You're using 'Known hosts file' strategy to verify ssh host keys, but your known_hosts file does not exist, please go to 'Manage Jenkins' -> 'Security' -> 'Git Host Key Verification Configuration' and configure host key verification. > git fetch --tags --force --progress -- git://devvexx.opendaylight.org/mirror/integration/test master # timeout=10 > git rev-parse FETCH_HEAD^{commit} # timeout=10 Checking out Revision cc140aba40d20c279ad305eb08204917e9a97348 (origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f cc140aba40d20c279ad305eb08204917e9a97348 # timeout=10 Commit message: "Add bgpcep variables for vanadium" > git rev-parse FETCH_HEAD^{commit} # timeout=10 > git rev-list --no-walk 2791c5dea1951b0af9405c9ac43be0884a2c5414 # timeout=10 No emails were triggered. provisioning config files... copy managed file [npmrc] to file:/home/jenkins/.npmrc copy managed file [pipconf] to file:/home/jenkins/.config/pip/pip.conf copy managed file [clouds-yaml] to file:/home/jenkins/.config/openstack/clouds.yaml [controller-csit-3node-clustering-ask-all-vanadium] $ /bin/bash /tmp/jenkins5810741745364791989.sh ---> python-tools-install.sh Setup pyenv: system * 3.8.13 (set by /opt/pyenv/version) * 3.9.13 (set by /opt/pyenv/version) * 3.10.13 (set by /opt/pyenv/version) * 3.11.7 (set by /opt/pyenv/version) lf-activate-venv(): INFO: Creating python3 venv at /tmp/venv-9I7X lf-activate-venv(): INFO: Save venv in file: /tmp/.os_lf_venv lf-activate-venv(): INFO: Installing base packages (pip, setuptools, virtualenv) lf-activate-venv(): INFO: Attempting to install with network-safe options... lf-activate-venv(): INFO: Base packages installed successfully lf-activate-venv(): INFO: Installing additional packages: lftools lf-activate-venv(): INFO: Adding /tmp/venv-9I7X/bin to PATH Generating Requirements File ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. httplib2 0.30.2 requires pyparsing<4,>=3.0.4, but you have pyparsing 2.4.7 which is incompatible. Python 3.11.7 pip 25.3 from /tmp/venv-9I7X/lib/python3.11/site-packages/pip (python 3.11) appdirs==1.4.4 argcomplete==3.6.3 aspy.yaml==1.3.0 attrs==25.4.0 autopage==0.5.2 beautifulsoup4==4.14.3 boto3==1.42.30 botocore==1.42.30 bs4==0.0.2 certifi==2026.1.4 cffi==2.0.0 cfgv==3.5.0 chardet==5.2.0 charset-normalizer==3.4.4 click==8.3.1 cliff==4.13.1 cmd2==3.1.0 cryptography==3.3.2 debtcollector==3.0.0 decorator==5.2.1 defusedxml==0.7.1 Deprecated==1.3.1 distlib==0.4.0 dnspython==2.8.0 docker==7.1.0 dogpile.cache==1.5.0 durationpy==0.10 email-validator==2.3.0 filelock==3.20.3 future==1.0.0 gitdb==4.0.12 GitPython==3.1.46 httplib2==0.30.2 identify==2.6.16 idna==3.11 importlib-resources==1.5.0 iso8601==2.1.0 Jinja2==3.1.6 jmespath==1.0.1 jsonpatch==1.33 jsonpointer==3.0.0 jsonschema==4.26.0 jsonschema-specifications==2025.9.1 keystoneauth1==5.12.0 kubernetes==35.0.0 lftools==0.37.21 lxml==6.0.2 markdown-it-py==4.0.0 MarkupSafe==3.0.3 mdurl==0.1.2 msgpack==1.1.2 multi_key_dict==2.0.3 munch==4.0.0 netaddr==1.3.0 niet==1.4.2 nodeenv==1.10.0 oauth2client==4.1.3 oauthlib==3.3.1 openstacksdk==4.8.0 os-service-types==1.8.2 osc-lib==4.3.0 oslo.config==10.2.0 oslo.context==6.2.0 oslo.i18n==6.7.1 oslo.log==8.0.0 oslo.serialization==5.9.0 oslo.utils==9.2.0 packaging==25.0 pbr==7.0.3 platformdirs==4.5.1 prettytable==3.17.0 psutil==7.2.1 pyasn1==0.6.2 pyasn1_modules==0.4.2 pycparser==2.23 pygerrit2==2.0.15 PyGithub==2.8.1 Pygments==2.19.2 PyJWT==2.10.1 PyNaCl==1.6.2 pyparsing==2.4.7 pyperclip==1.11.0 pyrsistent==0.20.0 python-cinderclient==9.8.0 python-dateutil==2.9.0.post0 python-heatclient==4.3.0 python-jenkins==1.8.3 python-keystoneclient==5.7.0 python-magnumclient==4.9.0 python-openstackclient==8.3.0 python-swiftclient==4.9.0 PyYAML==6.0.3 referencing==0.37.0 requests==2.32.5 requests-oauthlib==2.0.0 requestsexceptions==1.4.0 rfc3986==2.0.0 rich==14.2.0 rich-argparse==1.7.2 rpds-py==0.30.0 rsa==4.9.1 ruamel.yaml==0.19.1 ruamel.yaml.clib==0.2.15 s3transfer==0.16.0 simplejson==3.20.2 six==1.17.0 smmap==5.0.2 soupsieve==2.8.1 stevedore==5.6.0 tabulate==0.9.0 toml==0.10.2 tomlkit==0.14.0 tqdm==4.67.1 typing_extensions==4.15.0 tzdata==2025.3 urllib3==1.26.20 virtualenv==20.36.1 wcwidth==0.2.14 websocket-client==1.9.0 wrapt==2.0.1 xdg==6.0.0 xmltodict==1.0.2 yq==3.4.3 [EnvInject] - Injecting environment variables from a build step. [EnvInject] - Injecting as environment variables the properties content OS_STACK_TEMPLATE=csit-2-instance-type.yaml OS_CLOUD=vex OS_STACK_NAME=releng-controller-csit-3node-clustering-ask-all-vanadium-25 OS_STACK_TEMPLATE_DIR=openstack-hot [EnvInject] - Variables injected successfully. provisioning config files... copy managed file [clouds-yaml] to file:/home/jenkins/.config/openstack/clouds.yaml [controller-csit-3node-clustering-ask-all-vanadium] $ /bin/bash /tmp/jenkins9691839747739728869.sh ---> Create parameters file for OpenStack HOT OpenStack Heat parameters generated ----------------------------------- parameters: vm_0_count: '3' vm_0_flavor: 'v3-standard-4' vm_0_image: 'ZZCI - Ubuntu 22.04 - builder - x86_64 - 20260109-084045.554' vm_1_count: '0' vm_1_flavor: 'v3-standard-2' vm_1_image: 'ZZCI - Ubuntu 22.04 - mininet-ovs-217 - x86_64 - 20250917-133034.654' job_name: '34030-25' silo: 'releng' [controller-csit-3node-clustering-ask-all-vanadium] $ /bin/bash -l /tmp/jenkins1938102070093363241.sh ---> Create HEAT stack + source /home/jenkins/lf-env.sh + lf-activate-venv --python python3 'lftools[openstack]' kubernetes niet python-heatclient python-openstackclient python-magnumclient urllib3~=1.26.15 yq ++ mktemp -d /tmp/venv-XXXX + lf_venv=/tmp/venv-OY7H + local venv_file=/tmp/.os_lf_venv + local python=python3 + local options + local set_path=true + local install_args= ++ getopt -o np:v: -l no-path,system-site-packages,python:,venv-file: -n lf-activate-venv -- --python python3 'lftools[openstack]' kubernetes niet python-heatclient python-openstackclient python-magnumclient urllib3~=1.26.15 yq + options=' --python '\''python3'\'' -- '\''lftools[openstack]'\'' '\''kubernetes'\'' '\''niet'\'' '\''python-heatclient'\'' '\''python-openstackclient'\'' '\''python-magnumclient'\'' '\''urllib3~=1.26.15'\'' '\''yq'\''' + eval set -- ' --python '\''python3'\'' -- '\''lftools[openstack]'\'' '\''kubernetes'\'' '\''niet'\'' '\''python-heatclient'\'' '\''python-openstackclient'\'' '\''python-magnumclient'\'' '\''urllib3~=1.26.15'\'' '\''yq'\''' ++ set -- --python python3 -- 'lftools[openstack]' kubernetes niet python-heatclient python-openstackclient python-magnumclient urllib3~=1.26.15 yq + true + case $1 in + python=python3 + shift 2 + true + case $1 in + shift + break + case $python in + local pkg_list= + [[ -d /opt/pyenv ]] + echo 'Setup pyenv:' Setup pyenv: + export PYENV_ROOT=/opt/pyenv + PYENV_ROOT=/opt/pyenv + export PATH=/opt/pyenv/bin:/home/jenkins/.local/bin:/home/jenkins/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin + PATH=/opt/pyenv/bin:/home/jenkins/.local/bin:/home/jenkins/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin + pyenv versions system 3.8.13 3.9.13 3.10.13 * 3.11.7 (set by /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/.python-version) + command -v pyenv ++ pyenv init - --no-rehash + eval 'PATH="$(bash --norc -ec '\''IFS=:; paths=($PATH); for i in ${!paths[@]}; do if [[ ${paths[i]} == "'\'''\''/opt/pyenv/shims'\'''\''" ]]; then unset '\''\'\'''\''paths[i]'\''\'\'''\''; fi; done; echo "${paths[*]}"'\'')" export PATH="/opt/pyenv/shims:${PATH}" export PYENV_SHELL=bash source '\''/opt/pyenv/libexec/../completions/pyenv.bash'\'' pyenv() { local command command="${1:-}" if [ "$#" -gt 0 ]; then shift fi case "$command" in rehash|shell) eval "$(pyenv "sh-$command" "$@")" ;; *) command pyenv "$command" "$@" ;; esac }' +++ bash --norc -ec 'IFS=:; paths=($PATH); for i in ${!paths[@]}; do if [[ ${paths[i]} == "/opt/pyenv/shims" ]]; then unset '\''paths[i]'\''; fi; done; echo "${paths[*]}"' ++ PATH=/opt/pyenv/bin:/home/jenkins/.local/bin:/home/jenkins/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin ++ export PATH=/opt/pyenv/shims:/opt/pyenv/bin:/home/jenkins/.local/bin:/home/jenkins/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin ++ PATH=/opt/pyenv/shims:/opt/pyenv/bin:/home/jenkins/.local/bin:/home/jenkins/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin ++ export PYENV_SHELL=bash ++ PYENV_SHELL=bash ++ source /opt/pyenv/libexec/../completions/pyenv.bash +++ complete -F _pyenv pyenv ++ lf-pyver python3 ++ local py_version_xy=python3 ++ local py_version_xyz= ++ awk '{ print $1 }' ++ pyenv versions ++ local command ++ command=versions ++ '[' 1 -gt 0 ']' ++ shift ++ case "$command" in ++ command pyenv versions ++ pyenv versions ++ sed 's/^[ *]* //' ++ grep -E '^[0-9.]*[0-9]$' ++ [[ ! -s /tmp/.pyenv_versions ]] +++ tail -n 1 +++ grep '^3' /tmp/.pyenv_versions +++ sort -V ++ py_version_xyz=3.11.7 ++ [[ -z 3.11.7 ]] ++ echo 3.11.7 ++ return 0 + pyenv local 3.11.7 + local command + command=local + '[' 2 -gt 0 ']' + shift + case "$command" in + command pyenv local 3.11.7 + pyenv local 3.11.7 + for arg in "$@" + case $arg in + pkg_list+='lftools[openstack] ' + for arg in "$@" + case $arg in + pkg_list+='kubernetes ' + for arg in "$@" + case $arg in + pkg_list+='niet ' + for arg in "$@" + case $arg in + pkg_list+='python-heatclient ' + for arg in "$@" + case $arg in + pkg_list+='python-openstackclient ' + for arg in "$@" + case $arg in + pkg_list+='python-magnumclient ' + for arg in "$@" + case $arg in + pkg_list+='urllib3~=1.26.15 ' + for arg in "$@" + case $arg in + pkg_list+='yq ' + [[ -f /tmp/.os_lf_venv ]] ++ cat /tmp/.os_lf_venv + lf_venv=/tmp/venv-9I7X + echo 'lf-activate-venv(): INFO: Reuse venv:/tmp/venv-9I7X from' file:/tmp/.os_lf_venv lf-activate-venv(): INFO: Reuse venv:/tmp/venv-9I7X from file:/tmp/.os_lf_venv + echo 'lf-activate-venv(): INFO: Installing base packages (pip, setuptools, virtualenv)' lf-activate-venv(): INFO: Installing base packages (pip, setuptools, virtualenv) + local 'pip_opts=--upgrade --quiet' + pip_opts='--upgrade --quiet --trusted-host pypi.org' + pip_opts='--upgrade --quiet --trusted-host pypi.org --trusted-host files.pythonhosted.org' + pip_opts='--upgrade --quiet --trusted-host pypi.org --trusted-host files.pythonhosted.org --trusted-host pypi.python.org' + [[ -n '' ]] + [[ -n '' ]] + echo 'lf-activate-venv(): INFO: Attempting to install with network-safe options...' lf-activate-venv(): INFO: Attempting to install with network-safe options... + /tmp/venv-9I7X/bin/python3 -m pip install --upgrade --quiet --trusted-host pypi.org --trusted-host files.pythonhosted.org --trusted-host pypi.python.org pip 'setuptools<66' virtualenv + echo 'lf-activate-venv(): INFO: Base packages installed successfully' lf-activate-venv(): INFO: Base packages installed successfully + [[ -z lftools[openstack] kubernetes niet python-heatclient python-openstackclient python-magnumclient urllib3~=1.26.15 yq ]] + echo 'lf-activate-venv(): INFO: Installing additional packages: lftools[openstack] kubernetes niet python-heatclient python-openstackclient python-magnumclient urllib3~=1.26.15 yq ' lf-activate-venv(): INFO: Installing additional packages: lftools[openstack] kubernetes niet python-heatclient python-openstackclient python-magnumclient urllib3~=1.26.15 yq + /tmp/venv-9I7X/bin/python3 -m pip install --upgrade --quiet --trusted-host pypi.org --trusted-host files.pythonhosted.org --trusted-host pypi.python.org --upgrade-strategy eager 'lftools[openstack]' kubernetes niet python-heatclient python-openstackclient python-magnumclient urllib3~=1.26.15 yq + type python3 + true + echo 'lf-activate-venv(): INFO: Adding /tmp/venv-9I7X/bin to PATH' lf-activate-venv(): INFO: Adding /tmp/venv-9I7X/bin to PATH + PATH=/tmp/venv-9I7X/bin:/opt/pyenv/shims:/opt/pyenv/bin:/home/jenkins/.local/bin:/home/jenkins/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin + return 0 + openstack --os-cloud vex limits show --absolute +--------------------------+---------+ | Name | Value | +--------------------------+---------+ | maxTotalInstances | -1 | | maxTotalCores | 450 | | maxTotalRAMSize | 1000000 | | maxServerMeta | 128 | | maxImageMeta | 128 | | maxPersonality | 5 | | maxPersonalitySize | 10240 | | maxTotalKeypairs | 100 | | maxServerGroups | 10 | | maxServerGroupMembers | 10 | | maxTotalFloatingIps | -1 | | maxSecurityGroups | -1 | | maxSecurityGroupRules | -1 | | totalRAMUsed | 647168 | | totalCoresUsed | 158 | | totalInstancesUsed | 52 | | totalFloatingIpsUsed | 0 | | totalSecurityGroupsUsed | 0 | | totalServerGroupsUsed | 0 | | maxTotalVolumes | -1 | | maxTotalSnapshots | 10 | | maxTotalVolumeGigabytes | 4096 | | maxTotalBackups | 10 | | maxTotalBackupGigabytes | 1000 | | totalVolumesUsed | 0 | | totalGigabytesUsed | 0 | | totalSnapshotsUsed | 0 | | totalBackupsUsed | 0 | | totalBackupGigabytesUsed | 0 | +--------------------------+---------+ + pushd /opt/ciman/openstack-hot /opt/ciman/openstack-hot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium + lftools openstack --os-cloud vex stack create releng-controller-csit-3node-clustering-ask-all-vanadium-25 csit-2-instance-type.yaml /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/stack-parameters.yaml Creating stack releng-controller-csit-3node-clustering-ask-all-vanadium-25 Waiting to initialize infrastructure... Stack initialization successful. ------------------------------------ Stack Details ------------------------------------ {'added': None, 'capabilities': [], 'created_at': '2026-01-17T23:41:38Z', 'deleted': None, 'deleted_at': None, 'description': 'No description', 'environment': None, 'environment_files': None, 'files': None, 'files_container': None, 'id': 'b2c27ce9-d85e-41ab-b7c1-77b78d01337b', 'is_rollback_disabled': True, 'links': [{'href': 'https://orchestration.public.mtl1.vexxhost.net/v1/12c36e260d8e4bb2913965203b1b491f/stacks/releng-controller-csit-3node-clustering-ask-all-vanadium-25/b2c27ce9-d85e-41ab-b7c1-77b78d01337b', 'rel': 'self'}], 'location': Munch({'cloud': 'vex', 'region_name': 'ca-ymq-1', 'zone': None, 'project': Munch({'id': '12c36e260d8e4bb2913965203b1b491f', 'name': '61975f2c-7c17-4d69-82fa-c3ae420ad6fd', 'domain_id': None, 'domain_name': 'Default'})}), 'name': 'releng-controller-csit-3node-clustering-ask-all-vanadium-25', 'notification_topics': [], 'outputs': [{'description': 'IP addresses of the 1st vm types', 'output_key': 'vm_0_ips', 'output_value': ['10.30.170.104', '10.30.171.129', '10.30.170.156']}, {'description': 'IP addresses of the 2nd vm types', 'output_key': 'vm_1_ips', 'output_value': []}], 'owner_id': ****, 'parameters': {'OS::project_id': '12c36e260d8e4bb2913965203b1b491f', 'OS::stack_id': 'b2c27ce9-d85e-41ab-b7c1-77b78d01337b', 'OS::stack_name': 'releng-controller-csit-3node-clustering-ask-all-vanadium-25', 'job_name': '34030-25', 'silo': 'releng', 'vm_0_count': '3', 'vm_0_flavor': 'v3-standard-4', 'vm_0_image': 'ZZCI - Ubuntu 22.04 - builder - x86_64 - ' '20260109-084045.554', 'vm_1_count': '0', 'vm_1_flavor': 'v3-standard-2', 'vm_1_image': 'ZZCI - Ubuntu 22.04 - mininet-ovs-217 - x86_64 ' '- 20250917-133034.654'}, 'parent_id': None, 'replaced': None, 'status': 'CREATE_COMPLETE', 'status_reason': 'Stack CREATE completed successfully', 'tags': [], 'template': None, 'template_description': 'No description', 'template_url': None, 'timeout_mins': 15, 'unchanged': None, 'updated': None, 'updated_at': None, 'user_project_id': 'e8f1d6f692d04f248f5c95bbeb15e50c'} ------------------------------------ + popd /w/workspace/controller-csit-3node-clustering-ask-all-vanadium [controller-csit-3node-clustering-ask-all-vanadium] $ /bin/bash -l /tmp/jenkins5209227387157596989.sh ---> Copy SSH public keys to CSIT lab Setup pyenv: system 3.8.13 3.9.13 3.10.13 * 3.11.7 (set by /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/.python-version) lf-activate-venv(): INFO: Reuse venv:/tmp/venv-9I7X from file:/tmp/.os_lf_venv lf-activate-venv(): INFO: Installing base packages (pip, setuptools, virtualenv) lf-activate-venv(): INFO: Attempting to install with network-safe options... lf-activate-venv(): INFO: Base packages installed successfully lf-activate-venv(): INFO: Installing additional packages: lftools[openstack] kubernetes python-heatclient python-openstackclient urllib3~=1.26.15 lf-activate-venv(): INFO: Adding /tmp/venv-9I7X/bin to PATH SSH not responding on 10.30.170.104. Retrying in 10 seconds... SSH not responding on 10.30.170.156. Retrying in 10 seconds... SSH not responding on 10.30.171.129. Retrying in 10 seconds... Ping to 10.30.170.104 successful. Ping to 10.30.170.156 successful. Ping to 10.30.171.129 successful. SSH not responding on 10.30.170.156. Retrying in 10 seconds... SSH not responding on 10.30.171.129. Retrying in 10 seconds... Warning: Permanently added '10.30.170.104' (ECDSA) to the list of known hosts. releng-34030-25-0-builder-0 Successfully copied public keys to slave 10.30.170.104 Process 6499 ready. Ping to 10.30.170.156 successful. Ping to 10.30.171.129 successful. Warning: Permanently added '10.30.170.156' (ECDSA) to the list of known hosts. Warning: Permanently added '10.30.171.129' (ECDSA) to the list of known hosts. releng-34030-25-0-builder-2 Successfully copied public keys to slave 10.30.170.156 releng-34030-25-0-builder-1 Successfully copied public keys to slave 10.30.171.129 Process 6500 ready. Process 6501 ready. SSH ready on all stack servers. [controller-csit-3node-clustering-ask-all-vanadium] $ /bin/bash -l /tmp/jenkins14031658841935039455.sh Setup pyenv: system 3.8.13 3.9.13 3.10.13 * 3.11.7 (set by /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/.python-version) lf-activate-venv(): INFO: Creating python3 venv at /tmp/venv-i5C8 lf-activate-venv(): INFO: Save venv in file: /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/.robot_venv lf-activate-venv(): INFO: Installing base packages (pip, setuptools, virtualenv) lf-activate-venv(): INFO: Attempting to install with network-safe options... lf-activate-venv(): INFO: Base packages installed successfully lf-activate-venv(): INFO: Installing additional packages: setuptools wheel lf-activate-venv(): INFO: Adding /tmp/venv-i5C8/bin to PATH + echo 'Installing Python Requirements' Installing Python Requirements + cat + python -m pip install -r requirements.txt Looking in indexes: https://nexus3.opendaylight.org/repository/PyPi/simple Collecting docker-py (from -r requirements.txt (line 1)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/docker-py/1.10.6/docker_py-1.10.6-py2.py3-none-any.whl (50 kB) Collecting ipaddr (from -r requirements.txt (line 2)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/ipaddr/2.2.0/ipaddr-2.2.0.tar.gz (26 kB) Installing build dependencies: started Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started Getting requirements to build wheel: finished with status 'done' Preparing metadata (pyproject.toml): started Preparing metadata (pyproject.toml): finished with status 'done' Collecting netaddr (from -r requirements.txt (line 3)) Using cached https://nexus3.opendaylight.org/repository/PyPi/packages/netaddr/1.3.0/netaddr-1.3.0-py3-none-any.whl (2.3 MB) Collecting netifaces (from -r requirements.txt (line 4)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/netifaces/0.11.0/netifaces-0.11.0.tar.gz (30 kB) Installing build dependencies: started Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started Getting requirements to build wheel: finished with status 'done' Preparing metadata (pyproject.toml): started Preparing metadata (pyproject.toml): finished with status 'done' Collecting pyhocon (from -r requirements.txt (line 5)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/pyhocon/0.3.61/pyhocon-0.3.61-py3-none-any.whl (25 kB) Collecting requests (from -r requirements.txt (line 6)) Using cached https://nexus3.opendaylight.org/repository/PyPi/packages/requests/2.32.5/requests-2.32.5-py3-none-any.whl (64 kB) Collecting robotframework (from -r requirements.txt (line 7)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/robotframework/7.4.1/robotframework-7.4.1-py3-none-any.whl (806 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 806.9/806.9 kB 32.4 MB/s 0:00:00 Collecting robotframework-httplibrary (from -r requirements.txt (line 8)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/robotframework-httplibrary/0.4.2/robotframework-httplibrary-0.4.2.tar.gz (9.1 kB) Installing build dependencies: started Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started Getting requirements to build wheel: finished with status 'done' Preparing metadata (pyproject.toml): started Preparing metadata (pyproject.toml): finished with status 'done' Collecting robotframework-requests==0.9.7 (from -r requirements.txt (line 9)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/robotframework-requests/0.9.7/robotframework_requests-0.9.7-py3-none-any.whl (21 kB) Collecting robotframework-selenium2library (from -r requirements.txt (line 10)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/robotframework-selenium2library/3.0.0/robotframework_selenium2library-3.0.0-py2.py3-none-any.whl (6.2 kB) Collecting robotframework-sshlibrary==3.8.0 (from -r requirements.txt (line 11)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/robotframework-sshlibrary/3.8.0/robotframework-sshlibrary-3.8.0.tar.gz (51 kB) Installing build dependencies: started Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started Getting requirements to build wheel: finished with status 'done' Preparing metadata (pyproject.toml): started Preparing metadata (pyproject.toml): finished with status 'done' Collecting scapy (from -r requirements.txt (line 12)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/scapy/2.7.0/scapy-2.7.0-py3-none-any.whl (2.6 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.6/2.6 MB 40.5 MB/s 0:00:00 Collecting jsonpath-rw (from -r requirements.txt (line 15)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/jsonpath-rw/1.4.0/jsonpath-rw-1.4.0.tar.gz (13 kB) Installing build dependencies: started Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started Getting requirements to build wheel: finished with status 'done' Preparing metadata (pyproject.toml): started Preparing metadata (pyproject.toml): finished with status 'done' Collecting elasticsearch (from -r requirements.txt (line 18)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/elasticsearch/9.2.1/elasticsearch-9.2.1-py3-none-any.whl (963 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 963.6/963.6 kB 35.4 MB/s 0:00:00 Collecting elasticsearch-dsl (from -r requirements.txt (line 19)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/elasticsearch-dsl/8.18.0/elasticsearch_dsl-8.18.0-py3-none-any.whl (10 kB) Collecting pyangbind (from -r requirements.txt (line 22)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/pyangbind/0.8.7/pyangbind-0.8.7-py3-none-any.whl (52 kB) Collecting isodate (from -r requirements.txt (line 25)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/isodate/0.7.2/isodate-0.7.2-py3-none-any.whl (22 kB) Collecting jmespath (from -r requirements.txt (line 28)) Using cached https://nexus3.opendaylight.org/repository/PyPi/packages/jmespath/1.0.1/jmespath-1.0.1-py3-none-any.whl (20 kB) Collecting jsonpatch (from -r requirements.txt (line 31)) Using cached https://nexus3.opendaylight.org/repository/PyPi/packages/jsonpatch/1.33/jsonpatch-1.33-py2.py3-none-any.whl (12 kB) Collecting paramiko>=1.15.3 (from robotframework-sshlibrary==3.8.0->-r requirements.txt (line 11)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/paramiko/4.0.0/paramiko-4.0.0-py3-none-any.whl (223 kB) Collecting scp>=0.13.0 (from robotframework-sshlibrary==3.8.0->-r requirements.txt (line 11)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/scp/0.15.0/scp-0.15.0-py2.py3-none-any.whl (8.8 kB) Collecting docker-pycreds>=0.2.1 (from docker-py->-r requirements.txt (line 1)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/docker-pycreds/0.4.0/docker_pycreds-0.4.0-py2.py3-none-any.whl (9.0 kB) Collecting six>=1.4.0 (from docker-py->-r requirements.txt (line 1)) Using cached https://nexus3.opendaylight.org/repository/PyPi/packages/six/1.17.0/six-1.17.0-py2.py3-none-any.whl (11 kB) Collecting websocket-client>=0.32.0 (from docker-py->-r requirements.txt (line 1)) Using cached https://nexus3.opendaylight.org/repository/PyPi/packages/websocket-client/1.9.0/websocket_client-1.9.0-py3-none-any.whl (82 kB) Collecting pyparsing<4,>=2 (from pyhocon->-r requirements.txt (line 5)) Using cached https://nexus3.opendaylight.org/repository/PyPi/packages/pyparsing/3.3.1/pyparsing-3.3.1-py3-none-any.whl (121 kB) Collecting charset_normalizer<4,>=2 (from requests->-r requirements.txt (line 6)) Using cached https://nexus3.opendaylight.org/repository/PyPi/packages/charset-normalizer/3.4.4/charset_normalizer-3.4.4-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (151 kB) Collecting idna<4,>=2.5 (from requests->-r requirements.txt (line 6)) Using cached https://nexus3.opendaylight.org/repository/PyPi/packages/idna/3.11/idna-3.11-py3-none-any.whl (71 kB) Collecting urllib3<3,>=1.21.1 (from requests->-r requirements.txt (line 6)) Using cached https://nexus3.opendaylight.org/repository/PyPi/packages/urllib3/2.6.3/urllib3-2.6.3-py3-none-any.whl (131 kB) Collecting certifi>=2017.4.17 (from requests->-r requirements.txt (line 6)) Using cached https://nexus3.opendaylight.org/repository/PyPi/packages/certifi/2026.1.4/certifi-2026.1.4-py3-none-any.whl (152 kB) Collecting webtest>=2.0 (from robotframework-httplibrary->-r requirements.txt (line 8)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/webtest/3.0.7/webtest-3.0.7-py3-none-any.whl (32 kB) Collecting jsonpointer (from robotframework-httplibrary->-r requirements.txt (line 8)) Using cached https://nexus3.opendaylight.org/repository/PyPi/packages/jsonpointer/3.0.0/jsonpointer-3.0.0-py2.py3-none-any.whl (7.6 kB) Collecting robotframework-seleniumlibrary>=3.0.0 (from robotframework-selenium2library->-r requirements.txt (line 10)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/robotframework-seleniumlibrary/6.8.0/robotframework_seleniumlibrary-6.8.0-py3-none-any.whl (104 kB) Collecting ply (from jsonpath-rw->-r requirements.txt (line 15)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/ply/3.11/ply-3.11-py2.py3-none-any.whl (49 kB) Collecting decorator (from jsonpath-rw->-r requirements.txt (line 15)) Using cached https://nexus3.opendaylight.org/repository/PyPi/packages/decorator/5.2.1/decorator-5.2.1-py3-none-any.whl (9.2 kB) Collecting anyio (from elasticsearch->-r requirements.txt (line 18)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/anyio/4.12.1/anyio-4.12.1-py3-none-any.whl (113 kB) Collecting elastic-transport<10,>=9.2.0 (from elasticsearch->-r requirements.txt (line 18)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/elastic-transport/9.2.1/elastic_transport-9.2.1-py3-none-any.whl (65 kB) Collecting python-dateutil (from elasticsearch->-r requirements.txt (line 18)) Using cached https://nexus3.opendaylight.org/repository/PyPi/packages/python-dateutil/2.9.0.post0/python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB) Collecting sniffio (from elasticsearch->-r requirements.txt (line 18)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/sniffio/1.3.1/sniffio-1.3.1-py3-none-any.whl (10 kB) Collecting typing-extensions (from elasticsearch->-r requirements.txt (line 18)) Using cached https://nexus3.opendaylight.org/repository/PyPi/packages/typing-extensions/4.15.0/typing_extensions-4.15.0-py3-none-any.whl (44 kB) Collecting elasticsearch (from -r requirements.txt (line 18)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/elasticsearch/8.19.3/elasticsearch-8.19.3-py3-none-any.whl (952 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 952.8/952.8 kB 32.1 MB/s 0:00:00 INFO: pip is looking at multiple versions of elasticsearch-dsl to determine which version is compatible with other requirements. This could take a while. Collecting elasticsearch-dsl (from -r requirements.txt (line 19)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/elasticsearch-dsl/8.17.1/elasticsearch_dsl-8.17.1-py3-none-any.whl (158 kB) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/elasticsearch-dsl/8.17.0/elasticsearch_dsl-8.17.0-py3-none-any.whl (158 kB) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/elasticsearch-dsl/8.16.0/elasticsearch_dsl-8.16.0-py3-none-any.whl (158 kB) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/elasticsearch-dsl/8.15.4/elasticsearch_dsl-8.15.4-py3-none-any.whl (104 kB) Collecting elastic-transport<9,>=8.15.1 (from elasticsearch->-r requirements.txt (line 18)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/elastic-transport/8.17.1/elastic_transport-8.17.1-py3-none-any.whl (64 kB) Collecting pyang (from pyangbind->-r requirements.txt (line 22)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/pyang/2.7.1/pyang-2.7.1-py2.py3-none-any.whl (598 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 598.5/598.5 kB 20.2 MB/s 0:00:00 Collecting lxml (from pyangbind->-r requirements.txt (line 22)) Using cached https://nexus3.opendaylight.org/repository/PyPi/packages/lxml/6.0.2/lxml-6.0.2-cp311-cp311-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl (5.2 MB) Collecting regex (from pyangbind->-r requirements.txt (line 22)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/regex/2026.1.15/regex-2026.1.15-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (800 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 800.5/800.5 kB 36.7 MB/s 0:00:00 Collecting enum34 (from pyangbind->-r requirements.txt (line 22)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/enum34/1.1.10/enum34-1.1.10-py3-none-any.whl (11 kB) Collecting bcrypt>=3.2 (from paramiko>=1.15.3->robotframework-sshlibrary==3.8.0->-r requirements.txt (line 11)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/bcrypt/5.0.0/bcrypt-5.0.0-cp39-abi3-manylinux_2_28_x86_64.whl (278 kB) Collecting cryptography>=3.3 (from paramiko>=1.15.3->robotframework-sshlibrary==3.8.0->-r requirements.txt (line 11)) Using cached https://nexus3.opendaylight.org/repository/PyPi/packages/cryptography/46.0.3/cryptography-46.0.3-cp311-abi3-manylinux_2_28_x86_64.whl (4.5 MB) Collecting invoke>=2.0 (from paramiko>=1.15.3->robotframework-sshlibrary==3.8.0->-r requirements.txt (line 11)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/invoke/2.2.1/invoke-2.2.1-py3-none-any.whl (160 kB) Collecting pynacl>=1.5 (from paramiko>=1.15.3->robotframework-sshlibrary==3.8.0->-r requirements.txt (line 11)) Using cached https://nexus3.opendaylight.org/repository/PyPi/packages/pynacl/1.6.2/pynacl-1.6.2-cp38-abi3-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl (1.4 MB) Collecting cffi>=2.0.0 (from cryptography>=3.3->paramiko>=1.15.3->robotframework-sshlibrary==3.8.0->-r requirements.txt (line 11)) Using cached https://nexus3.opendaylight.org/repository/PyPi/packages/cffi/2.0.0/cffi-2.0.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (215 kB) Collecting pycparser (from cffi>=2.0.0->cryptography>=3.3->paramiko>=1.15.3->robotframework-sshlibrary==3.8.0->-r requirements.txt (line 11)) Using cached https://nexus3.opendaylight.org/repository/PyPi/packages/pycparser/2.23/pycparser-2.23-py3-none-any.whl (118 kB) Collecting selenium>=4.3.0 (from robotframework-seleniumlibrary>=3.0.0->robotframework-selenium2library->-r requirements.txt (line 10)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/selenium/4.39.0/selenium-4.39.0-py3-none-any.whl (9.7 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.7/9.7 MB 63.1 MB/s 0:00:00 Collecting robotframework-pythonlibcore>=4.4.1 (from robotframework-seleniumlibrary>=3.0.0->robotframework-selenium2library->-r requirements.txt (line 10)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/robotframework-pythonlibcore/4.5.0/robotframework_pythonlibcore-4.5.0-py3-none-any.whl (19 kB) Collecting click>=8.0 (from robotframework-seleniumlibrary>=3.0.0->robotframework-selenium2library->-r requirements.txt (line 10)) Using cached https://nexus3.opendaylight.org/repository/PyPi/packages/click/8.3.1/click-8.3.1-py3-none-any.whl (108 kB) Collecting trio<1.0,>=0.31.0 (from selenium>=4.3.0->robotframework-seleniumlibrary>=3.0.0->robotframework-selenium2library->-r requirements.txt (line 10)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/trio/0.32.0/trio-0.32.0-py3-none-any.whl (512 kB) Collecting trio-websocket<1.0,>=0.12.2 (from selenium>=4.3.0->robotframework-seleniumlibrary>=3.0.0->robotframework-selenium2library->-r requirements.txt (line 10)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/trio-websocket/0.12.2/trio_websocket-0.12.2-py3-none-any.whl (21 kB) Collecting attrs>=23.2.0 (from trio<1.0,>=0.31.0->selenium>=4.3.0->robotframework-seleniumlibrary>=3.0.0->robotframework-selenium2library->-r requirements.txt (line 10)) Using cached https://nexus3.opendaylight.org/repository/PyPi/packages/attrs/25.4.0/attrs-25.4.0-py3-none-any.whl (67 kB) Collecting sortedcontainers (from trio<1.0,>=0.31.0->selenium>=4.3.0->robotframework-seleniumlibrary>=3.0.0->robotframework-selenium2library->-r requirements.txt (line 10)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/sortedcontainers/2.4.0/sortedcontainers-2.4.0-py2.py3-none-any.whl (29 kB) Collecting outcome (from trio<1.0,>=0.31.0->selenium>=4.3.0->robotframework-seleniumlibrary>=3.0.0->robotframework-selenium2library->-r requirements.txt (line 10)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/outcome/1.3.0.post0/outcome-1.3.0.post0-py2.py3-none-any.whl (10 kB) Collecting wsproto>=0.14 (from trio-websocket<1.0,>=0.12.2->selenium>=4.3.0->robotframework-seleniumlibrary>=3.0.0->robotframework-selenium2library->-r requirements.txt (line 10)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/wsproto/1.3.2/wsproto-1.3.2-py3-none-any.whl (24 kB) Collecting pysocks!=1.5.7,<2.0,>=1.5.6 (from urllib3[socks]<3.0,>=2.5.0->selenium>=4.3.0->robotframework-seleniumlibrary>=3.0.0->robotframework-selenium2library->-r requirements.txt (line 10)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/pysocks/1.7.1/PySocks-1.7.1-py3-none-any.whl (16 kB) Collecting WebOb>=1.2 (from webtest>=2.0->robotframework-httplibrary->-r requirements.txt (line 8)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/webob/1.8.9/WebOb-1.8.9-py2.py3-none-any.whl (115 kB) Collecting waitress>=3.0.2 (from webtest>=2.0->robotframework-httplibrary->-r requirements.txt (line 8)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/waitress/3.0.2/waitress-3.0.2-py3-none-any.whl (56 kB) Collecting beautifulsoup4 (from webtest>=2.0->robotframework-httplibrary->-r requirements.txt (line 8)) Using cached https://nexus3.opendaylight.org/repository/PyPi/packages/beautifulsoup4/4.14.3/beautifulsoup4-4.14.3-py3-none-any.whl (107 kB) Collecting h11<1,>=0.16.0 (from wsproto>=0.14->trio-websocket<1.0,>=0.12.2->selenium>=4.3.0->robotframework-seleniumlibrary>=3.0.0->robotframework-selenium2library->-r requirements.txt (line 10)) Downloading https://nexus3.opendaylight.org/repository/PyPi/packages/h11/0.16.0/h11-0.16.0-py3-none-any.whl (37 kB) Collecting soupsieve>=1.6.1 (from beautifulsoup4->webtest>=2.0->robotframework-httplibrary->-r requirements.txt (line 8)) Using cached https://nexus3.opendaylight.org/repository/PyPi/packages/soupsieve/2.8.1/soupsieve-2.8.1-py3-none-any.whl (36 kB) Building wheels for collected packages: robotframework-sshlibrary, ipaddr, netifaces, robotframework-httplibrary, jsonpath-rw Building wheel for robotframework-sshlibrary (pyproject.toml): started Building wheel for robotframework-sshlibrary (pyproject.toml): finished with status 'done' Created wheel for robotframework-sshlibrary: filename=robotframework_sshlibrary-3.8.0-py3-none-any.whl size=55205 sha256=4eb6802a385df70a335f639f2579dea13d48e8c68c55a1254f60e6369349d25d Stored in directory: /home/jenkins/.cache/pip/wheels/f7/c9/b3/a977b7bcc410d45ae27d240df3d00a12585509180e373ecccc Building wheel for ipaddr (pyproject.toml): started Building wheel for ipaddr (pyproject.toml): finished with status 'done' Created wheel for ipaddr: filename=ipaddr-2.2.0-py3-none-any.whl size=18353 sha256=4a2f1736213062632ae5655c7f1b2e6606848925ec7b5b9be6e36a6049cd496e Stored in directory: /home/jenkins/.cache/pip/wheels/dc/6c/04/da2d847fa8d45c59af3e1d83e2acc29cb8adcbaf04c0898dbf Building wheel for netifaces (pyproject.toml): started Building wheel for netifaces (pyproject.toml): finished with status 'done' Created wheel for netifaces: filename=netifaces-0.11.0-cp311-cp311-linux_x86_64.whl size=41069 sha256=b91dc2700dde15568aecb92f608320db7c0e56da76ceb3508eaee311ab823657 Stored in directory: /home/jenkins/.cache/pip/wheels/f8/18/88/e61d54b995bea304bdb1d040a92b72228a1bf72ca2a3eba7c9 Building wheel for robotframework-httplibrary (pyproject.toml): started Building wheel for robotframework-httplibrary (pyproject.toml): finished with status 'done' Created wheel for robotframework-httplibrary: filename=robotframework_httplibrary-0.4.2-py3-none-any.whl size=10014 sha256=f46d43c0552115a4504dfa1f6733b8b789299e13232439e0e3a5854e79d9f393 Stored in directory: /home/jenkins/.cache/pip/wheels/aa/bc/0d/9a20dd51effef392aae2733cb4c7b66c6fa29fca33d88b57ed Building wheel for jsonpath-rw (pyproject.toml): started Building wheel for jsonpath-rw (pyproject.toml): finished with status 'done' Created wheel for jsonpath-rw: filename=jsonpath_rw-1.4.0-py3-none-any.whl size=15176 sha256=f334ff0a91dce2bb7fa06848632ceeaa36b4133fd165a7205bb3de866b746120 Stored in directory: /home/jenkins/.cache/pip/wheels/f1/54/63/9a8da38cefae13755097b36cc852decc25d8ef69c37d58d4eb Successfully built robotframework-sshlibrary ipaddr netifaces robotframework-httplibrary jsonpath-rw Installing collected packages: sortedcontainers, ply, netifaces, ipaddr, enum34, websocket-client, WebOb, waitress, urllib3, typing-extensions, soupsieve, sniffio, six, scapy, robotframework-pythonlibcore, robotframework, regex, pysocks, pyparsing, pycparser, netaddr, lxml, jsonpointer, jmespath, isodate, invoke, idna, h11, decorator, click, charset_normalizer, certifi, bcrypt, attrs, wsproto, requests, python-dateutil, pyhocon, pyang, outcome, jsonpath-rw, jsonpatch, elastic-transport, docker-pycreds, cffi, beautifulsoup4, webtest, trio, robotframework-requests, pynacl, pyangbind, elasticsearch, docker-py, cryptography, trio-websocket, robotframework-httplibrary, paramiko, elasticsearch-dsl, selenium, scp, robotframework-sshlibrary, robotframework-seleniumlibrary, robotframework-selenium2library Successfully installed WebOb-1.8.9 attrs-25.4.0 bcrypt-5.0.0 beautifulsoup4-4.14.3 certifi-2026.1.4 cffi-2.0.0 charset_normalizer-3.4.4 click-8.3.1 cryptography-46.0.3 decorator-5.2.1 docker-py-1.10.6 docker-pycreds-0.4.0 elastic-transport-8.17.1 elasticsearch-8.19.3 elasticsearch-dsl-8.15.4 enum34-1.1.10 h11-0.16.0 idna-3.11 invoke-2.2.1 ipaddr-2.2.0 isodate-0.7.2 jmespath-1.0.1 jsonpatch-1.33 jsonpath-rw-1.4.0 jsonpointer-3.0.0 lxml-6.0.2 netaddr-1.3.0 netifaces-0.11.0 outcome-1.3.0.post0 paramiko-4.0.0 ply-3.11 pyang-2.7.1 pyangbind-0.8.7 pycparser-2.23 pyhocon-0.3.61 pynacl-1.6.2 pyparsing-3.3.1 pysocks-1.7.1 python-dateutil-2.9.0.post0 regex-2026.1.15 requests-2.32.5 robotframework-7.4.1 robotframework-httplibrary-0.4.2 robotframework-pythonlibcore-4.5.0 robotframework-requests-0.9.7 robotframework-selenium2library-3.0.0 robotframework-seleniumlibrary-6.8.0 robotframework-sshlibrary-3.8.0 scapy-2.7.0 scp-0.15.0 selenium-4.39.0 six-1.17.0 sniffio-1.3.1 sortedcontainers-2.4.0 soupsieve-2.8.1 trio-0.32.0 trio-websocket-0.12.2 typing-extensions-4.15.0 urllib3-2.6.3 waitress-3.0.2 websocket-client-1.9.0 webtest-3.0.7 wsproto-1.3.2 + pip freeze attrs==25.4.0 bcrypt==5.0.0 beautifulsoup4==4.14.3 certifi==2026.1.4 cffi==2.0.0 charset-normalizer==3.4.4 click==8.3.1 cryptography==46.0.3 decorator==5.2.1 distlib==0.4.0 docker-py==1.10.6 docker-pycreds==0.4.0 elastic-transport==8.17.1 elasticsearch==8.19.3 elasticsearch-dsl==8.15.4 enum34==1.1.10 filelock==3.20.3 h11==0.16.0 idna==3.11 invoke==2.2.1 ipaddr==2.2.0 isodate==0.7.2 jmespath==1.0.1 jsonpatch==1.33 jsonpath-rw==1.4.0 jsonpointer==3.0.0 lxml==6.0.2 netaddr==1.3.0 netifaces==0.11.0 outcome==1.3.0.post0 paramiko==4.0.0 platformdirs==4.5.1 ply==3.11 pyang==2.7.1 pyangbind==0.8.7 pycparser==2.23 pyhocon==0.3.61 PyNaCl==1.6.2 pyparsing==3.3.1 PySocks==1.7.1 python-dateutil==2.9.0.post0 regex==2026.1.15 requests==2.32.5 robotframework==7.4.1 robotframework-httplibrary==0.4.2 robotframework-pythonlibcore==4.5.0 robotframework-requests==0.9.7 robotframework-selenium2library==3.0.0 robotframework-seleniumlibrary==6.8.0 robotframework-sshlibrary==3.8.0 scapy==2.7.0 scp==0.15.0 selenium==4.39.0 six==1.17.0 sniffio==1.3.1 sortedcontainers==2.4.0 soupsieve==2.8.1 trio==0.32.0 trio-websocket==0.12.2 typing_extensions==4.15.0 urllib3==2.6.3 virtualenv==20.36.1 waitress==3.0.2 WebOb==1.8.9 websocket-client==1.9.0 WebTest==3.0.7 wsproto==1.3.2 [EnvInject] - Injecting environment variables from a build step. [EnvInject] - Injecting as environment variables the properties file path 'env.properties' [EnvInject] - Variables injected successfully. [controller-csit-3node-clustering-ask-all-vanadium] $ /bin/bash -l /tmp/jenkins1606982271988448325.sh Setup pyenv: system 3.8.13 3.9.13 3.10.13 * 3.11.7 (set by /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/.python-version) lf-activate-venv(): INFO: Reuse venv:/tmp/venv-9I7X from file:/tmp/.os_lf_venv lf-activate-venv(): INFO: Installing base packages (pip, setuptools, virtualenv) lf-activate-venv(): INFO: Attempting to install with network-safe options... lf-activate-venv(): INFO: Base packages installed successfully lf-activate-venv(): INFO: Installing additional packages: python-heatclient python-openstackclient yq lf-activate-venv(): INFO: Adding /tmp/venv-9I7X/bin to PATH + ODL_SYSTEM=() + TOOLS_SYSTEM=() + OPENSTACK_SYSTEM=() + OPENSTACK_CONTROLLERS=() + mapfile -t ADDR ++ openstack stack show -f json -c outputs releng-controller-csit-3node-clustering-ask-all-vanadium-25 ++ jq -r '.outputs[] | select(.output_key | match("^vm_[0-9]+_ips$")) | .output_value | .[]' + for i in "${ADDR[@]}" ++ ssh 10.30.170.104 hostname -s Warning: Permanently added '10.30.170.104' (ECDSA) to the list of known hosts. + REMHOST=releng-34030-25-0-builder-0 + case ${REMHOST} in + ODL_SYSTEM=("${ODL_SYSTEM[@]}" "${i}") + for i in "${ADDR[@]}" ++ ssh 10.30.171.129 hostname -s Warning: Permanently added '10.30.171.129' (ECDSA) to the list of known hosts. + REMHOST=releng-34030-25-0-builder-1 + case ${REMHOST} in + ODL_SYSTEM=("${ODL_SYSTEM[@]}" "${i}") + for i in "${ADDR[@]}" ++ ssh 10.30.170.156 hostname -s Warning: Permanently added '10.30.170.156' (ECDSA) to the list of known hosts. + REMHOST=releng-34030-25-0-builder-2 + case ${REMHOST} in + ODL_SYSTEM=("${ODL_SYSTEM[@]}" "${i}") + echo NUM_ODL_SYSTEM=3 + echo NUM_TOOLS_SYSTEM=0 + '[' '' == yes ']' + NUM_OPENSTACK_SYSTEM=0 + echo NUM_OPENSTACK_SYSTEM=0 + '[' 0 -eq 2 ']' + echo ODL_SYSTEM_IP=10.30.170.104 ++ seq 0 2 + for i in $(seq 0 $(( ${#ODL_SYSTEM[@]} - 1 ))) + echo ODL_SYSTEM_1_IP=10.30.170.104 + for i in $(seq 0 $(( ${#ODL_SYSTEM[@]} - 1 ))) + echo ODL_SYSTEM_2_IP=10.30.171.129 + for i in $(seq 0 $(( ${#ODL_SYSTEM[@]} - 1 ))) + echo ODL_SYSTEM_3_IP=10.30.170.156 + echo TOOLS_SYSTEM_IP= ++ seq 0 -1 + openstack_index=0 + NUM_OPENSTACK_CONTROL_NODES=1 + echo NUM_OPENSTACK_CONTROL_NODES=1 ++ seq 0 0 + for i in $(seq 0 $((NUM_OPENSTACK_CONTROL_NODES - 1))) + echo OPENSTACK_CONTROL_NODE_1_IP= + NUM_OPENSTACK_COMPUTE_NODES=-1 + echo NUM_OPENSTACK_COMPUTE_NODES=-1 + '[' -1 -ge 2 ']' ++ seq 0 -2 + NUM_OPENSTACK_HAPROXY_NODES=0 + echo NUM_OPENSTACK_HAPROXY_NODES=0 ++ seq 0 -1 + echo 'Contents of slave_addresses.txt:' Contents of slave_addresses.txt: + cat slave_addresses.txt NUM_ODL_SYSTEM=3 NUM_TOOLS_SYSTEM=0 NUM_OPENSTACK_SYSTEM=0 ODL_SYSTEM_IP=10.30.170.104 ODL_SYSTEM_1_IP=10.30.170.104 ODL_SYSTEM_2_IP=10.30.171.129 ODL_SYSTEM_3_IP=10.30.170.156 TOOLS_SYSTEM_IP= NUM_OPENSTACK_CONTROL_NODES=1 OPENSTACK_CONTROL_NODE_1_IP= NUM_OPENSTACK_COMPUTE_NODES=-1 NUM_OPENSTACK_HAPROXY_NODES=0 [EnvInject] - Injecting environment variables from a build step. [EnvInject] - Injecting as environment variables the properties file path 'slave_addresses.txt' [EnvInject] - Variables injected successfully. [controller-csit-3node-clustering-ask-all-vanadium] $ /bin/sh /tmp/jenkins2913089409405057296.sh Preparing for JRE Version 21 Karaf artifact is karaf Karaf project is integration Java home is /usr/lib/jvm/java-21-openjdk-amd64 [EnvInject] - Injecting environment variables from a build step. [EnvInject] - Injecting as environment variables the properties file path 'set_variables.env' [EnvInject] - Variables injected successfully. [controller-csit-3node-clustering-ask-all-vanadium] $ /bin/bash /tmp/jenkins17186243862884605053.sh 2026-01-17 23:43:33 URL:https://raw.githubusercontent.com/opendaylight/integration-distribution/master/pom.xml [2619/2619] -> "pom.xml" [1] Bundle version is 0.24.0-SNAPSHOT --2026-01-17 23:43:33-- https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/integration/karaf/0.24.0-SNAPSHOT/maven-metadata.xml Resolving nexus.opendaylight.org (nexus.opendaylight.org)... 199.204.45.87, 2604:e100:1:0:f816:3eff:fe45:48d6 Connecting to nexus.opendaylight.org (nexus.opendaylight.org)|199.204.45.87|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 1404 (1.4K) [application/xml] Saving to: ‘maven-metadata.xml’ 0K . 100% 71.9M=0s 2026-01-17 23:43:33 (71.9 MB/s) - ‘maven-metadata.xml’ saved [1404/1404] org.opendaylight.integration karaf 0.24.0-SNAPSHOT 20260117.223656 12 20260117223656 pom 0.24.0-20260117.223656-12 20260117223656 tar.gz 0.24.0-20260117.223656-12 20260117223656 zip 0.24.0-20260117.223656-12 20260117223656 cyclonedx xml 0.24.0-20260117.223656-12 20260117223656 cyclonedx json 0.24.0-20260117.223656-12 20260117223656 Nexus timestamp is 0.24.0-20260117.223656-12 Distribution bundle URL is https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/integration/karaf/0.24.0-SNAPSHOT/karaf-0.24.0-20260117.223656-12.zip Distribution bundle is karaf-0.24.0-20260117.223656-12.zip Distribution bundle version is 0.24.0-SNAPSHOT Distribution folder is karaf-0.24.0-SNAPSHOT Nexus prefix is https://nexus.opendaylight.org [EnvInject] - Injecting environment variables from a build step. [EnvInject] - Injecting as environment variables the properties file path 'detect_variables.env' [EnvInject] - Variables injected successfully. [controller-csit-3node-clustering-ask-all-vanadium] $ /bin/bash -l /tmp/jenkins781854674466652567.sh Setup pyenv: system 3.8.13 3.9.13 3.10.13 * 3.11.7 (set by /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/.python-version) lf-activate-venv(): INFO: Reuse venv:/tmp/venv-9I7X from file:/tmp/.os_lf_venv lf-activate-venv(): INFO: Installing base packages (pip, setuptools, virtualenv) lf-activate-venv(): INFO: Attempting to install with network-safe options... lf-activate-venv(): INFO: Base packages installed successfully lf-activate-venv(): INFO: Installing additional packages: python-heatclient python-openstackclient lf-activate-venv(): INFO: Adding /tmp/venv-9I7X/bin to PATH Copying common-functions.sh to /tmp Copying common-functions.sh to 10.30.170.104:/tmp Warning: Permanently added '10.30.170.104' (ECDSA) to the list of known hosts. Copying common-functions.sh to 10.30.171.129:/tmp Warning: Permanently added '10.30.171.129' (ECDSA) to the list of known hosts. Copying common-functions.sh to 10.30.170.156:/tmp Warning: Permanently added '10.30.170.156' (ECDSA) to the list of known hosts. [controller-csit-3node-clustering-ask-all-vanadium] $ /bin/bash /tmp/jenkins17581712376214565351.sh common-functions.sh is being sourced common-functions environment: MAVENCONF: /tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.url.mvn.cfg ACTUALFEATURES: FEATURESCONF: /tmp/karaf-0.24.0-SNAPSHOT/etc/org.apache.karaf.features.cfg CUSTOMPROP: /tmp/karaf-0.24.0-SNAPSHOT/etc/custom.properties LOGCONF: /tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.logging.cfg MEMCONF: /tmp/karaf-0.24.0-SNAPSHOT/bin/setenv CONTROLLERMEM: 2048m AKKACONF: /tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/pekko.conf MODULESCONF: /tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/modules.conf MODULESHARDSCONF: /tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/module-shards.conf SUITES: ################################################# ## Configure Cluster and Start ## ################################################# ACTUALFEATURES: odl-integration-compatible-with-all,odl-jolokia,odl-restconf,odl-clustering-test-app SPACE_SEPARATED_FEATURES: odl-integration-compatible-with-all odl-jolokia odl-restconf odl-clustering-test-app Locating script plan to use... script plan exists!!! Changing the script plan path... # Place the scripts in run order: /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/scripts/car_people_shard.sh Executing /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/scripts/car_people_shard.sh... Add car-people shards file Copy shard config to member-1 with IP address 10.30.170.104 Warning: Permanently added '10.30.170.104' (ECDSA) to the list of known hosts. Copy shard config to member-2 with IP address 10.30.171.129 Warning: Permanently added '10.30.171.129' (ECDSA) to the list of known hosts. Copy shard config to member-3 with IP address 10.30.170.156 Warning: Permanently added '10.30.170.156' (ECDSA) to the list of known hosts. Finished running script plans Configuring member-1 with IP address 10.30.170.104 Warning: Permanently added '10.30.170.104' (ECDSA) to the list of known hosts. Warning: Permanently added '10.30.170.104' (ECDSA) to the list of known hosts. + source /tmp/common-functions.sh karaf-0.24.0-SNAPSHOT vanadium ++ [[ /tmp/common-functions.sh == \/\t\m\p\/\c\o\n\f\i\g\u\r\a\t\i\o\n\-\s\c\r\i\p\t\.\s\h ]] common-functions.sh is being sourced ++ echo 'common-functions.sh is being sourced' ++ BUNDLEFOLDER=karaf-0.24.0-SNAPSHOT ++ DISTROSTREAM=vanadium ++ export MAVENCONF=/tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.url.mvn.cfg ++ MAVENCONF=/tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.url.mvn.cfg ++ export FEATURESCONF=/tmp/karaf-0.24.0-SNAPSHOT/etc/org.apache.karaf.features.cfg ++ FEATURESCONF=/tmp/karaf-0.24.0-SNAPSHOT/etc/org.apache.karaf.features.cfg ++ export CUSTOMPROP=/tmp/karaf-0.24.0-SNAPSHOT/etc/custom.properties ++ CUSTOMPROP=/tmp/karaf-0.24.0-SNAPSHOT/etc/custom.properties ++ export LOGCONF=/tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.logging.cfg ++ LOGCONF=/tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.logging.cfg ++ export MEMCONF=/tmp/karaf-0.24.0-SNAPSHOT/bin/setenv ++ MEMCONF=/tmp/karaf-0.24.0-SNAPSHOT/bin/setenv ++ export CONTROLLERMEM= ++ CONTROLLERMEM= ++ case "${DISTROSTREAM}" in ++ CLUSTER_SYSTEM=pekko ++ export AKKACONF=/tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/pekko.conf ++ AKKACONF=/tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/pekko.conf ++ export MODULESCONF=/tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/modules.conf ++ MODULESCONF=/tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/modules.conf ++ export MODULESHARDSCONF=/tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/module-shards.conf ++ MODULESHARDSCONF=/tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/module-shards.conf ++ print_common_env ++ cat common-functions environment: MAVENCONF: /tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.url.mvn.cfg ACTUALFEATURES: FEATURESCONF: /tmp/karaf-0.24.0-SNAPSHOT/etc/org.apache.karaf.features.cfg CUSTOMPROP: /tmp/karaf-0.24.0-SNAPSHOT/etc/custom.properties LOGCONF: /tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.logging.cfg MEMCONF: /tmp/karaf-0.24.0-SNAPSHOT/bin/setenv CONTROLLERMEM: AKKACONF: /tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/pekko.conf MODULESCONF: /tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/modules.conf MODULESHARDSCONF: /tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/module-shards.conf SUITES: ++ SSH='ssh -t -t' ++ extra_services_cntl=' dnsmasq.service httpd.service libvirtd.service openvswitch.service ovs-vswitchd.service ovsdb-server.service rabbitmq-server.service ' ++ extra_services_cmp=' libvirtd.service openvswitch.service ovs-vswitchd.service ovsdb-server.service ' Changing to /tmp Downloading the distribution from https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/integration/karaf/0.24.0-SNAPSHOT/karaf-0.24.0-20260117.223656-12.zip + echo 'Changing to /tmp' + cd /tmp + echo 'Downloading the distribution from https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/integration/karaf/0.24.0-SNAPSHOT/karaf-0.24.0-20260117.223656-12.zip' + wget --progress=dot:mega https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/integration/karaf/0.24.0-SNAPSHOT/karaf-0.24.0-20260117.223656-12.zip --2026-01-17 23:43:46-- https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/integration/karaf/0.24.0-SNAPSHOT/karaf-0.24.0-20260117.223656-12.zip Resolving nexus.opendaylight.org (nexus.opendaylight.org)... 199.204.45.87, 2604:e100:1:0:f816:3eff:fe45:48d6 Connecting to nexus.opendaylight.org (nexus.opendaylight.org)|199.204.45.87|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 240365717 (229M) [application/zip] Saving to: ‘karaf-0.24.0-20260117.223656-12.zip’ 0K ........ ........ ........ ........ ........ ........ 1% 26.4M 9s 3072K ........ ........ ........ ........ ........ ........ 2% 32.5M 8s 6144K ........ ........ ........ ........ ........ ........ 3% 56.4M 6s 9216K ........ ........ ........ ........ ........ ........ 5% 36.5M 6s 12288K ........ ........ ........ ........ ........ ........ 6% 53.5M 6s 15360K ........ ........ ........ ........ ........ ........ 7% 168M 5s 18432K ........ ........ ........ ........ ........ ........ 9% 217M 4s 21504K ........ ........ ........ ........ ........ ........ 10% 253M 4s 24576K ........ ........ ........ ........ ........ ........ 11% 214M 3s 27648K ........ ........ ........ ........ ........ ........ 13% 238M 3s 30720K ........ ........ ........ ........ ........ ........ 14% 224M 3s 33792K ........ ........ ........ ........ ........ ........ 15% 229M 3s 36864K ........ ........ ........ ........ ........ ........ 17% 236M 2s 39936K ........ ........ ........ ........ ........ ........ 18% 277M 2s 43008K ........ ........ ........ ........ ........ ........ 19% 279M 2s 46080K ........ ........ ........ ........ ........ ........ 20% 282M 2s 49152K ........ ........ ........ ........ ........ ........ 22% 284M 2s 52224K ........ ........ ........ ........ ........ ........ 23% 291M 2s 55296K ........ ........ ........ ........ ........ ........ 24% 285M 2s 58368K ........ ........ ........ ........ ........ ........ 26% 245M 2s 61440K ........ ........ ........ ........ ........ ........ 27% 272M 2s 64512K ........ ........ ........ ........ ........ ........ 28% 344M 1s 67584K ........ ........ ........ ........ ........ ........ 30% 289M 1s 70656K ........ ........ ........ ........ ........ ........ 31% 269M 1s 73728K ........ ........ ........ ........ ........ ........ 32% 268M 1s 76800K ........ ........ ........ ........ ........ ........ 34% 263M 1s 79872K ........ ........ ........ ........ ........ ........ 35% 252M 1s 82944K ........ ........ ........ ........ ........ ........ 36% 232M 1s 86016K ........ ........ ........ ........ ........ ........ 37% 232M 1s 89088K ........ ........ ........ ........ ........ ........ 39% 222M 1s 92160K ........ ........ ........ ........ ........ ........ 40% 197M 1s 95232K ........ ........ ........ ........ ........ ........ 41% 303M 1s 98304K ........ ........ ........ ........ ........ ........ 43% 234M 1s 101376K ........ ........ ........ ........ ........ ........ 44% 263M 1s 104448K ........ ........ ........ ........ ........ ........ 45% 259M 1s 107520K ........ ........ ........ ........ ........ ........ 47% 257M 1s 110592K ........ ........ ........ ........ ........ ........ 48% 259M 1s 113664K ........ ........ ........ ........ ........ ........ 49% 262M 1s 116736K ........ ........ ........ ........ ........ ........ 51% 286M 1s 119808K ........ ........ ........ ........ ........ ........ 52% 293M 1s 122880K ........ ........ ........ ........ ........ ........ 53% 291M 1s 125952K ........ ........ ........ ........ ........ ........ 54% 284M 1s 129024K ........ ........ ........ ........ ........ ........ 56% 324M 1s 132096K ........ ........ ........ ........ ........ ........ 57% 291M 1s 135168K ........ ........ ........ ........ ........ ........ 58% 283M 1s 138240K ........ ........ ........ ........ ........ ........ 60% 287M 1s 141312K ........ ........ ........ ........ ........ ........ 61% 287M 1s 144384K ........ ........ ........ ........ ........ ........ 62% 271M 1s 147456K ........ ........ ........ ........ ........ ........ 64% 316M 1s 150528K ........ ........ ........ ........ ........ ........ 65% 288M 0s 153600K ........ ........ ........ ........ ........ ........ 66% 277M 0s 156672K ........ ........ ........ ........ ........ ........ 68% 282M 0s 159744K ........ ........ ........ ........ ........ ........ 69% 290M 0s 162816K ........ ........ ........ ........ ........ ........ 70% 272M 0s 165888K ........ ........ ........ ........ ........ ........ 71% 279M 0s 168960K ........ ........ ........ ........ ........ ........ 73% 275M 0s 172032K ........ ........ ........ ........ ........ ........ 74% 268M 0s 175104K ........ ........ ........ ........ ........ ........ 75% 301M 0s 178176K ........ ........ ........ ........ ........ ........ 77% 285M 0s 181248K ........ ........ ........ ........ ........ ........ 78% 303M 0s 184320K ........ ........ ........ ........ ........ ........ 79% 303M 0s 187392K ........ ........ ........ ........ ........ ........ 81% 299M 0s 190464K ........ ........ ........ ........ ........ ........ 82% 302M 0s 193536K ........ ........ ........ ........ ........ ........ 83% 284M 0s 196608K ........ ........ ........ ........ ........ ........ 85% 328M 0s 199680K ........ ........ ........ ........ ........ ........ 86% 294M 0s 202752K ........ ........ ........ ........ ........ ........ 87% 285M 0s 205824K ........ ........ ........ ........ ........ ........ 88% 303M 0s 208896K ........ ........ ........ ........ ........ ........ 90% 290M 0s 211968K ........ ........ ........ ........ ........ ........ 91% 281M 0s 215040K ........ ........ ........ ........ ........ ........ 92% 279M 0s 218112K ........ ........ ........ ........ ........ ........ 94% 294M 0s 221184K ........ ........ ........ ........ ........ ........ 95% 283M 0s 224256K ........ ........ ........ ........ ........ ........ 96% 283M 0s 227328K ........ ........ ........ ........ ........ ........ 98% 286M 0s 230400K ........ ........ ........ ........ ........ ........ 99% 274M 0s 233472K ........ ........ ... 100% 281M=1.2s 2026-01-17 23:43:47 (192 MB/s) - ‘karaf-0.24.0-20260117.223656-12.zip’ saved [240365717/240365717] Extracting the new controller... + echo 'Extracting the new controller...' + unzip -q karaf-0.24.0-20260117.223656-12.zip Adding external repositories... + echo 'Adding external repositories...' + sed -ie 's%org.ops4j.pax.url.mvn.repositories=%org.ops4j.pax.url.mvn.repositories=https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot@id=opendaylight-snapshot@snapshots, https://nexus.opendaylight.org/content/repositories/public@id=opendaylight-mirror, http://repo1.maven.org/maven2@id=central, http://repository.springsource.com/maven/bundles/release@id=spring.ebr.release, http://repository.springsource.com/maven/bundles/external@id=spring.ebr.external, http://zodiac.springsource.com/maven/bundles/release@id=gemini, http://repository.apache.org/content/groups/snapshots-group@id=apache@snapshots@noreleases, https://oss.sonatype.org/content/repositories/snapshots@id=sonatype.snapshots.deploy@snapshots@noreleases, https://oss.sonatype.org/content/repositories/ops4j-snapshots@id=ops4j.sonatype.snapshots.deploy@snapshots@noreleases%g' /tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.url.mvn.cfg + cat /tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.url.mvn.cfg ################################################################################ # # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ################################################################################ # # If set to true, the following property will not allow any certificate to be used # when accessing Maven repositories through SSL # #org.ops4j.pax.url.mvn.certificateCheck= # # Path to the local Maven settings file. # The repositories defined in this file will be automatically added to the list # of default repositories if the 'org.ops4j.pax.url.mvn.repositories' property # below is not set. # The following locations are checked for the existence of the settings.xml file # * 1. looks for the specified url # * 2. if not found looks for ${user.home}/.m2/settings.xml # * 3. if not found looks for ${maven.home}/conf/settings.xml # * 4. if not found looks for ${M2_HOME}/conf/settings.xml # #org.ops4j.pax.url.mvn.settings= # # Path to the local Maven repository which is used to avoid downloading # artifacts when they already exist locally. # The value of this property will be extracted from the settings.xml file # above, or defaulted to: # System.getProperty( "user.home" ) + "/.m2/repository" # org.ops4j.pax.url.mvn.localRepository=${karaf.home}/${karaf.default.repository} # # Default this to false. It's just weird to use undocumented repos # org.ops4j.pax.url.mvn.useFallbackRepositories=false # # Uncomment if you don't wanna use the proxy settings # from the Maven conf/settings.xml file # # org.ops4j.pax.url.mvn.proxySupport=false # # Comma separated list of repositories scanned when resolving an artifact. # Those repositories will be checked before iterating through the # below list of repositories and even before the local repository # A repository url can be appended with zero or more of the following flags: # @snapshots : the repository contains snaphots # @noreleases : the repository does not contain any released artifacts # # The following property value will add the system folder as a repo. # org.ops4j.pax.url.mvn.defaultRepositories=\ file:${karaf.home}/${karaf.default.repository}@id=system.repository@snapshots,\ file:${karaf.data}/kar@id=kar.repository@multi@snapshots,\ file:${karaf.base}/${karaf.default.repository}@id=child.system.repository@snapshots # Use the default local repo (e.g.~/.m2/repository) as a "remote" repo #org.ops4j.pax.url.mvn.defaultLocalRepoAsRemote=false # # Comma separated list of repositories scanned when resolving an artifact. # The default list includes the following repositories: # http://repo1.maven.org/maven2@id=central # http://repository.springsource.com/maven/bundles/release@id=spring.ebr # http://repository.springsource.com/maven/bundles/external@id=spring.ebr.external # http://zodiac.springsource.com/maven/bundles/release@id=gemini # http://repository.apache.org/content/groups/snapshots-group@id=apache@snapshots@noreleases # https://oss.sonatype.org/content/repositories/snapshots@id=sonatype.snapshots.deploy@snapshots@noreleases # https://oss.sonatype.org/content/repositories/ops4j-snapshots@id=ops4j.sonatype.snapshots.deploy@snapshots@noreleases # To add repositories to the default ones, prepend '+' to the list of repositories # to add. # A repository url can be appended with zero or more of the following flags: # @snapshots : the repository contains snapshots # @noreleases : the repository does not contain any released artifacts # @id=repository.id : the id for the repository, just like in the settings.xml this is optional but recommended # org.ops4j.pax.url.mvn.repositories=https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot@id=opendaylight-snapshot@snapshots, https://nexus.opendaylight.org/content/repositories/public@id=opendaylight-mirror, http://repo1.maven.org/maven2@id=central, http://repository.springsource.com/maven/bundles/release@id=spring.ebr.release, http://repository.springsource.com/maven/bundles/external@id=spring.ebr.external, http://zodiac.springsource.com/maven/bundles/release@id=gemini, http://repository.apache.org/content/groups/snapshots-group@id=apache@snapshots@noreleases, https://oss.sonatype.org/content/repositories/snapshots@id=sonatype.snapshots.deploy@snapshots@noreleases, https://oss.sonatype.org/content/repositories/ops4j-snapshots@id=ops4j.sonatype.snapshots.deploy@snapshots@noreleases ### ^^^ No remote repositories. This is the only ODL change compared to Karaf defaults.+ [[ True == \T\r\u\e ]] + echo 'Configuring the startup features...' Configuring the startup features... + sed -ie 's/\(featuresBoot=\|featuresBoot =\)/featuresBoot = odl-integration-compatible-with-all,odl-jolokia,odl-restconf,odl-clustering-test-app,/g' /tmp/karaf-0.24.0-SNAPSHOT/etc/org.apache.karaf.features.cfg + FEATURE_TEST_STRING=features-test + FEATURE_TEST_VERSION=0.24.0-SNAPSHOT + KARAF_VERSION=karaf4 + [[ integration == \i\n\t\e\g\r\a\t\i\o\n ]] + sed -ie 's%\(featuresRepositories=\|featuresRepositories =\)%featuresRepositories = mvn:org.opendaylight.integration/features-test/0.24.0-SNAPSHOT/xml/features,mvn:org.apache.karaf.decanter/apache-karaf-decanter/1.2.0/xml/features,%g' /tmp/karaf-0.24.0-SNAPSHOT/etc/org.apache.karaf.features.cfg + [[ ! -z '' ]] + cat /tmp/karaf-0.24.0-SNAPSHOT/etc/org.apache.karaf.features.cfg ################################################################################ # # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ################################################################################ # # Comma separated list of features repositories to register by default # featuresRepositories = mvn:org.opendaylight.integration/features-test/0.24.0-SNAPSHOT/xml/features,mvn:org.apache.karaf.decanter/apache-karaf-decanter/1.2.0/xml/features, file:${karaf.etc}/eef4f053-53e3-43e4-9c13-1e5bf5e49fc9.xml # # Comma separated list of features to install at startup # featuresBoot = odl-integration-compatible-with-all,odl-jolokia,odl-restconf,odl-clustering-test-app, e0ed419d-f0e5-4080-bffe-2232822c3309 # # Resource repositories (OBR) that the features resolver can use # to resolve requirements/capabilities # # The format of the resourceRepositories is # resourceRepositories=[xml:url|json:url],... # for Instance: # #resourceRepositories=xml:http://host/path/to/index.xml # or #resourceRepositories=json:http://host/path/to/index.json # # # Defines if the boot features are started in asynchronous mode (in a dedicated thread) # featuresBootAsynchronous=false # # Service requirements enforcement # # By default, the feature resolver checks the service requirements/capabilities of # bundles for new features (xml schema >= 1.3.0) in order to automatically installs # the required bundles. # The following flag can have those values: # - disable: service requirements are completely ignored # - default: service requirements are ignored for old features # - enforce: service requirements are always verified # #serviceRequirements=default # # Store cfg file for config element in feature # #configCfgStore=true # # Define if the feature service automatically refresh bundles # autoRefresh=true # # Configuration of features processing mechanism (overrides, blacklisting, modification of features) # XML file defines instructions related to features processing # versions.properties may declare properties to resolve placeholders in XML file # both files are relative to ${karaf.etc} # #featureProcessing=org.apache.karaf.features.xml #featureProcessingVersions=versions.properties + configure_karaf_log karaf4 '' + local -r karaf_version=karaf4 + local -r controllerdebugmap= + local logapi=log4j + grep log4j2 /tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.logging.cfg log4j2.pattern = %d{ISO8601} | %-5p | %-16t | %-32c{1} | %X{bundle.id} - %X{bundle.name} - %X{bundle.version} | %m%n log4j2.rootLogger.level = INFO #log4j2.rootLogger.type = asyncRoot #log4j2.rootLogger.includeLocation = false log4j2.rootLogger.appenderRef.RollingFile.ref = RollingFile log4j2.rootLogger.appenderRef.PaxOsgi.ref = PaxOsgi log4j2.rootLogger.appenderRef.Console.ref = Console log4j2.rootLogger.appenderRef.Console.filter.threshold.type = ThresholdFilter log4j2.rootLogger.appenderRef.Console.filter.threshold.level = ${karaf.log.console:-OFF} log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.type = ContextMapFilter log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.pair1.type = KeyValuePair log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.pair1.key = slf4j.marker log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.pair1.value = CONFIDENTIAL log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.operator = or log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.onMatch = DENY log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.onMismatch = NEUTRAL log4j2.logger.spifly.name = org.apache.aries.spifly log4j2.logger.spifly.level = WARN log4j2.logger.audit.name = org.apache.karaf.jaas.modules.audit log4j2.logger.audit.level = INFO log4j2.logger.audit.additivity = false log4j2.logger.audit.appenderRef.AuditRollingFile.ref = AuditRollingFile # Console appender not used by default (see log4j2.rootLogger.appenderRefs) log4j2.appender.console.type = Console log4j2.appender.console.name = Console log4j2.appender.console.layout.type = PatternLayout log4j2.appender.console.layout.pattern = ${log4j2.pattern} log4j2.appender.rolling.type = RollingRandomAccessFile log4j2.appender.rolling.name = RollingFile log4j2.appender.rolling.fileName = ${karaf.data}/log/karaf.log log4j2.appender.rolling.filePattern = ${karaf.data}/log/karaf.log.%i #log4j2.appender.rolling.immediateFlush = false log4j2.appender.rolling.append = true log4j2.appender.rolling.layout.type = PatternLayout log4j2.appender.rolling.layout.pattern = ${log4j2.pattern} log4j2.appender.rolling.policies.type = Policies log4j2.appender.rolling.policies.size.type = SizeBasedTriggeringPolicy log4j2.appender.rolling.policies.size.size = 64MB log4j2.appender.rolling.strategy.type = DefaultRolloverStrategy log4j2.appender.rolling.strategy.max = 7 log4j2.appender.audit.type = RollingRandomAccessFile log4j2.appender.audit.name = AuditRollingFile log4j2.appender.audit.fileName = ${karaf.data}/security/audit.log log4j2.appender.audit.filePattern = ${karaf.data}/security/audit.log.%i log4j2.appender.audit.append = true log4j2.appender.audit.layout.type = PatternLayout log4j2.appender.audit.layout.pattern = ${log4j2.pattern} log4j2.appender.audit.policies.type = Policies log4j2.appender.audit.policies.size.type = SizeBasedTriggeringPolicy log4j2.appender.audit.policies.size.size = 8MB log4j2.appender.audit.strategy.type = DefaultRolloverStrategy log4j2.appender.audit.strategy.max = 7 log4j2.appender.osgi.type = PaxOsgi log4j2.appender.osgi.name = PaxOsgi log4j2.appender.osgi.filter = * #log4j2.logger.aether.name = shaded.org.eclipse.aether #log4j2.logger.aether.level = TRACE #log4j2.logger.http-headers.name = shaded.org.apache.http.headers #log4j2.logger.http-headers.level = DEBUG #log4j2.logger.maven.name = org.ops4j.pax.url.mvn #log4j2.logger.maven.level = TRACE + logapi=log4j2 Configuring the karaf log... karaf_version: karaf4, logapi: log4j2 + echo 'Configuring the karaf log... karaf_version: karaf4, logapi: log4j2' + '[' log4j2 == log4j2 ']' + sed -ie 's/log4j2.appender.rolling.policies.size.size = 64MB/log4j2.appender.rolling.policies.size.size = 1GB/g' /tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.logging.cfg + orgmodule=org.opendaylight.yangtools.yang.parser.repo.YangTextSchemaContextResolver + orgmodule_=org_opendaylight_yangtools_yang_parser_repo_YangTextSchemaContextResolver + echo 'log4j2.logger.org_opendaylight_yangtools_yang_parser_repo_YangTextSchemaContextResolver.name = WARN' + echo 'log4j2.logger.org_opendaylight_yangtools_yang_parser_repo_YangTextSchemaContextResolver.level = WARN' controllerdebugmap: cat /tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.logging.cfg + unset IFS + echo 'controllerdebugmap: ' + '[' -n '' ']' + echo 'cat /tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.logging.cfg' + cat /tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.logging.cfg ################################################################################ # # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ################################################################################ # Common pattern layout for appenders log4j2.pattern = %d{ISO8601} | %-5p | %-16t | %-32c{1} | %X{bundle.id} - %X{bundle.name} - %X{bundle.version} | %m%n # Root logger log4j2.rootLogger.level = INFO # uncomment to use asynchronous loggers, which require mvn:com.lmax/disruptor/3.3.2 library #log4j2.rootLogger.type = asyncRoot #log4j2.rootLogger.includeLocation = false log4j2.rootLogger.appenderRef.RollingFile.ref = RollingFile log4j2.rootLogger.appenderRef.PaxOsgi.ref = PaxOsgi log4j2.rootLogger.appenderRef.Console.ref = Console log4j2.rootLogger.appenderRef.Console.filter.threshold.type = ThresholdFilter log4j2.rootLogger.appenderRef.Console.filter.threshold.level = ${karaf.log.console:-OFF} # Filters for logs marked by org.opendaylight.odlparent.Markers log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.type = ContextMapFilter log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.pair1.type = KeyValuePair log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.pair1.key = slf4j.marker log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.pair1.value = CONFIDENTIAL log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.operator = or log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.onMatch = DENY log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.onMismatch = NEUTRAL # Loggers configuration # Spifly logger log4j2.logger.spifly.name = org.apache.aries.spifly log4j2.logger.spifly.level = WARN # Security audit logger log4j2.logger.audit.name = org.apache.karaf.jaas.modules.audit log4j2.logger.audit.level = INFO log4j2.logger.audit.additivity = false log4j2.logger.audit.appenderRef.AuditRollingFile.ref = AuditRollingFile # Appenders configuration # Console appender not used by default (see log4j2.rootLogger.appenderRefs) log4j2.appender.console.type = Console log4j2.appender.console.name = Console log4j2.appender.console.layout.type = PatternLayout log4j2.appender.console.layout.pattern = ${log4j2.pattern} # Rolling file appender log4j2.appender.rolling.type = RollingRandomAccessFile log4j2.appender.rolling.name = RollingFile log4j2.appender.rolling.fileName = ${karaf.data}/log/karaf.log log4j2.appender.rolling.filePattern = ${karaf.data}/log/karaf.log.%i # uncomment to not force a disk flush #log4j2.appender.rolling.immediateFlush = false log4j2.appender.rolling.append = true log4j2.appender.rolling.layout.type = PatternLayout log4j2.appender.rolling.layout.pattern = ${log4j2.pattern} log4j2.appender.rolling.policies.type = Policies log4j2.appender.rolling.policies.size.type = SizeBasedTriggeringPolicy log4j2.appender.rolling.policies.size.size = 1GB log4j2.appender.rolling.strategy.type = DefaultRolloverStrategy log4j2.appender.rolling.strategy.max = 7 # Audit file appender log4j2.appender.audit.type = RollingRandomAccessFile log4j2.appender.audit.name = AuditRollingFile log4j2.appender.audit.fileName = ${karaf.data}/security/audit.log log4j2.appender.audit.filePattern = ${karaf.data}/security/audit.log.%i log4j2.appender.audit.append = true log4j2.appender.audit.layout.type = PatternLayout log4j2.appender.audit.layout.pattern = ${log4j2.pattern} log4j2.appender.audit.policies.type = Policies log4j2.appender.audit.policies.size.type = SizeBasedTriggeringPolicy log4j2.appender.audit.policies.size.size = 8MB log4j2.appender.audit.strategy.type = DefaultRolloverStrategy log4j2.appender.audit.strategy.max = 7 # OSGi appender log4j2.appender.osgi.type = PaxOsgi log4j2.appender.osgi.name = PaxOsgi log4j2.appender.osgi.filter = * # help with identification of maven-related problems with pax-url-aether #log4j2.logger.aether.name = shaded.org.eclipse.aether #log4j2.logger.aether.level = TRACE #log4j2.logger.http-headers.name = shaded.org.apache.http.headers #log4j2.logger.http-headers.level = DEBUG #log4j2.logger.maven.name = org.ops4j.pax.url.mvn #log4j2.logger.maven.level = TRACE log4j2.logger.org_opendaylight_yangtools_yang_parser_repo_YangTextSchemaContextResolver.name = WARN log4j2.logger.org_opendaylight_yangtools_yang_parser_repo_YangTextSchemaContextResolver.level = WARN Configure java home: /usr/lib/jvm/java-21-openjdk-amd64 max memory: 3072m memconf: /tmp/karaf-0.24.0-SNAPSHOT/bin/setenv + set_java_vars /usr/lib/jvm/java-21-openjdk-amd64 3072m /tmp/karaf-0.24.0-SNAPSHOT/bin/setenv + local -r java_home=/usr/lib/jvm/java-21-openjdk-amd64 + local -r controllermem=3072m + local -r memconf=/tmp/karaf-0.24.0-SNAPSHOT/bin/setenv + echo Configure + echo ' java home: /usr/lib/jvm/java-21-openjdk-amd64' + echo ' max memory: 3072m' + echo ' memconf: /tmp/karaf-0.24.0-SNAPSHOT/bin/setenv' + sed -ie 's%^# export JAVA_HOME%export JAVA_HOME=${JAVA_HOME:-/usr/lib/jvm/java-21-openjdk-amd64}%g' /tmp/karaf-0.24.0-SNAPSHOT/bin/setenv + sed -ie 's/JAVA_MAX_MEM="2048m"/JAVA_MAX_MEM=3072m/g' /tmp/karaf-0.24.0-SNAPSHOT/bin/setenv cat /tmp/karaf-0.24.0-SNAPSHOT/bin/setenv + echo 'cat /tmp/karaf-0.24.0-SNAPSHOT/bin/setenv' + cat /tmp/karaf-0.24.0-SNAPSHOT/bin/setenv #!/bin/sh # # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # # handle specific scripts; the SCRIPT_NAME is exactly the name of the Karaf # script: client, instance, shell, start, status, stop, karaf # # if [ "${KARAF_SCRIPT}" == "SCRIPT_NAME" ]; then # Actions go here... # fi # # general settings which should be applied for all scripts go here; please keep # in mind that it is possible that scripts might be executed more than once, e.g. # in example of the start script where the start script is executed first and the # karaf script afterwards. # # # The following section shows the possible configuration options for the default # karaf scripts # export JAVA_HOME=${JAVA_HOME:-/usr/lib/jvm/java-21-openjdk-amd64} # Location of Java installation # export JAVA_OPTS # Generic JVM options, for instance, where you can pass the memory configuration # export JAVA_NON_DEBUG_OPTS # Additional non-debug JVM options # export EXTRA_JAVA_OPTS # Additional JVM options # export KARAF_HOME # Karaf home folder # export KARAF_DATA # Karaf data folder # export KARAF_BASE # Karaf base folder # export KARAF_ETC # Karaf etc folder # export KARAF_LOG # Karaf log folder # export KARAF_SYSTEM_OPTS # First citizen Karaf options # export KARAF_OPTS # Additional available Karaf options # export KARAF_DEBUG # Enable debug mode # export KARAF_REDIRECT # Enable/set the std/err redirection when using bin/start # export KARAF_NOROOT # Prevent execution as root if set to true Set Java version + echo 'Set Java version' + sudo /usr/sbin/alternatives --install /usr/bin/java java /usr/lib/jvm/java-21-openjdk-amd64/bin/java 1 sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper sudo: a password is required + sudo /usr/sbin/alternatives --set java /usr/lib/jvm/java-21-openjdk-amd64/bin/java sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper JDK default version ... sudo: a password is required + echo 'JDK default version ...' + java -version openjdk version "21.0.9" 2025-10-21 OpenJDK Runtime Environment (build 21.0.9+10-Ubuntu-122.04) OpenJDK 64-Bit Server VM (build 21.0.9+10-Ubuntu-122.04, mixed mode, sharing) Set JAVA_HOME + echo 'Set JAVA_HOME' + export JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64 ++ readlink -e /usr/lib/jvm/java-21-openjdk-amd64/bin/java Java binary pointed at by JAVA_HOME: /usr/lib/jvm/java-21-openjdk-amd64/bin/java + JAVA_RESOLVED=/usr/lib/jvm/java-21-openjdk-amd64/bin/java + echo 'Java binary pointed at by JAVA_HOME: /usr/lib/jvm/java-21-openjdk-amd64/bin/java' Listing all open ports on controller system... + echo 'Listing all open ports on controller system...' + netstat -pnatu /tmp/configuration-script.sh: line 40: netstat: command not found Custom shard config exists!!! Copying the shard config... + '[' -f /tmp/custom_shard_config.txt ']' + echo 'Custom shard config exists!!!' + echo 'Copying the shard config...' + cp /tmp/custom_shard_config.txt /tmp/karaf-0.24.0-SNAPSHOT/bin/ Configuring cluster + echo 'Configuring cluster' + /tmp/karaf-0.24.0-SNAPSHOT/bin/configure_cluster.sh 1 10.30.170.104 10.30.171.129 10.30.170.156 ################################################ ## Configure Cluster ## ################################################ NOTE: Cluster configuration files not found. Copying from /tmp/karaf-0.24.0-SNAPSHOT/system/org/opendaylight/controller/sal-clustering-config/12.0.3 Configuring unique name in pekko.conf Configuring hostname in pekko.conf Configuring data and rpc seed nodes in pekko.conf modules = [ { name = "inventory" namespace = "urn:opendaylight:inventory" shard-strategy = "module" }, { name = "topology" namespace = "urn:TBD:params:xml:ns:yang:network-topology" shard-strategy = "module" }, { name = "toaster" namespace = "http://netconfcentral.org/ns/toaster" shard-strategy = "module" }, { name = "car" namespace = "urn:opendaylight:params:xml:ns:yang:controller:config:sal-clustering-it:car" shard-strategy = "module" }, { name = "people" namespace = "urn:opendaylight:params:xml:ns:yang:controller:config:sal-clustering-it:people" shard-strategy = "module" }, { name = "car-people" namespace = "urn:opendaylight:params:xml:ns:yang:controller:config:sal-clustering-it:car-people" shard-strategy = "module" } ] Configuring replication type in module-shards.conf ################################################ ## NOTE: Manually restart controller to ## ## apply configuration. ## ################################################ Dump pekko.conf + echo 'Dump pekko.conf' + cat /tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/pekko.conf odl-cluster-data { pekko { remote { artery { enabled = on transport = tcp canonical.hostname = "10.30.170.104" canonical.port = 2550 } } cluster { # Using artery. seed-nodes = ["pekko://opendaylight-cluster-data@10.30.170.104:2550", "pekko://opendaylight-cluster-data@10.30.171.129:2550", "pekko://opendaylight-cluster-data@10.30.170.156:2550"] roles = ["member-1"] # when under load we might trip a false positive on the failure detector # failure-detector { # heartbeat-interval = 4 s # acceptable-heartbeat-pause = 16s # } } persistence { # By default the snapshots/journal directories live in KARAF_HOME. You can choose to put it somewhere else by # modifying the following two properties. The directory location specified may be a relative or absolute path. # The relative path is always relative to KARAF_HOME. # snapshot-store.local.dir = "target/snapshots" } disable-default-actor-system-quarantined-event-handling = "false" } } Dump modules.conf + echo 'Dump modules.conf' + cat /tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/modules.conf modules = [ { name = "inventory" namespace = "urn:opendaylight:inventory" shard-strategy = "module" }, { name = "topology" namespace = "urn:TBD:params:xml:ns:yang:network-topology" shard-strategy = "module" }, { name = "toaster" namespace = "http://netconfcentral.org/ns/toaster" shard-strategy = "module" }, { name = "car" namespace = "urn:opendaylight:params:xml:ns:yang:controller:config:sal-clustering-it:car" shard-strategy = "module" }, { name = "people" namespace = "urn:opendaylight:params:xml:ns:yang:controller:config:sal-clustering-it:people" shard-strategy = "module" }, { name = "car-people" namespace = "urn:opendaylight:params:xml:ns:yang:controller:config:sal-clustering-it:car-people" shard-strategy = "module" } ] Dump module-shards.conf + echo 'Dump module-shards.conf' + cat /tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/module-shards.conf module-shards = [ { name = "default" shards = [ { name = "default" replicas = ["member-1", "member-2", "member-3"] } ] }, { name = "inventory" shards = [ { name="inventory" replicas = ["member-1", "member-2", "member-3"] } ] }, { name = "topology" shards = [ { name="topology" replicas = ["member-1", "member-2", "member-3"] } ] }, { name = "toaster" shards = [ { name="toaster" replicas = ["member-1", "member-2", "member-3"] } ] }, { name = "car" shards = [ { name="car" replicas = ["member-1", "member-2", "member-3"] } ] }, { name = "people" shards = [ { name="people" replicas = ["member-1", "member-2", "member-3"] } ] }, { name = "car-people" shards = [ { name="car-people" replicas = ["member-1", "member-2", "member-3"] } ] } ] Configuring member-2 with IP address 10.30.171.129 Warning: Permanently added '10.30.171.129' (ECDSA) to the list of known hosts. Warning: Permanently added '10.30.171.129' (ECDSA) to the list of known hosts. + source /tmp/common-functions.sh karaf-0.24.0-SNAPSHOT vanadium ++ [[ /tmp/common-functions.sh == \/\t\m\p\/\c\o\n\f\i\g\u\r\a\t\i\o\n\-\s\c\r\i\p\t\.\s\h ]] ++ echo 'common-functions.sh is being sourced' common-functions.sh is being sourced ++ BUNDLEFOLDER=karaf-0.24.0-SNAPSHOT ++ DISTROSTREAM=vanadium ++ export MAVENCONF=/tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.url.mvn.cfg ++ MAVENCONF=/tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.url.mvn.cfg ++ export FEATURESCONF=/tmp/karaf-0.24.0-SNAPSHOT/etc/org.apache.karaf.features.cfg ++ FEATURESCONF=/tmp/karaf-0.24.0-SNAPSHOT/etc/org.apache.karaf.features.cfg ++ export CUSTOMPROP=/tmp/karaf-0.24.0-SNAPSHOT/etc/custom.properties ++ CUSTOMPROP=/tmp/karaf-0.24.0-SNAPSHOT/etc/custom.properties ++ export LOGCONF=/tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.logging.cfg ++ LOGCONF=/tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.logging.cfg ++ export MEMCONF=/tmp/karaf-0.24.0-SNAPSHOT/bin/setenv ++ MEMCONF=/tmp/karaf-0.24.0-SNAPSHOT/bin/setenv ++ export CONTROLLERMEM= ++ CONTROLLERMEM= ++ case "${DISTROSTREAM}" in ++ CLUSTER_SYSTEM=pekko ++ export AKKACONF=/tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/pekko.conf ++ AKKACONF=/tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/pekko.conf ++ export MODULESCONF=/tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/modules.conf ++ MODULESCONF=/tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/modules.conf ++ export MODULESHARDSCONF=/tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/module-shards.conf ++ MODULESHARDSCONF=/tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/module-shards.conf ++ print_common_env ++ cat common-functions environment: MAVENCONF: /tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.url.mvn.cfg ACTUALFEATURES: FEATURESCONF: /tmp/karaf-0.24.0-SNAPSHOT/etc/org.apache.karaf.features.cfg CUSTOMPROP: /tmp/karaf-0.24.0-SNAPSHOT/etc/custom.properties LOGCONF: /tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.logging.cfg MEMCONF: /tmp/karaf-0.24.0-SNAPSHOT/bin/setenv CONTROLLERMEM: AKKACONF: /tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/pekko.conf MODULESCONF: /tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/modules.conf MODULESHARDSCONF: /tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/module-shards.conf SUITES: ++ SSH='ssh -t -t' ++ extra_services_cntl=' dnsmasq.service httpd.service libvirtd.service openvswitch.service ovs-vswitchd.service ovsdb-server.service rabbitmq-server.service ' ++ extra_services_cmp=' libvirtd.service openvswitch.service ovs-vswitchd.service ovsdb-server.service ' Changing to /tmp Downloading the distribution from https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/integration/karaf/0.24.0-SNAPSHOT/karaf-0.24.0-20260117.223656-12.zip + echo 'Changing to /tmp' + cd /tmp + echo 'Downloading the distribution from https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/integration/karaf/0.24.0-SNAPSHOT/karaf-0.24.0-20260117.223656-12.zip' + wget --progress=dot:mega https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/integration/karaf/0.24.0-SNAPSHOT/karaf-0.24.0-20260117.223656-12.zip --2026-01-17 23:43:50-- https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/integration/karaf/0.24.0-SNAPSHOT/karaf-0.24.0-20260117.223656-12.zip Resolving nexus.opendaylight.org (nexus.opendaylight.org)... 199.204.45.87, 2604:e100:1:0:f816:3eff:fe45:48d6 Connecting to nexus.opendaylight.org (nexus.opendaylight.org)|199.204.45.87|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 240365717 (229M) [application/zip] Saving to: ‘karaf-0.24.0-20260117.223656-12.zip’ 0K ........ ........ ........ ........ ........ ........ 1% 59.7M 4s 3072K ........ ........ ........ ........ ........ ........ 2% 93.6M 3s 6144K ........ ........ ........ ........ ........ ........ 3% 90.2M 3s 9216K ........ ........ ........ ........ ........ ........ 5% 109M 3s 12288K ........ ........ ........ ........ ........ ........ 6% 121M 2s 15360K ........ ........ ........ ........ ........ ........ 7% 165M 2s 18432K ........ ........ ........ ........ ........ ........ 9% 198M 2s 21504K ........ ........ ........ ........ ........ ........ 10% 211M 2s 24576K ........ ........ ........ ........ ........ ........ 11% 256M 2s 27648K ........ ........ ........ ........ ........ ........ 13% 252M 2s 30720K ........ ........ ........ ........ ........ ........ 14% 297M 1s 33792K ........ ........ ........ ........ ........ ........ 15% 286M 1s 36864K ........ ........ ........ ........ ........ ........ 17% 276M 1s 39936K ........ ........ ........ ........ ........ ........ 18% 273M 1s 43008K ........ ........ ........ ........ ........ ........ 19% 311M 1s 46080K ........ ........ ........ ........ ........ ........ 20% 289M 1s 49152K ........ ........ ........ ........ ........ ........ 22% 305M 1s 52224K ........ ........ ........ ........ ........ ........ 23% 328M 1s 55296K ........ ........ ........ ........ ........ ........ 24% 339M 1s 58368K ........ ........ ........ ........ ........ ........ 26% 339M 1s 61440K ........ ........ ........ ........ ........ ........ 27% 318M 1s 64512K ........ ........ ........ ........ ........ ........ 28% 336M 1s 67584K ........ ........ ........ ........ ........ ........ 30% 336M 1s 70656K ........ ........ ........ ........ ........ ........ 31% 322M 1s 73728K ........ ........ ........ ........ ........ ........ 32% 338M 1s 76800K ........ ........ ........ ........ ........ ........ 34% 310M 1s 79872K ........ ........ ........ ........ ........ ........ 35% 233M 1s 82944K ........ ........ ........ ........ ........ ........ 36% 289M 1s 86016K ........ ........ ........ ........ ........ ........ 37% 311M 1s 89088K ........ ........ ........ ........ ........ ........ 39% 320M 1s 92160K ........ ........ ........ ........ ........ ........ 40% 322M 1s 95232K ........ ........ ........ ........ ........ ........ 41% 326M 1s 98304K ........ ........ ........ ........ ........ ........ 43% 320M 1s 101376K ........ ........ ........ ........ ........ ........ 44% 328M 1s 104448K ........ ........ ........ ........ ........ ........ 45% 331M 1s 107520K ........ ........ ........ ........ ........ ........ 47% 329M 1s 110592K ........ ........ ........ ........ ........ ........ 48% 335M 1s 113664K ........ ........ ........ ........ ........ ........ 49% 330M 1s 116736K ........ ........ ........ ........ ........ ........ 51% 325M 0s 119808K ........ ........ ........ ........ ........ ........ 52% 301M 0s 122880K ........ ........ ........ ........ ........ ........ 53% 300M 0s 125952K ........ ........ ........ ........ ........ ........ 54% 236M 0s 129024K ........ ........ ........ ........ ........ ........ 56% 456M 0s 132096K ........ ........ ........ ........ ........ ........ 57% 257M 0s 135168K ........ ........ ........ ........ ........ ........ 58% 231M 0s 138240K ........ ........ ........ ........ ........ ........ 60% 212M 0s 141312K ........ ........ ........ ........ ........ ........ 61% 249M 0s 144384K ........ ........ ........ ........ ........ ........ 62% 269M 0s 147456K ........ ........ ........ ........ ........ ........ 64% 356M 0s 150528K ........ ........ ........ ........ ........ ........ 65% 352M 0s 153600K ........ ........ ........ ........ ........ ........ 66% 323M 0s 156672K ........ ........ ........ ........ ........ ........ 68% 299M 0s 159744K ........ ........ ........ ........ ........ ........ 69% 188M 0s 162816K ........ ........ ........ ........ ........ ........ 70% 397M 0s 165888K ........ ........ ........ ........ ........ ........ 71% 478M 0s 168960K ........ ........ ........ ........ ........ ........ 73% 503M 0s 172032K ........ ........ ........ ........ ........ ........ 74% 371M 0s 175104K ........ ........ ........ ........ ........ ........ 75% 325M 0s 178176K ........ ........ ........ ........ ........ ........ 77% 332M 0s 181248K ........ ........ ........ ........ ........ ........ 78% 332M 0s 184320K ........ ........ ........ ........ ........ ........ 79% 332M 0s 187392K ........ ........ ........ ........ ........ ........ 81% 340M 0s 190464K ........ ........ ........ ........ ........ ........ 82% 335M 0s 193536K ........ ........ ........ ........ ........ ........ 83% 318M 0s 196608K ........ ........ ........ ........ ........ ........ 85% 325M 0s 199680K ........ ........ ........ ........ ........ ........ 86% 324M 0s 202752K ........ ........ ........ ........ ........ ........ 87% 306M 0s 205824K ........ ........ ........ ........ ........ ........ 88% 318M 0s 208896K ........ ........ ........ ........ ........ ........ 90% 331M 0s 211968K ........ ........ ........ ........ ........ ........ 91% 335M 0s 215040K ........ ........ ........ ........ ........ ........ 92% 327M 0s 218112K ........ ........ ........ ........ ........ ........ 94% 330M 0s 221184K ........ ........ ........ ........ ........ ........ 95% 325M 0s 224256K ........ ........ ........ ........ ........ ........ 96% 338M 0s 227328K ........ ........ ........ ........ ........ ........ 98% 337M 0s 230400K ........ ........ ........ ........ ........ ........ 99% 329M 0s 233472K ........ ........ ... 100% 326M=0.9s 2026-01-17 23:43:51 (261 MB/s) - ‘karaf-0.24.0-20260117.223656-12.zip’ saved [240365717/240365717] Extracting the new controller... + echo 'Extracting the new controller...' + unzip -q karaf-0.24.0-20260117.223656-12.zip Adding external repositories... + echo 'Adding external repositories...' + sed -ie 's%org.ops4j.pax.url.mvn.repositories=%org.ops4j.pax.url.mvn.repositories=https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot@id=opendaylight-snapshot@snapshots, https://nexus.opendaylight.org/content/repositories/public@id=opendaylight-mirror, http://repo1.maven.org/maven2@id=central, http://repository.springsource.com/maven/bundles/release@id=spring.ebr.release, http://repository.springsource.com/maven/bundles/external@id=spring.ebr.external, http://zodiac.springsource.com/maven/bundles/release@id=gemini, http://repository.apache.org/content/groups/snapshots-group@id=apache@snapshots@noreleases, https://oss.sonatype.org/content/repositories/snapshots@id=sonatype.snapshots.deploy@snapshots@noreleases, https://oss.sonatype.org/content/repositories/ops4j-snapshots@id=ops4j.sonatype.snapshots.deploy@snapshots@noreleases%g' /tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.url.mvn.cfg + cat /tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.url.mvn.cfg ################################################################################ # # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ################################################################################ # # If set to true, the following property will not allow any certificate to be used # when accessing Maven repositories through SSL # #org.ops4j.pax.url.mvn.certificateCheck= # # Path to the local Maven settings file. # The repositories defined in this file will be automatically added to the list # of default repositories if the 'org.ops4j.pax.url.mvn.repositories' property # below is not set. # The following locations are checked for the existence of the settings.xml file # * 1. looks for the specified url # * 2. if not found looks for ${user.home}/.m2/settings.xml # * 3. if not found looks for ${maven.home}/conf/settings.xml # * 4. if not found looks for ${M2_HOME}/conf/settings.xml # #org.ops4j.pax.url.mvn.settings= # # Path to the local Maven repository which is used to avoid downloading # artifacts when they already exist locally. # The value of this property will be extracted from the settings.xml file # above, or defaulted to: # System.getProperty( "user.home" ) + "/.m2/repository" # org.ops4j.pax.url.mvn.localRepository=${karaf.home}/${karaf.default.repository} # # Default this to false. It's just weird to use undocumented repos # org.ops4j.pax.url.mvn.useFallbackRepositories=false # # Uncomment if you don't wanna use the proxy settings # from the Maven conf/settings.xml file # # org.ops4j.pax.url.mvn.proxySupport=false # # Comma separated list of repositories scanned when resolving an artifact. # Those repositories will be checked before iterating through the # below list of repositories and even before the local repository # A repository url can be appended with zero or more of the following flags: # @snapshots : the repository contains snaphots # @noreleases : the repository does not contain any released artifacts # # The following property value will add the system folder as a repo. # org.ops4j.pax.url.mvn.defaultRepositories=\ file:${karaf.home}/${karaf.default.repository}@id=system.repository@snapshots,\ file:${karaf.data}/kar@id=kar.repository@multi@snapshots,\ file:${karaf.base}/${karaf.default.repository}@id=child.system.repository@snapshots # Use the default local repo (e.g.~/.m2/repository) as a "remote" repo #org.ops4j.pax.url.mvn.defaultLocalRepoAsRemote=false # # Comma separated list of repositories scanned when resolving an artifact. # The default list includes the following repositories: # http://repo1.maven.org/maven2@id=central # http://repository.springsource.com/maven/bundles/release@id=spring.ebr # http://repository.springsource.com/maven/bundles/external@id=spring.ebr.external # http://zodiac.springsource.com/maven/bundles/release@id=gemini # http://repository.apache.org/content/groups/snapshots-group@id=apache@snapshots@noreleases # https://oss.sonatype.org/content/repositories/snapshots@id=sonatype.snapshots.deploy@snapshots@noreleases # https://oss.sonatype.org/content/repositories/ops4j-snapshots@id=ops4j.sonatype.snapshots.deploy@snapshots@noreleases # To add repositories to the default ones, prepend '+' to the list of repositories # to add. # A repository url can be appended with zero or more of the following flags: # @snapshots : the repository contains snapshots # @noreleases : the repository does not contain any released artifacts # @id=repository.id : the id for the repository, just like in the settings.xml this is optional but recommended # org.ops4j.pax.url.mvn.repositories=https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot@id=opendaylight-snapshot@snapshots, https://nexus.opendaylight.org/content/repositories/public@id=opendaylight-mirror, http://repo1.maven.org/maven2@id=central, http://repository.springsource.com/maven/bundles/release@id=spring.ebr.release, http://repository.springsource.com/maven/bundles/external@id=spring.ebr.external, http://zodiac.springsource.com/maven/bundles/release@id=gemini, http://repository.apache.org/content/groups/snapshots-group@id=apache@snapshots@noreleases, https://oss.sonatype.org/content/repositories/snapshots@id=sonatype.snapshots.deploy@snapshots@noreleases, https://oss.sonatype.org/content/repositories/ops4j-snapshots@id=ops4j.sonatype.snapshots.deploy@snapshots@noreleases ### ^^^ No remote repositories. This is the only ODL change compared to Karaf defaults.Configuring the startup features... + [[ True == \T\r\u\e ]] + echo 'Configuring the startup features...' + sed -ie 's/\(featuresBoot=\|featuresBoot =\)/featuresBoot = odl-integration-compatible-with-all,odl-jolokia,odl-restconf,odl-clustering-test-app,/g' /tmp/karaf-0.24.0-SNAPSHOT/etc/org.apache.karaf.features.cfg + FEATURE_TEST_STRING=features-test + FEATURE_TEST_VERSION=0.24.0-SNAPSHOT + KARAF_VERSION=karaf4 + [[ integration == \i\n\t\e\g\r\a\t\i\o\n ]] + sed -ie 's%\(featuresRepositories=\|featuresRepositories =\)%featuresRepositories = mvn:org.opendaylight.integration/features-test/0.24.0-SNAPSHOT/xml/features,mvn:org.apache.karaf.decanter/apache-karaf-decanter/1.2.0/xml/features,%g' /tmp/karaf-0.24.0-SNAPSHOT/etc/org.apache.karaf.features.cfg + [[ ! -z '' ]] + cat /tmp/karaf-0.24.0-SNAPSHOT/etc/org.apache.karaf.features.cfg ################################################################################ # # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ################################################################################ # # Comma separated list of features repositories to register by default # featuresRepositories = mvn:org.opendaylight.integration/features-test/0.24.0-SNAPSHOT/xml/features,mvn:org.apache.karaf.decanter/apache-karaf-decanter/1.2.0/xml/features, file:${karaf.etc}/eef4f053-53e3-43e4-9c13-1e5bf5e49fc9.xml # # Comma separated list of features to install at startup # featuresBoot = odl-integration-compatible-with-all,odl-jolokia,odl-restconf,odl-clustering-test-app, e0ed419d-f0e5-4080-bffe-2232822c3309 # # Resource repositories (OBR) that the features resolver can use # to resolve requirements/capabilities # # The format of the resourceRepositories is # resourceRepositories=[xml:url|json:url],... # for Instance: # #resourceRepositories=xml:http://host/path/to/index.xml # or #resourceRepositories=json:http://host/path/to/index.json # # # Defines if the boot features are started in asynchronous mode (in a dedicated thread) # featuresBootAsynchronous=false # # Service requirements enforcement # # By default, the feature resolver checks the service requirements/capabilities of # bundles for new features (xml schema >= 1.3.0) in order to automatically installs # the required bundles. # The following flag can have those values: # - disable: service requirements are completely ignored # - default: service requirements are ignored for old features # - enforce: service requirements are always verified # #serviceRequirements=default # # Store cfg file for config element in feature # #configCfgStore=true # # Define if the feature service automatically refresh bundles # autoRefresh=true # # Configuration of features processing mechanism (overrides, blacklisting, modification of features) # XML file defines instructions related to features processing # versions.properties may declare properties to resolve placeholders in XML file # both files are relative to ${karaf.etc} # #featureProcessing=org.apache.karaf.features.xml #featureProcessingVersions=versions.properties + configure_karaf_log karaf4 '' + local -r karaf_version=karaf4 + local -r controllerdebugmap= + local logapi=log4j + grep log4j2 /tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.logging.cfg log4j2.pattern = %d{ISO8601} | %-5p | %-16t | %-32c{1} | %X{bundle.id} - %X{bundle.name} - %X{bundle.version} | %m%n log4j2.rootLogger.level = INFO #log4j2.rootLogger.type = asyncRoot #log4j2.rootLogger.includeLocation = false log4j2.rootLogger.appenderRef.RollingFile.ref = RollingFile log4j2.rootLogger.appenderRef.PaxOsgi.ref = PaxOsgi log4j2.rootLogger.appenderRef.Console.ref = Console log4j2.rootLogger.appenderRef.Console.filter.threshold.type = ThresholdFilter log4j2.rootLogger.appenderRef.Console.filter.threshold.level = ${karaf.log.console:-OFF} log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.type = ContextMapFilter log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.pair1.type = KeyValuePair log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.pair1.key = slf4j.marker log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.pair1.value = CONFIDENTIAL log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.operator = or log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.onMatch = DENY log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.onMismatch = NEUTRAL log4j2.logger.spifly.name = org.apache.aries.spifly log4j2.logger.spifly.level = WARN log4j2.logger.audit.name = org.apache.karaf.jaas.modules.audit log4j2.logger.audit.level = INFO log4j2.logger.audit.additivity = false log4j2.logger.audit.appenderRef.AuditRollingFile.ref = AuditRollingFile # Console appender not used by default (see log4j2.rootLogger.appenderRefs) log4j2.appender.console.type = Console log4j2.appender.console.name = Console log4j2.appender.console.layout.type = PatternLayout log4j2.appender.console.layout.pattern = ${log4j2.pattern} log4j2.appender.rolling.type = RollingRandomAccessFile log4j2.appender.rolling.name = RollingFile log4j2.appender.rolling.fileName = ${karaf.data}/log/karaf.log log4j2.appender.rolling.filePattern = ${karaf.data}/log/karaf.log.%i #log4j2.appender.rolling.immediateFlush = false log4j2.appender.rolling.append = true log4j2.appender.rolling.layout.type = PatternLayout log4j2.appender.rolling.layout.pattern = ${log4j2.pattern} log4j2.appender.rolling.policies.type = Policies log4j2.appender.rolling.policies.size.type = SizeBasedTriggeringPolicy log4j2.appender.rolling.policies.size.size = 64MB log4j2.appender.rolling.strategy.type = DefaultRolloverStrategy log4j2.appender.rolling.strategy.max = 7 log4j2.appender.audit.type = RollingRandomAccessFile log4j2.appender.audit.name = AuditRollingFile log4j2.appender.audit.fileName = ${karaf.data}/security/audit.log log4j2.appender.audit.filePattern = ${karaf.data}/security/audit.log.%i log4j2.appender.audit.append = true log4j2.appender.audit.layout.type = PatternLayout log4j2.appender.audit.layout.pattern = ${log4j2.pattern} log4j2.appender.audit.policies.type = Policies log4j2.appender.audit.policies.size.type = SizeBasedTriggeringPolicy log4j2.appender.audit.policies.size.size = 8MB log4j2.appender.audit.strategy.type = DefaultRolloverStrategy log4j2.appender.audit.strategy.max = 7 log4j2.appender.osgi.type = PaxOsgi log4j2.appender.osgi.name = PaxOsgi log4j2.appender.osgi.filter = * #log4j2.logger.aether.name = shaded.org.eclipse.aether #log4j2.logger.aether.level = TRACE #log4j2.logger.http-headers.name = shaded.org.apache.http.headers #log4j2.logger.http-headers.level = DEBUG #log4j2.logger.maven.name = org.ops4j.pax.url.mvn #log4j2.logger.maven.level = TRACE + logapi=log4j2 Configuring the karaf log... karaf_version: karaf4, logapi: log4j2 + echo 'Configuring the karaf log... karaf_version: karaf4, logapi: log4j2' + '[' log4j2 == log4j2 ']' + sed -ie 's/log4j2.appender.rolling.policies.size.size = 64MB/log4j2.appender.rolling.policies.size.size = 1GB/g' /tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.logging.cfg controllerdebugmap: cat /tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.logging.cfg + orgmodule=org.opendaylight.yangtools.yang.parser.repo.YangTextSchemaContextResolver + orgmodule_=org_opendaylight_yangtools_yang_parser_repo_YangTextSchemaContextResolver + echo 'log4j2.logger.org_opendaylight_yangtools_yang_parser_repo_YangTextSchemaContextResolver.name = WARN' + echo 'log4j2.logger.org_opendaylight_yangtools_yang_parser_repo_YangTextSchemaContextResolver.level = WARN' + unset IFS + echo 'controllerdebugmap: ' + '[' -n '' ']' + echo 'cat /tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.logging.cfg' + cat /tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.logging.cfg ################################################################################ # # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ################################################################################ # Common pattern layout for appenders log4j2.pattern = %d{ISO8601} | %-5p | %-16t | %-32c{1} | %X{bundle.id} - %X{bundle.name} - %X{bundle.version} | %m%n # Root logger log4j2.rootLogger.level = INFO # uncomment to use asynchronous loggers, which require mvn:com.lmax/disruptor/3.3.2 library #log4j2.rootLogger.type = asyncRoot #log4j2.rootLogger.includeLocation = false log4j2.rootLogger.appenderRef.RollingFile.ref = RollingFile log4j2.rootLogger.appenderRef.PaxOsgi.ref = PaxOsgi log4j2.rootLogger.appenderRef.Console.ref = Console log4j2.rootLogger.appenderRef.Console.filter.threshold.type = ThresholdFilter log4j2.rootLogger.appenderRef.Console.filter.threshold.level = ${karaf.log.console:-OFF} # Filters for logs marked by org.opendaylight.odlparent.Markers log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.type = ContextMapFilter log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.pair1.type = KeyValuePair log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.pair1.key = slf4j.marker log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.pair1.value = CONFIDENTIAL log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.operator = or log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.onMatch = DENY log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.onMismatch = NEUTRAL # Loggers configuration # Spifly logger log4j2.logger.spifly.name = org.apache.aries.spifly log4j2.logger.spifly.level = WARN # Security audit logger log4j2.logger.audit.name = org.apache.karaf.jaas.modules.audit log4j2.logger.audit.level = INFO log4j2.logger.audit.additivity = false log4j2.logger.audit.appenderRef.AuditRollingFile.ref = AuditRollingFile # Appenders configuration # Console appender not used by default (see log4j2.rootLogger.appenderRefs) log4j2.appender.console.type = Console log4j2.appender.console.name = Console log4j2.appender.console.layout.type = PatternLayout log4j2.appender.console.layout.pattern = ${log4j2.pattern} # Rolling file appender log4j2.appender.rolling.type = RollingRandomAccessFile log4j2.appender.rolling.name = RollingFile log4j2.appender.rolling.fileName = ${karaf.data}/log/karaf.log log4j2.appender.rolling.filePattern = ${karaf.data}/log/karaf.log.%i # uncomment to not force a disk flush #log4j2.appender.rolling.immediateFlush = false log4j2.appender.rolling.append = true log4j2.appender.rolling.layout.type = PatternLayout log4j2.appender.rolling.layout.pattern = ${log4j2.pattern} log4j2.appender.rolling.policies.type = Policies log4j2.appender.rolling.policies.size.type = SizeBasedTriggeringPolicy log4j2.appender.rolling.policies.size.size = 1GB log4j2.appender.rolling.strategy.type = DefaultRolloverStrategy log4j2.appender.rolling.strategy.max = 7 # Audit file appender log4j2.appender.audit.type = RollingRandomAccessFile log4j2.appender.audit.name = AuditRollingFile log4j2.appender.audit.fileName = ${karaf.data}/security/audit.log log4j2.appender.audit.filePattern = ${karaf.data}/security/audit.log.%i log4j2.appender.audit.append = true log4j2.appender.audit.layout.type = PatternLayout log4j2.appender.audit.layout.pattern = ${log4j2.pattern} log4j2.appender.audit.policies.type = Policies log4j2.appender.audit.policies.size.type = SizeBasedTriggeringPolicy log4j2.appender.audit.policies.size.size = 8MB log4j2.appender.audit.strategy.type = DefaultRolloverStrategy log4j2.appender.audit.strategy.max = 7 # OSGi appender log4j2.appender.osgi.type = PaxOsgi log4j2.appender.osgi.name = PaxOsgi log4j2.appender.osgi.filter = * # help with identification of maven-related problems with pax-url-aether #log4j2.logger.aether.name = shaded.org.eclipse.aether #log4j2.logger.aether.level = TRACE #log4j2.logger.http-headers.name = shaded.org.apache.http.headers #log4j2.logger.http-headers.level = DEBUG #log4j2.logger.maven.name = org.ops4j.pax.url.mvn #log4j2.logger.maven.level = TRACE log4j2.logger.org_opendaylight_yangtools_yang_parser_repo_YangTextSchemaContextResolver.name = WARN log4j2.logger.org_opendaylight_yangtools_yang_parser_repo_YangTextSchemaContextResolver.level = WARN + set_java_vars /usr/lib/jvm/java-21-openjdk-amd64 3072m /tmp/karaf-0.24.0-SNAPSHOT/bin/setenv + local -r java_home=/usr/lib/jvm/java-21-openjdk-amd64 + local -r controllermem=3072m + local -r memconf=/tmp/karaf-0.24.0-SNAPSHOT/bin/setenv Configure java home: /usr/lib/jvm/java-21-openjdk-amd64 max memory: 3072m memconf: /tmp/karaf-0.24.0-SNAPSHOT/bin/setenv + echo Configure + echo ' java home: /usr/lib/jvm/java-21-openjdk-amd64' + echo ' max memory: 3072m' + echo ' memconf: /tmp/karaf-0.24.0-SNAPSHOT/bin/setenv' + sed -ie 's%^# export JAVA_HOME%export JAVA_HOME=${JAVA_HOME:-/usr/lib/jvm/java-21-openjdk-amd64}%g' /tmp/karaf-0.24.0-SNAPSHOT/bin/setenv + sed -ie 's/JAVA_MAX_MEM="2048m"/JAVA_MAX_MEM=3072m/g' /tmp/karaf-0.24.0-SNAPSHOT/bin/setenv cat /tmp/karaf-0.24.0-SNAPSHOT/bin/setenv + echo 'cat /tmp/karaf-0.24.0-SNAPSHOT/bin/setenv' + cat /tmp/karaf-0.24.0-SNAPSHOT/bin/setenv #!/bin/sh # # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # # handle specific scripts; the SCRIPT_NAME is exactly the name of the Karaf # script: client, instance, shell, start, status, stop, karaf # # if [ "${KARAF_SCRIPT}" == "SCRIPT_NAME" ]; then # Actions go here... # fi # # general settings which should be applied for all scripts go here; please keep # in mind that it is possible that scripts might be executed more than once, e.g. # in example of the start script where the start script is executed first and the # karaf script afterwards. # # # The following section shows the possible configuration options for the default # karaf scripts # export JAVA_HOME=${JAVA_HOME:-/usr/lib/jvm/java-21-openjdk-amd64} # Location of Java installation # export JAVA_OPTS # Generic JVM options, for instance, where you can pass the memory configuration # export JAVA_NON_DEBUG_OPTS # Additional non-debug JVM options # export EXTRA_JAVA_OPTS # Additional JVM options # export KARAF_HOME # Karaf home folder # export KARAF_DATA # Karaf data folder # export KARAF_BASE # Karaf base folder # export KARAF_ETC # Karaf etc folder # export KARAF_LOG # Karaf log folder # export KARAF_SYSTEM_OPTS # First citizen Karaf options # export KARAF_OPTS # Additional available Karaf options # export KARAF_DEBUG # Enable debug mode # export KARAF_REDIRECT # Enable/set the std/err redirection when using bin/start # export KARAF_NOROOT # Prevent execution as root if set to true Set Java version + echo 'Set Java version' + sudo /usr/sbin/alternatives --install /usr/bin/java java /usr/lib/jvm/java-21-openjdk-amd64/bin/java 1 sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper sudo: a password is required + sudo /usr/sbin/alternatives --set java /usr/lib/jvm/java-21-openjdk-amd64/bin/java sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper sudo: a password is required JDK default version ... + echo 'JDK default version ...' + java -version openjdk version "21.0.9" 2025-10-21 OpenJDK Runtime Environment (build 21.0.9+10-Ubuntu-122.04) OpenJDK 64-Bit Server VM (build 21.0.9+10-Ubuntu-122.04, mixed mode, sharing) Set JAVA_HOME + echo 'Set JAVA_HOME' + export JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64 ++ readlink -e /usr/lib/jvm/java-21-openjdk-amd64/bin/java Java binary pointed at by JAVA_HOME: /usr/lib/jvm/java-21-openjdk-amd64/bin/java + JAVA_RESOLVED=/usr/lib/jvm/java-21-openjdk-amd64/bin/java + echo 'Java binary pointed at by JAVA_HOME: /usr/lib/jvm/java-21-openjdk-amd64/bin/java' Listing all open ports on controller system... + echo 'Listing all open ports on controller system...' + netstat -pnatu /tmp/configuration-script.sh: line 40: netstat: command not found Custom shard config exists!!! Copying the shard config... + '[' -f /tmp/custom_shard_config.txt ']' + echo 'Custom shard config exists!!!' + echo 'Copying the shard config...' + cp /tmp/custom_shard_config.txt /tmp/karaf-0.24.0-SNAPSHOT/bin/ Configuring cluster + echo 'Configuring cluster' + /tmp/karaf-0.24.0-SNAPSHOT/bin/configure_cluster.sh 2 10.30.170.104 10.30.171.129 10.30.170.156 ################################################ ## Configure Cluster ## ################################################ NOTE: Cluster configuration files not found. Copying from /tmp/karaf-0.24.0-SNAPSHOT/system/org/opendaylight/controller/sal-clustering-config/12.0.3 Configuring unique name in pekko.conf Configuring hostname in pekko.conf Configuring data and rpc seed nodes in pekko.conf modules = [ { name = "inventory" namespace = "urn:opendaylight:inventory" shard-strategy = "module" }, { name = "topology" namespace = "urn:TBD:params:xml:ns:yang:network-topology" shard-strategy = "module" }, { name = "toaster" namespace = "http://netconfcentral.org/ns/toaster" shard-strategy = "module" }, { name = "car" namespace = "urn:opendaylight:params:xml:ns:yang:controller:config:sal-clustering-it:car" shard-strategy = "module" }, { name = "people" namespace = "urn:opendaylight:params:xml:ns:yang:controller:config:sal-clustering-it:people" shard-strategy = "module" }, { name = "car-people" namespace = "urn:opendaylight:params:xml:ns:yang:controller:config:sal-clustering-it:car-people" shard-strategy = "module" } ] Configuring replication type in module-shards.conf ################################################ ## NOTE: Manually restart controller to ## ## apply configuration. ## ################################################ Dump pekko.conf + echo 'Dump pekko.conf' + cat /tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/pekko.conf odl-cluster-data { pekko { remote { artery { enabled = on transport = tcp canonical.hostname = "10.30.171.129" canonical.port = 2550 } } cluster { # Using artery. seed-nodes = ["pekko://opendaylight-cluster-data@10.30.170.104:2550", "pekko://opendaylight-cluster-data@10.30.171.129:2550", "pekko://opendaylight-cluster-data@10.30.170.156:2550"] roles = ["member-2"] # when under load we might trip a false positive on the failure detector # failure-detector { # heartbeat-interval = 4 s # acceptable-heartbeat-pause = 16s # } } persistence { # By default the snapshots/journal directories live in KARAF_HOME. You can choose to put it somewhere else by # modifying the following two properties. The directory location specified may be a relative or absolute path. # The relative path is always relative to KARAF_HOME. # snapshot-store.local.dir = "target/snapshots" } disable-default-actor-system-quarantined-event-handling = "false" } } Dump modules.conf + echo 'Dump modules.conf' + cat /tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/modules.conf modules = [ { name = "inventory" namespace = "urn:opendaylight:inventory" shard-strategy = "module" }, { name = "topology" namespace = "urn:TBD:params:xml:ns:yang:network-topology" shard-strategy = "module" }, { name = "toaster" namespace = "http://netconfcentral.org/ns/toaster" shard-strategy = "module" }, { name = "car" namespace = "urn:opendaylight:params:xml:ns:yang:controller:config:sal-clustering-it:car" shard-strategy = "module" }, { name = "people" namespace = "urn:opendaylight:params:xml:ns:yang:controller:config:sal-clustering-it:people" shard-strategy = "module" }, { name = "car-people" namespace = "urn:opendaylight:params:xml:ns:yang:controller:config:sal-clustering-it:car-people" shard-strategy = "module" } ] Dump module-shards.conf + echo 'Dump module-shards.conf' + cat /tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/module-shards.conf module-shards = [ { name = "default" shards = [ { name = "default" replicas = ["member-1", "member-2", "member-3"] } ] }, { name = "inventory" shards = [ { name="inventory" replicas = ["member-1", "member-2", "member-3"] } ] }, { name = "topology" shards = [ { name="topology" replicas = ["member-1", "member-2", "member-3"] } ] }, { name = "toaster" shards = [ { name="toaster" replicas = ["member-1", "member-2", "member-3"] } ] }, { name = "car" shards = [ { name="car" replicas = ["member-1", "member-2", "member-3"] } ] }, { name = "people" shards = [ { name="people" replicas = ["member-1", "member-2", "member-3"] } ] }, { name = "car-people" shards = [ { name="car-people" replicas = ["member-1", "member-2", "member-3"] } ] } ] Configuring member-3 with IP address 10.30.170.156 Warning: Permanently added '10.30.170.156' (ECDSA) to the list of known hosts. Warning: Permanently added '10.30.170.156' (ECDSA) to the list of known hosts. + source /tmp/common-functions.sh karaf-0.24.0-SNAPSHOT vanadium ++ [[ /tmp/common-functions.sh == \/\t\m\p\/\c\o\n\f\i\g\u\r\a\t\i\o\n\-\s\c\r\i\p\t\.\s\h ]] ++ echo 'common-functions.sh is being sourced' common-functions.sh is being sourced ++ BUNDLEFOLDER=karaf-0.24.0-SNAPSHOT ++ DISTROSTREAM=vanadium ++ export MAVENCONF=/tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.url.mvn.cfg ++ MAVENCONF=/tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.url.mvn.cfg ++ export FEATURESCONF=/tmp/karaf-0.24.0-SNAPSHOT/etc/org.apache.karaf.features.cfg ++ FEATURESCONF=/tmp/karaf-0.24.0-SNAPSHOT/etc/org.apache.karaf.features.cfg ++ export CUSTOMPROP=/tmp/karaf-0.24.0-SNAPSHOT/etc/custom.properties ++ CUSTOMPROP=/tmp/karaf-0.24.0-SNAPSHOT/etc/custom.properties ++ export LOGCONF=/tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.logging.cfg ++ LOGCONF=/tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.logging.cfg ++ export MEMCONF=/tmp/karaf-0.24.0-SNAPSHOT/bin/setenv ++ MEMCONF=/tmp/karaf-0.24.0-SNAPSHOT/bin/setenv ++ export CONTROLLERMEM= ++ CONTROLLERMEM= ++ case "${DISTROSTREAM}" in ++ CLUSTER_SYSTEM=pekko ++ export AKKACONF=/tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/pekko.conf ++ AKKACONF=/tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/pekko.conf ++ export MODULESCONF=/tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/modules.conf ++ MODULESCONF=/tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/modules.conf ++ export MODULESHARDSCONF=/tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/module-shards.conf ++ MODULESHARDSCONF=/tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/module-shards.conf ++ print_common_env ++ cat common-functions environment: MAVENCONF: /tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.url.mvn.cfg ACTUALFEATURES: FEATURESCONF: /tmp/karaf-0.24.0-SNAPSHOT/etc/org.apache.karaf.features.cfg CUSTOMPROP: /tmp/karaf-0.24.0-SNAPSHOT/etc/custom.properties LOGCONF: /tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.logging.cfg MEMCONF: /tmp/karaf-0.24.0-SNAPSHOT/bin/setenv CONTROLLERMEM: AKKACONF: /tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/pekko.conf MODULESCONF: /tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/modules.conf MODULESHARDSCONF: /tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/module-shards.conf SUITES: ++ SSH='ssh -t -t' ++ extra_services_cntl=' dnsmasq.service httpd.service libvirtd.service openvswitch.service ovs-vswitchd.service ovsdb-server.service rabbitmq-server.service ' ++ extra_services_cmp=' libvirtd.service openvswitch.service ovs-vswitchd.service ovsdb-server.service ' Changing to /tmp + echo 'Changing to /tmp' + cd /tmp Downloading the distribution from https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/integration/karaf/0.24.0-SNAPSHOT/karaf-0.24.0-20260117.223656-12.zip + echo 'Downloading the distribution from https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/integration/karaf/0.24.0-SNAPSHOT/karaf-0.24.0-20260117.223656-12.zip' + wget --progress=dot:mega https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/integration/karaf/0.24.0-SNAPSHOT/karaf-0.24.0-20260117.223656-12.zip --2026-01-17 23:43:54-- https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/integration/karaf/0.24.0-SNAPSHOT/karaf-0.24.0-20260117.223656-12.zip Resolving nexus.opendaylight.org (nexus.opendaylight.org)... 199.204.45.87, 2604:e100:1:0:f816:3eff:fe45:48d6 Connecting to nexus.opendaylight.org (nexus.opendaylight.org)|199.204.45.87|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 240365717 (229M) [application/zip] Saving to: ‘karaf-0.24.0-20260117.223656-12.zip’ 0K ........ ........ ........ ........ ........ ........ 1% 55.4M 4s 3072K ........ ........ ........ ........ ........ ........ 2% 99.5M 3s 6144K ........ ........ ........ ........ ........ ........ 3% 118M 3s 9216K ........ ........ ........ ........ ........ ........ 5% 102M 3s 12288K ........ ........ ........ ........ ........ ........ 6% 141M 2s 15360K ........ ........ ........ ........ ........ ........ 7% 118M 2s 18432K ........ ........ ........ ........ ........ ........ 9% 106M 2s 21504K ........ ........ ........ ........ ........ ........ 10% 129M 2s 24576K ........ ........ ........ ........ ........ ........ 11% 165M 2s 27648K ........ ........ ........ ........ ........ ........ 13% 142M 2s 30720K ........ ........ ........ ........ ........ ........ 14% 155M 2s 33792K ........ ........ ........ ........ ........ ........ 15% 130M 2s 36864K ........ ........ ........ ........ ........ ........ 17% 153M 2s 39936K ........ ........ ........ ........ ........ ........ 18% 173M 2s 43008K ........ ........ ........ ........ ........ ........ 19% 198M 2s 46080K ........ ........ ........ ........ ........ ........ 20% 175M 1s 49152K ........ ........ ........ ........ ........ ........ 22% 150M 1s 52224K ........ ........ ........ ........ ........ ........ 23% 161M 1s 55296K ........ ........ ........ ........ ........ ........ 24% 156M 1s 58368K ........ ........ ........ ........ ........ ........ 26% 217M 1s 61440K ........ ........ ........ ........ ........ ........ 27% 272M 1s 64512K ........ ........ ........ ........ ........ ........ 28% 263M 1s 67584K ........ ........ ........ ........ ........ ........ 30% 242M 1s 70656K ........ ........ ........ ........ ........ ........ 31% 266M 1s 73728K ........ ........ ........ ........ ........ ........ 32% 264M 1s 76800K ........ ........ ........ ........ ........ ........ 34% 266M 1s 79872K ........ ........ ........ ........ ........ ........ 35% 289M 1s 82944K ........ ........ ........ ........ ........ ........ 36% 301M 1s 86016K ........ ........ ........ ........ ........ ........ 37% 289M 1s 89088K ........ ........ ........ ........ ........ ........ 39% 278M 1s 92160K ........ ........ ........ ........ ........ ........ 40% 291M 1s 95232K ........ ........ ........ ........ ........ ........ 41% 270M 1s 98304K ........ ........ ........ ........ ........ ........ 43% 267M 1s 101376K ........ ........ ........ ........ ........ ........ 44% 261M 1s 104448K ........ ........ ........ ........ ........ ........ 45% 278M 1s 107520K ........ ........ ........ ........ ........ ........ 47% 271M 1s 110592K ........ ........ ........ ........ ........ ........ 48% 258M 1s 113664K ........ ........ ........ ........ ........ ........ 49% 301M 1s 116736K ........ ........ ........ ........ ........ ........ 51% 287M 1s 119808K ........ ........ ........ ........ ........ ........ 52% 288M 1s 122880K ........ ........ ........ ........ ........ ........ 53% 287M 1s 125952K ........ ........ ........ ........ ........ ........ 54% 285M 1s 129024K ........ ........ ........ ........ ........ ........ 56% 287M 1s 132096K ........ ........ ........ ........ ........ ........ 57% 269M 1s 135168K ........ ........ ........ ........ ........ ........ 58% 230M 1s 138240K ........ ........ ........ ........ ........ ........ 60% 268M 0s 141312K ........ ........ ........ ........ ........ ........ 61% 278M 0s 144384K ........ ........ ........ ........ ........ ........ 62% 297M 0s 147456K ........ ........ ........ ........ ........ ........ 64% 281M 0s 150528K ........ ........ ........ ........ ........ ........ 65% 274M 0s 153600K ........ ........ ........ ........ ........ ........ 66% 287M 0s 156672K ........ ........ ........ ........ ........ ........ 68% 285M 0s 159744K ........ ........ ........ ........ ........ ........ 69% 288M 0s 162816K ........ ........ ........ ........ ........ ........ 70% 284M 0s 165888K ........ ........ ........ ........ ........ ........ 71% 287M 0s 168960K ........ ........ ........ ........ ........ ........ 73% 285M 0s 172032K ........ ........ ........ ........ ........ ........ 74% 283M 0s 175104K ........ ........ ........ ........ ........ ........ 75% 288M 0s 178176K ........ ........ ........ ........ ........ ........ 77% 249M 0s 181248K ........ ........ ........ ........ ........ ........ 78% 342M 0s 184320K ........ ........ ........ ........ ........ ........ 79% 295M 0s 187392K ........ ........ ........ ........ ........ ........ 81% 288M 0s 190464K ........ ........ ........ ........ ........ ........ 82% 283M 0s 193536K ........ ........ ........ ........ ........ ........ 83% 266M 0s 196608K ........ ........ ........ ........ ........ ........ 85% 300M 0s 199680K ........ ........ ........ ........ ........ ........ 86% 276M 0s 202752K ........ ........ ........ ........ ........ ........ 87% 281M 0s 205824K ........ ........ ........ ........ ........ ........ 88% 284M 0s 208896K ........ ........ ........ ........ ........ ........ 90% 282M 0s 211968K ........ ........ ........ ........ ........ ........ 91% 287M 0s 215040K ........ ........ ........ ........ ........ ........ 92% 285M 0s 218112K ........ ........ ........ ........ ........ ........ 94% 269M 0s 221184K ........ ........ ........ ........ ........ ........ 95% 282M 0s 224256K ........ ........ ........ ........ ........ ........ 96% 285M 0s 227328K ........ ........ ........ ........ ........ ........ 98% 283M 0s 230400K ........ ........ ........ ........ ........ ........ 99% 278M 0s 233472K ........ ........ ... 100% 277M=1.1s 2026-01-17 23:43:55 (215 MB/s) - ‘karaf-0.24.0-20260117.223656-12.zip’ saved [240365717/240365717] + echo 'Extracting the new controller...' Extracting the new controller... + unzip -q karaf-0.24.0-20260117.223656-12.zip Adding external repositories... + echo 'Adding external repositories...' + sed -ie 's%org.ops4j.pax.url.mvn.repositories=%org.ops4j.pax.url.mvn.repositories=https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot@id=opendaylight-snapshot@snapshots, https://nexus.opendaylight.org/content/repositories/public@id=opendaylight-mirror, http://repo1.maven.org/maven2@id=central, http://repository.springsource.com/maven/bundles/release@id=spring.ebr.release, http://repository.springsource.com/maven/bundles/external@id=spring.ebr.external, http://zodiac.springsource.com/maven/bundles/release@id=gemini, http://repository.apache.org/content/groups/snapshots-group@id=apache@snapshots@noreleases, https://oss.sonatype.org/content/repositories/snapshots@id=sonatype.snapshots.deploy@snapshots@noreleases, https://oss.sonatype.org/content/repositories/ops4j-snapshots@id=ops4j.sonatype.snapshots.deploy@snapshots@noreleases%g' /tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.url.mvn.cfg + cat /tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.url.mvn.cfg ################################################################################ # # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ################################################################################ # # If set to true, the following property will not allow any certificate to be used # when accessing Maven repositories through SSL # #org.ops4j.pax.url.mvn.certificateCheck= # # Path to the local Maven settings file. # The repositories defined in this file will be automatically added to the list # of default repositories if the 'org.ops4j.pax.url.mvn.repositories' property # below is not set. # The following locations are checked for the existence of the settings.xml file # * 1. looks for the specified url # * 2. if not found looks for ${user.home}/.m2/settings.xml # * 3. if not found looks for ${maven.home}/conf/settings.xml # * 4. if not found looks for ${M2_HOME}/conf/settings.xml # #org.ops4j.pax.url.mvn.settings= # # Path to the local Maven repository which is used to avoid downloading # artifacts when they already exist locally. # The value of this property will be extracted from the settings.xml file # above, or defaulted to: # System.getProperty( "user.home" ) + "/.m2/repository" # org.ops4j.pax.url.mvn.localRepository=${karaf.home}/${karaf.default.repository} # # Default this to false. It's just weird to use undocumented repos # org.ops4j.pax.url.mvn.useFallbackRepositories=false # # Uncomment if you don't wanna use the proxy settings # from the Maven conf/settings.xml file # # org.ops4j.pax.url.mvn.proxySupport=false # # Comma separated list of repositories scanned when resolving an artifact. # Those repositories will be checked before iterating through the # below list of repositories and even before the local repository # A repository url can be appended with zero or more of the following flags: # @snapshots : the repository contains snaphots # @noreleases : the repository does not contain any released artifacts # # The following property value will add the system folder as a repo. # org.ops4j.pax.url.mvn.defaultRepositories=\ file:${karaf.home}/${karaf.default.repository}@id=system.repository@snapshots,\ file:${karaf.data}/kar@id=kar.repository@multi@snapshots,\ file:${karaf.base}/${karaf.default.repository}@id=child.system.repository@snapshots # Use the default local repo (e.g.~/.m2/repository) as a "remote" repo #org.ops4j.pax.url.mvn.defaultLocalRepoAsRemote=false # # Comma separated list of repositories scanned when resolving an artifact. # The default list includes the following repositories: # http://repo1.maven.org/maven2@id=central # http://repository.springsource.com/maven/bundles/release@id=spring.ebr # http://repository.springsource.com/maven/bundles/external@id=spring.ebr.external # http://zodiac.springsource.com/maven/bundles/release@id=gemini # http://repository.apache.org/content/groups/snapshots-group@id=apache@snapshots@noreleases # https://oss.sonatype.org/content/repositories/snapshots@id=sonatype.snapshots.deploy@snapshots@noreleases # https://oss.sonatype.org/content/repositories/ops4j-snapshots@id=ops4j.sonatype.snapshots.deploy@snapshots@noreleases # To add repositories to the default ones, prepend '+' to the list of repositories # to add. # A repository url can be appended with zero or more of the following flags: # @snapshots : the repository contains snapshots # @noreleases : the repository does not contain any released artifacts # @id=repository.id : the id for the repository, just like in the settings.xml this is optional but recommended # org.ops4j.pax.url.mvn.repositories=https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot@id=opendaylight-snapshot@snapshots, https://nexus.opendaylight.org/content/repositories/public@id=opendaylight-mirror, http://repo1.maven.org/maven2@id=central, http://repository.springsource.com/maven/bundles/release@id=spring.ebr.release, http://repository.springsource.com/maven/bundles/external@id=spring.ebr.external, http://zodiac.springsource.com/maven/bundles/release@id=gemini, http://repository.apache.org/content/groups/snapshots-group@id=apache@snapshots@noreleases, https://oss.sonatype.org/content/repositories/snapshots@id=sonatype.snapshots.deploy@snapshots@noreleases, https://oss.sonatype.org/content/repositories/ops4j-snapshots@id=ops4j.sonatype.snapshots.deploy@snapshots@noreleases ### ^^^ No remote repositories. This is the only ODL change compared to Karaf defaults.Configuring the startup features... + [[ True == \T\r\u\e ]] + echo 'Configuring the startup features...' + sed -ie 's/\(featuresBoot=\|featuresBoot =\)/featuresBoot = odl-integration-compatible-with-all,odl-jolokia,odl-restconf,odl-clustering-test-app,/g' /tmp/karaf-0.24.0-SNAPSHOT/etc/org.apache.karaf.features.cfg + FEATURE_TEST_STRING=features-test + FEATURE_TEST_VERSION=0.24.0-SNAPSHOT + KARAF_VERSION=karaf4 + [[ integration == \i\n\t\e\g\r\a\t\i\o\n ]] + sed -ie 's%\(featuresRepositories=\|featuresRepositories =\)%featuresRepositories = mvn:org.opendaylight.integration/features-test/0.24.0-SNAPSHOT/xml/features,mvn:org.apache.karaf.decanter/apache-karaf-decanter/1.2.0/xml/features,%g' /tmp/karaf-0.24.0-SNAPSHOT/etc/org.apache.karaf.features.cfg + [[ ! -z '' ]] + cat /tmp/karaf-0.24.0-SNAPSHOT/etc/org.apache.karaf.features.cfg ################################################################################ # # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ################################################################################ # # Comma separated list of features repositories to register by default # featuresRepositories = mvn:org.opendaylight.integration/features-test/0.24.0-SNAPSHOT/xml/features,mvn:org.apache.karaf.decanter/apache-karaf-decanter/1.2.0/xml/features, file:${karaf.etc}/eef4f053-53e3-43e4-9c13-1e5bf5e49fc9.xml # # Comma separated list of features to install at startup # featuresBoot = odl-integration-compatible-with-all,odl-jolokia,odl-restconf,odl-clustering-test-app, e0ed419d-f0e5-4080-bffe-2232822c3309 # # Resource repositories (OBR) that the features resolver can use # to resolve requirements/capabilities # # The format of the resourceRepositories is # resourceRepositories=[xml:url|json:url],... # for Instance: # #resourceRepositories=xml:http://host/path/to/index.xml # or #resourceRepositories=json:http://host/path/to/index.json # # # Defines if the boot features are started in asynchronous mode (in a dedicated thread) # featuresBootAsynchronous=false # # Service requirements enforcement # # By default, the feature resolver checks the service requirements/capabilities of # bundles for new features (xml schema >= 1.3.0) in order to automatically installs # the required bundles. # The following flag can have those values: # - disable: service requirements are completely ignored # - default: service requirements are ignored for old features # - enforce: service requirements are always verified # #serviceRequirements=default # # Store cfg file for config element in feature # #configCfgStore=true # # Define if the feature service automatically refresh bundles # autoRefresh=true # # Configuration of features processing mechanism (overrides, blacklisting, modification of features) # XML file defines instructions related to features processing # versions.properties may declare properties to resolve placeholders in XML file # both files are relative to ${karaf.etc} # #featureProcessing=org.apache.karaf.features.xml #featureProcessingVersions=versions.properties + configure_karaf_log karaf4 '' + local -r karaf_version=karaf4 + local -r controllerdebugmap= + local logapi=log4j + grep log4j2 /tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.logging.cfg log4j2.pattern = %d{ISO8601} | %-5p | %-16t | %-32c{1} | %X{bundle.id} - %X{bundle.name} - %X{bundle.version} | %m%n log4j2.rootLogger.level = INFO #log4j2.rootLogger.type = asyncRoot #log4j2.rootLogger.includeLocation = false log4j2.rootLogger.appenderRef.RollingFile.ref = RollingFile log4j2.rootLogger.appenderRef.PaxOsgi.ref = PaxOsgi log4j2.rootLogger.appenderRef.Console.ref = Console log4j2.rootLogger.appenderRef.Console.filter.threshold.type = ThresholdFilter log4j2.rootLogger.appenderRef.Console.filter.threshold.level = ${karaf.log.console:-OFF} log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.type = ContextMapFilter log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.pair1.type = KeyValuePair log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.pair1.key = slf4j.marker log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.pair1.value = CONFIDENTIAL log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.operator = or log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.onMatch = DENY log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.onMismatch = NEUTRAL log4j2.logger.spifly.name = org.apache.aries.spifly log4j2.logger.spifly.level = WARN log4j2.logger.audit.name = org.apache.karaf.jaas.modules.audit log4j2.logger.audit.level = INFO log4j2.logger.audit.additivity = false log4j2.logger.audit.appenderRef.AuditRollingFile.ref = AuditRollingFile # Console appender not used by default (see log4j2.rootLogger.appenderRefs) log4j2.appender.console.type = Console log4j2.appender.console.name = Console log4j2.appender.console.layout.type = PatternLayout log4j2.appender.console.layout.pattern = ${log4j2.pattern} log4j2.appender.rolling.type = RollingRandomAccessFile log4j2.appender.rolling.name = RollingFile log4j2.appender.rolling.fileName = ${karaf.data}/log/karaf.log log4j2.appender.rolling.filePattern = ${karaf.data}/log/karaf.log.%i #log4j2.appender.rolling.immediateFlush = false log4j2.appender.rolling.append = true log4j2.appender.rolling.layout.type = PatternLayout log4j2.appender.rolling.layout.pattern = ${log4j2.pattern} log4j2.appender.rolling.policies.type = Policies log4j2.appender.rolling.policies.size.type = SizeBasedTriggeringPolicy log4j2.appender.rolling.policies.size.size = 64MB log4j2.appender.rolling.strategy.type = DefaultRolloverStrategy log4j2.appender.rolling.strategy.max = 7 log4j2.appender.audit.type = RollingRandomAccessFile log4j2.appender.audit.name = AuditRollingFile log4j2.appender.audit.fileName = ${karaf.data}/security/audit.log log4j2.appender.audit.filePattern = ${karaf.data}/security/audit.log.%i log4j2.appender.audit.append = true log4j2.appender.audit.layout.type = PatternLayout log4j2.appender.audit.layout.pattern = ${log4j2.pattern} log4j2.appender.audit.policies.type = Policies log4j2.appender.audit.policies.size.type = SizeBasedTriggeringPolicy log4j2.appender.audit.policies.size.size = 8MB log4j2.appender.audit.strategy.type = DefaultRolloverStrategy log4j2.appender.audit.strategy.max = 7 log4j2.appender.osgi.type = PaxOsgi log4j2.appender.osgi.name = PaxOsgi log4j2.appender.osgi.filter = * #log4j2.logger.aether.name = shaded.org.eclipse.aether #log4j2.logger.aether.level = TRACE #log4j2.logger.http-headers.name = shaded.org.apache.http.headers #log4j2.logger.http-headers.level = DEBUG #log4j2.logger.maven.name = org.ops4j.pax.url.mvn #log4j2.logger.maven.level = TRACE Configuring the karaf log... karaf_version: karaf4, logapi: log4j2 + logapi=log4j2 + echo 'Configuring the karaf log... karaf_version: karaf4, logapi: log4j2' + '[' log4j2 == log4j2 ']' + sed -ie 's/log4j2.appender.rolling.policies.size.size = 64MB/log4j2.appender.rolling.policies.size.size = 1GB/g' /tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.logging.cfg + orgmodule=org.opendaylight.yangtools.yang.parser.repo.YangTextSchemaContextResolver + orgmodule_=org_opendaylight_yangtools_yang_parser_repo_YangTextSchemaContextResolver + echo 'log4j2.logger.org_opendaylight_yangtools_yang_parser_repo_YangTextSchemaContextResolver.name = WARN' controllerdebugmap: + echo 'log4j2.logger.org_opendaylight_yangtools_yang_parser_repo_YangTextSchemaContextResolver.level = WARN' + unset IFS + echo 'controllerdebugmap: ' cat /tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.logging.cfg + '[' -n '' ']' + echo 'cat /tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.logging.cfg' + cat /tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.logging.cfg ################################################################################ # # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ################################################################################ # Common pattern layout for appenders log4j2.pattern = %d{ISO8601} | %-5p | %-16t | %-32c{1} | %X{bundle.id} - %X{bundle.name} - %X{bundle.version} | %m%n # Root logger log4j2.rootLogger.level = INFO # uncomment to use asynchronous loggers, which require mvn:com.lmax/disruptor/3.3.2 library #log4j2.rootLogger.type = asyncRoot #log4j2.rootLogger.includeLocation = false log4j2.rootLogger.appenderRef.RollingFile.ref = RollingFile log4j2.rootLogger.appenderRef.PaxOsgi.ref = PaxOsgi log4j2.rootLogger.appenderRef.Console.ref = Console log4j2.rootLogger.appenderRef.Console.filter.threshold.type = ThresholdFilter log4j2.rootLogger.appenderRef.Console.filter.threshold.level = ${karaf.log.console:-OFF} # Filters for logs marked by org.opendaylight.odlparent.Markers log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.type = ContextMapFilter log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.pair1.type = KeyValuePair log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.pair1.key = slf4j.marker log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.pair1.value = CONFIDENTIAL log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.operator = or log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.onMatch = DENY log4j2.rootLogger.appenderRef.RollingFile.filter.confidential.onMismatch = NEUTRAL # Loggers configuration # Spifly logger log4j2.logger.spifly.name = org.apache.aries.spifly log4j2.logger.spifly.level = WARN # Security audit logger log4j2.logger.audit.name = org.apache.karaf.jaas.modules.audit log4j2.logger.audit.level = INFO log4j2.logger.audit.additivity = false log4j2.logger.audit.appenderRef.AuditRollingFile.ref = AuditRollingFile # Appenders configuration # Console appender not used by default (see log4j2.rootLogger.appenderRefs) log4j2.appender.console.type = Console log4j2.appender.console.name = Console log4j2.appender.console.layout.type = PatternLayout log4j2.appender.console.layout.pattern = ${log4j2.pattern} # Rolling file appender log4j2.appender.rolling.type = RollingRandomAccessFile log4j2.appender.rolling.name = RollingFile log4j2.appender.rolling.fileName = ${karaf.data}/log/karaf.log log4j2.appender.rolling.filePattern = ${karaf.data}/log/karaf.log.%i # uncomment to not force a disk flush #log4j2.appender.rolling.immediateFlush = false log4j2.appender.rolling.append = true log4j2.appender.rolling.layout.type = PatternLayout log4j2.appender.rolling.layout.pattern = ${log4j2.pattern} log4j2.appender.rolling.policies.type = Policies log4j2.appender.rolling.policies.size.type = SizeBasedTriggeringPolicy log4j2.appender.rolling.policies.size.size = 1GB log4j2.appender.rolling.strategy.type = DefaultRolloverStrategy log4j2.appender.rolling.strategy.max = 7 # Audit file appender log4j2.appender.audit.type = RollingRandomAccessFile log4j2.appender.audit.name = AuditRollingFile log4j2.appender.audit.fileName = ${karaf.data}/security/audit.log log4j2.appender.audit.filePattern = ${karaf.data}/security/audit.log.%i log4j2.appender.audit.append = true log4j2.appender.audit.layout.type = PatternLayout log4j2.appender.audit.layout.pattern = ${log4j2.pattern} log4j2.appender.audit.policies.type = Policies log4j2.appender.audit.policies.size.type = SizeBasedTriggeringPolicy log4j2.appender.audit.policies.size.size = 8MB log4j2.appender.audit.strategy.type = DefaultRolloverStrategy log4j2.appender.audit.strategy.max = 7 # OSGi appender log4j2.appender.osgi.type = PaxOsgi log4j2.appender.osgi.name = PaxOsgi log4j2.appender.osgi.filter = * # help with identification of maven-related problems with pax-url-aether #log4j2.logger.aether.name = shaded.org.eclipse.aether #log4j2.logger.aether.level = TRACE #log4j2.logger.http-headers.name = shaded.org.apache.http.headers #log4j2.logger.http-headers.level = DEBUG #log4j2.logger.maven.name = org.ops4j.pax.url.mvn #log4j2.logger.maven.level = TRACE log4j2.logger.org_opendaylight_yangtools_yang_parser_repo_YangTextSchemaContextResolver.name = WARN log4j2.logger.org_opendaylight_yangtools_yang_parser_repo_YangTextSchemaContextResolver.level = WARN Configure java home: /usr/lib/jvm/java-21-openjdk-amd64 max memory: 3072m + set_java_vars /usr/lib/jvm/java-21-openjdk-amd64 3072m /tmp/karaf-0.24.0-SNAPSHOT/bin/setenv + local -r java_home=/usr/lib/jvm/java-21-openjdk-amd64 + local -r controllermem=3072m + local -r memconf=/tmp/karaf-0.24.0-SNAPSHOT/bin/setenv + echo Configure + echo ' java home: /usr/lib/jvm/java-21-openjdk-amd64' + echo ' max memory: 3072m' memconf: /tmp/karaf-0.24.0-SNAPSHOT/bin/setenv + echo ' memconf: /tmp/karaf-0.24.0-SNAPSHOT/bin/setenv' + sed -ie 's%^# export JAVA_HOME%export JAVA_HOME=${JAVA_HOME:-/usr/lib/jvm/java-21-openjdk-amd64}%g' /tmp/karaf-0.24.0-SNAPSHOT/bin/setenv + sed -ie 's/JAVA_MAX_MEM="2048m"/JAVA_MAX_MEM=3072m/g' /tmp/karaf-0.24.0-SNAPSHOT/bin/setenv cat /tmp/karaf-0.24.0-SNAPSHOT/bin/setenv + echo 'cat /tmp/karaf-0.24.0-SNAPSHOT/bin/setenv' + cat /tmp/karaf-0.24.0-SNAPSHOT/bin/setenv #!/bin/sh # # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # # handle specific scripts; the SCRIPT_NAME is exactly the name of the Karaf # script: client, instance, shell, start, status, stop, karaf # # if [ "${KARAF_SCRIPT}" == "SCRIPT_NAME" ]; then # Actions go here... # fi # # general settings which should be applied for all scripts go here; please keep # in mind that it is possible that scripts might be executed more than once, e.g. # in example of the start script where the start script is executed first and the # karaf script afterwards. # # # The following section shows the possible configuration options for the default # karaf scripts # export JAVA_HOME=${JAVA_HOME:-/usr/lib/jvm/java-21-openjdk-amd64} # Location of Java installation # export JAVA_OPTS # Generic JVM options, for instance, where you can pass the memory configuration # export JAVA_NON_DEBUG_OPTS # Additional non-debug JVM options # export EXTRA_JAVA_OPTS # Additional JVM options # export KARAF_HOME # Karaf home folder # export KARAF_DATA # Karaf data folder # export KARAF_BASE # Karaf base folder # export KARAF_ETC # Karaf etc folder # export KARAF_LOG # Karaf log folder # export KARAF_SYSTEM_OPTS # First citizen Karaf options # export KARAF_OPTS # Additional available Karaf options # export KARAF_DEBUG # Enable debug mode # export KARAF_REDIRECT # Enable/set the std/err redirection when using bin/start # export KARAF_NOROOT # Prevent execution as root if set to true Set Java version + echo 'Set Java version' + sudo /usr/sbin/alternatives --install /usr/bin/java java /usr/lib/jvm/java-21-openjdk-amd64/bin/java 1 sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper sudo: a password is required + sudo /usr/sbin/alternatives --set java /usr/lib/jvm/java-21-openjdk-amd64/bin/java sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper sudo: a password is required JDK default version ... + echo 'JDK default version ...' + java -version openjdk version "21.0.9" 2025-10-21 OpenJDK Runtime Environment (build 21.0.9+10-Ubuntu-122.04) OpenJDK 64-Bit Server VM (build 21.0.9+10-Ubuntu-122.04, mixed mode, sharing) Set JAVA_HOME + echo 'Set JAVA_HOME' + export JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64 ++ readlink -e /usr/lib/jvm/java-21-openjdk-amd64/bin/java Java binary pointed at by JAVA_HOME: /usr/lib/jvm/java-21-openjdk-amd64/bin/java + JAVA_RESOLVED=/usr/lib/jvm/java-21-openjdk-amd64/bin/java + echo 'Java binary pointed at by JAVA_HOME: /usr/lib/jvm/java-21-openjdk-amd64/bin/java' Listing all open ports on controller system... + echo 'Listing all open ports on controller system...' + netstat -pnatu /tmp/configuration-script.sh: line 40: netstat: command not found Custom shard config exists!!! Copying the shard config... + '[' -f /tmp/custom_shard_config.txt ']' + echo 'Custom shard config exists!!!' + echo 'Copying the shard config...' + cp /tmp/custom_shard_config.txt /tmp/karaf-0.24.0-SNAPSHOT/bin/ Configuring cluster + echo 'Configuring cluster' + /tmp/karaf-0.24.0-SNAPSHOT/bin/configure_cluster.sh 3 10.30.170.104 10.30.171.129 10.30.170.156 ################################################ ## Configure Cluster ## ################################################ NOTE: Cluster configuration files not found. Copying from /tmp/karaf-0.24.0-SNAPSHOT/system/org/opendaylight/controller/sal-clustering-config/12.0.3 Configuring unique name in pekko.conf Configuring hostname in pekko.conf Configuring data and rpc seed nodes in pekko.conf modules = [ { name = "inventory" namespace = "urn:opendaylight:inventory" shard-strategy = "module" }, { name = "topology" namespace = "urn:TBD:params:xml:ns:yang:network-topology" shard-strategy = "module" }, { name = "toaster" namespace = "http://netconfcentral.org/ns/toaster" shard-strategy = "module" }, { name = "car" namespace = "urn:opendaylight:params:xml:ns:yang:controller:config:sal-clustering-it:car" shard-strategy = "module" }, { name = "people" namespace = "urn:opendaylight:params:xml:ns:yang:controller:config:sal-clustering-it:people" shard-strategy = "module" }, { name = "car-people" namespace = "urn:opendaylight:params:xml:ns:yang:controller:config:sal-clustering-it:car-people" shard-strategy = "module" } ] Configuring replication type in module-shards.conf ################################################ ## NOTE: Manually restart controller to ## ## apply configuration. ## ################################################ Dump pekko.conf odl-cluster-data { pekko { remote { artery { enabled = on transport = tcp canonical.hostname = "10.30.170.156" canonical.port = 2550 } } cluster { # Using artery. seed-nodes = ["pekko://opendaylight-cluster-data@10.30.170.104:2550", "pekko://opendaylight-cluster-data@10.30.171.129:2550", "pekko://opendaylight-cluster-data@10.30.170.156:2550"] roles = ["member-3"] # when under load we might trip a false positive on the failure detector # failure-detector { # heartbeat-interval = 4 s # acceptable-heartbeat-pause = 16s # } } persistence { # By default the snapshots/journal directories live in KARAF_HOME. You can choose to put it somewhere else by # modifying the following two properties. The directory location specified may be a relative or absolute path. # The relative path is always relative to KARAF_HOME. # snapshot-store.local.dir = "target/snapshots" } disable-default-actor-system-quarantined-event-handling = "false" } } Dump modules.conf modules = [ { name = "inventory" namespace = "urn:opendaylight:inventory" shard-strategy = "module" }, { name = "topology" namespace = "urn:TBD:params:xml:ns:yang:network-topology" shard-strategy = "module" }, { name = "toaster" namespace = "http://netconfcentral.org/ns/toaster" shard-strategy = "module" }, { name = "car" namespace = "urn:opendaylight:params:xml:ns:yang:controller:config:sal-clustering-it:car" shard-strategy = "module" }, { name = "people" namespace = "urn:opendaylight:params:xml:ns:yang:controller:config:sal-clustering-it:people" shard-strategy = "module" }, { name = "car-people" namespace = "urn:opendaylight:params:xml:ns:yang:controller:config:sal-clustering-it:car-people" shard-strategy = "module" } ] Dump module-shards.conf + echo 'Dump pekko.conf' + cat /tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/pekko.conf + echo 'Dump modules.conf' + cat /tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/modules.conf + echo 'Dump module-shards.conf' module-shards = [ { name = "default" shards = [ { name = "default" replicas = ["member-1", "member-2", "member-3"] } ] }, { name = "inventory" shards = [ { name="inventory" replicas = ["member-1", "member-2", "member-3"] } ] }, { name = "topology" shards = [ { name="topology" replicas = ["member-1", "member-2", "member-3"] } ] }, { name = "toaster" shards = [ { name="toaster" replicas = ["member-1", "member-2", "member-3"] } ] }, { name = "car" shards = [ { name="car" replicas = ["member-1", "member-2", "member-3"] } ] }, { name = "people" shards = [ { name="people" replicas = ["member-1", "member-2", "member-3"] } ] }, { name = "car-people" shards = [ { name="car-people" replicas = ["member-1", "member-2", "member-3"] } ] } ] + cat /tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/module-shards.conf Locating config plan to use... Finished running config plans Starting member-1 with IP address 10.30.170.104 Warning: Permanently added '10.30.170.104' (ECDSA) to the list of known hosts. Warning: Permanently added '10.30.170.104' (ECDSA) to the list of known hosts. Redirecting karaf console output to karaf_console.log Starting controller... start: Redirecting Karaf output to /tmp/karaf-0.24.0-SNAPSHOT/data/log/karaf_console.log Starting member-2 with IP address 10.30.171.129 Warning: Permanently added '10.30.171.129' (ECDSA) to the list of known hosts. Warning: Permanently added '10.30.171.129' (ECDSA) to the list of known hosts. Redirecting karaf console output to karaf_console.log Starting controller... start: Redirecting Karaf output to /tmp/karaf-0.24.0-SNAPSHOT/data/log/karaf_console.log Starting member-3 with IP address 10.30.170.156 Warning: Permanently added '10.30.170.156' (ECDSA) to the list of known hosts. Warning: Permanently added '10.30.170.156' (ECDSA) to the list of known hosts. Redirecting karaf console output to karaf_console.log Starting controller... start: Redirecting Karaf output to /tmp/karaf-0.24.0-SNAPSHOT/data/log/karaf_console.log [controller-csit-3node-clustering-ask-all-vanadium] $ /bin/bash /tmp/jenkins5766251451336966384.sh common-functions.sh is being sourced common-functions environment: MAVENCONF: /tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.url.mvn.cfg ACTUALFEATURES: FEATURESCONF: /tmp/karaf-0.24.0-SNAPSHOT/etc/org.apache.karaf.features.cfg CUSTOMPROP: /tmp/karaf-0.24.0-SNAPSHOT/etc/custom.properties LOGCONF: /tmp/karaf-0.24.0-SNAPSHOT/etc/org.ops4j.pax.logging.cfg MEMCONF: /tmp/karaf-0.24.0-SNAPSHOT/bin/setenv CONTROLLERMEM: 2048m AKKACONF: /tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/pekko.conf MODULESCONF: /tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/modules.conf MODULESHARDSCONF: /tmp/karaf-0.24.0-SNAPSHOT/configuration/initial/module-shards.conf SUITES: + echo '#################################################' ################################################# + echo '## Verify Cluster is UP ##' ## Verify Cluster is UP ## + echo '#################################################' ################################################# + create_post_startup_script + cat + copy_and_run_post_startup_script + seed_index=1 ++ seq 1 3 + for i in $(seq 1 "${NUM_ODL_SYSTEM}") + CONTROLLERIP=ODL_SYSTEM_1_IP + echo 'Execute the post startup script on controller 10.30.170.104' Execute the post startup script on controller 10.30.170.104 + scp /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/post-startup-script.sh 10.30.170.104:/tmp/ Warning: Permanently added '10.30.170.104' (ECDSA) to the list of known hosts. + ssh 10.30.170.104 'bash /tmp/post-startup-script.sh 1' Warning: Permanently added '10.30.170.104' (ECDSA) to the list of known hosts. /tmp/post-startup-script.sh: line 4: netstat: command not found /tmp/post-startup-script.sh: line 4: netstat: command not found /tmp/post-startup-script.sh: line 4: netstat: command not found /tmp/post-startup-script.sh: line 4: netstat: command not found /tmp/post-startup-script.sh: line 4: netstat: command not found /tmp/post-startup-script.sh: line 4: netstat: command not found /tmp/post-startup-script.sh: line 4: netstat: command not found /tmp/post-startup-script.sh: line 4: netstat: command not found /tmp/post-startup-script.sh: line 4: netstat: command not found /tmp/post-startup-script.sh: line 4: netstat: command not found /tmp/post-startup-script.sh: line 4: netstat: command not found /tmp/post-startup-script.sh: line 4: netstat: command not found Waiting up to 3 minutes for controller to come up, checking every 5 seconds... 2026-01-17T23:44:42,580 | INFO | SystemReadyService-0 | SimpleSystemReadyMonitor | 323 - org.opendaylight.infrautils.ready-api - 7.1.9 | System ready; AKA: Aye captain, all warp coils are now operating at peak efficiency! [M.] Controller is UP 2026-01-17T23:44:42,580 | INFO | SystemReadyService-0 | SimpleSystemReadyMonitor | 323 - org.opendaylight.infrautils.ready-api - 7.1.9 | System ready; AKA: Aye captain, all warp coils are now operating at peak efficiency! [M.] Listing all open ports on controller system... /tmp/post-startup-script.sh: line 51: netstat: command not found looking for "BindException: Address already in use" in log file looking for "server is unhealthy" in log file + '[' 1 == 0 ']' + for i in $(seq 1 "${NUM_ODL_SYSTEM}") + CONTROLLERIP=ODL_SYSTEM_2_IP + echo 'Execute the post startup script on controller 10.30.171.129' Execute the post startup script on controller 10.30.171.129 + scp /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/post-startup-script.sh 10.30.171.129:/tmp/ Warning: Permanently added '10.30.171.129' (ECDSA) to the list of known hosts. + ssh 10.30.171.129 'bash /tmp/post-startup-script.sh 2' Warning: Permanently added '10.30.171.129' (ECDSA) to the list of known hosts. /tmp/post-startup-script.sh: line 4: netstat: command not found /tmp/post-startup-script.sh: line 4: netstat: command not found /tmp/post-startup-script.sh: line 4: netstat: command not found /tmp/post-startup-script.sh: line 4: netstat: command not found /tmp/post-startup-script.sh: line 4: netstat: command not found /tmp/post-startup-script.sh: line 4: netstat: command not found /tmp/post-startup-script.sh: line 4: netstat: command not found /tmp/post-startup-script.sh: line 4: netstat: command not found /tmp/post-startup-script.sh: line 4: netstat: command not found /tmp/post-startup-script.sh: line 4: netstat: command not found /tmp/post-startup-script.sh: line 4: netstat: command not found /tmp/post-startup-script.sh: line 4: netstat: command not found Waiting up to 3 minutes for controller to come up, checking every 5 seconds... 2026-01-17T23:44:42,680 | INFO | SystemReadyService-0 | SimpleSystemReadyMonitor | 323 - org.opendaylight.infrautils.ready-api - 7.1.9 | System ready; AKA: Aye captain, all warp coils are now operating at peak efficiency! [M.] Controller is UP 2026-01-17T23:44:42,680 | INFO | SystemReadyService-0 | SimpleSystemReadyMonitor | 323 - org.opendaylight.infrautils.ready-api - 7.1.9 | System ready; AKA: Aye captain, all warp coils are now operating at peak efficiency! [M.] Listing all open ports on controller system... /tmp/post-startup-script.sh: line 51: netstat: command not found looking for "BindException: Address already in use" in log file looking for "server is unhealthy" in log file + '[' 2 == 0 ']' + for i in $(seq 1 "${NUM_ODL_SYSTEM}") + CONTROLLERIP=ODL_SYSTEM_3_IP + echo 'Execute the post startup script on controller 10.30.170.156' Execute the post startup script on controller 10.30.170.156 + scp /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/post-startup-script.sh 10.30.170.156:/tmp/ Warning: Permanently added '10.30.170.156' (ECDSA) to the list of known hosts. + ssh 10.30.170.156 'bash /tmp/post-startup-script.sh 3' Warning: Permanently added '10.30.170.156' (ECDSA) to the list of known hosts. /tmp/post-startup-script.sh: line 4: netstat: command not found /tmp/post-startup-script.sh: line 4: netstat: command not found /tmp/post-startup-script.sh: line 4: netstat: command not found /tmp/post-startup-script.sh: line 4: netstat: command not found /tmp/post-startup-script.sh: line 4: netstat: command not found /tmp/post-startup-script.sh: line 4: netstat: command not found /tmp/post-startup-script.sh: line 4: netstat: command not found /tmp/post-startup-script.sh: line 4: netstat: command not found /tmp/post-startup-script.sh: line 4: netstat: command not found /tmp/post-startup-script.sh: line 4: netstat: command not found /tmp/post-startup-script.sh: line 4: netstat: command not found /tmp/post-startup-script.sh: line 4: netstat: command not found Waiting up to 3 minutes for controller to come up, checking every 5 seconds... 2026-01-17T23:44:42,630 | INFO | SystemReadyService-0 | SimpleSystemReadyMonitor | 323 - org.opendaylight.infrautils.ready-api - 7.1.9 | System ready; AKA: Aye captain, all warp coils are now operating at peak efficiency! [M.] Controller is UP 2026-01-17T23:44:42,630 | INFO | SystemReadyService-0 | SimpleSystemReadyMonitor | 323 - org.opendaylight.infrautils.ready-api - 7.1.9 | System ready; AKA: Aye captain, all warp coils are now operating at peak efficiency! [M.] Listing all open ports on controller system... /tmp/post-startup-script.sh: line 51: netstat: command not found looking for "BindException: Address already in use" in log file looking for "server is unhealthy" in log file + '[' 0 == 0 ']' + seed_index=1 + dump_controller_threads ++ seq 1 3 + for i in $(seq 1 "${NUM_ODL_SYSTEM}") + CONTROLLERIP=ODL_SYSTEM_1_IP + echo 'Let'\''s take the karaf thread dump' Let's take the karaf thread dump + ssh 10.30.170.104 'sudo ps aux' Warning: Permanently added '10.30.170.104' (ECDSA) to the list of known hosts. ++ grep -v grep ++ grep org.apache.karaf.main.Main /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/ps_before.log ++ tr -s ' ' ++ cut -f2 '-d ' + pid=2086 + echo 'karaf main: org.apache.karaf.main.Main, pid:2086' karaf main: org.apache.karaf.main.Main, pid:2086 + ssh 10.30.170.104 '/usr/lib/jvm/java-21-openjdk-amd64/bin/jstack -l 2086' Warning: Permanently added '10.30.170.104' (ECDSA) to the list of known hosts. + for i in $(seq 1 "${NUM_ODL_SYSTEM}") + CONTROLLERIP=ODL_SYSTEM_2_IP + echo 'Let'\''s take the karaf thread dump' Let's take the karaf thread dump + ssh 10.30.171.129 'sudo ps aux' Warning: Permanently added '10.30.171.129' (ECDSA) to the list of known hosts. ++ grep org.apache.karaf.main.Main /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/ps_before.log ++ grep -v grep ++ tr -s ' ' ++ cut -f2 '-d ' + pid=2097 + echo 'karaf main: org.apache.karaf.main.Main, pid:2097' karaf main: org.apache.karaf.main.Main, pid:2097 + ssh 10.30.171.129 '/usr/lib/jvm/java-21-openjdk-amd64/bin/jstack -l 2097' Warning: Permanently added '10.30.171.129' (ECDSA) to the list of known hosts. + for i in $(seq 1 "${NUM_ODL_SYSTEM}") + CONTROLLERIP=ODL_SYSTEM_3_IP + echo 'Let'\''s take the karaf thread dump' Let's take the karaf thread dump + ssh 10.30.170.156 'sudo ps aux' Warning: Permanently added '10.30.170.156' (ECDSA) to the list of known hosts. ++ grep org.apache.karaf.main.Main /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/ps_before.log ++ grep -v grep ++ tr -s ' ' ++ cut -f2 '-d ' + pid=2100 + echo 'karaf main: org.apache.karaf.main.Main, pid:2100' karaf main: org.apache.karaf.main.Main, pid:2100 + ssh 10.30.170.156 '/usr/lib/jvm/java-21-openjdk-amd64/bin/jstack -l 2100' Warning: Permanently added '10.30.170.156' (ECDSA) to the list of known hosts. + '[' 0 -gt 0 ']' + echo 'Generating controller variables...' Generating controller variables... ++ seq 1 3 + for i in $(seq 1 "${NUM_ODL_SYSTEM}") + CONTROLLERIP=ODL_SYSTEM_1_IP + odl_variables=' -v ODL_SYSTEM_1_IP:10.30.170.104' + for i in $(seq 1 "${NUM_ODL_SYSTEM}") + CONTROLLERIP=ODL_SYSTEM_2_IP + odl_variables=' -v ODL_SYSTEM_1_IP:10.30.170.104 -v ODL_SYSTEM_2_IP:10.30.171.129' + for i in $(seq 1 "${NUM_ODL_SYSTEM}") + CONTROLLERIP=ODL_SYSTEM_3_IP + odl_variables=' -v ODL_SYSTEM_1_IP:10.30.170.104 -v ODL_SYSTEM_2_IP:10.30.171.129 -v ODL_SYSTEM_3_IP:10.30.170.156' + echo 'Generating mininet variables...' Generating mininet variables... ++ seq 1 0 + get_test_suites SUITES + local __suite_list=SUITES + echo 'Locating test plan to use...' Locating test plan to use... + testplan_filepath=/w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/testplans/controller-clustering-ask-vanadium.txt + '[' '!' -f /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/testplans/controller-clustering-ask-vanadium.txt ']' + testplan_filepath=/w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/testplans/controller-clustering-ask.txt + '[' disabled '!=' disabled ']' + echo 'Changing the testplan path...' Changing the testplan path... + sed s:integration:/w/workspace/controller-csit-3node-clustering-ask-all-vanadium: /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/testplans/controller-clustering-ask.txt + cat testplan.txt # Place the suites in run order: /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_rpc_broker/rpc_provider_precedence.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_rpc_broker/rpc_provider_partition_and_heal.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_rpc_broker/action_provider_precedence.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_rpc_broker/action_provider_partition_and_heal.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/cluster_singleton/master_stability.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/cluster_singleton/partition_and_heal.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/cluster_singleton/chasing_the_leader.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/singleton_service/global_rpc_kill.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/singleton_service/global_rpc_freeze.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/singleton_service/global_rpc_isolate.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/carpeople_crud.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/car_failover_crud.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/car_persistence_recovery.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/buycar_failover.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/entity_isolate.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/buycar_failover_isolation.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/car_failover_crud_isolation.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/car_outage_corners.robot + '[' -z '' ']' ++ grep -E -v '(^[[:space:]]*#|^[[:space:]]*$)' testplan.txt ++ tr '\012' ' ' + suite_list='/w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_rpc_broker/rpc_provider_precedence.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_rpc_broker/rpc_provider_partition_and_heal.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_rpc_broker/action_provider_precedence.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_rpc_broker/action_provider_partition_and_heal.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/cluster_singleton/master_stability.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/cluster_singleton/partition_and_heal.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/cluster_singleton/chasing_the_leader.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/singleton_service/global_rpc_kill.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/singleton_service/global_rpc_freeze.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/singleton_service/global_rpc_isolate.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/carpeople_crud.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/car_failover_crud.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/car_persistence_recovery.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/buycar_failover.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/entity_isolate.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/buycar_failover_isolation.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/car_failover_crud_isolation.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/car_outage_corners.robot ' + eval 'SUITES='\''/w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_rpc_broker/rpc_provider_precedence.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_rpc_broker/rpc_provider_partition_and_heal.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_rpc_broker/action_provider_precedence.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_rpc_broker/action_provider_partition_and_heal.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/cluster_singleton/master_stability.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/cluster_singleton/partition_and_heal.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/cluster_singleton/chasing_the_leader.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/singleton_service/global_rpc_kill.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/singleton_service/global_rpc_freeze.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/singleton_service/global_rpc_isolate.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/carpeople_crud.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/car_failover_crud.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/car_persistence_recovery.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/buycar_failover.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/entity_isolate.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/buycar_failover_isolation.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/car_failover_crud_isolation.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/car_outage_corners.robot '\''' ++ SUITES='/w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_rpc_broker/rpc_provider_precedence.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_rpc_broker/rpc_provider_partition_and_heal.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_rpc_broker/action_provider_precedence.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_rpc_broker/action_provider_partition_and_heal.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/cluster_singleton/master_stability.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/cluster_singleton/partition_and_heal.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/cluster_singleton/chasing_the_leader.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/singleton_service/global_rpc_kill.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/singleton_service/global_rpc_freeze.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/singleton_service/global_rpc_isolate.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/carpeople_crud.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/car_failover_crud.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/car_persistence_recovery.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/buycar_failover.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/entity_isolate.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/buycar_failover_isolation.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/car_failover_crud_isolation.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/car_outage_corners.robot ' + echo 'Starting Robot test suites /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_rpc_broker/rpc_provider_precedence.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_rpc_broker/rpc_provider_partition_and_heal.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_rpc_broker/action_provider_precedence.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_rpc_broker/action_provider_partition_and_heal.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/cluster_singleton/master_stability.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/cluster_singleton/partition_and_heal.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/cluster_singleton/chasing_the_leader.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/singleton_service/global_rpc_kill.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/singleton_service/global_rpc_freeze.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/singleton_service/global_rpc_isolate.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/carpeople_crud.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/car_failover_crud.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/car_persistence_recovery.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/buycar_failover.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/entity_isolate.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/buycar_failover_isolation.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/car_failover_crud_isolation.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/car_outage_corners.robot ...' Starting Robot test suites /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_rpc_broker/rpc_provider_precedence.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_rpc_broker/rpc_provider_partition_and_heal.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_rpc_broker/action_provider_precedence.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_rpc_broker/action_provider_partition_and_heal.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/cluster_singleton/master_stability.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/cluster_singleton/partition_and_heal.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/cluster_singleton/chasing_the_leader.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/singleton_service/global_rpc_kill.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/singleton_service/global_rpc_freeze.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/singleton_service/global_rpc_isolate.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/carpeople_crud.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/car_failover_crud.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/car_persistence_recovery.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/buycar_failover.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/entity_isolate.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/buycar_failover_isolation.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/car_failover_crud_isolation.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/car_outage_corners.robot ... + robot -N controller-clustering-ask.txt --removekeywords wuks -e exclude -e skip_if_vanadium -v BUNDLEFOLDER:karaf-0.24.0-SNAPSHOT -v BUNDLE_URL:https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/integration/karaf/0.24.0-SNAPSHOT/karaf-0.24.0-20260117.223656-12.zip -v CONTROLLER:10.30.170.104 -v CONTROLLER1:10.30.171.129 -v CONTROLLER2:10.30.170.156 -v CONTROLLER_USER:jenkins -v JAVA_HOME:/usr/lib/jvm/java-21-openjdk-amd64 -v JDKVERSION:openjdk21 -v JENKINS_WORKSPACE:/w/workspace/controller-csit-3node-clustering-ask-all-vanadium -v MININET: -v MININET1: -v MININET2: -v MININET_USER:jenkins -v NEXUSURL_PREFIX:https://nexus.opendaylight.org -v NUM_ODL_SYSTEM:3 -v NUM_TOOLS_SYSTEM:0 -v ODL_STREAM:vanadium -v ODL_SYSTEM_IP:10.30.170.104 -v ODL_SYSTEM_1_IP:10.30.170.104 -v ODL_SYSTEM_2_IP:10.30.171.129 -v ODL_SYSTEM_3_IP:10.30.170.156 -v ODL_SYSTEM_USER:jenkins -v TOOLS_SYSTEM_IP: -v TOOLS_SYSTEM_USER:jenkins -v USER_HOME:/home/jenkins -v IS_KARAF_APPL:True -v WORKSPACE:/tmp /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_rpc_broker/rpc_provider_precedence.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_rpc_broker/rpc_provider_partition_and_heal.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_rpc_broker/action_provider_precedence.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_rpc_broker/action_provider_partition_and_heal.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/cluster_singleton/master_stability.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/cluster_singleton/partition_and_heal.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/cluster_singleton/chasing_the_leader.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/singleton_service/global_rpc_kill.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/singleton_service/global_rpc_freeze.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/singleton_service/global_rpc_isolate.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/dom_data_broker/restart_odl_with_tell_based_false.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/carpeople_crud.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/car_failover_crud.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/car_persistence_recovery.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/buycar_failover.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/entity_isolate.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/buycar_failover_isolation.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/car_failover_crud_isolation.robot /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/test/csit/suites/controller/Clustering_Datastore/car_outage_corners.robot ============================================================================== controller-clustering-ask.txt ============================================================================== controller-clustering-ask.txt.Restart Odl With Tell Based False :: Unset te... ============================================================================== Stop_All_Members :: Stop every odl node. | PASS | ------------------------------------------------------------------------------ Unset_Tell_Based_Protocol_Usage :: Comment out the flag usage in c... | PASS | ------------------------------------------------------------------------------ Start_All_And_Sync :: Start each member and wait for sync. | FAIL | Got rc: 1 or stderr was not empty: sudo: netstat: command not found ------------------------------------------------------------------------------ controller-clustering-ask.txt.Restart Odl With Tell Based False ::... | FAIL | 3 tests, 2 passed, 1 failed ============================================================================== controller-clustering-ask.txt.Rpc Provider Precedence :: DOMRpcBroker testi... ============================================================================== Register_Rpc_On_Each_Node :: Register global rpc on each node of t... | PASS | ------------------------------------------------------------------------------ Invoke_Rpc_On_Each_Node :: Verify that the rpc response comes from... | PASS | ------------------------------------------------------------------------------ Unregister_Rpc_On_Node :: Unregister the rpc on one of the cluster... | PASS | ------------------------------------------------------------------------------ Invoke_Rpc_On_Node_With_Unregistered_Rpc :: Invoke rcp on the node... | FAIL | ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_rpc_provider_precedence_invoke_rpc_on_node_with_unregistered_rpc" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_rpc_provider_precedence_invoke_rpc_on_node_with_unregistered_rpc&order=bug_status" ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) ------------------------------------------------------------------------------ Invoke_Rpc_On_Remaining_Nodes :: Verify that the rpc response come... | PASS | ------------------------------------------------------------------------------ Reregister_Rpc_On_Node :: Reregister the rpc. | PASS | ------------------------------------------------------------------------------ Invoke_Rpc_On_Each_Node_Again :: Verify that the rpc response come... | PASS | ------------------------------------------------------------------------------ Unregister_Rpc_On_Each_Node :: Unregister rpc on every node. | PASS | ------------------------------------------------------------------------------ controller-clustering-ask.txt.Rpc Provider Precedence :: DOMRpcBro... | FAIL | 8 tests, 7 passed, 1 failed ============================================================================== controller-clustering-ask.txt.Restart Odl With Tell Based False :: Unset te... ============================================================================== Stop_All_Members :: Stop every odl node. | FAIL | Keyword 'Verify_Karaf_Is_Not_Running_On_Member' failed after retrying for 6 minutes. The last error was: Found running Karaf count: 1: 0 != 1 ------------------------------------------------------------------------------ Unset_Tell_Based_Protocol_Usage :: Comment out the flag usage in c... | PASS | ------------------------------------------------------------------------------ Start_All_And_Sync :: Start each member and wait for sync. | FAIL | Keyword 'Verify_Members_Are_Ready' failed after retrying for 6 minutes. The last error was: ConnectionError: HTTPConnectionPool(host='10.30.170.156', port=8181): Max retries exceeded with url: /jolokia/read/org.opendaylight.controller:Category=ShardManager,name=shard-manager-config,type=DistributedConfigDatastore (Caused by NewConnectionError("HTTPConnection(host='10.30.170.156', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ controller-clustering-ask.txt.Restart Odl With Tell Based False ::... | FAIL | 3 tests, 1 passed, 2 failed ============================================================================== controller-clustering-ask.txt.Rpc Provider Partition And Heal :: DOMRpcBrok... ============================================================================== Register_Rpc_On_Two_Nodes :: Register rpc on two nodes of the odl ... | PASS | ------------------------------------------------------------------------------ Invoke_Rpc_On_Each_Node :: Invoke get-constant rpc on every node o... | FAIL | ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_rpc_provider_partition_and_heal_invoke_rpc_on_each_node" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_rpc_provider_partition_and_heal_invoke_rpc_on_each_node&order=bug_status" Keyword 'Verify_Constant_On_Active_Nodes' failed after retrying for 10 seconds. The last error was: ConnectionError: HTTPConnectionPool(host='10.30.170.156', port=8181): Max retries exceeded with url: /rests/operations/odl-mdsal-lowlevel-target:get-constant (Caused by NewConnectionError("HTTPConnection(host='10.30.170.156', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Isolate_One_Node :: Isolate one node with registered rpc. From the... | FAIL | ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_rpc_provider_partition_and_heal_isolate_one_node" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_rpc_provider_partition_and_heal_isolate_one_node&order=bug_status" ConnectionError: HTTPConnectionPool(host='10.30.170.156', port=8181): Max retries exceeded with url: /rests/operations/odl-mdsal-lowlevel-target:get-constant (Caused by NewConnectionError("HTTPConnection(host='10.30.170.156', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Invoke_Rpc_On_Nonisolated_Nodes :: Invoke rpc on non-islolated nodes. | FAIL | ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_rpc_provider_partition_and_heal_invoke_rpc_on_nonisolated_nodes" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_rpc_provider_partition_and_heal_invoke_rpc_on_nonisolated_nodes&order=bug_status" Keyword 'DrbCommons.Verify_Constant_On_Active_Nodes' failed after retrying for 1 minute. The last error was: Keyword 'Verify_Constant_On_Active_Nodes' failed after retrying for 10 seconds. The last error was: ConnectionError: HTTPConnectionPool(host='10.30.170.156', port=8181): Max retries exceeded with url: /rests/operations/odl-mdsal-lowlevel-target:get-constant (Caused by NewConnectionError("HTTPConnection(host='10.30.170.156', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Rejoin_Isolated_Member :: Rejoin isolated node | FAIL | ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_rpc_provider_partition_and_heal_rejoin_isolated_member" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_rpc_provider_partition_and_heal_rejoin_isolated_member&order=bug_status" Variable '${isolated_idx}' not found. ------------------------------------------------------------------------------ Invoke_Rpc_On_Each_Node_Again :: Invoke rpc get-constant on every ... | FAIL | ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_rpc_provider_partition_and_heal_invoke_rpc_on_each_node_again" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_rpc_provider_partition_and_heal_invoke_rpc_on_each_node_again&order=bug_status" Keyword 'Verify_Constant_On_Active_Nodes' failed after retrying for 10 seconds. The last error was: ConnectionError: HTTPConnectionPool(host='10.30.170.156', port=8181): Max retries exceeded with url: /rests/operations/odl-mdsal-lowlevel-target:get-constant (Caused by NewConnectionError("HTTPConnection(host='10.30.170.156', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Unregister_Rpc_On_Each_Node :: Inregister rpc on both nodes. | PASS | ------------------------------------------------------------------------------ controller-clustering-ask.txt.Rpc Provider Partition And Heal :: D... | FAIL | 7 tests, 2 passed, 5 failed ============================================================================== controller-clustering-ask.txt.Restart Odl With Tell Based False :: Unset te... ============================================================================== Stop_All_Members :: Stop every odl node. | PASS | ------------------------------------------------------------------------------ Unset_Tell_Based_Protocol_Usage :: Comment out the flag usage in c... | PASS | ------------------------------------------------------------------------------ Start_All_And_Sync :: Start each member and wait for sync. | FAIL | Keyword 'Verify_Members_Are_Ready' failed after retrying for 6 minutes. The last error was: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.170.104:8181/jolokia/read/org.opendaylight.controller:Category=ShardManager,name=shard-manager-config,type=DistributedConfigDatastore ------------------------------------------------------------------------------ controller-clustering-ask.txt.Restart Odl With Tell Based False ::... | FAIL | 3 tests, 2 passed, 1 failed ============================================================================== controller-clustering-ask.txt.Action Provider Precedence :: DOMRpcBroker te... ============================================================================== Register_Rpc_On_Each_Node :: Register routed rpc on each node of t... | FAIL | ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_action_provider_precedence_register_rpc_on_each_node" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_action_provider_precedence_register_rpc_on_each_node&order=bug_status" HTTPError: 401 Client Error: Unauthorized for url: http://10.30.170.104:8181/rests/operations/odl-mdsal-lowlevel-control:register-bound-constant ------------------------------------------------------------------------------ Invoke_Rpc_On_Each_Node :: Verify that the rpc response comes from... | FAIL | ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_action_provider_precedence_invoke_rpc_on_each_node" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_action_provider_precedence_invoke_rpc_on_each_node&order=bug_status" HTTPError: 401 Client Error: Unauthorized for url: http://10.30.170.104:8181/rests/operations/odl-mdsal-lowlevel-target:get-contexted-constant ------------------------------------------------------------------------------ Unregister_Rpc_On_Node :: Unregister the rpc on one of the cluster... | FAIL | ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_action_provider_precedence_unregister_rpc_on_node" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_action_provider_precedence_unregister_rpc_on_node&order=bug_status" HTTPError: 401 Client Error: Unauthorized for url: http://10.30.170.104:8181/rests/operations/odl-mdsal-lowlevel-control:unregister-bound-constant ------------------------------------------------------------------------------ Invoke_Rpc_On_Node_With_Unregistered_Rpc :: Invoke rcp on the node... | FAIL | ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_action_provider_precedence_invoke_rpc_on_node_with_unregistered_rpc" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_action_provider_precedence_invoke_rpc_on_node_with_unregistered_rpc&order=bug_status" HTTPError: 401 Client Error: Unauthorized for url: http://10.30.170.104:8181/rests/operations/odl-mdsal-lowlevel-target:get-contexted-constant ------------------------------------------------------------------------------ Invoke_Rpc_On_Remaining_Nodes :: Verify that the rpc response come... | PASS | ------------------------------------------------------------------------------ Reregister_Rpc_On_Node :: Reregister the rpc. | FAIL | ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_action_provider_precedence_reregister_rpc_on_node" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_action_provider_precedence_reregister_rpc_on_node&order=bug_status" HTTPError: 401 Client Error: Unauthorized for url: http://10.30.170.104:8181/rests/operations/odl-mdsal-lowlevel-control:register-bound-constant ------------------------------------------------------------------------------ Invoke_Rpc_On_Each_Node_Again :: Verify that the rpc response come... | FAIL | ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_action_provider_precedence_invoke_rpc_on_each_node_again" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_action_provider_precedence_invoke_rpc_on_each_node_again&order=bug_status" HTTPError: 401 Client Error: Unauthorized for url: http://10.30.170.104:8181/rests/operations/odl-mdsal-lowlevel-target:get-contexted-constant ------------------------------------------------------------------------------ Unregister_Rpc_On_Each_Node :: Unregister rpc on every node. | FAIL | ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_action_provider_precedence_unregister_rpc_on_each_node" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_action_provider_precedence_unregister_rpc_on_each_node&order=bug_status" HTTPError: 401 Client Error: Unauthorized for url: http://10.30.170.104:8181/rests/operations/odl-mdsal-lowlevel-control:unregister-bound-constant ------------------------------------------------------------------------------ controller-clustering-ask.txt.Action Provider Precedence :: DOMRpc... | FAIL | 8 tests, 1 passed, 7 failed ============================================================================== controller-clustering-ask.txt.Restart Odl With Tell Based False :: Unset te... ============================================================================== Stop_All_Members :: Stop every odl node. | FAIL | Keyword 'Verify_Karaf_Is_Not_Running_On_Member' failed after retrying for 6 minutes. The last error was: Found running Karaf count: 1: 0 != 1 ------------------------------------------------------------------------------ Unset_Tell_Based_Protocol_Usage :: Comment out the flag usage in c... | PASS | ------------------------------------------------------------------------------ Start_All_And_Sync :: Start each member and wait for sync. | FAIL | Keyword 'Verify_Members_Are_Ready' failed after retrying for 6 minutes. The last error was: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.129:8181/jolokia/read/org.opendaylight.controller:Category=ShardManager,name=shard-manager-config,type=DistributedConfigDatastore ------------------------------------------------------------------------------ controller-clustering-ask.txt.Restart Odl With Tell Based False ::... | FAIL | 3 tests, 1 passed, 2 failed ============================================================================== controller-clustering-ask.txt.Action Provider Partition And Heal :: DOMRpcB... ============================================================================== Register_Rpc_On_Two_Nodes :: Register rpc on two nodes of the odl ... | FAIL | ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_action_provider_partition_and_heal_register_rpc_on_two_nodes" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_action_provider_partition_and_heal_register_rpc_on_two_nodes&order=bug_status" HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.129:8181/rests/operations/odl-mdsal-lowlevel-control:register-bound-constant ------------------------------------------------------------------------------ Invoke_Rpc_On_Each_Node :: Invoke get-contexted-constant rpc on ev... | FAIL | ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_action_provider_partition_and_heal_invoke_rpc_on_each_node" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_action_provider_partition_and_heal_invoke_rpc_on_each_node&order=bug_status" Keyword 'Verify_Contexted_Constant_On_Active_Nodes' failed after retrying for 10 seconds. The last error was: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.129:8181/rests/operations/odl-mdsal-lowlevel-target:get-contexted-constant ------------------------------------------------------------------------------ Isolate_One_Node :: Isolate one node with registered rpc. | FAIL | ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_action_provider_partition_and_heal_isolate_one_node" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_action_provider_partition_and_heal_isolate_one_node&order=bug_status" ReadTimeout: HTTPConnectionPool(host='10.30.170.156', port=8181): Read timed out. (read timeout=125.0) ------------------------------------------------------------------------------ Invoke_Rpc_On_Remaining_Nodes :: Invoke rpc on non-islolated nodes... | FAIL | ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_action_provider_partition_and_heal_invoke_rpc_on_remaining_nodes" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_action_provider_partition_and_heal_invoke_rpc_on_remaining_nodes&order=bug_status" Keyword 'DrbCommons.Verify_Contexted_Constant_On_Active_Nodes' failed after retrying for 1 minute. The last error was: Keyword 'Verify_Contexted_Constant_On_Active_Nodes' failed after retrying for 10 seconds. The last error was: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.129:8181/rests/operations/odl-mdsal-lowlevel-target:get-contexted-constant ------------------------------------------------------------------------------ Rejoin_Isolated_Member :: Rejoin isolated node | FAIL | ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_action_provider_partition_and_heal_rejoin_isolated_member" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_action_provider_partition_and_heal_rejoin_isolated_member&order=bug_status" Variable '${isolated_idx}' not found. ------------------------------------------------------------------------------ Invoke_Rpc_On_Each_Node_Again :: Invoke rpc get-contexted-constant... | FAIL | ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_action_provider_partition_and_heal_invoke_rpc_on_each_node_again" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_action_provider_partition_and_heal_invoke_rpc_on_each_node_again&order=bug_status" Keyword 'Verify_Contexted_Constant_On_Active_Nodes' failed after retrying for 10 seconds. The last error was: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.129:8181/rests/operations/odl-mdsal-lowlevel-target:get-contexted-constant ------------------------------------------------------------------------------ Unregister_Rpc_On_Each_Node :: Inregister rpc on both nodes. | FAIL | ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_action_provider_partition_and_heal_unregister_rpc_on_each_node" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_action_provider_partition_and_heal_unregister_rpc_on_each_node&order=bug_status" HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.129:8181/rests/operations/odl-mdsal-lowlevel-control:unregister-bound-constant ------------------------------------------------------------------------------ controller-clustering-ask.txt.Action Provider Partition And Heal :... | FAIL | 7 tests, 0 passed, 7 failed ============================================================================== controller-clustering-ask.txt.Restart Odl With Tell Based False :: Unset te... ============================================================================== Stop_All_Members :: Stop every odl node. | FAIL | Keyword 'Verify_Karaf_Is_Not_Running_On_Member' failed after retrying for 6 minutes. The last error was: Found running Karaf count: 1: 0 != 1 ------------------------------------------------------------------------------ Unset_Tell_Based_Protocol_Usage :: Comment out the flag usage in c... | PASS | ------------------------------------------------------------------------------ Start_All_And_Sync :: Start each member and wait for sync. | FAIL | Keyword 'Verify_Members_Are_Ready' failed after retrying for 6 minutes. The last error was: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /jolokia/read/org.opendaylight.controller:Category=ShardManager,name=shard-manager-config,type=DistributedConfigDatastore (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ controller-clustering-ask.txt.Restart Odl With Tell Based False ::... | FAIL | 3 tests, 1 passed, 2 failed ============================================================================== controller-clustering-ask.txt.Master Stability :: Cluster Singleton testing... ============================================================================== Register_Singleton_Constant_On_Each_Node_And_Verify :: Register a ... | FAIL | ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_master_stability_register_singleton_constant_on_each_node_and_verify" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_master_stability_register_singleton_constant_on_each_node_and_verify&order=bug_status" ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/operations/odl-mdsal-lowlevel-control:register-singleton-constant (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Unregister_Singleton_Constant_On_Non_Master_Node :: Unregister the... | FAIL | ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_master_stability_unregister_singleton_constant_on_non_master_node" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_master_stability_unregister_singleton_constant_on_non_master_node&order=bug_status" Variable '@{cs_candidates}' not found. Did you mean: @{cs_exp_candidates} ------------------------------------------------------------------------------ Monitor_Stability_While_Unregistered :: Verify that the owner rema... | FAIL | ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_master_stability_monitor_stability_while_unregistered" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_master_stability_monitor_stability_while_unregistered&order=bug_status" Variable '${cs_owner}' not found. ------------------------------------------------------------------------------ Reregister_Singleton_Constant :: Re-register the unregistered cand... | FAIL | ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_master_stability_reregister_singleton_constant" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_master_stability_reregister_singleton_constant&order=bug_status" Variable '${unregistered_node}' not found. ------------------------------------------------------------------------------ Verify_Stability_After_Reregistration :: Verify that the owner rem... | FAIL | ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_master_stability_verify_stability_after_reregistration" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_master_stability_verify_stability_after_reregistration&order=bug_status" Variable '${cs_owner}' not found. ------------------------------------------------------------------------------ Unregister_Singleton_Constant_On_Each_Node :: Unregister the appli... | FAIL | ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_master_stability_unregister_singleton_constant_on_each_node" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_master_stability_unregister_singleton_constant_on_each_node&order=bug_status" ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/operations/odl-mdsal-lowlevel-control:unregister-singleton-constant (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ controller-clustering-ask.txt.Master Stability :: Cluster Singleto... | FAIL | 6 tests, 0 passed, 6 failed ============================================================================== controller-clustering-ask.txt.Restart Odl With Tell Based False :: Unset te... ============================================================================== Stop_All_Members :: Stop every odl node. | FAIL | Keyword 'Verify_Karaf_Is_Not_Running_On_Member' failed after retrying for 6 minutes. The last error was: Found running Karaf count: 1: 0 != 1 ------------------------------------------------------------------------------ Unset_Tell_Based_Protocol_Usage :: Comment out the flag usage in c... | PASS | ------------------------------------------------------------------------------ Start_All_And_Sync :: Start each member and wait for sync. | FAIL | Keyword 'Verify_Members_Are_Ready' failed after retrying for 6 minutes. The last error was: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.129:8181/jolokia/read/org.opendaylight.controller:Category=ShardManager,name=shard-manager-config,type=DistributedConfigDatastore ------------------------------------------------------------------------------ controller-clustering-ask.txt.Restart Odl With Tell Based False ::... | FAIL | 3 tests, 1 passed, 2 failed ============================================================================== controller-clustering-ask.txt.Partition And Heal :: Cluster Singleton testi... ============================================================================== Register_Singleton_Constant_On_Each_Node :: Register a candidate a... | FAIL | ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_partition_and_heal_register_singleton_constant_on_each_node" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_partition_and_heal_register_singleton_constant_on_each_node&order=bug_status" HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.129:8181/rests/operations/odl-mdsal-lowlevel-control:register-singleton-constant ------------------------------------------------------------------------------ Verify_Singleton_Constant_On_Each_Node :: Store the owner and cand... | FAIL | ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_partition_and_heal_verify_singleton_constant_on_each_node" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_partition_and_heal_verify_singleton_constant_on_each_node&order=bug_status" Could not parse owner and candidates for device get-singleton-constant-service'] ------------------------------------------------------------------------------ Isolate_Owner_Node :: Isolate the cluster node which is the owner.... | FAIL | ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_partition_and_heal_isolate_owner_node" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_partition_and_heal_isolate_owner_node&order=bug_status" Variable '${cs_owner}' not found. ------------------------------------------------------------------------------ Monitor_Stability_While_Isolated :: Monitor the stability of the s... | FAIL | ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_partition_and_heal_monitor_stability_while_isolated" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_partition_and_heal_monitor_stability_while_isolated&order=bug_status" Invalid IF condition: Evaluating expression '"${index}" == "${cs_isolated_index}"' failed: Variable '${cs_isolated_index}' not found. ------------------------------------------------------------------------------ Rejoin_Isolated_node :: Rejoin isolated node. | FAIL | ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_partition_and_heal_rejoin_isolated_node" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_partition_and_heal_rejoin_isolated_node&order=bug_status" Variable '${cs_isolated_index}' not found. ------------------------------------------------------------------------------ Unregister_Singleton_Constant_On_Each_Node :: Unregister the appli... | FAIL | ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_partition_and_heal_unregister_singleton_constant_on_each_node" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_partition_and_heal_unregister_singleton_constant_on_each_node&order=bug_status" HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.129:8181/rests/operations/odl-mdsal-lowlevel-control:unregister-singleton-constant ------------------------------------------------------------------------------ controller-clustering-ask.txt.Partition And Heal :: Cluster Single... | FAIL | 6 tests, 0 passed, 6 failed ============================================================================== controller-clustering-ask.txt.Restart Odl With Tell Based False :: Unset te... ============================================================================== Stop_All_Members :: Stop every odl node. | FAIL | Keyword 'Verify_Karaf_Is_Not_Running_On_Member' failed after retrying for 6 minutes. The last error was: Found running Karaf count: 1: 0 != 1 ------------------------------------------------------------------------------ Unset_Tell_Based_Protocol_Usage :: Comment out the flag usage in c... | PASS | ------------------------------------------------------------------------------ Start_All_And_Sync :: Start each member and wait for sync. | FAIL | Keyword 'Verify_Members_Are_Ready' failed after retrying for 6 minutes. The last error was: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.129:8181/jolokia/read/org.opendaylight.controller:Category=ShardManager,name=shard-manager-config,type=DistributedConfigDatastore ------------------------------------------------------------------------------ controller-clustering-ask.txt.Restart Odl With Tell Based False ::... | FAIL | 3 tests, 1 passed, 2 failed ============================================================================== controller-clustering-ask.txt.Chasing The Leader :: Cluster Singleton testi... ============================================================================== Register_Candidates :: Register a candidate application on each no... | FAIL | ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_chasing_the_leader_register_candidates" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_chasing_the_leader_register_candidates&order=bug_status" HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.129:8181/rests/operations/odl-mdsal-lowlevel-control:register-flapping-singleton ------------------------------------------------------------------------------ Do_Nothing :: Do nothing for the time of the test duration, becaus... | PASS | ------------------------------------------------------------------------------ Unregister_Candidates_And_Validate_Criteria :: Unregister the test... | FAIL | ... click for list of related bugs or create a new one if needed (with the "controller_clustering_ask_txt_chasing_the_leader_unregister_candidates_and_validate_criteria" reference somewhere inside) "https://bugs.opendaylight.org/buglist.cgi?f1=cf_external_ref&o1=substring&v1=controller_clustering_ask_txt_chasing_the_leader_unregister_candidates_and_validate_criteria&order=bug_status" HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.129:8181/rests/operations/odl-mdsal-lowlevel-control:unregister-flapping-singleton ------------------------------------------------------------------------------ controller-clustering-ask.txt.Chasing The Leader :: Cluster Single... | FAIL | 3 tests, 1 passed, 2 failed ============================================================================== controller-clustering-ask.txt.Restart Odl With Tell Based False :: Unset te... ============================================================================== Stop_All_Members :: Stop every odl node. | FAIL | Keyword 'Verify_Karaf_Is_Not_Running_On_Member' failed after retrying for 6 minutes. The last error was: Found running Karaf count: 1: 0 != 1 ------------------------------------------------------------------------------ Unset_Tell_Based_Protocol_Usage :: Comment out the flag usage in c... | PASS | ------------------------------------------------------------------------------ Start_All_And_Sync :: Start each member and wait for sync. | FAIL | Keyword 'Verify_Members_Are_Ready' failed after retrying for 6 minutes. The last error was: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.129:8181/jolokia/read/org.opendaylight.controller:Category=ShardManager,name=shard-manager-config,type=DistributedConfigDatastore ------------------------------------------------------------------------------ controller-clustering-ask.txt.Restart Odl With Tell Based False ::... | FAIL | 3 tests, 1 passed, 2 failed ============================================================================== controller-clustering-ask.txt.Global Rpc Kill :: Controller functional HA t... ============================================================================== Get_Basic_Rpc_Test_Owner :: Find a service owner and successors. | FAIL | Parent suite setup failed: SSHException: No existing session ------------------------------------------------------------------------------ Rpc_Before_Stopping_On_Owner :: Run rpc on the service owner. | FAIL | Parent suite setup failed: SSHException: No existing session ------------------------------------------------------------------------------ Rpc_Before_Stop_On_Successors :: Run rpc on non owher cluster nodes. | FAIL | Parent suite setup failed: SSHException: No existing session ------------------------------------------------------------------------------ Stop_Current_Owner_Member :: Stop cluster node which is the owner. | FAIL | Parent suite setup failed: SSHException: No existing session ------------------------------------------------------------------------------ Verify_New_Basic_Rpc_Test_Owner_Elected :: Verify new owner of the... | FAIL | Parent suite setup failed: SSHException: No existing session ------------------------------------------------------------------------------ Rpc_On_Remained_Cluster_Nodes :: Run rpc on remained cluster nodes. | FAIL | Parent suite setup failed: SSHException: No existing session ------------------------------------------------------------------------------ Restart_Stopped_Member :: Restart stopped node | FAIL | Parent suite setup failed: SSHException: No existing session ------------------------------------------------------------------------------ Verify_New_Owner_Remained_After_Rejoin :: Verify no owner change h... | FAIL | Parent suite setup failed: SSHException: No existing session ------------------------------------------------------------------------------ Rpc_After_Rejoin_On_New_Owner :: Run rpc on the new service owner ... | FAIL | Parent suite setup failed: SSHException: No existing session ------------------------------------------------------------------------------ Rpc_After_Rejoin_On_Old_Owner :: Run rpc on rejoined cluster node. | FAIL | Parent suite setup failed: SSHException: No existing session ------------------------------------------------------------------------------ Rpc_After_Rejoin_On_All :: Run rpc again on all nodes. | FAIL | Parent suite setup failed: SSHException: No existing session ------------------------------------------------------------------------------ controller-clustering-ask.txt.Global Rpc Kill :: Controller functi... | FAIL | Suite setup failed: SSHException: No existing session 11 tests, 0 passed, 11 failed ============================================================================== controller-clustering-ask.txt.Restart Odl With Tell Based False :: Unset te... ============================================================================== Stop_All_Members :: Stop every odl node. | FAIL | Keyword 'Verify_Karaf_Is_Not_Running_On_Member' failed after retrying for 6 minutes. The last error was: Found running Karaf count: 1: 0 != 1 ------------------------------------------------------------------------------ Unset_Tell_Based_Protocol_Usage :: Comment out the flag usage in c... | PASS | ------------------------------------------------------------------------------ Start_All_And_Sync :: Start each member and wait for sync. | FAIL | Keyword 'Verify_Members_Are_Ready' failed after retrying for 6 minutes. The last error was: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.129:8181/jolokia/read/org.opendaylight.controller:Category=ShardManager,name=shard-manager-config,type=DistributedConfigDatastore ------------------------------------------------------------------------------ controller-clustering-ask.txt.Restart Odl With Tell Based False ::... | FAIL | 3 tests, 1 passed, 2 failed ============================================================================== controller-clustering-ask.txt.Global Rpc Freeze :: Controller functional HA... ============================================================================== Get_Basic_Rpc_Test_Owner :: Find a service owner and successors. | FAIL | Parent suite setup failed: SSHException: No existing session ------------------------------------------------------------------------------ Rpc_Before_Freezing_On_Owner :: Run rpc on the service owner. | FAIL | Parent suite setup failed: SSHException: No existing session ------------------------------------------------------------------------------ Rpc_Before_Freeze_On_Successors :: Run rpc on non owher cluster no... | FAIL | Parent suite setup failed: SSHException: No existing session ------------------------------------------------------------------------------ Freeze_Current_Owner_Member :: Stop cluster node which is the owner. | FAIL | Parent suite setup failed: SSHException: No existing session ------------------------------------------------------------------------------ Verify_New_Basic_Rpc_Test_Owner_Elected :: Verify new owner of the... | FAIL | Parent suite setup failed: SSHException: No existing session ------------------------------------------------------------------------------ Rpc_On_Remained_Cluster_Nodes :: Run rpc on remained cluster nodes. | FAIL | Parent suite setup failed: SSHException: No existing session ------------------------------------------------------------------------------ Unfreeze_Frozen_Member :: Restart frozen node | FAIL | Parent suite setup failed: SSHException: No existing session ------------------------------------------------------------------------------ Rpc_After_Rejoin_On_New_Owner :: Run rpc on the new service owner ... | FAIL | Parent suite setup failed: SSHException: No existing session ------------------------------------------------------------------------------ Rpc_After_Rejoin_On_Old_Owner :: Run rpc on rejoined cluster node. | FAIL | Parent suite setup failed: SSHException: No existing session ------------------------------------------------------------------------------ Rpc_After_Rejoin_On_All :: Run rpc again on all nodes. | FAIL | Parent suite setup failed: SSHException: No existing session ------------------------------------------------------------------------------ controller-clustering-ask.txt.Global Rpc Freeze :: Controller func... | FAIL | Suite setup failed: SSHException: No existing session 10 tests, 0 passed, 10 failed ============================================================================== controller-clustering-ask.txt.Restart Odl With Tell Based False :: Unset te... ============================================================================== Stop_All_Members :: Stop every odl node. | FAIL | Keyword 'Verify_Karaf_Is_Not_Running_On_Member' failed after retrying for 6 minutes. The last error was: Found running Karaf count: 1: 0 != 1 ------------------------------------------------------------------------------ Unset_Tell_Based_Protocol_Usage :: Comment out the flag usage in c... | PASS | ------------------------------------------------------------------------------ Start_All_And_Sync :: Start each member and wait for sync. | FAIL | Keyword 'Verify_Members_Are_Ready' failed after retrying for 6 minutes. The last error was: HTTPError: 401 Client Error: Unauthorized for url: http://10.30.171.129:8181/jolokia/read/org.opendaylight.controller:Category=ShardManager,name=shard-manager-config,type=DistributedConfigDatastore ------------------------------------------------------------------------------ controller-clustering-ask.txt.Restart Odl With Tell Based False ::... | FAIL | 3 tests, 1 passed, 2 failed ============================================================================== controller-clustering-ask.txt.Global Rpc Isolate :: Controller functional H... ============================================================================== Get_Basic_Rpc_Test_Owner :: Find a service owner and successors. | FAIL | Parent suite setup failed: SSHException: No existing session ------------------------------------------------------------------------------ Rpc_Before_Isolation_On_Owner :: Run rpc on the service owner. | FAIL | Parent suite setup failed: SSHException: No existing session ------------------------------------------------------------------------------ Rpc_Before_Isolation_On_Successors :: Run rpc on non owher cluster... | FAIL | Parent suite setup failed: SSHException: No existing session ------------------------------------------------------------------------------ Isolate_Current_Owner_Member :: Isolating cluster node which is th... | FAIL | Parent suite setup failed: SSHException: No existing session ------------------------------------------------------------------------------ Verify_New_Basic_Rpc_Test_Owner_Elected :: Verify new owner of the... | FAIL | Parent suite setup failed: SSHException: No existing session ------------------------------------------------------------------------------ Rpc_On_Isolated_Node :: Run rpc on isolated cluster node. | FAIL | Parent suite setup failed: SSHException: No existing session ------------------------------------------------------------------------------ Rpc_On_Non_Isolated_Cluster_Nodes :: Run rpc on remained cluster n... | FAIL | Parent suite setup failed: SSHException: No existing session ------------------------------------------------------------------------------ Rejoin_Isolated_Member :: Rejoin isolated node | FAIL | Parent suite setup failed: SSHException: No existing session ------------------------------------------------------------------------------ Rpc_After_Rejoin_On_New_Owner :: Run rpc on the new service owner ... | FAIL | Parent suite setup failed: SSHException: No existing session ------------------------------------------------------------------------------ Rpc_After_Rejoin_On_Old_Owner :: Run rpc on rejoined cluster node. | FAIL | Parent suite setup failed: SSHException: No existing session ------------------------------------------------------------------------------ Rpc_After_Rejoin_On_All :: Run rpc again on all nodes. | FAIL | Parent suite setup failed: SSHException: No existing session ------------------------------------------------------------------------------ controller-clustering-ask.txt.Global Rpc Isolate :: Controller fun... | FAIL | Suite setup failed: SSHException: No existing session 11 tests, 0 passed, 11 failed ============================================================================== controller-clustering-ask.txt.Restart Odl With Tell Based False :: Unset te... ============================================================================== Stop_All_Members :: Stop every odl node. | FAIL | Keyword 'Verify_Karaf_Is_Not_Running_On_Member' failed after retrying for 6 minutes. The last error was: Found running Karaf count: 1: 0 != 1 ------------------------------------------------------------------------------ Unset_Tell_Based_Protocol_Usage :: Comment out the flag usage in c... | PASS | ------------------------------------------------------------------------------ Start_All_And_Sync :: Start each member and wait for sync. | FAIL | Keyword 'Verify_Members_Are_Ready' failed after retrying for 6 minutes. The last error was: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /jolokia/read/org.opendaylight.controller:Category=ShardManager,name=shard-manager-config,type=DistributedConfigDatastore (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ controller-clustering-ask.txt.Restart Odl With Tell Based False ::... | FAIL | 3 tests, 1 passed, 2 failed ============================================================================== controller-clustering-ask.txt.Carpeople Crud :: Suite for performing basic ... ============================================================================== Add_Cars_To_Leader :: Add 30 cars to car Leader by one big PUT. | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ See_Added_Cars_On_Leader :: GET response from Leader should match ... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ See_Added_Cars_On_Followers :: The same check on other members. | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Add_People_To_First_Follower :: Add 30 people to people first Foll... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ See_Added_People_On_Leader :: GET response from Leader should matc... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ See_Added_People_On_Followers :: The same check on other members. | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Buy_Cars_On_Leader :: Buy some cars on car-people Leader, loop of ... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Buy_Cars_On_Followers :: On each Follower buy corresponding ID seg... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ See_Added_CarPeople_On_Leader :: GET car-person mappings from Lead... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ See_Added_CarPeople_On_Followers :: The same check on other members. | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Delete_All_CarPeople_On_Leader :: DELETE car-people container. No ... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Delete_All_People_On_Leader :: DELETE people container. No verific... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Delete_All_Cars_On_Leader :: DELETE cars container. No verificatio... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ controller-clustering-ask.txt.Carpeople Crud :: Suite for performi... | FAIL | Suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) 13 tests, 0 passed, 13 failed ============================================================================== controller-clustering-ask.txt.Car Failover Crud :: Suite mixing basic opera... ============================================================================== Add_Original_Cars_On_Old_Leader_And_Verify :: Add initial cars on ... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Stop_Original_Car_Leader :: Stop the car Leader to cause a new lea... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Wait_For_New_Leader :: Wait until new car Leader is elected. | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ See_Original_Cars_On_New_Leader :: GET cars from new Leader, shoul... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ See_Original_Cars_On_New_Followers :: The same check on other exis... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Delete_Original_Cars_On_New_Leader :: Delete cars on the new Leader. | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Add_Leader_Cars_On_New_Leader :: Add cars on the new Leader. | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ See_Leader_Cars_On_New_Leader :: GET cars from new Leader, should ... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ See_Leader_Cars_On_New_Followers :: The same check on other existi... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Delete_Leader_Cars_On_New_First_Follower :: Delete cars in new fir... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Add_Follower_Cars_On_New_First_Follower :: Add cars on the new fir... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ See_Folower_Cars_On_New_Leader :: Get cars from the new Leader, sh... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ See_Follower_Cars_On_New_Followers :: The same check on other exis... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Start_Old_Car_Leader :: Start the stopped member without deleting ... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ See_Folower_Cars_On_Old_Leader :: GET cars from the restarted memb... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Delete_Follower_Cars_On_New_Leader :: Delete cars on the last Leader. | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ controller-clustering-ask.txt.Car Failover Crud :: Suite mixing ba... | FAIL | Suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) 16 tests, 0 passed, 16 failed ============================================================================== controller-clustering-ask.txt.Car Persistence Recovery :: This test restart... ============================================================================== Add_Cars_On_Leader_And_Verify :: Single big PUT to datastore to ad... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Stop_All_Members :: Stop all controllers. | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Start_All_Members :: Start all controllers (should restore the per... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Memorize_Leader_And_Followers :: Locate current Leader of car Shard. | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ See_Cars_On_Leader :: GET cars from Leader, should match the PUT d... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ See_Cars_On_Followers :: The same check on other members. | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Delete_Cars_On_Leader :: Delete cars on the new Leader. | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ controller-clustering-ask.txt.Car Persistence Recovery :: This tes... | FAIL | Suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) 7 tests, 0 passed, 7 failed ============================================================================== controller-clustering-ask.txt.Buycar Failover :: This test focuses on testi... ============================================================================== Add_Cars_To_Leader_And_Verify :: Add all needed cars to car Leader... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Add_People_To_First_Follower_And_Verify :: Add all needed people t... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Buy_Cars_On_Leader_And_Verify :: Buy some cars on the leader member. | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Buy_Cars_On_Follower_And_Verify :: Buy some cars on the first foll... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Reboot_People_Leader :: Previous people Leader is rebooted. We sho... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Buy_Cars_On_Leader_After_Reboot_And_Verify :: Buy some cars on the... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Buy_Cars_On_Follower_After_Reboot_And_Verify :: Buy some cars on t... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Delete_All_CarPeople :: DELETE car-people container. No verificati... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Delete_All_People :: DELETE people container. No verification beyo... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Delete_All_Cars :: DELETE cars container. No verification beyond h... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ controller-clustering-ask.txt.Buycar Failover :: This test focuses... | FAIL | Suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) 10 tests, 0 passed, 10 failed ============================================================================== controller-clustering-ask.txt.Entity Isolate :: Suite for performing member... ============================================================================== Check All Shards Before Isolate :: Check all shards in controller. | FAIL | Parent suite setup failed: Evaluating expression 'json.loads(\'\'\'{\n "error": "javax.management.InstanceNotFoundException : org.opendaylight.controller:Category=Shards,name=member-1-shard-entity-ownership-operational,type=DistributedOperationalDatastore",\n "error_type": "javax.management.InstanceNotFoundException",\n "request": {\n "mbean": "org.opendaylight.controller:Category=Shards,name=member-1-shard-entity-ownership-operational,type=DistributedOperationalDatastore",\n "type": "read"\n },\n "stacktrace": "javax.management.InstanceNotFoundException: org.opendaylight.controller:Category=Shards,name=member-1-shard-entity-ownership-operational,type=DistributedOperationalDatastore\\n\\tat java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1073)\\n\\tat java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBeanInfo(DefaultMBeanServerInterceptor.java:1343)\\n\\tat java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.getMBeanInfo(JmxMBeanServer.java:921)\\n\\tat org.jolokia.handler.ReadHandler$1.execute(ReadHandler.java:46)\\n\\tat org.jolokia.handler.ReadHandler$1.execute(ReadHandler.java:41)\\n\\tat org.jolokia.backend.executor.AbstractMBeanServerExecutor.call(AbstractMBeanServerExecutor.java:90)\\n\\tat org.jolokia.handler.ReadHandler.getMBeanInfo(ReadHandler.java:233)\\n\\tat org.jolokia.handler.ReadHandler.getAllAttributesNames(ReadHandler.java:245)\\n\\tat org.jolokia.handler.ReadHandler.resolveAttributes(ReadHandler.java:221)\\n\\tat org.jolokia.handler.ReadHandler.f... [ Message content over the limit has been removed. ] ...lipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\\n\\tat org.eclipse.jetty.server.Server.handle(Server.java:516)\\n\\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)\\n\\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)\\n\\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)\\n\\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)\\n\\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)\\n\\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)\\n\\tat org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)\\n\\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)\\n\\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)\\n\\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)\\n\\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)\\n\\tat org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)\\n\\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)\\n\\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)\\n\\tat java.base/java.lang.Thread.run(Thread.java:1583)\\n",\n "status": 404\n}\n\'\'\')' failed: JSONDecodeError: Invalid control character at: line 8 column 190 (char 619) ------------------------------------------------------------------------------ Isolate Entity Leader :: Isolate the entity-ownership Leader to ca... | FAIL | Parent suite setup failed: Evaluating expression 'json.loads(\'\'\'{\n "error": "javax.management.InstanceNotFoundException : org.opendaylight.controller:Category=Shards,name=member-1-shard-entity-ownership-operational,type=DistributedOperationalDatastore",\n "error_type": "javax.management.InstanceNotFoundException",\n "request": {\n "mbean": "org.opendaylight.controller:Category=Shards,name=member-1-shard-entity-ownership-operational,type=DistributedOperationalDatastore",\n "type": "read"\n },\n "stacktrace": "javax.management.InstanceNotFoundException: org.opendaylight.controller:Category=Shards,name=member-1-shard-entity-ownership-operational,type=DistributedOperationalDatastore\\n\\tat java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1073)\\n\\tat java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBeanInfo(DefaultMBeanServerInterceptor.java:1343)\\n\\tat java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.getMBeanInfo(JmxMBeanServer.java:921)\\n\\tat org.jolokia.handler.ReadHandler$1.execute(ReadHandler.java:46)\\n\\tat org.jolokia.handler.ReadHandler$1.execute(ReadHandler.java:41)\\n\\tat org.jolokia.backend.executor.AbstractMBeanServerExecutor.call(AbstractMBeanServerExecutor.java:90)\\n\\tat org.jolokia.handler.ReadHandler.getMBeanInfo(ReadHandler.java:233)\\n\\tat org.jolokia.handler.ReadHandler.getAllAttributesNames(ReadHandler.java:245)\\n\\tat org.jolokia.handler.ReadHandler.resolveAttributes(ReadHandler.java:221)\\n\\tat org.jolokia.handler.ReadHandler.f... [ Message content over the limit has been removed. ] ...lipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\\n\\tat org.eclipse.jetty.server.Server.handle(Server.java:516)\\n\\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)\\n\\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)\\n\\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)\\n\\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)\\n\\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)\\n\\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)\\n\\tat org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)\\n\\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)\\n\\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)\\n\\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)\\n\\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)\\n\\tat org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)\\n\\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)\\n\\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)\\n\\tat java.base/java.lang.Thread.run(Thread.java:1583)\\n",\n "status": 404\n}\n\'\'\')' failed: JSONDecodeError: Invalid control character at: line 8 column 190 (char 619) ------------------------------------------------------------------------------ Check All Shards After Isolate :: Check all shards in controller. | FAIL | Parent suite setup failed: Evaluating expression 'json.loads(\'\'\'{\n "error": "javax.management.InstanceNotFoundException : org.opendaylight.controller:Category=Shards,name=member-1-shard-entity-ownership-operational,type=DistributedOperationalDatastore",\n "error_type": "javax.management.InstanceNotFoundException",\n "request": {\n "mbean": "org.opendaylight.controller:Category=Shards,name=member-1-shard-entity-ownership-operational,type=DistributedOperationalDatastore",\n "type": "read"\n },\n "stacktrace": "javax.management.InstanceNotFoundException: org.opendaylight.controller:Category=Shards,name=member-1-shard-entity-ownership-operational,type=DistributedOperationalDatastore\\n\\tat java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1073)\\n\\tat java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBeanInfo(DefaultMBeanServerInterceptor.java:1343)\\n\\tat java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.getMBeanInfo(JmxMBeanServer.java:921)\\n\\tat org.jolokia.handler.ReadHandler$1.execute(ReadHandler.java:46)\\n\\tat org.jolokia.handler.ReadHandler$1.execute(ReadHandler.java:41)\\n\\tat org.jolokia.backend.executor.AbstractMBeanServerExecutor.call(AbstractMBeanServerExecutor.java:90)\\n\\tat org.jolokia.handler.ReadHandler.getMBeanInfo(ReadHandler.java:233)\\n\\tat org.jolokia.handler.ReadHandler.getAllAttributesNames(ReadHandler.java:245)\\n\\tat org.jolokia.handler.ReadHandler.resolveAttributes(ReadHandler.java:221)\\n\\tat org.jolokia.handler.ReadHandler.f... [ Message content over the limit has been removed. ] ...lipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\\n\\tat org.eclipse.jetty.server.Server.handle(Server.java:516)\\n\\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)\\n\\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)\\n\\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)\\n\\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)\\n\\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)\\n\\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)\\n\\tat org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)\\n\\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)\\n\\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)\\n\\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)\\n\\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)\\n\\tat org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)\\n\\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)\\n\\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)\\n\\tat java.base/java.lang.Thread.run(Thread.java:1583)\\n",\n "status": 404\n}\n\'\'\')' failed: JSONDecodeError: Invalid control character at: line 8 column 190 (char 619) ------------------------------------------------------------------------------ Rejoin Entity Leader :: Rejoin the entity-ownership Leader. | FAIL | Parent suite setup failed: Evaluating expression 'json.loads(\'\'\'{\n "error": "javax.management.InstanceNotFoundException : org.opendaylight.controller:Category=Shards,name=member-1-shard-entity-ownership-operational,type=DistributedOperationalDatastore",\n "error_type": "javax.management.InstanceNotFoundException",\n "request": {\n "mbean": "org.opendaylight.controller:Category=Shards,name=member-1-shard-entity-ownership-operational,type=DistributedOperationalDatastore",\n "type": "read"\n },\n "stacktrace": "javax.management.InstanceNotFoundException: org.opendaylight.controller:Category=Shards,name=member-1-shard-entity-ownership-operational,type=DistributedOperationalDatastore\\n\\tat java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1073)\\n\\tat java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBeanInfo(DefaultMBeanServerInterceptor.java:1343)\\n\\tat java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.getMBeanInfo(JmxMBeanServer.java:921)\\n\\tat org.jolokia.handler.ReadHandler$1.execute(ReadHandler.java:46)\\n\\tat org.jolokia.handler.ReadHandler$1.execute(ReadHandler.java:41)\\n\\tat org.jolokia.backend.executor.AbstractMBeanServerExecutor.call(AbstractMBeanServerExecutor.java:90)\\n\\tat org.jolokia.handler.ReadHandler.getMBeanInfo(ReadHandler.java:233)\\n\\tat org.jolokia.handler.ReadHandler.getAllAttributesNames(ReadHandler.java:245)\\n\\tat org.jolokia.handler.ReadHandler.resolveAttributes(ReadHandler.java:221)\\n\\tat org.jolokia.handler.ReadHandler.f... [ Message content over the limit has been removed. ] ...lipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\\n\\tat org.eclipse.jetty.server.Server.handle(Server.java:516)\\n\\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)\\n\\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)\\n\\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)\\n\\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)\\n\\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)\\n\\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)\\n\\tat org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)\\n\\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)\\n\\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)\\n\\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)\\n\\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)\\n\\tat org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)\\n\\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)\\n\\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)\\n\\tat java.base/java.lang.Thread.run(Thread.java:1583)\\n",\n "status": 404\n}\n\'\'\')' failed: JSONDecodeError: Invalid control character at: line 8 column 190 (char 619) ------------------------------------------------------------------------------ Check All Shards After Rejoin :: Check all shards in controller. | FAIL | Parent suite setup failed: Evaluating expression 'json.loads(\'\'\'{\n "error": "javax.management.InstanceNotFoundException : org.opendaylight.controller:Category=Shards,name=member-1-shard-entity-ownership-operational,type=DistributedOperationalDatastore",\n "error_type": "javax.management.InstanceNotFoundException",\n "request": {\n "mbean": "org.opendaylight.controller:Category=Shards,name=member-1-shard-entity-ownership-operational,type=DistributedOperationalDatastore",\n "type": "read"\n },\n "stacktrace": "javax.management.InstanceNotFoundException: org.opendaylight.controller:Category=Shards,name=member-1-shard-entity-ownership-operational,type=DistributedOperationalDatastore\\n\\tat java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1073)\\n\\tat java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBeanInfo(DefaultMBeanServerInterceptor.java:1343)\\n\\tat java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.getMBeanInfo(JmxMBeanServer.java:921)\\n\\tat org.jolokia.handler.ReadHandler$1.execute(ReadHandler.java:46)\\n\\tat org.jolokia.handler.ReadHandler$1.execute(ReadHandler.java:41)\\n\\tat org.jolokia.backend.executor.AbstractMBeanServerExecutor.call(AbstractMBeanServerExecutor.java:90)\\n\\tat org.jolokia.handler.ReadHandler.getMBeanInfo(ReadHandler.java:233)\\n\\tat org.jolokia.handler.ReadHandler.getAllAttributesNames(ReadHandler.java:245)\\n\\tat org.jolokia.handler.ReadHandler.resolveAttributes(ReadHandler.java:221)\\n\\tat org.jolokia.handler.ReadHandler.f... [ Message content over the limit has been removed. ] ...lipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\\n\\tat org.eclipse.jetty.server.Server.handle(Server.java:516)\\n\\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)\\n\\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)\\n\\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)\\n\\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)\\n\\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)\\n\\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)\\n\\tat org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)\\n\\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)\\n\\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)\\n\\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)\\n\\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)\\n\\tat org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)\\n\\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)\\n\\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)\\n\\tat java.base/java.lang.Thread.run(Thread.java:1583)\\n",\n "status": 404\n}\n\'\'\')' failed: JSONDecodeError: Invalid control character at: line 8 column 190 (char 619) ------------------------------------------------------------------------------ controller-clustering-ask.txt.Entity Isolate :: Suite for performi... | FAIL | Suite setup failed: Evaluating expression 'json.loads(\'\'\'{\n "error": "javax.management.InstanceNotFoundException : org.opendaylight.controller:Category=Shards,name=member-1-shard-entity-ownership-operational,type=DistributedOperationalDatastore",\n "error_type": "javax.management.InstanceNotFoundException",\n "request": {\n "mbean": "org.opendaylight.controller:Category=Shards,name=member-1-shard-entity-ownership-operational,type=DistributedOperationalDatastore",\n "type": "read"\n },\n "stacktrace": "javax.management.InstanceNotFoundException: org.opendaylight.controller:Category=Shards,name=member-1-shard-entity-ownership-operational,type=DistributedOperationalDatastore\\n\\tat java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1073)\\n\\tat java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBeanInfo(DefaultMBeanServerInterceptor.java:1343)\\n\\tat java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.getMBeanInfo(JmxMBeanServer.java:921)\\n\\tat org.jolokia.handler.ReadHandler$1.execute(ReadHandler.java:46)\\n\\tat org.jolokia.handler.ReadHandler$1.execute(ReadHandler.java:41)\\n\\tat org.jolokia.backend.executor.AbstractMBeanServerExecutor.call(AbstractMBeanServerExecutor.java:90)\\n\\tat org.jolokia.handler.ReadHandler.getMBeanInfo(ReadHandler.java:233)\\n\\tat org.jolokia.handler.ReadHandler.getAllAttributesNames(ReadHandler.java:245)\\n\\tat org.jolokia.handler.ReadHandler.resolveAttributes(ReadHandler.java:221)\\n\\tat org.jolokia.handler.ReadHandler.f... [ Message content over the limit has been removed. ] ...lipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\\n\\tat org.eclipse.jetty.server.Server.handle(Server.java:516)\\n\\tat org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)\\n\\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)\\n\\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)\\n\\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)\\n\\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)\\n\\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)\\n\\tat org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)\\n\\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)\\n\\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)\\n\\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)\\n\\tat org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)\\n\\tat org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)\\n\\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)\\n\\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)\\n\\tat java.base/java.lang.Thread.run(Thread.java:1583)\\n",\n "status": 404\n}\n\'\'\')' failed: JSONDecodeError: Invalid control character at: line 8 column 190 (char 619) 5 tests, 0 passed, 5 failed ============================================================================== controller-clustering-ask.txt.Buycar Failover Isolation :: This test focuse... ============================================================================== Add_Cars_To_Leader_And_Verify :: Add all needed cars to car Leader... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Add_People_To_First_Follower_And_Verify :: Add all needed people t... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Buy_Cars_On_Leader_And_Verify :: Buy some cars on the leader member. | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Buy_Cars_On_Follower_And_Verify :: Buy some cars on the first foll... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Isolate_and_Rejoin_People_Leader :: Previous people Leader is isol... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Buy_Cars_On_Leader_After_Rejoin_And_Verify :: Buy some cars on the... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Buy_Cars_On_Follower_After_Rejoin_And_Verify :: Buy some cars on t... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Delete_All_CarPeople :: DELETE car-people container. No verificati... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Delete_All_People :: DELETE people container. No verification beyo... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Delete_All_Cars :: DELETE cars container. No verification beyond h... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ controller-clustering-ask.txt.Buycar Failover Isolation :: This te... | FAIL | Suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) 10 tests, 0 passed, 10 failed ============================================================================== controller-clustering-ask.txt.Car Failover Crud Isolation :: Suite mixing b... ============================================================================== Add_Original_Cars_On_Old_Leader_And_Verify :: Add initial cars on ... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Isolate_Original_Car_Leader :: Isolate the car Leader to cause a n... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Wait_For_New_Leader :: Wait until new car Leader is elected. | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ See_Original_Cars_On_New_Leader :: GET cars from new Leader, shoul... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ See_Original_Cars_On_New_Followers :: The same check on other exis... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Delete_Original_Cars_On_New_Leader :: Delete cars on the new Leader. | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Add_Leader_Cars_On_New_Leader :: Add cars on the new Leader. | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ See_Leader_Cars_On_New_Leader :: GET cars from new Leader, should ... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ See_Leader_Cars_On_New_Followers :: The same check on other existi... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Delete_Leader_Cars_On_New_First_Follower :: Delete cars in new fir... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Add_Follower_Cars_On_New_First_Follower :: Add cars on the new fir... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ See_Folower_Cars_On_New_Leader :: Get cars from the new Leader, sh... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ See_Follower_Cars_On_New_Followers :: The same check on other exis... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Rejoin_Old_Car_Leader :: Rejoin the isolated member without deleti... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ See_Folower_Cars_On_Old_Leader :: GET cars from the restarted memb... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Delete_Follower_Cars_On_New_Leader :: Delete cars on the last Leader. | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ controller-clustering-ask.txt.Car Failover Crud Isolation :: Suite... | FAIL | Suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) 16 tests, 0 passed, 16 failed ============================================================================== controller-clustering-ask.txt.Car Outage Corners :: Cluster suite for testi... ============================================================================== Stop_Majority_Of_The_Followers :: Stop half plus one car Follower ... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Attempt_To_Add_Cars_To_Leader :: Adding cars should fail, as major... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Start_Tipping_Follower :: Start one Follower member without persis... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Add_Cars_On_Tipping_Follower :: Add cars on the tipping Follower. | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ See_Cars_On_Existing_Members :: On each up member: GET cars, shoul... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Start_Other_Followers :: Start other followers without persisted d... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ See_Cars_On_New_Follower_Leader :: GET cars from a new follower to... | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ Delete_Cars_On_Leader :: Delete cars on Leader. | FAIL | Parent suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) ------------------------------------------------------------------------------ controller-clustering-ask.txt.Car Outage Corners :: Cluster suite ... | FAIL | Suite setup failed: ConnectionError: HTTPConnectionPool(host='10.30.171.129', port=8181): Max retries exceeded with url: /rests/data/ietf-yang-library:modules-state?content=nonconfig (Caused by NewConnectionError("HTTPConnection(host='10.30.171.129', port=8181): Failed to establish a new connection: [Errno 111] Connection refused")) 8 tests, 0 passed, 8 failed ============================================================================== controller-clustering-ask.txt | FAIL | 195 tests, 24 passed, 171 failed ============================================================================== Output: /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/output.xml Log: /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/log.html Report: /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/report.html + true + echo 'Examining the files in data/log and checking filesize' Examining the files in data/log and checking filesize + ssh 10.30.170.104 'ls -altr /tmp/karaf-0.24.0-SNAPSHOT/data/log/' Warning: Permanently added '10.30.170.104' (ECDSA) to the list of known hosts. total 8668 drwxrwxr-x 2 jenkins jenkins 4096 Jan 17 23:44 . -rw-rw-r-- 1 jenkins jenkins 1720 Jan 17 23:44 karaf_console.log drwxrwxr-x 9 jenkins jenkins 4096 Jan 18 02:23 .. -rw-rw-r-- 1 jenkins jenkins 8855929 Jan 18 02:34 karaf.log + ssh 10.30.170.104 'du -hs /tmp/karaf-0.24.0-SNAPSHOT/data/log/*' Warning: Permanently added '10.30.170.104' (ECDSA) to the list of known hosts. 8.5M /tmp/karaf-0.24.0-SNAPSHOT/data/log/karaf.log 4.0K /tmp/karaf-0.24.0-SNAPSHOT/data/log/karaf_console.log + ssh 10.30.171.129 'ls -altr /tmp/karaf-0.24.0-SNAPSHOT/data/log/' Warning: Permanently added '10.30.171.129' (ECDSA) to the list of known hosts. total 7794096 -rw-rw-r-- 1 jenkins jenkins 1720 Jan 17 23:44 karaf_console.log -rw-rw-r-- 1 jenkins jenkins 1073742916 Jan 18 01:28 karaf.log.1 -rw-rw-r-- 1 jenkins jenkins 1073742512 Jan 18 01:41 karaf.log.2 -rw-rw-r-- 1 jenkins jenkins 1073742396 Jan 18 01:46 karaf.log.3 -rw-rw-r-- 1 jenkins jenkins 1073742680 Jan 18 01:56 karaf.log.4 -rw-rw-r-- 1 jenkins jenkins 1073742607 Jan 18 02:00 karaf.log.5 -rw-rw-r-- 1 jenkins jenkins 1073742769 Jan 18 02:10 karaf.log.6 -rw-rw-r-- 1 jenkins jenkins 1073741989 Jan 18 02:16 karaf.log.7 drwxrwxr-x 2 jenkins jenkins 4096 Jan 18 02:17 . drwxrwxr-x 9 jenkins jenkins 4096 Jan 18 02:23 .. -rw-rw-r-- 1 jenkins jenkins 464887044 Jan 18 02:25 karaf.log + ssh 10.30.171.129 'du -hs /tmp/karaf-0.24.0-SNAPSHOT/data/log/*' Warning: Permanently added '10.30.171.129' (ECDSA) to the list of known hosts. 444M /tmp/karaf-0.24.0-SNAPSHOT/data/log/karaf.log 1.1G /tmp/karaf-0.24.0-SNAPSHOT/data/log/karaf.log.1 1.1G /tmp/karaf-0.24.0-SNAPSHOT/data/log/karaf.log.2 1.1G /tmp/karaf-0.24.0-SNAPSHOT/data/log/karaf.log.3 1.1G /tmp/karaf-0.24.0-SNAPSHOT/data/log/karaf.log.4 1.1G /tmp/karaf-0.24.0-SNAPSHOT/data/log/karaf.log.5 1.1G /tmp/karaf-0.24.0-SNAPSHOT/data/log/karaf.log.6 1.1G /tmp/karaf-0.24.0-SNAPSHOT/data/log/karaf.log.7 4.0K /tmp/karaf-0.24.0-SNAPSHOT/data/log/karaf_console.log + ssh 10.30.170.156 'ls -altr /tmp/karaf-0.24.0-SNAPSHOT/data/log/' Warning: Permanently added '10.30.170.156' (ECDSA) to the list of known hosts. total 13544 drwxrwxr-x 2 jenkins jenkins 4096 Jan 17 23:44 . -rw-rw-r-- 1 jenkins jenkins 1720 Jan 17 23:44 karaf_console.log drwxrwxr-x 4 jenkins jenkins 4096 Jan 18 02:23 .. -rw-rw-r-- 1 jenkins jenkins 13852417 Jan 18 02:34 karaf.log + ssh 10.30.170.156 'du -hs /tmp/karaf-0.24.0-SNAPSHOT/data/log/*' Warning: Permanently added '10.30.170.156' (ECDSA) to the list of known hosts. 14M /tmp/karaf-0.24.0-SNAPSHOT/data/log/karaf.log 4.0K /tmp/karaf-0.24.0-SNAPSHOT/data/log/karaf_console.log + set +e ++ seq 1 3 + for i in $(seq 1 "${NUM_ODL_SYSTEM}") + CONTROLLERIP=ODL_SYSTEM_1_IP + echo 'Let'\''s take the karaf thread dump again' Let's take the karaf thread dump again + ssh 10.30.170.104 'sudo ps aux' Warning: Permanently added '10.30.170.104' (ECDSA) to the list of known hosts. ++ grep org.apache.karaf.main.Main /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/ps_after.log ++ grep -v grep ++ tr -s ' ' ++ cut -f2 '-d ' + pid=27921 + echo 'karaf main: org.apache.karaf.main.Main, pid:27921' karaf main: org.apache.karaf.main.Main, pid:27921 + ssh 10.30.170.104 '/usr/lib/jvm/java-21-openjdk-amd64/bin/jstack -l 27921' Warning: Permanently added '10.30.170.104' (ECDSA) to the list of known hosts. + echo 'killing karaf process...' killing karaf process... + ssh 10.30.170.104 bash -c 'ps axf | grep karaf | grep -v grep | awk '\''{print "kill -9 " $1}'\'' | sh' Warning: Permanently added '10.30.170.104' (ECDSA) to the list of known hosts. + for i in $(seq 1 "${NUM_ODL_SYSTEM}") + CONTROLLERIP=ODL_SYSTEM_2_IP + echo 'Let'\''s take the karaf thread dump again' Let's take the karaf thread dump again + ssh 10.30.171.129 'sudo ps aux' Warning: Permanently added '10.30.171.129' (ECDSA) to the list of known hosts. ++ grep org.apache.karaf.main.Main /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/ps_after.log ++ grep -v grep ++ tr -s ' ' ++ cut -f2 '-d ' + pid= + echo 'karaf main: org.apache.karaf.main.Main, pid:' karaf main: org.apache.karaf.main.Main, pid: + ssh 10.30.171.129 '/usr/lib/jvm/java-21-openjdk-amd64/bin/jstack -l ' Warning: Permanently added '10.30.171.129' (ECDSA) to the list of known hosts. Usage: jstack [-l][-e] (to connect to running process) Options: -l long listing. Prints additional information about locks -e extended listing. Prints additional information about threads -? -h --help -help to print this help message + true + echo 'killing karaf process...' killing karaf process... + ssh 10.30.171.129 bash -c 'ps axf | grep karaf | grep -v grep | awk '\''{print "kill -9 " $1}'\'' | sh' Warning: Permanently added '10.30.171.129' (ECDSA) to the list of known hosts. + for i in $(seq 1 "${NUM_ODL_SYSTEM}") + CONTROLLERIP=ODL_SYSTEM_3_IP + echo 'Let'\''s take the karaf thread dump again' Let's take the karaf thread dump again + ssh 10.30.170.156 'sudo ps aux' Warning: Permanently added '10.30.170.156' (ECDSA) to the list of known hosts. ++ grep org.apache.karaf.main.Main /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/ps_after.log ++ tr -s ' ' ++ grep -v grep ++ cut -f2 '-d ' + pid=25549 + echo 'karaf main: org.apache.karaf.main.Main, pid:25549' karaf main: org.apache.karaf.main.Main, pid:25549 + ssh 10.30.170.156 '/usr/lib/jvm/java-21-openjdk-amd64/bin/jstack -l 25549' Warning: Permanently added '10.30.170.156' (ECDSA) to the list of known hosts. + echo 'killing karaf process...' killing karaf process... + ssh 10.30.170.156 bash -c 'ps axf | grep karaf | grep -v grep | awk '\''{print "kill -9 " $1}'\'' | sh' Warning: Permanently added '10.30.170.156' (ECDSA) to the list of known hosts. + sleep 5 ++ seq 1 3 + for i in $(seq 1 "${NUM_ODL_SYSTEM}") + CONTROLLERIP=ODL_SYSTEM_1_IP + echo 'Compressing karaf.log 1' Compressing karaf.log 1 + ssh 10.30.170.104 gzip --best /tmp/karaf-0.24.0-SNAPSHOT/data/log/karaf.log Warning: Permanently added '10.30.170.104' (ECDSA) to the list of known hosts. + echo 'Fetching compressed karaf.log 1' Fetching compressed karaf.log 1 + scp 10.30.170.104:/tmp/karaf-0.24.0-SNAPSHOT/data/log/karaf.log.gz odl1_karaf.log.gz Warning: Permanently added '10.30.170.104' (ECDSA) to the list of known hosts. + ssh 10.30.170.104 rm -f /tmp/karaf-0.24.0-SNAPSHOT/data/log/karaf.log.gz Warning: Permanently added '10.30.170.104' (ECDSA) to the list of known hosts. + scp 10.30.170.104:/tmp/karaf-0.24.0-SNAPSHOT/data/log/karaf_console.log odl1_karaf_console.log Warning: Permanently added '10.30.170.104' (ECDSA) to the list of known hosts. + ssh 10.30.170.104 rm -f /tmp/karaf-0.24.0-SNAPSHOT/data/log/karaf_console.log Warning: Permanently added '10.30.170.104' (ECDSA) to the list of known hosts. + echo 'Fetch GC logs' Fetch GC logs + mkdir -p gclogs-1 + scp '10.30.170.104:/tmp/karaf-0.24.0-SNAPSHOT/data/log/*.log' gclogs-1/ Warning: Permanently added '10.30.170.104' (ECDSA) to the list of known hosts. scp: /tmp/karaf-0.24.0-SNAPSHOT/data/log/*.log: No such file or directory + for i in $(seq 1 "${NUM_ODL_SYSTEM}") + CONTROLLERIP=ODL_SYSTEM_2_IP + echo 'Compressing karaf.log 2' Compressing karaf.log 2 + ssh 10.30.171.129 gzip --best /tmp/karaf-0.24.0-SNAPSHOT/data/log/karaf.log Warning: Permanently added '10.30.171.129' (ECDSA) to the list of known hosts. + echo 'Fetching compressed karaf.log 2' Fetching compressed karaf.log 2 + scp 10.30.171.129:/tmp/karaf-0.24.0-SNAPSHOT/data/log/karaf.log.gz odl2_karaf.log.gz Warning: Permanently added '10.30.171.129' (ECDSA) to the list of known hosts. + ssh 10.30.171.129 rm -f /tmp/karaf-0.24.0-SNAPSHOT/data/log/karaf.log.gz Warning: Permanently added '10.30.171.129' (ECDSA) to the list of known hosts. + scp 10.30.171.129:/tmp/karaf-0.24.0-SNAPSHOT/data/log/karaf_console.log odl2_karaf_console.log Warning: Permanently added '10.30.171.129' (ECDSA) to the list of known hosts. + ssh 10.30.171.129 rm -f /tmp/karaf-0.24.0-SNAPSHOT/data/log/karaf_console.log Warning: Permanently added '10.30.171.129' (ECDSA) to the list of known hosts. + echo 'Fetch GC logs' Fetch GC logs + mkdir -p gclogs-2 + scp '10.30.171.129:/tmp/karaf-0.24.0-SNAPSHOT/data/log/*.log' gclogs-2/ Warning: Permanently added '10.30.171.129' (ECDSA) to the list of known hosts. scp: /tmp/karaf-0.24.0-SNAPSHOT/data/log/*.log: No such file or directory + for i in $(seq 1 "${NUM_ODL_SYSTEM}") + CONTROLLERIP=ODL_SYSTEM_3_IP + echo 'Compressing karaf.log 3' Compressing karaf.log 3 + ssh 10.30.170.156 gzip --best /tmp/karaf-0.24.0-SNAPSHOT/data/log/karaf.log Warning: Permanently added '10.30.170.156' (ECDSA) to the list of known hosts. + echo 'Fetching compressed karaf.log 3' Fetching compressed karaf.log 3 + scp 10.30.170.156:/tmp/karaf-0.24.0-SNAPSHOT/data/log/karaf.log.gz odl3_karaf.log.gz Warning: Permanently added '10.30.170.156' (ECDSA) to the list of known hosts. + ssh 10.30.170.156 rm -f /tmp/karaf-0.24.0-SNAPSHOT/data/log/karaf.log.gz Warning: Permanently added '10.30.170.156' (ECDSA) to the list of known hosts. + scp 10.30.170.156:/tmp/karaf-0.24.0-SNAPSHOT/data/log/karaf_console.log odl3_karaf_console.log Warning: Permanently added '10.30.170.156' (ECDSA) to the list of known hosts. + ssh 10.30.170.156 rm -f /tmp/karaf-0.24.0-SNAPSHOT/data/log/karaf_console.log Warning: Permanently added '10.30.170.156' (ECDSA) to the list of known hosts. + echo 'Fetch GC logs' Fetch GC logs + mkdir -p gclogs-3 + scp '10.30.170.156:/tmp/karaf-0.24.0-SNAPSHOT/data/log/*.log' gclogs-3/ Warning: Permanently added '10.30.170.156' (ECDSA) to the list of known hosts. scp: /tmp/karaf-0.24.0-SNAPSHOT/data/log/*.log: No such file or directory + echo 'Examine copied files' Examine copied files + ls -lt total 80420 drwxrwxr-x. 2 jenkins jenkins 6 Jan 18 02:35 gclogs-3 -rw-rw-r--. 1 jenkins jenkins 1720 Jan 18 02:35 odl3_karaf_console.log -rw-rw-r--. 1 jenkins jenkins 370867 Jan 18 02:34 odl3_karaf.log.gz drwxrwxr-x. 2 jenkins jenkins 6 Jan 18 02:34 gclogs-2 -rw-rw-r--. 1 jenkins jenkins 1720 Jan 18 02:34 odl2_karaf_console.log -rw-rw-r--. 1 jenkins jenkins 5105011 Jan 18 02:34 odl2_karaf.log.gz drwxrwxr-x. 2 jenkins jenkins 6 Jan 18 02:34 gclogs-1 -rw-rw-r--. 1 jenkins jenkins 1720 Jan 18 02:34 odl1_karaf_console.log -rw-rw-r--. 1 jenkins jenkins 677591 Jan 18 02:34 odl1_karaf.log.gz -rw-rw-r--. 1 jenkins jenkins 135288 Jan 18 02:34 karaf_3_25549_threads_after.log -rw-rw-r--. 1 jenkins jenkins 14224 Jan 18 02:34 ps_after.log -rw-rw-r--. 1 jenkins jenkins 0 Jan 18 02:34 karaf_2__threads_after.log -rw-rw-r--. 1 jenkins jenkins 130113 Jan 18 02:34 karaf_1_27921_threads_after.log -rw-rw-r--. 1 jenkins jenkins 304205 Jan 18 02:34 report.html -rw-rw-r--. 1 jenkins jenkins 3532327 Jan 18 02:34 log.html -rw-rw-r--. 1 jenkins jenkins 71599871 Jan 18 02:34 output.xml -rw-rw-r--. 1 jenkins jenkins 4120 Jan 17 23:47 testplan.txt -rw-rw-r--. 1 jenkins jenkins 116978 Jan 17 23:47 karaf_3_2100_threads_before.log -rw-rw-r--. 1 jenkins jenkins 13815 Jan 17 23:47 ps_before.log -rw-rw-r--. 1 jenkins jenkins 121624 Jan 17 23:47 karaf_2_2097_threads_before.log -rw-rw-r--. 1 jenkins jenkins 122670 Jan 17 23:47 karaf_1_2086_threads_before.log -rw-rw-r--. 1 jenkins jenkins 3106 Jan 17 23:43 post-startup-script.sh -rw-rw-r--. 1 jenkins jenkins 252 Jan 17 23:43 startup-script.sh -rw-rw-r--. 1 jenkins jenkins 3471 Jan 17 23:43 configuration-script.sh -rw-rw-r--. 1 jenkins jenkins 695 Jan 17 23:43 custom_shard_config.txt -rw-rw-r--. 1 jenkins jenkins 135 Jan 17 23:43 scriptplan.txt -rw-rw-r--. 1 jenkins jenkins 335 Jan 17 23:43 detect_variables.env -rw-rw-r--. 1 jenkins jenkins 2619 Jan 17 23:43 pom.xml -rw-rw-r--. 1 jenkins jenkins 92 Jan 17 23:43 set_variables.env -rw-rw-r--. 1 jenkins jenkins 314 Jan 17 23:43 slave_addresses.txt -rw-rw-r--. 1 jenkins jenkins 570 Jan 17 23:42 requirements.txt -rw-rw-r--. 1 jenkins jenkins 26 Jan 17 23:42 env.properties -rw-rw-r--. 1 jenkins jenkins 333 Jan 17 23:41 stack-parameters.yaml drwxrwxr-x. 7 jenkins jenkins 4096 Jan 17 23:40 test drwxrwxr-x. 2 jenkins jenkins 6 Jan 17 23:40 test@tmp -rw-rw-r--. 1 jenkins jenkins 1404 Jan 17 22:37 maven-metadata.xml + true [controller-csit-3node-clustering-ask-all-vanadium] $ /bin/sh /tmp/jenkins13118082093499839617.sh Cleaning up Robot installation... $ ssh-agent -k unset SSH_AUTH_SOCK; unset SSH_AGENT_PID; echo Agent pid 5287 killed; [ssh-agent] Stopped. Recording plot data Robot results publisher started... INFO: Checking test criticality is deprecated and will be dropped in a future release! -Parsing output xml: Done! -Copying log files to build dir: Done! -Assigning results to build: Done! -Checking thresholds: Done! Done publishing Robot results. Build step 'Publish Robot Framework test results' changed build result to UNSTABLE [PostBuildScript] - [INFO] Executing post build scripts. [controller-csit-3node-clustering-ask-all-vanadium] $ /bin/bash /tmp/jenkins2108307774376520780.sh Archiving csit artifacts mv: cannot stat '*_1.png': No such file or directory mv: cannot stat '/tmp/odl1_*': No such file or directory mv: cannot stat '*_2.png': No such file or directory mv: cannot stat '/tmp/odl2_*': No such file or directory mv: cannot stat '*_3.png': No such file or directory mv: cannot stat '/tmp/odl3_*': No such file or directory % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 4744k 0 4744k 0 0 5390k 0 --:--:-- --:--:-- --:--:-- 5384k 100 6851k 0 6851k 0 0 5403k 0 --:--:-- 0:00:01 --:--:-- 5407k Archive: robot-plugin.zip inflating: ./archives/robot-plugin/log.html inflating: ./archives/robot-plugin/output.xml inflating: ./archives/robot-plugin/report.html mv: cannot stat '*.log.gz': No such file or directory mv: cannot stat '*.csv': No such file or directory mv: cannot stat '*.png': No such file or directory [PostBuildScript] - [INFO] Executing post build scripts. [controller-csit-3node-clustering-ask-all-vanadium] $ /bin/bash /tmp/jenkins2850964754659801542.sh [PostBuildScript] - [INFO] Executing post build scripts. [EnvInject] - Injecting environment variables from a build step. [EnvInject] - Injecting as environment variables the properties content OS_CLOUD=vex OS_STACK_NAME=releng-controller-csit-3node-clustering-ask-all-vanadium-25 [EnvInject] - Variables injected successfully. provisioning config files... copy managed file [clouds-yaml] to file:/home/jenkins/.config/openstack/clouds.yaml [controller-csit-3node-clustering-ask-all-vanadium] $ /bin/bash /tmp/jenkins17755086738320582077.sh ---> openstack-stack-delete.sh Setup pyenv: system 3.8.13 3.9.13 3.10.13 * 3.11.7 (set by /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/.python-version) lf-activate-venv(): INFO: Reuse venv:/tmp/venv-9I7X from file:/tmp/.os_lf_venv lf-activate-venv(): INFO: Installing base packages (pip, setuptools, virtualenv) lf-activate-venv(): INFO: Attempting to install with network-safe options... lf-activate-venv(): INFO: Base packages installed successfully lf-activate-venv(): INFO: Installing additional packages: lftools[openstack] kubernetes python-heatclient python-openstackclient urllib3~=1.26.15 lf-activate-venv(): INFO: Adding /tmp/venv-9I7X/bin to PATH INFO: Stack cost retrieval disabled, setting cost to 0 INFO: Deleting stack releng-controller-csit-3node-clustering-ask-all-vanadium-25 Successfully deleted stack releng-controller-csit-3node-clustering-ask-all-vanadium-25 [PostBuildScript] - [INFO] Executing post build scripts. [controller-csit-3node-clustering-ask-all-vanadium] $ /bin/bash /tmp/jenkins2449999336259057739.sh ---> sysstat.sh [controller-csit-3node-clustering-ask-all-vanadium] $ /bin/bash /tmp/jenkins1428485476571795995.sh ---> package-listing.sh ++ facter osfamily ++ tr '[:upper:]' '[:lower:]' + OS_FAMILY=redhat + workspace=/w/workspace/controller-csit-3node-clustering-ask-all-vanadium + START_PACKAGES=/tmp/packages_start.txt + END_PACKAGES=/tmp/packages_end.txt + DIFF_PACKAGES=/tmp/packages_diff.txt + PACKAGES=/tmp/packages_start.txt + '[' /w/workspace/controller-csit-3node-clustering-ask-all-vanadium ']' + PACKAGES=/tmp/packages_end.txt + case "${OS_FAMILY}" in + rpm -qa + sort + '[' -f /tmp/packages_start.txt ']' + '[' -f /tmp/packages_end.txt ']' + diff /tmp/packages_start.txt /tmp/packages_end.txt + '[' /w/workspace/controller-csit-3node-clustering-ask-all-vanadium ']' + mkdir -p /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/archives/ + cp -f /tmp/packages_diff.txt /tmp/packages_end.txt /tmp/packages_start.txt /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/archives/ [controller-csit-3node-clustering-ask-all-vanadium] $ /bin/bash /tmp/jenkins11164214441812124924.sh ---> capture-instance-metadata.sh Setup pyenv: system 3.8.13 3.9.13 3.10.13 * 3.11.7 (set by /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/.python-version) lf-activate-venv(): INFO: Reuse venv:/tmp/venv-9I7X from file:/tmp/.os_lf_venv lf-activate-venv(): INFO: Installing base packages (pip, setuptools, virtualenv) lf-activate-venv(): INFO: Attempting to install with network-safe options... lf-activate-venv(): INFO: Base packages installed successfully lf-activate-venv(): INFO: Installing additional packages: lftools lf-activate-venv(): INFO: Adding /tmp/venv-9I7X/bin to PATH INFO: Running in OpenStack, capturing instance metadata [controller-csit-3node-clustering-ask-all-vanadium] $ /bin/bash /tmp/jenkins13484286286574912248.sh provisioning config files... Could not find credentials [logs] for controller-csit-3node-clustering-ask-all-vanadium #25 copy managed file [jenkins-log-archives-settings] to file:/w/workspace/controller-csit-3node-clustering-ask-all-vanadium@tmp/config10076306425634253321tmp Regular expression run condition: Expression=[^.*logs-s3.*], Label=[odl-logs-s3-cloudfront-index] Run condition [Regular expression match] enabling perform for step [Provide Configuration files] provisioning config files... copy managed file [jenkins-s3-log-ship] to file:/home/jenkins/.aws/credentials [EnvInject] - Injecting environment variables from a build step. [EnvInject] - Injecting as environment variables the properties content SERVER_ID=logs [EnvInject] - Variables injected successfully. [controller-csit-3node-clustering-ask-all-vanadium] $ /bin/bash /tmp/jenkins15568660732994033448.sh ---> create-netrc.sh WARN: Log server credential not found. [controller-csit-3node-clustering-ask-all-vanadium] $ /bin/bash /tmp/jenkins3115751165673638790.sh ---> python-tools-install.sh Setup pyenv: system 3.8.13 3.9.13 3.10.13 * 3.11.7 (set by /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/.python-version) lf-activate-venv(): INFO: Reuse venv:/tmp/venv-9I7X from file:/tmp/.os_lf_venv lf-activate-venv(): INFO: Installing base packages (pip, setuptools, virtualenv) lf-activate-venv(): INFO: Attempting to install with network-safe options... lf-activate-venv(): INFO: Base packages installed successfully lf-activate-venv(): INFO: Installing additional packages: lftools lf-activate-venv(): INFO: Adding /tmp/venv-9I7X/bin to PATH [controller-csit-3node-clustering-ask-all-vanadium] $ /bin/bash /tmp/jenkins10918069677402353003.sh ---> sudo-logs.sh Archiving 'sudo' log.. [controller-csit-3node-clustering-ask-all-vanadium] $ /bin/bash /tmp/jenkins2718933316172943091.sh ---> job-cost.sh Setup pyenv: system 3.8.13 3.9.13 3.10.13 * 3.11.7 (set by /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/.python-version) lf-activate-venv(): INFO: Reuse venv:/tmp/venv-9I7X from file:/tmp/.os_lf_venv lf-activate-venv(): INFO: Installing base packages (pip, setuptools, virtualenv) lf-activate-venv(): INFO: Attempting to install with network-safe options... lf-activate-venv(): INFO: Base packages installed successfully lf-activate-venv(): INFO: Installing additional packages: zipp==1.1.0 python-openstackclient urllib3~=1.26.15 lf-activate-venv(): INFO: Adding /tmp/venv-9I7X/bin to PATH DEBUG: total: 0 INFO: Retrieving Stack Cost... INFO: Retrieving Pricing Info for: v3-standard-2 INFO: Archiving Costs [controller-csit-3node-clustering-ask-all-vanadium] $ /bin/bash -l /tmp/jenkins12026973016740913418.sh ---> logs-deploy.sh Setup pyenv: system 3.8.13 3.9.13 3.10.13 * 3.11.7 (set by /w/workspace/controller-csit-3node-clustering-ask-all-vanadium/.python-version) lf-activate-venv(): INFO: Reuse venv:/tmp/venv-9I7X from file:/tmp/.os_lf_venv lf-activate-venv(): INFO: Installing base packages (pip, setuptools, virtualenv) lf-activate-venv(): INFO: Attempting to install with network-safe options... lf-activate-venv(): INFO: Base packages installed successfully lf-activate-venv(): INFO: Installing additional packages: lftools urllib3~=1.26.15 lf-activate-venv(): INFO: Adding /tmp/venv-9I7X/bin to PATH WARNING: Nexus logging server not set INFO: S3 path logs/releng/vex-yul-odl-jenkins-1/controller-csit-3node-clustering-ask-all-vanadium/25/ INFO: archiving logs to S3 ---> uname -a: Linux prd-centos8-robot-2c-8g-1588.novalocal 4.18.0-553.5.1.el8.x86_64 #1 SMP Tue May 21 05:46:01 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux ---> lscpu: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 2 On-line CPU(s) list: 0,1 Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 2 NUMA node(s): 1 Vendor ID: AuthenticAMD CPU family: 23 Model: 49 Model name: AMD EPYC-Rome Processor Stepping: 0 CPU MHz: 2799.998 BogoMIPS: 5599.99 Virtualization: AMD-V Hypervisor vendor: KVM Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 512K L3 cache: 16384K NUMA node0 CPU(s): 0,1 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd ibrs ibpb stibp vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr wbnoinvd arat npt nrip_save umip rdpid arch_capabilities ---> nproc: 2 ---> df -h: Filesystem Size Used Avail Use% Mounted on devtmpfs 3.8G 0 3.8G 0% /dev tmpfs 3.8G 0 3.8G 0% /dev/shm tmpfs 3.8G 17M 3.8G 1% /run tmpfs 3.8G 0 3.8G 0% /sys/fs/cgroup /dev/vda1 40G 8.5G 32G 22% / tmpfs 770M 0 770M 0% /run/user/1001 ---> free -m: total used free shared buff/cache available Mem: 7697 644 4697 19 2355 6754 Swap: 1023 0 1023 ---> ip addr: 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: mtu 1458 qdisc mq state UP group default qlen 1000 link/ether fa:16:3e:a4:1d:07 brd ff:ff:ff:ff:ff:ff altname enp0s3 altname ens3 inet 10.30.171.237/23 brd 10.30.171.255 scope global dynamic noprefixroute eth0 valid_lft 75773sec preferred_lft 75773sec inet6 fe80::f816:3eff:fea4:1d07/64 scope link valid_lft forever preferred_lft forever ---> sar -b -r -n DEV: Linux 4.18.0-553.5.1.el8.x86_64 (prd-centos8-robot-2c-8g-1588.novalocal) 01/18/2026 _x86_64_ (2 CPU) 12:00:01 AM tps rtps wtps bread/s bwrtn/s 12:01:01 AM 1.03 0.03 1.00 0.27 213.24 12:02:01 AM 0.47 0.05 0.42 1.60 19.69 12:03:01 AM 0.27 0.00 0.27 0.00 18.73 12:04:01 AM 0.23 0.00 0.23 0.00 22.11 12:05:01 AM 0.35 0.00 0.35 0.00 19.18 12:06:01 AM 0.42 0.00 0.42 0.00 20.46 12:07:01 AM 0.67 0.32 0.35 6.93 146.53 12:08:01 AM 1.02 0.07 0.95 7.60 139.65 12:09:01 AM 0.32 0.00 0.32 0.00 19.73 12:10:02 AM 0.27 0.00 0.27 0.00 18.80 12:11:01 AM 0.30 0.00 0.30 0.00 11.96 12:12:01 AM 0.22 0.00 0.22 0.00 28.57 12:13:01 AM 0.17 0.00 0.17 0.00 6.38 12:14:01 AM 0.18 0.00 0.18 0.00 4.28 12:15:01 AM 0.18 0.00 0.18 0.00 5.78 12:16:01 AM 0.33 0.00 0.33 0.00 7.11 12:17:01 AM 0.23 0.00 0.23 0.00 6.30 12:18:01 AM 0.17 0.00 0.17 0.00 6.81 12:19:01 AM 0.18 0.00 0.18 0.00 46.85 12:20:01 AM 0.40 0.00 0.40 0.00 19.15 12:21:01 AM 0.70 0.00 0.70 0.00 21.43 12:22:01 AM 0.48 0.00 0.48 0.00 15.98 12:23:01 AM 0.47 0.00 0.47 0.00 12.80 12:24:01 AM 0.13 0.00 0.13 0.00 16.01 12:25:01 AM 0.20 0.00 0.20 0.00 25.01 12:26:01 AM 0.22 0.00 0.22 0.00 9.90 12:27:01 AM 0.17 0.00 0.17 0.00 14.05 12:28:01 AM 0.43 0.00 0.43 0.00 15.49 12:29:01 AM 0.10 0.00 0.10 0.00 5.46 12:30:01 AM 0.17 0.00 0.17 0.00 13.91 12:31:01 AM 0.43 0.00 0.43 0.00 17.94 12:32:01 AM 0.50 0.00 0.50 0.00 12.28 12:33:01 AM 0.12 0.00 0.12 0.00 5.75 12:34:01 AM 0.18 0.00 0.18 0.00 27.75 12:35:01 AM 0.17 0.00 0.17 0.00 1.63 12:36:01 AM 0.20 0.03 0.17 1.33 3.25 12:37:01 AM 0.22 0.00 0.22 0.00 93.07 12:38:01 AM 0.30 0.00 0.30 0.00 36.88 12:39:01 AM 0.13 0.00 0.13 0.00 33.87 12:40:01 AM 0.20 0.00 0.20 0.00 8.73 12:41:01 AM 0.27 0.00 0.27 0.00 10.08 12:42:01 AM 0.27 0.00 0.27 0.00 18.81 12:43:01 AM 0.17 0.00 0.17 0.00 16.96 12:44:01 AM 0.55 0.00 0.55 0.00 17.76 12:45:01 AM 0.40 0.02 0.38 0.13 17.74 12:46:01 AM 8.68 5.73 2.95 555.41 77.74 12:47:01 AM 0.30 0.00 0.30 0.00 31.17 12:48:01 AM 0.37 0.00 0.37 0.00 10.15 12:49:01 AM 0.32 0.00 0.32 0.00 14.76 12:50:01 AM 0.17 0.00 0.17 0.00 14.31 12:51:01 AM 0.35 0.00 0.35 0.00 11.65 12:52:01 AM 0.23 0.00 0.23 0.00 11.35 12:53:01 AM 0.18 0.00 0.18 0.00 20.38 12:54:01 AM 0.13 0.00 0.13 0.00 4.95 12:55:01 AM 0.20 0.00 0.20 0.00 13.51 12:56:01 AM 0.35 0.00 0.35 0.00 16.43 12:57:01 AM 0.20 0.00 0.20 0.00 13.98 12:58:01 AM 0.15 0.00 0.15 0.00 11.60 12:59:01 AM 0.22 0.00 0.22 0.00 20.93 01:00:01 AM 0.72 0.02 0.70 0.13 21.58 01:01:01 AM 0.72 0.02 0.70 0.13 21.09 01:02:01 AM 0.53 0.00 0.53 0.00 17.84 01:03:01 AM 0.33 0.00 0.33 0.00 24.91 01:04:01 AM 0.48 0.00 0.48 0.00 13.58 01:05:01 AM 0.40 0.00 0.40 0.00 15.70 01:06:01 AM 0.22 0.00 0.22 0.00 11.06 01:07:01 AM 0.20 0.00 0.20 0.00 11.20 01:08:01 AM 0.15 0.00 0.15 0.00 8.15 01:09:01 AM 0.17 0.00 0.17 0.00 15.20 01:10:01 AM 0.28 0.00 0.28 0.00 6.61 01:11:01 AM 0.28 0.00 0.28 0.00 10.71 01:12:01 AM 0.23 0.00 0.23 0.00 5.27 01:13:01 AM 0.17 0.00 0.17 0.00 5.35 01:14:01 AM 0.18 0.00 0.18 0.00 8.80 01:15:01 AM 0.35 0.00 0.35 0.00 8.63 01:16:02 AM 0.23 0.00 0.23 0.00 11.28 01:17:01 AM 0.14 0.00 0.14 0.00 16.68 01:18:01 AM 0.15 0.00 0.15 0.00 16.26 01:19:01 AM 0.12 0.00 0.12 0.00 8.66 01:20:01 AM 0.32 0.00 0.32 0.00 20.21 01:21:01 AM 0.25 0.00 0.25 0.00 18.04 01:22:01 AM 0.32 0.00 0.32 0.00 22.99 01:23:01 AM 0.18 0.00 0.18 0.00 11.50 01:24:01 AM 0.18 0.00 0.18 0.00 8.53 01:25:01 AM 0.20 0.00 0.20 0.00 15.09 01:26:01 AM 0.23 0.00 0.23 0.00 10.33 01:27:01 AM 0.20 0.00 0.20 0.00 14.68 01:28:01 AM 0.32 0.00 0.32 0.00 16.94 01:29:01 AM 0.40 0.00 0.40 0.00 6.02 01:30:01 AM 0.27 0.00 0.27 0.00 8.72 01:31:01 AM 0.23 0.00 0.23 0.00 4.18 01:32:01 AM 0.18 0.00 0.18 0.00 10.85 01:33:01 AM 0.22 0.00 0.22 0.00 5.25 01:34:01 AM 0.42 0.00 0.42 0.00 22.29 01:35:01 AM 0.68 0.00 0.68 0.00 19.63 01:36:01 AM 0.60 0.00 0.60 0.00 20.14 01:37:01 AM 0.48 0.00 0.48 0.00 16.65 01:38:01 AM 0.32 0.00 0.32 0.00 11.31 01:39:01 AM 0.15 0.00 0.15 0.00 16.93 01:40:01 AM 0.17 0.00 0.17 0.00 22.71 01:41:01 AM 0.30 0.00 0.30 0.00 13.36 01:42:01 AM 0.37 0.00 0.37 0.00 14.66 01:43:01 AM 0.50 0.00 0.50 0.00 14.60 01:44:01 AM 0.37 0.00 0.37 0.00 11.58 01:45:01 AM 0.47 0.00 0.47 0.00 14.10 01:46:01 AM 0.52 0.00 0.52 0.00 17.91 01:47:01 AM 0.40 0.00 0.40 0.00 6.25 01:48:01 AM 0.55 0.00 0.55 0.00 16.60 01:49:01 AM 0.93 0.00 0.93 0.00 25.63 01:50:01 AM 0.38 0.00 0.38 0.00 13.85 01:51:01 AM 0.28 0.00 0.28 0.00 18.86 01:52:01 AM 0.38 0.00 0.38 0.00 19.73 01:53:01 AM 0.52 0.00 0.52 0.00 17.91 01:54:01 AM 0.37 0.00 0.37 0.00 26.12 01:55:01 AM 0.25 0.00 0.25 0.00 8.97 01:56:01 AM 0.33 0.00 0.33 0.00 17.51 01:57:01 AM 0.13 0.00 0.13 0.00 7.38 01:58:01 AM 0.13 0.00 0.13 0.00 7.35 01:59:01 AM 0.18 0.00 0.18 0.00 15.74 02:00:01 AM 0.23 0.00 0.23 0.00 13.51 02:01:01 AM 0.25 0.00 0.25 0.00 11.00 02:02:01 AM 0.57 0.00 0.57 0.00 9.06 02:03:01 AM 0.25 0.00 0.25 0.00 15.49 02:04:01 AM 0.23 0.00 0.23 0.00 15.98 02:05:01 AM 0.25 0.00 0.25 0.00 13.40 02:06:01 AM 0.25 0.00 0.25 0.00 18.58 02:07:01 AM 0.17 0.00 0.17 0.00 8.68 02:08:01 AM 0.13 0.00 0.13 0.00 16.53 02:09:01 AM 0.25 0.00 0.25 0.00 24.65 02:10:01 AM 0.37 0.00 0.37 0.00 12.76 02:11:01 AM 0.27 0.00 0.27 0.00 16.11 02:12:01 AM 0.20 0.00 0.20 0.00 13.08 02:13:01 AM 0.17 0.00 0.17 0.00 9.98 02:14:01 AM 0.13 0.00 0.13 0.00 7.23 02:15:01 AM 0.32 0.00 0.32 0.00 19.68 02:16:01 AM 0.23 0.00 0.23 0.00 7.82 02:17:01 AM 0.27 0.00 0.27 0.00 4.18 02:18:01 AM 0.13 0.00 0.13 0.00 12.80 02:19:01 AM 0.15 0.00 0.15 0.00 15.86 02:20:01 AM 0.28 0.00 0.28 0.00 11.96 02:21:01 AM 0.45 0.00 0.45 0.00 21.86 02:22:01 AM 0.52 0.00 0.52 0.00 17.98 02:23:01 AM 0.62 0.00 0.62 0.00 17.76 02:24:01 AM 0.37 0.00 0.37 0.00 25.05 02:25:02 AM 0.28 0.00 0.28 0.00 11.30 02:26:01 AM 0.15 0.00 0.15 0.00 10.05 02:27:01 AM 0.18 0.00 0.18 0.00 14.28 02:28:01 AM 0.18 0.00 0.18 0.00 14.33 02:29:01 AM 0.17 0.00 0.17 0.00 8.00 02:30:01 AM 0.30 0.00 0.30 0.00 12.80 02:31:01 AM 0.28 0.00 0.28 0.00 23.88 02:32:01 AM 0.23 0.00 0.23 0.00 13.78 02:33:01 AM 0.20 0.00 0.20 0.00 34.06 02:34:01 AM 0.18 0.00 0.18 0.00 14.13 02:35:01 AM 1.77 0.00 1.77 0.00 109.51 02:36:01 AM 30.18 0.50 29.68 54.52 4238.19 Average: 0.56 0.04 0.52 4.03 46.43 12:00:01 AM kbmemfree kbavail kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty 12:01:01 AM 4850168 6970804 3032256 38.47 2688 2288080 751236 8.41 268592 2377980 4 12:02:01 AM 4850832 6972024 3031592 38.46 2688 2288636 751236 8.41 268736 2377536 96 12:03:01 AM 4850468 6972152 3031956 38.46 2688 2289128 751236 8.41 268736 2377960 100 12:04:01 AM 4849708 6971876 3032716 38.47 2688 2289612 751236 8.41 268736 2378600 60 12:05:01 AM 4849616 6972272 3032808 38.48 2688 2290100 751236 8.41 268740 2378940 88 12:06:01 AM 4842032 6967484 3040392 38.57 2688 2292904 751236 8.41 268948 2385896 2460 12:07:01 AM 4835712 6965900 3046712 38.65 2688 2297632 770168 8.62 269048 2391904 2688 12:08:01 AM 4832488 6963932 3049936 38.69 2688 2298896 748624 8.38 269096 2395500 168 12:09:01 AM 4832476 6964296 3049948 38.69 2688 2299272 748624 8.38 269096 2395604 88 12:10:02 AM 4831796 6964000 3050628 38.70 2688 2299648 767492 8.59 269096 2395988 4 12:11:01 AM 4833544 6966120 3048880 38.68 2688 2300028 767492 8.59 269096 2394352 172 12:12:01 AM 4832988 6966224 3049436 38.69 2688 2300680 767492 8.59 269096 2394940 40 12:13:01 AM 4832908 6966240 3049516 38.69 2688 2300776 767492 8.59 269096 2395036 4 12:14:01 AM 4832872 6966288 3049552 38.69 2688 2300868 767492 8.59 269096 2395120 36 12:15:01 AM 4832856 6966372 3049568 38.69 2688 2300960 752028 8.42 269096 2395220 4 12:16:01 AM 4832480 6966092 3049944 38.69 2688 2301056 752028 8.42 269096 2395388 40 12:17:01 AM 4832600 6966304 3049824 38.69 2688 2301148 752028 8.42 269096 2395412 4 12:18:01 AM 4831200 6966076 3051224 38.71 2688 2302328 773384 8.66 269120 2396748 1040 12:19:01 AM 4830652 6965924 3051772 38.72 2688 2302724 773384 8.66 269120 2397192 112 12:20:01 AM 4830040 6965704 3052384 38.72 2688 2303116 767508 8.59 269120 2397724 24 12:21:01 AM 4829900 6965952 3052524 38.73 2688 2303496 767508 8.59 269120 2398052 4 12:22:01 AM 4829428 6965848 3052996 38.73 2688 2303872 767508 8.59 269120 2398352 40 12:23:01 AM 4829128 6965924 3053296 38.74 2688 2304248 767508 8.59 269120 2398728 188 12:24:01 AM 4829008 6966184 3053416 38.74 2688 2304628 767508 8.59 269120 2399024 116 12:25:01 AM 4827932 6965700 3054492 38.75 2688 2305212 767508 8.59 269120 2399964 16 12:26:01 AM 4827756 6965812 3054668 38.75 2688 2305500 767508 8.59 269120 2400136 104 12:27:01 AM 4827188 6965536 3055236 38.76 2688 2305792 767508 8.59 269120 2400576 16 12:28:01 AM 4827452 6966084 3054972 38.76 2688 2306076 767508 8.59 269148 2400492 4 12:29:01 AM 4827228 6966140 3055196 38.76 2688 2306364 767508 8.59 269148 2400772 152 12:30:01 AM 4826732 6965928 3055692 38.77 2688 2306648 764616 8.56 269148 2401172 56 12:31:01 AM 4826452 6965964 3055972 38.77 2688 2306956 780636 8.74 269360 2401332 4 12:32:01 AM 4826208 6965912 3056216 38.77 2688 2307148 764944 8.57 269360 2401436 4 12:33:01 AM 4824852 6965424 3057572 38.79 2688 2308016 764944 8.57 269360 2402524 740 12:34:01 AM 4824928 6965556 3057496 38.79 2688 2308072 764944 8.57 269360 2402608 4 12:35:01 AM 4825212 6965840 3057212 38.79 2688 2308072 764944 8.57 269360 2402452 4 12:36:01 AM 4824492 6965528 3057932 38.79 2688 2308480 764944 8.57 269360 2403272 372 12:37:01 AM 4821504 6965836 3060920 38.83 2688 2311776 764944 8.57 269360 2406244 940 12:38:01 AM 4821244 6965640 3061180 38.84 2688 2311840 764944 8.57 269360 2406520 24 12:39:01 AM 4819924 6965476 3062500 38.85 2688 2313004 764944 8.57 269360 2407456 204 12:40:01 AM 4819636 6965244 3062788 38.86 2688 2313052 747636 8.37 269360 2407748 32 12:41:01 AM 4819628 6965592 3062796 38.86 2688 2313408 747636 8.37 269360 2408120 204 12:42:01 AM 4818844 6965176 3063580 38.87 2688 2313784 747636 8.37 269360 2408544 112 12:43:01 AM 4818848 6965572 3063576 38.87 2688 2314176 747636 8.37 269360 2408776 44 12:44:01 AM 4818292 6965396 3064132 38.87 2688 2314556 747636 8.37 269360 2409304 40 12:45:01 AM 4817024 6964512 3065400 38.89 2688 2314932 771128 8.63 269360 2410452 52 12:46:01 AM 4800532 6965096 3081892 39.10 2688 2331964 761356 8.52 271664 2424528 204 12:47:01 AM 4800000 6965220 3082424 39.11 2688 2332608 761356 8.52 271664 2425112 40 12:48:01 AM 4799856 6965364 3082568 39.11 2688 2332900 761356 8.52 271660 2425340 200 12:49:01 AM 4800152 6965952 3082272 39.10 2688 2333192 761356 8.52 271660 2425272 156 12:50:01 AM 4799492 6965572 3082932 39.11 2688 2333480 758036 8.49 271660 2425708 56 12:51:01 AM 4799372 6965740 3083052 39.11 2688 2333776 758036 8.49 271664 2426148 160 12:52:01 AM 4799304 6965928 3083120 39.11 2688 2334016 758036 8.49 271664 2426336 148 12:53:01 AM 4798576 6965644 3083848 39.12 2688 2334460 758036 8.49 271664 2426668 44 12:54:01 AM 4798488 6965788 3083936 39.12 2688 2334692 758036 8.49 271672 2426808 168 12:55:01 AM 4798292 6965780 3084132 39.13 2688 2334880 758036 8.49 271672 2426996 4 12:56:01 AM 4797788 6965676 3084636 39.13 2688 2335280 758036 8.49 271676 2427576 72 12:57:01 AM 4797524 6965808 3084900 39.14 2688 2335676 758036 8.49 271676 2427796 128 12:58:01 AM 4796776 6965432 3085648 39.15 2688 2336056 758036 8.49 271676 2428504 216 12:59:01 AM 4796432 6965480 3085992 39.15 2688 2336448 758036 8.49 271676 2428772 32 01:00:01 AM 4795160 6964584 3087264 39.17 2688 2336824 759016 8.50 271676 2429760 24 01:01:01 AM 4794852 6964672 3087572 39.17 2688 2337212 759016 8.50 271680 2430224 36 01:02:01 AM 4795368 6965568 3087056 39.16 2688 2337588 759016 8.50 271712 2429688 36 01:03:01 AM 4794880 6965708 3087544 39.17 2688 2338208 759016 8.50 271712 2430316 4 01:04:01 AM 4794416 6965524 3088008 39.18 2688 2338496 759016 8.50 271712 2430800 56 01:05:01 AM 4793692 6965088 3088732 39.19 2688 2338784 759016 8.50 271712 2431020 56 01:06:01 AM 4793672 6965312 3088752 39.19 2688 2339020 759016 8.50 271712 2431196 4 01:07:01 AM 4793664 6965596 3088760 39.19 2688 2339312 759016 8.50 271712 2431420 4 01:08:01 AM 4792632 6964848 3089792 39.20 2688 2339596 759016 8.50 271712 2431976 100 01:09:01 AM 4793204 6965764 3089220 39.19 2688 2339940 774896 8.68 271712 2432064 24 01:10:01 AM 4792820 6965560 3089604 39.20 2688 2340120 764192 8.56 271712 2432476 152 01:11:01 AM 4792672 6965512 3089752 39.20 2688 2340220 764192 8.56 271728 2432484 24 01:12:01 AM 4792536 6965440 3089888 39.20 2688 2340284 764192 8.56 271728 2432480 20 01:13:01 AM 4792572 6965588 3089852 39.20 2688 2340396 764192 8.56 271728 2432592 4 01:14:01 AM 4792184 6965424 3090240 39.20 2688 2340620 764192 8.56 271728 2432812 4 01:15:01 AM 4792224 6965548 3090200 39.20 2688 2340704 764192 8.56 271728 2432836 8 01:16:02 AM 4791944 6965688 3090480 39.21 2688 2341124 764192 8.56 271728 2433368 160 01:17:01 AM 4791328 6965448 3091096 39.22 2688 2341500 764192 8.56 271728 2434180 84 01:18:01 AM 4790644 6965136 3091780 39.22 2688 2341880 764192 8.56 271728 2434584 16 01:19:01 AM 4790588 6965456 3091836 39.22 2688 2342256 764192 8.56 271728 2434620 164 01:20:01 AM 4789956 6965200 3092468 39.23 2688 2342632 762404 8.54 271728 2435344 84 01:21:01 AM 4789568 6965216 3092856 39.24 2688 2343032 762404 8.54 271728 2435580 24 01:22:01 AM 4788840 6965076 3093584 39.25 2688 2343612 762404 8.54 271728 2436316 68 01:23:01 AM 4789068 6965596 3093356 39.24 2688 2343904 762404 8.54 271728 2436288 56 01:24:01 AM 4788820 6965636 3093604 39.25 2688 2344192 762404 8.54 271728 2436576 152 01:25:01 AM 4788444 6965540 3093980 39.25 2688 2344480 762404 8.54 271728 2436856 56 01:26:01 AM 4787992 6965380 3094432 39.26 2688 2344772 762404 8.54 271728 2437216 152 01:27:01 AM 4787736 6965420 3094688 39.26 2688 2345060 762404 8.54 271728 2437744 68 01:28:01 AM 4787244 6965256 3095180 39.27 2688 2345392 785036 8.79 271728 2438004 4 01:29:01 AM 4787252 6965340 3095172 39.27 2688 2345468 763776 8.55 271728 2437872 4 01:30:01 AM 4786948 6965192 3095476 39.27 2688 2345624 764352 8.56 271732 2438252 4 01:31:01 AM 4787140 6965432 3095284 39.27 2688 2345668 764352 8.56 271732 2438136 48 01:32:01 AM 4786980 6965496 3095444 39.27 2688 2345892 764352 8.56 271736 2438288 4 01:33:01 AM 4786400 6965180 3096024 39.28 2688 2346164 764352 8.56 271736 2438752 216 01:34:01 AM 4786292 6965456 3096132 39.28 2688 2346544 764352 8.56 271736 2438928 24 01:35:01 AM 4785980 6965520 3096444 39.28 2688 2346920 764352 8.56 271736 2439304 24 01:36:01 AM 4785312 6965228 3097112 39.29 2688 2347296 764352 8.56 271736 2439748 36 01:37:01 AM 4785132 6965428 3097292 39.29 2688 2347676 764352 8.56 271736 2440180 24 01:38:01 AM 4784408 6965096 3098016 39.30 2688 2348068 764352 8.56 271736 2440672 188 01:39:01 AM 4783844 6965248 3098580 39.31 2688 2348784 764352 8.56 271736 2441308 428 01:40:01 AM 4783280 6964908 3099144 39.32 2688 2349008 759172 8.50 271736 2441764 4 01:41:01 AM 4783264 6965180 3099160 39.32 2688 2349296 759172 8.50 271736 2441996 56 01:42:01 AM 4782940 6965144 3099484 39.32 2688 2349584 759172 8.50 271736 2442216 4 01:43:01 AM 4782536 6965028 3099888 39.33 2688 2349872 759172 8.50 271736 2442504 56 01:44:01 AM 4782440 6965220 3099984 39.33 2688 2350160 759172 8.50 271736 2442688 56 01:45:01 AM 4782132 6965228 3100292 39.33 2688 2350468 759172 8.50 271736 2443012 76 01:46:01 AM 4781544 6964924 3100880 39.34 2688 2350760 757128 8.48 271736 2443516 32 01:47:01 AM 4781840 6965284 3100584 39.34 2688 2350808 757128 8.48 271736 2443372 4 01:48:01 AM 4781232 6965040 3101192 39.34 2688 2351176 757128 8.48 271736 2443924 36 01:49:01 AM 4780936 6965128 3101488 39.35 2688 2351560 779544 8.73 280772 2435192 52 01:50:01 AM 4779964 6964544 3102460 39.36 2688 2351936 746996 8.36 280776 2435892 204 01:51:01 AM 4780088 6965040 3102336 39.36 2688 2352316 746996 8.36 280772 2435964 128 01:52:01 AM 4779772 6965108 3102652 39.36 2688 2352692 746996 8.36 280772 2436196 24 01:53:01 AM 4779408 6965120 3103016 39.37 2688 2353068 746996 8.36 280772 2436592 4 01:54:01 AM 4778552 6964904 3103872 39.38 2688 2353716 762396 8.54 280772 2437640 4 01:55:01 AM 4778420 6965056 3104004 39.38 2688 2354000 762396 8.54 280772 2437664 108 01:56:01 AM 4778104 6965036 3104320 39.38 2688 2354288 762396 8.54 280772 2437912 4 01:57:01 AM 4777948 6965168 3104476 39.38 2688 2354576 762396 8.54 280772 2438212 104 01:58:01 AM 4777636 6965136 3104788 39.39 2688 2354864 762396 8.54 280772 2438412 200 01:59:01 AM 4777460 6965248 3104964 39.39 2688 2355152 762396 8.54 280772 2438704 56 02:00:01 AM 4776648 6964812 3105776 39.40 2688 2355520 756408 8.47 280772 2439336 96 02:01:01 AM 4775556 6963856 3106868 39.42 2688 2355656 756408 8.47 280772 2439996 20 02:02:01 AM 4776508 6964924 3105916 39.40 2688 2355780 778460 8.72 280776 2439468 80 02:03:01 AM 4776360 6965180 3106064 39.40 2688 2356172 778460 8.72 280776 2439748 84 02:04:01 AM 4775616 6964820 3106808 39.41 2688 2356564 778460 8.72 280776 2440324 84 02:05:01 AM 4775520 6965104 3106904 39.42 2688 2356944 778460 8.72 280776 2440620 144 02:06:01 AM 4775228 6965192 3107196 39.42 2688 2357324 778460 8.72 280776 2440952 68 02:07:01 AM 4774932 6965272 3107492 39.42 2688 2357700 778460 8.72 280776 2441260 220 02:08:01 AM 4774224 6964944 3108200 39.43 2688 2358080 762900 8.54 280776 2441904 152 02:09:01 AM 4773852 6965188 3108572 39.44 2688 2358696 762900 8.54 280776 2442268 108 02:10:01 AM 4773272 6964892 3109152 39.44 2688 2358980 756344 8.47 280776 2442900 196 02:11:01 AM 4773164 6965084 3109260 39.45 2688 2359276 756344 8.47 280776 2442992 104 02:12:01 AM 4772848 6965016 3109576 39.45 2688 2359516 756344 8.47 280776 2443288 36 02:13:01 AM 4772852 6965308 3109572 39.45 2688 2359804 756344 8.47 280776 2443384 56 02:14:01 AM 4772436 6965180 3109988 39.45 2688 2360092 756344 8.47 280776 2443788 152 02:15:01 AM 4772128 6965168 3110296 39.46 2688 2360388 756344 8.47 280776 2443980 24 02:16:01 AM 4771844 6965012 3110580 39.46 2688 2360532 756344 8.47 280776 2444172 24 02:17:01 AM 4771612 6964844 3110812 39.47 2688 2360580 756344 8.47 280776 2444172 32 02:18:01 AM 4771260 6964872 3111164 39.47 2688 2360964 756344 8.47 280776 2444812 84 02:19:01 AM 4770872 6964852 3111552 39.47 2688 2361328 756344 8.47 280776 2445064 4 02:20:01 AM 4770348 6964704 3112076 39.48 2688 2361704 757876 8.49 280776 2445536 160 02:21:01 AM 4770196 6964928 3112228 39.48 2688 2362080 757876 8.49 280776 2445852 24 02:22:01 AM 4770120 6965232 3112304 39.48 2688 2362460 757876 8.49 280776 2446060 4 02:23:01 AM 4769576 6965064 3112848 39.49 2688 2362836 757876 8.49 280776 2446436 40 02:24:01 AM 4768928 6965072 3113496 39.50 2688 2363492 757876 8.49 280776 2447096 52 02:25:02 AM 4768632 6965068 3113792 39.50 2688 2363784 757876 8.49 280776 2447388 152 02:26:01 AM 4768260 6964992 3114164 39.51 2688 2364072 757876 8.49 280776 2447868 200 02:27:01 AM 4768044 6965068 3114380 39.51 2688 2364364 757876 8.49 280776 2447976 104 02:28:01 AM 4767920 6965240 3114504 39.51 2688 2364660 757876 8.49 280776 2448272 4 02:29:01 AM 4767544 6965140 3114880 39.52 2688 2364944 757876 8.49 280776 2448548 100 02:30:01 AM 4766816 6964820 3115608 39.53 2688 2365344 756360 8.47 280776 2449196 264 02:31:01 AM 4766116 6964744 3116308 39.53 2688 2365968 756360 8.47 280776 2449788 284 02:32:01 AM 4765720 6964732 3116704 39.54 2688 2366348 756360 8.47 280776 2450120 336 02:33:01 AM 4764196 6963924 3118228 39.56 2688 2367064 790140 8.85 280776 2451520 88 02:34:01 AM 4761644 6961988 3120780 39.59 2688 2367680 790140 8.85 280776 2453964 336 02:35:01 AM 5005864 7030540 2876560 36.49 2688 2193460 670304 7.51 275760 2216460 10164 02:36:01 AM 4807352 6909192 3075072 39.01 2688 2275100 749868 8.40 548868 2167036 11832 Average: 4798441 6965591 3083983 39.12 2688 2334589 761020 8.52 275447 2423191 270 12:00:01 AM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil 12:01:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:01:01 AM eth0 2.32 1.35 0.81 0.44 0.00 0.00 0.00 0.00 12:02:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:02:01 AM eth0 1.72 0.97 0.39 0.18 0.00 0.00 0.00 0.00 12:03:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:03:01 AM eth0 1.52 1.10 0.39 0.20 0.00 0.00 0.00 0.00 12:04:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:04:01 AM eth0 1.98 1.10 0.63 0.37 0.00 0.00 0.00 0.00 12:05:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:05:01 AM eth0 1.47 1.08 0.38 0.19 0.00 0.00 0.00 0.00 12:06:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:06:01 AM eth0 31.43 29.90 5.18 2.91 0.00 0.00 0.00 0.00 12:07:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:07:01 AM eth0 13.30 11.28 1.77 1.15 0.00 0.00 0.00 0.00 12:08:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:08:01 AM eth0 25.84 22.64 5.42 3.41 0.00 0.00 0.00 0.00 12:09:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:09:01 AM eth0 11.33 9.23 2.73 1.79 0.00 0.00 0.00 0.00 12:10:02 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:10:02 AM eth0 12.78 8.88 2.88 1.76 0.00 0.00 0.00 0.00 12:11:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:11:01 AM eth0 11.23 9.20 2.80 1.88 0.00 0.00 0.00 0.00 12:12:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:12:01 AM eth0 18.14 15.24 4.50 2.87 0.00 0.00 0.00 0.00 12:13:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:13:01 AM eth0 0.47 0.28 0.10 0.05 0.00 0.00 0.00 0.00 12:14:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:14:01 AM eth0 0.53 0.22 0.10 0.04 0.00 0.00 0.00 0.00 12:15:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:15:01 AM eth0 0.62 0.23 0.10 0.05 0.00 0.00 0.00 0.00 12:16:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:16:01 AM eth0 0.78 0.33 0.23 0.11 0.00 0.00 0.00 0.00 12:17:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:17:01 AM eth0 0.32 0.20 0.09 0.04 0.00 0.00 0.00 0.00 12:18:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:18:01 AM eth0 80.74 80.27 12.85 6.72 0.00 0.00 0.00 0.00 12:19:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:19:01 AM eth0 11.85 9.43 2.88 1.84 0.00 0.00 0.00 0.00 12:20:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:20:01 AM eth0 11.38 9.25 2.77 1.80 0.00 0.00 0.00 0.00 12:21:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:21:01 AM eth0 11.31 9.28 2.86 1.86 0.00 0.00 0.00 0.00 12:22:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:22:01 AM eth0 10.81 8.95 2.61 1.78 0.00 0.00 0.00 0.00 12:23:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:23:01 AM eth0 11.43 9.15 2.78 1.80 0.00 0.00 0.00 0.00 12:24:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:24:01 AM eth0 11.86 9.10 2.84 1.79 0.00 0.00 0.00 0.00 12:25:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:25:01 AM eth0 14.60 11.63 3.85 2.39 0.00 0.00 0.00 0.00 12:26:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:26:01 AM eth0 1.25 0.95 0.64 0.39 0.00 0.00 0.00 0.00 12:27:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:27:01 AM eth0 1.50 0.88 0.57 0.33 0.00 0.00 0.00 0.00 12:28:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:28:01 AM eth0 0.77 0.67 0.26 0.13 0.00 0.00 0.00 0.00 12:29:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:29:01 AM eth0 0.80 0.60 0.26 0.13 0.00 0.00 0.00 0.00 12:30:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:30:01 AM eth0 0.75 0.60 0.26 0.13 0.00 0.00 0.00 0.00 12:31:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:31:01 AM eth0 8.31 7.20 1.64 1.06 0.00 0.00 0.00 0.00 12:32:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:32:01 AM eth0 5.50 5.12 0.77 0.42 0.00 0.00 0.00 0.00 12:33:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:33:01 AM eth0 6.95 3.72 1.08 0.65 0.00 0.00 0.00 0.00 12:34:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:34:01 AM eth0 0.85 0.60 0.32 0.27 0.00 0.00 0.00 0.00 12:35:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:35:01 AM eth0 3.67 0.00 0.39 0.00 0.00 0.00 0.00 0.00 12:36:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:36:01 AM eth0 6.21 4.05 1.88 1.24 0.00 0.00 0.00 0.00 12:37:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:37:01 AM eth0 5.93 5.70 0.78 1.06 0.00 0.00 0.00 0.00 12:38:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:38:01 AM eth0 9.00 3.12 1.62 0.62 0.00 0.00 0.00 0.00 12:39:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:39:01 AM eth0 6.21 5.81 1.03 0.91 0.00 0.00 0.00 0.00 12:40:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:40:01 AM eth0 4.25 3.52 0.56 0.27 0.00 0.00 0.00 0.00 12:41:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:41:01 AM eth0 11.51 9.50 2.76 1.84 0.00 0.00 0.00 0.00 12:42:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:42:01 AM eth0 12.10 9.22 3.01 1.98 0.00 0.00 0.00 0.00 12:43:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:43:01 AM eth0 11.45 9.02 2.78 1.79 0.00 0.00 0.00 0.00 12:44:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:44:01 AM eth0 12.61 8.97 2.84 1.79 0.00 0.00 0.00 0.00 12:45:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:45:01 AM eth0 12.70 9.51 3.35 2.25 0.00 0.00 0.00 0.00 12:46:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:46:01 AM eth0 12.78 10.03 3.00 1.93 0.00 0.00 0.00 0.00 12:47:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:47:01 AM eth0 19.34 14.93 4.58 2.81 0.00 0.00 0.00 0.00 12:48:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:48:01 AM eth0 1.85 1.12 0.87 0.59 0.00 0.00 0.00 0.00 12:49:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:49:01 AM eth0 0.85 0.57 0.19 0.10 0.00 0.00 0.00 0.00 12:50:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:50:01 AM eth0 3.62 0.63 0.54 0.10 0.00 0.00 0.00 0.00 12:51:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:51:01 AM eth0 1.12 0.67 0.33 0.16 0.00 0.00 0.00 0.00 12:52:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:52:01 AM eth0 0.93 0.48 0.18 0.08 0.00 0.00 0.00 0.00 12:53:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:53:01 AM eth0 7.48 6.67 1.42 0.89 0.00 0.00 0.00 0.00 12:54:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:54:01 AM eth0 8.07 7.80 1.14 0.69 0.00 0.00 0.00 0.00 12:55:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:55:01 AM eth0 6.20 5.68 0.88 0.53 0.00 0.00 0.00 0.00 12:56:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:56:01 AM eth0 10.68 9.73 2.07 1.27 0.00 0.00 0.00 0.00 12:57:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:57:01 AM eth0 17.33 9.60 3.46 1.90 0.00 0.00 0.00 0.00 12:58:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:58:01 AM eth0 12.96 9.63 3.48 2.27 0.00 0.00 0.00 0.00 12:59:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12:59:01 AM eth0 11.38 9.22 2.95 1.98 0.00 0.00 0.00 0.00 01:00:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:00:01 AM eth0 17.98 8.86 3.60 1.78 0.00 0.00 0.00 0.00 01:01:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:01:01 AM eth0 11.76 9.25 2.86 1.86 0.00 0.00 0.00 0.00 01:02:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:02:01 AM eth0 12.34 9.06 2.80 1.79 0.00 0.00 0.00 0.00 01:03:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:03:01 AM eth0 14.30 12.10 3.59 2.27 0.00 0.00 0.00 0.00 01:04:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:04:01 AM eth0 1.92 0.82 0.57 0.32 0.00 0.00 0.00 0.00 01:05:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:05:01 AM eth0 3.45 0.85 30.49 0.39 0.00 0.00 0.00 0.00 01:06:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:06:01 AM eth0 1.05 0.72 0.39 0.19 0.00 0.00 0.00 0.00 01:07:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:07:01 AM eth0 1.03 0.57 0.24 0.11 0.00 0.00 0.00 0.00 01:08:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:08:01 AM eth0 1.65 0.82 0.56 0.32 0.00 0.00 0.00 0.00 01:09:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:09:01 AM eth0 7.78 6.90 1.53 0.97 0.00 0.00 0.00 0.00 01:10:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:10:01 AM eth0 5.46 5.10 0.76 0.42 0.00 0.00 0.00 0.00 01:11:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:11:01 AM eth0 3.07 2.53 0.55 0.29 0.00 0.00 0.00 0.00 01:12:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:12:01 AM eth0 3.12 2.75 0.42 0.26 0.00 0.00 0.00 0.00 01:13:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:13:01 AM eth0 5.35 5.03 0.77 0.42 0.00 0.00 0.00 0.00 01:14:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:14:01 AM eth0 3.37 2.55 0.63 0.47 0.00 0.00 0.00 0.00 01:15:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:15:01 AM eth0 4.58 4.13 0.73 0.40 0.00 0.00 0.00 0.00 01:16:02 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:16:02 AM eth0 13.33 10.66 3.22 2.12 0.00 0.00 0.00 0.00 01:17:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:17:01 AM eth0 11.74 9.15 3.25 1.96 0.00 0.00 0.00 0.00 01:18:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:18:01 AM eth0 11.95 9.30 3.05 1.98 0.00 0.00 0.00 0.00 01:19:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:19:01 AM eth0 11.25 8.96 2.74 1.79 0.00 0.00 0.00 0.00 01:20:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:20:01 AM eth0 11.33 9.37 2.79 1.85 0.00 0.00 0.00 0.00 01:21:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:21:01 AM eth0 11.71 9.25 2.89 1.87 0.00 0.00 0.00 0.00 01:22:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:22:01 AM eth0 14.88 12.51 3.90 2.51 0.00 0.00 0.00 0.00 01:23:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:23:01 AM eth0 0.82 0.62 0.26 0.13 0.00 0.00 0.00 0.00 01:24:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:24:01 AM eth0 0.82 0.70 0.27 0.14 0.00 0.00 0.00 0.00 01:25:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:25:01 AM eth0 0.83 0.60 0.26 0.13 0.00 0.00 0.00 0.00 01:26:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:26:01 AM eth0 0.97 0.70 0.39 0.19 0.00 0.00 0.00 0.00 01:27:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:27:01 AM eth0 1.88 0.82 0.60 0.33 0.00 0.00 0.00 0.00 01:28:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:28:01 AM eth0 7.95 6.85 1.74 1.19 0.00 0.00 0.00 0.00 01:29:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:29:01 AM eth0 3.02 2.72 0.41 0.27 0.00 0.00 0.00 0.00 01:30:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:30:01 AM eth0 2.48 2.33 0.35 0.22 0.00 0.00 0.00 0.00 01:31:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:31:01 AM eth0 3.03 2.67 0.51 0.24 0.00 0.00 0.00 0.00 01:32:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:32:01 AM eth0 6.26 5.53 1.08 0.67 0.00 0.00 0.00 0.00 01:33:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:33:01 AM eth0 7.80 6.52 1.98 1.39 0.00 0.00 0.00 0.00 01:34:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:34:01 AM eth0 10.83 9.10 2.69 1.79 0.00 0.00 0.00 0.00 01:35:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:35:01 AM eth0 11.18 9.00 2.71 1.79 0.00 0.00 0.00 0.00 01:36:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:36:01 AM eth0 11.33 9.20 2.84 1.86 0.00 0.00 0.00 0.00 01:37:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:37:01 AM eth0 11.63 9.10 2.79 1.82 0.00 0.00 0.00 0.00 01:38:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:38:01 AM eth0 11.58 9.52 3.06 2.09 0.00 0.00 0.00 0.00 01:39:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:39:01 AM eth0 20.61 17.01 5.09 3.27 0.00 0.00 0.00 0.00 01:40:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:40:01 AM eth0 0.72 0.65 0.22 0.11 0.00 0.00 0.00 0.00 01:41:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:41:01 AM eth0 1.03 0.80 0.39 0.20 0.00 0.00 0.00 0.00 01:42:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:42:01 AM eth0 0.73 0.67 0.26 0.13 0.00 0.00 0.00 0.00 01:43:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:43:01 AM eth0 0.80 0.62 0.26 0.13 0.00 0.00 0.00 0.00 01:44:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:44:01 AM eth0 0.63 0.60 0.25 0.13 0.00 0.00 0.00 0.00 01:45:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:45:01 AM eth0 2.43 1.55 0.60 0.33 0.00 0.00 0.00 0.00 01:46:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:46:01 AM eth0 9.58 8.30 2.14 1.44 0.00 0.00 0.00 0.00 01:47:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:47:01 AM eth0 3.88 3.57 0.54 0.27 0.00 0.00 0.00 0.00 01:48:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:48:01 AM eth0 11.18 9.15 2.84 1.94 0.00 0.00 0.00 0.00 01:49:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:49:01 AM eth0 11.23 9.08 2.72 1.79 0.00 0.00 0.00 0.00 01:50:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:50:01 AM eth0 11.21 9.38 3.03 2.05 0.00 0.00 0.00 0.00 01:51:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:51:01 AM eth0 12.26 9.17 2.97 1.86 0.00 0.00 0.00 0.00 01:52:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:52:01 AM eth0 11.11 9.13 2.74 1.80 0.00 0.00 0.00 0.00 01:53:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:53:01 AM eth0 16.40 8.92 3.31 1.78 0.00 0.00 0.00 0.00 01:54:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:54:01 AM eth0 18.26 14.95 4.74 3.06 0.00 0.00 0.00 0.00 01:55:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:55:01 AM eth0 1.20 0.75 0.29 0.14 0.00 0.00 0.00 0.00 01:56:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:56:01 AM eth0 7.51 0.72 1.27 0.20 0.00 0.00 0.00 0.00 01:57:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:57:01 AM eth0 1.23 0.60 0.29 0.13 0.00 0.00 0.00 0.00 01:58:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:58:01 AM eth0 1.50 0.60 0.37 0.13 0.00 0.00 0.00 0.00 01:59:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 01:59:01 AM eth0 0.93 0.65 0.26 0.13 0.00 0.00 0.00 0.00 02:00:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 02:00:01 AM eth0 7.75 6.71 1.49 0.94 0.00 0.00 0.00 0.00 02:01:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 02:01:01 AM eth0 4.55 3.27 0.95 0.67 0.00 0.00 0.00 0.00 02:02:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 02:02:01 AM eth0 8.21 5.33 1.08 0.61 0.00 0.00 0.00 0.00 02:03:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 02:03:01 AM eth0 14.06 9.78 2.91 1.92 0.00 0.00 0.00 0.00 02:04:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 02:04:01 AM eth0 12.01 9.57 3.05 2.02 0.00 0.00 0.00 0.00 02:05:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 02:05:01 AM eth0 11.30 9.16 2.73 1.80 0.00 0.00 0.00 0.00 02:06:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 02:06:01 AM eth0 12.55 9.32 2.95 1.87 0.00 0.00 0.00 0.00 02:07:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 02:07:01 AM eth0 14.30 9.10 3.06 1.79 0.00 0.00 0.00 0.00 02:08:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 02:08:01 AM eth0 12.06 9.31 3.09 2.05 0.00 0.00 0.00 0.00 02:09:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 02:09:01 AM eth0 14.25 12.11 3.61 2.28 0.00 0.00 0.00 0.00 02:10:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 02:10:01 AM eth0 4.90 0.60 0.83 0.13 0.00 0.00 0.00 0.00 02:11:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 02:11:01 AM eth0 2.30 0.78 0.52 0.20 0.00 0.00 0.00 0.00 02:12:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 02:12:01 AM eth0 1.33 0.60 0.27 0.12 0.00 0.00 0.00 0.00 02:13:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 02:13:01 AM eth0 0.95 0.57 0.25 0.12 0.00 0.00 0.00 0.00 02:14:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 02:14:01 AM eth0 0.92 0.60 0.26 0.13 0.00 0.00 0.00 0.00 02:15:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 02:15:01 AM eth0 7.73 6.63 1.50 0.88 0.00 0.00 0.00 0.00 02:16:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 02:16:01 AM eth0 3.80 3.22 0.93 0.67 0.00 0.00 0.00 0.00 02:17:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 02:17:01 AM eth0 5.28 3.63 0.73 0.28 0.00 0.00 0.00 0.00 02:18:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 02:18:01 AM eth0 13.18 9.85 3.39 2.28 0.00 0.00 0.00 0.00 02:19:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 02:19:01 AM eth0 11.21 8.88 2.84 1.91 0.00 0.00 0.00 0.00 02:20:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 02:20:01 AM eth0 13.01 9.05 2.93 1.79 0.00 0.00 0.00 0.00 02:21:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 02:21:01 AM eth0 12.23 9.31 2.94 1.87 0.00 0.00 0.00 0.00 02:22:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 02:22:01 AM eth0 11.65 9.03 2.77 1.79 0.00 0.00 0.00 0.00 02:23:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 02:23:01 AM eth0 12.21 9.21 2.93 1.80 0.00 0.00 0.00 0.00 02:24:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 02:24:01 AM eth0 18.28 14.51 4.43 2.75 0.00 0.00 0.00 0.00 02:25:02 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 02:25:02 AM eth0 1.00 0.67 0.27 0.13 0.00 0.00 0.00 0.00 02:26:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 02:26:01 AM eth0 1.20 0.69 0.38 0.18 0.00 0.00 0.00 0.00 02:27:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 02:27:01 AM eth0 0.98 0.53 0.20 0.10 0.00 0.00 0.00 0.00 02:28:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 02:28:01 AM eth0 0.90 0.60 0.21 0.10 0.00 0.00 0.00 0.00 02:29:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 02:29:01 AM eth0 1.37 0.63 0.29 0.10 0.00 0.00 0.00 0.00 02:30:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 02:30:01 AM eth0 7.22 6.60 1.39 0.87 0.00 0.00 0.00 0.00 02:31:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 02:31:01 AM eth0 9.95 9.43 3.20 8.11 0.00 0.00 0.00 0.00 02:32:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 02:32:01 AM eth0 9.35 8.18 2.23 0.94 0.00 0.00 0.00 0.00 02:33:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 02:33:01 AM eth0 13.36 12.38 4.04 13.44 0.00 0.00 0.00 0.00 02:34:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 02:34:01 AM eth0 9.02 8.50 3.01 1.40 0.00 0.00 0.00 0.00 02:35:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 02:35:01 AM eth0 52.28 37.42 120.17 6.62 0.00 0.00 0.00 0.00 02:36:01 AM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 02:36:01 AM eth0 99.22 65.06 192.53 316.29 0.00 0.00 0.00 0.00 Average: lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: eth0 8.93 6.93 3.99 3.29 0.00 0.00 0.00 0.00 ---> sar -P ALL: Linux 4.18.0-553.5.1.el8.x86_64 (prd-centos8-robot-2c-8g-1588.novalocal) 01/18/2026 _x86_64_ (2 CPU) 12:00:01 AM CPU %user %nice %system %iowait %steal %idle 12:01:01 AM all 1.16 0.03 0.33 0.07 0.05 98.37 12:01:01 AM 0 0.54 0.00 0.20 0.03 0.05 99.18 12:01:01 AM 1 1.79 0.05 0.45 0.10 0.05 97.56 12:02:01 AM all 1.29 0.00 0.17 0.00 0.05 98.50 12:02:01 AM 0 0.85 0.00 0.15 0.00 0.03 98.96 12:02:01 AM 1 1.72 0.00 0.18 0.00 0.07 98.03 12:03:01 AM all 1.24 0.00 0.15 0.00 0.06 98.55 12:03:01 AM 0 1.10 0.00 0.15 0.00 0.07 98.68 12:03:01 AM 1 1.37 0.00 0.15 0.00 0.05 98.43 12:04:01 AM all 1.23 0.00 0.15 0.00 0.04 98.58 12:04:01 AM 0 0.99 0.00 0.13 0.00 0.03 98.85 12:04:01 AM 1 1.47 0.00 0.17 0.00 0.05 98.31 12:05:01 AM all 1.24 0.00 0.15 0.00 0.07 98.55 12:05:01 AM 0 1.60 0.00 0.22 0.00 0.08 98.10 12:05:01 AM 1 0.87 0.00 0.08 0.00 0.05 99.00 12:06:01 AM all 9.53 0.00 1.35 0.00 0.08 89.05 12:06:01 AM 0 4.12 0.00 0.94 0.00 0.07 94.88 12:06:01 AM 1 14.94 0.00 1.76 0.00 0.08 83.22 12:07:01 AM all 14.44 0.00 0.59 0.05 0.07 84.85 12:07:01 AM 0 2.52 0.00 0.27 0.08 0.05 97.07 12:07:01 AM 1 26.33 0.00 0.92 0.02 0.08 72.65 12:08:01 AM all 6.59 0.00 0.46 0.02 0.08 92.86 12:08:01 AM 0 2.94 0.00 0.38 0.02 0.07 96.59 12:08:01 AM 1 10.25 0.00 0.54 0.02 0.08 89.12 12:09:01 AM all 4.33 0.00 0.32 0.00 0.06 95.29 12:09:01 AM 0 1.67 0.00 0.15 0.00 0.05 98.13 12:09:01 AM 1 6.99 0.00 0.48 0.00 0.07 92.46 12:10:02 AM all 4.41 0.00 0.33 0.00 0.07 95.20 12:10:02 AM 0 5.37 0.00 0.38 0.00 0.07 94.18 12:10:02 AM 1 3.44 0.00 0.27 0.00 0.07 96.22 12:11:01 AM all 4.48 0.00 0.37 0.00 0.05 95.10 12:11:01 AM 0 2.52 0.00 0.24 0.00 0.05 97.19 12:11:01 AM 1 6.44 0.00 0.49 0.00 0.05 93.02 12:11:01 AM CPU %user %nice %system %iowait %steal %idle 12:12:01 AM all 6.24 0.00 0.37 0.00 0.07 93.33 12:12:01 AM 0 2.99 0.00 0.23 0.00 0.05 96.72 12:12:01 AM 1 9.49 0.00 0.50 0.00 0.08 89.93 12:13:01 AM all 0.58 0.00 0.08 0.00 0.05 99.29 12:13:01 AM 0 0.57 0.00 0.07 0.00 0.03 99.33 12:13:01 AM 1 0.58 0.00 0.10 0.00 0.07 99.25 12:14:01 AM all 0.50 0.00 0.07 0.00 0.06 99.37 12:14:01 AM 0 0.50 0.00 0.03 0.00 0.03 99.43 12:14:01 AM 1 0.50 0.00 0.10 0.00 0.08 99.31 12:15:01 AM all 0.58 0.00 0.08 0.00 0.05 99.29 12:15:01 AM 0 0.58 0.00 0.07 0.00 0.03 99.32 12:15:01 AM 1 0.57 0.00 0.10 0.00 0.07 99.26 12:16:01 AM all 0.58 0.00 0.08 0.00 0.06 99.28 12:16:01 AM 0 0.53 0.00 0.10 0.00 0.07 99.30 12:16:01 AM 1 0.62 0.00 0.07 0.00 0.05 99.27 12:17:01 AM all 0.54 0.00 0.10 0.00 0.05 99.31 12:17:01 AM 0 0.28 0.00 0.08 0.00 0.05 99.58 12:17:01 AM 1 0.80 0.00 0.12 0.00 0.05 99.03 12:18:01 AM all 6.24 0.00 0.65 0.00 0.06 93.06 12:18:01 AM 0 4.41 0.00 0.55 0.00 0.05 94.99 12:18:01 AM 1 8.08 0.00 0.74 0.00 0.07 91.11 12:19:01 AM all 4.39 0.00 0.33 0.00 0.06 95.22 12:19:01 AM 0 2.12 0.00 0.22 0.00 0.05 97.61 12:19:01 AM 1 6.66 0.00 0.45 0.00 0.07 92.83 12:20:01 AM all 4.38 0.00 0.33 0.00 0.06 95.24 12:20:01 AM 0 2.81 0.00 0.23 0.00 0.05 96.91 12:20:01 AM 1 5.95 0.00 0.42 0.00 0.07 93.56 12:21:01 AM all 4.35 0.00 0.36 0.01 0.06 95.22 12:21:01 AM 0 3.84 0.00 0.32 0.00 0.07 95.77 12:21:01 AM 1 4.86 0.00 0.40 0.02 0.05 94.67 12:22:01 AM all 4.34 0.00 0.30 0.00 0.05 95.31 12:22:01 AM 0 5.20 0.00 0.35 0.00 0.05 94.40 12:22:01 AM 1 3.48 0.00 0.25 0.00 0.05 96.22 12:22:01 AM CPU %user %nice %system %iowait %steal %idle 12:23:01 AM all 4.32 0.00 0.30 0.00 0.05 95.33 12:23:01 AM 0 4.08 0.00 0.28 0.00 0.05 95.58 12:23:01 AM 1 4.57 0.00 0.32 0.00 0.05 95.07 12:24:01 AM all 4.31 0.00 0.33 0.00 0.07 95.29 12:24:01 AM 0 4.08 0.00 0.30 0.00 0.07 95.55 12:24:01 AM 1 4.55 0.00 0.35 0.00 0.07 95.04 12:25:01 AM all 4.65 0.00 0.30 0.00 0.06 94.99 12:25:01 AM 0 6.88 0.00 0.38 0.00 0.07 92.67 12:25:01 AM 1 2.44 0.00 0.22 0.00 0.05 97.29 12:26:01 AM all 0.94 0.00 0.13 0.00 0.05 98.89 12:26:01 AM 0 0.69 0.00 0.10 0.00 0.03 99.18 12:26:01 AM 1 1.19 0.00 0.15 0.00 0.07 98.60 12:27:01 AM all 0.84 0.00 0.16 0.00 0.07 98.93 12:27:01 AM 0 0.60 0.00 0.15 0.00 0.05 99.20 12:27:01 AM 1 1.09 0.00 0.17 0.00 0.08 98.66 12:28:01 AM all 0.96 0.01 0.13 0.00 0.04 98.86 12:28:01 AM 0 0.74 0.00 0.13 0.00 0.05 99.08 12:28:01 AM 1 1.19 0.02 0.13 0.00 0.03 98.63 12:29:01 AM all 1.04 0.00 0.14 0.00 0.07 98.76 12:29:01 AM 0 1.25 0.00 0.18 0.00 0.07 98.49 12:29:01 AM 1 0.82 0.00 0.10 0.00 0.07 99.02 12:30:01 AM all 0.79 0.00 0.13 0.00 0.05 99.03 12:30:01 AM 0 0.34 0.00 0.10 0.00 0.03 99.53 12:30:01 AM 1 1.25 0.00 0.15 0.00 0.07 98.53 12:31:01 AM all 3.98 0.00 2.28 0.00 0.05 93.69 12:31:01 AM 0 1.05 0.00 0.13 0.00 0.05 98.76 12:31:01 AM 1 7.02 0.00 4.51 0.00 0.05 88.42 12:32:01 AM all 3.35 0.00 0.27 0.00 0.06 96.31 12:32:01 AM 0 3.06 0.00 0.22 0.00 0.07 96.66 12:32:01 AM 1 3.66 0.00 0.32 0.00 0.05 95.97 12:33:01 AM all 4.79 0.00 0.92 0.00 0.05 94.24 12:33:01 AM 0 0.77 0.00 0.10 0.00 0.05 99.08 12:33:01 AM 1 8.96 0.00 1.76 0.00 0.05 89.23 12:33:01 AM CPU %user %nice %system %iowait %steal %idle 12:34:01 AM all 2.71 0.00 0.21 0.00 0.04 97.04 12:34:01 AM 0 5.11 0.00 0.34 0.00 0.03 94.52 12:34:01 AM 1 0.37 0.00 0.08 0.00 0.05 99.50 12:35:01 AM all 0.35 0.00 0.07 0.00 0.04 99.54 12:35:01 AM 0 0.08 0.00 0.07 0.00 0.03 99.82 12:35:01 AM 1 0.62 0.00 0.07 0.00 0.05 99.27 12:36:01 AM all 3.80 0.00 0.30 0.00 0.03 95.86 12:36:01 AM 0 5.60 0.00 0.46 0.00 0.03 93.90 12:36:01 AM 1 2.04 0.00 0.15 0.00 0.03 97.78 12:37:01 AM all 9.92 0.00 0.46 0.00 0.08 89.54 12:37:01 AM 0 1.65 0.00 0.18 0.00 0.07 98.10 12:37:01 AM 1 18.18 0.00 0.73 0.00 0.10 80.98 12:38:01 AM all 3.00 0.00 0.29 0.01 0.04 96.65 12:38:01 AM 0 0.57 0.00 0.08 0.02 0.05 99.28 12:38:01 AM 1 5.55 0.00 0.51 0.00 0.04 93.91 12:39:01 AM all 7.01 0.00 0.38 0.00 0.05 92.56 12:39:01 AM 0 1.57 0.00 0.17 0.00 0.05 98.21 12:39:01 AM 1 12.75 0.00 0.60 0.00 0.05 86.60 12:40:01 AM all 2.16 0.00 0.20 0.00 0.04 97.60 12:40:01 AM 0 2.82 0.00 0.23 0.00 0.05 96.90 12:40:01 AM 1 1.49 0.00 0.17 0.00 0.03 98.31 12:41:01 AM all 5.76 0.00 0.43 0.00 0.05 93.76 12:41:01 AM 0 5.88 0.00 0.44 0.00 0.05 93.63 12:41:01 AM 1 5.64 0.00 0.42 0.00 0.05 93.89 12:42:01 AM all 4.32 0.00 0.31 0.00 0.07 95.30 12:42:01 AM 0 3.70 0.00 0.25 0.00 0.07 95.99 12:42:01 AM 1 4.95 0.00 0.37 0.00 0.07 94.61 12:43:01 AM all 4.35 0.00 0.31 0.00 0.06 95.28 12:43:01 AM 0 3.06 0.00 0.23 0.00 0.05 96.66 12:43:01 AM 1 5.65 0.00 0.38 0.00 0.07 93.90 12:44:01 AM all 4.44 0.00 0.34 0.00 0.06 95.16 12:44:01 AM 0 2.99 0.00 0.25 0.00 0.05 96.71 12:44:01 AM 1 5.89 0.00 0.43 0.00 0.07 93.61 12:44:01 AM CPU %user %nice %system %iowait %steal %idle 12:45:01 AM all 4.36 0.00 0.35 0.00 0.05 95.24 12:45:01 AM 0 3.64 0.00 0.28 0.00 0.05 96.02 12:45:01 AM 1 5.08 0.00 0.42 0.00 0.05 94.45 12:46:01 AM all 4.32 0.23 0.42 0.12 0.06 94.85 12:46:01 AM 0 2.24 0.02 0.28 0.03 0.05 97.38 12:46:01 AM 1 6.41 0.45 0.55 0.20 0.07 92.32 12:47:01 AM all 6.00 0.00 0.34 0.00 0.04 93.61 12:47:01 AM 0 3.63 0.00 0.23 0.00 0.03 96.10 12:47:01 AM 1 8.38 0.00 0.45 0.00 0.05 91.12 12:48:01 AM all 0.92 0.00 0.13 0.00 0.05 98.91 12:48:01 AM 0 0.67 0.00 0.07 0.00 0.03 99.23 12:48:01 AM 1 1.17 0.00 0.18 0.00 0.07 98.58 12:49:01 AM all 0.88 0.00 0.11 0.00 0.05 98.96 12:49:01 AM 0 1.09 0.00 0.15 0.00 0.07 98.70 12:49:01 AM 1 0.67 0.00 0.07 0.00 0.03 99.23 12:50:01 AM all 0.95 0.00 0.13 0.00 0.04 98.87 12:50:01 AM 0 1.10 0.00 0.17 0.00 0.05 98.68 12:50:01 AM 1 0.80 0.00 0.10 0.00 0.03 99.07 12:51:01 AM all 0.93 0.00 0.13 0.00 0.05 98.89 12:51:01 AM 0 0.97 0.00 0.15 0.00 0.05 98.83 12:51:01 AM 1 0.89 0.00 0.12 0.00 0.05 98.95 12:52:01 AM all 0.79 0.00 0.11 0.00 0.06 99.04 12:52:01 AM 0 0.79 0.00 0.12 0.00 0.07 99.03 12:52:01 AM 1 0.80 0.00 0.10 0.00 0.05 99.05 12:53:01 AM all 3.32 0.00 0.49 0.00 0.06 96.13 12:53:01 AM 0 5.54 0.00 0.85 0.00 0.05 93.56 12:53:01 AM 1 1.14 0.00 0.13 0.00 0.07 98.66 12:54:01 AM all 6.76 0.00 1.67 0.00 0.05 91.53 12:54:01 AM 0 8.92 0.00 2.96 0.00 0.05 88.06 12:54:01 AM 1 4.63 0.00 0.40 0.00 0.05 94.92 12:55:01 AM all 6.97 0.00 0.56 0.00 0.05 92.42 12:55:01 AM 0 9.98 0.00 0.67 0.00 0.05 89.30 12:55:01 AM 1 4.00 0.00 0.45 0.00 0.05 95.50 12:55:01 AM CPU %user %nice %system %iowait %steal %idle 12:56:01 AM all 7.45 0.00 0.53 0.00 0.05 91.97 12:56:01 AM 0 6.39 0.00 0.46 0.00 0.03 93.11 12:56:01 AM 1 8.54 0.00 0.60 0.00 0.07 90.79 12:57:01 AM all 4.58 0.00 0.34 0.00 0.07 95.01 12:57:01 AM 0 4.96 0.00 0.35 0.00 0.08 94.60 12:57:01 AM 1 4.20 0.00 0.33 0.00 0.05 95.42 12:58:01 AM all 4.55 0.00 0.33 0.00 0.07 95.06 12:58:01 AM 0 3.19 0.00 0.25 0.00 0.05 96.51 12:58:01 AM 1 5.90 0.00 0.40 0.00 0.08 93.62 12:59:01 AM all 4.31 0.00 0.31 0.00 0.06 95.32 12:59:01 AM 0 3.09 0.00 0.23 0.00 0.05 96.63 12:59:01 AM 1 5.54 0.00 0.38 0.00 0.07 94.01 01:00:01 AM all 4.40 0.00 0.32 0.00 0.07 95.22 01:00:01 AM 0 2.74 0.00 0.23 0.00 0.05 96.97 01:00:01 AM 1 6.05 0.00 0.40 0.00 0.08 93.46 01:01:01 AM all 4.42 0.00 0.33 0.01 0.07 95.17 01:01:01 AM 0 5.59 0.00 0.38 0.02 0.07 93.95 01:01:01 AM 1 3.26 0.00 0.28 0.00 0.07 96.39 01:02:01 AM all 4.40 0.00 0.33 0.00 0.06 95.21 01:02:01 AM 0 2.84 0.00 0.25 0.00 0.07 96.84 01:02:01 AM 1 5.95 0.00 0.42 0.00 0.05 93.58 01:03:01 AM all 4.90 0.00 0.28 0.00 0.06 94.77 01:03:01 AM 0 4.78 0.00 0.32 0.00 0.05 94.85 01:03:01 AM 1 5.01 0.00 0.23 0.00 0.07 94.69 01:04:01 AM all 0.97 0.00 0.12 0.00 0.05 98.86 01:04:01 AM 0 0.70 0.00 0.13 0.00 0.05 99.11 01:04:01 AM 1 1.23 0.00 0.10 0.00 0.05 98.61 01:05:01 AM all 0.94 0.00 0.13 0.00 0.06 98.86 01:05:01 AM 0 0.84 0.00 0.17 0.00 0.07 98.93 01:05:01 AM 1 1.05 0.00 0.10 0.00 0.05 98.80 01:06:01 AM all 0.84 0.00 0.13 0.00 0.05 98.98 01:06:01 AM 0 1.04 0.00 0.15 0.00 0.07 98.75 01:06:01 AM 1 0.64 0.00 0.12 0.00 0.03 99.21 01:06:01 AM CPU %user %nice %system %iowait %steal %idle 01:07:01 AM all 0.77 0.00 0.09 0.00 0.05 99.09 01:07:01 AM 0 1.00 0.00 0.13 0.00 0.07 98.79 01:07:01 AM 1 0.53 0.00 0.05 0.00 0.03 99.38 01:08:01 AM all 0.96 0.00 0.13 0.00 0.05 98.86 01:08:01 AM 0 1.10 0.00 0.17 0.00 0.05 98.68 01:08:01 AM 1 0.82 0.00 0.10 0.00 0.05 99.03 01:09:01 AM all 3.81 0.00 2.38 0.00 0.05 93.76 01:09:01 AM 0 2.62 0.00 0.22 0.00 0.05 97.11 01:09:01 AM 1 5.03 0.00 4.58 0.00 0.05 90.34 01:10:01 AM all 3.76 0.00 0.27 0.01 0.04 95.92 01:10:01 AM 0 5.30 0.00 0.33 0.02 0.05 94.31 01:10:01 AM 1 2.23 0.00 0.22 0.00 0.03 97.52 01:11:01 AM all 2.49 0.00 0.25 0.01 0.05 97.20 01:11:01 AM 0 2.57 0.00 0.24 0.00 0.05 97.15 01:11:01 AM 1 2.41 0.00 0.26 0.02 0.05 97.26 01:12:01 AM all 3.19 0.00 0.33 0.00 0.05 96.43 01:12:01 AM 0 0.13 0.00 0.12 0.00 0.05 99.70 01:12:01 AM 1 6.41 0.00 0.54 0.00 0.05 93.00 01:13:01 AM all 3.16 0.00 0.32 0.00 0.04 96.48 01:13:01 AM 0 0.22 0.00 0.12 0.00 0.05 99.61 01:13:01 AM 1 6.24 0.00 0.53 0.00 0.04 93.19 01:14:01 AM all 3.81 0.00 0.31 0.00 0.04 95.84 01:14:01 AM 0 2.88 0.00 0.27 0.00 0.05 96.80 01:14:01 AM 1 4.76 0.00 0.34 0.00 0.03 94.87 01:15:01 AM all 3.05 0.00 0.24 0.00 0.05 96.66 01:15:01 AM 0 5.32 0.00 0.36 0.00 0.05 94.27 01:15:01 AM 1 0.82 0.00 0.12 0.00 0.05 99.01 01:16:02 AM all 4.87 0.00 0.37 0.00 0.06 94.70 01:16:02 AM 0 4.31 0.00 0.33 0.00 0.05 95.30 01:16:02 AM 1 5.43 0.00 0.40 0.00 0.07 94.10 01:17:01 AM all 4.39 0.00 0.31 0.00 0.07 95.23 01:17:01 AM 0 4.66 0.00 0.32 0.00 0.07 94.95 01:17:01 AM 1 4.11 0.00 0.31 0.00 0.07 95.51 01:17:01 AM CPU %user %nice %system %iowait %steal %idle 01:18:01 AM all 4.35 0.00 0.32 0.00 0.07 95.27 01:18:01 AM 0 5.20 0.00 0.38 0.00 0.07 94.35 01:18:01 AM 1 3.49 0.00 0.25 0.00 0.07 96.19 01:19:01 AM all 4.33 0.00 0.33 0.00 0.06 95.28 01:19:01 AM 0 4.12 0.00 0.33 0.00 0.07 95.47 01:19:01 AM 1 4.53 0.00 0.33 0.00 0.05 95.08 01:20:01 AM all 4.45 0.00 0.28 0.00 0.06 95.21 01:20:01 AM 0 4.78 0.00 0.32 0.00 0.05 94.85 01:20:01 AM 1 4.11 0.00 0.25 0.00 0.07 95.57 01:21:01 AM all 4.34 0.00 0.33 0.00 0.06 95.28 01:21:01 AM 0 5.89 0.00 0.42 0.00 0.07 93.62 01:21:01 AM 1 2.79 0.00 0.23 0.00 0.05 96.93 01:22:01 AM all 5.06 0.00 0.25 0.00 0.06 94.63 01:22:01 AM 0 2.46 0.00 0.17 0.00 0.05 97.33 01:22:01 AM 1 7.67 0.00 0.33 0.00 0.07 91.93 01:23:01 AM all 0.94 0.00 0.13 0.00 0.06 98.86 01:23:01 AM 0 0.95 0.00 0.12 0.00 0.05 98.88 01:23:01 AM 1 0.94 0.00 0.15 0.00 0.07 98.85 01:24:01 AM all 0.84 0.00 0.12 0.00 0.05 99.00 01:24:01 AM 0 0.82 0.00 0.12 0.00 0.05 99.01 01:24:01 AM 1 0.85 0.00 0.12 0.00 0.05 98.98 01:25:01 AM all 0.92 0.00 0.12 0.00 0.06 98.91 01:25:01 AM 0 1.25 0.00 0.17 0.00 0.07 98.51 01:25:01 AM 1 0.58 0.00 0.07 0.00 0.05 99.30 01:26:01 AM all 0.91 0.00 0.13 0.00 0.04 98.92 01:26:01 AM 0 0.77 0.00 0.10 0.00 0.03 99.10 01:26:01 AM 1 1.05 0.00 0.15 0.00 0.05 98.75 01:27:01 AM all 0.90 0.00 0.13 0.00 0.05 98.91 01:27:01 AM 0 0.85 0.00 0.12 0.00 0.05 98.98 01:27:01 AM 1 0.95 0.00 0.15 0.00 0.05 98.85 01:28:01 AM all 3.29 0.00 1.35 0.00 0.05 95.32 01:28:01 AM 0 4.15 0.00 2.40 0.00 0.07 93.39 01:28:01 AM 1 2.42 0.00 0.29 0.00 0.03 97.26 01:28:01 AM CPU %user %nice %system %iowait %steal %idle 01:29:01 AM all 3.92 0.00 0.97 0.00 0.04 95.07 01:29:01 AM 0 4.85 0.00 0.36 0.00 0.03 94.75 01:29:01 AM 1 2.98 0.00 1.58 0.00 0.05 95.39 01:30:01 AM all 2.43 0.00 0.26 0.00 0.05 97.26 01:30:01 AM 0 0.95 0.00 0.08 0.00 0.05 98.92 01:30:01 AM 1 3.96 0.00 0.45 0.00 0.05 95.54 01:31:01 AM all 1.01 0.00 0.14 0.00 0.03 98.81 01:31:01 AM 0 1.92 0.00 0.19 0.00 0.03 97.86 01:31:01 AM 1 0.12 0.00 0.10 0.00 0.03 99.75 01:32:01 AM all 4.08 0.00 0.29 0.00 0.05 95.58 01:32:01 AM 0 7.02 0.00 0.46 0.00 0.05 92.46 01:32:01 AM 1 1.22 0.00 0.12 0.00 0.05 98.61 01:33:01 AM all 4.59 0.00 0.39 0.00 0.06 94.96 01:33:01 AM 0 2.64 0.00 0.25 0.00 0.05 97.06 01:33:01 AM 1 6.62 0.00 0.54 0.00 0.07 92.78 01:34:01 AM all 4.33 0.00 0.30 0.00 0.06 95.31 01:34:01 AM 0 2.29 0.00 0.15 0.00 0.05 97.51 01:34:01 AM 1 6.37 0.00 0.45 0.00 0.07 93.11 01:35:01 AM all 4.32 0.00 0.35 0.00 0.07 95.26 01:35:01 AM 0 6.91 0.00 0.52 0.00 0.08 92.49 01:35:01 AM 1 1.74 0.00 0.18 0.00 0.05 98.03 01:36:01 AM all 4.24 0.00 0.32 0.00 0.06 95.39 01:36:01 AM 0 2.99 0.00 0.23 0.00 0.05 96.72 01:36:01 AM 1 5.48 0.00 0.40 0.00 0.07 94.05 01:37:01 AM all 4.45 0.00 0.29 0.00 0.07 95.19 01:37:01 AM 0 2.56 0.00 0.20 0.00 0.05 97.19 01:37:01 AM 1 6.34 0.00 0.38 0.00 0.08 93.19 01:38:01 AM all 4.36 0.00 0.35 0.00 0.06 95.23 01:38:01 AM 0 2.31 0.00 0.22 0.00 0.05 97.43 01:38:01 AM 1 6.42 0.00 0.48 0.00 0.07 93.03 01:39:01 AM all 7.05 0.00 0.42 0.00 0.06 92.47 01:39:01 AM 0 7.85 0.00 0.50 0.00 0.07 91.58 01:39:01 AM 1 6.25 0.00 0.33 0.00 0.05 93.36 01:39:01 AM CPU %user %nice %system %iowait %steal %idle 01:40:01 AM all 0.83 0.00 0.12 0.00 0.06 98.99 01:40:01 AM 0 1.05 0.00 0.13 0.00 0.05 98.76 01:40:01 AM 1 0.62 0.00 0.10 0.00 0.07 99.22 01:41:01 AM all 0.92 0.00 0.11 0.00 0.04 98.93 01:41:01 AM 0 0.89 0.00 0.13 0.00 0.05 98.93 01:41:01 AM 1 0.95 0.00 0.08 0.00 0.03 98.93 01:42:01 AM all 0.91 0.00 0.12 0.00 0.05 98.92 01:42:01 AM 0 1.07 0.00 0.15 0.00 0.07 98.71 01:42:01 AM 1 0.75 0.00 0.08 0.00 0.03 99.13 01:43:01 AM all 0.97 0.00 0.12 0.00 0.07 98.85 01:43:01 AM 0 1.32 0.00 0.15 0.00 0.08 98.45 01:43:01 AM 1 0.62 0.00 0.08 0.00 0.05 99.25 01:44:01 AM all 0.93 0.00 0.13 0.00 0.04 98.91 01:44:01 AM 0 0.77 0.00 0.10 0.00 0.03 99.10 01:44:01 AM 1 1.08 0.00 0.15 0.00 0.05 98.71 01:45:01 AM all 1.29 0.00 0.13 0.00 0.05 98.53 01:45:01 AM 0 0.67 0.00 0.12 0.00 0.05 99.16 01:45:01 AM 1 1.92 0.00 0.13 0.00 0.05 97.89 01:46:01 AM all 5.00 0.00 1.67 0.00 0.06 93.27 01:46:01 AM 0 8.44 0.00 3.27 0.00 0.05 88.24 01:46:01 AM 1 1.72 0.00 0.15 0.00 0.07 98.06 01:47:01 AM all 2.27 0.00 0.14 0.00 0.04 97.54 01:47:01 AM 0 4.00 0.00 0.22 0.00 0.03 95.75 01:47:01 AM 1 0.58 0.00 0.07 0.00 0.05 99.30 01:48:01 AM all 5.72 0.00 0.40 0.00 0.08 93.81 01:48:01 AM 0 6.67 0.00 0.47 0.00 0.07 92.79 01:48:01 AM 1 4.78 0.00 0.32 0.00 0.08 94.82 01:49:01 AM all 4.35 0.16 0.37 0.01 0.05 95.06 01:49:01 AM 0 4.72 0.00 0.37 0.02 0.05 94.85 01:49:01 AM 1 3.99 0.32 0.37 0.00 0.05 95.27 01:50:01 AM all 4.32 0.00 0.31 0.00 0.06 95.31 01:50:01 AM 0 5.00 0.00 0.38 0.00 0.07 94.55 01:50:01 AM 1 3.64 0.00 0.23 0.00 0.05 96.07 01:50:01 AM CPU %user %nice %system %iowait %steal %idle 01:51:01 AM all 4.45 0.00 0.32 0.00 0.07 95.17 01:51:01 AM 0 4.51 0.00 0.35 0.00 0.07 95.07 01:51:01 AM 1 4.38 0.00 0.28 0.00 0.07 95.27 01:52:01 AM all 4.33 0.00 0.33 0.00 0.05 95.29 01:52:01 AM 0 5.42 0.00 0.38 0.00 0.05 94.14 01:52:01 AM 1 3.24 0.00 0.27 0.00 0.05 96.44 01:53:01 AM all 4.37 0.00 0.30 0.00 0.07 95.26 01:53:01 AM 0 2.02 0.00 0.20 0.00 0.07 97.71 01:53:01 AM 1 6.73 0.00 0.40 0.00 0.07 92.80 01:54:01 AM all 5.95 0.00 0.35 0.00 0.05 93.65 01:54:01 AM 0 6.58 0.00 0.35 0.00 0.05 93.02 01:54:01 AM 1 5.31 0.00 0.35 0.00 0.05 94.28 01:55:01 AM all 0.89 0.00 0.13 0.00 0.05 98.92 01:55:01 AM 0 0.78 0.00 0.12 0.00 0.05 99.05 01:55:01 AM 1 1.00 0.00 0.15 0.00 0.05 98.80 01:56:01 AM all 0.78 0.00 0.12 0.02 0.06 99.03 01:56:01 AM 0 0.54 0.00 0.08 0.03 0.05 99.30 01:56:01 AM 1 1.02 0.00 0.15 0.00 0.07 98.76 01:57:01 AM all 0.88 0.00 0.13 0.00 0.05 98.94 01:57:01 AM 0 0.82 0.00 0.15 0.00 0.05 98.98 01:57:01 AM 1 0.93 0.00 0.12 0.00 0.05 98.90 01:58:01 AM all 0.94 0.00 0.09 0.00 0.04 98.93 01:58:01 AM 0 0.62 0.00 0.07 0.00 0.03 99.28 01:58:01 AM 1 1.25 0.00 0.12 0.00 0.05 98.58 01:59:01 AM all 0.94 0.00 0.12 0.00 0.06 98.88 01:59:01 AM 0 0.87 0.00 0.12 0.00 0.05 98.96 01:59:01 AM 1 1.02 0.00 0.12 0.00 0.07 98.80 02:00:01 AM all 3.18 0.00 0.24 0.00 0.06 96.52 02:00:01 AM 0 4.21 0.00 0.34 0.00 0.05 95.40 02:00:01 AM 1 2.16 0.00 0.15 0.00 0.07 97.62 02:01:01 AM all 4.03 0.00 1.70 0.01 0.04 94.22 02:01:01 AM 0 7.61 0.00 3.32 0.00 0.03 89.04 02:01:01 AM 1 0.60 0.00 0.15 0.02 0.05 99.18 02:01:01 AM CPU %user %nice %system %iowait %steal %idle 02:02:01 AM all 3.81 0.00 0.27 0.00 0.04 95.88 02:02:01 AM 0 6.02 0.00 0.39 0.00 0.05 93.54 02:02:01 AM 1 1.64 0.00 0.15 0.00 0.03 98.18 02:03:01 AM all 4.58 0.00 0.34 0.00 0.08 95.00 02:03:01 AM 0 3.51 0.00 0.30 0.00 0.07 96.12 02:03:01 AM 1 5.65 0.00 0.38 0.00 0.08 93.88 02:04:01 AM all 4.39 0.00 0.33 0.00 0.06 95.23 02:04:01 AM 0 1.80 0.00 0.20 0.00 0.03 97.96 02:04:01 AM 1 6.97 0.00 0.45 0.00 0.08 92.49 02:05:01 AM all 4.29 0.00 0.32 0.00 0.06 95.33 02:05:01 AM 0 3.11 0.00 0.27 0.00 0.05 96.57 02:05:01 AM 1 5.47 0.00 0.37 0.00 0.07 94.10 02:06:01 AM all 4.35 0.00 0.33 0.00 0.06 95.27 02:06:01 AM 0 1.45 0.00 0.17 0.00 0.05 98.33 02:06:01 AM 1 7.24 0.00 0.48 0.00 0.07 92.21 02:07:01 AM all 4.38 0.00 0.34 0.00 0.07 95.21 02:07:01 AM 0 2.16 0.00 0.18 0.00 0.05 97.61 02:07:01 AM 1 6.61 0.00 0.50 0.00 0.08 92.81 02:08:01 AM all 4.42 0.00 0.32 0.00 0.06 95.20 02:08:01 AM 0 3.16 0.00 0.22 0.00 0.05 96.57 02:08:01 AM 1 5.69 0.00 0.42 0.00 0.07 93.83 02:09:01 AM all 4.95 0.00 0.28 0.00 0.05 94.72 02:09:01 AM 0 1.77 0.00 0.17 0.00 0.05 98.01 02:09:01 AM 1 8.13 0.00 0.40 0.00 0.05 91.42 02:10:01 AM all 0.97 0.00 0.15 0.00 0.06 98.82 02:10:01 AM 0 0.72 0.00 0.12 0.00 0.03 99.13 02:10:01 AM 1 1.22 0.00 0.18 0.00 0.08 98.51 02:11:01 AM all 0.94 0.00 0.13 0.00 0.05 98.87 02:11:01 AM 0 0.77 0.00 0.12 0.00 0.05 99.07 02:11:01 AM 1 1.12 0.00 0.15 0.00 0.05 98.68 02:12:01 AM all 0.91 0.00 0.12 0.00 0.06 98.91 02:12:01 AM 0 1.05 0.00 0.12 0.00 0.05 98.78 02:12:01 AM 1 0.77 0.00 0.12 0.00 0.07 99.05 02:12:01 AM CPU %user %nice %system %iowait %steal %idle 02:13:01 AM all 0.87 0.00 0.13 0.00 0.05 98.96 02:13:01 AM 0 0.94 0.00 0.17 0.00 0.07 98.83 02:13:01 AM 1 0.80 0.00 0.08 0.00 0.03 99.08 02:14:01 AM all 0.94 0.00 0.13 0.00 0.04 98.89 02:14:01 AM 0 1.02 0.00 0.15 0.00 0.05 98.78 02:14:01 AM 1 0.87 0.00 0.10 0.00 0.03 99.00 02:15:01 AM all 3.01 0.00 0.24 0.00 0.05 96.71 02:15:01 AM 0 3.33 0.00 0.35 0.00 0.05 96.27 02:15:01 AM 1 2.69 0.00 0.12 0.00 0.05 97.15 02:16:01 AM all 2.90 0.00 1.61 0.00 0.04 95.44 02:16:01 AM 0 5.13 0.00 3.20 0.00 0.03 91.64 02:16:01 AM 1 0.75 0.00 0.08 0.00 0.05 99.12 02:17:01 AM all 2.21 0.00 0.22 0.00 0.04 97.53 02:17:01 AM 0 0.22 0.00 0.13 0.00 0.05 99.60 02:17:01 AM 1 4.26 0.00 0.31 0.00 0.03 95.40 02:18:01 AM all 5.02 0.00 0.41 0.00 0.05 94.51 02:18:01 AM 0 5.39 0.00 0.43 0.00 0.05 94.13 02:18:01 AM 1 4.65 0.00 0.39 0.00 0.05 94.90 02:19:01 AM all 4.32 0.00 0.31 0.00 0.09 95.28 02:19:01 AM 0 6.34 0.00 0.42 0.00 0.07 93.17 02:19:01 AM 1 2.30 0.00 0.20 0.00 0.12 97.38 02:20:01 AM all 4.29 0.00 0.33 0.00 0.05 95.34 02:20:01 AM 0 3.80 0.00 0.28 0.00 0.05 95.87 02:20:01 AM 1 4.77 0.00 0.37 0.00 0.05 94.81 02:21:01 AM all 4.37 0.00 0.33 0.01 0.06 95.24 02:21:01 AM 0 2.00 0.00 0.20 0.02 0.05 97.73 02:21:01 AM 1 6.74 0.00 0.45 0.00 0.07 92.74 02:22:01 AM all 4.39 0.00 0.34 0.00 0.05 95.21 02:22:01 AM 0 2.31 0.00 0.27 0.00 0.05 97.37 02:22:01 AM 1 6.47 0.00 0.42 0.00 0.05 93.06 02:23:01 AM all 4.37 0.00 0.32 0.00 0.06 95.26 02:23:01 AM 0 3.25 0.00 0.23 0.00 0.05 96.47 02:23:01 AM 1 5.48 0.00 0.40 0.00 0.07 94.05 02:23:01 AM CPU %user %nice %system %iowait %steal %idle 02:24:01 AM all 5.94 0.00 0.32 0.00 0.06 93.68 02:24:01 AM 0 4.33 0.00 0.25 0.00 0.05 95.37 02:24:01 AM 1 7.55 0.00 0.38 0.00 0.07 92.00 02:25:02 AM all 0.91 0.00 0.16 0.00 0.03 98.90 02:25:02 AM 0 0.42 0.00 0.13 0.00 0.03 99.42 02:25:02 AM 1 1.40 0.00 0.18 0.00 0.03 98.38 02:26:01 AM all 0.88 0.00 0.13 0.00 0.07 98.92 02:26:01 AM 0 0.63 0.00 0.08 0.00 0.05 99.24 02:26:01 AM 1 1.14 0.00 0.17 0.00 0.08 98.61 02:27:01 AM all 0.90 0.00 0.11 0.00 0.04 98.95 02:27:01 AM 0 0.60 0.00 0.08 0.00 0.03 99.28 02:27:01 AM 1 1.20 0.00 0.13 0.00 0.05 98.61 02:28:01 AM all 0.99 0.00 0.13 0.00 0.04 98.84 02:28:01 AM 0 1.09 0.00 0.15 0.00 0.05 98.71 02:28:01 AM 1 0.88 0.00 0.12 0.00 0.03 98.96 02:29:01 AM all 0.89 0.00 0.13 0.00 0.05 98.92 02:29:01 AM 0 1.27 0.00 0.18 0.00 0.07 98.48 02:29:01 AM 1 0.52 0.00 0.08 0.00 0.03 99.36 02:30:01 AM all 3.29 0.00 0.94 0.00 0.06 95.71 02:30:01 AM 0 0.87 0.00 0.13 0.00 0.05 98.95 02:30:01 AM 1 5.76 0.00 1.77 0.00 0.07 92.40 02:31:01 AM all 6.77 0.00 3.22 0.00 0.08 89.94 02:31:01 AM 0 9.22 0.00 3.62 0.00 0.09 87.08 02:31:01 AM 1 4.37 0.00 2.83 0.00 0.07 92.73 02:32:01 AM all 6.46 0.00 4.30 0.00 0.05 89.19 02:32:01 AM 0 6.72 0.00 4.28 0.00 0.05 88.95 02:32:01 AM 1 6.19 0.00 4.31 0.00 0.05 89.44 02:33:01 AM all 7.18 0.00 2.62 0.00 0.07 90.13 02:33:01 AM 0 5.03 0.00 3.40 0.00 0.07 91.50 02:33:01 AM 1 9.36 0.00 1.83 0.00 0.07 88.73 02:34:01 AM all 6.86 0.00 3.61 0.00 0.06 89.47 02:34:01 AM 0 4.55 0.00 2.44 0.00 0.05 92.96 02:34:01 AM 1 9.23 0.00 4.81 0.00 0.07 85.90 02:34:01 AM CPU %user %nice %system %iowait %steal %idle 02:35:01 AM all 13.12 0.00 3.31 0.00 0.06 83.51 02:35:01 AM 0 1.19 0.00 0.62 0.00 0.05 98.15 02:35:01 AM 1 25.57 0.00 6.11 0.00 0.07 68.24 02:36:01 AM all 34.57 0.00 3.68 0.43 0.09 61.24 02:36:01 AM 0 34.70 0.00 3.90 0.58 0.10 60.72 02:36:01 AM 1 34.44 0.00 3.46 0.27 0.08 61.75 Average: all 3.64 0.00 0.46 0.00 0.06 95.83 Average: 0 3.07 0.00 0.43 0.01 0.05 96.44 Average: 1 4.21 0.01 0.50 0.00 0.06 95.22