00:35:24 Started by upstream project "autorelease-release-vanadium-mvn39-openjdk21" build number 309 00:35:24 originally caused by: 00:35:24 Started by timer 00:35:24 Running as SYSTEM 00:35:24 [EnvInject] - Loading node environment variables. 00:35:24 Building remotely on prd-queue-disttest-2c-1g-22936 (queue-disttest-2c-1g) in workspace /w/workspace/integration-distribution-test-vanadium 00:35:24 [ssh-agent] Looking for ssh-agent implementation... 00:35:25 [ssh-agent] Exec ssh-agent (binary ssh-agent on a remote machine) 00:35:25 $ ssh-agent 00:35:25 SSH_AUTH_SOCK=/tmp/ssh-YTrko7sS063b/agent.3155 00:35:25 SSH_AGENT_PID=3157 00:35:25 [ssh-agent] Started. 00:35:25 Running ssh-add (command line suppressed) 00:35:25 Identity added: /w/workspace/integration-distribution-test-vanadium@tmp/private_key_7021509162095821481.key (/w/workspace/integration-distribution-test-vanadium@tmp/private_key_7021509162095821481.key) 00:35:25 [ssh-agent] Using credentials jenkins (jenkins-ssh) 00:35:25 No emails were triggered. 00:35:25 provisioning config files... 00:35:25 copy managed file [npmrc] to file:/home/jenkins/.npmrc 00:35:25 copy managed file [pipconf] to file:/home/jenkins/.config/pip/pip.conf 00:35:25 [integration-distribution-test-vanadium] $ /bin/bash /tmp/jenkins11329983624965441330.sh 00:35:25 ---> python-tools-install.sh 00:35:25 Setup pyenv: 00:35:25 * system (set by /opt/pyenv/version) 00:35:25 * 3.8.20 (set by /opt/pyenv/version) 00:35:25 * 3.9.20 (set by /opt/pyenv/version) 00:35:25 3.10.15 00:35:25 3.11.10 00:35:29 lf-activate-venv(): INFO: Creating python3 venv at /tmp/venv-bAig 00:35:29 lf-activate-venv(): INFO: Save venv in file: /tmp/.os_lf_venv 00:35:29 lf-activate-venv(): INFO: Installing base packages (pip, setuptools, virtualenv) 00:35:29 lf-activate-venv(): INFO: Attempting to install with network-safe options... 00:35:34 lf-activate-venv(): INFO: Base packages installed successfully 00:35:34 lf-activate-venv(): INFO: Installing additional packages: lftools 00:36:00 lf-activate-venv(): INFO: Adding /tmp/venv-bAig/bin to PATH 00:36:00 Generating Requirements File 00:36:24 ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 00:36:24 httplib2 0.30.2 requires pyparsing<4,>=3.0.4, but you have pyparsing 2.4.7 which is incompatible. 00:36:24 Python 3.11.10 00:36:25 pip 26.1.2 from /tmp/venv-bAig/lib/python3.11/site-packages/pip (python 3.11) 00:36:25 aiohappyeyeballs==2.6.2 00:36:25 aiohttp==3.14.0 00:36:25 aiosignal==1.4.0 00:36:25 appdirs==1.4.4 00:36:25 argcomplete==3.6.3 00:36:25 aspy.yaml==1.3.0 00:36:25 attrs==26.1.0 00:36:25 autopage==0.6.0 00:36:25 beautifulsoup4==4.14.3 00:36:25 boto3==1.43.20 00:36:25 botocore==1.43.20 00:36:25 bs4==0.0.2 00:36:25 certifi==2026.5.20 00:36:25 cffi==2.0.0 00:36:25 cfgv==3.5.0 00:36:25 chardet==7.4.3 00:36:25 charset-normalizer==3.4.7 00:36:25 click==8.4.1 00:36:25 cliff==4.14.0 00:36:25 cmd2==3.5.1 00:36:25 cryptography==3.3.2 00:36:25 debtcollector==3.1.0 00:36:25 decorator==5.3.1 00:36:25 defusedxml==0.7.1 00:36:25 Deprecated==1.3.1 00:36:25 distlib==0.4.1 00:36:25 dnspython==2.8.0 00:36:25 docker==7.1.0 00:36:25 dogpile.cache==1.5.0 00:36:25 durationpy==0.10 00:36:25 email-validator==2.3.0 00:36:25 filelock==3.29.0 00:36:25 frozenlist==1.8.0 00:36:25 future==1.0.0 00:36:25 gitdb==4.0.12 00:36:25 GitPython==3.1.50 00:36:25 httplib2==0.30.2 00:36:25 identify==2.6.19 00:36:25 idna==3.18 00:36:25 importlib-resources==1.5.0 00:36:25 iso8601==2.1.0 00:36:25 Jinja2==3.1.6 00:36:25 jmespath==1.1.0 00:36:25 jsonpatch==1.33 00:36:25 jsonpointer==3.1.1 00:36:25 jsonschema==4.26.0 00:36:25 jsonschema-specifications==2025.9.1 00:36:25 keystoneauth1==5.14.0 00:36:25 kubernetes==36.0.2 00:36:25 lftools==0.37.22 00:36:25 lxml==6.1.1 00:36:25 markdown-it-py==4.2.0 00:36:25 MarkupSafe==3.0.3 00:36:25 mdurl==0.1.2 00:36:25 msgpack==1.1.2 00:36:25 multi_key_dict==2.0.3 00:36:25 multidict==6.7.1 00:36:25 munch==4.0.0 00:36:25 netaddr==1.3.0 00:36:25 niet==1.4.2 00:36:25 nodeenv==1.10.0 00:36:25 oauth2client==4.1.3 00:36:25 oauthlib==3.3.1 00:36:25 openstacksdk==4.14.0 00:36:25 os-service-types==1.8.2 00:36:25 osc-lib==4.6.0 00:36:25 oslo.config==10.4.0 00:36:25 oslo.context==6.4.0 00:36:25 oslo.i18n==6.8.0 00:36:25 oslo.log==8.2.0 00:36:25 oslo.serialization==5.10.0 00:36:25 oslo.utils==10.1.0 00:36:25 packaging==26.2 00:36:25 pbr==7.0.3 00:36:25 platformdirs==4.10.0 00:36:25 prettytable==3.17.0 00:36:25 propcache==0.5.2 00:36:25 psutil==7.2.2 00:36:25 pyasn1==0.6.3 00:36:25 pyasn1_modules==0.4.2 00:36:25 pycparser==3.0 00:36:25 pygerrit2==2.0.15 00:36:25 PyGithub==2.9.1 00:36:25 Pygments==2.20.0 00:36:25 PyJWT==2.13.0 00:36:25 PyNaCl==1.6.2 00:36:25 pyparsing==2.4.7 00:36:25 pyperclip==1.11.0 00:36:25 pyrsistent==0.20.0 00:36:25 python-cinderclient==9.9.0 00:36:25 python-dateutil==2.9.0.post0 00:36:25 python-discovery==1.4.0 00:36:25 python-heatclient==5.2.0 00:36:25 python-jenkins==1.8.3 00:36:25 python-keystoneclient==5.8.0 00:36:25 python-magnumclient==4.10.0 00:36:25 python-openstackclient==10.0.0 00:36:25 python-swiftclient==4.10.0 00:36:25 PyYAML==6.0.3 00:36:25 referencing==0.37.0 00:36:25 requests==2.34.2 00:36:25 requests-oauthlib==2.0.0 00:36:25 rfc3986==2.0.0 00:36:25 rich==15.0.0 00:36:25 rich-argparse==1.8.0 00:36:25 rpds-py==2026.5.1 00:36:25 rsa==4.9.1 00:36:25 ruamel.yaml==0.19.1 00:36:25 ruamel.yaml.clib==0.2.15 00:36:25 s3transfer==0.18.0 00:36:25 simplejson==4.1.1 00:36:25 six==1.17.0 00:36:25 smmap==5.0.3 00:36:25 soupsieve==2.8.4 00:36:25 stevedore==5.8.0 00:36:25 tabulate==0.10.0 00:36:25 toml==0.10.2 00:36:25 tomlkit==0.15.0 00:36:25 tqdm==4.67.3 00:36:25 typing_extensions==4.15.0 00:36:25 urllib3==1.26.20 00:36:25 virtualenv==21.4.2 00:36:25 wcwidth==0.7.0 00:36:25 websocket-client==1.9.0 00:36:25 wrapt==2.2.1 00:36:25 xdg==6.0.0 00:36:25 xmltodict==1.0.4 00:36:25 yarl==1.24.2 00:36:25 yq==3.4.3 00:36:25 [integration-distribution-test-vanadium] $ /bin/sh /tmp/jenkins9018114735093892511.sh 00:36:25 ---> uv-install.sh 00:36:25 Installing uv/uvx (latest) using shell installer 00:36:25 2026-06-03 00:36:25 URL:https://releases.astral.sh/installers/uv/latest/uv-installer.sh [71233/71233] -> "/tmp/uv-install-LeRiK4.sh" [1] 00:36:25 downloading uv 0.11.18 x86_64-unknown-linux-gnu 00:36:26 installing to /home/jenkins/.local/bin 00:36:26 uv 00:36:26 uvx 00:36:26 everything's installed! 00:36:26 00:36:26 To add $HOME/.local/bin to your PATH, either restart your shell or run: 00:36:26 00:36:26 source $HOME/.local/bin/env (sh, bash, zsh) 00:36:26 source $HOME/.local/bin/env.fish (fish) 00:36:26 Adding install location to PATH 00:36:26 ---> Validating uv/uvx install 00:36:26 uvx 0.11.18 (x86_64-unknown-linux-gnu) 00:36:26 Waiting for the completion of bgpcep-csit-1node-bgp-ingest-all-vanadium 00:36:26 bgpcep-csit-1node-bgp-ingest-all-vanadium #281 started. 04:27:44 bgpcep-csit-1node-bgp-ingest-all-vanadium #281 completed. Result was UNSTABLE 04:27:44 Waiting for the completion of bgpcep-csit-1node-bgp-ingest-mixed-all-vanadium 04:27:44 bgpcep-csit-1node-bgp-ingest-mixed-all-vanadium #281 started. 04:27:44 bgpcep-csit-1node-bgp-ingest-mixed-all-vanadium #281 completed. Result was UNSTABLE 04:27:44 Waiting for the completion of bgpcep-csit-1node-throughpcep-all-vanadium 04:27:44 bgpcep-csit-1node-throughpcep-all-vanadium #281 started. 04:27:44 bgpcep-csit-1node-throughpcep-all-vanadium #281 completed. Result was UNSTABLE 04:27:44 Waiting for the completion of bgpcep-csit-1node-userfeatures-all-vanadium 04:27:44 bgpcep-csit-1node-userfeatures-all-vanadium #281 started. 04:27:44 bgpcep-csit-1node-userfeatures-all-vanadium #281 completed. Result was UNSTABLE 04:27:44 Waiting for the completion of daexim-csit-1node-basic-only-vanadium 04:27:44 daexim-csit-1node-basic-only-vanadium #281 started. 04:27:44 daexim-csit-1node-basic-only-vanadium #281 completed. Result was UNSTABLE 04:27:44 Waiting for the completion of daexim-csit-3node-clustering-basic-only-vanadium 04:27:44 daexim-csit-3node-clustering-basic-only-vanadium #281 started. 04:27:44 daexim-csit-3node-clustering-basic-only-vanadium #281 completed. Result was FAILURE 04:27:44 Waiting for the completion of distribution-csit-managed-vanadium 04:27:44 distribution-csit-managed-vanadium #281 started. 04:27:44 distribution-csit-managed-vanadium #281 completed. Result was SUCCESS 04:27:44 Waiting for the completion of jsonrpc-csit-1node-basic-only-vanadium 04:27:44 jsonrpc-csit-1node-basic-only-vanadium #282 started. 04:27:44 jsonrpc-csit-1node-basic-only-vanadium #282 completed. Result was UNSTABLE 04:27:44 Waiting for the completion of openflowplugin-csit-1node-cbench-only-vanadium 04:27:44 openflowplugin-csit-1node-cbench-only-vanadium #281 started. 04:27:44 openflowplugin-csit-1node-cbench-only-vanadium #281 completed. Result was UNSTABLE 04:27:44 Waiting for the completion of openflowplugin-csit-1node-flow-services-all-vanadium 04:27:44 openflowplugin-csit-1node-flow-services-all-vanadium #281 started. 04:27:44 openflowplugin-csit-1node-flow-services-all-vanadium #281 completed. Result was UNSTABLE 04:27:44 Waiting for the completion of openflowplugin-csit-1node-perf-bulkomatic-only-vanadium 04:27:44 openflowplugin-csit-1node-perf-bulkomatic-only-vanadium #281 started. 04:27:44 openflowplugin-csit-1node-perf-bulkomatic-only-vanadium #281 completed. Result was UNSTABLE 04:27:44 Waiting for the completion of openflowplugin-csit-1node-perf-stats-collection-only-vanadium 04:27:44 openflowplugin-csit-1node-perf-stats-collection-only-vanadium #281 started. 04:27:44 openflowplugin-csit-1node-perf-stats-collection-only-vanadium #281 completed. Result was UNSTABLE 04:27:44 Waiting for the completion of openflowplugin-csit-1node-scale-link-only-vanadium 04:27:44 openflowplugin-csit-1node-scale-link-only-vanadium #281 started. 04:27:44 openflowplugin-csit-1node-scale-link-only-vanadium #281 completed. Result was UNSTABLE 04:27:44 Waiting for the completion of openflowplugin-csit-1node-scale-only-vanadium 04:27:44 openflowplugin-csit-1node-scale-only-vanadium #281 started. 04:27:44 openflowplugin-csit-1node-scale-only-vanadium #281 completed. Result was UNSTABLE 04:27:44 Waiting for the completion of openflowplugin-csit-1node-scale-switch-only-vanadium 04:27:44 openflowplugin-csit-1node-scale-switch-only-vanadium #281 started. 04:27:44 openflowplugin-csit-1node-scale-switch-only-vanadium #281 completed. Result was UNSTABLE 04:27:44 Waiting for the completion of openflowplugin-csit-3node-clustering-bulkomatic-only-vanadium 04:27:44 openflowplugin-csit-3node-clustering-bulkomatic-only-vanadium #281 started. 04:27:44 openflowplugin-csit-3node-clustering-bulkomatic-only-vanadium #281 completed. Result was FAILURE 04:27:44 Waiting for the completion of openflowplugin-csit-3node-clustering-only-vanadium 04:27:44 openflowplugin-csit-3node-clustering-only-vanadium #281 started. 04:27:44 openflowplugin-csit-3node-clustering-only-vanadium #281 completed. Result was FAILURE 04:27:44 Waiting for the completion of openflowplugin-csit-3node-clustering-perf-bulkomatic-only-vanadium 04:27:44 openflowplugin-csit-3node-clustering-perf-bulkomatic-only-vanadium #281 started. 04:27:44 openflowplugin-csit-3node-clustering-perf-bulkomatic-only-vanadium #281 completed. Result was FAILURE 04:27:44 Waiting for the completion of ovsdb-csit-1node-upstream-southbound-all-vanadium 04:27:44 ovsdb-csit-1node-upstream-southbound-all-vanadium #282 started. 04:27:44 ovsdb-csit-1node-upstream-southbound-all-vanadium #282 completed. Result was UNSTABLE 04:27:44 Waiting for the completion of ovsdb-csit-3node-upstream-clustering-only-vanadium 04:27:44 ovsdb-csit-3node-upstream-clustering-only-vanadium #281 started. 04:27:44 ovsdb-csit-3node-upstream-clustering-only-vanadium #281 completed. Result was FAILURE 04:27:44 Build step 'Trigger/call builds on other projects' changed build result to UNSTABLE 04:27:44 Build step 'Trigger/call builds on other projects' marked build as failure 04:27:44 $ ssh-agent -k 04:27:44 unset SSH_AUTH_SOCK; 04:27:44 unset SSH_AGENT_PID; 04:27:44 echo Agent pid 3157 killed; 04:27:44 [ssh-agent] Stopped. 04:27:44 [PostBuildScript] - [INFO] Executing post build scripts. 04:27:44 [integration-distribution-test-vanadium] $ /bin/sh /tmp/jenkins10940322606149331421.sh 04:27:56 python /tmp/tmp.97JQ5fe6nh https://jenkins.opendaylight.org/releng/job/integration-distribution-test-vanadium/283/ 04:27:56 /tmp/tmp.97JQ5fe6nh:26: DeprecationWarning: Call to deprecated method findAll. (Replaced by find_all) -- Deprecated since version 4.0.0. 04:27:56 links = soup.findAll("a", { "class" : "model-link" }) 04:27:56 [PostBuildScript] - [INFO] Executing post build scripts. 04:27:56 [integration-distribution-test-vanadium] $ /bin/bash /tmp/jenkins13092621291712654658.sh 04:27:56 ---> sysstat.sh 04:27:56 /tmp/jenkins13092621291712654658.sh: line 19: facter: command not found 04:27:56 [integration-distribution-test-vanadium] $ /bin/bash /tmp/jenkins12607388255763933210.sh 04:27:56 ---> package-listing.sh 04:27:56 ++ facter osfamily 04:27:56 /tmp/jenkins12607388255763933210.sh: line 19: facter: command not found 04:27:56 ++ tr '[:upper:]' '[:lower:]' 04:27:56 + OS_FAMILY= 04:27:56 [integration-distribution-test-vanadium] $ /bin/bash /tmp/jenkins1884162669449772573.sh 04:27:56 ---> capture-instance-metadata.sh 04:27:56 Setup pyenv: 04:27:56 system 04:27:56 3.8.20 04:27:56 3.9.20 04:27:56 3.10.15 04:27:56 * 3.11.10 (set by /w/workspace/integration-distribution-test-vanadium/.python-version) 04:27:57 lf-activate-venv(): INFO: Reuse venv:/tmp/venv-bAig from file:/tmp/.os_lf_venv 04:27:57 lf-activate-venv(): INFO: Installing base packages (pip, setuptools, virtualenv) 04:27:57 lf-activate-venv(): INFO: Attempting to install with network-safe options... 04:27:59 lf-activate-venv(): INFO: Base packages installed successfully 04:27:59 lf-activate-venv(): INFO: Installing additional packages: lftools 04:28:10 lf-activate-venv(): INFO: Adding /tmp/venv-bAig/bin to PATH 04:28:10 INFO: Running in OpenStack, capturing instance metadata 04:28:11 [integration-distribution-test-vanadium] $ /bin/bash /tmp/jenkins3148641087120766933.sh 04:28:11 provisioning config files... 04:28:11 Could not find credentials [logs] for integration-distribution-test-vanadium #283 04:28:11 copy managed file [jenkins-log-archives-settings] to file:/w/workspace/integration-distribution-test-vanadium@tmp/config3241247991605250580tmp 04:28:11 Regular expression run condition: Expression=[^.*logs-s3.*], Label=[odl-logs-s3-cloudfront-index] 04:28:11 Run condition [Regular expression match] enabling perform for step [Provide Configuration files] 04:28:11 provisioning config files... 04:28:11 copy managed file [jenkins-s3-log-ship] to file:/home/jenkins/.aws/credentials 04:28:11 [EnvInject] - Injecting environment variables from a build step. 04:28:11 [EnvInject] - Injecting as environment variables the properties content 04:28:11 SERVER_ID=logs 04:28:11 04:28:11 [EnvInject] - Variables injected successfully. 04:28:11 [integration-distribution-test-vanadium] $ /bin/bash /tmp/jenkins4131596177051350190.sh 04:28:11 ---> create-netrc.sh 04:28:11 WARN: Log server credential not found. 04:28:11 [integration-distribution-test-vanadium] $ /bin/bash /tmp/jenkins8781377924249993739.sh 04:28:11 ---> python-tools-install.sh 04:28:12 Setup pyenv: 04:28:12 system 04:28:12 3.8.20 04:28:12 3.9.20 04:28:12 3.10.15 04:28:12 * 3.11.10 (set by /w/workspace/integration-distribution-test-vanadium/.python-version) 04:28:12 lf-activate-venv(): INFO: Reuse venv:/tmp/venv-bAig from file:/tmp/.os_lf_venv 04:28:12 lf-activate-venv(): INFO: Installing base packages (pip, setuptools, virtualenv) 04:28:12 lf-activate-venv(): INFO: Attempting to install with network-safe options... 04:28:14 lf-activate-venv(): INFO: Base packages installed successfully 04:28:14 lf-activate-venv(): INFO: Installing additional packages: lftools 04:28:28 lf-activate-venv(): INFO: Adding /tmp/venv-bAig/bin to PATH 04:28:28 [integration-distribution-test-vanadium] $ /bin/sh /tmp/jenkins17957543273684431194.sh 04:28:28 ---> uv-install.sh 04:28:28 uv 0.11.18 is already installed 04:28:28 uvx 0.11.18 (x86_64-unknown-linux-gnu) 04:28:28 [integration-distribution-test-vanadium] $ /bin/bash /tmp/jenkins10788757122543265719.sh 04:28:28 ---> sudo-logs.sh 04:28:28 Archiving 'sudo' log.. 04:28:28 /tmp/jenkins10788757122543265719.sh: line 41: facter: command not found 04:28:28 [integration-distribution-test-vanadium] $ /bin/bash /tmp/jenkins10737594855022023144.sh 04:28:28 ---> job-cost.sh 04:28:28 INFO: Activating Python virtual environment... 04:28:28 Setup pyenv: 04:28:28 system 04:28:28 3.8.20 04:28:28 3.9.20 04:28:28 3.10.15 04:28:28 * 3.11.10 (set by /w/workspace/integration-distribution-test-vanadium/.python-version) 04:28:28 lf-activate-venv(): INFO: Reuse venv:/tmp/venv-bAig from file:/tmp/.os_lf_venv 04:28:28 lf-activate-venv(): INFO: Installing base packages (pip, setuptools, virtualenv) 04:28:28 lf-activate-venv(): INFO: Attempting to install with network-safe options... 04:28:30 lf-activate-venv(): INFO: Base packages installed successfully 04:28:30 lf-activate-venv(): INFO: Installing additional packages: zipp==1.1.0 python-openstackclient urllib3~=1.26.15 04:28:39 lf-activate-venv(): INFO: Adding /tmp/venv-bAig/bin to PATH 04:28:39 INFO: No stack-cost file found 04:28:39 INFO: Instance uptime: 14021s 04:28:39 INFO: Fetching instance metadata (attempt 1 of 3)... 04:28:39 DEBUG: URL: http://169.254.169.254/latest/meta-data/instance-type 04:28:39 INFO: Successfully fetched instance metadata 04:28:39 INFO: Instance type: v3-standard-2 04:28:39 INFO: Retrieving pricing info for: v3-standard-2 04:28:39 INFO: Fetching Vexxhost pricing API (attempt 1 of 3)... 04:28:39 DEBUG: URL: https://pricing.vexxhost.net/v1/pricing/v3-standard-2/cost?seconds=14021 04:28:40 INFO: Successfully fetched Vexxhost pricing API 04:28:40 INFO: Retrieved cost: 0.22 04:28:40 INFO: Retrieved resource: v3-standard-2 04:28:40 INFO: Creating archive directory: /w/workspace/integration-distribution-test-vanadium/archives/cost 04:28:40 INFO: Archiving costs to: /w/workspace/integration-distribution-test-vanadium/archives/cost.csv 04:28:40 INFO: Successfully archived job cost data 04:28:40 DEBUG: Cost data: integration-distribution-test-vanadium,283,2026-06-03 04:28:40,v3-standard-2,14021,0.22,0.00,FAILURE 04:28:40 [integration-distribution-test-vanadium] $ /bin/bash -l /tmp/jenkins18014397570939284333.sh 04:28:40 ---> logs-deploy.sh 04:28:40 Setup pyenv: 04:28:40 system 04:28:40 3.8.20 04:28:40 3.9.20 04:28:40 3.10.15 04:28:40 * 3.11.10 (set by /w/workspace/integration-distribution-test-vanadium/.python-version) 04:28:40 lf-activate-venv(): INFO: Reuse venv:/tmp/venv-bAig from file:/tmp/.os_lf_venv 04:28:40 lf-activate-venv(): INFO: Installing base packages (pip, setuptools, virtualenv) 04:28:40 lf-activate-venv(): INFO: Attempting to install with network-safe options... 04:28:42 lf-activate-venv(): INFO: Base packages installed successfully 04:28:42 lf-activate-venv(): INFO: Installing additional packages: lftools urllib3~=1.26.15 04:28:52 lf-activate-venv(): INFO: Adding /tmp/venv-bAig/bin to PATH 04:28:52 WARNING: Nexus logging server not set 04:28:52 INFO: S3 path logs/releng/vex-yul-odl-jenkins-1/integration-distribution-test-vanadium/283/ 04:28:52 INFO: archiving logs to S3 04:28:53 ---> uname -a: 04:28:53 Linux prd-queue-disttest-2c-1g-22936 6.8.0-90-generic #91-Ubuntu SMP PREEMPT_DYNAMIC Tue Nov 18 14:14:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux 04:28:53 04:28:53 04:28:53 ---> lscpu: 04:28:53 Architecture: x86_64 04:28:53 CPU op-mode(s): 32-bit, 64-bit 04:28:53 Address sizes: 40 bits physical, 48 bits virtual 04:28:53 Byte Order: Little Endian 04:28:53 CPU(s): 2 04:28:53 On-line CPU(s) list: 0,1 04:28:53 Vendor ID: AuthenticAMD 04:28:53 Model name: AMD EPYC-Rome Processor 04:28:53 CPU family: 23 04:28:53 Model: 49 04:28:53 Thread(s) per core: 1 04:28:53 Core(s) per socket: 1 04:28:53 Socket(s): 2 04:28:53 Stepping: 0 04:28:53 BogoMIPS: 5599.99 04:28:53 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd ibrs ibpb stibp vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr wbnoinvd arat npt nrip_save umip rdpid arch_capabilities 04:28:53 Virtualization: AMD-V 04:28:53 Hypervisor vendor: KVM 04:28:53 Virtualization type: full 04:28:53 L1d cache: 64 KiB (2 instances) 04:28:53 L1i cache: 64 KiB (2 instances) 04:28:53 L2 cache: 1 MiB (2 instances) 04:28:53 L3 cache: 32 MiB (2 instances) 04:28:53 NUMA node(s): 1 04:28:53 NUMA node0 CPU(s): 0,1 04:28:53 Vulnerability Gather data sampling: Not affected 04:28:53 Vulnerability Itlb multihit: Not affected 04:28:53 Vulnerability L1tf: Not affected 04:28:53 Vulnerability Mds: Not affected 04:28:53 Vulnerability Meltdown: Not affected 04:28:53 Vulnerability Mmio stale data: Not affected 04:28:53 Vulnerability Reg file data sampling: Not affected 04:28:53 Vulnerability Retbleed: Mitigation; untrained return thunk; SMT disabled 04:28:53 Vulnerability Spec rstack overflow: Vulnerable: Safe RET, no microcode 04:28:53 Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl 04:28:53 Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization 04:28:53 Vulnerability Spectre v2: Mitigation; Retpolines; IBPB conditional; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected 04:28:53 Vulnerability Srbds: Not affected 04:28:53 Vulnerability Tsx async abort: Not affected 04:28:53 Vulnerability Vmscape: Not affected 04:28:53 04:28:53 04:28:53 ---> nproc: 04:28:53 2 04:28:53 04:28:53 04:28:53 ---> df -h: 04:28:53 Filesystem Size Used Avail Use% Mounted on 04:28:53 tmpfs 795M 1.1M 794M 1% /run 04:28:53 /dev/vda1 38G 7.8G 30G 21% / 04:28:53 tmpfs 3.9G 0 3.9G 0% /dev/shm 04:28:53 tmpfs 5.0M 0 5.0M 0% /run/lock 04:28:53 /dev/vda16 881M 117M 703M 15% /boot 04:28:53 /dev/vda15 105M 6.2M 99M 6% /boot/efi 04:28:53 tmpfs 795M 16K 795M 1% /run/user/1001 04:28:53 04:28:53 04:28:53 ---> free -m: 04:28:53 total used free shared buff/cache available 04:28:53 Mem: 7941 713 5366 4 2156 7228 04:28:53 Swap: 1023 0 1023 04:28:53 04:28:53 04:28:53 ---> ip addr: 04:28:53 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 04:28:53 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 04:28:53 inet 127.0.0.1/8 scope host lo 04:28:53 valid_lft forever preferred_lft forever 04:28:53 inet6 ::1/128 scope host noprefixroute 04:28:53 valid_lft forever preferred_lft forever 04:28:53 2: ens3: mtu 1458 qdisc mq state UP group default qlen 1000 04:28:53 link/ether fa:16:3e:c6:5b:9e brd ff:ff:ff:ff:ff:ff 04:28:53 altname enp0s3 04:28:53 inet 10.30.171.34/23 metric 100 brd 10.30.171.255 scope global dynamic ens3 04:28:53 valid_lft 72374sec preferred_lft 72374sec 04:28:53 inet6 fe80::f816:3eff:fec6:5b9e/64 scope link 04:28:53 valid_lft forever preferred_lft forever 04:28:53 04:28:53 04:28:53 ---> sar -b -r -n DEV: 04:28:53 Linux 6.8.0-90-generic (prd-queue-disttest-2c-1g-22936) 06/03/26 _x86_64_ (2 CPU) 04:28:53 04:28:53 00:35:10 LINUX RESTART (2 CPU) 04:28:53 04:28:53 00:40:32 tps rtps wtps dtps bread/s bwrtn/s bdscd/s 04:28:53 00:50:00 1.38 0.03 1.34 0.01 1.31 15.90 0.89 04:28:53 01:00:12 1.59 0.01 1.56 0.02 0.05 18.63 0.34 04:28:53 01:10:22 1.35 0.00 1.35 0.00 0.00 15.76 0.00 04:28:53 01:20:00 1.32 0.00 1.32 0.00 0.01 15.22 0.00 04:28:53 01:30:00 15.08 0.26 1.86 12.97 31.12 78.12 107467.30 04:28:53 01:40:00 1.38 0.00 1.37 0.01 1.05 16.18 0.05 04:28:53 01:50:01 1.34 0.00 1.34 0.00 0.00 15.74 0.00 04:28:53 02:00:02 1.56 0.00 1.55 0.02 0.00 19.32 0.23 04:28:53 02:10:02 1.36 0.00 1.36 0.00 0.00 15.68 0.00 04:28:53 02:20:01 1.34 0.00 1.34 0.00 0.00 15.60 0.00 04:28:53 02:30:03 1.55 0.00 1.53 0.02 0.00 18.86 0.23 04:28:53 02:40:03 15.59 13.52 2.02 0.05 150.49 44.41 9.31 04:28:53 02:50:01 1.54 0.00 1.53 0.01 0.00 18.81 0.08 04:28:53 03:00:03 1.59 0.00 1.57 0.02 0.00 19.58 0.23 04:28:53 03:10:02 1.48 0.00 1.48 0.00 0.00 17.95 0.13 04:28:53 03:20:01 1.45 0.00 1.45 0.01 0.00 17.33 0.05 04:28:53 03:30:02 1.64 0.00 1.62 0.02 0.00 19.90 0.23 04:28:53 03:40:04 1.40 0.00 1.40 0.00 0.00 16.21 0.00 04:28:53 03:50:01 1.39 0.00 1.39 0.00 0.00 16.07 0.00 04:28:53 04:00:07 1.62 0.00 1.61 0.02 0.00 19.94 0.22 04:28:53 04:10:07 1.55 0.00 1.54 0.01 0.00 18.63 0.08 04:28:53 04:20:01 1.42 0.00 1.42 0.00 0.00 16.52 0.00 04:28:53 Average: 2.73 0.63 1.50 0.60 8.39 21.40 4899.79 04:28:53 04:28:53 00:40:32 kbmemfree kbavail kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty 04:28:53 00:50:00 5712536 7508696 278392 3.42 63052 1910048 536592 5.84 463760 1689988 124 04:28:53 01:00:12 5703084 7499780 286980 3.53 63212 1910428 535320 5.83 463072 1690520 204 04:28:53 01:10:22 5702844 7500032 286760 3.53 63380 1910752 535492 5.83 463152 1691012 136 04:28:53 01:20:00 5702844 7500464 286332 3.52 63524 1911040 535620 5.83 463408 1691444 176 04:28:53 01:30:00 5668128 7492324 294132 3.62 63804 1937076 730564 7.96 477580 1708972 152 04:28:53 01:40:00 5667624 7492720 293748 3.61 63932 1937836 535620 5.83 472380 1709424 124 04:28:53 01:50:01 5667120 7492684 293780 3.61 64068 1938160 535620 5.83 472572 1709884 240 04:28:53 02:00:02 5664888 7490956 295516 3.63 64228 1938500 535620 5.83 472580 1710376 264 04:28:53 02:10:02 5664888 7491456 295016 3.63 64392 1938820 535620 5.83 472584 1710868 188 04:28:53 02:20:01 5664648 7491688 294772 3.62 64540 1939144 535620 5.83 472512 1711332 268 04:28:53 02:30:03 5663564 7491260 295192 3.63 64720 1939600 535620 5.83 472788 1711848 184 04:28:53 02:40:03 5607824 7492264 285128 3.51 68524 1984224 535620 5.83 473400 1759668 196 04:28:53 02:50:01 5601032 7485976 291360 3.58 68688 1984556 535684 5.83 473444 1760160 236 04:28:53 03:00:03 5598776 7484232 293088 3.60 68856 1984876 535684 5.83 473476 1760652 204 04:28:53 03:10:02 5598776 7484864 292380 3.60 69024 1985336 590760 6.43 478816 1761148 280 04:28:53 03:20:01 5598776 7485392 291872 3.59 69212 1985664 535836 5.84 473580 1761668 240 04:28:53 03:30:02 5597268 7484396 292848 3.60 69380 1985992 535836 5.84 473752 1762164 240 04:28:53 03:40:04 5596272 7483896 293340 3.61 69548 1986316 535836 5.84 473768 1762656 208 04:28:53 03:50:01 5596680 7484800 292432 3.60 69708 1986644 535836 5.84 473772 1763144 140 04:28:53 04:00:07 5600768 7489544 287640 3.54 69904 1987096 535836 5.84 473920 1763664 192 04:28:53 04:10:07 5599816 7489132 288012 3.54 70108 1987428 535900 5.84 473964 1764200 192 04:28:53 04:20:01 5599896 7489716 287420 3.53 70276 1987756 535900 5.84 473976 1764696 204 04:28:53 Average: 5639911 7491194 290734 3.57 66640 1957150 547093 5.96 471921 1732704 200 04:28:53 04:28:53 00:40:32 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil 04:28:53 00:50:00 lo 1.00 1.00 0.05 0.05 0.00 0.00 0.00 0.00 04:28:53 00:50:00 ens3 2.75 0.15 0.46 0.13 0.00 0.00 0.00 0.00 04:28:53 01:00:12 lo 1.00 1.00 0.05 0.05 0.00 0.00 0.00 0.00 04:28:53 01:00:12 ens3 0.64 0.05 0.09 0.03 0.00 0.00 0.00 0.00 04:28:53 01:10:22 lo 1.00 1.00 0.05 0.05 0.00 0.00 0.00 0.00 04:28:53 01:10:22 ens3 0.38 0.05 0.08 0.03 0.00 0.00 0.00 0.00 04:28:53 01:20:00 lo 1.00 1.00 0.05 0.05 0.00 0.00 0.00 0.00 04:28:53 01:20:00 ens3 0.49 0.07 0.12 0.05 0.00 0.00 0.00 0.00 04:28:53 01:30:00 lo 1.02 1.02 0.05 0.05 0.00 0.00 0.00 0.00 04:28:53 01:30:00 ens3 0.66 0.32 3.16 0.05 0.00 0.00 0.00 0.00 04:28:53 01:40:00 lo 1.00 1.00 0.05 0.05 0.00 0.00 0.00 0.00 04:28:53 01:40:00 ens3 0.34 0.06 0.08 0.03 0.00 0.00 0.00 0.00 04:28:53 01:50:01 lo 1.00 1.00 0.05 0.05 0.00 0.00 0.00 0.00 04:28:53 01:50:01 ens3 0.33 0.09 0.11 0.06 0.00 0.00 0.00 0.00 04:28:53 02:00:02 lo 1.00 1.00 0.05 0.05 0.00 0.00 0.00 0.00 04:28:53 02:00:02 ens3 0.20 0.05 0.07 0.03 0.00 0.00 0.00 0.00 04:28:53 02:10:02 lo 1.00 1.00 0.05 0.05 0.00 0.00 0.00 0.00 04:28:53 02:10:02 ens3 0.15 0.03 0.04 0.01 0.00 0.00 0.00 0.00 04:28:53 02:20:01 lo 1.00 1.00 0.05 0.05 0.00 0.00 0.00 0.00 04:28:53 02:20:01 ens3 0.17 0.04 0.04 0.01 0.00 0.00 0.00 0.00 04:28:53 02:30:03 lo 1.00 1.00 0.05 0.05 0.00 0.00 0.00 0.00 04:28:53 02:30:03 ens3 0.16 0.03 0.04 0.01 0.00 0.00 0.00 0.00 04:28:53 02:40:03 lo 1.00 1.00 0.05 0.05 0.00 0.00 0.00 0.00 04:28:53 02:40:03 ens3 0.24 0.06 0.07 0.03 0.00 0.00 0.00 0.00 04:28:53 02:50:01 lo 1.00 1.00 0.05 0.05 0.00 0.00 0.00 0.00 04:28:53 02:50:01 ens3 0.23 0.05 0.07 0.03 0.00 0.00 0.00 0.00 04:28:53 03:00:03 lo 1.00 1.00 0.05 0.05 0.00 0.00 0.00 0.00 04:28:53 03:00:03 ens3 0.29 0.07 0.10 0.05 0.00 0.00 0.00 0.00 04:28:53 03:10:02 lo 1.00 1.00 0.05 0.05 0.00 0.00 0.00 0.00 04:28:53 03:10:02 ens3 0.15 0.03 0.04 0.01 0.00 0.00 0.00 0.00 04:28:53 03:20:01 lo 1.00 1.00 0.05 0.05 0.00 0.00 0.00 0.00 04:28:53 03:20:01 ens3 0.15 0.04 0.04 0.01 0.00 0.00 0.00 0.00 04:28:53 03:30:02 lo 1.00 1.00 0.05 0.05 0.00 0.00 0.00 0.00 04:28:53 03:30:02 ens3 0.29 0.07 0.10 0.05 0.00 0.00 0.00 0.00 04:28:53 03:40:04 lo 1.00 1.00 0.05 0.05 0.00 0.00 0.00 0.00 04:28:53 03:40:04 ens3 0.23 0.05 0.07 0.03 0.00 0.00 0.00 0.00 04:28:53 03:50:01 lo 1.00 1.00 0.05 0.05 0.00 0.00 0.00 0.00 04:28:53 03:50:01 ens3 0.16 0.02 0.04 0.01 0.00 0.00 0.00 0.00 04:28:53 04:00:07 lo 1.00 1.00 0.05 0.05 0.00 0.00 0.00 0.00 04:28:53 04:00:07 ens3 0.18 0.06 0.06 0.03 0.00 0.00 0.00 0.00 04:28:53 04:10:07 lo 1.00 1.00 0.05 0.05 0.00 0.00 0.00 0.00 04:28:53 04:10:07 ens3 0.14 0.02 0.04 0.01 0.00 0.00 0.00 0.00 04:28:53 04:20:01 lo 1.00 1.00 0.05 0.05 0.00 0.00 0.00 0.00 04:28:53 04:20:01 ens3 0.14 0.03 0.04 0.01 0.00 0.00 0.00 0.00 04:28:53 Average: lo 1.00 1.00 0.05 0.05 0.00 0.00 0.00 0.00 04:28:53 Average: ens3 0.38 0.07 0.22 0.04 0.00 0.00 0.00 0.00 04:28:53 04:28:53 04:28:53 ---> sar -P ALL: 04:28:53 Linux 6.8.0-90-generic (prd-queue-disttest-2c-1g-22936) 06/03/26 _x86_64_ (2 CPU) 04:28:53 04:28:53 00:35:10 LINUX RESTART (2 CPU) 04:28:53 04:28:53 00:40:32 CPU %user %nice %system %iowait %steal %idle 04:28:53 00:50:00 all 0.28 0.00 0.07 0.01 0.02 99.62 04:28:53 00:50:00 0 0.47 0.00 0.05 0.01 0.01 99.46 04:28:53 00:50:00 1 0.08 0.00 0.10 0.00 0.03 99.78 04:28:53 01:00:12 all 0.28 0.00 0.17 0.01 0.02 99.52 04:28:53 01:00:12 0 0.44 0.00 0.15 0.00 0.01 99.39 04:28:53 01:00:12 1 0.12 0.00 0.19 0.01 0.03 99.65 04:28:53 01:10:22 all 0.23 0.00 0.08 0.02 0.02 99.65 04:28:53 01:10:22 0 0.05 0.00 0.09 0.00 0.02 99.84 04:28:53 01:10:22 1 0.40 0.00 0.08 0.03 0.02 99.47 04:28:53 01:20:00 all 0.23 0.00 0.07 0.01 0.02 99.67 04:28:53 01:20:00 0 0.05 0.00 0.08 0.01 0.02 99.85 04:28:53 01:20:00 1 0.42 0.00 0.06 0.01 0.01 99.50 04:28:53 01:30:00 all 0.38 0.00 0.19 0.04 0.02 99.37 04:28:53 01:30:00 0 0.52 0.00 0.19 0.02 0.02 99.26 04:28:53 01:30:00 1 0.25 0.00 0.19 0.06 0.03 99.48 04:28:53 01:40:00 all 0.25 0.00 0.06 0.01 0.02 99.66 04:28:53 01:40:00 0 0.28 0.00 0.06 0.00 0.02 99.64 04:28:53 01:40:00 1 0.23 0.00 0.07 0.01 0.02 99.68 04:28:53 01:50:01 all 0.21 0.00 0.07 0.01 0.02 99.69 04:28:53 01:50:01 0 0.36 0.00 0.06 0.01 0.02 99.55 04:28:53 01:50:01 1 0.05 0.00 0.08 0.00 0.02 99.84 04:28:53 02:00:02 all 0.26 0.00 0.16 0.02 0.02 99.55 04:28:53 02:00:02 0 0.45 0.00 0.14 0.03 0.01 99.37 04:28:53 02:00:02 1 0.06 0.00 0.18 0.01 0.03 99.72 04:28:53 02:10:02 all 0.25 0.00 0.06 0.01 0.02 99.66 04:28:53 02:10:02 0 0.28 0.00 0.05 0.01 0.02 99.64 04:28:53 02:10:02 1 0.22 0.00 0.08 0.01 0.02 99.67 04:28:53 02:20:01 all 0.25 0.00 0.06 0.00 0.02 99.66 04:28:53 02:20:01 0 0.45 0.00 0.04 0.00 0.01 99.49 04:28:53 02:20:01 1 0.05 0.00 0.09 0.01 0.03 99.83 04:28:53 02:30:03 all 0.21 0.00 0.17 0.01 0.02 99.59 04:28:53 02:30:03 0 0.23 0.00 0.16 0.01 0.02 99.57 04:28:53 02:30:03 1 0.19 0.00 0.17 0.00 0.03 99.62 04:28:53 04:28:53 02:30:03 CPU %user %nice %system %iowait %steal %idle 04:28:53 02:40:03 all 0.25 0.17 0.14 0.13 0.02 99.29 04:28:53 02:40:03 0 0.06 0.33 0.21 0.25 0.03 99.13 04:28:53 02:40:03 1 0.44 0.01 0.07 0.01 0.02 99.45 04:28:53 02:50:01 all 0.26 0.00 0.08 0.01 0.02 99.63 04:28:53 02:50:01 0 0.27 0.00 0.06 0.01 0.02 99.65 04:28:53 02:50:01 1 0.26 0.00 0.10 0.01 0.02 99.61 04:28:53 03:00:03 all 0.27 0.00 0.16 0.01 0.02 99.54 04:28:53 03:00:03 0 0.41 0.00 0.14 0.01 0.01 99.42 04:28:53 03:00:03 1 0.13 0.00 0.17 0.00 0.02 99.67 04:28:53 03:10:02 all 0.25 0.00 0.07 0.01 0.02 99.66 04:28:53 03:10:02 0 0.07 0.00 0.10 0.01 0.02 99.80 04:28:53 03:10:02 1 0.42 0.00 0.05 0.00 0.01 99.51 04:28:53 03:20:01 all 0.27 0.00 0.07 0.01 0.02 99.63 04:28:53 03:20:01 0 0.44 0.00 0.05 0.01 0.02 99.47 04:28:53 03:20:01 1 0.09 0.00 0.09 0.01 0.02 99.80 04:28:53 03:30:02 all 0.31 0.00 0.16 0.01 0.02 99.50 04:28:53 03:30:02 0 0.37 0.00 0.14 0.02 0.02 99.46 04:28:53 03:30:02 1 0.24 0.00 0.19 0.00 0.02 99.55 04:28:53 03:40:04 all 0.22 0.00 0.08 0.01 0.02 99.67 04:28:53 03:40:04 0 0.25 0.00 0.06 0.01 0.01 99.66 04:28:53 03:40:04 1 0.19 0.00 0.09 0.00 0.02 99.69 04:28:53 03:50:01 all 0.18 0.00 0.08 0.01 0.02 99.71 04:28:53 03:50:01 0 0.32 0.00 0.07 0.01 0.01 99.58 04:28:53 03:50:01 1 0.05 0.00 0.09 0.00 0.02 99.85 04:28:53 04:00:07 all 0.27 0.00 0.17 0.01 0.02 99.54 04:28:53 04:00:07 0 0.25 0.00 0.17 0.00 0.01 99.56 04:28:53 04:00:07 1 0.29 0.00 0.17 0.01 0.02 99.52 04:28:53 04:10:07 all 0.28 0.00 0.08 0.01 0.02 99.61 04:28:53 04:10:07 0 0.07 0.00 0.11 0.01 0.03 99.78 04:28:53 04:10:07 1 0.49 0.00 0.05 0.01 0.01 99.44 04:28:53 04:20:01 all 0.25 0.00 0.07 0.01 0.02 99.65 04:28:53 04:20:01 0 0.05 0.00 0.09 0.01 0.03 99.82 04:28:53 04:20:01 1 0.46 0.00 0.05 0.00 0.01 99.48 04:28:53 04:28:53 Average: CPU %user %nice %system %iowait %steal %idle 04:28:53 Average: all 0.26 0.01 0.11 0.02 0.02 99.60 04:28:53 Average: 0 0.28 0.01 0.10 0.02 0.02 99.56 04:28:53 Average: 1 0.23 0.00 0.11 0.01 0.02 99.63 04:28:53 04:28:53 04:28:53