Started by user Robert Varga Running as SYSTEM [EnvInject] - Loading node environment variables. Building remotely on prd-centos8-builder-4c-4g-4506 (centos8-builder-4c-4g) in workspace /w/workspace/autorelease-gerrit-branch-lock-argon [ssh-agent] Looking for ssh-agent implementation... [ssh-agent] Exec ssh-agent (binary ssh-agent on a remote machine) $ ssh-agent SSH_AUTH_SOCK=/tmp/ssh-4Q7Wf6ZBzRJn/agent.5537 SSH_AGENT_PID=5538 [ssh-agent] Started. Running ssh-add (command line suppressed) Identity added: /w/workspace/autorelease-gerrit-branch-lock-argon@tmp/private_key_4618172484669686659.key (/w/workspace/autorelease-gerrit-branch-lock-argon@tmp/private_key_4618172484669686659.key) [ssh-agent] Using credentials jenkins (jenkins-ssh) The recommended git tool is: NONE using credential jenkins-ssh Wiping out workspace first. Cloning the remote Git repository Cloning repository git://devvexx.opendaylight.org/mirror/releng/autorelease > git init /w/workspace/autorelease-gerrit-branch-lock-argon # timeout=10 Fetching upstream changes from git://devvexx.opendaylight.org/mirror/releng/autorelease > git --version # timeout=10 > git --version # 'git version 2.43.0' using GIT_SSH to set credentials jenkins-ssh [INFO] Currently running in a labeled security context [INFO] Currently SELinux is 'enforcing' on the host > /usr/bin/chcon --type=ssh_home_t /w/workspace/autorelease-gerrit-branch-lock-argon@tmp/jenkins-gitclient-ssh2426470937209405271.key Verifying host key using known hosts file You're using 'Known hosts file' strategy to verify ssh host keys, but your known_hosts file does not exist, please go to 'Manage Jenkins' -> 'Security' -> 'Git Host Key Verification Configuration' and configure host key verification. > git fetch --tags --force --progress -- git://devvexx.opendaylight.org/mirror/releng/autorelease +refs/heads/*:refs/remotes/origin/* # timeout=10 > git config remote.origin.url git://devvexx.opendaylight.org/mirror/releng/autorelease # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch > git rev-parse refs/remotes/origin/stable/argon^{commit} # timeout=10 Checking out Revision 236e1d68c513032c849cf5f357b1758e04e627c5 (refs/remotes/origin/stable/argon) > git config core.sparsecheckout # timeout=10 > git checkout -f 236e1d68c513032c849cf5f357b1758e04e627c5 # timeout=10 Commit message: "Update git submodules" > git rev-list --no-walk 1ecbaf4c56c985bb0fecf4380222fdab51d13ae9 # timeout=10 provisioning config files... copy managed file [npmrc] to file:/home/jenkins/.npmrc copy managed file [pipconf] to file:/home/jenkins/.config/pip/pip.conf [autorelease-gerrit-branch-lock-argon] $ /bin/bash /tmp/jenkins8885818755073382997.sh ---> autorelease-gerrit-branch-lock.sh Cloning into 'All-Projects'... Warning: Permanently added the RSA host key for IP address '[13.39.230.180]:29418' to the list of known hosts. Total 1738 (delta 603), reused 1738 (delta 603) Note: switching to '24a3ec21ce4acdf355c15b50ac8bf065f456099f'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by switching back to a branch. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -c with the switch command. Example: git switch -c Or undo this operation with: git switch - Turn off this advice by setting config variable advice.detachedHead to false Warning: Permanently added the RSA host key for IP address '[13.39.230.180]:29418' to the list of known hosts. From ssh://git.opendaylight.org:29418/All-Projects * [new ref] refs/meta/config -> config Switched to branch 'config' Warning: Permanently added the RSA host key for IP address '[13.39.230.180]:29418' to the list of known hosts. Warning: Permanently added the RSA host key for IP address '[13.39.230.180]:29418' to the list of known hosts. Warning: Permanently added the RSA host key for IP address '[35.181.223.58]:29418' to the list of known hosts. + case $mode in + echo 'INFO: Unlocking branch: stable/argon' INFO: Unlocking branch: stable/argon + git config -f project.config --remove-section access.refs/heads/stable/argon + git commit -asm 'Chore: Unlock branch stable/argon' [config 6f4c8ea] Chore: Unlock branch stable/argon 1 file changed, 6 deletions(-) + git diff HEAD~1 diff --git a/project.config b/project.config index 68e98bf..268f5c0 100644 --- a/project.config +++ b/project.config @@ -148,9 +148,3 @@ removeReviewer = group Release Engineering Team label-Code-Review = -2..+2 group Release Engineering Team label-Verified = -1..+1 group Release Engineering Team -[access "refs/heads/stable/argon"] - exclusiveGroupPermissions = submit - submit = group Release Engineering Team - removeReviewer = group Release Engineering Team - label-Code-Review = -2..+2 group Release Engineering Team - label-Verified = -1..+1 group Release Engineering Team + '[' false = true ']' + echo 'INFO: Pushing changes to the repository.' INFO: Pushing changes to the repository. + git push origin HEAD:refs/meta/config Warning: Permanently added the RSA host key for IP address '[13.39.230.180]:29418' to the list of known hosts. remote: remote: Processing changes: refs: 1 remote: Processing changes: refs: 1, done To ssh://git.opendaylight.org:29418/All-Projects 24a3ec2..6f4c8ea HEAD -> refs/meta/config $ ssh-agent -k unset SSH_AUTH_SOCK; unset SSH_AGENT_PID; echo Agent pid 5538 killed; [ssh-agent] Stopped. [PostBuildScript] - [INFO] Executing post build scripts. [autorelease-gerrit-branch-lock-argon] $ /bin/bash /tmp/jenkins15797697307669150448.sh ---> sysstat.sh [autorelease-gerrit-branch-lock-argon] $ /bin/bash /tmp/jenkins4528188825859229630.sh ---> package-listing.sh ++ facter osfamily ++ tr '[:upper:]' '[:lower:]' + OS_FAMILY=redhat + workspace=/w/workspace/autorelease-gerrit-branch-lock-argon + START_PACKAGES=/tmp/packages_start.txt + END_PACKAGES=/tmp/packages_end.txt + DIFF_PACKAGES=/tmp/packages_diff.txt + PACKAGES=/tmp/packages_start.txt + '[' /w/workspace/autorelease-gerrit-branch-lock-argon ']' + PACKAGES=/tmp/packages_end.txt + case "${OS_FAMILY}" in + rpm -qa + sort + '[' -f /tmp/packages_start.txt ']' + '[' -f /tmp/packages_end.txt ']' + diff /tmp/packages_start.txt /tmp/packages_end.txt + '[' /w/workspace/autorelease-gerrit-branch-lock-argon ']' + mkdir -p /w/workspace/autorelease-gerrit-branch-lock-argon/archives/ + cp -f /tmp/packages_diff.txt /tmp/packages_end.txt /tmp/packages_start.txt /w/workspace/autorelease-gerrit-branch-lock-argon/archives/ [autorelease-gerrit-branch-lock-argon] $ /bin/bash /tmp/jenkins14156967054283624868.sh ---> capture-instance-metadata.sh Setup pyenv: system * 3.8.13 (set by /opt/pyenv/version) * 3.9.13 (set by /opt/pyenv/version) * 3.10.13 (set by /opt/pyenv/version) * 3.11.7 (set by /opt/pyenv/version) lf-activate-venv(): INFO: Creating python3 venv at /tmp/venv-i9e8 lf-activate-venv(): INFO: Save venv in file: /tmp/.os_lf_venv lf-activate-venv(): INFO: Installing: lftools lf-activate-venv(): INFO: Adding /tmp/venv-i9e8/bin to PATH INFO: Running in OpenStack, capturing instance metadata [autorelease-gerrit-branch-lock-argon] $ /bin/bash /tmp/jenkins10107727858583731952.sh provisioning config files... Could not find credentials [logs] for autorelease-gerrit-branch-lock-argon #53 copy managed file [jenkins-log-archives-settings] to file:/w/workspace/autorelease-gerrit-branch-lock-argon@tmp/config9491490640669881135tmp Regular expression run condition: Expression=[^.*logs-s3.*], Label=[odl-logs-s3-cloudfront-index] Run condition [Regular expression match] enabling perform for step [Provide Configuration files] provisioning config files... copy managed file [jenkins-s3-log-ship] to file:/home/jenkins/.aws/credentials [EnvInject] - Injecting environment variables from a build step. [EnvInject] - Injecting as environment variables the properties content SERVER_ID=logs [EnvInject] - Variables injected successfully. [autorelease-gerrit-branch-lock-argon] $ /bin/bash /tmp/jenkins12977941714892760153.sh ---> create-netrc.sh WARN: Log server credential not found. [autorelease-gerrit-branch-lock-argon] $ /bin/bash /tmp/jenkins16503852187601970456.sh ---> python-tools-install.sh Setup pyenv: system 3.8.13 3.9.13 3.10.13 * 3.11.7 (set by /w/workspace/autorelease-gerrit-branch-lock-argon/.python-version) lf-activate-venv(): INFO: Reuse venv:/tmp/venv-i9e8 from file:/tmp/.os_lf_venv lf-activate-venv(): INFO: Installing: lftools lf-activate-venv(): INFO: Adding /tmp/venv-i9e8/bin to PATH Generating Requirements File Python 3.11.7 pip 24.0 from /tmp/venv-i9e8/lib/python3.11/site-packages/pip (python 3.11) appdirs==1.4.4 argcomplete==3.2.3 aspy.yaml==1.3.0 attrs==23.2.0 autopage==0.5.2 beautifulsoup4==4.12.3 boto3==1.34.77 botocore==1.34.77 bs4==0.0.2 cachetools==5.3.3 certifi==2024.2.2 cffi==1.16.0 cfgv==3.4.0 chardet==5.2.0 charset-normalizer==3.3.2 click==8.1.7 cliff==4.6.0 cmd2==2.4.3 cryptography==3.3.2 debtcollector==3.0.0 decorator==5.1.1 defusedxml==0.7.1 Deprecated==1.2.14 distlib==0.3.8 dnspython==2.6.1 docker==4.2.2 dogpile.cache==1.3.2 email_validator==2.1.1 filelock==3.13.3 future==1.0.0 gitdb==4.0.11 GitPython==3.1.43 google-auth==2.29.0 httplib2==0.22.0 identify==2.5.35 idna==3.6 importlib-resources==1.5.0 iso8601==2.1.0 Jinja2==3.1.3 jmespath==1.0.1 jsonpatch==1.33 jsonpointer==2.4 jsonschema==4.21.1 jsonschema-specifications==2023.12.1 keystoneauth1==5.6.0 kubernetes==29.0.0 lftools==0.37.10 lxml==5.2.1 MarkupSafe==2.1.5 msgpack==1.0.8 multi_key_dict==2.0.3 munch==4.0.0 netaddr==1.2.1 netifaces==0.11.0 niet==1.4.2 nodeenv==1.8.0 oauth2client==4.1.3 oauthlib==3.2.2 openstacksdk==3.0.0 os-client-config==2.1.0 os-service-types==1.7.0 osc-lib==3.0.1 oslo.config==9.4.0 oslo.context==5.5.0 oslo.i18n==6.3.0 oslo.log==5.5.1 oslo.serialization==5.4.0 oslo.utils==7.1.0 packaging==24.0 pbr==6.0.0 platformdirs==4.2.0 prettytable==3.10.0 pyasn1==0.6.0 pyasn1_modules==0.4.0 pycparser==2.22 pygerrit2==2.0.15 PyGithub==2.3.0 pyinotify==0.9.6 PyJWT==2.8.0 PyNaCl==1.5.0 pyparsing==2.4.7 pyperclip==1.8.2 pyrsistent==0.20.0 python-cinderclient==9.5.0 python-dateutil==2.9.0.post0 python-heatclient==3.5.0 python-jenkins==1.8.2 python-keystoneclient==5.4.0 python-magnumclient==4.4.0 python-novaclient==18.6.0 python-openstackclient==6.6.0 python-swiftclient==4.5.0 PyYAML==6.0.1 referencing==0.34.0 requests==2.31.0 requests-oauthlib==2.0.0 requestsexceptions==1.4.0 rfc3986==2.0.0 rpds-py==0.18.0 rsa==4.9 ruamel.yaml==0.18.6 ruamel.yaml.clib==0.2.8 s3transfer==0.10.1 simplejson==3.19.2 six==1.16.0 smmap==5.0.1 soupsieve==2.5 stevedore==5.2.0 tabulate==0.9.0 toml==0.10.2 tomlkit==0.12.4 tqdm==4.66.2 typing_extensions==4.10.0 tzdata==2024.1 urllib3==1.26.18 virtualenv==20.25.1 wcwidth==0.2.13 websocket-client==1.7.0 wrapt==1.16.0 xdg==6.0.0 xmltodict==0.13.0 yq==3.2.3 [autorelease-gerrit-branch-lock-argon] $ /bin/bash /tmp/jenkins119357849485310199.sh ---> sudo-logs.sh Archiving 'sudo' log.. [autorelease-gerrit-branch-lock-argon] $ /bin/bash /tmp/jenkins8689415246590644245.sh ---> job-cost.sh Setup pyenv: system 3.8.13 3.9.13 3.10.13 * 3.11.7 (set by /w/workspace/autorelease-gerrit-branch-lock-argon/.python-version) lf-activate-venv(): INFO: Reuse venv:/tmp/venv-i9e8 from file:/tmp/.os_lf_venv lf-activate-venv(): INFO: Installing: zipp==1.1.0 python-openstackclient urllib3~=1.26.15 lf-activate-venv(): INFO: Adding /tmp/venv-i9e8/bin to PATH INFO: No Stack... INFO: Retrieving Pricing Info for: v3-standard-4 INFO: Archiving Costs [autorelease-gerrit-branch-lock-argon] $ /bin/bash -l /tmp/jenkins4636270511010376949.sh ---> logs-deploy.sh Setup pyenv: system 3.8.13 3.9.13 3.10.13 * 3.11.7 (set by /w/workspace/autorelease-gerrit-branch-lock-argon/.python-version) lf-activate-venv(): INFO: Reuse venv:/tmp/venv-i9e8 from file:/tmp/.os_lf_venv lf-activate-venv(): INFO: Installing: lftools lf-activate-venv(): INFO: Adding /tmp/venv-i9e8/bin to PATH WARNING: Nexus logging server not set INFO: S3 path logs/releng/vex-yul-odl-jenkins-1/autorelease-gerrit-branch-lock-argon/53/ INFO: archiving logs to S3 ---> uname -a: Linux prd-centos8-builder-4c-4g-4506.novalocal 4.18.0-547.el8.x86_64 #1 SMP Wed Mar 20 00:35:01 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux ---> lscpu: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 4 On-line CPU(s) list: 0-3 Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 4 NUMA node(s): 1 Vendor ID: AuthenticAMD CPU family: 23 Model: 49 Model name: AMD EPYC-Rome Processor Stepping: 0 CPU MHz: 2799.998 BogoMIPS: 5599.99 Virtualization: AMD-V Hypervisor vendor: KVM Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 512K L3 cache: 16384K NUMA node0 CPU(s): 0-3 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core ssbd ibrs ibpb stibp vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves clzero xsaveerptr wbnoinvd arat npt nrip_save umip rdpid arch_capabilities ---> nproc: 4 ---> df -h: Filesystem Size Used Avail Use% Mounted on devtmpfs 7.7G 0 7.7G 0% /dev tmpfs 7.7G 0 7.7G 0% /dev/shm tmpfs 7.7G 17M 7.7G 1% /run tmpfs 7.7G 0 7.7G 0% /sys/fs/cgroup /dev/vda1 80G 8.7G 72G 11% / tmpfs 1.6G 0 1.6G 0% /run/user/1001 ---> free -m: total used free shared buff/cache available Mem: 15761 716 12821 19 2222 14718 Swap: 1023 0 1023 ---> ip addr: 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: mtu 1458 qdisc mq state UP group default qlen 1000 link/ether fa:16:3e:8a:7d:2f brd ff:ff:ff:ff:ff:ff altname enp0s3 altname ens3 inet 10.30.171.100/23 brd 10.30.171.255 scope global dynamic noprefixroute eth0 valid_lft 85923sec preferred_lft 85923sec inet6 fe80::f816:3eff:fe8a:7d2f/64 scope link valid_lft forever preferred_lft forever 3: docker0: mtu 1458 qdisc noqueue state DOWN group default link/ether 02:42:6f:66:67:d3 brd ff:ff:ff:ff:ff:ff inet 10.250.0.254/24 brd 10.250.0.255 scope global docker0 valid_lft forever preferred_lft forever ---> sar -b -r -n DEV: Linux 4.18.0-547.el8.x86_64 (centos-stream-8-builder-140785a5-b754-4c0c-b31b-2b4ade92ecf9.nov) 04/03/2024 _x86_64_ (4 CPU) 23:05:00 LINUX RESTART (4 CPU) 11:06:02 PM tps rtps wtps bread/s bwrtn/s 11:07:02 PM 2.22 0.00 2.22 0.00 634.99 11:08:01 PM 0.98 0.02 0.97 2.17 13.89 11:09:01 PM 0.42 0.03 0.38 0.53 3.68 11:10:01 PM 0.43 0.00 0.43 0.00 6.01 11:11:01 PM 121.38 2.97 118.41 472.91 2010.21 11:12:01 PM 68.97 14.10 54.87 2488.79 4044.46 11:13:01 PM 84.40 0.77 83.64 69.19 7242.19 Average: 40.01 2.57 37.44 435.40 2001.44 11:06:02 PM kbmemfree kbavail kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty 11:07:02 PM 13761996 15244888 2377448 14.73 2688 1712200 1038848 6.04 158708 1918320 4 11:08:01 PM 13761848 15244808 2377596 14.73 2688 1712268 1038848 6.04 158748 1918456 4 11:09:01 PM 13761688 15244664 2377756 14.73 2688 1712284 1038848 6.04 158832 1918396 4 11:10:01 PM 13760808 15243784 2378636 14.74 2688 1712284 1029400 5.99 158828 1918824 4 11:11:01 PM 13674508 15210932 2464936 15.27 2688 1765620 1068816 6.22 174264 1989444 7932 11:12:01 PM 13259172 15084716 2880272 17.85 2688 2041044 1154080 6.71 374716 2152904 135320 11:13:01 PM 13182700 15124144 2956744 18.32 2688 2151032 1086452 6.32 379620 2201444 97524 Average: 13594674 15199705 2544770 15.77 2688 1829533 1065042 6.20 223388 2002541 34399 11:06:02 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil 11:07:02 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:07:02 PM eth0 0.24 0.12 0.02 0.01 0.00 0.00 0.00 0.00 11:07:02 PM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:08:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:08:01 PM eth0 0.17 0.10 0.01 0.01 0.00 0.00 0.00 0.00 11:08:01 PM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:09:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:09:01 PM eth0 0.32 0.18 0.04 0.01 0.00 0.00 0.00 0.00 11:09:01 PM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:10:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:10:01 PM eth0 0.17 0.12 0.01 0.01 0.00 0.00 0.00 0.00 11:10:01 PM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:11:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:11:01 PM eth0 155.56 83.69 652.73 39.03 0.00 0.00 0.00 0.00 11:11:01 PM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:12:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:12:01 PM eth0 66.82 58.06 695.10 9.69 0.00 0.00 0.00 0.00 11:12:01 PM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:13:01 PM lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 11:13:01 PM eth0 27.95 26.65 235.56 6.56 0.00 0.00 0.00 0.00 11:13:01 PM docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: eth0 36.06 24.24 227.28 7.94 0.00 0.00 0.00 0.00 Average: docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 ---> sar -P ALL: Linux 4.18.0-547.el8.x86_64 (centos-stream-8-builder-140785a5-b754-4c0c-b31b-2b4ade92ecf9.nov) 04/03/2024 _x86_64_ (4 CPU) 23:05:00 LINUX RESTART (4 CPU) 11:06:02 PM CPU %user %nice %system %iowait %steal %idle 11:07:02 PM all 0.16 0.00 0.05 0.26 0.01 99.52 11:07:02 PM 0 0.02 0.00 0.02 0.53 0.00 99.44 11:07:02 PM 1 0.34 0.00 0.05 0.00 0.02 99.59 11:07:02 PM 2 0.24 0.00 0.10 0.53 0.03 99.10 11:07:02 PM 3 0.03 0.00 0.03 0.00 0.00 99.93 11:08:01 PM all 0.18 0.00 0.05 0.09 0.02 99.66 11:08:01 PM 0 0.05 0.00 0.05 0.36 0.02 99.53 11:08:01 PM 1 0.05 0.00 0.05 0.00 0.00 99.90 11:08:01 PM 2 0.54 0.00 0.03 0.00 0.02 99.41 11:08:01 PM 3 0.07 0.00 0.07 0.00 0.03 99.83 11:09:01 PM all 0.16 0.00 0.04 0.00 0.02 99.78 11:09:01 PM 0 0.05 0.00 0.03 0.00 0.02 99.90 11:09:01 PM 1 0.42 0.00 0.05 0.00 0.03 99.50 11:09:01 PM 2 0.10 0.00 0.02 0.00 0.00 99.88 11:09:01 PM 3 0.07 0.00 0.05 0.00 0.03 99.85 11:10:01 PM all 0.13 0.00 0.05 0.00 0.02 99.80 11:10:01 PM 0 0.07 0.00 0.03 0.00 0.02 99.88 11:10:01 PM 1 0.40 0.00 0.08 0.00 0.02 99.50 11:10:01 PM 2 0.05 0.00 0.02 0.00 0.02 99.92 11:10:01 PM 3 0.02 0.00 0.05 0.00 0.02 99.92 11:11:01 PM all 4.42 0.00 1.10 0.83 0.03 93.62 11:11:01 PM 0 3.46 0.00 0.77 0.00 0.03 95.74 11:11:01 PM 1 4.15 0.00 1.96 2.84 0.05 91.00 11:11:01 PM 2 4.42 0.00 0.62 0.15 0.02 94.79 11:11:01 PM 3 5.64 0.00 1.05 0.32 0.03 92.95 11:12:01 PM all 16.98 0.00 2.76 1.53 0.07 78.67 11:12:01 PM 0 1.82 0.00 1.42 0.68 0.07 96.01 11:12:01 PM 1 10.93 0.00 1.97 1.84 0.05 85.21 11:12:01 PM 2 37.03 0.00 4.75 1.59 0.08 56.55 11:12:01 PM 3 18.16 0.00 2.89 2.01 0.07 76.88 11:13:01 PM all 19.82 0.00 2.07 1.34 0.05 76.71 11:13:01 PM 0 2.27 0.00 0.90 0.68 0.05 96.09 11:13:01 PM 1 12.11 0.00 1.62 1.22 0.05 84.99 11:13:01 PM 2 27.15 0.00 2.31 1.44 0.05 69.06 11:13:01 PM 3 37.75 0.00 3.46 2.02 0.07 56.71 Average: all 6.00 0.00 0.88 0.58 0.03 92.51 Average: 0 1.11 0.00 0.46 0.32 0.03 98.08 Average: 1 4.07 0.00 0.83 0.85 0.03 94.22 Average: 2 9.96 0.00 1.12 0.53 0.03 88.35 Average: 3 8.85 0.00 1.09 0.62 0.04 89.40