Message boards : CMS Application : CMS is using failover frontier servers
Message board moderation

To post messages, you must log in.

AuthorMessage
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2442
Credit: 230,337,530
RAC: 117,989
Message 41343 - Posted: 24 Jan 2020, 8:24:42 UTC
Last modified: 24 Jan 2020, 9:13:19 UTC

CMS tasks are configured to get frontier data from cms-frontier.openhtc.io and to use the following servers as failovers:
cms1-frontier.openhtc.io
cms2-frontier.openhtc.io
cms3-frontier.openhtc.io
cms4-frontier.openhtc.io

Since yesterday I notice that roughly 30% of all frontier request are send to the failover servers.
This should be investigated.

<edit>
The task's logfile looks like this:
cmsRun  -j FrameworkJobReport.xml PSet.py
warn  [frontier.c:1014]: Request 125 on chan 1 failed at Fri Jan 24 08:44:25 2020: -8 [fn-htclient.c:438]: server error (HTTP/1.1 530 Unassigned) proxy=</hostname_censored_by_volunteer>[</IP_censored_by_volunteer>] server=cms-frontier.openhtc.io
warn  [frontier.c:1136]: Trying next server cms1-frontier.openhtc.io
warn  [frontier.c:1014]: Request 250 on chan 1 failed at Fri Jan 24 08:44:25 2020: -8 [fn-htclient.c:438]: server error (HTTP/1.1 530 Unassigned) proxy=</hostname_censored_by_volunteer>[</IP_censored_by_volunteer>] server=cms1-frontier.openhtc.io
warn  [frontier.c:1136]: Trying next server cms2-frontier.openhtc.io
warn  [frontier.c:1014]: Request 450 on chan 1 failed at Fri Jan 24 08:44:26 2020: -8 [fn-htclient.c:438]: server error (HTTP/1.1 530 Unassigned) proxy=</hostname_censored_by_volunteer>[</IP_censored_by_volunteer>] server=cms2-frontier.openhtc.io
warn  [frontier.c:1136]: Trying next server cms3-frontier.openhtc.io
warn  [frontier.c:1014]: Request 531 on chan 1 failed at Fri Jan 24 08:44:26 2020: -8 [fn-htclient.c:438]: server error (HTTP/1.1 530 Unassigned) proxy=</hostname_censored_by_volunteer>[</IP_censored_by_volunteer>] server=cms3-frontier.openhtc.io
warn  [frontier.c:1136]: Trying next server cms4-frontier.openhtc.io
warn  [frontier.c:1014]: Request 542 on chan 1 failed at Fri Jan 24 08:44:26 2020: -8 [fn-htclient.c:438]: server error (HTTP/1.1 530 Unassigned) proxy=</hostname_censored_by_volunteer>[</IP_censored_by_volunteer>] server=cms4-frontier.openhtc.io
warn  [frontier.c:1127]: Trying direct connect to server cms-frontier.openhtc.io
warn  [frontier.c:1014]: Request 604 on chan 1 failed at Fri Jan 24 08:44:27 2020: -8 [fn-htclient.c:438]: server error (HTTP/1.1 530 Unassigned) proxy= server=cms-frontier.openhtc.io[172.64.173.7]
warn  [frontier.c:1136]: Trying next server cms-frontier.openhtc.io[172.64.172.7]
warn  [frontier.c:1014]: Request 605 on chan 1 failed at Fri Jan 24 08:44:27 2020: -8 [fn-htclient.c:438]: server error (HTTP/1.1 530 Unassigned) proxy= server=cms-frontier.openhtc.io[172.64.172.7]
warn  [frontier.c:1136]: Trying next server cms-frontier.openhtc.io[2606:4700:e4::ac40:ac07]
warn  [frontier.c:1014]: Request 606 on chan 1 failed at Fri Jan 24 08:44:27 2020: -9 [fn-socket.c:85]: network error on connect to 2606:4700:e4::ac40:ac07: Network is unreachable
warn  [frontier.c:1136]: Trying next server cms-frontier.openhtc.io[2606:4700:e4::ac40:ad07]
warn  [frontier.c:1014]: Request 607 on chan 1 failed at Fri Jan 24 08:44:27 2020: -9 [fn-socket.c:85]: network error on connect to 2606:4700:e4::ac40:ad07: Network is unreachable
warn  [frontier.c:1136]: Trying next server cms1-frontier.openhtc.io
warn  [frontier.c:1014]: Request 608 on chan 1 failed at Fri Jan 24 08:44:37 2020: -6 [fn-socket.c:239]: read from 172.64.172.7 timed out after 10 seconds
warn  [frontier.c:1136]: Trying next server cms1-frontier.openhtc.io[172.64.173.7]
warn  [frontier.c:1014]: Request 609 on chan 1 failed at Fri Jan 24 08:44:47 2020: -6 [fn-socket.c:239]: read from 172.64.173.7 timed out after 10 seconds
warn  [frontier.c:1136]: Trying next server cms1-frontier.openhtc.io[2606:4700:e4::ac40:ad07]
warn  [frontier.c:1014]: Request 610 on chan 1 failed at Fri Jan 24 08:44:47 2020: -9 [fn-socket.c:85]: network error on connect to 2606:4700:e4::ac40:ad07: Network is unreachable
warn  [frontier.c:1136]: Trying next server cms1-frontier.openhtc.io[2606:4700:e4::ac40:ac07]
warn  [frontier.c:1014]: Request 611 on chan 1 failed at Fri Jan 24 08:44:47 2020: -9 [fn-socket.c:85]: network error on connect to 2606:4700:e4::ac40:ac07: Network is unreachable
warn  [frontier.c:1136]: Trying next server cms2-frontier.openhtc.io
warn  [frontier.c:1014]: Request 612 on chan 1 failed at Fri Jan 24 08:44:57 2020: -6 [fn-socket.c:239]: read from 172.64.173.7 timed out after 10 seconds
warn  [frontier.c:1136]: Trying next server cms2-frontier.openhtc.io[172.64.172.7]
warn  [frontier.c:1014]: Request 613 on chan 1 failed at Fri Jan 24 08:45:07 2020: -6 [fn-socket.c:239]: read from 172.64.172.7 timed out after 10 seconds
warn  [frontier.c:1136]: Trying next server cms2-frontier.openhtc.io[2606:4700:e4::ac40:ad07]
warn  [frontier.c:1014]: Request 614 on chan 1 failed at Fri Jan 24 08:45:07 2020: -9 [fn-socket.c:85]: network error on connect to 2606:4700:e4::ac40:ad07: Network is unreachable
warn  [frontier.c:1136]: Trying next server cms2-frontier.openhtc.io[2606:4700:e4::ac40:ac07]
warn  [frontier.c:1014]: Request 615 on chan 1 failed at Fri Jan 24 08:45:07 2020: -9 [fn-socket.c:85]: network error on connect to 2606:4700:e4::ac40:ac07: Network is unreachable
warn  [frontier.c:1136]: Trying next server cms3-frontier.openhtc.io
warn  [frontier.c:1014]: Request 616 on chan 1 failed at Fri Jan 24 08:45:17 2020: -6 [fn-socket.c:239]: read from 172.64.173.7 timed out after 10 seconds
warn  [frontier.c:1136]: Trying next server cms3-frontier.openhtc.io[172.64.172.7]
warn  [frontier.c:1014]: Request 617 on chan 1 failed at Fri Jan 24 08:45:27 2020: -6 [fn-socket.c:239]: read from 172.64.172.7 timed out after 10 seconds
warn  [frontier.c:1136]: Trying next server cms3-frontier.openhtc.io[2606:4700:e4::ac40:ad07]
warn  [frontier.c:1014]: Request 618 on chan 1 failed at Fri Jan 24 08:45:27 2020: -9 [fn-socket.c:85]: network error on connect to 2606:4700:e4::ac40:ad07: Network is unreachable
warn  [frontier.c:1136]: Trying next server cms3-frontier.openhtc.io[2606:4700:e4::ac40:ac07]
warn  [frontier.c:1014]: Request 619 on chan 1 failed at Fri Jan 24 08:45:27 2020: -9 [fn-socket.c:85]: network error on connect to 2606:4700:e4::ac40:ac07: Network is unreachable
warn  [frontier.c:1136]: Trying next server cms4-frontier.openhtc.io
warn  [frontier.c:1014]: Request 620 on chan 1 failed at Fri Jan 24 08:45:37 2020: -6 [fn-socket.c:239]: read from 172.64.172.7 timed out after 10 seconds
warn  [frontier.c:1136]: Trying next server cms4-frontier.openhtc.io[172.64.173.7]
warn  [frontier.c:1014]: Request 621 on chan 1 failed at Fri Jan 24 08:45:47 2020: -6 [fn-socket.c:239]: read from 172.64.173.7 timed out after 10 seconds
warn  [frontier.c:1136]: Trying next server cms4-frontier.openhtc.io[2606:4700:e4::ac40:ad07]
warn  [frontier.c:1014]: Request 622 on chan 1 failed at Fri Jan 24 08:45:47 2020: -9 [fn-socket.c:85]: network error on connect to 2606:4700:e4::ac40:ad07: Network is unreachable
warn  [frontier.c:1136]: Trying next server cms4-frontier.openhtc.io[2606:4700:e4::ac40:ac07]
warn  [frontier.c:1014]: Request 623 on chan 1 failed at Fri Jan 24 08:45:47 2020: -9 [fn-socket.c:85]: network error on connect to 2606:4700:e4::ac40:ac07: Network is unreachable
error [frontier.c:1159]: No more servers/proxies. Last error was: Request 623 on chan 1 failed at Fri Jan 24 08:45:47 2020: -9 [fn-socket.c:85]: network error on connect to 2606:4700:e4::ac40:ac07: Network is unreachable
----- Begin Fatal Exception 24-Jan-2020 08:45:47 UTC-----------------------
An exception of category 'StdException' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing ESSource: class=PoolDBESSource label='GlobalTag'
Exception Message:
A std::exception was thrown.
Can not get data (Additional Information: [frontier.c:1159]: No more servers/proxies. Last error was: Request 623 on chan 1 failed at Fri Jan 24 08:45:47 2020: -9 [fn-socket.c:85]: network error on connect to 2606:4700:e4::ac40:ac07: Network is unreachable) ( CORAL : "coral::FrontierAccess::Statement::execute" from "CORAL/RelationalPlugins/frontier" )
----- End Fatal Exception -------------------------------------------------
Complete
process id is 260 status is 66

</edit>
ID: 41343 · Report as offensive     Reply Quote

Message boards : CMS Application : CMS is using failover frontier servers


©2024 CERN