LHC@home

CMS tasks are configured to get frontier data from cms-frontier.openhtc.io and to use the following servers as failovers:
cms1-frontier.openhtc.io
cms2-frontier.openhtc.io
cms3-frontier.openhtc.io
cms4-frontier.openhtc.io

Since yesterday I notice that roughly 30% of all frontier request are send to the failover servers.
This should be investigated.

<edit>
The task's logfile looks like this:

cmsRun  -j FrameworkJobReport.xml PSet.py
warn  [frontier.c:1014]: Request 125 on chan 1 failed at Fri Jan 24 08:44:25 2020: -8 [fn-htclient.c:438]: server error (HTTP/1.1 530 Unassigned) proxy=</hostname_censored_by_volunteer>[</IP_censored_by_volunteer>] server=cms-frontier.openhtc.io
warn  [frontier.c:1136]: Trying next server cms1-frontier.openhtc.io
warn  [frontier.c:1014]: Request 250 on chan 1 failed at Fri Jan 24 08:44:25 2020: -8 [fn-htclient.c:438]: server error (HTTP/1.1 530 Unassigned) proxy=</hostname_censored_by_volunteer>[</IP_censored_by_volunteer>] server=cms1-frontier.openhtc.io
warn  [frontier.c:1136]: Trying next server cms2-frontier.openhtc.io
warn  [frontier.c:1014]: Request 450 on chan 1 failed at Fri Jan 24 08:44:26 2020: -8 [fn-htclient.c:438]: server error (HTTP/1.1 530 Unassigned) proxy=</hostname_censored_by_volunteer>[</IP_censored_by_volunteer>] server=cms2-frontier.openhtc.io
warn  [frontier.c:1136]: Trying next server cms3-frontier.openhtc.io
warn  [frontier.c:1014]: Request 531 on chan 1 failed at Fri Jan 24 08:44:26 2020: -8 [fn-htclient.c:438]: server error (HTTP/1.1 530 Unassigned) proxy=</hostname_censored_by_volunteer>[</IP_censored_by_volunteer>] server=cms3-frontier.openhtc.io
warn  [frontier.c:1136]: Trying next server cms4-frontier.openhtc.io
warn  [frontier.c:1014]: Request 542 on chan 1 failed at Fri Jan 24 08:44:26 2020: -8 [fn-htclient.c:438]: server error (HTTP/1.1 530 Unassigned) proxy=</hostname_censored_by_volunteer>[</IP_censored_by_volunteer>] server=cms4-frontier.openhtc.io
warn  [frontier.c:1127]: Trying direct connect to server cms-frontier.openhtc.io
warn  [frontier.c:1014]: Request 604 on chan 1 failed at Fri Jan 24 08:44:27 2020: -8 [fn-htclient.c:438]: server error (HTTP/1.1 530 Unassigned) proxy= server=cms-frontier.openhtc.io[172.64.173.7]
warn  [frontier.c:1136]: Trying next server cms-frontier.openhtc.io[172.64.172.7]
warn  [frontier.c:1014]: Request 605 on chan 1 failed at Fri Jan 24 08:44:27 2020: -8 [fn-htclient.c:438]: server error (HTTP/1.1 530 Unassigned) proxy= server=cms-frontier.openhtc.io[172.64.172.7]
warn  [frontier.c:1136]: Trying next server cms-frontier.openhtc.io[2606:4700:e4::ac40:ac07]
warn  [frontier.c:1014]: Request 606 on chan 1 failed at Fri Jan 24 08:44:27 2020: -9 [fn-socket.c:85]: network error on connect to 2606:4700:e4::ac40:ac07: Network is unreachable
warn  [frontier.c:1136]: Trying next server cms-frontier.openhtc.io[2606:4700:e4::ac40:ad07]
warn  [frontier.c:1014]: Request 607 on chan 1 failed at Fri Jan 24 08:44:27 2020: -9 [fn-socket.c:85]: network error on connect to 2606:4700:e4::ac40:ad07: Network is unreachable
warn  [frontier.c:1136]: Trying next server cms1-frontier.openhtc.io
warn  [frontier.c:1014]: Request 608 on chan 1 failed at Fri Jan 24 08:44:37 2020: -6 [fn-socket.c:239]: read from 172.64.172.7 timed out after 10 seconds
warn  [frontier.c:1136]: Trying next server cms1-frontier.openhtc.io[172.64.173.7]
warn  [frontier.c:1014]: Request 609 on chan 1 failed at Fri Jan 24 08:44:47 2020: -6 [fn-socket.c:239]: read from 172.64.173.7 timed out after 10 seconds
warn  [frontier.c:1136]: Trying next server cms1-frontier.openhtc.io[2606:4700:e4::ac40:ad07]
warn  [frontier.c:1014]: Request 610 on chan 1 failed at Fri Jan 24 08:44:47 2020: -9 [fn-socket.c:85]: network error on connect to 2606:4700:e4::ac40:ad07: Network is unreachable
warn  [frontier.c:1136]: Trying next server cms1-frontier.openhtc.io[2606:4700:e4::ac40:ac07]
warn  [frontier.c:1014]: Request 611 on chan 1 failed at Fri Jan 24 08:44:47 2020: -9 [fn-socket.c:85]: network error on connect to 2606:4700:e4::ac40:ac07: Network is unreachable
warn  [frontier.c:1136]: Trying next server cms2-frontier.openhtc.io
warn  [frontier.c:1014]: Request 612 on chan 1 failed at Fri Jan 24 08:44:57 2020: -6 [fn-socket.c:239]: read from 172.64.173.7 timed out after 10 seconds
warn  [frontier.c:1136]: Trying next server cms2-frontier.openhtc.io[172.64.172.7]
warn  [frontier.c:1014]: Request 613 on chan 1 failed at Fri Jan 24 08:45:07 2020: -6 [fn-socket.c:239]: read from 172.64.172.7 timed out after 10 seconds
warn  [frontier.c:1136]: Trying next server cms2-frontier.openhtc.io[2606:4700:e4::ac40:ad07]
warn  [frontier.c:1014]: Request 614 on chan 1 failed at Fri Jan 24 08:45:07 2020: -9 [fn-socket.c:85]: network error on connect to 2606:4700:e4::ac40:ad07: Network is unreachable
warn  [frontier.c:1136]: Trying next server cms2-frontier.openhtc.io[2606:4700:e4::ac40:ac07]
warn  [frontier.c:1014]: Request 615 on chan 1 failed at Fri Jan 24 08:45:07 2020: -9 [fn-socket.c:85]: network error on connect to 2606:4700:e4::ac40:ac07: Network is unreachable
warn  [frontier.c:1136]: Trying next server cms3-frontier.openhtc.io
warn  [frontier.c:1014]: Request 616 on chan 1 failed at Fri Jan 24 08:45:17 2020: -6 [fn-socket.c:239]: read from 172.64.173.7 timed out after 10 seconds
warn  [frontier.c:1136]: Trying next server cms3-frontier.openhtc.io[172.64.172.7]
warn  [frontier.c:1014]: Request 617 on chan 1 failed at Fri Jan 24 08:45:27 2020: -6 [fn-socket.c:239]: read from 172.64.172.7 timed out after 10 seconds
warn  [frontier.c:1136]: Trying next server cms3-frontier.openhtc.io[2606:4700:e4::ac40:ad07]
warn  [frontier.c:1014]: Request 618 on chan 1 failed at Fri Jan 24 08:45:27 2020: -9 [fn-socket.c:85]: network error on connect to 2606:4700:e4::ac40:ad07: Network is unreachable
warn  [frontier.c:1136]: Trying next server cms3-frontier.openhtc.io[2606:4700:e4::ac40:ac07]
warn  [frontier.c:1014]: Request 619 on chan 1 failed at Fri Jan 24 08:45:27 2020: -9 [fn-socket.c:85]: network error on connect to 2606:4700:e4::ac40:ac07: Network is unreachable
warn  [frontier.c:1136]: Trying next server cms4-frontier.openhtc.io
warn  [frontier.c:1014]: Request 620 on chan 1 failed at Fri Jan 24 08:45:37 2020: -6 [fn-socket.c:239]: read from 172.64.172.7 timed out after 10 seconds
warn  [frontier.c:1136]: Trying next server cms4-frontier.openhtc.io[172.64.173.7]
warn  [frontier.c:1014]: Request 621 on chan 1 failed at Fri Jan 24 08:45:47 2020: -6 [fn-socket.c:239]: read from 172.64.173.7 timed out after 10 seconds
warn  [frontier.c:1136]: Trying next server cms4-frontier.openhtc.io[2606:4700:e4::ac40:ad07]
warn  [frontier.c:1014]: Request 622 on chan 1 failed at Fri Jan 24 08:45:47 2020: -9 [fn-socket.c:85]: network error on connect to 2606:4700:e4::ac40:ad07: Network is unreachable
warn  [frontier.c:1136]: Trying next server cms4-frontier.openhtc.io[2606:4700:e4::ac40:ac07]
warn  [frontier.c:1014]: Request 623 on chan 1 failed at Fri Jan 24 08:45:47 2020: -9 [fn-socket.c:85]: network error on connect to 2606:4700:e4::ac40:ac07: Network is unreachable
error [frontier.c:1159]: No more servers/proxies. Last error was: Request 623 on chan 1 failed at Fri Jan 24 08:45:47 2020: -9 [fn-socket.c:85]: network error on connect to 2606:4700:e4::ac40:ac07: Network is unreachable
----- Begin Fatal Exception 24-Jan-2020 08:45:47 UTC-----------------------
An exception of category 'StdException' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing ESSource: class=PoolDBESSource label='GlobalTag'
Exception Message:
A std::exception was thrown.
Can not get data (Additional Information: [frontier.c:1159]: No more servers/proxies. Last error was: Request 623 on chan 1 failed at Fri Jan 24 08:45:47 2020: -9 [fn-socket.c:85]: network error on connect to 2606:4700:e4::ac40:ac07: Network is unreachable) ( CORAL : "coral::FrontierAccess::Statement::execute" from "CORAL/RelationalPlugins/frontier" )
----- End Fatal Exception -------------------------------------------------
Complete
process id is 260 status is 66

</edit>