LHC@home

From the BOINC point of view it seems ok, my client downloads and uploads ATLAS files:

01-Aug-2017 08:33:12 [LHC@home] Started download of 5W5KDmokewqnSu7Ccp2YYBZmABFKDmABFKDmXNGKDmnFMKDm0GAUfn_input.tar.gz
01-Aug-2017 08:33:12 [LHC@home] Finished upload of w-c8_n20_lhc2016_40_MD-105-16-476-2.5-0.9157__24__s__64.31_59.32__1_2__6__13.5_1_sixvf_boinc5378_1_0
01-Aug-2017 08:33:13 [LHC@home] Finished download of 5W5KDmokewqnSu7Ccp2YYBZmABFKDmABFKDmXNGKDmnFMKDm0GAUfn_input.tar.gz
01-Aug-2017 08:33:13 [LHC@home] Started download of rte_5W5KDmokewqnSu7Ccp2YYBZmABFKDmABFKDmXNGKDmnFMKDm0GAUfn.tar.gz
01-Aug-2017 08:33:14 [LHC@home] Finished download of rte_5W5KDmokewqnSu7Ccp2YYBZmABFKDmABFKDmXNGKDmnFMKDm0GAUfn.tar.gz
01-Aug-2017 08:33:14 [LHC@home] Started download of boinc_job_script.xp0zEy
01-Aug-2017 08:33:15 [LHC@home] Finished download of boinc_job_script.xp0zEy


We've notified our ATLAS colleagues about the possible application/Frontier problem.

Thank you Nils.

From the BOINC point of view it seems ok, my client downloads and uploads ATLAS files

this seems to be the strange thing: some people do NOT have this problem, others do (here, also ping yields a timeout, again).

We've notified our ATLAS colleagues about the possible application/Frontier problem.

so let's wait and see what they can/will find out.

I now run ATLAS on two CPU cores at a time with no problems for either downloading work units or errors running them.

I think CERN is sort of ignoring the single-CPU users, to encourage the multi-core version. It is supposed to save on bandwidth, etc. I liked the single-core version for efficiency, but two cores will work for me.

Cern-IT found the solution yesterday morning for this problem and they find this also today. So take a break.

Erich56 wrote:

... also ping yields a timeout, again ...

Hi Erich,

As already stated by Yeti, a failing ping is not a criterion to see if the server is running or not.
Ping uses the ICMP protocol and may be dropped/rejected by any of the hops between your host and the target system.

The file transfer done by the projects is (mostly) done via HTTP.
To check the availability of a service on a distinct server the VMs typically use a command like

nc -z -v -w 5 lhchomeproxy.cern.ch 80

and an answer like

Connection to lhchomeproxy.cern.ch 80 port [tcp/http] succeeded!

shows that the service is up and also that the route to the server is not blocked.

Unfortunately this is only the network part of a connection.
Some services are protected by special credentials on a higher comunication level and sometimes also cause failures.

It seems that the 120MB files should be downloaded from the boincai04 server and that server is not reachable for me.
Maybe for Nils cause he's on CERN's LAN.

Yes the boincai04 server is down once again so you probably should just try one of the other project tasks for now.

---

Volunteer Mad Scientist For Life

Hi,
The server which is hosting this file was down, that is why there was a download error.. Now we have brought back the machine, and the file should be available.

Cheers!

Ever since the day that there was the issue after the cleanup of old files, I have been experiencing the same issues. LHC was running Atlas fine for a long time, where it would download 4 tasks and run them through without issue.. What I am seeing now (and since the day of the server cleanup) is my machine will attempt to download files for tasks, and get stuck retrying on a few for several hours.

7/29/2017 7:22:43 PM | LHC@home | Started download of jf_f3ff3ac08153d0ee04ea606f0dea9a0e
7/29/2017 7:23:05 PM | | Project communication failed: attempting access to reference site
7/29/2017 7:23:05 PM | LHC@home | Temporarily failed download of jf_f3ff3ac08153d0ee04ea606f0dea9a0e: connect() failed
7/29/2017 7:23:05 PM | LHC@home | Backing off 01:05:41 on download of jf_f3ff3ac08153d0ee04ea606f0dea9a0e
7/29/2017 7:23:06 PM | | Internet access OK - project servers may be temporarily down.

I have Updated, Restarted, Removed and re-added the LHC project several times, over several days. Suggestions? Tried 2 different networks (home and work), same issues. Connection to other projects are no issue.

Hi,
The server which is hosting this file was down, that is why there was a download error.. Now we have brought back the machine, and the file should be available.

Cheers!

Ever since the day that there was the issue after the cleanup of old files, I have been experiencing the same issues. LHC was running Atlas fine for a long time, where it would download 4 tasks and run them through without issue.. What I am seeing now (and since the day of the server cleanup) is my machine will attempt to download files for tasks, and get stuck retrying on a few for several hours.

7/29/2017 7:22:43 PM | LHC@home | Started download of jf_f3ff3ac08153d0ee04ea606f0dea9a0e
7/29/2017 7:23:05 PM | | Project communication failed: attempting access to reference site
7/29/2017 7:23:05 PM | LHC@home | Temporarily failed download of jf_f3ff3ac08153d0ee04ea606f0dea9a0e: connect() failed
7/29/2017 7:23:05 PM | LHC@home | Backing off 01:05:41 on download of jf_f3ff3ac08153d0ee04ea606f0dea9a0e
7/29/2017 7:23:06 PM | | Internet access OK - project servers may be temporarily down.

I have Updated, Restarted, Removed and re-added the LHC project several times, over several days. Suggestions? Tried 2 different networks (home and work), same issues. Connection to other projects are no issue.

Server error: feeder not running

Server can't open log file (../log_boincai04/scheduler.log)


(did you happen to check your pm over at TEST lately? )

Problems with boincai04 affect the download of the EVNT files.
It's good to hear that they are solved.

Instead there may be additional problems independent from boincai04 as the heavy use of the spare server ccfrontier.in2p3.fr and the missing output at console 2 point out.
Has this also been checked/solved?

As already stated by Yeti, a failing ping is not a criterion to see if the server is running or not.
Ping uses the ICMP protocol and may be dropped/rejected by any of the hops between your host and the target system.

well, the experience I have made in the past days was that ping worked fine when the server was alive, and that ping showed a timeout when the server was down. Maybe it was rather a coincidence that it worked that way. I personally am NOT a network specialist at all.

To check the availability of a service on a distinct server the VMs typically use a command like

nc -z -v -w 5 lhchomeproxy.cern.ch 80

I now tried this, but I got the message that "nc" is a wrong command.

Erich56 wrote:

I now tried this, but I got the message that "nc" is a wrong command.

Well, it's a command that was originally written for unix (short form of netcat) and therefore isn't available by default on windows.

netcat

"netcat" - sounds sweet :-)

Yes, there was a permission issue with the scheduler file on boincai04, it is fixed now..

Cheers!

Hi,
The server which is hosting this file was down, that is why there was a download error.. Now we have brought back the machine, and the file should be available.

Cheers!

Ever since the day that there was the issue after the cleanup of old files, I have been experiencing the same issues. LHC was running Atlas fine for a long time, where it would download 4 tasks and run them through without issue.. What I am seeing now (and since the day of the server cleanup) is my machine will attempt to download files for tasks, and get stuck retrying on a few for several hours.

7/29/2017 7:22:43 PM | LHC@home | Started download of jf_f3ff3ac08153d0ee04ea606f0dea9a0e
7/29/2017 7:23:05 PM | | Project communication failed: attempting access to reference site
7/29/2017 7:23:05 PM | LHC@home | Temporarily failed download of jf_f3ff3ac08153d0ee04ea606f0dea9a0e: connect() failed
7/29/2017 7:23:05 PM | LHC@home | Backing off 01:05:41 on download of jf_f3ff3ac08153d0ee04ea606f0dea9a0e
7/29/2017 7:23:06 PM | | Internet access OK - project servers may be temporarily down.

I have Updated, Restarted, Removed and re-added the LHC project several times, over several days. Suggestions? Tried 2 different networks (home and work), same issues. Connection to other projects are no issue.

Server error: feeder not running

Server can't open log file (../log_boincai04/scheduler.log)


(did you happen to check your pm over at TEST lately? )

Just to summarize the cause of the failure for some files with ATLAS@home:
1. some of the input files are stored on a test server boincai04, and it was down yesterday due to heavy load. We modified the job submission script, so all the input files are stored on more powerful and reliable servers, which should prevent this from happening again.
2. For people who still attach the hosts to the Atlas-test project, the server (boincai04) was stuck a few times in the past a few days due to the heavy workload on it as a tiny machine. Now we split the workload on different machines, and the boincai04 machine still dispatches a small amount of test jobs..

Cheers!

Thanks, Wenjing, for the update; have a nice day :-)

Just got a download error for an Atlas task: https://lhcathome.cern.ch/lhcathome/workunit.php?wuid=76058390 like my wingman did.

---

Since about one week, the download-file of Atlas (200MByte) is dropping very slow.
The Counter of the network starts with for example 100 kps and reduced up to Zero.
It need about 1 hour instead of one minute regulary.

Since about one week, the download-file of Atlas (200MByte) is dropping very slow.
The Counter of the network starts with for example 100 kps and reduced up to Zero.
It need about 1 hour instead of one minute regulary.

hm, that's strange. Here, all downloads run with same fast speed as ever before (i.e. in about one minute).