LHC@home

Spring and Autumn is therefor a good time.
We Volunteers would be happy if this would be solved.
Ivan, we stay with you!

All Atlas tasks complete and validate both on my Windows 10 PC and the SUN Linux box. All other LHC tasks fail, save SixTrack. What is the difference between Atlas tasks and other VM tasks?
Tullio

Spring and Autumn is therefor a good time.
We Volunteers would be happy if this would be solved.
Ivan, we stay with you!

Thanks, I need the feedback sometimes. Especially as I need to get this into real CMS production before I retire. I'm contracted until 31/3/18; there might be funds until the end of next year. After that I may be forced back to Byron Bay. :-( (The health service is free here, not so in Oz.)

---

... I retire ...

:-( :-( :-(

Spring and Autumn is therefor a good time.
We Volunteers would be happy if this would be solved.
Ivan, we stay with you!

Thanks, I need the feedback sometimes. Especially as I need to get this into real CMS production before I retire. I'm contracted until 31/3/18; there might be funds until the end of next year. After that I may be forced back to Byron Bay. :-( (The health service is free here, not so in Oz.)

Moving to New South Wales?
Australia is too far away for me to ever travel.
Is this *semi-retired*?? (that is what I have always claimed myself)

But I have lived here since 1971 so most of my travel is on the bigscreen via satellite (which ends up costing as much as actually traveling)

I just might have to run a few hundred CMS for you since I was just leaving those to you over at -dev

I do like that contract date since I will hit that dreaded *60* mark in January .....and start my 14th year here in a couple days (10-24-2004)

Over here most people move to live by the ocean or for some hot reason down to Arizona when they retire........but I have lived by the water pretty much all 60 years so Byron Bay sounds like the place for that Ivan (humid subtropical climate)

It will be warmer December down there.
And it looks like not a lot of people there (I wish some where I live would move there)

Here it is rain until June and occasional wind just enough to knock out the power and shut down my fleet of computers.....and everything else (yeah you would think I would have got a generator by now)

I only have an ale on a rare occasion but switching from Samuel Smiths to a Fosters would be impossible (ok it does look like they have hundreds of micro brewers just like we have here)
I would take care of that 20-core Xenon for you of course

But let me know if you want me to run some CMS multi-cores

---

Volunteer Mad Scientist For Life

But let me know if you want me to run some CMS multi-cores

Magic,

you can set your preferences in -dev to get both multicore tasks(Theory and CMS).
This is running for me in production very well (only single-core).

Spring and Autumn is therefor a good time.
We Volunteers would be happy if this would be solved.
Ivan, we stay with you!

Thanks, I need the feedback sometimes. Especially as I need to get this into real CMS production before I retire. I'm contracted until 31/3/18; there might be funds until the end of next year. After that I may be forced back to Byron Bay. :-( (The health service is free here, not so in Oz.)

Moving to New South Wales?
Australia is too far away for me to ever travel.
Is this *semi-retired*?? (that is what I have always claimed myself)

Yeah, I was born there, 66 years ago. In the bush. In a banana shed!
Not sure about semi-retired, but I'd go mental being fully retired. Perhaps write that SF novel that's been in my head for decades?

But I have lived here since 1971 so most of my travel is on the bigscreen via satellite (which ends up costing as much as actually traveling)

I do like that contract date since I will hit that dreaded *60* mark in January .....and start my 14th year here in a couple days (10-24-2004)

I realised recently that this is the longest I've worked at one place, the longest I've worked in one country (17 years UK, 13 years CH, 14 years Oz including Uni, and 4 years CA), and the longest I've lived at one address. :-(

Over here most people move to live by the ocean or for some hot reason down to Arizona when they retire........but I have lived by the water pretty much all 60 years so Byron Bay sounds like the place for that Ivan (humid subtropical climate)

It will be warmer December down there.
And it looks like not a lot of people there (I wish some where I live would move there)

Here it is rain until June and occasional wind just enough to knock out the power and shut down my fleet of computers.....and everything else (yeah you would think I would have got a generator by now)

I know what you mean about the rain, I lived for four years in Vancouver BC. I once asked a chap I was sharing a Whistler chair-lift with where he was from, "Everwet, Washington!" Surprisingly, Expo '86 was completely dry except for the last day -- which was also the day I was riding my motorcycle from Coos Bay (IIRC) to YVR on my way back from a conference in SF. I was a bit bedraggled at the border crossing...
On the other hand, Byron Bay gets 100 inches per year, mostly as summer thunderstorms and the rain-depression aftermaths of cyclones (Pacific hurricanes) that have run out of steam on their way south down the Queensland coast. The humidity gets me ever since I moved away to Uni in Canberra -- I ended up not going home for Christmas while I was doing my PhD; 42 C and 20% humidity was preferable to 32 C and 90% humidity.

---

I would take care of that 20-core Xenon for you of course

Unfortunately, that machine and its twin (they each take up a half-width slot in a 2U rack chassis) belong to work, not to me. They both run SETI@Home in the background as a "stress test". :-) So far, everyone seems happy with that. Running CMS@Home on brphab and another 12-core machine is easier to justify, of course, given my involvement with this project.

---

All Atlas tasks complete and validate both on my Windows 10 PC and the SUN Linux box. All other LHC tasks fail, save SixTrack. What is the difference between Atlas tasks and other VM tasks?
Tullio

If you watch the VM's of each progress till they are crunching, ATLAS seems to be on a different development path while CMS, Theory and LHCb seem to be in near identical scripts and setups.

If you watch the VM's of each progress till they are crunching, ATLAS seems to be on a different development path while CMS, Theory and LHCb seem to be in near identical scripts and setups.

That is why I am running only Atlas@home and SixTrack, when available. I abort all the rest.
Tullio

Have CMS and Theory selected in preferences and they are running on a Server very well without problems!

For the last couple weeks I've been stress testing LHC@Home work on this server by overloading BOINC with high disk usage tasks (Pogs, Rosetta) running alongside the various VM-based LHC WU styles and CMS has the highest failure rate.
"Unable to mount root" is the typical failure.

If Theory, CMS, LHCb and ATLAS are starved for CPU cycles by running competing applications that use most of the available threads (simulating game playing. etc), again, CMS has the highest failure rates on this server.

The only way to get 12 CMS tasks to eventually all suspend successfully on the hardware RAID0 3 drive partition, dedicated to BOINC, is to accept the first suspend failures and defragment the partition, which will have a 95%+ fragmentation.
Subsequent suspension attempts will almost always succeed once drive is defragmented.
This says something about how BOINC moves the large WU VDI's into the slots when a whole batch of WU are started. Maybe there is a BOINC setting to force it to move one slot at a time instead of multi-tasking that procedure? (This project is certainly designed for SSD drives, which are not in my budget)

Could the project be modified for WU's to use link clone to the base VDI for each slot and save on space, fragmentations and setup times?

again, same problem with a task this morning: it errored out after a few minutes:

2017-10-27 09:29:48 (6820): Guest Log: [DEBUG] Testing connection to Condor server on port 9618
2017-10-27 09:30:08 (6820): VM Completion File Detected.
2017-10-27 09:30:08 (6820): VM Completion Message: Could not connect to Condor server on port 9618

next probably Condor-related problem this afternoon:

2017-10-28 14:12:41 (3616): Guest Log: [DEBUG] HTCondor ping
2017-10-28 14:14:21 (3616): Guest Log: [DEBUG] 1
2017-10-28 14:14:21 (3616): Guest Log: [DEBUG] DC_NOP failed!
2017-10-28 14:14:21 (3616): Guest Log: SECMAN:2007:Failed to end classad message.
2017-10-28 14:14:21 (3616): Guest Log: 10/28/17 14:13:23 recognized DC_NOP as command name, using command 60011.
2017-10-28 14:14:21 (3616): Guest Log: 10/28/17 14:15:03 SECMAN: no classad from server, failing
2017-10-28 14:14:21 (3616): Guest Log: [ERROR] Could not ping HTCondor.
2017-10-28 14:14:21 (3616): Guest Log: [INFO] Shutting Down.

what is this all about?

This morning I noticed a CMS task which had been running for more than 16 hours, but the Windows Task Manager showed no CPU usage.

I terminated the task, and from stderr I could see that total CPU usage was only 75 secs.
For more information see here:
https://lhcathome.cern.ch/lhcathome/result.php?resultid=163118321

very strange to me sounds the "Hypervisor System log" within the stderr (in the lower part) - I had never seen something like this before.

Can anyone tell me what was going wrong?

They all fail on my Windows 10 PC while Atlas tasks run perfectly on the same machine. Nobody has given me any explanation.
Tullio

something seems strange with the CMS tasks lately.

2 days ago, for example, I had a few cases where tasks errored out after several hours, with

-1073740791 (0xC0000409) STATUS_STACK_BUFFER_OVERRUN

shown under "final status".

More Information on such a task here:
https://lhcathome.cern.ch/lhcathome/result.php?resultid=163021052

.
.
Can anyone tell me what was going wrong?

After Requesting an X509 credential from LHC@home
there is no fast benchmark and no HTCondor ping,
so somehow the request did not get an answer.
The job should have been aborted.

hm, so it seems that there may indeed be some kind of problem with the Condor Server - which occurs not too often, but once in a while

Ivan wrote:

There is, still, a very big [authentication] problem with the Condor server. However, Volunteer jobs should not be communicating with it.
tl;dr: what communicates with Condor is the log-merge processes, and these should only run on CMS resources. If they try to run on Volunteer hosts, we really need to look into it.
We are trying to solve these remaining problems, but the scattered and disparate nature of the people who need to be involved is a drawback. Northern hemisphere summer was a problem, due to holidays. I'd like it to be fixed soon but, you know, winter and Christmas...

Once more, yesterday evening I had this problem:

2017-10-30 21:11:32 (3808): Guest Log: [DEBUG] Testing connection to Condor server on port 9618
2017-10-30 21:11:52 (3808): VM Completion File Detected.
2017-10-30 21:11:52 (3808): VM Completion Message: Could not connect to Condor server on port 9618

So, obviously there is still some kind of issue with the Condor server.

again, same problem with a task this morning: it errored out after a few minutes:

2017-10-27 09:29:48 (6820): Guest Log: [DEBUG] Testing connection to Condor server on port 9618
2017-10-27 09:30:08 (6820): VM Completion File Detected.
2017-10-27 09:30:08 (6820): VM Completion Message: Could not connect to Condor server on port 9618

This error and the other error you posted are occurring in Theory sims also.
I posted the variety of errors from my last 3 days of Theory runs on your other thread https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=4496&postid=32939#32939.