Message boards : Theory Application : 6+ day task?
Message board moderation

To post messages, you must log in.

AuthorMessage
doug

Send message
Joined: 28 Mar 20
Posts: 33
Credit: 218,222
RAC: 2
Message 52330 - Posted: 25 Sep 2025, 15:34:33 UTC

I have a Theory task (Theory_2922-4905151-252_1) that has now been running for over 6d, 5.5 hr.

Latest Windows 10, on an older machine. The BOINC app is "301.00 (vbox64_theory)".

BoincTasks shows the task taking around 45% CPU, so it's apparently still going. Another Theory task (same application) that is still in progress shows in the BOINC Manager a total elapsed/remaining runtime of about 3.5 hours, which is a pretty huge difference.

Is all this expected? Should I be concerned about this 6+ day task?

Thanks.

doug
ID: 52330 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1923
Credit: 149,467,009
RAC: 143,612
Message 52331 - Posted: 25 Sep 2025, 16:02:58 UTC - in response to Message 52330.  

Is all this expected? Should I be concerned about this 6+ day task?
no, don't be concerned. The lengh of Theory tasks varies to a high extent (recently I had one which ran about 16 days, and also before my machines received rather long tasks at times).
With the LHC tasks, the progess and time indications from the BOINC manager don't tell a thing.
However, for Theory tasks, you can click on the "Graphics" button in the left hand part of the BOINC Manager, then a Browser window opens, there you click on "logs", and then on "running log" - this shows the progress of the task
ID: 52331 · Report as offensive     Reply Quote
Crystal Pellet
Volunteer moderator
Volunteer tester

Send message
Joined: 14 Jan 10
Posts: 1469
Credit: 9,926,782
RAC: 1,799
Message 52332 - Posted: 25 Sep 2025, 16:08:49 UTC - in response to Message 52331.  

To add to Erich56's reply:
On the first line of that running.log you'll find something like ===> [runRivet] Thu Sep 25 02:13:29 PM UTC 2025 [boinc pp jets 13000 25 - pythia8 8.244 CP2-CR1 100000 282]
The forelast number (mostly 100000) is the number of events to be processed.
Go to the end of the running log to find how many events are processed so far.

Of course at first: Is the task using cpu-cycles?
ID: 52332 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2710
Credit: 292,041,669
RAC: 145,478
Message 52333 - Posted: 25 Sep 2025, 16:10:07 UTC - in response to Message 52330.  

Some of them run up to 10 days.
Locate the runRivet.log in ...\slots\x\shared\ and check it's "last modified" timestamp.
If that is older than a day the task most likely got stuck and you should cancel it.

If the log gets updated at least every now and then check it's content for the newest "nnn events processed" lines.
Tasks usually process 100000 events, so you can estimate the time left.

BOINC can't do this as it doesn't know where it can find those numbers.
ID: 52333 · Report as offensive     Reply Quote
doug

Send message
Joined: 28 Mar 20
Posts: 33
Credit: 218,222
RAC: 2
Message 52334 - Posted: 25 Sep 2025, 16:47:54 UTC - in response to Message 52333.  

Thanks all. You've set my mind at ease.

Following Crystal Pellet's instructions, I found this:

===> [runRivet] Thu Sep 25 01:55:56 PM UTC 2025 [boinc ppbar jets 1960 37 - pythia8 8.230 tune-4c 100000 282]

Following Erich56's instructions, I found this:

Pythia::next(): 49000 events have been generated
49000 events processed
49100 events processed
49200 events processed
49300 events processed
49400 events processed
49500 events processed
49600 events processed
49700 events processed

So apparently only HALF done after well over 6 days!

Also, BoincTasks shows the task taking around 45% CPU, so I was pretty sure the task was still alive in some way or other.

Finally, the timedate stamp on the file mentioned by computezrmle was updated in the last few minutes.

Thanks to all of you!

doug
ID: 52334 · Report as offensive     Reply Quote
Crystal Pellet
Volunteer moderator
Volunteer tester

Send message
Joined: 14 Jan 10
Posts: 1469
Credit: 9,926,782
RAC: 1,799
Message 52335 - Posted: 26 Sep 2025, 6:04:06 UTC - in response to Message 52334.  
Last modified: 26 Sep 2025, 6:51:19 UTC

Also, BoincTasks shows the task taking around 45% CPU, so I was pretty sure the task was still alive in some way or other.
From another Theory task of yours:

Run time 5 hours 0 min 11 sec
CPU time 2 hours 15 min

You should find the reason that your tasks are running < 50% CPU
Maybe something in your BOINC settings, if that is not the intention.
ID: 52335 · Report as offensive     Reply Quote
Profile Magic Quantum Mechanic
Avatar

Send message
Joined: 24 Oct 04
Posts: 1242
Credit: 85,072,025
RAC: 135,993
Message 52336 - Posted: 26 Sep 2025, 6:16:17 UTC

I have had many long Valid Theory's over the years but I always watch the running logs since you can get a Sherpa that will run 10 days and fail
( I save stuff) here is one from last June
Run time 8 days 18 hours 41 min 28 sec
CPU time 8 days 17 hours 14 min 35 sec
Validate state Valid
Credit 7,539.68
and many others but you can always abort if you want to run another project
ID: 52336 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2710
Credit: 292,041,669
RAC: 145,478
Message 52337 - Posted: 26 Sep 2025, 6:24:10 UTC - in response to Message 52335.  

Also, BoincTasks shows the task taking around 45% CPU, so I was pretty sure the task was still alive in some way or other.
From another Theory task of yours:

Run time 5 hours 0 min 11 sec
CPU time 2 hours 15 min

You should find the reasom that your tasks are running < 50% CPU
Maybe something in your BOINC settings, if that is not the intention.

The reason can be found in stderr.txt:

Setting CPU throttle for VM. (40%)

@doug
For vbox apps it is not recommended to throttle the CPU via BOINC's computing preferences since it may cause timing issues.
Instead, leave it at 100 % and limit the # CPUs via <ncpus>N</ncpus> in cc_config.xml.
ID: 52337 · Report as offensive     Reply Quote
Crystal Pellet
Volunteer moderator
Volunteer tester

Send message
Joined: 14 Jan 10
Posts: 1469
Credit: 9,926,782
RAC: 1,799
Message 52338 - Posted: 26 Sep 2025, 8:36:27 UTC - in response to Message 52337.  

.... Instead, leave it at 100 % and limit the # CPUs via <ncpus>N</ncpus> in cc_config.xml.
.... or in BoincTasks -> Menu Extra -> BOINC Preference -> On multiprocessor systems, use at most ...... % of the processors
ID: 52338 · Report as offensive     Reply Quote
saigon
Avatar

Send message
Joined: 8 Jul 12
Posts: 6
Credit: 1,648,031
RAC: 7,248
Message 52632 - Posted: 7 Nov 2025, 5:13:17 UTC

A similar situation: for example, task Theory_2922-4895579-343 has been running for over a week. Considering that the PC runs 6-8 hours a day, the calculation is still running, and the running.log is already over 3MB. The project is updated periodically, and today it says "Time expired - no response." It's a shame so much time was wasted. And about 40% of these extremely long tasks. Isn't it possible to increase the deadline?
ID: 52632 · Report as offensive     Reply Quote
Crystal Pellet
Volunteer moderator
Volunteer tester

Send message
Joined: 14 Jan 10
Posts: 1469
Credit: 9,926,782
RAC: 1,799
Message 52633 - Posted: 7 Nov 2025, 8:50:08 UTC - in response to Message 52632.  

A similar situation: for example, task Theory_2922-4895579-343 has been running for over a week. Considering that the PC runs 6-8 hours a day, the calculation is still running...

ATLAS, CMS and Theory's on your system are running within a Virtual Machine and are considered to run without any interruption, so no pausing and suspending.
After a overnight suspend they will error out or for Theory's with a bit of luck will restart from the beginning.
ID: 52633 · Report as offensive     Reply Quote
saigon
Avatar

Send message
Joined: 8 Jul 12
Posts: 6
Credit: 1,648,031
RAC: 7,248
Message 52634 - Posted: 7 Nov 2025, 11:12:20 UTC - in response to Message 52633.  

After a overnight suspend they will error out or for Theory's with a bit of luck will restart from the beginning.

Yes, the task is paused and continues correctly the next day, with no errors. This is evident in the running.log logs. But the 10-day limit is woefully inadequate for extremely long tasks. Is it possible to increase the limit if the task is running correctly and periodically announces itself to the server? Not 864,000 seconds, but 1M or more?
ID: 52634 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2710
Credit: 292,041,669
RAC: 145,478
Message 52635 - Posted: 7 Nov 2025, 12:56:35 UTC - in response to Message 52634.  

Theory due dates set for BOINC are inter-coordinated with due dates in the backend systems.
Changes on one end require changes on the other end and vice versa.

Your computer's mc-plots record shows a Theory error rate of only 1 %.
This is pretty low and does not justify those changes, especially since this 1 % covers all kind of errors.
ID: 52635 · Report as offensive     Reply Quote
saigon
Avatar

Send message
Joined: 8 Jul 12
Posts: 6
Credit: 1,648,031
RAC: 7,248
Message 52636 - Posted: 7 Nov 2025, 13:35:01 UTC

I understand, thanks for the clarification. Task Theory_2922-4895579-343 is still running, even though the status is "Time expired - no response." Can I stop it? It won't do any good?
running.log 2025-Oct-31 10:51:58 6.0M text/plain;charset=utf-8
A small part running.log https://dpaste.com/BE34838BN
ID: 52636 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2710
Credit: 292,041,669
RAC: 145,478
Message 52637 - Posted: 7 Nov 2025, 13:45:31 UTC - in response to Message 52636.  

Whatever you do, I doubt you will get credit for this task.
So, it might be the best to cancel it and run a fresh task.
ID: 52637 · Report as offensive     Reply Quote
Crystal Pellet
Volunteer moderator
Volunteer tester

Send message
Joined: 14 Jan 10
Posts: 1469
Credit: 9,926,782
RAC: 1,799
Message 52638 - Posted: 7 Nov 2025, 13:47:39 UTC - in response to Message 52636.  
Last modified: 7 Nov 2025, 13:52:11 UTC

Did you take that from the runRivet.log from disk or from the running.log shown by using the "Show graphics" button from BOINC Manager?
As computezrmle said, you probably will not get credit. The wingman that did the resend, returned a 'valid' result, however scientifical is was not valid: job: run exitcode=1
ID: 52638 · Report as offensive     Reply Quote
saigon
Avatar

Send message
Joined: 8 Jul 12
Posts: 6
Credit: 1,648,031
RAC: 7,248
Message 52640 - Posted: 7 Nov 2025, 14:06:04 UTC - in response to Message 52638.  

Crystal Pellet From the running.log shown by using the "Show graphics" button from BOINC Manager.
ID: 52640 · Report as offensive     Reply Quote
Crystal Pellet
Volunteer moderator
Volunteer tester

Send message
Joined: 14 Jan 10
Posts: 1469
Credit: 9,926,782
RAC: 1,799
Message 52642 - Posted: 7 Nov 2025, 14:48:13 UTC - in response to Message 52640.  

Crystal Pellet From the running.log shown by using the "Show graphics" button from BOINC Manager.
That's OK. The runRivet.log from disk is not updated after the task was suspended (Leave in memory off), BOINC client restarted or system rebooted.
Maybe you could find something in stderr.txt in the corresponding slot, why the task takes so long.
BTW: the deadline for the client is 10 days, but from the server you get 11 days as deadline (1 day grace period).
ID: 52642 · Report as offensive     Reply Quote
ServicEnginIC
Avatar

Send message
Joined: 28 May 16
Posts: 5
Credit: 5,506,537
RAC: 8,940
Message 52663 - Posted: 16 Nov 2025, 0:53:25 UTC - in response to Message 52331.  

However, for Theory tasks, you can click on the "Graphics" button in the left hand part of the BOINC Manager, then a Browser window opens, there you click on "logs", and then on "running log" - this shows the progress of the task

This tip has been very useful for me.
Previously, I blindly aborted Theory tasks when lasting more than 5-6 days.

Then I read your comment (thank you very much) and explored "Show graphics" command and beyond at BOINC Manager.
When selecting a running Theory task, command "Show graphics" gets enabled.
When pressing "Show graphics", an explorer window opens, containing a link to "logs".
If that link is pressed, a very useful "Index of /logs/" page opens.
That page contains a "Last Modified:" label that gives a clue on whether "running.log" file has been recently updated or not.
When updating "running.log" file becomes frozen, it can be taken as a warning to abort the corresponding task.
And if "running.log" is being periodically updated, it can be taken as a signal that the task is still alive.
"Index of /logs/" page also contains a link to "running.log" file itself.
At the beginning of "running.log" file, a definition of the task is shown.
The penultimate parameter on the first line indicates the stated number of events for the task. 55000 in this example.
And at the end of the file, progress of the task can be followed.

Based on previous information, I continued to process this overdued task. I was following that "running.log" file" was being periodically updated, and the task was slow but continuously approaching its stated number of events (55000).
Finally, that task was successfully reported, more than three days past its due date/time.
It holds my current record: 1,239,153.57 seconds of execution time, 16,659.96 credits awarded.
Task details
It was also the longest task reported on November 7th: 344.21 hours
That is: 14 days, 8 hours, 12 minutes, 33 seconds
ID: 52663 · Report as offensive     Reply Quote
Crystal Pellet
Volunteer moderator
Volunteer tester

Send message
Joined: 14 Jan 10
Posts: 1469
Credit: 9,926,782
RAC: 1,799
Message 52665 - Posted: 16 Nov 2025, 10:38:17 UTC - in response to Message 52663.  
Last modified: 16 Nov 2025, 10:39:37 UTC

However, for Theory tasks, you can click on the "Graphics" button in the left hand part of the BOINC Manager, then a Browser window opens, there you click on "logs", and then on "running log" - this shows the progress of the task

This tip has been very useful for me.
Previously, I blindly aborted Theory tasks when lasting more than 5-6 days.
Then I read your comment (thank you very much) and explored "Show graphics" command and beyond at BOINC Manager.
Congratulations for returning this overdue valid task and many thanks for your extended comments and images!

It was also the longest task reported on November 7th: 344.21 hours
That is: 14 days, 8 hours, 12 minutes, 33 seconds
Unfortunately someone broke your record during the last 100 tasks:

Theory Simulation 4930 6784 3.52 (0.03 - 489.41)
20 days - 9 hours - 25 minutes
ID: 52665 · Report as offensive     Reply Quote

Message boards : Theory Application : 6+ day task?


©2025 CERN