LHC@home

Hi,
I have received a couple of ATLAS tasks which lasted incredibly long for usual length of ATLAS tasks - ie 2d instead of few hours:
https://lhcathome.cern.ch/lhcathome/result.php?resultid=188619213
The same seems to be happening on an ATLAS task form

lhcathome-dev

In general, on my Ubuntu machine native ATLAS tasks run fine. I did not have a look at the logs, but maybe you can find something interesting...
Cheers,
A.[/code]

I had very long ATLAS tasks several days ago, they finally failed :-(

See my posting here: https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=4620&postid=35085#35085

Hello,
I got another couple of these tasks with ATLAS version 2.54 (native_mt) , e.g.:
https://lhcathome.cern.ch/lhcathome/result.php?resultid=191180856
The expected duration was few hours, but now it is running since a couple of days. The second one comes from lhcathome-dev, with ATLAS version 0.50 (native_mt).

Could someone tell if ATLAS jobs so long are normal or expected? If yes, then there is something wrong with the expected running time; if not, do you have a fix?
These long tasks take all the available CPU slots (not many, as I make available my work desktop computer and I use it more as monitoring from the volunteer point of view) for entire days, preventing other tasks to be processed.

Are these tasks also the reason behind the drop in the GigaFLOPs reported by the server status page? 1-2 weeks ago we were at >80, but now we are at ~50-60...
Thanks,
Cheers,
A.

Have Linux-native Tasks with 40 hours with single CPU and 36 hours with two CPU's.
Yes, they are heavy. You get more than 1k Cobblestones.
https://lhcathome.cern.ch/lhcathome/img/progresschart.png
Edit: Of course only 200 Collisions!

A few days ago I had a 1-core longrunner with the following values:

WallTime=144152.41s
KernelTime=298.04s
UserTime=143632.60s
CPUUsage=99%

Monitoring tip:
Open a console window, cd to your BOINC client's base directory and run the following oneliner

watch -n10 "find ./slots/ \( -name \"log.EVNTtoHITS\" -o -name \"AthenaMP.log\" \) |sort |xargs -I {} -n1 sh -c \"egrep 'INFO.*Event nr. ' {} |tail -n1; echo -e '\n'\""

Indeed the current tasks are rather heavier than the previous ones - on average each event takes twice or even three times as long. Some may call them "deadly", others might appreciate the extra credit :)

In general if the task is using close to 100% CPU it is still good - this is especially true for native linux tasks where we don't have VirtualBox causing trouble.

... others might appreciate the extra credit :)

What looks like extra credit now will turn into extra low credit once we get work with shorter runtimes.
This is caused by the method the credit is calculated.
At the end it will average out.

In general if the task is using close to 100% CPU it is still good - this is especially true for native linux tasks where we don't have VirtualBox causing trouble.

👍

Indeed the current tasks are rather heavier than the previous ones - on average each event takes twice or even three times as long.

I took this as a reason for trying out 4-core tasks. Although I remember having seen from some charts and comments that 4-core is (markedly?) less efficient, in comparison to 1- or 2-core. So I'll see.

Just FYI...It is taking 90 seconds of run time at 100% to accomplish 1 second of estimated time. I have roughly 660 minutes left, so that comes out about 16-17 hours at a CPU set at 100% usage to complete this task. WOW!

---

Well, so much for my predictions. The task suddenly completed and was verified. Interestingly, iit ran 3X normal time , but I got less credit.

J

Hi Jim,

If you look into your task logs
https://lhcathome.cern.ch/lhcathome/result.php?resultid=191413557
https://lhcathome.cern.ch/lhcathome/result.php?resultid=191541487
you may notice lots of lines that show your computer is struggling very hard to run ATLAS.

Examples:

2018-05-18 14:51:16 (4482): Guest Log: PyJobTransforms.trfExe.validate 2018-05-18 20:49:36,502 ERROR Validation of return code failed: EVNTtoHITS got a SIGKILL signal (exit code 137) (Error code 65)
2018-05-18 14:51:16 (4482): Guest Log: PyJobTransforms.trfExe.validate 2018-05-18 20:49:36,517 INFO Scanning logfile log.EVNTtoHITS for errors
2018-05-18 14:51:16 (4482): Guest Log: PyJobTransforms.transform.execute 2018-05-18 20:49:36,792 CRITICAL Transform executor raised TransformValidationException: EVNTtoHITS got a SIGKILL signal (exit code 137)
2018-05-18 14:51:16 (4482): Guest Log: PyJobTransforms.transform.execute 2018-05-18 20:49:40,156 WARNING Transform now exiting early with exit code 65 (EVNTtoHITS got a SIGKILL signal (exit code 137))

2018-05-19 14:41:54 (13200): VM state change detected. (old = 'running', new = 'paused')
2018-05-19 14:42:06 (13200): VM state change detected. (old = 'paused', new = 'running')
2018-05-19 14:42:14 (13200): VM state change detected. (old = 'running', new = 'paused')
2018-05-19 14:42:25 (13200): VM state change detected. (old = 'paused', new = 'running')
2018-05-19 14:47:28 (13200): VM state change detected. (old = 'running', new = 'paused')

The reason is that the recent tasks - when you run them as 1-core or 2-core - need much more RAM than it is configured by the project server.
Your logs show that you run them as 1-core (with 3500 MB RAM).

As your host has enough RAM you may consider to use an app_config.xml like this:

<app_config>
  <app>
    <name>ATLAS</name>
    <max_concurrent>2</max_concurrent>
    <report_results_immediately/>
  </app>
  <app_version>
    <app_name>ATLAS</app_name>
    <plan_class>vbox64_mt_mcore_atlas</plan_class>
    <avg_ncpus>1.0</avg_ncpus>
    <cmdline>--nthreads 1 --memory_size_mb 4800</cmdline>
  </app_version>
</app_config>

The settings become active with the 1st fresh task that starts after you "reload config files" in your BOINC manager.

On my Linux laptop I have a one core Atlas task that has been running for 6 days and 22 hours. My last Atlas task on the SUN M20 Linux workstation has completed with a HITS file, so it is a good task. All Windows 10 Atlas tasks (2 CPUs) complete in about 20 minutes and validate, but they produce no HITS files.
Tullio

Your task logs show a couple of different error and warning messages for all of your hosts.
It seems like you configured your #cores and your VM's RAM setting only via the project's web preferences.
This leads (most likely) to a too low RAM setting to run the recent ATLAS tasks.

You may use the following app_config.xml files to solve the problems.

https://lhcathome.cern.ch/lhcathome/show_host_detail.php?hostid=10517701
https://lhcathome.cern.ch/lhcathome/show_host_detail.php?hostid=10510582

<app_config>
  <app>
    <name>ATLAS</name>
    <max_concurrent>1</max_concurrent>
    <report_results_immediately/>
  </app>
  <app_version>
    <app_name>ATLAS</app_name>
    <plan_class>vbox64_mt_mcore_atlas</plan_class>
    <avg_ncpus>1.0</avg_ncpus>
    <cmdline>--nthreads 1 --memory_size_mb 4800</cmdline>
  </app_version>
</app_config>

https://lhcathome.cern.ch/lhcathome/show_host_detail.php?hostid=10407309

<app_config>
  <app>
    <name>ATLAS</name>
    <max_concurrent>1</max_concurrent>
    <report_results_immediately/>
  </app>
  <app_version>
    <app_name>ATLAS</app_name>
    <plan_class>vbox64_mt_mcore_atlas</plan_class>
    <avg_ncpus>2.0</avg_ncpus>
    <cmdline>--nthreads 2 --memory_size_mb 4800</cmdline>
  </app_version>
</app_config>

In addition your windows host shows a rather uncommon error message:

2018-05-24 20:40:00 (568): Error creating VirtualBox instance! rc = 0x80004002

This may point out a problem regarding your VirtualBox installation.
I'm not sure how to solve this - other volunteers may - but you may try a reinstall of VirtualBox.


Also be aware that David Cameron announced today that the ATLAS task queue may be dry during this weekend:
https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=4331&postid=35370

Thanks. I have installed VBox 5.2.12 on the SUN Linux WS and the Windows 10 PC. I am waiting for the Linux laptop to finish its task to do the same. I am against all app_config.xml files. Tasks should run out of the box.
Tullio

... I am against all app_config.xml files. Tasks should run out of the box.

Using an app_config.xml in this case is like helping a small child when it makes it's first steps.
The difference is that a child will learn to walk with or without your help.