LHC@home

The build is running Linux Mint 21.1, CVMFS appears to be installed ok, cvmfs_config probe shows ok on all counts.
Atlas tasks seem to run fine however Theory tasks are failing instantly:

<core_client_version>7.20.5</core_client_version>
<![CDATA[
<message>
process exited with code 195 (0xc3, -61)</message>
<stderr_txt>
11:02:10 (50718): wrapper (7.15.26016): starting
11:02:10 (50718): wrapper (7.15.26016): starting
11:02:10 (50718): wrapper: running ../../projects/lhcathome.cern.ch_lhcathome/cranky-0.0.32 ()
11:02:10 BST +01:00 2023-06-07: cranky-0.0.32: [INFO] Detected Theory App
11:02:10 BST +01:00 2023-06-07: cranky-0.0.32: [INFO] Checking CVMFS.
11:02:11 BST +01:00 2023-06-07: cranky-0.0.32: [INFO] Checking runc.
11:02:11 BST +01:00 2023-06-07: cranky-0.0.32: [INFO] Creating the filesystem.
11:02:11 BST +01:00 2023-06-07: cranky-0.0.32: [INFO] Using /cvmfs/cernvm-prod.cern.ch/cvm3
11:02:11 BST +01:00 2023-06-07: cranky-0.0.32: [INFO] Updating config.json.
11:02:11 BST +01:00 2023-06-07: cranky-0.0.32: [INFO] Running Container 'runc'.
operation not permitted
11:02:11 BST +01:00 2023-06-07: cranky-0.0.32: [INFO] Container 'runc' finished with status code 1.
11:02:11 BST +01:00 2023-06-07: cranky-0.0.32: [INFO] Preparing output.
11:02:11 BST +01:00 2023-06-07: cranky-0.0.32: [ERROR] No output found.
11:02:12 (50718): cranky exited; CPU time 0.580012
11:02:12 (50718): app exit status: 0xce
11:02:12 (50718): called boinc_finish(195)

</stderr_txt>
]]>

See task https://lhcathome.cern.ch/lhcathome/result.php?resultid=394608415 as an example.

Any ideas?

I have followed that guide and the probe comes back ok.

What does /etc/systemd/system/boinc-client.service contain?
Try to append systemd.unified_cgroup_hierarchy=false systemd.legacy_systemd_cgroup_controller=false vsyscall=emulate to GRUB_CMDLINE_LINUX_DEFAULT at /etc/default/grub and then run "sudo update-grub".

I have had to do a reinstall, setting things up again on Mint 21.2 all the Theory apps are failing

<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
process exited with code 195 (0xc3, -61)</message>
<stderr_txt>
22:45:00 (15665): wrapper (7.15.26016): starting
22:45:00 (15665): wrapper (7.15.26016): starting
22:45:00 (15665): wrapper: running ../../projects/lhcathome.cern.ch_lhcathome/cranky-0.0.32 ()
22:45:00 GMT +00:00 2023-10-29: cranky-0.0.32: [INFO] Detected Theory App
22:45:00 GMT +00:00 2023-10-29: cranky-0.0.32: [INFO] Checking CVMFS.
22:45:02 GMT +00:00 2023-10-29: cranky-0.0.32: [INFO] Checking runc.
22:45:02 GMT +00:00 2023-10-29: cranky-0.0.32: [INFO] Creating the filesystem.
22:45:02 GMT +00:00 2023-10-29: cranky-0.0.32: [INFO] Using /cvmfs/cernvm-prod.cern.ch/cvm3
22:45:02 GMT +00:00 2023-10-29: cranky-0.0.32: [INFO] Updating config.json.
22:45:02 GMT +00:00 2023-10-29: cranky-0.0.32: [INFO] Running Container 'runc'.
EOF
container_linux.go:336: starting container process caused "process_linux.go:293: applying cgroup configuration for process caused \"mountpoint for cgroup not found\""
22:45:02 GMT +00:00 2023-10-29: cranky-0.0.32: [INFO] Container 'runc' finished with status code 1.
22:45:02 GMT +00:00 2023-10-29: cranky-0.0.32: [INFO] Preparing output.
22:45:02 GMT +00:00 2023-10-29: cranky-0.0.32: [ERROR] No output found.
22:45:02 (15665): cranky exited; CPU time 0.373332
22:45:02 (15665): app exit status: 0xce
22:45:02 (15665): called boinc_finish(195)

</stderr_txt>
]]>

Any thoughts?

https://lhcathome.cern.ch/lhcathome/show_host_detail.php?hostid=10838833
(are you trying to run Native....looks like you were.....never mind )

ATLAS seems to be running, though after 5% and 30 mins the CPU hasn't done anything yet, totally idle

Yea weirdly when I first setup this machine they worked fine, then stopped working.

A lot of Theory tasks from several users got this error after a full run where all events were processed successful.

2024-02-15 11:48:04 (10724): Guest Log: 11:48:04 CET +01:00 2024-02-15: cranky: [INFO] Container 'runc' finished with status code 0.
2024-02-15 11:48:04 (10724): Guest Log: 11:48:04 CET +01:00 2024-02-15: cranky: [INFO] Preparing output.
2024-02-15 11:48:04 (10724): Guest Log: 11:48:04 CET +01:00 2024-02-15: cranky: [ERROR] No output found.
2024-02-15 11:48:04 (10724): Guest Log: [ERROR] Job Failed
2024-02-15 11:48:05 (10724): Guest Log: [INFO] Shutting Down.
2024-02-15 11:48:05 (10724): VM Completion File Detected.
2024-02-15 11:48:05 (10724): VM Completion Message: Job Failed

2024-02-15 12:20:50 (24440): Guest Log: 12:20:45 CET +01:00 2024-02-15: cranky: [INFO] Container 'runc' finished with status code 0.
2024-02-15 12:20:50 (24440): Guest Log: 12:20:45 CET +01:00 2024-02-15: cranky: [INFO] Preparing output.
2024-02-15 12:20:50 (24440): Guest Log: 12:20:45 CET +01:00 2024-02-15: cranky: [ERROR] No output found.
2024-02-15 12:20:50 (24440): Guest Log: [ERROR] Job Failed

In the past, there was an output.tgz in slotfolder not deleted.
Now the slotfolder is empty.
Can this be a reason for this message outputfile not found?

There are no new workunits from Theory at the moment.
All tasks you get now are resends and a lot of them may suffer from cranky: [ERROR] No output found.

Feel free to check the wingman's error task, whether this is the case and feel free to abort your task.

First Theory finished successful :-).
https://lhcathome.cern.ch/lhcathome/workunit.php?wuid=219262572

It also happens on Linux OS:

02:29:27 PST -08:00 2024-02-15: cranky-0.1.4: [INFO] mcplots runspec: boinc ppbar ue 1800 15 - pythia6 6.425 a 100000 488
03:03:20 PST -08:00 2024-02-15: cranky-0.1.4: [INFO] Container 'runc' finished with status code 0.
03:03:20 PST -08:00 2024-02-15: cranky-0.1.4: [INFO] Preparing output.
03:03:20 PST -08:00 2024-02-15: cranky-0.1.4: [ERROR] No output found.
03:03:20 (2643468): cranky exited; CPU time 2011.126290
03:03:20 (2643468): app exit status: 0xce
03:03:20 (2643468): called boinc_finish(195)

First Theory finished successful :-).
https://lhcathome.cern.ch/lhcathome/workunit.php?wuid=219262572

Not a relevant task... was a resend!
Tasks first sent before 12th Febr midday are o.k.

Hi

the same happened to my machines, e.g.:

<core_client_version>7.20.5</core_client_version>
<![CDATA[
<message>
process exited with code 195 (0xc3, -61)</message>
<stderr_txt>
14:15:33 (487647): wrapper (7.15.26016): starting
14:15:33 (487647): wrapper (7.15.26016): starting
14:15:33 (487647): wrapper: running ../../projects/lhcathome.cern.ch_lhcathome/cranky-0.1.4 ()
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Detected Theory App
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] This application must have permanent access to
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] online repositories via a local CVMFS service.
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] It supports suspend/resume if a couple of
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] requirements are fulfilled.
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Most important:
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] - init process is systemd
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] - cgroups v2 is enabled and 'freezer' is available
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] - the user running this application is a member of the 'boinc' group
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] - sudo is at least version 1.9.10
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] - sudoer file provided by LHC@home is installed
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Checking local requirements.
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Found Sudo-Version 1.9.14p2.
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Probing /cvmfs/alice.cern.ch... OK
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Probing /cvmfs/cernvm-prod.cern.ch... OK
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Probing /cvmfs/grid.cern.ch... OK
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Probing /cvmfs/sft.cern.ch... OK
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Excerpt from "cvmfs_config stat": VERSION HOST PROXY
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] 2.11.2.0 http://s1cern-cvmfs.openhtc.io/cvmfs/alice.cern.ch DIRECT
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Found 'runc version spec: 1.0.2-dev' at '/cvmfs/grid.cern.ch/vc/containers/runc.new'.
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Creating container filesystem.
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Using /cvmfs/cernvm-prod.cern.ch/cvm4
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Starting runc container.
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] To get some details on systemd level run
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] systemctl status Theory_2687-2673981-506_2.scope
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] mcplots runspec: boinc pp zinclusive 8000 -,-,280,520 - herwig++ 2.6.3 LHC-UE-EE-SCR-CTEQ6L1 100000 506
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] ----,^^^^,<<<~_____---,^^^,<<~____--,^^,<~__;_
15:29:22 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Container Theory_2687-2673981-506_2 finished with status code 0.
15:29:22 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Preparing output.
15:29:22 CET +01:00 2024-02-15: cranky-0.1.4: [ERROR] No output found.
15:29:23 (487647): cranky exited; CPU time 4928.509906
15:29:23 (487647): app exit status: 0xce
15:29:23 (487647): called boinc_finish(195)

</stderr_txt>
]]>

Is it worth trying with new work-units or am I missing out somethimg in the setup??

Best regards
Dr. B

count me in, I have the same issue
<core_client_version>7.24.1</core_client_version>
<![CDATA[
<message>
Unzul�ssige Funktion.
(0x1) - exit code 1 (0x1)</message>
<stderr_txt>
2024-02-16 15:13:43 (13272): Detected: vboxwrapper 26206
2024-02-16 15:13:43 (13272): Detected: BOINC client v7.24.1
2024-02-16 15:14:15 (13272): Detected: VirtualBox VboxManage Interface (Version: 7.0.14)
2024-02-16 15:14:15 (13272): Detected: Heartbeat check (file: 'heartbeat' every 1200.000000 seconds)
2024-02-16 15:14:15 (13272): Successfully copied 'init_data.xml' to the shared directory.
2024-02-16 15:14:15 (13272): Successfully copied 'input' to the shared directory.
...
2024-02-16 15:15:51 (13272): Guest Log: job: htmld=/shared/html/job
2024-02-16 15:15:56 (13272): Guest Log: job: unpack exitcode=0
2024-02-16 15:15:57 (13272): Guest Log: INFO: activated the work-around for ld:
2024-02-16 15:15:57 (13272): Guest Log: lrwxrwxrwx 1 0 0 15 Feb 16 14:16 /tmp/tmp.slQVZPg0L8/ld -> /usr/bin/ld.bfd
2024-02-16 15:15:59 (13272): Guest Log: 15:16:05 CET +01:00 2024-02-16: cranky: [INFO] ===> [runRivet] Fri Feb 16 14:16:03 UTC 2024 [boinc pp jets 7000 80,-,960 - herwig++ 2.5.1 default 100000 636]
2024-02-16 16:54:38 (13272): Status Report: Job Duration: '864000.000000'
2024-02-16 16:54:38 (13272): Status Report: Elapsed Time: '6000.000000'
2024-02-16 16:54:38 (13272): Status Report: CPU Time: '5221.843750'
2024-02-16 16:56:01 (13272): Guest Log: job: run exitcode=0
2024-02-16 16:56:01 (13272): Guest Log: job: diskusage=3800
2024-02-16 16:56:01 (13272): Guest Log: job: logsize=856 k
2024-02-16 16:56:01 (13272): Guest Log: job: times=
2024-02-16 16:56:01 (13272): Guest Log: 0m0.013s 0m0.019s
2024-02-16 16:56:01 (13272): Guest Log: 85m7.489s 0m35.604s
2024-02-16 16:56:02 (13272): Guest Log: job: cpuusage=5143
2024-02-16 16:56:08 (13272): Guest Log: 16:56:11 CET +01:00 2024-02-16: cranky: [INFO] Container 'runc' finished with status code 0.
2024-02-16 16:56:08 (13272): Guest Log: 16:56:11 CET +01:00 2024-02-16: cranky: [INFO] Preparing output.
2024-02-16 16:56:08 (13272): Guest Log: 16:56:11 CET +01:00 2024-02-16: cranky: [ERROR] No output found.
2024-02-16 16:56:09 (13272): Guest Log: [ERROR] Job Failed
2024-02-16 16:56:09 (13272): Guest Log: [INFO] Shutting Down.
2024-02-16 16:56:09 (13272): VM Completion File Detected.
2024-02-16 16:56:09 (13272): VM Completion Message: Job Failed
.
2024-02-16 16:56:09 (13272): Powering off VM.
2024-02-16 16:56:10 (13272): Successfully stopped VM.
2024-02-16 16:56:10 (13272): Deregistering VM. (boinc_c920f1f06ef991de, slot#10)
2024-02-16 16:56:10 (13272): Removing network bandwidth throttle group from VM.
2024-02-16 16:56:10 (13272): Removing VM from VirtualBox.
16:56:16 (13272): called boinc_finish(1)

</stderr_txt>
https://lhcathome.cern.ch/lhcathome/result.php?resultid=406183164
https://lhcathome.cern.ch/lhcathome/result.php?resultid=406181094
https://lhcathome.cern.ch/lhcathome/result.php?resultid=406130991
https://lhcathome.cern.ch/lhcathome/result.php?resultid=406131183
https://lhcathome.cern.ch/lhcathome/result.php?resultid=406131213
https://lhcathome.cern.ch/lhcathome/result.php?resultid=406125901

---

Matthias

same thing here, a few minutes ago:
...
2024-02-17 07:23:20 (19824): Guest Log: 07:17:59 CET +01:00 2024-02-17: cranky: [INFO] Container 'runc' finished with status code 0.
2024-02-17 07:23:20 (19824): Guest Log: 07:17:59 CET +01:00 2024-02-17: cranky: [INFO] Preparing output.
2024-02-17 07:23:20 (19824): Guest Log: 07:17:59 CET +01:00 2024-02-17: cranky: [ERROR] No output found.
2024-02-17 07:23:20 (19824): Guest Log: [ERROR] Job Failed
2024-02-17 07:23:20 (19824): Guest Log: [INFO] Shutting Down.
...
https://lhcathome.cern.ch/lhcathome/result.php?resultid=406202842

I was not aware that obviously ALL Theory tasks are faulty now.
I downloaded them on several hosts, and all tasks errored out after some time, up to after more than 1 hour :-(

So I am wondering why no-one back at LHC&Home stops the download of Theory tasks and empties the queue. The result is a real waste of ressources on the volunteers' side.