Message boards : Theory Application : cranky: [ERROR] No output found - SOLVED
Message board moderation

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
Ryan Munro

Send message
Joined: 17 Aug 17
Posts: 81
Credit: 6,770,932
RAC: 24,675
Message 48201 - Posted: 7 Jun 2023, 10:19:00 UTC

The build is running Linux Mint 21.1, CVMFS appears to be installed ok, cvmfs_config probe shows ok on all counts.
Atlas tasks seem to run fine however Theory tasks are failing instantly:

<core_client_version>7.20.5</core_client_version>
<![CDATA[
<message>
process exited with code 195 (0xc3, -61)</message>
<stderr_txt>
11:02:10 (50718): wrapper (7.15.26016): starting
11:02:10 (50718): wrapper (7.15.26016): starting
11:02:10 (50718): wrapper: running ../../projects/lhcathome.cern.ch_lhcathome/cranky-0.0.32 ()
11:02:10 BST +01:00 2023-06-07: cranky-0.0.32: [INFO] Detected Theory App
11:02:10 BST +01:00 2023-06-07: cranky-0.0.32: [INFO] Checking CVMFS.
11:02:11 BST +01:00 2023-06-07: cranky-0.0.32: [INFO] Checking runc.
11:02:11 BST +01:00 2023-06-07: cranky-0.0.32: [INFO] Creating the filesystem.
11:02:11 BST +01:00 2023-06-07: cranky-0.0.32: [INFO] Using /cvmfs/cernvm-prod.cern.ch/cvm3
11:02:11 BST +01:00 2023-06-07: cranky-0.0.32: [INFO] Updating config.json.
11:02:11 BST +01:00 2023-06-07: cranky-0.0.32: [INFO] Running Container 'runc'.
operation not permitted
11:02:11 BST +01:00 2023-06-07: cranky-0.0.32: [INFO] Container 'runc' finished with status code 1.
11:02:11 BST +01:00 2023-06-07: cranky-0.0.32: [INFO] Preparing output.
11:02:11 BST +01:00 2023-06-07: cranky-0.0.32: [ERROR] No output found.
11:02:12 (50718): cranky exited; CPU time 0.580012
11:02:12 (50718): app exit status: 0xce
11:02:12 (50718): called boinc_finish(195)

</stderr_txt>
]]>


See task https://lhcathome.cern.ch/lhcathome/result.php?resultid=394608415 as an example.

Any ideas?
ID: 48201 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 2158
Credit: 162,593,175
RAC: 122,989
Message 48203 - Posted: 7 Jun 2023, 12:33:50 UTC - in response to Message 48201.  

ID: 48203 · Report as offensive     Reply Quote
Ryan Munro

Send message
Joined: 17 Aug 17
Posts: 81
Credit: 6,770,932
RAC: 24,675
Message 48204 - Posted: 7 Jun 2023, 12:44:21 UTC - in response to Message 48203.  

I have followed that guide and the probe comes back ok.
ID: 48204 · Report as offensive     Reply Quote
kotenok2000
Avatar

Send message
Joined: 21 Feb 11
Posts: 65
Credit: 548,171
RAC: 298
Message 48292 - Posted: 12 Jul 2023, 17:42:19 UTC

What does /etc/systemd/system/boinc-client.service contain?
Try to append systemd.unified_cgroup_hierarchy=false systemd.legacy_systemd_cgroup_controller=false vsyscall=emulate to GRUB_CMDLINE_LINUX_DEFAULT at /etc/default/grub and then run "sudo update-grub".
ID: 48292 · Report as offensive     Reply Quote
Ryan Munro

Send message
Joined: 17 Aug 17
Posts: 81
Credit: 6,770,932
RAC: 24,675
Message 48852 - Posted: 29 Oct 2023, 22:47:13 UTC

I have had to do a reinstall, setting things up again on Mint 21.2 all the Theory apps are failing

<core_client_version>7.18.1</core_client_version>
<![CDATA[
<message>
process exited with code 195 (0xc3, -61)</message>
<stderr_txt>
22:45:00 (15665): wrapper (7.15.26016): starting
22:45:00 (15665): wrapper (7.15.26016): starting
22:45:00 (15665): wrapper: running ../../projects/lhcathome.cern.ch_lhcathome/cranky-0.0.32 ()
22:45:00 GMT +00:00 2023-10-29: cranky-0.0.32: [INFO] Detected Theory App
22:45:00 GMT +00:00 2023-10-29: cranky-0.0.32: [INFO] Checking CVMFS.
22:45:02 GMT +00:00 2023-10-29: cranky-0.0.32: [INFO] Checking runc.
22:45:02 GMT +00:00 2023-10-29: cranky-0.0.32: [INFO] Creating the filesystem.
22:45:02 GMT +00:00 2023-10-29: cranky-0.0.32: [INFO] Using /cvmfs/cernvm-prod.cern.ch/cvm3
22:45:02 GMT +00:00 2023-10-29: cranky-0.0.32: [INFO] Updating config.json.
22:45:02 GMT +00:00 2023-10-29: cranky-0.0.32: [INFO] Running Container 'runc'.
EOF
container_linux.go:336: starting container process caused "process_linux.go:293: applying cgroup configuration for process caused \"mountpoint for cgroup not found\""
22:45:02 GMT +00:00 2023-10-29: cranky-0.0.32: [INFO] Container 'runc' finished with status code 1.
22:45:02 GMT +00:00 2023-10-29: cranky-0.0.32: [INFO] Preparing output.
22:45:02 GMT +00:00 2023-10-29: cranky-0.0.32: [ERROR] No output found.
22:45:02 (15665): cranky exited; CPU time 0.373332
22:45:02 (15665): app exit status: 0xce
22:45:02 (15665): called boinc_finish(195)

</stderr_txt>
]]>


Any thoughts?
ID: 48852 · Report as offensive     Reply Quote
Profile Magic Quantum Mechanic
Avatar

Send message
Joined: 24 Oct 04
Posts: 1137
Credit: 49,947,598
RAC: 2,649
Message 48853 - Posted: 30 Oct 2023, 0:57:20 UTC - in response to Message 48852.  
Last modified: 30 Oct 2023, 1:37:51 UTC

https://lhcathome.cern.ch/lhcathome/show_host_detail.php?hostid=10838833
(are you trying to run Native....looks like you were.....never mind )
ID: 48853 · Report as offensive     Reply Quote
Ryan Munro

Send message
Joined: 17 Aug 17
Posts: 81
Credit: 6,770,932
RAC: 24,675
Message 48858 - Posted: 30 Oct 2023, 21:04:35 UTC

ATLAS seems to be running, though after 5% and 30 mins the CPU hasn't done anything yet, totally idle
ID: 48858 · Report as offensive     Reply Quote
Ryan Munro

Send message
Joined: 17 Aug 17
Posts: 81
Credit: 6,770,932
RAC: 24,675
Message 49539 - Posted: 13 Feb 2024, 8:21:43 UTC - in response to Message 48853.  

Yea weirdly when I first setup this machine they worked fine, then stopped working.
ID: 49539 · Report as offensive     Reply Quote
Crystal Pellet
Volunteer moderator
Volunteer tester

Send message
Joined: 14 Jan 10
Posts: 1330
Credit: 8,761,505
RAC: 5,803
Message 49562 - Posted: 15 Feb 2024, 10:59:48 UTC - in response to Message 49539.  

A lot of Theory tasks from several users got this error after a full run where all events were processed successful.

2024-02-15 11:48:04 (10724): Guest Log: 11:48:04 CET +01:00 2024-02-15: cranky: [INFO] Container 'runc' finished with status code 0.
2024-02-15 11:48:04 (10724): Guest Log: 11:48:04 CET +01:00 2024-02-15: cranky: [INFO] Preparing output.
2024-02-15 11:48:04 (10724): Guest Log: 11:48:04 CET +01:00 2024-02-15: cranky: [ERROR] No output found.
2024-02-15 11:48:04 (10724): Guest Log: [ERROR] Job Failed
2024-02-15 11:48:05 (10724): Guest Log: [INFO] Shutting Down.
2024-02-15 11:48:05 (10724): VM Completion File Detected.
2024-02-15 11:48:05 (10724): VM Completion Message: Job Failed
ID: 49562 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 2158
Credit: 162,593,175
RAC: 122,989
Message 49563 - Posted: 15 Feb 2024, 11:28:04 UTC - in response to Message 49561.  

2024-02-15 12:20:50 (24440): Guest Log: 12:20:45 CET +01:00 2024-02-15: cranky: [INFO] Container 'runc' finished with status code 0.
2024-02-15 12:20:50 (24440): Guest Log: 12:20:45 CET +01:00 2024-02-15: cranky: [INFO] Preparing output.
2024-02-15 12:20:50 (24440): Guest Log: 12:20:45 CET +01:00 2024-02-15: cranky: [ERROR] No output found.
2024-02-15 12:20:50 (24440): Guest Log: [ERROR] Job Failed
ID: 49563 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 2158
Credit: 162,593,175
RAC: 122,989
Message 49564 - Posted: 15 Feb 2024, 11:58:26 UTC - in response to Message 49563.  

In the past, there was an output.tgz in slotfolder not deleted.
Now the slotfolder is empty.
Can this be a reason for this message outputfile not found?
ID: 49564 · Report as offensive     Reply Quote
Crystal Pellet
Volunteer moderator
Volunteer tester

Send message
Joined: 14 Jan 10
Posts: 1330
Credit: 8,761,505
RAC: 5,803
Message 49565 - Posted: 15 Feb 2024, 12:13:09 UTC - in response to Message 49563.  

There are no new workunits from Theory at the moment.
All tasks you get now are resends and a lot of them may suffer from cranky: [ERROR] No output found.

Feel free to check the wingman's error task, whether this is the case and feel free to abort your task.
ID: 49565 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 2158
Credit: 162,593,175
RAC: 122,989
Message 49566 - Posted: 15 Feb 2024, 12:24:21 UTC - in response to Message 49565.  

ID: 49566 · Report as offensive     Reply Quote
Crystal Pellet
Volunteer moderator
Volunteer tester

Send message
Joined: 14 Jan 10
Posts: 1330
Credit: 8,761,505
RAC: 5,803
Message 49567 - Posted: 15 Feb 2024, 12:34:21 UTC - in response to Message 49565.  

It also happens on Linux OS:

02:29:27 PST -08:00 2024-02-15: cranky-0.1.4: [INFO] mcplots runspec: boinc ppbar ue 1800 15 - pythia6 6.425 a 100000 488
03:03:20 PST -08:00 2024-02-15: cranky-0.1.4: [INFO] Container 'runc' finished with status code 0.
03:03:20 PST -08:00 2024-02-15: cranky-0.1.4: [INFO] Preparing output.
03:03:20 PST -08:00 2024-02-15: cranky-0.1.4: [ERROR] No output found.
03:03:20 (2643468): cranky exited; CPU time 2011.126290
03:03:20 (2643468): app exit status: 0xce
03:03:20 (2643468): called boinc_finish(195)
ID: 49567 · Report as offensive     Reply Quote
Saturn911

Send message
Joined: 3 Nov 12
Posts: 49
Credit: 123,410,089
RAC: 137,870
Message 49568 - Posted: 15 Feb 2024, 14:23:56 UTC - in response to Message 49566.  
Last modified: 15 Feb 2024, 14:44:19 UTC

First Theory finished successful :-).
https://lhcathome.cern.ch/lhcathome/workunit.php?wuid=219262572


Not a relevant task... was a resend!
Tasks first sent before 12th Febr midday are o.k.
ID: 49568 · Report as offensive     Reply Quote
yama

Send message
Joined: 9 Aug 23
Posts: 2
Credit: 14,516,609
RAC: 84,128
Message 49569 - Posted: 15 Feb 2024, 14:58:24 UTC

Hi

the same happened to my machines, e.g.:

<core_client_version>7.20.5</core_client_version>
<![CDATA[
<message>
process exited with code 195 (0xc3, -61)</message>
<stderr_txt>
14:15:33 (487647): wrapper (7.15.26016): starting
14:15:33 (487647): wrapper (7.15.26016): starting
14:15:33 (487647): wrapper: running ../../projects/lhcathome.cern.ch_lhcathome/cranky-0.1.4 ()
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Detected Theory App
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] This application must have permanent access to
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] online repositories via a local CVMFS service.
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] It supports suspend/resume if a couple of
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] requirements are fulfilled.
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Most important:
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] - init process is systemd
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] - cgroups v2 is enabled and 'freezer' is available
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] - the user running this application is a member of the 'boinc' group
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] - sudo is at least version 1.9.10
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] - sudoer file provided by LHC@home is installed
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Checking local requirements.
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Found Sudo-Version 1.9.14p2.
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Probing /cvmfs/alice.cern.ch... OK
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Probing /cvmfs/cernvm-prod.cern.ch... OK
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Probing /cvmfs/grid.cern.ch... OK
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Probing /cvmfs/sft.cern.ch... OK
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Excerpt from "cvmfs_config stat": VERSION HOST PROXY
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] 2.11.2.0 http://s1cern-cvmfs.openhtc.io/cvmfs/alice.cern.ch DIRECT
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Found 'runc version spec: 1.0.2-dev' at '/cvmfs/grid.cern.ch/vc/containers/runc.new'.
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Creating container filesystem.
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Using /cvmfs/cernvm-prod.cern.ch/cvm4
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Starting runc container.
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] To get some details on systemd level run
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] systemctl status Theory_2687-2673981-506_2.scope
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] mcplots runspec: boinc pp zinclusive 8000 -,-,280,520 - herwig++ 2.6.3 LHC-UE-EE-SCR-CTEQ6L1 100000 506
14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] ----,^^^^,<<<~_____---,^^^,<<~____--,^^,<~__;_
15:29:22 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Container Theory_2687-2673981-506_2 finished with status code 0.
15:29:22 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Preparing output.
15:29:22 CET +01:00 2024-02-15: cranky-0.1.4: [ERROR] No output found.
15:29:23 (487647): cranky exited; CPU time 4928.509906
15:29:23 (487647): app exit status: 0xce
15:29:23 (487647): called boinc_finish(195)

</stderr_txt>
]]>

Is it worth trying with new work-units or am I missing out somethimg in the setup??

Best regards
Dr. B
ID: 49569 · Report as offensive     Reply Quote
Crystal Pellet
Volunteer moderator
Volunteer tester

Send message
Joined: 14 Jan 10
Posts: 1330
Credit: 8,761,505
RAC: 5,803
Message 49579 - Posted: 16 Feb 2024, 9:39:47 UTC

A new batch of Theory tasks submitted this morning seems to suffer from the same error.
ID: 49579 · Report as offensive     Reply Quote
Matthias Lehmkuhl

Send message
Joined: 15 Jul 05
Posts: 23
Credit: 2,156,651
RAC: 2,147
Message 49585 - Posted: 16 Feb 2024, 17:14:45 UTC

count me in, I have the same issue
<core_client_version>7.24.1</core_client_version>
<![CDATA[
<message>
Unzul�ssige Funktion.
(0x1) - exit code 1 (0x1)</message>
<stderr_txt>
2024-02-16 15:13:43 (13272): Detected: vboxwrapper 26206
2024-02-16 15:13:43 (13272): Detected: BOINC client v7.24.1
2024-02-16 15:14:15 (13272): Detected: VirtualBox VboxManage Interface (Version: 7.0.14)
2024-02-16 15:14:15 (13272): Detected: Heartbeat check (file: 'heartbeat' every 1200.000000 seconds)
2024-02-16 15:14:15 (13272): Successfully copied 'init_data.xml' to the shared directory.
2024-02-16 15:14:15 (13272): Successfully copied 'input' to the shared directory.
...
2024-02-16 15:15:51 (13272): Guest Log: job: htmld=/shared/html/job
2024-02-16 15:15:56 (13272): Guest Log: job: unpack exitcode=0
2024-02-16 15:15:57 (13272): Guest Log: INFO: activated the work-around for ld:
2024-02-16 15:15:57 (13272): Guest Log: lrwxrwxrwx 1 0 0 15 Feb 16 14:16 /tmp/tmp.slQVZPg0L8/ld -> /usr/bin/ld.bfd
2024-02-16 15:15:59 (13272): Guest Log: 15:16:05 CET +01:00 2024-02-16: cranky: [INFO] ===> [runRivet] Fri Feb 16 14:16:03 UTC 2024 [boinc pp jets 7000 80,-,960 - herwig++ 2.5.1 default 100000 636]
2024-02-16 16:54:38 (13272): Status Report: Job Duration: '864000.000000'
2024-02-16 16:54:38 (13272): Status Report: Elapsed Time: '6000.000000'
2024-02-16 16:54:38 (13272): Status Report: CPU Time: '5221.843750'
2024-02-16 16:56:01 (13272): Guest Log: job: run exitcode=0
2024-02-16 16:56:01 (13272): Guest Log: job: diskusage=3800
2024-02-16 16:56:01 (13272): Guest Log: job: logsize=856 k
2024-02-16 16:56:01 (13272): Guest Log: job: times=
2024-02-16 16:56:01 (13272): Guest Log: 0m0.013s 0m0.019s
2024-02-16 16:56:01 (13272): Guest Log: 85m7.489s 0m35.604s
2024-02-16 16:56:02 (13272): Guest Log: job: cpuusage=5143
2024-02-16 16:56:08 (13272): Guest Log: 16:56:11 CET +01:00 2024-02-16: cranky: [INFO] Container 'runc' finished with status code 0.
2024-02-16 16:56:08 (13272): Guest Log: 16:56:11 CET +01:00 2024-02-16: cranky: [INFO] Preparing output.
2024-02-16 16:56:08 (13272): Guest Log: 16:56:11 CET +01:00 2024-02-16: cranky: [ERROR] No output found.
2024-02-16 16:56:09 (13272): Guest Log: [ERROR] Job Failed
2024-02-16 16:56:09 (13272): Guest Log: [INFO] Shutting Down.
2024-02-16 16:56:09 (13272): VM Completion File Detected.
2024-02-16 16:56:09 (13272): VM Completion Message: Job Failed
.
2024-02-16 16:56:09 (13272): Powering off VM.
2024-02-16 16:56:10 (13272): Successfully stopped VM.
2024-02-16 16:56:10 (13272): Deregistering VM. (boinc_c920f1f06ef991de, slot#10)
2024-02-16 16:56:10 (13272): Removing network bandwidth throttle group from VM.
2024-02-16 16:56:10 (13272): Removing VM from VirtualBox.
16:56:16 (13272): called boinc_finish(1)

</stderr_txt>
https://lhcathome.cern.ch/lhcathome/result.php?resultid=406183164
https://lhcathome.cern.ch/lhcathome/result.php?resultid=406181094
https://lhcathome.cern.ch/lhcathome/result.php?resultid=406130991
https://lhcathome.cern.ch/lhcathome/result.php?resultid=406131183
https://lhcathome.cern.ch/lhcathome/result.php?resultid=406131213
https://lhcathome.cern.ch/lhcathome/result.php?resultid=406125901
Matthias

ID: 49585 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1721
Credit: 107,700,897
RAC: 102,203
Message 49588 - Posted: 17 Feb 2024, 6:27:44 UTC
Last modified: 17 Feb 2024, 6:28:27 UTC

same thing here, a few minutes ago:
...
2024-02-17 07:23:20 (19824): Guest Log: 07:17:59 CET +01:00 2024-02-17: cranky: [INFO] Container 'runc' finished with status code 0.
2024-02-17 07:23:20 (19824): Guest Log: 07:17:59 CET +01:00 2024-02-17: cranky: [INFO] Preparing output.
2024-02-17 07:23:20 (19824): Guest Log: 07:17:59 CET +01:00 2024-02-17: cranky: [ERROR] No output found.
2024-02-17 07:23:20 (19824): Guest Log: [ERROR] Job Failed
2024-02-17 07:23:20 (19824): Guest Log: [INFO] Shutting Down.
...
https://lhcathome.cern.ch/lhcathome/result.php?resultid=406202842
ID: 49588 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1721
Credit: 107,700,897
RAC: 102,203
Message 49589 - Posted: 17 Feb 2024, 8:02:44 UTC

I was not aware that obviously ALL Theory tasks are faulty now.
I downloaded them on several hosts, and all tasks errored out after some time, up to after more than 1 hour :-(

So I am wondering why no-one back at LHC&Home stops the download of Theory tasks and empties the queue. The result is a real waste of ressources on the volunteers' side.
ID: 49589 · Report as offensive     Reply Quote
1 · 2 · Next

Message boards : Theory Application : cranky: [ERROR] No output found - SOLVED


©2024 CERN