Message boards :
Theory Application :
cranky: [ERROR] No output found - SOLVED
Message board moderation
Author | Message |
---|---|
Send message Joined: 17 Aug 17 Posts: 81 Credit: 8,123,274 RAC: 11,445 |
The build is running Linux Mint 21.1, CVMFS appears to be installed ok, cvmfs_config probe shows ok on all counts. Atlas tasks seem to run fine however Theory tasks are failing instantly: <core_client_version>7.20.5</core_client_version> <![CDATA[ <message> process exited with code 195 (0xc3, -61)</message> <stderr_txt> 11:02:10 (50718): wrapper (7.15.26016): starting 11:02:10 (50718): wrapper (7.15.26016): starting 11:02:10 (50718): wrapper: running ../../projects/lhcathome.cern.ch_lhcathome/cranky-0.0.32 () 11:02:10 BST +01:00 2023-06-07: cranky-0.0.32: [INFO] Detected Theory App 11:02:10 BST +01:00 2023-06-07: cranky-0.0.32: [INFO] Checking CVMFS. 11:02:11 BST +01:00 2023-06-07: cranky-0.0.32: [INFO] Checking runc. 11:02:11 BST +01:00 2023-06-07: cranky-0.0.32: [INFO] Creating the filesystem. 11:02:11 BST +01:00 2023-06-07: cranky-0.0.32: [INFO] Using /cvmfs/cernvm-prod.cern.ch/cvm3 11:02:11 BST +01:00 2023-06-07: cranky-0.0.32: [INFO] Updating config.json. 11:02:11 BST +01:00 2023-06-07: cranky-0.0.32: [INFO] Running Container 'runc'. operation not permitted 11:02:11 BST +01:00 2023-06-07: cranky-0.0.32: [INFO] Container 'runc' finished with status code 1. 11:02:11 BST +01:00 2023-06-07: cranky-0.0.32: [INFO] Preparing output. 11:02:11 BST +01:00 2023-06-07: cranky-0.0.32: [ERROR] No output found. 11:02:12 (50718): cranky exited; CPU time 0.580012 11:02:12 (50718): app exit status: 0xce 11:02:12 (50718): called boinc_finish(195) </stderr_txt> ]]> See task https://lhcathome.cern.ch/lhcathome/result.php?resultid=394608415 as an example. Any ideas? |
Send message Joined: 2 May 07 Posts: 2176 Credit: 172,465,898 RAC: 76,559 |
|
Send message Joined: 17 Aug 17 Posts: 81 Credit: 8,123,274 RAC: 11,445 |
I have followed that guide and the probe comes back ok. |
Send message Joined: 21 Feb 11 Posts: 65 Credit: 548,727 RAC: 46 |
What does /etc/systemd/system/boinc-client.service contain? Try to append systemd.unified_cgroup_hierarchy=false systemd.legacy_systemd_cgroup_controller=false vsyscall=emulate to GRUB_CMDLINE_LINUX_DEFAULT at /etc/default/grub and then run "sudo update-grub". |
Send message Joined: 17 Aug 17 Posts: 81 Credit: 8,123,274 RAC: 11,445 |
I have had to do a reinstall, setting things up again on Mint 21.2 all the Theory apps are failing <core_client_version>7.18.1</core_client_version> <![CDATA[ <message> process exited with code 195 (0xc3, -61)</message> <stderr_txt> 22:45:00 (15665): wrapper (7.15.26016): starting 22:45:00 (15665): wrapper (7.15.26016): starting 22:45:00 (15665): wrapper: running ../../projects/lhcathome.cern.ch_lhcathome/cranky-0.0.32 () 22:45:00 GMT +00:00 2023-10-29: cranky-0.0.32: [INFO] Detected Theory App 22:45:00 GMT +00:00 2023-10-29: cranky-0.0.32: [INFO] Checking CVMFS. 22:45:02 GMT +00:00 2023-10-29: cranky-0.0.32: [INFO] Checking runc. 22:45:02 GMT +00:00 2023-10-29: cranky-0.0.32: [INFO] Creating the filesystem. 22:45:02 GMT +00:00 2023-10-29: cranky-0.0.32: [INFO] Using /cvmfs/cernvm-prod.cern.ch/cvm3 22:45:02 GMT +00:00 2023-10-29: cranky-0.0.32: [INFO] Updating config.json. 22:45:02 GMT +00:00 2023-10-29: cranky-0.0.32: [INFO] Running Container 'runc'. EOF container_linux.go:336: starting container process caused "process_linux.go:293: applying cgroup configuration for process caused \"mountpoint for cgroup not found\"" 22:45:02 GMT +00:00 2023-10-29: cranky-0.0.32: [INFO] Container 'runc' finished with status code 1. 22:45:02 GMT +00:00 2023-10-29: cranky-0.0.32: [INFO] Preparing output. 22:45:02 GMT +00:00 2023-10-29: cranky-0.0.32: [ERROR] No output found. 22:45:02 (15665): cranky exited; CPU time 0.373332 22:45:02 (15665): app exit status: 0xce 22:45:02 (15665): called boinc_finish(195) </stderr_txt> ]]> Any thoughts? |
Send message Joined: 24 Oct 04 Posts: 1155 Credit: 51,448,841 RAC: 39,684 |
https://lhcathome.cern.ch/lhcathome/show_host_detail.php?hostid=10838833 (are you trying to run Native....looks like you were.....never mind ) |
Send message Joined: 17 Aug 17 Posts: 81 Credit: 8,123,274 RAC: 11,445 |
ATLAS seems to be running, though after 5% and 30 mins the CPU hasn't done anything yet, totally idle |
Send message Joined: 17 Aug 17 Posts: 81 Credit: 8,123,274 RAC: 11,445 |
Yea weirdly when I first setup this machine they worked fine, then stopped working. |
Send message Joined: 14 Jan 10 Posts: 1346 Credit: 9,081,032 RAC: 4,176 |
A lot of Theory tasks from several users got this error after a full run where all events were processed successful. 2024-02-15 11:48:04 (10724): Guest Log: 11:48:04 CET +01:00 2024-02-15: cranky: [INFO] Container 'runc' finished with status code 0. 2024-02-15 11:48:04 (10724): Guest Log: 11:48:04 CET +01:00 2024-02-15: cranky: [INFO] Preparing output. 2024-02-15 11:48:04 (10724): Guest Log: 11:48:04 CET +01:00 2024-02-15: cranky: [ERROR] No output found. 2024-02-15 11:48:04 (10724): Guest Log: [ERROR] Job Failed 2024-02-15 11:48:05 (10724): Guest Log: [INFO] Shutting Down. 2024-02-15 11:48:05 (10724): VM Completion File Detected. 2024-02-15 11:48:05 (10724): VM Completion Message: Job Failed |
Send message Joined: 2 May 07 Posts: 2176 Credit: 172,465,898 RAC: 76,559 |
2024-02-15 12:20:50 (24440): Guest Log: 12:20:45 CET +01:00 2024-02-15: cranky: [INFO] Container 'runc' finished with status code 0. 2024-02-15 12:20:50 (24440): Guest Log: 12:20:45 CET +01:00 2024-02-15: cranky: [INFO] Preparing output. 2024-02-15 12:20:50 (24440): Guest Log: 12:20:45 CET +01:00 2024-02-15: cranky: [ERROR] No output found. 2024-02-15 12:20:50 (24440): Guest Log: [ERROR] Job Failed |
Send message Joined: 2 May 07 Posts: 2176 Credit: 172,465,898 RAC: 76,559 |
In the past, there was an output.tgz in slotfolder not deleted. Now the slotfolder is empty. Can this be a reason for this message outputfile not found? |
Send message Joined: 14 Jan 10 Posts: 1346 Credit: 9,081,032 RAC: 4,176 |
There are no new workunits from Theory at the moment. All tasks you get now are resends and a lot of them may suffer from cranky: [ERROR] No output found. Feel free to check the wingman's error task, whether this is the case and feel free to abort your task. |
Send message Joined: 2 May 07 Posts: 2176 Credit: 172,465,898 RAC: 76,559 |
First Theory finished successful :-). https://lhcathome.cern.ch/lhcathome/workunit.php?wuid=219262572 |
Send message Joined: 14 Jan 10 Posts: 1346 Credit: 9,081,032 RAC: 4,176 |
It also happens on Linux OS: 02:29:27 PST -08:00 2024-02-15: cranky-0.1.4: [INFO] mcplots runspec: boinc ppbar ue 1800 15 - pythia6 6.425 a 100000 488 03:03:20 PST -08:00 2024-02-15: cranky-0.1.4: [INFO] Container 'runc' finished with status code 0. 03:03:20 PST -08:00 2024-02-15: cranky-0.1.4: [INFO] Preparing output. 03:03:20 PST -08:00 2024-02-15: cranky-0.1.4: [ERROR] No output found. 03:03:20 (2643468): cranky exited; CPU time 2011.126290 03:03:20 (2643468): app exit status: 0xce 03:03:20 (2643468): called boinc_finish(195) |
Send message Joined: 3 Nov 12 Posts: 54 Credit: 134,817,843 RAC: 111,360 |
First Theory finished successful :-). Not a relevant task... was a resend! Tasks first sent before 12th Febr midday are o.k. |
Send message Joined: 9 Aug 23 Posts: 2 Credit: 19,159,070 RAC: 21,869 |
Hi the same happened to my machines, e.g.: <core_client_version>7.20.5</core_client_version> <![CDATA[ <message> process exited with code 195 (0xc3, -61)</message> <stderr_txt> 14:15:33 (487647): wrapper (7.15.26016): starting 14:15:33 (487647): wrapper (7.15.26016): starting 14:15:33 (487647): wrapper: running ../../projects/lhcathome.cern.ch_lhcathome/cranky-0.1.4 () 14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Detected Theory App 14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] This application must have permanent access to 14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] online repositories via a local CVMFS service. 14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] It supports suspend/resume if a couple of 14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] requirements are fulfilled. 14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Most important: 14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] - init process is systemd 14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] - cgroups v2 is enabled and 'freezer' is available 14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] - the user running this application is a member of the 'boinc' group 14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] - sudo is at least version 1.9.10 14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] - sudoer file provided by LHC@home is installed 14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Checking local requirements. 14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Found Sudo-Version 1.9.14p2. 14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Probing /cvmfs/alice.cern.ch... OK 14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Probing /cvmfs/cernvm-prod.cern.ch... OK 14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Probing /cvmfs/grid.cern.ch... OK 14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Probing /cvmfs/sft.cern.ch... OK 14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Excerpt from "cvmfs_config stat": VERSION HOST PROXY 14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] 2.11.2.0 http://s1cern-cvmfs.openhtc.io/cvmfs/alice.cern.ch DIRECT 14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Found 'runc version spec: 1.0.2-dev' at '/cvmfs/grid.cern.ch/vc/containers/runc.new'. 14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Creating container filesystem. 14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Using /cvmfs/cernvm-prod.cern.ch/cvm4 14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Starting runc container. 14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] To get some details on systemd level run 14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] systemctl status Theory_2687-2673981-506_2.scope 14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] mcplots runspec: boinc pp zinclusive 8000 -,-,280,520 - herwig++ 2.6.3 LHC-UE-EE-SCR-CTEQ6L1 100000 506 14:15:33 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] ----,^^^^,<<<~_____---,^^^,<<~____--,^^,<~__;_ 15:29:22 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Container Theory_2687-2673981-506_2 finished with status code 0. 15:29:22 CET +01:00 2024-02-15: cranky-0.1.4: [INFO] Preparing output. 15:29:22 CET +01:00 2024-02-15: cranky-0.1.4: [ERROR] No output found. 15:29:23 (487647): cranky exited; CPU time 4928.509906 15:29:23 (487647): app exit status: 0xce 15:29:23 (487647): called boinc_finish(195) </stderr_txt> ]]> Is it worth trying with new work-units or am I missing out somethimg in the setup?? Best regards Dr. B |
Send message Joined: 14 Jan 10 Posts: 1346 Credit: 9,081,032 RAC: 4,176 |
A new batch of Theory tasks submitted this morning seems to suffer from the same error. |
Send message Joined: 15 Jul 05 Posts: 23 Credit: 2,332,408 RAC: 2,419 |
count me in, I have the same issue <core_client_version>7.24.1</core_client_version> <![CDATA[ <message> Unzul�ssige Funktion. (0x1) - exit code 1 (0x1)</message> <stderr_txt> 2024-02-16 15:13:43 (13272): Detected: vboxwrapper 26206 2024-02-16 15:13:43 (13272): Detected: BOINC client v7.24.1 2024-02-16 15:14:15 (13272): Detected: VirtualBox VboxManage Interface (Version: 7.0.14) 2024-02-16 15:14:15 (13272): Detected: Heartbeat check (file: 'heartbeat' every 1200.000000 seconds) 2024-02-16 15:14:15 (13272): Successfully copied 'init_data.xml' to the shared directory. 2024-02-16 15:14:15 (13272): Successfully copied 'input' to the shared directory. ... 2024-02-16 15:15:51 (13272): Guest Log: job: htmld=/shared/html/job 2024-02-16 15:15:56 (13272): Guest Log: job: unpack exitcode=0 2024-02-16 15:15:57 (13272): Guest Log: INFO: activated the work-around for ld: 2024-02-16 15:15:57 (13272): Guest Log: lrwxrwxrwx 1 0 0 15 Feb 16 14:16 /tmp/tmp.slQVZPg0L8/ld -> /usr/bin/ld.bfd 2024-02-16 15:15:59 (13272): Guest Log: 15:16:05 CET +01:00 2024-02-16: cranky: [INFO] ===> [runRivet] Fri Feb 16 14:16:03 UTC 2024 [boinc pp jets 7000 80,-,960 - herwig++ 2.5.1 default 100000 636] 2024-02-16 16:54:38 (13272): Status Report: Job Duration: '864000.000000' 2024-02-16 16:54:38 (13272): Status Report: Elapsed Time: '6000.000000' 2024-02-16 16:54:38 (13272): Status Report: CPU Time: '5221.843750' 2024-02-16 16:56:01 (13272): Guest Log: job: run exitcode=0 2024-02-16 16:56:01 (13272): Guest Log: job: diskusage=3800 2024-02-16 16:56:01 (13272): Guest Log: job: logsize=856 k 2024-02-16 16:56:01 (13272): Guest Log: job: times= 2024-02-16 16:56:01 (13272): Guest Log: 0m0.013s 0m0.019s 2024-02-16 16:56:01 (13272): Guest Log: 85m7.489s 0m35.604s 2024-02-16 16:56:02 (13272): Guest Log: job: cpuusage=5143 2024-02-16 16:56:08 (13272): Guest Log: 16:56:11 CET +01:00 2024-02-16: cranky: [INFO] Container 'runc' finished with status code 0. 2024-02-16 16:56:08 (13272): Guest Log: 16:56:11 CET +01:00 2024-02-16: cranky: [INFO] Preparing output. 2024-02-16 16:56:08 (13272): Guest Log: 16:56:11 CET +01:00 2024-02-16: cranky: [ERROR] No output found. 2024-02-16 16:56:09 (13272): Guest Log: [ERROR] Job Failed 2024-02-16 16:56:09 (13272): Guest Log: [INFO] Shutting Down. 2024-02-16 16:56:09 (13272): VM Completion File Detected. 2024-02-16 16:56:09 (13272): VM Completion Message: Job Failed . 2024-02-16 16:56:09 (13272): Powering off VM. 2024-02-16 16:56:10 (13272): Successfully stopped VM. 2024-02-16 16:56:10 (13272): Deregistering VM. (boinc_c920f1f06ef991de, slot#10) 2024-02-16 16:56:10 (13272): Removing network bandwidth throttle group from VM. 2024-02-16 16:56:10 (13272): Removing VM from VirtualBox. 16:56:16 (13272): called boinc_finish(1) </stderr_txt> https://lhcathome.cern.ch/lhcathome/result.php?resultid=406183164 https://lhcathome.cern.ch/lhcathome/result.php?resultid=406181094 https://lhcathome.cern.ch/lhcathome/result.php?resultid=406130991 https://lhcathome.cern.ch/lhcathome/result.php?resultid=406131183 https://lhcathome.cern.ch/lhcathome/result.php?resultid=406131213 https://lhcathome.cern.ch/lhcathome/result.php?resultid=406125901 Matthias |
Send message Joined: 18 Dec 15 Posts: 1735 Credit: 113,596,858 RAC: 80,700 |
same thing here, a few minutes ago: ... 2024-02-17 07:23:20 (19824): Guest Log: 07:17:59 CET +01:00 2024-02-17: cranky: [INFO] Container 'runc' finished with status code 0. 2024-02-17 07:23:20 (19824): Guest Log: 07:17:59 CET +01:00 2024-02-17: cranky: [INFO] Preparing output. 2024-02-17 07:23:20 (19824): Guest Log: 07:17:59 CET +01:00 2024-02-17: cranky: [ERROR] No output found. 2024-02-17 07:23:20 (19824): Guest Log: [ERROR] Job Failed 2024-02-17 07:23:20 (19824): Guest Log: [INFO] Shutting Down. ... https://lhcathome.cern.ch/lhcathome/result.php?resultid=406202842 |
Send message Joined: 18 Dec 15 Posts: 1735 Credit: 113,596,858 RAC: 80,700 |
I was not aware that obviously ALL Theory tasks are faulty now. I downloaded them on several hosts, and all tasks errored out after some time, up to after more than 1 hour :-( So I am wondering why no-one back at LHC&Home stops the download of Theory tasks and empties the queue. The result is a real waste of ressources on the volunteers' side. |
©2024 CERN