LHC@home

I got a whole lot of invalid results due to this error across multiple machines starting today. The tasks don't show up as errors, but invalid. Example: https://lhcathome.cern.ch/lhcathome/result.php?resultid=416180255

My normal troubleshooting command shows cvmfs is OK on the host. I do occasionally get WUs actually crunch for a while instead of giving up right away.

$ cvmfs_config probe
Probing /cvmfs/atlas.cern.ch... OK
Probing /cvmfs/atlas-condb.cern.ch... OK
Probing /cvmfs/grid.cern.ch... OK
Probing /cvmfs/cernvm-prod.cern.ch... OK
Probing /cvmfs/sft.cern.ch... OK
Probing /cvmfs/alice.cern.ch... OK

Is this a problem with my setup or are the latest batch of tasks bad?

+1
https://lhcathome.cern.ch/lhcathome/result.php?resultid=416182268

[2024-11-09 06:29:49] 2024-11-09 05:29:35,541 | INFO | extracted standard info from prmon json
[2024-11-09 06:29:49] 2024-11-09 05:29:35,541 | INFO | extracted standard memory fields from prmon json
[2024-11-09 06:29:49] 2024-11-09 05:29:35,542 | WARNING | GPU info not found in prmon json: 'gpu'
[2024-11-09 06:29:49] 2024-11-09 05:29:35,542 | WARNING | found no stored workdir sizes
[2024-11-09 06:29:49] 2024-11-09 05:29:35,542 | INFO | will not add max space = 0 B to job metrics
[2024-11-09 06:29:49] 2024-11-09 05:29:35,542 | WARNING | wrong length of table data, x=[1731130164.0], y=[0.0] (must be same and length>=4)

What's prmon json

I also got of bunch of tasks that finished with error. I don't think this is client side error. However, yesterday there were a few good pairs that ran with 50 events.

pilotErrorDiags = ['Failed to execute payload:/bin/bash: Sim_tf.py: command not found\n', 'Payload metadata does not exist

same problem here, see:

https://lhcathome.cern.ch/lhcathome/results.php?hostid=10853437

I have over 450 invalids. Initially I blamed my own machines for conflicts or memory issues but after poking around uninstalling things, rebooting, removing and re-adding the project etc, and then observing it happening on more than one machine it seems it's the units themselves. Very frustrating.
https://lhcathome.cern.ch/lhcathome/results.php?userid=268818&offset=0&show_names=0&state=5&appid=

I also got of bunch of tasks that finished with error. I don't think this is client side error. However, yesterday there were a few good pairs that ran with 50 events.

pilotErrorDiags = ['Failed to execute payload:/bin/bash: Sim_tf.py: command not found\n', 'Payload metadata does not exist

Same here. Therefore this might be on server side (CVMFS) or the WUs themselves are not correct ('Payload metadata').
So i put it on no new Work since this is solved.

Same here https://lhcathome.cern.ch/lhcathome/result.php?resultid=416524959

I got also one result with this error
https://lhcathome.cern.ch/lhcathome/result.php?resultid=417576672
I did a project reset some days before and this was the first time I got an Atlas task. So the Atlas vdi was just newly downloaded for this result

---

Matthias

As I see it works now. Since yesterday, I’ve had 79 successful runs. I also checked the logs containing Sim_tf.py lines, so it’s probably using it. However, the download speed varies between 50 kbps or 500 kbps. At least in native version.
...
09:41:39 Py:Sim_tf INFO **** Setting-up configuration flags
09:42:10 Py:AutoConfigFlags INFO Obtaining metadata of auto-configuration by peeking into 'EVNT.41942093._000662.pool.root.1'
09:42:12 Py:MetaReader INFO Current mode used: lite
09:42:15 Py:PileUp INFO There are 400 events in this run.
...

Started to get a few tasks with this error again today. https://lhcathome.cern.ch/lhcathome/result.php?resultid=417634622
Most of the tasks working OK though.

---

Started to get a few tasks with this error again today. https://lhcathome.cern.ch/lhcathome/result.php?resultid=417634622
Most of the tasks working OK though.

same here