Stderr output

<core_client_version>7.9.3</core_client_version>
<![CDATA[
<stderr_txt>
11:05:29 (4398): wrapper (7.7.26015): starting
11:05:29 (4398): wrapper: running run_atlas (--nthreads 6)
singularity image is /cvmfs/atlas.cern.ch/repo/images/singularity/x86_64-slc6.img
sys.argv = ['run_atlas', '--nthreads', '6']
THREADS=6
Checking for CVMFS
CVMFS is installed
OS:cat: /etc/redhat-release: No such file or directory

This is not SLC6, need to run with Singularity....
Checking Singularity...
Singularity is installed
copy /var/lib/boinc-client/slots/10/shared/input.tar.gz
copy /var/lib/boinc-client/slots/10/shared/RTE.tar.gz
copy /var/lib/boinc-client/slots/10/shared/ATLAS.root_0
copy /var/lib/boinc-client/slots/10/shared/start_atlas.sh
export ATHENA_PROC_NUMBER=6;start atlas job with PandaID=4184709929
Testing the function of Singularity...
check singularity with cmd:singularity exec -B /cvmfs /cvmfs/atlas.cern.ch/repo/images/singularity/x86_64-slc6.img hostname
Singularity Works...
cmd = singularity exec --pwd /var/lib/boinc-client/slots/10 -B /cvmfs,/var /cvmfs/atlas.cern.ch/repo/images/singularity/x86_64-slc6.img sh start_atlas.sh > runtime_log 2> runtime_log.err
running cmd return value is 0

***********************log_extracts.txt*************************
- Last 10 lines from /var/lib/boinc-client/slots/10/Panda_Pilot_5666_1545300349/PandaJob/athena_stdout.txt -
PyJobTransforms.trfExe.preExecute 2018-12-20 11:06:15,289 INFO Batch/grid running - command outputs will not be echoed. Logs for EVNTtoHITS are in log.EVNTtoHITS
PyJobTransforms.trfExe.preExecute 2018-12-20 11:06:15,290 INFO Now writing wrapper for substep executor EVNTtoHITS
PyJobTransforms.trfExe._writeAthenaWrapper 2018-12-20 11:06:15,291 INFO Valgrind not engaged
PyJobTransforms.trfExe.preExecute 2018-12-20 11:06:15,291 INFO Athena will be executed in a subshell via ['./runwrapper.EVNTtoHITS.sh']
PyJobTransforms.trfExe.execute 2018-12-20 11:06:15,291 INFO Starting execution of EVNTtoHITS (['./runwrapper.EVNTtoHITS.sh'])
PyJobTransforms.trfExe.execute 2018-12-20 11:08:04,486 INFO EVNTtoHITS executor returns 65
PyJobTransforms.trfExe.validate 2018-12-20 11:08:05,392 ERROR Validation of return code failed: Non-zero return code from EVNTtoHITS (65) (Error code 65)
PyJobTransforms.trfExe.validate 2018-12-20 11:08:05,407 INFO Scanning logfile log.EVNTtoHITS for errors
PyJobTransforms.transform.execute 2018-12-20 11:08:05,721 CRITICAL Transform executor raised TransformValidationException: Non-zero return code from EVNTtoHITS (65); Logfile error in log.EVNTtoHITS: "AthMpEvtLoopMgr     FATAL makePool failed for AthMpEvtLoopMgr.SharedEvtQueueProvider"
PyJobTransforms.transform.execute 2018-12-20 11:08:08,828 WARNING Transform now exiting early with exit code 65 (Non-zero return code from EVNTtoHITS (65); Logfile error in log.EVNTtoHITS: "AthMpEvtLoopMgr     FATAL makePool failed for AthMpEvtLoopMgr.SharedEvtQueueProvider")

- Walltime -
JobRetrival=0, StageIn=8, Execution=143, StageOut=0, CleanUp=9

***********************pilot_error_report.json*********************
{
    "4184709929": {
        "2": [
            {
                "pilotErrorCode": 0,
                "pilotErrorDiag": "Job failed: Non-zero failed job return code: 65"
            }
        ]
    }
}
*****************The last 100 lines of the pilot log******************
    "sepath": "/atlas/disk/atlasdatadisk/rucio", 
    "seprodpath": "/atlas/disk/atlasdatadisk/rucio", 
    "setokens": "ATLASDATADISK", 
    "site": "BOINC", 
    "siteid": "BOINC_MCORE", 
    "sitershare": null, 
    "space": 0, 
    "special_par": null, 
    "stageinretry": 2, 
    "stageoutretry": 2, 
    "status": "brokeroff", 
    "statusoverride": "offline", 
    "sysconfig": "manual", 
    "system": "arc", 
    "tags": "arc", 
    "tier": "T3", 
    "timefloor": 0, 
    "tmpdir": null, 
    "transferringlimit": 20000, 
    "tspace": "2070-01-01T00:00:00", 
    "use_newmover": "True", 
    "validatedreleases": "True", 
    "version": null, 
    "wansinklimit": null, 
    "wansourcelimit": null, 
    "wnconnectivity": "full", 
    "wntmpdir": null, 
    "workflow": null
}

2018-12-20 10:05:49|5666|SiteInformat| Queuedata was successfully downloaded by pilot wrapper script
2018-12-20 10:05:49|5666|ATLASSiteInf| curl command returned valid queuedata
2018-12-20 10:05:49|5666|ATLASSiteInf| Site BOINC_MCORE is currently in brokeroff mode
2018-12-20 10:05:49|5666|ATLASSiteInf| Job recovery turned off
2018-12-20 10:05:49|5666|ATLASSiteInf| Confirmed correctly formatted rucio sepath
2018-12-20 10:05:49|5666|ATLASSiteInf| Confirmed correctly formatted rucio seprodpath
2018-12-20 10:05:49|5666|SiteInformat| Evaluating queuedata
2018-12-20 10:05:49|5666|SiteInformat| Setting unset pilot variables using queuedata
2018-12-20 10:05:49|5666|SiteInformat| appdir: 
2018-12-20 10:05:49|5666|pUtil.py    | File registration will be done by server
2018-12-20 10:05:49|5666|pUtil.py    | Updated stage-in retry number to 2
2018-12-20 10:05:49|5666|pUtil.py    | Updated stage-out retry number to 2
2018-12-20 10:05:49|5666|pUtil.py    | Detected unset (NULL) release/homepackage string
2018-12-20 10:05:49|5666|ATLASExperim| Application dir confirmed: /var/lib/boinc-client/slots/10/
2018-12-20 10:05:49|5666|pilot.py    | Pilot will serve experiment: Nordugrid-ATLAS
2018-12-20 10:05:49|5666|ATLASExperim| Architecture information:
2018-12-20 10:05:49|5666|ATLASExperim| Excuting command: lsb_release -a
2018-12-20 10:05:49|5666|ATLASExperim| 
sh: lsb_release: command not found
2018-12-20 10:05:49|5666|pUtil.py    | getSiteInformation: got experiment=ATLAS
2018-12-20 10:05:49|5666|ATLASExperim| appdirs = ['/cvmfs/atlas.cern.ch/repo/sw']
2018-12-20 10:05:49|5666|ATLASExperim| head of /cvmfs/atlas.cern.ch/repo/sw/ChangeLog: 
--------------------------------------------------------------------------------
2018-12-20 11:00:41 Alessandro De Salvo
	* + AGISData 20181220110041

2018-12-20 10:00:51 Alessandro De Salvo
	* + AGISData 20181220100051

2018-12-20 09:01:00 Alessandro De Salvo
	* + AGISData 20181220090100

2018-12-20 08:02:19 Alessandro De Salvo
--------------------------------------------------------------------------------
2018-12-20 10:05:49|5666|ATLASExperim| ATLAS_PYTHON_PILOT set to /usr/bin/python
2018-12-20 10:05:49|5666|pUtil.py    | getSiteInformation: got experiment=ATLAS
2018-12-20 10:05:49|5666|ATLASExperim| Executing command: export ATLAS_LOCAL_ROOT_BASE=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase;$ATLAS_LOCAL_ROOT_BASE/utilities/checkValidity.sh (time-out: 300)
2018-12-20 10:05:49|5666|pUtil.py    | Executing command: export ATLAS_LOCAL_ROOT_BASE=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase;$ATLAS_LOCAL_ROOT_BASE/utilities/checkValidity.sh (protected by timed_command, timeout: 300 s)
2018-12-20 10:05:50|5666|pUtil.py    | Elapsed time: 1
2018-12-20 10:05:50|5666|ATLASExperim| Diagnostics tool has verified CVMFS
2018-12-20 10:05:50|5666|Node.py     | Collecting machine features
2018-12-20 10:05:50|5666|Node.py     | $MACHINEFEATURES not defined locally
2018-12-20 10:05:50|5666|Node.py     | $JOBFEATURES not defined locally
2018-12-20 10:05:50|5666|Node.py     | Executing command: hostname -i
2018-12-20 10:05:50|5666|Node.py     | IP number of worker node: 127.0.1.1
2018-12-20 10:05:50|5666|pUtil.py    | getSiteInformation: got experiment=Nordugrid-ATLAS
2018-12-20 10:05:50|5666|pilot.py    | Using site information for experiment: Nordugrid-ATLAS
2018-12-20 10:05:50|5666|pilot.py    | Will attempt to create workdir: /var/lib/boinc-client/slots/10/Panda_Pilot_5666_1545300349
2018-12-20 10:05:50|5666|pilot.py    | Creating file: /var/lib/boinc-client/slots/10/CURRENT_SITEWORKDIR
2018-12-20 10:05:50|5666|pUtil.py    | Wrote string "/var/lib/boinc-client/slots/10/Panda_Pilot_5666_1545300349" to file: /var/lib/boinc-client/slots/10/CURRENT_SITEWORKDIR
2018-12-20 10:05:50|5666|ATLASExperim| ATLAS_POOLCOND_PATH not set by wrapper
2018-12-20 10:05:50|5666|pilot.py    | Preparing to execute Cleaner
2018-12-20 10:05:50|5666|pilot.py    | Cleaning /var/lib/boinc-client/slots/10
2018-12-20 10:05:50|5666|Cleaner.py  | Cleaner initialized with clean-up limit: 2 hours
2018-12-20 10:05:50|5666|Cleaner.py  | Cleaner will scan for lost directories in verified path: /var/lib/boinc-client/slots/10
2018-12-20 10:05:50|5666|Cleaner.py  | Executing empty dirs clean-up, stage 1/5
2018-12-20 10:05:50|5666|Cleaner.py  | Purged 0 empty directories
2018-12-20 10:05:50|5666|Cleaner.py  | Executing work dir clean-up, stage 2/5
2018-12-20 10:05:50|5666|Cleaner.py  | Purged 0 single workDirs directories
2018-12-20 10:05:50|5666|Cleaner.py  | Executing maxed-out dirs clean-up, stage 3/5
2018-12-20 10:05:50|5666|Cleaner.py  | Purged 0 empty directories
2018-12-20 10:05:50|5666|Cleaner.py  | Executing AthenaMP clean-up, stage 4/5 <SKIPPED>
2018-12-20 10:05:50|5666|Cleaner.py  | Executing PanDA Pilot dir clean-up, stage 5/5
2018-12-20 10:05:50|5666|Cleaner.py  | Number of found job state files: 0
2018-12-20 10:05:50|5666|Cleaner.py  | No job state files were found, aborting clean-up
2018-12-20 10:05:50|5666|pilot.py    | Update frequencies:
2018-12-20 10:05:50|5666|pilot.py    | ...Processes: 300 s
2018-12-20 10:05:50|5666|pilot.py    | .......Space: 600 s
2018-12-20 10:05:50|5666|pilot.py    | ......Server: 1800 s
2018-12-20 10:05:50|5666|pUtil.py    | Timefloor set to zero in queuedata (multi-jobs disabled)
***************diag file************
runtimeenvironments=APPS/HEP/ATLAS-SITE;
Processors=1
WallTime=282.46s
KernelTime=8.41s
UserTime=124.15s
CPUUsage=46%
MaxResidentMemory=2036204kB
AverageResidentMemory=0kB
AverageTotalMemory=0kB
AverageUnsharedMemory=0kB
AverageUnsharedStack=0kB
AverageSharedMemory=0kB
PageSize=4096B
MajorPageFaults=8421
MinorPageFaults=2333466
Swaps=0
ForcedSwitches=127507
WaitSwitches=505087
Inputs=3760273
Outputs=63071
SocketReceived=0
SocketSent=0
Signals=0

nodename=Hydrosaure@boinc-ab350m
exitcode=0
******************************WorkDir***********************
total 192134
drwxrwx--x  6 boinc boinc        48 Dec 20 11:10 .
drwxrwx--x 25 boinc boinc        25 Dec 18 16:52 ..
-rw-------  1 boinc boinc   7158688 Dec 20 11:05 agis_ddmendpoints.cvmfs.json
-rw-------  1 boinc boinc   5694493 Dec 20 11:06 agis_schedconf.cvmfs.json
drwx------  2 boinc boinc         2 Dec 20 11:06 .alrb
drwxr-xr-x  3 boinc boinc         3 Dec 20 11:05 APPS
-rwx------  1 boinc boinc      2441 Dec 20 07:45 ARCpilot
-rw-------  1 boinc boinc       550 Dec 20 11:05 .asetup
-rw-------  1 boinc boinc     11051 Dec 20 11:06 .asetup.save
-rw-r--r--  1 boinc boinc         0 Dec 20 11:05 boinc_lockfile
-rw-r--r--  1 boinc boinc      8192 Dec 20 11:10 boinc_mmap_file
-rw-r--r--  1 boinc boinc       525 Dec 20 11:08 boinc_task_state.xml
-rw-------  1 boinc boinc        58 Dec 20 11:05 CURRENT_SITEWORKDIR
-rw-r--r--  1 boinc boinc 193938300 Dec 20 11:05 EVNT.15754943._003171.pool.root.1
-rw-r--r--  1 boinc boinc      6534 Dec 20 11:05 init_data.xml
-rw-r--r--  1 boinc boinc   1088718 Dec 20 11:05 input.tar.gz
-rw-------  1 boinc boinc      4032 Dec 20 11:10 jobSmallFiles.tgz
-rw-r--r--  1 boinc boinc       105 Dec 20 11:05 job.xml
-rw-------  1 boinc boinc    165253 Dec 20 11:10 log.16442994._137453.job.log.1
-rw-------  1 boinc boinc    156364 Dec 20 11:08 log.16442994._137453.job.log.tgz.1
-rw-------  1 boinc boinc      1730 Dec 20 11:09 log_extracts.txt
-rw-------  1 boinc boinc       311 Dec 20 11:08 memory_monitor_summary.json
-rw-------  1 boinc boinc       599 Dec 20 11:10 metadata-surl.xml
-rw-------  1 boinc boinc       241 Dec 20 11:09 output.list
-rw-------  1 boinc boinc        11 Dec 20 11:05 pandaIDs.out
-rw-------  1 boinc boinc      2919 Dec 20 11:05 pandaJobData_1.out
-rw-------  1 boinc boinc      2919 Dec 20 11:05 pandaJobData.out
-rw-------  1 boinc boinc     10529 Dec 20 11:09 panda_node_struct.pickle
-rw-------  1 boinc boinc       203 Dec 20 11:08 pilot_error_report.json
-rw-------  1 boinc boinc        30 Dec 20 11:05 PILOT_INITDIR
-rw-------  1 boinc boinc       137 Dec 20 11:10 pilotlog-last.txt
-rw-------  1 boinc boinc     11375 Dec 20 11:05 pilotlog.txt
drwx------  3 boinc boinc         3 Dec 20 11:06 .pki
-rw-------  1 boinc boinc      3801 Dec 20 11:05 queuedata.json
-rw-r--r--  1 boinc boinc      4436 Dec 20 07:40 queuedata.pilot.json
-rw-r--r--  1 boinc boinc       606 Dec 20 11:05 RTE.tar.gz
-rwxr-xr-x  1 boinc boinc      8356 Dec 20 11:05 run_atlas
-rw-r--r--  1 boinc boinc       604 Dec 20 11:10 runtime_log
-rw-r--r--  1 boinc boinc     10380 Dec 20 11:10 runtime_log.err
drwxrwx--x  2 boinc boinc         7 Dec 20 11:10 shared
-rw-r--r--  1 boinc boinc     14361 Dec 20 11:05 start_atlas.sh
-rw-------  1 boinc boinc        19 Dec 20 11:05 START_TIME_4184709929
-rw-------  1 boinc boinc         1 Dec 20 11:05 STATUSCODE
-rw-r--r--  1 boinc boinc      9877 Dec 20 11:10 stderr.txt
-rw-------  1 boinc boinc        46 Dec 20 11:08 workdir_size-4184709929.json
-rw-r--r--  1 boinc boinc       100 Dec 20 11:05 wrapper_26015_x86_64-pc-linux-gnu
-rw-r--r--  1 boinc boinc        23 Dec 20 11:10 wrapper_checkpoint.txt
-rw-------  1 boinc boinc       494 Dec 20 11:10 zR4KDm7F7ttnlyackoJh5iwnABFKDmABFKDmX6hbDmABFKDmOMowfm.diag
running start_atlas return value is 0
Parent exit 0
child process exit 0
11:10:31 (4398): run_atlas exited; CPU time 124.469259
11:10:31 (4398): called boinc_finish(0)

</stderr_txt>
]]>

©2026 CERN