Stderr output

<core_client_version>7.7.0</core_client_version>
<![CDATA[
<stderr_txt>
14:44:43 (197387): wrapper (7.7.26015): starting
14:44:43 (197387): wrapper: running run_atlas (--nthreads 8)
[2024-05-15 14:44:43] Arguments: --nthreads 8
[2024-05-15 14:44:43] Threads: 8
[2024-05-15 14:44:43] Checking for CVMFS
[2024-05-15 14:44:44] Probing /cvmfs/atlas.cern.ch... OK
[2024-05-15 14:44:44] Probing /cvmfs/atlas-condb.cern.ch... OK
[2024-05-15 14:44:44] Running cvmfs_config stat atlas.cern.ch
[2024-05-15 14:44:45] VERSION PID UPTIME(M) MEM(K) REVISION EXPIRES(M) NOCATALOGS CACHEUSE(K) CACHEMAX(K) NOFDUSE NOFDMAX NOIOERR NOOPEN HITRATE(%) RX(K) SPEED(K/S) HOST PROXY ONLINE
[2024-05-15 14:44:45] 2.11.2.0 237673 2873 149652 132771 2 271 16403634 18432001 34700 130560 0 17309832 98.630 50519825 37765 http://cvmfs-s1fnal.opensciencegrid.org:8000/cvmfs/atlas.cern.ch http://192.41.231.237:6081 1
[2024-05-15 14:44:45] CVMFS is ok
[2024-05-15 14:44:45] Efficiency of ATLAS tasks can be improved by the following measure(s):
[2024-05-15 14:44:45] The CVMFS client on this computer should be configured to use Cloudflare's openhtc.io.
[2024-05-15 14:44:45] Further information can be found at the LHC@home message board.
[2024-05-15 14:44:45] Using apptainer image /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos7
[2024-05-15 14:44:45] Checking for apptainer binary...
[2024-05-15 14:44:45] Using apptainer found in PATH at /usr/bin/apptainer
[2024-05-15 14:44:45] Running /usr/bin/apptainer --version
[2024-05-15 14:44:45] apptainer version 1.3.0-1.el7
[2024-05-15 14:44:45] Checking apptainer works with /usr/bin/apptainer exec -B /cvmfs /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos7 hostname
[2024-05-15 14:44:54] c6-12-27-4.aglt2.org
[2024-05-15 14:44:54] apptainer works
[2024-05-15 14:44:54] Set ATHENA_PROC_NUMBER=8
[2024-05-15 14:44:54] Set ATHENA_CORE_NUMBER=8
[2024-05-15 14:44:54] Starting ATLAS job with PandaID=6206806110
[2024-05-15 14:44:54] Running command: /usr/bin/apptainer exec -B /cvmfs,/tmp/boinchome/slots/1 /cvmfs/atlas.cern.ch/repo/containers/fs/singularity/x86_64-centos7 sh start_atlas.sh
[2024-05-15 18:26:54]  *** The last 200 lines of the pilot log: ***
[2024-05-15 18:26:54]   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.18-x86_64-centos7/lib/python3.9/urllib/request.py", line 494, in _call_chain
[2024-05-15 18:26:54]     result = func(*args)
[2024-05-15 18:26:54]   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.18-x86_64-centos7/lib/python3.9/urllib/request.py", line 1389, in https_open
[2024-05-15 18:26:54]     return self.do_open(http.client.HTTPSConnection, req,
[2024-05-15 18:26:54]   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.18-x86_64-centos7/lib/python3.9/urllib/request.py", line 1349, in do_open
[2024-05-15 18:26:54]     raise URLError(err)
[2024-05-15 18:26:54] urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)>
[2024-05-15 18:26:54] 
[2024-05-15 18:26:54] During handling of the above exception, another exception occurred:
[2024-05-15 18:26:54] 
[2024-05-15 18:26:54] Traceback (most recent call last):
[2024-05-15 18:26:54]   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.18-x86_64-centos7/lib/python3.9/logging/__init__.py", line 1083, in emit
[2024-05-15 18:26:54]     msg = self.format(record)
[2024-05-15 18:26:54]   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.18-x86_64-centos7/lib/python3.9/logging/__init__.py", line 927, in format
[2024-05-15 18:26:54]     return fmt.format(record)
[2024-05-15 18:26:54]   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.18-x86_64-centos7/lib/python3.9/logging/__init__.py", line 663, in format
[2024-05-15 18:26:54]     record.message = record.getMessage()
[2024-05-15 18:26:54]   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.18-x86_64-centos7/lib/python3.9/logging/__init__.py", line 367, in getMessage
[2024-05-15 18:26:54]     msg = msg % self.args
[2024-05-15 18:26:54] TypeError: not all arguments converted during string formatting
[2024-05-15 18:26:54] Call stack:
[2024-05-15 18:26:54]   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.18-x86_64-centos7/lib/python3.9/threading.py", line 937, in _bootstrap
[2024-05-15 18:26:54]     self._bootstrap_inner()
[2024-05-15 18:26:54]   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.18-x86_64-centos7/lib/python3.9/threading.py", line 980, in _bootstrap_inner
[2024-05-15 18:26:54]     self.run()
[2024-05-15 18:26:54]   File "/tmp/boinchome/slots/1/pilot3/pilot/common/exception.py", line 450, in run
[2024-05-15 18:26:54]     self.target(**self.kwargs)
[2024-05-15 18:26:54]   File "/tmp/boinchome/slots/1/pilot3/pilot/control/job.py", line 2568, in queue_monitor
[2024-05-15 18:26:54]     update_server(job, args)
[2024-05-15 18:26:54]   File "/tmp/boinchome/slots/1/pilot3/pilot/control/job.py", line 2611, in update_server
[2024-05-15 18:26:54]     user.update_server(job)
[2024-05-15 18:26:54]   File "/tmp/boinchome/slots/1/pilot3/pilot/user/atlas/common.py", line 2692, in update_server
[2024-05-15 18:26:54]     status = upload_file(url, new_path)
[2024-05-15 18:26:54]   File "/tmp/boinchome/slots/1/pilot3/pilot/util/https.py", line 837, in upload_file
[2024-05-15 18:26:54]     logger.warning("URL Error:", e)
[2024-05-15 18:26:54] Message: 'URL Error:'
[2024-05-15 18:26:54] Arguments: (URLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')),)
[2024-05-15 18:26:54] --- Logging error ---
[2024-05-15 18:26:54] Traceback (most recent call last):
[2024-05-15 18:26:54]   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.18-x86_64-centos7/lib/python3.9/urllib/request.py", line 1346, in do_open
[2024-05-15 18:26:54]     h.request(req.get_method(), req.selector, req.data, headers,
[2024-05-15 18:26:54]   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.18-x86_64-centos7/lib/python3.9/http/client.py", line 1285, in request
[2024-05-15 18:26:54]     self._send_request(method, url, body, headers, encode_chunked)
[2024-05-15 18:26:54]   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.18-x86_64-centos7/lib/python3.9/http/client.py", line 1331, in _send_request
[2024-05-15 18:26:54]     self.endheaders(body, encode_chunked=encode_chunked)
[2024-05-15 18:26:54]   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.18-x86_64-centos7/lib/python3.9/http/client.py", line 1280, in endheaders
[2024-05-15 18:26:54]     self._send_output(message_body, encode_chunked=encode_chunked)
[2024-05-15 18:26:54]   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.18-x86_64-centos7/lib/python3.9/http/client.py", line 1040, in _send_output
[2024-05-15 18:26:54]     self.send(msg)
[2024-05-15 18:26:54]   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.18-x86_64-centos7/lib/python3.9/http/client.py", line 980, in send
[2024-05-15 18:26:54]     self.connect()
[2024-05-15 18:26:54]   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.18-x86_64-centos7/lib/python3.9/http/client.py", line 1454, in connect
[2024-05-15 18:26:54]     self.sock = self._context.wrap_socket(self.sock,
[2024-05-15 18:26:54]   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.18-x86_64-centos7/lib/python3.9/ssl.py", line 501, in wrap_socket
[2024-05-15 18:26:54]     return self.sslsocket_class._create(
[2024-05-15 18:26:54]   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.18-x86_64-centos7/lib/python3.9/ssl.py", line 1074, in _create
[2024-05-15 18:26:54]     self.do_handshake()
[2024-05-15 18:26:54]   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.18-x86_64-centos7/lib/python3.9/ssl.py", line 1343, in do_handshake
[2024-05-15 18:26:54]     self._sslobj.do_handshake()
[2024-05-15 18:26:54] ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)
[2024-05-15 18:26:54] 
[2024-05-15 18:26:54] During handling of the above exception, another exception occurred:
[2024-05-15 18:26:54] 
[2024-05-15 18:26:54] Traceback (most recent call last):
[2024-05-15 18:26:54]   File "/tmp/boinchome/slots/1/pilot3/pilot/util/https.py", line 831, in upload_file
[2024-05-15 18:26:54]     with urllib.request.urlopen(request) as response:
[2024-05-15 18:26:54]   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.18-x86_64-centos7/lib/python3.9/urllib/request.py", line 214, in urlopen
[2024-05-15 18:26:54]     return opener.open(url, data, timeout)
[2024-05-15 18:26:54]   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.18-x86_64-centos7/lib/python3.9/urllib/request.py", line 517, in open
[2024-05-15 18:26:54]     response = self._open(req, data)
[2024-05-15 18:26:54]   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.18-x86_64-centos7/lib/python3.9/urllib/request.py", line 534, in _open
[2024-05-15 18:26:54]     result = self._call_chain(self.handle_open, protocol, protocol +
[2024-05-15 18:26:54]   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.18-x86_64-centos7/lib/python3.9/urllib/request.py", line 494, in _call_chain
[2024-05-15 18:26:54]     result = func(*args)
[2024-05-15 18:26:54]   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.18-x86_64-centos7/lib/python3.9/urllib/request.py", line 1389, in https_open
[2024-05-15 18:26:54]     return self.do_open(http.client.HTTPSConnection, req,
[2024-05-15 18:26:54]   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.18-x86_64-centos7/lib/python3.9/urllib/request.py", line 1349, in do_open
[2024-05-15 18:26:54]     raise URLError(err)
[2024-05-15 18:26:54] urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)>
[2024-05-15 18:26:54] 
[2024-05-15 18:26:54] During handling of the above exception, another exception occurred:
[2024-05-15 18:26:54] 
[2024-05-15 18:26:54] Traceback (most recent call last):
[2024-05-15 18:26:54]   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.18-x86_64-centos7/lib/python3.9/logging/__init__.py", line 1083, in emit
[2024-05-15 18:26:54]     msg = self.format(record)
[2024-05-15 18:26:54]   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.18-x86_64-centos7/lib/python3.9/logging/__init__.py", line 927, in format
[2024-05-15 18:26:54]     return fmt.format(record)
[2024-05-15 18:26:54]   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.18-x86_64-centos7/lib/python3.9/logging/__init__.py", line 663, in format
[2024-05-15 18:26:54]     record.message = record.getMessage()
[2024-05-15 18:26:54]   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.18-x86_64-centos7/lib/python3.9/logging/__init__.py", line 367, in getMessage
[2024-05-15 18:26:54]     msg = msg % self.args
[2024-05-15 18:26:54] TypeError: not all arguments converted during string formatting
[2024-05-15 18:26:54] Call stack:
[2024-05-15 18:26:54]   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.18-x86_64-centos7/lib/python3.9/threading.py", line 937, in _bootstrap
[2024-05-15 18:26:54]     self._bootstrap_inner()
[2024-05-15 18:26:54]   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.18-x86_64-centos7/lib/python3.9/threading.py", line 980, in _bootstrap_inner
[2024-05-15 18:26:54]     self.run()
[2024-05-15 18:26:54]   File "/tmp/boinchome/slots/1/pilot3/pilot/common/exception.py", line 450, in run
[2024-05-15 18:26:54]     self.target(**self.kwargs)
[2024-05-15 18:26:54]   File "/tmp/boinchome/slots/1/pilot3/pilot/control/job.py", line 2568, in queue_monitor
[2024-05-15 18:26:54]     update_server(job, args)
[2024-05-15 18:26:54]   File "/tmp/boinchome/slots/1/pilot3/pilot/control/job.py", line 2611, in update_server
[2024-05-15 18:26:54]     user.update_server(job)
[2024-05-15 18:26:54]   File "/tmp/boinchome/slots/1/pilot3/pilot/user/atlas/common.py", line 2692, in update_server
[2024-05-15 18:26:54]     status = upload_file(url, new_path)
[2024-05-15 18:26:54]   File "/tmp/boinchome/slots/1/pilot3/pilot/util/https.py", line 837, in upload_file
[2024-05-15 18:26:54]     logger.warning("URL Error:", e)
[2024-05-15 18:26:54] Message: 'URL Error:'
[2024-05-15 18:26:54] Arguments: (URLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)')),)
[2024-05-15 18:26:54] 2024-05-15 22:26:37,180 | WARNING  | failed to send data to https://pilot.atlas-ml.org: response=<urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer c
[2024-05-15 18:26:54] 2024-05-15 22:26:37,181 | INFO     | executing command: curl --connect-timeout 20 --max-time 120 -H "Content-Type: application/json" -X POST --upload-file /tmp/boinchome/slots/1/PanDA_Pilot-6206806110
[2024-05-15 18:26:54] 2024-05-15 22:26:37,192 | WARNING  | exception caught: [Errno 107] Transport endpoint is not connected: '/bin/bash'
[2024-05-15 18:26:54] 2024-05-15 22:26:37,192 | INFO     | this job has now completed (state=failed)
[2024-05-15 18:26:54] 2024-05-15 22:26:37,192 | INFO     | pilot will not update the server (heartbeat message will be written to file)
[2024-05-15 18:26:54] 2024-05-15 22:26:37,193 | INFO     | job 6206806110 has failed - writing final server update
[2024-05-15 18:26:54] 2024-05-15 22:26:37,193 | WARNING  | making sure that job.state is set to failed since a pilot error code is set
[2024-05-15 18:26:54] 2024-05-15 22:26:37,193 | INFO     | payload/TRF did not report the number of read events
[2024-05-15 18:26:54] 2024-05-15 22:26:37,198 | INFO     | executing command: lscpu
[2024-05-15 18:26:54]   File "/tmp/boinchome/slots/1/pilot3/pilot/common/exception.py", line 450, in run
[2024-05-15 18:26:54]     self.target(**self.kwargs)
[2024-05-15 18:26:54]   File "/tmp/boinchome/slots/1/pilot3/pilot/control/job.py", line 2568, in queue_monitor
[2024-05-15 18:26:54]     update_server(job, args)
[2024-05-15 18:26:54]   File "/tmp/boinchome/slots/1/pilot3/pilot/control/job.py", line 2617, in update_server
[2024-05-15 18:26:54]     send_state(job, args, job.state, metadata=metadata)
[2024-05-15 18:26:54]   File "/tmp/boinchome/slots/1/pilot3/pilot/control/job.py", line 420, in send_state
[2024-05-15 18:26:54]     data = get_data_structure(job, state, args, xml=xml, metadata=metadata)
[2024-05-15 18:26:54]   File "/tmp/boinchome/slots/1/pilot3/pilot/control/job.py", line 747, in get_data_structure
[2024-05-15 18:26:54]     cpumodel = get_cpu_cores(cpumodel)  # add the CPU cores if not present
[2024-05-15 18:26:54]   File "/tmp/boinchome/slots/1/pilot3/pilot/util/workernode.py", line 353, in get_cpu_cores
[2024-05-15 18:26:54]     ec, stdout = lscpu()
[2024-05-15 18:26:54]   File "/tmp/boinchome/slots/1/pilot3/pilot/util/workernode.py", line 333, in lscpu
[2024-05-15 18:26:54]     ec, stdout, _ = execute(cmd)
[2024-05-15 18:26:54]   File "/tmp/boinchome/slots/1/pilot3/pilot/util/container.py", line 94, in execute
[2024-05-15 18:26:54]     process = subprocess.Popen(exe,
[2024-05-15 18:26:54]   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.18-x86_64-centos7/lib/python3.9/subprocess.py", line 951, in __init__
[2024-05-15 18:26:54]     self._execute_child(args, executable, preexec_fn, close_fds,
[2024-05-15 18:26:54]   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.18-x86_64-centos7/lib/python3.9/subprocess.py", line 1837, in _execute_child
[2024-05-15 18:26:54]     raise child_exception_type(errno_num, err_msg, err_filename)
[2024-05-15 18:26:54] exception caught by thread run() function: (<class 'OSError'>, OSError(107, 'Transport endpoint is not connected'), <traceback object at 0x7f4b2d72cb80>)
[2024-05-15 18:26:54] Traceback (most recent call last):
[2024-05-15 18:26:54]   File "/tmp/boinchome/slots/1/pilot3/pilot/common/exception.py", line 450, in run
[2024-05-15 18:26:54]     self.target(**self.kwargs)
[2024-05-15 18:26:54]   File "/tmp/boinchome/slots/1/pilot3/pilot/control/job.py", line 2568, in queue_monitor
[2024-05-15 18:26:54]     update_server(job, args)
[2024-05-15 18:26:54]   File "/tmp/boinchome/slots/1/pilot3/pilot/control/job.py", line 2617, in update_server
[2024-05-15 18:26:54]     send_state(job, args, job.state, metadata=metadata)
[2024-05-15 18:26:54]   File "/tmp/boinchome/slots/1/pilot3/pilot/control/job.py", line 420, in send_state
[2024-05-15 18:26:54]     data = get_data_structure(job, state, args, xml=xml, metadata=metadata)
[2024-05-15 18:26:54]   File "/tmp/boinchome/slots/1/pilot3/pilot/control/job.py", line 747, in get_data_structure
[2024-05-15 18:26:54]     cpumodel = get_cpu_cores(cpumodel)  # add the CPU cores if not present
[2024-05-15 18:26:54]   File "/tmp/boinchome/slots/1/pilot3/pilot/util/workernode.py", line 353, in get_cpu_cores
[2024-05-15 18:26:54]     ec, stdout = lscpu()
[2024-05-15 18:26:54]   File "/tmp/boinchome/slots/1/pilot3/pilot/util/workernode.py", line 333, in lscpu
[2024-05-15 18:26:54]     ec, stdout, _ = execute(cmd)
[2024-05-15 18:26:54]   File "/tmp/boinchome/slots/1/pilot3/pilot/util/container.py", line 94, in execute
[2024-05-15 18:26:54]     process = subprocess.Popen(exe,
[2024-05-15 18:26:54]   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.18-x86_64-centos7/lib/python3.9/subprocess.py", line 951, in __init__
[2024-05-15 18:26:54]     self._execute_child(args, executable, preexec_fn, close_fds,
[2024-05-15 18:26:54]   File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/python/3.9.18-x86_64-centos7/lib/python3.9/subprocess.py", line 1837, in _execute_child
[2024-05-15 18:26:54]     raise child_exception_type(errno_num, err_msg, err_filename)
[2024-05-15 18:26:54] OSError: [Errno 107] Transport endpoint is not connected: '/bin/bash'
[2024-05-15 18:26:54] 
[2024-05-15 18:26:54] None
[2024-05-15 18:26:54] exception has been put in bucket queue belonging to thread 'queue_monitor'
[2024-05-15 18:26:54] setting graceful stop in 10 s since there is no point in continuing
[2024-05-15 18:26:54] 2024-05-15 22:26:38,190 | INFO     | waiting for thread to finish: ['<_MainThread(MainThread, started 139961002305344)>', '<ExcThread(queue_monitor, started 139960342529792)>']
[2024-05-15 18:26:54] 2024-05-15 22:26:40,201 | INFO     | waiting for thread to finish: ['<_MainThread(MainThread, started 139961002305344)>', '<ExcThread(queue_monitor, started 139960342529792)>']
[2024-05-15 18:26:54] 2024-05-15 22:26:42,212 | INFO     | waiting for thread to finish: ['<_MainThread(MainThread, started 139961002305344)>', '<ExcThread(queue_monitor, started 139960342529792)>']
[2024-05-15 18:26:54] 2024-05-15 22:26:44,222 | INFO     | waiting for thread to finish: ['<_MainThread(MainThread, started 139961002305344)>', '<ExcThread(queue_monitor, started 139960342529792)>']
[2024-05-15 18:26:54] 2024-05-15 22:26:46,232 | INFO     | waiting for thread to finish: ['<_MainThread(MainThread, started 139961002305344)>', '<ExcThread(queue_monitor, started 139960342529792)>']
[2024-05-15 18:26:54] 2024-05-15 22:26:48,243 | INFO     | caller=run is remaining thread - safe to abort (names=['<_MainThread(MainThread, started 139961002305344)>'])
[2024-05-15 18:26:54] 2024-05-15 22:26:53,269 | INFO     | all workflow threads have been joined
[2024-05-15 18:26:54] 2024-05-15 22:26:53,269 | INFO     | end of generic workflow (traces error code: 0)
[2024-05-15 18:26:54] 2024-05-15 22:26:53,270 | INFO     | traces error code: 0
[2024-05-15 18:26:54] 2024-05-15 22:26:53,270 | INFO     | pilot has finished (exit code=0, shell exit code=0)
[2024-05-15 18:26:54] ./runpilot2-wrapper.sh: line 15: /usr/bin/date: Transport endpoint is not connected
[2024-05-15 18:26:54]  ==== pilot stdout END ====
[2024-05-15 18:26:54] ./runpilot2-wrapper.sh: line 15: /usr/bin/date: Transport endpoint is not connected
[2024-05-15 18:26:54]  ==== wrapper stdout RESUME ====
[2024-05-15 18:26:54] ./runpilot2-wrapper.sh: line 15: /usr/bin/date: Transport endpoint is not connected
[2024-05-15 18:26:54]  pilotpid: 223037
[2024-05-15 18:26:54] ./runpilot2-wrapper.sh: line 15: /usr/bin/date: Transport endpoint is not connected
[2024-05-15 18:26:54]  Pilot exit status: 0
[2024-05-15 18:26:54] ./runpilot2-wrapper.sh: line 949: /usr/bin/cat: Transport endpoint is not connected
[2024-05-15 18:26:54] ./runpilot2-wrapper.sh: line 949: /usr/bin/cut: Transport endpoint is not connected
[2024-05-15 18:26:54] ./runpilot2-wrapper.sh: line 949: xargs: command not found
[2024-05-15 18:26:54] ./runpilot2-wrapper.sh: line 15: /usr/bin/date: Transport endpoint is not connected
[2024-05-15 18:26:54]  pandaids: 
[2024-05-15 18:26:54] ./runpilot2-wrapper.sh: line 501: /usr/bin/ps: Transport endpoint is not connected
[2024-05-15 18:26:54] ./runpilot2-wrapper.sh: line 15: /usr/bin/date: Transport endpoint is not connected
[2024-05-15 18:26:54]  No supervise_pilot CHILD process found
[2024-05-15 18:26:54] ./runpilot2-wrapper.sh: line 15: /usr/bin/date: Transport endpoint is not connected
[2024-05-15 18:26:54]  Test setup, not cleaning
[2024-05-15 18:26:54] ./runpilot2-wrapper.sh: line 15: /usr/bin/date: Transport endpoint is not connected
[2024-05-15 18:26:54]  ==== wrapper stdout END ====
[2024-05-15 18:26:54] ./runpilot2-wrapper.sh: line 10: /usr/bin/date: Transport endpoint is not connected
[2024-05-15 18:26:54]  ==== wrapper stderr END ====
[2024-05-15 18:26:54] ./runpilot2-wrapper.sh: line 522: /usr/bin/date: Transport endpoint is not connected
[2024-05-15 18:26:54] ./runpilot2-wrapper.sh: line 15: /usr/bin/date: Transport endpoint is not connected
[2024-05-15 18:26:54]  apfmon messages muted
[2024-05-15 18:26:54]  *** Error codes and diagnostics ***
[2024-05-15 18:26:54]  *** Listing of results directory ***
[2024-05-15 18:26:54] total 592972
[2024-05-15 18:26:54] -rw-r--r-- 1 boincer umatlas      5633 May 15 10:16 queuedata.json
[2024-05-15 18:26:54] -rw-r--r-- 1 boincer umatlas    467845 May 15 10:16 pilot3.tar.gz
[2024-05-15 18:26:54] -rwx------ 1 boincer umatlas     32251 May 15 10:16 runpilot2-wrapper.sh
[2024-05-15 18:26:54] -rw-r--r-- 1 boincer umatlas       100 May 15 14:43 wrapper_26015_x86_64-pc-linux-gnu
[2024-05-15 18:26:54] -rwxr-xr-x 1 boincer umatlas      7986 May 15 14:43 run_atlas
[2024-05-15 18:26:54] -rw-r--r-- 1 boincer umatlas       105 May 15 14:43 job.xml
[2024-05-15 18:26:54] -rw-r--r-- 2 boincer umatlas 601375264 May 15 14:44 EVNT.38776100._000252.pool.root.1
[2024-05-15 18:26:54] -rw-r--r-- 2 boincer umatlas     17537 May 15 14:44 start_atlas.sh
[2024-05-15 18:26:54] drwxrwx--x 2 boincer umatlas      4096 May 15 14:44 shared
[2024-05-15 18:26:54] -rw-r--r-- 2 boincer umatlas    480004 May 15 14:44 input.tar.gz
[2024-05-15 18:26:54] -rw-r--r-- 1 boincer umatlas         0 May 15 14:44 boinc_lockfile
[2024-05-15 18:26:54] -rw-r--r-- 1 boincer umatlas      2640 May 15 14:44 pandaJob.out
[2024-05-15 18:26:54] -rw------- 1 boincer umatlas       467 May 15 14:45 setup.sh.local
[2024-05-15 18:26:54] -rw------- 1 boincer umatlas   1011265 May 15 14:45 agis_schedconf.cvmfs.json
[2024-05-15 18:26:54] -rw------- 1 boincer umatlas         0 May 15 14:45 agis_ddmendpoints.agis.ALL.json
[2024-05-15 18:26:54] -rw------- 1 boincer umatlas   1324701 May 15 14:45 cric_ddmendpoints.json
[2024-05-15 18:26:54] drwx------ 4 boincer umatlas      4096 May 15 14:45 pilot3
[2024-05-15 18:26:54] -rw-r--r-- 1 boincer umatlas      6385 May 15 18:19 init_data.xml
[2024-05-15 18:26:54] -rw------- 1 boincer umatlas       893 May 15 18:22 heartbeat.json
[2024-05-15 18:26:54] -rw------- 1 boincer umatlas        95 May 15 18:24 pilot_heartbeat.json
[2024-05-15 18:26:54] -rw-r--r-- 1 boincer umatlas       532 May 15 18:24 boinc_task_state.xml
[2024-05-15 18:26:54] -rw------- 1 boincer umatlas      1068 May 15 18:24 memory_monitor_summary.json
[2024-05-15 18:26:54] drwxrwx--- 2 boincer umatlas      4096 May 15 18:26 PanDA_Pilot-6206806110
[2024-05-15 18:26:54] -rw-r--r-- 1 boincer umatlas      8192 May 15 18:26 boinc_mmap_file
[2024-05-15 18:26:54] -rw-r--r-- 1 boincer umatlas        28 May 15 18:26 wrapper_checkpoint.txt
[2024-05-15 18:26:54] -rw------- 1 boincer umatlas   1148584 May 15 18:26 pilotlog.txt
[2024-05-15 18:26:54] -rw------- 1 boincer umatlas   1183943 May 15 18:26 log.38776104._006301.job.log.1
[2024-05-15 18:26:54] -rw-r--r-- 1 boincer umatlas       428 May 15 18:26 runtime_log
[2024-05-15 18:26:54] -rw------- 1 boincer umatlas       632 May 15 18:26 Po4LDm4bUR5nsSi4apGgGQJmABFKDmABFKDm4ySLDmZQOKDmPR0EOo.diag
[2024-05-15 18:26:54] -rw-r--r-- 1 boincer umatlas     11694 May 15 18:26 runtime_log.err
[2024-05-15 18:26:54] -rw-r--r-- 1 boincer umatlas     22248 May 15 18:26 stderr.txt
[2024-05-15 18:26:54] No HITS result produced
[2024-05-15 18:26:54]  *** Contents of shared directory: ***
[2024-05-15 18:26:54] total 587780
[2024-05-15 18:26:54] -rw-r--r-- 2 boincer umatlas 601375264 May 15 14:44 ATLAS.root_0
[2024-05-15 18:26:54] -rw-r--r-- 2 boincer umatlas     17537 May 15 14:44 start_atlas.sh
[2024-05-15 18:26:54] -rw-r--r-- 2 boincer umatlas    480004 May 15 14:44 input.tar.gz
18:26:56 (197387): run_atlas exited; CPU time 36157.393688
18:26:56 (197387): called boinc_finish(0)

</stderr_txt>
<message>
upload failure: <file_xfer_error>
  <file_name>Po4LDm4bUR5nsSi4apGgGQJmABFKDmABFKDm4ySLDmZQOKDmPR0EOo_0_r1546576704_ATLAS_result</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>

</message>
]]>

©2024 CERN