Message boards : ATLAS application : Native ATLAS: Too many levels of symbolic links and CVMFS troubles
Message board moderation

To post messages, you must log in.

AuthorMessage
archie

Send message
Joined: 29 Dec 19
Posts: 1
Credit: 66,829
RAC: 0
Message 50543 - Posted: 11 Aug 2024, 6:21:44 UTC
Last modified: 11 Aug 2024, 6:41:13 UTC

I've had trouble getting the native ATLAS application running on my Gentoo Linux PC---when CVMFS is not "warm" (when the folders in /cvmfs are gone) it fails CVMFS checks, but if I run cvmfs_config probe before un-suspending the job it passes this point, but then crashes with
FATAL in sysInitialize(): standard std::exception is caught
from AthDictLoaderSvc in log.EVNTtoHITS.

It might be of note that I haven't restarted my PC since installing BOINC, but even if this issue is fixed that easily it is potentially useful for others having the same problem to know this.

This is some perhaps salient output from log.EVNTtoHITS:
00:35:41  --------------- HepPDT Version 2.06.01 ---------------
00:35:41 found 298 particles
00:35:41 ISF_LongLivedInputConverter                                  0    INFO   AlgTool: ISF_LongLivedInputConverter.ISF_ParticleSimWhiteList_ExtraParticles
00:35:41 ISF_LongLivedInputConverter                                  0    INFO   AlgTool: ISF_LongLivedInputConverter.ISF_ParticlePositionFilterDynamic
00:35:41 ISF_LongLivedInputConverter                                  0    INFO   AlgTool: ISF_LongLivedInputConverter.ISF_EtaPhiFilter
00:35:41 ISF_LongLivedInputConverter                                  0    INFO   AlgTool: ISF_LongLivedInputConverter.ISF_GenParticleInteractingFilter
00:35:41 ISF_Kernel_FullG4MT_QS                                       0 WARNING Two ISimulatorTool instances (ISF_Kernel_FullG4MT_QS.ISF_ParticleKillerTool,ToolSvc.ISF_ParticleKillerTool) with the same flavor in this job!
00:35:41  Check your configuration!
00:35:41 ISF_Kernel_FullG4MT_QS                                       0    INFO The following Simulators will be used in this job:      SimulationTools = PrivateToolHandleArray(['ISF::ParticleKillerSimTool/ISF_ParticleKillerTool','iGeant4::G4TransportTool/ISF_LongLivedGeant4Tool'])
00:35:41 ParticleKillerTool = PublicToolHandle('ISF::ParticleKillerSimTool/ISF_ParticleKillerTool')
00:35:41 ISF_Kernel_FullG4MT_QS                                       0    INFO The following routing chains are defined:
00:35:41 ISF_Kernel_FullG4MT_QS                                       0    INFO AtlasID (GeoID=1):      IDSimulationSelectors = PublicToolHandleArray(['ISF::DefaultSimSelector/ISF_FullGeant4Selector'])
00:35:41 ISF_Kernel_FullG4MT_QS                                       0    INFO AtlasForward (GeoID=2):         BeamPipeSimulationSelectors = PublicToolHandleArray(['ISF::DefaultSimSelector/ISF_FullGeant4Selector'])
00:35:41 ISF_Kernel_FullG4MT_QS                                       0    INFO AtlasCalo (GeoID=3):    CaloSimulationSelectors = PublicToolHandleArray(['ISF::DefaultSimSelector/ISF_FullGeant4Selector'])
00:35:41 ISF_Kernel_FullG4MT_QS                                       0    INFO AtlasMS (GeoID=4):      MSSimulationSelectors = PublicToolHandleArray(['ISF::DefaultSimSelector/ISF_FullGeant4Selector'])
00:35:41 ISF_Kernel_FullG4MT_QS                                       0    INFO AtlasCavern (GeoID=5):  CavernSimulationSelectors = PublicToolHandleArray(['ISF::DefaultSimSelector/ISF_DefaultParticleKillerSelector'])
00:35:41 ISF_MC15aPlusTruthService                                    0    INFO   AlgTool: ISF_MC15aPlusTruthService.ISF_MCTruthStrategyGroupID_MC15
00:35:41 ISF_MC15aPlusTruthService                                    0    INFO   AlgTool: ISF_MC15aPlusTruthService.ISF_MCTruthStrategyGroupIDHadInt_MC15
00:35:41 ISF_MC15aPlusTruthService                                    0    INFO   AlgTool: ISF_MC15aPlusTruthService.ISF_MCTruthStrategyGroupCaloMuBrem
00:35:41 ISF_MC15aPlusTruthService                                    0    INFO   AlgTool: ISF_MC15aPlusTruthService.ISF_MCTruthStrategyGroupCaloDecay_MC15
00:35:41 ClassIDSvc                                                   0    INFO getRegistryEntries: read 1519 CLIDRegistry entries for module ALL
00:35:41 CondInputLoader                                              0    INFO Initializing CondInputLoader...
00:35:41 AthDictLoaderSvc                                             0    INFO in initialize...
00:35:41 AthDictLoaderSvc                                             0   FATAL in sysInitialize(): standard std::exception is caught
00:35:41 AthDictLoaderSvc                                             0   ERROR boost::filesystem::status: Too many levels of symbolic links [system:40]: "/cvmfs/sft.cern.ch/lcg/releases/gcc/11.2.0-8a51a/x86_64-centos7/lib"
00:35:41 ServiceManager                                               0   ERROR Unable to initialize service "AthDictLoaderSvc"
00:35:41 CondInputLoader.DictLoaderSvc                                0   ERROR ServiceLocatorHelper::service: can not locate service AthDictLoaderSvc
00:35:41 CondInputLoader                                              0   ERROR IOVSvc/src/CondInputLoader.cxx:94 (StatusCode CondInputLoader::initialize()): code FAILURE: m_dictLoader.retrieve()
00:35:41 CondInputLoader                                              0   ERROR AthenaBaseComps/AthenaBaseComps/AthCommonDataStore.icc:43 (StatusCode AthCommonDataStore<PBASE>::sysInitialize() [with PBASE = AthCommonMsg<Algorithm>]): code FAILURE: PBASE::sysInitialize()
00:35:41 AthCondSeq                                                   0   ERROR Unable to initialize Algorithm CondInputLoader/CondInputLoader
00:35:41 AthCondSeq                                                   0   ERROR AthenaBaseComps/AthenaBaseComps/AthCommonDataStore.icc:43 (StatusCode AthCommonDataStore<PBASE>::sysInitialize() [with PBASE = AthCommonMsg<Gaudi::Sequence>]): code FAILURE: PBASE::sysInitialize()
00:35:41 AthAllAlgSeq                                                 0   ERROR Unable to initialize Algorithm AthSequencer/AthCondSeq
00:35:41 AthAllAlgSeq                                                 0   ERROR AthenaBaseComps/AthenaBaseComps/AthCommonDataStore.icc:43 (StatusCode AthCommonDataStore<PBASE>::sysInitialize() [with PBASE = AthCommonMsg<Gaudi::Sequence>]): code FAILURE: PBASE::sysInitialize()
00:35:41 AthAlgEvtSeq                                                 0   ERROR Unable to initialize Algorithm AthSequencer/AthAllAlgSeq
00:35:41 AthAlgEvtSeq                                                 0   ERROR AthenaBaseComps/AthenaBaseComps/AthCommonDataStore.icc:43 (StatusCode AthCommonDataStore<PBASE>::sysInitialize() [with PBASE = AthCommonMsg<Gaudi::Sequence>]): code FAILURE: PBASE::sysInitialize()
00:35:41 AthMasterSeq                                                 0   ERROR Unable to initialize Algorithm AthSequencer/AthAlgEvtSeq
00:35:41 ClassIDSvc                                                   0    INFO getRegistryEntries: read 348 CLIDRegistry entries for module ALL
00:35:41 OutputStreamHITS.OutputStreamHITS_TopFolder                  0 WARNING add: can not find type [JetCollection] in clid db
00:35:41 OutputStreamHITS.OutputStreamHITS_TopFolder                  0 WARNING add: can not find type [xAOD::JetContainer] in clid db
00:35:41 OutputStreamHITS.OutputStreamHITS_TopFolder                  0 WARNING add: can not find type [xAOD::JetContainer] in clid db
00:35:41 DecisionSvc                                                  0    INFO Inserting stream: OutputStreamHITS with no Algs
00:35:41 AthDictLoaderSvc                                             0    INFO in initialize...
00:35:41 AthDictLoaderSvc                                             0   FATAL in sysInitialize(): standard std::exception is caught
00:35:41 AthDictLoaderSvc                                             0   ERROR boost::filesystem::status: Too many levels of symbolic links [system:40]: "/cvmfs/sft.cern.ch/lcg/releases/gcc/11.2.0-8a51a/x86_64-centos7/lib"
00:35:41 ServiceManager                                               0   ERROR Unable to initialize service "AthDictLoaderSvc"
00:35:41 OutputStreamHITS.ServiceHandle('AthDictLoaderSvc')           0   ERROR ServiceLocatorHelper::service: can not locate service AthDictLoaderSvc
00:35:41 OutputStreamHITS                                             0   ERROR AthenaServices/src/AthenaOutputStream.cxx:232 (StatusCode AthenaOutputStream::initialize()): code FAILURE: m_dictLoader.retrieve()
00:35:41 OutputStreamHITS                                             0   ERROR AthenaBaseComps/AthenaBaseComps/AthCommonDataStore.icc:43 (StatusCode AthCommonDataStore<PBASE>::sysInitialize() [with PBASE = AthCommonMsg<Algorithm>]): code FAILURE: PBASE::sysInitialize()
00:35:41 AthOutSeq                                                    0   ERROR Unable to initialize Algorithm AthenaOutputStream/OutputStreamHITS
00:35:41 AthOutSeq                                                    0   ERROR AthenaBaseComps/AthenaBaseComps/AthCommonDataStore.icc:43 (StatusCode AthCommonDataStore<PBASE>::sysInitialize() [with PBASE = AthCommonMsg<Gaudi::Sequence>]): code FAILURE: PBASE::sysInitialize()
00:35:41 AthMasterSeq                                                 0   ERROR Unable to initialize Algorithm AthSequencer/AthOutSeq
00:35:41 AthMasterSeq                                                 0   ERROR AthenaBaseComps/AthenaBaseComps/AthCommonDataStore.icc:43 (StatusCode AthCommonDataStore<PBASE>::sysInitialize() [with PBASE = AthCommonMsg<Gaudi::Sequence>]): code FAILURE: PBASE::sysInitialize()
00:35:41 AthenaHiveEventLoopMgr                                       0   ERROR Unable to initialize Algorithm: AthMasterSeq
00:35:41 AthenaHiveEventLoopMgr                                       0   ERROR Failed to initialize base class MinimalEventLoopMgr
00:35:41 ServiceManager                                               0   ERROR Unable to initialize Service: AthenaHiveEventLoopMgr
00:35:41 Py:ComponentAccumulator   ERROR Failed to initialize AppMgr
00:35:41 Py:Sim_tf            INFO Run ISF simulation in 903.6656563282013 seconds
00:35:41 WARNING - Attempt to delete the physical volume store while geometry closed !
00:35:41 WARNING - Attempt to delete the solid store while geometry closed !
00:35:41 WARNING - Attempt to delete the logical volume store while geometry closed !
00:35:41 WARNING - Attempt to delete the region store while geometry closed !
00:35:41 leaving with code 1: "failure"



I originally had /var/lib/boinc symlinked to a directory on another drive as my root drive is low on space, but after removing it the problem persists.

The following URL links to the most recent task I have attempted. It first failed in the first way, but because I suspended it quickly enough it was able to restart when I resumed it. Then, once I resumed it it failed in the second way.
https://lhcathome.cern.ch/lhcathome/result.php?resultid=413396911

I've had one other task fail in the second way, but it was "validated" despite not having any actual useful results (unless these are just test tasks---I do have them enabled, but I assumed that they would say that they were somewhere).

edit: nicer title

edit 2: just thought I should add that I can probably only run native ATLAS on my PC as I don't have VirtualBox so I can only run native builds, and Theory's suspend and resume relies on systemd, which I don't have (Gentoo uses OpenRC by default)
ID: 50543 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2626
Credit: 266,225,201
RAC: 126,229
Message 50545 - Posted: 12 Aug 2024, 8:22:50 UTC - in response to Message 50543.  

Some comments

... when CVMFS is not "warm" (when the folders in /cvmfs are gone) it fails CVMFS checks ...

This means, CVMFS did not mount the repositories.
Since CVMFS relies on autofs, check that service.
A reboot might help to start from a well defined point.
You may also check your CVMFS configuration against this threads:
https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=5594
https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=5595


... if I run cvmfs_config probe before un-suspending the job ...

Never suspend/resume or restart an ATLAS native job as it will drop all previously completed calculation and restart from scratch.


... because I suspended it quickly enough it was able to restart when I resumed it ...

Same here.
Do not suspend/resume any ATLAS native task.



I don't have VirtualBox

There's a recent VirtualBox packet for Gentoo.
See the Gentoo manual/wiki.


Gentoo uses OpenRC by default

Gentoo can be told to use systemd.
See the Gentoo manual/wiki.


I originally had /var/lib/boinc symlinked to a directory on another drive as my root drive is low on space, but after removing it the problem persists.

Check your mountpoints (esp. the basic permissions) and symbolic links (relative vs. absolute).

If you have trouble using a non-standard BOINC installation, try a standard installation.
ID: 50545 · Report as offensive     Reply Quote

Message boards : ATLAS application : Native ATLAS: Too many levels of symbolic links and CVMFS troubles


©2025 CERN