Message boards : ATLAS application : ATLAS vbox and native 3.01
Message board moderation

To post messages, you must log in.

1 · 2 · 3 · 4 · Next

AuthorMessage
David Cameron
Project administrator
Project developer
Project scientist

Send message
Joined: 13 May 14
Posts: 387
Credit: 15,314,184
RAC: 0
Message 47914 - Posted: 28 Mar 2023, 11:15:43 UTC

Hi all,

We have just made release 3.01 for vbox and native apps in preparation for running updated simulation software. Until now all ATLAS tasks have been simulations of the ATLAS detector during "Run 2" of the LHC that ran from 2015 to 2018. Even 5 years later ATLAS physicists are still analysing the data from that period and require new simulations to be done. However last year "Run 3" of the LHC started and it will last until the end of 2025. Soon we will switch the ATLAS tasks here to Run 3 simulations, where many things have changed in the detector and the software used to simulate and analyse data.

Version 3 of the app allows us to run both Run 2 and Run 3 simulations in the same version during the transition period, however this does make the image for vbox a bit larger (4.4GB, or 1.8GB compressed). The Linux native app is the same as before with one very minor change to set an environment variable required by the new software. The main benefit of the software used for Run 3 simulations is that it uses far less memory, less than 3GB for an 8-core task.

At the moment we are not yet submitting Run 3 tasks here so everything should work exactly the same as before, but we'll start doing this soon and let you know. We have been doing some extensive testing on the LHC-dev project and there are still a couple of things to fix.
ID: 47914 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 2096
Credit: 159,633,218
RAC: 142,040
Message 47915 - Posted: 28 Mar 2023, 12:38:52 UTC - in response to Message 47914.  
Last modified: 28 Mar 2023, 12:49:58 UTC

Have in the last two hours 100 Tasks with download Error.
<core_client_version>7.20.2</core_client_version>
<![CDATA[
<message>
app_version download error: couldn't get input files:
<file_xfer_error>

<file_name>ATLAS_vbox_3.01_image.vdi</file_name>
<error_code>-224 (permanent HTTP error)</error_code>
<error_message>permanent HTTP error</error_message>
</file_xfer_error>
</message>
]]>
ID: 47915 · Report as offensive     Reply Quote
David Cameron
Project administrator
Project developer
Project scientist

Send message
Joined: 13 May 14
Posts: 387
Credit: 15,314,184
RAC: 0
Message 47916 - Posted: 28 Mar 2023, 13:01:26 UTC - in response to Message 47915.  

Sorry, the file had the wrong permissions on the server. It should work now.
ID: 47916 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2411
Credit: 226,126,557
RAC: 128,184
Message 47917 - Posted: 28 Mar 2023, 13:06:29 UTC - in response to Message 47916.  

vdi as well as vdi.gz are now allowed to be downloaded (tested with wget).
ID: 47917 · Report as offensive     Reply Quote
Profile Yeti
Volunteer moderator
Avatar

Send message
Joined: 2 Sep 04
Posts: 453
Credit: 193,569,815
RAC: 13,099
Message 47918 - Posted: 28 Mar 2023, 13:14:37 UTC

I have got a new 3.01 WU, but it seems still to contain this big 1,2 GB file.

What will happen with it, as far as I understood it is useless for BOINC-Crunchers? It would be a waste of resources, electricity, time, bandwith and lifetime of all SSDs if it is still downloaded even it isn't needed.


Supporting BOINC, a great concept !
ID: 47918 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2411
Credit: 226,126,557
RAC: 128,184
Message 47919 - Posted: 28 Mar 2023, 13:18:34 UTC - in response to Message 47918.  

as far as I understood it is useless for BOINC-Crunchers?

I must have missed that.
Where did you get that idea from?
ID: 47919 · Report as offensive     Reply Quote
Profile Yeti
Volunteer moderator
Avatar

Send message
Joined: 2 Sep 04
Posts: 453
Credit: 193,569,815
RAC: 13,099
Message 47920 - Posted: 28 Mar 2023, 13:23:31 UTC - in response to Message 47919.  

Where did you get that idea from?
https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=5976&postid=47891#47891


Supporting BOINC, a great concept !
ID: 47920 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2411
Credit: 226,126,557
RAC: 128,184
Message 47921 - Posted: 28 Mar 2023, 13:31:47 UTC - in response to Message 47920.  

I agree, those huge EVNT files are annoying and shouldn't have been sent to BOINC for a couple of reasons.
But I still can't find any words that claim they are useless for science.
ID: 47921 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 2096
Credit: 159,633,218
RAC: 142,040
Message 47922 - Posted: 28 Mar 2023, 13:38:51 UTC - in response to Message 47916.  

Sorry, the file had the wrong permissions on the server. It should work now.

Download now ok,
Important is the work we do with this Tasks and not the size of the input file.
ID: 47922 · Report as offensive     Reply Quote
Profile Yeti
Volunteer moderator
Avatar

Send message
Joined: 2 Sep 04
Posts: 453
Credit: 193,569,815
RAC: 13,099
Message 47923 - Posted: 28 Mar 2023, 13:42:03 UTC - in response to Message 47921.  

But I still can't find any words that claim they are useless for science.
These are not my words
a scientist added a batch to the ATLAS-queue not meant for BOINC

They are not meant for BOINC


Supporting BOINC, a great concept !
ID: 47923 · Report as offensive     Reply Quote
Profile Yeti
Volunteer moderator
Avatar

Send message
Joined: 2 Sep 04
Posts: 453
Credit: 193,569,815
RAC: 13,099
Message 47924 - Posted: 28 Mar 2023, 13:47:56 UTC - in response to Message 47922.  
Last modified: 28 Mar 2023, 13:48:43 UTC

Important is the work we do with this Tasks
Shure
and not the size of the input file.

If I can do the same science with 130 / 200 / 300 MB Downloads like before, why should we now download 1,2 GB for the same amount of science?

Maybe it is okay for you, fine, but for me it is a wastefulness


Supporting BOINC, a great concept !
ID: 47924 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 2096
Credit: 159,633,218
RAC: 142,040
Message 47925 - Posted: 28 Mar 2023, 13:50:48 UTC - in response to Message 47924.  
Last modified: 28 Mar 2023, 14:02:40 UTC

Therefore we need a comment from science group generating the input.
As long as the Errors are smaller than the successful Tasks, no problem so long.
ID: 47925 · Report as offensive     Reply Quote
David Cameron
Project administrator
Project developer
Project scientist

Send message
Joined: 13 May 14
Posts: 387
Credit: 15,314,184
RAC: 0
Message 47926 - Posted: 28 Mar 2023, 14:08:03 UTC - in response to Message 47925.  

The science output is the same regardless of the size of input file. The unfortunate large input files are the result of accidentally creating files with too many events instead of splitting them into smaller chunks. But they are not a waste or useless for science since the processing and output is the same.
ID: 47926 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2411
Credit: 226,126,557
RAC: 128,184
Message 47927 - Posted: 28 Mar 2023, 14:12:40 UTC - in response to Message 47926.  
Last modified: 28 Mar 2023, 14:14:24 UTC

Thanks.
Just wondering why this is mentioned in (all?) log.EVNTtoHITS:
maxEvents = 200


<edit>
How many of those big dogs are left?
</edit>
ID: 47927 · Report as offensive     Reply Quote
David Cameron
Project administrator
Project developer
Project scientist

Send message
Joined: 13 May 14
Posts: 387
Credit: 15,314,184
RAC: 0
Message 47928 - Posted: 28 Mar 2023, 14:36:53 UTC - in response to Message 47927.  

Thanks.
Just wondering why this is mentioned in (all?) log.EVNTtoHITS:
maxEvents = 200


This is because each task is configured to simulate 200 events, even though there are more events than that in the input EVNT file. The EVNT file may contain 10000 events for example, so this would be split into 50 tasks of 200 events and each of those 50 tasks would download the same file.


How many of those big dogs are left?


I think this may be all of the remaining tasks for Run 2 simulation, which is why I would like to get the Run 3 simulation going as soon as possible.
ID: 47928 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 2096
Credit: 159,633,218
RAC: 142,040
Message 47929 - Posted: 28 Mar 2023, 17:17:02 UTC - in response to Message 47928.  

Is it a problem to do 500 instead of 200 events for this tasks, or
have this input file not the data therefore?
ID: 47929 · Report as offensive     Reply Quote
Profile rbpeake

Send message
Joined: 17 Sep 04
Posts: 99
Credit: 30,698,195
RAC: 5,637
Message 47930 - Posted: 28 Mar 2023, 17:42:18 UTC - in response to Message 47929.  

I agree with this.

Thanks.
Regards,
Bob P.
ID: 47930 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2411
Credit: 226,126,557
RAC: 128,184
Message 47931 - Posted: 28 Mar 2023, 17:48:28 UTC - in response to Message 47930.  

As far as I understand the task generation process this is not possible for tasks that are already in the queue.
At least not without time intensive and error-prone manual intervention.
ID: 47931 · Report as offensive     Reply Quote
Crystal Pellet
Volunteer moderator
Volunteer tester

Send message
Joined: 14 Jan 10
Posts: 1279
Credit: 8,484,048
RAC: 1,651
Message 47932 - Posted: 28 Mar 2023, 19:18:12 UTC

I ran a few tasks with the new image-vdi and exe and had setup the VM to run with 4096 MB RAM and 8 threads.

Not much swapping is used.
The ALT-F2 monitoring seems to work OK here. Not stuck like on -dev.
ID: 47932 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 2096
Credit: 159,633,218
RAC: 142,040
Message 47934 - Posted: 29 Mar 2023, 0:02:28 UTC - in response to Message 47932.  


The ALT-F2 monitoring seems to work OK here. Not stuck like on -dev.


+1
ID: 47934 · Report as offensive     Reply Quote
1 · 2 · 3 · 4 · Next

Message boards : ATLAS application : ATLAS vbox and native 3.01


©2024 CERN