Message boards : LHCb Application : Huge LHCb Result
Message board moderation

To post messages, you must log in.

AuthorMessage
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2386
Credit: 222,987,246
RAC: 136,572
Message 34919 - Posted: 8 Apr 2018, 18:12:20 UTC

LHCb result file was nearly 1GB!
:-O

[08/Apr/2018:19:30:17 +0200] "CONNECT lbboinc01.cern.ch:9148 HTTP/1.0" 200 994685415 "-" "-" TCP_TUNNEL:HIER_DIRECT
ID: 34919 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2386
Credit: 222,987,246
RAC: 136,572
Message 34957 - Posted: 11 Apr 2018, 12:11:27 UTC

Found some more of that size in the logs:
1.3 GB
1.1 GB
1.0 GB
1.1 GB
1.2 GB

Will this become standard now?
What should volunteers with lower bandwidth do?

[07/Apr/2018:13:48:37 +0200] "CONNECT lbboinc01.cern.ch:9148 HTTP/1.0" 200 1267818530 "-" "-" TCP_TUNNEL:HIER_DIRECT
[07/Apr/2018:17:02:46 +0200] "CONNECT lbboinc01.cern.ch:9148 HTTP/1.0" 200 1085743755 "-" "-" TCP_TUNNEL:HIER_DIRECT
[09/Apr/2018:01:29:42 +0200] "CONNECT lbboinc01.cern.ch:9148 HTTP/1.0" 200 997377919 "-" "-" TCP_TUNNEL:HIER_DIRECT
[11/Apr/2018:09:36:35 +0200] "CONNECT lbboinc01.cern.ch:9148 HTTP/1.0" 200 1140769680 "-" "-" TCP_TUNNEL:HIER_DIRECT
[11/Apr/2018:13:35:39 +0200] "CONNECT lbboinc01.cern.ch:9148 HTTP/1.0" 200 1188767940 "-" "-" TCP_TUNNEL:HIER_DIRECT
ID: 34957 · Report as offensive     Reply Quote
Erich56

Send message
Joined: 18 Dec 15
Posts: 1686
Credit: 100,428,730
RAC: 102,734
Message 34960 - Posted: 11 Apr 2018, 15:35:12 UTC - in response to Message 34957.  

Found some more of that size in the logs:
1.3 GB
1.1 GB
1.0 GB
1.1 GB
1.2 GB
that's interesting - I havn't had any such huge one yet
ID: 34960 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2386
Credit: 222,987,246
RAC: 136,572
Message 34961 - Posted: 11 Apr 2018, 16:10:02 UTC

BTW, it's about a factor of 10 compared to the normal result size.
ID: 34961 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2386
Credit: 222,987,246
RAC: 136,572
Message 35001 - Posted: 13 Apr 2018, 20:40:20 UTC

It's climbing to new heights and happens on different hosts:
1.2 GB
1.3 GB
1.4 GB
1.6 GB

Is it due to a configuration error or did some tasks "escape" that were thought to be processed inside the CERN network?
Comments from the project team would be appreciated.

[12/Apr/2018:06:24:16 +0200] "CONNECT lbboinc01.cern.ch:9148 HTTP/1.0" 200 1197658388 "-" "-" TCP_TUNNEL:HIER_DIRECT
[13/Apr/2018:10:45:15 +0200] "CONNECT lbboinc01.cern.ch:9148 HTTP/1.0" 200 1255994250 "-" "-" TCP_TUNNEL:HIER_DIRECT
[13/Apr/2018:11:59:37 +0200] "CONNECT lbboinc01.cern.ch:9148 HTTP/1.0" 200 1447436119 "-" "-" TCP_TUNNEL:HIER_DIRECT
[13/Apr/2018:22:21:42 +0200] "CONNECT lbboinc01.cern.ch:9148 HTTP/1.0" 200 1550429788 "-" "-" TCP_TUNNEL:HIER_DIRECT
ID: 35001 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2386
Credit: 222,987,246
RAC: 136,572
Message 35119 - Posted: 30 Apr 2018, 8:49:41 UTC

During the past 2.5 weeks there were 78 uploads with more than 0.5 GB each.
That sums up to nearly 80 GB.
ID: 35119 · Report as offensive     Reply Quote
Kennywor

Send message
Joined: 23 Feb 08
Posts: 5
Credit: 127,194
RAC: 0
Message 35900 - Posted: 14 Jul 2018, 22:40:38 UTC - in response to Message 34957.  

What should volunteers with lower bandwidth do?


Guess.
ID: 35900 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2386
Credit: 222,987,246
RAC: 136,572
Message 36374 - Posted: 12 Aug 2018, 8:58:25 UTC

LHCb big result statistics between 2018-05-01 and 2018-08-11


Month
total_results/big_results_(>200MB)/%
max_upload_size


May
3122/18/0.6%
1.86 GiB

June
2739/36/1.3%
1.29 GiB

July
3889/45/1.2%
0.96 GiB

August
1424/88/6.2%
1.01 GiB


Please avoid sending out LHCb tasks to private volunteers that produce result files of that size.
ID: 36374 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2386
Credit: 222,987,246
RAC: 136,572
Message 36389 - Posted: 13 Aug 2018, 4:28:22 UTC - in response to Message 36374.  

Statistics as of last weekend 2018-08-11/12.

total_results/big_results_(>200MB)/%
max_upload_size

267/64/24.0%
1.01 GiB
ID: 36389 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 2071
Credit: 156,173,570
RAC: 105,533
Message 36392 - Posted: 13 Aug 2018, 6:51:53 UTC

Have you seen the same in -dev. Because there is a Multithread-Program?
ID: 36392 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2386
Credit: 222,987,246
RAC: 136,572
Message 36393 - Posted: 13 Aug 2018, 7:17:25 UTC - in response to Message 36392.  
Last modified: 13 Aug 2018, 7:17:43 UTC

Had only 3 valid LHCb WUs at -dev (May 2018).
It would be lots of work to extract the relevant information from the logfiles based on the timestamps.

As there were no changes regarding my software environment it's most likely an issue at CERN.
ID: 36393 · Report as offensive     Reply Quote
maeax

Send message
Joined: 2 May 07
Posts: 2071
Credit: 156,173,570
RAC: 105,533
Message 36394 - Posted: 13 Aug 2018, 7:23:22 UTC - in response to Message 36393.  
Last modified: 13 Aug 2018, 7:23:54 UTC

Have two-Core running in -dev (2 Tasks at the same moment) since weeks without problems!
ID: 36394 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2386
Credit: 222,987,246
RAC: 136,572
Message 36395 - Posted: 13 Aug 2018, 7:46:43 UTC - in response to Message 36394.  

There is no real problem nor any error message except the upload size of the (intermediate) result files which is occasionally 10 times as much as usual.
All of my last >120 LHCb WUs from different hosts finished with success.
ID: 36395 · Report as offensive     Reply Quote
Profile Magic Quantum Mechanic
Avatar

Send message
Joined: 24 Oct 04
Posts: 1114
Credit: 49,503,137
RAC: 3,956
Message 36398 - Posted: 13 Aug 2018, 15:33:15 UTC

I also have them running ok here and the 2-core version over at -dev
(Atlas is a waste of time for me since they take 10 hours to d/l and 4hrs at most to run)
ID: 36398 · Report as offensive     Reply Quote

Message boards : LHCb Application : Huge LHCb Result


©2024 CERN