Message boards : CMS Application : New Version 70.00
Message board moderation

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
Profile Laurence
Project administrator
Project developer

Send message
Joined: 20 Jun 14
Posts: 355
Credit: 238,395
RAC: 0
Message 47448 - Posted: 1 Nov 2022, 9:39:42 UTC

This new version of the CMS app now supports the multimode attach feature. This means that the image is no longer copied to the slot directory with each new job. Let us know if there are any issues.
ID: 47448 · Report as offensive     Reply Quote
ivan
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Project scientist
Avatar

Send message
Joined: 29 Aug 05
Posts: 918
Credit: 6,039,108
RAC: 1,409
Message 47449 - Posted: 1 Nov 2022, 10:26:32 UTC - in response to Message 47448.  

My Windows 10 machine hasn't downloaded the new version.
ID: 47449 · Report as offensive     Reply Quote
Profile Laurence
Project administrator
Project developer

Send message
Joined: 20 Jun 14
Posts: 355
Credit: 238,395
RAC: 0
Message 47450 - Posted: 1 Nov 2022, 10:31:27 UTC - in response to Message 47449.  

Please try again
ID: 47450 · Report as offensive     Reply Quote
ivan
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Project scientist
Avatar

Send message
Joined: 29 Aug 05
Posts: 918
Credit: 6,039,108
RAC: 1,409
Message 47451 - Posted: 1 Nov 2022, 11:32:28 UTC - in response to Message 47450.  

OK, I reset the project and now it's downloading.
ID: 47451 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jun 08
Posts: 2121
Credit: 169,265,748
RAC: 113,995
Message 47452 - Posted: 1 Nov 2022, 13:03:35 UTC - in response to Message 47448.  

The app version has changed to multicore.
By intention?
ID: 47452 · Report as offensive     Reply Quote
Profile Laurence
Project administrator
Project developer

Send message
Joined: 20 Jun 14
Posts: 355
Credit: 238,395
RAC: 0
Message 47453 - Posted: 1 Nov 2022, 13:59:59 UTC - in response to Message 47452.  

No. Will fix this asap. v70.10 on it's way.
ID: 47453 · Report as offensive     Reply Quote
Jim1348

Send message
Joined: 15 Nov 14
Posts: 602
Credit: 24,371,321
RAC: 11
Message 47454 - Posted: 1 Nov 2022, 14:34:32 UTC

I am wondering if this means that CMS is "operational", or still doing test units.
I will try it if they are doing actual work units.
ID: 47454 · Report as offensive     Reply Quote
hadron

Send message
Joined: 4 Sep 22
Posts: 15
Credit: 1,763,419
RAC: 18,688
Message 47455 - Posted: 1 Nov 2022, 15:31:34 UTC

Just what are we looking for here?
ID: 47455 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jun 08
Posts: 2121
Credit: 169,265,748
RAC: 113,995
Message 47456 - Posted: 1 Nov 2022, 16:28:27 UTC

Each of my VMs starts with log entries like these:
2022-11-01 12:50:43 (39334): Guest Log: [INFO] CMS application starting. Check log files.
2022-11-01 12:50:43 (39334): Guest Log: [INFO] Requesting an idtoken from LHC@home
2022-11-01 12:50:44 (39334): Guest Log: [INFO] Requesting an idtoken from vLHC@home-dev
2022-11-01 12:51:15 (39334): Guest Log: [INFO] Requesting an idtoken from LHC@home
2022-11-01 12:51:15 (39334): Guest Log: [INFO] Requesting an idtoken from vLHC@home-dev
2022-11-01 12:51:45 (39334): Guest Log: [INFO] Requesting an idtoken from LHC@home
2022-11-01 12:51:46 (39334): Guest Log: [INFO] Requesting an idtoken from vLHC@home-dev
2022-11-01 12:52:17 (39334): Guest Log: [INFO] Requesting an idtoken from LHC@home
2022-11-01 12:52:17 (39334): Guest Log: [INFO] Requesting an idtoken from vLHC@home-dev
2022-11-01 12:52:48 (39334): Guest Log: [INFO] Requesting an idtoken from LHC@home
2022-11-01 12:52:49 (39334): Guest Log: [INFO] Requesting an idtoken from vLHC@home-dev
2022-11-01 12:53:20 (39334): Guest Log: [INFO] Requesting an idtoken from LHC@home
2022-11-01 12:53:20 (39334): Guest Log: [INFO] Requesting an idtoken from vLHC@home-dev
2022-11-01 12:53:52 (39334): Guest Log: [DEBUG]   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2022-11-01 12:53:52 (39334): Guest Log:                                  Dload  Upload   Total   Spent    Left  Speed
2022-11-01 12:53:52 (39334): Guest Log:   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
2022-11-01 12:53:52 (39334): Guest Log: 100   221  100   221    0     0    436      0 --:--:-- --:--:-- --:--:--   437
2022-11-01 12:53:52 (39334): Guest Log: [DEBUG]   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2022-11-01 12:53:52 (39334): Guest Log:                                  Dload  Upload   Total   Spent    Left  Speed
2022-11-01 12:53:52 (39334): Guest Log:   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
2022-11-01 12:53:52 (39334): Guest Log: 100   221  100   221    0     0    436      0 --:--:-- --:--:-- --:--:--   437
2022-11-01 12:53:52 (39334): Guest Log: [ERROR] Could not get an x509 credential

Nonetheless the CMS jobs seem to run fine.
ID: 47456 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jun 08
Posts: 2121
Credit: 169,265,748
RAC: 113,995
Message 47457 - Posted: 1 Nov 2022, 16:34:19 UTC

Got a fresh v70.00 task after a couple of v70.10 tasks.
This usually points out that not all server instances have been restarted after the recent app upgrade.
ID: 47457 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jun 08
Posts: 2121
Credit: 169,265,748
RAC: 113,995
Message 47458 - Posted: 1 Nov 2022, 16:36:54 UTC - in response to Message 47454.  

CMS never runs test units, not even on the dev server.
The scientific payload comes from the same backend queue.
ID: 47458 · Report as offensive     Reply Quote
Jim1348

Send message
Joined: 15 Nov 14
Posts: 602
Credit: 24,371,321
RAC: 11
Message 47459 - Posted: 1 Nov 2022, 16:54:24 UTC - in response to Message 47458.  
Last modified: 1 Nov 2022, 16:54:36 UTC

CMS never runs test units, not even on the dev server.

OK, I had gotten the opposite impression.
https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=5855&postid=47122#47122
ID: 47459 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jun 08
Posts: 2121
Credit: 169,265,748
RAC: 113,995
Message 47461 - Posted: 1 Nov 2022, 17:10:51 UTC - in response to Message 47459.  

You probably stumbled over this:
"We've been running the same test-flow for a couple of years now ...".

It means that Ivan has to watch the backend queue and manually creates fresh work when it starts getting dry.
The BOINC related queue automatically generates "envelope" tasks.
ID: 47461 · Report as offensive     Reply Quote
Jim1348

Send message
Joined: 15 Nov 14
Posts: 602
Credit: 24,371,321
RAC: 11
Message 47462 - Posted: 1 Nov 2022, 19:31:43 UTC - in response to Message 47461.  

OK, I have been wanting to do it. I am in again.
ID: 47462 · Report as offensive     Reply Quote
Toby Broom
Volunteer moderator

Send message
Joined: 27 Sep 08
Posts: 742
Credit: 539,921,127
RAC: 298,227
Message 47463 - Posted: 1 Nov 2022, 22:18:51 UTC - in response to Message 47457.  

Same for me, I keep aborting the 70.00
ID: 47463 · Report as offensive     Reply Quote
ivan
Volunteer moderator
Project tester
Volunteer developer
Volunteer tester
Project scientist
Avatar

Send message
Joined: 29 Aug 05
Posts: 918
Credit: 6,039,108
RAC: 1,409
Message 47464 - Posted: 1 Nov 2022, 23:26:28 UTC - in response to Message 47456.  

Yes, I noticed that message about the x509 credentials too but, as you say, the jobs seem to be running fine. We do still need those credentials to write results to the Data Bridge; I'll check that that's working properly tomorrow.
ID: 47464 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jun 08
Posts: 2121
Credit: 169,265,748
RAC: 113,995
Message 47466 - Posted: 2 Nov 2022, 7:12:55 UTC - in response to Message 47455.  

Just what are we looking for here?

Not sure what you are asking for.
- the scientific background?
- the task setup from the IT perspective?
- anything else?

Would be nice if you could post a hint.
ID: 47466 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jun 08
Posts: 2121
Credit: 169,265,748
RAC: 113,995
Message 47467 - Posted: 2 Nov 2022, 8:16:57 UTC

CMS v70.00 and v70.10 are based on the same vdi file "CMS_2022_09_07_prod.vdi".
The kernel boot parameters of that vdi contain an old bug that was introduced long ago.
It affects the CVMFS configuration when the VM starts to boot.

The server lists passed to the kernel look as follows:
cvmfs_server=cvmfs-stratum-one.cern.ch,cvmfs-s1fnal.opensciencegrid.org,cvmfs-s1bnl.opensciencegrid.org,grid-cvmfs-one.desy.de
cvmfs_cdn=s1cern-cvmfs.openhtc.io,s1ral-cvmfs.openhtc.io;s1bnl-cvmfs.openhtc.io;s1fnal-cvmfs.openhtc.io;s1unl-cvmfs.openhtc.io


Although a ";" (semicolon) must be used as separator plus double quotes to enclose the list later within the /etc/cvmfs configuration files it is not allowed here.
Instead, a "," (comma) must be used here.
The "cvmfs_server" list looks fine but not the "cvmfs_cdn" list.

As a result the "cvmfs_cdn" list configures only 2 servers (s1cern-cvmfs.openhtc.io and s1ral-cvmfs.openhtc.io).
All others behind the 1st ";" will be ignored.

This affects load-balancing as well as fail-over.

Load-balancing now happens only between the backends at cern and ral.
Example:
A client located near Melbourne will get CVMFS data from the Cloudflare proxy nearby (most likely also Melbourne) but if this proxy has to refresh it's cache it will send requests to Europe although it could get the data from Swinburne (see updated lists below).

Fail-over will never switch to the backends at bnl, fnal and unl if cern and ral are both down.


@Laurence
If you touch this setting you might also want to sync the server lists with the recent lists from the CVMFS master.
The result should then look like:
cvmfs_server=cvmfs-stratum-one.cern.ch:8000,cernvmfs.gridpp.rl.ac.uk:8000,cvmfs-s1bnl.opensciencegrid.org:8000,cvmfs-s1fnal.opensciencegrid.org:8000,cvmfsrep.grid.sinica.edu.tw:8000,cvmfs-stratum-one.ihep.ac.cn:8000,cvmfs-s1.hpc.swin.edu.au:8000
cvmfs_cdn=s1cern-cvmfs.openhtc.io,s1ral-cvmfs.openhtc.io,s1bnl-cvmfs.openhtc.io,s1fnal-cvmfs.openhtc.io:8080,s1asgc-cvmfs.openhtc.io:8080,s1ihep-cvmfs.openhtc.io:8080,s1swinburne-cvmfs.openhtc.io:8080
ID: 47467 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Avatar

Send message
Joined: 15 Jun 08
Posts: 2121
Credit: 169,265,748
RAC: 113,995
Message 47470 - Posted: 2 Nov 2022, 12:07:46 UTC

The following files inside the vdi are also not up to date.
They should be changed according to:
https://github.com/cvmfs-contrib/config-repo/tree/master/etc/cvmfs/domain.d

[/persistent]/etc/cvmfs/domain.d/cern.ch.conf
# These are here so cvmfs will notice them if common.conf sets them
CVMFS_USE_CDN="$CVMFS_USE_CDN"
CVMFS_HTTP_PROXY="$CVMFS_HTTP_PROXY"
CVMFS_FALLBACK_PROXY="$CVMFS_FALLBACK_PROXY"
. ../common.conf

if [ "$CVMFS_USE_CDN" = "yes" ]; then
    CVMFS_SERVER_URL="http://s1cern-cvmfs.openhtc.io/cvmfs/@fqrn@;http://s1ral-cvmfs.openhtc.io/cvmfs/@fqrn@;http://s1bnl-cvmfs.openhtc.io/cvmfs/@fqrn@;http://s1fnal-cvmfs.openhtc.io:8080/cvmfs/@fqrn@;http://s1asgc-cvmfs.openhtc.io:8080/cvmfs/@fqrn@;http://s1ihep-cvmfs.openhtc.io:8080/cvmfs/@fqrn@;http://s1swinburne-cvmfs.openhtc.io:8080/cvmfs/@fqrn@"
else
    CVMFS_SERVER_URL="http://cvmfs-stratum-one.cern.ch:8000/cvmfs/@fqrn@;http://cernvmfs.gridpp.rl.ac.uk:8000/cvmfs/@fqrn@;http://cvmfs-s1bnl.opensciencegrid.org:8000/cvmfs/@fqrn@;http://cvmfs-s1fnal.opensciencegrid.org:8000/cvmfs/@fqrn@;http://cvmfsrep.grid.sinica.edu.tw:8000/cvmfs/@fqrn@;http://cvmfs-stratum-one.ihep.ac.cn:8000/cvmfs/@fqrn@;http://cvmfs-s1.hpc.swin.edu.au:8000/cvmfs/@fqrn@"
fi
CVMFS_KEYS_DIR=$CVMFS_MOUNT_DIR/$CVMFS_CONFIG_REPOSITORY/etc/cvmfs/keys/cern.ch



[/persistent]/etc/cvmfs/domain.d/opensciencegrid.org.conf
# These are here so cvmfs will notice them if common.conf sets them
CVMFS_USE_CDN="$CVMFS_USE_CDN"
CVMFS_HTTP_PROXY="$CVMFS_HTTP_PROXY"
CVMFS_FALLBACK_PROXY="$CVMFS_FALLBACK_PROXY"
. ../common.conf

if [ "$CVMFS_USE_CDN" = "yes" ]; then
    CVMFS_SERVER_URL="http://s1ral-cvmfs.openhtc.io/cvmfs/@fqrn@;http://s1nikhef-cvmfs.openhtc.io/cvmfs/@fqrn@;http://s1bnl-cvmfs.openhtc.io/cvmfs/@fqrn@;http://s1fnal-cvmfs.openhtc.io:8080/cvmfs/@fqrn@;http://s1asgc-cvmfs.openhtc.io:8080/cvmfs/@fqrn@;http://s1ihep-cvmfs.openhtc.io:8080/cvmfs/@fqrn@;http://s1swinburne-cvmfs.openhtc.io:8080/cvmfs/@fqrn@"
else
    CVMFS_SERVER_URL="http://cvmfs-egi.gridpp.rl.ac.uk:8000/cvmfs/@fqrn@;http://klei.nikhef.nl:8000/cvmfs/@fqrn@;http://cvmfs-s1bnl.opensciencegrid.org:8000/cvmfs/@fqrn@;http://cvmfs-s1fnal.opensciencegrid.org:8000/cvmfs/@fqrn@;http://cvmfsrep.grid.sinica.edu.tw:8000/cvmfs/@fqrn@;http://cvmfs-stratum-one.ihep.ac.cn:8000/cvmfs/@fqrn@;http://cvmfs-s1.hpc.swin.edu.au:8000/cvmfs/@fqrn@"
fi
CVMFS_KEYS_DIR=$CVMFS_MOUNT_DIR/$CVMFS_CONFIG_REPOSITORY/etc/cvmfs/keys/opensciencegrid.org
ID: 47470 · Report as offensive     Reply Quote
hadron

Send message
Joined: 4 Sep 22
Posts: 15
Credit: 1,763,419
RAC: 18,688
Message 47471 - Posted: 2 Nov 2022, 15:24:47 UTC - in response to Message 47466.  
Last modified: 2 Nov 2022, 15:25:06 UTC

Just what are we looking for here?

Not sure what you are asking for.
- the scientific background?
- the task setup from the IT perspective?
- anything else?

Would be nice if you could post a hint.

What files on my system are new/changed? Are any old ones left behind that can be deleted?
ID: 47471 · Report as offensive     Reply Quote
1 · 2 · Next

Message boards : CMS Application : New Version 70.00


©2023 CERN