log in

The Dreaded .....not fast enough isp that Cern server won't just.....


Advanced search

Message boards : Theory Application : The Dreaded .....not fast enough isp that Cern server won't just.....

Author Message
Profile MAGIC Quantum Mechanic
Avatar
Send message
Joined: 24 Oct 04
Posts: 512
Credit: 14,661,078
RAC: 3,859
Message 32172 - Posted: 1 Sep 2017, 13:09:06 UTC

PLEASE move the multi-core Theory tasks over to here!!

I know some people on the planet do have high-speed internet usually via cable but I also know many of us have a room full of computers but at times have it running at a speed under 1Mbps and after running several thousand of these tasks I know internet speed that ever causes these particular tasks from starting......once they start......guaranteed Valid.

It is because of the combination of the Cern server and the dreaded Oracle VirtualBox

These VB tasks are required to also run connected to the internet the entire time.

Not that problem with SixTracks or GPU tasks (I run those at Einstein)

I can run hundreds of those and unplug the ethernet from all 9 computers.

Each month my monthly data transfer is gone after 6 days so I get throttled down from 45Mbps to at many times under 1Mbps

I know after we moved VB over here new VB members could not understand why these tasks never started and it can be watched on our VM Console as it will get stuck on page one.

This is the part that they will not get to in less than 20 minutes if that internet speed is not fast enough.



So after I get all my multi-cores started over at -dev I have to struggle here trying to get 24 single cores to start running and I have to do one at a time.

And I do this after midnight until 4am and finally give up......last night I got 23 running and this time (it is now 6am so I have skipped sleeping) I also got 23 of the 24 running so on my tasks page you can see all the ones that didn't start and that can be seen by the fact they are Invalid after that 20 minute wall.

Now I don't want that doubled are updated at all since they could just sit there running but NOT starting........and I watch all of mine and after they hit the 20 minute wall and the VM Console tells me I abort them otherwise they run another 5+ minutes on top of that 20 minutes of nothing.

SO to make my long story a bit longer........it sure would be nice to run these here multi-core.

In fact at dev and other test place I have run thousands of multi-core Theory,CMS.and LHCb tasks Valid and ALL the Invalids were because of the internet speed problem.

Imagine if we still had dialups like back when I first started here in 2004

Guaranteed disaster......but then it was all SixTracks so that didn't even matter since internet speed makes no problems.


____________
Volunteer Mad Scientist For Life

computezrmle
Send message
Joined: 15 Jun 08
Posts: 347
Credit: 3,501,142
RAC: 1,838
Message 32174 - Posted: 1 Sep 2017, 14:26:33 UTC - in response to Message 32172.

We discussed that a while ago.
A small linux box with a squid and a few iptables rules would be very helpful.
Especially for that:

MAGIC Quantum Mechanic wrote:
... my monthly data transfer is gone after 6 days ...
... watched on our VM Console as it will get stuck on page one.

The VMs typically download huge amounts of data during their setup and to fill their local CVMFS.
This data gets lost when the VM shuts down and is downloaded again by the next VM.
A perfect scenario for a cache.

Profile MAGIC Quantum Mechanic
Avatar
Send message
Joined: 24 Oct 04
Posts: 512
Credit: 14,661,078
RAC: 3,859
Message 32188 - Posted: 2 Sep 2017, 1:07:42 UTC - in response to Message 32174.
Last modified: 2 Sep 2017, 1:08:24 UTC

I guess you missed the part where I said I have run THOUSANDS of test multicore versions of Theory,CMS,and LHCb tasks Valid.
(I have them running as I type this)

I also did the alpha version of multi-core Atlas tasks before the came here.

Remember I have been doing VB tasks since March 1st 2011 and was the only member to run those at T4T from day one until the very last seconds before we moved them here and had the Top 5 computers in the 6+ years over there so I know what I am talking about computezrmle

You can also stop by at vLHC-dev and see who has done most of those tests.
____________
Volunteer Mad Scientist For Life

Message boards : Theory Application : The Dreaded .....not fast enough isp that Cern server won't just.....