21) Message boards : Number crunching : Bandwidth and ram for vb and native tasks? (Message 41043)
Posted 24 Dec 2019 by wolfman1360
Post:
I can only speak for Linux:

Native ATLAS takes 2 GB, and CMS (VBox) takes 3 GB. At least that is what they show in BoincTasks, but actual usage is less. I expect you will need to plan for the worst-case in order to run the number you want.
Theory (native) is very small - averaging around 20 MB, though occasionally more, and SixTrack only about 100 MB in Linux.

Okay, so apart from Theory, which I thought took tons more, even on Linux, I had everything almost right, if a little backwards. Thanks
The RAM usage is well known but I think the bandwidth is not. You can get a total daily transfer history from BOINC on the command line:

boinccmd --get_daily_xfer_history


But it will not be broken down by project and/or sub-project. It will just show the daily transfer totals for that host. If you have a proxy server running for your BOINC clients you might be able to save a lot of network traffic with it on the native workloads.

Oh that will be a life saver - thank you. I plan on making LHC exclusive, at least for a little while, apart from maybe CPDN. Local proxies are far, far down the road for me. This will be by far one of the most involved projects I'll be in - or maybe that's me overthinking and overcomplicating everything....

I guess this means 75 Mbit/s download and 3-4 Mbit/s upload?
75 down are far away from being a problem in this case.

3-4 up might be a problem.

Yes, that's what I meant - Mbit.
I do have other machines that are in datacenters and internet isn't an issue. One day that 300/25 will come my way and I can be happy. And maybe a few more Ryzens too.
I'm aiming for theoretically using up around 75% of each machine if at all possible. Should I be aiming at more or less? Basically 2 threads not being used on an 8 threads machine and similar on 16 - though might bump that up to 3 not in use.
22) Message boards : Number crunching : Bandwidth and ram for vb and native tasks? (Message 41038)
Posted 23 Dec 2019 by wolfman1360
Post:
So all Windows and Linux machines have finally run dry over here. Time to actually migrate over to LHC.
Do all tasks using native and / or virtual box require constant internet access? I have 75 down and a claimed 6 up, yet only get around 3-4. Will this be a problem with around 5 8-core machines and 1 16? I'm assuming CMS, Theory and Atlas use similar amounts of bandwidth with more incoming than outgoing.

Now for ram - from my reading, Atlas takes the most for a singlecore task with CMS behind that and theory taking the least, even less so for native tasks under Linux.
Just as an example on my 12 thread i7-8750H, which has 32 GB ram running Windows 10. I should be fine running 10 single core theory or CMS tasks, however should run 3 4-core Atlas tasks to prevent the use of too much ram?

Thanks. I will of course be experimenting slowly. I don't want to start off with more than the machines or internet can handle. All of the machines I'm referencing apart from the laptop are dedicated crunchers.
23) Message boards : Number crunching : Max # jobs and Max # CPUs (Message 41037)
Posted 23 Dec 2019 by wolfman1360
Post:
I don't think anyone answered one of your questions:

... on the single little core 2 duo I have running on here, if there was an Asteroids at home task running, and boinc decided a native theory Atlas task should start, with max number of jobs CPUs set to 2, there would be 2 theory Atlas tasks using 0.5 CPUs (sharing one core) and asteroids would use the second if max CPUs was also set to 2. Am I understanding this correctly …?
No - the number of CPUs to be used by an (Atlas, either native or VBox) task is specified by that setting and then passed as a requirement down the chain. So if you set it to 2, the client will have to set aside two cores for it, and will therefore either wait until your Asteroids task has finished leaving the other core idle, or else find another single-core job (e.g. Sixtrack) to do in the meantime which won't finish at the same time and the cycle restarts...

So it's best not to set #CPUs to completely fill the machine, unless you dedicate it to only running Atlas. (In which case you also have to remember to untick the "If no work for selected applications is available, accept work from other applications?" box!)

Thanks for this. Sorry for the late reply.
I've got a few questions before I actually migrate all my hosts over here that I'll throw up in a separate thread. I think I have most of them figured out, but reading through the forums is still leaving a few questions unanswered, in particular the network and bandwidth aspect.
24) Message boards : Number crunching : Max # jobs and Max # CPUs (Message 40969)
Posted 15 Dec 2019 by wolfman1360
Post:
you are already running Theory OK, you have CVMFS installed properly, so try native ATLAS.
And since you have 32 GB of memory, you can set Max # CPUs 1 and use some of it, as it is a little more efficient than using more cores per work unity (which is used if you want to save on memory).

BUT: I have found that to run native ATLAS, you have to grant additional permissions after attaching to LHC and downloading at least one native ATLAS.
Then: Your run "sudo chmod -R 777 /var/lib/boinc-client" and reboot. Then it might work.

So, set number of cores to 1, number of tasks to unlimited for the 32 gb machine. The core 2 duo only has 4 gb, but is also running Linux. Should that be okay with Atlas too? It seems to be crunching away at theory right now. I'll keep that command in mind when I get an Atlas task.[/quote]

And don't even think of upgrading to BOINC 7.16.3, or native ATLAS falls apart.

Everything else is pretty easy beyond that. To run CMS all you need is VirtualBox.
And for SixTrack you don't even need that.

Well. I am unfortunately running Boinc 7.16.3 thanks to costamagnagianfranco/boinc. Is there a quick method to downgrade to an earlier version or am I out of luck?

So to summarize: Number of CPUs only applies to atlas and I should set it to 1 (with 32 gb ram), number of tasks set to unlimited since I want all cores filled in that case. With machines with half the ram I should set an alternative profile with CPU cores set to 2 and jobs again set to unlimited. Should I specifically run Atlas tasks to start with under both since I hear multicore tasks can reek havoc on single core tasks?

I do have Windows hosts, they're just finishing up on another project and then I'll start porting everything over here, though this does beg an interesting question since I can't find it in the forums. I do have a Ryzen 1800x which for some unknown reason does not have virtualization able to be selected in the bios. Does native in Linux allow one not to have that running and still get by, or does it need to be enabled even if you're not using virtual box? On a similar note, do I need to install additional packages under virtual box or is that just to be able to view the current VM status? Under Linux most of my machines are headless so this would be pointless?
25) Message boards : Number crunching : LHC on android (Message 40968)
Posted 15 Dec 2019 by wolfman1360
Post:
2 months late, but does this still only support Android 8 and below?
I'm running Android 9 on one (and 10) on 2 other fairly high end Android devices so hopefully this can get fixed soon.
Really hard to find boinc projects that will run on Android these days. A waste of 24 cores, even if they don't hold a candle to x86.
26) Message boards : Number crunching : Max # jobs and Max # CPUs (Message 40965)
Posted 15 Dec 2019 by wolfman1360
Post:
Hello,
Please let me know if this is the wrong thread. I've been doing some forum searching and just want to make sure I'm understanding this correctly.
I am one of those new users and am struggling to understand, but think I have it figured out.
All projects, apart from Atlas, are single threaded, regardless if they run on Native for Linux or vbox for Windows. This is why max number of CPUs is offered and if it is changed, vms it will use that many cores, regardless if a task from another project was running. For instance, on the single little core 2 duo I have running on here, if there was an Asteroids at home task running, and boinc decided a native theory task should start, with max number of jobs set to 2, there would be 2 theory tasks using 0.5 CPUs (sharing one core) and asteroids would use the second if max CPUs was also set to 2. Am I understanding this correctly or does this also apply to Atlas?

I have a few Haswell machines with 32 GB of ram, an older Sandy bridge i7 with 16, and an fx8350. I take it setting max number of CPU and jobs is dependent on what subprojects I'd like to run? Keep in mind these are dedicated crunchers - 3 out of the 4 are running linux but I'm unsure of how to start off and don't want to overburden myself with work or end up having failed tasks because I didn't configure something correctly.

Any recommendations on a per project basis? I'm still trying to figure out the ins and outs of each - runtimes, resources, etc. i like how it's a little more involved and requires a little more work on the Linux side. I feel like I'm actually contributing to something special.

thanks.
27) Message boards : Theory Application : Taking the plunge, but making sure I understand everything correctly first. (Message 40064)
Posted 2 Oct 2019 by wolfman1360
Post:
Hello.
After seeing https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=5136 I figured I'd give this another shot. My last attempt didn't go too well, but I think it was due to a late hour and several missed steps on my part. Coincidentally, shortly after receiving errors, the machine entered a boot loop after restarting - but only now have I figured out that it just required reseating the ram. On to the questions.

Does this, as well as Atlas, require constant internet connectivity? From some reading, though I'm not sure if I am understanding this correctly, using openhtc.io improves CPU efficiency because of improved latency / ping times?
I've been reading about Squid proxy. Does involve two machines on the same network? Is this needed or is this just for added convenience and less bandwidth? Right now, at least on this network, there will only be this machine.
After installing this, Virtual box is no longer needed in conjunction? If it is, can the entire thing be managed headless, for the most part?
How much disk space am I looking at? The machine has a 120 gb ssd and 16 gb of ram paired with an fx8350. Any specific settings in project preferences I should be paying attention to with this in mind?
I'll be using Boinc tasks to monitor this remotely. I'm assuming that while this is running there can be no other projects alongside?
What bandwidth usage am I looking at?
I do have several Windows hosts. Would WSL (Windows subsystem for Linux) work as a substitute, or is this too complex for Microsoft's integration to handle? I'm sure there are countless limitations and I would just assume that Windows Boinc would simply run the needed CVMFS or other applications, after they were installed of course. Probably just a dream at this point.
Any other tips, tricks, hints or things that I am clearly missing?

I'm super excited to get this going. While some may see it as a negative, I feel like these few extra steps really add something to the eventual crunching. It makes it fun and engaging when things aren't always what they seem and we find new ways to crunch those numbers without the vm errors cropping up. It's like being a part of something not many delve into. And it gives me an excuse to work in a Linux environment.

thank you for any help.[/list][/url]
28) Message boards : Theory Application : Simple Bash script which sets everything up automatically to run native apps (Message 40063)
Posted 2 Oct 2019 by wolfman1360
Post:
I want to give you a great big hug right now.
I am far from a Linux expert. I completely ( and I mean completely) borked up a machine running Linux (Ubuntu 1804) attempting to install this (apart from boinc) all on my own. At this point, I'm failing to remember the specifics, but I do remember quite clearly that I was having issues with CVMFS. I vaguely remember it attempting to replace various libraries having to do with boinc as well as a few other things. Thinking nothing of it, as the machine was running headless and Linux has never given me any issues before, I went ahead with it.
I received several errors after this - again, I don't specifically remember the steps taken here - only that when attempting to reboot the system, it failed to come back up, and physically getting to it later found it to be in an infinit bootloop. I do remember that cvmfs_config probe was getting me nowhere at the time. Said machine is still sitting here, so I will throw Ubuntu 1804 on it again and start from scratch.

Regardless, I look forward to giving this another shot. This is my excuse to delve into Linux and break things. I'm still confused as to what exactly this application does vs. virtualbox. Does this require an alternative version of Boinc to run alongside, or am I missing something completely and utterly obvious? Of course I am probably overcomplicating just about everything. Seems like this also uses a ton of internet traffic, which I have no issues with.
Sorry for the very drawn out post and perhaps silly questions.
thank you again for putting this together and if there is anything I can do to help let me know.
29) Questions and Answers : Windows : Utterly confused in terms of projects to run - some questions (Message 39765)
Posted 30 Aug 2019 by wolfman1360
Post:
https://www.virtualbox.org/wiki/Download_Old_Builds_5_2

The newest of the older versions is VirtualBox 5.2.32 (released July 16 2019)


You need a much newer version of VB to start with and also d/l the Extension Pack that goes with it.

And in your Boinc Preferences if you want to run Atlas then maybe test it set at Max # jobs - 4
Max # CPUs 2
on the location you have that pc set at (Home,School,or Work)


Ah okay, thank you. I will do this. I have one linux box - I will follow the instructions to get theory up and running on there later today as well. I have all projects selected since I am not sure what is most prevalent here. Am I to assume everything that uses VB is multithreaded?
I am assuming max CPUs is max cores? So far this machine has completed a few tasks at the default configuration - 8 CPUS - but I think I will limit it as you suggest so I'm not using 100% CPU and the vm has problems.

From what I gather, theory is Linux specific. Anything else to keep in mind on that front - is VB still required for Linux, too? My linux box is completely headless so not sure how to go about grabbing the latest vb and extension pack using ssh as I'm still a Linux newbie...
30) Questions and Answers : Windows : Utterly confused in terms of projects to run - some questions (Message 39759)
Posted 30 Aug 2019 by wolfman1360
Post:
Hello!
Utterly new here (at least, new now that I am paying attention to requirements) and feeling just slightly overwhelmed, but perhaps I am overcomplicating matters. I did attempt to read the FAQ but was having difficulty trying to understand some of the wording. I did, however, successfully get one task started. I call that a win win in my book!

I have virtual box installed - version 5.2.8.

I figured I should attempt this project on one of my most well equipped machines. a Ryzen pro 1700x with an abundance of ram - 64 GB - using Windows 10 professional and plenty of solid state space on the internal drive. Said Ryzen may be replaced with an i7-7700 or 8700, however, but I digress.

I found requirements for ram and disk usage, but I'm not clear on how that equates to CPU usage. E.g. if a task is multithreaded, as this Atlas task I'm currently crunching apparently is, will that increase ram size the more CPUs (or threads/cores) are assigned to that task?

Which tasks are multithreaded and which are one task per thread/core?
I'm noticing that this task has a fairly decent deadline (I think, though I also think it's going to be well above the 4 plus hours ETA). What kind of runtimes can I expect with each task type?

Most of my machines don't have near this amount of ram and one of them does not allow me to enable virtualization in the bios (don't ask).

Any help/tips are very much appreciated. I am probably overthinking this, but I also don't want to crunch for hours/days only to find that I could have done it more efficiently or optimized it.
thanks!


Previous 20


©2021 CERN