Message boards : Number crunching : VM image ready to crunch
Message board moderation

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
Guiri-One[Andalucia]

Send message
Joined: 1 Feb 06
Posts: 66
Credit: 9,723
RAC: 0
Message 44138 - Posted: 18 Jan 2021, 11:51:36 UTC

Hi team,

I was just wondering: Isn't there an image with all stuff done ready to crunch? I mean, I download it, use Virtualbox and run it, having insise the native apps.

In the end of the day, we are running VM anyway, seems to be "easier", isn't it? At least we would not be loading and unloading 1 image per task

On the other hand, is a million times easier for a non extra experienced user, to just download and load a linux VM and crunch than following the instructions (is quite long...)

Thx¡
Javi
ID: 44138 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2386
Credit: 223,034,233
RAC: 136,753
Message 44140 - Posted: 18 Jan 2021, 12:15:55 UTC - in response to Message 44138.  

The VM images distributed by the project include all you need for a wide range of calculations.
Nonetheless each task requires individual input parameters as well as access to software updates.
All of that is distributed via Frontier and CVMFS, a set of repositories containing many hundreds of TB.

Nobody would be happy if all of that would be send prior to the very 1st task.
ID: 44140 · Report as offensive     Reply Quote
Guiri-One[Andalucia]

Send message
Joined: 1 Feb 06
Posts: 66
Credit: 9,723
RAC: 0
Message 44141 - Posted: 18 Jan 2021, 12:23:11 UTC - in response to Message 44140.  

Then, I misundersttod something: What are those "how to run native apps" forum explanations for then?

Do you know where can I download such image, load in VBox and "forget"?

Thx in advance
ID: 44141 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2386
Credit: 223,034,233
RAC: 136,753
Message 44142 - Posted: 18 Jan 2021, 12:38:19 UTC - in response to Message 44141.  

Do you know where can I download such image, load in VBox and "forget"?

This happens automatically when you get the 1st VBox task (or after a project reset).


What are those "how to run native apps" forum explanations for then?

That's only for users running native apps on Linux instead of VBox apps.
Since you still don't show your computers nor post any valuable link nobody can tell you whether this is relevant for you or not.
ID: 44142 · Report as offensive     Reply Quote
Guiri-One[Andalucia]

Send message
Joined: 1 Feb 06
Posts: 66
Credit: 9,723
RAC: 0
Message 44143 - Posted: 18 Jan 2021, 12:46:07 UTC - in response to Message 44142.  

Hi,

I understand what you mean.

But I was talking about something else: If it is possible to run a native Atlas in a linux environment ..why not running such linux in a VM instad of being running "normal" VM tasks which are downloading images, loading them 1 by one, etc? You turn on your pc, load your linux image with native and let it be. Instead of 1 image per task.

Who cares how am I running now? Question remains the same:)

Why Am I wonring about this?

- Resumes and suspends are not working very well in Vm tasks.
- Time is wasted each time task is over (unload image, load new one...etc).
ID: 44143 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2386
Credit: 223,034,233
RAC: 136,753
Message 44144 - Posted: 18 Jan 2021, 12:56:12 UTC - in response to Message 44143.  

Who cares how am I running now? Question remains the same:)

As an unpayed volunteer I invest my time in answering your questions.
To give you qualified answers you should give qualified information.
If you always complain about that you may expect no further answer.
Instead you may go through the thousands of posts in this forum as they already tell you everything you asked above.
ID: 44144 · Report as offensive     Reply Quote
Guiri-One[Andalucia]

Send message
Joined: 1 Feb 06
Posts: 66
Credit: 9,723
RAC: 0
Message 44145 - Posted: 18 Jan 2021, 13:16:38 UTC - in response to Message 44144.  

This is my host.

https://lhcathome.cern.ch/lhcathome/show_host_detail.php?hostid=10670205

And now, I still have the question: we are just trying to figure out if this is possible (yes, is not only me who has this "doubt")

We are all unpayed crunchers who just want to find out better (and more efficient) ways to use our machines.

In the thousands of posts, there is no "answer" for this topic, if it was that easy we would have found it as we found out how to install and run native apps in a linux system. Now we are thinking about running such linux from VM directly, instead of using time installing all the stuff neeeded in each host...

Thx in advance if anyone has ever fought with this :(
ID: 44145 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2386
Credit: 223,034,233
RAC: 136,753
Message 44149 - Posted: 18 Jan 2021, 21:21:51 UTC - in response to Message 44145.  

The computer behind the link is running Windows.
Hence, there's no way to directly run ATLAS/Theory native on it.
https://lhcathome.cern.ch/lhcathome/show_host_detail.php?hostid=10670205

A possible solution would be to create your own VM and install your preferred Linux on it.
Then install CVMFS and BOINC on that VM and run native apps from there.

In this scenario the CVMFS inside the VM must be configured to use openhtc.io following the description here:
https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=5342


Be aware that this setup is not recommended and will not be supported.
ID: 44149 · Report as offensive     Reply Quote
Guiri-One[Andalucia]

Send message
Joined: 1 Feb 06
Posts: 66
Credit: 9,723
RAC: 0
Message 44150 - Posted: 19 Jan 2021, 8:19:35 UTC - in response to Message 44149.  

Hi,

I didn't want to talk about my host because of that. I know I am on windows but my doubt has nothing to do with my current host.

Nevertheless, I think you pointed the answer: ->"Is not recommended to install a Linux in a VM and run native from there "<- This is thew point

Therefore, if would need to keep using VM as usual.

Thx anyway¡
Javi F
ID: 44150 · Report as offensive     Reply Quote
marmot
Avatar

Send message
Joined: 5 Nov 15
Posts: 144
Credit: 6,301,268
RAC: 0
Message 44167 - Posted: 21 Jan 2021, 12:18:49 UTC - in response to Message 44138.  
Last modified: 21 Jan 2021, 12:20:59 UTC

Hi team,

I was just wondering: Isn't there an image with all stuff done ready to crunch? I mean, I download it, use Virtualbox and run it, having insise the native apps.

In the end of the day, we are running VM anyway, seems to be "easier", isn't it? At least we would not be loading and unloading 1 image per task

On the other hand, is a million times easier for a non extra experienced user, to just download and load a linux VM and crunch than following the instructions (is quite long...)

Thx¡
Javi


This discussion has all gone off track.

It's the same question I and others have had:

'Is there a downloadable, prebuilt, Linux VDI that is ready to run LHC@Home native apps as soon as you create your own VM using that VDI drive and then change the machine name?'

I built an antiX (Debian 9 based) VM that runs on as little as 128MB RAM which has been running various Linus only WU's for BOINC projects on my Windows test laptop.
This VM was going to be setup for LHC WU's as well; but it now has keys download issues and no upgrade path to Debian 10 and so I'm considering developing a new version of my LinuxForBOINC VM.

This seems like a GitHub project waiting to be developed.

Maybe it could be called "BoincedLinuxVM"?

I'm sure many Windows users would love a downloadable Linux VM to run BOINC apps that are Linux only and not have the headaches of managing Linux or having to go through the 10 to 40 hours of creation time when the wheel has already been created by a GitHub team.
It would also save on RAM as native ATLAS only needs a VM to have 2560MB for 4 cores.
ID: 44167 · Report as offensive     Reply Quote
tullio

Send message
Joined: 19 Feb 08
Posts: 708
Credit: 4,336,250
RAC: 0
Message 44169 - Posted: 21 Jan 2021, 14:35:42 UTC

I am running QuChemPedIA@home tasks on a Windows 10 PC using VirtualBox. QuChem is a Linux project and 94% of its users run a native Linux client, most on Debian. Strangely enough, when my PC completes a task, 90% of the times it was faster than its Linux wingman, even if the Linux host has a much more powerful CPU, mostly AMD Ryzen Threadripper.My CPU is an Intel i5 9400F, with 6 processors, that is 3 cores. Some of my Linux wingmen reach 128 processors. I am an old UNIX and Linux user, but at least in this case Windows is faster.
Tullio
ID: 44169 · Report as offensive     Reply Quote
marmot
Avatar

Send message
Joined: 5 Nov 15
Posts: 144
Credit: 6,301,268
RAC: 0
Message 44172 - Posted: 21 Jan 2021, 17:35:17 UTC - in response to Message 44169.  

I am running QuChemPedIA@home tasks on a Windows 10 PC using VirtualBox. QuChem is a Linux project and 94% of its users run a native Linux client, most on Debian. Strangely enough, when my PC completes a task, 90% of the times it was faster than its Linux wingman, even if the Linux host has a much more powerful CPU, mostly AMD Ryzen Threadripper.My CPU is an Intel i5 9400F, with 6 processors, that is 3 cores. Some of my Linux wingmen reach 128 processors. I am an old UNIX and Linux user, but at least in this case Windows is faster.
Tullio

How did you build your Linux VM?
Did you use the Cern product?
ID: 44172 · Report as offensive     Reply Quote
marmot
Avatar

Send message
Joined: 5 Nov 15
Posts: 144
Credit: 6,301,268
RAC: 0
Message 44173 - Posted: 21 Jan 2021, 18:05:17 UTC

Tested my first ATLAS in my antiX VM after increasing it's RAM to 2560 and using BleachBit to purge extraneous files to get enough disk on the meager 16GB VDI ( I need to grow it to 24-32GB).

I forgot to install Singularity but CVMFS is installed and the WU completed by grabbing what it needed of Singularity over the shared fs
The 3 core WU barely used any CPU and the OS showed less than 1300MB usage the entire time.

https://lhcathome.cern.ch/lhcathome/result.php?resultid=296573069


BTW, Microsoft has built a Linux subsystem (WSL) for Windows 10. Some Linux distros are then available from the Microsoft store ( gotta grow that store).
https://subhankarsarkar.com/run-native-linux-on-windows/

I wonder how many users have successfully run native Linux BOINC WU with that system installed?
Probably BOINC would need 2 installs and I don't think BOINC WU reports would be able to discern the Linux app was run on a machine in a WSL subsystem. Maybe WUProps could detect Linux apps run on WSL?
ID: 44173 · Report as offensive     Reply Quote
computezrmle
Volunteer moderator
Volunteer developer
Volunteer tester
Help desk expert
Avatar

Send message
Joined: 15 Jun 08
Posts: 2386
Credit: 223,034,233
RAC: 136,753
Message 44174 - Posted: 21 Jan 2021, 18:28:38 UTC - in response to Message 44173.  

Success?
No, since the task did not create a HITS file.
[2021-01-21 10:38:47] No HITS result produced



In addition the CVMFS client is installed but not configured to use openhtc.io.
This disrespects the project's requirements not to swamp the CVMFS stratum-one-servers.
[2021-01-21 09:33:23] 2.6.0.0 14739 0 66852 77616 3 1 74634 4096001 0 65024 0 0 n/a 525 196 http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch DIRECT 1
[2021-01-21 09:33:23] CVMFS is ok
[2021-01-21 09:33:23] Efficiency of ATLAS tasks can be improved by the following measure(s):
[2021-01-21 09:33:23] The CVMFS client on this computer should be configured to use Cloudflare's openhtc.io.
[2021-01-21 09:33:23] Small home clusters do not require a local http proxy but it is suggested if
[2021-01-21 09:33:23] more than 10 cores throughout the same LAN segment are regularly running ATLAS like tasks.
[2021-01-21 09:33:23] Further information can be found at the LHC@home message board.
ID: 44174 · Report as offensive     Reply Quote
Guiri-One[Andalucia]

Send message
Joined: 1 Feb 06
Posts: 66
Credit: 9,723
RAC: 0
Message 44177 - Posted: 22 Jan 2021, 8:40:52 UTC - in response to Message 44167.  

Morning,

This is exactly what I meant, I didn't want to tlak about what my host is running or not running :)

I understood there is no "official" image built with such conf...maybe is time to spend some effort in such image by the community?
ID: 44177 · Report as offensive     Reply Quote
tullio

Send message
Joined: 19 Feb 08
Posts: 708
Credit: 4,336,250
RAC: 0
Message 44184 - Posted: 23 Jan 2021, 5:12:11 UTC - in response to Message 44172.  
Last modified: 23 Jan 2021, 5:30:31 UTC

I am running QuChemPedIA@home tasks on a Windows 10 PC using VirtualBox. QuChem is a Linux project and 94% of its users run a native Linux client, most on Debian. Strangely enough, when my PC completes a task, 90% of the times it was faster than its Linux wingman, even if the Linux host has a much more powerful CPU, mostly AMD Ryzen Threadripper.My CPU is an Intel i5 9400F, with 6 processors, that is 3 cores. Some of my Linux wingmen reach 128 processors. I am an old UNIX and Linux user, but at least in this case Windows is faster.
Tullio

How did you build your Linux VM?
Did you use the Cern product?

No, i am using the wrapper QuChem provided me. They complain that they don't have a developer and I don't know where they got it. I am 50 in their ranking list of RAC.
Tullio
In stderr.txt the vboxwrapper is indicated as 7.9.26200
I've completed a number of BOINC@TACC tasks on the same Windows 10 PC. They seem to use a docker 4.1.19-boot2 docker. Frankly, I don't know what it is.
ID: 44184 · Report as offensive     Reply Quote
marmot
Avatar

Send message
Joined: 5 Nov 15
Posts: 144
Credit: 6,301,268
RAC: 0
Message 44191 - Posted: 24 Jan 2021, 20:53:33 UTC - in response to Message 44174.  
Last modified: 24 Jan 2021, 21:46:22 UTC

Success?
No, since the task did not create a HITS file.
[2021-01-21 10:38:47] No HITS result produced


According to the WU database; the WU is valid and received credit.

That is the only criteria a user is to judge the success or failure of BOINC WU's, and it has been this way for decades.

If no HITS file created is an error state; then do not give credit and mark the WU as one ending in an error.


In addition the CVMFS client is installed but not configured to use openhtc.io.
This disrespects the project's requirements not to swamp the CVMFS stratum-one-servers.
[2021-01-21 09:33:23] 2.6.0.0 14739 0 66852 77616 3 1 74634 4096001 0 65024 0 0 n/a 525 196 http://cvmfs-s1fnal.opensciencegrid.org/cvmfs/atlas.cern.ch DIRECT 1
[2021-01-21 09:33:23] CVMFS is ok
[2021-01-21 09:33:23] Efficiency of ATLAS tasks can be improved by the following measure(s):
[2021-01-21 09:33:23] The CVMFS client on this computer should be configured to use Cloudflare's openhtc.io.
[2021-01-21 09:33:23] Small home clusters do not require a local http proxy but it is suggested if
[2021-01-21 09:33:23] more than 10 cores throughout the same LAN segment are regularly running ATLAS like tasks.
[2021-01-21 09:33:23] Further information can be found at the LHC@home message board.


Again, the WU succeeded and reports valid.
If the HITS file is not being created properly then that is an issue that needs correcting.
Cloudflare is 3rd party and was never necessary for performing BOINC work.

This VM is a work in progress and correcting the keys error file; getting updates and then installing Singularity, then figuring out why the HITS file is not being created (might be corrected once Singularity is installed) take precedence over Cloudflare connection (which I do not even like to use. It is a for-profit company that does not respect data privacy. It appears they have improved their privacy regulations and so I will need to change my opinion here. I need a few days to read more about their changes. It takes time to change your emotions towards a company that you held personal animosity towards.)
ID: 44191 · Report as offensive     Reply Quote
marmot
Avatar

Send message
Joined: 5 Nov 15
Posts: 144
Credit: 6,301,268
RAC: 0
Message 44192 - Posted: 24 Jan 2021, 20:56:06 UTC - in response to Message 44184.  

I am running QuChemPedIA@home tasks on a Windows 10 PC using VirtualBox. QuChem is a Linux project and 94% of its users run a native Linux client, most on Debian. Strangely enough, when my PC completes a task, 90% of the times it was faster than its Linux wingman, even if the Linux host has a much more powerful CPU, mostly AMD Ryzen Threadripper.My CPU is an Intel i5 9400F, with 6 processors, that is 3 cores. Some of my Linux wingmen reach 128 processors. I am an old UNIX and Linux user, but at least in this case Windows is faster.
Tullio

How did you build your Linux VM?
Did you use the Cern product?

No, i am using the wrapper QuChem provided me. They complain that they don't have a developer and I don't know where they got it. I am 50 in their ranking list of RAC.
Tullio
In stderr.txt the vboxwrapper is indicated as 7.9.26200
I've completed a number of BOINC@TACC tasks on the same Windows 10 PC. They seem to use a docker 4.1.19-boot2 docker. Frankly, I don't know what it is.


Same BOINC turnkey solution that Kryptos@Home used to get their project up and sending work quickly.
Here's their GitHub page:
https://github.com/boot2docker/boot2dockerz
ID: 44192 · Report as offensive     Reply Quote
marmot
Avatar

Send message
Joined: 5 Nov 15
Posts: 144
Credit: 6,301,268
RAC: 0
Message 44193 - Posted: 24 Jan 2021, 21:01:37 UTC - in response to Message 44177.  

Morning,

This is exactly what I meant, I didn't want to tlak about what my host is running or not running :)

I understood there is no "official" image built with such conf...maybe is time to spend some effort in such image by the community?



I'm working on my 1st version with antiX but am looking at Puppy's newest Ubuntu version as antiX broke the upgrade path from Debian 9 to 10. Will need a full rebuild. I have so many labor hours in this antiX 17.2 VM for BOINC now that I'll finish it and put it into production before developing another VM.

I'll check into Puppy's upgrade path continuity.

Thanks for bringing up the topic and hopefully some other people help with a BOINCix project.
I still think it will be more useful for managing workloads over a native Linux install on Windows 10 WSL subsystem.
ID: 44193 · Report as offensive     Reply Quote
tullio

Send message
Joined: 19 Feb 08
Posts: 708
Credit: 4,336,250
RAC: 0
Message 44195 - Posted: 25 Jan 2021, 6:35:26 UTC

I've seen that both QuChemPedIA@home and BOINC@TACC use the same vboxwrapper 26200, while LHC@home Atlas uses 26197.
Tullio
ID: 44195 · Report as offensive     Reply Quote
1 · 2 · Next

Message boards : Number crunching : VM image ready to crunch


©2024 CERN