Please make work units self-contained


Advanced search

Message boards : Number crunching : Please make work units self-contained

Author Message
Jesse Viviano
Send message
Joined: 20 Dec 15
Posts: 16
Credit: 328,453
RAC: 0
    
Message 5793 - Posted: 9 Dec 2016, 0:21:45 UTC

Please make work units self-contained. I do not like computation errors caused by unreliable internet connections, like Wi-Fi connections that are wrecked by microwave ovens or laptop users who are not always connected to the Internet. All of the data that should be required to run a work unit should be contained within the work unit. I have had several work units become computation errors due to the fact that they cannot get the data they need due to a microwave oven that temporarily ruins my 2.4 GHz Wi-Fi connection when it is running. Unfortunately, I am in a situation that prevents me from using a wired Ethernet connection, and the walls between the internet gateway and my Wi-Fi card absorb the 5.8 GHz Wi-Fi so well that it is unusably slow.

David Cameron
Project administrator
Project developer
Project tester
Project scientist
Send message
Joined: 13 May 14
Posts: 252
Credit: 2,028,556
RAC: 1
    
Message 5811 - Posted: 12 Dec 2016, 11:17:10 UTC - in response to Message 5793.

While the ATLAS WU run, they regularly contact external database services to query information on the ATLAS detector setup. The information that each WU requests can be different so that's why it's not as simple as cacheing it inside the VM image. We do sometimes run ATLAS simulation in environments where there is no network connectivity but then we have to make sure all possible database information is available locally to the task, which we do by putting it all into a massive (~7GB) file and storing this file locally. If we did this for ATLAS@Home it would make the image way too large so unfortunately we prefer to rely on internet connectivity. Also, if your microwave is leaking enough to affect your wifi I would be a little worried....

Nick Name
Send message
Joined: 22 Jun 14
Posts: 13
Credit: 135,078
RAC: 0
    
Message 5818 - Posted: 14 Dec 2016, 3:21:59 UTC

Magical microwaves aside, there are too many links in the ATLAS chain.

BOINC host requirements:
VM capable hardware
ATLAS capable hardware
VirtualBox
Adequate bandwidth and a stable internet connection

ATLAS requirements:
Stable servers (all over the world!) with adequate bandwidth and connectivity for data to be downloaded during the run
Stable database - or maybe databases?
Good work units

I'm sure I overlooked something. A breakdown at any one of these points, only some of which the BOINC user can control means the work is wasted.

By the way, how many of those connections around the world should be expected? I've seen at least 21 separate addresses so far.
____________
Team USA form

Toby Broom
Send message
Joined: 1 Jul 14
Posts: 70
Credit: 12,032,688
RAC: 0
    
Message 5819 - Posted: 14 Dec 2016, 9:39:27 UTC

The work from sixtrack is self contained.

https://lhcathome.cern.ch/lhcathome/


I imagine from the side of Atlas team they wanted something easy for there situation so they pick the networked VM.

nothing is perfect :)

David Cameron
Project administrator
Project developer
Project tester
Project scientist
Send message
Joined: 13 May 14
Posts: 252
Credit: 2,028,556
RAC: 1
    
Message 5825 - Posted: 15 Dec 2016, 8:16:00 UTC - in response to Message 5819.

Believe it or not, the tasks running on ATLAS@Home are the simplest type of task in ATLAS :) Most other kinds of workflows require much more network access, input/output data or memory, and unfortunately we don't really have anything which is self-contained or lightweight. This is due to the fact that the ATLAS detector is an extremely complicated machine which requires very complex software and infrastructure around it. I hope that you all enjoy the challenge of this tough project :)

Toby Broom
Send message
Joined: 1 Jul 14
Posts: 70
Credit: 12,032,688
RAC: 0
    
Message 5826 - Posted: 15 Dec 2016, 8:20:05 UTC - in response to Message 5825.

I enjoy that I can contribute to the valuable science, I enjoy working out how to make my computer run well for the project.

I'd be happy to take the complex work :)

Message boards : Number crunching : Please make work units self-contained