Old data
7 Sep 2011 7:53:54 UTC
Topic 195948
(moderation:
Hi there,
I haven't had a task in almost a week, but boinc still shows a lot of data for this project. I did a project reset, which resulted in a decrease from about 420 to 320 MB of Einstein data. What is this?
Old data
)
Even though you may not have had work from this project --for whatever reason, be it debt being paid back to other projects, or you've set this project to No new tasks, or your computer isn't on long enough to send work at this time-- the data of the main task(s) you were working on stays on your computer, until those are done, plus the science applications stay on your computer.
The main thing is the locality scheduling. This works by sending you one main data file, which is then sliced up on your computer whenever you ask for and get work. By sending you this main file as a whole, both you and the project use up a lot less bandwidth.
However, when you reset the project, normally all data on your system gets thrown away and you have nothing of the project on disk until it contacts that project and it sends back the science applications. It doesn't work like that on this project as it has the option "Resend lost work" on, on the server. So it will then check against the database what main data files should be on your system and send that/those back to you. Meaning that in the end you still have those main data files sitting on your system. You also still have the debt to work with, albeit that you may be appointed work now quicker, as the debt for this project was also reset. Still, that depends on a lot of other factors.
Thanks for the reply. Sounds
)
Thanks for the reply. Sounds pretty complicated...
Well, I detached and reattached and now I only have 120MB with one task in line. That seems ok.
RE: Thanks for the reply.
)
Well, yes, it is complicated for those who haven't had locality scheduling explained to them. I think E@H is pretty much unique in making full use of this BOINC feature for handling Gravitational Wave tasks.
Most projects supply tasks as a single packet of data which gets analysed by your computer and then gets thrown away when the result is successfully returned. With E@H GW tasks, the task itself is extremely small and contains no data. It simply contains a series of parameters that are fed into the science app to allow it to crunch a fixed set of data files which are separately sent to your host.
As an example, consider this task which was sent to you on 30th August and which your host successfully returned on 6th September. Its name was h1_0400.70_S6GC1__2272_S6BucketA_1. The 400.70 part of the name is a frequency in Hz and the 2272 is a sequence number to keep track of tasks that require exactly the same data. That sequence number will count down to zero so you can see that there are potentially thousands of tasks that require exactly the same data. By resetting the project or detaching and reattaching, you will have thrown away that data and your next task (which you now have) will be for a different frequency and your host will have needed to download a whole new set of data files just to crunch it. If you hadn't reset, your next task would most likely have been a lower sequence number for the 0400.70Hz frequency and you wouldn't have had to download a single byte of data.
To crunch a single task, say the one whose name i listed, you would need at least 8 to 10 large data files that would have names different to the above but which would include the precise frequency of 0400.70Hz plus a number of related and slightly higher frequencies like 0400.75, 0400.80, 0400.85, ...., etc. There would be two data files for each particular frequency, one starting with h1_ and one starting with l1_ standing for the Hanford and Livingstone observatories where the data was originally accumulated.
As you could appreciate, the volume of data per task would simply be prohibitive if it had to be sent completely, every time a single task was issued. Locality scheduling is the solution to the problem. Issue the data once and then issue potentially thousands of tasks that rely on exactly that same data payload. Sure, you need to keep that data on your computer, but that is a small price to pay for the saving in download bandwidth. Without locality scheduling, the GW project simply could not operate.
OK for the moment. If you take a look at your new task, you will see it is for the 0421.15Hz frequency. Its sequence number is 2509 but if you look at the end of its name you will see an _2 suffix. For validation purposes, two identical tasks are issued to two separate computers. These tasks have suffixes of _0 and _1. If one or more of the original tasks fails, resends will be generated which will have higher integer suffixes. Your new task is a resend task. There is no particular problem with that. When your new task is finished, there may not be further tasks for that same frequency so you might get another frequency change requiring further large data files to be downloaded. Your best bet is to let BOINC manage this and accept that you may eventually need to have a few hundred MB of data living for a time on your machine. Those files will eventually be deleted when no potential tasks for those frequencies remain. This happens automatically when your host contacts the project and the project decides that the files are no longer needed.
Hopefully you can understand a bit more about locality scheduling. Please feel free to ask questions if anything is unclear. The other part of your original message pointed out that there were times when you had no tasks for E@H. This is quite normal when you are supporting quite a number of projects simultaneously. You will probably always find that a few projects of that larger number are temporarily missing from the list of tasks on board. BOINC tries to prevent work from becoming 'stale' by delaying new tasks until they are really needed. A lot depends on how you have your resource shares allocated and what you have chosen as a work cache size. You could possibly get more projects to supply a task with a larger cache size but it would be more difficult for BOINC to manage things and you'd probably see BOINC dropping into 'panic mode' just to get 'at risk' tasks completed in time. It's far better to avoid that by keeping the cache size relatively small and not always having every project represented in the tasks list all the time.
Cheers,
Gary.