- We cannot let the participant have a setting that would allow them to say that BOINC should run tasks to completion as a default, but we can use the switch time to force the issue; even though this is worse because switch time is used in the calculations for work fetch and CPU scheduling. And when you use 12 hours as your switch time you seriously bias these calculations.
Wait a second, that is not strictly correct. The TSI does not have anything to do with calculating the size of work fetches, other than it sets the LTD trigger point where they are allowed. So if anything, a larger TSI will allow the host to fetch sooner when it is overworked.
Actually, the bias seems to be to reduce the size of the cache and to force many tasks into High Priority Mode when they do not need to be based on the run time of the task and the deadline.
If needs be I can get the actual snippet of John's comment. But, my point was based on my Felix Unger monitoring of my work queue as it flexes about based on what the system has lined up to work on.
Hmmm...
OK, no argument there on the queue 'flex'. I see that even with 5.10.x. So I'm going to assume you're talking about a late model 6x CC. If so the change to the way LTD works would certainly explain what you're seeing. Having the max at zero and everything else is negative is really bad idea, for anumber of reasons, all of which John pointed out, and was overruled on. ;-)
Bottom line for me is there isn't anything I need in 6x, even for Vista, and what has been/changed added has only served to make it even less attractive than was initially. :-(
- We cannot let the participant have a setting that would allow them to say that BOINC should run tasks to completion as a default, but we can use the switch time to force the issue; even though this is worse because switch time is used in the calculations for work fetch and CPU scheduling. And when you use 12 hours as your switch time you seriously bias these calculations.
Ask projects to stop adding checkpointing to their applications, then the task will run from start to finish. Of course the downside is that when you reboot, or otherwise exit and restart BOINC, you have to start over.
Using a wrapper will also run the task from start to finish. At least it still checkpoints, but unknown to BOINC.
When the devs say this or that is a bad idea they quite often mean "we just don't have time to think of all the implications of coding that functionality, how it's going to affect all the other parts, etc."
Gerry, the fact that there are tasks sitting in your cache that were preempted with only 15 secs to go is insignificant in the grand scheme of things. The only adverse affect it has is that it bothers pedantic little neat freaks, the Felix Ungers of the world. The good thing about it is that watching the Felixes of the world moan and pine about nothing supplys comedic relief for the well adjusted. If they ever revive the Odd Couple serial this thread could surely inspire a storyline for 1 episode.
As I said in another thread... 200 chiefs thinking of code for 2 or 3 Indians to write.
My point has been and always will be to move the ball forward for the science or the development of BOINC. I can make that claim for probably every post I've ever written. I don't know who or when the 15 seconds thing ever came up. Indeed, with a pair a quad cores in my quiver, my caches often have WUs with many hours to completion, even above ten hours or more, not just minutes, so this is not that level of nit-picking.
My point in my first post to this thread was to discuss the merits of what I think is a fair idea (an idea that I still believe is a good one, by the way), although even from the start I recognized that statistically there is really no difference. It is only a matter of preference. Good idea or not, discussions of ideas to improve BOINC wherever we can as a community should not be discouraged.
- We cannot let the participant have a setting that would allow them to say that BOINC should run tasks to completion as a default, but we can use the switch time to force the issue; even though this is worse because switch time is used in the calculations for work fetch and CPU scheduling. And when you use 12 hours as your switch time you seriously bias these calculations.
Ask projects to stop adding checkpointing to their applications, then the task will run from start to finish. Of course the downside is that when you reboot, or otherwise exit and restart BOINC, you have to start over.
You could fix that by putting in a checkpoint that only runs when you exit the program, or the pc reboots. Yes I know sometimes there is no warning when the pc crashes, but sometimes there is. If Boinc could recognize those, then checkpointing could be eliminated from some projects. Projects with extremely long run times, IMO, should keep checkpointing and allow switching just so the User feels Boinc is doing what he wants, crunching for multiple projects.
Another thing they could do is allow a different project on each cpu, set by the User. ie I choose to set project A run on cpu 0 and project B to run on cpu 1. This could be done thru Boinc with little programming, I think, as it can be done by the User now. You just have to redo it after a restart and again after each workunit starts. That might solve some of the problems too, forcing project A to only use cpu 0 would not let project B swap projects A's workunit out so it can take over. Letting project A's workunit finish to completion. Now those with fewer cpu's than projects would still be swapping. But you could, in this idea, force one project to only use cpu 0 and all others to use cpu 1, for example.
Quote:
Using a wrapper will also run the task from start to finish. At least it still checkpoints, but unknown to BOINC.
Using a wrapper works fine but often gives trouble for micro-managers because ti just doesn't keep up with the stats. Meaning it doesn't show the time to completion or progress thru the workunit properly.
When the devs say this or that is a bad idea they quite often mean "we just don't have time to think of all the implications of coding that functionality, how it's going to affect all the other parts, etc."
Gerry, the fact that there are tasks sitting in your cache that were preempted with only 15 secs to go is insignificant in the grand scheme of things. The only adverse affect it has is that it bothers pedantic little neat freaks, the Felix Ungers of the world. The good thing about it is that watching the Felixes of the world moan and pine about nothing supplys comedic relief for the well adjusted. If they ever revive the Odd Couple serial this thread could surely inspire a storyline for 1 episode.
As I said in another thread... 200 chiefs thinking of code for 2 or 3 Indians to write.
My point has been and always will be to move the ball forward for the science or the development of BOINC. I can make that claim for probably every post I've ever written. I don't know who or when the 15 seconds thing ever came up. Indeed, with a pair a quad cores in my quiver, my caches often have WUs with many hours to completion, even above ten hours or more, not just minutes, so this is not that level of nit-picking.
The 15 seconds thing is my fault and my mistake. In the past there have been several requests to have BOINC ignore the "switch between tasks about every" setting if a task needs only a few more minutes/seconds to complete. I mistakenly thought that is what you were asking for and I apologize for not reading your post more carefully.
Now I see that what you want is for BOINC to run tasks from start to finish rather than preempt tasks. I can't think of a reason why that's not a good idea and perhaps that is how BOINC should have been built from the beginning.
Quote:
My point in my first post to this thread was to discuss the merits of what I think is a fair idea (an idea that I still believe is a good one, by the way), although even from the start I recognized that statistically there is really no difference. It is only a matter of preference.
Having one's own preferences catered to is a luxury. The devs are short on time and manpower. With the scheduler working as badly as it has since 6.4.5, is this the best time to request luxuries?
On the other hand, maybe the preempting is the luxury we don't really need? Maybe getting rid of it would simplify the problem of scheduling tasks and help the devs get the scheduler working again? OK, maybe it isn't really broken but it doesn't seem to be handling GPU tasks very well.
Quote:
Good idea or not, discussions of ideas to improve BOINC wherever we can as a community should not be discouraged.
Agreed though I hope we can discuss more than just ideas to improve the software. The community and what projects do in the community wants inspection and discussion as well. A lot of people don't want the latter.
Actually I am running mostly 6.5.0 to take advantage of the GPU capability that it has. For the Mac Pro I have to use 5.10.45 because many projects have still not made the needed change to recognize the modified CPU ID string. The last project that I ran into with this issue is RCN ... but have also seen it with Sztaki and CPDN ...
But, the queue issue has been with us for a long time and is more prevalent with multi-CPU systems and has to do with the way that the model works in the CPU scheduler and work fetch. In simplistic terms it adds up the hours of work and divides it out with the assumption that there is really only one resource. Instead it ought to be laying out the tasks in "lanes" so to speak so that these long tasks (like from CPDN, Sztaki, Lattice, Orbit, etc.) are allocated to one CPU and then tests made to see if the other CPUs have enough work.
A simplistic example might be that I have one task form each of the projects listed above. On a 8 CPU system these tasks would occupy one CPU each leaving 4 with nothing to do. Yet a simple addition of the number of hours of work would seem to indicate that there is sufficient to keep the system occupied. So, you see that the system "runs down" the queue because it thinks that it has sufficient work based on this simplistic addition of the totality of work. When the number of hours has been worked down, then BOINC suddenly realizes that it does not have enough work and orders up some more ...
In my case I like to run "lean" because that is what the designers had in mind as conveyed in their many posts about the subject. But, this does mean that I do run risks of running out of work at times.
@Gerry,
In my case it came up because I had one task that indeed had 15 seconds to run to completion. I also had another that had 0 seconds to run, 100% done and was in "Waiting to run" status when I made my comment. Even with my settings, it is not enough to "Force" BOINC to act as I would desire to stay "clean" and to keep as few models at risk as possible. In recent years BOINC has improved and my loss of models is low. Yet, I have 3 models for Tanpaku on one system, and 8-10 for Reisel Seive on two others. Now, should I "purge" those models? Did the projects get the science? If I hang onto these will my sending them in "late" retain the science?
Again, in the grand scheme of things these are small questions. But, this is supposed to be about the science and processing work and ensuring that the project scientists have the fruits of our labor. By running lean, running models to completion as soon as possible and reporting the results this is accomplished in the most efficient manner. By having models seconds from completion after hours of processing puts all that labor at risk for a system crash and all the vagaries of nature.
@Ageless
Yes, by removing essential components of BOINC we could reduce its effectiveness and utility. Or, we could continue to give people tools to manage their systems without unduly biasing internal operations. I tired to make my point that I can almost get the effect I desire with the change in setting. But, in making that setting I actually adversely affect internal calculations. If I had the capability to "instruct" the system on a project by project basis that i would strongly prefer that the tasks from specific projects should be run to completion that the system would be more effective.
The sole objection to the change is that it would allow "micromanagement" which is a straw-man answer because many changes to this point allow if not encourage the very micromanagement so despised in this case. For example you cannot say that allowing me to "flag" Einstein as a project I would like to have their tasks run to completion is adding a feature that allows micromanagement and then in the next breath tell me that setting the "switch interval" high allows me to achieve that very end ... If getting to my goal by abusing one setting is Ok, then why isn't having a specific setting evil?
In my case it came up because I had one task that indeed had 15 seconds to run to completion. I also had another that had 0 seconds to run, 100% done and was in "Waiting to run" status when I made my comment. Even with my settings, it is not enough to "Force" BOINC to act as I would desire to stay "clean" and to keep as few models at risk as possible. In recent years BOINC has improved and my loss of models is low.
Again, in the grand scheme of things these are small questions. But, this is supposed to be about the science and processing work and ensuring that the project scientists have the fruits of our labor. By running lean, running models to completion as soon as possible and reporting the results this is accomplished in the most efficient manner. By having models seconds from completion after hours of processing puts all that labor at risk for a system crash and all the vagaries of nature.
I very much agree with this. I have sometimes wondered, as others have, although I have not followed these lines of discussion as much, if there could be a TTC override on the switching of tasks, meaning that if there are less than, say, five minutes to completion, BOINC would finish the WU before switching tasks. I would think this would be simple to program for BOINC, but then again, I am no programmer. I call it the "finish the puppy" priority override tag. I think my idea as described/clarified below overlaps with your idea, perhaps there is much to having both ideas incorporated.
@Dagorath,
Quote:
Now I see that what you want is for BOINC to run tasks from start to finish rather than preempt tasks. I can't think of a reason why that's not a good idea and perhaps that is how BOINC should have been built from the beginning.
Actually no, believe it or not. My idea is far simpler than what others have discussed that are related to this but not the same. My idea is for BOINC to give priority to the task with the most CPU time completed when switching tasks, not completing the task outright; that is an entirely different matter. So, for example, BOINC will tell an Einstein WU with five hours completed to continue its run first according to user preferences, so that if the WU completes, BOINC will then start a new WU from Einstein. If BOINC tells Einstien to run a WU that still does not complete, it will, again, start that Einstein WU first until it completes. The point being that priority is given to a WU with the most CPU time completed. If that one completes, then BOINC will then start a new WU. If a WU is already running for Einstein, and BOINC decides it wants to run a second Einstein WU simultaneously on a multi-core system, then BOINC will, again, look for the next WU with the most time completed from that project, and continue that WU first. If there are no Einstein WUs with CPU time completed, BOINC will then start a new WU from that project.
In other words, push what is already started out the door first before beginning a new task without violating user preferences. That's it! Telling BOINC to complete the task before switching causes its own problems, especially with WUs over five or more hours. This in my view violates the preferences set by the user. Paul's idea of "Lanes" as he decribes them would work well with my idea I think, especially with the longer run WUs. For shorter run WUs, I think my idea would get the work out the door efficiently without violating user preferences, which in all cases need to be respected by BOINC.
Now I see that what you want is for BOINC to run tasks from start to finish rather than preempt tasks. I can't think of a reason why that's not a good idea and perhaps that is how BOINC should have been built from the beginning.
...
On the other hand, maybe the preempting is the luxury we don't really need?
Think about projects with very long running WUs. For examples, I've run CPDN tasks that took several CPU months (!!!) to complete. Running one of those monsters exclusively from start to finish would have shut me out from all other BOINC projects for half a year.
Now I see that what you want is for BOINC to run tasks from start to finish rather than preempt tasks. I can't think of a reason why that's not a good idea and perhaps that is how BOINC should have been built from the beginning.
...
On the other hand, maybe the preempting is the luxury we don't really need?
Think about projects with very long running WUs. For examples, I've run CPDN tasks that took several CPU months (!!!) to complete. Running one of those monsters exclusively from start to finish would have shut me out from all other BOINC projects for half a year.
CU
Bikeman
It would shut you out of other projects if run on a single core but not on a multi-core. But assume it's on a single core.... over the longterm it doesn't matter if one doesn't crunch other projects for 6 months because after the CPDN task is done you won't crunch CPDN for a long while. In the end it all balances out. Isn't the task and project switching just a show for the user to instil the faith that over the short term his shares are being respected? Is the show really necessary?
@Gerry,
Thanks for additional details. OK you're not proposing to run until finished but now it sounds like what you want is what we already have. Maybe you're seeing tasks with 2 hours time being run when tasks with 4 hours are waiting because the task with 4 hours was running on 1 core when the other core(s) switched?
RE: RE: RE: - We
)
Hmmm...
OK, no argument there on the queue 'flex'. I see that even with 5.10.x. So I'm going to assume you're talking about a late model 6x CC. If so the change to the way LTD works would certainly explain what you're seeing. Having the max at zero and everything else is negative is really bad idea, for anumber of reasons, all of which John pointed out, and was overruled on. ;-)
Bottom line for me is there isn't anything I need in 6x, even for Vista, and what has been/changed added has only served to make it even less attractive than was initially. :-(
Alinator
RE: Anyway I wonder whether
)
Ask Kathryn to move it to the BOINC Dev forums. ;-)
RE: - We cannot let the
)
Ask projects to stop adding checkpointing to their applications, then the task will run from start to finish. Of course the downside is that when you reboot, or otherwise exit and restart BOINC, you have to start over.
Using a wrapper will also run the task from start to finish. At least it still checkpoints, but unknown to BOINC.
RE: When the devs say this
)
My point has been and always will be to move the ball forward for the science or the development of BOINC. I can make that claim for probably every post I've ever written. I don't know who or when the 15 seconds thing ever came up. Indeed, with a pair a quad cores in my quiver, my caches often have WUs with many hours to completion, even above ten hours or more, not just minutes, so this is not that level of nit-picking.
My point in my first post to this thread was to discuss the merits of what I think is a fair idea (an idea that I still believe is a good one, by the way), although even from the start I recognized that statistically there is really no difference. It is only a matter of preference. Good idea or not, discussions of ideas to improve BOINC wherever we can as a community should not be discouraged.
(Click for detailed stats)
RE: RE: - We cannot let
)
You could fix that by putting in a checkpoint that only runs when you exit the program, or the pc reboots. Yes I know sometimes there is no warning when the pc crashes, but sometimes there is. If Boinc could recognize those, then checkpointing could be eliminated from some projects. Projects with extremely long run times, IMO, should keep checkpointing and allow switching just so the User feels Boinc is doing what he wants, crunching for multiple projects.
Another thing they could do is allow a different project on each cpu, set by the User. ie I choose to set project A run on cpu 0 and project B to run on cpu 1. This could be done thru Boinc with little programming, I think, as it can be done by the User now. You just have to redo it after a restart and again after each workunit starts. That might solve some of the problems too, forcing project A to only use cpu 0 would not let project B swap projects A's workunit out so it can take over. Letting project A's workunit finish to completion. Now those with fewer cpu's than projects would still be swapping. But you could, in this idea, force one project to only use cpu 0 and all others to use cpu 1, for example.
Using a wrapper works fine but often gives trouble for micro-managers because ti just doesn't keep up with the stats. Meaning it doesn't show the time to completion or progress thru the workunit properly.
RE: RE: When the devs say
)
The 15 seconds thing is my fault and my mistake. In the past there have been several requests to have BOINC ignore the "switch between tasks about every" setting if a task needs only a few more minutes/seconds to complete. I mistakenly thought that is what you were asking for and I apologize for not reading your post more carefully.
Now I see that what you want is for BOINC to run tasks from start to finish rather than preempt tasks. I can't think of a reason why that's not a good idea and perhaps that is how BOINC should have been built from the beginning.
Having one's own preferences catered to is a luxury. The devs are short on time and manpower. With the scheduler working as badly as it has since 6.4.5, is this the best time to request luxuries?
On the other hand, maybe the preempting is the luxury we don't really need? Maybe getting rid of it would simplify the problem of scheduling tasks and help the devs get the scheduler working again? OK, maybe it isn't really broken but it doesn't seem to be handling GPU tasks very well.
Agreed though I hope we can discuss more than just ideas to improve the software. The community and what projects do in the community wants inspection and discussion as well. A lot of people don't want the latter.
BOINC FAQ Service
Official BOINC wiki
Installing BOINC on Linux
@Alinator Actually I am
)
@Alinator
Actually I am running mostly 6.5.0 to take advantage of the GPU capability that it has. For the Mac Pro I have to use 5.10.45 because many projects have still not made the needed change to recognize the modified CPU ID string. The last project that I ran into with this issue is RCN ... but have also seen it with Sztaki and CPDN ...
But, the queue issue has been with us for a long time and is more prevalent with multi-CPU systems and has to do with the way that the model works in the CPU scheduler and work fetch. In simplistic terms it adds up the hours of work and divides it out with the assumption that there is really only one resource. Instead it ought to be laying out the tasks in "lanes" so to speak so that these long tasks (like from CPDN, Sztaki, Lattice, Orbit, etc.) are allocated to one CPU and then tests made to see if the other CPUs have enough work.
A simplistic example might be that I have one task form each of the projects listed above. On a 8 CPU system these tasks would occupy one CPU each leaving 4 with nothing to do. Yet a simple addition of the number of hours of work would seem to indicate that there is sufficient to keep the system occupied. So, you see that the system "runs down" the queue because it thinks that it has sufficient work based on this simplistic addition of the totality of work. When the number of hours has been worked down, then BOINC suddenly realizes that it does not have enough work and orders up some more ...
In my case I like to run "lean" because that is what the designers had in mind as conveyed in their many posts about the subject. But, this does mean that I do run risks of running out of work at times.
@Gerry,
In my case it came up because I had one task that indeed had 15 seconds to run to completion. I also had another that had 0 seconds to run, 100% done and was in "Waiting to run" status when I made my comment. Even with my settings, it is not enough to "Force" BOINC to act as I would desire to stay "clean" and to keep as few models at risk as possible. In recent years BOINC has improved and my loss of models is low. Yet, I have 3 models for Tanpaku on one system, and 8-10 for Reisel Seive on two others. Now, should I "purge" those models? Did the projects get the science? If I hang onto these will my sending them in "late" retain the science?
Again, in the grand scheme of things these are small questions. But, this is supposed to be about the science and processing work and ensuring that the project scientists have the fruits of our labor. By running lean, running models to completion as soon as possible and reporting the results this is accomplished in the most efficient manner. By having models seconds from completion after hours of processing puts all that labor at risk for a system crash and all the vagaries of nature.
@Ageless
Yes, by removing essential components of BOINC we could reduce its effectiveness and utility. Or, we could continue to give people tools to manage their systems without unduly biasing internal operations. I tired to make my point that I can almost get the effect I desire with the change in setting. But, in making that setting I actually adversely affect internal calculations. If I had the capability to "instruct" the system on a project by project basis that i would strongly prefer that the tasks from specific projects should be run to completion that the system would be more effective.
The sole objection to the change is that it would allow "micromanagement" which is a straw-man answer because many changes to this point allow if not encourage the very micromanagement so despised in this case. For example you cannot say that allowing me to "flag" Einstein as a project I would like to have their tasks run to completion is adding a feature that allows micromanagement and then in the next breath tell me that setting the "switch interval" high allows me to achieve that very end ... If getting to my goal by abusing one setting is Ok, then why isn't having a specific setting evil?
RE: @Gerry, In my case it
)
I very much agree with this. I have sometimes wondered, as others have, although I have not followed these lines of discussion as much, if there could be a TTC override on the switching of tasks, meaning that if there are less than, say, five minutes to completion, BOINC would finish the WU before switching tasks. I would think this would be simple to program for BOINC, but then again, I am no programmer. I call it the "finish the puppy" priority override tag. I think my idea as described/clarified below overlaps with your idea, perhaps there is much to having both ideas incorporated.
@Dagorath,
Actually no, believe it or not. My idea is far simpler than what others have discussed that are related to this but not the same. My idea is for BOINC to give priority to the task with the most CPU time completed when switching tasks, not completing the task outright; that is an entirely different matter. So, for example, BOINC will tell an Einstein WU with five hours completed to continue its run first according to user preferences, so that if the WU completes, BOINC will then start a new WU from Einstein. If BOINC tells Einstien to run a WU that still does not complete, it will, again, start that Einstein WU first until it completes. The point being that priority is given to a WU with the most CPU time completed. If that one completes, then BOINC will then start a new WU. If a WU is already running for Einstein, and BOINC decides it wants to run a second Einstein WU simultaneously on a multi-core system, then BOINC will, again, look for the next WU with the most time completed from that project, and continue that WU first. If there are no Einstein WUs with CPU time completed, BOINC will then start a new WU from that project.
In other words, push what is already started out the door first before beginning a new task without violating user preferences. That's it! Telling BOINC to complete the task before switching causes its own problems, especially with WUs over five or more hours. This in my view violates the preferences set by the user. Paul's idea of "Lanes" as he decribes them would work well with my idea I think, especially with the longer run WUs. For shorter run WUs, I think my idea would get the work out the door efficiently without violating user preferences, which in all cases need to be respected by BOINC.
(Click for detailed stats)
RE: Now I see that what
)
Think about projects with very long running WUs. For examples, I've run CPDN tasks that took several CPU months (!!!) to complete. Running one of those monsters exclusively from start to finish would have shut me out from all other BOINC projects for half a year.
CU
Bikeman
RE: RE: Now I see that
)
It would shut you out of other projects if run on a single core but not on a multi-core. But assume it's on a single core.... over the longterm it doesn't matter if one doesn't crunch other projects for 6 months because after the CPDN task is done you won't crunch CPDN for a long while. In the end it all balances out. Isn't the task and project switching just a show for the user to instil the faith that over the short term his shares are being respected? Is the show really necessary?
@Gerry,
Thanks for additional details. OK you're not proposing to run until finished but now it sounds like what you want is what we already have. Maybe you're seeing tasks with 2 hours time being run when tasks with 4 hours are waiting because the task with 4 hours was running on 1 core when the other core(s) switched?
BOINC FAQ Service
Official BOINC wiki
Installing BOINC on Linux