Scheduler Bug with work requests for O2MD1 work - ### Staff Please Read ###

That is exactly how lost tasks are treated.  They are not new tasks but tasks that have already been allocated to you and have become 'lost' (by whatever mechanism).

I actually use this 'feature' to my (and the project's) advantage.  Because the performance of all my hosts is closely monitored automatically, I get alerted very quickly when something catastrophic happens.  A recent example for one host was that just as one task was finishing and a new one was about to start, BOINC suddenly decided that the file JPLEPH.405 was corrupt (MD5 check failed).  Of course, the whole work cache of maybe 200 tasks immediately were declared as computation errors since all tasks rely on that file.

The nice thing is that for major problems like this, the tasks sit there and don't get reported until the balance of (at max) a 24hr backoff counts down.  Once alerted that the machine has stopped crunching, it's about a 5 min job to shut BOINC down and cause all the failed tasks sitting in the state file to become 'lost'.   This is achieved by editing the state file so you really need to understand what you are doing.

Whenever I investigate failures like this, the data file that has failed the MD5 check seems OK, but as a precaution (in case a disk sector belonging to the file gave a transient read error) I rename the file to <filename>.BAD and then install a fresh copy of the file to a new spot on the disk.

Then I restart BOINC, which shows no compute errors any more but may show tasks completed prior to the error.  The restart automatically clears the backoff and BOINC contacts the server.  Immediately, 12 'lost tasks' are resent and crunching resumes.  Periodically, I will 'update' the project to get further batches of 12 until all the lost tasks have been fully recovered.  Everything is back to normal.  It's as if the problem never occurred in the first place.

The machine that I'm referring to in the above description was crunching FGRPB1G and not O2MD1 tasks.  Since the O2MD1 GPU tasks are test tasks where only 1 test task per quorum is 'allowed', the above procedure doesn't work.  You would think it should work if all the 'lost' tasks were GPU tasks.  Each quorum would no longer have its one GPU task so it should be OK for the scheduler to resend the task as a GPU variant.  I have specifically tested this, and others have reported as well, so I know that such a lost task gets replaced by a CPU version and each affected quorum will no longer contain its allowance of 1 test task.

Hopefully, when Bernd gets to look at all this, something might be done about that as well.  In other words, a 'lost' GPU task should be able to be resent as a GPU task and not a CPU task, particularly if a volunteer has deliberately not allowed CPU tasks in the preferences.  I know from actual testing that NOT allowing CPU tasks does NOT stop the scheduler from resending a lost GPU task as a CPU task.