Deletion of data files on hosts by the server

I've entertained similar thoughts because of a number of things that happen from time to time which seem to be not optimal.

In your case however, my own observations seem to suggest that the initial request for deletion was triggered by the server thinking that every task that needed 807.10 data had been issued at that point. Of course, 18 days later there are bound to be some that need to be reissued due to non return of the original issue. So if the server had made the decision to request deletion based on all tasks issued rather than all tasks returned and validated, what you saw in your case is bound to happen from time to time.

Common sense says that file deletion should only occur after all tasks have been returned and validated. My own observations suggest that this is not the case. I've been trying to get "runs" of tasks with sequential sequence numbers by increasing the preference for work_buf_additional_days in the local prefs override file on a particular host. I have observed, after getting several in sequence, that the server will sometimes "jump up" the frequency of the next task delivered by 0.05 or 0.10. This can often be accompanied by a drastic change in sequence number. However, I've also noticed that if I keep increasing the cache a day at a time, quite often the server will revert to the previous frequency and the correct next sequence number, allowing me to continue that particular run.

Mostly, this happens with no request for deletion but often a request for slightly higher frequency data. However, I have seen at least two cases where there were requests for deletion at the time of the small step change in frequency and then a reversion to the lower frequency with additional "correct" sequence numbers. This clearly seems to suggest that on at least two occasions that I've directly observed, the server was requesting deletion of data for which all tasks had not even been fully issued.

I think the local client must store up the request and only act on it after all tasks depending on a particular data file had been completed. I say this because there has not ever been a problem in crunching the additional tasks delivered after the request for deletion was issued by the server.

Another nasty thing I've seen happen a large number of times is that a new host will often be issued with "old" work for its very first task. This "old" work (sometimes even in the 7xx.xx frequency range) will often be a single task for which the complete suite of large data files has just been downloaded. Subsequent requests for work will result in a complete set of "newer" data files (in the 8xx.xx or 9xx.xx frequency range) which then can be used for a much larger number of tasks. I think it's quite unfair to force a new host to effectively do two complete data downloads in this way. The server should wait longer for someone who already has the data before picking a new host to be given this "reissue" work.