Tasks being trashed due to MD5 checksum errors for LIGO data files

I should have moved the first message about this before responding to it. My apologies to all concerned. Once I've fixed this by shifting all the discussion into this new thread, I'll answer all the subsequent responses that have arisen.

It did - at least on the couple of occasions that I took the trouble to check it out. I quickly came to the conclusion that NEITHER the data itself NOR the server supplied checksums stored in the state file were corrupt. My assumption was that it had to be something to do with the way the checksum was being calculated/tested within the BOINC client.

My situation is quite special. I have ALL the LIGO data in question stored locally on a central host. I have the blocks for this data (extracted from state files) also stored locally. I have dozens of hosts sharing this data. I had only one host trashing groups of tasks that depended on this data. Initially, all the hosts sharing this data were "seeded" with a selection of blocks from the local repository to get them started on the frequency range I wanted them to use. Apart from the issues with this one host, I had nothing else to suggest there was any problem with the data or the server supplied checksums.

I would have retrieved groups of trashed tasks on this particular host at least 20 - 30 times over the (quite extended) period that I experimented with this particular problem. When I first noticed this problem, I had a large multi-day cache and NNT was set. There were about a dozen successfully completed tasks and perhaps 50+ tasks that had been trashed. There were more tasks that didn't depend on the supposedly corrupt LIGO file and the host was still working on these.

Naturally, I was quite dismayed so I stopped BOINC and started browsing the state file to try to get a handle on what was going on. It's a long story but I spent a lot of time studying the internals of the state file and I decided to attempt a recovery, knowing that E@H had the "resend lost tasks" feature enabled. By comparing a copy of a "good" state file with the current one, I found it quite easy to see what needed to be done to "undo" the damage. It was easy to see the particular LIGO file that caused the problem and "fix" the block for that file. I had a freeware MD5 tester and I used it to create a sum to compare against the value in the state file. It checked out OK but I did replace the file with a fresh copy from the local repository - which also checked out OK as well.

There were a couple of other minor things that needed fixing eg status values of -119 and -161 instead of zero (or 1) spring to mind. These were quite obvious when comparing good with bad. The biggest deal was what to do about all the .... blocks for the 50+ errored tasks. I decided to delete them all completely and see if the server would resend them once I restarted BOINC. I did spend a lot of time checking things before finally saving the changes.

So, with fingers crossed, I restarted BOINC and was quite surprised to see no complaints at all. The fully completed tasks were still there ready to report. The 50+ errored tasks were simply not listed any more. The 4 tasks in progress (quad core) all restarted from where they had left off. The remaining unstarted tasks were still there waiting to go. I left NNT in place and updated the project. The completed tasks were reported and validated. The server resent all the "lost tasks", a dozen for each time I updated. I kept a close watch on this machine as further tasks completed and there were no issues with validation. I enabled new tasks and refilled the cache without any issues - until a day or so later when it all hit the fan once again. The only difference this time was that a different LIGO file was being complained about.

As I mentioned before, I've performed the above procedure at least 20 times on this same host and never had any issue recovering the cache fully each time. There were never any issues with validation of any tasks at any time, that I noticed. I made refinements to the technique. For instance, while most of the trashed tasks had no accumulated CPU time, there were usually a couple that were part crunched at the time that BOINC decided that a LIGO file was suddenly corrupt. When this occured, I noticed that in the block at the very bottom of the state file, there were still s for each of the partly crunched failed tasks. As these had unique numbers associated with them, this gave me the idea that rather than delete the block for each failed task, maybe I could simply edit them to "fix" what BOINC had done to them and thereby get them to reload the checkpoints that were still in the slot directories. And, yes, the editing was quite easy to figure out and it worked perfectly. The checkpoints for each partly crunched task were reloaded and crunched to completion and I didn't find a single task that failed subsequently to validate. I was able to extend this editing to all failed tasks and so I stopped creating lost tasks to be resent, entirely.

I formed the view that data and checksums must all be OK, otherwise a task would never have got to the partly crunched stage in the first place. The only other thing I could think of was that the disk surface might be on the way out and perhaps there were a quite small number of occasions where a disk read was failing. I tested that by renaming each "complained about" LIGO file with a _BAD extension and then making a new copy of the file. My thinking was that, over time, I should be able to "blank out" the bad spots on the disk in this way. After 20 - 30 iterations I came to the conclusion that this theory wasn't likely to be true. That's when I decided to play with the RAM timings - with immediate success.

I don't claim I fully understand this problem but I know how much I persisted with the experiments and I know what ultimately worked for me. If you have a better explanation I'll be very happy to hear it.

OK, first the usual warning. Mistakes in editing your state file can trash your entire cache. Be very careful and make sure you use a "plain text" editor. I use notepad2 for Windows, kwrite for Linux and Text Wrangler for Mac OS X. The following notes assume you have already browsed your state file and have a basic understanding of its structure and that you can identify what I'm talking about without further hand holding. If any person reading this is not in this category, I strongly suggest that you make the decision not to fiddle with your state file. Please don't ask for step by step details. I don't have time to do that, sorry.

There are 6 areas of the state file to be aware of in particular:-

• 1. The blocks for data files (sun, earth, skygrid, LIGO data, etc).
  2. The blocks that contain task name and output information. These contain and tags, amongst other things.
  3. The block which separates the previous two from the next one
  4. The blocks
  5. The blocks
  6. The block which contains all the s
  

I'm describing the state file from the point of view of a single project (E@H) and a single application (the GW app). Obviously things are more complicated if you have multiple projects and multiple apps per project.

The only things you need to worry about are items 1, 2, 5, and 6 in the above list.

1. Look for a negative (-119) for the 'complained abou't data file. There will also be an block added in. Change the status to '1' and completely remove the error message. If you are satisfied the file is not corrupt, leave it alone. If you are not sure, either check it manually or perhaps use one of the mirror URLs to grab a new copy and replace it.

2. At the time of the failure some of these may have acquired a negative (-161 springs to mind). These should all have a status of zero. I started by actually deleting all the blocks that had -161 status on the basis that they were and so would be recreated. That worked fine but I also found that just changing the negative status to zero also worked.

5. This is where most of the fixup work resides. These are task/result templates where things change and/or get added at various stages during the crunching of a task. This is what finally gets sent to the scheduler via a sched_request when a completed task (successfully or after erroring out) is reported. Take a look at an 'unstarted' result. Notice things like the is zero, the is 2 and the line is immediately after the line. Then take a look at a block for an 'in progress' task and play 'spot the difference'. From memory, the main difference is that there is an entry between and .

If you get tasks that error out but are not reported, there are a few extra things to notice. The will have a negative value (-185 springs to mind in my case), the will be a value larger than 2, and there will be added lines between and , including error message lines. When I was into fixing trashed caches, all I did was search for the -185 and zero it, change the in the next line back to 2 and then remove all the added lines between and . The only exception was for partly crunched tasks where there was an entry for the task in . In that case I would look for and leave the line in place, since I was expecting the partly crunched task to be retrieved from its checkpoint (which it always was). In those cases I believe the line was always immediately above the line.

The above is all from memory but I did it so many times without incident that I'm pretty confident it's close to the mark. Browse your current state file and see if you can see all the things I'm talking about.