Progress stops

Einstein@Home uses different applications to search for different things.  So, yes, each 'search' uses a particular application to perform the search.  As an example, there is a CPU search for pulsars that emit gamma rays using data from the Large Area Telescope (LAT) on the Fermi satellite. There is also a search for continuous gravitational wave (GW) emissions from massive rotating objects like neutron stars, some of which may be spinning at millisecond rates.  That search uses data from the LIGO observatories.

There are also two searches that use GPUs to look for similar things.  The GW GPU search is via a test app at the moment so you shouldn't be getting any of those unless you have changed the setting to allow test apps.  Now that you've 'unhidden' your computers (thank you) I can see that you aren't getting test tasks and that's a good thing because they are having validation problems at the moment.  Also, they have other issues when run simultaneously with the gamma ray pulsar GPU tasks.

Virtually all the tasks you aborted were for the FGRP5 CPU search (22 tasks).  There were no GPU tasks aborted and only 2 for the O2AS20-500 GW search.  I looked at all your tasks for the FGRP5 search and here is a link to what shows up.  Over time this will change as old tasks get removed and new tasks get added to the online database.

The first 5 tasks in that list show a progressive deterioration in the crunch time - from 18k secs to 31k secs.  That is not normal and tends to suggest that something else is going on.  Since both CPU time and run time are slowing down, it looks like CPU frequency is reducing enormously.  You would normally expect throttling like that to be due to overheating but you say your machine is water cooled - so it seems a bit of a mystery.  Perhaps you should give your machine a good check for virus/malware in case it's something like that?

FGRP5 tasks have a pretty standard run time.  The 18k secs figure for your machine seems reasonable.   Does your workload from Seti change very much?  There's got to be something that has changed that's causing the slowdown.  There is no sign that the tasks you aborted were in any sort of trouble.  You can click on the task ID link of any task to see what was returned to the project up to the point the task was aborted.  I looked at a few and there seems to be quite a lot of evidence for tasks being stopped and restarted from saved checkpoints.

You will likely lose quite a bit of progress every time that happens but all you should see when you restart is that the time and progress values should go back to what they were at the time of the saved checkpoint.  I saw one that was aborted when it was in the followup stage - probably not many minutes before it would have completed and been returned.  There is no progress beyond ~90% in the followup stage until the final jump to 100%.  Don't ever abort a task that seems stuck at ~90% without allowing 30-60 mins for it to finish normally.

I think you'd just end up with a more confusing situation than you already have.  Different projects have apps that use your resources differently.  Some make heavy use, some make lighter use.  If one project will run without issue, that doesn't really help you explain why another app has problems.

Lets look at some preference settings first - although I can't really see how this would cause what you see.  For computing preferences, you've already mentioned your 50% time setting.  There's a particular reason I dislike that setting.  I run CPUs 24/7/365.  After starting, they reach a stable operating temperature and stay there with no short term thermal cycling.  By using the 50% time setting you are essentially turning your CPU from idle to full on and back again every couple of seconds.  I worry that the expansion/contraction effects may have long term consequences for CPUs and motherboards.  I think a better mode of operation is to leave that setting at 100% and control the total heat generated by reducing the % of cores - or you could turn HT off and go back to the 4 'real' cores and run 3 CPU tasks plus 1 GPU task.  That will give a lower operating temperature without the internal fluctuations.

I have machines still running fine (Q6600 quad core CPUs) that I built in 2008.  The CPU temps are around 80-90C.  I've never had a CPU failure.  If your CPU temps never exceed 65C, you have no issue with heat.  I don't know what sort of damage might eventuate from very short term thermal cycling though.

Can you also advise of other processor usage settings that start, "Suspend when .... "?  If crunching is suspended on a regular basis, it's pretty important that the setting for keeping tasks in memory when suspended is set to yes.  That won't actually change the reported values for crunch times but it could change your perception of how long it's taking for a task to make headway if the values keep resetting to those of a former checkpoint.  If you keep tasks in memory, you can eliminate the wasted progress that gets lost every time a task has to restart from a checkpoint.

In the tasks list link that I gave you last time, there is a new FGRP5 completed task (returned 15 Sep 10:18:22 UTC) with an elapsed time of 25,733 secs.  There is also a new aborted task that was returned about an hour later.  Here is the task ID data that was returned for that aborted task. If you look down towards the bottom before the Windows debugger stuff, you can see that 7 out of 79 sky points had been completed and the 8th had just been started.  The line that says "% checkpoint read: skypoint 7 binarypoint 0" tells us that the app was restarting by reading in from a saved checkpoint.  Two lines later it tells us "% Sky point 8/79" which signifies the start of computing for the 8th checkpoint when the task was aborted.

For comparison, here is the task ID data that was returned for the successful task.  This is exactly what crunching should look like - 79 completed loops (each with 56 'dots' - the nf1dots parameter was 56) and after each one, the new checkpoint was written.  These 79 loops are the main calculations and are followed by the followup stage where the top 10 candidate signals are assessed.  No sign of a single stoppage or restart from a checkpoint anywhere in the whole run.  The followup stage is the bit that happens when the progress seems to pause just below the ~90% mark.

So, the real description of your problem is that a normal task can proceed through all stages in a certain amount of time with no signs of stopping/restarting or any other sorts of hiccups.  On the other hand you have what should be very similar tasks that get around 10% of the way through in rougly the same total time with lots of evidence of stops/restarts and extremely slow rates of progress.  Something on your machine is likely doing this.  I would think it's very unlikely to be the hardware itself.  A hardware issue should affect all tasks.

I have no idea why there is such a difference between your 'good' and 'bad' results.  It's not the data - the same data file LATeah0060F.dat was being crunched in both those tasks.  It has to be something else.  If you can't work out what it is, I would suggest not running the FGRP5 CPU tasks (you've only got a couple left) and start working on the much larger number of O2AS GW tasks that you already have.  They seem to run more successfully.

Maybe someone else might have some ideas.  I've done lots of FGRP5 tasks over the years and never seen this sort of behaviour before.

Assuming you still have hyper threading enabled, I would reverse these settings. 100% time usage, and 50% of CPU's. If the issue resolves itself, you may gradually increase the percentage of CPUs used until you start having errors. Ideally, I like to run one task per physical core, allowing at least 2 additional threads for GPU use. In my case, I have a DL360 server with 2 processors, six physical cores each. I have run it successfully between 50% and 65% of CPU utilization. I do know, when I first started the box, It was running at 100% and every task ran with errors, including the GPU. Since I was planning to reduce the utilization I didn't give it much thought, but reading your comment sheds new light on the issue.

I'm currently running at 60%, giving me 12 concurrent CPU tasks, and 2 concurrent GPU tasks. Unless I want to add the other power supply, 65% is about as much as I can push that box.

As Gary has mentioned, different applications will have different characteristics as to how they will perform. Some are more CPU intensive than others, the result being, higher power consumption and higher operating temperature. Since you seem to have a good setup for CPU cooling, I doubt temperature is an issue, but power consumption might be.

Please realise that the % of cores BOINC is allowed to use refers to CPU tasks only and not GPU tasks.  If your "7 tasks running" consisted of 6 CPU tasks plus 1 GPU task, that is what is allowed for a setting of 75%.

For the case of a Seti task stopping and then restarting after about a minute, that could be due to a restriction like suspend when non-BOINC usage is above 35%, for example.  In other words, BOINC saw a limit being exceeded and so paused a task until whatever it was cleared.  The fact that everything went back to zero could have been due to no checkpoint being available to restart from.  You have to crunch for a while until the first checkpoint is written.  I don't run Seti and have no knowledge of how their tasks work.