GPU work units fail, AMD gpu

Hi Kristian, Welcome to Einstein!  Thank you for deciding to devote resources here.

Yesterday, I posted a small guide in the Getting Started forum that might be useful to review.  Since you are successfully running, you may be needing more targeted information to get the best performance.

You have a pair of very capable machines with AMD GPUs.  I use very similar AMD GPUs (no nvidia) so I can give some advice about how to optimise.

A lot depends on your objectives.  At point 2. in the guide, I posed a question.  At the end of the guide, I tried to suggest why detecting continuous GW is likely to be a big deal and therefore is very much why the project would love you to run the GW app.  If you make a decision about what you would like to run, I'll help you get the best out of what you decide to run, on whichever machine you want to run it.

Think about it and let me know what you want to do.

The GW GPU app is relatively new and still with some raw edges.  There are improvements but they are quite slow to arrive.  The biggest problem is that there are still some procedures/functions in the algorithm that can only run on a CPU core so GPU utilisation appears low and crunch times are slow compared to the much more efficient GRP GPU app.  However, you can considerably improve performance of the GW app by running concurrent tasks.  It does make a substantial difference to output.

For example, I have a Ryzen 5 2600 (6C/12T) host with a 4GB RX 570.  It runs 3 concurrent instances of the GW GPU tasks in a bit over 30 mins currently - a completed task every 10-12 mins on average.  It runs GPU tasks only and has a current RAC of over 200K.  This is probably close to double what could be achieved by running GPU tasks one at a time.

I run Linux exclusively, not Windows.  I used to frequently see a similar problem - it essentially got fixed with a new amdgpu kernel module (the GPU driver) sometime in early 2018 if I remember correctly.  It still happens very occasionally to a couple of machines and it's the GPU crashing (fans stopped) whilst the CPU in the machine runs normally.  Connecting to the machine remotely shows the symptoms you describe - the task in question struggling extremely slowly towards 99.999%.

It never gets to that for me these days.  I have control scripts running on a server machine which closely monitor the app running on each host.  If a GPU crashes, the CPU clock ticks being used to support the task(s) running on that GPU decline to zero per second so the situation can be detected quite quickly.  A cold restart of the affected machine always solves the problem.  The BOINC client then restarts the tasks that were 'spinning their wheels', using the last saved checkpoint.  The tasks always complete successfully without further issue.

Unfortunately, stopping and restarting BOINC without completely resetting the GPU is not enough.  Hence the cold boot.  Next time you see this issue, try stopping and restarting BOINC.  Things might work differently with Windows.  If the task doesn't immediately start making normal progress, you will need to reboot the machine.