Fulfill O3 GPU quorums not validating
14 Dec 2023 5:19:20 UTC
Topic 230493
(moderation:
I run exclusively GW tasks of the GPU O3 flavor recently. This week I've had a sudden big surge in number of tasks pending. On reviewing some today I found it common to see WUs for which both tasks had been returned, but both were currently reported with status "Completed, waiting for validation" The server status page does not report validator suspended or otherwise not running, but does report a large number of O3AS pending tasks.
Here are links for a few such WUs:
https://einsteinathome.org/workunit/771901252
https://einsteinathome.org/workunit/771916542
https://einsteinathome.org/workunit/771916652
https://einsteinathome.org/workunit/771916754
https://einsteinathome.org/workunit/771945775
https://einsteinathome.org/workunit/771887645
These were as of 5:14 UTC December 14, 2023
I also have noticed a growing
)
I also have noticed a growing number of pending tasks that have a minimum quorum of two that have not been validated.
Server Status Page O3AS -
)
Server Status Page
O3AS - 55,000+ waiting for validation.
something stuck project-side.
_________________________________________________________________________
The filesystem on the project
)
The filesystem on the project server (einstein3) is doing some "scrubbing" that slows down the validator. It's not completely stuck, but slowed down (current delay 1-2d). The scrubbing should be finished in <20h, after that, the validator should be able to catch up. I also moved the second search (BRP7) to another server, which should also help to reduce I/O load.
BM