GRP tasks not being generated for the last six hours?
9 Sep 2019 14:25:39 UTC
Topic 219557
(moderation:
I've noticed that my machines running GPU GRP tasks have had their queues drop over night.
On review I notice that the last time either machine received tasks with _0 or _1 suffixes was somewhat over six hours ago, with just a few _2 and such (reissues) delivered since then.
On the Einstein server status page the 14:20 UTC update shows zero tasks to send for FGRPB1G, and that work generator as "not running".
Something amiss?
+1
)
+1
archae86 wrote:I've noticed
)
Gary wrote in another thread:
It's nothing to do with tasks being "messed up". Where does that idea come from? It's entirely to do with the fact that there are 2 different applications - a long standing and trusted app running on a CPU core and a new application designed to perform parts of the calculations (hopefully much faster) by taking advantage of the parallel processing capabilities of a modern GPU.
The GPU app is under development and the testing of it is revealing that it's not always getting close enough answers to those of the trusted CPU app, in certain circumstances. By analyzing the reasons for the discrepancies in the results, the Devs hope to achieve a GPU app that will both save time and mostly always get close enough to the same results. It's an ongoing process and the invalid results should hopefully lead to newer versions of the app that can achieve the required computational accuracy.
mikey wrote:Gary wrote in
)
Mikey,
What I wrote in that "other thread" was in connection to a totally different science app - the test app for the O2AS GW search using GPUs.
Archae86 has started this thread because (for some as yet unexplained reason) the workunit generator for the Gamma Ray Pulsar search that uses GPUs (FGRPB1G) is currently not running and there are no tasks ready to send.
Even if you had simply confused the two searches, thinking they were one and the same, I still don't understand how a test app getting some inconclusive/invalid results could in any way explain the puzzling failure of a workunit generator to keep producing new tasks ready to send out. I guess we'll get some information once the issue has been resolved.
Cheers,
Gary.
The GRP generators have not
)
The GRP generators have not been running for a lot longer than 6 hours.
https://einsteinathome.org/goto/comment/173245
Finally not only are GPU GRP
)
Finally not only are GPU GRP tasks being generated, but the rest of the plumbing is intact enough that all of my machines have reported old work and received fresh work.
The Radeon VII is off to the "sit and wait a while" punishment box as it reached a daily task download quota, but the other two are fully caught up.
I was down to two tasks in queue on one machine when I got up (the plumbing and task availability had been OK for over two hours, but it was waiting out a deferral), so two days was a good cache size this time.