Discussion Thread for the Continuous GW Search known as O2MD1 (now O2MDF - GPUs only)

For the last couple of weeks, I've been running O2MDF GPU tasks on a new build (amongst several others) - a Ryzen 2600 with a 4GB RX 570 GPU.  I started with tasks directed at the G347.3 supernova remnant and found the machine would run very nicely with a 4x multiplicity and produce 250 results per day.   Having settled on 4x, the machine ran continuously for 17 days 16 hours until the transition to data directed at the Vela pulsar.  There were 4329 results uploaded during that period with 0 invalids and 0 errors.  That works out to 245 results per day.

The transition to crunching Vela (frequencies just over 400Hz) happened at 6:45am 1/12/19 (UTC +10).  The big difference was a lot longer crunch time causing a drop to around 103 tasks for the first full day.  There were still no problems with errors or invalids - until just recently.

The machine moved from crunching tasks named h1_0461.20_......  to a higher frequency band, eg h1_0654.20_.......  and now the problems have started - 1 validate error and 3 TIME_LIMIT_EXCEEDED pretty quickly.  I actually saw the last one.  The % done was just under 98% and the elapsed time was 2h 8m 16s when the plug was pulled.  There was another task around 80% and approaching the 2hr mark so I suspended the entire cache of work except for that one and it was able to finish quickly without error.

I decided to transition back to 3x, and as a result, the TIME_LIMIT problem seems resolved.  There is a task about to finish now (98%) and the elapsed time is currently just over 1hr 20m.   There are two more following at lower % completed but with commensurately lower elapsed times.  They will be fine.

It just remains to be seen about validate errors.  It would appear that the memory requirements and/or crunch time of Vela tasks might increase with frequency, something that wasn't so noticeable with the G347.3 directed data.  It would be nice if we could be warned about this sort of stuff.

I remember a post from Bernd about getting back a lot more GW results than expected and that this was causing issues for the servers.  I wonder if they have deliberately increased the work content of Vela tasks so as to reduce the numbers of results, perhaps without adjusting the value that controls TIME_LIMIT.  Perhaps this also leads to increased memory requirements.  If that's the case, it sure would be nice to get a bit of advance warning.