GPU slow %
20 Jul 2018 1:29:25 UTC
Topic 215860
(moderation:
GPU is fluctuating all over the place. It doesn't go past 80%.
Windows 10, GTX 1080Ti - Driver v398.36, BOINC v7.10.2, Einstein@Home - Gamma-ray pulsar binary search #1 on GPUs 1.20 (FGRPopencl 1K-nvidia.
You may find the GPU
)
You may find the GPU utilization number goes higher on the Gamma-ray pulsar binary search here if you enable running two of them at a time on your GPU.
But if you have other work going on for your CPU which gets in the way of the CPU instantly servicing the GPU, then running 2X may not help. Based on your description, I suspect that is the case. But you could also be banging into other limits.
I'm running 2X on a GTX 1070 on this host
Windows 10, Driver 388.71, BOINC 7.10.2
GPU-Z reports the GPU utilization at 99%.
"CPU instantly servicing the
)
"CPU instantly servicing the GPU". Well that's just it other GPU work from other work like Prime GPU runs at 99% all the time even with all CPUs at 100%. It's the only GPU work that I see not fluctuating. I'll try the 2x ok.
The current Windows Nvidia
)
The current Windows Nvidia application for Gamma Ray pulsar GPU work here at Einstein is built using openCL. While I'm no expert on the details, I believe that previous Einstein applications built using CUDA handled CPU servicing of GPU requirements by a completely different communication mechanism. Possibly the key difference is that CUDA uses interrupts for circumstances for which openCL relies a a continuous-polling wait loop.
Whether I've got any of that right or not, the readily observable thing is that the current Einstein Windows Nvidia application occupies a full CPU instance if available, even though very little actual CPU computation is done, and the considerable data transfer activity does not add up to that either. I think that a side-effect you may be observing is that it is much easier to starve the GPU by having the rest of the system in a state which does not give the GPU access to CPU services as quickly as in a responsive system. I, personally, don't run any other BOINC work than Einstein GPU Gamma-Ray Pulsar tasks.
We volunteer participants have hoped for a CUDA build for quite some time, but that appears not to have worked out well the last time it was tried, and another trial may not be high enough on the priority list to reach us any time soon.
Sandman192 wrote:... Well
)
You can't legitimately compare the Einstein GPU app with other GPU apps from other projects. Some apps will need a lot of 'cpu support' - others very little. There is also quite a difference (so it would seem) in the OpenCL implementations provided by different GPU manufacturers. At Einstein, the same app runs quite differently on AMD hardware compared to nVidia and it's not clear to me why that should be. Maybe it's a function of different compilers or different compiler flags. I don't think there's been any sort of definitive answer provided. It's not the overall elapsed time that is the problem - modern nVidia GPUs can be very productive, provided you are not trying to run other CPU intensive work on all the CPU cores. You need to experiment to find the optimum conditions.
The only working nVidia card I'm using is a 750Ti. It's not quite modern enough to be a decent performer. It runs in a Pentium dual core host (G640). All my hosts run Linux and there aren't any GUI apps to show GPU utilisation. However, there is a handy utility called 'top' which shows a big ordered list of running processes and the (CPU and memory) resources they consume. On this host, I'm running a single CPU task only (so there is an 'unloaded' CPU core) along with a single GPU task. When I run top, it shows the CPU task consuming 99% of a CPU core and the GPU task consuming 98% of the other CPU core. The machine runs headless. If I actually hook up peripherals and login to the desktop, it does feel slow and if I run even basic commands, etc, the EAH GPU tasks do slow down as a result.
I can easily compare that to a slightly older Pentium dual core (E6300) running a AMD RX 570. It's also running a single CPU task but the second core supports 3 concurrent GPU tasks. When I run top on this machine, the CPU task is consuming 99% of a CPU core and the 3 GPU tasks, supported by the other core, are using 4%, 2%, and 2% respectively, just now when I took the snapshot. The values do fluctuate - the top utility updates the displayed full page list every second - so I just quit the utility which leaves the last snapshot as a static display. If I hook up peripherals to this machine, it always feels fully alive and responsive and there is very little (if any) change in crunch times from using the desktop for a while.
I have a bunch of older machines that were CPU only that I've hardware upgraded with AMD 'Polaris' series GPUs (RX 460 to RX 580) approximately a year ago. The above machine was one of those. They had not been software updated since original installation for the new GPU and I decided very recently to see what affect on performance there might be by bringing them right up to date now. It's pretty easy to do this because I keep a regularly updated full local mirror copy of my distro's repository on an external USB hard drive. I started this little exercise about 10 days ago.
As an experiment on the first one I updated, I decided to see what would happen if I did the update whilst BOINC was still running. I was expecting problems but was curious to see what would happen. There were in excess of 700 packages that needed updating and it took about 30 - 40 mins for the update to complete from the USB3 connected hard drive repo copy. There were no issues and the Einstein apps kept running throughout. There were a number of tasks completed during this time and each had added about a minute or two to the normal crunch time. In other words, even though there were fairly extensive package database transactions going on (the repo contains over 15,000 individual packages) the crunching was barely affected. When the update completed, I rebooted, installed the latest kernel (4.17.x), and rebooted again so the new kernel could run. I restarted BOINC and the apps just picked up their last saved checkpoints and continued on. Some of them had been running BOINC 7.2.42 so, for those, I just threw in the boinc, boincmgr and boinccmd executables for 7.6.33 and they restarted just fine with the newer BOINC version. My distro doesn't package BOINC so I had built 7.6.33 from source some time ago.
One of the packages that got updated was the amdgpu open source graphics driver. The new version seems to provide around a 5+% performance improvement. I install the OpenCL libs that come with the official AMD supplied AMDGPU-PRO package for Red Hat Linux. My distro uses RPM style packaging so I was able to unpack the Red Hat packages and extract the bits that provide OpenCl. These bits are installed on top of the amdgpu graphics driver. I had been using the 16.60 version of the libs. I've now tried a series of later versions - 17.50, 18.10 and the very latest 18.20. None of the more recent versions seem to do any better than 16.60 so I haven't been too bothered about fully updating OpenCL as well.
I'm not a programmer so I have no real knowledge or understanding of why there is such a huge difference in the CPU resources needed to support OpenCL on nVidia as opposed to AMD. If you have adequate CPU resources, modern nVidia cards perform very well. If you have poor performance at Einstein, you may need to reduce the number of concurrent CPU tasks. Running multiple GPU tasks could help with this because the system will 'reserve' an extra CPU core for each extra concurrent GPU task. Just make sure you don't try to run serious 'non-BOINC' work on those reserved cores or else performance will suffer significantly.
Cheers,
Gary.