Is there a possibility of using a different compression algorithm for BRP4? Would using LZMA2 for example reduce the size of each task some? I am not sure what the existing compression algorithm is. With each task being 32MB and GPUs being able to process these tasks in as little as 20-minutes, the transfer requirements become significant. This is even more so the case when multiple GPUs are running.
well, there are several projects which use 7z http://www.7-zip.org/ to compress data.
and applying 7-zip to a random data file reduces its size from 4,098 KB to 3,979 KB - about 3% compression.
I do think the project might have thought of that one, if it was going to be any significant use ;-)
yup - it depends on the kind of data. at least worth a test..
Is there a possibility of using a different compression algorithm for BRP4? Would using LZMA2 for example reduce the size of each task some? I am not sure what the existing compression algorithm is. With each task being 32MB and GPUs being able to process these tasks in as little as 20-minutes, the transfer requirements become significant. This is even more so the case when multiple GPUs are running.
well, there are several projects which use 7z http://www.7-zip.org/ to compress data.
and applying 7-zip to a random data file reduces its size from 4,098 KB to 3,979 KB - about 3% compression.
I do think the project might have thought of that one, if it was going to be any significant use ;-)
yup - it depends on the kind of data. at least worth a test..
Well. I did the test. BRP4 can be compressed to 91% of original size according to WinRAR. They are partly text files and partly data files with lots of zeroes in there. So it is the moment to do something with data. Not with the network bandwidth.
Well. I did the test. BRP4 can be compressed to 91% of original size according to WinRAR. They are partly text files and partly data files with lots of zeroes in there. So it is the moment to do something with data. Not with the network bandwidth.
alexander's winrar probably is not an option because it's payware. 7z is open-source and to my experience often performs better..
Well. I did the test. BRP4 can be compressed to 91% of original size according to WinRAR. They are partly text files and partly data files with lots of zeroes in there. So it is the moment to do something with data. Not with the network bandwidth.
alexander's winrar probably is not an option because it's payware. 7z is open-source and to my experience often performs better..
I think there will be no difference in compression between both WinRAR and 7z.
Well. I did the test. BRP4 can be compressed to 91% of original size according to WinRAR. They are partly text files and partly data files with lots of zeroes in there. So it is the moment to do something with data. Not with the network bandwidth.
alexander's winrar probably is not an option because it's payware. 7z is open-source and to my experience often performs better..
I think there will be no difference in compression between both WinRAR and 7z.
Both WinRAR and 7z can handle multiple compression formats. It isn't the tool that you use that matters, it's the compression algorithm that you choose which has to be one which balances compression ratios with server CPU load.
Remember too that the servers run Linux, so whatever tool is chosen, it won't be one with "Win..." in its name ;-)
If you try gzip under Linux, the result is not significantly different, around 8% compression which is not that much.
The work unit generation was kind of a bottleneck before (and compression would be just another, final stage for this), but now that is solved and one can start to balance the 8% or so bandwidths saving against the "never change a running system" thing :-).
There is even a transparent file decompression feature that is built into BOINC so that BOINC will handle the decompression on the client side (BOINC > 5.4) if instructed by the header response of the HTTP download server? So there would not have to be a new client version for this even, if I understand this correctly. Only server side compression, cconfig changes in (ALL!) download servers.
It looks like gzip has the best compression for this particular file format although the difference is not that much in any case. I tried a couple different filters with xz as well but these did not improve compression.
From reading the BOINC documentation, client 5.4 or newer can handle both deflate and gzip using HTTP content encoding. The files would be compressed on the server side and the boinc client can handle decompression after download. If Apache is being used by the project then mod_deflate can handle compression automatically.
The data rates here are prodigious - BRP4 requires roughly four times as much data per unit of computation time, as even the fastest 'shorty' SETI work. And with SETI being largely down at the moment, a lot of that data demand will have transferred here.
Downloadspeeds are very low @ the time. Was away for some hours and get only 7 WUs downloaded, the over 20 others are still loading O.o Had to primegrid one WU before the first einstein WU was ready to crunch. So the packing is not the real clue that seems to help out of speed problems ^^
RE: RE: RE: Is there a
)
yup - it depends on the kind of data. at least worth a test..
RE: RE: RE: RE: Is
)
Well. I did the test. BRP4 can be compressed to 91% of original size according to WinRAR. They are partly text files and partly data files with lots of zeroes in there. So it is the moment to do something with data. Not with the network bandwidth.
RE: Well. I did the test.
)
alexander's winrar probably is not an option because it's payware. 7z is open-source and to my experience often performs better..
RE: RE: Well. I did the
)
I think there will be no difference in compression between both WinRAR and 7z.
RE: RE: RE: Well. I did
)
Both WinRAR and 7z can handle multiple compression formats. It isn't the tool that you use that matters, it's the compression algorithm that you choose which has to be one which balances compression ratios with server CPU load.
Remember too that the servers run Linux, so whatever tool is chosen, it won't be one with "Win..." in its name ;-)
Hi! If you try gzip under
)
Hi!
If you try gzip under Linux, the result is not significantly different, around 8% compression which is not that much.
The work unit generation was kind of a bottleneck before (and compression would be just another, final stage for this), but now that is solved and one can start to balance the 8% or so bandwidths saving against the "never change a running system" thing :-).
There is even a transparent file decompression feature that is built into BOINC so that BOINC will handle the decompression on the client side (BOINC > 5.4) if instructed by the header response of the HTTP download server? So there would not have to be a new client version for this even, if I understand this correctly. Only server side compression, cconfig changes in (ALL!) download servers.
Right?
CU
HBE
I tried a few different
)
I tried a few different compression options with one of the 4MB files:
$ du -sk p2030.20100331.G45.61-01.91.C.b0s0g0.00000_3465.*
3988 p2030.20100331.G45.61-01.91.C.b0s0g0.00000_3465.7z (lzma2)
4100 p2030.20100331.G45.61-01.91.C.b0s0g0.00000_3465.binary
4064 p2030.20100331.G45.61-01.91.C.b0s0g0.00000_3465.binary.bz2 (-9)
3952 p2030.20100331.G45.61-01.91.C.b0s0g0.00000_3465.binary.gz (-9)
3988 p2030.20100331.G45.61-01.91.C.b0s0g0.00000_3465.binary.xz (lzma2)
It looks like gzip has the best compression for this particular file format although the difference is not that much in any case. I tried a couple different filters with xz as well but these did not improve compression.
From reading the BOINC documentation, client 5.4 or newer can handle both deflate and gzip using HTTP content encoding. The files would be compressed on the server side and the boinc client can handle decompression after download. If Apache is being used by the project then mod_deflate can handle compression automatically.
http://boinc.berkeley.edu/trac/wiki/FileCompression
RE: The data rates here are
)
Guilty as charged.
Steve
Crunching as member of The GPU Users Group team.
I think we should try. If it
)
I think we should try. If it will overload servers alittle, than it is no trouble to get everything back to normal. Am I right?
Downloadspeeds are very low @
)
Downloadspeeds are very low @ the time. Was away for some hours and get only 7 WUs downloaded, the over 20 others are still loading O.o Had to primegrid one WU before the first einstein WU was ready to crunch. So the packing is not the real clue that seems to help out of speed problems ^^
DSKAG Austria Research Team: [LINK]http://www.research.dskag.at[/LINK]