OK using i7z which i think is giving meaningful values than turbostat, and i can say i'm a long way short of Arthur's results running 32 tasks. Currently it looks like ~40 hours if i can wait that long.
i7z reports 2698MHz and Turbo mode ON.
Also found cpu-world showing what turbo actually means so i'm happy the CPUs are clocking right.
The difference between Arthur’s and my setup seems memory channel 4 vs 2 and speed 1600MHz vs 1033MHz.
I shall now explore the dark void of memory bandwidth. I may be some time...
... looks like a quad channel is the way forward. Suggestions anyone?
That was certainly my experience with host 831490 (dual-quad Xeon workstation, Dell chassis with Intel 500-based motherboard) nearly 10 years ago. Switching to quad-channel (automatic by putting four matched Rambus DIMMs in the right slots) increased crunching performance by about 25%.
The Mem Read (MB/s) value is now rather dynamic, spiking to 17000+ if measuring over one second intervals whereas before it would not get past 13000. So fingers crossed...
I've been working on building a "CPU-only-cruncher" myself.
Here's my build so far...
MB: SuperMicro X8DTT-F (2x)
CPU: Intel Xeon CPU E5620 @ 2.40GHz (2/board)
RAM: TBD, trying to find some cheap bulk RAM for this...has about 12 slots per board. Anyone know a good source for this?
PSUa: Supermicro 24Pin 1200W 1U Server Power Supply 80 Plus Gold - PWS-1K21P-1R
PSUb: PDB-PT808-S20 SuperMicro Accessory
These SuperMicro boards can be a royal pain getting to work correctly. Took me a while to realize it has the crazy proprietary 20-Pin power connection to the MB.
I'll post some pics later...
It's been 'fairly' cheap to put together too....
MB: $20 (x2 is $40)
PSUa: $25
PSUb: $35
CPU: $12 (x4 is $48)
Total: ~$148
Not bad for a two-dual CPU cruncher with 32 total cores. :P Again this doesn't include RAM but hopefully I can weasel some cheap sticks.
I buy my RAM at NewEgg or Amazon because I like those colorful heat sinks on gamer memory. If there is a super-secret source of cheap RAM, I don't know about it. BUT, you will be able to use dual channel mode with those CPUs without bottlenecking the system, so you only need 2 sticks per CPU. Also, running ~28 simultaneous units on dual 2670's with an additional 3 GPU tasks, I'm only using 14% of 32GB memory. If budget is your main concern, I think 4 sticks of 4GB each will be just fine, possibly even 4 2GB sticks given the smaller number of tasks you will be running.
You could get 4 sticks of 2GB for $50, for example:
OK do you have a single 1U rack style style case for these two or the plug in blade types?
Quote:
CPU: Intel Xeon CPU E5620 @ 2.40GHz (2/board)
Quote:
RAM: TBD, trying to find some cheap bulk RAM for this...has about 12 slots per board. Anyone know a good source for this?
I'm not 100% sure what RAM you need, mobo suggest DDR3 1333, but Intel says DDR3 800/1066. Note it's triple channel, and there is lot of the 1066 on ebay for peanuts.
Quote:
These SuperMicro boards can be a royal pain getting to work correctly. Took me a while to realize it has the crazy proprietary 20-Pin power connection to the MB.
Yes the supermicro (well any vendor) server boards take you towards a pretty much to their PSU, and then their case. What disk drives?
Quote:
I'll post some pics later...
cool, i'd be interested to know if it's noisy.
Quote:
It's been 'fairly' cheap to put together too....
very cheap! Those E56xx Xeons were my first thought starting point.
Could y'all help me try and understand how the memory configuration effects the CPU/BOINC performance; specifically what do you all mean about Dual or Triple channel memory configurations. I threw 6x4gb 1333Mhz sticks on it but I'm not sure if the configuration was correct. Any insight would be of great help!
I'll try and get some pics to y'all..it's nothing pretty at the moment, just a couple boards sitting on some cardboard boxes. @AgentB No Rack Case at the moment. I'm working on fabricating a case for it myself but it's been quite slow in the making.
Could y'all help me try and understand how the memory configuration effects the CPU/BOINC performance; specifically what do you all mean about Dual or Triple channel memory configurations. I threw 6x4gb 1333Mhz sticks on it but I'm not sure if the configuration was correct. Any insight would be of great help!
I just looked the Supermicro mobo user manual section 2-5 has the optimal arrangement (all in A slots) . I posted earlier Frank Denneman's blog as being helpful.
Simply put, each channel provides an access path to memory, so the more paths available then more data can be moved. Think lanes on a highway.
Quote:
I'll try and get some pics to y'all..it's nothing pretty at the moment, just a couple boards sitting on some cardboard boxes.
Here it is...it is now to be known as, "The Socki.Box"....
I'm running it now with just one cpu and a full module (6x) of 4gb 1333mhz sticks to see how one CPU fully loaded does. It is running 8 tasks simultaneously.
OK using i7z which i think is
)
OK using i7z which i think is giving meaningful values than turbostat, and i can say i'm a long way short of Arthur's results running 32 tasks. Currently it looks like ~40 hours if i can wait that long.
i7z reports 2698MHz and Turbo mode ON.
Also found cpu-world showing what turbo actually means so i'm happy the CPUs are clocking right.
The difference between Arthur’s and my setup seems memory channel 4 vs 2 and speed 1600MHz vs 1033MHz.
I shall now explore the dark void of memory bandwidth. I may be some time...
RE: I shall now explore the
)
OK that wasn't too painful, Intel's PCM tools downloaded, make and follow instructions lead me to
pcm-memory.x
Apologies for the long post
[pre]
running at 0
|---------------------------------------||---------------------------------------|
|-- Socket 0 --||-- Socket 1 --|
|---------------------------------------||---------------------------------------|
|-- Memory Channel Monitoring --||-- Memory Channel Monitoring --|
|---------------------------------------||---------------------------------------|
|-- Mem Ch 0: Reads (MB/s): 0.52 --||-- Mem Ch 0: Reads (MB/s): 0.19 --|
|-- Writes(MB/s): 0.19 --||-- Writes(MB/s): 0.10 --|
|-- Mem Ch 1: Reads (MB/s): 0.50 --||-- Mem Ch 1: Reads (MB/s): 0.19 --|
|-- Writes(MB/s): 0.18 --||-- Writes(MB/s): 0.08 --|
|-- NODE 0 Mem Read (MB/s) : 1.02 --||-- NODE 1 Mem Read (MB/s) : 0.38 --|
|-- NODE 0 Mem Write(MB/s) : 0.37 --||-- NODE 1 Mem Write(MB/s) : 0.18 --|
|-- NODE 0 P. Write (T/s): 437 --||-- NODE 1 P. Write (T/s): 0 --|
|-- NODE 0 Memory (MB/s): 1.39 --||-- NODE 1 Memory (MB/s): 0.56 --|
|---------------------------------------||---------------------------------------|
running at 25% (8)
|---------------------------------------||---------------------------------------|
|-- Socket 0 --||-- Socket 1 --|
|---------------------------------------||---------------------------------------|
|-- Memory Channel Monitoring --||-- Memory Channel Monitoring --|
|---------------------------------------||---------------------------------------|
|-- Mem Ch 0: Reads (MB/s): 1425.77 --||-- Mem Ch 0: Reads (MB/s): 1806.74 --|
|-- Writes(MB/s): 380.18 --||-- Writes(MB/s): 492.18 --|
|-- Mem Ch 1: Reads (MB/s): 1424.30 --||-- Mem Ch 1: Reads (MB/s): 1806.92 --|
|-- Writes(MB/s): 379.74 --||-- Writes(MB/s): 492.54 --|
|-- NODE 0 Mem Read (MB/s) : 2850.07 --||-- NODE 1 Mem Read (MB/s) : 3613.66 --|
|-- NODE 0 Mem Write(MB/s) : 759.93 --||-- NODE 1 Mem Write(MB/s) : 984.72 --|
|-- NODE 0 P. Write (T/s): 551 --||-- NODE 1 P. Write (T/s): 3 --|
|-- NODE 0 Memory (MB/s): 3610.00 --||-- NODE 1 Memory (MB/s): 4598.38 --|
|---------------------------------------||---------------------------------------|
Running at 50% (16)
|---------------------------------------||---------------------------------------|
|-- Socket 0 --||-- Socket 1 --|
|---------------------------------------||---------------------------------------|
|-- Memory Channel Monitoring --||-- Memory Channel Monitoring --|
|---------------------------------------||---------------------------------------|
|-- Mem Ch 0: Reads (MB/s): 3822.52 --||-- Mem Ch 0: Reads (MB/s): 4258.93 --|
|-- Writes(MB/s): 1368.96 --||-- Writes(MB/s): 1425.41 --|
|-- Mem Ch 1: Reads (MB/s): 3818.65 --||-- Mem Ch 1: Reads (MB/s): 4254.18 --|
|-- Writes(MB/s): 1368.25 --||-- Writes(MB/s): 1424.40 --|
|-- NODE 0 Mem Read (MB/s) : 7641.17 --||-- NODE 1 Mem Read (MB/s) : 8513.11 --|
|-- NODE 0 Mem Write(MB/s) : 2737.22 --||-- NODE 1 Mem Write(MB/s) : 2849.81 --|
|-- NODE 0 P. Write (T/s): 874 --||-- NODE 1 P. Write (T/s): 0 --|
|-- NODE 0 Memory (MB/s): 10378.39 --||-- NODE 1 Memory (MB/s): 11362.91 --|
|---------------------------------------||---------------------------------------|
Running at 62.5% (20)
|---------------------------------------||---------------------------------------|
|-- Socket 0 --||-- Socket 1 --|
|---------------------------------------||---------------------------------------|
|-- Memory Channel Monitoring --||-- Memory Channel Monitoring --|
|---------------------------------------||---------------------------------------|
|-- Mem Ch 0: Reads (MB/s): 5335.20 --||-- Mem Ch 0: Reads (MB/s): 4294.53 --|
|-- Writes(MB/s): 1988.72 --||-- Writes(MB/s): 1513.29 --|
|-- Mem Ch 1: Reads (MB/s): 5327.53 --||-- Mem Ch 1: Reads (MB/s): 4291.10 --|
|-- Writes(MB/s): 1986.82 --||-- Writes(MB/s): 1512.99 --|
|-- NODE 0 Mem Read (MB/s) : 10662.72 --||-- NODE 1 Mem Read (MB/s) : 8585.63 --|
|-- NODE 0 Mem Write(MB/s) : 3975.54 --||-- NODE 1 Mem Write(MB/s) : 3026.29 --|
|-- NODE 0 P. Write (T/s): 1035 --||-- NODE 1 P. Write (T/s): 195 --|
|-- NODE 0 Memory (MB/s): 14638.26 --||-- NODE 1 Memory (MB/s): 11611.92 --|
|---------------------------------------||---------------------------------------|
Running at 75% (24)
|---------------------------------------||---------------------------------------|
|-- Socket 0 --||-- Socket 1 --|
|---------------------------------------||---------------------------------------|
|-- Memory Channel Monitoring --||-- Memory Channel Monitoring --|
|---------------------------------------||---------------------------------------|
|-- Mem Ch 0: Reads (MB/s): 5131.37 --||-- Mem Ch 0: Reads (MB/s): 6004.28 --|
|-- Writes(MB/s): 1940.39 --||-- Writes(MB/s): 2441.99 --|
|-- Mem Ch 1: Reads (MB/s): 5126.06 --||-- Mem Ch 1: Reads (MB/s): 5991.51 --|
|-- Writes(MB/s): 1939.08 --||-- Writes(MB/s): 2437.45 --|
|-- NODE 0 Mem Read (MB/s) : 10257.42 --||-- NODE 1 Mem Read (MB/s) : 11995.79 --|
|-- NODE 0 Mem Write(MB/s) : 3879.48 --||-- NODE 1 Mem Write(MB/s) : 4879.45 --|
|-- NODE 0 P. Write (T/s): 1394 --||-- NODE 1 P. Write (T/s): 0 --|
|-- NODE 0 Memory (MB/s): 14136.90 --||-- NODE 1 Memory (MB/s): 16875.23 --|
|---------------------------------------||---------------------------------------|
Running at 87.5% (28)
|---------------------------------------||---------------------------------------|
|-- Socket 0 --||-- Socket 1 --|
|---------------------------------------||---------------------------------------|
|-- Memory Channel Monitoring --||-- Memory Channel Monitoring --|
|---------------------------------------||---------------------------------------|
|-- Mem Ch 0: Reads (MB/s): 6100.31 --||-- Mem Ch 0: Reads (MB/s): 6021.69 --|
|-- Writes(MB/s): 2728.76 --||-- Writes(MB/s): 2517.61 --|
|-- Mem Ch 1: Reads (MB/s): 6082.72 --||-- Mem Ch 1: Reads (MB/s): 6010.76 --|
|-- Writes(MB/s): 2723.05 --||-- Writes(MB/s): 2515.13 --|
|-- NODE 0 Mem Read (MB/s) : 12183.03 --||-- NODE 1 Mem Read (MB/s) : 12032.45 --|
|-- NODE 0 Mem Write(MB/s) : 5451.80 --||-- NODE 1 Mem Write(MB/s) : 5032.75 --|
|-- NODE 0 P. Write (T/s): 1415 --||-- NODE 1 P. Write (T/s): 0 --|
|-- NODE 0 Memory (MB/s): 17634.83 --||-- NODE 1 Memory (MB/s): 17065.20 --|
|---------------------------------------||---------------------------------------|
Running at 100% (32)
|---------------------------------------||---------------------------------------|
|-- Socket 0 --||-- Socket 1 --|
|---------------------------------------||---------------------------------------|
|-- Memory Channel Monitoring --||-- Memory Channel Monitoring --|
|---------------------------------------||---------------------------------------|
|-- Mem Ch 0: Reads (MB/s): 6224.04 --||-- Mem Ch 0: Reads (MB/s): 6115.16 --|
|-- Writes(MB/s): 3692.56 --||-- Writes(MB/s): 3102.25 --|
|-- Mem Ch 1: Reads (MB/s): 6200.76 --||-- Mem Ch 1: Reads (MB/s): 6105.67 --|
|-- Writes(MB/s): 3684.86 --||-- Writes(MB/s): 3098.92 --|
|-- NODE 0 Mem Read (MB/s) : 12424.79 --||-- NODE 1 Mem Read (MB/s) : 12220.82 --|
|-- NODE 0 Mem Write(MB/s) : 7377.42 --||-- NODE 1 Mem Write(MB/s) : 6201.17 --|
|-- NODE 0 P. Write (T/s): 1221 --||-- NODE 1 P. Write (T/s): 14 --|
|-- NODE 0 Memory (MB/s): 19802.21 --||-- NODE 1 Memory (MB/s): 18422.00 --|
|---------------------------------------||---------------------------------------|
[/pre]
There is additional QPI settings in the bios i need to set to get more detail but i think the above is revealing.
I double checked the memory and it is actually 1600 MHz not 1033 but it is dual channel.
From Intel the maximum memory bandwidth is 51.2 GB/s and Frank Denneman's blog about memory was really helpful.
The DDR3 Memory has a maximum of 12800 MB/s - so Dual channel will be 25600 maximum, and that looks to be very close to the figures above.
... looks like a quad channel is the way forward. Suggestions anyone?
RE: ... looks like a quad
)
That was certainly my experience with host 831490 (dual-quad Xeon workstation, Dell chassis with Intel 500-based motherboard) nearly 10 years ago. Switching to quad-channel (automatic by putting four matched Rambus DIMMs in the right slots) increased crunching performance by about 25%.
OK running at quad channel
)
OK running at quad channel seems to have the desired effects,
[pre]
Running at 100% (32)
|---------------------------------------||---------------------------------------|
|-- Socket 0 --||-- Socket 1 --|
|---------------------------------------||---------------------------------------|
|-- Memory Channel Monitoring --||-- Memory Channel Monitoring --|
|---------------------------------------||---------------------------------------|
|-- Mem Ch 0: Reads (MB/s): 3751.87 --||-- Mem Ch 0: Reads (MB/s): 3675.54 --|
|-- Writes(MB/s): 1572.01 --||-- Writes(MB/s): 1550.15 --|
|-- Mem Ch 1: Reads (MB/s): 3751.83 --||-- Mem Ch 1: Reads (MB/s): 3671.97 --|
|-- Writes(MB/s): 1572.91 --||-- Writes(MB/s): 1549.91 --|
|-- Mem Ch 2: Reads (MB/s): 3748.07 --||-- Mem Ch 2: Reads (MB/s): 3673.55 --|
|-- Writes(MB/s): 1568.89 --||-- Writes(MB/s): 1547.90 --|
|-- Mem Ch 3: Reads (MB/s): 3760.26 --||-- Mem Ch 3: Reads (MB/s): 3693.29 --|
|-- Writes(MB/s): 1573.46 --||-- Writes(MB/s): 1555.25 --|
|-- NODE 0 Mem Read (MB/s) : 15012.03 --||-- NODE 1 Mem Read (MB/s) : 14714.35 --|
|-- NODE 0 Mem Write(MB/s) : 6287.27 --||-- NODE 1 Mem Write(MB/s) : 6203.22 --|
|-- NODE 0 P. Write (T/s): 515 --||-- NODE 1 P. Write (T/s): 66 --|
|-- NODE 0 Memory (MB/s): 21299.30 --||-- NODE 1 Memory (MB/s): 20917.57 --|
|---------------------------------------||---------------------------------------|
[/pre]
The Mem Read (MB/s) value is now rather dynamic, spiking to 17000+ if measuring over one second intervals whereas before it would not get past 13000. So fingers crossed...
I've been working on building
)
I've been working on building a "CPU-only-cruncher" myself.
Here's my build so far...
MB: SuperMicro X8DTT-F (2x)
CPU: Intel Xeon CPU E5620 @ 2.40GHz (2/board)
RAM: TBD, trying to find some cheap bulk RAM for this...has about 12 slots per board. Anyone know a good source for this?
PSUa: Supermicro 24Pin 1200W 1U Server Power Supply 80 Plus Gold - PWS-1K21P-1R
PSUb: PDB-PT808-S20 SuperMicro Accessory
These SuperMicro boards can be a royal pain getting to work correctly. Took me a while to realize it has the crazy proprietary 20-Pin power connection to the MB.
I'll post some pics later...
It's been 'fairly' cheap to put together too....
MB: $20 (x2 is $40)
PSUa: $25
PSUb: $35
CPU: $12 (x4 is $48)
Total: ~$148
Not bad for a two-dual CPU cruncher with 32 total cores. :P Again this doesn't include RAM but hopefully I can weasel some cheap sticks.
Adam: I buy my RAM at
)
Adam:
I buy my RAM at NewEgg or Amazon because I like those colorful heat sinks on gamer memory. If there is a super-secret source of cheap RAM, I don't know about it. BUT, you will be able to use dual channel mode with those CPUs without bottlenecking the system, so you only need 2 sticks per CPU. Also, running ~28 simultaneous units on dual 2670's with an additional 3 GPU tasks, I'm only using 14% of 32GB memory. If budget is your main concern, I think 4 sticks of 4GB each will be just fine, possibly even 4 2GB sticks given the smaller number of tasks you will be running.
You could get 4 sticks of 2GB for $50, for example:
http://www.newegg.com/Product/Product.aspx?Item=N82E16820239848&cm_re=2gb_ddr3-_-20-239-848-_-Product
RE: build so far... MB:
)
OK do you have a single 1U rack style style case for these two or the plug in blade types?
I'm not 100% sure what RAM you need, mobo suggest DDR3 1333, but Intel says DDR3 800/1066. Note it's triple channel, and there is lot of the 1066 on ebay for peanuts.
Yes the supermicro (well any vendor) server boards take you towards a pretty much to their PSU, and then their case. What disk drives?
cool, i'd be interested to know if it's noisy.
very cheap! Those E56xx Xeons were my first thought starting point.
Could y'all help me try and
)
Could y'all help me try and understand how the memory configuration effects the CPU/BOINC performance; specifically what do you all mean about Dual or Triple channel memory configurations. I threw 6x4gb 1333Mhz sticks on it but I'm not sure if the configuration was correct. Any insight would be of great help!
I'll try and get some pics to y'all..it's nothing pretty at the moment, just a couple boards sitting on some cardboard boxes. @AgentB No Rack Case at the moment. I'm working on fabricating a case for it myself but it's been quite slow in the making.
RE: Could y'all help me try
)
I just looked the Supermicro mobo user manual section 2-5 has the optimal arrangement (all in A slots) . I posted earlier Frank Denneman's blog as being helpful.
Simply put, each channel provides an access path to memory, so the more paths available then more data can be moved. Think lanes on a highway.
Excellent - a frankencruncher :)
Here it is...it is now to be
)
Here it is...it is now to be known as, "The Socki.Box"....
I'm running it now with just one cpu and a full module (6x) of 4gb 1333mhz sticks to see how one CPU fully loaded does. It is running 8 tasks simultaneously.