Jump to content
*Coming November/December* Signature Rigs & Content Embed Wizard (Google Docs,Soundcloud,Spotify & More) ×
*Coming December* EHW Marketplace ×

Welcome to ExtremeHW

Welcome to ExtremeHW, like most online communities you must register to view or post in our community, but don't worry this is a simple free process that requires minimal information for you to signup. Be a part of ExtremeHW by signing in or creating an account.

  • Start new topics and reply to others
  • Subscribe to topics and forums to get email updates
  • Get your own profile page and make new friends
  • Send personal messages to other members.
  • Take advantage of site exclusive features.

Threadripper Home Server


Recommended Posts

It has been about 6 years since I've posted anything resembling a build log online.  That ended up being more of a summary due to time constraints, and that log has pretty much gone the way of the dodo.

 

Spoiler

 

Dualin' 540

Current state of that thread:

540thread.png.c55cfaba70133baa1bdfdaef592db8af.png

 

Final pics:

27_26_cropped.thumb.png.2b519a4979be9f883a9ecb7b4f764baf.png

39_17_cropped.thumb.png.143725e9db509defdab42ebbeb31f322.png

39_43_cropped.thumb.png.9efb075ae12ec197cf5cbd5fcf772ef0.png

55_43_cropped.thumb.png.d5a00356e8bae10e7848313696b88025.png

 

 

 

 

This build will be somewhat of an evolution of that machine which housed my daily driver and my HTPC in one case.

 

For this machine I am planning on retiring most of the gear in my rack (ASRock Rack w/ 2x 2680 v2, X99-E WS w/ 2697 v4, ASRock X570M Pro 4 w/ Ryzen 3 3200G) and consolidating down to a single machine running in a Thermaltake Core X5.

 

I already have most of the parts for this build, so I hope to have it on the bench to start some testing this weekend.

 

Parts list

Case - Thermaltake Core X5

Motherboard - ASRock Rack X399D8A-2T

PSU - EVGA Supernova G2 1000W

CPU - AMD Threadripper 2970WX

RAM - 4x 8x 16GB DDR4-3200 ECC-UDIMM (eventually 8x 16GB)

GPUs - Radeon 5700XT, Vega 64

SSDs - Intel Optane 905P 480GB U.2, ADATA SX8200 Pro 1TB, SK Hynix P31 1TB, 2x crucial MX500 2TB

HDDs - 2x HGST Deskstar NAS 4TB, 1x Seagate IronWolf Pro 4TB, 5x Seagate EXOS 8TB, 1x 4TB Toshiba N300 (Hot swap bay for backups) 

 

Cooling

CPU - Watercool Heatkiller IV Pro Copper

GPU - Watercool Heatkiller IV Copper/Acetal

Radiator - 2x XSPC EX360

Pump - Watercool D5 w/ EK-XRES 100 Reservoir 

 

The first potato pic of this log will be a quick shot of the ASRock board. :D

 

tRipperServer_mBoard.thumb.jpg.80cc23d3374932924fe948eb49e4810b.jpg

 

Edited by tictoc
Update Parts List
Link to post
Share on other sites

Nice ! Will be a solid build. Got nosey and checked out that motherboard, not bad and comes with 2x 10Gbe ports which certainly makes the networking side of things a little more futureproof. Any more GPU's going in that beast for folding ? 😛 

Link to post
Share on other sites
On 11/4/2020 at 8:21 AM, axipher said:

Looking forward to this project, love the black tubing.

It is the only tube I've used for the last 10 years. For the last 6-7 years it's been EK ZMT, and before that it was just some random EPDM tubing.  Best tubing there is. Zero maintenance, no funky plasticizer, blocks all UV light to help minimize any growth, and can be turned at pretty tight angles without collapsing.  The tubing being matte black is the icing on the cake.

 

On 11/4/2020 at 12:23 PM, ENTERPRISE said:

Nice ! Will be a solid build. Got nosey and checked out that motherboard, not bad and comes with 2x 10Gbe ports which certainly makes the networking side of things a little more futureproof. Any more GPU's going in that beast for folding ? 😛 

The pair of X550's is a nice bonus and frees up a PCIe slot.  Just a few GPUs in this machine, one for the HTPC and one to be used in various VMs. Although I'm sure there will be a bit of F@H run from time to time.

Edited by tictoc
typos
Link to post
Share on other sites
12 hours ago, tictoc said:

It is the only tube I've used for the last 10 years. For the last 6-7 years it's been EK ZMT, and before that it was just some random EPDM tubing.  Best tubing there is. Zero maintenance, no funky platicizer, blocks all UV light to help minimize any growth, and can it be turned at pretty tight angles without collapsing.  The tubing being matte black is the icing on the cake.

 

The pair of X550's is a nice bonus and frees up a PCIe slot.  Just a few GPUs in this machine, one for the HTPC and one to be used in various VMs. Although I'm sure there will be a bit of F@H run from time to time.

Keeping PCIe slots free with having certain items on mobo instead is a big bonus. A. For airflow or B. You can go crazy and bang in loads of add in cards.

Link to post
Share on other sites

That's a nice Threadripper home server build you got there !

 

Re. cooling, I have been using a CPU-only loop with a Heatkiller IV Pro Nickel-Copper for a TR2+  2950X @ 4.275 to 4.325 all-core, along with 2x XSPC RX 360s. Absolutely zero problems for two years now, even with heavy use

Edited by J7SC_Orion
Link to post
Share on other sites

I've been running the Heatkiller block since I first jumped on the Threadripper train with a 2970WX on a Taichi board. Currently using the same block on my main machine which is running a 3960X @ 4.25GHz all-core OC 24/7. 

 

This will be running a very mild OC, since my 2970WX is not a great clocker and hits a pretty steep wall at about 3.75-3.8GHz all-core.  The GPU in the loop (Vega 64 for now) will be undervolted, so the pair of 360s should keep things nice and cool.

 

Getting everything set up on air on my bench to start doing some testing today.  It will be a few weeks before the build actually starts going together, since there is a bunch of testing and validation that I need to do before I migrate everything over to this build.  As part of my down-sizing I am purging a ton of data from storage, since I am a recovering data hoarder. 😃

 

The build itself is pretty straight forward, but the underlying stack will take a bit to iron out.  I am going to be moving all my data from mdadm/xfs to btrfs, and before I do that there will be a fair amount of testing to do.  Also, I am going to do a bit of poking at bcache, since I haven't played around with that in 4ish years.

Link to post
Share on other sites

I can also attest to the Heatkiller blocks, fantastic ! Managed to keep my 2990WX from going to mental in the heat department back when I had that system. I remember when I first got it and I was like WOAH heavy chunk of metal haha.

Link to post
Share on other sites

I forgot the post office closed early on Saturday, so I didn't make it down from my house in time to pick up my Optane drive. 😢  Looks like I'll just be tuning RAM this weekend.

 

Tuning RAM is going to be a real pain on this board.  If the RAM fails to train at a given speed/timings, it won't post, and then I have to go through the following process.

 

Power down -> remove two sticks -> power up -> pray to the memory gods for a successful post -> adjust speed/timings -> reboot with two sticks -> verify post -> power down -> install removed sticks -> power on -> cross fingers and pray for post -> boot into OS -> rinse and repeat.  If after pulling two sticks it still fails to post, then it takes a CMOS reset to get back to a state where the board will post. 

 

Running at 2400 with pretty trash timings right now, hopefully I can just take very small steps and avoid the whole shut down remove RAM reset CMOS shenanigans.

Edited by tictoc
Link to post
Share on other sites
3 hours ago, ENTERPRISE said:

I can also attest to the Heatkiller blocks, fantastic ! Managed to keep my 2990WX from going to mental in the heat department back when I had that system. I remember when I first got it and I was like WOAH heavy chunk of metal haha.

 

...indeed they are - I paid express delivery from Germany to Canada for that weighty cooling wonder 😱

   

1 hour ago, tictoc said:

I forgot the post office closed early on Saturday, so I didn't make it down from my house in time to pick up my Optane drive. 😢  Looks like I'll just be tuning RAM this weekend.

 

Tuning RAM is going to be a real pain on this board.  If the RAM fails to train at a given speed/timings, it won't post, and then I have to go through the following process.

 

Power down -> remove two sticks -> power up -> pray to the memory gods for a successful post -> adjust speed/timings -> reboot with two sticks -> verify post -> power down -> install removed sticks -> power on -> cross fingers and pray for post -> boot into OS -> rinse and repeat.  If after pulling two sticks it still fails to post, then it takes a CMOS reset to get back to a state where the board will post. 

 

Running at 2400 with pretty trash timings right now, hopefully I can just take very small steps and avoid the whole shut down remove RAM reset CMOS shenanigans.

 

...RAM tuning on server boards / CMOS is fun (> not). 🙈

 

But as you know from your prior 2970WX Taichi, it can pay off big time with Zen...just hoping you don't loose your sanity before then, like when you take your morning toast from the toaster and it feels just like pulling RAM sticks off a server mobo to you

Edited by J7SC_Orion
Link to post
Share on other sites

I'm probably just going to end up pretending that all the Threadripper memory settings don't exist in the BIOS, and just treat it like my my C602 board.

 

That board has a nice and easy memory OC. Set speed to 1866MHz, up memory voltage until it's stable, and then just be content that you're running cheap 1333MHz RDIMMs at 1866MHz. :wheee_orange:

Link to post
Share on other sites

Currently I am testing my RAM at 3200 16-20-16-40.  So far it looks to be stable. 

 

The ICs on these no-name ECC 3200 sticks are Micron rev.E, so now that I've already gone this far, I guess I'll see just how far I can push them while maintaining 100% stability. 

Edited by tictoc
Link to post
Share on other sites

I wouldn't have even started messing with the memory, but originally I couldn't get the board to post with the 3200MHz ECC sticks.  To get it to post the first time, I had to put in a couple of non-ECC sticks.  Once I did that, I was able to power down and swap in a pair of the ECC sticks.  That got me into the UEFI, but it would almost instantly crash because the speed/timings were way off from what the sticks wanted to run at.

 

After the initial brain damage of finding a starting point for the RAM, it has actually been fairly simple to up the speed and tighten the timings.  Two big gotchas for this board/memory are that the board will not even post if TRCD is set below 20, and almost any speed will fail to post or be wildly unstable with Command Rate set at 1T.

Link to post
Share on other sites

64GB (4x16GB) ended up being 100% stable at 3200|16-20-16-38|1.395v.

 

128GB (8x 16GB) is not stable at the same timings. I have a working (and at least mostly stable) profile for 2933.  Unless it will run at 3200 CL18, which is highly unlikely from my past experience of trying to run 3200 with 128GB on my Taichi board, I'll be running it at 2933. :D 

 

 

Link to post
Share on other sites
19 hours ago, tictoc said:

64GB (4x16GB) ended up being 100% stable at 3200|16-20-16-38|1.395v.

 

128GB (8x 16GB) is not stable at the same timings. I have a working (and at least mostly stable) profile for 2933.  Unless it will run at 3200 CL18, which is highly unlikely from my past experience of trying to run 3200 with 128GB on my Taichi board, I'll be running it at 2933. :D 

 

 

For a 128Gb, that is not bad performance mate.  Do you actually use that much or could you go down to 64gb and harness the additional performance in speeds ? Not sure if what you run is memory performance sensitive or just the fact you just need as much RAM as you can get.

Link to post
Share on other sites

As much as I can get, so even if I have to drop down to something like 2733 or 2800 with cleaned up timings that would be fine. 👍

Edited by tictoc
Link to post
Share on other sites

Not much happening other than continuing some extended testing.

 

tRipperServer_testing2.thumb.jpg.ef971cf3d25c6098ab3c8cb75d8a3a28.jpg

 

Right now I'm doing some misc performance/reliability testing on btrfs, and trying to decide what the final storage layout will be.

 

Also, I'll be going with a 200mm Noctua fan in the front panel of the case, which will hopefully move enough air to keep things cool.  Only thing that I might need to address temp wise is the VRMs.  With the goal being silence I'd rather not add a fan mid case to cool the VRMs.  The server will see fairly high load compiling and running test applications, but with no OC, VRM temps should be fine.  If not, then I can always add one of my Heatkiller SW-X 80 DIYs to the loop.  I used a SW-X 60 on my Taichi X399 board, and it cooled the VRMs without a hitch.   

Link to post
Share on other sites

Nothing too exciting, just making up some new cables for the drives.

 

tRipperServer_driveCables1.thumb.jpg.d3ce257280fc4dc58649a6284a84e3d5.jpg

 

I will be starting to backup/transfer the data on the current server and start migrating the data to the new drive setup this weekend. 😀

Edited by tictoc
Link to post
Share on other sites

10Gb networking makes pretty quick work of transferring data around.  I'll probably spin up a temporary 6-8TB SSD RAID-0 to pull the data off the server, add the old drives to the new storage pool, and then it's just a local rsync to put the data in the new storage pool.  Since I purged a bunch of stuff off my current file server to cold archives, I should be able to get everything moved in 2 batches.  

Edited by tictoc
Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...

Important Information

This Website may place and access certain Cookies on your computer. ExtremeHW uses Cookies to improve your experience of using the Website and to improve our range of products and services. ExtremeHW has carefully chosen these Cookies and has taken steps to ensure that your privacy is protected and respected at all times. All Cookies used by this Website are used in accordance with current UK and EU Cookie Law. For more information please see our Privacy Policy