Building 4xGPU rack-mount rendering node

Few months ago we build 2 custom GPU rendering nodes. I’ve decided to share all the specs of what we ended up buying to make it easier to anyone who wants get something similar.  

List of components:

  1. Rack mount case – Chenbro RM413 4U + 8 PCI rear back window (55H103413B007 + 55H173413B001) – 142 Eur + 15 Eur  = 157 Eur . You have to unscrew the rear window that comes with the case and replace it with extended one or instead look for Chenbro RM41300-FS81 which already has 8 slots. I wasn’t able to buy this one but it’s out there.
  2. Motherboard – MSI X99A XPOWER 390 Eur . This one is relatively cheap and has 2 x 1Gbit Ethernet connection.
  3. PSU – Enermax Platimax 1500W modular with 90-95% efficiency 290 Eur, enough to supply power for all GPUs,
  4. Memory – Crucial Ballistix Sport 8GB DDR4 x 4 = 211 Eur.
  5. CPU –  Intel i7-5930K 3.50GHz 15MB BOX = 560 Eur
  6. SSD –  Samsung 250GB 850 EVO = 85 Eur.
  7. OS – Windows 7 PRO = 90 Eur

Total without GPUs = 1783 Eur

To put this into perspective. Before I decided to build those I got quotes for 4xGPU server nodes that would cost 7000 Eur. GPUs not included.

DSC00324

We ended up doing two configurations:

4 x Nvidia GTX TItan X 12GB at around 1150 Eur x 4 = 4600 Eur 
Total cost = 6383 Eur (23% VAT included) 

And

4 x Nvidia GTX 980Ti 6GB at around 2840 Eur with Asus motherboard
Total cost = 4723 Eur (23% VAT included)

Overall this is quite cost-effective solution if you need rack mounted nodes. You could use standard desktop case if you want, there are many around and they are easy to find. Look for 4-way SLI PC case and it will do the trick. If you’re really hardcore, you can go with some 7-PCI-E motherboard and custom case. For inspiration look at bitcoin mining builds.

And here are some photos from the building process:

DSC00312

DSC00333

DSC00336

DSC00334

Though I don’t have more photos from the build process. It’s quite straight forward. If you’ve built any PC in your life it will be just as easy. This is basically desktop build in rack mount server format.

One more note about cooling. Those builds are working basically 24/7 for 3 months now without any issues. Temperatures are in range of 78-84 while fan speed is auto managed by GPU (default) and is going at 60-70% so it still has some room in case things will get too hot. NVidia rates those GPU safe up to 91 degrees so I don’t think there are any reasons to worry about. At the time of this post up-time of both render nodes was 22 days (since last electricity malfunction). Depending on your environment temperatures you could add some fans to the case to help with airflow. I’m using stock setup at the moment.

So far those bad boys helped us with those shots (among many others) :

002
https://www.behance.net/gallery/33632113/Guitar-CGI

9c69b934685953.56d99b307174c
https://www.behance.net/gallery/34685953/Excavator-CGI

Best,

Tomasz Wyszołmirski

42 Comments

  • Andrew says:

    Hi Tomasz,

    First of all, congratulations for your build and article, it’s very explanatory. The projects you’re using this for are awesome, as well!
    I’m looking into assembling a similar render node and I have a few questions:

    Did you have to strip the GPUs from their cases or did they fit as they were out of the box?
    Did you have other options in terms of rack-mounted cases, besides the Chenbro you chose?
    How come you didn’t go for the 1080 TI FE GPUs?

    Thank you!

    • Hi Andrew,

      GPUs fit as they are, no need for any adjustments.
      Chenbro was the best solution I found in terms of price and fitting 4GPUs, but it might have changed.

      I ended up replacing 980ti with 1080ti. Keep in mind that this blog post is 15 months old, 1080ti didn’t exist back than.

      Best,
      Tomasz

  • Mike says:

    Hi Tomasz,

    you wrote:
    “I don’t know how I missed your comment. But regarding the PCI speed, from my tests it seems like it doesn’t matter as much with V-Ray RT. It will only affect the time it takes to transfer all the data to GPU, which even with 3.0 x1 wouldn’t be that bad. After initial loading, there is very little data going to and from GPU.”

    Does that mean, that this rig would be totaly fine for rendering? https://www.miningrigs.net/?product=gray-matter-gpu-server-case-v3-0

  • Alexander Zovic says:

    Hi Thomas. Love your and thanks for sharing so much information with us. Were looking in to buying a gpu cluster, but Im Wonderign about network speeds and how much dr rendering for example would affect the network. Do you have any experience with that?

  • Dragos says:

    Hi Tomasz

    With rendering times at around 10 minutes per FHD frame – this would mean that a 3 minutes video shot at 24FPS would take around 10 days to render.

    Is this correct ? Any issues with heating if you leave your station to work at full power for 10 days ? Electricity must cost a ton 🙂

    We plan at building a rendering station this summer, using the future 2080 chips from Nvidia, maybe 4 x 2080 Ti with 64GB RAM – any chance we can drop down those times to around 2 minutes/frame ?

    Thanks,

    • wyszolmirski says:

      Hi Dragos,

      In general on those nodes are frames render from 30s to 10 minutes. Recently we calculated some 4k shots that took only 1-2 minutes per frame. But there is no rule other than more complex scenes = longer the render. I did calculate that rendering on GPUs with taking into account electricity, render nodes costs, licences etc is around 4 times cheaper than than on CPU.

      As for heating or issues, there were none so far. For Nvidia 2080 etc… I wouldn’t trust any rumors until there is solid statement from Nvidia.

      Best,

    • Dragos says:

      Absolutely, nothing’s certain until Nvidia says so. A couple of days ago they’ve launched Titan Xp, with around 10% better performance compared to a 1080 Ti, but almost doubling the cost.

      In this scenario, we’d be looking at investing around 10.000€ to build two nodes, each with 4 X 1080 Ti’s.

    • wyszolmirski says:

      I will drop some additional information on this as it might be useful.

      1080 Ti is the great deal with it’s 11GB of memory, but 1070 is still the best choice when it comes to power per dollar probably 6×1070 in some kind of open air case could be the cheapest render node.

      I did some crazy estimations a while ago: http://www.dabarti.com/screens/ec027f8d5-d479-4e91-b9a7-198c4b01f4d6294642b5809-093b-4b61-b55a-de8f7b3dc505.png Assuming I would need 1 mln frames in a year, what would be the cheapest option.

      New Ryzen CPUs are looking pretty good, but they are at least 2 times more expensive than Titan X nodes. But it might be a good choice if you need a lot of memory. But on the other hand Nvidia offers great professional Quadro GPUs with 24GB on board. Also dual Quadro P100 with NVlink can scale to 32GB of memory.

      Anyway. Rendering on GPU is looking pretty good.

  • Sebastian says:

    Tomasz, thank You so much for posting this! I’m building my own rig right now and I cannot decide between 5820K and 5930K. I will use dual GTX 1070 FE for GPU rendering with V-Ray RT (I don’t need much more at the moment). What is your stance on having a CPU with 28 vs 40 PCIe lanes? Is there any real difference between rendering performance while using 16x/16x and 16x/8x PCIe lane speed? The pricetag on the 5930K is substancially higher and I cannot decide whether it is worth the additional cost.

    Any advice on that matter will be highly appreciated. Especially from someone like you.

    Greetings from a fellow Pole.

    • Hi Sebastian,
      I don’t know how I missed your comment. But regarding the PCI speed, from my tests it seems like it doesn’t matter as much with V-Ray RT. It will only affect the time it takes to transfer all the data to GPU, which even with 3.0 x1 wouldn’t be that bad. After initial loading, there is very little data going to and from GPU.
      Best,

    • Sebastian says:

      Thank You very much for your response. I went with 5930K anyway (I got it for a great price-only 50$ more expensive than 5820K which was a bargain if you ask me). Double GTX 1070 FE should do the trick for now with plenty of headroom for future improvement, including additional GPUs. Anyway-thanks again. I admire your work. Keep it up!

  • Judith Chu says:

    Hello, I have recently bought 4x EVGA GTX1080 FTW DT GAMING ACX3.0 8GB cars for my new rendering workstation. However, the Nvidia guy kept telling us the 4way SLI won’t work and the GTX1080 only supports 2way. I used to have a GTX980ti SLI for rendering. Will it have a huge performance boost or it will just work like one card like the Nvidia guy said?
    Thanks

  • Hi, when you say: 4 x Nvidia GTX 980Ti 6GB at around 2840 Eur with Asus motherboard

    What ASUS motherboard?

    • It was Asus Rampage V Extreme Intel X99 LGA2011-3 BOX . Most likely it is a bit overkill, I’m thinking about revisiting this with an most affordable build possible… Something with open air case and 6 GPUs could work be interesting.

  • Carl says:

    Hello Tomasz, first of all thank you for your article. Secondly are you using GTX 980s etc.. as opposed to the quadro cards because they’re a more cost effective solution?

    • Yes. That’s the main reason. They are more cost effective, though there are claims that Quadros are more stable and reliable.

      Personally I never got any issues with GTX cards and I can recommend them for rendering.

  • Vas Tcaci says:

    Tomasz, thanks a lot for this great info and the script you have done for VrayRT. I have tried GPU rendering few years ago and I didn’t like it. However, after reading your post I gave it another go and.. it’s great!. Just on time, as I was planning to upgrade my PC.

    I am wondering if the core i7 6700k processor will be able to run 4 GPU’s and a PCIe SSD, as I have noticed that it has only 16 PCIe lanes. Where it gets a bit confusing for me is that some of the motherboards offer 4 way SLI.

    I thought I would ask you as you have more knowledge in this department.

    • Vas, GPUs for rendering are not supposed to be connected through SLI, it will work without it just fine, it may even slow down the rendering. When it comes to PCI lanes and PCIe speed as far as I know it may only affect the time it needs to transfer all geometry, textures onto GPGPU memory, but difference would be so small that overall performance will be pretty much identical.

      Best,
      Tomasz

    • Vas Tcaci says:

      Thanks Tomasz, I have done some research and those motherboards have PLX chips to split the PCI lanes to the GPU’s. As much as I like the 6700k CPU I will probably go for a core I7 6850k just to avoid any issues.

      One other question regarding V-ray RT. On production rendering mode I noticed a high peak usage of the CPU , for few seconds, while the light cache is calculated and then the GPU kicks once this is complete. Is this normal? Is this not suppose to be calculated by the GPU? This is not the case in ActiveShade mode where Light Chace is not used.

    • Light Cache is always calculated on CPU at the moment.

  • daniel says:

    Hi guys,

    Thank you for the quality informations on the site, and congradulations for your work!

    I am thinking about building a system with i7 4790k, 32gb ram, 2 x gtx690 4gb for rendering. Do you think is a good system for rendering architectural interior and exterior scenes ( houses, offices, etc) with vray rt ?

  • Paul says:

    Have you used Red Shift on this setup yet? If so what are your render times on images like the ones you’ve shown?
    Im looking to build a render farm soon.
    Thanks
    paul

    • Hi Paul,

      I tried Red Shift but on 970 and than later on 1070 to compare against other GPU engines, but I ended up sticking with V-Ray RT.

      Render times for 1080p frames are usually in range of 2-6 minutes for our shots. It can go up to 10-15min for heavy scenes.

      Best,

  • Ivan Schuler says:

    Hello I want build a 4x GTX Titan Workstation. A GPU Workstation for rendering. Because I´m gone render everything with the GPU, how much CPU you would recommend for the new Workstation. icore 7 o xeon? Other applications that are not rendering are running very smooth with a normal CPU.

    Right now i have this setup:
    CPU: 2.66 GHz Dual Quad Core Xeon X5550 (8core / 16 threads)
    RAM:32GB 1066 MHz DDR3
    GPU: Nvidia GeForce GTX 770 3GB
    iOS: Win 8
    5 year old.

    Its ok for work, it just need more speed for rendering.

  • Vu Pham says:

    Hi, ever you render something like this scale before with this setup ? https://www.flickr.com/photos/phamduongvu/23293917054/in/dateposted-public/

    Love to study more

  • Rob says:

    Looking forward to hearing more details about using RT in production. Keep up the great work. Currently debating whether to build dual Xeon or quad GPU setup and seeking info!

  • Dennis Persson says:

    Hi Tomasz

    Is it possible to use this machine as an DR node from within Max or Maya with RT and the GPUs?

  • Dennis says:

    Hi Tomasz, is it possible to use this as a DR node from another workstation that doesnt have an RT compatible GPU?
    I was thinking of something like this in order to speed up lookdev and not having to send out frames individually to each node

  • I have z97 pro ac wifi 32 GB ddr3 ram dual GTX 970 4790K CPU if I upgrade to dual GPU to GTX 980 TI do you think I could take render in 15 -20 min ?

    • Dual GTX 980Ti Would be definitely way faster than 4790K. When it comes to comparing GPUs. You can test your render times on 970 that you have and use it as baseline. 980Ti would be around 60% faster. I will post in near future a little bit more details about using RT in production.

  • I do architectural design and I on my pc it takes around 30-40 min Im trying to understand if I should go for workstation pc or invest on graphic card 🙂

    Basically what Im asking is that which way is the fast way to take render dual chip set mother board and single GPU or single cpu +4gpu

  • can you pls tell me how long does it take to render something professional print out board ?

    • What exactly do you mean by “professional print out board” ? In general we render animations at 1080p if you check our vimeo page most of the latest clips were rendered on this farm with times per frame usually in the range of 1-10 minutes depending on the complexity. Sometimes it does go up to 20-30 minutes but that is quite rare.

  • Ohnet says:

    awesome! thanks for the anwswer.

  • Ohnet says:

    Hello!! nice setup, I m trying to build this atm. I a few titans and another rig that is going with 980ti.

    how you deal with the temps with all the cards stacked like that?

    what is the highest that you have.

    • Temperatures are in range of 78-84 usually. Fan speed is auto managed by GPU (default) and is going at 60-70% so it still has some room in case things will get too hot. It’s going almost non-stop for 3 months so far, so I would say it’s very stable.

      Just checked. Both nodes have up time at 22 days at the moment. Since last electricity malfunction 🙂

Leave a Reply