Building 4xGPU rack-mount rendering node

DSC00371

Few months ago we build 2 custom GPU rendering nodes. I’ve decided to share all the specs of what we ended up buying to make it easier to anyone who wants get something similar.  

List of components:

  1. Rack mount case – Chenbro RM413 4U + 8 PCI rear back window (55H103413B007 + 55H173413B001) – 142 Eur + 15 Eur  = 157 Eur . You have to unscrew the rear window that comes with the case and replace it with extended one or instead look for Chenbro RM41300-FS81 which already has 8 slots. I wasn’t able to buy this one but it’s out there.
  2. Motherboard – MSI X99A XPOWER 390 Eur . This one is relatively cheap and has 2 x 1Gbit Ethernet connection.
  3. PSU – Enermax Platimax 1500W modular with 90-95% efficiency 290 Eur, enough to supply power for all GPUs,
  4. Memory – Crucial Ballistix Sport 8GB DDR4 x 4 = 211 Eur.
  5. CPU –  Intel i7-5930K 3.50GHz 15MB BOX = 560 Eur
  6. SSD –  Samsung 250GB 850 EVO = 85 Eur.
  7. OS – Windows 7 PRO = 90 Eur

Total without GPUs = 1783 Eur

To put this into perspective. Before I decided to build those I got quotes for 4xGPU server nodes that would cost 7000 Eur. GPUs not included.

DSC00324

We ended up doing two configurations:

4 x Nvidia GTX TItan X 12GB at around 1150 Eur x 4 = 4600 Eur 
Total cost = 6383 Eur (23% VAT included) 

And

4 x Nvidia GTX 980Ti 6GB at around 2840 Eur with Asus motherboard
Total cost = 4723 Eur (23% VAT included)

Overall this is quite cost-effective solution if you need rack mounted nodes. You could use standard desktop case if you want, there are many around and they are easy to find. Look for 4-way SLI PC case and it will do the trick. If you’re really hardcore, you can go with some 7-PCI-E motherboard and custom case. For inspiration look at bitcoin mining builds.

And here are some photos from the building process:

DSC00312

DSC00333

DSC00336

DSC00334

Though I don’t have more photos from the build process. It’s quite straight forward. If you’ve built any PC in your life it will be just as easy. This is basically desktop build in rack mount server format.

One more note about cooling. Those builds are working basically 24/7 for 3 months now without any issues. Temperatures are in range of 78-84 while fan speed is auto managed by GPU (default) and is going at 60-70% so it still has some room in case things will get too hot. NVidia rates those GPU safe up to 91 degrees so I don’t think there are any reasons to worry about. At the time of this post up-time of both render nodes was 22 days (since last electricity malfunction). Depending on your environment temperatures you could add some fans to the case to help with airflow. I’m using stock setup at the moment.

So far those bad boys helped us with those shots (among many others) :

002
https://www.behance.net/gallery/33632113/Guitar-CGI

9c69b934685953.56d99b307174c
https://www.behance.net/gallery/34685953/Excavator-CGI

Best,

Tomasz Wyszołmirski

22 Comments

  • Vas Tcaci says:

    Tomasz, thanks a lot for this great info and the script you have done for VrayRT. I have tried GPU rendering few years ago and I didn’t like it. However, after reading your post I gave it another go and.. it’s great!. Just on time, as I was planning to upgrade my PC.

    I am wondering if the core i7 6700k processor will be able to run 4 GPU’s and a PCIe SSD, as I have noticed that it has only 16 PCIe lanes. Where it gets a bit confusing for me is that some of the motherboards offer 4 way SLI.

    I thought I would ask you as you have more knowledge in this department.

    • Vas, GPUs for rendering are not supposed to be connected through SLI, it will work without it just fine, it may even slow down the rendering. When it comes to PCI lanes and PCIe speed as far as I know it may only affect the time it needs to transfer all geometry, textures onto GPGPU memory, but difference would be so small that overall performance will be pretty much identical.

      Best,
      Tomasz

    • Vas Tcaci says:

      Thanks Tomasz, I have done some research and those motherboards have PLX chips to split the PCI lanes to the GPU’s. As much as I like the 6700k CPU I will probably go for a core I7 6850k just to avoid any issues.

      One other question regarding V-ray RT. On production rendering mode I noticed a high peak usage of the CPU , for few seconds, while the light cache is calculated and then the GPU kicks once this is complete. Is this normal? Is this not suppose to be calculated by the GPU? This is not the case in ActiveShade mode where Light Chace is not used.

    • Light Cache is always calculated on CPU at the moment.

  • daniel says:

    Hi guys,

    Thank you for the quality informations on the site, and congradulations for your work!

    I am thinking about building a system with i7 4790k, 32gb ram, 2 x gtx690 4gb for rendering. Do you think is a good system for rendering architectural interior and exterior scenes ( houses, offices, etc) with vray rt ?

  • Paul says:

    Have you used Red Shift on this setup yet? If so what are your render times on images like the ones you’ve shown?
    Im looking to build a render farm soon.
    Thanks
    paul

    • Hi Paul,

      I tried Red Shift but on 970 and than later on 1070 to compare against other GPU engines, but I ended up sticking with V-Ray RT.

      Render times for 1080p frames are usually in range of 2-6 minutes for our shots. It can go up to 10-15min for heavy scenes.

      Best,

  • Ivan Schuler says:

    Hello I want build a 4x GTX Titan Workstation. A GPU Workstation for rendering. Because I´m gone render everything with the GPU, how much CPU you would recommend for the new Workstation. icore 7 o xeon? Other applications that are not rendering are running very smooth with a normal CPU.

    Right now i have this setup:
    CPU: 2.66 GHz Dual Quad Core Xeon X5550 (8core / 16 threads)
    RAM:32GB 1066 MHz DDR3
    GPU: Nvidia GeForce GTX 770 3GB
    iOS: Win 8
    5 year old.

    Its ok for work, it just need more speed for rendering.

  • Vu Pham says:

    Hi, ever you render something like this scale before with this setup ? https://www.flickr.com/photos/phamduongvu/23293917054/in/dateposted-public/

    Love to study more

  • Rob says:

    Looking forward to hearing more details about using RT in production. Keep up the great work. Currently debating whether to build dual Xeon or quad GPU setup and seeking info!

  • Dennis Persson says:

    Hi Tomasz

    Is it possible to use this machine as an DR node from within Max or Maya with RT and the GPUs?

  • Dennis says:

    Hi Tomasz, is it possible to use this as a DR node from another workstation that doesnt have an RT compatible GPU?
    I was thinking of something like this in order to speed up lookdev and not having to send out frames individually to each node

  • I have z97 pro ac wifi 32 GB ddr3 ram dual GTX 970 4790K CPU if I upgrade to dual GPU to GTX 980 TI do you think I could take render in 15 -20 min ?

    • Dual GTX 980Ti Would be definitely way faster than 4790K. When it comes to comparing GPUs. You can test your render times on 970 that you have and use it as baseline. 980Ti would be around 60% faster. I will post in near future a little bit more details about using RT in production.

  • I do architectural design and I on my pc it takes around 30-40 min Im trying to understand if I should go for workstation pc or invest on graphic card 🙂

    Basically what Im asking is that which way is the fast way to take render dual chip set mother board and single GPU or single cpu +4gpu

  • can you pls tell me how long does it take to render something professional print out board ?

    • What exactly do you mean by “professional print out board” ? In general we render animations at 1080p if you check our vimeo page most of the latest clips were rendered on this farm with times per frame usually in the range of 1-10 minutes depending on the complexity. Sometimes it does go up to 20-30 minutes but that is quite rare.

  • Ohnet says:

    awesome! thanks for the anwswer.

  • Ohnet says:

    Hello!! nice setup, I m trying to build this atm. I a few titans and another rig that is going with 980ti.

    how you deal with the temps with all the cards stacked like that?

    what is the highest that you have.

    • Temperatures are in range of 78-84 usually. Fan speed is auto managed by GPU (default) and is going at 60-70% so it still has some room in case things will get too hot. It’s going almost non-stop for 3 months so far, so I would say it’s very stable.

      Just checked. Both nodes have up time at 22 days at the moment. Since last electricity malfunction 🙂

Leave a Reply