New(ish) E3 Xeon build for Scope PCI cards

PC Configurations, motherboards, etc, etc

Moderators: valis, garyb

Post Reply
User avatar
valis
Posts: 7306
Joined: Sun Sep 23, 2001 4:00 pm
Location: West Coast USA
Contact:

New(ish) E3 Xeon build for Scope PCI cards

Post by valis »

Well, my aging 2001 era dual Prestonia (first p4 'netburst') Xeon rig is probably beyond ready for retirement, as it has been resuscitated many times. It also runs so toasty inside, that it apparently has fried all of my original PCI cards (although this took a decade or so to take its toll). Dawman was nice enough to offer to send over an extra ASRock Rack E3C236D4U Motherboard that he wasn't using. Originally I was going to replace the role that my 2008 era Harpertown dual Xeon fills and migrate that over to fill the 2001 era... but all of these machines are getting quite old, hot and heavy (nudge nudge wink wink).

This made me reconsider the ASRock board for now, and in its place I decided to get a similar class (and era) Xeon board that supports PCI slots instead. More or less combining two machines into one now, to make room for new things in here as well.

So here's my pivot plan for a new build around my also geriatric (but supremely reliable and beloved) Scope PCI cards:

Core parts list As this system is assembled from parts bought over time, you'll note that many parts were purchased based on cost/performance (on sale) rather than 100% parts selection.

Note a few things the Xeon+Supermicro AMI BIOS board lets me do:
  • Set 1 setting to toggle between full TDP (base speed all cores), full TDP (turbo on all cores, not peaking core(s)), and full turbo (enable higher speeds with less cores)
  • Completely enable and disable modern power saving controls and sleep states
  • ECC ram and BMC+IPMI will log errors with onboard components into the EFI and readable by 3rd party software
  • Control whether specific peripheral devices are enabled via EFI, older/compatible BIOS or both
  • The m.2 slot is compatible with a much wider array of devices
  • No onboard sound and the onboard video can be disabled via jumper
  • all other onboard devices easily disabled via EFI/BIOS, jumper or both
  • 4 different PCI slots off of a single PCIe bridge
  • Allocate IRQ, power levels, speeds and compatibility states for the entire PCI & PCIe spec for ALL PCI slots, PCIe slots, NVME/M.2 and so on.
It's worth noting that rather than getting the Lian Li case I had originally intended to get, I found a gaming PC case and watercooling kit on sale (separate sales) that combined saved me about $120 on the Noctua/Lian Li setup I had originally planned. It does have silly LED everywhere, and a perty window to show off the green PCB's that I'll stuff into this everywhere (motherboard & Scope cards) but hey, at least I can show my rig during a Twitch stream without seeming completely unfashionable. I also have become a fan of the all-in-1 cooling units over the last few years, the failure rates are way down and the ease of installation has gone way up (compared to DIY cooling solutions). And almost all modern cases (gaming & workstation) come with multiple mounting options for the reservoirs such that any reasonably sized case can really house 2-3 all-in-1 units for both the CPU and the GPU(s). Anyway, we'll see how these hold up, I expect the fans to get noisy within a year or so and need to be upgraded anyway.

I also will have 2 spinning HD's in this box, which are pulled from another working build that I have repurposed into a gaming rig for my wife, but those are already on hand and not listed. So in total I will have the following drive configuration:
  • 1TB NVMe boot/app drive.
  • 500GB "projects" drive (samsung 860 EVO SATA SSD)
  • 120 GB "samples" drive (HP SATA SSD, this is NOT my primary daw rig so this is plenty)
  • 2TB WD Black SATA HD (1TB partition for storage & more samples, 2TB partition for backup volume to image the 1TB ssd)
  • 2TB WD Black SATA HD (storage & archival for all installers used on this system which also get mirrored to a RAID NAS volume that has every installer for all machines in here going back 2.5 decades)
I haven't yet made it through the entire build, as I am doing it during spare time between family & client duties, but I will be finishing up this week and will post results as I go. It might take longer to get the Scope PCI cards in place, and I plan to send them to our new Capacitor Service resource to check all caps and anything else he's able to check/fix while he's at it. This latter part I'll likely do one card at a time, but once done I hope to get another 1-2 decades from these PCI cards AND will likely still get an Xite should my work prove fruitful enough to support that.
User avatar
Peter Drake
Posts: 214
Joined: Thu Nov 08, 2018 12:27 pm
Location: Outer Milky Way

Re: New build for Scope PCI cards

Post by Peter Drake »

Interesting. I assume from the drive sizes that this will be a 32 bit OS? Also, any criteria other than price and availability for the graphic card?
User avatar
valis
Posts: 7306
Joined: Sun Sep 23, 2001 4:00 pm
Location: West Coast USA
Contact:

Re: New build for Scope PCI cards

Post by valis »

Win10 64bit will be the OS, though I'll have to refresh my recall on what doesn't work under Scope in 64bit. STS isn't of interest to me for years, but I know there's a specific (very metallic) delay/mod plugin I'll miss and a few Modular modules I'll have to remove.

In regards to the GPU
, it supports 4K on the HDMI output and will still handle any opengl/directx related drawing commands for web browsers and plugins alike. The main criteria was power draw, but keep in mind I was building a non-PCI slot Xeon for a different purpose at the start of this. Ie, the machine I had originally planned to build was going to inherit the Quadro card that's in my older (2008 era Harpertown) dual Xeon, which sucks about 1300 Watts under a full render load.

Interestingly, this 2017 era E3-1285 V6 Xeon actually only comes in slightly above the other machine in terms of multicore performance for the apps that I was concerned with (3dsmax, maya, etc), and though the ram speed is faster the older machine has 4 channel ram AND 8 banks (interleaved this really cuts latency, and so the DDR2 era ram is about 2400 equiv for many tasks). It's really surprising how well the older 2008 era machine has held up, but it was the peak of the Core2 era CPU's after all. As such it lacks much of the power saving that was bolted on starting with Nehelem era, and so in theory was a better match to Scope.

It was for this reason that I was going to move the 2008 harpertown Xeon over to the Scope role (it has 4 32bit & 2 64bit PCI slots) and reduce the CPU speeds, amount of ram, and use this low power GPU in that case (if you'll pardon the roundabout way we got to the point here). That would have allowed me to cut the PSU's in that box down to the one I'm using here, 650W, whereas the box currently has both a 960W & 750W (PC Power & cooling units). I know from experience that dropping the FSB speed to 1333 (instead of 1600) and using 2.6 or 2.8 Ghz chips (which I have on hand already) cuts power usage and thermals dramatically. Why? Well I've fried 2 northbridge chips on those boards (this is my 3rd, and I still have an extra in the closet if it had failed too) due to how much power the RAM and CPU's suck under a heavy load, and that box was a workhorse for many many years. This of course was after 6-12 months of constant render usage for things like After Effects, Nuke/Fusion and Autodesk 3d apps.

It can no longer handle any of those roles, so it's been largely a storage machine for about 3-4 years, and does backup render duties (things like matte passes/alpha channel/depth mask etc run great due to their simplified processing). But it bottlenecks severely with realtime tasks (4K video editing, compositing and realtime visuals at anything above 720p) and is far too old for playing games. Which is why I have other machines of course..

In any case the original plan was to build the ASRock box to replace THAT, cut things down and put Scope in the 2008 era box where I can disable EIST & etc, and use the newer box for 3d duties. As luck would have it, my big 3D project for this winter was just rescheduled until January anyway...and so the 'shift' to making a Scope box seems to be timely for a few reasons. I am currently rethinking whether this box should stay in service at all, but I have so much inventory to service and maintain it still that I'll have to see if any of it has secondhand value.

In the long run, I will finish this new build and I have another AMD build to replace that Xeon rig already started as well. This will STILL cut my power usage significantly in this room, boost performance by a few magnitudes (for billable work, this is good) and cut heat/noise in here significantly which is my main goal. That should put me on good footing for a few years, which currently seems wise as it's hard to know what we'll face right now in terms of both tech costs and supply of goods in general.

----------------

As for the current build, the core of the build is mostly done today, I pulled the ASRock board on Saturday and mounted the new one, re-mounted the PSU and SSD's yesterday and spent a good deal of time today routing the wiring on the backside of the motherboard tray etc. Not the greatest case, and I expect the LED fans to look silly enough, but overall seems like airflow will be up considerably for the PCI cards and this has enough filters to make the dusting easier too. Will post images and results as they come...

Also for reference not that while the Xeon CPU's for V5/V6 era are too old for Win11, these motherboards do have TPM 2.0 headers and accept aftermarket compatible modules. So I may very well still add that to enable secure boot before finalizing the box (even if that means I stay on Win10). If I do that after already doing a base install to validate all the hardware, I'll clean the boot SSD with a few commands like dd/du (and achieve something similar to the clear all command in diskpart, but this I will do from a freebsd install) and ensure the boot sector is solid and the drive is all 0's, then start over with the TPM 2.0 enabled installation. I haven't done a modern secure boot with bios & boot sector encryption yet, so that might be interesting to learn about even if it's not necessary for Scope itself.
User avatar
valis
Posts: 7306
Joined: Sun Sep 23, 2001 4:00 pm
Location: West Coast USA
Contact:

Re: New(ish) build for Scope PCI cards

Post by valis »

Understand now why the ASRock board I started this project with wouldn't boot. I need dual rank with different chips, specifically any DDR4 that offers the option for *higher* than 16GB sticks (in the same model range) uses chips that are higher density and don't work without a buffer/registered socket in this era (which this board does not have/support). Ie, the 2 sticks I bought were 'low profile' but also came in 32GB & 64GB capacity, and so the 16GB is single sided/single rank with a higher density chip and the boards either don't recognize the chips, don't support the signaling, or both.

Already ordered 2 new sticks from the SM QVal list, and will check bench booting the ASrock board with them as well when I get them. Should be here Friday. Checking the QVal list for SM shows the older low profile chips are now deprecated and gives a newer part instead.
dawman
Posts: 14368
Joined: Sun Jul 24, 2005 4:00 pm
Location: PROJECT WINDOW

Re: New(ish) build for Scope PCI cards

Post by dawman »

Lots of IT guys here with experience on ASRock/SMicro.


https://forum.level1techs.com/c/hardwar ... rboards/30
User avatar
valis
Posts: 7306
Joined: Sun Sep 23, 2001 4:00 pm
Location: West Coast USA
Contact:

Re: New(ish) build for Scope PCI cards

Post by valis »

Awesome, thanks. My old resource (2cpu.com & an irc channel that was unrelated) are gone for some years now.

New RAM arrives today, will be able to test that and perhaps build out the first Win10 OS install attempt by this weekend, but Scope cards may not follow for another week or so...
Berny Shoes
Posts: 77
Joined: Fri Dec 21, 2018 3:22 am

Re: New(ish) build for Scope PCI cards

Post by Berny Shoes »

dawman wrote: Tue Oct 26, 2021 7:04 am Lots of IT guys here with experience on ASRock/SMicro.
https://forum.level1techs.com/c/hardwar ... rboards/30
Thank for this link 👆
User avatar
valis
Posts: 7306
Joined: Sun Sep 23, 2001 4:00 pm
Location: West Coast USA
Contact:

Re: New(ish) build for Scope PCI cards

Post by valis »

Both motherboards like this ram, so single rank 16GB ECC sticks are no go (higher density chips) but dual rank 16GB ECC works great.

Supermicro motherboard BIOS was updated (skipping the ASRock until I do that build), an oddity here was the fact that the onboard video (not IGP, but a low end VGA chipset for BIOS/EFI/IPMI usage in server or remotely managed scenarios) was the only one that would work for a DOS disk. One that was taken care of, a few BIOS modifications needed (mostly to disable remote IPMI wait and PXE boot on both NICs) and then it's off to install Win10.
User avatar
valis
Posts: 7306
Joined: Sun Sep 23, 2001 4:00 pm
Location: West Coast USA
Contact:

Re: New(ish) build for Scope PCI cards

Post by valis »

Now that I can get into the EFI/BIOS, the only other thing I've experience so far was the Optimized Defaults changed the nvme port to default to an onboard firmware loader rather than via bios (raid cards etc). Setting it from "Third party provider" (the default) back to AMI firmware was the answer, which makes sense as the default option is probably more sensible for the type of devices this board typically houses (the board supports the largest m.2 socket size which would not be for consumer devices).

In both instances, the defaults and ram issues wouldn't be a concern if this wasn't a Xeon board, and so most of this is probably anecdotal to anyone reading along.

Win10 is installed and updating, and I'll have things stripped down by tomorrow (no cortana, weather taskbar crap, O&O/WIn10Privacy and various other things removed). Scope cards soon after, will report again.

Oh, and the cheapo RGB controller in the case already failed, losing all but 1 port (port 4). Sale items...nice air movement and intake filtering and I won't likely allow the iCue software to stay running once things are set. I am in 2 discord servers for open source projects that attempt to play a 'unified' role in addressing RGB devices on the market, and I'll figure out which of those allows me to set low level ambient lighting and impacts the runtime overhead for Scope the least (verifying dpc latency and polling frequency etc). Such things don't really matter in Scope contexts, but when I was running visuals at shows being able to integrate my rig with the lighting would have been cool.

Now that consumer penetration of the RGB crap has reached the point where you can address your facemask and your headphones while streaming on Twitch...well might as well join in the dystopian fun? Or at least the 40% off sale case presents an opportunity to tinker.

Here are the projects I was referring to in the event anyone is interested:
https://rgbsync.com/
https://www.project-aurora.com/
https://github.com/DarthAffe/RGB.NET (another option, not as current)

I typically run Ableton synced to Vezer, Resolume and other software on stage (all 3 talk midi, osc & dmx--though Live needs Max to do this) so having something I can at least scene change with isn't a bad idea.
User avatar
valis
Posts: 7306
Joined: Sun Sep 23, 2001 4:00 pm
Location: West Coast USA
Contact:

Re: New(ish) build for Scope PCI cards

Post by valis »

Corsair will ship the RGB hub (for those that aren't falling off their chair laughing yet) at no cost, so no major loss there.

The system is now running at full speed, and I've returned the excess parts via ebay/amazon etc. All is well, and I'm archiving things from my old Scope box again just to ensure I don't lose anything (most if not all is already on a newer machine and an external drive or two). Once that's done I'll see about the Scope cards.

Once this is stable, it's time to give the Scope cards a once over from our friendly refurbisher.
User avatar
dante
Posts: 5043
Joined: Sat Nov 24, 2001 4:00 pm
Location: Melbourne Australia
Contact:

Re: New(ish) build for Scope PCI cards

Post by dante »

valis wrote: Sat Oct 30, 2021 5:18 pm Once this is stable, it's time to give the Scope cards a once over from our friendly refurbisher.
Who dat ?
User avatar
valis
Posts: 7306
Joined: Sun Sep 23, 2001 4:00 pm
Location: West Coast USA
Contact:

Re: New(ish) E3 Xeon build for Scope PCI cards

Post by valis »

User avatar
valis
Posts: 7306
Joined: Sun Sep 23, 2001 4:00 pm
Location: West Coast USA
Contact:

Re: New(ish) E3 Xeon build for Scope PCI cards

Post by valis »

Ready for retirement after 20 years, this Supermicro P4DC6+ housed my DSP cards reliably for a long time, but it’s time for new digs.
Attachments
Supermicro-64DC6+.jpg
Supermicro-64DC6+.jpg (471.42 KiB) Viewed 4590 times
User avatar
valis
Posts: 7306
Joined: Sun Sep 23, 2001 4:00 pm
Location: West Coast USA
Contact:

Re: New(ish) E3 Xeon build for Scope PCI cards

Post by valis »

Cards in new build, I think my primary key card is flaky though as with it in the system is taking a very long time to post, and Scope launches reliably only about 75% of the time. Cold boots especially seem to be an issue, which is what I noticed with the last cards that went south. Might be time for a refurb visit for each card...
Attachments
Supermicro_X11SSA-F.jpg
Supermicro_X11SSA-F.jpg (420.22 KiB) Viewed 4589 times
User avatar
valis
Posts: 7306
Joined: Sun Sep 23, 2001 4:00 pm
Location: West Coast USA
Contact:

Re: New(ish) E3 Xeon build for Scope PCI cards

Post by valis »

Btw I see what you mean about the onboard video now Dawman, on the Xeon boards I think that's an especially smart route in the future. Sadly the two variants of this board that connect the IGP to external headers aren't on newegg, so i opted for this. Quicksync will still work with the IGP and other things, but the BMC connection to the VGA chipset onboard only does passthrough to the IGP for certain things, which is why I opted for a low power GTX130.

However I am now wondering if a low power GPU that doesn't draw from the PCIe slot might have been better, the card only draws 25-30W and the slot is 8x only. PCIe only specifies 75W delivery for 16x, 4x & 8x are both 25W. So while I don't think I'll ever hit 30W anyway, it might have been better to find a 1 slot card that has a 6pin power. Many PSU's don't give as much 3.3/5V (or derive it from a lesser 12V rail), and so allowing the GPU to pull from the 12V that any PSU will have in plenty in this era might be a good idea.

In any case, other than realizing that dual rank vs. single rank is still an issue in the modern era this system was very simple to build. Lower end case and integrated cooling parts are quite capable now, so that was interesting as well. The system's noise footprint is so far below that old box, I can't wait until I can replace my other big rig (my 2008 Harpertown era dual xeon that has sat next to the Scope box since 2009). At least the whine of high rpm scsi drives is finally gone for the first time in literally decades (I've used SCSI since about 95/96).

I have my head around most of the BIOS options, but there are also a few power related settings for the CPU (and turbo states) that I will need to learn. The power limits for the package dictate how long states can be active, and while I don't care about Max turbo I will be doing some DPC/system latency tests to see if all-core turbo vs single core turbo (higher speeds) makes a difference, for example. Or higher overall package TDP for all-core turbo (4 core 'max' speed without single core turbo) vs. keeping it at stock speeds, and so on. Ie, I will stress test balancing cpu load against PCI/PCIe availability so I know how far I can push GPU/gfx vs. PCI traffic (Scope) vs. DAW/bidule/cpu load etc.

Moving forward, I am quite interested in seeing if I can use Kron to control VCV Rack and other modulars, and then extend that workflow into the hardware world...
User avatar
valis
Posts: 7306
Joined: Sun Sep 23, 2001 4:00 pm
Location: West Coast USA
Contact:

Re: New(ish) E3 Xeon build for Scope PCI cards

Post by valis »

11 Masterverbs (9 MV Pro) before PCI limit reached, currently running on two 6 dsp cards.
User avatar
dante
Posts: 5043
Joined: Sat Nov 24, 2001 4:00 pm
Location: Melbourne Australia
Contact:

Re: New(ish) E3 Xeon build for Scope PCI cards

Post by dante »

Looks impressive.
User avatar
valis
Posts: 7306
Joined: Sun Sep 23, 2001 4:00 pm
Location: West Coast USA
Contact:

Re: New(ish) E3 Xeon build for Scope PCI cards

Post by valis »

Impressive thanks to Scope, the case, PSU, SSD's and cooling parts (inc. the Fans and the Corsair closed loop cooler) were all from sales. Cooling has improved dramatically over the years, plenty satisfied with this build.

3rd card is back in the Xeon system system (2 of the 3 I'm running are refurb'd at this point). I get 11 Masterverbs is where I hit the PCI limit every time, 10 seems to be the limit for this box without more severe tweaking.

For reference I still have hyperthreading enabled, all cores states enabled (max turbo & all sleep states) and I have (for now) largely rolled back all of the exotic tweaks we typically do to systems to dial this in one by one. My guess is that the main issue will be the PCI latency setting (which is still exposed in this BIOS), slot sharing with the primary card (none right now) and how much I load up on the host/ASIO side vs. Scope.

So far I am sure I can go well beyond my old system on the host side, run modern tools, and look forward to helping Sonic|Core iron out their 64bit issues so that we can have another decade ahead.
User avatar
valis
Posts: 7306
Joined: Sun Sep 23, 2001 4:00 pm
Location: West Coast USA
Contact:

Re: New(ish) E3 Xeon build for Scope PCI cards

Post by valis »

PCI limit proof, and again I am probably able to improve this with performance related tweaks in the EFI (BIOS) that I reversed for the time being.
MV10.png
MV10.png (190.62 KiB) Viewed 4222 times
Post Reply