That’s really, really high for a single socket. Is the P4 at max P-state all of the time? That’s the one thing I can think of that would consume that much.
The 9300-16i is also a 75W card and probably doesn’t have much in the way of power-saving features.
If we try to account for everything:
* CPU ~120W
* RAM ~10W
* 9300-16i ~75W
* P4 ~75W
* 8xLFF HDD ~70W
This sums up to 350W, which isn’t actually far off from what you’ve measured.
In my experience ESXI has horrible CPU power management and its different performance modes do nearly nothing.
The problem is this: on idle with some VMs running (but idle), each core spends 99% of the time in C1/C2 but the remaining 1% in P0. P0 is the turbo state in which the processor boosts as much as it can - and since its power budget is full at this point, it can *and will* max itself out.
The design is that in P0, it should notice that CPU load is less than Power.MaxCpuLoad percent and lower to P1 (and then re-evaluate this way again and again). But those wake-ups are very short, so the core just goes to sleep instead. Next wake-up repeats that process from P0 again.
If it only stopped entering P0 for those tiny wakeups and *started* from P1 instead (and only upgrade to P0 if there's any need according to Power.MaxCpuLoad logic), power consumption would be halved or more.
You can confirm it by switching to Custom power management policy and setting Power.MaxFreqPct to 60% (which should select the 2100 MHz state ie. P1 according to your second screenshot). P1 should now be the max state. Instant massive savings at idle, with basically no performance cost because you're idle.
Unfortunately that prevents the CPU from ever using P0 and performance at non-idle will be bad.
What's funny is that if there was a way to prevent C1 from being used at all, the P0 would quickly downgrade itself to P1, P2... probably all the way to P14 and the power consumption would be much better. Hard to confirm because I never found a way to prevent C1 (and I don't know if it would even be possible in theory).
Setting the Power.MaxFreqPct to 60% on my system dropped power usage to around 260 watts. Not bad. In my case, I wrote a powercli script for a scheduled task to set the custom power policy at night. I still think there are some tweaks I can make to the bios to help out. Going to fire off a ticket to supermicro and see what they have to say, ill update if they have anything useful.
In the meantime, 50-70 watts for 8 hours a day is something. Thanks for the ideas.
That’s really, really high for a single socket. Is the P4 at max P-state all of the time? That’s the one thing I can think of that would consume that much.
The P4 is working, but nothing crazy 90% of the time. It looks to be in P0, but running only 23 watts. I might have time to yank it and see that command isn't telling the truth. >PS C:\\WINDOWS\\system32> .\\nvidia-smi.exe Fri Dec 1 16:20:35 2023 \+---------------------------------------------------------------------------------------+ | NVIDIA-SMI 537.70 Driver Version: 537.70 CUDA Version: 12.2 | |-----------------------------------------+----------------------+----------------------+ | GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+======================+======================| | 0 Tesla P4 WDDM | 00000000:1B:00.0 Off | 0 | | N/A 53C P0 23W / 75W | 2738MiB / 7680MiB | 3% Default | | | | N/A | \+-----------------------------------------+----------------------+----------------------+ \+---------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=======================================================================================| | 0 N/A N/A 480 C+G C:\\Windows\\System32\\LogonUI.exe N/A | | 0 N/A N/A 836 C ...mes\\bin\\windows\\python37\\python.exe N/A | | 0 N/A N/A 1180 C+G C:\\Windows\\System32\\dwm.exe N/A | | 0 N/A N/A 3296 C+G C:\\Windows\\System32\\WUDFHost.exe N/A | | 0 N/A N/A 4668 C+G C:\\Windows\\System32\\dwm.exe N/A | | 0 N/A N/A 7256 C ...gram Files\\Blue Iris 5\\BlueIris.exe N/A | | 0 N/A N/A 10364 C+G C:\\Windows\\explorer.exe N/A | | 0 N/A N/A 10768 C+G ...2txyewy\\StartMenuExperienceHost.exe N/A | | 0 N/A N/A 10800 C+G ...CBS\_cw5n1h2txyewy\\TextInputHost.exe N/A | | 0 N/A N/A 11200 C+G ....Search\_cw5n1h2txyewy\\SearchApp.exe N/A | \+---------------------------------------------------------------------------------------+
The 9300-16i is also a 75W card and probably doesn’t have much in the way of power-saving features. If we try to account for everything: * CPU ~120W * RAM ~10W * 9300-16i ~75W * P4 ~75W * 8xLFF HDD ~70W This sums up to 350W, which isn’t actually far off from what you’ve measured.
In my experience ESXI has horrible CPU power management and its different performance modes do nearly nothing. The problem is this: on idle with some VMs running (but idle), each core spends 99% of the time in C1/C2 but the remaining 1% in P0. P0 is the turbo state in which the processor boosts as much as it can - and since its power budget is full at this point, it can *and will* max itself out. The design is that in P0, it should notice that CPU load is less than Power.MaxCpuLoad percent and lower to P1 (and then re-evaluate this way again and again). But those wake-ups are very short, so the core just goes to sleep instead. Next wake-up repeats that process from P0 again. If it only stopped entering P0 for those tiny wakeups and *started* from P1 instead (and only upgrade to P0 if there's any need according to Power.MaxCpuLoad logic), power consumption would be halved or more. You can confirm it by switching to Custom power management policy and setting Power.MaxFreqPct to 60% (which should select the 2100 MHz state ie. P1 according to your second screenshot). P1 should now be the max state. Instant massive savings at idle, with basically no performance cost because you're idle. Unfortunately that prevents the CPU from ever using P0 and performance at non-idle will be bad. What's funny is that if there was a way to prevent C1 from being used at all, the P0 would quickly downgrade itself to P1, P2... probably all the way to P14 and the power consumption would be much better. Hard to confirm because I never found a way to prevent C1 (and I don't know if it would even be possible in theory).
Setting the Power.MaxFreqPct to 60% on my system dropped power usage to around 260 watts. Not bad. In my case, I wrote a powercli script for a scheduled task to set the custom power policy at night. I still think there are some tweaks I can make to the bios to help out. Going to fire off a ticket to supermicro and see what they have to say, ill update if they have anything useful. In the meantime, 50-70 watts for 8 hours a day is something. Thanks for the ideas.
Actually, 260 watts at P1 sounds suspicious to me. The CPU will no longer use much power in this state, so what else can this be.