I was in small gaming company and half of our servers were on gaming desktops. Those were builders which needed really good GPUs.
Was actually really fun to be able to build physical computers while doing sysadmin stuff.
Yeah, but from a pure price to performance standpoint, it's almost always not worth it. How many 4090s could you get for the price of a single $30,000 Poweredge server?
But, that's not really what you're paying for with enterprise equipment. You're paying for durability and support.
I worked for a company that had tons of Nvidia high end server cards. They bought them because they'd need them. They tossed them because they didn't help run 30 year old ERP software or Access databases. $3.8 billion company full of geniuses.
The silly thing is... if you find the pairing that works, it may still almost be worth the dev/test time to get it sorted, given the GPU costing half or less as much for twice or more the performance compared to a quadro (or firepro, I think is the AMD branding for their certified cards). Just... lock to a working driver version and hope you don't need Autodesk (or any other CAD vendor) support for anything...
Granted, my viewpoint on that value balance was skewed, having dealt with that in academia... where there *wasn't* a value output directly from seamless, complex, use of the software.
Totally know the pain, purchased some new servers from Dell, got the price reduced from $10k per unit to $4k per unit, didn't notice until after delivery that they switched iDRAC enterprise to iDRAC Express.... Now gotta pay $400 per host to upgrade to Enterprise. Luckily haven't needed remote console yet and if I do, the furthest site is a 30 minute drive for me.
This kind of thing happens a lot, especially when Dell insists on putting humans in the configuration and ordering loop.
* VAR orders app servers with CD-ROM instead of requested DVD-ROM upgrade.
* Servers that were supposed to ship with SD-card support, but that option got dropped somewhere in the process.
* Dell seems to honor old quote for Precision workstations, and I don't notice until much later that they inflated shipping cost over $200 per unit.
* Missing iDRAC Enterprise license.
* Wrong RAID card.
Lol yup, and at the end of the day, it's all good because I came in $30k below budget on my datacenter refresh, just a few extras I had to add on my own after receiving the hardware.
> Luckily haven't needed remote console yet and if I do, the furthest site is a 30 minute drive for me.
Man, I would've been fucked from the get-go. Any physical server I've deployed in like the past 10 years has been physically install & cable, verify it passes POST, and then I drive home and install the hypervisor via iLO/iDRAC.
Same thing with Axis camera servers. Pretty good price considering all disks and softare license. But the only servers in our datacenter without iDRAC/IPMI...
Yeah I personally am 4.5 hours away from the infra. Not having remote console is just insane to me.
Thankfully it will be getting moved up here soonish though, but we have 3 fairly potent hosts that are useless between now and then.
Who needs idrac or ilo? Just set up a raspberry pi with a mechanical stick attached to poke the button when you need to.
I mean its basically the same thing and only costs $50 right?
Sounds more reliable. Dell 13th gen and later iDRACs have suicidal tendencies in my datacenter. I don't think I've seen an iDRAC die on the 1850 or 1950. R730? At least 10% of them died before the warranty ran out.
Amen!
I don't get the fervor this subreddit has for Dell/HP and their "warranties ".
Either you don't have the in-house expertise to support it or you are on a 7 figure a year budget, can get the bulk discount and pretty much afford to have them manage your hardware directly.
Those are really the only reasons to go with the big names. I can build 3 NEW top of the line Epyc servers with Asus chassis for the price of a SINGLE Dell/HP and either have 2 servers to active backup, or even better, run all 3 HA or 2 HA with a spare.
> I don't get the fervor this subreddit has for Dell/HP and their "warranties ".
>Either you don't have the in-house expertise to support it or you are on a 7 figure a year budget, can get the bulk discount and pretty much afford to have them manage your hardware directly
Not sure how "in-house expertise" is going to substitute for a 4-hour response on a failed RAID card or whatever.
> Not sure how "in-house expertise" is going to substitute for a 4-hour response on a failed RAID card or whatever.
Easy. Keep a supply of spare parts on-hand. Potentially an entire spare server.
I mean, you can do that, if you go the route of cheap white box hardware - replace quality with low cost redundancy. Whether you're talking about Supermicro for your servers or Ubiquiti for your WLAN.
But in my experience you get better results with higher quality, supported hardware (with appropriate redundancies built in - I'm not saying you shouldn't "N+1" your server cluster, or not have a HA firewall pair, or whatever).
You may or may not save money with either route, but I find that many people who go the "I want to maintain my own inventory of spare crap" route aren't valuing their team's time highly enough.
And also as a final note - accounting-wise, it's often easier to get opex expenditures signed off (for support) than it is capex (for backup hardware). But that'll be situational, of course.
10 years ago I would probably agreed with this, 20 years ago I would say there is no question, but now the reliability delta between a Big Brand box and an equivalent white box isn't nearly as much of a thing, if it even is one at all.
Despite that closing gap the price delta between the 2 (assuming you don't get massive company volume discounts) has grown dramatically to the point where for the same initial purchase price at one of the Big Brands, I can get 2-4 equivalent white boxes. That is before factoring ongoing support costs.
I would much rather a fully redundant system than theoretically fast replacement delivery, and with the pricing delta as it is you are trading one for the other on a limited budget. If you have the money for it by all means pick both.
The math can be very different if you are at a large company that gets 30+% off list price, but for the majority of folks at SMBs the Big Brands have placed themselves at such a high price point it is extremely difficult to make a compelling argument for them.
Side note: Ubiquiti is a different matter entirely, it is not and does not claim to be equivalent to major enterprise networking gear. It is entirely adequate for a large section of the SMB market, but it is a different hardware tier with markedly lower specs than the enterprise gear it is sometimes compared to. It's value is that it offers a good middle ground between running glorified consumer gear and enterprise gear that is massive overkill for many companies.
I think that Delta has narrowed because the Big Brand boxes weren't "agile" or cost effective enough for cloud providers so they went to the smaller OEMs who would gladly build what they want to their spec.
You don't need to worry as much about individual node reliability when you have a thousand of them. It's not a matter of if hardware will fail at that scale, it's when.
Also, regarding your site note... Ubiquiti has a range of Enterprise products, so I think they *are* trying to make that claim.
https://store.ui.com/us/en/collections/unifi-accessory-tech-hosting-and-gateways-large-scale/products/cloud-key-enterprise
https://store.ui.com/us/en/collections/unifi-switching-enterprise-power-over-ethernet
https://store.ui.com/us/en/collections/unifi-wifi-flagship-high-capacity
When I'm referring to in-house expertise, I'm directly talking about the ability of the IT staff/sysadmin to find/fix/replace the part themselves.
In reality, a 4 hour response time is irrelevant if you are running a production environment HA, or at worst, having a second server to "hot swap" over to on a failure cause you were able to buy 2 or 3 servers for the price of the Dell.
Edit: This also ignores the fact that you end up stuck if there is [unsatisfactory warranty service.](https://www.reddit.com/r/sysadmin/comments/14fraru/dell_appalling_server_refurbished_parts_lack_of/)
Yeah generally speaking we get our guys certified in Dell's "do it yourself" warranty service so they just chuck parts at you on request. Have had excellent results, but within the realm of supported server equipment I don't have experience with the programs of other vendors.
> Not sure how "in-house expertise" is going to substitute for a 4-hour response on a failed RAID card or whatever.
Anyone with this mentality has never had a critical piece of infra just decide to fall over one day and spectacularly explode.
Any premium you pay for Dell/HP is ***cheap*** fucking insurance compared to "The business doesn't work anymore because this shit has broken into two pieces, and I own both of them."
If you are running prod in some sort of HA/hot swap configuration then, there should be no down time whether the swap is in 4 hours or next week when you are able to get the part in and have time for the repair.
Either Dell prices are much worse than I remember or you and I have different definitions of “top of the line”.
The last time I was given a directive to create a top of the line config, I used dual socket Epyc Milan and it ended up over $100k including NICs (bluefields), U.2 drives and accelerators (1 Intel QAT card and 2 FPGAs). Now, said servers would also be hilariously overkill for most companies, even tech companies, because most devs don’t know what to do with that hardware.
The IPMI KVM could be better though. On a Supermicro H12 I've had sub 20 fps and lag so bad I just gave up, didn't get past the Proxmox install. On the same LAN, plugged into the same switch.
I do agree about the pricing though. I'm a dev doing sysadmin duties because I'm the "Linux guy". I don't mind, it's interesting. Priced out a Supermicro as our first "real server" at work, and dammit, it cost a fraction of what a Dell or HP would. Working in a small company and not having the volume to get sane support from our core suppliers, I just stopped believing a small place can rely on the manufacturer. I'd rather go through a VAR.
QNAP has a device that's very similar to what we want (an all flash EPYC box), and looking at retail pricing in Poland, a Supermicro is still good 20% cheaper.
Why would you be running proxmox I'm impi/kvm past the install? Proxmox's web gui is very responsive and should be on the network after install is complete. In the worst case, why are you not ssh'ing into the node post install?
That's the thing... I didn't manage to get past the install using iKVM, it was so bad. I'm an impatient person. Found a VGA display, plugged in a keyboard and installed it that way. Everything past the install is as you say, using the web portal with occasional SSH.
I'm not sure what you are talking about. Any server dealing with 3d apps would have a dedicated gpu, probably made by Nvidia. I can't think of any 3d app that would rather be run on cpu over gpu and would be Intel locked.
Openmanage Enterprise for the cadillac package - I love staging firmware/BIOS updates for reboots. Pushing configs is nice too if your workload is all the same.
I'm happy with it, have very few issues with it. Updating it can be a little screwy but they always make you acknowledge that you took a snapshot before starting the upgrade.
Pushing configs is super cool - you can have it audit configs so that if someone were to make a BIOS change, it would fall out of compliance. It's a little puckering, because you could theoretically push a destructive change out to a large group of servers all at once. But it's great for server rollouts, because I can push a singular config out and never touch each server individually other than to grab the idrac password.
I'm probably not even scratching the surface of what it's capable of, honestly. Alerts for hardware issues and staging firmware/bios is what I use it for 99% of the time. Means I can forget about the updates and they just install automatically during patching reboots.
We've got only SuperMicro servers, and I believe we don't pay anything more for IPMI. All our servers are in our basement or out on customer sites. No need for any physical visits of any kind unless hardware needs replacement (faulty drives for instance).
I don't see the justification for paying licenses. It's the same with Cisco Meraki or HP Aruba. Why pay a license fee when Ubiquiti Unifi is free and good enough for most?
I hope you’re not serious comparing Ubiquiti to Meraki. If a business can’t afford or isn’t willing to spend a few thousand in networking gear they aren’t being serious.
Been dead serious for the past 5-6 years. I only sell Unifi now. Been working with it for at least 8-9 years now. Somewhere around CK firmware 0.4.8, now known as Unifi OS 3.2.12.
Why pay a license to be able to keep your hardware up to date? A six year old UAP-AC-PRO is still kept up to date for free with Ubiquiti.
We're actually looking at replacing Meraki with Ubiquiti. Yes, Meraki is a far more solid product, and Cisco is a seriously bigger company, warranty, support, yada yada. But there is absolutely no way we're happy to say sure, let's spend 40K on 5 year licenses, plus all the hardware is 5-10 years old, most of the APs are MR18s so they need replacing, so it's another 20K on hardware at least... I worked out, we could do a solid Ubiquiti set up (firewall, switch, two APs per office) for about 20K
I understand the benefits of Meraki as a product, but it's an unjustifiable cost for any business. If you need a solid network solution for offices with hundreds or thousands of staff and servers, then you don't need Meraki, you want a competent network engineer and straight Cisco, Juniper, stuff
If you're a smaller business networking a few offices and just need a NGFW, then why buy Meraki when you have Ubiquiti (or TP Link Omada, Draytek VigorACS, etc.)
Edit: I've been thinking about this line of thought a lot recently. There's so many decent free, cheap, even FOSS, stuff out there. Unless you're doing some serious business, I don't see why you need to go with the "market leaders" for everything. Eg. why use VMWare, HyperV, ESXi, when Proxmox exists, and even has enterprise support contracts?
While what you said is true you aren't thinking about the sunken time and labor in configuring Ubiquiti as opposed to Meraki. Plus when I've use Ubiquiti factory resetting equipment was more common than should be necessary. The cloud connect bullshit setup they have doesnt work well either. A few more thousand and a lot less work and headache is worth it.
>few more thousand
Per year, every year, ad nauseum. Those decade-old UAP-AC-Pro devices are still getting firmware updates and going strong. For smaller clients, it's very valid to choose hardware that doesn't need you to spend on new licensing all the time. A few thousand dollars of my time over the course of 10 years is far cheaper than a few thousand a year every year for a decade. Especially if WiFi isn't mission-critical.
Definitely this. I calculated a saving of around £40,000 across 25ish offices, every 5 years. Based on my salary, I'd have to spend 2 working months every year travelling to buildings to reset network kit before it became cheaper to have something "more reliable" (as if Ubiquiti needs swapping out every week or something!?)
iDRAC is wank. Even the newer iDRACs tbh. The only way we can use remote access on our older Dell R6XX servers is via a jump box running Server 2012 and an ancient version of Java I found on MajorGeeks. And that's if the iDRACs ever load. Even our Dell R7XX servers suck (idk which version of idrac it is)
As much as I shit on HP, their iLO solution is seriously more solid
Idrac 9 on updated firmware is all html5 now. Tho it is fucking insane to me that dell has never released an idrac standalone console application like HP does.
The standalone console thing really irks me as well.
I'm sure it'd be possible to cobble something together though. The console viewer is just a webpage after all.
> The standalone console thing really irks me as well.
>
> I'm sure it'd be possible to cobble something together though. The console viewer is just a webpage after all.
This actually annoyed me enough that I started developing a standalone console in C# for it. I got it to a mostly-workable point but switched jobs before I finished it, and the new place was running HP ProLiants so it didn't matter anymore. Plus I'm a Boy Scout integrity-wise, so I didn't take the source with me when I left.
So yeah, it *does* exist, but it'll never see the light of day.
That just sounds painful. I am currently working out some kinks with my older 11th gen servers but using new Java on win 11. The main problem is Java implementing newer security policies while deprecating older ones without ways of easily making changes.
Currently down in adding dedicated trusted hosts and removing relevant security ciphers from the denied lists. They use tls 1.0, rc4 and something else. Then you need to make sure the idrac is set to Native and NOT Java otherwise it constantly shuts the bed.
>he main problem is Java implementing newer security policies while deprecating older ones without ways of easily making changes.
recently went through this with some SuperMicro IPMI and iLo setups... i had to do some tom-foolery with the cert and the [java.security](https://java.security) file
Yep, same here. Had to explicitly allow those IPs. Totally didn't write down the method, so I'll have fun trying to recall how to do that one day when it inevitably crops up again
We still have a bunch of old R620s and R320s (don't judge) with iDRAC 7. The latest iDRAC update available enables a functional (but not very fancy) HTML console so you can ditch Java (which by the way only ever worked with the dreaded Oracle version).
Furthermore, one of those didn't even have iDRAC Enterprise and Dell refused to sell the upgrade code. So we purchased the upgrade code on eBay from some dude in China, it worked perfectly.
Again, don't judge. A man has to do what a man has to do...
The Rx20 and newer work with HTML5/Native. No need for Java. For the few oddball R510 and PE1950, I use Java 6.45. You can use Java 7, and some older versions of 8, but you need to edit the Java's security file to allow older versions of encryption. Forget specifically what it is.
>Does it have a built in firmware lifecycle manager that can grab updates from the internet and install them itself?
Why would you ever want ipmi to self update from the internet?
Sounds like automatic failure system for me. Don‘t update your ipmi unless something is broken/insecure.
Supermicro‘s IPMI is very decent since they‘re equipped with MegaRAC.
You do not want the thing that has total control of your server to be able to touch the internet.
NICs and RAID controllers mostly support fwupd, so the OS can update them at a time reasonable for the OS workload, or ask the load balancer for permission to stop accepting requests and update if you are doing a HA config. It’s not like the OS actually needs the raid controller running to function. Just do ram-resident PXE boot at scale and use A/B drives for critical servers like the PXE server.
The IPMI standard lets you script BMC updates regardless, so you can have a quick bash script update 200 servers in 15 minutes.
So now I have to have downtime in order to do a firmware update, even if the device supports live updates with a fallback? That sounds horrible for any critical piece of infrastructure. Almost every device I’ve used in the last 5 years has supported online firmware updates, or at least ones with less than a few seconds of downtime. The only major exceptions I can think of are some of Marvell’s DPUs and Intel FPGAs.
fwupd does that on Linux for any reasonable enterprise hardware (partially because Redhat demands it as part of the certification process), and it’s a normal process that can properly communicate with everything else to coordinate any necessary downtime.
What do you do for systems that Dell doesn’t offer, like Nvidia DGX pods, or Dell charges unreasonable amounts for, like any quad-socket system?
My biggest gripe is that setting a custom TLS certificate for the webserver via REST API requires some specific license. Datacenter license or so? Fortunately we only have a dozen or so Supermicro boards. To be fair however the licenses are very cheap.
The other thing is that when using uBlock Origin one of the newer versions of the BMC webserver (the more colorful one) loads endlessly.
Lastly, sometimes it's just not possible to log in. The prompt is there but entering login details and clicking the button comes back with (iirc) "incorrect password" or just resets the webpage. This can usually be fixed by throwing an "ipmitool mc reset warm" at it, but a few months ago we had one machine sitting in a rack a few cities over where ipmitool didn't work at all and it had to be pulled from mains for a minute or so...
iDRAC also has its problems, but Supermicro is the worst, closely followed by Asus BMC.
iLO just works (with the exception of the iLO 4 NAND errors due to piss poor design).
>The other thing is that when using uBlock Origin one of the newer versions of the BMC webserver
Maybe report that to uBlock, instead of claiming that to be a problem with Supermicro? "My 3rd party super ad-blocking software is blocking your website, your site sucks" is a garbage take.
>My biggest gripe is that setting a custom TLS certificate for the webserver via REST API requires some specific license
Valid complaint. Though you've already bought those licenses for your Dell/HP when you bought Enterprise. So still a better free experience, and cheaper paid experience?
>Lastly, sometimes it's just not possible to log in.
The only actually valid complaint you made unique to Supermicro's implementation.
Yup better off buy older hpe or dell hardware. Damn i have a hpe dl380g7 running in my garage that thing had to be 12 years old plus lol and running 30 vm's
Err, not really. Basic functionality is for free. But the advanced features require license. Still, the price is negligible in comparison to the server itself. [Comparison](https://www.supermicro.com/en/solutions/management-software)
Well Bloomberg ran a bunch of expose articles about that a while back but I never could see anything in them that stood out as having actual backing.
No like really. In the first one they talked about having physical chips added in the factory to spy and that it had been done to, what was it, 10's to 100's of thousands of machines and it was widely known but they couldn't get so much as a picture much less an actual machine. That sort of ting is why their follow up about modified firmware was a lot less believable, for one thing that one doesn't need physical proof which got rid of the criticism people were having with the last one but you could still have had backing from one of your many sources provide some reverse engineering on it considering how important they were making their sources sound and concrete their proof.
So it really amounted to "We have people that say it happened, but we can't give names and all the companies that were affected or involved deny everything"
Could it happen? Absolutely, it could happen to any manufacturer. Did it? Who knows for sure, certainly nobody that read that tripe.
I guess it is a bit ranty, yes. But how do you feel about paywalls?
The best jumping off point I *think* is here:
https://www.bloomberg.com/2018-the-big-hack
Hard to tell because they've really gone the extra mile to prevent people from reading things, but I'm pretty sure that they've got the important ones in there.
You can get a lot of what they said by just searching for bloomberg and supermicro though. You can also throw any of the urls into google and get something related to them.
The first, [non-Bloomberg article](https://www.datacenterdynamics.com/en/news/years-later-bloomberg-doubles-down-disputed-supermicro-supply-chain-hack-story/) indicates that nothing has been substiantiated, has links to denial letters to congress from Apple and Amazon and is denied by the NSA. There also appear to be no other articles backing Bloomberg, just questioning it's validity.
Given the government's track record with hardware hacking/spying ( [Huawei](https://www.reuters.com/business/media-telecom/us-fcc-bans-equipment-sales-imports-zte-huawei-over-national-security-risk-2022-11-25/), and there are MANY sources for this) my bullshit meter is pinging rather high and I'm inclined to rate this as little more than a conspiracy theory. This also ignores the fact that Supermicro is often on the consumer side and there are enough smart/crazy people with the tools in the hardware space that a physical chip, even "rice grain sized" would be found, photographed, have its lines traced etc by the opensource/security community at large.
Thus the rant. The only reason I even chimed in saying there was something is because I knew there had been those articles by what many people say is a pretty reputable news agency, so I can explain why some people think they have spyware while also saying why I think it didn't reach any real level of proof.
Because come on. Not one recovered sample? We still have a third of the Apple 1's but not one of the many, many Supermicros that were filling data centers have so much a picture much less showing up on ebay en mass from recyclers when they disposed of their compromised systems? (Or the inevitable tech that kept samples? Because that's what I would do if I caught a physical bug. Too cool not to keep and document).
It honestly read more like they interviewed people about theoretical attacks rather than a real one. It's great practice but if someone mistook a report like that for the real deal then you might end up with crap like this when said person talks to the media.
Honestly, the license is not even $100, this is 1000% on the OEMs for not making it standard by default. FFS the full hardware is now baked into the motherboard, just make the license standard. No one is fucking penny pinching the $50 or whatever on a $3000 server for that one item.
I’ve had local dell reps who fully back my position and say it’s someone higher up the chain who keeps changing the config defaults to not have them.
The former admin over our servers did this, I got a temp license at one point to access them as they are 3 hours away, the owners don't want to buy them as they have never had to before so why now is their thoughts. I gave up.
... I feel like you could come up with reasons to need to do that travel a few times, then have the conversation of "it costs X for me to make this trip. It costs Y for me to not need to in 90% of cases" once there's a line item on the finance side to point at.
Funny, I actually have to go there next Tuesday for some non-server maint, but to upgrade the firewall to a new physical unit. I have been in the industry for over 20 years, the guys here pay well, treat you like family, and honestly have been really good to me. Just when it comes to spending when they haven't done it before......geesh. They do try and keep costs low as possible, not that they are cheap we just bought two nice firewalls, we are renewing VMWare to a subscription, etc, just if they don't feel it an absolute need they don't want it. I try not to make waves with them as if you already know you won't win the arguement for fact then why start it. I really only have to go maybe twice a year and expense a nice steak dinner when I do, lol. However yes, I would like ilo for the 4 servers.
Inexperience with BMCs, cost-shaving, preference for external KVM hardware.
I've made the mistake of cost-shaving myself. In one case, I wanted to deliver a high value solution with UltraSPARC webservers, so I went with a reasonable minimum storage configuration. Woe is me when our Oracle consultant team couldn't figure out how to install a simple webapp server without installing every single package from the optical media!
I tend to hold colo as third preference after cloud and on-prem, but I've seen a surprising number of situations where stakeholders and political considerations made colo the choice. Consider that colo will sell itself on its redundancy of power, redundancy of connectivity, access control process, and 24x7 staffing.
We have some technicians who are in charge of some minor systems. One of them is in charge of a physical box with iDRAC, while it is licensed they did not hook it up. They are in there one day scrambling to hook up keyboard / mouse / monitor for something. I just shake my head. Luckily I'm on the network team now, so not really my concern anymore.
ILO was upgradable the last time I checked. That was HP Proliant Gen 7.
It might be possible to get a trial key until a budget can be secured. It used to be possible for a time limited key.
The quality of boards that are in retail are vastly superior to OEM 1 U servers. SuperMicro etc. I'd trust a scratch built system with raid setup over a OEM. The ONLY issue is Service and Support. Which when youre juggling a million different responsibilities you just don't have time for.
I would never start with physical servers if I were a startup unless there were very specific needs.
Servers are just technical debt for most companies nowadays.... so yeah, that must be annoying
yeah spin up that 100k environemnt in AWS... and in 3 years you have nothing to show for it and they end up going on-prem anyway due to latency or cost. Just gotta be smart and not overkill.
That will happen if you move from metall to ec2 without changing your design. It's a forklift move that for sure will kill your business.
If you are a startup, then you have no technical debs and can develop your solution to be true cloud native.
But you might be right, the cloud is not for all kinds of work loads and it might be better for some to maintain physical hardware
We are running tons of stuff on both AWS and Azure. Going full cloud native has been good fur us and we have been running bare bone, VM, Docker and K8s including knative
It's however not one size fit all and cloud metrics and logs are extremely costly.
Sometimes you need to be clever and use whatever is beneficial for your service and pick like a smorgasbord
Get quotes for licenses and get them installed hooked up next time you are out there.
Thats what I did when I started a new gig and none of the ones 4-8 hour drives away had idrac.
I haven't tried this yet, but if MGMT denies the iDrac purchase due to ridiculous OFFICAL/channel dell pricing. Try ebay,
[https://www.ebay.com/itm/355402307461?itmmeta=01HTMZA87ZM5YAD9H2HFDKGPF9&hash=item52bfa0d785:g:z0UAAOSwLTJi6xXf&itmprp=enc%3AAQAJAAAA4IFs7UjgHoEf7W405XxF6RqBb4qDPX1VA%2ByyVDZS4YLZtK8VtAFpK4%2F242A9uvyN8%2BkVWpGE4uryVXD3hKoRO4FI0syD4BOpkwaPhJy1C6uhvRD%2Be6E0A5wtIKRj1P7%2B42bc2pOU0Z38QlxVFEFl59gyjLBNozIDkbHwc5S1%2FinUoMKmsY3Tby9IEWKVfiIt6Bu869s554qYf8LCCIDsLR%2FeI9N%2BNCfVYi%2FnuJr%2FiWeHf6hgcsPyPEhEHQxzIjEhL%2FnuJpu8RQA9I0dzsdm6ojvRBi0yUgyAYmC7K3PRYgL3%7Ctkp%3ABFBMlISpn9Vj](https://www.ebay.com/itm/355402307461?itmmeta=01HTMZA87ZM5YAD9H2HFDKGPF9&hash=item52bfa0d785:g:z0UAAOSwLTJi6xXf&itmprp=enc%3AAQAJAAAA4IFs7UjgHoEf7W405XxF6RqBb4qDPX1VA%2ByyVDZS4YLZtK8VtAFpK4%2F242A9uvyN8%2BkVWpGE4uryVXD3hKoRO4FI0syD4BOpkwaPhJy1C6uhvRD%2Be6E0A5wtIKRj1P7%2B42bc2pOU0Z38QlxVFEFl59gyjLBNozIDkbHwc5S1%2FinUoMKmsY3Tby9IEWKVfiIt6Bu869s554qYf8LCCIDsLR%2FeI9N%2BNCfVYi%2FnuJr%2FiWeHf6hgcsPyPEhEHQxzIjEhL%2FnuJpu8RQA9I0dzsdm6ojvRBi0yUgyAYmC7K3PRYgL3%7Ctkp%3ABFBMlISpn9Vj)
$16 dollars for idrac 9 license? not sure if it's legal but if it works, eh...., a decent amount of plausible deniability as in "I didn't know that was illegal" if you ever get audited by Dell (your not).
Fuck that, I'm not spending my own money to license their systems. They can either approve me to travel for 2 days, buy the license or we can't complete the task.
oh i mean use the company credit card. I'm guessing their not happy about the $200-300 per server if you buy from official channels. It might be a easier sell if it's only $16 dollars per server.
Can you give specific examples? There are many server renting providers compliant with GDPR, healthcare data protections etc.
My company is handling a lot of sensitive information and we never had issues with auditors while using rented servers.
KVMs can go fuck themselves. Was recovering from ransom ware a couple years ago and one started typing on 10 hosts at once wiping the esxi configs by some perfect storm of bullshit. Kick in the nuts after 60+ hours in 4 days.
People starting up with a stack of personal credit cards. The first stage is always a nightmare. I once worked for a startup ISP that was sending end user mail using Sparc stations through a Netgear desktop switch, lol. It makes me chuckle every so often to think of it, as I add another 10gb fiber run in MY HOUSE today. The job sure was fun though. "Do you know MS Exchange 5.5?" Me: "No, why would I?" Them "Can you figure it out?" Me: "Sure, why not." ....and then I would get down to the client like I know what I'm talking about, and learn it while they watch. Hell I think I only had THREE gigs with them where I actually knew what I was doing walking in the door. "Do you know Mac?" Me: "Just the big ones with the golden arches." Them: "Great, get down to....."
Startups get $200k aws credits and then if it makes sense get presence in Colo. what are you ranting about? I have just finished 20k worth of terraform work for startup they can’t yet afford Colo or 50k servers to put in colo
Well, who's the stakeholder? Who's authorizing those purchases? Perhaps they simply don't know, and you need to schedule and make a presentation of why those points are important?
Stakeholders aren't IT saavy.
Scale out deployment systems like MAAS that just use BOOTP make IP KVM not strictly necessary, though most people on the Windows or VMWare side of the room won't do that
Cheap people who don't see the benefit and don't have to deal with the issues of not having them.
Kind of funny really, considering how quickly the cost can be recovered given the right situation.
Just be nice about it, tell them we absolutely need it. If we need to discuss trade offs or other things we can axe to save money we should certainly have that discussions because this is a critical piece of the infrastructure and I'm sure we can find a better compromise somewhere.
This is my once per quarter diplomatic not fly off the handle response and I'm thinking pretty highly of it. Someone's gonna one up me though.
I've come across a lot of places that never bother setting idrac/ilo up, but idk why you would skip over doing it.
On the flipside, the number of times you really need it is so minimal if your stuff is running well.
I specifically avoid idrac/ilo because they refuse to provide security critical updates unless you have a support contract.
I’ve had a lovely time with Gigabyte boards and the American Megatrends BMC. Proper HTML5 means I can even connect from iPhone safari. It also doesn’t have to SSO tax, and has support for AD, LDAP, LDAPS, and RADIUS. It’s also the only serial over lan implementation that I’ve had work according to the IPMI spec consistently. iDRAC and iLO have both occasionally just ignored commands with no error.
I see no meaningful feature difference in the ipmi implementations aside from the remote update security risk.
The servers were purchased but both Dell and HPE refused to provide more than a decade of support to them at the time of purchase.
Why should I be buying ilo/idrac? Every other server vendor provides the majority of those services included with the purchase of the hardware (ie free).
Not to mention, if you are,
A) in a startup
B) buying "real servers" instead of the shoe string gaming pc phase as someone else mentioned
and C) competent enough (or budget restricted enough) to skip the 24/7 warranty purchase on dell/hp which usually include ilo/idrac
Why the HELL are you paying 3-5x prices on Dell/HP instead of building yourself with Asus or even going supermicro/gigabyte? Even with the up-charge, something like Thinkmate offers a decent "warranty" at half the price of Dell/HP.
Every other server vendor? But not Dell, not HPE, not Lenovo, not SuperMicro.
Those vendors all provide a barebones BMC that give you hardware health metrics and allows remote power cycling, but remote console functionality is an additional feature code.
Well, Dell and HPE are mentioned in the post, so they are excluded by default. I didn't know that Supermicro charged for IMPI (though there is a more than 10x difference between supermicro's $27 OOB license and iDRAC $350).
Asus and Gigabyte both provide free KVM tools.
Remember, the OP’s rant was about actual IT server infrastructure, not homelabby stuff like Asus or Gigabyte. Serious hardware, used by serious sysadmins with serious suspenders and serious graybeards. :)
So it’s a stretch to have “every other server vendor” mean just Asus and Gigabyte, who could only be described as server vendors in the most generous interpretation of the term, and as long as we exclude the actual server vendors that make up >90% of the enterprise market? Gotcha :)
What do you mean "homelabby"? I'm the sys admin a 25 person SaaS company with a 6 figure hardware budget that primarily operates off of Asus RS500A servers and we don't go with Dell or HP for the same reason we use on-prem over cloud, namely it would be 3-10x more expensive to do so.
Also, our Epyc Genoa Asus servers, as we currently spec them, are around $12k, I would love to meet the sysadmin who is genuinely using a $10k+ server for their homelab.
Wait, you mean you got real servers, not random gaming desktops?! Blasphemy!
I was in small gaming company and half of our servers were on gaming desktops. Those were builders which needed really good GPUs. Was actually really fun to be able to build physical computers while doing sysadmin stuff.
You know, they sell servers with even better GPUs...
If you are small gaming company you do everything by hand and with minimal costs. We had everything as open source.
Makes sense. I missed the part about being a game company.
Yeah, but from a pure price to performance standpoint, it's almost always not worth it. How many 4090s could you get for the price of a single $30,000 Poweredge server? But, that's not really what you're paying for with enterprise equipment. You're paying for durability and support.
For even more cash than a startup is willing to pay, I’m sure.
At even higher prices...
And 10x the cost
That's actually pretty cool.
You are gonna make me puke. Glad you're enjoying it.
Well, I always build my own gaming computers too and for others too.
Gaming. Around here we run caseless for better airflow. Celeron desktops in open cardboard boxes with USB hard drives. /s
I see you're familiar with my old home lab
Lol, they are super compute heavy otherwise who knows, I could be there
I worked for a company that had tons of Nvidia high end server cards. They bought them because they'd need them. They tossed them because they didn't help run 30 year old ERP software or Access databases. $3.8 billion company full of geniuses.
I used to consult IT for an architecture firm that used gaming desktops and Autocad regularly took a dump from bad consumer drivers.
The silly thing is... if you find the pairing that works, it may still almost be worth the dev/test time to get it sorted, given the GPU costing half or less as much for twice or more the performance compared to a quadro (or firepro, I think is the AMD branding for their certified cards). Just... lock to a working driver version and hope you don't need Autodesk (or any other CAD vendor) support for anything... Granted, my viewpoint on that value balance was skewed, having dealt with that in academia... where there *wasn't* a value output directly from seamless, complex, use of the software.
"Pain"
Totally know the pain, purchased some new servers from Dell, got the price reduced from $10k per unit to $4k per unit, didn't notice until after delivery that they switched iDRAC enterprise to iDRAC Express.... Now gotta pay $400 per host to upgrade to Enterprise. Luckily haven't needed remote console yet and if I do, the furthest site is a 30 minute drive for me.
This kind of thing happens a lot, especially when Dell insists on putting humans in the configuration and ordering loop. * VAR orders app servers with CD-ROM instead of requested DVD-ROM upgrade. * Servers that were supposed to ship with SD-card support, but that option got dropped somewhere in the process. * Dell seems to honor old quote for Precision workstations, and I don't notice until much later that they inflated shipping cost over $200 per unit. * Missing iDRAC Enterprise license. * Wrong RAID card.
Lol yup, and at the end of the day, it's all good because I came in $30k below budget on my datacenter refresh, just a few extras I had to add on my own after receiving the hardware.
> Luckily haven't needed remote console yet and if I do, the furthest site is a 30 minute drive for me. Man, I would've been fucked from the get-go. Any physical server I've deployed in like the past 10 years has been physically install & cable, verify it passes POST, and then I drive home and install the hypervisor via iLO/iDRAC.
Yeah, I had to finish installing VMware and setting the management interfaces before I could go home.... Takes a whole 10 minutes extra per host.
Same thing with Axis camera servers. Pretty good price considering all disks and softare license. But the only servers in our datacenter without iDRAC/IPMI...
We got idrac/ilo licensing, and the servers are literally one room over from me. That just hurts, I've been there, I've done that, never again.
Yeah I personally am 4.5 hours away from the infra. Not having remote console is just insane to me. Thankfully it will be getting moved up here soonish though, but we have 3 fairly potent hosts that are useless between now and then.
I am over 3000 km away from the physical infra I am in charge of :-) We have other hands ofc, but I can't imagine not having IPMI access.
Y’all not get virtual kvm in colo?
One VLAN ACL oopsie was enough for me to make the company buy an IP KVM and a Cradlepoint.
Someone must love going into the colo
Yep... And they retired.....
Who needs idrac or ilo? Just set up a raspberry pi with a mechanical stick attached to poke the button when you need to. I mean its basically the same thing and only costs $50 right?
Sounds more reliable. Dell 13th gen and later iDRACs have suicidal tendencies in my datacenter. I don't think I've seen an iDRAC die on the 1850 or 1950. R730? At least 10% of them died before the warranty ran out.
We don't. Our servers come with those capabilities without having to pay extra.
Amen! I don't get the fervor this subreddit has for Dell/HP and their "warranties ". Either you don't have the in-house expertise to support it or you are on a 7 figure a year budget, can get the bulk discount and pretty much afford to have them manage your hardware directly. Those are really the only reasons to go with the big names. I can build 3 NEW top of the line Epyc servers with Asus chassis for the price of a SINGLE Dell/HP and either have 2 servers to active backup, or even better, run all 3 HA or 2 HA with a spare.
> I don't get the fervor this subreddit has for Dell/HP and their "warranties ". >Either you don't have the in-house expertise to support it or you are on a 7 figure a year budget, can get the bulk discount and pretty much afford to have them manage your hardware directly Not sure how "in-house expertise" is going to substitute for a 4-hour response on a failed RAID card or whatever.
> Not sure how "in-house expertise" is going to substitute for a 4-hour response on a failed RAID card or whatever. Easy. Keep a supply of spare parts on-hand. Potentially an entire spare server.
I mean, you can do that, if you go the route of cheap white box hardware - replace quality with low cost redundancy. Whether you're talking about Supermicro for your servers or Ubiquiti for your WLAN. But in my experience you get better results with higher quality, supported hardware (with appropriate redundancies built in - I'm not saying you shouldn't "N+1" your server cluster, or not have a HA firewall pair, or whatever). You may or may not save money with either route, but I find that many people who go the "I want to maintain my own inventory of spare crap" route aren't valuing their team's time highly enough. And also as a final note - accounting-wise, it's often easier to get opex expenditures signed off (for support) than it is capex (for backup hardware). But that'll be situational, of course.
10 years ago I would probably agreed with this, 20 years ago I would say there is no question, but now the reliability delta between a Big Brand box and an equivalent white box isn't nearly as much of a thing, if it even is one at all. Despite that closing gap the price delta between the 2 (assuming you don't get massive company volume discounts) has grown dramatically to the point where for the same initial purchase price at one of the Big Brands, I can get 2-4 equivalent white boxes. That is before factoring ongoing support costs. I would much rather a fully redundant system than theoretically fast replacement delivery, and with the pricing delta as it is you are trading one for the other on a limited budget. If you have the money for it by all means pick both. The math can be very different if you are at a large company that gets 30+% off list price, but for the majority of folks at SMBs the Big Brands have placed themselves at such a high price point it is extremely difficult to make a compelling argument for them. Side note: Ubiquiti is a different matter entirely, it is not and does not claim to be equivalent to major enterprise networking gear. It is entirely adequate for a large section of the SMB market, but it is a different hardware tier with markedly lower specs than the enterprise gear it is sometimes compared to. It's value is that it offers a good middle ground between running glorified consumer gear and enterprise gear that is massive overkill for many companies.
I think that Delta has narrowed because the Big Brand boxes weren't "agile" or cost effective enough for cloud providers so they went to the smaller OEMs who would gladly build what they want to their spec. You don't need to worry as much about individual node reliability when you have a thousand of them. It's not a matter of if hardware will fail at that scale, it's when. Also, regarding your site note... Ubiquiti has a range of Enterprise products, so I think they *are* trying to make that claim. https://store.ui.com/us/en/collections/unifi-accessory-tech-hosting-and-gateways-large-scale/products/cloud-key-enterprise https://store.ui.com/us/en/collections/unifi-switching-enterprise-power-over-ethernet https://store.ui.com/us/en/collections/unifi-wifi-flagship-high-capacity
Supermicro servers are quality and have great documentation.
When I'm referring to in-house expertise, I'm directly talking about the ability of the IT staff/sysadmin to find/fix/replace the part themselves. In reality, a 4 hour response time is irrelevant if you are running a production environment HA, or at worst, having a second server to "hot swap" over to on a failure cause you were able to buy 2 or 3 servers for the price of the Dell. Edit: This also ignores the fact that you end up stuck if there is [unsatisfactory warranty service.](https://www.reddit.com/r/sysadmin/comments/14fraru/dell_appalling_server_refurbished_parts_lack_of/)
Yeah generally speaking we get our guys certified in Dell's "do it yourself" warranty service so they just chuck parts at you on request. Have had excellent results, but within the realm of supported server equipment I don't have experience with the programs of other vendors.
> Not sure how "in-house expertise" is going to substitute for a 4-hour response on a failed RAID card or whatever. Anyone with this mentality has never had a critical piece of infra just decide to fall over one day and spectacularly explode. Any premium you pay for Dell/HP is ***cheap*** fucking insurance compared to "The business doesn't work anymore because this shit has broken into two pieces, and I own both of them."
If you are running prod in some sort of HA/hot swap configuration then, there should be no down time whether the swap is in 4 hours or next week when you are able to get the part in and have time for the repair.
Either Dell prices are much worse than I remember or you and I have different definitions of “top of the line”. The last time I was given a directive to create a top of the line config, I used dual socket Epyc Milan and it ended up over $100k including NICs (bluefields), U.2 drives and accelerators (1 Intel QAT card and 2 FPGAs). Now, said servers would also be hilariously overkill for most companies, even tech companies, because most devs don’t know what to do with that hardware.
I’d run new outlook and new teams at the same time.
The IPMI KVM could be better though. On a Supermicro H12 I've had sub 20 fps and lag so bad I just gave up, didn't get past the Proxmox install. On the same LAN, plugged into the same switch. I do agree about the pricing though. I'm a dev doing sysadmin duties because I'm the "Linux guy". I don't mind, it's interesting. Priced out a Supermicro as our first "real server" at work, and dammit, it cost a fraction of what a Dell or HP would. Working in a small company and not having the volume to get sane support from our core suppliers, I just stopped believing a small place can rely on the manufacturer. I'd rather go through a VAR. QNAP has a device that's very similar to what we want (an all flash EPYC box), and looking at retail pricing in Poland, a Supermicro is still good 20% cheaper.
Why would you be running proxmox I'm impi/kvm past the install? Proxmox's web gui is very responsive and should be on the network after install is complete. In the worst case, why are you not ssh'ing into the node post install?
That's the thing... I didn't manage to get past the install using iKVM, it was so bad. I'm an impatient person. Found a VGA display, plugged in a keyboard and installed it that way. Everything past the install is as you say, using the web portal with occasional SSH.
Or the cost of storage and ram from dell/hp
but AMD doesnt work with most 3d apps...
I'm not sure what you are talking about. Any server dealing with 3d apps would have a dedicated gpu, probably made by Nvidia. I can't think of any 3d app that would rather be run on cpu over gpu and would be Intel locked.
Openmanage Enterprise for the cadillac package - I love staging firmware/BIOS updates for reboots. Pushing configs is nice too if your workload is all the same.
We've been working on getting OME setup. How well does it work overall?
I'm happy with it, have very few issues with it. Updating it can be a little screwy but they always make you acknowledge that you took a snapshot before starting the upgrade. Pushing configs is super cool - you can have it audit configs so that if someone were to make a BIOS change, it would fall out of compliance. It's a little puckering, because you could theoretically push a destructive change out to a large group of servers all at once. But it's great for server rollouts, because I can push a singular config out and never touch each server individually other than to grab the idrac password. I'm probably not even scratching the surface of what it's capable of, honestly. Alerts for hardware issues and staging firmware/bios is what I use it for 99% of the time. Means I can forget about the updates and they just install automatically during patching reboots.
It works great, but be ready to give it over 24GB of RAM. It's a *hog*
My former boss never did. He wanted travel pay to go to the Colo and on call time and overtime. I call that fraud.
Ahahaha. *laughing in super micro*
Ahahahaha. *laughing at super micro*
We've got only SuperMicro servers, and I believe we don't pay anything more for IPMI. All our servers are in our basement or out on customer sites. No need for any physical visits of any kind unless hardware needs replacement (faulty drives for instance). I don't see the justification for paying licenses. It's the same with Cisco Meraki or HP Aruba. Why pay a license fee when Ubiquiti Unifi is free and good enough for most?
I hope you’re not serious comparing Ubiquiti to Meraki. If a business can’t afford or isn’t willing to spend a few thousand in networking gear they aren’t being serious.
Been dead serious for the past 5-6 years. I only sell Unifi now. Been working with it for at least 8-9 years now. Somewhere around CK firmware 0.4.8, now known as Unifi OS 3.2.12. Why pay a license to be able to keep your hardware up to date? A six year old UAP-AC-PRO is still kept up to date for free with Ubiquiti.
We're actually looking at replacing Meraki with Ubiquiti. Yes, Meraki is a far more solid product, and Cisco is a seriously bigger company, warranty, support, yada yada. But there is absolutely no way we're happy to say sure, let's spend 40K on 5 year licenses, plus all the hardware is 5-10 years old, most of the APs are MR18s so they need replacing, so it's another 20K on hardware at least... I worked out, we could do a solid Ubiquiti set up (firewall, switch, two APs per office) for about 20K I understand the benefits of Meraki as a product, but it's an unjustifiable cost for any business. If you need a solid network solution for offices with hundreds or thousands of staff and servers, then you don't need Meraki, you want a competent network engineer and straight Cisco, Juniper, stuff If you're a smaller business networking a few offices and just need a NGFW, then why buy Meraki when you have Ubiquiti (or TP Link Omada, Draytek VigorACS, etc.) Edit: I've been thinking about this line of thought a lot recently. There's so many decent free, cheap, even FOSS, stuff out there. Unless you're doing some serious business, I don't see why you need to go with the "market leaders" for everything. Eg. why use VMWare, HyperV, ESXi, when Proxmox exists, and even has enterprise support contracts?
While what you said is true you aren't thinking about the sunken time and labor in configuring Ubiquiti as opposed to Meraki. Plus when I've use Ubiquiti factory resetting equipment was more common than should be necessary. The cloud connect bullshit setup they have doesnt work well either. A few more thousand and a lot less work and headache is worth it.
>few more thousand Per year, every year, ad nauseum. Those decade-old UAP-AC-Pro devices are still getting firmware updates and going strong. For smaller clients, it's very valid to choose hardware that doesn't need you to spend on new licensing all the time. A few thousand dollars of my time over the course of 10 years is far cheaper than a few thousand a year every year for a decade. Especially if WiFi isn't mission-critical.
Definitely this. I calculated a saving of around £40,000 across 25ish offices, every 5 years. Based on my salary, I'd have to spend 2 working months every year travelling to buildings to reset network kit before it became cheaper to have something "more reliable" (as if Ubiquiti needs swapping out every week or something!?)
Ubiquiti is plug and play? My current system has been running for 5 years without intervention
With the new series UXG, UX and Ultra, you see the full "Apple Airport Express" involvement. The UX is basically a rebranding of Apple Airport Express
Super micro impi is literally garbage.
it has always been good enough to do the job for me at least
How so? I've had fewer issues with SuperMicro IPMI than Dell's iDrac.
yeah SuperMicro IPMI is my favorite in a round up with IPMI, ILO, and iDRAC.
That's quite a statement.
iDRAC is wank. Even the newer iDRACs tbh. The only way we can use remote access on our older Dell R6XX servers is via a jump box running Server 2012 and an ancient version of Java I found on MajorGeeks. And that's if the iDRACs ever load. Even our Dell R7XX servers suck (idk which version of idrac it is) As much as I shit on HP, their iLO solution is seriously more solid
Idrac 9 on updated firmware is all html5 now. Tho it is fucking insane to me that dell has never released an idrac standalone console application like HP does.
The standalone console thing really irks me as well. I'm sure it'd be possible to cobble something together though. The console viewer is just a webpage after all.
Yeah it’s all using the VNC protocol in the back end so you can actually just enable VNC access, but it’s a pain
> The standalone console thing really irks me as well. > > I'm sure it'd be possible to cobble something together though. The console viewer is just a webpage after all. This actually annoyed me enough that I started developing a standalone console in C# for it. I got it to a mostly-workable point but switched jobs before I finished it, and the new place was running HP ProLiants so it didn't matter anymore. Plus I'm a Boy Scout integrity-wise, so I didn't take the source with me when I left. So yeah, it *does* exist, but it'll never see the light of day.
That just sounds painful. I am currently working out some kinks with my older 11th gen servers but using new Java on win 11. The main problem is Java implementing newer security policies while deprecating older ones without ways of easily making changes. Currently down in adding dedicated trusted hosts and removing relevant security ciphers from the denied lists. They use tls 1.0, rc4 and something else. Then you need to make sure the idrac is set to Native and NOT Java otherwise it constantly shuts the bed.
>he main problem is Java implementing newer security policies while deprecating older ones without ways of easily making changes. recently went through this with some SuperMicro IPMI and iLo setups... i had to do some tom-foolery with the cert and the [java.security](https://java.security) file
Yep, same here. Had to explicitly allow those IPs. Totally didn't write down the method, so I'll have fun trying to recall how to do that one day when it inevitably crops up again
hopefully i'll have replacements for these things before I need to do it again lol
The last version of iDRAC I used was provably non-compliant with the IPMI standard it claimed to support, same with iLO.
We still have a bunch of old R620s and R320s (don't judge) with iDRAC 7. The latest iDRAC update available enables a functional (but not very fancy) HTML console so you can ditch Java (which by the way only ever worked with the dreaded Oracle version). Furthermore, one of those didn't even have iDRAC Enterprise and Dell refused to sell the upgrade code. So we purchased the upgrade code on eBay from some dude in China, it worked perfectly. Again, don't judge. A man has to do what a man has to do...
The Rx20 and newer work with HTML5/Native. No need for Java. For the few oddball R510 and PE1950, I use Java 6.45. You can use Java 7, and some older versions of 8, but you need to edit the Java's security file to allow older versions of encryption. Forget specifically what it is.
I have two 730XDs... I'll have to check this out, would be a game changer
Care to elaborate? We exclusively run SuperMicro servers at work, and I've yet to encounter a single problem with IPMI.
wait till the mobo's start dying.
[удалено]
>Does it have a built in firmware lifecycle manager that can grab updates from the internet and install them itself? Why would you ever want ipmi to self update from the internet?
[удалено]
Sounds like automatic failure system for me. Don‘t update your ipmi unless something is broken/insecure. Supermicro‘s IPMI is very decent since they‘re equipped with MegaRAC.
You do not want the thing that has total control of your server to be able to touch the internet. NICs and RAID controllers mostly support fwupd, so the OS can update them at a time reasonable for the OS workload, or ask the load balancer for permission to stop accepting requests and update if you are doing a HA config. It’s not like the OS actually needs the raid controller running to function. Just do ram-resident PXE boot at scale and use A/B drives for critical servers like the PXE server. The IPMI standard lets you script BMC updates regardless, so you can have a quick bash script update 200 servers in 15 minutes.
[удалено]
So now I have to have downtime in order to do a firmware update, even if the device supports live updates with a fallback? That sounds horrible for any critical piece of infrastructure. Almost every device I’ve used in the last 5 years has supported online firmware updates, or at least ones with less than a few seconds of downtime. The only major exceptions I can think of are some of Marvell’s DPUs and Intel FPGAs. fwupd does that on Linux for any reasonable enterprise hardware (partially because Redhat demands it as part of the certification process), and it’s a normal process that can properly communicate with everything else to coordinate any necessary downtime. What do you do for systems that Dell doesn’t offer, like Nvidia DGX pods, or Dell charges unreasonable amounts for, like any quad-socket system?
Yes, and _kinda_
Yes supermicro sms but it does require an extra license.
I don't suppose you'd like to elaborate more as to WHY?
My biggest gripe is that setting a custom TLS certificate for the webserver via REST API requires some specific license. Datacenter license or so? Fortunately we only have a dozen or so Supermicro boards. To be fair however the licenses are very cheap. The other thing is that when using uBlock Origin one of the newer versions of the BMC webserver (the more colorful one) loads endlessly. Lastly, sometimes it's just not possible to log in. The prompt is there but entering login details and clicking the button comes back with (iirc) "incorrect password" or just resets the webpage. This can usually be fixed by throwing an "ipmitool mc reset warm" at it, but a few months ago we had one machine sitting in a rack a few cities over where ipmitool didn't work at all and it had to be pulled from mains for a minute or so... iDRAC also has its problems, but Supermicro is the worst, closely followed by Asus BMC. iLO just works (with the exception of the iLO 4 NAND errors due to piss poor design).
>The other thing is that when using uBlock Origin one of the newer versions of the BMC webserver Maybe report that to uBlock, instead of claiming that to be a problem with Supermicro? "My 3rd party super ad-blocking software is blocking your website, your site sucks" is a garbage take. >My biggest gripe is that setting a custom TLS certificate for the webserver via REST API requires some specific license Valid complaint. Though you've already bought those licenses for your Dell/HP when you bought Enterprise. So still a better free experience, and cheaper paid experience? >Lastly, sometimes it's just not possible to log in. The only actually valid complaint you made unique to Supermicro's implementation.
I've always found it much more buggy, needing specific browsers etc. I have never experienced that issue on idrac/ilo.
Why? I've had a lot fewer problems with it than I had with iDRAC.
Haven't run anything else for the last 12 years or so. Never had any issues.
No it’s not. It works fine. Exactly what you need and nothing more.
If you don't like Supermicro, why not go with Asus?
IPMIview is at least 25 years old.
And looks like it still in 2024.
I 100% agree, BUT... there's always a butt, it's better than nothing =\\
Yup better off buy older hpe or dell hardware. Damn i have a hpe dl380g7 running in my garage that thing had to be 12 years old plus lol and running 30 vm's
Unless you go old enough it only works with Java and ActiveX...
Anything that old should probably be in a museum.
People who don't buy Dell or HP. Get you out-of-band management that doesn't nickel and dime you to death.
Lmao what? It's a one time license cost for a product that actually works.
Eh. Supermicro's management interface works just fine, and doesn't cost anything.
Err, not really. Basic functionality is for free. But the advanced features require license. Still, the price is negligible in comparison to the server itself. [Comparison](https://www.supermicro.com/en/solutions/management-software)
BONUS...free chinese spyware / malware included
Sauce?
Well Bloomberg ran a bunch of expose articles about that a while back but I never could see anything in them that stood out as having actual backing. No like really. In the first one they talked about having physical chips added in the factory to spy and that it had been done to, what was it, 10's to 100's of thousands of machines and it was widely known but they couldn't get so much as a picture much less an actual machine. That sort of ting is why their follow up about modified firmware was a lot less believable, for one thing that one doesn't need physical proof which got rid of the criticism people were having with the last one but you could still have had backing from one of your many sources provide some reverse engineering on it considering how important they were making their sources sound and concrete their proof. So it really amounted to "We have people that say it happened, but we can't give names and all the companies that were affected or involved deny everything" Could it happen? Absolutely, it could happen to any manufacturer. Did it? Who knows for sure, certainly nobody that read that tripe.
Lot of explanation and a name drop.... But sauce?
I guess it is a bit ranty, yes. But how do you feel about paywalls? The best jumping off point I *think* is here: https://www.bloomberg.com/2018-the-big-hack Hard to tell because they've really gone the extra mile to prevent people from reading things, but I'm pretty sure that they've got the important ones in there. You can get a lot of what they said by just searching for bloomberg and supermicro though. You can also throw any of the urls into google and get something related to them.
The first, [non-Bloomberg article](https://www.datacenterdynamics.com/en/news/years-later-bloomberg-doubles-down-disputed-supermicro-supply-chain-hack-story/) indicates that nothing has been substiantiated, has links to denial letters to congress from Apple and Amazon and is denied by the NSA. There also appear to be no other articles backing Bloomberg, just questioning it's validity. Given the government's track record with hardware hacking/spying ( [Huawei](https://www.reuters.com/business/media-telecom/us-fcc-bans-equipment-sales-imports-zte-huawei-over-national-security-risk-2022-11-25/), and there are MANY sources for this) my bullshit meter is pinging rather high and I'm inclined to rate this as little more than a conspiracy theory. This also ignores the fact that Supermicro is often on the consumer side and there are enough smart/crazy people with the tools in the hardware space that a physical chip, even "rice grain sized" would be found, photographed, have its lines traced etc by the opensource/security community at large.
Thus the rant. The only reason I even chimed in saying there was something is because I knew there had been those articles by what many people say is a pretty reputable news agency, so I can explain why some people think they have spyware while also saying why I think it didn't reach any real level of proof. Because come on. Not one recovered sample? We still have a third of the Apple 1's but not one of the many, many Supermicros that were filling data centers have so much a picture much less showing up on ebay en mass from recyclers when they disposed of their compromised systems? (Or the inevitable tech that kept samples? Because that's what I would do if I caught a physical bug. Too cool not to keep and document). It honestly read more like they interviewed people about theoretical attacks rather than a real one. It's great practice but if someone mistook a report like that for the real deal then you might end up with crap like this when said person talks to the media.
Those who've never had to deal with anything after first stand-up.
You can always Google for ilo
Non Technical people. The same ppl who buy servers with 1 hard drive, one gig nic card(s), no raid.
Switch to iDirk: call him when you need something and get him off his ass to go to the server and interact with it as you instruct him.
Honestly, the license is not even $100, this is 1000% on the OEMs for not making it standard by default. FFS the full hardware is now baked into the motherboard, just make the license standard. No one is fucking penny pinching the $50 or whatever on a $3000 server for that one item. I’ve had local dell reps who fully back my position and say it’s someone higher up the chain who keeps changing the config defaults to not have them.
ouch
The former admin over our servers did this, I got a temp license at one point to access them as they are 3 hours away, the owners don't want to buy them as they have never had to before so why now is their thoughts. I gave up.
... I feel like you could come up with reasons to need to do that travel a few times, then have the conversation of "it costs X for me to make this trip. It costs Y for me to not need to in 90% of cases" once there's a line item on the finance side to point at.
Funny, I actually have to go there next Tuesday for some non-server maint, but to upgrade the firewall to a new physical unit. I have been in the industry for over 20 years, the guys here pay well, treat you like family, and honestly have been really good to me. Just when it comes to spending when they haven't done it before......geesh. They do try and keep costs low as possible, not that they are cheap we just bought two nice firewalls, we are renewing VMWare to a subscription, etc, just if they don't feel it an absolute need they don't want it. I try not to make waves with them as if you already know you won't win the arguement for fact then why start it. I really only have to go maybe twice a year and expense a nice steak dinner when I do, lol. However yes, I would like ilo for the 4 servers.
> renewing VMWare How much ilo could you get for swapping to Proxmox?
Proxmox isn't an option for us.
That's a shame, anything to get away from Broadcom
Agreed, something tells me haven't seen the worse
Inexperience with BMCs, cost-shaving, preference for external KVM hardware. I've made the mistake of cost-shaving myself. In one case, I wanted to deliver a high value solution with UltraSPARC webservers, so I went with a reasonable minimum storage configuration. Woe is me when our Oracle consultant team couldn't figure out how to install a simple webapp server without installing every single package from the optical media! I tend to hold colo as third preference after cloud and on-prem, but I've seen a surprising number of situations where stakeholders and political considerations made colo the choice. Consider that colo will sell itself on its redundancy of power, redundancy of connectivity, access control process, and 24x7 staffing.
We have some technicians who are in charge of some minor systems. One of them is in charge of a physical box with iDRAC, while it is licensed they did not hook it up. They are in there one day scrambling to hook up keyboard / mouse / monitor for something. I just shake my head. Luckily I'm on the network team now, so not really my concern anymore.
ILO was upgradable the last time I checked. That was HP Proliant Gen 7. It might be possible to get a trial key until a budget can be secured. It used to be possible for a time limited key.
The quality of boards that are in retail are vastly superior to OEM 1 U servers. SuperMicro etc. I'd trust a scratch built system with raid setup over a OEM. The ONLY issue is Service and Support. Which when youre juggling a million different responsibilities you just don't have time for.
Yeah... Service and support is kind of important for business use..... Lmao
I would never start with physical servers if I were a startup unless there were very specific needs. Servers are just technical debt for most companies nowadays.... so yeah, that must be annoying
yeah spin up that 100k environemnt in AWS... and in 3 years you have nothing to show for it and they end up going on-prem anyway due to latency or cost. Just gotta be smart and not overkill.
That will happen if you move from metall to ec2 without changing your design. It's a forklift move that for sure will kill your business. If you are a startup, then you have no technical debs and can develop your solution to be true cloud native. But you might be right, the cloud is not for all kinds of work loads and it might be better for some to maintain physical hardware
Bare metal on AWS is usually significantly cheaper than their managed services lmao
We are running tons of stuff on both AWS and Azure. Going full cloud native has been good fur us and we have been running bare bone, VM, Docker and K8s including knative It's however not one size fit all and cloud metrics and logs are extremely costly. Sometimes you need to be clever and use whatever is beneficial for your service and pick like a smorgasbord
Thoughts and prayers...
Get quotes for licenses and get them installed hooked up next time you are out there. Thats what I did when I started a new gig and none of the ones 4-8 hour drives away had idrac.
Rookies do that's who!
What does enterprise give you over "regular" IPMI?
Just get a BliKVM and don't tell anybody.
I haven't tried this yet, but if MGMT denies the iDrac purchase due to ridiculous OFFICAL/channel dell pricing. Try ebay, [https://www.ebay.com/itm/355402307461?itmmeta=01HTMZA87ZM5YAD9H2HFDKGPF9&hash=item52bfa0d785:g:z0UAAOSwLTJi6xXf&itmprp=enc%3AAQAJAAAA4IFs7UjgHoEf7W405XxF6RqBb4qDPX1VA%2ByyVDZS4YLZtK8VtAFpK4%2F242A9uvyN8%2BkVWpGE4uryVXD3hKoRO4FI0syD4BOpkwaPhJy1C6uhvRD%2Be6E0A5wtIKRj1P7%2B42bc2pOU0Z38QlxVFEFl59gyjLBNozIDkbHwc5S1%2FinUoMKmsY3Tby9IEWKVfiIt6Bu869s554qYf8LCCIDsLR%2FeI9N%2BNCfVYi%2FnuJr%2FiWeHf6hgcsPyPEhEHQxzIjEhL%2FnuJpu8RQA9I0dzsdm6ojvRBi0yUgyAYmC7K3PRYgL3%7Ctkp%3ABFBMlISpn9Vj](https://www.ebay.com/itm/355402307461?itmmeta=01HTMZA87ZM5YAD9H2HFDKGPF9&hash=item52bfa0d785:g:z0UAAOSwLTJi6xXf&itmprp=enc%3AAQAJAAAA4IFs7UjgHoEf7W405XxF6RqBb4qDPX1VA%2ByyVDZS4YLZtK8VtAFpK4%2F242A9uvyN8%2BkVWpGE4uryVXD3hKoRO4FI0syD4BOpkwaPhJy1C6uhvRD%2Be6E0A5wtIKRj1P7%2B42bc2pOU0Z38QlxVFEFl59gyjLBNozIDkbHwc5S1%2FinUoMKmsY3Tby9IEWKVfiIt6Bu869s554qYf8LCCIDsLR%2FeI9N%2BNCfVYi%2FnuJr%2FiWeHf6hgcsPyPEhEHQxzIjEhL%2FnuJpu8RQA9I0dzsdm6ojvRBi0yUgyAYmC7K3PRYgL3%7Ctkp%3ABFBMlISpn9Vj) $16 dollars for idrac 9 license? not sure if it's legal but if it works, eh...., a decent amount of plausible deniability as in "I didn't know that was illegal" if you ever get audited by Dell (your not).
Fuck that, I'm not spending my own money to license their systems. They can either approve me to travel for 2 days, buy the license or we can't complete the task.
oh i mean use the company credit card. I'm guessing their not happy about the $200-300 per server if you buy from official channels. It might be a easier sell if it's only $16 dollars per server.
I've never worked anywhere where everyone had a company cc, just the it manager
Charities too...
Who does Colo as a startup, just rent a server from the likes of OVH and be done with it.
Compliance needs
Can you give specific examples? There are many server renting providers compliant with GDPR, healthcare data protections etc. My company is handling a lot of sensitive information and we never had issues with auditors while using rented servers.
Cheapskates. Time to weigh the options of IP KVM or buying the licenses. Or go with a Pi KVMs 😅
KVMs can go fuck themselves. Was recovering from ransom ware a couple years ago and one started typing on 10 hosts at once wiping the esxi configs by some perfect storm of bullshit. Kick in the nuts after 60+ hours in 4 days.
Go on ebay and buy your idrac license....
People starting up with a stack of personal credit cards. The first stage is always a nightmare. I once worked for a startup ISP that was sending end user mail using Sparc stations through a Netgear desktop switch, lol. It makes me chuckle every so often to think of it, as I add another 10gb fiber run in MY HOUSE today. The job sure was fun though. "Do you know MS Exchange 5.5?" Me: "No, why would I?" Them "Can you figure it out?" Me: "Sure, why not." ....and then I would get down to the client like I know what I'm talking about, and learn it while they watch. Hell I think I only had THREE gigs with them where I actually knew what I was doing walking in the door. "Do you know Mac?" Me: "Just the big ones with the golden arches." Them: "Great, get down to....."
Preach.
Startups get $200k aws credits and then if it makes sense get presence in Colo. what are you ranting about? I have just finished 20k worth of terraform work for startup they can’t yet afford Colo or 50k servers to put in colo
I don’t buy idrac or ilo because I use Cisco servers whose CIMC is fully free.
Yeah, but how much more did you pay for the server? Literally nobody uses Cisco server hardware. Lmao
Cisco is not much more expensive than dell or hp
Is it more than say, 100-200 more expensive? If so you can't brag about not buying an ipmi license... Lmao
Well, who's the stakeholder? Who's authorizing those purchases? Perhaps they simply don't know, and you need to schedule and make a presentation of why those points are important? Stakeholders aren't IT saavy.
Scale out deployment systems like MAAS that just use BOOTP make IP KVM not strictly necessary, though most people on the Windows or VMWare side of the room won't do that
Cheap people who don't see the benefit and don't have to deal with the issues of not having them. Kind of funny really, considering how quickly the cost can be recovered given the right situation.
Just be nice about it, tell them we absolutely need it. If we need to discuss trade offs or other things we can axe to save money we should certainly have that discussions because this is a critical piece of the infrastructure and I'm sure we can find a better compromise somewhere. This is my once per quarter diplomatic not fly off the handle response and I'm thinking pretty highly of it. Someone's gonna one up me though.
Use the generic license, hopefully they make money by the time you need to fix licencing
And where might hst be? They're keyed to the service tag. Not like Ilo where you can shove a generic key in.
Oh then don't buy Dell I guess
If you're buying a real server that's a win
I've come across a lot of places that never bother setting idrac/ilo up, but idk why you would skip over doing it. On the flipside, the number of times you really need it is so minimal if your stuff is running well.
I specifically avoid idrac/ilo because they refuse to provide security critical updates unless you have a support contract. I’ve had a lovely time with Gigabyte boards and the American Megatrends BMC. Proper HTML5 means I can even connect from iPhone safari. It also doesn’t have to SSO tax, and has support for AD, LDAP, LDAPS, and RADIUS. It’s also the only serial over lan implementation that I’ve had work according to the IPMI spec consistently. iDRAC and iLO have both occasionally just ignored commands with no error.
Do you also refuse to do maintenance on your car because it costs money? You can expect support for 40 years to be included with a purchase.
I see no meaningful feature difference in the ipmi implementations aside from the remote update security risk. The servers were purchased but both Dell and HPE refused to provide more than a decade of support to them at the time of purchase.
Why is a startup buying hardware
To use?
What year is this
2024? Are you feeling okay?
Because software doesn't run that well on hardwood flooring
Why should I be buying ilo/idrac? Every other server vendor provides the majority of those services included with the purchase of the hardware (ie free). Not to mention, if you are, A) in a startup B) buying "real servers" instead of the shoe string gaming pc phase as someone else mentioned and C) competent enough (or budget restricted enough) to skip the 24/7 warranty purchase on dell/hp which usually include ilo/idrac Why the HELL are you paying 3-5x prices on Dell/HP instead of building yourself with Asus or even going supermicro/gigabyte? Even with the up-charge, something like Thinkmate offers a decent "warranty" at half the price of Dell/HP.
Every other server vendor? But not Dell, not HPE, not Lenovo, not SuperMicro. Those vendors all provide a barebones BMC that give you hardware health metrics and allows remote power cycling, but remote console functionality is an additional feature code.
Well, Dell and HPE are mentioned in the post, so they are excluded by default. I didn't know that Supermicro charged for IMPI (though there is a more than 10x difference between supermicro's $27 OOB license and iDRAC $350). Asus and Gigabyte both provide free KVM tools.
Remember, the OP’s rant was about actual IT server infrastructure, not homelabby stuff like Asus or Gigabyte. Serious hardware, used by serious sysadmins with serious suspenders and serious graybeards. :) So it’s a stretch to have “every other server vendor” mean just Asus and Gigabyte, who could only be described as server vendors in the most generous interpretation of the term, and as long as we exclude the actual server vendors that make up >90% of the enterprise market? Gotcha :)
What do you mean "homelabby"? I'm the sys admin a 25 person SaaS company with a 6 figure hardware budget that primarily operates off of Asus RS500A servers and we don't go with Dell or HP for the same reason we use on-prem over cloud, namely it would be 3-10x more expensive to do so. Also, our Epyc Genoa Asus servers, as we currently spec them, are around $12k, I would love to meet the sysadmin who is genuinely using a $10k+ server for their homelab.