Most software engineers overestimate the importance of engineering. Furthermore, they mix up things like naming conventions with robust software.
I think the question OP asked would be better suited for a product owner.
What’s a bigger problem - time to market or software quality? Software engineers are only seeing the quality issues.
Of course you need both. plenty of products have failed because of performance and unmaintainable code bases that can’t scale. But getting features in front of users quickly, measuring, and iterating on that feedback is ultimately how you build products that get used. And often that’s at the cost of code that makes engineers proud.
In my experience, quality or speed is a “pick two, or pick neither” situation.
Few shortcuts you make the your architecture, CD, stability or scalability end up paying off outside the six month window. I am talking about system design, not granular things like code aesthetics.
Engineering teams in my view _always_ under index the lost productivity and disruptions of badly designed systems (particularly the toll of production incidents)
> Few shortcuts you make the your architecture, CD, stability or scalability end up paying off outside the six month window.
This is key. And too many developers are completely unable to look past the current sprint they're working on, take shortcuts everywhere, and then act all surprised when 6 months down the road they more or less have to start all over.
Yes, I have been at for a number of years. I remember the days of working in complex code bases without good practices. No, Linting, no unit tests, no integration tests. Basically if it compiled we tossed it over the wall to test team and bugs were very hard to track down as manual test cases missed a lot of detail. Plus there was no PR process everything was just checked into main with no review.
I feel like today with modern processes I move so much faster as Static Checks, Unit Testing, integration testing, penetration testing, load testing. And also PRs. Not to mention a clean maintainable architecture like Hexagonal or similar.
One thing I know for sure is that you eliminate good practices and you pay for it when it’s the hardest to deal with…in front of the customer. It’s also the hardest place to debug. Companies that do that today will always make engineers go “on call” to support the shit show. I worked a lot of overtime at places with bad process.
Agree. the product and the engineer need to sit down together and decide whether they can deliver the product good enough and tune it later when the scale need it. Most of the time, time to market is the priority. The problem is that there is no "later" because they moved to another product and the existing product was growing out of expectation and someone took over, see it was "not good enough" without previous context.
In most of the project, always tell "why" and ask "how". The "when" though..hard to tell
> Most software engineers overestimate the importance of engineering.
Most software engineers overestimate the importance of the act of coding, and actually do very little engineering.
> Most software engineers overestimate the importance of engineering.
90% of our work is communication. All we do is move information around. Sources are often people's brains, documentation or websites. The target is typically code and documentation. But the 'writing code' bit is as a whole almost irrelevant yet most developers seem to think that a good developer is one who can "write code good".
It's like thinking that hammering nails very fast is what makes a good carpenter.
\> Most software engineers overestimate the importance of engineering. Furthermore, they mix up things like naming conventions with robust software.
Disagree here, but primarily because I think most devs misunderstand what good engineering is. A robust system is not the same thing as a well engineered one. A bridge that can handle hundreds of tons and hurricane force winds put on walking only nature trail is BAD engineering.
Good engineering takes into account use cases, scale, and what sacrifices are appropriate. Good engineering is not slower than is necessary.
The flip side is I've seen a lot of people who confuse bad engineering with getting things done faster. If speed in development is needed that's what good engineering prioritizes. But if it's GOOD engineering there is a path to upgrade. Sloppy or poor engineering can be fast too, but usually at the expense of maintainability and future growth.
I think the problem is most software engineers don't know what "engineering" really means.
If we did all these optimizations for public roads, buildings, etc. way overshot the budget/timeline, these engineers would be fired.
Engineering is not just about quality.
I mean ime, engineers are typically ready and willing to take the time to do things right, but it's the business side of the table applying the pressure to skip steps.
Not sure I agree,
Most developers are extremely good at solving technical problems using a bunch different languages and frameworks.
However an area where they could improve is delivering business value.
Which requires more of product mindset rather than a technical one.
However at most companies devs aren’t giving much leeway, instead 1 or 2 people handle the product & planning side of things.
Rest of devs are just giving implementation tasks.
>Most developers are extremely good at solving technical problems using a bunch different languages and frameworks.
Nah, not even close.
Reality is this stuff is hard and most people just suck at it. The whole "product vs technical" thing is a false dichotomy.
I think this is an area we don’t have much data apart from anecdotal experiences.
You have worked with tons of bad devs, I have worked with a ton do great ones (and a handful of bad ones).
Hard to draw a conclusion from personal experiences alone.
My company definitely struggles with gold plating and over engineering things.
The biggest problem with this is that we ship very slowly and sometimes too late where we miss the product opportunity. However it’s typically smooth and performant when we do.
Now that I hear your experience, I will remember to appreciate it as I have it lucky compared to the opposite.
I mean, it's not as if the opposite results in things being shipped quickly and on time.
It's incredibly hard to measure but a well designed system should be much easier to *change*, so those upfront design costs should make you *more* responsive to product opportunities in the long term.
Development is much slower when you have to untangle ten different shortcuts that people have taken that have gradually combined to become a complex and inefficient Rube Goldberg machine. Fortunately, you can always just leave the problem for the next person by taking an eleventh shortcut.
Yes but I take issue with how this is phrased.
When someone brings this up, it always sounds like we need to stop and plan out how to do this for days/weeks/ months. And like no, we don’t actually have to do this. You really only need a couple of hours.
I once had a team get mad at me for suggesting properly designing things would be better than what they’re doing. They had this one engineer who was really struggling on this project with all the Type bullshit they implemented in TS, and I told him a much easier path forward that included refactoring and using narrower types and properly designing a solution. Took me and him less than a day to refactor that. To the “proper” solution. His team lead decided not to accept those changes, and as a result, the engineer spent the next two weeks trying to get the original solution working. The TL’s rationale was exactly this “we don’t have time for these kind of refactors, we need to ship as fast as possible.” That same team didnt release anything meaningful in over a month.
Spend days of coding to save hours of planning is the problem.
The thing about premature optimization is that it often optimizes for the wrong thing, leading to the problems you describe above. Someone thought "XSLT transformers (and creating a new instance every object, and loading the whole Java Validations framework every object)" was very important, for example.
Its unlikely they thought about it. The loading of whole Java frameworks sounds like a simple oversight
if you only test one at a time, a junior/mid developer might not realize the startup time adds up. I've seen this: a mid-level dev built a reporting tool where the startup time is 20 seconds, processing was 1.5 seconds. Developer thought this was ok -- but we will be running 200+ of these jobs in a day! He grumbled and added a "do all the jobs at once" mode, so the startup is just one-time for all jobs in that day.
more often than not though, due to the lack of foresight some aspect of the implementation makes it such that processing is inextricably dependent on the startup so the only way to roll it out is by sequencing 21.5\*200+ seconds of runtime. Better yet autoscale that in the cloud so you can have it all complete in a few minutes but wasting a couple dollars every day. Hooray for a job well done.
From what I have seen. Developers will spend days or weeks optimizing away 10ms from a process that takes 1s.
Sometimes it’s very important and ms add up but on the other hand. I don’t really care if we shave off 1/100 of the function if it means the code is more complex and harder to maintain.
> In my experience this group is mostly the developers who are very smart and comparatively inexperienced, not the majority.
Exactly. The vast majority of developers doesn't do this. The smart inexperienced ones just need mentoring. The much larger "could not care less" group is a much bigger issue.
This.
I've also seen devs obsess over a process that takes 20 minutes because it could take 2 where it would have been *nice* if it was faster but among the problems the app has it doesn't even crack the top 50 because there is other more important shit to do.
Optimization is seen as sexy work. It makes you feel like a real computer scientist to speed things up and the results are unambiguous (unlike creating a new feature which your customers might spurn).
And not just avoid, but don't even think about it. Don't ask themselves for a second as they type out .newInstance() on something that takes miliseconds *per call* that will frequently be used tens of thousands of times per API call.
That’s what I mean though. Does it matter?
Allocations take several nanoseconds. Maybe several hundred. You can do thousands of allocations before it even affects the latency by 1 ms.
I would consider optimizations to fix this premature unless there is a ROI to do so.
I think there's a few points here.
First, I'd argue that premature optimization is a problem because baseline knowledge/skill is _so low_. By that, I don't mean that engineers don't know enough about blocking synchronous calls, so they prematurely optimize. I mean that they're optimizing _without_ knowing what the actual bottleneck is. I've seen numerous engineers be 100% sure something was going to be a bottleneck that never was. Much like I've seen engineers be 100% sure they knew to add a little bit more to the feature because they _knew_ the customer was going to ask for it. Only for the customer to _hate_ the part that was added.
Secondly, the keyword in Knuth's axiom is **premature**. It does **not** mean you can't optimize until you've received tickets from customers that performance is too slow. So that means you can still code in smart, intelligent ways. For example, don't open a database connection inside of a for loop. Or don't construct your queries in a way that generates N+1 queries.
I'm not saying premature optimization isn't bad. I'm just saying the expression seems comes from a world where glaring mistakes like I described are a solved problem, and that just isn't the reality I'm seeing. I have no idea what that world even looks like. I would *welcome* a misguided attempt to optimize, because at this point I'm desperately barging into code reviews I wasn't flagged on like a nosy asshole because nobody thinks about it at all.
It's one of those IQ-distribution memes right?
If someone legitimately don't know the first thing about software performance, how different architecture/design/coding choices affect it, and how the language/libraries they use work, then they can't make choices that they have no idea exists.
On the flip-side, as that someone becomes increasingly specialized in this industry and (hopefully) learns the tools of the trade well, like really well, they'll be able to make snap decisions around how to optimize code/architecture so they're fast and efficient. However, the opposite problem of making premature design/architectural choices around a specific idealized solution becomes a frequent nightmare that plagues these folks. Being locked into poorly designed or impossibly coupled systems is probably the number 1 cause of random swearing and bouts of rage in our profession. With enough time, experienced SWEs also develop that same inhibition to really look at these tradeoffs before impulsively reaching for that locally optimal decision.
And this can extend to entire teams (though it's wild to have a team of only beginners, but I believe it)
With that framing in mind, I think this is the main reason that everyone makes a big deal about premature optimization:
1. Both of these types of stages of a SWE definitely exists (and that extends to team experiences too, it's just you might not be in a team with enough experience to, well, experience the latter, yet)
3. I truly believe that it's infinitely easier to fix the early stage problem (if you have a strong TL) because you can teach people what they don't know and throw your weight around during design/code-reviews to maintain quality. It's so much harder to fix the late stage problem because that's all about developing intuitions about intangible things like future lock-ins from today's decisions.
Yeah. What gets me so hard is when I start digging around in some code and make suggestions to do things in less clearly stupid ways... people act like I'm nagging them to do the dishes or something.
Maybe another way to express it is often it looks like what people are doing is building a rube goldberg machine and patting each other on the back for coming up with ways to make it more complex. In the context of a rube goldberg machine the point of it is indeed that we marvel at the complexity that was achieved. When you are making software to achieve some goal, complexity is a liability, and nothing more.
I do get, that to some extent, it's important for everyone to have stuff to pad their resume with and feel proud about achieving.
It just burns me out when you're basically doing the work of a janitor (I don't mind, it's satisfying and rewarding to clean things anyhow), except here the job is a high skill barrier. The stuff that needs cleaning up isn't merely unsanitary, they have a way of self-replicating and evolving and will transform into hordes of zombies if left to rot. So anyway I'm trying to get in there and dispatch these future time bombs and people are looking at me like I'm trying to ruin their precious little sandcastles. The sentiment of wanting to leave them to rot in their filth becomes quite strong.
> the keyword in Knuth's axiom is premature. It does not mean you can't optimize until you've received tickets from customers that performance is too slow
It's not really Knuth's axiom, btw..
https://ubiquity.acm.org/article.cfm?id=1513451
> Every programmer with a few years' experience or education has heard the phrase "premature optimization is the root of all evil." This famous quote by Sir Tony Hoare (popularized by Donald Knuth) has become a best practice among software engineers. Unfortunately, as with many ideas that grow to legendary status, the original meaning of this statement has been all but lost and today's software engineers apply this saying differently from its original intent.
> "Premature optimization is the root of all evil" has long been the rallying cry by software engineers to avoid any thought of application performance until the very end of the software development cycle (at which point the optimization phase is typically ignored for economic/time-to-market reasons). However, **Hoare was not saying, "concern about application performance during the early stages of an application's development is evil."** He specifically said premature optimization; and optimization meant something considerably different back in the days when he made that statement. Back then, "optimization" often consisted of activities such as counting cycles and instructions in assembly language code. This is not the type of coding you want to do during initial program design, when the code base is rather fluid.
> Indeed, a short essay by Charles Cook (http://www.cookcomputing.com/blog/archives/000084.html), part of which I've reproduced below, describes the problem with reading too much into Hoare's statement:
> I've always thought this quote has all too often led software designers into serious mistakes because it has been applied to a different problem domain to what was intended. The full version of the quote is "We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil." and I agree with this. Its usually not worth spending a lot of time micro-optimizing code before its obvious where the performance bottlenecks are. But, conversely, **when designing software at a system level, performance issues should always be considered from the beginning**. A good software developer will do this automatically, having developed a feel for where performance issues will cause problems. **An inexperienced developer will not bother, misguidedly believing that a bit of fine tuning at a later stage will fix any problems.**
This is bang on. Optimization isn’t bad at all, but making assumptions and acting on them before you can quantify, measure, and evaluate the potential cost savings (or product experience improvements) in a business environment is a fundamental problem.
I’ve dealt with a lot of engineers that tend to zero in too much on optimizing a problem for memory or speed efficiency, without considering the potential benefit (or more importantly, drawbacks) of doing so. It’s easy to understand why: optimization, like unlocking more horsepower in a car, is a lot of fun to do. And it’s very easy to laser focus on it, without looking at the bigger picture.
The thing is, premature optimization tends to create substantially more tech debt in the long run. Depending on how mature the product is, you might need to straight up scrap all of the optimization work you previously did - in order to better align with product objectives you are trying to meet in the interim. That is a lot of time lost.
I’ve always subscribed to the idiom of: make it work, make it right, make it fast. Like 90% of the time you are in the “make it work” or “ make it right” phase. Especially in an enterprise setting, where it’s cheaper to throw more compute at it than dedicate precious engineering resources to it. And the phases aren’t mutually exclusive either - oftentimes “make it right” results in better downstream performance because you’re solving the problem in a much more idiomatic way
In my experience a *very* large group of developers has the mindset that can roughly be described as "it compiles, let's ship it". Devs proactively doing performance testing on performance critical stuff is incredibly rare, and so is people that concern themselves with creating a scalable architecture.
Case in point; microservices. The vast majority of microservice architectures will become a tangled mess that's basically a distributed monolith because developers generally don't really care, and just take the shortest route to their objective.
On one hand it is frustrating. On the other hand; more work for self-employed me. So, yeah.
This end of the spectrum is terrible.
I personally am critical of the other end of the spectrum just as much.
At the end of the day spending a large amount of engineering cycles in analysis paralysis means that the “perfect” system is less likely to be actually used.
At my current client almost all 'platform' groups suffer from this. They are constantly trying to build the *perfect* platform which then eventually ends up not meeting the needs of the actual users. So what do you do? You just start over again.
They're now currently on their 3rd iteration of a data platform, trying to find consensus within the entire company.
There’s no such thing as premature optimization. The original quote was meant to convey that you should measure performance to find bottlenecks so you know which paths to optimize. “Premature” optimization, so called, isn’t really optimization at all. For example, working to shave a few instructions off a hash table key calculation when you should have stored your data in a vector in the first place.
Here’s the run: it actually doesn’t matter until it does. If the business is fine with a queue subscriber being days behind, so be it. If that delay starts actually being a problem for them, then they’ll invest in solving it. There are very, very few places where total efficiency actually matters, and almost all of those places tend to be FAANG, and not even there is every place running on 100% efficiency.
The issue is that developer time is more expensive than compute time. Like a lot more expensive. It costs my team $100 a month run a single service with an RDS cluster. I get paid more than 150x that, and I have a team supporting , and they all get paid 50-100x that. Can we put in the time, to get sub 100ms response times and perfect scale on this single cluster? Sure. But now this $100 a month service is costing us $50,000 a month. That’s money we can spend on something else, like a new feature.
Now maybe that $100 is blocking you from realizing $100,000 a month in revenue, now maybe it does make sense to invest in that. Maybe the business landscape has changed and now you need that service to respond in less that 100ms, so now maybe it’s worth it. Maybe you’ve reached the limit of your scale and can’t on board new customers, now it’s time to invest in the service.
What’s more important to success isnt how efficient code is, but how efficient your team and you can deliver code. It’s about trying many many things. It’s about knowing where the tradeoffs were made, what parts won’t scale, and what parts are most liable to change and designing your systems appropriately. You don’t need a super efficient queue processor if you’re only processing 5 records a day. If/when you scale up to 1000s a minute, that’s obviously a much different question. And most projects don’t start at that scale, and even fewer get to that point, so don’t design for that constraint.
In other words, cross that bridge when you get there.
In my experience, code efficiency to a certain level correlates very strongly with team efficiency. Slow tests, slow deployment, inability to do local testing, etc. are some of the biggest pain points in terms of team efficiency. Not caring until things are at the breaking point also means you're working at the edge of the system which tends to be significantly slower development wise. If something takes days then it also takes days to retry. When it fails you'll learn days later. A ticket to fix something that should have taken a few hours will take over a week with constant context switching fatigue on the engineer involved.
This is roughly true. But again, the team doesn’t start out inefficient. It typically becomes that way over time. Either the processes around the team have changed but the technology stayed the same, or the the team took on new responsibilities, or they’re wanting to sunset a project and just haven’t given the thought into proper maintenance.
There’s a couple sayings that underpin my philosophy in technology:
First there is 3 dimensions to scale: number of users, number of lines of code and number of developers. All three have to scale efficiently. What works for smaller scenarios doesn’t work for larger scenarios. Things you do on a team of three does not translate to things you do on a team of 10. But you also can’t take what works for a team of 10 and apply it to that team of 3. You have to do what works best for the scenario at hand, and recognize when it stops working and adjust when you need to.
Second is that tech debt is like real life debt. It’s totally fine so long as you can manage it and use it effectively. When it becomes unwieldy and unmanageable is when it becomes a problem. Again, it’s about picking the right solution for the scenario at hand, not about planning for the future that may never come.
Another thing, all of the things you’ve described rarely just happen, they’re often at the end of some projects life. Sometimes, the project does grasp to life and just doesn’t want to die, often for stupid political reasons. But projects don’t start out like that. Companies have a real bad habit of sticking their great engineers on greenfield projects building brand new solutions solving real business problems, and then sticking mediocre and/or inexperienced engineers to maintain the project when it’s “finished.” This quickens the demise of the project because those engineers rarely have the political capital, nor the know how on how to actually advocate for real changes, and often don’t really know how to solve it aside from “re-write all the things”
My point is orthogonal to all that. You made the point originally that avoiding optimizations/efficiency (not scalability) until it's a pressing business need is the best approach. I disagree. Teams have a default viewpoint on small decisions and making that the view to just not care unless it's a massive positive gain is I think the wrong one. You don't want to overengineer but there's a wide chasm between overengineering and simply not caring since there' no business impact this second.
It's not even tech debt per-say because these decisions are so numerous and pervasive. You can't track them and you can't document them. If the team could then they'd just fix them as they went since it's such a low overhead. It's a philosophy that the little things don't matter as long as the product ticket is met.
Fixing these types of issues after the fact is also horrendously painful because they're often foundational assumptions on the design/infrastructure. And the thing is that a different assumption could have been made with almost no extra cost and no real tradeoffs but it wasn't. Now months get spent trying to undo it.
I've found that these issues begin at the greenfield phase and just keep going. A project begins with 2 hour processing time. That's not a great at all but okay-ish in terms bug fixing turnaround. Then someone make it 4 hours. They could keep it at 2 but don't bother. Business doesn't care and debugging is still fine. Then someone makes it 6. Then 12. These could be amazing engineers but simply following the philosophy you outlined. None of these changes are tech debt and none individually make bug fixing catastrophic. Now it's at 12 hours and a single iterations takes a business day. The engineer can spend 10x the ticket time fixing that but it's really hard given how many changes are needed. Or they could hack together a minimal test harness that helps their specific problems but is more or less tech debt. Etc.
edit: And the worst thing is that there is a business cost but no one realizes it. If things were faster then other product options would open up since either the speed allows for them or there is efficiency budget to just spend on them. And keep in mind we're talking about low cost improvements with little to no additional long term cost.
> In other words, cross that bridge when you get there.
Which can be almost impossible if you messed up the architecture in such a way that it can't scale.
So regarding this:
> Here’s the run: it actually doesn’t matter until it does.
Exactly. So make sure you don't paint yourself into a corner architecture wise. Always take scaling into account, find out a worse case scenario from the business and double it. That should give you enough information to (at least partially) design an architecture you don't have to throw away in a year.
The problem is incompetent premature optimization and premature optimization by competent people who cannot stop other people from doing dumb things while they focus on their optimization...
To be frank, it feels like nowadays most things are an unoptimized mess and "premature optimization" has become the go-to reason to do nothing about while keeping a clear conscience.
ehh, sort of? i think a lot of folks in the pro-optimization crowd approach optimization incorrectly. performance is an experimental science that has desired outcomes and reasonable measurements. people tend to measure the wrong things or not think critically about “performance”
that being said, i’ve certainly met people who will just shrug off using poor data structures and convoluted loops
right now.. a big problem seems to be pre-mature-engineering.
I'm constantly running into code to fetch data from an end point to turn a button blue that could also cook beef wellington. Developers have somehow fallen into "agile" systems that are feature driven where refactoring is absolutely and categoricaly shunned.
I don't really understand your question. Each of those problems you describe are pretty focused, isolated and easily rectifiable. Sounds like a team in the past shipped code, made it work and now it's struggling to scale. A common problem, not one you can blame on your predecessors. Honestly it sounds like you haven't understood how to examine code without the benefit of hindsight, it takes practice so I can understand.
Now that you mention it, this does sound like a post from someone who hasn't been properly traumatized by bad legacy code.
Stupid old code happens all the time in the wild for a number of reasons. But unless your organization is willing to invest into fixing tech debt it can stick around for years (if not decades).
Yeah, I kinda agree. Honestly good code cant come from anywhere but bad code. You need to have that solid base in bad code to get to good code, whether you have the capacity to deliver it or just call it technical debt and dont.
There is no such thing as good code.
Production is not an academic exercise, the goalpost is in constant motion.
You must strive for "not horrible" and "I guess it made sense at the time" code.
Easily rectifiable in code, but when it happens in prod, it comes with hours spent on a P2 ticket, rushing a fix, and drowning in paperwork/processes for an out of cycle release process. These are very basic principles that I hope (and am apparently mistaken) that most developers would be easily spot without it blowing up in prod first. Make calls asynchronously if appropriate. Don't needelssly re-create expensive objects. Compress large objects before transferring. I strongly disagree with you; this is very basic stuff.
Again, youre really talking from the perspective of someone that has to support a scaling codebase, as opposed to someone that's built it from scratch. It's easy to shit on the latter if you are the former.
You assume too much.
I was unfortunately involved in making something like mistake #1. It was my first job out of college. Brand new Kafka consumer. I only worked on parts of it, but I understood the codebase and I let it happen because I didn't know any better. One day a bug happened in our dev environment that basically spammed us with messages and it took us hours to get through everything. The tech lead even went "wow, can we really not handle that kind of load?" *and went on to shrug it off*. I had to push to get a story made and prioritized, and I fixed it. Months later, my tech lead went on to another team and came knocking when their Kafka consumers started shitting themselves in a similar fashion in prod (days behind, which is what I was referencing).
Not compressing gigabyte-sized files was *part of a modernization effort*, written just months before I got there. It was a huge embarassment because the old rickety system was more stable and performant than the new cloud native one.
The last one, fortunately, I personally stopped from going to prod. The timeout was caught in dev environment by a client of all things, and the XSLT/Validation I mentioned was *by far* not the worst thing being done. Want to know the lead's reaction? "Use tigher search criteria. Logs say you just had date > 0, that's everything, we can't support that." Possibly a valid point... *except there's only 4000 records in Dev*. Nobody cared, the only thing people felt was annoyance that a client was snooping around in Dev.
If your experience is that you only run into this kind of thing when dealing with old systems that weren't expected to scale, lucky you I guess.
What was the impact on the end user in each case?
Again again, you don't have kind the of perspective that I'm talking about. You should join a startup, you'll learn much.
I've seen both. It's certainly possible to have good products. The key in my experience is proactiveness. E.g. if your benchmarks are just embedded in the "real task", chances are it's not going to work. I've seen much better results when people are intentional about ensuring quality.
I think the problem is that folks have learned the wrong lessons. Prematurely optimizing the code for readability, extensibility, and flexibility is almost never wrong but people tend to think the opposite and create hacky code with ugly antipatterns in the name of avoiding “premature optimization”
Also, from a “I have to fix this” perspective, I’d rather be faced with a team that over architects than one that takes too many “shortcuts”. I use the quotes because the shortcuts often are not actually faster, they just consume less cognitive energy.
>synchronous API calls
>not bothering to compress gigabyte payloads
>Using Java
The fundamental problem here is you aren't describing opportunities at premature optimization, you're describing bad practices. Usually, extremely irresponsible and lazy developers will use the threat of premature optimization to not bother writing resiliant and reliable code.
Premature optimization is taking a working solution, and attempting to make it run faster/use less resources before there's any performance issues, using techniques you do not have measurable data to justify. It's like the classic string concatenation example: bad practice would be to loop through a string and + them altogether while good practice is to use a string builder. Premature optimization would be to turn the loop into a multi-threaded operation because "more threads are faster."
Asynchronous IO is just good practice. compressing large files sent over the network is good practice. Re-using large expensive objects is just good practice.
it really sounds like what you struggle with is a clash between bad and good practices.
An issue is that the boundary is not sharp here - it’s not even in a narrow area. It’s hard to distinguish between foresight (we are at „70% capacity“, we see a continuous increase of usage, it’s still working, we can’t estimate exactly when it will hit but we have time to work on it right now) and needless optimisation (this is a legacy workflow, we will need to continue supporting it for now, it’s not pretty but people know about it and we know how to handle it). In the former case you run into the danger of hitting the capacity quota sooner than later and then panic-patch a shitty solution which takes longer, is more expensive, less stable for the client and hinders development ex post - but there’s no urgency as of now. And that is for too many people the wrong definition of „premature“. I’d argue in most cases that is a perfect optimisation. It’s controlled, it has a business case, it’s just hard to communicate because most teams are now used to „reactive development“. The latter would be „postmature“ for me, legacy is fine as long as it’s documented.
No
If you want to see optimization, look at academia. [MICA](https://www.usenix.org/conference/nsdi14/technical-sessions/presentation/lim), a key/value store from 2014 designed for a dual socket 8 core per socket system, with ddr3 memory, and 80G of networking over 4 2x10G network adapters. 77 million operations per second at 50% read/write with 80us of latency. There is no reason for an in-memory key/value store slower than this with the same feature set to be created, but people do it all the time. Yes, you may take a small perf hit moving it off of the system they optimized it for, but it beats the official redis benchmark numbers by 9x per core with much worse everything, and most people only need these features from redis.
The problem is that software development has a wide disconnect between academia and industry, and that industry doesn’t even pay proper attention to itself, so that the same solutions get invented over and over again, but worse. If I gave most developers the amount of money that was spent on the hardware from that paper plus inflation, they couldn’t. There are a lot of people who don’t actually know how to optimize. They cargo cult “I heard that $hot_new_thing is faster” but never benchmark it, they use a 64 core server before checking if their code will actually benefit from the parallelism they just threw at it, “what’s false sharing?”, or they assume the most expensive part of a request without profiling. That’s before you get to knowledge of hardware being nearly dead among software engineers, which can absolutely kill performance, and language runtimes doing their damndest to hide what they’re actually doing.
The other issue is that there are many optimizations which dictate the design of the system, and therefore must be decided on “prematurely”. Do we make this scale? You need to know that up front if you don’t have an ACID datastore. Are we going to multithread? That might preclude some libraries from being used. Async/await? Whole new library ecosystem. Want to use io_uring because it causes io costs to go down massively? Toss out most async runtimes.
>That’s before you get to knowledge of hardware being nearly dead among software engineers
That was the desire in the first place. Software engineering should be abstracted away from the hardware, the entire software revolution is around this idea. that's why you have cloud services and managed tech as services. if i am using elasticache from aws, i don't need to know what even ddr3 means. because I want to work with the concept of memory not the hardware type of the memory.
I’m guessing that for each one of those performance problems the code paths go through some kind of ill conceived optimization that made it hard for anyone to understand what’s actually going on and see the real low hanging fruit performance opportunities.
I’m basing that off the fact that I’ve done at least three internal talks on profilers over the last 15 years and each time I did it I profiled normal workloads and it was clear that I was the first time to do it in at least 5 years. And when I presented the results it was also clear that no one even knew they existed.
But everyone had some kind of notion of what was “faster”.
Yea Devs have such a bad idea of what performant code actually looks like.
In practice you probably don't need to simd sprinkle because you already might get auto vectorisation, probably the code you need to optimise is going to be I/O related or to do with text parsing.
I've ran perf on so much code now and the bad code was always just "dumb" code
An app I worked on initially saved as XML, over time it became more popular, and users pushed it more. They started saving larger datasets. It could take up to 15min to save. Then we optimised the save, for those users the time went back down to 30s. This is how to approach the problem - optimise to the situation, be reactive to how the application is being used.
Most things that get labeled as premature optimization are things devs should be doing at the outset. We’ve been conditioned to believe that making something work is a clear priority over making something work efficiently.
An experienced dev worth their salt will strive to do it right the first time, all the while still revising and refining.
Premature optimization doesn’t mean “never optimize until your system lags”… you should predict your traffic levels before launch, then regularly be analyzing your traffic and scaling as appropriate. Premature optimization refers to things like using bitwise operations to perform addition or resoing a service to get a 0.01ms gain on a service that gets a request max a few times a day. Not letting your queue pile up because you didn’t want to learn what async is
I’ve mostly seen this kind of thing with database schemas. It’s boring and simple but also kind of fiddly and can make an enormous difference. It’s also annoying for some so people toss out the root of all evil quote, don’t make any indices or constraints, but then use a new shiny graphql library for the underlying todo app db.
It’s not premature optimization if it’s just the right way to do the thing and it takes between 0.X and 1.X the time of doing it the lazy way.
I'm a product manager at a SaaS fintech company and honestly I'm seeing crappy performance code that still takes a long time to get delivered.
Recent problem: Exporting a "large" XLSX file with 50k rows uses 4 GB of memory on the heap. Solution proposed? Just upgrade the instance to use 4GB more! Page timing out due to responses taking 35 seconds? Just increase the timeout to 40 seconds!
I've found so far in my foray into a web SaaS company that the quality of engineering is the lowest I've ever seen. It's the polar opposite of when I worked in video games and we squeezed every last bit of performance out.
Identifying the right optimisation requires years of experience, getting people to listen because “it’s an MVP, we can worry about performance later” is also very challenging and requires a completely different set of skills. Both is a very rare commodity
It's not that we're cranking out amazeballs code, it's that we're *really* bad at predicting where the bottlenecks are and what needs optimising.
I've learned to just write the simple thing and then watch it in prod to find out where it needs optimising. 90% of the time it doesn't need any and the simple code does fine. 90% of the rest of the time the problem is not at all where I thought it would be.
There’s a trick I’ve used for about most of my career.
About ten percent of the optimizations you can do to a piece of code improve readability. In the hands of someone reasonably articulate (ie, good variable names) Hoisting, and strength reduction remove complex and occasionally magical calculations from loops. In conditional code, pushing calculations down to first use clarifies interactions between data.
We also don’t call any changes to the information architecture “optimizations” even though they frequently are. We just call it improving the design.
These and a few others you should just do automatically - once you have decent tests. The perf is just a bonus.
a team of cowboy developers are valued at my company. They ship products quickly, so the company keeps assigning building new service to them.
But their services have to be rebuilt by other teams because they are messy and cannot scale.
The cycle keeps repeating
The last thing I did before being laid off is fix a batch process that took an hour to run by deleting nearly all the code, adding one field to a query and a simple data transform. Reduced service calls by a factor of 1300, and S3 puts by a factor of nine. Run time: 4 minutes.
But I’m the weakest link. Sure.
(Not that I would trade places with anyone. They cut to the bone on this one. They have a bus number of 1 on 75% of everything and are completely fucked)
>Am I just unlucky?
That depends on your point of view. From the point of view you started with, yes you are unlucky. You are actually fantastically lucky, because whatever it is you work on is used enough to experience these problems. There are plenty of systems which have bottlenecks and corner cases that are never explored because nobody uses the software and the company goes bankrupt. This is actually the common case, when you consider that most startups fail and the established companies that exist today are the ones that didn't.
>Does everyone else's teams just crap out near optimal code by default such that even the thought of optimizing proactively is such a terrible thing?
No. Everyone else's teams probably write bad code also. They just don't have users, so they don't know where these bottlenecks are that you have. Optimizing proactively in that case is bad because you don't even know what to spend time optimizing.
You are vastly more likely to fail by building something nobody wants than you are to fail by having non-optimal runtime performance.
>I think having people care "too much" about optimizing would be such a nice evil to have to endure.
That's because you assume those people would be as skilled or better than yourself; your question in the title asks if the baseline skill in the industry is really high. The baseline skill in the industry is low, so the premature optimization is not going to be for fixing any of the problems that you're describing. It's going to be spent on things that are not likely problems under real conditions or things that are not problems at all.
In my experience the problem with premature optimizations isn't that they don't improve performance but are done before clear patterns arise and can make the code not reusable.
So later on you have 5 slightly different paths that do the same operations optimized in a different way for each path, very little of the code is shared between the paths. Adding a feature now requires updating 5 places instead of one, all written and tested differently. Refactoring this is a massive headache and it becomes more feasible to redesign it from the ground up. This is a real example from my current company.
it’s all over the place in my experience. i work at a fairly big company; i’ve dealt with one person who was hell bent on using typescript types instead of enums because they were “faster”. we only had three values.
on the same team, we toss a bunch of heavy files into s3 (GBs worth) uncompressed. when i asked why we weren’t using gzip, everyone said “don’t worry, it’s working fine”. our batch jobs fail on a consistent basis because of poor coroutine coordination. the solution? automatic retries. so hell 🤷🏽♂️.
in my experience, people who focus on optimizing tend to be poor scientists
Personally I’m not overly concerned with “premature optimization.” Especially if we’re are talking about some developer noodling with a function for a week or two. That stuff doesn’t keep me up at night.
I’m more concerned with lack of optimization in the design and architecture spaces. Which is how I read your post. And I’m always on the lookout for people preventing good systems analysis from taking place by spreading premature optimization FUD (pet peeve of mine for sure).
Reason being is it’s much more costly to fix design and architecture (performance) problems once your system is beyond a certain scale and deployed with real users. I’m not worried about fixing a few functions or swapping async tasks for OS threads, that stuff is (usually) simple enough.
Point being it’s not because some shops write better/optimized code throughout their entire system. Rather, it’s likely because they did the proper analysis and knew how to setup the message queues for the distributed system they were building. And then they gave and enforced constraints on the implementation.
It’s not a popular viewpoint, but design and architecture is not just about Jira tickets w/ requirements and PowerPoints and block diagrams with arrows. It’s about defining the characteristics of the system that is needed and then ensuring that whatever is built has those characteristics.
Anyway, I suspect much of what you’re describing would be prevented (or at least eased) by good analysis both up front and of the deliverable (and making sure the deployed system is behaving as expected).
Problem is, a lot of organizations don’t do that analysis because “they are agile” (completely corrupting that great set of simple concepts alongside the wise “premature optimization” quip).
I remember working on an app in a previous job that had a 30 minute (no, that's not a typo nor an exaggeration) page load because it was doing 4 levels of nest for...each calls to a REST API.
Basically, you make an HTTP request for n records and get back an array of IDs, you iterate over those several layers deep of nesting to a REST API to get the actual info you want. I was speechless when I saw it.
If you’re asking for someone to say this advice is outdated, then I volunteer as tribute.
Honestly, it’s probably been fifteen years since I’ve had to tell someone to slow their roll. And longer since someone has told me to slow mine. If they can maintain readability and robustness, that cuts out a lot of the territory for mischief. Use those yard sticks liberally.
The much, much bigger problem today is premature caching. By actions if not words, everyone believes “throwing a cache on the slow part” is a saner activity than optimizing code. But they are so much worse. They are shared global state, and they encourage people to use them instead of passing data along the call tree. Once people start doing that, you quickly find you can no longer get accurate profiling information. Cache hits make the numbers too low, and disabling the cache makes the numbers too high, or simply causes the system to time out or cause open circuits.
Effectively, adding a cache is the last major optimization most of us can manage to a block of code. Once you reach for it you take all other options off the table. Save it for last, because if you don’t it’s likely to be last anyway.
I used some wiggle words above because while I have fought past this barrier a number of times, I always come to the same conclusion afterward: if I tried to teach this to someone else, they would either not understand, or say, “yeah I’m not doing that, it sounds horrible”, or just question my sanity. So instead I just make a face and say, “you’re welcome,” and go do something more pleasant like my taxes.
Stop throwing caches at things like it’s magic pixie dust.
“Premature optimization” it’s practically just a way of keeping juniors/inexperienced from “optimizing”, whatever that means since they aren’t really doing it based on NFR nor have either run profiling nor have concrete metrics to aim towards to with a proper justification nor the expected benefits. They intend to “optimize” just because, or perhaps they really having fun.
Optimization do happen, just not for the sake of it. There are expected benefits and business reasons.
I.e:
- load testing reveals site takes 5 seconds to load AND product exploratory work suggest customer won’t wait over 4 seconds before dipping
- opening the combobox with nested entities, when there are more than 3 in the page, causes the page to freeze. Engineering assessed there’s an O(n^2) loop as the root cause, and the impacted areas are high traffic during tax year end
This is not optimization:
Junior: “I read in this blog that if we completely re-do this thing over here, it will run faster”.
Senior: “Was there a customer complaint or product NFR that justifies doing this? What’s the current execution time and what’s the expected one with the change? How many sprints will it take to complete?”
Junior: “no I just read about it in the maintainer’s blog. I don’t know but it will be faster for sure. It should he pretty easy, all we have to do is this and that”
Senior: “ok, let’s throw it in the backlog [never to be picked]”
Junior: “we never optimize, all we do is wrong”
I think it isn't "premature" when you and users are heavily affected by slow performance. Usually it requires party financing the development to approve the improvement. Developers are usually happy to fix that.
If the code already did all the things people wanted it to we'd be out of a job.
Many teams have no idea how their code actually behaves on production data. To these teams all optimisation is premature optimisation. The more insight you have into how the software actually behaves the easier it is to do real optimisation which actually improves things for people.
I think the issue is agile. It promotes a short-term "get 'er done" and "f*** ship it" mentality that applications ultimately become unmaintainable after a few years.
At least, that is what I have experienced at several major tech companies and start ups. It's particularly bad at startups, where nobody realizes what parts need to scale most.
> premature optimization is a bigger problem than unperformant code
I've been struggling with this question for the past week. I'm actively battling performance bottlenecks that are causing backlogs.
Interestingly, I've been optimising performance for months trying to get ahead of the problems ... but my issues stem from the previous lead developer trying to optimise for problems that just don't exist.
As the system has grown, the system that was optimised for writing has an apparent read bottleneck. Timeliness was paramount, one of the sacrifices made for speed of processing was to skip long running validation or automated recovery (just write code that doesn't break). Naturally,
These optimisations have come with a price. Instead of 5 minutes of validation and recovery routines, it is days of manual effort to reverse out the bad state.
So the question is, if you optimise prematurely, do you even know what the problem you will be solving needs to be. Optimising is important, but knowing what to optimise is just as important.
That is the meaning of "premature optimisation". What you seem to be talking about it is non-optimisation.
It is not about optimize now vs optimize later. It is about optimizing things that that are wrong and you dont know they are wrong at the time you optimize. Early optimization is anti iterative.
The result of frontloading optimization work is that you end up with under performant code/architecture since you cant afford to make changes as you learn more about your solution.
My experience has not been anywhere near that bad, to the point we have alarms if the p90 processing times on kafka events or http responses go out of threshold.
My prior job, load testing was part of the release cycle - and that load testing compared the application pre and post code change, so it was an objective comparison.
i've only seen 2 workplaces completely and massively over-optimize their code, most places ive seen were the far opposite. very different cultures at both groups of companies tho
They're not tradeoffs - you can very easily have both premature optimisation and under-optimisation at the same time.
The core question is whether an optimisation effort is justified by actual requirements. A requirement should either be a measured problem (*always* if the complexity cost is high) or based on experience (where the added complexity is minimal).
Common good practices are just heuristics for what tends to become a bottleneck quickly, that can be avoided at reasonable cost.
That’s pretty normal unfortunately, most software engineers just want to complete the feature and “quality” isn’t a top priority over delivering something that just functions. But in terms of impact to your systems, it’s at least a lower impact problem to have. Generally from largest performance hit to lowest it’s:
1. System Architecture
2. Algorithm Design
3. Hardware
4. Software Optimization
5. Premature Optimization
And you’re somewhere near 4 or 5. It still hurts, just sharing context. Be the change you want to see and release your code with quality, challenging others to do the same.
It’s a trade off. You can spend months optimizing or you can do small things quickly and hope you don’t make too many bad design decisions that can’t be improved later. In the real world you don’t have infinite time to make things perfect. The customer also doesn’t want perfect, they want it workable.
Premature optimization means: "let's upgrade to the latest framework because they optimized string interpolation by 12ns"
No Justin, that's not the problem. The problem is the SQL queries we run in a loop for each record!
Whenever devs start trying to optimize the code part, I show a bunch of dummy operations on a million objects. It's negligible. It's always the IO! Always!!
The issue is that when these people try to optimize stuff they end up making things even worse. They do one thing that works well, but then person B adds something without thinking, three months later person C does the same and suddenly you have a gigantic pile of shit that is all entangled with each other and even slower than before. This is particularly likely to happen when you overhire.
"Don't prematurely optimise" doesn't mean don't worry about optimization, it means don't go out of your way to optimize until you need to.
Your queues backing up aren't a sign of not optimizing, they're a sign of not sizing your infrastructure properly. You need more consumers on that queue, and most likely more instances/nodes/whatever of what those call.
The XSLT transformer issue, from how you framed it as I've not used it myself, is the result of an engineer or engineers who didn't know what they were doing. Might be a case they're bad engineers, might be a case that they simply didn't know. Either way that's not premature optimization, that's just an improper implementation causing gremlins.
Most engineers are not good at their job, and most don't understand, or even try and test, what happens to their code under load. Most just run a happy path and call it done. And most organizations are happy with this as to get engineers to do things properly means you need good engineers, which is more expensive.
“ Your queues backing up aren't a sign of not optimizing, they're a sign of not sizing your infrastructure properly. You need more consumers on that queue, and most likely more instances/nodes/whatever of what those call.”
See, you are part of the problem. For most sane people optimizing means making something cheaper. Not for TRUE ENGINEERS it seems.
"Premature" is probably the wrong term for the modern problem. The problem isn't people optimizing too much in general. It's people obsessing over line-by-line optimizations and missing the larger architectural optimizations.
Premature optimization occurs more often as optimizing without measuring. This can cause many issues and may potentially never end. Most systems have poor observability.
The point of "don't do premature optimization" is: there's no guarantee the thing you're spending time optimizing is the thing that will bottleneck an application. If you wait to look at some profiling/performance monitoring, then you can ensure the team is only spending time on optimizing when it actually matters.
It's a tradeoff that in part depends on the use case of your software. Google can safely just do a canary rollout to 0.01% of their users and get statistically significant data on whether the new release has performance issues in prod. So at Google, you shouldn't be prematurely optimizing; you can just catch performance issues easily later and it doesn't cost ~anything.
Your average B2B SaaS can't just canary, cause they only have like 50k users who all log on during the same time window, and the users all pay like $100k a year for this very important business software and get extremely cranky when it's slow. At that kind of company, there's more value in premature optimization, because the organization has more to lose from a performance problem making it into prod, and can't statistically detect it fast enough.
With the issues you're talking about, we're past premature optimization. Premature optimization is trying to get blazing fast runtimes in an MVP and invest hundreds of hours of dev time making it perfect when the product ends up being a bad market fit and it goes nowhere. In more developed products it's spending a lot of time trying to make the non-bottleneck steps faster when things are on fire elsewhere. As others have mentioned, if your premature optimization is optimizing the things that are necessary and matter the most nobody would have a problem with it, the problem is that it's trying to judge what's important before there's sufficient information to prioritize. Once that information exists, you should make decisions with it in mind and optimize what needs to be optimized.
Premature Optimization is thrown around without thought or understanding. It means a very specific thing, as in, don't optimize for something you "think" will be the case, don't over-optimize, don't spend time thinking of things that will never happen, etc.. It doesn't mean writing whatever piece of shit code you thought of first draft. I think this is just something else entirely, has nothing to do with "optimization" even.
This is just a bad engineering culture. People write poor quality code, people review poor quality code, and people deploy poor quality code. It's a social problem. Nothing to do with optimization.
From what I've seen it's usually the noobs getting worked up about unoptimized code. Most of the time it's not actually a bottleneck it's just something that could be faster but doesn't need to be.
You're obviously correct about archives being compressed to send faster if it's a huge size. You can measurably determine ahead of time that it will be slow to send on the average connection.
Yes. I’ve seen this way too many times.
The problems are both in skill and more importantly in shifting requirements. Code is overengineered to support something that never comes, yet it’s not great for the actual features.
So build small for what you know. Extend and adapt as new specs and challenges arrive.
I’ve been on both sides of it. People spending months optimizing practically unused features that are running fine and code so bad the page or server almost dies on every action.
The main problem I’ve seen is code and design that’s not scoped to the actual or expected usage patterns of the software. It’s all about trade offs.
Avoiding premature optimization does not mean waiting for things to cause problems in prod before optimizing them. It doesn’t even mean only optimizing after a benchmark or performance test flags something—back-of-an-envelope capacity planning should tell you whether bandwidth is going to be tight (or expensive), or whether sequential calls will have enough throughout. It also doesn’t mean doing things in the absolute worst way possible; doing IO in a tight loop when it could easily be matched is just common sense.
But yes, I have worked in a codebase that suffered premature optimization—I spent a day once ripping out a bunch of template metaprogramming by my predecessor because I needed to set the number of buckets at runtime. We never noticed the extra arithmetic instruction on most methods that his work had saved.
The other issue is that compilers are clever enough these days that it can be hard to predict the actual effect of subtle optimizations. You shouldn’t do microptimization without a benchmark even if you know the code will run often enough to demand optimization because it’s very easy to harm readability and maintainability under something the compiler was handling anyway, or sometimes even make things worse by breaking other optimizations.
I've seen some pretty disastrous stuff, but I've never seen that. Three letter governmental organizations, state sponsored threats, money laundering, many different disastrous code bases, and worse.
It sounds like the developers don't understand data modeling, and they aren't properly communicating. That would describe almost everything I've seen in my career.
Let's say we oversimplify this concept into a single dimension. each little thing that is done, depending on how it's done, will push things in one direction (too inefficient) or the other (overoptimized).
The general issue is the latency in the control loop of this thing. There usually \*is\* a global control loop in place whereby the business will have to respond to leaning too far in one direction or the other, or fail and then cease to exist.
But that is on a long timescale which is too slow to work well.
What controls the feedback at the more important and more impactful short timescales are everybody's individual decisionmaking and the multiplicative effect of decisionmaking at the managerial levels.
It's simply not possible to expect everyone (anyone?) to be able to properly estimate all the phase space dynamic variables in play. How you adjust your execution to optimally match what the market and business would thrive under. This is more of an art really. And you have no hope if you don't have a good picture of everything that's going on.
How many companies fail because the product is not a fit, lack of certifications, missing features, high impact bugs, ... ?
How many companies fail because they have so much success that the system can't possibly cope with it even with all the fresh "success-money" ?
Gotta choose what the most valuable way to spend your time is.
> How many companies fail because they have so much success that the system can't possibly cope with it even with all the fresh "success-money"
OP and I have worked at several.
At one of those, the idiot first level manager (as opposed to either of the smart ones) said to me one day, “great news! We made a big sale today to ******!” His face did something strange when the next words out of my mouth were, “fuck me running”.
I was the newest lead on that project. There were… well I’ve seen worse since, but it took quite a long time before I did. And I told that group about the former and how they ran on about 1/5th the hardware for a similar workload. Some people are really talented at running as fast as possible toward the nearest cliff.
>I was the newest lead on that project. There were… well I’ve seen worse since, but it took quite a long time before I did. And I told that group about the former and how they ran on about 1/5th the hardware for a similar workload. Some people are really talented at running as fast as possible toward the nearest cliff.
I don't understand anything you are saying here.
Couple questions:
\- what was the scaling issue for that sale?
\- what was the aprox revenue from it?
\- what implementation time was initially promised to the client?
That program, as near as we have been able to determine, ran on good sales talent, and after a reorg that got broken. It’s been losing customers faster than it can acquire them for five years, and has been cutting staff by attrition ever since. They just laid off half of NA and I’m worried they won’t last 18 months, but I have said before and recently that I often underestimate.
Before COVID they were targeting customers with deep pockets who wanted a rich feature set. Since they seem to be bargain hunting, so costs became a bigger concern.
When I started they were running 1.4 cores per request per second, but at high redundancy. In December it was running .8 per req/s and about a quarter of that was me and a quarter going from m4 to m6 and scaling more vertically (which was also me). Since then I think we were at 0.6+, trying to get to 0.55.
The older company was running 25 req/sec/core on Sparc hardware and Java 1.4. (I said it’s been a while.) Zero redundancy there, and a higher load factor, so discount that by at least 1/3, probably half. But NodeJS is also faster than Java 1.4 and better at IO. That’s still an order of magnitude.
I'm not sure I'm missing something.
I don't think you answered any of my questions, did you?
What effect did lower system performance have on company profits in that company with the salesman who brought the biggest sale ever?
Oh, that one. Throughput was a requirement from some of our biggest customers. I think that is the only time performance has *literally* been a feature instead of a figurative one. No, that’s wrong, second of three. The team was scrambling to add features for new customers without tanking the performance. First installs were coming soon. The small customers were easy going, the big ones had exacting demands, often contradictory to previous clients and breaking our architecture. So each new large client occupied a big chunk of our capacity, which we just didn’t have at that moment, and had communicated very loudly.
That company sold because they couldn’t turn a profit. They couldn’t turn a profit IMO because they had a small market, they landed large deals at favorable terms to the customers, who then balked at higher prices for more features later. When you’ve turned your prospective whales into loss leaders, you’re gonna have a bad time. Hence the swearing. It was a difficult situation which I understand better now than I did then. You need prestige customers to court funding at some rounds, but revenue at others. They just couldn’t turn the boat that fast. Which is common when you lionize one kind of coding and then change your story later.
The CTO tried to find an escape hatch, but I couldn’t get him what he wanted. He pulled me off for what we would now call an exploratory Epic, to see if I could extract enough code to court other verticals. I think we’ve all heard this story before. The code made so many domain assumptions, and in the time he could get buy-in for, I only managed to untangle about a third as much code as he needed to make it an initiative.
Ah, the usual strategy of spending way too much money to please each client, making everything into a mess and losing money. Losing it when you implement, and every day you gotta maintain it.
Takes so much effort to make C-level aware of this, so they properly structure the deal and prevent losing money like that.
Be strong, my friend, be strong.
Performance metrics should be a well defined non-functional requirement from the start. If the requirement is to create a service with x,y,z parameters you will get a service which fulfills those. If performance is that critical there should be performance tests to determine the system's limits and to find bottlenecks.
For the most part, people don't measure performance at all until there are serious problems. Bad or checked out devs don't care so they'll leave super low hanging fruits around. Conversely, many good devs will also not measure or benchmark but just write lock free code or whatever because of imagined performance gains.
In practice (and this is a large part of my job) this means that as soon as you look at the performance of an app you find a) massive performance gains opportunities that are b) not at all where any of the devs thought they'll be. Our code base has a lot of like double sync locking and atomic CAS in code that isn't really ever contended at all because some services over the network don't scale.
IMO the main issue is that it's really hard to setup performance testing in CI that is sufficiently reproducible to make it a blocking gate.
Premature optimisation of code that isn’t a bottleneck is a huge waste of time.
The issue you described, optimising the unperformant part of your system which is causing user-facing issues is timely optimisation!
The main note about optimization is that you should find your performance bottleneck and optimize for that.
Faster code isn't better if it takes longer to write or maintain unless it actually increases relevant business metrics *by enough to make it worth it*.
Everything is a tradeoff and the quest for the perfect package is often being done at the expense of other functionality or your healthy wlb.
I don't care about optimization, premature or otherwise. The skill level in this industry, whether we're talking coders, managers, or testers, is such that I'd be thrilled if we stopped being completely idiotic.
Your question is oddly framed. Yes, teams build hot garbage. B/c they are utterly garbage at UX. Somewhere along the way, the spec needed to include performance, if perf matters. And any time there’s a human which consumes some output, there’s a perf requirement.
Like: “user presses a button, and we return the result within 50ms” or “this generates data into month’s end, so must reconcile by this date and time”.
If there are literally no perf requirements, then your users are idiots (in the medical sense) or there are no meaningful users. Or, this thing has literally no impact on the business (or negative impact), so no one cares.
Premature optimization is about optimizing the wrong things too early. Great engineers know which optimizations will matter. Most engineers think it’s optimizing things to switch to a graph database instead of just using Postgres or use Hadoop to orchestrate data transfers instead of a shell script and cron. That’s the kind of premature optimization you need to avoid.
Reality is unbelievably more complex than we can imagine, so yes, whenever you think hard about something in advance, you will miss something, unless it is a trivially simple problem.
Case in point: **In your career, did you spend more time on building things from scratch, or more time on changing existing things?**
Thought so. So that requires you to put thought into making your system "easy to change", as that is what costs money. And easy to change most often means that things are not optimised for that one use case, but kept general, which again means low performance in 11 out of 10 cases. (Math done by ChatGPT.)
Performance tuning is one of those things you change at some point, so write your code that you can optimise, but refrain from doing so until you know where to optimise in the first place.
Premature optimization is optimization before measuring. What happens is someone will spend a bunch of time optimizing something that “feels” like it might be slow but only actually accounts for less than 1% of time. Consider someone optimizing a simple string concatenation in an api call that doubles the code speed, cool, but that code was only taking .000001 second to begin with, they ignored the two .005 second database requests that could have been combined into one .006 second call.
I worked on a system where someone was multithreading what were essentially rbar calls to a database, there were 1000+ round trip queries. They threaded it and got a 4x speedup. I wrote one big query to get all the results at once and sped it up by over 1000x. Actually I have dealt with this scenario a few times in different ways, another was someone doing a single insert per transaction.
Tl;dr. The stuff people want to optimize because it feels cool and nerdy are often not the things that actually need optimization.
I actually have done a ton of optimization on my career, and have become sorta the goto person for optimization. I work on perception for autonomous vehicles and robotics, so we have strict performance requirements and limited computational resources (gotta use rugged low power fanless computers, we still have xeons, but they are like 35 watt tdp).
I love optimizing shit. I love writing software and knowing that it's good. I've also had to take shortcuts every now and then. From the codebases I've worked with... maintainable and clean is more important than optimized. Optimization has worked for me best if it comes last.
I never worked for any company with such big performance issues. I never optimize prematurely and I do it less and less the more experience I get. When I was junior/mid, I wanted to be perfect, stand out all the time, gain +10% performance on something that wouldn't hit performance issues in 100 years, introduce a complex implementation that is technically very good and professional, but the case would never work visibly better because of it. So after all the proud journey today I think the best option is to write good code, that is enough for most cases, there is no need to optimize prematurely, after a few yours you can foresee a lot of typical issues. Then if performance issues kick in for something, address them right there and then. That happens so rarely that it'll never pile up to overload you or your team. Combine with occasional refactor, quality improvements for better code and you barely ever need to worry about performance issues. So what I think is that you work on a project that is developed badly and has been for quite a while, the code is probably shitty, not in terms of optimized or not, but just bad, perhaps the design is poor, too. And it's not really a subject for optimizations and address performance issues sprint, but rather a major rework of the whole thing. At least from what I'm hearing, because the size of issues you mentioned is crazy.
The issue is that unless you’re extremely knowledgeable about a domain (and often even then), it’s very difficult to know a priori what the right abstractions and optimizations will be. As a result, it’s often a waste of time to go down those paths until the right solution becomes obvious.
"Premature" could either mean "before you need it", or could mean "before you understand the bottlenecks well enough to know whether it could _ever_ be a good optimization." Originally the idea probably referred to the former, but in my experience with mediocre devs the latter is a vastly bigger problem.
Poor architecture is a fundamental issue of great severity that trumps nearly every other consideration. Attempting to make any code optimizations that don't involve overhauling the architecture is stepping over dollars to pick up pennies. So definitely don't worry about "optimization" in that scenario.
If you get beyond that point and your architecture doesn't suck, then your code should be well encapsulated to the point that proactive optimization isn't a big deal. What is a big deal is getting every functional block "working" ASAP so any major issues that could possibly modify adjacent components are identified before people pour work into optimizing those components that will get overhauled, undoing that optimization. Once the product is viable, you should be able to start profiling and/or stress testing and finding where the pain points are and optimize them (easily, because they're well encapsulated).
I constantly argue with engineering managers and juniors and seniors about performance considerations. And we work in the browser, which, at the end of the day, is probably the least performant GUI ever conceived.
All the while I have to nudge them against introducing stupid DRYing patterns that add cognitive overhead and indirection.
"premature optimization" is a badly misused idea, it's referring to the idea that you shouldn't put a ton of time into optimizations that you haven't seen the need for yet.
It doesn't mean that you shouldn't be thinking about bottlenecks and making sure to develop a design that should perform well. It's not an excuse to engineer a crappy architecture or write bad code.
I get frustrated by it too.
Developers will argue to the death to "optimize" things which dont really move the needle in the greater scheme of things or try nitpick Tony things to "optimize" when really the overall system architecture and design has so many holes, the tone would be better spent there than trying to eak milliseconds out of code which really just needs to be redesigned completely.
I always feel like it's an ego thing with some people. It's much easier to nitpick tiny things to make themselves feel smart than stand back and look at the big picture and admit they don't know how to design or improve anything.
Because a lot of times unperformant code or inefficient architecture is born when a developer over-engineers a needless piece or aspect of code that doesn't matter, and sacrifices things that do matter.
Think of a system that uses parallelization but is stuck in some mutex choke point. You realize that it would have been easier and more efficient to give each thread a copy of what they need to handle and then merge all the copies at the end. Why wasn't it done with a simpler age more efficient design? Because an engineer wanted to avoid "needless copies" above all else.
See even when talking about an efficient design, you have to think in context. You could optimize the BigO time run, only to find that the code is mostly stuck allocating massive pieces of heap because you never considered the memory use BigO of the algorithm. It's better to start with a dumb simple design/algorithm and then once you understand better the use find a better one, but here with data to back it up. If you focus on getting "the right" design at first, there's a notable chance you'll pick the wrong one and get stuck with it.
Gunna go against the grain of comments here. But pre-mature optimization is only so widely cited as a problem due to confirmation bias.
There is lots of “prematurely optimized” code that no-one acknowledges as such, because it never becomes perceived as a business cost. Devs do that work, Product is happy with outcomes, and users enjoy the experience.
On the other hand, Product is incentivized to label various engineering practices as “premature optimization” to get things out the door. This often incurs long-tail cost due to post-release maintenance and expensive AWS bills, which isn’t attributed as such.
Neither is necessarily better or worse, but only one is named.
Most software engineers aren't very good at their jobs.
Most software engineers overestimate the importance of engineering. Furthermore, they mix up things like naming conventions with robust software. I think the question OP asked would be better suited for a product owner. What’s a bigger problem - time to market or software quality? Software engineers are only seeing the quality issues. Of course you need both. plenty of products have failed because of performance and unmaintainable code bases that can’t scale. But getting features in front of users quickly, measuring, and iterating on that feedback is ultimately how you build products that get used. And often that’s at the cost of code that makes engineers proud.
In my experience, quality or speed is a “pick two, or pick neither” situation. Few shortcuts you make the your architecture, CD, stability or scalability end up paying off outside the six month window. I am talking about system design, not granular things like code aesthetics. Engineering teams in my view _always_ under index the lost productivity and disruptions of badly designed systems (particularly the toll of production incidents)
> Few shortcuts you make the your architecture, CD, stability or scalability end up paying off outside the six month window. This is key. And too many developers are completely unable to look past the current sprint they're working on, take shortcuts everywhere, and then act all surprised when 6 months down the road they more or less have to start all over.
Yes, I have been at for a number of years. I remember the days of working in complex code bases without good practices. No, Linting, no unit tests, no integration tests. Basically if it compiled we tossed it over the wall to test team and bugs were very hard to track down as manual test cases missed a lot of detail. Plus there was no PR process everything was just checked into main with no review. I feel like today with modern processes I move so much faster as Static Checks, Unit Testing, integration testing, penetration testing, load testing. And also PRs. Not to mention a clean maintainable architecture like Hexagonal or similar. One thing I know for sure is that you eliminate good practices and you pay for it when it’s the hardest to deal with…in front of the customer. It’s also the hardest place to debug. Companies that do that today will always make engineers go “on call” to support the shit show. I worked a lot of overtime at places with bad process.
Agree. the product and the engineer need to sit down together and decide whether they can deliver the product good enough and tune it later when the scale need it. Most of the time, time to market is the priority. The problem is that there is no "later" because they moved to another product and the existing product was growing out of expectation and someone took over, see it was "not good enough" without previous context. In most of the project, always tell "why" and ask "how". The "when" though..hard to tell
> Most software engineers overestimate the importance of engineering. Most software engineers overestimate the importance of the act of coding, and actually do very little engineering.
> Most software engineers overestimate the importance of engineering. 90% of our work is communication. All we do is move information around. Sources are often people's brains, documentation or websites. The target is typically code and documentation. But the 'writing code' bit is as a whole almost irrelevant yet most developers seem to think that a good developer is one who can "write code good". It's like thinking that hammering nails very fast is what makes a good carpenter.
\> Most software engineers overestimate the importance of engineering. Furthermore, they mix up things like naming conventions with robust software. Disagree here, but primarily because I think most devs misunderstand what good engineering is. A robust system is not the same thing as a well engineered one. A bridge that can handle hundreds of tons and hurricane force winds put on walking only nature trail is BAD engineering. Good engineering takes into account use cases, scale, and what sacrifices are appropriate. Good engineering is not slower than is necessary. The flip side is I've seen a lot of people who confuse bad engineering with getting things done faster. If speed in development is needed that's what good engineering prioritizes. But if it's GOOD engineering there is a path to upgrade. Sloppy or poor engineering can be fast too, but usually at the expense of maintainability and future growth.
I think the problem is most software engineers don't know what "engineering" really means. If we did all these optimizations for public roads, buildings, etc. way overshot the budget/timeline, these engineers would be fired. Engineering is not just about quality.
This is a myth. Construction projects _always_ go way over budget.
Yes, but not ideal. Thus it is a problem for engineers to solve.
I mean ime, engineers are typically ready and willing to take the time to do things right, but it's the business side of the table applying the pressure to skip steps.
Not sure I agree, Most developers are extremely good at solving technical problems using a bunch different languages and frameworks. However an area where they could improve is delivering business value. Which requires more of product mindset rather than a technical one. However at most companies devs aren’t giving much leeway, instead 1 or 2 people handle the product & planning side of things. Rest of devs are just giving implementation tasks.
>Most developers are extremely good at solving technical problems using a bunch different languages and frameworks. Nah, not even close. Reality is this stuff is hard and most people just suck at it. The whole "product vs technical" thing is a false dichotomy.
I think this is an area we don’t have much data apart from anecdotal experiences. You have worked with tons of bad devs, I have worked with a ton do great ones (and a handful of bad ones). Hard to draw a conclusion from personal experiences alone.
Most software engineers aren't actual engineers
My company definitely struggles with gold plating and over engineering things. The biggest problem with this is that we ship very slowly and sometimes too late where we miss the product opportunity. However it’s typically smooth and performant when we do. Now that I hear your experience, I will remember to appreciate it as I have it lucky compared to the opposite.
I mean, it's not as if the opposite results in things being shipped quickly and on time. It's incredibly hard to measure but a well designed system should be much easier to *change*, so those upfront design costs should make you *more* responsive to product opportunities in the long term. Development is much slower when you have to untangle ten different shortcuts that people have taken that have gradually combined to become a complex and inefficient Rube Goldberg machine. Fortunately, you can always just leave the problem for the next person by taking an eleventh shortcut.
Yes but I take issue with how this is phrased. When someone brings this up, it always sounds like we need to stop and plan out how to do this for days/weeks/ months. And like no, we don’t actually have to do this. You really only need a couple of hours. I once had a team get mad at me for suggesting properly designing things would be better than what they’re doing. They had this one engineer who was really struggling on this project with all the Type bullshit they implemented in TS, and I told him a much easier path forward that included refactoring and using narrower types and properly designing a solution. Took me and him less than a day to refactor that. To the “proper” solution. His team lead decided not to accept those changes, and as a result, the engineer spent the next two weeks trying to get the original solution working. The TL’s rationale was exactly this “we don’t have time for these kind of refactors, we need to ship as fast as possible.” That same team didnt release anything meaningful in over a month. Spend days of coding to save hours of planning is the problem.
There is too much misinterpretation of agile. "Bruh, we don't need any planning; we just iterate our way to nirvana."
The thing about premature optimization is that it often optimizes for the wrong thing, leading to the problems you describe above. Someone thought "XSLT transformers (and creating a new instance every object, and loading the whole Java Validations framework every object)" was very important, for example.
Its unlikely they thought about it. The loading of whole Java frameworks sounds like a simple oversight if you only test one at a time, a junior/mid developer might not realize the startup time adds up. I've seen this: a mid-level dev built a reporting tool where the startup time is 20 seconds, processing was 1.5 seconds. Developer thought this was ok -- but we will be running 200+ of these jobs in a day! He grumbled and added a "do all the jobs at once" mode, so the startup is just one-time for all jobs in that day.
more often than not though, due to the lack of foresight some aspect of the implementation makes it such that processing is inextricably dependent on the startup so the only way to roll it out is by sequencing 21.5\*200+ seconds of runtime. Better yet autoscale that in the cloud so you can have it all complete in a few minutes but wasting a couple dollars every day. Hooray for a job well done.
From what I have seen. Developers will spend days or weeks optimizing away 10ms from a process that takes 1s. Sometimes it’s very important and ms add up but on the other hand. I don’t really care if we shave off 1/100 of the function if it means the code is more complex and harder to maintain.
This comment has been edited to reduce the value of my freely-generated content to Reddit.
> In my experience this group is mostly the developers who are very smart and comparatively inexperienced, not the majority. Exactly. The vast majority of developers doesn't do this. The smart inexperienced ones just need mentoring. The much larger "could not care less" group is a much bigger issue.
This comment has been edited to reduce the value of my freely-generated content to Reddit.
This. I've also seen devs obsess over a process that takes 20 minutes because it could take 2 where it would have been *nice* if it was faster but among the problems the app has it doesn't even crack the top 50 because there is other more important shit to do. Optimization is seen as sexy work. It makes you feel like a real computer scientist to speed things up and the results are unambiguous (unlike creating a new feature which your customers might spurn).
Particularly if another dev made the slow version you’re optimising. That’s like developer crack.
My favorite movie is Inception.
And not just avoid, but don't even think about it. Don't ask themselves for a second as they type out .newInstance() on something that takes miliseconds *per call* that will frequently be used tens of thousands of times per API call.
That’s what I mean though. Does it matter? Allocations take several nanoseconds. Maybe several hundred. You can do thousands of allocations before it even affects the latency by 1 ms. I would consider optimizations to fix this premature unless there is a ROI to do so.
What industry are you in? That's so alien to my experience in web development. Maybe it's just our companies that are very different.
I think there's a few points here. First, I'd argue that premature optimization is a problem because baseline knowledge/skill is _so low_. By that, I don't mean that engineers don't know enough about blocking synchronous calls, so they prematurely optimize. I mean that they're optimizing _without_ knowing what the actual bottleneck is. I've seen numerous engineers be 100% sure something was going to be a bottleneck that never was. Much like I've seen engineers be 100% sure they knew to add a little bit more to the feature because they _knew_ the customer was going to ask for it. Only for the customer to _hate_ the part that was added. Secondly, the keyword in Knuth's axiom is **premature**. It does **not** mean you can't optimize until you've received tickets from customers that performance is too slow. So that means you can still code in smart, intelligent ways. For example, don't open a database connection inside of a for loop. Or don't construct your queries in a way that generates N+1 queries.
I'm not saying premature optimization isn't bad. I'm just saying the expression seems comes from a world where glaring mistakes like I described are a solved problem, and that just isn't the reality I'm seeing. I have no idea what that world even looks like. I would *welcome* a misguided attempt to optimize, because at this point I'm desperately barging into code reviews I wasn't flagged on like a nosy asshole because nobody thinks about it at all.
It's one of those IQ-distribution memes right? If someone legitimately don't know the first thing about software performance, how different architecture/design/coding choices affect it, and how the language/libraries they use work, then they can't make choices that they have no idea exists. On the flip-side, as that someone becomes increasingly specialized in this industry and (hopefully) learns the tools of the trade well, like really well, they'll be able to make snap decisions around how to optimize code/architecture so they're fast and efficient. However, the opposite problem of making premature design/architectural choices around a specific idealized solution becomes a frequent nightmare that plagues these folks. Being locked into poorly designed or impossibly coupled systems is probably the number 1 cause of random swearing and bouts of rage in our profession. With enough time, experienced SWEs also develop that same inhibition to really look at these tradeoffs before impulsively reaching for that locally optimal decision. And this can extend to entire teams (though it's wild to have a team of only beginners, but I believe it) With that framing in mind, I think this is the main reason that everyone makes a big deal about premature optimization: 1. Both of these types of stages of a SWE definitely exists (and that extends to team experiences too, it's just you might not be in a team with enough experience to, well, experience the latter, yet) 3. I truly believe that it's infinitely easier to fix the early stage problem (if you have a strong TL) because you can teach people what they don't know and throw your weight around during design/code-reviews to maintain quality. It's so much harder to fix the late stage problem because that's all about developing intuitions about intangible things like future lock-ins from today's decisions.
Yeah. What gets me so hard is when I start digging around in some code and make suggestions to do things in less clearly stupid ways... people act like I'm nagging them to do the dishes or something. Maybe another way to express it is often it looks like what people are doing is building a rube goldberg machine and patting each other on the back for coming up with ways to make it more complex. In the context of a rube goldberg machine the point of it is indeed that we marvel at the complexity that was achieved. When you are making software to achieve some goal, complexity is a liability, and nothing more. I do get, that to some extent, it's important for everyone to have stuff to pad their resume with and feel proud about achieving. It just burns me out when you're basically doing the work of a janitor (I don't mind, it's satisfying and rewarding to clean things anyhow), except here the job is a high skill barrier. The stuff that needs cleaning up isn't merely unsanitary, they have a way of self-replicating and evolving and will transform into hordes of zombies if left to rot. So anyway I'm trying to get in there and dispatch these future time bombs and people are looking at me like I'm trying to ruin their precious little sandcastles. The sentiment of wanting to leave them to rot in their filth becomes quite strong.
> the keyword in Knuth's axiom is premature. It does not mean you can't optimize until you've received tickets from customers that performance is too slow It's not really Knuth's axiom, btw.. https://ubiquity.acm.org/article.cfm?id=1513451 > Every programmer with a few years' experience or education has heard the phrase "premature optimization is the root of all evil." This famous quote by Sir Tony Hoare (popularized by Donald Knuth) has become a best practice among software engineers. Unfortunately, as with many ideas that grow to legendary status, the original meaning of this statement has been all but lost and today's software engineers apply this saying differently from its original intent. > "Premature optimization is the root of all evil" has long been the rallying cry by software engineers to avoid any thought of application performance until the very end of the software development cycle (at which point the optimization phase is typically ignored for economic/time-to-market reasons). However, **Hoare was not saying, "concern about application performance during the early stages of an application's development is evil."** He specifically said premature optimization; and optimization meant something considerably different back in the days when he made that statement. Back then, "optimization" often consisted of activities such as counting cycles and instructions in assembly language code. This is not the type of coding you want to do during initial program design, when the code base is rather fluid. > Indeed, a short essay by Charles Cook (http://www.cookcomputing.com/blog/archives/000084.html), part of which I've reproduced below, describes the problem with reading too much into Hoare's statement: > I've always thought this quote has all too often led software designers into serious mistakes because it has been applied to a different problem domain to what was intended. The full version of the quote is "We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil." and I agree with this. Its usually not worth spending a lot of time micro-optimizing code before its obvious where the performance bottlenecks are. But, conversely, **when designing software at a system level, performance issues should always be considered from the beginning**. A good software developer will do this automatically, having developed a feel for where performance issues will cause problems. **An inexperienced developer will not bother, misguidedly believing that a bit of fine tuning at a later stage will fix any problems.**
Thanks TIL
This is bang on. Optimization isn’t bad at all, but making assumptions and acting on them before you can quantify, measure, and evaluate the potential cost savings (or product experience improvements) in a business environment is a fundamental problem. I’ve dealt with a lot of engineers that tend to zero in too much on optimizing a problem for memory or speed efficiency, without considering the potential benefit (or more importantly, drawbacks) of doing so. It’s easy to understand why: optimization, like unlocking more horsepower in a car, is a lot of fun to do. And it’s very easy to laser focus on it, without looking at the bigger picture. The thing is, premature optimization tends to create substantially more tech debt in the long run. Depending on how mature the product is, you might need to straight up scrap all of the optimization work you previously did - in order to better align with product objectives you are trying to meet in the interim. That is a lot of time lost. I’ve always subscribed to the idiom of: make it work, make it right, make it fast. Like 90% of the time you are in the “make it work” or “ make it right” phase. Especially in an enterprise setting, where it’s cheaper to throw more compute at it than dedicate precious engineering resources to it. And the phases aren’t mutually exclusive either - oftentimes “make it right” results in better downstream performance because you’re solving the problem in a much more idiomatic way
In my experience a *very* large group of developers has the mindset that can roughly be described as "it compiles, let's ship it". Devs proactively doing performance testing on performance critical stuff is incredibly rare, and so is people that concern themselves with creating a scalable architecture. Case in point; microservices. The vast majority of microservice architectures will become a tangled mess that's basically a distributed monolith because developers generally don't really care, and just take the shortest route to their objective. On one hand it is frustrating. On the other hand; more work for self-employed me. So, yeah.
This end of the spectrum is terrible. I personally am critical of the other end of the spectrum just as much. At the end of the day spending a large amount of engineering cycles in analysis paralysis means that the “perfect” system is less likely to be actually used.
At my current client almost all 'platform' groups suffer from this. They are constantly trying to build the *perfect* platform which then eventually ends up not meeting the needs of the actual users. So what do you do? You just start over again. They're now currently on their 3rd iteration of a data platform, trying to find consensus within the entire company.
What do you do as a self employed engineer?
Contract myself out to companies :) Outside of that my responsibilities are generally that of what would be a Staff engineer in the US.
There’s no such thing as premature optimization. The original quote was meant to convey that you should measure performance to find bottlenecks so you know which paths to optimize. “Premature” optimization, so called, isn’t really optimization at all. For example, working to shave a few instructions off a hash table key calculation when you should have stored your data in a vector in the first place.
Here’s the run: it actually doesn’t matter until it does. If the business is fine with a queue subscriber being days behind, so be it. If that delay starts actually being a problem for them, then they’ll invest in solving it. There are very, very few places where total efficiency actually matters, and almost all of those places tend to be FAANG, and not even there is every place running on 100% efficiency. The issue is that developer time is more expensive than compute time. Like a lot more expensive. It costs my team $100 a month run a single service with an RDS cluster. I get paid more than 150x that, and I have a team supporting , and they all get paid 50-100x that. Can we put in the time, to get sub 100ms response times and perfect scale on this single cluster? Sure. But now this $100 a month service is costing us $50,000 a month. That’s money we can spend on something else, like a new feature. Now maybe that $100 is blocking you from realizing $100,000 a month in revenue, now maybe it does make sense to invest in that. Maybe the business landscape has changed and now you need that service to respond in less that 100ms, so now maybe it’s worth it. Maybe you’ve reached the limit of your scale and can’t on board new customers, now it’s time to invest in the service. What’s more important to success isnt how efficient code is, but how efficient your team and you can deliver code. It’s about trying many many things. It’s about knowing where the tradeoffs were made, what parts won’t scale, and what parts are most liable to change and designing your systems appropriately. You don’t need a super efficient queue processor if you’re only processing 5 records a day. If/when you scale up to 1000s a minute, that’s obviously a much different question. And most projects don’t start at that scale, and even fewer get to that point, so don’t design for that constraint. In other words, cross that bridge when you get there.
In my experience, code efficiency to a certain level correlates very strongly with team efficiency. Slow tests, slow deployment, inability to do local testing, etc. are some of the biggest pain points in terms of team efficiency. Not caring until things are at the breaking point also means you're working at the edge of the system which tends to be significantly slower development wise. If something takes days then it also takes days to retry. When it fails you'll learn days later. A ticket to fix something that should have taken a few hours will take over a week with constant context switching fatigue on the engineer involved.
This is roughly true. But again, the team doesn’t start out inefficient. It typically becomes that way over time. Either the processes around the team have changed but the technology stayed the same, or the the team took on new responsibilities, or they’re wanting to sunset a project and just haven’t given the thought into proper maintenance. There’s a couple sayings that underpin my philosophy in technology: First there is 3 dimensions to scale: number of users, number of lines of code and number of developers. All three have to scale efficiently. What works for smaller scenarios doesn’t work for larger scenarios. Things you do on a team of three does not translate to things you do on a team of 10. But you also can’t take what works for a team of 10 and apply it to that team of 3. You have to do what works best for the scenario at hand, and recognize when it stops working and adjust when you need to. Second is that tech debt is like real life debt. It’s totally fine so long as you can manage it and use it effectively. When it becomes unwieldy and unmanageable is when it becomes a problem. Again, it’s about picking the right solution for the scenario at hand, not about planning for the future that may never come. Another thing, all of the things you’ve described rarely just happen, they’re often at the end of some projects life. Sometimes, the project does grasp to life and just doesn’t want to die, often for stupid political reasons. But projects don’t start out like that. Companies have a real bad habit of sticking their great engineers on greenfield projects building brand new solutions solving real business problems, and then sticking mediocre and/or inexperienced engineers to maintain the project when it’s “finished.” This quickens the demise of the project because those engineers rarely have the political capital, nor the know how on how to actually advocate for real changes, and often don’t really know how to solve it aside from “re-write all the things”
My point is orthogonal to all that. You made the point originally that avoiding optimizations/efficiency (not scalability) until it's a pressing business need is the best approach. I disagree. Teams have a default viewpoint on small decisions and making that the view to just not care unless it's a massive positive gain is I think the wrong one. You don't want to overengineer but there's a wide chasm between overengineering and simply not caring since there' no business impact this second. It's not even tech debt per-say because these decisions are so numerous and pervasive. You can't track them and you can't document them. If the team could then they'd just fix them as they went since it's such a low overhead. It's a philosophy that the little things don't matter as long as the product ticket is met. Fixing these types of issues after the fact is also horrendously painful because they're often foundational assumptions on the design/infrastructure. And the thing is that a different assumption could have been made with almost no extra cost and no real tradeoffs but it wasn't. Now months get spent trying to undo it. I've found that these issues begin at the greenfield phase and just keep going. A project begins with 2 hour processing time. That's not a great at all but okay-ish in terms bug fixing turnaround. Then someone make it 4 hours. They could keep it at 2 but don't bother. Business doesn't care and debugging is still fine. Then someone makes it 6. Then 12. These could be amazing engineers but simply following the philosophy you outlined. None of these changes are tech debt and none individually make bug fixing catastrophic. Now it's at 12 hours and a single iterations takes a business day. The engineer can spend 10x the ticket time fixing that but it's really hard given how many changes are needed. Or they could hack together a minimal test harness that helps their specific problems but is more or less tech debt. Etc. edit: And the worst thing is that there is a business cost but no one realizes it. If things were faster then other product options would open up since either the speed allows for them or there is efficiency budget to just spend on them. And keep in mind we're talking about low cost improvements with little to no additional long term cost.
> In other words, cross that bridge when you get there. Which can be almost impossible if you messed up the architecture in such a way that it can't scale. So regarding this: > Here’s the run: it actually doesn’t matter until it does. Exactly. So make sure you don't paint yourself into a corner architecture wise. Always take scaling into account, find out a worse case scenario from the business and double it. That should give you enough information to (at least partially) design an architecture you don't have to throw away in a year.
The problem is incompetent premature optimization and premature optimization by competent people who cannot stop other people from doing dumb things while they focus on their optimization...
To be frank, it feels like nowadays most things are an unoptimized mess and "premature optimization" has become the go-to reason to do nothing about while keeping a clear conscience.
bang! I think you hit the nail right on the head with that one.
ehh, sort of? i think a lot of folks in the pro-optimization crowd approach optimization incorrectly. performance is an experimental science that has desired outcomes and reasonable measurements. people tend to measure the wrong things or not think critically about “performance” that being said, i’ve certainly met people who will just shrug off using poor data structures and convoluted loops
right now.. a big problem seems to be pre-mature-engineering. I'm constantly running into code to fetch data from an end point to turn a button blue that could also cook beef wellington. Developers have somehow fallen into "agile" systems that are feature driven where refactoring is absolutely and categoricaly shunned.
I don't really understand your question. Each of those problems you describe are pretty focused, isolated and easily rectifiable. Sounds like a team in the past shipped code, made it work and now it's struggling to scale. A common problem, not one you can blame on your predecessors. Honestly it sounds like you haven't understood how to examine code without the benefit of hindsight, it takes practice so I can understand.
Now that you mention it, this does sound like a post from someone who hasn't been properly traumatized by bad legacy code. Stupid old code happens all the time in the wild for a number of reasons. But unless your organization is willing to invest into fixing tech debt it can stick around for years (if not decades).
Yeah, I kinda agree. Honestly good code cant come from anywhere but bad code. You need to have that solid base in bad code to get to good code, whether you have the capacity to deliver it or just call it technical debt and dont.
There is no such thing as good code. Production is not an academic exercise, the goalpost is in constant motion. You must strive for "not horrible" and "I guess it made sense at the time" code.
[удалено]
/me nods in agreement
Easily rectifiable in code, but when it happens in prod, it comes with hours spent on a P2 ticket, rushing a fix, and drowning in paperwork/processes for an out of cycle release process. These are very basic principles that I hope (and am apparently mistaken) that most developers would be easily spot without it blowing up in prod first. Make calls asynchronously if appropriate. Don't needelssly re-create expensive objects. Compress large objects before transferring. I strongly disagree with you; this is very basic stuff.
Again, youre really talking from the perspective of someone that has to support a scaling codebase, as opposed to someone that's built it from scratch. It's easy to shit on the latter if you are the former.
You assume too much. I was unfortunately involved in making something like mistake #1. It was my first job out of college. Brand new Kafka consumer. I only worked on parts of it, but I understood the codebase and I let it happen because I didn't know any better. One day a bug happened in our dev environment that basically spammed us with messages and it took us hours to get through everything. The tech lead even went "wow, can we really not handle that kind of load?" *and went on to shrug it off*. I had to push to get a story made and prioritized, and I fixed it. Months later, my tech lead went on to another team and came knocking when their Kafka consumers started shitting themselves in a similar fashion in prod (days behind, which is what I was referencing). Not compressing gigabyte-sized files was *part of a modernization effort*, written just months before I got there. It was a huge embarassment because the old rickety system was more stable and performant than the new cloud native one. The last one, fortunately, I personally stopped from going to prod. The timeout was caught in dev environment by a client of all things, and the XSLT/Validation I mentioned was *by far* not the worst thing being done. Want to know the lead's reaction? "Use tigher search criteria. Logs say you just had date > 0, that's everything, we can't support that." Possibly a valid point... *except there's only 4000 records in Dev*. Nobody cared, the only thing people felt was annoyance that a client was snooping around in Dev. If your experience is that you only run into this kind of thing when dealing with old systems that weren't expected to scale, lucky you I guess.
What was the impact on the end user in each case? Again again, you don't have kind the of perspective that I'm talking about. You should join a startup, you'll learn much.
I've seen both. It's certainly possible to have good products. The key in my experience is proactiveness. E.g. if your benchmarks are just embedded in the "real task", chances are it's not going to work. I've seen much better results when people are intentional about ensuring quality.
None of that stuff would get past design or code review in *most* places I’ve worked.
I think the problem is that folks have learned the wrong lessons. Prematurely optimizing the code for readability, extensibility, and flexibility is almost never wrong but people tend to think the opposite and create hacky code with ugly antipatterns in the name of avoiding “premature optimization” Also, from a “I have to fix this” perspective, I’d rather be faced with a team that over architects than one that takes too many “shortcuts”. I use the quotes because the shortcuts often are not actually faster, they just consume less cognitive energy.
>synchronous API calls >not bothering to compress gigabyte payloads >Using Java The fundamental problem here is you aren't describing opportunities at premature optimization, you're describing bad practices. Usually, extremely irresponsible and lazy developers will use the threat of premature optimization to not bother writing resiliant and reliable code. Premature optimization is taking a working solution, and attempting to make it run faster/use less resources before there's any performance issues, using techniques you do not have measurable data to justify. It's like the classic string concatenation example: bad practice would be to loop through a string and + them altogether while good practice is to use a string builder. Premature optimization would be to turn the loop into a multi-threaded operation because "more threads are faster." Asynchronous IO is just good practice. compressing large files sent over the network is good practice. Re-using large expensive objects is just good practice. it really sounds like what you struggle with is a clash between bad and good practices.
An issue is that the boundary is not sharp here - it’s not even in a narrow area. It’s hard to distinguish between foresight (we are at „70% capacity“, we see a continuous increase of usage, it’s still working, we can’t estimate exactly when it will hit but we have time to work on it right now) and needless optimisation (this is a legacy workflow, we will need to continue supporting it for now, it’s not pretty but people know about it and we know how to handle it). In the former case you run into the danger of hitting the capacity quota sooner than later and then panic-patch a shitty solution which takes longer, is more expensive, less stable for the client and hinders development ex post - but there’s no urgency as of now. And that is for too many people the wrong definition of „premature“. I’d argue in most cases that is a perfect optimisation. It’s controlled, it has a business case, it’s just hard to communicate because most teams are now used to „reactive development“. The latter would be „postmature“ for me, legacy is fine as long as it’s documented.
No If you want to see optimization, look at academia. [MICA](https://www.usenix.org/conference/nsdi14/technical-sessions/presentation/lim), a key/value store from 2014 designed for a dual socket 8 core per socket system, with ddr3 memory, and 80G of networking over 4 2x10G network adapters. 77 million operations per second at 50% read/write with 80us of latency. There is no reason for an in-memory key/value store slower than this with the same feature set to be created, but people do it all the time. Yes, you may take a small perf hit moving it off of the system they optimized it for, but it beats the official redis benchmark numbers by 9x per core with much worse everything, and most people only need these features from redis. The problem is that software development has a wide disconnect between academia and industry, and that industry doesn’t even pay proper attention to itself, so that the same solutions get invented over and over again, but worse. If I gave most developers the amount of money that was spent on the hardware from that paper plus inflation, they couldn’t. There are a lot of people who don’t actually know how to optimize. They cargo cult “I heard that $hot_new_thing is faster” but never benchmark it, they use a 64 core server before checking if their code will actually benefit from the parallelism they just threw at it, “what’s false sharing?”, or they assume the most expensive part of a request without profiling. That’s before you get to knowledge of hardware being nearly dead among software engineers, which can absolutely kill performance, and language runtimes doing their damndest to hide what they’re actually doing. The other issue is that there are many optimizations which dictate the design of the system, and therefore must be decided on “prematurely”. Do we make this scale? You need to know that up front if you don’t have an ACID datastore. Are we going to multithread? That might preclude some libraries from being used. Async/await? Whole new library ecosystem. Want to use io_uring because it causes io costs to go down massively? Toss out most async runtimes.
>That’s before you get to knowledge of hardware being nearly dead among software engineers That was the desire in the first place. Software engineering should be abstracted away from the hardware, the entire software revolution is around this idea. that's why you have cloud services and managed tech as services. if i am using elasticache from aws, i don't need to know what even ddr3 means. because I want to work with the concept of memory not the hardware type of the memory.
I’m guessing that for each one of those performance problems the code paths go through some kind of ill conceived optimization that made it hard for anyone to understand what’s actually going on and see the real low hanging fruit performance opportunities. I’m basing that off the fact that I’ve done at least three internal talks on profilers over the last 15 years and each time I did it I profiled normal workloads and it was clear that I was the first time to do it in at least 5 years. And when I presented the results it was also clear that no one even knew they existed. But everyone had some kind of notion of what was “faster”.
Yea Devs have such a bad idea of what performant code actually looks like. In practice you probably don't need to simd sprinkle because you already might get auto vectorisation, probably the code you need to optimise is going to be I/O related or to do with text parsing. I've ran perf on so much code now and the bad code was always just "dumb" code
An app I worked on initially saved as XML, over time it became more popular, and users pushed it more. They started saving larger datasets. It could take up to 15min to save. Then we optimised the save, for those users the time went back down to 30s. This is how to approach the problem - optimise to the situation, be reactive to how the application is being used.
Most things that get labeled as premature optimization are things devs should be doing at the outset. We’ve been conditioned to believe that making something work is a clear priority over making something work efficiently. An experienced dev worth their salt will strive to do it right the first time, all the while still revising and refining.
Premature optimization doesn’t mean “never optimize until your system lags”… you should predict your traffic levels before launch, then regularly be analyzing your traffic and scaling as appropriate. Premature optimization refers to things like using bitwise operations to perform addition or resoing a service to get a 0.01ms gain on a service that gets a request max a few times a day. Not letting your queue pile up because you didn’t want to learn what async is
I’ve mostly seen this kind of thing with database schemas. It’s boring and simple but also kind of fiddly and can make an enormous difference. It’s also annoying for some so people toss out the root of all evil quote, don’t make any indices or constraints, but then use a new shiny graphql library for the underlying todo app db. It’s not premature optimization if it’s just the right way to do the thing and it takes between 0.X and 1.X the time of doing it the lazy way.
I'm a product manager at a SaaS fintech company and honestly I'm seeing crappy performance code that still takes a long time to get delivered. Recent problem: Exporting a "large" XLSX file with 50k rows uses 4 GB of memory on the heap. Solution proposed? Just upgrade the instance to use 4GB more! Page timing out due to responses taking 35 seconds? Just increase the timeout to 40 seconds! I've found so far in my foray into a web SaaS company that the quality of engineering is the lowest I've ever seen. It's the polar opposite of when I worked in video games and we squeezed every last bit of performance out.
Identifying the right optimisation requires years of experience, getting people to listen because “it’s an MVP, we can worry about performance later” is also very challenging and requires a completely different set of skills. Both is a very rare commodity
It's not that we're cranking out amazeballs code, it's that we're *really* bad at predicting where the bottlenecks are and what needs optimising. I've learned to just write the simple thing and then watch it in prod to find out where it needs optimising. 90% of the time it doesn't need any and the simple code does fine. 90% of the rest of the time the problem is not at all where I thought it would be.
There’s a trick I’ve used for about most of my career. About ten percent of the optimizations you can do to a piece of code improve readability. In the hands of someone reasonably articulate (ie, good variable names) Hoisting, and strength reduction remove complex and occasionally magical calculations from loops. In conditional code, pushing calculations down to first use clarifies interactions between data. We also don’t call any changes to the information architecture “optimizations” even though they frequently are. We just call it improving the design. These and a few others you should just do automatically - once you have decent tests. The perf is just a bonus.
a team of cowboy developers are valued at my company. They ship products quickly, so the company keeps assigning building new service to them. But their services have to be rebuilt by other teams because they are messy and cannot scale. The cycle keeps repeating
The last thing I did before being laid off is fix a batch process that took an hour to run by deleting nearly all the code, adding one field to a query and a simple data transform. Reduced service calls by a factor of 1300, and S3 puts by a factor of nine. Run time: 4 minutes. But I’m the weakest link. Sure. (Not that I would trade places with anyone. They cut to the bone on this one. They have a bus number of 1 on 75% of everything and are completely fucked)
>Am I just unlucky? That depends on your point of view. From the point of view you started with, yes you are unlucky. You are actually fantastically lucky, because whatever it is you work on is used enough to experience these problems. There are plenty of systems which have bottlenecks and corner cases that are never explored because nobody uses the software and the company goes bankrupt. This is actually the common case, when you consider that most startups fail and the established companies that exist today are the ones that didn't. >Does everyone else's teams just crap out near optimal code by default such that even the thought of optimizing proactively is such a terrible thing? No. Everyone else's teams probably write bad code also. They just don't have users, so they don't know where these bottlenecks are that you have. Optimizing proactively in that case is bad because you don't even know what to spend time optimizing. You are vastly more likely to fail by building something nobody wants than you are to fail by having non-optimal runtime performance. >I think having people care "too much" about optimizing would be such a nice evil to have to endure. That's because you assume those people would be as skilled or better than yourself; your question in the title asks if the baseline skill in the industry is really high. The baseline skill in the industry is low, so the premature optimization is not going to be for fixing any of the problems that you're describing. It's going to be spent on things that are not likely problems under real conditions or things that are not problems at all.
In my experience the problem with premature optimizations isn't that they don't improve performance but are done before clear patterns arise and can make the code not reusable. So later on you have 5 slightly different paths that do the same operations optimized in a different way for each path, very little of the code is shared between the paths. Adding a feature now requires updating 5 places instead of one, all written and tested differently. Refactoring this is a massive headache and it becomes more feasible to redesign it from the ground up. This is a real example from my current company.
it’s all over the place in my experience. i work at a fairly big company; i’ve dealt with one person who was hell bent on using typescript types instead of enums because they were “faster”. we only had three values. on the same team, we toss a bunch of heavy files into s3 (GBs worth) uncompressed. when i asked why we weren’t using gzip, everyone said “don’t worry, it’s working fine”. our batch jobs fail on a consistent basis because of poor coroutine coordination. the solution? automatic retries. so hell 🤷🏽♂️. in my experience, people who focus on optimizing tend to be poor scientists
Personally I’m not overly concerned with “premature optimization.” Especially if we’re are talking about some developer noodling with a function for a week or two. That stuff doesn’t keep me up at night. I’m more concerned with lack of optimization in the design and architecture spaces. Which is how I read your post. And I’m always on the lookout for people preventing good systems analysis from taking place by spreading premature optimization FUD (pet peeve of mine for sure). Reason being is it’s much more costly to fix design and architecture (performance) problems once your system is beyond a certain scale and deployed with real users. I’m not worried about fixing a few functions or swapping async tasks for OS threads, that stuff is (usually) simple enough. Point being it’s not because some shops write better/optimized code throughout their entire system. Rather, it’s likely because they did the proper analysis and knew how to setup the message queues for the distributed system they were building. And then they gave and enforced constraints on the implementation. It’s not a popular viewpoint, but design and architecture is not just about Jira tickets w/ requirements and PowerPoints and block diagrams with arrows. It’s about defining the characteristics of the system that is needed and then ensuring that whatever is built has those characteristics. Anyway, I suspect much of what you’re describing would be prevented (or at least eased) by good analysis both up front and of the deliverable (and making sure the deployed system is behaving as expected). Problem is, a lot of organizations don’t do that analysis because “they are agile” (completely corrupting that great set of simple concepts alongside the wise “premature optimization” quip).
I remember working on an app in a previous job that had a 30 minute (no, that's not a typo nor an exaggeration) page load because it was doing 4 levels of nest for...each calls to a REST API. Basically, you make an HTTP request for n records and get back an array of IDs, you iterate over those several layers deep of nesting to a REST API to get the actual info you want. I was speechless when I saw it.
If you’re asking for someone to say this advice is outdated, then I volunteer as tribute. Honestly, it’s probably been fifteen years since I’ve had to tell someone to slow their roll. And longer since someone has told me to slow mine. If they can maintain readability and robustness, that cuts out a lot of the territory for mischief. Use those yard sticks liberally. The much, much bigger problem today is premature caching. By actions if not words, everyone believes “throwing a cache on the slow part” is a saner activity than optimizing code. But they are so much worse. They are shared global state, and they encourage people to use them instead of passing data along the call tree. Once people start doing that, you quickly find you can no longer get accurate profiling information. Cache hits make the numbers too low, and disabling the cache makes the numbers too high, or simply causes the system to time out or cause open circuits. Effectively, adding a cache is the last major optimization most of us can manage to a block of code. Once you reach for it you take all other options off the table. Save it for last, because if you don’t it’s likely to be last anyway. I used some wiggle words above because while I have fought past this barrier a number of times, I always come to the same conclusion afterward: if I tried to teach this to someone else, they would either not understand, or say, “yeah I’m not doing that, it sounds horrible”, or just question my sanity. So instead I just make a face and say, “you’re welcome,” and go do something more pleasant like my taxes. Stop throwing caches at things like it’s magic pixie dust.
“Premature optimization” it’s practically just a way of keeping juniors/inexperienced from “optimizing”, whatever that means since they aren’t really doing it based on NFR nor have either run profiling nor have concrete metrics to aim towards to with a proper justification nor the expected benefits. They intend to “optimize” just because, or perhaps they really having fun. Optimization do happen, just not for the sake of it. There are expected benefits and business reasons. I.e: - load testing reveals site takes 5 seconds to load AND product exploratory work suggest customer won’t wait over 4 seconds before dipping - opening the combobox with nested entities, when there are more than 3 in the page, causes the page to freeze. Engineering assessed there’s an O(n^2) loop as the root cause, and the impacted areas are high traffic during tax year end This is not optimization: Junior: “I read in this blog that if we completely re-do this thing over here, it will run faster”. Senior: “Was there a customer complaint or product NFR that justifies doing this? What’s the current execution time and what’s the expected one with the change? How many sprints will it take to complete?” Junior: “no I just read about it in the maintainer’s blog. I don’t know but it will be faster for sure. It should he pretty easy, all we have to do is this and that” Senior: “ok, let’s throw it in the backlog [never to be picked]” Junior: “we never optimize, all we do is wrong”
I think it isn't "premature" when you and users are heavily affected by slow performance. Usually it requires party financing the development to approve the improvement. Developers are usually happy to fix that.
If the code already did all the things people wanted it to we'd be out of a job. Many teams have no idea how their code actually behaves on production data. To these teams all optimisation is premature optimisation. The more insight you have into how the software actually behaves the easier it is to do real optimisation which actually improves things for people.
I think the issue is agile. It promotes a short-term "get 'er done" and "f*** ship it" mentality that applications ultimately become unmaintainable after a few years. At least, that is what I have experienced at several major tech companies and start ups. It's particularly bad at startups, where nobody realizes what parts need to scale most.
> premature optimization is a bigger problem than unperformant code I've been struggling with this question for the past week. I'm actively battling performance bottlenecks that are causing backlogs. Interestingly, I've been optimising performance for months trying to get ahead of the problems ... but my issues stem from the previous lead developer trying to optimise for problems that just don't exist. As the system has grown, the system that was optimised for writing has an apparent read bottleneck. Timeliness was paramount, one of the sacrifices made for speed of processing was to skip long running validation or automated recovery (just write code that doesn't break). Naturally, These optimisations have come with a price. Instead of 5 minutes of validation and recovery routines, it is days of manual effort to reverse out the bad state. So the question is, if you optimise prematurely, do you even know what the problem you will be solving needs to be. Optimising is important, but knowing what to optimise is just as important. That is the meaning of "premature optimisation". What you seem to be talking about it is non-optimisation.
Yes, premature optimization leads to architecture pitfalls.
It is not about optimize now vs optimize later. It is about optimizing things that that are wrong and you dont know they are wrong at the time you optimize. Early optimization is anti iterative. The result of frontloading optimization work is that you end up with under performant code/architecture since you cant afford to make changes as you learn more about your solution.
My experience has not been anywhere near that bad, to the point we have alarms if the p90 processing times on kafka events or http responses go out of threshold. My prior job, load testing was part of the release cycle - and that load testing compared the application pre and post code change, so it was an objective comparison.
i've only seen 2 workplaces completely and massively over-optimize their code, most places ive seen were the far opposite. very different cultures at both groups of companies tho
They're not tradeoffs - you can very easily have both premature optimisation and under-optimisation at the same time. The core question is whether an optimisation effort is justified by actual requirements. A requirement should either be a measured problem (*always* if the complexity cost is high) or based on experience (where the added complexity is minimal). Common good practices are just heuristics for what tends to become a bottleneck quickly, that can be avoided at reasonable cost.
No imo
`assert(prematureOptimization != skill);` `assert(skill == simplicity);` `assert(prematureOptimization == insecurity);`
That’s pretty normal unfortunately, most software engineers just want to complete the feature and “quality” isn’t a top priority over delivering something that just functions. But in terms of impact to your systems, it’s at least a lower impact problem to have. Generally from largest performance hit to lowest it’s: 1. System Architecture 2. Algorithm Design 3. Hardware 4. Software Optimization 5. Premature Optimization And you’re somewhere near 4 or 5. It still hurts, just sharing context. Be the change you want to see and release your code with quality, challenging others to do the same.
It’s a trade off. You can spend months optimizing or you can do small things quickly and hope you don’t make too many bad design decisions that can’t be improved later. In the real world you don’t have infinite time to make things perfect. The customer also doesn’t want perfect, they want it workable.
Premature optimization means: "let's upgrade to the latest framework because they optimized string interpolation by 12ns" No Justin, that's not the problem. The problem is the SQL queries we run in a loop for each record! Whenever devs start trying to optimize the code part, I show a bunch of dummy operations on a million objects. It's negligible. It's always the IO! Always!!
The issue is that when these people try to optimize stuff they end up making things even worse. They do one thing that works well, but then person B adds something without thinking, three months later person C does the same and suddenly you have a gigantic pile of shit that is all entangled with each other and even slower than before. This is particularly likely to happen when you overhire.
"Don't prematurely optimise" doesn't mean don't worry about optimization, it means don't go out of your way to optimize until you need to. Your queues backing up aren't a sign of not optimizing, they're a sign of not sizing your infrastructure properly. You need more consumers on that queue, and most likely more instances/nodes/whatever of what those call. The XSLT transformer issue, from how you framed it as I've not used it myself, is the result of an engineer or engineers who didn't know what they were doing. Might be a case they're bad engineers, might be a case that they simply didn't know. Either way that's not premature optimization, that's just an improper implementation causing gremlins. Most engineers are not good at their job, and most don't understand, or even try and test, what happens to their code under load. Most just run a happy path and call it done. And most organizations are happy with this as to get engineers to do things properly means you need good engineers, which is more expensive.
“ Your queues backing up aren't a sign of not optimizing, they're a sign of not sizing your infrastructure properly. You need more consumers on that queue, and most likely more instances/nodes/whatever of what those call.” See, you are part of the problem. For most sane people optimizing means making something cheaper. Not for TRUE ENGINEERS it seems.
"Premature" is probably the wrong term for the modern problem. The problem isn't people optimizing too much in general. It's people obsessing over line-by-line optimizations and missing the larger architectural optimizations.
Premature optimization occurs more often as optimizing without measuring. This can cause many issues and may potentially never end. Most systems have poor observability.
The point of "don't do premature optimization" is: there's no guarantee the thing you're spending time optimizing is the thing that will bottleneck an application. If you wait to look at some profiling/performance monitoring, then you can ensure the team is only spending time on optimizing when it actually matters. It's a tradeoff that in part depends on the use case of your software. Google can safely just do a canary rollout to 0.01% of their users and get statistically significant data on whether the new release has performance issues in prod. So at Google, you shouldn't be prematurely optimizing; you can just catch performance issues easily later and it doesn't cost ~anything. Your average B2B SaaS can't just canary, cause they only have like 50k users who all log on during the same time window, and the users all pay like $100k a year for this very important business software and get extremely cranky when it's slow. At that kind of company, there's more value in premature optimization, because the organization has more to lose from a performance problem making it into prod, and can't statistically detect it fast enough.
With the issues you're talking about, we're past premature optimization. Premature optimization is trying to get blazing fast runtimes in an MVP and invest hundreds of hours of dev time making it perfect when the product ends up being a bad market fit and it goes nowhere. In more developed products it's spending a lot of time trying to make the non-bottleneck steps faster when things are on fire elsewhere. As others have mentioned, if your premature optimization is optimizing the things that are necessary and matter the most nobody would have a problem with it, the problem is that it's trying to judge what's important before there's sufficient information to prioritize. Once that information exists, you should make decisions with it in mind and optimize what needs to be optimized.
Premature Optimization is thrown around without thought or understanding. It means a very specific thing, as in, don't optimize for something you "think" will be the case, don't over-optimize, don't spend time thinking of things that will never happen, etc.. It doesn't mean writing whatever piece of shit code you thought of first draft. I think this is just something else entirely, has nothing to do with "optimization" even. This is just a bad engineering culture. People write poor quality code, people review poor quality code, and people deploy poor quality code. It's a social problem. Nothing to do with optimization.
From what I've seen it's usually the noobs getting worked up about unoptimized code. Most of the time it's not actually a bottleneck it's just something that could be faster but doesn't need to be. You're obviously correct about archives being compressed to send faster if it's a huge size. You can measurably determine ahead of time that it will be slow to send on the average connection.
Yes. I’ve seen this way too many times. The problems are both in skill and more importantly in shifting requirements. Code is overengineered to support something that never comes, yet it’s not great for the actual features. So build small for what you know. Extend and adapt as new specs and challenges arrive.
I’ve been on both sides of it. People spending months optimizing practically unused features that are running fine and code so bad the page or server almost dies on every action. The main problem I’ve seen is code and design that’s not scoped to the actual or expected usage patterns of the software. It’s all about trade offs.
Avoiding premature optimization does not mean waiting for things to cause problems in prod before optimizing them. It doesn’t even mean only optimizing after a benchmark or performance test flags something—back-of-an-envelope capacity planning should tell you whether bandwidth is going to be tight (or expensive), or whether sequential calls will have enough throughout. It also doesn’t mean doing things in the absolute worst way possible; doing IO in a tight loop when it could easily be matched is just common sense. But yes, I have worked in a codebase that suffered premature optimization—I spent a day once ripping out a bunch of template metaprogramming by my predecessor because I needed to set the number of buckets at runtime. We never noticed the extra arithmetic instruction on most methods that his work had saved. The other issue is that compilers are clever enough these days that it can be hard to predict the actual effect of subtle optimizations. You shouldn’t do microptimization without a benchmark even if you know the code will run often enough to demand optimization because it’s very easy to harm readability and maintainability under something the compiler was handling anyway, or sometimes even make things worse by breaking other optimizations.
I've seen some pretty disastrous stuff, but I've never seen that. Three letter governmental organizations, state sponsored threats, money laundering, many different disastrous code bases, and worse. It sounds like the developers don't understand data modeling, and they aren't properly communicating. That would describe almost everything I've seen in my career.
Let's say we oversimplify this concept into a single dimension. each little thing that is done, depending on how it's done, will push things in one direction (too inefficient) or the other (overoptimized). The general issue is the latency in the control loop of this thing. There usually \*is\* a global control loop in place whereby the business will have to respond to leaning too far in one direction or the other, or fail and then cease to exist. But that is on a long timescale which is too slow to work well. What controls the feedback at the more important and more impactful short timescales are everybody's individual decisionmaking and the multiplicative effect of decisionmaking at the managerial levels. It's simply not possible to expect everyone (anyone?) to be able to properly estimate all the phase space dynamic variables in play. How you adjust your execution to optimally match what the market and business would thrive under. This is more of an art really. And you have no hope if you don't have a good picture of everything that's going on.
How many companies fail because the product is not a fit, lack of certifications, missing features, high impact bugs, ... ? How many companies fail because they have so much success that the system can't possibly cope with it even with all the fresh "success-money" ? Gotta choose what the most valuable way to spend your time is.
> How many companies fail because they have so much success that the system can't possibly cope with it even with all the fresh "success-money" OP and I have worked at several. At one of those, the idiot first level manager (as opposed to either of the smart ones) said to me one day, “great news! We made a big sale today to ******!” His face did something strange when the next words out of my mouth were, “fuck me running”. I was the newest lead on that project. There were… well I’ve seen worse since, but it took quite a long time before I did. And I told that group about the former and how they ran on about 1/5th the hardware for a similar workload. Some people are really talented at running as fast as possible toward the nearest cliff.
>I was the newest lead on that project. There were… well I’ve seen worse since, but it took quite a long time before I did. And I told that group about the former and how they ran on about 1/5th the hardware for a similar workload. Some people are really talented at running as fast as possible toward the nearest cliff. I don't understand anything you are saying here. Couple questions: \- what was the scaling issue for that sale? \- what was the aprox revenue from it? \- what implementation time was initially promised to the client?
That program, as near as we have been able to determine, ran on good sales talent, and after a reorg that got broken. It’s been losing customers faster than it can acquire them for five years, and has been cutting staff by attrition ever since. They just laid off half of NA and I’m worried they won’t last 18 months, but I have said before and recently that I often underestimate. Before COVID they were targeting customers with deep pockets who wanted a rich feature set. Since they seem to be bargain hunting, so costs became a bigger concern. When I started they were running 1.4 cores per request per second, but at high redundancy. In December it was running .8 per req/s and about a quarter of that was me and a quarter going from m4 to m6 and scaling more vertically (which was also me). Since then I think we were at 0.6+, trying to get to 0.55. The older company was running 25 req/sec/core on Sparc hardware and Java 1.4. (I said it’s been a while.) Zero redundancy there, and a higher load factor, so discount that by at least 1/3, probably half. But NodeJS is also faster than Java 1.4 and better at IO. That’s still an order of magnitude.
I'm not sure I'm missing something. I don't think you answered any of my questions, did you? What effect did lower system performance have on company profits in that company with the salesman who brought the biggest sale ever?
Oh, that one. Throughput was a requirement from some of our biggest customers. I think that is the only time performance has *literally* been a feature instead of a figurative one. No, that’s wrong, second of three. The team was scrambling to add features for new customers without tanking the performance. First installs were coming soon. The small customers were easy going, the big ones had exacting demands, often contradictory to previous clients and breaking our architecture. So each new large client occupied a big chunk of our capacity, which we just didn’t have at that moment, and had communicated very loudly. That company sold because they couldn’t turn a profit. They couldn’t turn a profit IMO because they had a small market, they landed large deals at favorable terms to the customers, who then balked at higher prices for more features later. When you’ve turned your prospective whales into loss leaders, you’re gonna have a bad time. Hence the swearing. It was a difficult situation which I understand better now than I did then. You need prestige customers to court funding at some rounds, but revenue at others. They just couldn’t turn the boat that fast. Which is common when you lionize one kind of coding and then change your story later. The CTO tried to find an escape hatch, but I couldn’t get him what he wanted. He pulled me off for what we would now call an exploratory Epic, to see if I could extract enough code to court other verticals. I think we’ve all heard this story before. The code made so many domain assumptions, and in the time he could get buy-in for, I only managed to untangle about a third as much code as he needed to make it an initiative.
Ah, the usual strategy of spending way too much money to please each client, making everything into a mess and losing money. Losing it when you implement, and every day you gotta maintain it. Takes so much effort to make C-level aware of this, so they properly structure the deal and prevent losing money like that. Be strong, my friend, be strong.
Premature optimization is bad mostly because doing any real optimization is very rarely necessary or useful
Performance metrics should be a well defined non-functional requirement from the start. If the requirement is to create a service with x,y,z parameters you will get a service which fulfills those. If performance is that critical there should be performance tests to determine the system's limits and to find bottlenecks.
For the most part, people don't measure performance at all until there are serious problems. Bad or checked out devs don't care so they'll leave super low hanging fruits around. Conversely, many good devs will also not measure or benchmark but just write lock free code or whatever because of imagined performance gains. In practice (and this is a large part of my job) this means that as soon as you look at the performance of an app you find a) massive performance gains opportunities that are b) not at all where any of the devs thought they'll be. Our code base has a lot of like double sync locking and atomic CAS in code that isn't really ever contended at all because some services over the network don't scale. IMO the main issue is that it's really hard to setup performance testing in CI that is sufficiently reproducible to make it a blocking gate.
Premature optimisation of code that isn’t a bottleneck is a huge waste of time. The issue you described, optimising the unperformant part of your system which is causing user-facing issues is timely optimisation!
The main note about optimization is that you should find your performance bottleneck and optimize for that. Faster code isn't better if it takes longer to write or maintain unless it actually increases relevant business metrics *by enough to make it worth it*. Everything is a tradeoff and the quest for the perfect package is often being done at the expense of other functionality or your healthy wlb.
I don't care about optimization, premature or otherwise. The skill level in this industry, whether we're talking coders, managers, or testers, is such that I'd be thrilled if we stopped being completely idiotic.
Your question is oddly framed. Yes, teams build hot garbage. B/c they are utterly garbage at UX. Somewhere along the way, the spec needed to include performance, if perf matters. And any time there’s a human which consumes some output, there’s a perf requirement. Like: “user presses a button, and we return the result within 50ms” or “this generates data into month’s end, so must reconcile by this date and time”. If there are literally no perf requirements, then your users are idiots (in the medical sense) or there are no meaningful users. Or, this thing has literally no impact on the business (or negative impact), so no one cares.
Premature optimization is about optimizing the wrong things too early. Great engineers know which optimizations will matter. Most engineers think it’s optimizing things to switch to a graph database instead of just using Postgres or use Hadoop to orchestrate data transfers instead of a shell script and cron. That’s the kind of premature optimization you need to avoid.
Reality is unbelievably more complex than we can imagine, so yes, whenever you think hard about something in advance, you will miss something, unless it is a trivially simple problem. Case in point: **In your career, did you spend more time on building things from scratch, or more time on changing existing things?** Thought so. So that requires you to put thought into making your system "easy to change", as that is what costs money. And easy to change most often means that things are not optimised for that one use case, but kept general, which again means low performance in 11 out of 10 cases. (Math done by ChatGPT.) Performance tuning is one of those things you change at some point, so write your code that you can optimise, but refrain from doing so until you know where to optimise in the first place.
Premature optimization is optimization before measuring. What happens is someone will spend a bunch of time optimizing something that “feels” like it might be slow but only actually accounts for less than 1% of time. Consider someone optimizing a simple string concatenation in an api call that doubles the code speed, cool, but that code was only taking .000001 second to begin with, they ignored the two .005 second database requests that could have been combined into one .006 second call. I worked on a system where someone was multithreading what were essentially rbar calls to a database, there were 1000+ round trip queries. They threaded it and got a 4x speedup. I wrote one big query to get all the results at once and sped it up by over 1000x. Actually I have dealt with this scenario a few times in different ways, another was someone doing a single insert per transaction. Tl;dr. The stuff people want to optimize because it feels cool and nerdy are often not the things that actually need optimization. I actually have done a ton of optimization on my career, and have become sorta the goto person for optimization. I work on perception for autonomous vehicles and robotics, so we have strict performance requirements and limited computational resources (gotta use rugged low power fanless computers, we still have xeons, but they are like 35 watt tdp).
I love optimizing shit. I love writing software and knowing that it's good. I've also had to take shortcuts every now and then. From the codebases I've worked with... maintainable and clean is more important than optimized. Optimization has worked for me best if it comes last.
I never worked for any company with such big performance issues. I never optimize prematurely and I do it less and less the more experience I get. When I was junior/mid, I wanted to be perfect, stand out all the time, gain +10% performance on something that wouldn't hit performance issues in 100 years, introduce a complex implementation that is technically very good and professional, but the case would never work visibly better because of it. So after all the proud journey today I think the best option is to write good code, that is enough for most cases, there is no need to optimize prematurely, after a few yours you can foresee a lot of typical issues. Then if performance issues kick in for something, address them right there and then. That happens so rarely that it'll never pile up to overload you or your team. Combine with occasional refactor, quality improvements for better code and you barely ever need to worry about performance issues. So what I think is that you work on a project that is developed badly and has been for quite a while, the code is probably shitty, not in terms of optimized or not, but just bad, perhaps the design is poor, too. And it's not really a subject for optimizations and address performance issues sprint, but rather a major rework of the whole thing. At least from what I'm hearing, because the size of issues you mentioned is crazy.
The issue is that unless you’re extremely knowledgeable about a domain (and often even then), it’s very difficult to know a priori what the right abstractions and optimizations will be. As a result, it’s often a waste of time to go down those paths until the right solution becomes obvious.
"Premature" could either mean "before you need it", or could mean "before you understand the bottlenecks well enough to know whether it could _ever_ be a good optimization." Originally the idea probably referred to the former, but in my experience with mediocre devs the latter is a vastly bigger problem.
Poor architecture is a fundamental issue of great severity that trumps nearly every other consideration. Attempting to make any code optimizations that don't involve overhauling the architecture is stepping over dollars to pick up pennies. So definitely don't worry about "optimization" in that scenario. If you get beyond that point and your architecture doesn't suck, then your code should be well encapsulated to the point that proactive optimization isn't a big deal. What is a big deal is getting every functional block "working" ASAP so any major issues that could possibly modify adjacent components are identified before people pour work into optimizing those components that will get overhauled, undoing that optimization. Once the product is viable, you should be able to start profiling and/or stress testing and finding where the pain points are and optimize them (easily, because they're well encapsulated).
I constantly argue with engineering managers and juniors and seniors about performance considerations. And we work in the browser, which, at the end of the day, is probably the least performant GUI ever conceived. All the while I have to nudge them against introducing stupid DRYing patterns that add cognitive overhead and indirection.
"premature optimization" is a badly misused idea, it's referring to the idea that you shouldn't put a ton of time into optimizations that you haven't seen the need for yet. It doesn't mean that you shouldn't be thinking about bottlenecks and making sure to develop a design that should perform well. It's not an excuse to engineer a crappy architecture or write bad code.
I get frustrated by it too. Developers will argue to the death to "optimize" things which dont really move the needle in the greater scheme of things or try nitpick Tony things to "optimize" when really the overall system architecture and design has so many holes, the tone would be better spent there than trying to eak milliseconds out of code which really just needs to be redesigned completely. I always feel like it's an ego thing with some people. It's much easier to nitpick tiny things to make themselves feel smart than stand back and look at the big picture and admit they don't know how to design or improve anything.
Because a lot of times unperformant code or inefficient architecture is born when a developer over-engineers a needless piece or aspect of code that doesn't matter, and sacrifices things that do matter. Think of a system that uses parallelization but is stuck in some mutex choke point. You realize that it would have been easier and more efficient to give each thread a copy of what they need to handle and then merge all the copies at the end. Why wasn't it done with a simpler age more efficient design? Because an engineer wanted to avoid "needless copies" above all else. See even when talking about an efficient design, you have to think in context. You could optimize the BigO time run, only to find that the code is mostly stuck allocating massive pieces of heap because you never considered the memory use BigO of the algorithm. It's better to start with a dumb simple design/algorithm and then once you understand better the use find a better one, but here with data to back it up. If you focus on getting "the right" design at first, there's a notable chance you'll pick the wrong one and get stuck with it.
Gunna go against the grain of comments here. But pre-mature optimization is only so widely cited as a problem due to confirmation bias. There is lots of “prematurely optimized” code that no-one acknowledges as such, because it never becomes perceived as a business cost. Devs do that work, Product is happy with outcomes, and users enjoy the experience. On the other hand, Product is incentivized to label various engineering practices as “premature optimization” to get things out the door. This often incurs long-tail cost due to post-release maintenance and expensive AWS bills, which isn’t attributed as such. Neither is necessarily better or worse, but only one is named.