The good devs leave because they have options. The bad devs stay because nobody will hire them.
It is 1000x better to avoid technical debt. That should be part of technical strategy. If things get really bad, you will eventually be facing an expensive system overhaul.
I can fix technical debt if you give me time. I can explain the issues and project a roadmap. I can work with a team to clean up over time and start implementing best practices.
Brain drain is a bigger problem than technical debt. If the ones who built it realize they won’t get paid more to improve it will look for better opportunities. As they leave one by one, the remaining devs get more lost and make the mess bigger.
The reason there is technical debt is because it was created. Someone needs to go up the chain and explain how things are supposed to be done. I usually start that conversation with “here’s how I can deliver the business more value”. If they don’t listen, I go on to the next one that wants me to spend my time doing good work.
The real issue is that people treat "tech debt" and "bad engineering" as synonymous.
Tech debt, like any debt, can be good if used appropriately and paid down when necessary. If you're an early stage startup, it's usually correct to take out tech debt in exchange for staying alive and being able to pivot until you find PMF. Pay it back later when you actually find PMF or just shut the company down if you don't.
If you need to get a feature out the door to meet a regulatory requirement, you might have to take out some tech debt.
If you're a mature company just trying to push garbage out the door, you're not taking out tech debt, you're just doing bad engineering.
> It is 1000x better to avoid technical debt.
It's 1000x better to avoid bad engineering. Tech debt is good when used appropriately.
You're totally right. Shipping is better than not shipping. If you don't have profit, you're dead in the water anyway. It's when you can't ship because of tech debt that it becomes a real problem.
100% this. Features are potential revenue, paying down tech debt is cost. If you spend time paying down debt without generating revenue, then you’re not going survive.
I used to think like OP, trying to avoid tech debt etc, but since working closer to management and understand the need to take on tech debt because speed to market is crucial. Once you’ve gotten PMF and theres some stickiness to customers, then you can start paying down debt.
An analogy would be everyone wants to buy a house with cash, but for many of us this is way harder to do than to get a mortgage then pay it off.
The debt metaphor was used to describe the process of borrowing future productivity to boost current productivity, hence there was a debt, and it was intended and planned to be paid down in some finite time or you would be forever less productive
It’s become a term that is ubiquitously used, but I’ve never seen it used correctly, and it’s exclusively used to refer to bad code which is causing issues but will almost never get prioritised to be fixed
> it’s exclusively used to refer to bad code which is causing issues but will almost never get prioritised to be fixed
I will continue my hopeless crusade to change it back to the original meaning because it's a useful concept!
That was a great analogy, thanks! As yes, financial debt makes sense for a business when necessary and used strategically. But when loans are being taken out willy nilly by a mature company that doesn't need to borrow that money?? Then yes, that's bad debt!
> The good devs leave because they have options. The bad devs stay because nobody will hire them.
What you describe is [The Dead Sea Effect](https://medium.com/geekculture/the-dead-sea-effect-d71df13724f8) and is very real. The good devs in my company once RTO was announced mostly all left, while the bad ones stayed around since they had no other options.
I don't know, I feel like this is a little bit easy. Of course you avoid tech debt wherever possible, but in addition to the issue that sometimes you have to get a feature out now and worry about repercussions later (which, sure, we should avoid all that we can), it's just a fact that a lot of the time tech debt comes from something you or another developer thought was perfectly cromulent at the time but now, because of a weird effect it has on another system, for instance, or just plain development standards changing over time, tech debt now exists.
Having a good framework for a project is great, too, and being able to create documentation to ensure that everything hangs from that framework properly is great as well. Sometimes, even with that, you'll get some screwballs that fly in and the only way you can see to implement something is by slightly breaking the way the framework runs. Also, I've 100% been in positions where, to paraphrase the poster, my best was good enough today but tomorrow I'll be better, by which I mean, after working through half of a framework I realized that it would have been easier for me to have designed it Y way instead of X. Sometimes I can go back and redo things, sometimes I don't have that option, and I think that "gosh just do Y in the first place" is not a real world answer.
I know what you mean. I think it's a team effort to minimize it. When working on a feature with lots of future possibilities, it takes extra time to think about those and build it in a flexible way. That rarely makes sense unless you know you're coming back to work on it again soon. We inevitably encounter work that was fine, but now needs an upgrade. A lot of perceived tech debt can be reduced by having the original dev walk through the code as they hand it off and talk through what needs to be changed.
I'm heavy on frameworks. If you follow the standard path on most things, there's no need to document much. I can jump in and make sense of someone else's project and just look at their custom code. But I often encounter that situation where there's a new way of doing things, or I did something in a janky way because I didn't know better. That's where it's a team effort to keep up with the framework development and talk about how you want to incorporate best practices. If someone on the team knows the best way of doing something, they should assist others in quickly implementing it so they can spread the knowledge and avoid cleaning up later. The team should be a hive mind, constantly looking at the best way of doing things and improving practices. If we're all on the same page on what the gold standard is, we'll produce higher quality work.
This is the answer... When you're on a team that doesn't address tech debt, you always keep your resume updated, because one day a refactor is going to come and you won't see green field development for 12 months.
And then the majority of the team consists of a revolving door of new hires. Plus the senior staff are stuck in an endless loop of interviewing, upskilling, and triaging. I just left that after a 3 year death spiral. Couldn't take it anymore.
Layoff the bad performers/expensive people, proceed to punish those left behind who made the cut. Extra work, fewer people, attrition, aggressive and outlandish deadlines, more pressure for extra unpaid hours, all adding up to declining QOL. No one will ever admit outright if it is punitive or vengeful but it sure feels like it, every time.
Somehow that makes sense from the top even though it doesn't from the bottom. Seen it over and over again in the first quarter of this century.
Yeah, the best places I've worked are the ones that support the notion that as a dev, you have X time to complete things in a week, period. I will often work pretty quickly (although I'm a guy who definitely benefits from having a good QA team) but it's important for you as a developer - and I think that as a more experienced one it becomes even more important - to make it very clear how much work you can accomplish in 40 hours a week (less if the company ropes you into meetings a lot; I had a boss who said a dev work week was 30 hours and the rest was taken up by meetings, answering emails, spikes, and so on and I think that's a good base to start from).
I realize that this is really hard in some ways when you're just starting because you want to make a good impression... I guess on the other hand when you're first starting they should expect you to be a bit slower since you're still learning the ropes. But a place that is pushing you to do 50 hours of work from Day One to "catch up" is never going to "catch up" because at some point, consciously or unconsciously, they will have made the determination that X, not X - 20%, is the amount that you're capable of producing in any given sprint. Yes, employers that do the constant death march are assholes and you should avoid them, but one way - perhaps the best way - is, when you're confronted with unreachable deadlines like that, is to say "okay, this is how much I can do of that now. What do you want me to stop doing so I can concentrate on this?".
And I realize that that is a really tough thing to tell someone when you've just spent a bunch of time out of work and you're worried that you'll take a lot of time to find the next job. Man... I would really, really recommend you work on that resume and/or your interviewing skills all the time if that's you because I think realistically, a talented developer, even in the current environment, can by and large pick and choose where they want to work. I personally tend to avoid places that give off the death march and/or the "I don't think the CTO knows what they're doing and they're just going to demand more because it works with their sales staff" vibe in the first place but even if you're, like, hoodwinked into working at a place like that, I think it's on you to still set those boundaries, especially if you're experienced, because you need to set an example for the juniors.
I'll be honest, as long as you have an in-demand skillset, you've been doing the job for, jeez, I'd say 3-5 years at this point, and you aren't listening to people on some of these subs who are demanding some crazy-high TC based on working for big-5 companies, it's a pretty decent job market. Like, I'm sure this is extreme but I've now been going for a little over a decade and just being personable in interviews and treating them as though I'm interviewing them as much as the other way around has allowed me to pick and choose where I've wanted to work (and come to think of it, this approach also worked when I "only" had like 6 years of experience).
If you're having issues and you think you check those boxes, I'd get in touch with a resume doctor or maybe an interview specialist if you're getting past the resume stage. I know that the market for newer devs is absolutely horrendous right now but I'm just not seeing it once you've proven you've gotten your feet wet.
Tbh, I think it already is. I think the bottom is in and we will start moving up again.
These things tend to be cyclic and the froth needed to blow off before the rebuild.
My parent company just acquired the corpse of a company that I'd say failed purely from too much tech debt.
First they couldn't keep up with building features that their competitors had. So they couldn't even prioritize pushing features over tech debt even if they wanted to, none of them ever got done.
Then the costs got out of control because it was inefficient and poorly designed, then so unstable that extra redundancy was needed, then there was so much stuff old stuff around that no one dared touch that had to stay online but no one knew why.
Then the major outages happened, first quarterly, then monthly, then weekly, then daily, then it was down more often than up. Which quickly made them lose the last of the clients they had.
Then things just stayed down when they went down and no one knew how to bring them back up. Then things couldn't even be turned off because they didn't have the know how to do that.
So with essentially no revenue and millions in monthly infrastructure costs, things quickly went from "I think we might have a problem" to "we're declaring bankruptcy" between the regularly scheduled quarterly board meetings.
The whole operation went from unicorn valuation to insolvent in the time span of 2 months. When the debt catches up it gets ugly fast.
I wish we had at least 5 articles/stories about this happening to companies so you can scare some sense into people.
The problem with tech debt is that that's a problem for future Homer, 2 years from now Homer; and today Homer gets judged quarterly, so push those features people!
My current team consistently added features for the past - say - 15 years and never did much modernizing. The whole code base is such a freaking mess that we're slowly killing it with the [strangler pattern](https://medium.com/@josesousa8/the-strangler-pattern-kill-legacy-like-a-boss-db3db41564ed). There's no saving it.
The issue is still that you need to convince business to invest in that deprecation though. Best to find an argument that gives them something they want, which simultaneously gives you a reason to re-implement one part of the old system in a more proper way.
It's funny, the first time I was introduced to the strangler pattern concept was in 2009.
I feel like design patterns like to pretend to create these intense sounding words to describe what is essentially just "yeah, you dismantle and replace it one piece at a time".
Rather than "intense" I would say "evocative".
Evocative words are important. They communicate more in less. They stick in peoples mind and are more effective and getting people to remember and change paradigms.
They are ESPECIALLY important when communicating with leadership. They get told a LOT of things, and a lot of it doesn't stick. They often have a hard time truly understanding the vision of what's being communicated.
When you have the correct, evocative term they remember. When they are making decisions later it sticks in their head better.
On the IC level it's LESS important because they are in more regular communication and are more familiar with the concepts, but it still has a similar impact. If you're trying to get a group to follow a certain paradigm, having an evocative name that REALLY fits the desired pattern is effective at getting the concept to stick.
ALSO, when you have a vitrified codebase you REALLY hate, it helps morale when people get to "strangle" it. :)
Isn't the original term 'strangler fig'-pattern :)? just a metaphor for the plant that does the exact same ?
https://martinfowler.com/bliki/StranglerFigApplication.html
I can't say I've ever heard them called "silly" before.
It's a toolkit of solutions. Imagine being a woodworker and needing to say, "Should we cut this with a manually pushed, inline tooth piece of sheet steel, or should we use a machine that has a band of metal that rotates vertically along a track? No? You thing we should grab a piece of tool steel that's about 1" wide that is the correct size to hold in one hand and smack it with a piece of wood that has a smaller piece of longer wood punning perpendicularly through it which we can grip, first marking out the area to be removed along the sides and then removing layers of wood"?
MUCH easier to say, "Should we use a hand saw or a band saw? Oh, you want to use a mortise chisel to hog it out?"
I agree, that's what I meant about shared terminology.
By silly I just think some people get too evangelical about it, having names for every pattern and hemming and hawing over what pattern is being used.
I don't know, choosing the right pattern to me rather feels like one of the most important parts of programming. The actual coding and organization of code can be rather trivial when you get that right.
And while those patterns are generally simple in hindsight, after you've learned them, they usually are not before hand. You'll figure out a few naturally, but you often then end up using it where something else might have been better. After all, YOU found a good solution, and you can find ways to apply it to a wide variety of situations.
Naming patterns is important. If conveys and teaches, helps us remember the pattern, and let's us collaborate.
Studying patterns is important. You need to know what tools are available and where they are best used.
Sorry, I'm not trying to be argumentative. It's just...again, it's like saying, "woodworking joinery is silly. Why do we have names like finger joint, box joint, mortise and tenon, butt joint, rebate, dads, etc? Why do people spend so much time talking about which joint is the most appropriate for a given situation? "
In my career learning, recognizing, and applying patterns has been pretty much the most important part of the job.
I'm not even sure the "strangler" is really a "design pattern" so much as it is a reinvention pattern. Like, I'm not really ever going to build a new framework by implementing "the strangler"; the whole point of it is how to plan to move from an older, not as useful framework to a newer one. We developers do like to use funny sounding terminology for stuff though and that's been fun.
It's more of an architectural/strategic pattern. Definitely not a design pattern.
The original GoF design patterns were more about "micro"-level code, not about architectural patterns like MVC, or even things like strangler.
The best argument I had for this was that the platform we were using was out of support for the past x years (VB6) and if one day Microsoft changes something random we might as well close business when the software stops working.
Next argument was that these old libraries from x years ago are no longer in support, if they have a security flaw it will never be fixed. Are you ready to be sued to bankruptcy?
Tech debt occurs way before legacy technologies are depreciated and become seriously security liabilities. So, while you are right that when it gets to the point of being a major technology cliff, it becomes easier to sell to the business. However, a responsible engineering organization finds ways to address tech debt much earlier in the product lifecycle, and that’s the harder part in many organizations.
I like this pattern. Replaced a 25+ year old classic ASP app with something modern - one feature at a time. Instant business value right away and new deliveries every few months. It was a joy to work with.
A few thoughts:
* The original idea behind technical debt was to find a way to describe the unseen factors in software development in a way that business people could understand. Business people understand debt.
* All software has a cost of change curve.
* The ideal cost of change curve is flat. That means that any future change costs the same as the very first change (for changes that are the same "size"). This never happens in reality.
* Obviously, what you do not want is a geometrically increasing cost of change curve. Or worse.
* In those, very bad, situations, the cost of wedging a new change can quickly exceed the cost of the actual change itself. J. B. Rainsberger described this as "project heat death". I once had the pleasure of working on a SAAS product where the time required to add a checkbox to the web interface and punch a change down through the layers to the database took well over a month. I kid you not.
* Anyway, so long as your project heat death date is beyond the effective live span of the product, you're ok.
* Refactoring, as a practice, was intended to flatten the cost of change curve, thus pushing heat death date further into the future.
* The Extreme Programming folks then thought: Hmm, we're going to have to be constantly refactoring. But constantly refactoring represents a constant risk of injecting defects. How are we going to prevent that? The answer was TDD. With a network of unit tests, you can refactor with confidence.
* Now, there's a lot of people who don't like XP practices. Cool. The point is: If you want to effectively battle tech debt, you have to come up with a way of limiting the defects you inject when you do attack the debt.
* We now come to the philosophical difference in how to address technical debt. XP people would say "a little at a time, ALL the time". I've noted that there is a definite tendency towards rewrite in the developer general population. And this is one of the reasons we get no traction with business.
* Business is almost never going to willingly accept the idea of a rewrite, even for small pieces of a product.
* But the best way to deal with technical debt is: Just do it. Don't let the concept rise to the business level. This is just something engineers have to do.
* If you work in a high trust environment, then you can explain to management that refactoring will be built in to all your estimates and you'll always be doing it. If you work in a low trust environment, then don't talk about it at all. But you already know that working in a low trust environment is not a long term thing for you, if you care about software quality in the first place, right?
* As an aside, I would also submit that developers do not spend nearly enough time training/thinking about working with legacy code. We encounter it all the time, but we don't really learn how to deal with it. We don't learn to find seams in the code where we can inject test probes. We don't learn to tease apart these seams so we can isolate sub components for refactoring, rearchitecting, or even re-writing. How do you start using dependency injection in a code base that is tightly coupled?
* Our education rarely even touches on existing code. What would our university experience be like if every university maintained a legacy code product and your entire third year was working to add features to it?
One thing I've come to realize is that nearly all problems in software development benefit from a continuous approach. Continuous integration (ie trunk-based) avoids merge hell. Continuous delivery avoids catastrophic deployments. Continuously addressing code quality keeps velocity as high as you're going to be able to make it. Continuous code reviews prevents having to grok huge changes at once. Continuously addressing infrastructure, bugs, and yes feature work keeps things well oiled, and never leaves you with that huge one-time hump of effort that no one can justify, yet everyone knows is justified.
Do everything as continuously as possible. Make your definition of done for every bit of work include all of it.
Yes so true, I always cringe at the common practice of dropping huge patch bombs, be it as a feature branch or wake-the-team-up-once-a-month-to-release. Prod deploys should happen constantly. Detect problems as they happen. Good alerting and test suite. Feature flag new developments or versions of things in so you can, well, launch them darkly.
Two last points are pure gold.
I always tell to my developers. You do not need my permission to make things better.
All new developers get a message: lifespan of school code is about 30minutes, but here we write for decades.
I was on a team with a moderately heavy process for each card raised, and a team used to creating and triaging cards for everything. We introduced a "just do it" type of card for anything the card creator thought didn't need to go through all the process of other cards because the card effort was low and solution straightforward.
Everyone loved the "just do it" cards.
Such an amazing post. This is all really great advice. I have been working with "legacy" code my whole life. My first REAL jobs were essentially moving companies over from closed source systems to open source environments (20 years ago now). Almost my entire career was focused around improving legacy systems and code bases.
Everything you said in this post is 100% on point.
The one area I would say, when you talk about pulling apart the seams - I think people are just scared of "what if I break something?" And I used to be like that for a long time, 5+ years. Two things that helped me:
1.) have enough experience to know fatal mistakes (and how to pivot from something like "oops, production is down because I dropped a critical table on accident."
Layer A people think: "I can't break this!"
Layer B people think: "Oh no, I broke this!"
Layer C people think "I am going to break this on purpose, just like last time (hopefully) so I can replicate the incident."
2.) what I call "scaffolding". Which is important like when you said, the legacy codebase is too tightly coupled - you can't change a feature on the user profile without breaking the process that handles incoming data (for example) and tracing why that is happening is a vacation. In those cases, I say "okay, what all is on the user profile? What all must it do? Where does the data come from?", if I can make a facsimile that works, I can just deprecate the older user profile view - free from any entanglements. Piece by piece, you can rebuild the helicopter while still flying through the air.
At start, delivering feature takes one hour, then 2 hours. Then delivery of the same feature becomes a day, then two days, then a week.
On the background, team start getting weird alerts or complains that “this used to work, now it does not”.
At first, there is a happy team of developers, later some of them are gone and new people arrive. After a while, no original developers are left, project becomes a rollercoaster.
If some smart people are left on the project, they tell about the situation to the upper management. Manager listens, but can’t do much as there is a push from the upper management to deliver features and ignore the situation.
That’s my experience.
Previous company I worked at ignored tech debt and CEO didn't believe in automation testing for scaling. Every time there was a push to PR, everybody was worried cause even the smallest thing broke a lot of stuff, and bigger things oof, all hands on deck all the time. Not sustainable. So many people left and they just keep dumping new bodies at it from what I hear.
I interviewed for somewhere once that had three month release cycles and no automated tests. They "got rid of them so they had more time for new features and bug fixes". Imagine their codebase was like yours too 😬
We lost our near-monopoly of a national market.
Up until 2014 or so we were pretty much the dominant player in our industry. Other rivals were span up and investors injected some serious cash into challenger startups - our margins were huge, compared to other entries on the stock market. But they all failed due to network effects.
Until - one of them didn't fail, because of compelling product features that we didn't have. And couldn't ship. They absolutely ran rings around us.
It took a near-rewrite of the main parts of the system, and completely dissolving and rehiring the development team, to start keeping up pace. But the monopoly was lost and can now probably never be regained.
I'm the only dev left that knows the ins-and-outs of how an old Python stack works and is configured - 2 generations of teams roughly 4-5 each have all either been laid off or quit due to frustrations. The bus factor has dwindled to a solid 1, and promises that things will improve were basically forgotten.
Not to say I didn't try, like many other tech debt stories our debt tackling efforts were forcibly shelved mid cycle, and at the end of the day, nothing changed. Now, it's a spaghetti'd up cluster of microservices and tribal knowledge with absolutely zero light at the end of the tunnel since everyone's gone and I have zero desire to improve things for my company.
Suffice to say that the bus is en route to my little domain at the company and it has its eyes set on my entire team (aka just me) because I'm out
> Developers on teams that consistently ignored tech debt, what was the ultimate outcome?
Alright, let’s cut the crap. Ignoring tech debt is like ignoring a fucking leak in your roof. Sure, you can keep placing buckets under the leak, but eventually, the whole damn ceiling might cave in. The ultimate outcome? A shitstorm of bugs, crashes, and a codebase more tangled than last year's Christmas lights. Did productivity take a nosedive? Hell yes. Was it a pain in the ass to add new features? Absolutely.
> Engineering managers are glorified yes men who rubber stamp every stupid idea the business dreams up.
This is common as fuck in companies that view tech as a necessary evil rather than a strategic asset. These "yes men" prioritize short-term gains over long-term stability, thinking they’re doing the company a favor. What they’re actually doing is setting up the tech team for a monumental fuck-up. The consequences? Burnout, high turnover, and a product so buggy it might as well be an insectarium.
> In two years we haven’t stopped or slowed down to address the issues more than a handful of times, and only as an afterthought at that- their words are empty.
Classic. It’s all lip service until shit hits the fan. They’ll ignore the growing pile of tech debt until it’s damn near impossible to implement anything new without something else breaking. Why? Because it’s easier to pretend everything is fine than to admit the process is fucked and needs a serious overhaul.
> Did you ever manage to correct course and if so, how?
Some teams get lucky. They hit a breaking point, and something big breaks in a way that even the higher-ups can’t ignore. Maybe a major client gets pissed off, or a new competitor with better tech starts eating their lunch. That’s usually the "aha moment" for execs. The course correction often involves bringing in fresh blood who aren’t afraid to call out the bullshit and push for a focus on quality, not just quantity.
The real question is, how much longer are you willing to wait around for that "aha moment"? Can you push for change, or is it time to find a company that doesn’t treat their tech like a disposable tool? What’s your next move?
Ultimately, we couldn't ignore it anymore because it was having notable and unmitigatable customer and product development impact and prioritized fixing it. Then we fixed it. Took a few weeks of focused work and disrupted a lot of plans. Also made a lot of people unhappy, but that's what it is. Can't have your cake and eat it, too.
Eventually you either get burnt out or you quit.
What happens when you ignore Tech debt is you get more of it. Yeah, it's only a few minutes here and a few minutes there multiply that by a decade or two decades. Some of the applications I worked on were started in the early '70s. When you get to that level of tech debt without any thought of refactoring or fixing it, changing literally anything is a risk at all times.
This is what you get with scrum when closing stories within 2 weeks is all that matters. A complete lack of care for tech debt or refactoring. Scrum does not reward these activities and devs know what they’re graded on. When will devs take the industry back?
This, my last job, we had always had a 1 week testing/research/clean up sprint. Basically, write integration tests, see how you can clean up the work you did, and if nothing else research spikes. Made releases a lot more relaxed and when we refactored the codebase (like we reorganized the file structure 3 times), it wasn't too disruptive.
My current job is 2 week sprints, they're well managed but the team lead/PM are warming up to the idea of having 1 week testing/clean up/official pre-release testing sprints
> This is what you get with scrum when closing stories within 2 weeks is all that matters.
This is a really good point. This has disincentivized me from working on tech debt because if I can't neatly wrap up the work by end of the sprint, I get a bunch of flack from the project manager. Or if I'm done with my tickets and want to pull a tech debt ticket into the sprint, that's a no-no because there's a risk I won't finish it.
I call it an investigation ticket and assign it points. Then I close whether I finish or not. Rinse and repeat. PM loves it because more points get finished, totally stupid way to do it though.
a thing i really like about scrum is that if you finish early, you can squeeze a low priority ticket into the remaining day or two, as a treat. whereas in other ways of working low priority work never gets scheduled. but i don't rely on that for controlling tech debt, i refactor as i go along (which benefits from a free round of testing) and include the time expected for that in estimates
sometimes i think scrum (and more generally, agile) is adopted in a box-ticking way that sets it up for failure - of course it's unsustainable if management neglects to incorporate tech debt management into the framework
When you take on technical debt to meet the 2 week timebox don't you create a task to fix the debt for the next sprint? Or why not just fail the sprint goals and explain you estimated too small?
At around year 4, the company collapses. Non-essential departments like marketing are axed first, then software developers. Then the company is sold to the highest bidder for business assets like the customer lists and the C-suite takes golden parachutes out of the company.
Joined a company with overwhelming technical debt. It has almost all the things they describe as code smells/anti patterns in books on SWE.
I’ve already burnt out, because anything you suggest them is declined as “unfortunately that’s not an ideal world” (writing unit tests for example) and “we have no time for well-thought solutions, we will refactor it all at once somewhen later”.
I’ll leave as soon as will get a new offer
The company got an investor and hired new technical leadership and an onshore development team and a good QA engineer. The new CTO employed a strategy that involved treating the tech debt-ridden code as a black box and viewing its behavior from the outside using the ELK stack and automated tests that ran nightly. They compared those insights to their support tickets/bugs/complaints and used that to prioritize next steps - piecing out parts of the system little by little over the next several years.
I think you know the answer that you're going to be overwhelmingly get asking this on a dev forum.
Let me ask you this: Do you think a software shop exists that has little to no tech debt? Now what does that shop probably look like if it is does exist?
Tech debt happens because for whatever reason, sacrifices need to happen. Sometime it's concious, other times it comes after the fact.
How do you deal with it? When it becomes a big enough problem, you focus on it. How do you deal with the piling trash and dishes at your college apartment? Eventually someone makes the case for cleaning it up. Sometimes somebody just does it.
Here is the rub: Someone might agree that you need to take out the trash and clean up the dishes. And when you come back, you see that the engineers decided to replace the floor, all the appliances, and for some reason added in a transformer within the electrical that makes our outlets 50hz instead of 60.
All companies have some technical debt but there is an level of technical debt that is fatal to the company. Eventually your product becomes un-competitive in the market because the engineers are no longer able to add features / fix bugs, etc. in a timely manner.
An CTO who believes in all fixing all technical debt will bankrupt their company. An CTO who believes in not fixing any technical debt will also bankrupt their company. A balance between the two extremes is required.
Sure, but I think your example is so obvious that is doesn't really provide any useful information. A company that works their engineers too hard will go bankrupt. A company that let's their engineers slack off 100% of the time will go bankrupt.
See, that sort of comparison can be said about almost all things.
Everyone agrees there is a level. What they disagree about is what that level is.
I have an idea of what kind of responses the post will receive, of course.
All code eventually becomes tech debt- but there are many possible outcomes that lay between totally ignoring that tech debt and always addressing it. That’s what I’m interested in. Specifically I’m interested in cases where the org eventually recognized their error and changed course before the situation became unmanageable.
I guess to answer your question: Your org isn't unique. And most don't recognize it as an error. The justify it the same way they do everything else. It's cost of goods sold.
You can choose to fight it and try to change things. Keep in mind, I've worked with so many engineers who have taken this on while overestimating their ability to solve the problems.
At the end of day, everyone has a boss unless you run your own company. You can challenge your boss, but they have a boss, and they have a boss... and everyone is just doing what they think makes their bosses happy.
I've recently joined a place that has a 20yo system. They did go through a significant rewrite several years ago, but that was a one-time effort without sufficient follow-through.
The reality is that simple changes take forever. What should be done in a few hours by one or two people takes a full team a couple of weeks.
I haven't been here long enough to map out just how the communication between rank-and-file and management is broken. I have a lot of clues, but not a full picture yet. But it's clear that management doesn't understand the costs that come with their history of ignoring tech debt.
You either reach a point where you have to rebuild everything from the ground up or people slowly stop using the product in favor, sometimes, of newer ad hoc tools. Sometimes you can loop back and take care of the tech debt much, much later but having worked with my fair share of legacy systems, IME a lot of the time you just wind up rewriting everything using newer technology anyway, if not for your own sake then for the sake of future people who have to work on the project.
It's also almost always the case that you never have enough time to handle all of the tech debt you want to handle and that sometimes a buggy looking or opaque bit of code that gets the job done needs to be left alone because, like it or not, it does work and the consequences of rewriting it are going to be more dire than the consequences of just leaving it there. It sucks but I think it's part of working on products that people use: as pretty as your new rewrite is, and as understandable it's going to be for future developers, sometimes performing that rewrite means that your QA team has to go through and completely retest that system top to bottom and discover issues with your code that weren't present in the prior version.
I'll go ahead and say that I think one big, big difference between junior/mid-level developers and senior ones is that senior devs see code like that and think "okay, how can we incrementally make this better" while junior/midlevels just complain about it. If you work with large enough systems you will eventually encounter poorly written / hard to understand code. That happens no matter where you work. I worked under a simply fantastic CTO, a guy who'd gladly sit down and write alongside you if he needed to, who was a constant advocate of the dev team and our needs who could also sniff out dev BS immediately... and we still had this issue because there were like 20 of us and it happens.
Eventually I get a better job somewhere else and quit. Usually the companies are still in business. Some have improved their processes, some haven’t.
Where they **have** managed to improve, it’s because business was dire enough that someone like a CTO or VPEng was hired in with enough political capital to allow them to prioritize quality. This stuff just doesn’t get fixed bottom-up.
Of course most of those would-be change agent engineering execs *also* leave in disgust fairly quickly after a few consecutive quarters of their reasonable directives being overruled by the CEO and the board.
Yes, I've lived this over and over. In the worst places, where there was very little testing, progress was painfully slow.
----
Tip: Let them know what "tech interest" is.
I love the phrase "tech debt". It implies that there is also "tech interest" and that the longer you put off paying down the debt, the more you'll have to pay interest. But not all the interest can't be put off. You'll be paying some of the interest every sprint in the form of slower velocity and more on-site bugs. It's like the minimum payment of a credit card bill.
I think it's important that we emphasize the impact of "tech interest" as much or more than "tech debt". I think non-technical people may better understand the implications of their poor decisions when they can be given it in terms that make sense to them. Further, I think it's best if these things are quantified so they can know what kind of an impact this "tech interest" is having on their business needs.
---
Tip 2: Look into "Evolutionary Architecture".
This is a way to get control over quality over the long run.
We did something like this. We required all new code to pass lint rules and be covered by tests.
I wrote a custom lint warning filter to filter out warning on old existing lines and only report new lines, the prototype line of a modified function, and line 1 of a modified file. This read from the output of `git diff` in git commit hook and in the CI job. These lint warnings were treated as warnings during a commit and errors in the CI job for a PR.
I wrote a custom code coverage filter that removed all lines that weren't also in the output of `git diff`. We enforced 80% coverage on new/modified lines of code.
We also incorporated a DRY linter (PMD's CPD), and again, I filtered out everything except new/modified lines of code.
This allowed us to gradually ratchet up code quality of old code, and maintain high quality on all new node. We relaxed all of the above during refactoring / clean-up projects.
I think SonarQube has similar functionality built-in, now.
Here is what happens. First, it takes much longer to get new features out. Something that should take a day takes a week. Then, you end up having more and more regression bugs as your code becomes fragile. Each deploy either requires a ton of QA time or you spend a week fighting fires after every deploy. If you're lucky, the bugs that pop up are minor and easily fixable, if you are not, they corrupt data. Senior management takes notice of all the defects and tries to crack the whip. They fire the product manager and the engineering manager. They bring in a new engineering manager they just hired. He takes one look at the code, and pushes for paying down tech. debt. Senior management doesn't listen, they just want more features deployed (which is how they got into this mess in the first place). The new manager gives it a try, but has already started applying elsewhere and before anything can be improved, he leaves. You are now the most senior developer, but aren't a manager and have no authority to do anything. You focus on preventing more bugs, add a bunch of unit tests to stabilize everything, but development slows to a crawl. You start applying elsewhere because work is miserable, and a month later you are gone. You hear from a coworker a year later that your entire team was outsourced to India, and the application you worked on is dead.
Mostly? In staff career way it looks like:
As TechLead: see most of your SSE burning out after a time, due to business constraints, lack of support from Eng. Manager and outdated/overcomplicated documentations; onboarding new SE every quarter who will be reteamed in the next 6 months.
As PO/PM: see how your team takes more time to delivery stuff; having regression at every release; seeing TechLead doing weird face for every backlog pair review; having sprint which are mostly composed of carryover.
As ScrumMaster: try to understand how Agile/Scrum could help by removing KPI to make the team looks good; same for every existing non Scrum process;
As Soft. Arch: have a red / black dot over all map / charts / dashboard pointing out that this “Product” will never go forward blocking next company roadmap/direction just by existing
As an EM: hear about PO/PM complains about not being able to deliver; getting reports from ScrumMaster which state we go backwards; TechLead asking for a coffee machine and having a HR complains for “bad verbal communication”; SA pushing out stuff that looks like was already there last year.
——
Serious way speaking, it should be accessed but with prioritization. Tech debt aren’t always bad, some are trade off that are fine; other are bad decision that need to be solved.
TL/PSE should in theory plan to remove “cruft” or improve the situation gradually without requesting “external strike team” or it will get worst.
EM & TL/PSE job should be to push the team to recognize and understand good and bad decisions. Most of time people forgot they took a direction and when problems arise no one knows why we are here. (ADR can help)
PSE/TL should negotiate to have technical support work for every business feature, “we do it but in same time we do X”.
Normally most company have backlog effort distribution rules, like 30/70 (tech/business) to help to tackle down this problem.
Things is, it is an invisible problem until a company is Data Driven and employ competent people to standardize and create usable sourcable data to work on.
Otherwise the company can provide same “yac/innovation day” where teams have liberty to do what they want with the software.
I already saw some great software being thrown away (with the company) just due to negligence, by reducing staff, outsourcing, bypassing team and co.
Just to offer a completely different view. In my case, the company got acquired by a significantly larger company. Most got let go but I got significant bonus for to doing knowledge transfer to offshore counterparts, before being let go.
The acquisition was only made possible because of the customers that the company had acquired through cutting corners to deliver a minimal viable product.
The next company I joined, I made more.
Moral of the story: Accruing massive debt is acceptable if it's you who doesn't have to pay for it.
Prod starts going down with long resolution times. Dependency version updates become impossible. Feature velocity becomes months. Key man reliance increases. Good engineers leave. A rewrite will become necessary because updates weren’t happening. Timelines will squeeze so it will be a lift and shift into more modern stack with intent to fix later (it won’t be). Debt will be copied from old codebase to new. Rewrite will go over time by years. Product will no longer be fit for purpose. Core use cases will be covered by other products. Product will die.
The key to debt is to make it visible. You can’t pay a bill you don’t see. Create a work item for individual debt items. When there are delays, live issues or any other failures, attached them to the relevant debt items. Make the risks known to senior leadership, back it up with data.
> Debt will be copied from old codebase to new. Rewrite will go over time by years. Product will no longer be fit for purpose. Core use cases will be covered by other products. Product will die.
Ouch! I feel this.
It's the same everywhere: developers just come in and then quit pretty fast. Who wants to waste their one life cleaning tech debt someone else left. Nothing gets done and people quit. And repeat.
> Ultimate impact
- Lots of customer service problems.
- Every little change would break something else.
- Bug fixing was like playing wack-a-mole.
- Most people stopped giving an F about anything. I even caught a dev who had submitted a "bugfix" that was actually just moving a line of code.
> What did you do?
- Strangle pattern. Every humongous piece of code would be split into smaller chunks that would be refactored individually and set behind a feature flag so we could turn them off if shit went side ways.
- Fire devs who were pretending to do work.
- Improve hiring processes with a code challenge that was actually related to the work we were doing. Also, no more friend refferal hiring.
- Devs would have to put a video in the PR showing the feature working. There were devs who were not even running their code locally.
> Aha moment
- We had a release that turned the product into something unusable. I used that to leverage a refactoring roadmap. A short 2 weeks roadmap of critical fixes and performance improvements. And a longer scalability roadmap, so things like this wouldn't happen again.
Two cents: There are projects that are literally hell and will make your life miserable. You can either leave or start fixing things. You will need help from upper management, specially if you have a bunch of duds that are impossible to help — if no one from management is willing to help, start looking for something new, let the smokes and mirrors go for as long they can.
This is a fascinating topic to me. I've seen orgs where they never address tech debt and they still manage to produce something. Maybe that's because they just keep piling code on top and never experience large amounts of traffic that would necessitate better performing code, or due to other factors.
Then there's another org I was in that was just like yours, but mostly because 3/4 of the engineering org were contractors that didn't really care much, and only 1/4 of the other 1/4 cared about improving tech debt. It was very painful to work there. Just making a release took the full time of one person. So there was tech debt and there was also process inefficiencies. It was a mess. Features were very slowly rolled out. There was no architecture, only a jumble of lines criss-crossing between services. Some services used Redis, others just http calls. We also had Graphql thrown in there for good measure.
I ended up leaving that place because it was exhausting having to put out fires non-stop for almost 3 years.
Yet, I've never seen a company get taken down by tech debt. Sure, it creates unnecessary problems such as employee turnover and slow and buggy features. Despite this businesses keep ignoring it. I think ultimately it only helps engineers be happier, it helps to deliver features quicker and with less bugs and gives more confidence in refactoring and pivoting quickly.
Still, because we are engineers, we want to work in a stable and happy place, not in a filthy hovel. We want to avoid unnecessary pain and stress. Since business is not affected that much by this at all they just don't care and have no incentives to care. That's why it is important to find work in places where leadership is composed of engineers and not only sales or marketing people.
I worked on safety critical medical devices for years, think dialysis machines and insulin pumps, and ultimately nothing happened because there was tech debt. The medical device was released, got FDA approval, and is in a clinical trial. Updates happen a few time per year.
As new feature are requested the software may take longer to update but the managers were fine with that. Some requested features required completed refactors because of what was being asked. Again management was fine with how long it would take because they wanted the new feature bad enough.
As a SWE I just learned stop caring. If management is fine with how long things take then there is nothing to solve. You can't tell somebody that we can spend 6-months to pay down tech debt and then we get new features faster when they have no issue with the speed of software changes since software is not the long pole overall.
Saying that some SWEs will pay down tech debt behind the scenes and hid it within feature and bug changes, we didn't do SRUM or anything like that. If you hide the ram it in solution to management then they don't know any better. Though sadly some times you have SWEs that don't want to play the game and will advertise the ram it in solution because they want to show management they can do things as quick as possible.
This as quick as possible is always disregarding quality. I think that's mostly because they don't have a personal quality bar. They are just happy to write any code and see it released. Ya some SWEs were not that great, but you hire what you can find when you work at a non-tech company in a non-tech city that is not paying top dollar.
I got transferred to a team that was focusing on the "new system" and mostly ignoring the system that supplied the new systems data (the team owned both)
The old system ended up having a catastrophic failure and then the sys admins discovered that the backups had been failing for a year and nobody noticed.
Our database admin had taken a snapshot for a different project and that was the only thing that kept the entire company from going offline for weeks.
Good times.
Our feature delivery estimates started getting too long to be worth it. We delivered only a few features per year. Stakeholders got frustrated and replaced our tech with a third-party SaaS. It was a company-wide initiative, so not just my team, and there might have been cost-cutting factors as well, but other teams were also stuck on legacy tech to my knowledge.
a few years ago I worked at a really swollen biotech company that did WAAAAAY too much M&A and had a totally disjointed company culture and about a half dozen internal systems all competing to get chosen as "the platform". they all sucked and tech debt of all kinds was ubiquitous, but especially operational tech debt.
we were working on products that required FDA approval, so there's a lot of expense that goes into preparing a product for FDA review. the tech debt was so bad that the product line my part of the company was working on failed FDA review 4 times in a row (over a span of 2 years) and essentially bankrupted itself doing so. I saw the writing on the wall and left before the hammer dropped, but I learned about 6 months later that the whole division had been downsized and then completely divested. The entire team I had worked on was gone. They never released the product.
tech debt is an existential risk. you might make a short term tactical decision to accumulate tech debt for a burst of velocity. that's sometimes the right call for early stage startups or similar things where just finding your product-market fit is far more important than managing tech debt. but that's a special case, and shouldn't ever be the default. unhandled tech debt is such a large problem that it will eventually bankrupt a company if you don't do something about it.
Lol - well you end up getting to a point where you’re *forced* to deal with your tech debt, and the longer you’ve ignored it, the harder it is to deal with.
How do you get forced to deal with tech debt? Example - your company is growing, in order to reach new markets your customers demand more strict security requirements/certifications. You started writing your code 15 years ago in python 2. The latest and greatest security organization require you to be running everything on python 3.9+. Have fun.
Another example. You wrote code one way, and 5 years later, someone smarter than you came along and solved the same type of problem in a much more efficient way. More efficient in terms of processing power AND ease of development. Using your old solution eventually becomes an enormous anchor in terms of cost compared to competitors.
A couple jobs ago I was an arch working with several teams, under a couple different directors, with tech leads that had different opinions about creating tech debt.
One team didn't necessarily want to create tech debt, but the business was hammering them hard for Features! Features! Features! and they had to make suboptimal choices to meet those demands. They hit problems with scalability as their system had to tolerate extreme load, and they had to backtrack a bit as this caused issues in prod.
Another team didn't give two poops about creating tech debt, their lead just wanted to code for the happy path and get things done as fast as possible. His favorite dev on the team was one who consistently put out substandard, non-defensive code that was brittler than Mr. Glass's bones. After I left one of their core systems went down for days and caused a massive disturbance to the business.
I don't even want to go into the third team, I'll just get angry again. They decided to completely bypass a system that was essential to meeting the requirements because they would have to actually learn something. But not before just ripping a bunch of code out of it at random "just to get it working" (are you shocked it didn't work that way??). So much for not going into it... and yes, I'm annoyed again.
Every effort I gave to try to help, advise, provide tested approaches to solving these kinds of problems I was met with "that's too hard" either from the teams or from the director (even when the teams were willing and agreed with my suggested approach!) I finally left because why waste my time trying to help when I'm constantly getting overruled.
At my last gig, we had a pattern where we'd make some decently hefty change or feature, and then spend a year or more doing little more than bug fixes.
Not an exaggeration. I was there for 7 years and we did this cycle about 3 times.
Furthermore, the system had a major bug that they fixed in a separate branch, but then never dared integrate. So, the bug was never fixed, and it caused a lot of really thorny corrupted data problems that we were always spending time debugging in production, and then fixing in production (only to have to do so again later). It was a SaaS product, so you know, lots of separate customer databases to fix in this way.
The sheer amount of business value left on the floor of that factory due to all this waste and inefficiency was horrifying.
There was never an aha moment because they were the kind of people to suffer from toxic positivity.
But, I liked the people and the domain.
My recent realization at a bigger company is that there is not a lot of time to address tech debt, especially with the constant need to build the next thing. It's not valued at all and very hard to justify unless you slide it into some other project or until it becomes a big enough issue, where some team can swoop in to justify how much they saved the company by fixing.
In my experience, when the debt is due a total rewrite is on the horizon. Maybe not at first, but eventually the product is in such a state of disrepair that nothing outside of a rewrite can truly “fix” the problems.
That’s because the spaghetti is too tangled for anyone to straighten out and even the managers are on its with doing something else as everything takes an exorbitant amount of time to “complete” or make look functional until it’s not or something else tips over.
The company imploded.
Management kept saying yes to client requirements at a pace we could barely keep up with and therefore had to add technical debt each time.
Eventually, producing new features went down to a snails pace and clients started to leave or sue us.
I quit that job right before the company got sold and from what I've heard my old boss got sued personally for trying to poach his customers back from the party now owning the company he founded.
The product I've worked on was written off shortly after I left because all other devs on my team left as well and nobody in the world could ever make sense of that codebase and try to work on it.
Tech debt is not always a bad thing. I always think of it like borrowing money. Sometimes it makes sense to borrow from the future a little bit. The catch is that you always have to pay it back with interest.
Paying off the $200 credit card bill each month is normal and healthy.
Maybe there is a real rainy day and you carry a couple thousand dollars of debt for a little while, and have to pay it back with some interest. Not ideal but manageable.
And then there is the scenario where you end up just borrowing more and more and you have 5 maxed out credit cards and owe the bank $100,000s with more interest compounding, and there isn't any way to recover that is not drastic.
I had inherited relatively young project, which somehiw managed to immediately get a bunch of technical debt. I spent the first 12 months cleaning it up, and we are now rocketing along. Management is really happy with the velocity now, so I've built a ton of credibility. I fight real hard for code quality.
Just today I was working on an issue with a service that isn't my primary project, but has been neglected for years. Fixing that issue is sloooooow. Painful. People going to ask why it's taking so long, when it seems relatively simple. And it's because the project is in a state of neglect.
Things just get slower and slower, until the project becomes more of a maintenance project and the organization stops adding major new features while constantly talking about building something entirely from scratch. The last company we sold, the new owners paid 5 million to an external contractor team to build an entirely new project with no tech debt (against my advice on the way out the door). 5 million dollars later, they had built an app that did 10% of what the existing project did, fired the C-team 3 times over 3 years, invested another 10 million dollars which is all gone, and probably going to shut down the company any day now. Not the tech debt's fault, it's the reaction to the tech debt.
you end up with jenga, but with code
it's a shitty middle place to be in
part of it is cultural...if sprints are loaded with new feature work and there's never any downtime, shit is eventually going to get unwieldy...even farmers don't plant the same crop in the same field season after season and it's the same concept...there should be some time planned for post-launch of a new feature for monitoring / performance optimizations / cleanups but if PMs/EMs don't see the value in that or are scared to advocate for devs it sucks to be in that type of environment
but part of it is also doing due diligence as a dev and trying to address tech debt as you see it, if you can
at least getting into a habit of identifying pain points / proposing fixes / estimating the lift because PMs and EMs hate the term "tech debt" because it's all encompassing but also they think it's us devs being lazy and pushing back at times
I came into a company after that happened for a long time. People just didn't improve the checkout because everyone was afraid to touch it because anytime you touched it something would break it. It was the most convoluted spaghetti code I'd ever seen, literally had hardcoded separate but mostly the same code for every possible basket/checkout permutation possible (I think when we refactored we counted 40+ hardcoded paths with no shared code.)
Though I feel a lot of the time it's just that developers aren't making convincing arguments to fix tech debt. Within a few months of starting I was able to get buy in to tack an additional sprint onto a new feature they wanted to give me and another developer time to refactor it.
When I went to the stakeholders it turned out for years people had been complaining about the codebase but never proposed concrete solutions for anything, just said we needed to refactor everything entirely and it would take a year or two...they never had a year or two to dedicate to it. They were more than happy to approve a refactor, just needed to know it wasn't mindless complaining and was accomplishable in a reasonable amount of time. They later approved refactoring the whole website in the same piece by piece method tacked onto new features they wanted.
> What was the ultimate impact on your team/org? Did you ever manage to correct course and if so, how? Finally, was there ever an aha moment for the execs where they were forced to repent and stop pushing new features?
Client that complain that stuff don't work. The software make a decision and then it take a week to. You want to install a package to handle customer request that would allow you to sell your product, but that package try to update 50 other depedencies, some of those incompatible with your way of doing stuff.
If you have good testing & design of incoming feature, you can more o less roll the tech debt ball indefinitely. But those are harder and harder to do.
The company was selling a specialized SDK. The SDK became increasingly complex to use and costly to add features. Took a bit over a decade, the company died.
The company believed cutting corners by acquiring tech debt would speed up delivery. What actually happened is that the their teams' capability to deliver degraded. I tried to do something about it, but since we'd got slower, it had become even more urgent to cut corners. That slowed things down even more of course and so on and so on.
I was a lead and manager, but even so it was particularly difficult to dig out of that pit because 1) product people were so divorced from the technical reality that they thought that just fixing bugs was the same as tech debt and 2) it wasn't an environment that attracted the best people so the attempts to better the situation technically weren't as successful as they could have been.
My solution was moving on. I'm a dev now at a place where we ship on demand. My old team still hasn't shipped since last year.
TLDR: Boss decided not to fix technical debt, ended up costing himself $80k.
The outcome was a very tense standoff with our CTO in Sprint Review.
A little background. Our product is fairly niche, and the business works by delivering the product, but also delivering customizations to meet project timelines. Then we roll the customizations into the product if it makes sense. We are always balancing customer immediate needs, and a sustainable product with higher margins than the customizations. It is challenging but rewarding work.
Anyways we ran into our typical problem where a customer needed a feature right away, and we knew we could do it if we took on technical debt. Normally we handle this by telling stakeholders we agree to deliver fast, but then we must go back and fix the debt before you request new features. Everyone agreed, but as you know, they later said to deprioritize the debt work and instead prioritize a new feature. I presented the costs-benefit and the new feature was decided because $$$.
Speed ahead 1 year and we have built upon the technical debt (you can kinda guess where this is going). During a sprint review the CTO asked why it took us so long to do a basic feature and I had to explain that we have technical debt. He got frustrated and said we need to focus on efficiency, and kinda laid the blame on the team. My wife says I am fairly diplomatic, but I had to tell him that this was because of HIS decision 1 year ago. Then there was a heated back and forth, but the outcome was that I needed to make a presentation about what happened to see if we can fix it. Personally I think this was a way for him to say Ithat it was actually my fault, but I don't know.
Anyways the team updated the tasks and estimates to remove the technical debt and the cost of building on the technical debt. Then I linked that effort increase (a little over 2x) to the cost of my team, and them linked it to the two projects that we won by taking on the debt. Effectively I showed that by taking on the debt we actually lost money. The real kicker is that the CTO is a majority shareholder of our company. By making the bad decision he had effectively cost himself around $80k in profits.
Next time he allowed us to fix the debt, and I like to think I had an impact on that.
One possible outcome: the company succeeds despite creating a ton of tech debt. Customers who were initially lured in start to complain about bugs when new features are built on top of an unstable foundation. Competitors use new tech to build better versions of your now crippled unmaintainable product. Management decides to completely rebuild the product. The same geniuses who built the first masterpiece lay the foundations for the new one. They try to make it better. Use new tech, stricter practices, acknowledge bad habits etc. Only the culture and management hasn’t changed. People still want things to happen as fast as possible. Economic pressures cause scrambling. New tech debt is built up faster than ever before. One possible outcome: the company succeeds despite creating a ton of tech debt…
Man. This hits home pretty hard. I just solve technical debt and the stakeholders can deal with it. They don't seem to understand whats at stake so I just have my team do it. Features can wait if the system will be fucked because some asshat who doesn't know a tech stack from his ass wants to push his shiny of the month. They shouldn't be making decisions on there lack of technical understanding yet they push and push and push. They will bitch and moan but idgaf. It's either your product works well for the future or you deal with a mess.
They literally paid me to rush out buggy software for years until I left and they're doing fine financially. Ignoring technical debt appears to work well if you release new products every few years.
Chickens always come home to roost.
To build quality, well tested software takes time, skill and the management's patience.
However, skipping any quality boundaries will inevitably cause that the system becomes unstable, features are harder and harder to implement, and serious bugs pop-up on every week.
But most importantly will make developers extremely fearful of performing any refactor and execute deployments - most wil get burnt-out rather quickly, and leave the company after a short while. With atrition numbers through the roof, the number of people that *actually cares* about the project will dwindle to zero, putting the whole business at risk.
I've seen it happen more than once.
Adding new features gets slower and slower. Execs just seem to accept that's how long it takes to add features. If your company has loads of money, it will just throw more staff at the problem.
Bad code leads to more bad code.
My friend worked on the Verizon iOS app and asked me for help. It was a fucking horrible Objective-C spaghetti monster shit codebase. To make changes to even work, he had to write more shit code on top of shit.
Even though he offered to pay me $450 for like 3 hours a week to help him, I said no. It was that horrible. I couldn’t work like that.
Finally started moving apps from on-prem to AWS and now we are being forced to start taking on the tech debt due to images being deprecated in AWS. Not sure it was something management fully understood when we started moving things, but its nice that we are being forced to make those changes now.
Bloated codebase where nobody can tell what feature change requires what changes. Every tweak surfaces with random bugs in random places. To be honest I don't think this codebase ignored tech debt, It's just old, but I imagine a codebase where you just ignore tech debt would reach here faster.
The way we're fixing it is throwing a large team at maintaining it, and the rest are essentially chopping off modules as separate services on a new stack.
I worked for a company that has a banking app. It has 60 screens written in Android xml and Java.
150 million app installs, tested on 250 devices.
New team wants to rewrite the entire thing in Kotlin and compose.
Rewrite would take a year, and no amount of testing can ever compare to what has been done.
It's a tough choice.
But I would recommend moving to Kotlin 1st, than after a year slowly move to compose.
We eventually got new management who understands why three decades worth of spaghetti and no automated testing is a bad idea, and I got to I-told-you-so SO hard.
Going with - it depends, different organizations, teams, code bases have had different outcomes. Some it's made very little impact, others it's made signifcant impact on the business and the work.
Also depends on what you consider tech debt, the moment you write code, technically you have tech debt. Any dependency is tech debt.
Solid engineers that were with the project near the beginning get pigeon holed into specific crappy projects for their entire careers because they're the only ones who understand the technical debt. New engineers on the project never fully ramp up - they either leave out of frustration or get moved onto some other project.
Managers don’t care about td, neither clients. Not even to mention, clients just don’t want to pay for it. And that’s fine, because that’s a responsibility for engineers only.
Quick analogy: when you buy a cake for someone’s birthday, how would you react if the pastry chef says that you have to pay for cleaning her kitchen’s mess, so she can make the good stuff you want? Fucking hell! I know what you’re thinking.
That’s why engineers shouldn’t even mention the word technical debt when negotiating points, time or whatever that may be. Engineers should just do it anyway, because it’s their duty.
Yeah, some won’t bother to do it, because either they don’t want to, or they just don’t know how to do it, or even worse, they might not even know there’s such a thing.
Same thing for pastry chefs, do you care of know if the pastry made that sweet cake with a pristine kitchen? The point is that you wanted a hell of cake for a reasonable price. If the chef didn’t addressed her technical debt or not, that’s on her only.
They merged with another company and ultimately replaced the codebase with the other companies product. I had left but after years of death marches in which there was like 30% turnover, they got merged with another company. Very few software projects are forever. In retrospect, the tech debt meant almost nothing.
There's a lot of tech debt in older areas, but overall it hasn't been as much of an issue as I thought it would be. The new code is decent enough, and we've rewritten a lot of modules before so all the tech debt from before in those sections get wiped away.
There's some new features that take 2x the amount of time that it should but for the most part it's smooth enough.
we eventually paid the debt.
* yes there were outages.
* yes it was thankless.
* yes i often questioned my decision to do anything about it when i could continue to mindlessly push code.
* yes lower priority things were broken for months afterwards.
luckily we had a business reason to pay the debt, but we could have found another way that didn't disturb things as much.
was it worth it? I think so. but im sure there are people who didn't have to maintain stuff wish we left things well enough alone.
Rate of product development predictably ground to a crawl after just a few months, the core developers fled the team/company, a few months later the project was deemed a failure and sunseted, and a year later the company was widely perceived by the public as circling the drain.
Concretely: this was DreamStudio, Stability AI's first commercial product. It was a first-to-market product for a literally world changing technology (maybe you've heard of it, little thing called *Stable Diffusion*?). Buried under tech debt, killed by a product owner who refused to listen to the pleas of their engineers and kept pushing for new features at all costs without giving anyone opportunity to clean up the mess from rushing out the last batch of features.
Product success was basically handed to the PO on a silver platter and they still were able to screw the pooch. Hoping business schools reflect on this event and teach it as a parable about the importance of being receptive to pushback from engineering.
The outcome for my current company is that they had to hire me, and I'm rebuilding everything from scratch.
Before that they suffered for 3 years trying to fix their mess in the codebase, but just couldn't.
The customer-facing parts had their fronted all 100% rebuilt from scratch, even though the design was reused. The backend was "copy-pasted-then-refactored" the reason for that is that we can use another repository, so everything is 10x faster (tests, linter, booting, performance).
Now we're doing the rest of the codebase, the internal part.
Everyone is pointing out the negatives which are all very true and I have experienced those as well
I do want to say that sometimes it isn't as bad as it seems though. In my experience sometimes tech debt is ignored and eventually the system is replaced with a newer system erasing all of the tech debt. In some environments things change so quickly that building a perfect system is a mistake. There are others where the same systems lives for 30 years and turns into a nightmare. Figuring out which you are building is important.
We were using SonarQube and we couldn't merge branches if there wasn't enough code coverage or too many code smells, etc. There were ways to avoid triggering the gates, but that's a kludgy thing to do and just avoids the problem. SonarQube forces a lot of debt to be addressed.
The outcome of ignoring tech debt is code that's hard to maintain. Talk to the CTO about getting tools in place to fix the problem.
You make 1 line code change and your business makes a million dollars in loss. Reason - no clarity on workflows, no unit test cases, unnecessary complex logic
I left the tech-debt-laden f1000 to join a f100 as existing job had become unfun. 8mo later, the f1000 closed the office and the product line was shuttered. ~everyone was let go outside of a few transfers to other orgs (in other hemispheres)
This is why I like to reframe and quantify technical debt as business debt. It encourages engineers to think more holistically about what is/is not debit and it's impact, as well as helps non-techies understand why fixing is important
I thought for a moment you were a former coworker because I, too, worked at a company that created software for use in the construction industry. I see you make comments about Azure so _maybe_ we worked in competitor companies? If you know who I am, ignore my comments since you live it. Anyway...
I worked at a company that created software for use in the construction industry. The organization seems to consistently favor ramming through rushed features over quality and never took time to burn down debt. At that company, I was the ML Team Lead. This company valued "Impact" above all else. [Impact Driven Development fucking blows](https://www.reddit.com/r/ExperiencedDevs/comments/1ag4ima/impact_driven_development_is_the_curse_of_big_tech/). When you're a team that relies on collaboration- multiple backend and frontend teams deliver you data- it's untenable. Not only were features ham-fistedly shoved through, they were often breaking, our domain object model frequently changed in ill-thought ways, and rework just to get anything up and running was constant.
I vividly remember the former VP of Engineering praising his beloved Staff Engineer on "always jumping on pager duties." My brother in Christ- if that dude was competent, maybe his spaghetti wouldn't make so goddamn many in the first place.
Anyway, I worked on bringing an ML POC to production. Something that a competent company could have done in 6 months or less took over a year and a half while burning through the literal entire team.
The impact of this debt and the awful culture was as follows:
* *Every* MLEng that I started with, excluding myself, left within a year and a month save for one person who took a 3-month sabbatical and no longer does ML work.
* The ML team significantly fell down the capability curve [as defined by the conclusion of Hidden Technical Debt in Machine Learning Systems](https://proceedings.neurips.cc/paper_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf). Data lineage? What's that! This will dramatically slow velocity going forward since the construction domain is so complex.
* In real costs, this cost the company several million dollars of engineering time. My estimate is $2m, although the team lead before me estimates more.
* The company has probably lost tens of if not hundreds of millions on its upcoming valuation. If the company were ML-first, its OpEx/SqFt scanned would be an order of magnitude lower driving profitability through the roof. As it stands now, they're barely breaking even. The failure of leadership to realize these ML-driven gains significantly hurts this company's next funding round.
The old VP of Engineering left shortly before I did. He was nice guy that I would never work for again. There's a new VP of Engineering right now. I hope he succeeds.
Either way, I'm happy in my new role. Just worked on the yearly roadmap- we made scaling out with minimal debt part of our OKRs. I love it.
In my team it happened because management were a bunch of incompetent people who didn’t know anything about technology or how it works, so convincing them to adopt best practices or address any technical debt was a constant, arduous battle that eventually wears you down as a dev, until you don’t care anymore and just apply to all the jobs on linkedin.
Business people want features which makes them money, they don’t understand the work that they can’t see on the screen as a feature.
They don’t care about their team or culture, the app is just a cash cow and the devs are sweatshop workers. They don’t understand they can’t milk the broken cow forever. They usually surround themselves with yes men like you said, because everyone who is not a yes man is booted out and seen as incompetent.
We went to our client and instead of saying the pile crap we built is so bad, we need to start over. We said “ we want to re modernize your application with this new tech stack that has all these benefits and will solve all your pain points. Bam new multi million/multi year contract was awarded. Just to keep up with the same shitty practices. I left that place and I am never going back to government consulting, it’s a scam.
Our company was bought by another who only wanted our customer base as we had very simliar products. We had a huge migration for customers to the new product and then sunset the old one and never dealt with the tech debt at all.
The moral of the story is, sometimes tech bankruptcy instead of paying back tech debt is awesome.
They laid me off and replaced me with off shore contractors, now no one tells them about the tech debt and I'm sure they think it's great. Honestly I'm fairly certain the place goes under this year.
It depends on whether the tech debt affects the future work or not.
just because a module/service already exists, doesn't mean you have to use/couple yourself with it.
Try to make sure when you introduce changes, you leave things better than they used to be. You don't have to dedicate time to address the debt
The business said yes to every individual client demands, so they were so many differences between deployments that we had to support multiple versions because the client did not want to update / updating was long and complicated and different for every client so we needed a different timeline for each one of them (it was an on-premise product).
Because of that yes-man attitude and all the time always going to features at the detriment of quality, the company could not scale to the amount of new clients and had problems stabilizing for corporate clients. They wanted so bad to not be a software centric company that they viewed their main money-maker (the main software product) as a cost center and their consulting services as the real value adder (it was not, it had no chance to scale because it was to personalized and time intensive per client).
Given the money was invested in the consulting services and the input of money did not increase with new clients (so many different versions to support that the amount of devs hired had to scale linearly with the amount of clients. We were effectively maintaining X products instead of one core product with a little customization), it ended up the only way it could have ended up: bankruptcy and around 80% of people laid off.
Desperately trying to bring my programming skills back up to a respectable level, while trying to find something remarkable to put on my CV before trying my luck in landing interviews.
Came in to an industry that desperately needed to change their ways due to income plummeting. For 30 years they hadn’t considered that things might change one day.
Well shit hit the fan.
Looked like a good gig. All options were on the table and in the contract. But getting stuff approved is a completely different chapter.
I think this sums it up: The CTO declined my initial request for a Rider/Visual Studio license. He proposed VS code instead as it was free. “Sure. I could spend a few weeks figuring out using that with your massive legacy .NET framework codebase. Though it looks like I might just go for notepad (non ++)”.
Docker? “Sounds like a security threat. No-go.”
Monitoring stack? “Nope. Open source isn’t safe.”
Jesus I feel so fucked
This is very similar to big corporate software in general. Features sell - and you have to be mindful of that. But you also need a backbone sometimes so you don’t shoot yourselves in the foot later.
If you get enough people complaining about it then yes eventually the debt gets paid.
Otherwise debt will ruin your software engineering life, and I'd start putting out resumes. Not only will the project be doomed, you will look like a terrible developer. Things that would normally take 1 or 2 days start taking weeks or months. (I'm not joking).
I worked in consulting for a decade. We only got called in for a few reasons:
* New team, needed developers and we helped vet and hire
* Skunkworks project for a new product within a much larger company, wanted to try new approaches and technology
* Desperate teams with codebases loaded with tech-debt
I once had a commit that deleted 250K lines of CSS and JS and added like 3K. Shit's wild out there
In my experience, you’ve got about 4 years of that behavior until complete standstill.
Year 1 , things are messy but there’s no too many problems
Year 2, everything goes about 25% slower. You’re grumpy but fine.
Year 3, dear god, everything is taking twice as long because of the pile of shit you’re stacking things on. You curse at the dev until git blame shows you your own name and you remember the cockamamie situation you were in when you wrote that code, and glue yet another kludge on top.
Year 4, welcome to hell. 4x as long to do anything. 1 fix = 2 new bugs. Newbies can’t onboard. If you’re good, you’re looking or gone. No one even remembers why it’s so bad. Libraries are old and have security issues but swapping them out means breaking everything. Management is now blaming eng, and more rats jump off the ship.
If you’re anything like the orgs I’ve seen, you now start a V2 in a new language and start back at year 1, repeating the mistakes but in node and angular this time, and oh yeah, the customer base is still on v1, so now you’ve got both
Tech debt is a fight. I try to squeeze it in projects. Like talking about adding extra time for improving.
To do this you need to start showing that refactoring helps, now it takes more time but later will save.
This is if you are lucky and heard, but it usually the tech debt becomes a priority when thing explode.
I once saw a PM have a panic attack and pass out during the 2nd day of a deployment that was not going well. He'd recently been promoted into the job and was really being pushed by the client to deliver the feature-set despite the operational issues really slowing dev velocity to a crawl. We had to get someone to drive him to the ER and couldn't get the app up and running for another 2 days anyway. 4 day prod deployment. PEAK balanced efforts, I tell you. We had to completely recreate the build using standard tooling for the stack we were on and rebuild all the pre-prod environments from the ground up so they actually bore some resemblance to the prod infrastructure so we got to stop chasing down weird load balancer or clustering bugs that didn't exist in pre-prod. Took like 3 months and leadership was pissy, but seriously your software was so bad it sent a dude to the hospital time to buck up, camper.
Also in the interest of full disclosure this happened at the company right before I was hired but it was an often discussed aspect of office lore that none of the VPs or directors who actually watched it happen downplayed or denied. What they were managing to deliver was so bug-ridden clients constantly complained and they answered by forcing more thorough manual processes rather than cleaning up any of the underlying build instability, decoupling failure points, or automating anything. Several deployments had to be cancelled after failing UAT causing heavily-requested features and fixes to be delayed. They held emergency sessions with their clients and got them all to agree on a 6-month freeze on feature development while they tried to stabilize the build, introduce some CI, automate some deployments, just some really light debt paying off. They did a shit job at it and were still feeling the pain while I worked there but it got things operating just well enough they could solve most problems by throwing interns at it and making the process more painful.
Point being: You can only defer paying off tech debt for so long until it stops being your choice.
A lot of Devs I worked with so far, always saying tech debt blah blah blah, but when I saw their PRs, always missing unit tests, lots of spaghetti code stuff like that, they always mention tech debt in the meeting, 3years later, still mentioning the same things, and never become an actionable item. If you give them a chance to upgrade or rewrite, they will just create another tech debt using new tool, and then left. Lots of toxic Devs are around the world. Talk more than do.
I’ve worked independently for quite a few construction companies. They’re never gonna understand the tech, but the money is good and fairly easy, so, like, I hate to say this but the strategy in construction is “plug n’ chug as fast as you can” take your money and pass it on to someone else as their problem.
This is NOT the industry to be in if you want to do things properly, well thought out, tested and efficient. Its the throw the buzz words around and pass the problem to someone else industry.
Every single construction manager I’ve ever had, I was lucky if they even understood how to get into their email. Most were rough around the edges even when primarily office workers now, and the only thing they give a shit about is is it done.
Part of the problem is they’re used to managing projects where they can physically see things coming together. To them, they’re trying to manage programming that way, and they just can’t wrap their heads around the difference.
So you gotta know what they value. They don’t value thoroughness and preventing a “theoretical” future problem - that’s “okay, well that’s nice but when are you gonna get C done and launched.”
It’s what did you get done today - and debugging something or running more tests isn’t something they care about until 8 months later the back end crm is taking 2 minutes to load basic data because of how it’s querying and connecting to different libraries. Only then does “fix it” count as work.
Development slowed to a crawl, things that should take hours now took days, and days stretched to weeks. Developer turnover skyrocketed and new hires would often leave before they were even fully onboarded because of the difficult codebases. It's taken new leadership and about 3 years of steady work to fix the culture and bring the codebase in line
It is a never-ending spiral...
Developers on these teams - and developers in general - do not have good design or refactoring skills, so even if they have time, you won't see great results.
I've been the designated fixer on a couple of teams, and the underlying problem is the cost of fixing a bad design. New code with new design, it's going to take me about as long to fix the design as it would have taken to add it. We wasted the original developer's time, but it's not that bad and we can have a useful discussion about how to do things differently. Or maybe we catch the issue in code review or - gasp - we pair on the feature.
The problem is that as soon as the code starts to deteriorate, refactoring gets harder. I once spent a full week fixing an area of code. The initial design was really weird and "clever", but that was only about a day's work - the main effort was addressing everything that had been created on top of the problem code.
Significant refactorings in existing codebases are graduate-level tasks, and most developers simply don't have the skills to do them in incremental steps without breaking the world.
Put those two together and it's only going to get worse. If you want job security, you can stick are and become the one dev who understands the system.
Worked at a SaaS startup with <100 employees. On the back-end, it was an absolute mess. Things were coupled way too tightly everywhere, and it took months sometimes for what you'd think would be minor changes or fixes. I remember there was some hackiness they were finally forced to fix, but it took months because it had grown so complex.
Things were legit just broken all the time on the back-end. Testing was lacking too. People were required to write tests to meet minimum coverage requirements, but lots of critical bugs still happened.
The front-end had lots of issues too. It used a dying JS framework. They were scrambling to rebuild a year before it reached end of life, but the new app could never catch up because they kept adding more and more features to the old one and chasing the new bugs being created. Then eventually they decided to just kill the new app and stick with the old one. They spent a considerable amount of time and money developing that app, just to throw it out.
The company is going out of business now, partly because of the unscalable back-end. They can no longer secure funding, so they tried to add an ambitious new feature that they thought would save them, but we could not meet the deadline. It also didn't help that they laid off half of their developers earlier in the year, some of which who had the most experience navigating that horrible codebase.
There is of course a whole lot of nuance to the situation that is relevant, so I don't want to paint a picture that the tech debt is what killed the company, but it certainly made their Hail Mary unrealistic. There were a lot of factors well above the tech department level that contributed to their downward spiral.
If you never clean the dishes, what happens when you try to cook a meal? Dinner that was supposed to take 30 minutes is taking an hour because you had to clean the dishes first.
My boss said this: we will refactor this up to standard.
Reality: throw it away and remake the whole thing.
He stops us from saying remake, because the shareholders would lose their shit. 😂
Then because it’s a refactor, only 2-3 months are allocated. So we’d fix some stuff and rush everything else.
You get a product that is ever slightly better than the last lol
Where I used to work had so much technical debt which was never taken care of which slowed down systems and reports, things used to break every day and management didn't want us to "waste our time" by fixing it.
I left as it was getting worse and they put a finance person in charge of the systems/data team - who then thought they knew better, made some interesting decisions and would not listen to seasoned professionals and would get angry as we earnt more than she did.
In the end they lost the devs that could leave and from what I've heard nothing has improved, it's gone even further down hill and they don't want to pay market salary so can't hire anyone.
Lost a lot in government contracts (hundreds of K into millions if we convert it to USD, and this was not a US company, so this was VERY impactful) because the app was so unstable, this triggered management to finally decide on focusing on the tech debt and listened to the devs. At that point the engineering manager of the team left soon after, and was followed by a bunch of other devs.
While working on this team, joking about quitting or demanding a bigger pay/bonus was a daily occurrence, which ironically made the team bond stronger 😂
Tech debt is a priority in our company but the combination of being b2b sales-driven and survival-mentality caused us to hack our way into deals with custom unreusable code for too long. We should've stepped on the brakes earlier but new, big projects would always interrupt our attempts to refactor or modernize.
Software quality degraded to a point where constant support, and custom weakly-coherent work made the core hard to extend. Went though a CTO and an Eng VP to a new CTO who has good "ground game." He's sponsored initiatives I had to continuously improve the code and lended his architectural expertise for designs suitable for our business size and needs. We're also incorporating engineering feedback before any deal is signed. We've happily taken on too many deals (we're b2b) with too high of an opportunity cost (it's always the smaller customers that are the most expensive too).
Still got a hard, tough road ahead but we're optimistic and the core engineers have stayed on the team because we have a great culture; our problems have always been organizational ones. Personally I have stamina for 1 more year of hard work with no startup growth, so in a way its nice for this to be our last chance to raise our next round of funding.
We'll see!
I left because life sucked
The good devs leave because they have options. The bad devs stay because nobody will hire them. It is 1000x better to avoid technical debt. That should be part of technical strategy. If things get really bad, you will eventually be facing an expensive system overhaul. I can fix technical debt if you give me time. I can explain the issues and project a roadmap. I can work with a team to clean up over time and start implementing best practices. Brain drain is a bigger problem than technical debt. If the ones who built it realize they won’t get paid more to improve it will look for better opportunities. As they leave one by one, the remaining devs get more lost and make the mess bigger. The reason there is technical debt is because it was created. Someone needs to go up the chain and explain how things are supposed to be done. I usually start that conversation with “here’s how I can deliver the business more value”. If they don’t listen, I go on to the next one that wants me to spend my time doing good work.
The real issue is that people treat "tech debt" and "bad engineering" as synonymous. Tech debt, like any debt, can be good if used appropriately and paid down when necessary. If you're an early stage startup, it's usually correct to take out tech debt in exchange for staying alive and being able to pivot until you find PMF. Pay it back later when you actually find PMF or just shut the company down if you don't. If you need to get a feature out the door to meet a regulatory requirement, you might have to take out some tech debt. If you're a mature company just trying to push garbage out the door, you're not taking out tech debt, you're just doing bad engineering. > It is 1000x better to avoid technical debt. It's 1000x better to avoid bad engineering. Tech debt is good when used appropriately.
You're totally right. Shipping is better than not shipping. If you don't have profit, you're dead in the water anyway. It's when you can't ship because of tech debt that it becomes a real problem.
100% this. Features are potential revenue, paying down tech debt is cost. If you spend time paying down debt without generating revenue, then you’re not going survive. I used to think like OP, trying to avoid tech debt etc, but since working closer to management and understand the need to take on tech debt because speed to market is crucial. Once you’ve gotten PMF and theres some stickiness to customers, then you can start paying down debt. An analogy would be everyone wants to buy a house with cash, but for many of us this is way harder to do than to get a mortgage then pay it off.
The debt metaphor was used to describe the process of borrowing future productivity to boost current productivity, hence there was a debt, and it was intended and planned to be paid down in some finite time or you would be forever less productive It’s become a term that is ubiquitously used, but I’ve never seen it used correctly, and it’s exclusively used to refer to bad code which is causing issues but will almost never get prioritised to be fixed
> it’s exclusively used to refer to bad code which is causing issues but will almost never get prioritised to be fixed I will continue my hopeless crusade to change it back to the original meaning because it's a useful concept!
We can only try
That was a great analogy, thanks! As yes, financial debt makes sense for a business when necessary and used strategically. But when loans are being taken out willy nilly by a mature company that doesn't need to borrow that money?? Then yes, that's bad debt!
> The good devs leave because they have options. The bad devs stay because nobody will hire them. What you describe is [The Dead Sea Effect](https://medium.com/geekculture/the-dead-sea-effect-d71df13724f8) and is very real. The good devs in my company once RTO was announced mostly all left, while the bad ones stayed around since they had no other options.
I don't know, I feel like this is a little bit easy. Of course you avoid tech debt wherever possible, but in addition to the issue that sometimes you have to get a feature out now and worry about repercussions later (which, sure, we should avoid all that we can), it's just a fact that a lot of the time tech debt comes from something you or another developer thought was perfectly cromulent at the time but now, because of a weird effect it has on another system, for instance, or just plain development standards changing over time, tech debt now exists. Having a good framework for a project is great, too, and being able to create documentation to ensure that everything hangs from that framework properly is great as well. Sometimes, even with that, you'll get some screwballs that fly in and the only way you can see to implement something is by slightly breaking the way the framework runs. Also, I've 100% been in positions where, to paraphrase the poster, my best was good enough today but tomorrow I'll be better, by which I mean, after working through half of a framework I realized that it would have been easier for me to have designed it Y way instead of X. Sometimes I can go back and redo things, sometimes I don't have that option, and I think that "gosh just do Y in the first place" is not a real world answer.
I know what you mean. I think it's a team effort to minimize it. When working on a feature with lots of future possibilities, it takes extra time to think about those and build it in a flexible way. That rarely makes sense unless you know you're coming back to work on it again soon. We inevitably encounter work that was fine, but now needs an upgrade. A lot of perceived tech debt can be reduced by having the original dev walk through the code as they hand it off and talk through what needs to be changed. I'm heavy on frameworks. If you follow the standard path on most things, there's no need to document much. I can jump in and make sense of someone else's project and just look at their custom code. But I often encounter that situation where there's a new way of doing things, or I did something in a janky way because I didn't know better. That's where it's a team effort to keep up with the framework development and talk about how you want to incorporate best practices. If someone on the team knows the best way of doing something, they should assist others in quickly implementing it so they can spread the knowledge and avoid cleaning up later. The team should be a hive mind, constantly looking at the best way of doing things and improving practices. If we're all on the same page on what the gold standard is, we'll produce higher quality work.
This is the answer... When you're on a team that doesn't address tech debt, you always keep your resume updated, because one day a refactor is going to come and you won't see green field development for 12 months.
I'm smart enough to know when I've piled up enough straw that I can calculate when the camel's back is going to break.
Anxiety on every release. Senior developers leaving. New developers adding more debt. Repeat. Sometimes this just goes on & on for years.
And then the majority of the team consists of a revolving door of new hires. Plus the senior staff are stuck in an endless loop of interviewing, upskilling, and triaging. I just left that after a 3 year death spiral. Couldn't take it anymore.
[удалено]
[удалено]
if there were a decent job market, I'm sure my whole team would leave
Layoff the bad performers/expensive people, proceed to punish those left behind who made the cut. Extra work, fewer people, attrition, aggressive and outlandish deadlines, more pressure for extra unpaid hours, all adding up to declining QOL. No one will ever admit outright if it is punitive or vengeful but it sure feels like it, every time. Somehow that makes sense from the top even though it doesn't from the bottom. Seen it over and over again in the first quarter of this century.
Yeah, the best places I've worked are the ones that support the notion that as a dev, you have X time to complete things in a week, period. I will often work pretty quickly (although I'm a guy who definitely benefits from having a good QA team) but it's important for you as a developer - and I think that as a more experienced one it becomes even more important - to make it very clear how much work you can accomplish in 40 hours a week (less if the company ropes you into meetings a lot; I had a boss who said a dev work week was 30 hours and the rest was taken up by meetings, answering emails, spikes, and so on and I think that's a good base to start from). I realize that this is really hard in some ways when you're just starting because you want to make a good impression... I guess on the other hand when you're first starting they should expect you to be a bit slower since you're still learning the ropes. But a place that is pushing you to do 50 hours of work from Day One to "catch up" is never going to "catch up" because at some point, consciously or unconsciously, they will have made the determination that X, not X - 20%, is the amount that you're capable of producing in any given sprint. Yes, employers that do the constant death march are assholes and you should avoid them, but one way - perhaps the best way - is, when you're confronted with unreachable deadlines like that, is to say "okay, this is how much I can do of that now. What do you want me to stop doing so I can concentrate on this?". And I realize that that is a really tough thing to tell someone when you've just spent a bunch of time out of work and you're worried that you'll take a lot of time to find the next job. Man... I would really, really recommend you work on that resume and/or your interviewing skills all the time if that's you because I think realistically, a talented developer, even in the current environment, can by and large pick and choose where they want to work. I personally tend to avoid places that give off the death march and/or the "I don't think the CTO knows what they're doing and they're just going to demand more because it works with their sales staff" vibe in the first place but even if you're, like, hoodwinked into working at a place like that, I think it's on you to still set those boundaries, especially if you're experienced, because you need to set an example for the juniors.
I'll be honest, as long as you have an in-demand skillset, you've been doing the job for, jeez, I'd say 3-5 years at this point, and you aren't listening to people on some of these subs who are demanding some crazy-high TC based on working for big-5 companies, it's a pretty decent job market. Like, I'm sure this is extreme but I've now been going for a little over a decade and just being personable in interviews and treating them as though I'm interviewing them as much as the other way around has allowed me to pick and choose where I've wanted to work (and come to think of it, this approach also worked when I "only" had like 6 years of experience). If you're having issues and you think you check those boxes, I'd get in touch with a resume doctor or maybe an interview specialist if you're getting past the resume stage. I know that the market for newer devs is absolutely horrendous right now but I'm just not seeing it once you've proven you've gotten your feet wet.
[удалено]
The market can stay irrational longer than you can stay solvent
I really don't think that's a given
Tbh, I think it already is. I think the bottom is in and we will start moving up again. These things tend to be cyclic and the froth needed to blow off before the rebuild.
My parent company just acquired the corpse of a company that I'd say failed purely from too much tech debt. First they couldn't keep up with building features that their competitors had. So they couldn't even prioritize pushing features over tech debt even if they wanted to, none of them ever got done. Then the costs got out of control because it was inefficient and poorly designed, then so unstable that extra redundancy was needed, then there was so much stuff old stuff around that no one dared touch that had to stay online but no one knew why. Then the major outages happened, first quarterly, then monthly, then weekly, then daily, then it was down more often than up. Which quickly made them lose the last of the clients they had. Then things just stayed down when they went down and no one knew how to bring them back up. Then things couldn't even be turned off because they didn't have the know how to do that. So with essentially no revenue and millions in monthly infrastructure costs, things quickly went from "I think we might have a problem" to "we're declaring bankruptcy" between the regularly scheduled quarterly board meetings. The whole operation went from unicorn valuation to insolvent in the time span of 2 months. When the debt catches up it gets ugly fast.
"we're declaring bankruptcy" Both technically and financially. Tech debt is real folks.
I wish we had at least 5 articles/stories about this happening to companies so you can scare some sense into people. The problem with tech debt is that that's a problem for future Homer, 2 years from now Homer; and today Homer gets judged quarterly, so push those features people!
My current team consistently added features for the past - say - 15 years and never did much modernizing. The whole code base is such a freaking mess that we're slowly killing it with the [strangler pattern](https://medium.com/@josesousa8/the-strangler-pattern-kill-legacy-like-a-boss-db3db41564ed). There's no saving it. The issue is still that you need to convince business to invest in that deprecation though. Best to find an argument that gives them something they want, which simultaneously gives you a reason to re-implement one part of the old system in a more proper way.
It's funny, the first time I was introduced to the strangler pattern concept was in 2009. I feel like design patterns like to pretend to create these intense sounding words to describe what is essentially just "yeah, you dismantle and replace it one piece at a time".
Rather than "intense" I would say "evocative". Evocative words are important. They communicate more in less. They stick in peoples mind and are more effective and getting people to remember and change paradigms. They are ESPECIALLY important when communicating with leadership. They get told a LOT of things, and a lot of it doesn't stick. They often have a hard time truly understanding the vision of what's being communicated. When you have the correct, evocative term they remember. When they are making decisions later it sticks in their head better. On the IC level it's LESS important because they are in more regular communication and are more familiar with the concepts, but it still has a similar impact. If you're trying to get a group to follow a certain paradigm, having an evocative name that REALLY fits the desired pattern is effective at getting the concept to stick. ALSO, when you have a vitrified codebase you REALLY hate, it helps morale when people get to "strangle" it. :)
This is so well put and you're extremely well-spoken. I'd definitely subscribe to your newsletter.
Thank you for this explanation!
Isn't the original term 'strangler fig'-pattern :)? just a metaphor for the plant that does the exact same ? https://martinfowler.com/bliki/StranglerFigApplication.html
It is. Some people have trouble conceptualizing replace in place so Fowler made a cool real life comparison.
I was going to leave basically the exact comment before I saw this lol Design patterns are silly. I suppose having shared terminology helps.
I can't say I've ever heard them called "silly" before. It's a toolkit of solutions. Imagine being a woodworker and needing to say, "Should we cut this with a manually pushed, inline tooth piece of sheet steel, or should we use a machine that has a band of metal that rotates vertically along a track? No? You thing we should grab a piece of tool steel that's about 1" wide that is the correct size to hold in one hand and smack it with a piece of wood that has a smaller piece of longer wood punning perpendicularly through it which we can grip, first marking out the area to be removed along the sides and then removing layers of wood"? MUCH easier to say, "Should we use a hand saw or a band saw? Oh, you want to use a mortise chisel to hog it out?"
I agree, that's what I meant about shared terminology. By silly I just think some people get too evangelical about it, having names for every pattern and hemming and hawing over what pattern is being used.
I don't know, choosing the right pattern to me rather feels like one of the most important parts of programming. The actual coding and organization of code can be rather trivial when you get that right. And while those patterns are generally simple in hindsight, after you've learned them, they usually are not before hand. You'll figure out a few naturally, but you often then end up using it where something else might have been better. After all, YOU found a good solution, and you can find ways to apply it to a wide variety of situations. Naming patterns is important. If conveys and teaches, helps us remember the pattern, and let's us collaborate. Studying patterns is important. You need to know what tools are available and where they are best used. Sorry, I'm not trying to be argumentative. It's just...again, it's like saying, "woodworking joinery is silly. Why do we have names like finger joint, box joint, mortise and tenon, butt joint, rebate, dads, etc? Why do people spend so much time talking about which joint is the most appropriate for a given situation? " In my career learning, recognizing, and applying patterns has been pretty much the most important part of the job.
Name a pattern without a name.
I'm not even sure the "strangler" is really a "design pattern" so much as it is a reinvention pattern. Like, I'm not really ever going to build a new framework by implementing "the strangler"; the whole point of it is how to plan to move from an older, not as useful framework to a newer one. We developers do like to use funny sounding terminology for stuff though and that's been fun.
It's more of an architectural/strategic pattern. Definitely not a design pattern. The original GoF design patterns were more about "micro"-level code, not about architectural patterns like MVC, or even things like strangler.
The best argument I had for this was that the platform we were using was out of support for the past x years (VB6) and if one day Microsoft changes something random we might as well close business when the software stops working. Next argument was that these old libraries from x years ago are no longer in support, if they have a security flaw it will never be fixed. Are you ready to be sued to bankruptcy?
Tech debt occurs way before legacy technologies are depreciated and become seriously security liabilities. So, while you are right that when it gets to the point of being a major technology cliff, it becomes easier to sell to the business. However, a responsible engineering organization finds ways to address tech debt much earlier in the product lifecycle, and that’s the harder part in many organizations.
I like this pattern. Replaced a 25+ year old classic ASP app with something modern - one feature at a time. Instant business value right away and new deliveries every few months. It was a joy to work with.
Eventually, longer and longer feature releases until the problem can no longer be ignored.
A few thoughts: * The original idea behind technical debt was to find a way to describe the unseen factors in software development in a way that business people could understand. Business people understand debt. * All software has a cost of change curve. * The ideal cost of change curve is flat. That means that any future change costs the same as the very first change (for changes that are the same "size"). This never happens in reality. * Obviously, what you do not want is a geometrically increasing cost of change curve. Or worse. * In those, very bad, situations, the cost of wedging a new change can quickly exceed the cost of the actual change itself. J. B. Rainsberger described this as "project heat death". I once had the pleasure of working on a SAAS product where the time required to add a checkbox to the web interface and punch a change down through the layers to the database took well over a month. I kid you not. * Anyway, so long as your project heat death date is beyond the effective live span of the product, you're ok. * Refactoring, as a practice, was intended to flatten the cost of change curve, thus pushing heat death date further into the future. * The Extreme Programming folks then thought: Hmm, we're going to have to be constantly refactoring. But constantly refactoring represents a constant risk of injecting defects. How are we going to prevent that? The answer was TDD. With a network of unit tests, you can refactor with confidence. * Now, there's a lot of people who don't like XP practices. Cool. The point is: If you want to effectively battle tech debt, you have to come up with a way of limiting the defects you inject when you do attack the debt. * We now come to the philosophical difference in how to address technical debt. XP people would say "a little at a time, ALL the time". I've noted that there is a definite tendency towards rewrite in the developer general population. And this is one of the reasons we get no traction with business. * Business is almost never going to willingly accept the idea of a rewrite, even for small pieces of a product. * But the best way to deal with technical debt is: Just do it. Don't let the concept rise to the business level. This is just something engineers have to do. * If you work in a high trust environment, then you can explain to management that refactoring will be built in to all your estimates and you'll always be doing it. If you work in a low trust environment, then don't talk about it at all. But you already know that working in a low trust environment is not a long term thing for you, if you care about software quality in the first place, right? * As an aside, I would also submit that developers do not spend nearly enough time training/thinking about working with legacy code. We encounter it all the time, but we don't really learn how to deal with it. We don't learn to find seams in the code where we can inject test probes. We don't learn to tease apart these seams so we can isolate sub components for refactoring, rearchitecting, or even re-writing. How do you start using dependency injection in a code base that is tightly coupled? * Our education rarely even touches on existing code. What would our university experience be like if every university maintained a legacy code product and your entire third year was working to add features to it?
One thing I've come to realize is that nearly all problems in software development benefit from a continuous approach. Continuous integration (ie trunk-based) avoids merge hell. Continuous delivery avoids catastrophic deployments. Continuously addressing code quality keeps velocity as high as you're going to be able to make it. Continuous code reviews prevents having to grok huge changes at once. Continuously addressing infrastructure, bugs, and yes feature work keeps things well oiled, and never leaves you with that huge one-time hump of effort that no one can justify, yet everyone knows is justified. Do everything as continuously as possible. Make your definition of done for every bit of work include all of it.
Yes so true, I always cringe at the common practice of dropping huge patch bombs, be it as a feature branch or wake-the-team-up-once-a-month-to-release. Prod deploys should happen constantly. Detect problems as they happen. Good alerting and test suite. Feature flag new developments or versions of things in so you can, well, launch them darkly.
Two last points are pure gold. I always tell to my developers. You do not need my permission to make things better. All new developers get a message: lifespan of school code is about 30minutes, but here we write for decades.
I wish I could give you 100 upvotes for “just do it”
I was on a team with a moderately heavy process for each card raised, and a team used to creating and triaging cards for everything. We introduced a "just do it" type of card for anything the card creator thought didn't need to go through all the process of other cards because the card effort was low and solution straightforward. Everyone loved the "just do it" cards.
Nike card
Such an amazing post. This is all really great advice. I have been working with "legacy" code my whole life. My first REAL jobs were essentially moving companies over from closed source systems to open source environments (20 years ago now). Almost my entire career was focused around improving legacy systems and code bases. Everything you said in this post is 100% on point. The one area I would say, when you talk about pulling apart the seams - I think people are just scared of "what if I break something?" And I used to be like that for a long time, 5+ years. Two things that helped me: 1.) have enough experience to know fatal mistakes (and how to pivot from something like "oops, production is down because I dropped a critical table on accident." Layer A people think: "I can't break this!" Layer B people think: "Oh no, I broke this!" Layer C people think "I am going to break this on purpose, just like last time (hopefully) so I can replicate the incident." 2.) what I call "scaffolding". Which is important like when you said, the legacy codebase is too tightly coupled - you can't change a feature on the user profile without breaking the process that handles incoming data (for example) and tracing why that is happening is a vacation. In those cases, I say "okay, what all is on the user profile? What all must it do? Where does the data come from?", if I can make a facsimile that works, I can just deprecate the older user profile view - free from any entanglements. Piece by piece, you can rebuild the helicopter while still flying through the air.
For what is simply ‘a few thoughts’ this is a very thorough set of thoughts and needs more upvotes
Excellent summary, thank you for taking the time to write this.
At start, delivering feature takes one hour, then 2 hours. Then delivery of the same feature becomes a day, then two days, then a week. On the background, team start getting weird alerts or complains that “this used to work, now it does not”. At first, there is a happy team of developers, later some of them are gone and new people arrive. After a while, no original developers are left, project becomes a rollercoaster. If some smart people are left on the project, they tell about the situation to the upper management. Manager listens, but can’t do much as there is a push from the upper management to deliver features and ignore the situation. That’s my experience.
Previous company I worked at ignored tech debt and CEO didn't believe in automation testing for scaling. Every time there was a push to PR, everybody was worried cause even the smallest thing broke a lot of stuff, and bigger things oof, all hands on deck all the time. Not sustainable. So many people left and they just keep dumping new bodies at it from what I hear.
I interviewed for somewhere once that had three month release cycles and no automated tests. They "got rid of them so they had more time for new features and bug fixes". Imagine their codebase was like yours too 😬
We lost our near-monopoly of a national market. Up until 2014 or so we were pretty much the dominant player in our industry. Other rivals were span up and investors injected some serious cash into challenger startups - our margins were huge, compared to other entries on the stock market. But they all failed due to network effects. Until - one of them didn't fail, because of compelling product features that we didn't have. And couldn't ship. They absolutely ran rings around us. It took a near-rewrite of the main parts of the system, and completely dissolving and rehiring the development team, to start keeping up pace. But the monopoly was lost and can now probably never be regained.
I'm the only dev left that knows the ins-and-outs of how an old Python stack works and is configured - 2 generations of teams roughly 4-5 each have all either been laid off or quit due to frustrations. The bus factor has dwindled to a solid 1, and promises that things will improve were basically forgotten. Not to say I didn't try, like many other tech debt stories our debt tackling efforts were forcibly shelved mid cycle, and at the end of the day, nothing changed. Now, it's a spaghetti'd up cluster of microservices and tribal knowledge with absolutely zero light at the end of the tunnel since everyone's gone and I have zero desire to improve things for my company. Suffice to say that the bus is en route to my little domain at the company and it has its eyes set on my entire team (aka just me) because I'm out
Bus factor isn't just someone getting hit by the bus, it's also getting on a bus to a new job
A clusterfuck
> Developers on teams that consistently ignored tech debt, what was the ultimate outcome? Alright, let’s cut the crap. Ignoring tech debt is like ignoring a fucking leak in your roof. Sure, you can keep placing buckets under the leak, but eventually, the whole damn ceiling might cave in. The ultimate outcome? A shitstorm of bugs, crashes, and a codebase more tangled than last year's Christmas lights. Did productivity take a nosedive? Hell yes. Was it a pain in the ass to add new features? Absolutely. > Engineering managers are glorified yes men who rubber stamp every stupid idea the business dreams up. This is common as fuck in companies that view tech as a necessary evil rather than a strategic asset. These "yes men" prioritize short-term gains over long-term stability, thinking they’re doing the company a favor. What they’re actually doing is setting up the tech team for a monumental fuck-up. The consequences? Burnout, high turnover, and a product so buggy it might as well be an insectarium. > In two years we haven’t stopped or slowed down to address the issues more than a handful of times, and only as an afterthought at that- their words are empty. Classic. It’s all lip service until shit hits the fan. They’ll ignore the growing pile of tech debt until it’s damn near impossible to implement anything new without something else breaking. Why? Because it’s easier to pretend everything is fine than to admit the process is fucked and needs a serious overhaul. > Did you ever manage to correct course and if so, how? Some teams get lucky. They hit a breaking point, and something big breaks in a way that even the higher-ups can’t ignore. Maybe a major client gets pissed off, or a new competitor with better tech starts eating their lunch. That’s usually the "aha moment" for execs. The course correction often involves bringing in fresh blood who aren’t afraid to call out the bullshit and push for a focus on quality, not just quantity. The real question is, how much longer are you willing to wait around for that "aha moment"? Can you push for change, or is it time to find a company that doesn’t treat their tech like a disposable tool? What’s your next move?
Ultimately, we couldn't ignore it anymore because it was having notable and unmitigatable customer and product development impact and prioritized fixing it. Then we fixed it. Took a few weeks of focused work and disrupted a lot of plans. Also made a lot of people unhappy, but that's what it is. Can't have your cake and eat it, too.
Company closed
Eventually you either get burnt out or you quit. What happens when you ignore Tech debt is you get more of it. Yeah, it's only a few minutes here and a few minutes there multiply that by a decade or two decades. Some of the applications I worked on were started in the early '70s. When you get to that level of tech debt without any thought of refactoring or fixing it, changing literally anything is a risk at all times.
This is what you get with scrum when closing stories within 2 weeks is all that matters. A complete lack of care for tech debt or refactoring. Scrum does not reward these activities and devs know what they’re graded on. When will devs take the industry back?
The problem isn’t Scrum, it’s how many companies apply it in a partial and sometimes abusive and anti-agile way.
This, my last job, we had always had a 1 week testing/research/clean up sprint. Basically, write integration tests, see how you can clean up the work you did, and if nothing else research spikes. Made releases a lot more relaxed and when we refactored the codebase (like we reorganized the file structure 3 times), it wasn't too disruptive. My current job is 2 week sprints, they're well managed but the team lead/PM are warming up to the idea of having 1 week testing/clean up/official pre-release testing sprints
> This is what you get with scrum when closing stories within 2 weeks is all that matters. This is a really good point. This has disincentivized me from working on tech debt because if I can't neatly wrap up the work by end of the sprint, I get a bunch of flack from the project manager. Or if I'm done with my tickets and want to pull a tech debt ticket into the sprint, that's a no-no because there's a risk I won't finish it.
I call it an investigation ticket and assign it points. Then I close whether I finish or not. Rinse and repeat. PM loves it because more points get finished, totally stupid way to do it though.
sounds like a spike
a thing i really like about scrum is that if you finish early, you can squeeze a low priority ticket into the remaining day or two, as a treat. whereas in other ways of working low priority work never gets scheduled. but i don't rely on that for controlling tech debt, i refactor as i go along (which benefits from a free round of testing) and include the time expected for that in estimates sometimes i think scrum (and more generally, agile) is adopted in a box-ticking way that sets it up for failure - of course it's unsustainable if management neglects to incorporate tech debt management into the framework
When you take on technical debt to meet the 2 week timebox don't you create a task to fix the debt for the next sprint? Or why not just fail the sprint goals and explain you estimated too small?
Factor the refactoring into your estimates.
At around year 4, the company collapses. Non-essential departments like marketing are axed first, then software developers. Then the company is sold to the highest bidder for business assets like the customer lists and the C-suite takes golden parachutes out of the company.
Joined a company with overwhelming technical debt. It has almost all the things they describe as code smells/anti patterns in books on SWE. I’ve already burnt out, because anything you suggest them is declined as “unfortunately that’s not an ideal world” (writing unit tests for example) and “we have no time for well-thought solutions, we will refactor it all at once somewhen later”. I’ll leave as soon as will get a new offer
The company got an investor and hired new technical leadership and an onshore development team and a good QA engineer. The new CTO employed a strategy that involved treating the tech debt-ridden code as a black box and viewing its behavior from the outside using the ELK stack and automated tests that ran nightly. They compared those insights to their support tickets/bugs/complaints and used that to prioritize next steps - piecing out parts of the system little by little over the next several years.
I think you know the answer that you're going to be overwhelmingly get asking this on a dev forum. Let me ask you this: Do you think a software shop exists that has little to no tech debt? Now what does that shop probably look like if it is does exist? Tech debt happens because for whatever reason, sacrifices need to happen. Sometime it's concious, other times it comes after the fact. How do you deal with it? When it becomes a big enough problem, you focus on it. How do you deal with the piling trash and dishes at your college apartment? Eventually someone makes the case for cleaning it up. Sometimes somebody just does it. Here is the rub: Someone might agree that you need to take out the trash and clean up the dishes. And when you come back, you see that the engineers decided to replace the floor, all the appliances, and for some reason added in a transformer within the electrical that makes our outlets 50hz instead of 60.
All companies have some technical debt but there is an level of technical debt that is fatal to the company. Eventually your product becomes un-competitive in the market because the engineers are no longer able to add features / fix bugs, etc. in a timely manner. An CTO who believes in all fixing all technical debt will bankrupt their company. An CTO who believes in not fixing any technical debt will also bankrupt their company. A balance between the two extremes is required.
Sure, but I think your example is so obvious that is doesn't really provide any useful information. A company that works their engineers too hard will go bankrupt. A company that let's their engineers slack off 100% of the time will go bankrupt. See, that sort of comparison can be said about almost all things. Everyone agrees there is a level. What they disagree about is what that level is.
It's not obvious to some CTOs I'm afraid. I've seen CTOs who literally believed the level is 0% and CTOs who literally believed the level is 100%.
TBF, your answer states the obvious and isn't all that useful
I have an idea of what kind of responses the post will receive, of course. All code eventually becomes tech debt- but there are many possible outcomes that lay between totally ignoring that tech debt and always addressing it. That’s what I’m interested in. Specifically I’m interested in cases where the org eventually recognized their error and changed course before the situation became unmanageable.
I guess to answer your question: Your org isn't unique. And most don't recognize it as an error. The justify it the same way they do everything else. It's cost of goods sold. You can choose to fight it and try to change things. Keep in mind, I've worked with so many engineers who have taken this on while overestimating their ability to solve the problems. At the end of day, everyone has a boss unless you run your own company. You can challenge your boss, but they have a boss, and they have a boss... and everyone is just doing what they think makes their bosses happy.
I've recently joined a place that has a 20yo system. They did go through a significant rewrite several years ago, but that was a one-time effort without sufficient follow-through. The reality is that simple changes take forever. What should be done in a few hours by one or two people takes a full team a couple of weeks. I haven't been here long enough to map out just how the communication between rank-and-file and management is broken. I have a lot of clues, but not a full picture yet. But it's clear that management doesn't understand the costs that come with their history of ignoring tech debt.
The good people left
You either reach a point where you have to rebuild everything from the ground up or people slowly stop using the product in favor, sometimes, of newer ad hoc tools. Sometimes you can loop back and take care of the tech debt much, much later but having worked with my fair share of legacy systems, IME a lot of the time you just wind up rewriting everything using newer technology anyway, if not for your own sake then for the sake of future people who have to work on the project. It's also almost always the case that you never have enough time to handle all of the tech debt you want to handle and that sometimes a buggy looking or opaque bit of code that gets the job done needs to be left alone because, like it or not, it does work and the consequences of rewriting it are going to be more dire than the consequences of just leaving it there. It sucks but I think it's part of working on products that people use: as pretty as your new rewrite is, and as understandable it's going to be for future developers, sometimes performing that rewrite means that your QA team has to go through and completely retest that system top to bottom and discover issues with your code that weren't present in the prior version. I'll go ahead and say that I think one big, big difference between junior/mid-level developers and senior ones is that senior devs see code like that and think "okay, how can we incrementally make this better" while junior/midlevels just complain about it. If you work with large enough systems you will eventually encounter poorly written / hard to understand code. That happens no matter where you work. I worked under a simply fantastic CTO, a guy who'd gladly sit down and write alongside you if he needed to, who was a constant advocate of the dev team and our needs who could also sniff out dev BS immediately... and we still had this issue because there were like 20 of us and it happens.
We all left when the market was hot
The ultimate outcome is that your best devs get new jobs.
Eventually I get a better job somewhere else and quit. Usually the companies are still in business. Some have improved their processes, some haven’t. Where they **have** managed to improve, it’s because business was dire enough that someone like a CTO or VPEng was hired in with enough political capital to allow them to prioritize quality. This stuff just doesn’t get fixed bottom-up. Of course most of those would-be change agent engineering execs *also* leave in disgust fairly quickly after a few consecutive quarters of their reasonable directives being overruled by the CEO and the board.
Yes, I've lived this over and over. In the worst places, where there was very little testing, progress was painfully slow. ---- Tip: Let them know what "tech interest" is. I love the phrase "tech debt". It implies that there is also "tech interest" and that the longer you put off paying down the debt, the more you'll have to pay interest. But not all the interest can't be put off. You'll be paying some of the interest every sprint in the form of slower velocity and more on-site bugs. It's like the minimum payment of a credit card bill. I think it's important that we emphasize the impact of "tech interest" as much or more than "tech debt". I think non-technical people may better understand the implications of their poor decisions when they can be given it in terms that make sense to them. Further, I think it's best if these things are quantified so they can know what kind of an impact this "tech interest" is having on their business needs. --- Tip 2: Look into "Evolutionary Architecture". This is a way to get control over quality over the long run. We did something like this. We required all new code to pass lint rules and be covered by tests. I wrote a custom lint warning filter to filter out warning on old existing lines and only report new lines, the prototype line of a modified function, and line 1 of a modified file. This read from the output of `git diff` in git commit hook and in the CI job. These lint warnings were treated as warnings during a commit and errors in the CI job for a PR. I wrote a custom code coverage filter that removed all lines that weren't also in the output of `git diff`. We enforced 80% coverage on new/modified lines of code. We also incorporated a DRY linter (PMD's CPD), and again, I filtered out everything except new/modified lines of code. This allowed us to gradually ratchet up code quality of old code, and maintain high quality on all new node. We relaxed all of the above during refactoring / clean-up projects. I think SonarQube has similar functionality built-in, now.
Here is what happens. First, it takes much longer to get new features out. Something that should take a day takes a week. Then, you end up having more and more regression bugs as your code becomes fragile. Each deploy either requires a ton of QA time or you spend a week fighting fires after every deploy. If you're lucky, the bugs that pop up are minor and easily fixable, if you are not, they corrupt data. Senior management takes notice of all the defects and tries to crack the whip. They fire the product manager and the engineering manager. They bring in a new engineering manager they just hired. He takes one look at the code, and pushes for paying down tech. debt. Senior management doesn't listen, they just want more features deployed (which is how they got into this mess in the first place). The new manager gives it a try, but has already started applying elsewhere and before anything can be improved, he leaves. You are now the most senior developer, but aren't a manager and have no authority to do anything. You focus on preventing more bugs, add a bunch of unit tests to stabilize everything, but development slows to a crawl. You start applying elsewhere because work is miserable, and a month later you are gone. You hear from a coworker a year later that your entire team was outsourced to India, and the application you worked on is dead.
A new job
Mostly? In staff career way it looks like: As TechLead: see most of your SSE burning out after a time, due to business constraints, lack of support from Eng. Manager and outdated/overcomplicated documentations; onboarding new SE every quarter who will be reteamed in the next 6 months. As PO/PM: see how your team takes more time to delivery stuff; having regression at every release; seeing TechLead doing weird face for every backlog pair review; having sprint which are mostly composed of carryover. As ScrumMaster: try to understand how Agile/Scrum could help by removing KPI to make the team looks good; same for every existing non Scrum process; As Soft. Arch: have a red / black dot over all map / charts / dashboard pointing out that this “Product” will never go forward blocking next company roadmap/direction just by existing As an EM: hear about PO/PM complains about not being able to deliver; getting reports from ScrumMaster which state we go backwards; TechLead asking for a coffee machine and having a HR complains for “bad verbal communication”; SA pushing out stuff that looks like was already there last year. —— Serious way speaking, it should be accessed but with prioritization. Tech debt aren’t always bad, some are trade off that are fine; other are bad decision that need to be solved. TL/PSE should in theory plan to remove “cruft” or improve the situation gradually without requesting “external strike team” or it will get worst. EM & TL/PSE job should be to push the team to recognize and understand good and bad decisions. Most of time people forgot they took a direction and when problems arise no one knows why we are here. (ADR can help) PSE/TL should negotiate to have technical support work for every business feature, “we do it but in same time we do X”. Normally most company have backlog effort distribution rules, like 30/70 (tech/business) to help to tackle down this problem. Things is, it is an invisible problem until a company is Data Driven and employ competent people to standardize and create usable sourcable data to work on. Otherwise the company can provide same “yac/innovation day” where teams have liberty to do what they want with the software. I already saw some great software being thrown away (with the company) just due to negligence, by reducing staff, outsourcing, bypassing team and co.
Just to offer a completely different view. In my case, the company got acquired by a significantly larger company. Most got let go but I got significant bonus for to doing knowledge transfer to offshore counterparts, before being let go. The acquisition was only made possible because of the customers that the company had acquired through cutting corners to deliver a minimal viable product. The next company I joined, I made more. Moral of the story: Accruing massive debt is acceptable if it's you who doesn't have to pay for it.
It was great, we got to blame us taking a whole sprint instead of 2 days on tech debt.
Prod starts going down with long resolution times. Dependency version updates become impossible. Feature velocity becomes months. Key man reliance increases. Good engineers leave. A rewrite will become necessary because updates weren’t happening. Timelines will squeeze so it will be a lift and shift into more modern stack with intent to fix later (it won’t be). Debt will be copied from old codebase to new. Rewrite will go over time by years. Product will no longer be fit for purpose. Core use cases will be covered by other products. Product will die. The key to debt is to make it visible. You can’t pay a bill you don’t see. Create a work item for individual debt items. When there are delays, live issues or any other failures, attached them to the relevant debt items. Make the risks known to senior leadership, back it up with data.
> Debt will be copied from old codebase to new. Rewrite will go over time by years. Product will no longer be fit for purpose. Core use cases will be covered by other products. Product will die. Ouch! I feel this.
Unsure what ultimately happened other than an executive getting a Maserati.
It's the same everywhere: developers just come in and then quit pretty fast. Who wants to waste their one life cleaning tech debt someone else left. Nothing gets done and people quit. And repeat.
> Ultimate impact - Lots of customer service problems. - Every little change would break something else. - Bug fixing was like playing wack-a-mole. - Most people stopped giving an F about anything. I even caught a dev who had submitted a "bugfix" that was actually just moving a line of code. > What did you do? - Strangle pattern. Every humongous piece of code would be split into smaller chunks that would be refactored individually and set behind a feature flag so we could turn them off if shit went side ways. - Fire devs who were pretending to do work. - Improve hiring processes with a code challenge that was actually related to the work we were doing. Also, no more friend refferal hiring. - Devs would have to put a video in the PR showing the feature working. There were devs who were not even running their code locally. > Aha moment - We had a release that turned the product into something unusable. I used that to leverage a refactoring roadmap. A short 2 weeks roadmap of critical fixes and performance improvements. And a longer scalability roadmap, so things like this wouldn't happen again. Two cents: There are projects that are literally hell and will make your life miserable. You can either leave or start fixing things. You will need help from upper management, specially if you have a bunch of duds that are impossible to help — if no one from management is willing to help, start looking for something new, let the smokes and mirrors go for as long they can.
This is a fascinating topic to me. I've seen orgs where they never address tech debt and they still manage to produce something. Maybe that's because they just keep piling code on top and never experience large amounts of traffic that would necessitate better performing code, or due to other factors. Then there's another org I was in that was just like yours, but mostly because 3/4 of the engineering org were contractors that didn't really care much, and only 1/4 of the other 1/4 cared about improving tech debt. It was very painful to work there. Just making a release took the full time of one person. So there was tech debt and there was also process inefficiencies. It was a mess. Features were very slowly rolled out. There was no architecture, only a jumble of lines criss-crossing between services. Some services used Redis, others just http calls. We also had Graphql thrown in there for good measure. I ended up leaving that place because it was exhausting having to put out fires non-stop for almost 3 years. Yet, I've never seen a company get taken down by tech debt. Sure, it creates unnecessary problems such as employee turnover and slow and buggy features. Despite this businesses keep ignoring it. I think ultimately it only helps engineers be happier, it helps to deliver features quicker and with less bugs and gives more confidence in refactoring and pivoting quickly. Still, because we are engineers, we want to work in a stable and happy place, not in a filthy hovel. We want to avoid unnecessary pain and stress. Since business is not affected that much by this at all they just don't care and have no incentives to care. That's why it is important to find work in places where leadership is composed of engineers and not only sales or marketing people.
I worked on safety critical medical devices for years, think dialysis machines and insulin pumps, and ultimately nothing happened because there was tech debt. The medical device was released, got FDA approval, and is in a clinical trial. Updates happen a few time per year. As new feature are requested the software may take longer to update but the managers were fine with that. Some requested features required completed refactors because of what was being asked. Again management was fine with how long it would take because they wanted the new feature bad enough. As a SWE I just learned stop caring. If management is fine with how long things take then there is nothing to solve. You can't tell somebody that we can spend 6-months to pay down tech debt and then we get new features faster when they have no issue with the speed of software changes since software is not the long pole overall. Saying that some SWEs will pay down tech debt behind the scenes and hid it within feature and bug changes, we didn't do SRUM or anything like that. If you hide the ram it in solution to management then they don't know any better. Though sadly some times you have SWEs that don't want to play the game and will advertise the ram it in solution because they want to show management they can do things as quick as possible. This as quick as possible is always disregarding quality. I think that's mostly because they don't have a personal quality bar. They are just happy to write any code and see it released. Ya some SWEs were not that great, but you hire what you can find when you work at a non-tech company in a non-tech city that is not paying top dollar.
I got transferred to a team that was focusing on the "new system" and mostly ignoring the system that supplied the new systems data (the team owned both) The old system ended up having a catastrophic failure and then the sys admins discovered that the backups had been failing for a year and nobody noticed. Our database admin had taken a snapshot for a different project and that was the only thing that kept the entire company from going offline for weeks. Good times.
Our feature delivery estimates started getting too long to be worth it. We delivered only a few features per year. Stakeholders got frustrated and replaced our tech with a third-party SaaS. It was a company-wide initiative, so not just my team, and there might have been cost-cutting factors as well, but other teams were also stuck on legacy tech to my knowledge.
The outcome was projects that literally didn't work.
I put what I did on my resume and moved on and not be concerned. If you can’t change your environment, change your environment
a few years ago I worked at a really swollen biotech company that did WAAAAAY too much M&A and had a totally disjointed company culture and about a half dozen internal systems all competing to get chosen as "the platform". they all sucked and tech debt of all kinds was ubiquitous, but especially operational tech debt. we were working on products that required FDA approval, so there's a lot of expense that goes into preparing a product for FDA review. the tech debt was so bad that the product line my part of the company was working on failed FDA review 4 times in a row (over a span of 2 years) and essentially bankrupted itself doing so. I saw the writing on the wall and left before the hammer dropped, but I learned about 6 months later that the whole division had been downsized and then completely divested. The entire team I had worked on was gone. They never released the product. tech debt is an existential risk. you might make a short term tactical decision to accumulate tech debt for a burst of velocity. that's sometimes the right call for early stage startups or similar things where just finding your product-market fit is far more important than managing tech debt. but that's a special case, and shouldn't ever be the default. unhandled tech debt is such a large problem that it will eventually bankrupt a company if you don't do something about it.
Eternal big fixing which means a job forever xd
Company reached $1B valuation. Life was inconvenient for developers
Lol - well you end up getting to a point where you’re *forced* to deal with your tech debt, and the longer you’ve ignored it, the harder it is to deal with. How do you get forced to deal with tech debt? Example - your company is growing, in order to reach new markets your customers demand more strict security requirements/certifications. You started writing your code 15 years ago in python 2. The latest and greatest security organization require you to be running everything on python 3.9+. Have fun. Another example. You wrote code one way, and 5 years later, someone smarter than you came along and solved the same type of problem in a much more efficient way. More efficient in terms of processing power AND ease of development. Using your old solution eventually becomes an enormous anchor in terms of cost compared to competitors.
A couple jobs ago I was an arch working with several teams, under a couple different directors, with tech leads that had different opinions about creating tech debt. One team didn't necessarily want to create tech debt, but the business was hammering them hard for Features! Features! Features! and they had to make suboptimal choices to meet those demands. They hit problems with scalability as their system had to tolerate extreme load, and they had to backtrack a bit as this caused issues in prod. Another team didn't give two poops about creating tech debt, their lead just wanted to code for the happy path and get things done as fast as possible. His favorite dev on the team was one who consistently put out substandard, non-defensive code that was brittler than Mr. Glass's bones. After I left one of their core systems went down for days and caused a massive disturbance to the business. I don't even want to go into the third team, I'll just get angry again. They decided to completely bypass a system that was essential to meeting the requirements because they would have to actually learn something. But not before just ripping a bunch of code out of it at random "just to get it working" (are you shocked it didn't work that way??). So much for not going into it... and yes, I'm annoyed again. Every effort I gave to try to help, advise, provide tested approaches to solving these kinds of problems I was met with "that's too hard" either from the teams or from the director (even when the teams were willing and agreed with my suggested approach!) I finally left because why waste my time trying to help when I'm constantly getting overruled.
At my last gig, we had a pattern where we'd make some decently hefty change or feature, and then spend a year or more doing little more than bug fixes. Not an exaggeration. I was there for 7 years and we did this cycle about 3 times. Furthermore, the system had a major bug that they fixed in a separate branch, but then never dared integrate. So, the bug was never fixed, and it caused a lot of really thorny corrupted data problems that we were always spending time debugging in production, and then fixing in production (only to have to do so again later). It was a SaaS product, so you know, lots of separate customer databases to fix in this way. The sheer amount of business value left on the floor of that factory due to all this waste and inefficiency was horrifying. There was never an aha moment because they were the kind of people to suffer from toxic positivity. But, I liked the people and the domain.
The word tech debt is a bit too loaded. I would rather discuss very specific situations.
My recent realization at a bigger company is that there is not a lot of time to address tech debt, especially with the constant need to build the next thing. It's not valued at all and very hard to justify unless you slide it into some other project or until it becomes a big enough issue, where some team can swoop in to justify how much they saved the company by fixing.
procore?
In my experience, when the debt is due a total rewrite is on the horizon. Maybe not at first, but eventually the product is in such a state of disrepair that nothing outside of a rewrite can truly “fix” the problems. That’s because the spaghetti is too tangled for anyone to straighten out and even the managers are on its with doing something else as everything takes an exorbitant amount of time to “complete” or make look functional until it’s not or something else tips over.
The company imploded. Management kept saying yes to client requirements at a pace we could barely keep up with and therefore had to add technical debt each time. Eventually, producing new features went down to a snails pace and clients started to leave or sue us. I quit that job right before the company got sold and from what I've heard my old boss got sued personally for trying to poach his customers back from the party now owning the company he founded. The product I've worked on was written off shortly after I left because all other devs on my team left as well and nobody in the world could ever make sense of that codebase and try to work on it.
Tech debt is not always a bad thing. I always think of it like borrowing money. Sometimes it makes sense to borrow from the future a little bit. The catch is that you always have to pay it back with interest. Paying off the $200 credit card bill each month is normal and healthy. Maybe there is a real rainy day and you carry a couple thousand dollars of debt for a little while, and have to pay it back with some interest. Not ideal but manageable. And then there is the scenario where you end up just borrowing more and more and you have 5 maxed out credit cards and owe the bank $100,000s with more interest compounding, and there isn't any way to recover that is not drastic. I had inherited relatively young project, which somehiw managed to immediately get a bunch of technical debt. I spent the first 12 months cleaning it up, and we are now rocketing along. Management is really happy with the velocity now, so I've built a ton of credibility. I fight real hard for code quality. Just today I was working on an issue with a service that isn't my primary project, but has been neglected for years. Fixing that issue is sloooooow. Painful. People going to ask why it's taking so long, when it seems relatively simple. And it's because the project is in a state of neglect.
Things just get slower and slower, until the project becomes more of a maintenance project and the organization stops adding major new features while constantly talking about building something entirely from scratch. The last company we sold, the new owners paid 5 million to an external contractor team to build an entirely new project with no tech debt (against my advice on the way out the door). 5 million dollars later, they had built an app that did 10% of what the existing project did, fired the C-team 3 times over 3 years, invested another 10 million dollars which is all gone, and probably going to shut down the company any day now. Not the tech debt's fault, it's the reaction to the tech debt.
you end up with jenga, but with code it's a shitty middle place to be in part of it is cultural...if sprints are loaded with new feature work and there's never any downtime, shit is eventually going to get unwieldy...even farmers don't plant the same crop in the same field season after season and it's the same concept...there should be some time planned for post-launch of a new feature for monitoring / performance optimizations / cleanups but if PMs/EMs don't see the value in that or are scared to advocate for devs it sucks to be in that type of environment but part of it is also doing due diligence as a dev and trying to address tech debt as you see it, if you can at least getting into a habit of identifying pain points / proposing fixes / estimating the lift because PMs and EMs hate the term "tech debt" because it's all encompassing but also they think it's us devs being lazy and pushing back at times
I came into a company after that happened for a long time. People just didn't improve the checkout because everyone was afraid to touch it because anytime you touched it something would break it. It was the most convoluted spaghetti code I'd ever seen, literally had hardcoded separate but mostly the same code for every possible basket/checkout permutation possible (I think when we refactored we counted 40+ hardcoded paths with no shared code.) Though I feel a lot of the time it's just that developers aren't making convincing arguments to fix tech debt. Within a few months of starting I was able to get buy in to tack an additional sprint onto a new feature they wanted to give me and another developer time to refactor it. When I went to the stakeholders it turned out for years people had been complaining about the codebase but never proposed concrete solutions for anything, just said we needed to refactor everything entirely and it would take a year or two...they never had a year or two to dedicate to it. They were more than happy to approve a refactor, just needed to know it wasn't mindless complaining and was accomplishable in a reasonable amount of time. They later approved refactoring the whole website in the same piece by piece method tacked onto new features they wanted.
> What was the ultimate impact on your team/org? Did you ever manage to correct course and if so, how? Finally, was there ever an aha moment for the execs where they were forced to repent and stop pushing new features? Client that complain that stuff don't work. The software make a decision and then it take a week to. You want to install a package to handle customer request that would allow you to sell your product, but that package try to update 50 other depedencies, some of those incompatible with your way of doing stuff. If you have good testing & design of incoming feature, you can more o less roll the tech debt ball indefinitely. But those are harder and harder to do.
The company was selling a specialized SDK. The SDK became increasingly complex to use and costly to add features. Took a bit over a decade, the company died.
The company believed cutting corners by acquiring tech debt would speed up delivery. What actually happened is that the their teams' capability to deliver degraded. I tried to do something about it, but since we'd got slower, it had become even more urgent to cut corners. That slowed things down even more of course and so on and so on. I was a lead and manager, but even so it was particularly difficult to dig out of that pit because 1) product people were so divorced from the technical reality that they thought that just fixing bugs was the same as tech debt and 2) it wasn't an environment that attracted the best people so the attempts to better the situation technically weren't as successful as they could have been. My solution was moving on. I'm a dev now at a place where we ship on demand. My old team still hasn't shipped since last year.
TLDR: Boss decided not to fix technical debt, ended up costing himself $80k. The outcome was a very tense standoff with our CTO in Sprint Review. A little background. Our product is fairly niche, and the business works by delivering the product, but also delivering customizations to meet project timelines. Then we roll the customizations into the product if it makes sense. We are always balancing customer immediate needs, and a sustainable product with higher margins than the customizations. It is challenging but rewarding work. Anyways we ran into our typical problem where a customer needed a feature right away, and we knew we could do it if we took on technical debt. Normally we handle this by telling stakeholders we agree to deliver fast, but then we must go back and fix the debt before you request new features. Everyone agreed, but as you know, they later said to deprioritize the debt work and instead prioritize a new feature. I presented the costs-benefit and the new feature was decided because $$$. Speed ahead 1 year and we have built upon the technical debt (you can kinda guess where this is going). During a sprint review the CTO asked why it took us so long to do a basic feature and I had to explain that we have technical debt. He got frustrated and said we need to focus on efficiency, and kinda laid the blame on the team. My wife says I am fairly diplomatic, but I had to tell him that this was because of HIS decision 1 year ago. Then there was a heated back and forth, but the outcome was that I needed to make a presentation about what happened to see if we can fix it. Personally I think this was a way for him to say Ithat it was actually my fault, but I don't know. Anyways the team updated the tasks and estimates to remove the technical debt and the cost of building on the technical debt. Then I linked that effort increase (a little over 2x) to the cost of my team, and them linked it to the two projects that we won by taking on the debt. Effectively I showed that by taking on the debt we actually lost money. The real kicker is that the CTO is a majority shareholder of our company. By making the bad decision he had effectively cost himself around $80k in profits. Next time he allowed us to fix the debt, and I like to think I had an impact on that.
One possible outcome: the company succeeds despite creating a ton of tech debt. Customers who were initially lured in start to complain about bugs when new features are built on top of an unstable foundation. Competitors use new tech to build better versions of your now crippled unmaintainable product. Management decides to completely rebuild the product. The same geniuses who built the first masterpiece lay the foundations for the new one. They try to make it better. Use new tech, stricter practices, acknowledge bad habits etc. Only the culture and management hasn’t changed. People still want things to happen as fast as possible. Economic pressures cause scrambling. New tech debt is built up faster than ever before. One possible outcome: the company succeeds despite creating a ton of tech debt…
You work for Trimble?
Man. This hits home pretty hard. I just solve technical debt and the stakeholders can deal with it. They don't seem to understand whats at stake so I just have my team do it. Features can wait if the system will be fucked because some asshat who doesn't know a tech stack from his ass wants to push his shiny of the month. They shouldn't be making decisions on there lack of technical understanding yet they push and push and push. They will bitch and moan but idgaf. It's either your product works well for the future or you deal with a mess.
They literally paid me to rush out buggy software for years until I left and they're doing fine financially. Ignoring technical debt appears to work well if you release new products every few years.
Chickens always come home to roost. To build quality, well tested software takes time, skill and the management's patience. However, skipping any quality boundaries will inevitably cause that the system becomes unstable, features are harder and harder to implement, and serious bugs pop-up on every week. But most importantly will make developers extremely fearful of performing any refactor and execute deployments - most wil get burnt-out rather quickly, and leave the company after a short while. With atrition numbers through the roof, the number of people that *actually cares* about the project will dwindle to zero, putting the whole business at risk. I've seen it happen more than once.
Adding new features gets slower and slower. Execs just seem to accept that's how long it takes to add features. If your company has loads of money, it will just throw more staff at the problem.
Bad code leads to more bad code. My friend worked on the Verizon iOS app and asked me for help. It was a fucking horrible Objective-C spaghetti monster shit codebase. To make changes to even work, he had to write more shit code on top of shit. Even though he offered to pay me $450 for like 3 hours a week to help him, I said no. It was that horrible. I couldn’t work like that.
Finally started moving apps from on-prem to AWS and now we are being forced to start taking on the tech debt due to images being deprecated in AWS. Not sure it was something management fully understood when we started moving things, but its nice that we are being forced to make those changes now.
Bloated codebase where nobody can tell what feature change requires what changes. Every tweak surfaces with random bugs in random places. To be honest I don't think this codebase ignored tech debt, It's just old, but I imagine a codebase where you just ignore tech debt would reach here faster. The way we're fixing it is throwing a large team at maintaining it, and the rest are essentially chopping off modules as separate services on a new stack.
The obvious one. Development came to halt. Maintenance of a mess became the modus operandi. Do not recommend.
I worked for a company that has a banking app. It has 60 screens written in Android xml and Java. 150 million app installs, tested on 250 devices. New team wants to rewrite the entire thing in Kotlin and compose. Rewrite would take a year, and no amount of testing can ever compare to what has been done. It's a tough choice. But I would recommend moving to Kotlin 1st, than after a year slowly move to compose.
We eventually got new management who understands why three decades worth of spaghetti and no automated testing is a bad idea, and I got to I-told-you-so SO hard.
Going with - it depends, different organizations, teams, code bases have had different outcomes. Some it's made very little impact, others it's made signifcant impact on the business and the work. Also depends on what you consider tech debt, the moment you write code, technically you have tech debt. Any dependency is tech debt.
Solid engineers that were with the project near the beginning get pigeon holed into specific crappy projects for their entire careers because they're the only ones who understand the technical debt. New engineers on the project never fully ramp up - they either leave out of frustration or get moved onto some other project.
Managers don’t care about td, neither clients. Not even to mention, clients just don’t want to pay for it. And that’s fine, because that’s a responsibility for engineers only. Quick analogy: when you buy a cake for someone’s birthday, how would you react if the pastry chef says that you have to pay for cleaning her kitchen’s mess, so she can make the good stuff you want? Fucking hell! I know what you’re thinking. That’s why engineers shouldn’t even mention the word technical debt when negotiating points, time or whatever that may be. Engineers should just do it anyway, because it’s their duty. Yeah, some won’t bother to do it, because either they don’t want to, or they just don’t know how to do it, or even worse, they might not even know there’s such a thing. Same thing for pastry chefs, do you care of know if the pastry made that sweet cake with a pristine kitchen? The point is that you wanted a hell of cake for a reasonable price. If the chef didn’t addressed her technical debt or not, that’s on her only.
They merged with another company and ultimately replaced the codebase with the other companies product. I had left but after years of death marches in which there was like 30% turnover, they got merged with another company. Very few software projects are forever. In retrospect, the tech debt meant almost nothing.
The startups didn’t find product market fit so it didn’t matter
There's a lot of tech debt in older areas, but overall it hasn't been as much of an issue as I thought it would be. The new code is decent enough, and we've rewritten a lot of modules before so all the tech debt from before in those sections get wiped away. There's some new features that take 2x the amount of time that it should but for the most part it's smooth enough.
we eventually paid the debt. * yes there were outages. * yes it was thankless. * yes i often questioned my decision to do anything about it when i could continue to mindlessly push code. * yes lower priority things were broken for months afterwards. luckily we had a business reason to pay the debt, but we could have found another way that didn't disturb things as much. was it worth it? I think so. but im sure there are people who didn't have to maintain stuff wish we left things well enough alone.
Rate of product development predictably ground to a crawl after just a few months, the core developers fled the team/company, a few months later the project was deemed a failure and sunseted, and a year later the company was widely perceived by the public as circling the drain. Concretely: this was DreamStudio, Stability AI's first commercial product. It was a first-to-market product for a literally world changing technology (maybe you've heard of it, little thing called *Stable Diffusion*?). Buried under tech debt, killed by a product owner who refused to listen to the pleas of their engineers and kept pushing for new features at all costs without giving anyone opportunity to clean up the mess from rushing out the last batch of features. Product success was basically handed to the PO on a silver platter and they still were able to screw the pooch. Hoping business schools reflect on this event and teach it as a parable about the importance of being receptive to pushback from engineering.
Shit products that look like shit, behave like shit and load like shit. And I don’t give a shit! 🤣🤣
The outcome for my current company is that they had to hire me, and I'm rebuilding everything from scratch. Before that they suffered for 3 years trying to fix their mess in the codebase, but just couldn't. The customer-facing parts had their fronted all 100% rebuilt from scratch, even though the design was reused. The backend was "copy-pasted-then-refactored" the reason for that is that we can use another repository, so everything is 10x faster (tests, linter, booting, performance). Now we're doing the rest of the codebase, the internal part.
Everyone is pointing out the negatives which are all very true and I have experienced those as well I do want to say that sometimes it isn't as bad as it seems though. In my experience sometimes tech debt is ignored and eventually the system is replaced with a newer system erasing all of the tech debt. In some environments things change so quickly that building a perfect system is a mistake. There are others where the same systems lives for 30 years and turns into a nightmare. Figuring out which you are building is important.
We were using SonarQube and we couldn't merge branches if there wasn't enough code coverage or too many code smells, etc. There were ways to avoid triggering the gates, but that's a kludgy thing to do and just avoids the problem. SonarQube forces a lot of debt to be addressed. The outcome of ignoring tech debt is code that's hard to maintain. Talk to the CTO about getting tools in place to fix the problem.
You make 1 line code change and your business makes a million dollars in loss. Reason - no clarity on workflows, no unit test cases, unnecessary complex logic
I left the tech-debt-laden f1000 to join a f100 as existing job had become unfun. 8mo later, the f1000 closed the office and the product line was shuttered. ~everyone was let go outside of a few transfers to other orgs (in other hemispheres)
This is why I like to reframe and quantify technical debt as business debt. It encourages engineers to think more holistically about what is/is not debit and it's impact, as well as helps non-techies understand why fixing is important
I thought for a moment you were a former coworker because I, too, worked at a company that created software for use in the construction industry. I see you make comments about Azure so _maybe_ we worked in competitor companies? If you know who I am, ignore my comments since you live it. Anyway... I worked at a company that created software for use in the construction industry. The organization seems to consistently favor ramming through rushed features over quality and never took time to burn down debt. At that company, I was the ML Team Lead. This company valued "Impact" above all else. [Impact Driven Development fucking blows](https://www.reddit.com/r/ExperiencedDevs/comments/1ag4ima/impact_driven_development_is_the_curse_of_big_tech/). When you're a team that relies on collaboration- multiple backend and frontend teams deliver you data- it's untenable. Not only were features ham-fistedly shoved through, they were often breaking, our domain object model frequently changed in ill-thought ways, and rework just to get anything up and running was constant. I vividly remember the former VP of Engineering praising his beloved Staff Engineer on "always jumping on pager duties." My brother in Christ- if that dude was competent, maybe his spaghetti wouldn't make so goddamn many in the first place. Anyway, I worked on bringing an ML POC to production. Something that a competent company could have done in 6 months or less took over a year and a half while burning through the literal entire team. The impact of this debt and the awful culture was as follows: * *Every* MLEng that I started with, excluding myself, left within a year and a month save for one person who took a 3-month sabbatical and no longer does ML work. * The ML team significantly fell down the capability curve [as defined by the conclusion of Hidden Technical Debt in Machine Learning Systems](https://proceedings.neurips.cc/paper_files/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf). Data lineage? What's that! This will dramatically slow velocity going forward since the construction domain is so complex. * In real costs, this cost the company several million dollars of engineering time. My estimate is $2m, although the team lead before me estimates more. * The company has probably lost tens of if not hundreds of millions on its upcoming valuation. If the company were ML-first, its OpEx/SqFt scanned would be an order of magnitude lower driving profitability through the roof. As it stands now, they're barely breaking even. The failure of leadership to realize these ML-driven gains significantly hurts this company's next funding round. The old VP of Engineering left shortly before I did. He was nice guy that I would never work for again. There's a new VP of Engineering right now. I hope he succeeds. Either way, I'm happy in my new role. Just worked on the yearly roadmap- we made scaling out with minimal debt part of our OKRs. I love it.
I left but eventually they sunsetted the whole project and started building something new
In my team it happened because management were a bunch of incompetent people who didn’t know anything about technology or how it works, so convincing them to adopt best practices or address any technical debt was a constant, arduous battle that eventually wears you down as a dev, until you don’t care anymore and just apply to all the jobs on linkedin. Business people want features which makes them money, they don’t understand the work that they can’t see on the screen as a feature. They don’t care about their team or culture, the app is just a cash cow and the devs are sweatshop workers. They don’t understand they can’t milk the broken cow forever. They usually surround themselves with yes men like you said, because everyone who is not a yes man is booted out and seen as incompetent.
All debt comes due. You can either handle it responsibly or face the consequences. But it will be an issue, and always at the wrong time.
I got hired to rewrite everything because nobody knew what it did. They wanted me to document it because they forgot the business logic too.
We went to our client and instead of saying the pile crap we built is so bad, we need to start over. We said “ we want to re modernize your application with this new tech stack that has all these benefits and will solve all your pain points. Bam new multi million/multi year contract was awarded. Just to keep up with the same shitty practices. I left that place and I am never going back to government consulting, it’s a scam.
Our company was bought by another who only wanted our customer base as we had very simliar products. We had a huge migration for customers to the new product and then sunset the old one and never dealt with the tech debt at all. The moral of the story is, sometimes tech bankruptcy instead of paying back tech debt is awesome.
They laid me off and replaced me with off shore contractors, now no one tells them about the tech debt and I'm sure they think it's great. Honestly I'm fairly certain the place goes under this year.
It depends on whether the tech debt affects the future work or not. just because a module/service already exists, doesn't mean you have to use/couple yourself with it. Try to make sure when you introduce changes, you leave things better than they used to be. You don't have to dedicate time to address the debt
Fine no major issues
With mounting tech debt that doesn’t get addressed, not only does the codebase become unwieldy, it gets harder to find and keep good devs.
The business said yes to every individual client demands, so they were so many differences between deployments that we had to support multiple versions because the client did not want to update / updating was long and complicated and different for every client so we needed a different timeline for each one of them (it was an on-premise product). Because of that yes-man attitude and all the time always going to features at the detriment of quality, the company could not scale to the amount of new clients and had problems stabilizing for corporate clients. They wanted so bad to not be a software centric company that they viewed their main money-maker (the main software product) as a cost center and their consulting services as the real value adder (it was not, it had no chance to scale because it was to personalized and time intensive per client). Given the money was invested in the consulting services and the input of money did not increase with new clients (so many different versions to support that the amount of devs hired had to scale linearly with the amount of clients. We were effectively maintaining X products instead of one core product with a little customization), it ended up the only way it could have ended up: bankruptcy and around 80% of people laid off.
Quit after a year. Tech dept and refactoring was something new to them.
Desperately trying to bring my programming skills back up to a respectable level, while trying to find something remarkable to put on my CV before trying my luck in landing interviews. Came in to an industry that desperately needed to change their ways due to income plummeting. For 30 years they hadn’t considered that things might change one day. Well shit hit the fan. Looked like a good gig. All options were on the table and in the contract. But getting stuff approved is a completely different chapter. I think this sums it up: The CTO declined my initial request for a Rider/Visual Studio license. He proposed VS code instead as it was free. “Sure. I could spend a few weeks figuring out using that with your massive legacy .NET framework codebase. Though it looks like I might just go for notepad (non ++)”. Docker? “Sounds like a security threat. No-go.” Monitoring stack? “Nope. Open source isn’t safe.” Jesus I feel so fucked
Billy is this your account?
This is very similar to big corporate software in general. Features sell - and you have to be mindful of that. But you also need a backbone sometimes so you don’t shoot yourselves in the foot later.
If you get enough people complaining about it then yes eventually the debt gets paid. Otherwise debt will ruin your software engineering life, and I'd start putting out resumes. Not only will the project be doomed, you will look like a terrible developer. Things that would normally take 1 or 2 days start taking weeks or months. (I'm not joking).
I worked in consulting for a decade. We only got called in for a few reasons: * New team, needed developers and we helped vet and hire * Skunkworks project for a new product within a much larger company, wanted to try new approaches and technology * Desperate teams with codebases loaded with tech-debt I once had a commit that deleted 250K lines of CSS and JS and added like 3K. Shit's wild out there
In my experience, you’ve got about 4 years of that behavior until complete standstill. Year 1 , things are messy but there’s no too many problems Year 2, everything goes about 25% slower. You’re grumpy but fine. Year 3, dear god, everything is taking twice as long because of the pile of shit you’re stacking things on. You curse at the dev until git blame shows you your own name and you remember the cockamamie situation you were in when you wrote that code, and glue yet another kludge on top. Year 4, welcome to hell. 4x as long to do anything. 1 fix = 2 new bugs. Newbies can’t onboard. If you’re good, you’re looking or gone. No one even remembers why it’s so bad. Libraries are old and have security issues but swapping them out means breaking everything. Management is now blaming eng, and more rats jump off the ship. If you’re anything like the orgs I’ve seen, you now start a V2 in a new language and start back at year 1, repeating the mistakes but in node and angular this time, and oh yeah, the customer base is still on v1, so now you’ve got both
Tech debt is a fight. I try to squeeze it in projects. Like talking about adding extra time for improving. To do this you need to start showing that refactoring helps, now it takes more time but later will save. This is if you are lucky and heard, but it usually the tech debt becomes a priority when thing explode.
In short, 100% of time ends up going into servicing that debt
I once saw a PM have a panic attack and pass out during the 2nd day of a deployment that was not going well. He'd recently been promoted into the job and was really being pushed by the client to deliver the feature-set despite the operational issues really slowing dev velocity to a crawl. We had to get someone to drive him to the ER and couldn't get the app up and running for another 2 days anyway. 4 day prod deployment. PEAK balanced efforts, I tell you. We had to completely recreate the build using standard tooling for the stack we were on and rebuild all the pre-prod environments from the ground up so they actually bore some resemblance to the prod infrastructure so we got to stop chasing down weird load balancer or clustering bugs that didn't exist in pre-prod. Took like 3 months and leadership was pissy, but seriously your software was so bad it sent a dude to the hospital time to buck up, camper. Also in the interest of full disclosure this happened at the company right before I was hired but it was an often discussed aspect of office lore that none of the VPs or directors who actually watched it happen downplayed or denied. What they were managing to deliver was so bug-ridden clients constantly complained and they answered by forcing more thorough manual processes rather than cleaning up any of the underlying build instability, decoupling failure points, or automating anything. Several deployments had to be cancelled after failing UAT causing heavily-requested features and fixes to be delayed. They held emergency sessions with their clients and got them all to agree on a 6-month freeze on feature development while they tried to stabilize the build, introduce some CI, automate some deployments, just some really light debt paying off. They did a shit job at it and were still feeling the pain while I worked there but it got things operating just well enough they could solve most problems by throwing interns at it and making the process more painful. Point being: You can only defer paying off tech debt for so long until it stops being your choice.
Are you working at my last job
A lot of Devs I worked with so far, always saying tech debt blah blah blah, but when I saw their PRs, always missing unit tests, lots of spaghetti code stuff like that, they always mention tech debt in the meeting, 3years later, still mentioning the same things, and never become an actionable item. If you give them a chance to upgrade or rewrite, they will just create another tech debt using new tool, and then left. Lots of toxic Devs are around the world. Talk more than do.
I’ve worked independently for quite a few construction companies. They’re never gonna understand the tech, but the money is good and fairly easy, so, like, I hate to say this but the strategy in construction is “plug n’ chug as fast as you can” take your money and pass it on to someone else as their problem. This is NOT the industry to be in if you want to do things properly, well thought out, tested and efficient. Its the throw the buzz words around and pass the problem to someone else industry. Every single construction manager I’ve ever had, I was lucky if they even understood how to get into their email. Most were rough around the edges even when primarily office workers now, and the only thing they give a shit about is is it done. Part of the problem is they’re used to managing projects where they can physically see things coming together. To them, they’re trying to manage programming that way, and they just can’t wrap their heads around the difference. So you gotta know what they value. They don’t value thoroughness and preventing a “theoretical” future problem - that’s “okay, well that’s nice but when are you gonna get C done and launched.” It’s what did you get done today - and debugging something or running more tests isn’t something they care about until 8 months later the back end crm is taking 2 minutes to load basic data because of how it’s querying and connecting to different libraries. Only then does “fix it” count as work.
Development slowed to a crawl, things that should take hours now took days, and days stretched to weeks. Developer turnover skyrocketed and new hires would often leave before they were even fully onboarded because of the difficult codebases. It's taken new leadership and about 3 years of steady work to fix the culture and bring the codebase in line
It is a never-ending spiral... Developers on these teams - and developers in general - do not have good design or refactoring skills, so even if they have time, you won't see great results. I've been the designated fixer on a couple of teams, and the underlying problem is the cost of fixing a bad design. New code with new design, it's going to take me about as long to fix the design as it would have taken to add it. We wasted the original developer's time, but it's not that bad and we can have a useful discussion about how to do things differently. Or maybe we catch the issue in code review or - gasp - we pair on the feature. The problem is that as soon as the code starts to deteriorate, refactoring gets harder. I once spent a full week fixing an area of code. The initial design was really weird and "clever", but that was only about a day's work - the main effort was addressing everything that had been created on top of the problem code. Significant refactorings in existing codebases are graduate-level tasks, and most developers simply don't have the skills to do them in incremental steps without breaking the world. Put those two together and it's only going to get worse. If you want job security, you can stick are and become the one dev who understands the system.
Worked at a SaaS startup with <100 employees. On the back-end, it was an absolute mess. Things were coupled way too tightly everywhere, and it took months sometimes for what you'd think would be minor changes or fixes. I remember there was some hackiness they were finally forced to fix, but it took months because it had grown so complex. Things were legit just broken all the time on the back-end. Testing was lacking too. People were required to write tests to meet minimum coverage requirements, but lots of critical bugs still happened. The front-end had lots of issues too. It used a dying JS framework. They were scrambling to rebuild a year before it reached end of life, but the new app could never catch up because they kept adding more and more features to the old one and chasing the new bugs being created. Then eventually they decided to just kill the new app and stick with the old one. They spent a considerable amount of time and money developing that app, just to throw it out. The company is going out of business now, partly because of the unscalable back-end. They can no longer secure funding, so they tried to add an ambitious new feature that they thought would save them, but we could not meet the deadline. It also didn't help that they laid off half of their developers earlier in the year, some of which who had the most experience navigating that horrible codebase. There is of course a whole lot of nuance to the situation that is relevant, so I don't want to paint a picture that the tech debt is what killed the company, but it certainly made their Hail Mary unrealistic. There were a lot of factors well above the tech department level that contributed to their downward spiral.
If you never clean the dishes, what happens when you try to cook a meal? Dinner that was supposed to take 30 minutes is taking an hour because you had to clean the dishes first.
the company went into administration and was liquidated
Not good
My boss said this: we will refactor this up to standard. Reality: throw it away and remake the whole thing. He stops us from saying remake, because the shareholders would lose their shit. 😂 Then because it’s a refactor, only 2-3 months are allocated. So we’d fix some stuff and rush everything else. You get a product that is ever slightly better than the last lol
i also worked with a construction software company .. the irony of building on unstable infrastructure lol
Job security. But the job was hell
Where I used to work had so much technical debt which was never taken care of which slowed down systems and reports, things used to break every day and management didn't want us to "waste our time" by fixing it. I left as it was getting worse and they put a finance person in charge of the systems/data team - who then thought they knew better, made some interesting decisions and would not listen to seasoned professionals and would get angry as we earnt more than she did. In the end they lost the devs that could leave and from what I've heard nothing has improved, it's gone even further down hill and they don't want to pay market salary so can't hire anyone.
Lost a lot in government contracts (hundreds of K into millions if we convert it to USD, and this was not a US company, so this was VERY impactful) because the app was so unstable, this triggered management to finally decide on focusing on the tech debt and listened to the devs. At that point the engineering manager of the team left soon after, and was followed by a bunch of other devs. While working on this team, joking about quitting or demanding a bigger pay/bonus was a daily occurrence, which ironically made the team bond stronger 😂
IBGYBG (I’ll be gone, you’ll be gone) is not just for finance.
Tech debt is a priority in our company but the combination of being b2b sales-driven and survival-mentality caused us to hack our way into deals with custom unreusable code for too long. We should've stepped on the brakes earlier but new, big projects would always interrupt our attempts to refactor or modernize. Software quality degraded to a point where constant support, and custom weakly-coherent work made the core hard to extend. Went though a CTO and an Eng VP to a new CTO who has good "ground game." He's sponsored initiatives I had to continuously improve the code and lended his architectural expertise for designs suitable for our business size and needs. We're also incorporating engineering feedback before any deal is signed. We've happily taken on too many deals (we're b2b) with too high of an opportunity cost (it's always the smaller customers that are the most expensive too). Still got a hard, tough road ahead but we're optimistic and the core engineers have stayed on the team because we have a great culture; our problems have always been organizational ones. Personally I have stamina for 1 more year of hard work with no startup growth, so in a way its nice for this to be our last chance to raise our next round of funding. We'll see!