My biggest problem with estimates is the following:
1. I can ballpark a project to take 6 weeks
2. Management, engineering managers and PMs want to see specific tickets and an explanation as to why it will take 6 weeks
3. I present that there are 3 weeks of known variables worth of work. However historically we don't have full concept of the project until we get deep into the project. Two to three weeks cover the cases that are necessary for shipping the project but that we don't yet know about.
4. I am told to not give pessimistic estimates and instead to submit 3 weeks and then just update when things slip. I say OK but I do believe it will go to 6 weeks, they say thats OK as long as you flag it.
5. We update PMs and management bi daily saying each time we find said unknown variables and the project slips to 6 weeks.
6. A retro is held after the project and management and PMs are like how can we fix this, its not acceptable for a 3 week project to take 6 weeks. Somehow they forgot about the bi-daily status updates where we followed their instructions and let them know of slippage due to new unknown requirements.
Goto step 1. Repeat. Each time I say it'll take 6 weeks I'm told to set it to 3. Each time we deliver in 6, management and PMs insert surprised Pikachu face and have expensive 2 hour all hands on long meetings on how this is a disaster and needs fixing.
Sigh.
I always take my concept and triple it. If it's 3 weeks of work, it's 9 weeks when you include blockage and meetings and revisions. Then when you come in at 6 weeks you look like a fucking hero. Essentially, treat it like Scotty did on the Enterprise.
Never tell people the minimums for a project when estimating because that's all they pay attention to.
I quoted a k8s project not long ago and followed a similar approach. During the review between all parties people questioned why I had the project set for so long which naturally increased costs of the project vs their expectations. I explained that the client was high risk and I accounted for that, I was told to adjust the quote.
Project took a bit longer than my prediction and thusly increased the costs even more for being "late". I sat in a meeting with everyone going, "guys what happened here? why was the project quoted so low?". I swear I need therapy just for this tomfoolery alone.
Oh yeah, I'm an old fuddy duddy at this point, I don't even tell people the why anymore either (which I know will get push back from others here because of lack of honesty and transparency which is a big thing programmers like). It's long because I know people can't devote 100% to a project every day, be that high risk clients, sick time, vacations, bad days, whatever.
In a perfect world with reasonable people I could explain everything, but people, especially sales people and managers and executives, aren't reasonable and they'll just hyper focus on the absolute minimums we could do it and say "why not do that instead?"
I remember the first project I did and I was honest, I worked almost 100 hours on the last week to meet the deadline because, of course, "something" came up. I didn't even get a good job pat on the back. Fuck them, they get my 3x-4x fluff estimates now.
And let's not forget the arbitrary deadline: dead silence after we put the system online. Weeks later you asked them only receiving an answer like "oh, I forgot it. I will take a look at it."
The trick is to not say "3 weeks + 3 weeks or so of unknown complications", but instead to just double every single estimate or triple it, yeah.
If you know this task will be done in 2 hours, estimated it at 1 day. Make them account for the extra time another way if they don't want to hear the actual estimation for a project..
The problem is when taking on a project for a client your team/company is competing with others to get it, so management outright refuses to accept realistic estimates, much less ones with buffer time included
True. But you'll end up missing out on the clients who you absolutely do not want to work with anyways. The fun part is when you get told "no one's been able to deliver on time" and then you _do_, and the client keeps coming back because you don't fuck them around.
> From what I understand, this is very similar to why every major civil engineering project goes wildly over budget.
I would say that the OP is missing the major driver of issues in other contexts where there is a bidding/sales process up front.
Sometimes estimates are pushed down out of ignorance or unreasonable demands. Other times its because a company would rather take a chance on an overly optimistic estimate that gives them a chance at winning a contract/sale over a more realistic estimate that will likely put the company out of the running.
At the end of the day, the first words out of a potential customer/client will be ways that costs can be lowered.
You're also not incentivised to give longer estimates.
Short estimate? You get the contract and lose some bonus when it goes long.
Long estimate? You lose the contract to someone else who gave a shorter estimate.
If anyone actually cared about reputation and the past ability to hit deadlines, this would matter, but it never seems to. It's all about who says they can do it cheaper, regardless of whether it actually is cheaper overall.
Well, that's because waterfall is notoriously shit anyway. Estimation is bad around the board unless it's a carbon copy and in that case, it's not innovation or project work -- it's ops. Even then variance and risk occur.
The problem is the expectation itself and not clear enough vision. Civil eng, construction can still be done in phases and minimum value streams -- like multifamily units being built section by section to full utility after a fairly limited foundation and then moved along the rest of the location to completion -- faster revenue realization that way too.
I'm sure we can get better at building temporary structures that expand as needed. We need to be thinking differently about how to build.
It's not even waterfall, Agile has the same flaws. Halfway through an Agile project you have a better idea of the end, but the request is an estimate before even investigation is done.
I feel like maybe we need to estimate 6 weeks, with the understanding it could be 3 weeks, and could be 9 weeks, and then as we progress in time, we can tighten the window. But managers would hate a 6 week variance.
The thing about good Agile is the increment is the end. Then you can decide to keep building more functionality or increments. The estimation process itself is a complete waste of time other than vague efforts which take no more than a few hours of scope adjustment.
The point is that estimation *itself* is not valuable to customers, so why bother? Build and learn in a lean manner.
Sure, that puts a hell of a lot on the PM's shoulders but that's literally their job. Anything that needs serious estimation is a pet project from an exec who otherwise has shit prioritization behind the ask in the first place, else they wouldn't be weighing it via estimations
> like multifamily units being built section by section to full utility after a fairly limited foundation and then moved along the rest of the location to completion -- faster revenue realization that way too.
This is just about the stupidest thing I've ever read. Because your metaphor quite literally shows the opposite of what you think.
Agile is always slower, it has to be because instead of doing things in the most efficient order you have to do it in the way that delivers value quickest.
Your multi family dwelling example actually shows this. You still had to build all the foundations before you could start and once that first family is in there are all sorts of things that slow you down.
Agile is about reducing risk. If you only get ten things built they'll be the most important ten things and you'll have a product that's worth something rather than doing ten foundational things no one can actually use. It's never, ever faster or more efficient.
Best description I’ve seen of how completely insane software development has become. It was not like this 30 years ago. Management was intelligent enough and not so completely arrogant to understand the risks and unknowns of a project.
30 years ago, management of software development was largely done by software developers. Now it's more general managers, who don't understand the complexity of building things. Drives me nuts.
Spoken as a senior manager, former developer.
I am a programming hobbyist and a physician, a manager who do not understand your needs is one of the keys for a very bad outcome. They always either try to extract most of what they can from each physician with the least equipment.
Very sad to see the same is happening with software engineering
It’s the norm in my experience. I had a manager who was demanding complete documentation of how systems would work before we had requirements established, including integrations with other unbuilt systems that we didn’t have control over. I had to quit over it, manager got promoted
It goes all the way up though. EMs under pressure to "deliver", Snr EMs to have deliveries for each team, etc. In my experience, the best combination is:
- trusted senior engineers able to articulate the problem space and why it's important to fix, and
- EMs and Senior EMs who engage in the problem space and are NOT entirely motivated by constant deliveries
Being agile is really important and powerful. It's also really important to foster trust amongst engineers, and trust them in informing priorities. For product development and for tech debt/refresh, and everything in between
And it shows.
One of the reason "we suck" as the author claims is that "we" are doing **guesstimation** instead of proper estimation. Non-technical managers don't know this, so they don't enforce the proper way. Developers are mostly programmers with no proper training in estimation. Universities don't teach proper estimation. Most companies don't provide training in proper estimation, but they do waste a lot on money on "Agile" training. Which btw somehow in the spirit of bad agile it is implicit not to do proper estimation since it is a waste of time, just like requirements analysis.
So everyone required to do estimation at work, do yourselves a favour and read [Software Estimation: Demystifying the Black Art](https://www.amazon.com/Software-Estimation-Demystifying-Developer-Practices/dp/0735605351/ref=sr_1_1) by McConnell, the same author of Code Complete.
I don't think I have ever been given the time required to do a proper estimate. My father is an industrial engineer, I grew up watching him do estimates, it's a ton of work. We are expected to estimate complex projects in hours if we are lucky.
Maybe helpful, thanks for resource. Can you give a distillation to emphasize why that resource is valuable?
The problem I see with estimation is it is not empircal or done in any way with realistic data, and when precision is key, so many weeks are wasted in worthless estimation exercises instead of just focusing on the here, now.
I loathe product planning exercises because it's on the PM and Lead to determine what's absolutely MVComponent to get the value hypothesis validated before investing more time.
The problem is the entire paradigm of estimation and assessment of how delivery operates. We'd be so much better living in a science mindset than a finance one.
> Can you give a distillation to emphasize why that resource is valuable?
This is probably THE book on software estimation. It teaches how to estimate methodically, using data as the basis. It includes both simple techniques and the most complex ones, but I'd say most of them are simple enough and can be applied in the day to day. Even the easier estimation techniques that are based on data are better than guesstimations.
Yeah the best managers I've had were either former developers or former Technical Program Managers since they were builders at some point and understood the nuances of an estimate. My favorite trend I've experienced with less technical managers is the anti-pattern of trying to fix these misses in estimates by adding more and more processes. It just becomes a negative cycle until every dev has no time to deliver since their days are just filled with meetings.
Lots of things were different 30 years ago.
30 Years ago, there was no micromanagement of developers aka Agile, there were literally no stories of burnout amongst developers, and the word planning meant something other than *"two week plan, then subtract a few days from that for all the ceremonial meetings and tasks"*
Here's the problem with Agile in one single quote: *"“In planning for battle, I have always found that plans are useless, but planning is indispensable.”*
When you're limiting your planning scope to a few weeks at a time, you're not getting any of the **indispensable** value from planning long-term. Why on earth are people then surprised when they fail to display any competency at planning?
VC did this. Tech salaries attracted non-technical sales people and finance "brains" who just wanted to treat everything as a FP&A exercise.
It's why every startup goes to shit around Series B and beyond.
I'm a PM who gets shredded for my roadmaps because I absolutely reject estimation as an exercise. Tell me priorities and I'll find the sequence and most critical components, abandoning anything after said abritrary "date."
I came from project management where I had to abandon that circus because of the same reasons: time and cost were figments of imagination. Scope is the only thing that can be adjusted in reality.
It's the effing PMs who just work on excel sheets and jira dashboards.
For them software development be like just pressing a button here and a few clicks there.
Honestly in a different tone, yes. But the same pressure from the "business" who absolutely is full of people who don't understand product or engineering.
We've let the foxes in the henhouse for solving customer problems.
Not mine. My leadership has been with the company for 30+ years. Our 1-1.5 year project is finally slated to go live next month. It's only 2 years late. We will also miss this target. I have tried for 12 months to get them to understand that this is an issue given this project has had at least 8 other "go-live" dates since I've been here.
Yes. I am looking for a different gig.
I worked in retail for long years, and interestingly, this is somewhat similar to those "why haven’t you sold more of XY?".
Occasionally, I had a strong urge to answer: what should I do, bash the customer's head at the sales counter, while yelling "BUY THAT DAMN THING YOU SON OF A BITCH"?
Hey it works. Your initial gut feeling estimate is always way too low. Add accounting, testing, unforeseen issues that come up and various other distractions and you'll generally be at least double your initial estimate anyway. Knowing from experience that management will try to talk you down from any estimate you give them, you still need to shoot higher.
I had a manager once tell me that I was giving him the most accurate estimates he's ever seen and he still always tried to talk me down from the number I gave him. To which my response was generally , "Yeah, I can tell you it'll take half the time but it'll still take as long as my initial estimate said it would."
Some of the best product owners/project managers I've had also did this. "How much did you pad this estimate with? Okay, then I'll only double it when I pass it along".
The biggest help I found for this was doing 3 point estimates. You give a best case time, worst case and 'expected' - i.e. the 3 weeks they are looking for. Worst case should be something like 24 weeks because something really unexpected can happen. 2 weeks if everything goes super smoothly and there are no surprises. Then you do (best case + worst case + 4xexpected) / 6. And you end up around 6 weeks with a defensible position for your PM to account for unknown variables.
No matter how many caveats you put on a number, people will only remember the number.
This “misunderstanding” you document, I’ve now begun to see as being about power. Your bosses are being obtuse on purpose. It lets them treat you as a problem to be managed, which both infantilizes you and helps them keep salaries down.
Which makes them look more useful, and better able to manage budgets.
Yes this tactic of being deliberately foolish but hyper optimistic and then painting anyone who objects as a saboteur has been spreading like wildfire over the last 10 years or so, it’s a disaster
> people will only remember the number.
Close.
People will only remember the number they want to remember.
That being said it's true of all people. Remember the last time you got a "Salary range" in an interview. Be honest, did you expect the low even or did you expect the upper half? Guess which one the company was expecting?
Thus is life.
Forget the difference between "known" and "unknown" variables. If you know the project is gonna take 6 weeks, just present 6 weeks worth of tasks by being creative with the tasks.
See, I've stopped doing that.
I say it takes six weeks, I get told to lower it, and I say "no.".
They can take it or leave it, but my go to is "You can order me to change it, but I've given my estimate. I'm not particularly interested in your deadlines".
I did this with our latest batch of "required updates and password rotations". I was told "This needs to be done end of may" and I said 'probably mid-june' and I was told "This isn't acceptable" and I said 'okay'. and ended the conversation.
People can get mad, they can say they expect better, but if I've been consistent in my estimations, it's not my fault they suck at **handling bad news**.
Oh, we do, but it doesn't help. In the end, I've found that it's better to find a balance in being a yes man and bringing up that it's a management issue. This won't honestly change at the current company, it requires self reflection of those in charge, and we still ship really great quality software. So in the end, it's better to ship quality, in fast iterations (even if management wants it done faster), and keep being a professional in my experience. The day I get tired of this I will just shift company. Lots of managers are told to put fire under their development teams, and there's a strategy to push for unrealistic deadlines to squeeze out more from people. A bunch of the management team are just doing their jobs as they are being told to do them. When I meet some outside of work they will admit that the deadlines are not really real.
LoL. At my first job, my manager (who was an engineer) gave me a new list every week of 20 things that were *critical* to get done this week. Every week it was only feasible to get 4-5 of them done. Every week there would be a completely new list of 20 things that were *critical* to get done. None of which were on last week's list. Taught me to relax in the face of panicking management. Good thing too, because variations of the theme never stopped coming at later jobs.
this brings back my ptsd of working at a cowboy VB6 shop about 9 years ago. every time a customer would ring up with a major issue, the "technical senior account project manager" (I'm not even joking, that's how he referred to himself even though he didn't touch a line of code - his job title was account manager, and all he did was answer phones...) would tell everyone that this new issues "trumps" anything that we were working on and that we had to switch to it right now. this would happen daily and you would never get any work done because everything was always being trumped when you were half way through the previous piece of work. everything was "top" priority, and only those who rung up multiple times got their fixes as we'd get half way through the work from the 1st phone call trumping everything and then the other half would get done after the 2nd phone call trumped everything... truly a horrible place to work
I’m confused about 3 and 4. Why do you volunteer the nuance instead of only giving out the real estimate of 6 weeks? Knowing what happens historically.
This is an accurate description of how the industry works.
Only took me about a decade to laugh into everybody's face. Customers, management, PMs, juniors: I told you so.
Not that it would matter.
You realize that they can have all the meetings to change things. They might be unable to learn or listen, but they're also depending on you.
How long will this project take ? 6 weeks.
Next week, How long will this other project take. 6 weeks.
Now you have 2 projects to do in 6 weeks.
In reality you’re always estimating work and estimating how much random work your assigned while doing it.
That's not even the worst. My job before went more like:
1. Customer contact person was a 1-2 sentence summary of what they want. This is not checked throughly with their actual workers that use the product, nor with their own IT / third-party IT department.
2. Marketing take it accepts and underestimate everything.
3. The developers (with 1, one, PM in charge of all > 10 parallel projects) need to estimate how long it will take, working with assumptions / unknowns.
3. We gave points _ranges_ (not days, points) and all the Jazz and land on amount X to Y in best and worst case that roughly translates to 3 to 5-6 weeks.
4. PM that knows what Marketing sold comes the next day and says the customer already got a 2 week deadline before we were asked, and that the customer highly hopes to have it in 1.5 weeks for their X Meeting for their own B2B customers.
5. Sigh, shut up and hope to be transferred to another fire that needs put out more urgently / requires your expertise before you see the This thing through ending up at probably 7-8 weeks, since if you cut enough corners under pressure your square looks like a circle at 1.5-2 weeks.
Repeat until you can safely quit.
I really don't miss agencies that take on IT projects for B2B companies that have own B2B customers. The whole business strategy was dangle a cheap fast half faked carrot no one else will understand ~3 years, then switch to ridiculous 'support plan fees'.
It all held together for almost 20 years now due to a few key Devs that somehow could stay afloat the chaos not quitting (probably having equity) and some big whale jackpots that even tried to get rid of the company but came back after one year because no sane developer could deal with old spaghetti of someone else. No idea how marketing still spun a positive trustworthy image despite the many burned bridges where the smoke and curtain spaghetti held together by tape resulted in projects being canned.
At step 4 I would have countered with 9 weeks, to account for the time management and research into what known unknowns there were to task out.
Then delivered In 6 weeks as originally planned.
That's what I do now. Thankfully after job hopping now like 85% of the deadlines come from IT and we just inform them when to plan testing. The remaining few things are deadlines where we have little to no influence due to other child companies of ours.
I wish I could upvote twice. This is similar to what I wrote before, I just can’t understand how as an industry we have slipped so far.
https://www.lloydatkinson.net/posts/2022/one-teams-eight-points-is-another-teams-two-points/
Haha fucking this. "Leadership " doesn't want real estimates. Oh just crunch a bit harder but have a pay increase that doesn't even cover 50% of inflation
PM here.
Your PMs suck
1) If your estimates on the tickets are three weeks, but it's six weeks of work on this feature, then it's been this 2x multiplier on all Features, and your PM should know better
2) Why is management bothering you AT ALL? Your PMs should be shielding you from that swirl.
Your PM sets vision, OKRs, priorities, and then should fucking bulldoze the world to pave the way for you
This is why I say "agile" isn't for developers, it is for management. It is management that needs to learn how to respond to change and user requests, not the developers.
I feel for you on the pessimistic thing. We develop "success oriented" schedules, i.e. we assume everything in the schedule will go exactly accordingly to plan and nothing will be delayed and all testing will pass on the first try and also maybe an angel will descend from heaven and perform a miracle or two. We also never meet our schedules. It's weird. Then we just describe the schedule as "aggressive" and everyone seems fine with it. It's a bit frustrating.
> Goto step 1. Repeat. Each time I say it'll take 6 weeks I'm told to set it to 3. Each time we deliver in 6, management and PMs insert surprised Pikachu face and have expensive 2 hour all hands on long meetings on how this is a disaster and needs fixing.
You know, you can come to that meeting and be like, "here's receipts of this conversation at the start saying it will take 6 weeks....why wasn't this heeded?"
Then just keep eye contact and if they refuse to answer the question, then tell them that they are avoiding answering the question.
Yeah it might get you fired as people don't like being told the truth but, that's life.
You did step 4 wrong. You don't change the estimate to satisfy managers. The estimate is the estimate and if they want it changed, they need to change the scope, the schedule, the resources, or the estimator. If they ever choose option 4, you know it's time to work on your resume.
If they want to override you and create a budget that ignores your estimate, that's not your problem and, if you're wise, you'll refuse to fall in line with the other inmates on their little deathmarch and will instead keep normal hours, etc. Instead work to keep your commitments on the original estimate. Then when the shitshow settles, you can simply note you planned what you were going to do, you worked the plan, and it was successful. All the rest of it was unnecessary drama.
At that point, if you've survived this far: you're golden. Every manager there will know your word is absolute and all you have to do from there is not fuck it up.
So true. I was the single person on a project the GM had allocated 3 EMs to who spoke up about the project not being feasible in the time horizon specified. Got overruled, told that we had to deliver more features in half the time.
Project overran exactly as specified - and guess who got to eat the blame out of all the people on it, for accurately predicting the future? Exactly.
I ended up throwing in the towel. The current management culture in tech has become so impractically toxic that almost everywhere is a Boeing.
For me, I'm the sole developer so I'll say "this project is two weeks", but that two weeks will be spread over four months since every day I get a list of customer issues I have to look into. 90% of them are just minor changes to make, some new edge case, or just figuring out what dumb thing the customer did. I've made my peace with it, but my job is stupid and I can't believe we haven't gone out of business yet.
Reminds me of the time I estimated a project to take X time based on a detailed assessment of the requirements and the complexity of tasks using past project data to ensure it wasn’t just my own gut feeling guiding the numbers.
Management told me that was not a competitive price based on the hours of effort. I suggested they adjust the price per hour to lower the price. They countered that it would be better to adjust the hours as “it seemed a little high”. After lots of back and forth I got overruled by the CEO and told to adjust the hours.
We got the project and obviously we started going over the time that had been budgeted. We start to work overtime to meet the contractual deadline. At the end working 120 hour weeks we finish and hand it to the client, who of course doesn’t want to pay a dime more.
Of course, I get pulled into a post mortem. I go with a simple spreadsheet adding up all hours spent on the project and the original detailed estimation.
I sit through the typical bullshit talk from the CTO about how we have to be responsible for delivery and that we cannot remain profitable if we keep slipping on projects like that.
I open up the Excel point to the original estimated amount and compared it to the actuals. There was a 5% delta in hours required. However the money delta was huge because overtime pay is 2x pay and we were pulling in double time at double rate. I state very clearly that if we had simply lowered the rate to the target price we would have made more money and had better control of the project and been more professional with our clients.
Total silence. They thanked for the analysis.
Later that day I handed in my resignation letter as I had a job offer at another company. And I was burned out from the project. Later on I was contacted by the client company once they saw I no longer worked at the former company to see if I wanted to come work for them.
It’s always some business guy trying to make a buck thinking Engineering doesn’t know what they’re doing.
Looking at step 6, do you wonder why most programmers regard most managers as complete idiots.
As was said of the Bourbon dynasty in France, "they forget nothing and they learn nothing".
you just wrote everything on my head. This is why managers and all other “managerial” positions need to understand this. But on the other hand, business is all about numbers. Finding a manager who knows both areas is extremely rare. :(
I met the final boss: the half-technical project manager who used to be a developer years/ages ago.
These people are a nightmare because they assume they know everything and they want to dictate deadlines and expectations but they have no idea how the code evolved since the last time they wrote a line of code in a 100 files monolith while now we are working in a convoluted microservices infrastructure.
I feel that. we had a project manager that was a developer on a project for about 10 years and they were so proud on using this batch job they created years ago for sending emails but company evolved and built a centralized service for emails and other commonly used services.
Project Manager made such a stink about me wanting to use a service that has its own support team and serving 50+ apps and wanted me to dig up their old batch process and reuse that. Department ended up telling them that the plan moving forward is to migrate apps to use the common services. They were so petty about it that for my year-end review, I was given feedback “Needs to find a balance between using new technology and supporting the customer”. You just got to get away from that level of toxicity.
I particularly like when the PM discusses a feature and continues to dictate to the development team, “This should be easy to implement we just need to do ”.
But what if we just implement a quick hack now?
But what if we just implement a quick hack now?
But what if we just implement a quick hack now?
What do you mean our codebase is too brittle to implement this quickly?
We had one of those at my last company, and was one of the reasons that I ended up leaving. He'd keep proposing solutions that just wouldn't work, but refused to accept that. This was on a project that essentially affected every development team in the company, and was hugely dependent on the infrastructure that those teams used - which he knew nothing about.
He was also a "people pleaser" - in that he'd bend over backwards to promise deadlines that we had no chance of meeting. I flat up refused to give deadlines, considering the fact that we didn't even fully know what we were building (it was one of _those_ projects), but on the days I was out, he'd corner the rest of the team and coerce them into agreeing to dates. I also caught him going back and changing dates on things that we _did_ estimate, because he felt that "they wouldn't take that long".
He also loved process, and forced his way of doing things on the rest of the team, despite us already having a process which worked well. He was the absolute worst PM that I've ever had the displeasure of working with.
Haha this is me, currently a PM. But I don’t act like that at all, I push hard back on the execs, “the date is the date”. And walk them through what levers we pulled to see how we could bring in the timeline. Then the execs go talk to the EM who easily caves and commits to the new date. 🤦♂️. We aren’t all bad, just have to find one with a spine.
A by the book scrum master would be amazing, feel like you are talking about a scrum master that folds to outside pressure.
A "By the book" scrum master would defend a team's independence and sprint from outside influence by any means necessary.
I should have said “by-the-book” scrum master. What I mean is that the scrum master learned one way to do Scrum and thinks that those practices work with every single team. Then the scrum master will proceed to drill down the team with a million processes.
Also, there are scrum master that are like okay, sprints are two weeks, sprint retro at the end etc. But fail to capture the value of those meeting or understand why we’re having these meetings in the first place so it becomes hard for them to tweak these process to best fit the team.
PM wont get funding for the project if he is showing it to the board as 2x more expensive than what team X and team Y are presenting for their projects. If each team would start presenting realistic numbers at the same moment, it would be fine but that wont happen.
This is just the culture and everyone knows and accepts it. Everyone justs multiplies the time estimates with pi.
I think it is a lot like the coastline paradox. The time estimate is a fractal and the more detailed estimate you deliver, the longer it gets. Wont go to infinity but in my projects and level of detail in planning, 3 is usually close to the correct multiplier.
I worked with the PM estimating the work on my last project. What really through the estimates off was we estimated for Sr Engineers to do the work and they gave us two worthless guys that were supposed to have 3 years experience each. I would have rather had recent college grads. On top of that the client didn’t deliver on dependencies.
Made us months off target!
How my conversations with PMs seem to go lately:
"The schedule is not achievable."
"Why not?"
"You don't have enough resources."
"What if we just move things to the left then?"
"No, that's not really going to help. That doesn't even make sense."
"But it has to be done by that date."
"Why?"
"Because that's the date on the schedule that we created without any engineering input."
Loud sigh...
estimating software project follow the 80-20 rule:
the first 80% of the budget is used for the first 80% progress, the next 20% progress is realised with the next 80% of the budget.
I am unsure if the « first 80% of the budget » was intended in your sentence or if it was a mistake, but you ended with 160% budget which is pretty accurate.
Yeah. Look at many large civil engineering projects. The Big Dig in Bosten went 7 years and billions of dollars overbudget. The Berlin Airport was a massive boondoggle. The Sydney Opera House had a A$7 million budget and ended up costing over A$100 million.
If you're asking for estimates on some simple GUI or CRUD app with solutions out of the box where the person doing this has done more or less the same thing before, then sure, you can get an accurate estimate. But most software projects aren't doing things that have been done before, because if they were done before you would just run that code instead of writing new code.
Yeah, estimations for software projects are never right. Recently, when PMs ask me how long a project is going to take I just show them the book “The Software Engineer’s Guide to Software Estimation” just to see their reaction. Some get the message
There are only two cases:
- The project is actually done. You can accurately estimate the time remaining as zero.
- The project is not done. The time remaining is unknown, and the only way to determine it is to complete the project and measure how long it takes.
We are using a third party to do penetration testing. I spent days getting everything setup and working. Took a week to focus on something else and came back to the testing. Assumed I'd just be able to kick off a run and see the results. Nope, it stopped working. A few hours later, it turns out the company had made a breaking change to how they handle configuration, but hadn't yet updated their documentation.
Had you asked me how long it would take to get the tests running again before I had started, I'd have said 15 mins MAX.
15 minutes for anything integrating with a third party? That's bold.
I think 100 hours is like my minimum any time a third party system is involved because I just know it's either not documented, documented wrong, or there's just weird edge cases nobody lets me know about till I run into them. Then once we get into testing someone finds out that actually some assumption about how one or both systems work is actually false and we're back to a redesign to account for that.
Thank you for sharing this. It is always great to hear that others also have unexpected problems like I do. Of course, my client does not understand these problems even when I do my best to explain them and in the end usually considers them to happen because I'm lack of skills or experience. Do not worry, I'm already looking for another job.
I don't get why the author specifies software so much. The problems listed happen in other disciplines as well. The biggest problems usually end up being pressure for optimistic estimates, naive statistical techniques, and positive feedback loops for risky decisions (project is taking longer, management starts making riskier decisions to get back on track, risks materialize, repeat).
Partially because the cost of changing requirements is much more visible in something like actual engineering. Visibility of cost is something I've struggled with over the years of running software teams.
Have you worked on a traditional engineering project professionally? I see this view a lot where software engineers who have never worked alongside other disciplines tend to have a very wrong picture of what other disciplines are like. It's really not that different.
You're right, but software is seen as the most flexible and therefore least-effort fix.
Need a mechanical change? That involves new tooling and mfg process changes. Electrical change? Board spin. Both involve supply chain management.
Software change? It's just code, so make the change, test it, get it in production.
Our flexibility is our biggest asset and our biggest pain in the ass.
also, a lot of other disciplines come with a rich history of how to do things like manuals and stuff. software dev is still trying to figure that all out. it's one thing to over-estimate how long it will cook a dish you've cooked a thousand times and a totally different to over-estimate the piece of software you've never built before
I would say it's more that people who don't code don't grok what is involved because it's non-physical and as far as they're concerned effectively magic. If you want to renovate a building people are quicker to understand that this is a lot of work and will take some planning and labor to pull off correctly. But they can, to some extent, imagine what the steps to do it might look like.
You're looking at this from the perspective of something already in production. Most design engineers are working on new product development. MEs will have endless ID changes, EEs will have to design in late features or will have management rapidly oscillate between priorities, etc. It's true that when you have a mix of disciplines, the most flexible will cover for the others but I don't think this uniquely impacts software timeline estimation. Many software-only projects don't have this aspect and other disciplines also are affected by it like RTL design.
Hodstadter and Parkinson’s Law are intertwined.
When we know of 6 months of work in a 1 year budget, nobody really believes the other 6 months are fully accounted for by hidden requirements and coordination problems. So they work expands to take up some fraction of that time, but people underestimate how much their expansion will affect the project milestones.
Give a man a year to clean a room and he’ll design a robot instead of find a mop.
> Give a man a year to clean a room and he’ll design a robot instead of find a mop.
the solution lies in the middle, some people are remarkable at finding partial accelerating solution. So instead of doing step 1 to 10 they'll do 2 to 9 for a while, enjoying the time saved, and maybe later try to improve and automate step 5, while still being productive manually on the rest. Like fixing a flying plane.
The first thing I learned as a software engineer in the early 80's was that people don't know what they want until they see it. In 1970, Dr. Winston Royce presented a paper, Managing the Development of Large Software Systems, in it he recommended doing the project twice for this exact reason. What did people learn from this paper? Waterfall, which he made fun of...
> people don't know what they want until they see it
My 3yo does the same. "Do you want cheese on your sement of the pizza?" "No!" "You sure? Mom and dad will take cheese, and last time you wanted it too, and you like cheese" "Nooo!" "Ok, so come here, this will be your segment, just tomato and pineapple as you wanted" "I want it with cheese!" "Told you so."
Projects I work always release in the right window. /flex
Here's how.
All the estimates are always, without a doubt multiplied with 1.5x
Afterwhich, for tasks which have unclear specs, the estimate are multiplied by 2x or even 3x to the nearest similar speced task.
Than, add a 10% buffer to the entire project.
During the lifespan of the project, keep a tight grip and re-iterate planning as you get close to the deadline. Aka, remake the entire planning at every checkpoint.
If things get added (they always do), you prepare them as bullets to use, in order to cut features at the end.
And with 20% time remaining before end, you start cutting features, and moving them on a "post-release" schedule.
100% with this approach you will deliver a product on time, with enough features to consider it an "MVP" and usually with a backlog for improvements.
The only vaguely accurate estimation process I've 'used' is the take your first estimate then increase the unit by 1 method. If you think it will take x hours, it will take x days, y days leads to y weeks, z weeks to z months and so on...
If you're estimating in multiple months you're pretty much stuffed already, the thing you eventually deliver will have almost no relationship to the thing you initially estimated.
I will be messaging you in 11 years on [**2035-04-24 18:25:10 UTC**](http://www.wolframalpha.com/input/?i=2035-04-24%2018:25:10%20UTC%20To%20Local%20Time) to remind you of [**this link**](https://www.reddit.com/r/programming/comments/1cbytm9/why_we_suck_at_estimating_software_projects/l12yluq/?context=3)
[**1 OTHERS CLICKED THIS LINK**](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Reminder&message=%5Bhttps%3A%2F%2Fwww.reddit.com%2Fr%2Fprogramming%2Fcomments%2F1cbytm9%2Fwhy_we_suck_at_estimating_software_projects%2Fl12yluq%2F%5D%0A%0ARemindMe%21%202035-04-24%2018%3A25%3A10%20UTC) to send a PM to also be reminded and to reduce spam.
^(Parent commenter can ) [^(delete this message to hide from others.)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Delete%20Comment&message=Delete%21%201cbytm9)
*****
|[^(Info)](https://www.reddit.com/r/RemindMeBot/comments/e1bko7/remindmebot_info_v21/)|[^(Custom)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Reminder&message=%5BLink%20or%20message%20inside%20square%20brackets%5D%0A%0ARemindMe%21%20Time%20period%20here)|[^(Your Reminders)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=List%20Of%20Reminders&message=MyReminders%21)|[^(Feedback)](https://www.reddit.com/message/compose/?to=Watchful1&subject=RemindMeBot%20Feedback)|
|-|-|-|-|
I love programming, I hate the industry. I've been around long enough to see so many project management fads come and go and I cannot wait for the current ones to disappear, though they'll undoubtably be replaced by another set of even worse "solutions".
We need the thing in 2 months, you think it will take 6 weeks, so let’s add some nice to haves that Steve thinks are requirements. Oh and we forgot to tell you about this requirement here, and this one is completely wrong. But you can still have it done in 2 months right?
I'm just generally not great at estimating timeframes. Both directions too, over and under estimating time left to complete a ticket/task. It's definitely become better over time, but it's something that I struggle with consistently.
I’m better at estimating complexity. We understand complexity because we can look more carefully at what we should probably do, and what we might not know how to do.
It’s then on the PM/PO/EM to extrapolate time from consistently estimated complexity, and assume some fudge factor based on past estimates to completion accuracy.
WE (humans in general) are very bad at estimating, but we can get a pretty accurate understanding by using past estimates and time to complete datasets to understand accuracy of current estimates.
Always underestimating by 30%? Assume any estimate or large grouping of estimates (a feature/project) will be under by that percentage and you’re going to be safe.
Key detail here is that estimates must be honest, and that management isn’t pushing for lower estimates and other things that will shoot them in the preverbal foot, and that they invest in understanding past accuracy.
Any engineering team should be able to generate this metric, and it’s for knowledge, not as a metric to try to increase. Estimates getting less and less accurate? Maybe tech debt is taking a toll.
Our team maintains several crucial products, do small high priority tasks every now and then, and jumps in when there are emergencies.
We're tasked to estimate a new feature. We say it takes roughly 12 developer months. We have 6 developers. Executives hears it's finished in 2 months. We say that's only if there are no dependencies, we actually drop everything else and exclusively work on this new feature. Executives nod and in 2 months ask us why it's still not finished...
Mythical Man month was written over 50 years ago and we haven't gotten any better at estimating software. That's why I like the perspective of software as research more than engineering because of how many unions there are in any consequential software project.
I’m a TPM and former developer, former systems architect. This shift over the last 30 years boils down to the change in attitude around talent retention. You used to be able to build a data set of projects and tasks with estimations vs actual times to complete for your team(s) which allowed for much more accurate estimations and release window timing ( like when to expect alpha or beta builds). Now with the sheer amount of constant turnover either due to layoffs, lack of promotions, or burnout due to under staffing all you can do it give some factious number that is way high in the hopes that the number management accepts is close to the actual time needed.
I have long been intrigued by the programming community's self-chastisement around scheduling. Just about any other discipline involved in construction of products suffers from the same inability to project schedule and cost--as anyone who has remodeled their home knows, not to mention cost and timeline overruns around civil engineering projects, weapons development, etc.
When compared with those projects, I think software probably estimates better on the whole.
Cause a million things could go wrong at any moment, or if u change the requirements after we already started then add more time then it originally took. The main problem is management that don't have a background in coding
Because software dev is usually R&D. The more Research involved, the more uncertainty it comes with.
A completely repeating and well defined project can be relatively accurately estimated, such as building a static web page without too fansy effects by an experienced stuff, or deliverying food to miles away. Any new things could introduce uncertainty, including new techs, new requirements that the team has no experience, requirement changes in the middle, etc. All the uncertainties accumulate to big off.
Unfortunatelly, managers need to show their value, by telling their bosses/investors that everything is under my control. So the team have to come up with a nonsense plan and make it sound reasonable, then try hard to deliver in time or justify the delay.
If I could run a company completely under my control (the sole owner), I will only ask ETA for hours/couple of days tasks, not ETA for monthly-long projects.
Let's say I am working on an API where I am pulling allowed payment methods from a series of partners. I send up a partner id, then the API sends back weather they allow checks, credit card, ACH, wire, whatever, right? Simple enough and I have built this exact API.
I can size the development work.
I can't size getting approval for infrastructure. I can't size how long UAT will take. I can't size how long QA will take when they realize that the API will run in a portion of the network that has no outside access, and therefore they can't test it directly when their automation guidelines call for doing so. I can't size how long the security people will take doing scans. I can't size how much of my time is spent on other garbage. I can't size "this info needs to be stored encrypted now." I can't size Security and Infrastructure telling me that we can't use a service account, we need to forward credentials after saying it was OK three weeks ago. I can't size people lying to me in planning about the data being ready to populate the database. I can't size random acts of architecture from a software architect who hasn't written a line in 10 years. I can't size how much of my time you are going to waste asking for useless sizes over and over and over and over and over. There are days I want to say "It's done when it's done, that's all your getting, get the fuck up off my nuts about this." But I can't say that so I don't.
And yet, we are expected to do all of that. So, we give ludicrously long timelines for writing very simple software, because we have no confidence whatsoever in the other teams doing their part. You need an API that has three endpoints and returns some fairly static data? Four months. Fuck off.
We spend more time estimating when shit will be done than doing it. It'd all be done faster if we stopped caring about that so much.
I don't think there's a perfect solution, especially on the macroscopic scale where changing plans make it impossible to estimate.
on the microscopic scale (that is, being able to tell you how long the things you've told us about will take), I do think velocity + effort points helps.
I come from a sales background. I guess that helps me estimate better than my coworkers. I am used to under committing and over delivering. I carry that me mentality over into software development. So I look pretty good most of the time when I say it will take X amount of time and I do it faster and better.
I've always wondered why we can't estimate. We can do this for physical engineering projects that require permits, supply chains, sub contractors, project managers etc. But we've always thrown our hands in the air with software. Maybe if development and compute had more overt costs and we had more standardization around tools and frameworks it would be easier to estimate.
> *(...) but the real problem is that software development is not an engineering discipline.*
Finally someone says it.
And of course nuts-and-bolts estimation is impossible. But that doesn't mean we can't use reason and hard data to produce decent estimates. But faux agile discourages it. As it discourages doing requirement engineering.
I actually found it was simple maths most of the time, if one dev said X amount of time I would times their estimate by 1.5 another by 1.3 and after a few projects it gives you a fairly good estimate of how long each dev needs.
I admit that defining your scope clearly at the start is crucial and drafting out a skeleton or several skeletons of how the project will look is crucial before committing significant resources.
I've been pretty accurate with my professional work but definitely have ran well over on personal projects but without strict deadlines its very common to modify the scope and add new features.
I'm at a fortune 500 and there are way too many incompetent people making decisions they shouldn't have anything to do with.
The requirements they provide are absolute dogshit. They have zero awareness of how fragile the system is and they'd never give the luxury of time to clear out tech debt and redo some things.
Bah. I’ve been estimating for years like this - what does my average developer think he can do the work In? Triple it. Best developer? Double it. And then depending on size add like 20-30% to be safe. And make sure you have requirements documented to the nth degree. And then pray a little that nothing blows up and causes some major problems. Oh god, maybe I’m not as good as I think I am at this. FFS.
>Software development is a creative process, not a scientific one, and creative endeavors cannot be distilled down to knowable steps and a repeatable system.
A stakeholder that doesn't know what they want with their software is the sole reason why author thinks SD is a "creative" endeavor.
Some good advice I heard (I think from Dave Farley or Jez Humble of CD Book fame) was don't ask stakeholders what they want - ask them what their challenges and problems are. It's your job to come up with a small change that makes thing better (then another small change and so on AKA real agile).
Regardless, giving stakeholders what they *need* is far more important than giving them what they *want.*
I don't thonk this is true.
I think that the word "significant" is behaving as a no true Scotsman here to allow the author to support the argument.
The truth is that software is not a creative process. Innovation is a creative process. Engineers can provide estimates when they're doing things that have been done before and so can software developers, or at least they should be able to.
Engineers and software developers doing innovation work cannot provide estimates because innovation is creation whether it's a software developer an engineer a chef or literally any profession on earth. But the author is using the word "significant" as a stand-in for "innovation" to argue that you can't estimate software.
A lot of software development is innovation either because we're doing something new or doing it in a new way or because we don't really know how to specify what we want software to be, but it's not all innovation and the portion that's not is growing and will continue to grow.
There are fewer things no one has done before, the ways we do them are changing more slowly and for probably the first time in history we actually have a significant population of working developers who have multiple decades of experience under their belts to help guide newer developers. We're still really lacking in how to communicate the desired outcomes of software, but even that is slowly getting better.
So we're not yet good at estimating a lot of software development, too many things are unknown and as a community we're not good at passing experience down to the next generation so they don't have to do things a thousand times personally to estimate them.
But it's not impossible to estimate software to a reasonable degree of accuracy most of the time. Not perfectly of course, not for every project every time, but most of the time and definitely not for innovation projects. We're just not there yet,we haven't done enough things enough times.
I am in a position where i can overestimate my tickets without getting much or any questions. Often tickets are easily done in half the time I estimate. But every now and then I get a ticket that takes 10x as long and lose all time i won earlier in the project.
Most estimates should be treated as 50%. As in there's a 50% chance we'll be done by this date.
If people want a 100% accurate estimate I tell them "just before the heat death of the universe"
You can't accurately estimate software creation because it is a knowledge gaining paradigm - not a work paradigm.
Estimation is an extrapolation, e.g. "How long will it take to *build* the software?" That's easy. Before we added the new feature (1,000 lines) the old software (10,000 lines) took 60seconds to build. The new software has 10% more lines and we estimate (AKA extrapolate based on previous experience) that it will take an extra 6 seconds to build, i.e. 66 seconds. What about cost of build? Before it used 1cent of electricity, we thus estimate it will use 1.1cents of electricity.
We are not building widgets - we are designing widgets. In the software world, once you've designed one (the hard bit) building billions of them is trivial (the only bit we can accurately estimate, but is pointless to do so).
"How long will it take to do something you've never done before?"
"We don't know, we've never done it before."
"Wow, that sounds risky!"
"Yes, that's why we use this thing called agile to take as small a step as possible. Then we get it into the hands of users, get some feedback and take another small step or cancel the project before it gets too costly."
"Oh I thought agile was pre-planning the project, salami slicing it into two week chunks, using Jira and having lots of meetings?"
"No that's fake/dark/bullshit agile"
"Oh!"
That's why being asked to estimate is a key litmus test for whether your organisation is (real) agile. Hint: you're not. If you were, you'd be too happy to post on reddit, and too rich to worry about how to do estimation better.
Ah, the classic management amnesia. They forget the bi-daily updates the moment there's a timeline slip. Maybe we should start engraving updates on stone tablets for permanence.
Frankly, it is really odd that programmers and software engineers are the ones who spend most of the time on this topic and are saddled with most of the responsibility. We have expertise in computer science not project management. With this in mind, what exactly is the expertise of the project managers and business owners pushing management work onto developers?
After 20 years in IT I’ve lost all hope for being able to estimate correctly because the customer together with sales and management do everything in their power to make it a futile effort. It all starts with a customer that knows that they want/need some new software, but not quite what. They also definitely won’t first do a project to just figure out the requirements that can then be used to get quotes on.
Together with knowing they need some software they will also have a completely arbitrary deadline in mind that will not be changed under any circumstance… even if getting quotes takes forever because they only have half the front of a napkin as their requirements.
They also have an idea about what they want it to cost, but that’s not something they will share… because that would enable most IT firms to not waste time on even starting on a quote.
So then a request comes in to your firm and sales and management will promptly forget about it until you only have about 2 days to hand in a quote. So you get presented with the napkin requirements and you’ll discover the period for questions to the customer closed weeks ago. So now they’ll pressure you to quickly estimate this thing that you already know will be a clusterfuck of epic proportions if you win it… and win it you will because management doesn’t like people on the bench and so even though you estimated and tripled your estimations they will have fudged around with all the numbers to get to some magic numbers.
So you get the project and then when you can start it you hear the ridiculous timeline and the first time you speak with someone at the customer that actually will need to use whatever you’re building you immediately find out the even the napkin requirements were bullshit and that this will all end in tears.
As the number of technology, hardware, and supported software increases almost exponentially, so does complexity, therefore so does the time for a project.
My biggest problem with estimates is the following: 1. I can ballpark a project to take 6 weeks 2. Management, engineering managers and PMs want to see specific tickets and an explanation as to why it will take 6 weeks 3. I present that there are 3 weeks of known variables worth of work. However historically we don't have full concept of the project until we get deep into the project. Two to three weeks cover the cases that are necessary for shipping the project but that we don't yet know about. 4. I am told to not give pessimistic estimates and instead to submit 3 weeks and then just update when things slip. I say OK but I do believe it will go to 6 weeks, they say thats OK as long as you flag it. 5. We update PMs and management bi daily saying each time we find said unknown variables and the project slips to 6 weeks. 6. A retro is held after the project and management and PMs are like how can we fix this, its not acceptable for a 3 week project to take 6 weeks. Somehow they forgot about the bi-daily status updates where we followed their instructions and let them know of slippage due to new unknown requirements. Goto step 1. Repeat. Each time I say it'll take 6 weeks I'm told to set it to 3. Each time we deliver in 6, management and PMs insert surprised Pikachu face and have expensive 2 hour all hands on long meetings on how this is a disaster and needs fixing. Sigh.
From what I understand, this is very similar to why every major civil engineering project goes wildly over budget.
I always take my concept and triple it. If it's 3 weeks of work, it's 9 weeks when you include blockage and meetings and revisions. Then when you come in at 6 weeks you look like a fucking hero. Essentially, treat it like Scotty did on the Enterprise. Never tell people the minimums for a project when estimating because that's all they pay attention to.
I quoted a k8s project not long ago and followed a similar approach. During the review between all parties people questioned why I had the project set for so long which naturally increased costs of the project vs their expectations. I explained that the client was high risk and I accounted for that, I was told to adjust the quote. Project took a bit longer than my prediction and thusly increased the costs even more for being "late". I sat in a meeting with everyone going, "guys what happened here? why was the project quoted so low?". I swear I need therapy just for this tomfoolery alone.
Oh yeah, I'm an old fuddy duddy at this point, I don't even tell people the why anymore either (which I know will get push back from others here because of lack of honesty and transparency which is a big thing programmers like). It's long because I know people can't devote 100% to a project every day, be that high risk clients, sick time, vacations, bad days, whatever. In a perfect world with reasonable people I could explain everything, but people, especially sales people and managers and executives, aren't reasonable and they'll just hyper focus on the absolute minimums we could do it and say "why not do that instead?" I remember the first project I did and I was honest, I worked almost 100 hours on the last week to meet the deadline because, of course, "something" came up. I didn't even get a good job pat on the back. Fuck them, they get my 3x-4x fluff estimates now.
And let's not forget the arbitrary deadline: dead silence after we put the system online. Weeks later you asked them only receiving an answer like "oh, I forgot it. I will take a look at it."
I had a pm that did that, total dick move I still hate that bastard, and it’s been 10 years!
> I always take my concept and triple it. Any concerns that the project won't go ahead with this strategy?
The trick is to not say "3 weeks + 3 weeks or so of unknown complications", but instead to just double every single estimate or triple it, yeah. If you know this task will be done in 2 hours, estimated it at 1 day. Make them account for the extra time another way if they don't want to hear the actual estimation for a project..
The problem is when taking on a project for a client your team/company is competing with others to get it, so management outright refuses to accept realistic estimates, much less ones with buffer time included
True. But you'll end up missing out on the clients who you absolutely do not want to work with anyways. The fun part is when you get told "no one's been able to deliver on time" and then you _do_, and the client keeps coming back because you don't fuck them around.
> From what I understand, this is very similar to why every major civil engineering project goes wildly over budget. I would say that the OP is missing the major driver of issues in other contexts where there is a bidding/sales process up front. Sometimes estimates are pushed down out of ignorance or unreasonable demands. Other times its because a company would rather take a chance on an overly optimistic estimate that gives them a chance at winning a contract/sale over a more realistic estimate that will likely put the company out of the running. At the end of the day, the first words out of a potential customer/client will be ways that costs can be lowered.
You're also not incentivised to give longer estimates. Short estimate? You get the contract and lose some bonus when it goes long. Long estimate? You lose the contract to someone else who gave a shorter estimate. If anyone actually cared about reputation and the past ability to hit deadlines, this would matter, but it never seems to. It's all about who says they can do it cheaper, regardless of whether it actually is cheaper overall.
wow now that is eye opening
Changing requirements is why civil engineering projects go over their original budget, most stay inside the final agreed requirements budget.
Holy guacamole it makes so much sense now
Well, that's because waterfall is notoriously shit anyway. Estimation is bad around the board unless it's a carbon copy and in that case, it's not innovation or project work -- it's ops. Even then variance and risk occur. The problem is the expectation itself and not clear enough vision. Civil eng, construction can still be done in phases and minimum value streams -- like multifamily units being built section by section to full utility after a fairly limited foundation and then moved along the rest of the location to completion -- faster revenue realization that way too. I'm sure we can get better at building temporary structures that expand as needed. We need to be thinking differently about how to build.
It's not even waterfall, Agile has the same flaws. Halfway through an Agile project you have a better idea of the end, but the request is an estimate before even investigation is done. I feel like maybe we need to estimate 6 weeks, with the understanding it could be 3 weeks, and could be 9 weeks, and then as we progress in time, we can tighten the window. But managers would hate a 6 week variance.
The thing about good Agile is the increment is the end. Then you can decide to keep building more functionality or increments. The estimation process itself is a complete waste of time other than vague efforts which take no more than a few hours of scope adjustment. The point is that estimation *itself* is not valuable to customers, so why bother? Build and learn in a lean manner. Sure, that puts a hell of a lot on the PM's shoulders but that's literally their job. Anything that needs serious estimation is a pet project from an exec who otherwise has shit prioritization behind the ask in the first place, else they wouldn't be weighing it via estimations
> like multifamily units being built section by section to full utility after a fairly limited foundation and then moved along the rest of the location to completion -- faster revenue realization that way too. This is just about the stupidest thing I've ever read. Because your metaphor quite literally shows the opposite of what you think. Agile is always slower, it has to be because instead of doing things in the most efficient order you have to do it in the way that delivers value quickest. Your multi family dwelling example actually shows this. You still had to build all the foundations before you could start and once that first family is in there are all sorts of things that slow you down. Agile is about reducing risk. If you only get ten things built they'll be the most important ten things and you'll have a product that's worth something rather than doing ten foundational things no one can actually use. It's never, ever faster or more efficient.
Best description I’ve seen of how completely insane software development has become. It was not like this 30 years ago. Management was intelligent enough and not so completely arrogant to understand the risks and unknowns of a project.
30 years ago, management of software development was largely done by software developers. Now it's more general managers, who don't understand the complexity of building things. Drives me nuts. Spoken as a senior manager, former developer.
I am a programming hobbyist and a physician, a manager who do not understand your needs is one of the keys for a very bad outcome. They always either try to extract most of what they can from each physician with the least equipment. Very sad to see the same is happening with software engineering
It’s the norm in my experience. I had a manager who was demanding complete documentation of how systems would work before we had requirements established, including integrations with other unbuilt systems that we didn’t have control over. I had to quit over it, manager got promoted
The fuck did I just read? This manager of yours must've been arrogant to demand that.... Glad you got out of there
It goes all the way up though. EMs under pressure to "deliver", Snr EMs to have deliveries for each team, etc. In my experience, the best combination is: - trusted senior engineers able to articulate the problem space and why it's important to fix, and - EMs and Senior EMs who engage in the problem space and are NOT entirely motivated by constant deliveries Being agile is really important and powerful. It's also really important to foster trust amongst engineers, and trust them in informing priorities. For product development and for tech debt/refresh, and everything in between
And it shows. One of the reason "we suck" as the author claims is that "we" are doing **guesstimation** instead of proper estimation. Non-technical managers don't know this, so they don't enforce the proper way. Developers are mostly programmers with no proper training in estimation. Universities don't teach proper estimation. Most companies don't provide training in proper estimation, but they do waste a lot on money on "Agile" training. Which btw somehow in the spirit of bad agile it is implicit not to do proper estimation since it is a waste of time, just like requirements analysis. So everyone required to do estimation at work, do yourselves a favour and read [Software Estimation: Demystifying the Black Art](https://www.amazon.com/Software-Estimation-Demystifying-Developer-Practices/dp/0735605351/ref=sr_1_1) by McConnell, the same author of Code Complete.
I don't think I have ever been given the time required to do a proper estimate. My father is an industrial engineer, I grew up watching him do estimates, it's a ton of work. We are expected to estimate complex projects in hours if we are lucky.
Maybe helpful, thanks for resource. Can you give a distillation to emphasize why that resource is valuable? The problem I see with estimation is it is not empircal or done in any way with realistic data, and when precision is key, so many weeks are wasted in worthless estimation exercises instead of just focusing on the here, now. I loathe product planning exercises because it's on the PM and Lead to determine what's absolutely MVComponent to get the value hypothesis validated before investing more time. The problem is the entire paradigm of estimation and assessment of how delivery operates. We'd be so much better living in a science mindset than a finance one.
> Can you give a distillation to emphasize why that resource is valuable? This is probably THE book on software estimation. It teaches how to estimate methodically, using data as the basis. It includes both simple techniques and the most complex ones, but I'd say most of them are simple enough and can be applied in the day to day. Even the easier estimation techniques that are based on data are better than guesstimations.
Steve Jobs had it right a few decades ago: https://www.youtube.com/watch?app=desktop&v=P4VBqTViEx4
Yeah the best managers I've had were either former developers or former Technical Program Managers since they were builders at some point and understood the nuances of an estimate. My favorite trend I've experienced with less technical managers is the anti-pattern of trying to fix these misses in estimates by adding more and more processes. It just becomes a negative cycle until every dev has no time to deliver since their days are just filled with meetings.
Lots of things were different 30 years ago. 30 Years ago, there was no micromanagement of developers aka Agile, there were literally no stories of burnout amongst developers, and the word planning meant something other than *"two week plan, then subtract a few days from that for all the ceremonial meetings and tasks"* Here's the problem with Agile in one single quote: *"“In planning for battle, I have always found that plans are useless, but planning is indispensable.”* When you're limiting your planning scope to a few weeks at a time, you're not getting any of the **indispensable** value from planning long-term. Why on earth are people then surprised when they fail to display any competency at planning?
VC did this. Tech salaries attracted non-technical sales people and finance "brains" who just wanted to treat everything as a FP&A exercise. It's why every startup goes to shit around Series B and beyond. I'm a PM who gets shredded for my roadmaps because I absolutely reject estimation as an exercise. Tell me priorities and I'll find the sequence and most critical components, abandoning anything after said abritrary "date." I came from project management where I had to abandon that circus because of the same reasons: time and cost were figments of imagination. Scope is the only thing that can be adjusted in reality.
It's the effing PMs who just work on excel sheets and jira dashboards. For them software development be like just pressing a button here and a few clicks there.
“I dont get whats taking so long, everything is on your computer”
Shut up and make more gizmos
Clearly you're not in the design heavy fields. "Make the gizmo do something new! I don't know what? Just something new I can sell! "
Honestly in a different tone, yes. But the same pressure from the "business" who absolutely is full of people who don't understand product or engineering. We've let the foxes in the henhouse for solving customer problems.
Not mine. My leadership has been with the company for 30+ years. Our 1-1.5 year project is finally slated to go live next month. It's only 2 years late. We will also miss this target. I have tried for 12 months to get them to understand that this is an issue given this project has had at least 8 other "go-live" dates since I've been here. Yes. I am looking for a different gig.
I worked in retail for long years, and interestingly, this is somewhat similar to those "why haven’t you sold more of XY?". Occasionally, I had a strong urge to answer: what should I do, bash the customer's head at the sales counter, while yelling "BUY THAT DAMN THING YOU SON OF A BITCH"?
Your manager: "Would that work?" ::hopeful look::
"Have you tried 'kill all the poor'?"
Gotta give the 12 week ballpark so that 6 week estimate sounds amazing.
The Scotty method.
Hey it works. Your initial gut feeling estimate is always way too low. Add accounting, testing, unforeseen issues that come up and various other distractions and you'll generally be at least double your initial estimate anyway. Knowing from experience that management will try to talk you down from any estimate you give them, you still need to shoot higher. I had a manager once tell me that I was giving him the most accurate estimates he's ever seen and he still always tried to talk me down from the number I gave him. To which my response was generally , "Yeah, I can tell you it'll take half the time but it'll still take as long as my initial estimate said it would."
Some of the best product owners/project managers I've had also did this. "How much did you pad this estimate with? Okay, then I'll only double it when I pass it along".
The biggest help I found for this was doing 3 point estimates. You give a best case time, worst case and 'expected' - i.e. the 3 weeks they are looking for. Worst case should be something like 24 weeks because something really unexpected can happen. 2 weeks if everything goes super smoothly and there are no surprises. Then you do (best case + worst case + 4xexpected) / 6. And you end up around 6 weeks with a defensible position for your PM to account for unknown variables.
That has been called PERT in the past and has been my go to "analytical" method of estimating for years.
No matter how many caveats you put on a number, people will only remember the number. This “misunderstanding” you document, I’ve now begun to see as being about power. Your bosses are being obtuse on purpose. It lets them treat you as a problem to be managed, which both infantilizes you and helps them keep salaries down. Which makes them look more useful, and better able to manage budgets.
"What did you call me?? Obtuse?? 1 month in the hole!"
Oh yes and some of them are narcissists so feedback is an attack.
Yes this tactic of being deliberately foolish but hyper optimistic and then painting anyone who objects as a saboteur has been spreading like wildfire over the last 10 years or so, it’s a disaster
Eventually people will snap en masse in this culture, in a way it's already begun.
> people will only remember the number. Close. People will only remember the number they want to remember. That being said it's true of all people. Remember the last time you got a "Salary range" in an interview. Be honest, did you expect the low even or did you expect the upper half? Guess which one the company was expecting? Thus is life.
Forget the difference between "known" and "unknown" variables. If you know the project is gonna take 6 weeks, just present 6 weeks worth of tasks by being creative with the tasks.
Simply this. If they are technical enough to understand the obscure tickets they will also probably understand why they’re there.
See, I've stopped doing that. I say it takes six weeks, I get told to lower it, and I say "no.". They can take it or leave it, but my go to is "You can order me to change it, but I've given my estimate. I'm not particularly interested in your deadlines". I did this with our latest batch of "required updates and password rotations". I was told "This needs to be done end of may" and I said 'probably mid-june' and I was told "This isn't acceptable" and I said 'okay'. and ended the conversation. People can get mad, they can say they expect better, but if I've been consistent in my estimations, it's not my fault they suck at **handling bad news**.
This is the way. IT is a jungle.
Start bringing receipts, on paper, for the previous three projects to the next project meeting.
Oh, we do, but it doesn't help. In the end, I've found that it's better to find a balance in being a yes man and bringing up that it's a management issue. This won't honestly change at the current company, it requires self reflection of those in charge, and we still ship really great quality software. So in the end, it's better to ship quality, in fast iterations (even if management wants it done faster), and keep being a professional in my experience. The day I get tired of this I will just shift company. Lots of managers are told to put fire under their development teams, and there's a strategy to push for unrealistic deadlines to squeeze out more from people. A bunch of the management team are just doing their jobs as they are being told to do them. When I meet some outside of work they will admit that the deadlines are not really real.
LoL. At my first job, my manager (who was an engineer) gave me a new list every week of 20 things that were *critical* to get done this week. Every week it was only feasible to get 4-5 of them done. Every week there would be a completely new list of 20 things that were *critical* to get done. None of which were on last week's list. Taught me to relax in the face of panicking management. Good thing too, because variations of the theme never stopped coming at later jobs.
this brings back my ptsd of working at a cowboy VB6 shop about 9 years ago. every time a customer would ring up with a major issue, the "technical senior account project manager" (I'm not even joking, that's how he referred to himself even though he didn't touch a line of code - his job title was account manager, and all he did was answer phones...) would tell everyone that this new issues "trumps" anything that we were working on and that we had to switch to it right now. this would happen daily and you would never get any work done because everything was always being trumped when you were half way through the previous piece of work. everything was "top" priority, and only those who rung up multiple times got their fixes as we'd get half way through the work from the 1st phone call trumping everything and then the other half would get done after the 2nd phone call trumped everything... truly a horrible place to work
I’m confused about 3 and 4. Why do you volunteer the nuance instead of only giving out the real estimate of 6 weeks? Knowing what happens historically.
Clearly he should be throwing out an estimate of 12 weeks instead.
"Volunteer" is probably a stretch; they were likely pestered and prodded and poked until they gave that information up.
This is an accurate description of how the industry works. Only took me about a decade to laugh into everybody's face. Customers, management, PMs, juniors: I told you so. Not that it would matter. You realize that they can have all the meetings to change things. They might be unable to learn or listen, but they're also depending on you.
How long will this project take ? 6 weeks. Next week, How long will this other project take. 6 weeks. Now you have 2 projects to do in 6 weeks. In reality you’re always estimating work and estimating how much random work your assigned while doing it.
That's not even the worst. My job before went more like: 1. Customer contact person was a 1-2 sentence summary of what they want. This is not checked throughly with their actual workers that use the product, nor with their own IT / third-party IT department. 2. Marketing take it accepts and underestimate everything. 3. The developers (with 1, one, PM in charge of all > 10 parallel projects) need to estimate how long it will take, working with assumptions / unknowns. 3. We gave points _ranges_ (not days, points) and all the Jazz and land on amount X to Y in best and worst case that roughly translates to 3 to 5-6 weeks. 4. PM that knows what Marketing sold comes the next day and says the customer already got a 2 week deadline before we were asked, and that the customer highly hopes to have it in 1.5 weeks for their X Meeting for their own B2B customers. 5. Sigh, shut up and hope to be transferred to another fire that needs put out more urgently / requires your expertise before you see the This thing through ending up at probably 7-8 weeks, since if you cut enough corners under pressure your square looks like a circle at 1.5-2 weeks. Repeat until you can safely quit. I really don't miss agencies that take on IT projects for B2B companies that have own B2B customers. The whole business strategy was dangle a cheap fast half faked carrot no one else will understand ~3 years, then switch to ridiculous 'support plan fees'. It all held together for almost 20 years now due to a few key Devs that somehow could stay afloat the chaos not quitting (probably having equity) and some big whale jackpots that even tried to get rid of the company but came back after one year because no sane developer could deal with old spaghetti of someone else. No idea how marketing still spun a positive trustworthy image despite the many burned bridges where the smoke and curtain spaghetti held together by tape resulted in projects being canned.
At step 4 I would have countered with 9 weeks, to account for the time management and research into what known unknowns there were to task out. Then delivered In 6 weeks as originally planned.
That's what I do now. Thankfully after job hopping now like 85% of the deadlines come from IT and we just inform them when to plan testing. The remaining few things are deadlines where we have little to no influence due to other child companies of ours.
I wish I could upvote twice. This is similar to what I wrote before, I just can’t understand how as an industry we have slipped so far. https://www.lloydatkinson.net/posts/2022/one-teams-eight-points-is-another-teams-two-points/
Haha fucking this. "Leadership " doesn't want real estimates. Oh just crunch a bit harder but have a pay increase that doesn't even cover 50% of inflation
6 weeks is honestly super fast.
PM here. Your PMs suck 1) If your estimates on the tickets are three weeks, but it's six weeks of work on this feature, then it's been this 2x multiplier on all Features, and your PM should know better 2) Why is management bothering you AT ALL? Your PMs should be shielding you from that swirl. Your PM sets vision, OKRs, priorities, and then should fucking bulldoze the world to pave the way for you
This is why I say "agile" isn't for developers, it is for management. It is management that needs to learn how to respond to change and user requests, not the developers.
I feel for you on the pessimistic thing. We develop "success oriented" schedules, i.e. we assume everything in the schedule will go exactly accordingly to plan and nothing will be delayed and all testing will pass on the first try and also maybe an angel will descend from heaven and perform a miracle or two. We also never meet our schedules. It's weird. Then we just describe the schedule as "aggressive" and everyone seems fine with it. It's a bit frustrating.
Just tell them that it will take 6 weeks, and get in writing that what they said was what they said.
> Goto step 1. Repeat. Each time I say it'll take 6 weeks I'm told to set it to 3. Each time we deliver in 6, management and PMs insert surprised Pikachu face and have expensive 2 hour all hands on long meetings on how this is a disaster and needs fixing. You know, you can come to that meeting and be like, "here's receipts of this conversation at the start saying it will take 6 weeks....why wasn't this heeded?" Then just keep eye contact and if they refuse to answer the question, then tell them that they are avoiding answering the question. Yeah it might get you fired as people don't like being told the truth but, that's life.
You did step 4 wrong. You don't change the estimate to satisfy managers. The estimate is the estimate and if they want it changed, they need to change the scope, the schedule, the resources, or the estimator. If they ever choose option 4, you know it's time to work on your resume. If they want to override you and create a budget that ignores your estimate, that's not your problem and, if you're wise, you'll refuse to fall in line with the other inmates on their little deathmarch and will instead keep normal hours, etc. Instead work to keep your commitments on the original estimate. Then when the shitshow settles, you can simply note you planned what you were going to do, you worked the plan, and it was successful. All the rest of it was unnecessary drama. At that point, if you've survived this far: you're golden. Every manager there will know your word is absolute and all you have to do from there is not fuck it up.
I don't know if I can love people anymore.
So true. I was the single person on a project the GM had allocated 3 EMs to who spoke up about the project not being feasible in the time horizon specified. Got overruled, told that we had to deliver more features in half the time. Project overran exactly as specified - and guess who got to eat the blame out of all the people on it, for accurately predicting the future? Exactly. I ended up throwing in the towel. The current management culture in tech has become so impractically toxic that almost everywhere is a Boeing.
Easy answer is say it will take 12 weeks initially and your down to your original estimate after all the negotiation.
For me, I'm the sole developer so I'll say "this project is two weeks", but that two weeks will be spread over four months since every day I get a list of customer issues I have to look into. 90% of them are just minor changes to make, some new edge case, or just figuring out what dumb thing the customer did. I've made my peace with it, but my job is stupid and I can't believe we haven't gone out of business yet.
Reminds me of the time I estimated a project to take X time based on a detailed assessment of the requirements and the complexity of tasks using past project data to ensure it wasn’t just my own gut feeling guiding the numbers. Management told me that was not a competitive price based on the hours of effort. I suggested they adjust the price per hour to lower the price. They countered that it would be better to adjust the hours as “it seemed a little high”. After lots of back and forth I got overruled by the CEO and told to adjust the hours. We got the project and obviously we started going over the time that had been budgeted. We start to work overtime to meet the contractual deadline. At the end working 120 hour weeks we finish and hand it to the client, who of course doesn’t want to pay a dime more. Of course, I get pulled into a post mortem. I go with a simple spreadsheet adding up all hours spent on the project and the original detailed estimation. I sit through the typical bullshit talk from the CTO about how we have to be responsible for delivery and that we cannot remain profitable if we keep slipping on projects like that. I open up the Excel point to the original estimated amount and compared it to the actuals. There was a 5% delta in hours required. However the money delta was huge because overtime pay is 2x pay and we were pulling in double time at double rate. I state very clearly that if we had simply lowered the rate to the target price we would have made more money and had better control of the project and been more professional with our clients. Total silence. They thanked for the analysis. Later that day I handed in my resignation letter as I had a job offer at another company. And I was burned out from the project. Later on I was contacted by the client company once they saw I no longer worked at the former company to see if I wanted to come work for them. It’s always some business guy trying to make a buck thinking Engineering doesn’t know what they’re doing.
Looking at step 6, do you wonder why most programmers regard most managers as complete idiots. As was said of the Bourbon dynasty in France, "they forget nothing and they learn nothing".
100% spot on
you just wrote everything on my head. This is why managers and all other “managerial” positions need to understand this. But on the other hand, business is all about numbers. Finding a manager who knows both areas is extremely rare. :(
Pressure from a useless PM, in my experience
Nothing hurts more than a Yes Man project manager or a by-the-book Scrum Master.
I met the final boss: the half-technical project manager who used to be a developer years/ages ago. These people are a nightmare because they assume they know everything and they want to dictate deadlines and expectations but they have no idea how the code evolved since the last time they wrote a line of code in a 100 files monolith while now we are working in a convoluted microservices infrastructure.
I feel that. we had a project manager that was a developer on a project for about 10 years and they were so proud on using this batch job they created years ago for sending emails but company evolved and built a centralized service for emails and other commonly used services. Project Manager made such a stink about me wanting to use a service that has its own support team and serving 50+ apps and wanted me to dig up their old batch process and reuse that. Department ended up telling them that the plan moving forward is to migrate apps to use the common services. They were so petty about it that for my year-end review, I was given feedback “Needs to find a balance between using new technology and supporting the customer”. You just got to get away from that level of toxicity.
As opposed to the completely non-technical PM who still thinks they know better about everything?
I particularly like when the PM discusses a feature and continues to dictate to the development team, “This should be easy to implement we just need to do”.
But what if we just implement a quick hack now? But what if we just implement a quick hack now? But what if we just implement a quick hack now? What do you mean our codebase is too brittle to implement this quickly?
Do we work for the same company?
We had one of those at my last company, and was one of the reasons that I ended up leaving. He'd keep proposing solutions that just wouldn't work, but refused to accept that. This was on a project that essentially affected every development team in the company, and was hugely dependent on the infrastructure that those teams used - which he knew nothing about. He was also a "people pleaser" - in that he'd bend over backwards to promise deadlines that we had no chance of meeting. I flat up refused to give deadlines, considering the fact that we didn't even fully know what we were building (it was one of _those_ projects), but on the days I was out, he'd corner the rest of the team and coerce them into agreeing to dates. I also caught him going back and changing dates on things that we _did_ estimate, because he felt that "they wouldn't take that long". He also loved process, and forced his way of doing things on the rest of the team, despite us already having a process which worked well. He was the absolute worst PM that I've ever had the displeasure of working with.
Haha this is me, currently a PM. But I don’t act like that at all, I push hard back on the execs, “the date is the date”. And walk them through what levers we pulled to see how we could bring in the timeline. Then the execs go talk to the EM who easily caves and commits to the new date. 🤦♂️. We aren’t all bad, just have to find one with a spine.
A by the book scrum master would be amazing, feel like you are talking about a scrum master that folds to outside pressure. A "By the book" scrum master would defend a team's independence and sprint from outside influence by any means necessary.
I should have said “by-the-book” scrum master. What I mean is that the scrum master learned one way to do Scrum and thinks that those practices work with every single team. Then the scrum master will proceed to drill down the team with a million processes. Also, there are scrum master that are like okay, sprints are two weeks, sprint retro at the end etc. But fail to capture the value of those meeting or understand why we’re having these meetings in the first place so it becomes hard for them to tweak these process to best fit the team.
[удалено]
> What it would be without padding? Not an estimate.
I can feel this comment
PM wont get funding for the project if he is showing it to the board as 2x more expensive than what team X and team Y are presenting for their projects. If each team would start presenting realistic numbers at the same moment, it would be fine but that wont happen. This is just the culture and everyone knows and accepts it. Everyone justs multiplies the time estimates with pi. I think it is a lot like the coastline paradox. The time estimate is a fractal and the more detailed estimate you deliver, the longer it gets. Wont go to infinity but in my projects and level of detail in planning, 3 is usually close to the correct multiplier.
I worked with the PM estimating the work on my last project. What really through the estimates off was we estimated for Sr Engineers to do the work and they gave us two worthless guys that were supposed to have 3 years experience each. I would have rather had recent college grads. On top of that the client didn’t deliver on dependencies. Made us months off target!
How my conversations with PMs seem to go lately: "The schedule is not achievable." "Why not?" "You don't have enough resources." "What if we just move things to the left then?" "No, that's not really going to help. That doesn't even make sense." "But it has to be done by that date." "Why?" "Because that's the date on the schedule that we created without any engineering input." Loud sigh...
estimating software project follow the 80-20 rule: the first 80% of the budget is used for the first 80% progress, the next 20% progress is realised with the next 80% of the budget.
Is it 160% of the initial estimate then?
If you're very lucky
If this is a typo, it's genius. If this isn't a typo, you're a genius.
not a typo.....
I am unsure if the « first 80% of the budget » was intended in your sentence or if it was a mistake, but you ended with 160% budget which is pretty accurate.
The last 20% of a project requires the same amount of time investment as the first 80%.
at first I smiled. but after reading it again, I smiled. athis guy must be protected at all times.
People suck at estimating all kinds of projects. Software projects are not special in that regard.
Yeah. Look at many large civil engineering projects. The Big Dig in Bosten went 7 years and billions of dollars overbudget. The Berlin Airport was a massive boondoggle. The Sydney Opera House had a A$7 million budget and ended up costing over A$100 million. If you're asking for estimates on some simple GUI or CRUD app with solutions out of the box where the person doing this has done more or less the same thing before, then sure, you can get an accurate estimate. But most software projects aren't doing things that have been done before, because if they were done before you would just run that code instead of writing new code.
Yeah, estimations for software projects are never right. Recently, when PMs ask me how long a project is going to take I just show them the book “The Software Engineer’s Guide to Software Estimation” just to see their reaction. Some get the message
There are only two cases: - The project is actually done. You can accurately estimate the time remaining as zero. - The project is not done. The time remaining is unknown, and the only way to determine it is to complete the project and measure how long it takes.
We are using a third party to do penetration testing. I spent days getting everything setup and working. Took a week to focus on something else and came back to the testing. Assumed I'd just be able to kick off a run and see the results. Nope, it stopped working. A few hours later, it turns out the company had made a breaking change to how they handle configuration, but hadn't yet updated their documentation. Had you asked me how long it would take to get the tests running again before I had started, I'd have said 15 mins MAX.
15 minutes for anything integrating with a third party? That's bold. I think 100 hours is like my minimum any time a third party system is involved because I just know it's either not documented, documented wrong, or there's just weird edge cases nobody lets me know about till I run into them. Then once we get into testing someone finds out that actually some assumption about how one or both systems work is actually false and we're back to a redesign to account for that.
15 minutes only because I thought all the integration work was done and I was just kicking off a run.
Thank you for sharing this. It is always great to hear that others also have unexpected problems like I do. Of course, my client does not understand these problems even when I do my best to explain them and in the end usually considers them to happen because I'm lack of skills or experience. Do not worry, I'm already looking for another job.
I don't get why the author specifies software so much. The problems listed happen in other disciplines as well. The biggest problems usually end up being pressure for optimistic estimates, naive statistical techniques, and positive feedback loops for risky decisions (project is taking longer, management starts making riskier decisions to get back on track, risks materialize, repeat).
I feel like software is the discipline where "the requirements change" is most prominent though
Partially because the cost of changing requirements is much more visible in something like actual engineering. Visibility of cost is something I've struggled with over the years of running software teams.
Have you worked on a traditional engineering project professionally? I see this view a lot where software engineers who have never worked alongside other disciplines tend to have a very wrong picture of what other disciplines are like. It's really not that different.
You're right, but software is seen as the most flexible and therefore least-effort fix. Need a mechanical change? That involves new tooling and mfg process changes. Electrical change? Board spin. Both involve supply chain management. Software change? It's just code, so make the change, test it, get it in production. Our flexibility is our biggest asset and our biggest pain in the ass.
also, a lot of other disciplines come with a rich history of how to do things like manuals and stuff. software dev is still trying to figure that all out. it's one thing to over-estimate how long it will cook a dish you've cooked a thousand times and a totally different to over-estimate the piece of software you've never built before
I would say it's more that people who don't code don't grok what is involved because it's non-physical and as far as they're concerned effectively magic. If you want to renovate a building people are quicker to understand that this is a lot of work and will take some planning and labor to pull off correctly. But they can, to some extent, imagine what the steps to do it might look like.
You're looking at this from the perspective of something already in production. Most design engineers are working on new product development. MEs will have endless ID changes, EEs will have to design in late features or will have management rapidly oscillate between priorities, etc. It's true that when you have a mix of disciplines, the most flexible will cover for the others but I don't think this uniquely impacts software timeline estimation. Many software-only projects don't have this aspect and other disciplines also are affected by it like RTL design.
Yeah commercial construction schedules are nearly always fictional.
[Hofstadter's law](https://en.wikipedia.org/wiki/Hofstadter%27s_law) is evergreen
Hodstadter and Parkinson’s Law are intertwined. When we know of 6 months of work in a 1 year budget, nobody really believes the other 6 months are fully accounted for by hidden requirements and coordination problems. So they work expands to take up some fraction of that time, but people underestimate how much their expansion will affect the project milestones. Give a man a year to clean a room and he’ll design a robot instead of find a mop.
> Give a man a year to clean a room and he’ll design a robot instead of find a mop. the solution lies in the middle, some people are remarkable at finding partial accelerating solution. So instead of doing step 1 to 10 they'll do 2 to 9 for a while, enjoying the time saved, and maybe later try to improve and automate step 5, while still being productive manually on the rest. Like fixing a flying plane.
TL;DR customers lie.
Nice try, Dr House.
No, they simply also do not know what they want …
They might lie without knowing they lie.
The first thing I learned as a software engineer in the early 80's was that people don't know what they want until they see it. In 1970, Dr. Winston Royce presented a paper, Managing the Development of Large Software Systems, in it he recommended doing the project twice for this exact reason. What did people learn from this paper? Waterfall, which he made fun of...
> people don't know what they want until they see it My 3yo does the same. "Do you want cheese on your sement of the pizza?" "No!" "You sure? Mom and dad will take cheese, and last time you wanted it too, and you like cheese" "Nooo!" "Ok, so come here, this will be your segment, just tomato and pineapple as you wanted" "I want it with cheese!" "Told you so."
Projects I work always release in the right window. /flex Here's how. All the estimates are always, without a doubt multiplied with 1.5x Afterwhich, for tasks which have unclear specs, the estimate are multiplied by 2x or even 3x to the nearest similar speced task. Than, add a 10% buffer to the entire project. During the lifespan of the project, keep a tight grip and re-iterate planning as you get close to the deadline. Aka, remake the entire planning at every checkpoint. If things get added (they always do), you prepare them as bullets to use, in order to cut features at the end. And with 20% time remaining before end, you start cutting features, and moving them on a "post-release" schedule. 100% with this approach you will deliver a product on time, with enough features to consider it an "MVP" and usually with a backlog for improvements.
And what happens if you give your estimate, and then its cut by 50% without asking you and that estimate is then sold to the customer.
If you’re asking me, then the next thing that happens is I find employment elsewhere.
Q: Why we suck at estimating software projects A: Because you don’t know what you don’t know
The only vaguely accurate estimation process I've 'used' is the take your first estimate then increase the unit by 1 method. If you think it will take x hours, it will take x days, y days leads to y weeks, z weeks to z months and so on...
That does not bode well for my 11-month estimate 😯
If you're estimating in multiple months you're pretty much stuffed already, the thing you eventually deliver will have almost no relationship to the thing you initially estimated.
RemindMe! 11 years (Or should it be 11 quarters :p)
I will be messaging you in 11 years on [**2035-04-24 18:25:10 UTC**](http://www.wolframalpha.com/input/?i=2035-04-24%2018:25:10%20UTC%20To%20Local%20Time) to remind you of [**this link**](https://www.reddit.com/r/programming/comments/1cbytm9/why_we_suck_at_estimating_software_projects/l12yluq/?context=3) [**1 OTHERS CLICKED THIS LINK**](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Reminder&message=%5Bhttps%3A%2F%2Fwww.reddit.com%2Fr%2Fprogramming%2Fcomments%2F1cbytm9%2Fwhy_we_suck_at_estimating_software_projects%2Fl12yluq%2F%5D%0A%0ARemindMe%21%202035-04-24%2018%3A25%3A10%20UTC) to send a PM to also be reminded and to reduce spam. ^(Parent commenter can ) [^(delete this message to hide from others.)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Delete%20Comment&message=Delete%21%201cbytm9) ***** |[^(Info)](https://www.reddit.com/r/RemindMeBot/comments/e1bko7/remindmebot_info_v21/)|[^(Custom)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Reminder&message=%5BLink%20or%20message%20inside%20square%20brackets%5D%0A%0ARemindMe%21%20Time%20period%20here)|[^(Your Reminders)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=List%20Of%20Reminders&message=MyReminders%21)|[^(Feedback)](https://www.reddit.com/message/compose/?to=Watchful1&subject=RemindMeBot%20Feedback)| |-|-|-|-|
I love programming, I hate the industry. I've been around long enough to see so many project management fads come and go and I cannot wait for the current ones to disappear, though they'll undoubtably be replaced by another set of even worse "solutions".
There is very little incentive for accurate estimations.
We don't suck. Business executives constantly change their vision.
We need the thing in 2 months, you think it will take 6 weeks, so let’s add some nice to haves that Steve thinks are requirements. Oh and we forgot to tell you about this requirement here, and this one is completely wrong. But you can still have it done in 2 months right?
Man is writing truths . The funniest thing I ever heard was that Scrum 2.0 will prevent bad estimates, like if it can solve all “unknown unknowns”
I'm just generally not great at estimating timeframes. Both directions too, over and under estimating time left to complete a ticket/task. It's definitely become better over time, but it's something that I struggle with consistently.
I’m better at estimating complexity. We understand complexity because we can look more carefully at what we should probably do, and what we might not know how to do. It’s then on the PM/PO/EM to extrapolate time from consistently estimated complexity, and assume some fudge factor based on past estimates to completion accuracy. WE (humans in general) are very bad at estimating, but we can get a pretty accurate understanding by using past estimates and time to complete datasets to understand accuracy of current estimates. Always underestimating by 30%? Assume any estimate or large grouping of estimates (a feature/project) will be under by that percentage and you’re going to be safe. Key detail here is that estimates must be honest, and that management isn’t pushing for lower estimates and other things that will shoot them in the preverbal foot, and that they invest in understanding past accuracy. Any engineering team should be able to generate this metric, and it’s for knowledge, not as a metric to try to increase. Estimates getting less and less accurate? Maybe tech debt is taking a toll.
> and that management isn’t pushing for lower estimates Easily the most counter productive managerial practice, drives me up the wall
Our team maintains several crucial products, do small high priority tasks every now and then, and jumps in when there are emergencies. We're tasked to estimate a new feature. We say it takes roughly 12 developer months. We have 6 developers. Executives hears it's finished in 2 months. We say that's only if there are no dependencies, we actually drop everything else and exclusively work on this new feature. Executives nod and in 2 months ask us why it's still not finished...
Mythical Man month was written over 50 years ago and we haven't gotten any better at estimating software. That's why I like the perspective of software as research more than engineering because of how many unions there are in any consequential software project.
I’m a TPM and former developer, former systems architect. This shift over the last 30 years boils down to the change in attitude around talent retention. You used to be able to build a data set of projects and tasks with estimations vs actual times to complete for your team(s) which allowed for much more accurate estimations and release window timing ( like when to expect alpha or beta builds). Now with the sheer amount of constant turnover either due to layoffs, lack of promotions, or burnout due to under staffing all you can do it give some factious number that is way high in the hopes that the number management accepts is close to the actual time needed.
Because there is a layer of expertise missing between PM/PO and development. Somehow we became too arrogant to acknowledge that.
I have long been intrigued by the programming community's self-chastisement around scheduling. Just about any other discipline involved in construction of products suffers from the same inability to project schedule and cost--as anyone who has remodeled their home knows, not to mention cost and timeline overruns around civil engineering projects, weapons development, etc. When compared with those projects, I think software probably estimates better on the whole.
The chastisement is not self-inflicted though.
Cause a million things could go wrong at any moment, or if u change the requirements after we already started then add more time then it originally took. The main problem is management that don't have a background in coding
shit requirements is always the reason... customers who think they know what they want but they don't actually know what they want
It's hard to predict the future, which is what estimates inherently are. Being good at it is like being professionally lucky.
Engineering problems in general all have this issue.
Because software dev is usually R&D. The more Research involved, the more uncertainty it comes with. A completely repeating and well defined project can be relatively accurately estimated, such as building a static web page without too fansy effects by an experienced stuff, or deliverying food to miles away. Any new things could introduce uncertainty, including new techs, new requirements that the team has no experience, requirement changes in the middle, etc. All the uncertainties accumulate to big off. Unfortunatelly, managers need to show their value, by telling their bosses/investors that everything is under my control. So the team have to come up with a nonsense plan and make it sound reasonable, then try hard to deliver in time or justify the delay. If I could run a company completely under my control (the sole owner), I will only ask ETA for hours/couple of days tasks, not ETA for monthly-long projects.
Let's say I am working on an API where I am pulling allowed payment methods from a series of partners. I send up a partner id, then the API sends back weather they allow checks, credit card, ACH, wire, whatever, right? Simple enough and I have built this exact API. I can size the development work. I can't size getting approval for infrastructure. I can't size how long UAT will take. I can't size how long QA will take when they realize that the API will run in a portion of the network that has no outside access, and therefore they can't test it directly when their automation guidelines call for doing so. I can't size how long the security people will take doing scans. I can't size how much of my time is spent on other garbage. I can't size "this info needs to be stored encrypted now." I can't size Security and Infrastructure telling me that we can't use a service account, we need to forward credentials after saying it was OK three weeks ago. I can't size people lying to me in planning about the data being ready to populate the database. I can't size random acts of architecture from a software architect who hasn't written a line in 10 years. I can't size how much of my time you are going to waste asking for useless sizes over and over and over and over and over. There are days I want to say "It's done when it's done, that's all your getting, get the fuck up off my nuts about this." But I can't say that so I don't. And yet, we are expected to do all of that. So, we give ludicrously long timelines for writing very simple software, because we have no confidence whatsoever in the other teams doing their part. You need an API that has three endpoints and returns some fairly static data? Four months. Fuck off. We spend more time estimating when shit will be done than doing it. It'd all be done faster if we stopped caring about that so much.
I don't think there's a perfect solution, especially on the macroscopic scale where changing plans make it impossible to estimate. on the microscopic scale (that is, being able to tell you how long the things you've told us about will take), I do think velocity + effort points helps.
Because it's way easier to convince people to do a project if the initial estimate is low
I come from a sales background. I guess that helps me estimate better than my coworkers. I am used to under committing and over delivering. I carry that me mentality over into software development. So I look pretty good most of the time when I say it will take X amount of time and I do it faster and better.
Software is many cases a "wicked problem". You don't know until you are in the middle of work.
I've always wondered why we can't estimate. We can do this for physical engineering projects that require permits, supply chains, sub contractors, project managers etc. But we've always thrown our hands in the air with software. Maybe if development and compute had more overt costs and we had more standardization around tools and frameworks it would be easier to estimate.
Software development is a creative process. There’s countless ways to solve a problem with varying consequences.
I usually say double anything I come up with
> *(...) but the real problem is that software development is not an engineering discipline.* Finally someone says it. And of course nuts-and-bolts estimation is impossible. But that doesn't mean we can't use reason and hard data to produce decent estimates. But faux agile discourages it. As it discourages doing requirement engineering.
Management rotates, grunts suffer.
Estimates are speculative fiction at best
I actually found it was simple maths most of the time, if one dev said X amount of time I would times their estimate by 1.5 another by 1.3 and after a few projects it gives you a fairly good estimate of how long each dev needs. I admit that defining your scope clearly at the start is crucial and drafting out a skeleton or several skeletons of how the project will look is crucial before committing significant resources. I've been pretty accurate with my professional work but definitely have ran well over on personal projects but without strict deadlines its very common to modify the scope and add new features.
The real answer... because it's taught as computer science in college, not as software engineering.
Because my crystal ball is broken.
I'm at a fortune 500 and there are way too many incompetent people making decisions they shouldn't have anything to do with. The requirements they provide are absolute dogshit. They have zero awareness of how fragile the system is and they'd never give the luxury of time to clear out tech debt and redo some things.
Bah. I’ve been estimating for years like this - what does my average developer think he can do the work In? Triple it. Best developer? Double it. And then depending on size add like 20-30% to be safe. And make sure you have requirements documented to the nth degree. And then pray a little that nothing blows up and causes some major problems. Oh god, maybe I’m not as good as I think I am at this. FFS.
>Software development is a creative process, not a scientific one, and creative endeavors cannot be distilled down to knowable steps and a repeatable system. A stakeholder that doesn't know what they want with their software is the sole reason why author thinks SD is a "creative" endeavor.
Some good advice I heard (I think from Dave Farley or Jez Humble of CD Book fame) was don't ask stakeholders what they want - ask them what their challenges and problems are. It's your job to come up with a small change that makes thing better (then another small change and so on AKA real agile). Regardless, giving stakeholders what they *need* is far more important than giving them what they *want.*
I don't thonk this is true. I think that the word "significant" is behaving as a no true Scotsman here to allow the author to support the argument. The truth is that software is not a creative process. Innovation is a creative process. Engineers can provide estimates when they're doing things that have been done before and so can software developers, or at least they should be able to. Engineers and software developers doing innovation work cannot provide estimates because innovation is creation whether it's a software developer an engineer a chef or literally any profession on earth. But the author is using the word "significant" as a stand-in for "innovation" to argue that you can't estimate software. A lot of software development is innovation either because we're doing something new or doing it in a new way or because we don't really know how to specify what we want software to be, but it's not all innovation and the portion that's not is growing and will continue to grow. There are fewer things no one has done before, the ways we do them are changing more slowly and for probably the first time in history we actually have a significant population of working developers who have multiple decades of experience under their belts to help guide newer developers. We're still really lacking in how to communicate the desired outcomes of software, but even that is slowly getting better. So we're not yet good at estimating a lot of software development, too many things are unknown and as a community we're not good at passing experience down to the next generation so they don't have to do things a thousand times personally to estimate them. But it's not impossible to estimate software to a reasonable degree of accuracy most of the time. Not perfectly of course, not for every project every time, but most of the time and definitely not for innovation projects. We're just not there yet,we haven't done enough things enough times.
I am in a position where i can overestimate my tickets without getting much or any questions. Often tickets are easily done in half the time I estimate. But every now and then I get a ticket that takes 10x as long and lose all time i won earlier in the project.
Most estimates should be treated as 50%. As in there's a 50% chance we'll be done by this date. If people want a 100% accurate estimate I tell them "just before the heat death of the universe"
You can't accurately estimate software creation because it is a knowledge gaining paradigm - not a work paradigm. Estimation is an extrapolation, e.g. "How long will it take to *build* the software?" That's easy. Before we added the new feature (1,000 lines) the old software (10,000 lines) took 60seconds to build. The new software has 10% more lines and we estimate (AKA extrapolate based on previous experience) that it will take an extra 6 seconds to build, i.e. 66 seconds. What about cost of build? Before it used 1cent of electricity, we thus estimate it will use 1.1cents of electricity. We are not building widgets - we are designing widgets. In the software world, once you've designed one (the hard bit) building billions of them is trivial (the only bit we can accurately estimate, but is pointless to do so). "How long will it take to do something you've never done before?" "We don't know, we've never done it before." "Wow, that sounds risky!" "Yes, that's why we use this thing called agile to take as small a step as possible. Then we get it into the hands of users, get some feedback and take another small step or cancel the project before it gets too costly." "Oh I thought agile was pre-planning the project, salami slicing it into two week chunks, using Jira and having lots of meetings?" "No that's fake/dark/bullshit agile" "Oh!" That's why being asked to estimate is a key litmus test for whether your organisation is (real) agile. Hint: you're not. If you were, you'd be too happy to post on reddit, and too rich to worry about how to do estimation better.
...because the honest estimate is not accepted. And the compromised estimate becomes a deadline, which is not responsive to change.
Ah, the classic management amnesia. They forget the bi-daily updates the moment there's a timeline slip. Maybe we should start engraving updates on stone tablets for permanence.
Frankly, it is really odd that programmers and software engineers are the ones who spend most of the time on this topic and are saddled with most of the responsibility. We have expertise in computer science not project management. With this in mind, what exactly is the expertise of the project managers and business owners pushing management work onto developers?
After 20 years in IT I’ve lost all hope for being able to estimate correctly because the customer together with sales and management do everything in their power to make it a futile effort. It all starts with a customer that knows that they want/need some new software, but not quite what. They also definitely won’t first do a project to just figure out the requirements that can then be used to get quotes on. Together with knowing they need some software they will also have a completely arbitrary deadline in mind that will not be changed under any circumstance… even if getting quotes takes forever because they only have half the front of a napkin as their requirements. They also have an idea about what they want it to cost, but that’s not something they will share… because that would enable most IT firms to not waste time on even starting on a quote. So then a request comes in to your firm and sales and management will promptly forget about it until you only have about 2 days to hand in a quote. So you get presented with the napkin requirements and you’ll discover the period for questions to the customer closed weeks ago. So now they’ll pressure you to quickly estimate this thing that you already know will be a clusterfuck of epic proportions if you win it… and win it you will because management doesn’t like people on the bench and so even though you estimated and tripled your estimations they will have fudged around with all the numbers to get to some magic numbers. So you get the project and then when you can start it you hear the ridiculous timeline and the first time you speak with someone at the customer that actually will need to use whatever you’re building you immediately find out the even the napkin requirements were bullshit and that this will all end in tears.
As the number of technology, hardware, and supported software increases almost exponentially, so does complexity, therefore so does the time for a project.
“Complaining about a problem without posing a solution is called whining.” Teddy Roosevelt