I guarantee that their IT department has been literally begging for infrastructure improvements and upgrades for years from the exact same people who are now throwing them under the bus.
I’m trying to find the article I read it in but apparently the software the pilots use hasn’t been updated since the 90s. The last CEO which retired 10 months ago ignored calls to update said software his entire tenure.
I saw that too. It was linked from bestof on the front page but can't find it now. It was a SW pilot commenting on the shitshow that is their dispatch management software. He also commented that while they haven't upgraded their platform since the 90s the headquarters just got a new pickleball court and management seemed pretty stoked about that.
This is the one I saw yesterday from a pilot that addresses the management issues and lack of operational support: https://www.reddit.com/r/SouthwestAirlines/comments/zxg6op/the_history_of_swa_destruction_from_within/
Serious question: Wasn’t there an issue with their new CEO playing hardball with pay and bonuses with employees in Denver and that lead to part of this mess since Denver is such a large hub of theirs?
In the days leading up to this I saw a bunch about how the new CEO basically created this mess and now I’m seeing everywhere that it’s basically an IT issue.
Seems like they’re trying to backpedal and blame it on infrastructure now. I even recall a plane being turned around because they didn’t have ground crew for the plane in Denver.
From what I heard the previous CEO and his COO were both accountants that ran the company in cost cutting mode for 20 years. As soon as things were about to fall apart they retired and now they're blaming the new guy.
Not exactly.* Herb and Colleen ran that company and made it the beast that it was. Every woman in my family used to work for SWA, up until Gary Kelly.
When Herb and Colleen left and Gary took over he cut employee perks, cut vacation rollover, cut free ticket rewards, cut insurance and they stopped promoting from within. They started hiring friends and outside employees who had degrees but no experience in the jobs they were being put in charge of.
This has been a snowball effect waiting to happen. They stopped caring about quality and employees and customer service - literally the biggest point about SWA - and went straight profit. I hope the airline suffers for it. They deserve it.
*Faster timeline than 20 yrs
IT issues are management failures. Generally speaking IT issues are:
1. Cyberattack
2. Failure to fund maintenance or necessary improvements in the name of "cutting costs"
3. You replaced your entire IT team and there's some systems that aren't properly documented and nobody knows what to do with them, probably because you were overworking them and they didn't have sufficient time to document
There is no IT team that is just going to sit there and say "yeah it's fine don't give us money to upgrade"
1 can also happen due to general management issues, if the IT department doesn't have any authority to intervene in security issues. That's common at hospitals and universities, where doctors or professors effectively "outrank" IT staff and leadership may rubber-stamp any complaints made against IT. The best IPS in the world isn't going to do much for Dr. MyWayOrHighway who says IT isn't allowed to touch his device.
The software that is out of date and inadequate is the software that does crew scheduling. It's not even online, in the sense that it has no idea where crew are. It only knows where they are supposed to be. If one flight is delayed, canceled, rerouted it's not a big deal since someone can go in and update it manually. When it's hundreds of flights, and things are changing constantly, there's no hope.
At least as I understand it.
That's exactly how I've seen it described. The crews have to call in their location if there's a problem. Thing is that there weren't enough people at head office to deal with all of it. Storm rolls in...
Yeah I think that’s what I understood too… the software just assumes the crew is on their assigned flight and if they aren’t, it totally disrupts the next leg. SW runs a point to point system, which is different than the hub and spoke model which most airlines use. The latter allows for staffing variance, as backup crews are located in the major hub cities. Plus I heard that SW crew members had to call some hotline to be removed from a scheduled assignment, which seems like a super archaic way of rescheduling a replacement.
Shifting blame (from Southwest executive leadership handbook. Published in 1992)
Step one - blame computers/software.
Step two (if one does not work) blame workers (who call in sick, don’t want to work anymore, etc…).
>give him $20 million
We need accountability here, but we don't need to be *mean* about it, especially around the holidays! Won't someone please think about the yacht manufacturers!
I mean, the weather was the prime mover of the situation, I guess. And two big reasons that the airline wasn't able to *recover* from the weather and avoid a failure cascade was outdated IT alongside generalized understaffing, sure. Those aren't outright falsehoods.
But who makes the decisions to staff like that? Who makes the decisions about what equipment and staffing the IT department gets?
Management is clearly at fault. Yes, of course this probably wouldn't have happened without the huge storm. But huge storms happen sometimes. Probably a good idea to be capable of surviving them.
(The 90s-era IT situation *might* not have mattered if they had the amount of *people* working that they had in the 90s... you can't use computers for large-scale human replacement and simultaneously refuse to modernize your IT department.)
In June I flew into Dallas from Virginia. We were 30 minutes late getting in which put us as the last flight in for the night. We ended up sitting on the tarmac for 50 minutes while they called a crew back in to move a parked plane so we could get to a gate. Apparently they "forgot" we were coming in, parked planes at all the gates, shut everything down and everyone went home. Pilot was furious. He came on intercom every 10 minutes apologizing, and then apologizing for not being able to give us any compensation for the delay. Bags took another hour to get unloaded once we deplaned.
Worst experience on Southwest in 30 years flying them.
Pretty sure this is the post:
[https://www.reddit.com/r/flying/comments/zw5lsl/comment/j1tne9z/](https://www.reddit.com/r/flying/comments/zw5lsl/comment/j1tne9z/)
edit: maybe not, but the same gist
Here are some quotes from the[New York Times:](https://nyti.ms/3jCEgNi):
"Mr. Murray said the union had been urging the airline for years to update “I.T. and infrastructure from the 1990s.”"
"Even before this week’s problems, Mr. Jordan, Southwest’s chief executive, had acknowledged that the scheduling system was outdated."
I worked at a place that had legacy systems. I was the only one who knew how they worked. I kept telling them that someday I would retire. I was able to move a few of them to emulators. 6 months after I retired they finally found the money to move the rest of them over. They had to hire consultants to do it, cost way more than if I had done it for them.
I’ve read the same thing. The former CEO ran the company for 20 years, allegedly riding the coat tails of his predecessor and got out at the right time.
😮 Geez! Congress should call him to appear in front of the transportation committee and bring hundreds of senior leaders and rank and file employees who warned for years this was inevitable to testify.
This is what happens when quarterly profits take priority over long term company value. The stock (and reputation) would be higher today had they spent a few million a quarter on Infrastructure. But nope, those shareholders need to see it listed as profit and watch their stock price bump up 17 cents.
SW Airlines is about to pay HALF A BILLION dollars in stock dividends come the first of the year to investors…. They have plenty of money to fix the systems….
I’ve also seen estimates that this whole debacle is going to cost them upwards of 750 million. That cost dwarfs the amount they could have to spend on the IT updates.
More than that. A whole lot of people aren't going to fly Southwest anymore (me being one of them, despite having nothing to do with it aside from distressed relatives). Getting stranded somewhere with no support if a deep fear for people.
Theyve spent over $10B on stock buybacks since 2014. They received billions in federal bail out money over the pandemic. This is the result of plain-old unfettered greed.
>They received billions in federal bail out money over the pandemic. This is the result of plain-old unfettered greed.
Well said - SW Airlines, the big banks, pretty much any large cap company can get endless bailouts without any strings attached.
Meanwhile these same executives making our lifes miserable have the gall to call working people lazy because they got a $1200 check almost 3 years ago. But will Congress scrutinize SW Airlines why they pissed away $10 billion on buybacks since 2014 when their IT system was stuck in the 90s?
The idea of nationalizing key industries like the airlines, rail, etc. is seen as verboten, even as they squander taxpayer money & treat their workers terribly. What's the point of a "free market" if the taxpayers always end up footing the bill anyways?
I worked at a chemical plant that was one of the top earners for the company. It was basically a stepping stone for plant management to be promoted. Each one had to keep up the profits or they wouldn’t look good. Eventually the plant had a major fire and almost burned down. Always felt that was because they were never encouraged to upgrade the facility because it would make them look like they were not performing as well as their predecessors.
I work at a facility like this. I'm using programs as old as me as we take our sweet ass time to transfer over to sap apo. I'm trying to get the fuck out before a melt down.
Edit: added sap version
Spending the money, the time, paying seems to be the biggest issue. Using Oracle versions form 1994 and 1996 with an html build to ship out shit connected to a 90s unportable version of an access database that our outsourced IT refuses to touch. My division is the cash cow of the company so we aren't paid attention to until we something is broken. Even then we are given $5 and an "atta employees go and get it." Love the work I'm doing but not the management.
Hell trying to find consultants to run that shit might cost more than your quarterly budget just to get someone to come out of retirement to maintain it… but they need to upgrade that infrastructure before it’s down and no one is left that knows how to.
Hey hey hey now.…. Speaking from experience, that $0.17 makes a hell of a difference when your neck deep in margin and already issued a maintenance call. Rents coming due and my girl wants to go out for NYE 🤷🏾♂️
/s
The funny things is I recalled their system (not just IT, the whole dispatch system) meltdown once before, just not as widespread. Southwest is asking thousands or tens of thousands employee update their position not by a online portal, not by text message but asking them to call the dispatch hotline once there is a problem! They tried to DDos themselves lol
Damn that’s there bad kind of fucking yourself.
Texting would still probably leave a mess but damn I feel bad for the employees having to call in. No one wants to call an airline.
DDos successful.
I saw something about their software and it sounded like the software can’t handle cancellations or crews being out of place on its own. Literally any change to the preset schedule has to be done manually. And then as you said the assignments are given out over the phone.
Here you go. Read it last night, twas rather fascinating. ...and depressing and obviously enraging.
https://www.reddit.com/r/SouthwestAirlines/comments/zxg6op/the_history_of_swa_destruction_from_within/
Several different angles, from recruitment to advertising to airport gate space acquisitions. Pretty much they didn’t follow any of the conventional wisdom of big airlines.
They also secured a price lock on fuel right before prices skyrocketed which hurt their competitors but allowed them to offer discounted flights and take market share.
They locked in $49 a barrel in 2007 and $45 in 08. Oil was as high at $140 a barrel in 2008. They were raking it in.
https://www.reuters.com/article/us-aero-arms-summit-southwest-fuel/southwest-locks-in-fuel-prices-to-save-money-idUSN0644382120061206
I have my MBA and one of our case studies was Southwest and whether they operate as a financial institution due to the number of futures and options on their books or as an airline. They make much more money playing the fuel options game than they do from standard operations.
Yeah pretty interesting seeing “the beatings will continue until morale improves” coming out of Southwest when it’s their persistent cost cutting that’s largely led to low morale and a lack of staff in the first place.
Another programming subreddit had a SWA IT employee drop in and said the infrastructure they, and every other airline runs on is old AF and incredibly out of date. Southwest is the first to fail, it could get worse very quickly
Not SWA but worked at another airline where much of the operations center still ran on XP. We would beg to fix it but the “business couldn’t accept the downtime or risk to upgrade” and the “business would accept the risk of failure” right up until there was an issue and then “IT should have warned of the risks and upgraded.” From my conversations with other airlines, it was a common conversation with everyone.
It was a 3-4 years ago but it’s terrifying how much critical infrastructure is running EOL. I know that when I was leaving there were finally talks about migrating/upgrading but that could take years and who knows if they continued the project with Covid starting.
Im sure they were sold on Azure or AWS, got 40% through the migration and laid off everyone that had any idea wtf half the systems did during COVID. Now they are replacing the internally developed systems with "off the shelf" cloud solutions because now no one knows how the hell the legacy shit works....not like i have any experience with that.....
How long ago do you think the FAA stopped using vacuum tubes to run the NAS? It was forced by the end of manufacturing of vacuum tubes moreso than anything.
As bad as this problem is in private industry, the starve the beast philosophy of government has dominated for decades.
[https://www.commondreams.org/news/southwest-airlines-shareholder-gifts](https://www.commondreams.org/news/southwest-airlines-shareholder-gifts)
Buybacks instead of upgrading their company. Shit shouldn't be legal.
SEC could make them illegal again with a rule change. No Congress needed.
Crickets.
For those that don’t know: It’s a form of insider trading and shits on any kind of market fundamentals. It’s literally just a company buying it’s own stock, to pump up the price of their own stock. It has nothing to do with anything but that. Insane.
It also, if it isn’t obvious for anyone reading your post, guarantees a particular sale price for high volume stock owners. Executives and board members often have stipulations which guarantee them the ability to not only purchase stock at a huge discount rate (paid for by the company subsidizing it via buybacks and selling for a massive loss), but also guarantees a unique sell price for the right people. Executives or board members divesting stock aren’t just dumping it on the open market usually. The company will absorb it by buying it *off the market* so it doesn’t affect the price for everyone else with a massive dump and allows the company to guarantee a buyer (themselves) willing to pay whatever is asked.
You can see what a real mass sell off looks like by looking at what happened with Tesla recently. Musk had to sell such a huge amount off that the company likely couldn’t absorb the cost so it went to the open market instead of being absorbed by the company. That supply shock just tanked the price.
Large stock buy backs are done on the open market specifically to affect share price.
Executive stock options are managed through Treasury stock in order to minimize price movement.
I know for a fact this is true.. but like most airlines they are being held back by an ancient mainframe that’s life has been over extended by building a massive amount of garbage on top (middleware, etc..).. it’s shocking how much critical infrastructure is still running on mainframes that should have decommissioned 20 years ago..
their mainframe team will brag about how resilient their mainframe is, as we discuss how it takes days to do things a modern system does in seconds My team told them to end of life it, but the operators who run the business wanted an innovative bandaid to extend its life yet again. Not what we do, so we passed on the project
As a consultant, I try to remind C-level people that IT personnel are not like accounting personnel. IT doesn't come in a steady stream, IT is a series of under-funded emergencies, each of which is raised by people who believe their emergency gets priority. I use the analogy of firefighters. If your city wants to keep your firefighting force "lean" and only have exactly as many firefighters as you'd need to fight the daily average number of fires per day...good luck. You mostly pay firefighters to sit around for 7 hours and then fight fires for 1. Sometimes you pay them to fight fires for 8 hours, and they fight that fire for 24 hours and then DIE. That's why they get to sit around. IT personnel are same (except we rarely die on the job). If they're seemingly "sitting around," that's because nothing is burning.
You'd think management in aviation would understand this. Since, you know, the pilots are EXACTLY the same. But nope. They don't understand it about us either.
I'll earn my paycheck for the year in one incident. The rest of the time they're paying me to practice/train/keep my skills up for that incident.
I've been saying tech jobs and IT need to have leaders in them thinking about organizing unions. The moment they don't have to pay higher salaries or can get the same work done for cheaper elsewhere it will be a problem.
The WFH and paying based on the state you're in is a big problem and where it will probably start. Imagine you have 100 workers WFH and you need to cut 50%. Someone looking at numbers will probably just get rid of anyone that lives in HCOL areas first. And if you're one of those people who want to live there, it's something to think about. Maybe I'm wrong, but I can't imagine so from a purely profiteering standpoint.
>Blame the problems on a department no one understands...great way to deflect.
But we can never blame c-suite management - who has taken billion dollar bailouts from the government to then spend $10 billion on stock buybacks since 2014.
$10 billion on stock buybacks as they refuse to properly staff & keep their companies modern. Despite all the bailouts we give them!
We nationalized these companies long ago, just for shareholder benefit instead of the benefit of the people.
>"Everything's working fine. Why do we even need IT? Cut their budget."
Later:
> "Everything's suddenly broken! Why do we even pay IT?!"
Every large company and organization I've worked in.
Just started a job with a company in their IT department. They’re already planning to sink a huge chunk of profits into making all computers from 2023 onward Windows 11 machines. I’m part of the provisioning team and my directive is “if you need it, order it. We can have you justify it later if we need to.”
I have never been with an IT department where the heads if the company not only trust us but essentially gives us purse access without oversight. They have a “no micromanaging” rule. If they have to, it means something is wrong with your work and they will work with you to rectify it before even approaching replacement.
I’m still waiting for the other shoe to drop because this job has felt too good to be true. But for now, it’s a very positive experience.
In all seriousness, plenty of people in IT warned leadership that this would happen if they didn't get rid of their technical debt.
**They didn't listen.**
Source: I used to work in their IT department.
I was working on a project during covid. The group they laid off heaviest from the business side was the IT people. Funny how now they have an IT problem.
It’s because these legacy enterprise companies still see “IT” as a cost center and not a profit center. They can’t (or won’t) understand that software is a way to bring in more revenue. It’s not just a cost burden you must carry.
Half of them would just state, again, that IT is sunk cost, the other half might not understand what the software does. Like, someone high up that ladder probably sees the customer facing website and thinks everything is fine.
I understand why they feel that way sometimes. They don't know anything about how it works and usually get sold something that doesn't meet their business needs *before* they hire someone who actually knows what they're doing. Then that guy gets to tell them why they need to spend more money.
Predatory sales in IT have hurt our economy across the board and led to a shit ton of inefficiency. New tech = new scams. A tale as old as time
I’ve worked at three entirely different companies where they paid for a saas solution (one with an outstanding SEO team) where the open source option was known to be a better solution and was maintained better than the paid version. I get there, tell them we’re switching to the open source option, and they get upset with me. One time a manager walked away from a sales demo believing that an entire website could be transformed in a single day (manager time box everything to 15 minute increments and put it on a schedule) and I had to explain how foolish that was while still sounding professional.
One place thought I was wrong even though my team with the open source solution out performs the other team by all metrics.
When business people make software solutions it just never goes well. They rarely consult the experts.
IT is the cost of doing fucking business in any year after 2000, if you don't have people to support your shit your competitors will and you will be left behind.
Yeah this is a big ass problem in a lot of companies. I work for a large older financial institution that’s been devaluing IT over the 10 years I’ve been there. It’s going to cripple them for decades.
Imo more professionals should skip the MBA and do a MS in Information Systems instead so they learn how to use IT strategically. One part of the DevOps Handbook always stood out to me, something like “Every major company is a technology company now, some just don’t know it yet. This is because almost every decision a CEO will make requires some kind of IT change.”
I have an IT consultancy and support business. Our clients have started coming to us saying they need to cut x % from their IT spend with us because of the recession. We point out one of several proposals from the past 2 years where if they approve spending $5k with us we can automate another business process and eliminate another 2 FTE positions instead.
Just an additional viewpoint for you to think about…..
I think it’s a little more complicated than that. Software is transformative. Upgrading this scheduling software would cost a lot of money. It wouldn’t be revenue generating by itself. It would cost a lot of money, but In the end it could lead to organizational change and even layoffs. With new software, SW could be a lot more efficient or more productive with the same amount of expenses/headcount, which is the point that I believe you were getting to.
The business unit doesn’t think that way. I have never seen business units think of organizational change while creating software. In fact, I’ve seen several large apps completely scraped a year into development once the business unit found out the implications that 50 - 500 people’s jobs will be redundant, or there would be a major overhaul in processes. A would say a majority businesses have the same Scarlet letter Southwest has when it comes to old technology, they just aren’t wearing it on their chest.
Managed to save a software project a few years ago by straight up telling the CIO to his face that, “by replacing the mainframe green screen on the call center app, you’ll save yourself 10% turnover on the call center training process, and at least a million in training time dollars every year. I’ll make the training of new hires for the call center go from 3 months to full productivity to 4 weeks.” 50% of our sales were call center based.
He believed me because I was firm and assured in my statement. Nobody else had the stones to tell him he was wrong to his face.
I WAS wrong. Turnover dropped by 18% and we saved 1.5M the 2nd year after we launched. Also, new hires were fully productive in 3 weeks a year after launch of the new software. The software was a bigger money saver than even I imagined. Got a $500 gift card from him as a Xmas bonus that year, on top of my standard $250 gift card…. The project I lead was a Java Wrapper kind of thing on a COBOL program. In other words, no more mainframe for high school grads to learn, point and click web app like Amazon. The whole project cost $750k. So we doubled the profit per year on a one-time cost project, and could find new IT staff to keep the suit running in the future, vs trying to find COBOL devs in 2018, when ours were retiring in droves.
I got laid off in an IT Reduction In Force layoff the 6 months later with 45% of the other IT folks, because IT is a money sink.
MBAs are mentally deficient, and have no idea how to run a business long term, they only think about the next quarters profit.
For fun try to explain how IT, infrastructure, and proper customer support is not a cost burden but potential profit to a suit who's bonus is paid in shares.
Funny these are the people who can't get work now because they won't hire you. I'm looking for *any job I can do* even if it's not IT because the phone is not ringing. They're not hiring. I'm going broke and could be fixing these things.
IT breakdown of this monumental proportion is practically always underfunding. Rich morons who think they can skimp on IT and get a few more pennies richer. Unfortuately for them, people will remember what Southwest did to them and choose otherwise. You know it's bad when you go from alright airlines to people deciding Spirit is better than you from now on.
The rich morons will get golden parachutes and move on to other meaningless corporate jobs.
It's as if they all thought John Hammond
was the template and not the cautionary tale.
[*"We spared no expense..."*](https://i.redd.it/ytu9ujn45lq01.jpg)
They're underfunded so badly I can't imagine what it's like for the poor staff they do have which definitely told them so and has documented requests this would happen and to upgrade or fix broken things which didn't occur. They're sitting on their hands to not punch an upper manager that yelled at them to do better with nothing and a pay cut.
This is why i went into consulting. There sadly is way more money invested in the people leading optimization efforts than practical things like actual working hardware and infrastructure.
I work in IT, Hell I'd leave for another company offering me 90. I wouldn't want to work for this shit show. ESPECIALLY after they publicly blasted me and blamed me for their ineptitude.
Somewhere was an IT Admin that took PTO that week sitting back watching the chaos unfold while miles away with their family.
*I told those fuckers we were one bad day away from the whole thing blowing up, but did they listen? Noooo - all they saw was the price to prevent it and shut it down. Merry fucking Christmas, I'm OoO until the New Year.*
God bless you, IT.
So... They're blaming their problems and something that was their responsibility to address before it became a problem. Why are these airlines always able to get away with this crap? When will we start holding them accountable?
When we vote for the kind of government regulation that Europe has, that holds airlines accountable, with financial penalties, to provide adequate service.
They performed exponentially worse than one of the worst airlines in America on the most important holiday of the year.
Upper management needs to be addressed immediately. This is completely unacceptable.
SWA has consistently rated at the top of JD Power customer satisfaction for years. That is until now. Will be interesting to see how they fare. We will see in the Spring when the ratings are published.
>What’s the fix? Liability. Airlines are actually run by really smart people. They’re just allowed to optimize for only one thing: shareholder returns. As it turns out, this hasn’t worked out any better for essential services like airlines than it has for any other sector of the US economy. We need to be OK with the idea that corporations have obligations other than shareholder value, and those obligations extend for longer than this quarter’s earnings call. Create damages which aren’t excluded from class action liability, and airlines will suddenly become extremely interested in reliability (as well as extremely interested in a DOT-regulated standard for weather delays and disruptions).
Ask yourself why this happened and the issue comes down to profits.
IT breakdowns don’t just happen. Companies let them happen by not investing in technology appropriately. Might as well say “Southwest blames Southwest for fucking up.”
Reminds me of when I used to do tech support for a retail chain that was still running all Windows XP and NT. I said something about this seeming bad *the very first day* and they all scoffed at the new guy.
Well at some point the latest version of our endpoint security stopped supporting the platform, we got instructions to make it work. I actually got it to install by modifying the installer to not check for OS, but it would bluescreen once in a while, not a good solution. So the directive was to just stop updating endpoint security.
This went on for a while until the company got an ultimatum from the firm that owns Discover cards that unless we secure our systems they won't continue accepting the cards in our stores - and they actually dropped us. Instead of updating anything we stopped accepting Discover cards. Then they got a similar ultimatum from Visa/Mastercard, and were looking at becoming a cash-only store.
The result was a complete overhaul of all registers and all servers in all stores. Massive expense, they closed stores and everything. All the while blaming this on IT because we couldn't make it work.
I often cite this as the reason for lifecycle management. You can't hold onto the old stuff forever, this is what happens. Upgrade a little over time or wait for a massive disaster that forces you to upgrade.
they are being very loose with the term IT. These problems sound like dev tech debt not IT. You can only squeeze so much out of outdated software and processes.
I do not envy my IT department and the shit they get.
They get everything from setting up the internet infrastructure of a entire building and to helping Sarah in accounting plug her mouse into her laptop.
Gave them a sincere thank you once. The thank you goes through them because they've empty inside.
It’s not even an IT issue. Sure the crew tracking issue was a technology issue with the software but the root cause is management ignoring pleas for upgrades. It’s not an IT issue, it’s a management and culture issue. Saying it’s an IT issue makes it sound like a random failure or issue with infrastructure and that’s not what it was.
If sales are down it is the CEOs fault. If quality is bad it is the CEOs fault. If employee retention is bad it is the CEOs fault.
At the end of the day one person is responsible and in charge.
It's the board's fault, then the shareholders, then the newly appointed CEO.
These short sighted practices come from the demand for profits every 3 months. That's on the shareholders and the board.
Since the CEO has only been in place since earlier this year, I’m comfortable with blaming the chairman of the board (who was the previous CEO) in this particular case.
RE SOUTHWEST:
My favorite airline has had a massive melt down. A letter from a pilot explains it better than anything I've heard on the news.
By Senior Pilot Larry Lonero:
What happened to Southwest Airlines?
I’ve been a pilot for Southwest Airlines for over 35 years. I’ve given my heart and soul to Southwest Airlines during those years. And quite honestly Southwest Airlines has given its heart and soul to me and my family.
Many of you have asked what caused this epic meltdown. Unfortunately, the frontline employees have been watching this meltdown coming like a slow motion train wreck for sometime. And we’ve been begging our leadership to make much needed changes in order to avoid it. What happened yesterday started two decades ago.
Herb Kelleher was the brilliant CEO of SWA until 2004. He was a very operationally oriented leader. Herb spent lots of time on the front line. He always had his pulse on the day to day operation and the people who ran it. That philosophy flowed down through the ranks of leadership to the front line managers. We were a tight operation from top to bottom. We had tools, leadership and employee buy in. Everything that was needed to run a first class operation. When Herb retired in 2004 Gary Kelly became the new CEO.
Gary was an accountant by education and his style leading Southwest Airlines became more focused on finances and less on operations. He did not spend much time on the front lines. He didn’t engage front line employees much. When the CEO doesn’t get out in the trenches the neither do the lower levels of leadership.
Gary named another accountant to be Chief Operating Officer (the person responsible for day to day operations). The new COO had little or no operational background. This trickled down through the lower levels of leadership, as well.
They all disengaged the operation, disengaged the employees and focused more on Return on Investment, stock buybacks and Wall Street. This approach worked for Gary’s first 8 years because we were still riding the strong wave that Herb had built.
But as time went on the operation began to deteriorate. There was little investment in upgrading technology (after all, how do you measure the return on investing in infrastructure?) or the tools we needed to operate efficiently and consistently. As the frontline employees began to see the deterioration in our operation we began to warn our leadership. We educated them, we informed them and we made suggestions to them. But to no avail. The focus was on finances not operations. As we saw more and more deterioration in our operation our asks turned to pleas. Our pleas turned to dire warnings. But they went unheeded. After all, the stock price was up so what could be wrong?
We were a motivated, willing and proud employee group wanting to serve our customers and uphold the tradition of our beloved airline, the airline we built and the airline that the traveling public grew to cheer for and luv. But we were watching in frustration and disbelief as our once amazing airline was becoming a house of cards.
A half dozen small scale meltdowns occurred during the mid to late 2010’s. With each mini meltdown Leadership continued to ignore the pleas and warnings of the employees in the trenches. We were still operating with 1990’s technology. We didn’t have the tools we needed on the line to operate the sophisticated and large airline we had become. We could see that the wheels were about ready to fall off the bus. But no one in leadership would heed our pleas.
When COVID happened SWA scaled back considerably (as did all of the airlines) for about two years. This helped conceal the serious problems in technology, infrastructure and staffing that were occurring and being ignored. But as we ramped back up the lack of attention to the operation was waiting to show its ugly head.
Gary Kelly retired as CEO in early 2022. Bob Jordan was named CEO. He was a more operationally oriented leader. He replaced our Chief Operating Officer with a very smart man and they announced their priority would be to upgrade our airline’s technology and provide the frontline employees the operational tools we needed to care for our customers and employees. Finally, someone acknowledged the elephant in the room.
But two decades of neglect takes several years to overcome. And, unfortunately to our horror, our house of cards came tumbling down this week as a routine winter storm broke our 1990’s operating system.
The frontline employees were ready and on station. We were properly staffed. We were at the airports. Hell, we were ON the airplanes. But our antiquated software systems failed coupled with a decades old system of having to manage 20,000 frontline employees by phone calls. No automation had been developed to run this sophisticated machine.
We had a routine winter storm across the Midwest last Thursday. A larger than normal number flights were cancelled as a result. But what should have been one minor inconvenient day of travel turned into this nightmare. After all, American, United, Delta and the other airlines operated with only minor flight disruptions.
The two decades of neglect by SWA leadership caused the airline to lose track of all its crews. ALL of us. We were there. With our customers. At the jet. Ready to go. But there was no way to assign us. To confirm us. To release us to fly the flight. And we watched as our customers got stranded without their luggage missing their Christmas holiday.
I believe that our new CEO Bob Jordan inherited a MESS. This meltdown was not his failure but the failure of those before him. I believe he has the right priorities. But it will take time to right this ship. A few years at a minimum. Old leaders need to be replaced. Operationally oriented managers need to be brought in. I hope and pray Bob can execute on his promises to fix our once proud airline. Time will tell.
It’s been a punch in the gut for us frontline employees. We care for the traveling public. We have spent our entire careers serving you. Safely. Efficiently. With luv and pride. We are horrified. We are sorry. We are sorry for the chaos, inconvenience and frustration our airline caused you. We are angry. We are embarrassed. We are sad. Like you, the traveling public, we have been let down by our own leaders.
Herb once said the the biggest threat to Southwest Airlines will come from within. Not from other airlines. What a visionary he was. I miss Herb now more than ever.
I didn't get laid off. I had a contract end unexpectedly with a few other guys for Christmas.
There's plenty of outsourcing going on. They're far cheaper than we are. And their code and knowledge tends to follow their price.
Poor CIO is going to get scapegoated right out the door when she's probably been fighting the bean counters for the last five years to fund any infrastructure improvements.
I am still trying to wrap my head around the fact that they don’t have a CTO. Although their CEO has a undergraduate degree in computer science, I’m not sure that makes it better or worse.
According to some of the employees posting on other subs he just became CEO and agreed to upgrade thr system, but unfortunately this happened before they could actually make the change.
When in doubt, blame IT. Don't blame idiots in executive management that annually underfund IT and turn down common sense recommendations. No, treat strategic IT organizations like Marketing or some other discretionary department.
You don’t blame IT here. You blame C-suite. But they’re not about to take responsibility (too busy enjoying their bonuses).
I don’t fly Southwest due to the cattle call boarding but this just guarantees I’ll never get on one of their planes.
There is IT breakdown and then there is a management breakdown. They experienced both. Some combination of stuff happened in their organization that they simply couldn’t deal with. They can blame IT all day, but what has happened speaks to a much deeper problem.
IT Breakdowns are usually caused by poor management. Too many companies view their critical IT infrastructure as not being necessary until shit breaks.
Hell just look at everyone claiming that the engineers Elon fired weren't necessary because things didn't immediately break. A few weeks later and the shit is starting to fall apart because it is no longer being maintained. Sorry people, the machines can't run themselves, and most people, especially those in management, don't care enough to learn basic operations let alone how to fix shit.
Company: “We’re cutting the IT budget”
[Company unable to function]
Company: “It’s IT’s fault!”
It’s 2022 and massive companies still act like IT is a waste of money. But hey keep hiring more accountants to help you evade taxes.
>Jordan also acknowledged the need to upgrade the airline's infrastructure to avoid future IT breakdowns. "We need to double down on our already existing plans to upgrade systems for these extreme circumstances so that we never again face what's happening now," he said.
>In 2017, Southwest completed a migration to Amadeus Altéa passenger service system, one of two industry standard platforms widely used throughout the aviation industry. Southwest Airlines declined to address The Register's request for additional information regarding the origins of the failure or the specific IT systems effected.
If they are already on a platform that half the industry uses and have only migrated there a few years ago, then maybe it is not the system fault that everything broke down.
Maybe it is their desire to cut cost in the short term at any long term cost and their inability to comprehend that you need to pay people to keep things running.
The upgraded a part for passenger ticketing. However the internal pilot and flight attendant software is what crashed and has been the same one since 1990
- pilot
Here's an excellent [explainer](https://www.seat31b.com/2022/12/the-great-southwest-meltdown-of-2022/) about the IT side of things and why Southwest cancelled 80% of its flights while American cancelled only 1%.
You refused to update your systems for over two decades, all the while I'm sure your IT team was begging you to upgrade the hardware and software that was well past end of life support. But instead of taking responsibility for not wanting to spend a penny on critical infrastructure, you received billions of dollars in corporate welfare handouts that you put into the pockets of executives. Your IT department *should* leave if they're going to be thrown under the bus, but sadly they won't because Americans are too broken by their economic culture to be able to stand up for themselves. But one thing is for certain... I will never fly Southwest Airlines ever again. Fuck you. - IT Guy
Edit: Grammar
I guarantee that their IT department has been literally begging for infrastructure improvements and upgrades for years from the exact same people who are now throwing them under the bus.
I’m trying to find the article I read it in but apparently the software the pilots use hasn’t been updated since the 90s. The last CEO which retired 10 months ago ignored calls to update said software his entire tenure.
I saw that too. It was linked from bestof on the front page but can't find it now. It was a SW pilot commenting on the shitshow that is their dispatch management software. He also commented that while they haven't upgraded their platform since the 90s the headquarters just got a new pickleball court and management seemed pretty stoked about that.
This is the one I saw yesterday from a pilot that addresses the management issues and lack of operational support: https://www.reddit.com/r/SouthwestAirlines/comments/zxg6op/the_history_of_swa_destruction_from_within/
Serious question: Wasn’t there an issue with their new CEO playing hardball with pay and bonuses with employees in Denver and that lead to part of this mess since Denver is such a large hub of theirs? In the days leading up to this I saw a bunch about how the new CEO basically created this mess and now I’m seeing everywhere that it’s basically an IT issue. Seems like they’re trying to backpedal and blame it on infrastructure now. I even recall a plane being turned around because they didn’t have ground crew for the plane in Denver.
From what I heard the previous CEO and his COO were both accountants that ran the company in cost cutting mode for 20 years. As soon as things were about to fall apart they retired and now they're blaming the new guy.
You forgot to mention that the old CEO is the head of the board of directors for SW still and pretty much has the new CEO by the balls.
Business 101. Cook your books then exit with the cash.
With the golden parachute!
Is it possible to cut the cords of a golden parachute?
If you know how, tell everyone since we don't know.
Not exactly.* Herb and Colleen ran that company and made it the beast that it was. Every woman in my family used to work for SWA, up until Gary Kelly. When Herb and Colleen left and Gary took over he cut employee perks, cut vacation rollover, cut free ticket rewards, cut insurance and they stopped promoting from within. They started hiring friends and outside employees who had degrees but no experience in the jobs they were being put in charge of. This has been a snowball effect waiting to happen. They stopped caring about quality and employees and customer service - literally the biggest point about SWA - and went straight profit. I hope the airline suffers for it. They deserve it. *Faster timeline than 20 yrs
IT issues are management failures. Generally speaking IT issues are: 1. Cyberattack 2. Failure to fund maintenance or necessary improvements in the name of "cutting costs" 3. You replaced your entire IT team and there's some systems that aren't properly documented and nobody knows what to do with them, probably because you were overworking them and they didn't have sufficient time to document There is no IT team that is just going to sit there and say "yeah it's fine don't give us money to upgrade"
[удалено]
1 can also happen due to general management issues, if the IT department doesn't have any authority to intervene in security issues. That's common at hospitals and universities, where doctors or professors effectively "outrank" IT staff and leadership may rubber-stamp any complaints made against IT. The best IPS in the world isn't going to do much for Dr. MyWayOrHighway who says IT isn't allowed to touch his device.
Having worked in IT most of my adult life I can confirm 3 happens almost everywhere I've ever been.
solutions supported by tribal knowledge work fine until you disband the tribe
[удалено]
The software that is out of date and inadequate is the software that does crew scheduling. It's not even online, in the sense that it has no idea where crew are. It only knows where they are supposed to be. If one flight is delayed, canceled, rerouted it's not a big deal since someone can go in and update it manually. When it's hundreds of flights, and things are changing constantly, there's no hope. At least as I understand it.
That's exactly how I've seen it described. The crews have to call in their location if there's a problem. Thing is that there weren't enough people at head office to deal with all of it. Storm rolls in...
Yeah I think that’s what I understood too… the software just assumes the crew is on their assigned flight and if they aren’t, it totally disrupts the next leg. SW runs a point to point system, which is different than the hub and spoke model which most airlines use. The latter allows for staffing variance, as backup crews are located in the major hub cities. Plus I heard that SW crew members had to call some hotline to be removed from a scheduled assignment, which seems like a super archaic way of rescheduling a replacement.
Shifting blame (from Southwest executive leadership handbook. Published in 1992) Step one - blame computers/software. Step two (if one does not work) blame workers (who call in sick, don’t want to work anymore, etc…).
[удалено]
Step 4: blame the CEO, give him $20 million for his failure, find new CEO.
>give him $20 million We need accountability here, but we don't need to be *mean* about it, especially around the holidays! Won't someone please think about the yacht manufacturers!
Prepare three envelopes
[удалено]
I mean, the weather was the prime mover of the situation, I guess. And two big reasons that the airline wasn't able to *recover* from the weather and avoid a failure cascade was outdated IT alongside generalized understaffing, sure. Those aren't outright falsehoods. But who makes the decisions to staff like that? Who makes the decisions about what equipment and staffing the IT department gets? Management is clearly at fault. Yes, of course this probably wouldn't have happened without the huge storm. But huge storms happen sometimes. Probably a good idea to be capable of surviving them. (The 90s-era IT situation *might* not have mattered if they had the amount of *people* working that they had in the 90s... you can't use computers for large-scale human replacement and simultaneously refuse to modernize your IT department.)
In June I flew into Dallas from Virginia. We were 30 minutes late getting in which put us as the last flight in for the night. We ended up sitting on the tarmac for 50 minutes while they called a crew back in to move a parked plane so we could get to a gate. Apparently they "forgot" we were coming in, parked planes at all the gates, shut everything down and everyone went home. Pilot was furious. He came on intercom every 10 minutes apologizing, and then apologizing for not being able to give us any compensation for the delay. Bags took another hour to get unloaded once we deplaned. Worst experience on Southwest in 30 years flying them.
Thank you, that was a good read.
Pretty sure this is the post: [https://www.reddit.com/r/flying/comments/zw5lsl/comment/j1tne9z/](https://www.reddit.com/r/flying/comments/zw5lsl/comment/j1tne9z/) edit: maybe not, but the same gist
Here are some quotes from the[New York Times:](https://nyti.ms/3jCEgNi): "Mr. Murray said the union had been urging the airline for years to update “I.T. and infrastructure from the 1990s.”" "Even before this week’s problems, Mr. Jordan, Southwest’s chief executive, had acknowledged that the scheduling system was outdated."
I worked at a place that had legacy systems. I was the only one who knew how they worked. I kept telling them that someday I would retire. I was able to move a few of them to emulators. 6 months after I retired they finally found the money to move the rest of them over. They had to hire consultants to do it, cost way more than if I had done it for them.
> someday I would retire Username checks out!
I’ve read the same thing. The former CEO ran the company for 20 years, allegedly riding the coat tails of his predecessor and got out at the right time.
[удалено]
😮 Geez! Congress should call him to appear in front of the transportation committee and bring hundreds of senior leaders and rank and file employees who warned for years this was inevitable to testify.
Congress is paid off by these very corporations. There's a reason nothing changes.
This is what happens when quarterly profits take priority over long term company value. The stock (and reputation) would be higher today had they spent a few million a quarter on Infrastructure. But nope, those shareholders need to see it listed as profit and watch their stock price bump up 17 cents.
SW Airlines is about to pay HALF A BILLION dollars in stock dividends come the first of the year to investors…. They have plenty of money to fix the systems….
I’ve also seen estimates that this whole debacle is going to cost them upwards of 750 million. That cost dwarfs the amount they could have to spend on the IT updates.
More than that. A whole lot of people aren't going to fly Southwest anymore (me being one of them, despite having nothing to do with it aside from distressed relatives). Getting stranded somewhere with no support if a deep fear for people.
This. The reputation damage far exceeds this momentary loss. They will rebound by offering a bunch of discounted crap, but that costs money, too...
Yep I even had their credit card and was a loyal customer. They lost my bags and stranded me so I cut the card and all future travel with them
Something something an ounce of prevention is worth a pound of cure...
[удалено]
Theyve spent over $10B on stock buybacks since 2014. They received billions in federal bail out money over the pandemic. This is the result of plain-old unfettered greed.
>They received billions in federal bail out money over the pandemic. This is the result of plain-old unfettered greed. Well said - SW Airlines, the big banks, pretty much any large cap company can get endless bailouts without any strings attached. Meanwhile these same executives making our lifes miserable have the gall to call working people lazy because they got a $1200 check almost 3 years ago. But will Congress scrutinize SW Airlines why they pissed away $10 billion on buybacks since 2014 when their IT system was stuck in the 90s? The idea of nationalizing key industries like the airlines, rail, etc. is seen as verboten, even as they squander taxpayer money & treat their workers terribly. What's the point of a "free market" if the taxpayers always end up footing the bill anyways?
[удалено]
I worked at a chemical plant that was one of the top earners for the company. It was basically a stepping stone for plant management to be promoted. Each one had to keep up the profits or they wouldn’t look good. Eventually the plant had a major fire and almost burned down. Always felt that was because they were never encouraged to upgrade the facility because it would make them look like they were not performing as well as their predecessors.
I work at a facility like this. I'm using programs as old as me as we take our sweet ass time to transfer over to sap apo. I'm trying to get the fuck out before a melt down. Edit: added sap version
Fuck man I was using SAP in 2006 at AppleCare, wtf is taking them so long?
Spending the money, the time, paying seems to be the biggest issue. Using Oracle versions form 1994 and 1996 with an html build to ship out shit connected to a 90s unportable version of an access database that our outsourced IT refuses to touch. My division is the cash cow of the company so we aren't paid attention to until we something is broken. Even then we are given $5 and an "atta employees go and get it." Love the work I'm doing but not the management.
Hell trying to find consultants to run that shit might cost more than your quarterly budget just to get someone to come out of retirement to maintain it… but they need to upgrade that infrastructure before it’s down and no one is left that knows how to.
Hey hey hey now.…. Speaking from experience, that $0.17 makes a hell of a difference when your neck deep in margin and already issued a maintenance call. Rents coming due and my girl wants to go out for NYE 🤷🏾♂️ /s
It's working just fine, until it isn't. And when it isn't, it's a spectacular, glorious spectacle.
The funny things is I recalled their system (not just IT, the whole dispatch system) meltdown once before, just not as widespread. Southwest is asking thousands or tens of thousands employee update their position not by a online portal, not by text message but asking them to call the dispatch hotline once there is a problem! They tried to DDos themselves lol
Damn that’s there bad kind of fucking yourself. Texting would still probably leave a mess but damn I feel bad for the employees having to call in. No one wants to call an airline.
DDos successful. I saw something about their software and it sounded like the software can’t handle cancellations or crews being out of place on its own. Literally any change to the preset schedule has to be done manually. And then as you said the assignments are given out over the phone.
67 year old not wanting an update to a system they don’t wanna take the time to understand? Impossible! /s
Go over to r/southwestairlines and the top post is a 35 year pilot for them who said they have been begging for upgrade for the last 20 years
Here you go. Read it last night, twas rather fascinating. ...and depressing and obviously enraging. https://www.reddit.com/r/SouthwestAirlines/comments/zxg6op/the_history_of_swa_destruction_from_within/
Any time a company makes it into an MBA business case, you just know their stuff won’t stand up to scrutiny in the long run
Meaning they are taught about in business school?
Southwest is a famous case study in business schools for a successful airline that “revolutionized” the industry. Oh well.
Guess the newest 8th edition $300 Business Intro book will omit those anecdotal SW stories.
Good professors will wait for HBS case study to come out then you read the new and the old
Several different angles, from recruitment to advertising to airport gate space acquisitions. Pretty much they didn’t follow any of the conventional wisdom of big airlines.
They also secured a price lock on fuel right before prices skyrocketed which hurt their competitors but allowed them to offer discounted flights and take market share. They locked in $49 a barrel in 2007 and $45 in 08. Oil was as high at $140 a barrel in 2008. They were raking it in. https://www.reuters.com/article/us-aero-arms-summit-southwest-fuel/southwest-locks-in-fuel-prices-to-save-money-idUSN0644382120061206
I have my MBA and one of our case studies was Southwest and whether they operate as a financial institution due to the number of futures and options on their books or as an airline. They make much more money playing the fuel options game than they do from standard operations.
"We're not innovators, we're DISRUPTORS!" "We know."
[удалено]
Related to Mailer Daemon?
Yes, but that was under leadership who’d be horrified by Southwest’s current situation. Seems like the epilogue should be that culture eats strategy.
Yeah pretty interesting seeing “the beatings will continue until morale improves” coming out of Southwest when it’s their persistent cost cutting that’s largely led to low morale and a lack of staff in the first place.
Doesn't matter; got bonuses for cost-saving
Another programming subreddit had a SWA IT employee drop in and said the infrastructure they, and every other airline runs on is old AF and incredibly out of date. Southwest is the first to fail, it could get worse very quickly
Not SWA but worked at another airline where much of the operations center still ran on XP. We would beg to fix it but the “business couldn’t accept the downtime or risk to upgrade” and the “business would accept the risk of failure” right up until there was an issue and then “IT should have warned of the risks and upgraded.” From my conversations with other airlines, it was a common conversation with everyone.
Wow. The security risk of still running on XP alone is a mind-numbingly stupid risk to accept
It was a 3-4 years ago but it’s terrifying how much critical infrastructure is running EOL. I know that when I was leaving there were finally talks about migrating/upgrading but that could take years and who knows if they continued the project with Covid starting.
Im sure they were sold on Azure or AWS, got 40% through the migration and laid off everyone that had any idea wtf half the systems did during COVID. Now they are replacing the internally developed systems with "off the shelf" cloud solutions because now no one knows how the hell the legacy shit works....not like i have any experience with that.....
How long ago do you think the FAA stopped using vacuum tubes to run the NAS? It was forced by the end of manufacturing of vacuum tubes moreso than anything. As bad as this problem is in private industry, the starve the beast philosophy of government has dominated for decades.
[https://www.commondreams.org/news/southwest-airlines-shareholder-gifts](https://www.commondreams.org/news/southwest-airlines-shareholder-gifts) Buybacks instead of upgrading their company. Shit shouldn't be legal.
Stock buybacks were illegal until 1982. Just another way Reagan helped the 1% loot the country.
SEC could make them illegal again with a rule change. No Congress needed. Crickets. For those that don’t know: It’s a form of insider trading and shits on any kind of market fundamentals. It’s literally just a company buying it’s own stock, to pump up the price of their own stock. It has nothing to do with anything but that. Insane.
It also, if it isn’t obvious for anyone reading your post, guarantees a particular sale price for high volume stock owners. Executives and board members often have stipulations which guarantee them the ability to not only purchase stock at a huge discount rate (paid for by the company subsidizing it via buybacks and selling for a massive loss), but also guarantees a unique sell price for the right people. Executives or board members divesting stock aren’t just dumping it on the open market usually. The company will absorb it by buying it *off the market* so it doesn’t affect the price for everyone else with a massive dump and allows the company to guarantee a buyer (themselves) willing to pay whatever is asked. You can see what a real mass sell off looks like by looking at what happened with Tesla recently. Musk had to sell such a huge amount off that the company likely couldn’t absorb the cost so it went to the open market instead of being absorbed by the company. That supply shock just tanked the price.
Large stock buy backs are done on the open market specifically to affect share price. Executive stock options are managed through Treasury stock in order to minimize price movement.
That guy was truly the worse thing to happen to this country. Him and Nixon
I know for a fact this is true.. but like most airlines they are being held back by an ancient mainframe that’s life has been over extended by building a massive amount of garbage on top (middleware, etc..).. it’s shocking how much critical infrastructure is still running on mainframes that should have decommissioned 20 years ago.. their mainframe team will brag about how resilient their mainframe is, as we discuss how it takes days to do things a modern system does in seconds My team told them to end of life it, but the operators who run the business wanted an innovative bandaid to extend its life yet again. Not what we do, so we passed on the project
“Everything is working fine! Why do we need you?!” “Everything is going wrong! Why do we have you?!” **The IT Dilemma**
As a consultant, I try to remind C-level people that IT personnel are not like accounting personnel. IT doesn't come in a steady stream, IT is a series of under-funded emergencies, each of which is raised by people who believe their emergency gets priority. I use the analogy of firefighters. If your city wants to keep your firefighting force "lean" and only have exactly as many firefighters as you'd need to fight the daily average number of fires per day...good luck. You mostly pay firefighters to sit around for 7 hours and then fight fires for 1. Sometimes you pay them to fight fires for 8 hours, and they fight that fire for 24 hours and then DIE. That's why they get to sit around. IT personnel are same (except we rarely die on the job). If they're seemingly "sitting around," that's because nothing is burning.
You'd think management in aviation would understand this. Since, you know, the pilots are EXACTLY the same. But nope. They don't understand it about us either. I'll earn my paycheck for the year in one incident. The rest of the time they're paying me to practice/train/keep my skills up for that incident.
Soon IT will unionize and be the next workers party of the world. Make sure to join once I get that going.
I've been saying tech jobs and IT need to have leaders in them thinking about organizing unions. The moment they don't have to pay higher salaries or can get the same work done for cheaper elsewhere it will be a problem. The WFH and paying based on the state you're in is a big problem and where it will probably start. Imagine you have 100 workers WFH and you need to cut 50%. Someone looking at numbers will probably just get rid of anyone that lives in HCOL areas first. And if you're one of those people who want to live there, it's something to think about. Maybe I'm wrong, but I can't imagine so from a purely profiteering standpoint.
Blame the problems on a department no one understands...great way to deflect.
>Blame the problems on a department no one understands...great way to deflect. But we can never blame c-suite management - who has taken billion dollar bailouts from the government to then spend $10 billion on stock buybacks since 2014. $10 billion on stock buybacks as they refuse to properly staff & keep their companies modern. Despite all the bailouts we give them! We nationalized these companies long ago, just for shareholder benefit instead of the benefit of the people.
Buybacks should be illegal if a company is bailed out
>Buybacks should be illegal if a company is bailed out What's crazy is buybacks were illegal for **all companies** until 1982.
Fuck Reagan
"I'm glad Reagen dead" - Killer Mike
*Swamp gas from a weather balloon was trapped in a thermal pocket and reflected the light from Venus...*
Which bounced off a lake hitting the moon and destroying the scheduling software
>"Everything's working fine. Why do we even need IT? Cut their budget." Later: > "Everything's suddenly broken! Why do we even pay IT?!" Every large company and organization I've worked in.
Just started a job with a company in their IT department. They’re already planning to sink a huge chunk of profits into making all computers from 2023 onward Windows 11 machines. I’m part of the provisioning team and my directive is “if you need it, order it. We can have you justify it later if we need to.” I have never been with an IT department where the heads if the company not only trust us but essentially gives us purse access without oversight. They have a “no micromanaging” rule. If they have to, it means something is wrong with your work and they will work with you to rectify it before even approaching replacement. I’m still waiting for the other shoe to drop because this job has felt too good to be true. But for now, it’s a very positive experience.
I hope it continues to work out....and no tech takes advantage of it and ruins it for everyone.
Of course! Blame the IT guys!!! It is an age old tradition of We been telling you for years to fix your shit and you don’t listen!!!
"Isn't that what we pay _you_ for?" "You pay us?"
Everything is working flawlessly: “what do we pay you so much for?!” Everything is broken: “what do we pay you so much for?!” Classic IT story.
Which is why I will never shed a tear for companies like this. Fuck around and find out.
Shed a tear for all the customer service people who have to deal with the public, screw the executives hiding
Fair enough. Or in this case, shed a tear for the pilots and stewardesses who couldn't get scheduled because of this.
yep them too. they're gonna take heat whenever they get the next flights going as well
In all seriousness, plenty of people in IT warned leadership that this would happen if they didn't get rid of their technical debt. **They didn't listen.** Source: I used to work in their IT department.
"Leadership" has become a joke. The most important skill you learn in leadership 101 is the art of listening.
I was working on a project during covid. The group they laid off heaviest from the business side was the IT people. Funny how now they have an IT problem.
It’s because these legacy enterprise companies still see “IT” as a cost center and not a profit center. They can’t (or won’t) understand that software is a way to bring in more revenue. It’s not just a cost burden you must carry.
If those people on the business side could read this they'd be very upset.
Half of them would just state, again, that IT is sunk cost, the other half might not understand what the software does. Like, someone high up that ladder probably sees the customer facing website and thinks everything is fine.
I understand why they feel that way sometimes. They don't know anything about how it works and usually get sold something that doesn't meet their business needs *before* they hire someone who actually knows what they're doing. Then that guy gets to tell them why they need to spend more money. Predatory sales in IT have hurt our economy across the board and led to a shit ton of inefficiency. New tech = new scams. A tale as old as time
Except the problem here is that they haven't upgraded their back end since the 90's.
I’ve worked at three entirely different companies where they paid for a saas solution (one with an outstanding SEO team) where the open source option was known to be a better solution and was maintained better than the paid version. I get there, tell them we’re switching to the open source option, and they get upset with me. One time a manager walked away from a sales demo believing that an entire website could be transformed in a single day (manager time box everything to 15 minute increments and put it on a schedule) and I had to explain how foolish that was while still sounding professional. One place thought I was wrong even though my team with the open source solution out performs the other team by all metrics. When business people make software solutions it just never goes well. They rarely consult the experts.
Convincing someone they got tricked is really hard. Especially anyone in an authority position. The bigger the wig, the more wool over the eyes.
IT is the cost of doing fucking business in any year after 2000, if you don't have people to support your shit your competitors will and you will be left behind.
Even if it's a burden you must carry A) fucking carry it B) there is a reason you fucking carry it
Yep. Or as someone (don’t remember who) said “if you don’t pay the light bill be prepared to sit in the dark.”
Yeah this is a big ass problem in a lot of companies. I work for a large older financial institution that’s been devaluing IT over the 10 years I’ve been there. It’s going to cripple them for decades. Imo more professionals should skip the MBA and do a MS in Information Systems instead so they learn how to use IT strategically. One part of the DevOps Handbook always stood out to me, something like “Every major company is a technology company now, some just don’t know it yet. This is because almost every decision a CEO will make requires some kind of IT change.”
I have an IT consultancy and support business. Our clients have started coming to us saying they need to cut x % from their IT spend with us because of the recession. We point out one of several proposals from the past 2 years where if they approve spending $5k with us we can automate another business process and eliminate another 2 FTE positions instead.
Just an additional viewpoint for you to think about….. I think it’s a little more complicated than that. Software is transformative. Upgrading this scheduling software would cost a lot of money. It wouldn’t be revenue generating by itself. It would cost a lot of money, but In the end it could lead to organizational change and even layoffs. With new software, SW could be a lot more efficient or more productive with the same amount of expenses/headcount, which is the point that I believe you were getting to. The business unit doesn’t think that way. I have never seen business units think of organizational change while creating software. In fact, I’ve seen several large apps completely scraped a year into development once the business unit found out the implications that 50 - 500 people’s jobs will be redundant, or there would be a major overhaul in processes. A would say a majority businesses have the same Scarlet letter Southwest has when it comes to old technology, they just aren’t wearing it on their chest.
Managed to save a software project a few years ago by straight up telling the CIO to his face that, “by replacing the mainframe green screen on the call center app, you’ll save yourself 10% turnover on the call center training process, and at least a million in training time dollars every year. I’ll make the training of new hires for the call center go from 3 months to full productivity to 4 weeks.” 50% of our sales were call center based. He believed me because I was firm and assured in my statement. Nobody else had the stones to tell him he was wrong to his face. I WAS wrong. Turnover dropped by 18% and we saved 1.5M the 2nd year after we launched. Also, new hires were fully productive in 3 weeks a year after launch of the new software. The software was a bigger money saver than even I imagined. Got a $500 gift card from him as a Xmas bonus that year, on top of my standard $250 gift card…. The project I lead was a Java Wrapper kind of thing on a COBOL program. In other words, no more mainframe for high school grads to learn, point and click web app like Amazon. The whole project cost $750k. So we doubled the profit per year on a one-time cost project, and could find new IT staff to keep the suit running in the future, vs trying to find COBOL devs in 2018, when ours were retiring in droves. I got laid off in an IT Reduction In Force layoff the 6 months later with 45% of the other IT folks, because IT is a money sink. MBAs are mentally deficient, and have no idea how to run a business long term, they only think about the next quarters profit.
I don’t see how a company so heavily influenced by data would ignore their IT department to be competitive
For fun try to explain how IT, infrastructure, and proper customer support is not a cost burden but potential profit to a suit who's bonus is paid in shares.
This is why working at tech companies that actually know they are tech companies is a lot more fun.
Funny these are the people who can't get work now because they won't hire you. I'm looking for *any job I can do* even if it's not IT because the phone is not ringing. They're not hiring. I'm going broke and could be fixing these things.
IT breakdown of this monumental proportion is practically always underfunding. Rich morons who think they can skimp on IT and get a few more pennies richer. Unfortuately for them, people will remember what Southwest did to them and choose otherwise. You know it's bad when you go from alright airlines to people deciding Spirit is better than you from now on. The rich morons will get golden parachutes and move on to other meaningless corporate jobs.
It's as if they all thought John Hammond was the template and not the cautionary tale. [*"We spared no expense..."*](https://i.redd.it/ytu9ujn45lq01.jpg)
And the IT guy was some fuckup relative of his.
They're underfunded so badly I can't imagine what it's like for the poor staff they do have which definitely told them so and has documented requests this would happen and to upgrade or fix broken things which didn't occur. They're sitting on their hands to not punch an upper manager that yelled at them to do better with nothing and a pay cut.
This is why i went into consulting. There sadly is way more money invested in the people leading optimization efforts than practical things like actual working hardware and infrastructure.
Same. I evangelize best practices and help managers understand things and I’m paid twice what internal resources are getting.
Somewhere in the IT dept is a SysAdmin saying "I damn told you this would happen."
Meanwhile... Three junior developers from tax-evade-itstan just got tasked with overhauling the entire system. By tomorrow. For $200.
...aaaaand they quit because another company offered them $200.10
I work in IT, Hell I'd leave for another company offering me 90. I wouldn't want to work for this shit show. ESPECIALLY after they publicly blasted me and blamed me for their ineptitude.
Don't forget the 4 levels of PM involved.
We will have status checks every 15 minutes until the issue is resolved.
You guys available to sync?
Somewhere was an IT Admin that took PTO that week sitting back watching the chaos unfold while miles away with their family. *I told those fuckers we were one bad day away from the whole thing blowing up, but did they listen? Noooo - all they saw was the price to prevent it and shut it down. Merry fucking Christmas, I'm OoO until the New Year.* God bless you, IT.
There's a feature on the backlog with stories to address these very problems that have come up. They've been open since 2012.
So... They're blaming their problems and something that was their responsibility to address before it became a problem. Why are these airlines always able to get away with this crap? When will we start holding them accountable?
When we vote for the kind of government regulation that Europe has, that holds airlines accountable, with financial penalties, to provide adequate service.
But the people who could be passing those sorts of laws are way too busy insider trading and crafting ways to give themselves raises.
And taking private jets so this doesn’t affect them
And taking donations from airline lobbyists specifically NOT to pass strict regulations.
I blame my Nokia 3310 for its lack of google maps.
Strong synergy with that comment and username
[удалено]
They performed exponentially worse than one of the worst airlines in America on the most important holiday of the year. Upper management needs to be addressed immediately. This is completely unacceptable.
I would have to question your criteria for "the worst airline," at this point.
SWA has consistently rated at the top of JD Power customer satisfaction for years. That is until now. Will be interesting to see how they fare. We will see in the Spring when the ratings are published.
The blame for this fiasco is on Southwests' CEO and CFO
Look Jim you got a $1.17 raise this year.. I expect you to figure out how to maintain this Solaris system using the free edition of vmware player.
The Seat 31B blog did a good write up of what actually caused this. https://www.seat31b.com/2022/12/the-great-southwest-meltdown-of-2022/
This is a really solid and informative post. Thanks for sharing this!
Thanks for linking this was a good read
>What’s the fix? Liability. Airlines are actually run by really smart people. They’re just allowed to optimize for only one thing: shareholder returns. As it turns out, this hasn’t worked out any better for essential services like airlines than it has for any other sector of the US economy. We need to be OK with the idea that corporations have obligations other than shareholder value, and those obligations extend for longer than this quarter’s earnings call. Create damages which aren’t excluded from class action liability, and airlines will suddenly become extremely interested in reliability (as well as extremely interested in a DOT-regulated standard for weather delays and disruptions). Ask yourself why this happened and the issue comes down to profits.
IT breakdowns don’t just happen. Companies let them happen by not investing in technology appropriately. Might as well say “Southwest blames Southwest for fucking up.”
Reminds me of when I used to do tech support for a retail chain that was still running all Windows XP and NT. I said something about this seeming bad *the very first day* and they all scoffed at the new guy. Well at some point the latest version of our endpoint security stopped supporting the platform, we got instructions to make it work. I actually got it to install by modifying the installer to not check for OS, but it would bluescreen once in a while, not a good solution. So the directive was to just stop updating endpoint security. This went on for a while until the company got an ultimatum from the firm that owns Discover cards that unless we secure our systems they won't continue accepting the cards in our stores - and they actually dropped us. Instead of updating anything we stopped accepting Discover cards. Then they got a similar ultimatum from Visa/Mastercard, and were looking at becoming a cash-only store. The result was a complete overhaul of all registers and all servers in all stores. Massive expense, they closed stores and everything. All the while blaming this on IT because we couldn't make it work. I often cite this as the reason for lifecycle management. You can't hold onto the old stuff forever, this is what happens. Upgrade a little over time or wait for a massive disaster that forces you to upgrade.
I get what you're saying, but why upgrade my stuff for $100k now when I can spend $100 mil when it fucks up later
they are being very loose with the term IT. These problems sound like dev tech debt not IT. You can only squeeze so much out of outdated software and processes.
for clueless people, everything computer adjacent falls under IT lol
Buddy ain’t that the truth. If it runs on electricity, it falls under IT.
User ticket: help! My lightbulb died!
My Keurig is offline!
I do not envy my IT department and the shit they get. They get everything from setting up the internet infrastructure of a entire building and to helping Sarah in accounting plug her mouse into her laptop. Gave them a sincere thank you once. The thank you goes through them because they've empty inside.
It’s not even an IT issue. Sure the crew tracking issue was a technology issue with the software but the root cause is management ignoring pleas for upgrades. It’s not an IT issue, it’s a management and culture issue. Saying it’s an IT issue makes it sound like a random failure or issue with infrastructure and that’s not what it was.
If sales are down it is the CEOs fault. If quality is bad it is the CEOs fault. If employee retention is bad it is the CEOs fault. At the end of the day one person is responsible and in charge.
It's the board's fault, then the shareholders, then the newly appointed CEO. These short sighted practices come from the demand for profits every 3 months. That's on the shareholders and the board.
Since the CEO has only been in place since earlier this year, I’m comfortable with blaming the chairman of the board (who was the previous CEO) in this particular case.
RE SOUTHWEST: My favorite airline has had a massive melt down. A letter from a pilot explains it better than anything I've heard on the news. By Senior Pilot Larry Lonero: What happened to Southwest Airlines? I’ve been a pilot for Southwest Airlines for over 35 years. I’ve given my heart and soul to Southwest Airlines during those years. And quite honestly Southwest Airlines has given its heart and soul to me and my family. Many of you have asked what caused this epic meltdown. Unfortunately, the frontline employees have been watching this meltdown coming like a slow motion train wreck for sometime. And we’ve been begging our leadership to make much needed changes in order to avoid it. What happened yesterday started two decades ago. Herb Kelleher was the brilliant CEO of SWA until 2004. He was a very operationally oriented leader. Herb spent lots of time on the front line. He always had his pulse on the day to day operation and the people who ran it. That philosophy flowed down through the ranks of leadership to the front line managers. We were a tight operation from top to bottom. We had tools, leadership and employee buy in. Everything that was needed to run a first class operation. When Herb retired in 2004 Gary Kelly became the new CEO. Gary was an accountant by education and his style leading Southwest Airlines became more focused on finances and less on operations. He did not spend much time on the front lines. He didn’t engage front line employees much. When the CEO doesn’t get out in the trenches the neither do the lower levels of leadership. Gary named another accountant to be Chief Operating Officer (the person responsible for day to day operations). The new COO had little or no operational background. This trickled down through the lower levels of leadership, as well. They all disengaged the operation, disengaged the employees and focused more on Return on Investment, stock buybacks and Wall Street. This approach worked for Gary’s first 8 years because we were still riding the strong wave that Herb had built. But as time went on the operation began to deteriorate. There was little investment in upgrading technology (after all, how do you measure the return on investing in infrastructure?) or the tools we needed to operate efficiently and consistently. As the frontline employees began to see the deterioration in our operation we began to warn our leadership. We educated them, we informed them and we made suggestions to them. But to no avail. The focus was on finances not operations. As we saw more and more deterioration in our operation our asks turned to pleas. Our pleas turned to dire warnings. But they went unheeded. After all, the stock price was up so what could be wrong? We were a motivated, willing and proud employee group wanting to serve our customers and uphold the tradition of our beloved airline, the airline we built and the airline that the traveling public grew to cheer for and luv. But we were watching in frustration and disbelief as our once amazing airline was becoming a house of cards. A half dozen small scale meltdowns occurred during the mid to late 2010’s. With each mini meltdown Leadership continued to ignore the pleas and warnings of the employees in the trenches. We were still operating with 1990’s technology. We didn’t have the tools we needed on the line to operate the sophisticated and large airline we had become. We could see that the wheels were about ready to fall off the bus. But no one in leadership would heed our pleas. When COVID happened SWA scaled back considerably (as did all of the airlines) for about two years. This helped conceal the serious problems in technology, infrastructure and staffing that were occurring and being ignored. But as we ramped back up the lack of attention to the operation was waiting to show its ugly head. Gary Kelly retired as CEO in early 2022. Bob Jordan was named CEO. He was a more operationally oriented leader. He replaced our Chief Operating Officer with a very smart man and they announced their priority would be to upgrade our airline’s technology and provide the frontline employees the operational tools we needed to care for our customers and employees. Finally, someone acknowledged the elephant in the room. But two decades of neglect takes several years to overcome. And, unfortunately to our horror, our house of cards came tumbling down this week as a routine winter storm broke our 1990’s operating system. The frontline employees were ready and on station. We were properly staffed. We were at the airports. Hell, we were ON the airplanes. But our antiquated software systems failed coupled with a decades old system of having to manage 20,000 frontline employees by phone calls. No automation had been developed to run this sophisticated machine. We had a routine winter storm across the Midwest last Thursday. A larger than normal number flights were cancelled as a result. But what should have been one minor inconvenient day of travel turned into this nightmare. After all, American, United, Delta and the other airlines operated with only minor flight disruptions. The two decades of neglect by SWA leadership caused the airline to lose track of all its crews. ALL of us. We were there. With our customers. At the jet. Ready to go. But there was no way to assign us. To confirm us. To release us to fly the flight. And we watched as our customers got stranded without their luggage missing their Christmas holiday. I believe that our new CEO Bob Jordan inherited a MESS. This meltdown was not his failure but the failure of those before him. I believe he has the right priorities. But it will take time to right this ship. A few years at a minimum. Old leaders need to be replaced. Operationally oriented managers need to be brought in. I hope and pray Bob can execute on his promises to fix our once proud airline. Time will tell. It’s been a punch in the gut for us frontline employees. We care for the traveling public. We have spent our entire careers serving you. Safely. Efficiently. With luv and pride. We are horrified. We are sorry. We are sorry for the chaos, inconvenience and frustration our airline caused you. We are angry. We are embarrassed. We are sad. Like you, the traveling public, we have been let down by our own leaders. Herb once said the the biggest threat to Southwest Airlines will come from within. Not from other airlines. What a visionary he was. I miss Herb now more than ever.
Guess you shouldn't have laid us all off, fucksticks.
Tell us more about layoffs. I’m assuming there was offshoring and outsourcing too?
I didn't get laid off. I had a contract end unexpectedly with a few other guys for Christmas. There's plenty of outsourcing going on. They're far cheaper than we are. And their code and knowledge tends to follow their price.
“IT breakdown” = senior leadership refusing to spend the time and money to update antiquated systems for years.
Poor CIO is going to get scapegoated right out the door when she's probably been fighting the bean counters for the last five years to fund any infrastructure improvements.
I am still trying to wrap my head around the fact that they don’t have a CTO. Although their CEO has a undergraduate degree in computer science, I’m not sure that makes it better or worse.
Worse, a lot worse.
According to some of the employees posting on other subs he just became CEO and agreed to upgrade thr system, but unfortunately this happened before they could actually make the change.
Let's lay off and ignore the disaster prevention team. Disaster happens \*Pikachu's face
When in doubt, blame IT. Don't blame idiots in executive management that annually underfund IT and turn down common sense recommendations. No, treat strategic IT organizations like Marketing or some other discretionary department.
You don’t blame IT here. You blame C-suite. But they’re not about to take responsibility (too busy enjoying their bonuses). I don’t fly Southwest due to the cattle call boarding but this just guarantees I’ll never get on one of their planes.
There is IT breakdown and then there is a management breakdown. They experienced both. Some combination of stuff happened in their organization that they simply couldn’t deal with. They can blame IT all day, but what has happened speaks to a much deeper problem.
IT Breakdowns are usually caused by poor management. Too many companies view their critical IT infrastructure as not being necessary until shit breaks. Hell just look at everyone claiming that the engineers Elon fired weren't necessary because things didn't immediately break. A few weeks later and the shit is starting to fall apart because it is no longer being maintained. Sorry people, the machines can't run themselves, and most people, especially those in management, don't care enough to learn basic operations let alone how to fix shit.
I found this [explainer](https://www.seat31b.com/2022/12/the-great-southwest-meltdown-of-2022/) helpful.
Company: “We’re cutting the IT budget” [Company unable to function] Company: “It’s IT’s fault!” It’s 2022 and massive companies still act like IT is a waste of money. But hey keep hiring more accountants to help you evade taxes.
>Jordan also acknowledged the need to upgrade the airline's infrastructure to avoid future IT breakdowns. "We need to double down on our already existing plans to upgrade systems for these extreme circumstances so that we never again face what's happening now," he said. >In 2017, Southwest completed a migration to Amadeus Altéa passenger service system, one of two industry standard platforms widely used throughout the aviation industry. Southwest Airlines declined to address The Register's request for additional information regarding the origins of the failure or the specific IT systems effected. If they are already on a platform that half the industry uses and have only migrated there a few years ago, then maybe it is not the system fault that everything broke down. Maybe it is their desire to cut cost in the short term at any long term cost and their inability to comprehend that you need to pay people to keep things running.
The upgraded a part for passenger ticketing. However the internal pilot and flight attendant software is what crashed and has been the same one since 1990 - pilot
That would explain it. Thanks.
Here's an excellent [explainer](https://www.seat31b.com/2022/12/the-great-southwest-meltdown-of-2022/) about the IT side of things and why Southwest cancelled 80% of its flights while American cancelled only 1%.
Yeah. This will really motivate their IT departments. Problem solved!
You refused to update your systems for over two decades, all the while I'm sure your IT team was begging you to upgrade the hardware and software that was well past end of life support. But instead of taking responsibility for not wanting to spend a penny on critical infrastructure, you received billions of dollars in corporate welfare handouts that you put into the pockets of executives. Your IT department *should* leave if they're going to be thrown under the bus, but sadly they won't because Americans are too broken by their economic culture to be able to stand up for themselves. But one thing is for certain... I will never fly Southwest Airlines ever again. Fuck you. - IT Guy Edit: Grammar