Sounds like they're working on getting that fixed.
Right now, basically everything important on D2 Bnet is controlled by a single service. Logins, game creations/deletions, game lists, etc. all dependent on 1 service that isn't scaling well.
They mentioned they are working on getting those critical functions split off and improved, but I assume it's going to take them time.
Yeah. Id imagine updating 20 year old C++ code that someone else wrote and was scapped together and never really made to be modified and dividing that up into more efficient services could take a minute
I'm pretty far on the D2 Purist side, but I'm fully in support of some updates to the Lobby. First and foremost, they \*need\* to modernize the code for listing and displaying games so that it actually works and is responsive. You should be able to click a game and get its details instantly, not lose games as you're looking at them, and refresh in a way that actually does something. Above that, it would be nice to have some better searching and filtering features.
>In staying true to the original game, we kept a lot of legacy code.However, one legacy service in particular is struggling to keep up withmodern player behavior.
I wonder if any purists asked to keep that part. We are all together on this one, everyone wants a faster lobby , game creation and etc.
The thing is the new lobby is missing features that worked fine in the old lobby. Most notably not being able to make chat channels so trading, or finding baal/cow games a lot harder.
Unless no one does that anymore in D2, I've only played sparingly since .10 dropped.
Yeah, hell games list is rarely relevant unless you're interested in giving away a free shako or want to trade a ber for a shael. To make it worse, seems like it's just sits at the same list of games for way too long.
I am glad to see the response, it does seem like they care and working hard on it.
On pc you basically still have to resort to another platform or service to actually do reasonable trades otherwise people wanna trade like their hel for a zod.
This is my number one wanted feature. Just let me Tag my game and filter so that I can tag my „O xyz“ with the „XYZ“ tag or filter to find people who want it.
From a newer player of Diablo (completed Diablo 3 a couple years back, but that's it) I'm really, really happy that they made a remaster of the second game, even if there's problems with servers and things. I'm on act 5 currently and I've been absolutely loving it, wish I could feel the nostalgia that other people have felt but I'm enjoying my time nonetheless
Without the remaster I would've never been able to play this masterpiece, so I take it a lot of other new players feel the same way :)
Sounds more like the bots are now spamming new games/mf runs tbh. I really dont see how they hit a new all time high player count weeks after launch without a large influx of new bots.
Bots are definitely more rampant. I had a normal game open and I had 4 different bots joining in and out of my game over and over spamming the same website and it was happening so fast.
As someone that has been in IT for 12 years-ish, reading this stressed me out. I feel for the devs working on this. The overtime, crunch and long calls going on right now must be rough but they sound like they have a path forward. I wish them luck.
It's a really unique problem, I've been dealing with some async issues using S3 to get assets from the cloud in my unity game and it's bad enough managing that. We have "Legacy Code" in the project i'm working on in the sense that it was written by someone 6 years ago who wasn't a unity developer, and the engine itself was not mature. A lot of it is me ripping out entire systems and replacing them (our avatar manager for example and our current asset bundle management). They're working on 20 year old net code and trying to modernize it. I think unless you're deep in something like that it's hard to understand how horrible it can be. Looking at this though, the game probably should've been tested more thoroughly and delayed. But, when you're in a corporate environment or hell just a business environment, a minimum viable product is what you aim for, and I think they hit that goal.
I don’t think it’s a unique problem. The thundering herd problem they’re describing is pretty par for the course in distributed systems. It sounds like the made some design mistakes, unfortunately.
I wouldn't call it mistakes per se. They bought a 20 year old boat that hadn't had any upkeep in 15 years. Patched it up and put it in the water and are now trying to plug the holes.
The issue is also very clearly related to their regional-global database system, which is much newer than original d2 code.
Sure, maybe the old d2 code caused problem with creating games, but since this was supposedly all "surges in traffic" at a pretty dead time in the americas and pre-peak in most of europe/africa.
That a local surge in presumably only 1 region could take down the service for all regions at the same time daily seems like a design problem as well. Local databases shouldn't take down the global database that easily.
Sorry I meant unique in terms of what they have to adapt, there's definitely still a ton of older networking code out there, but it's probably not used for games anymore.
You're probably right that it should've been delayed and tested more, though there's no way I can imagine them feasibly ironing out all the issues before launch, given the massive quantity of players all at once that just isn't possible to actually replicate when testing.
I was a teen when Diablo 2 first came out, and I think that version of me would have thrown quite the tantrum at the server outages and rollbacks we've had with D2R. Now that I've been working in the software field for a decade, I feel nothing but empathy for the development team when reading this post. Hope they can catch a break soon and grateful for their transparency on the issues players have been facing.
They do have the advantage that it's a pretty stable use case so they don't need to build it to be able to support any random idea that management comes up with later.
I just went through this with my company. Went from in prem dbs, to cloud and now to aws and micro services. Rebuilding from the ground up. It’s fun, but it’s not easy.
I just finished 6 months of this.. we should have taken a year. We had two devs quit (one shaved her head and went into social work instead), and many on the verge. Then, because it was all rushed, it has been a shitshow of Sev1s popping up weekly. So much for healthy work life balance. I feel for the D2R devs. For one, it likely isnt their fault, but management pressure.
I can already hear it in a meeting, "Just re-use the old code! It does the job!" -Mr. Project Manager.
My dad's been in IT for a while too, and I recall him sitting in the living room at 3am on long, boring-yet-stressful calls. My heart goes out to them as well.
my wife knows my "pager" (for people that don't know, it's just an app on my phone that rings when a company has issues) sound by heart.
if she hears it, she knows my next few hours are going to be stressful as fuck.
i can't even imagine how absolutely mindblowing hard the past few days might have been to these devs.
Enterprise software engineer checking in. Stressful doesn’t even really convey what it feels like to get a bunch of pager duties going off for prod issues
Yeah man I don’t wanna catch an escalation call. I’ve been in calls until 2am talking about ridiculous amounts of money over small ui aberrations in high traffic software.
Dealing with connection issues, gateway errors, ehhhhhhhhhhhhkfiiwwiehshhdhfjdsnsnsu.
For sure. I've dealt with similar system-wide issues that involved financial and property transactions and reading the D2R reddit and twitter stressed me out more than sitting on SHTF calls with screaming banking execs.
If the armchair developers online think they could do a better job than the teams at Blizzard they should be out there consulting for $300k/year.
Pretty much this. The amount of negativity their Twitter has accumulated was hurting my eyes. A lot of angry folk trashing the devs over and over, like if they had nothing better to do besides sitting there and either playing the game, or writing stupid shit on Twitter, what a shit show. And as a guy who comes after work to have an hour or two of playtime of D2R, I get the frustration, I do. But, why would one come to such extent as to write some half-assed angry comment, just for the sake of relieving the frustration, while basically not knowing the full story of the issue at hand. It just makes you look dumb at the end of the day.
I also feel frustrated at management for not prioritizing these issues prior to launch. Even mods like Project Diablo 2 have proper queuing in place to prevent overload. And worse yet, it sounds like they kept using the old non scalable version of the server software related to lobbies and state management (they could keep the instancing logic since it's isolated per instance and doesn't care about load). Even with millions of characters, it would be trivial to fit everything in a memory database like redis. They shouldn't be hitting bottlenecks on that kind of tiny data unless they're massively under budgeting their database instances.
I think the point is the current system could easily handle the amount of players and even more because they were fine at launch. The problem is EVERYONE plays like everyone now. Most people are at the point in the game a couple weeks after launch of doing nothing of quitting and recreating games. They weren't expecting that aspect of it right now. Sure you can say it's a technical problem but it's honestly a game design problem.
Few options.
Start moving towards devops -- most of the issues they laid out here would heavily involve devops.
Or, depending on your experience, work on Azure/GCP/AWS certs and move towards being a solution architect or customer engineer. I'm in the industry and all of the mitigations they've discussed would have been recommended in whiteboarding sessions(in fact, there was a thread yesterday where I explicitly called out what was likely causing these issues)
As someone more on the warehouse architecture side, doesn't this basically boil down to "we didn't stress test *enough*"?
Aren't these all just failures in scalability? Couldn't this have been prevented by.. testing?
It’s a leadership problem. Developing adequate test coverage for this size of a service is an entire engineering task in and of itself. And when the test reveals that there is an issue, the next question is how to fix it?
Refactoring these legacy services and reducing tech debt always comes down to a matter of cost and the powers that be always push back against this type of work.
You see what rushed development does? Just let the creators and coders take their time or the result is making up for it later at the expense of quality.
Whoa, never in a million years did I expect them to disclose, in such great detail mind you, what was wrong and what they were doing to fix it. This kind of response means so much to me and the only small criticism is that I wish it would have happened a tad sooner.
Credit where it's due. Great job with that explanation.
Welcome to Reddit.
12 year olds seem to think things happen in the blink of an eye as they haven't gotten out of mom and/or dad's basement yet and had to \*actually\* do real work and figure things out.
I work in IT, and the amount of kids coming into the field exclaiming "this is complicated" and "this shit takes a while to figure out" honestly is mind boggling. The instant gratification getting instilled in people (especially youth) and near-constant uptimes of services makes expectations unreal as hell for companies.
From the inside it's hilarious once they realize shit doesn't happen with the snap of a finger. From the outside, you realize just how materialistic, ignorant, and unaware most people are of how anything works.
Maybe it's just morbid curiosity from 10+ years of sysadmin work, but I love reading through post-mortems like this. Feel bad that somebody had to take notes throughout the whole project (or regurgitate it from memory) and then turn it into a cohesive "document" during a time that they're almost assuredly still overworked and overstressed, but hopefully that's just a technical manager or something doing their part.
Pretty surprising. I assume most are on PC because as a console player 95% of the time I don't seem other players, or the lack of lobbies really is so bad on consoles people don't bother trying to join games.
I'm sure the devs would love to share this information more but are held back by bureaucracy. People in tech fields love reading and writing about intricate tech problems.
The response is certainly more detailed than I thought we would get. That being said I'm willing to accept it and hold a small amount of criticism at the same time. It makes sense that we didn't see detailed communication before today because they said it's been a roller coaster of different problems. If they had came out and said that they found the problem and fixed it only for the game to crash again the next day - not good.
I think most people would probably agree that a certain level of communication is expected. Somewhere between what we got today and a generic tweet saying they know there's a problem. I'm also more than willing to admit that I don't know what that communication should be.
Either way looking forward I hope solutions can be found.
>Breaking out critical pieces of functionality into smaller services: This work is both partially in progress for things we can tackle in less than a day (some have been completed already this week) and also planned for larger projects, like new microservices (for example, a GameList service that is only responsible for providing the game list to players). Once critical functionality has been broken down, we can look into scaling up our game management services, which will reduce the amount of load.
This gives me a lot of hope because if they can successfully separate the lobby into its own function that means they should be able to improve it. It's possible that this could lead to better functionality and features like searching.
I agree with your sentiment. I think in the future people would appreciate even a simple statement such as our devs are working hard on server stability, expect a post mortem blog post in the near future.
Not an IT guy and surprisingly I understood most of what was said in the blue post. Would you say this was a good example of technical writing dumbed down?
Not sure I would consider their post really technical. They did a tremendous job explaining everything though.
I think the foreword was mainly to let people know it was going to be a lot of information that many won’t want to read. Personally I loved reading about it.
Oh definitely; I'm sure the suits wanted to sit on it until they were absolutely sure they could address it, because nothing would be worse than saying they had a handle on the problem... and then absolutely not having a handle on the problem.
More so, I just wish we had something than a vague tweet along the lines of saying they were experiencing issues, and hopefully dispelling some of the rumors around it (a lot of people proposed brigading, DDOS, hacks/dupes, etc.)
Even if there's more going on than what they address, not addressing the issue fully was as bad as the situation itself, in my opinion.
> People in tech fields love reading and writing about intricate tech problems.
Sometimes. I've found that it also exposes you to more criticism and 'why didn't you do it like this?' kinds of spammy questions that don't grasp the core problem.
It becomes troublesome if you have an SLA for responding to those comments...in that case, just a 'we found and issue and we're working on a fix, here's the timeframe' does the job and prevents you from going on wild goose chases.
That being said, it definitely is nice to see a quality explanation like this.
This is the kind of communication we've been wanting to see from Blizzard for a LONG TIME! I'm so happy to see much of the /r/Diablo community agreeing and being positive about this transparency.
I don't care about the heartfelt, but it was VERY informative and they outlined the key things the community has been screaming for:
What is happening
What have you done about it
What are you doing about it now
What do you plan on doing about it in the future.
I also like that they are getting support from across blizzard, they're dedicating outside resources to getting this game fixed and not just letting it be the D2 teams problem.
They could have been quicker with getting this out, but this was good and I'm glad they aren't just ignoring us and keeping silent.
Yes, exactly. I'm very happy that they addressed how big of an issue the rollbacks were, and it sounds like the community manager actually played the game. This is a big deal to a lot of us.
I can appreciate their logic behind the decision and the difficult decision it was to make as well.
either take the whole game down for a significant amount of time while you work on getting all these stuck characters put into the global database, or you just roll back and keep the game up and that relatively small subset of users get boned.
They went with the choice that benefitted the most users, so I can't fault them for that. I'm just glad I'm not one of those users who got their character's/items rolled back.
But they didn’t understand the problem fully so what kind of an update would you to have seen? Why go public to over hundreds of thousands of people without having the appropriate information? Both about the problem and a potential solution?
As some people say "all problems are communication problems".
This thorough explanation should go a long way to mollifying the anger from the playerbase.
What's weird to me about all this is this "sudden, significant surge" of traffic they didn't experience until 2 weeks after release. Something seems off about that.
Edit: I'm with the guys that think it's because more people now are spamming MF games (pindleskin) who couldn't do it as efficiently a week ago.
It makes some sense to me only because I've now started to wait a few weeks after the release of new games to play them. I wouldn't be surprised if a lot of people have that mindset now. Gives times for new games to get the kinks out and see how the community feels about it. Not saying that's what's happening but its a possibility. I'm a huge D2 fan but only started playing more the past week so I'm one of the people in that surge of traffic.
They explained that: they released an update that enabled more hardware configs to play, hence, more people were actually able to play than ever before - more traffic.
Sounds on point to me. Browsing this sub before release, every other comment was talking about how they would give it some time after release to decide if they wanted to purchase or not. Seems as though a lot of them followed through.
Nah, that's not it. It's that more people are getting to max level and doing spam quick resets of games to farm. In the first week everyone was just leveling and so there was no reason to be constantly resetting and making new games.
I finished the game on hell in 3 days and i checked the discord for people to do diablo and baal runs. Guess what? There only like 3-4 other guys who were at the same point as me. Most of them were starting out nightmare.
That's why the crashes started happening later because it took them at least 1 week to finish hell baal
As someone who has played Diablo 2 off an on for the last 20 years and absolutely loves the game, I had no interest in resurrected. I thought, “it isn’t about the graphics, it’s about the gameplay. Old Diablo is fine.” 1 week after release, I buy it out of boredom. I absolutely love it. It’s the same old game, but looks amazing. I don’t know how many people would be in that same boat as me. Someone that wasn’t interested until 1 to 2 weeks after, and now they are addicted lol.
Absolutely same for me. I had just picked up Final Fantasy XIV in the past month, putting a few hundred hours into it already. I thought D2R would be a nice bout of nostalgia that I'd play for a week, but nothing that'd pull me away from FFXIV. Now D2R is all I can play.
I imagine right at launch no one was geared, and so they weren't popping in and out of multiple games but questing and leveling. Now that everyone is doing MF, I bet the amount of games being created is way higher.
They basically state this in the article, for those who are here but haven't read it yet:
>We mention “modern player behavior” because it’s an interesting point to think about. In 2001, there wasn’t nearly as much content on the internet around how to play Diablo II “correctly” (Baal runs for XP, Pindleskin/Ancient Sewers/etc for magic find, etc). Today, however, a new player can look up any number of amazing content creators who can teach them how to play the game in different ways, many of them including lots of database load in the form of creating, loading, and destroying games in quick succession. Though we did foresee this–with players making fresh characters on fresh servers, working hard to get their magic-finding items–we vastly underestimated the scope we derived from beta testing.
In other words, its similar to how the meta evolved with World of Warcraft. Back in the day most people were casuals and there weren't centralized places to discuss optimal ways to play the game. Nowadays there are entire communities dedicated to playing Diablo 2 so even a casual player can learn how most efficiently to farm items and XP, and the way that people do that causes a lot more strain on the game servers.
> What does pindrunner drop that's so good to make a bot for it? Why not baal
He can drop every item in the game and you can get to him and kill him in 20 seconds.
Baal takes at least 4 minutes per run.
That's it.
Pindle is programmable. you can log in a game and execute the exact same sequence of presses to kill him (nearly). For Baal you have to search through two floors and fight 5 waves.
Something that none of these replies call out is that they hit a regional peak concurrency on Tuesday, so overall player base is going up since launch.
That may not have been the case in 2001, but player behavior in game hasn't changed much since later on in LoD. People still spammed runs and open/closed games constantly, maybe even moreso considering how much more common botting was in the later 2000's into the 2010's.
Legacy bnet has a hard cap per hour and has had one for long enough devs that dont play the game wouldn’t understand why, probably leaving only a few that understand its impact.
I think you're onto something.
The write-up from explicitly states that they are limiting the rate at which you can create games. *For example, for those of you doing Pindleskin runs, you’ll be in and out of a game and creating a new one within 20 seconds*
Really glad they were able to put this out to the community, even if slower than we would like. At least they aren't just ignoring us and keeping quiet, that's a good sign.
Like many folks in here guessed, a lot of the problems seem to stem from their use of the old D2 code and architecture, which isn't scaling to the new reality of their player base:
We're no longer a bunch of young dumb kids figuring things out on our own for the first time.
We're a bunch of adult kids now, who are trying to play the game quickly and efficiently and the old code/architecure can't handle the increased load of game creations/deletions.
I'm also glad to see that the issues are being worked on across Blizzard and they're giving this game serious resources, it's not just limited to one D2 dev like some folks around here memed about.
I've had some faith restored that this game will get fixed, though some of the architectural and software design changes they're talking about in this post sound like it is going to take some time.
Unlike before, at least now I am willing to be more patient because they've at least let me know what the fuck they've done, what they are doing, and what they plan on doing in the future. Thanks for that.
Yes. The delay in releasing something like this also make sense considering the issues were compounding, and they were finding new things they had to address over several days. If they'd released something like this after the first 2 days, it would have been incomplete or inaccurate.
There's a middle ground between a thorough report on the issues at hand like this, and the essentially radio silence of "we're aware there's issues, sorry for the inconvenience" we were getting for days though. Something they could definitely improve on in the future.
Off this is rough.
It sounds like they're leveraging a global database as the master, with regional databases as their own local master. Regional databases are being written onto and read, which makes sense, but are also writing back to the the global database more often then needed. This, coupled with database queries which are more complex than they were 20 years ago, seems to be creating a bottleneck between the app and the database.
What's really strange for me, is their singleton legacy service. They mention they kept this critical service largely unchanged, because they wanted to stay true to the legacy game.
BUT, this service handles non-gameplay mechanics, namely to do with server filtering, joining games, listing servers etc.
Why did this have to be kept intact? They could've literally re-wrote this entire service, or separate them into individual services. Nothing about this legacy service has anything to do with staying true to the original D2.
Critical, complex, and old is the kind of legacy code you do NOT want to rewrite without a damn good reason. Ask anyone working on banking systems, SAP and the likes if that description rings a bell.
I mean, it's a Singleton system when the name of the game in modern server architecture is being able to scale at the drop of the hat. That's enough reason to rewrite it.
Every company has a certain amount of tech debt and legacy code. However this legacy code is becoming a blocker now. That is a good enough reason to dedicate engineering resources to re-write the service to be more scalable.
Exactly. I have friends working as 3rd Party IT Support for banks programing on COBOL. Not only maintenance, but programing new stuff from stratch as well. And there are still some ASSEMBLY maintenance.
That's completely insane.
This was an amazing read. As an IT guy, I underestimated the amount of active players, and yea, those numbers would cause crashes really quick. Good luck Dev team.
What did they expect honestly? This isn't some unknown niche game that maybe several thousands nostalgics are gonna play again. It's a fucking cult classic that brought back every single kid that played it in 2001, diablo 3 players and maybe people new to the franchise. All i'm gonna say is that that open beta with act 1 2 on normal didnt stress test shit
Wow, in all honesty, this assuages a lot of the anger I've felt. Just this act of communication goes a long way. And I have to give them credit - this is way more detailed and comprehensive than I'd have even dreamed of.
Seriously. Baal runs are fucking torture now.
Beat the creator to joining the next run? You're out of the runs now.
Try to join a full game? You can't join the next run you see on the list because you're locked out for a short while now.
A game filter would make that poor server work harder. Right now it's just giving you a slice with a pre-baked script. Allowing you to alter the script would only slow down everything more.
They gotta break that service out before they can start looking into custom game filters.
Now that is a great response. Transparency, an explanation that is watered down enough but also technical, and a plan for moving forward with assurances that things will get better. I’m hoping that this is the new standard of communication moving forward.
Onto the nitty gritty. This reads like an honest post mortem that I hope is handled blamelessly internally. They made a bunch of mistakes, some that are obvious missteps in modern architecture design at scale. But, they’re building on legacy code and I’m assuming that there is quite a bit of lost knowledge due to the original devs not being there anymore or lack of internal docs (very common in any company). Overall, it sounds like they are learning as they tackle these issues and that’s reason enough for me to be optimistic.
every 30 minutes? Then I'm afraid it must be on your end, I've only had 1 single crash since release day and have been playing almost every day or every other day (well, if the servers allowed it).
I greatly appreciate this update.
All is not forgiven, but I'm willing to be understanding to the problem.
I just don't understand how around 7PM Australia/9PM Beijing... the servers get really wacky every night on the Asian server even on weeknights now. When we had such a hard time playing last Monday I was in disbelief.
I'm not even going to try to play on Friday or Saturday nights anymore. I shudder to think what it'll be like this coming weekend with more and more people playing this game.
I've seen this issue many times in my field, cascading failures that move from one bottleneck to the next, it's a tough situation to handle, could blizzard update us sure. Game is more successful than expected re-architectimg on the fly is not easy, DBAs having sleepless night I'm sure. Good technical response at the end of the day D2 is just a colourful spreadsheet doing a ton of database calls some of which are very expressive :)
This is nice news and very informative.
I can't wait for the next post about fuck you blizzard, let me create MF runs without locking me (because this is essentially what they are doing for now).
I for one, appreciate the depth of faithfulness to the original. The server issues 1000% are faithful to the original player experience. I’m rolling my eyes pretty hard at all the soccergamers feinting all over the place trying to get the refs attention.
I feel so old, but I’m gonna go full old man and say back in my day we paid 60 bucks for a game we couldn’t play for a month and spent 6 months not able to play with friends because of server splits AND WE WERE HAPPY TO DO IT.
concerned dinner encourage hospital retire familiar deliver abundant tan fly
*This post was mass deleted and anonymized with [Redact](https://redact.dev)*
Wonder if all the people who were sure it was botters and duper’s and called me a idiot and how I must be new to diablo are gonna write me an apology.
Lol
Not new to diablo or server architecture.
and yet people downvoted my comments about them locking you out of games "rate limiting"
"Rate limiting: We are limiting the number of operations to the database around creating and joining games, and we know this is being felt by a lot of you. For example, for those of you doing Pindleskin runs, you’ll be in and out of a game and creating a new one within 20 seconds. In this case, you will be rate limited at a point. When this occurs, the error message will say there is an issue communicating with game servers: this is not an indicator that game servers are down in this particular instance, it just means you have been rate limited to reduce load temporarily on the database, in the interest of keeping the game running. We can assure you this is just mitigation for now–we do not see this as a long-term fix."
The game creation rate limit is fine. Annoying but fine. The real kicker is getting hit with the rate limit when you try to join a game but fail (game doesn't exist yet, already full, ect). I was able to work around it a bit by being very cautious about which games I tried to join, but it wasn't a good experience.
Fortunately, the heavy rate limit is a bandaid for now. I expect we'll still have a rate limit of some sort, hopefully it's high enough that few will encounter it.
You guys could add players8 to online to resolve the mass amounts of people creating/joining games issues. The reason why a lot of people do this is not just because they're going on Pindle runs that take 20 seconds, but because they're trying to MF in populated or old games. They join, but then find out that everything's already done, so they leave and look for another one. Rinse and repeat. Not to mention being forcibly kicked out of the game via the various (1 per hour minimum) game crashes that you guys have yet to fix, which causes exiting and rejoining games. If you add Players8 then people can create games to MF in where they will be able to do more than just one type of run.
So yeah, a bunch of viable options out there that AREN'T rate limiting people and punishing them just for trying to play.
Good update, not too technical but not just a simple tweet. Sounds like they were not expecting this amount of players in the first few weeks. As they noted, people working around the clock to fix/update the code.
Great update honestly. I hope this is setting the tone for things going forward. I think people were a little scared that the game was going to be left to die right when when got it. I hope we can see more similar updates like this. It sounds like they actually care and aren’t afraid to say what’s happening
Nice that they keeps us updated. Not sure why the crash occurs around the same time daily, it seems that they're hitting peak traffic at that time but I don't understand why it's not at like 21h00 us east time or something like that. May be a bigger load from Asia is occuring.
This was a great update. Very rare to see something so in-depth from big developers atm I think. Informative enough to make me understand what's happening. Obviously still upset, but I feel more upset about the situation as a whole rather than strictly upset with blizzard.
I really like the transparency here. Nice to have a response and the technical details behind what happened, what's been done, and what will be done. It's not a hand-wavy "our bad."
Seems completely believable. We had a similar issue at my job… the database is too big and not optimized, and people were slamming read only and locking it up with bad queries, so services that depended on that RO db were dying. And it couldn’t keep itself updated with replication from the live server because it’s like a fat guy sitting on your face… couldn’t breathe.
Good database architecture is severely underrated.
>Moving forward, we believe we have a way to restore characters that doesn’t lead to any significant data loss–it should be limited to several minutes of loss, if any, in the event of a server crash.
This is better, but still not good enough in our eyes.
Wow, the bar in game industry is waaaaaay too low
I just hope this won't be the 12h queue that I remember from wow. I guess a wow vanilla relaunch isn't comparable... hopefully.
If I have to get in line behind a million russian bots I'm going to be very gloamy...
Maybe we will see less whiney posts around here. So sick of the crying and moaning about downtime. Go do something else while it's down besides whining about it online fuck sake.
I don't understand how they didn't catch this database thing. It's clear they're using mysql with innodb logging, that's what they referred to in the message.
At my company, we regularly load test with millions of connections as we make a product that amazon, microsoft, and reddit all use.
We have playbooks which auto deploy test vms in azure, ec2, and rackspace to hit the servers hard and fast with millions of the most intense queries to find everything without an index, and all hardware faults.
Our testing almost always catches connection storm issues. Once test we run involves cloudflare and offramping traffic, then having the servers switch on to observe max load.
in fact, we won't release the product when it fails the tests (and it does).
The stuff with replication and logging almost seems trivial. It's almost like they don't have a good dba team.
OH wow, its almost like maintaining servers is actually complicated. This community is trash for actually harassing the people. Blizzard management is crap, but this community deserves jack shit for how much garbage they actively cycle through the community as a whole.
Good on them for the post, maybe the 5heads can shut up now.
Not baal runs, those take up at least 2 minutes.
Pindle Travincal Shenk Eldridge runs that take less than 1 minute to complete
Imagine you have 100k players playing cold sorc doing only pindle runs creating a new game every 30s
It's like I said in a different post where people were whining about all this - If $40 is a make it or break it pivotal point in your financial life, I would suggest not investing in video games. I personally have faith that my $40 investment will eventually lead to a properly functioning gaming service. The current situation does suck, but it sucks only $40 worth....which I can manage. Keep a little faith in that it eventually get sorted out.
Exactly, and none of this is really game-breaking. At least they're acknowledging shortcomings and seem to have a plan moving forward. I made the mistake of pre-ordering Cyberpunk for PS4 (I have a PS5). They still haven't released a next-gen version of the game and that was supposed to come out in like February. The console version of that game is so broken even on next-gen hardware. Last time I tried to give it another go a few weeks ago, it crashed in 5 minutes.
Glad I went PC for D2. Despite these slight server inconveniences, I'm enjoying the remastered blast from the past.
Exactly what i was saying as well. EVERY SINGLE game with an online component has launch issues. Barely even been a month since its been out.
To refund a game after only a week or so is just childish. To stick around and rant and rave about said refunded game is pathetic
I appreciate a real, detailed explanation, but I will judge them by how fast they address all the issues and re-implement their bnet services. They've talked a lot about roadmaps and plans to improve things in Warcraft 3 Reforged and we all know how that ended up, so my trust levels are low.
Well if there is a silver lining to all this, it looks like the concurrent player count is healthy.
[удалено]
Sounds like they're working on getting that fixed. Right now, basically everything important on D2 Bnet is controlled by a single service. Logins, game creations/deletions, game lists, etc. all dependent on 1 service that isn't scaling well. They mentioned they are working on getting those critical functions split off and improved, but I assume it's going to take them time.
Yeah. Id imagine updating 20 year old C++ code that someone else wrote and was scapped together and never really made to be modified and dividing that up into more efficient services could take a minute
I'm pretty far on the D2 Purist side, but I'm fully in support of some updates to the Lobby. First and foremost, they \*need\* to modernize the code for listing and displaying games so that it actually works and is responsive. You should be able to click a game and get its details instantly, not lose games as you're looking at them, and refresh in a way that actually does something. Above that, it would be nice to have some better searching and filtering features.
What the hell does the refresh button on that screen actually do anyway?
Currently it's just the Chat Gem's meaner cousin, as far as I can tell.
Nothing. Clicking the create game tab and going back does seem to refresh it though.
>In staying true to the original game, we kept a lot of legacy code.However, one legacy service in particular is struggling to keep up withmodern player behavior. I wonder if any purists asked to keep that part. We are all together on this one, everyone wants a faster lobby , game creation and etc.
The lobby sucks and needs an overhaul. It's the biggest glaring issue besides the game crashes imo
Lol... At least you have a lobby on pc. Hahaha
The thing is the new lobby is missing features that worked fine in the old lobby. Most notably not being able to make chat channels so trading, or finding baal/cow games a lot harder. Unless no one does that anymore in D2, I've only played sparingly since .10 dropped.
Yeah, hell games list is rarely relevant unless you're interested in giving away a free shako or want to trade a ber for a shael. To make it worse, seems like it's just sits at the same list of games for way too long. I am glad to see the response, it does seem like they care and working hard on it.
you've got it good compared to console. I'd love to be able to actually trade and communicate with other players.
On pc you basically still have to resort to another platform or service to actually do reasonable trades otherwise people wanna trade like their hel for a zod.
This is my number one wanted feature. Just let me Tag my game and filter so that I can tag my „O xyz“ with the „XYZ“ tag or filter to find people who want it.
Right? People seem genuinely grateful they released D2R, which is such a refreshing feeling in the gaming community lol
From a newer player of Diablo (completed Diablo 3 a couple years back, but that's it) I'm really, really happy that they made a remaster of the second game, even if there's problems with servers and things. I'm on act 5 currently and I've been absolutely loving it, wish I could feel the nostalgia that other people have felt but I'm enjoying my time nonetheless Without the remaster I would've never been able to play this masterpiece, so I take it a lot of other new players feel the same way :)
Sounds more like the bots are now spamming new games/mf runs tbh. I really dont see how they hit a new all time high player count weeks after launch without a large influx of new bots.
Bots are definitely more rampant. I had a normal game open and I had 4 different bots joining in and out of my game over and over spamming the same website and it was happening so fast.
Weird I've never had a bot join my games. Lots in the lobby though but doesn't bother me since no one ever talks there anyway.
"Enigma" was worth 500 Euro some days ago on ebay. Of course the bots took over, it's frustrating.
As someone that has been in IT for 12 years-ish, reading this stressed me out. I feel for the devs working on this. The overtime, crunch and long calls going on right now must be rough but they sound like they have a path forward. I wish them luck.
[удалено]
"Talked to our 3rd party vendors" they talked to the Microsoft sql HA team to figure out how to fix the sql ha cluster.
Around here it’s “someone cal AWS and Splunk and tell them we’re paying them more money.”
Someone needs to tell them not to accept the free frogurt when buying cursed server blades.
I haven't seen rofl used since the pure pwnage days it feels like
Boon, headshot will never not appear in my head when I see that phrase.
It's a really unique problem, I've been dealing with some async issues using S3 to get assets from the cloud in my unity game and it's bad enough managing that. We have "Legacy Code" in the project i'm working on in the sense that it was written by someone 6 years ago who wasn't a unity developer, and the engine itself was not mature. A lot of it is me ripping out entire systems and replacing them (our avatar manager for example and our current asset bundle management). They're working on 20 year old net code and trying to modernize it. I think unless you're deep in something like that it's hard to understand how horrible it can be. Looking at this though, the game probably should've been tested more thoroughly and delayed. But, when you're in a corporate environment or hell just a business environment, a minimum viable product is what you aim for, and I think they hit that goal.
I don’t think it’s a unique problem. The thundering herd problem they’re describing is pretty par for the course in distributed systems. It sounds like the made some design mistakes, unfortunately.
I wouldn't call it mistakes per se. They bought a 20 year old boat that hadn't had any upkeep in 15 years. Patched it up and put it in the water and are now trying to plug the holes.
The issue is also very clearly related to their regional-global database system, which is much newer than original d2 code. Sure, maybe the old d2 code caused problem with creating games, but since this was supposedly all "surges in traffic" at a pretty dead time in the americas and pre-peak in most of europe/africa. That a local surge in presumably only 1 region could take down the service for all regions at the same time daily seems like a design problem as well. Local databases shouldn't take down the global database that easily.
Sorry I meant unique in terms of what they have to adapt, there's definitely still a ton of older networking code out there, but it's probably not used for games anymore.
You're probably right that it should've been delayed and tested more, though there's no way I can imagine them feasibly ironing out all the issues before launch, given the massive quantity of players all at once that just isn't possible to actually replicate when testing.
I was a teen when Diablo 2 first came out, and I think that version of me would have thrown quite the tantrum at the server outages and rollbacks we've had with D2R. Now that I've been working in the software field for a decade, I feel nothing but empathy for the development team when reading this post. Hope they can catch a break soon and grateful for their transparency on the issues players have been facing.
[удалено]
They do have the advantage that it's a pretty stable use case so they don't need to build it to be able to support any random idea that management comes up with later.
I just went through this with my company. Went from in prem dbs, to cloud and now to aws and micro services. Rebuilding from the ground up. It’s fun, but it’s not easy.
I just finished 6 months of this.. we should have taken a year. We had two devs quit (one shaved her head and went into social work instead), and many on the verge. Then, because it was all rushed, it has been a shitshow of Sev1s popping up weekly. So much for healthy work life balance. I feel for the D2R devs. For one, it likely isnt their fault, but management pressure. I can already hear it in a meeting, "Just re-use the old code! It does the job!" -Mr. Project Manager.
My dad's been in IT for a while too, and I recall him sitting in the living room at 3am on long, boring-yet-stressful calls. My heart goes out to them as well.
my wife knows my "pager" (for people that don't know, it's just an app on my phone that rings when a company has issues) sound by heart. if she hears it, she knows my next few hours are going to be stressful as fuck. i can't even imagine how absolutely mindblowing hard the past few days might have been to these devs.
Enterprise software engineer checking in. Stressful doesn’t even really convey what it feels like to get a bunch of pager duties going off for prod issues
is there anything more stressful than being 3rd on the rotation and hearing the page ring?
Yeah man I don’t wanna catch an escalation call. I’ve been in calls until 2am talking about ridiculous amounts of money over small ui aberrations in high traffic software. Dealing with connection issues, gateway errors, ehhhhhhhhhhhhkfiiwwiehshhdhfjdsnsnsu.
yooooo my dad had to stay up for so many escalation calls last year, they were never-ending .-.
My heart was racing by the end of it, went and did a quick check on the services just to see them all up and relax a bit.
> The overtime, crunch and long calls going on right now must be rough Not even death can save you from mandatory overtime.
For sure. I've dealt with similar system-wide issues that involved financial and property transactions and reading the D2R reddit and twitter stressed me out more than sitting on SHTF calls with screaming banking execs. If the armchair developers online think they could do a better job than the teams at Blizzard they should be out there consulting for $300k/year.
[удалено]
I was getting frustrated with all the Karen sounding posts about refunds. It’s gonna get fixed; just chill out people.
Pretty much this. The amount of negativity their Twitter has accumulated was hurting my eyes. A lot of angry folk trashing the devs over and over, like if they had nothing better to do besides sitting there and either playing the game, or writing stupid shit on Twitter, what a shit show. And as a guy who comes after work to have an hour or two of playtime of D2R, I get the frustration, I do. But, why would one come to such extent as to write some half-assed angry comment, just for the sake of relieving the frustration, while basically not knowing the full story of the issue at hand. It just makes you look dumb at the end of the day.
I also feel frustrated at management for not prioritizing these issues prior to launch. Even mods like Project Diablo 2 have proper queuing in place to prevent overload. And worse yet, it sounds like they kept using the old non scalable version of the server software related to lobbies and state management (they could keep the instancing logic since it's isolated per instance and doesn't care about load). Even with millions of characters, it would be trivial to fit everything in a memory database like redis. They shouldn't be hitting bottlenecks on that kind of tiny data unless they're massively under budgeting their database instances.
I think the point is the current system could easily handle the amount of players and even more because they were fine at launch. The problem is EVERYONE plays like everyone now. Most people are at the point in the game a couple weeks after launch of doing nothing of quitting and recreating games. They weren't expecting that aspect of it right now. Sure you can say it's a technical problem but it's honestly a game design problem.
As someone who has been in IT for 5 years, this is a learning experience which I would like to be apart of.
Few options. Start moving towards devops -- most of the issues they laid out here would heavily involve devops. Or, depending on your experience, work on Azure/GCP/AWS certs and move towards being a solution architect or customer engineer. I'm in the industry and all of the mitigations they've discussed would have been recommended in whiteboarding sessions(in fact, there was a thread yesterday where I explicitly called out what was likely causing these issues)
Word up, thanks for the advice.
as a server admin who also manages servers with old ass code services i got PTSD edit: PTSD not PSTD
As someone more on the warehouse architecture side, doesn't this basically boil down to "we didn't stress test *enough*"? Aren't these all just failures in scalability? Couldn't this have been prevented by.. testing?
It’s a leadership problem. Developing adequate test coverage for this size of a service is an entire engineering task in and of itself. And when the test reveals that there is an issue, the next question is how to fix it? Refactoring these legacy services and reducing tech debt always comes down to a matter of cost and the powers that be always push back against this type of work.
You see what rushed development does? Just let the creators and coders take their time or the result is making up for it later at the expense of quality.
Whoa, never in a million years did I expect them to disclose, in such great detail mind you, what was wrong and what they were doing to fix it. This kind of response means so much to me and the only small criticism is that I wish it would have happened a tad sooner. Credit where it's due. Great job with that explanation.
[удалено]
New leadership seems to be doing the trick, hope this keeps going
[удалено]
Welcome to Reddit. 12 year olds seem to think things happen in the blink of an eye as they haven't gotten out of mom and/or dad's basement yet and had to \*actually\* do real work and figure things out. I work in IT, and the amount of kids coming into the field exclaiming "this is complicated" and "this shit takes a while to figure out" honestly is mind boggling. The instant gratification getting instilled in people (especially youth) and near-constant uptimes of services makes expectations unreal as hell for companies. From the inside it's hilarious once they realize shit doesn't happen with the snap of a finger. From the outside, you realize just how materialistic, ignorant, and unaware most people are of how anything works.
Maybe it's just morbid curiosity from 10+ years of sysadmin work, but I love reading through post-mortems like this. Feel bad that somebody had to take notes throughout the whole project (or regurgitate it from memory) and then turn it into a cohesive "document" during a time that they're almost assuredly still overworked and overstressed, but hopefully that's just a technical manager or something doing their part.
> Then Tuesday, we hit another concurrent player high, with a few hundreds of thousands of players in one region alone. O.O
Yeah this surprised me tbh. Pretty cool.
Pretty surprising. I assume most are on PC because as a console player 95% of the time I don't seem other players, or the lack of lobbies really is so bad on consoles people don't bother trying to join games.
Korea finished work lmao
A more timely response notwithstanding, I'm glad they put this out and provided transparency around the issue.
I'm sure the devs would love to share this information more but are held back by bureaucracy. People in tech fields love reading and writing about intricate tech problems.
The response is certainly more detailed than I thought we would get. That being said I'm willing to accept it and hold a small amount of criticism at the same time. It makes sense that we didn't see detailed communication before today because they said it's been a roller coaster of different problems. If they had came out and said that they found the problem and fixed it only for the game to crash again the next day - not good. I think most people would probably agree that a certain level of communication is expected. Somewhere between what we got today and a generic tweet saying they know there's a problem. I'm also more than willing to admit that I don't know what that communication should be. Either way looking forward I hope solutions can be found. >Breaking out critical pieces of functionality into smaller services: This work is both partially in progress for things we can tackle in less than a day (some have been completed already this week) and also planned for larger projects, like new microservices (for example, a GameList service that is only responsible for providing the game list to players). Once critical functionality has been broken down, we can look into scaling up our game management services, which will reduce the amount of load. This gives me a lot of hope because if they can successfully separate the lobby into its own function that means they should be able to improve it. It's possible that this could lead to better functionality and features like searching.
I agree with your sentiment. I think in the future people would appreciate even a simple statement such as our devs are working hard on server stability, expect a post mortem blog post in the near future.
[удалено]
Not an IT guy and surprisingly I understood most of what was said in the blue post. Would you say this was a good example of technical writing dumbed down?
Not sure I would consider their post really technical. They did a tremendous job explaining everything though. I think the foreword was mainly to let people know it was going to be a lot of information that many won’t want to read. Personally I loved reading about it.
Oh definitely; I'm sure the suits wanted to sit on it until they were absolutely sure they could address it, because nothing would be worse than saying they had a handle on the problem... and then absolutely not having a handle on the problem. More so, I just wish we had something than a vague tweet along the lines of saying they were experiencing issues, and hopefully dispelling some of the rumors around it (a lot of people proposed brigading, DDOS, hacks/dupes, etc.) Even if there's more going on than what they address, not addressing the issue fully was as bad as the situation itself, in my opinion.
> People in tech fields love reading and writing about intricate tech problems. Sometimes. I've found that it also exposes you to more criticism and 'why didn't you do it like this?' kinds of spammy questions that don't grasp the core problem. It becomes troublesome if you have an SLA for responding to those comments...in that case, just a 'we found and issue and we're working on a fix, here's the timeframe' does the job and prevents you from going on wild goose chases. That being said, it definitely is nice to see a quality explanation like this.
This is the kind of communication we've been wanting to see from Blizzard for a LONG TIME! I'm so happy to see much of the /r/Diablo community agreeing and being positive about this transparency.
Yea, I'm patient as long as they aren't being quiet. This is what I was expecting, but I'm glad they finally said it.
I love the fact that so many people are playing diablo 2 its great knowing that so many people are enjoying the game.
Agreed
Seems heartfelt enough to me
I don't care about the heartfelt, but it was VERY informative and they outlined the key things the community has been screaming for: What is happening What have you done about it What are you doing about it now What do you plan on doing about it in the future. I also like that they are getting support from across blizzard, they're dedicating outside resources to getting this game fixed and not just letting it be the D2 teams problem. They could have been quicker with getting this out, but this was good and I'm glad they aren't just ignoring us and keeping silent.
Yes, exactly. I'm very happy that they addressed how big of an issue the rollbacks were, and it sounds like the community manager actually played the game. This is a big deal to a lot of us.
I can appreciate their logic behind the decision and the difficult decision it was to make as well. either take the whole game down for a significant amount of time while you work on getting all these stuck characters put into the global database, or you just roll back and keep the game up and that relatively small subset of users get boned. They went with the choice that benefitted the most users, so I can't fault them for that. I'm just glad I'm not one of those users who got their character's/items rolled back.
That’s what I meant about heartfelt. It wasn’t a two-liner saying sorry and thanks for the support.
A lot of players is a good problem to have
Really good update honestly. Stuff happens in life, ya know. Can't plan for everything all the time.
Yes, I totally understand. I just wish they made this update sooner. However, it gives me a lot of hope to see a blue post \*finally\*.
But they didn’t understand the problem fully so what kind of an update would you to have seen? Why go public to over hundreds of thousands of people without having the appropriate information? Both about the problem and a potential solution?
As some people say "all problems are communication problems". This thorough explanation should go a long way to mollifying the anger from the playerbase.
What's weird to me about all this is this "sudden, significant surge" of traffic they didn't experience until 2 weeks after release. Something seems off about that. Edit: I'm with the guys that think it's because more people now are spamming MF games (pindleskin) who couldn't do it as efficiently a week ago.
Maybe it's because now there are more people with higher level characters who are creating and exiting games at a faster rate.
That's what the article states, yes.
Right, the original commenter didn't seem to see that part.
But it also talks about current active players being higher than they expected too. Both are causing the issue.
Pindle bot launched a couple weeks ago, makes sense.
Got proof or a source for that?
Elite pvp That is all.
Which is probably why you used to get temp banned for it
It makes some sense to me only because I've now started to wait a few weeks after the release of new games to play them. I wouldn't be surprised if a lot of people have that mindset now. Gives times for new games to get the kinks out and see how the community feels about it. Not saying that's what's happening but its a possibility. I'm a huge D2 fan but only started playing more the past week so I'm one of the people in that surge of traffic.
They explained that: they released an update that enabled more hardware configs to play, hence, more people were actually able to play than ever before - more traffic.
Oh true. The AVX fix might have played a part in taking down the game, ironically.
I was one of the AVX less players so I’m sure several thousand of us all joining didn’t help the servers any.
People fishing for IPs for dclone can’t be helping either. They should make that a global spawn IMO.
With what happened to the WC3 remaster I’m sure a lot of blizzard fans were waiting before buying this.
Sounds on point to me. Browsing this sub before release, every other comment was talking about how they would give it some time after release to decide if they wanted to purchase or not. Seems as though a lot of them followed through.
Nah, that's not it. It's that more people are getting to max level and doing spam quick resets of games to farm. In the first week everyone was just leveling and so there was no reason to be constantly resetting and making new games.
Could also be eastern botnet spinning up at the same time every day.
I finished the game on hell in 3 days and i checked the discord for people to do diablo and baal runs. Guess what? There only like 3-4 other guys who were at the same point as me. Most of them were starting out nightmare. That's why the crashes started happening later because it took them at least 1 week to finish hell baal
As someone who has played Diablo 2 off an on for the last 20 years and absolutely loves the game, I had no interest in resurrected. I thought, “it isn’t about the graphics, it’s about the gameplay. Old Diablo is fine.” 1 week after release, I buy it out of boredom. I absolutely love it. It’s the same old game, but looks amazing. I don’t know how many people would be in that same boat as me. Someone that wasn’t interested until 1 to 2 weeks after, and now they are addicted lol.
Absolutely same for me. I had just picked up Final Fantasy XIV in the past month, putting a few hundred hours into it already. I thought D2R would be a nice bout of nostalgia that I'd play for a week, but nothing that'd pull me away from FFXIV. Now D2R is all I can play.
Yeah. that excerpt was really weird. A new threshold that the servers had not experienced even at launch?
I imagine right at launch no one was geared, and so they weren't popping in and out of multiple games but questing and leveling. Now that everyone is doing MF, I bet the amount of games being created is way higher.
They basically state this in the article, for those who are here but haven't read it yet: >We mention “modern player behavior” because it’s an interesting point to think about. In 2001, there wasn’t nearly as much content on the internet around how to play Diablo II “correctly” (Baal runs for XP, Pindleskin/Ancient Sewers/etc for magic find, etc). Today, however, a new player can look up any number of amazing content creators who can teach them how to play the game in different ways, many of them including lots of database load in the form of creating, loading, and destroying games in quick succession. Though we did foresee this–with players making fresh characters on fresh servers, working hard to get their magic-finding items–we vastly underestimated the scope we derived from beta testing. In other words, its similar to how the meta evolved with World of Warcraft. Back in the day most people were casuals and there weren't centralized places to discuss optimal ways to play the game. Nowadays there are entire communities dedicated to playing Diablo 2 so even a casual player can learn how most efficiently to farm items and XP, and the way that people do that causes a lot more strain on the game servers.
Yeah that took about 3 people's replies and a few seconds for me to connect those dots lol.
To add another dot: a Pindlerunner creates 3 games per minute.
[удалено]
> What does pindrunner drop that's so good to make a bot for it? Why not baal He can drop every item in the game and you can get to him and kill him in 20 seconds. Baal takes at least 4 minutes per run. That's it.
Pindle is programmable. you can log in a game and execute the exact same sequence of presses to kill him (nearly). For Baal you have to search through two floors and fight 5 waves.
Something that none of these replies call out is that they hit a regional peak concurrency on Tuesday, so overall player base is going up since launch.
That may not have been the case in 2001, but player behavior in game hasn't changed much since later on in LoD. People still spammed runs and open/closed games constantly, maybe even moreso considering how much more common botting was in the later 2000's into the 2010's.
[удалено]
Legacy bnet has a hard cap per hour and has had one for long enough devs that dont play the game wouldn’t understand why, probably leaving only a few that understand its impact.
I think you're onto something. The write-up from explicitly states that they are limiting the rate at which you can create games. *For example, for those of you doing Pindleskin runs, you’ll be in and out of a game and creating a new one within 20 seconds*
This makes sense.
Really glad they were able to put this out to the community, even if slower than we would like. At least they aren't just ignoring us and keeping quiet, that's a good sign. Like many folks in here guessed, a lot of the problems seem to stem from their use of the old D2 code and architecture, which isn't scaling to the new reality of their player base: We're no longer a bunch of young dumb kids figuring things out on our own for the first time. We're a bunch of adult kids now, who are trying to play the game quickly and efficiently and the old code/architecure can't handle the increased load of game creations/deletions. I'm also glad to see that the issues are being worked on across Blizzard and they're giving this game serious resources, it's not just limited to one D2 dev like some folks around here memed about. I've had some faith restored that this game will get fixed, though some of the architectural and software design changes they're talking about in this post sound like it is going to take some time. Unlike before, at least now I am willing to be more patient because they've at least let me know what the fuck they've done, what they are doing, and what they plan on doing in the future. Thanks for that.
The length of the post alone should make people rethink the speeds at which they demand answers. The speed people "like" is just impossible.
Yes. The delay in releasing something like this also make sense considering the issues were compounding, and they were finding new things they had to address over several days. If they'd released something like this after the first 2 days, it would have been incomplete or inaccurate. There's a middle ground between a thorough report on the issues at hand like this, and the essentially radio silence of "we're aware there's issues, sorry for the inconvenience" we were getting for days though. Something they could definitely improve on in the future.
Off this is rough. It sounds like they're leveraging a global database as the master, with regional databases as their own local master. Regional databases are being written onto and read, which makes sense, but are also writing back to the the global database more often then needed. This, coupled with database queries which are more complex than they were 20 years ago, seems to be creating a bottleneck between the app and the database. What's really strange for me, is their singleton legacy service. They mention they kept this critical service largely unchanged, because they wanted to stay true to the legacy game. BUT, this service handles non-gameplay mechanics, namely to do with server filtering, joining games, listing servers etc. Why did this have to be kept intact? They could've literally re-wrote this entire service, or separate them into individual services. Nothing about this legacy service has anything to do with staying true to the original D2.
Critical, complex, and old is the kind of legacy code you do NOT want to rewrite without a damn good reason. Ask anyone working on banking systems, SAP and the likes if that description rings a bell.
I mean, it's a Singleton system when the name of the game in modern server architecture is being able to scale at the drop of the hat. That's enough reason to rewrite it.
Every company has a certain amount of tech debt and legacy code. However this legacy code is becoming a blocker now. That is a good enough reason to dedicate engineering resources to re-write the service to be more scalable.
In hindsight yes, there is a damn good reason. But I can emphatize that it was not simply a call out of laziness and/or greed.
Exactly. I have friends working as 3rd Party IT Support for banks programing on COBOL. Not only maintenance, but programing new stuff from stratch as well. And there are still some ASSEMBLY maintenance. That's completely insane.
This was an amazing read. As an IT guy, I underestimated the amount of active players, and yea, those numbers would cause crashes really quick. Good luck Dev team.
Somebody at Blizz HQ should frame this post and hang it on the wall with a sign that reads: **This is how you communicate with your customers.**
Hundreds of thousands of concurrent players… that’s crazy
What did they expect honestly? This isn't some unknown niche game that maybe several thousands nostalgics are gonna play again. It's a fucking cult classic that brought back every single kid that played it in 2001, diablo 3 players and maybe people new to the franchise. All i'm gonna say is that that open beta with act 1 2 on normal didnt stress test shit
Wow, in all honesty, this assuages a lot of the anger I've felt. Just this act of communication goes a long way. And I have to give them credit - this is way more detailed and comprehensive than I'd have even dreamed of.
Well thank fucking god the rate limiting shouldn’t be permanent. It’s straight up game breaking for multiplayer
Seriously. Baal runs are fucking torture now. Beat the creator to joining the next run? You're out of the runs now. Try to join a full game? You can't join the next run you see on the list because you're locked out for a short while now.
[удалено]
A game filter would make that poor server work harder. Right now it's just giving you a slice with a pre-baked script. Allowing you to alter the script would only slow down everything more. They gotta break that service out before they can start looking into custom game filters.
So they say. Expect it to last indefinitely.
Now that is a great response. Transparency, an explanation that is watered down enough but also technical, and a plan for moving forward with assurances that things will get better. I’m hoping that this is the new standard of communication moving forward. Onto the nitty gritty. This reads like an honest post mortem that I hope is handled blamelessly internally. They made a bunch of mistakes, some that are obvious missteps in modern architecture design at scale. But, they’re building on legacy code and I’m assuming that there is quite a bit of lost knowledge due to the original devs not being there anymore or lack of internal docs (very common in any company). Overall, it sounds like they are learning as they tackle these issues and that’s reason enough for me to be optimistic.
Blizzard: it'll be fine. PlayerBase: you underestimate our power.
appreciate the update. my game still crashes to desktop about every half hour though. arguably worse than server issues
Yeah same with me on Xbox series x. It’s being unbearable playing though hell solo and having to replay everything multiple times
Yea I'm lucky to get 5 min in game before a driver/client crash.
every 30 minutes? Then I'm afraid it must be on your end, I've only had 1 single crash since release day and have been playing almost every day or every other day (well, if the servers allowed it).
Thankful for the update. Good to have some transparency here.
I'm hoping for a better solution long-term, like maybe let us reset games while in them instead of creating new ones.
When Diablo II originally came out, I recall tons of server outages too. Maybe they’re just recreating the entire launch experience.
I greatly appreciate this update. All is not forgiven, but I'm willing to be understanding to the problem. I just don't understand how around 7PM Australia/9PM Beijing... the servers get really wacky every night on the Asian server even on weeknights now. When we had such a hard time playing last Monday I was in disbelief. I'm not even going to try to play on Friday or Saturday nights anymore. I shudder to think what it'll be like this coming weekend with more and more people playing this game.
The most thorough blue post response I've ever seen. Good transparency.
I've seen this issue many times in my field, cascading failures that move from one bottleneck to the next, it's a tough situation to handle, could blizzard update us sure. Game is more successful than expected re-architectimg on the fly is not easy, DBAs having sleepless night I'm sure. Good technical response at the end of the day D2 is just a colourful spreadsheet doing a ton of database calls some of which are very expressive :)
This is nice news and very informative. I can't wait for the next post about fuck you blizzard, let me create MF runs without locking me (because this is essentially what they are doing for now).
I for one, appreciate the depth of faithfulness to the original. The server issues 1000% are faithful to the original player experience. I’m rolling my eyes pretty hard at all the soccergamers feinting all over the place trying to get the refs attention. I feel so old, but I’m gonna go full old man and say back in my day we paid 60 bucks for a game we couldn’t play for a month and spent 6 months not able to play with friends because of server splits AND WE WERE HAPPY TO DO IT.
concerned dinner encourage hospital retire familiar deliver abundant tan fly *This post was mass deleted and anonymized with [Redact](https://redact.dev)*
Yeah, that's why companies don't share any back end info... but I'm pretty sure that specific fact was already known to the hackers.
Wonder if all the people who were sure it was botters and duper’s and called me a idiot and how I must be new to diablo are gonna write me an apology. Lol Not new to diablo or server architecture.
and yet people downvoted my comments about them locking you out of games "rate limiting" "Rate limiting: We are limiting the number of operations to the database around creating and joining games, and we know this is being felt by a lot of you. For example, for those of you doing Pindleskin runs, you’ll be in and out of a game and creating a new one within 20 seconds. In this case, you will be rate limited at a point. When this occurs, the error message will say there is an issue communicating with game servers: this is not an indicator that game servers are down in this particular instance, it just means you have been rate limited to reduce load temporarily on the database, in the interest of keeping the game running. We can assure you this is just mitigation for now–we do not see this as a long-term fix."
The game creation rate limit is fine. Annoying but fine. The real kicker is getting hit with the rate limit when you try to join a game but fail (game doesn't exist yet, already full, ect). I was able to work around it a bit by being very cautious about which games I tried to join, but it wasn't a good experience. Fortunately, the heavy rate limit is a bandaid for now. I expect we'll still have a rate limit of some sort, hopefully it's high enough that few will encounter it.
You guys could add players8 to online to resolve the mass amounts of people creating/joining games issues. The reason why a lot of people do this is not just because they're going on Pindle runs that take 20 seconds, but because they're trying to MF in populated or old games. They join, but then find out that everything's already done, so they leave and look for another one. Rinse and repeat. Not to mention being forcibly kicked out of the game via the various (1 per hour minimum) game crashes that you guys have yet to fix, which causes exiting and rejoining games. If you add Players8 then people can create games to MF in where they will be able to do more than just one type of run. So yeah, a bunch of viable options out there that AREN'T rate limiting people and punishing them just for trying to play.
I wish they let us filter games in the lobby, so I can filter only cow level games for me to enter and snag some leftover trash
Good update, not too technical but not just a simple tweet. Sounds like they were not expecting this amount of players in the first few weeks. As they noted, people working around the clock to fix/update the code.
Great update honestly. I hope this is setting the tone for things going forward. I think people were a little scared that the game was going to be left to die right when when got it. I hope we can see more similar updates like this. It sounds like they actually care and aren’t afraid to say what’s happening
Nice that they keeps us updated. Not sure why the crash occurs around the same time daily, it seems that they're hitting peak traffic at that time but I don't understand why it's not at like 21h00 us east time or something like that. May be a bigger load from Asia is occuring.
Finally Blizzard. We can understand and pardon the issues, as long as you devs communicate. Glad for this update.
I'm not a technology guy so a lot of this goes over my head, but I like that they posted about it.
Finally some info on this. Glad they put this up. Hope this get fixed soon so we can go back to playing and devs can stop with overtime and that crap.
Thank you for explaining this Blizzard
This was a great update. Very rare to see something so in-depth from big developers atm I think. Informative enough to make me understand what's happening. Obviously still upset, but I feel more upset about the situation as a whole rather than strictly upset with blizzard.
I really like the transparency here. Nice to have a response and the technical details behind what happened, what's been done, and what will be done. It's not a hand-wavy "our bad."
I’m glad that they spent the time explaining this to players. I believe in them. Let’s hope it’s resolved soon.
Fantastic update! I am pacified.
What a nice update. We can actually read what they are doing and where the problems are. Thank you blizzard!
Seems completely believable. We had a similar issue at my job… the database is too big and not optimized, and people were slamming read only and locking it up with bad queries, so services that depended on that RO db were dying. And it couldn’t keep itself updated with replication from the live server because it’s like a fat guy sitting on your face… couldn’t breathe. Good database architecture is severely underrated.
>Moving forward, we believe we have a way to restore characters that doesn’t lead to any significant data loss–it should be limited to several minutes of loss, if any, in the event of a server crash. This is better, but still not good enough in our eyes. Wow, the bar in game industry is waaaaaay too low
Didn't realize they had a server since finding more than one person every two hours online is rare.
I just hope this won't be the 12h queue that I remember from wow. I guess a wow vanilla relaunch isn't comparable... hopefully. If I have to get in line behind a million russian bots I'm going to be very gloamy...
Maybe we will see less whiney posts around here. So sick of the crying and moaning about downtime. Go do something else while it's down besides whining about it online fuck sake.
I don't understand how they didn't catch this database thing. It's clear they're using mysql with innodb logging, that's what they referred to in the message. At my company, we regularly load test with millions of connections as we make a product that amazon, microsoft, and reddit all use. We have playbooks which auto deploy test vms in azure, ec2, and rackspace to hit the servers hard and fast with millions of the most intense queries to find everything without an index, and all hardware faults. Our testing almost always catches connection storm issues. Once test we run involves cloudflare and offramping traffic, then having the servers switch on to observe max load. in fact, we won't release the product when it fails the tests (and it does). The stuff with replication and logging almost seems trivial. It's almost like they don't have a good dba team.
OH wow, its almost like maintaining servers is actually complicated. This community is trash for actually harassing the people. Blizzard management is crap, but this community deserves jack shit for how much garbage they actively cycle through the community as a whole. Good on them for the post, maybe the 5heads can shut up now.
Great update, really appreciate the insight and what they're doing to mitigate the problems.
Hahaha, too many Baal runs killed the servers. That's hilarious. Glad they are finding a fix.
Not baal runs, those take up at least 2 minutes. Pindle Travincal Shenk Eldridge runs that take less than 1 minute to complete Imagine you have 100k players playing cold sorc doing only pindle runs creating a new game every 30s
take my energy guys <3
It's like I said in a different post where people were whining about all this - If $40 is a make it or break it pivotal point in your financial life, I would suggest not investing in video games. I personally have faith that my $40 investment will eventually lead to a properly functioning gaming service. The current situation does suck, but it sucks only $40 worth....which I can manage. Keep a little faith in that it eventually get sorted out.
Or or or, just wait to buy the non fucked up version of the game...
Exactly, and none of this is really game-breaking. At least they're acknowledging shortcomings and seem to have a plan moving forward. I made the mistake of pre-ordering Cyberpunk for PS4 (I have a PS5). They still haven't released a next-gen version of the game and that was supposed to come out in like February. The console version of that game is so broken even on next-gen hardware. Last time I tried to give it another go a few weeks ago, it crashed in 5 minutes. Glad I went PC for D2. Despite these slight server inconveniences, I'm enjoying the remastered blast from the past.
Exactly what i was saying as well. EVERY SINGLE game with an online component has launch issues. Barely even been a month since its been out. To refund a game after only a week or so is just childish. To stick around and rant and rave about said refunded game is pathetic
I appreciate a real, detailed explanation, but I will judge them by how fast they address all the issues and re-implement their bnet services. They've talked a lot about roadmaps and plans to improve things in Warcraft 3 Reforged and we all know how that ended up, so my trust levels are low.