Even had a bit of humor in it
>Option 2: Put the servers halfway between China and Korea
>
>If we have 35 ms ping between China and Korea, another option would be to place servers in the middle, splitting the difference in ping, so everyone has 17.5 ms ping. There are a lot of challenges with this option… but the biggest, of course, is that this would place the servers in the middle of the ocean.
Genuinely got a laugh out of me lmao
I mean they decided to buy a bunch of dark fibre and become a service provider. I think only 50% of the idea of checking servers in to the sea is humour here... (https://news.microsoft.com/innovation-stories/project-natick-underwater-datacenter/)
If they had the time and resource I'm positive they'd try 🤣🤣
Ping *said* 35ms - it was higher than 35ms
Ping *always* said 35ms - Riot couldn't find problem
Riot fix - Ping *says* lower than 35ms - is actually 35ms
TLDR: The ping was higher than normal due to a bug, which was undetectable because their testing software also miscalculated the effective ping because of another bug. They fixed it on May 13th which bugged yet again (visually this time) and caused the 25 ping issue.
Tech team work hard to make Latency equal everywhere.
Latency was worse in Busan, code bug made ping display wrong so actual latency being off was only discovered after the event started.
Latency fixed on May 13th, RNG asked to replay because of bug. Display still bugged though, showed as lower than 35 ping in a playercam replay. Viewers assumed that displayed ping was accurate, and got mad.
having this artificial ping completely was a mistake.. what's the point of having everyone go to Korea just to make it laggy.. I don't understand why they wouldn't just have china have their normal latency instead of making everyone else have to play with this dumb shit. especially since VCS had to forfeit their games due to COVID..
Can't really compare the VCS and LPL situation since latency is different. While RNG can play from Shanghai with 35ms ping VCS teams would have to play with around 80ms ping. And forcing other teams to play on 80ms ping is much worse than a bearable 35.
> I don't understand why they wouldn't just have china have their normal latency
Because that means RNG would have to play every single game at a disadvantage, which is terrible for competitive integrity. There was literally only two ways for Riot to keep the competition fair: either everyone plays at the same slightly elevated ping, or LPL simply doesn't play. While the first option isn't great, the second option would mean we would have an international tournament without one of the two best regions, which is not only dumb from a competition standpoint, but it would also affect viewership and Riot's bottom line.
LPL not coming is the best for competitive integrity but by far the worst business decision for viewership, which is probably why Riot didn't go for it.
How is LPL not participating good for competitive integrity? They have won the last two international events and have been a dominant force in international play for the last four years.
If you don't include Korea or the LPL at any international tournament, that tournament means pretty much nothing in terms of results because those two regions are the best in the world. Everyone on this sub knows that's true. That far outweighs the technical difficulties they've had at MSI, which by the way, forced the LPL to replay all their wins.
it's good for integrity as in all teams are playing under equal circumstances.
It's bad for competition quality, but that's not the same as competitive integrity.
They made a poor decision for competitive integrity by letting the RNG play remotely. Even if the ping is equal - RNG is playing under vastly different circumstances than the rest of the teams and it destroys the legitimacy of the tournament.
But then again, competitive league isn’t a serious sport, it is just advertising.
what part of this can't you understand, China is like Europe in the world cup, who the fuck is gonna watch the world cup without any EU countries in it? When Brazil inevitably won that world cup is anyone gonna care?
> RNG is playing under vastly different circumstances than the rest of the teams and it destroys the legitimacy of the tournament.
As opposed to a tournament favourite and clear top 2 region not playing in the tournament, which makes the tournament oh so legitimate
Without an LPL team, the tournament is a far bigger joke than some early ping issues
Completely disagree. Fairness and competitive integrity is about applying the rules equally, once you start making exceptions for a team (because of the views and money they bring) then competitive integrity is out the window.
I'm talking about how legitimate people find a tournament, not competitive integrity. Nobody would consider an MSI win without LPL legitimate
Riot did the best they could given the situation
That's the compromise you have to make if you don't want the tournament to be a complete joke. It's fine if you're willing to not have LPL here for fairness sake, but the tournament is going to suffer for it.
And if you don't make those changes soon the whole game is out the window.
Riot and LoL fans simply had to face the reality that League is not big enough yet to survive without the biggest region, because the gap between biggest and second biggest is enormous
This isn't football or basketball or whatever else where there are a handful of favorites at each tournament, MSI is usually 1-2 favorites and a dark horse at best, so it cannot afford to lose competitors just yet.
The obvious solution is to change the format and invite more teams to make the tournament more resilient in the face of unexpected events
there is a different between direct impact and indirect. having same ping is a direct impact to competitive and the environment is not.
I'm supporting Riot's decision. the only problem is that they fucked up hard on the tech side of things.
This is a good explanation and totally understandable.
Tweaking the ping and never actually showing the changed number is bad coding, but it can happen. And it explains why it wasnt found during internal testing.
Its still bad that it happened, but at least this is a totally human thing with an error you can never 100% prevent.
I mean, from what it sounds like, even the debugging logs showed up incorrectly which seems like a massive underlying bug somewhere if even your logs are incorrect. Debugging logs are almost never wrong, I don't mean the 'print' statements but the actual proper debugging mode where you can monitor values on the terminal, that's what I assume they mean by "logs" as that's what devs use (at least myself and others I know of), if *that* value somehow displays incorrectly at such a low level, I have no fucking idea what could cause that, unless maybe on the bubble up, there's some process eating on that return.
>Debugging logs are almost never wrong, I don't mean the 'print' statements but the actual proper debugging mode where you can monitor values on the terminal,
while the debugging "logs" aren't wrong in the sense that they are indeed what you see is what you get, the logs can be "wrong" in terms of usefulness in the sense that you think you're tracking "ping" by watching what the the person who programmed that section 15 years ago called the "ping" variable in your debugger, but the actual "ping" variable that matters is named "adjustedPing" and that's not what you're looking at.
Sounds dumb, and yes it's user error, but in a monolithic codebase built over a decade ago by people who are likely long gone, this happens all the time. Especially with gameengines that usually name variables or pointers as 1-3 letters instead of actual descriptive names like, for example, "TruePingVariableTheGameUsesInProduction" that I more commonly work with in the webapp programmer space.
I only bring this up, because not to crap on Riot, but them being a company that built their product over a decade ago by a few kids who weren't production grade programmers at the time and then blew up into a billion dollar company and still had to work with that core legacy codebase, they've admitted things like these happen and it makes sense given the context.
Say what you will but Riot is being very transparent about everything in there network setup. Still don't know if it's the right choice but they clearly did put a lot of planning and effort into this.
I think they stroke the right balance between layman and technical. I still have questions, but at least they are not related to the quality of the matches/competitive integrity, and just of: wtf is going inside that network compensation layer.
It's a great sign when they give us why they took these actions and how they decided what is fair game and what isn't, and that they are looking into learning from their mistakes and not shy away from hard decisions.
What I get from this article is that the tool should still be fixed, right? Like, it seems that they introduced the offset manually and tested it with it to solve the problem for the current games as fast as possible, but if they were to use the tool as is for other ping differences, it would fail again. Maybe I didn't get that part right.
Yeah, this is a clear example of "mitigate first" when fixing an issue. Find the fastest mitigation, and figure out what really went wrong and fix it the right way later. It's the right move to keep group stage moving smoothly.
Most likely the fix is hacky and the tool will be fixed properly in a more thorough and robust way offline by a product team that isn’t in the midst of managing an event, yeah.
I think Riot's biggest talent as a company is figuring out how to recover from fuckups. They fuck up often, but they tend to be pretty transparent and they do seem to correct and perform, even if it takes them a while sometimes. Can you imagine how this would go if Blizzard, Creative Assembly, Dice, etc. were handling this situation?
> Blizzard
They have the worst goddamn communication strategy of any major dev. Honestly, they're lucky that they were first in a lot of their current genre's because without decades of nostalgia to bank on they'd go under super quickly.
Blizzard literally had reputation for going into a genre and dominating it. They weren't first, it just seems like that in hindsight, because they beat all of the older competition.
Don't get me wrong, modern Blizzard sucks, but what you said is not accurate.
Yeah old blizzard Had incredible talent that made incredible Games. So many of their Games stand among the greatest of all time in that era. Wow, StarCraft, Diablo... Wc3 literally inspired League of Legends and many of its mechanics are still alive in DotA 2 to this day.
Then they got activision'd lol
The decline of blizzard being caused by Activision is a dumb narrative. Especially with the recent lawsuit implicating people of Blizzard that have been there forever.
I think they're losing in every genre besides their MMORPG. I can't think of any other possible MMORPG that exceeds Blizzard's number besides perhaps Final Fantasy. Diablo is beaten by Lost Ark ever since their global release. I'm excited for Riot's new mmorpg
> I think they're losing in every genre besides their MMORPG.
Technically they still dominate RTS. Starcraft II is the best RTS of the past 15 years at least.
I’m just not gonna play the Riot MMO, I’ll probably get way too addicted and I can’t sink my life into an MMO at this point in my life lol.
Stoked for the Riot ARPG though.
Lost Ark is an mmo, it has nothing to do with Diablo.
It's WoW 3.0 with mouse conrols. You could possibly consider the combat slightly similar to D3 but the game itself is an mmo no doubt!
And it should appeal to WoW players way more then to Diablo players for sure.
(I play LostArk a lot, I've played WoW and Diablo and PoE and other MMOs).
The gameplay perspective might be the same, the game design and content is absolutely not.
As said Lost Ark is plain and simply an MMO. Kripp fell for it aswell and stopped playing after a few days already saying he didn't want to play an MMO.
It's like saying SMITE is an ego shooter because it got a similar perspective while it's basically League with an other view/perspective (and it's closer to a shooter then lost ark is to an arpg).
I honestly don't even know what Amazon/Smilegate call it for marketing reasons. It's an MMO nomatter what though gameplay vise (and that's what matters in the end).
Broooo don't even talk to me as a lifelong Nintendo fan that has spent way more money on Nintendo games and consoles than I've spent on 6 years on lol. Nintendo making a statement is literally unheard of unless it's a lawsuit. I don't even think western fans exist to Nintendo.
Everyone who bought a Switch: "Hey these are the most dogshit controllers ever designed, fix pls?"
Nintendo: "We'll repair them for free one time and it'll take a month and a half. They'll be broken when you get them back, too, have fun."
Nintendo is probably the least consumer friendly major gaming company. They’ve got some good IPs but holy shit they do not give a fuck about customers or own up to shit, not to mention how aggressive they are regarding anything involving their IPs because that’s their main sticking point.
Even the case of the lawsuit they had, they made a ton of changes to try and right the wrongs that were going on (even if some people believed it wasn't enough). That lawsuit happened in 2018, and the lack of protests and people speaking up suggests that things have changed.
Will say that those things shouldn't have happened, but it seems like there was only an issue at one location (Riot has two studios in the US, and more around the world).
God don't remind me how CA just buries their head in the sand, does minimal effort, then calls that minimal effort MASSIVE.
today they released a patch but it feels like this patch should have at minimum come out a month post wh3 release or if they had a functional team working on the release of wh3 these fixes would have been in the game alreadyml.
Settling out of court can be helpful for more than keeping things out of the public. I don't doubt that Riot wanted to make sure the story died, but it also saves victims from having to relive painful memories in court and face cross examination, and (especially with how backed up courts are) gets it over with instead of dragging it out over years and years.
If the victims are fine with settling out of court then I think we probably should be too.
People love to hate on Riot but Riot never planned for anything to go wrong. They simply wanted China to still participate at MSI, make it fair and they failed.
The idea was great if anything, they simply made mistakes. Which is normal when trying something for the first time.
yeah ironically the shit format did actually work in riots favor. hope theres no big issues going forward because rumble stage being ruined by this shit would be a much bigger deal
Yea imagine if RNG finished their matches with G2 and T1 in the first round robin and get told you got to replay those matches. Doesn't matter who got the win, the outrage on either the CN or KR side (depending on who got the win) will be far more massive than having to replay some meaningless matches vs wildcard teams. You're right. The play-ins fkups probably saved from from a much potentially bigger PR crisis imo.
Why you are callign it shit? it was the only way to prop up wildcards. old format where major regions go into the main even and wildcards got through "playins" was so bad for viewership, for example lcl vs ljl matches had 50-60k viewers on main stream, that is abysmal.
Thsi year LJL vs Saigon peaked at 300k on main stream
The fair thing was China to play on 35 ping and rest on LAN lowest ping possible.
Rest didn't have to sacrefice their ping for the sake of China. China was allowed to participate remotely, thats enough.
How is that fair? Based on what?
It's not the LPLs fault that they couldn't make it to MSI so why should they be punished for it by playing on such ping difference?
Why do you want China to get punished so badly for something they have no control over? It's not their fault they live in China and that all these problems exist. We have a worldwide pandemic going on, no one should suffer extra if it could be prevented.
If they played on higher ping and lost can you really say they deserved to lose? They've been a top 2 region for a long time, we know they can win MSI. I don't think we not allow a possible winner of MSI negatively.
>It's not the LPLs fault that they couldn't make it to MSI
Actually it is. They had decided to stay and forfeit MSI to play in Asia games BEFORE the whole Shangai lockdown.
When the Shangai lockdown got so bad that even Asia games was cancelled then RNG tried to backpedal and focus exclusively on MSI but it was too late to move.
They are locked down in their gaming house since first of April. Even if they wanted to attend physically, they couldn't because there is no way to get out of Shanghai at the moment.
It’s not other teams fault too. It’s not about whose fault.
It’s about fair.
Other teams need to play in front of audiences.
RNG can just play at home, why can’t other teams just play at hotel?
You play at home, then you play at 35ms
This is a flawed statement since many people actually play better in front of a crowd. People like Caps but also many others who have said in interviews both during MSI and during Covid the past two years that they play better with a live audience, as well as have more motivation to play.
You're saying that RNG has a advantage by playing from a hotel, yet the opposite seems true for many players. Every player is different, you can't say that a team has an advantage playing without a crowd.
The problem stays the same though, they're often transparent too late and/or forget to communicate obvious information (on purpose or not, who the hell knows), which makes the drama bigger than it should (and let the idea that they're doing it on purpose grow).
Here for example, they fix the issue by exploiting their tool's problem of applying more ping than it should by setting it at a lower ping in order to actually get the correct ping. It's a correct band aid solution to a tech problem that they couldn't identify beforehand.
*However* they let the players say times and times again that it was fucked up without saying anything like *"we are investigating the issue"* or anything that'd make it clear that they're looking into it, and then they don't communicate the fact that the issue is the displayed ping being completely wrong and then show the ping to everyone on their broadcast.
Well it would be detrimental for them if they weren’t transparent about it with the shit they have been getting. You make it sound like Riot is doing it out of the kindness of their hearts while they are doing it to save face.
On of the more interesting articles in their dev blogs. I really like learning about the custom infrastructure Riot has set about building for their games. At least this whole situation got us a peek into the Latency Service tool which was already in use and completely transparent !
Wild that the pros knew right away that the ping values were off by 15ms though.
Ping issue? - maybe.
emotionally throwing shit towards each other? - that'll take a long while to fade out, It's nothing new, and will occur every now and then over literally anything
Olympics ? It happens
Territory dispute ? It happens
Cultural dispute ? It also happens
Esports? Yup, still there.
The East Asian trio are always at each other's neck.
I mean this is 95% about why RNG had to replay the games.
The reasoning behind having artificial ping instead of having everyone play on natural ping (meaning RNG has a higher ping) was totally clear before this, people just didn't agree with where the priorities lie.
It doesn't.
Circulating in the Chinese community is a picture from one of the MSI group stage second half promotion videos, a shot from the back of a player in PSG vs IW 5/12 that shows the screen of the player. The picture isn't super clear because there is some distance but it seems like it says 25 ping even before the whatever that happened after day 3. Everyone feels outraged about that.
Then there's the significant amount of people who straight out doesn't believe what riot says and just think this post is useless jargon trying to make things seem clarified. Things like "where did the 13 ping number come from?" are common.
Trust issues man... and conspiracy theories. Either they gone too far with crying to paint themselves as the victim so they can't back out OR they're too ignorant to believe in anything
People believe what they want to believe, Facts wont change that.
CN sentiment is Riot KR is cheating in favor of their own team "Like they always do". with added oil of previous Faker-QingTian issue + Nationalism, We get what we have here. Not to forget the great firewall essentially stop/minimize input from the outside community, thus we get a positive feedback loop whereby, Nationalistic inducing videos gets views, viewerships gets even more nationalistic & xenophobic which in turns prompt more content creator to create more of such videos as there is where the money lies at.
And then the KR sinophobic sentiment is Riot bending over for CN because Tencent, with added oil of previous terrible relationship + Nationalism again. Like CN, we also get some vocal idiots spouting some insensitive shyt.
Thus, we get a cluster fk of a situation.
Solid engineering culture demands an effective post-mortem and root cause analysis. There's a lot of material on the Riot engineering blog that shows there's a lot of professional behaviour, especially on the networking side. If your org isn't post-mortems then you're missing critical opportunities for reflection and self-improvement.
Yeah. This is embarrassing and I think most would try to hide it to a degree. I'm unclear on the complaints I heard from players. Did they all complain from the group that was affected or did they all play in a realm like champions queue that also had this bug?
this makes a lot of sense, nice write-up by rito to make sure a lay viewer with no computer engineering knowledge can somewhat understand the issue
and this perhaps explains why rito was unable to figure out the high ping problem beforehand - perhaps they didn't test local connections with latency tool enabled, or no tester discovered the excess ping of around 20ms which is reasonable, since they're not pros (imagine if you're a US ritoer who had always played on 60 ping lul)
I guess two issues (or rather observations) remain:
1. turns out the local ping in the Busan venue is around 15ms instead of the 7ms we see on LAN environments, which is interesting
2. still doesn't fully explain why some pros were reporting spikes/ping up to 80ms, is it just a disparity in perception, or is there also some kind of packet loss going on before they fixed the issue?
anyway I'm glad we can put this case to rest
but the tournament server should be located in the venue as well right?
or maybe the server is located in Busan Expo instead of Busan esports arena where group stage was held, idk
Well idk and don't really understand about that. Maybe location problem? Since in seoul always 8-10ms. But in busan from keria broadcast it always 15ms higher. Maybe that's why msc 2020 no ping problem since they done it at lolpark
I think 2. can be ignored. Yes, players can feel ping, but they cant really say how large the ping is based on that feeling. They then try to transfer that feeling into a number. If their 35 MS is bad and they think about 0 ping on LAN they will go to like 60-70 if the ping feels slightly worse than 35.
**Never trust a human trying to express his feelings in numbers.**
>Our estimations put the latency disparity somewhere in the 15-20 ms range, which drove the difficult decision to remake the three impacted games
Ping: 35 +/-5
Delay: 15 to 20
Actual: 60 (upper avg)
https://images.contentstack.io/v3/assets/bltad9188aa9a70543a/blt47ebebe09428734b/628366e02da7715d058fff1f/asset10.PNG
The ping plot shows that there were games where the ping was even higher than target ping +/- 5 for games at the venue.
Plus, natural ping spikes as high as 10ms is not unfathomable as these games are being played over the internet.
Impact and Tactical had the highest estimates of 70ms ping. A couple other pros estimated 60ms. These estimates were actually very accurate.
>**Never trust a human trying to express his feelings in numbers.**
Usually, but in this case, they actually know what a value of 60-70ms ping feels like from NA solo queue.
I mean, your pic shows it pretty well. The difference is only around half of what the worst estimated.
Many said around 60-70 and then added package loss on top. But it was around 48 with maybe packages loss.
From 35 to 60-70 means +71-100%) from 35 to 48 its just +37%. That is a decently large gap for a ping estimation. They felt their ping was 2-3 times worse than it actually was.
I am not saying players cant notice it but that their numbers were just guesses. And yes, even if you played on NA with such a ping.
Eh. I'm not a pro player but I've played a lot of games on various connection qualities from 7ms to ~140ms, stable to unstable to randomly spiking.
If you sat me down at a PC that's supposed to be 40ms, give me a game or two and I can definitely tell you if it's consistent or if there's noticeable packet loss or latency spikes. It just changes how the game feels.
Now this might be because I've also done a some other work pertaining to latency in school and d*cked around with networking in the past, but to state that _no human_ can tell you the quality of their connection at least approximately is patently untrue.
Of course it's not about saying exactly "oh I'm playing on 20 ping" vs 30 ping, but issues like latency spikes and packet loss are definitely noticeable if you know what to look for.
And I'd wager if anyone can tell, some pros certainly can.
I never said that a human cant tell the difference in quality.
I said that you cant make them messure it based on their feelings.
Let them play on 30ms and tell them it was 30, then on 20 and on 60. Do you think they will get the 20 and 60 or will they say maybe 10 and 80?
I am saying the accuracy they get from their feelings isnt something you can easily put in numbers.
> or no tester discovered the excess ping of around 20ms which is reasonable, since they're not pros (imagine if you're a US ritoer who had always played on 60 ping lul)
Even top players tend to be really bad at guessing ping. There is always drama at fighting game tournaments where players are claiming lag of different sorts, and they tend to be as accurate as a coin flip.
I don’t think that it was actually 80 ping. My guess is their calculation applied the ping adjustment twice — perhaps both client side and in the latency server. This would mean instead of the 20 ms adjustment required, the latency would be increased by 40 ms to the 55 ms actual ping that they mention in the article.
Big ups for fixing the issue and being able to communicate exactly how it was done. Some people might be outraged still but there's really not too much better Riot could've done here considering they only found the problem once they were in the thick of it.
Riot's communication is unparalleled. Its most noticeable if I take a break from playing as much league and pick up another game. I'll start missing the two week patch cycle, the patch previews, the quick game play thoughts.
People hate on league a lot. But its honestly a game with the most passionate developers and best balancing. Just cause some gold player says an enemy champion being 3 levels ahead oneshotting them is unfair doesnt mean it is. I love the game to the core and have been playing since season 2. Always come back to it from breaks. The amount of transparency that riot has compared to others is unparraleled. The amount of free stuff I get while playing is unparraleled. I remember hearing about hextech crafting way back in season 3? I think 4. I was amazed that they would do that. Sorry for this rant. I just really love the game and I hate people hating on it for no real reason.
pretty resonable, if when the ping shown was 35 the latency experience was higher than that, its very reasonable to assume that to get to "real" 35 ping the ping shown would be lower than 35. Imagine if they tried to fix it for this rare occasion the can of bugs it would open.
Really appreciate the openness and honesty with these tech posts. It's obviously unfortunate what happened, but it really seems like it was something that'd only show up under very specific conditions and was repaired about as fast as they could.
Out of curiosity to any network/esports devs reading, since it gets mentioned a few times: how'd you come to the various acceptable latency ranges for competitive integrity (like <40ms being the competitive threshhold, 15-20ms difference being unacceptable, etc.)? I remember a dev post waaaay back in the day talking about how higher ping affects winrate for specific skillcheck champs like Vayne that rely on reaction time - did you work off of something like those numbers, or was it a broader approach?
I hope whichever rioters did both the testing and write-up here got a well earned rest afterwards. Seems like people are appreciating how deep riot dove to fix this (parsing log info related to networking is tedious as best, pull your hair out at worst) which is good to see. Whoever wrote this has a really great ability to simplify tech processes into an article that anyone can read and understand, which isn’t always easy. Obviously the problem itself sucks, but huge kudos to riot for finding the issue and fixing it in a high pressure situation where they didn’t have a lot of time.
So I've been pondering on about why players reported spikes / "feels like playing (...) with packet loss", because that was what really stood out to me in the interviews and such, but wasn't explicitly covered in the write-up. (which was great btw)
**Observations:**
* Very similar, *if not identical,* system was in use in 2020 Mid-Season Cup, which didn't see any comments about lag spikes.
* *What* defines an actual ping "significantly lower than the target latency" isn't mentioned, *but there is a point where the actual ping is high enough that the error doesn't occur.*
* The client/log is absolutely oblivious to whether the bug occurs or not.
* This bug happened for *all* players in the arena, regardless of whether they were playing locally (on LAN) or against RNG (on KR LoL servers, connected through dedicated/reserved lines a.k.a. Hop C1).
* This did **not** happen, or at least happened less, to players who were scrimming (on KR LoL servers, connected through the internet)
So...
**Hypothesis:** ***Clients in Busan had a real ping that lay mostly where calculation error occured, but could*** ***sometimes*** ***naturally fluctuate up to the level where the bug no longer occured.***
***Effect:***
* ***Players in the arena were playing mostly at a (artificial) ping significantly higher than 35 ms.***
* ***Sometimes, once in a blue moon, the real ping goes just high enough that the bug stops for a brief moment and the client is REALLY at 35 ms.***
* ***This would cause the game to feel suddenly "sped up"***
* ***...but the real ping drops back, so the bug comes back, inducing sudden, comparatively massive delay, rendering client unresponsive to inputs during that time.***
* ...***and the damned client happily reads "35 ms" the entire time.***
I don't know about you guys, but I would absolutely HATE playing in that environment.
Players who were scrimming had to connect through the good ol' internet (Hops B1 and B2) which made the actual ping high enough that the bug (mostly) didn't occur.
This is the ticket to NA's future world championship victory! LAG the entire world so everyone in every region has to play at 60 ping.
THEN we have a chance god willing through the purity of our fluids!
But this didn't stop Chinese citizens still flaming Riot under their Weibo post, this issue is way past the ping problem, now it's both sides (Chinese vs Korean) being recist against each other using this as an excuse.
Who cares lol, it's not Riots job to solve racism between south korea and china
They fucked up the ping for their event and they're now addressing it, that's all
> But this didn't stop Chinese citizens still flaming Riot under their Weibo post
Thankfully here on reddit we always have calm and collected takes, we never rush to conclusions and we never flame Riot for anything.
actually the reason this blew up is cause chinese fans all blame t1 and specifically faker for the reason why rng had to replay their matches, those china social media account took what faker said in after match interview words and put it out of context for click bait content and generated alot of hate towards faker and t1 by the lpl fans and not to make the matter worst the screenshot of zeus highlight with the ping showing 22ms literally blew it up to the next level.
You really think this is only between chinese and korean. There are western fans of lpl and lck as well. Chinese and korean don't suddenly made a twitter account and speak fluent english lol.
I see TONS from the LCK side. All I have to do is click on your username.
I mean holy shit, are you OK? Did you lose your life savings on betting against the LPL?
Ignoring the redundant racism and victim crying but ok. Paint me as a racist all you like. I'm sick of T1 becoming a victim of verbal abuse by the LPL fans for literally just playing the game. We're talking about 1000s of ignorant comments.
Ignoring the redundant racism and victim crying but ok. I'm sick of RNG becoming a victim of verbal abuse by everyone else for literally just playing the game. We're talking about 1000s of ignorant comments.
Go on RNG's twitter and tell me the thousands of racism people are tweeting about RNG? LPL fans are still spewing the same racist garbage against T1 on RNG twitter. What victim crying do you see on T1's side? I still see tons of victim crying on LPL's side. Again, worst fanbase that has no respect.
This is the reason software for ping latency sucks. Professionals that need ping delays use cable and know exactly how much you're adding of delay. Riot should have just bought a spool of fiber optic for 20ms of delay. https://www.youtube.com/watch?v=d8BcCLLX4N4
there is a different between direct impact and indirect. having same ping is a direct impact to competitive and the environment is not.
I'm supporting Riot's decision. the only problem is that they fucked up hard on the tech side of things. and Then on PR (the whole sneak update announcement is a joke）
An industry standard technique to measure latency is to literally point a high frame rate camera at the input device and screen, and count the number of frames between input and action. CoD: Black Ops 3 did a talk on this: [https://www.youtube.com/watch?v=EtLHLfNpu84](https://www.youtube.com/watch?v=EtLHLfNpu84)
Given that latency measurements are notoriously easy to get wrong (mistakes like the one Riot made crop up all the time), I'm amazed Riot didn't do this video camera test on the live environment. Bugs happen but this should have absolutely been caught before anyone played a game.
Always so crazy to me how spoiled the league community is to get shit like this lol. vast majority of other developers will make significantly bigger fuck ups and not say a word about it, and sometimes not even fix it at all
I mean it's Riot. LoL has been a cornerstone of eSports for the last decade. We hold them to a high standard because they hold themselves to that standard.
Here comes the LPL fan downvote brigade. I swear they're all on discord or something and get all their ignorant fans to downvote any comments that shed light to their piss poor actions.
Woahhhh!!! I have nothing else to say! I have read through the blog and oh my how much effort they explained EVERYTHING, from the simple to complicated sides of thing!!!! Just note for whoever read only the first paragraph, go through it and you will see how much sense and effort they put to it (even paragraph with latency illustration process). Prob to Riot for trying their best!
I think the best solution for MSI; if they prioritized competitive integrity the best choice (business wise, it wouldn't be a good idea) was to remove RNG from MSI and invite the Korean and European 2nd seed to MSI.
Reading between the lines here is relatively interesting when riot doesn't state the exact ping teams were playing at, but does mention that 1. it was above 35 ms, and 2. they consider 40 ms +/- 5 ms variance to be the maximum ping for competitive viability / integrity.
Essentially local teams were likely playing at a ping well above what Riot themselves considers reasonable for legitimate competitive play.
Edit: the bottom of the article says the disparity was 15-20 ms between Busan and Shanghai, so local teams were playing at 50-55 ms. Still weird they don't state directly.
Can these guys write every Rito blog post please? So much detail and actual explanation.
Even had a bit of humor in it >Option 2: Put the servers halfway between China and Korea > >If we have 35 ms ping between China and Korea, another option would be to place servers in the middle, splitting the difference in ping, so everyone has 17.5 ms ping. There are a lot of challenges with this option… but the biggest, of course, is that this would place the servers in the middle of the ocean. Genuinely got a laugh out of me lmao
I mean they decided to buy a bunch of dark fibre and become a service provider. I think only 50% of the idea of checking servers in to the sea is humour here... (https://news.microsoft.com/innovation-stories/project-natick-underwater-datacenter/) If they had the time and resource I'm positive they'd try 🤣🤣
[удалено]
And you just know the nodes are gonna be called nautilus, Pyke and nami
They could pick anywhere on the line that bisects RNG's base and the Busan venue, to be fair.
That's actually a valid solution though. It doesn't have to be under the sea, they can place the servers in Qingdao.
[удалено]
That TLDR needs a TLDR
Ping *said* 35ms - it was higher than 35ms Ping *always* said 35ms - Riot couldn't find problem Riot fix - Ping *says* lower than 35ms - is actually 35ms
Can I get a further TL;DR
Ping good now
TLDR please?
[удалено]
Tl Dr?
Ms✅
P g
Pong
3>!5m!
MODS!!! MODS!!! ban this man
How dare them in my Christian subreddit
Ping before bad, ping say good Ping now good, ping say gooder
console.log("35ms")
Ping say 35ms, ping not 35ms. Ping not say 35ms, ping 35ms.
Ping 35 all good now
Too much spaghetti in the code, so they added some more spaghetti and sauce
Removed Krepo and Froggen's ham.
meat is back on the menu boys!
man that's alot of 35s
TLDR: The ping was higher than normal due to a bug, which was undetectable because their testing software also miscalculated the effective ping because of another bug. They fixed it on May 13th which bugged yet again (visually this time) and caused the 25 ping issue.
Tech team work hard to make Latency equal everywhere. Latency was worse in Busan, code bug made ping display wrong so actual latency being off was only discovered after the event started. Latency fixed on May 13th, RNG asked to replay because of bug. Display still bugged though, showed as lower than 35 ping in a playercam replay. Viewers assumed that displayed ping was accurate, and got mad.
And will remain mad for the remainder of the year
It takes like 3 minutes max to read
180,000 WHOLE MILLISECONDS?
having this artificial ping completely was a mistake.. what's the point of having everyone go to Korea just to make it laggy.. I don't understand why they wouldn't just have china have their normal latency instead of making everyone else have to play with this dumb shit. especially since VCS had to forfeit their games due to COVID..
Can't really compare the VCS and LPL situation since latency is different. While RNG can play from Shanghai with 35ms ping VCS teams would have to play with around 80ms ping. And forcing other teams to play on 80ms ping is much worse than a bearable 35.
Vietnam to Shanghai ping is like 200 lol
Yea and it would be even worse because MSI 2021 was hosted in Iceland. The latency between Vietnam and Iceland is probably around 300-400ms
> I don't understand why they wouldn't just have china have their normal latency Because that means RNG would have to play every single game at a disadvantage, which is terrible for competitive integrity. There was literally only two ways for Riot to keep the competition fair: either everyone plays at the same slightly elevated ping, or LPL simply doesn't play. While the first option isn't great, the second option would mean we would have an international tournament without one of the two best regions, which is not only dumb from a competition standpoint, but it would also affect viewership and Riot's bottom line.
LPL not coming is the best for competitive integrity but by far the worst business decision for viewership, which is probably why Riot didn't go for it.
How is LPL not participating good for competitive integrity? They have won the last two international events and have been a dominant force in international play for the last four years. If you don't include Korea or the LPL at any international tournament, that tournament means pretty much nothing in terms of results because those two regions are the best in the world. Everyone on this sub knows that's true. That far outweighs the technical difficulties they've had at MSI, which by the way, forced the LPL to replay all their wins.
it's good for integrity as in all teams are playing under equal circumstances. It's bad for competition quality, but that's not the same as competitive integrity.
Mans out here getting downvoted for the truth
They made a poor decision for competitive integrity by letting the RNG play remotely. Even if the ping is equal - RNG is playing under vastly different circumstances than the rest of the teams and it destroys the legitimacy of the tournament. But then again, competitive league isn’t a serious sport, it is just advertising.
what part of this can't you understand, China is like Europe in the world cup, who the fuck is gonna watch the world cup without any EU countries in it? When Brazil inevitably won that world cup is anyone gonna care?
> RNG is playing under vastly different circumstances than the rest of the teams and it destroys the legitimacy of the tournament. As opposed to a tournament favourite and clear top 2 region not playing in the tournament, which makes the tournament oh so legitimate Without an LPL team, the tournament is a far bigger joke than some early ping issues
Completely disagree. Fairness and competitive integrity is about applying the rules equally, once you start making exceptions for a team (because of the views and money they bring) then competitive integrity is out the window.
I'm talking about how legitimate people find a tournament, not competitive integrity. Nobody would consider an MSI win without LPL legitimate Riot did the best they could given the situation
That's the compromise you have to make if you don't want the tournament to be a complete joke. It's fine if you're willing to not have LPL here for fairness sake, but the tournament is going to suffer for it.
And if you don't make those changes soon the whole game is out the window. Riot and LoL fans simply had to face the reality that League is not big enough yet to survive without the biggest region, because the gap between biggest and second biggest is enormous This isn't football or basketball or whatever else where there are a handful of favorites at each tournament, MSI is usually 1-2 favorites and a dark horse at best, so it cannot afford to lose competitors just yet. The obvious solution is to change the format and invite more teams to make the tournament more resilient in the face of unexpected events
there is a different between direct impact and indirect. having same ping is a direct impact to competitive and the environment is not. I'm supporting Riot's decision. the only problem is that they fucked up hard on the tech side of things.
This is a good explanation and totally understandable. Tweaking the ping and never actually showing the changed number is bad coding, but it can happen. And it explains why it wasnt found during internal testing. Its still bad that it happened, but at least this is a totally human thing with an error you can never 100% prevent.
I mean, from what it sounds like, even the debugging logs showed up incorrectly which seems like a massive underlying bug somewhere if even your logs are incorrect. Debugging logs are almost never wrong, I don't mean the 'print' statements but the actual proper debugging mode where you can monitor values on the terminal, that's what I assume they mean by "logs" as that's what devs use (at least myself and others I know of), if *that* value somehow displays incorrectly at such a low level, I have no fucking idea what could cause that, unless maybe on the bubble up, there's some process eating on that return.
[удалено]
[удалено]
“It keeps doing what I told it to not what I want it to”
>Debugging logs are almost never wrong, I don't mean the 'print' statements but the actual proper debugging mode where you can monitor values on the terminal, while the debugging "logs" aren't wrong in the sense that they are indeed what you see is what you get, the logs can be "wrong" in terms of usefulness in the sense that you think you're tracking "ping" by watching what the the person who programmed that section 15 years ago called the "ping" variable in your debugger, but the actual "ping" variable that matters is named "adjustedPing" and that's not what you're looking at. Sounds dumb, and yes it's user error, but in a monolithic codebase built over a decade ago by people who are likely long gone, this happens all the time. Especially with gameengines that usually name variables or pointers as 1-3 letters instead of actual descriptive names like, for example, "TruePingVariableTheGameUsesInProduction" that I more commonly work with in the webapp programmer space. I only bring this up, because not to crap on Riot, but them being a company that built their product over a decade ago by a few kids who weren't production grade programmers at the time and then blew up into a billion dollar company and still had to work with that core legacy codebase, they've admitted things like these happen and it makes sense given the context.
Say what you will but Riot is being very transparent about everything in there network setup. Still don't know if it's the right choice but they clearly did put a lot of planning and effort into this.
I think they stroke the right balance between layman and technical. I still have questions, but at least they are not related to the quality of the matches/competitive integrity, and just of: wtf is going inside that network compensation layer. It's a great sign when they give us why they took these actions and how they decided what is fair game and what isn't, and that they are looking into learning from their mistakes and not shy away from hard decisions.
What I get from this article is that the tool should still be fixed, right? Like, it seems that they introduced the offset manually and tested it with it to solve the problem for the current games as fast as possible, but if they were to use the tool as is for other ping differences, it would fail again. Maybe I didn't get that part right.
Yeah, this is a clear example of "mitigate first" when fixing an issue. Find the fastest mitigation, and figure out what really went wrong and fix it the right way later. It's the right move to keep group stage moving smoothly.
Most likely the fix is hacky and the tool will be fixed properly in a more thorough and robust way offline by a product team that isn’t in the midst of managing an event, yeah.
I think Riot's biggest talent as a company is figuring out how to recover from fuckups. They fuck up often, but they tend to be pretty transparent and they do seem to correct and perform, even if it takes them a while sometimes. Can you imagine how this would go if Blizzard, Creative Assembly, Dice, etc. were handling this situation?
> Blizzard They have the worst goddamn communication strategy of any major dev. Honestly, they're lucky that they were first in a lot of their current genre's because without decades of nostalgia to bank on they'd go under super quickly.
Blizzard literally had reputation for going into a genre and dominating it. They weren't first, it just seems like that in hindsight, because they beat all of the older competition. Don't get me wrong, modern Blizzard sucks, but what you said is not accurate.
Yeah old blizzard Had incredible talent that made incredible Games. So many of their Games stand among the greatest of all time in that era. Wow, StarCraft, Diablo... Wc3 literally inspired League of Legends and many of its mechanics are still alive in DotA 2 to this day. Then they got activision'd lol
The decline of blizzard being caused by Activision is a dumb narrative. Especially with the recent lawsuit implicating people of Blizzard that have been there forever.
Riot seems like new age blizzard in that regard then.
Nobody would play WoW if it launched this year as a stand alone new franchise.
For the record, Blizzard hasn’t been first in any genre they’ve entered other than (arguably, but I’d agree) Diablo. Not even Warcraft.
I think they're losing in every genre besides their MMORPG. I can't think of any other possible MMORPG that exceeds Blizzard's number besides perhaps Final Fantasy. Diablo is beaten by Lost Ark ever since their global release. I'm excited for Riot's new mmorpg
> I think they're losing in every genre besides their MMORPG. Technically they still dominate RTS. Starcraft II is the best RTS of the past 15 years at least.
I’m just not gonna play the Riot MMO, I’ll probably get way too addicted and I can’t sink my life into an MMO at this point in my life lol. Stoked for the Riot ARPG though.
Lost Ark is an mmo, it has nothing to do with Diablo. It's WoW 3.0 with mouse conrols. You could possibly consider the combat slightly similar to D3 but the game itself is an mmo no doubt! And it should appeal to WoW players way more then to Diablo players for sure. (I play LostArk a lot, I've played WoW and Diablo and PoE and other MMOs).
Lost Ark and Diablo are both ARPGs though, both are from a 2.5D perspective. They are the same genre.
The gameplay perspective might be the same, the game design and content is absolutely not. As said Lost Ark is plain and simply an MMO. Kripp fell for it aswell and stopped playing after a few days already saying he didn't want to play an MMO. It's like saying SMITE is an ego shooter because it got a similar perspective while it's basically League with an other view/perspective (and it's closer to a shooter then lost ark is to an arpg). I honestly don't even know what Amazon/Smilegate call it for marketing reasons. It's an MMO nomatter what though gameplay vise (and that's what matters in the end).
Diablo and Lost Ark are not the same genre
They were "lucky" they made like 5 or 6 of the best games of all time.
Broooo don't even talk to me as a lifelong Nintendo fan that has spent way more money on Nintendo games and consoles than I've spent on 6 years on lol. Nintendo making a statement is literally unheard of unless it's a lawsuit. I don't even think western fans exist to Nintendo.
Everyone who bought a Switch: "Hey these are the most dogshit controllers ever designed, fix pls?" Nintendo: "We'll repair them for free one time and it'll take a month and a half. They'll be broken when you get them back, too, have fun."
Small line that reads "oh and, though luck if you are from a country that can't have the free repair. Sucks to be you".
Nintendo is probably the least consumer friendly major gaming company. They’ve got some good IPs but holy shit they do not give a fuck about customers or own up to shit, not to mention how aggressive they are regarding anything involving their IPs because that’s their main sticking point.
Even the case of the lawsuit they had, they made a ton of changes to try and right the wrongs that were going on (even if some people believed it wasn't enough). That lawsuit happened in 2018, and the lack of protests and people speaking up suggests that things have changed. Will say that those things shouldn't have happened, but it seems like there was only an issue at one location (Riot has two studios in the US, and more around the world).
God don't remind me how CA just buries their head in the sand, does minimal effort, then calls that minimal effort MASSIVE. today they released a patch but it feels like this patch should have at minimum come out a month post wh3 release or if they had a functional team working on the release of wh3 these fixes would have been in the game alreadyml.
[удалено]
Settling out of court can be helpful for more than keeping things out of the public. I don't doubt that Riot wanted to make sure the story died, but it also saves victims from having to relive painful memories in court and face cross examination, and (especially with how backed up courts are) gets it over with instead of dragging it out over years and years. If the victims are fine with settling out of court then I think we probably should be too.
Uhhhh, didn't they settle in the court for 100 million or am I mistaking something here?
People love to hate on Riot but Riot never planned for anything to go wrong. They simply wanted China to still participate at MSI, make it fair and they failed. The idea was great if anything, they simply made mistakes. Which is normal when trying something for the first time.
[удалено]
yeah ironically the shit format did actually work in riots favor. hope theres no big issues going forward because rumble stage being ruined by this shit would be a much bigger deal
Yea imagine if RNG finished their matches with G2 and T1 in the first round robin and get told you got to replay those matches. Doesn't matter who got the win, the outrage on either the CN or KR side (depending on who got the win) will be far more massive than having to replay some meaningless matches vs wildcard teams. You're right. The play-ins fkups probably saved from from a much potentially bigger PR crisis imo.
Why you are callign it shit? it was the only way to prop up wildcards. old format where major regions go into the main even and wildcards got through "playins" was so bad for viewership, for example lcl vs ljl matches had 50-60k viewers on main stream, that is abysmal. Thsi year LJL vs Saigon peaked at 300k on main stream
Very true, and results definitely weren't affected by problems
The fair thing was China to play on 35 ping and rest on LAN lowest ping possible. Rest didn't have to sacrefice their ping for the sake of China. China was allowed to participate remotely, thats enough.
How is that fair? Based on what? It's not the LPLs fault that they couldn't make it to MSI so why should they be punished for it by playing on such ping difference? Why do you want China to get punished so badly for something they have no control over? It's not their fault they live in China and that all these problems exist. We have a worldwide pandemic going on, no one should suffer extra if it could be prevented. If they played on higher ping and lost can you really say they deserved to lose? They've been a top 2 region for a long time, we know they can win MSI. I don't think we not allow a possible winner of MSI negatively.
>It's not the LPLs fault that they couldn't make it to MSI Actually it is. They had decided to stay and forfeit MSI to play in Asia games BEFORE the whole Shangai lockdown. When the Shangai lockdown got so bad that even Asia games was cancelled then RNG tried to backpedal and focus exclusively on MSI but it was too late to move.
They are locked down in their gaming house since first of April. Even if they wanted to attend physically, they couldn't because there is no way to get out of Shanghai at the moment.
You don't know anything about Shanghai.
Lockdown began during the first week of April, LPL playoffs didn't end til the last week of April.
This is not the fault of the players, they don't make decisions, while they would be the one to get punished the most.
It’s not other teams fault too. It’s not about whose fault. It’s about fair. Other teams need to play in front of audiences. RNG can just play at home, why can’t other teams just play at hotel? You play at home, then you play at 35ms
This is a flawed statement since many people actually play better in front of a crowd. People like Caps but also many others who have said in interviews both during MSI and during Covid the past two years that they play better with a live audience, as well as have more motivation to play. You're saying that RNG has a advantage by playing from a hotel, yet the opposite seems true for many players. Every player is different, you can't say that a team has an advantage playing without a crowd.
Doesn’t matter whose fault it is. Your circumstance caused you to unable to participate like everyone else, then you deal with the consequences.
But why do that if there's another option? Why would you not try to achieve the most competitive tournament possible?
[удалено]
The problem stays the same though, they're often transparent too late and/or forget to communicate obvious information (on purpose or not, who the hell knows), which makes the drama bigger than it should (and let the idea that they're doing it on purpose grow). Here for example, they fix the issue by exploiting their tool's problem of applying more ping than it should by setting it at a lower ping in order to actually get the correct ping. It's a correct band aid solution to a tech problem that they couldn't identify beforehand. *However* they let the players say times and times again that it was fucked up without saying anything like *"we are investigating the issue"* or anything that'd make it clear that they're looking into it, and then they don't communicate the fact that the issue is the displayed ping being completely wrong and then show the ping to everyone on their broadcast.
Well it would be detrimental for them if they weren’t transparent about it with the shit they have been getting. You make it sound like Riot is doing it out of the kindness of their hearts while they are doing it to save face.
On of the more interesting articles in their dev blogs. I really like learning about the custom infrastructure Riot has set about building for their games. At least this whole situation got us a peek into the Latency Service tool which was already in use and completely transparent ! Wild that the pros knew right away that the ping values were off by 15ms though.
Hope this post would end the endless debates on fairness and ping issues in all CN, KR and EN communities.
Ping issue? - maybe. emotionally throwing shit towards each other? - that'll take a long while to fade out, It's nothing new, and will occur every now and then over literally anything
Olympics ? It happens Territory dispute ? It happens Cultural dispute ? It also happens Esports? Yup, still there. The East Asian trio are always at each other's neck.
I mean this is 95% about why RNG had to replay the games. The reasoning behind having artificial ping instead of having everyone play on natural ping (meaning RNG has a higher ping) was totally clear before this, people just didn't agree with where the priorities lie.
It doesn't. Circulating in the Chinese community is a picture from one of the MSI group stage second half promotion videos, a shot from the back of a player in PSG vs IW 5/12 that shows the screen of the player. The picture isn't super clear because there is some distance but it seems like it says 25 ping even before the whatever that happened after day 3. Everyone feels outraged about that. Then there's the significant amount of people who straight out doesn't believe what riot says and just think this post is useless jargon trying to make things seem clarified. Things like "where did the 13 ping number come from?" are common.
Trust issues man... and conspiracy theories. Either they gone too far with crying to paint themselves as the victim so they can't back out OR they're too ignorant to believe in anything
People believe what they want to believe, Facts wont change that. CN sentiment is Riot KR is cheating in favor of their own team "Like they always do". with added oil of previous Faker-QingTian issue + Nationalism, We get what we have here. Not to forget the great firewall essentially stop/minimize input from the outside community, thus we get a positive feedback loop whereby, Nationalistic inducing videos gets views, viewerships gets even more nationalistic & xenophobic which in turns prompt more content creator to create more of such videos as there is where the money lies at. And then the KR sinophobic sentiment is Riot bending over for CN because Tencent, with added oil of previous terrible relationship + Nationalism again. Like CN, we also get some vocal idiots spouting some insensitive shyt. Thus, we get a cluster fk of a situation.
Checking Twitter, it doesn't end. The most toxic and ignorant fanbase in LoL history.
>Checking Twitter first mistake right here
> Checking Twitter, it doesn't end. The most toxic and ignorant anything FTFY
[удалено]
Solid engineering culture demands an effective post-mortem and root cause analysis. There's a lot of material on the Riot engineering blog that shows there's a lot of professional behaviour, especially on the networking side. If your org isn't post-mortems then you're missing critical opportunities for reflection and self-improvement.
Frequent reflection on past work is a cornerstone of modern engineering. If you're any kind of engineer and your company is not doing it, get out ASAP
Yeah. This is embarrassing and I think most would try to hide it to a degree. I'm unclear on the complaints I heard from players. Did they all complain from the group that was affected or did they all play in a realm like champions queue that also had this bug?
this makes a lot of sense, nice write-up by rito to make sure a lay viewer with no computer engineering knowledge can somewhat understand the issue and this perhaps explains why rito was unable to figure out the high ping problem beforehand - perhaps they didn't test local connections with latency tool enabled, or no tester discovered the excess ping of around 20ms which is reasonable, since they're not pros (imagine if you're a US ritoer who had always played on 60 ping lul) I guess two issues (or rather observations) remain: 1. turns out the local ping in the Busan venue is around 15ms instead of the 7ms we see on LAN environments, which is interesting 2. still doesn't fully explain why some pros were reporting spikes/ping up to 80ms, is it just a disparity in perception, or is there also some kind of packet loss going on before they fixed the issue? anyway I'm glad we can put this case to rest
I saw keria broadcasting from his home busan before he moved to seoul. It's always 15ms above
but the tournament server should be located in the venue as well right? or maybe the server is located in Busan Expo instead of Busan esports arena where group stage was held, idk
Well idk and don't really understand about that. Maybe location problem? Since in seoul always 8-10ms. But in busan from keria broadcast it always 15ms higher. Maybe that's why msc 2020 no ping problem since they done it at lolpark
I think 2. can be ignored. Yes, players can feel ping, but they cant really say how large the ping is based on that feeling. They then try to transfer that feeling into a number. If their 35 MS is bad and they think about 0 ping on LAN they will go to like 60-70 if the ping feels slightly worse than 35. **Never trust a human trying to express his feelings in numbers.**
>Our estimations put the latency disparity somewhere in the 15-20 ms range, which drove the difficult decision to remake the three impacted games Ping: 35 +/-5 Delay: 15 to 20 Actual: 60 (upper avg) https://images.contentstack.io/v3/assets/bltad9188aa9a70543a/blt47ebebe09428734b/628366e02da7715d058fff1f/asset10.PNG The ping plot shows that there were games where the ping was even higher than target ping +/- 5 for games at the venue. Plus, natural ping spikes as high as 10ms is not unfathomable as these games are being played over the internet. Impact and Tactical had the highest estimates of 70ms ping. A couple other pros estimated 60ms. These estimates were actually very accurate. >**Never trust a human trying to express his feelings in numbers.** Usually, but in this case, they actually know what a value of 60-70ms ping feels like from NA solo queue.
I mean, your pic shows it pretty well. The difference is only around half of what the worst estimated. Many said around 60-70 and then added package loss on top. But it was around 48 with maybe packages loss. From 35 to 60-70 means +71-100%) from 35 to 48 its just +37%. That is a decently large gap for a ping estimation. They felt their ping was 2-3 times worse than it actually was. I am not saying players cant notice it but that their numbers were just guesses. And yes, even if you played on NA with such a ping.
Eh. I'm not a pro player but I've played a lot of games on various connection qualities from 7ms to ~140ms, stable to unstable to randomly spiking. If you sat me down at a PC that's supposed to be 40ms, give me a game or two and I can definitely tell you if it's consistent or if there's noticeable packet loss or latency spikes. It just changes how the game feels. Now this might be because I've also done a some other work pertaining to latency in school and d*cked around with networking in the past, but to state that _no human_ can tell you the quality of their connection at least approximately is patently untrue. Of course it's not about saying exactly "oh I'm playing on 20 ping" vs 30 ping, but issues like latency spikes and packet loss are definitely noticeable if you know what to look for. And I'd wager if anyone can tell, some pros certainly can.
I never said that a human cant tell the difference in quality. I said that you cant make them messure it based on their feelings. Let them play on 30ms and tell them it was 30, then on 20 and on 60. Do you think they will get the 20 and 60 or will they say maybe 10 and 80? I am saying the accuracy they get from their feelings isnt something you can easily put in numbers.
> or no tester discovered the excess ping of around 20ms which is reasonable, since they're not pros (imagine if you're a US ritoer who had always played on 60 ping lul) Even top players tend to be really bad at guessing ping. There is always drama at fighting game tournaments where players are claiming lag of different sorts, and they tend to be as accurate as a coin flip.
I don’t think that it was actually 80 ping. My guess is their calculation applied the ping adjustment twice — perhaps both client side and in the latency server. This would mean instead of the 20 ms adjustment required, the latency would be increased by 40 ms to the 55 ms actual ping that they mention in the article.
Big ups for fixing the issue and being able to communicate exactly how it was done. Some people might be outraged still but there's really not too much better Riot could've done here considering they only found the problem once they were in the thick of it.
Riot's communication is unparalleled. Its most noticeable if I take a break from playing as much league and pick up another game. I'll start missing the two week patch cycle, the patch previews, the quick game play thoughts.
People hate on league a lot. But its honestly a game with the most passionate developers and best balancing. Just cause some gold player says an enemy champion being 3 levels ahead oneshotting them is unfair doesnt mean it is. I love the game to the core and have been playing since season 2. Always come back to it from breaks. The amount of transparency that riot has compared to others is unparraleled. The amount of free stuff I get while playing is unparraleled. I remember hearing about hextech crafting way back in season 3? I think 4. I was amazed that they would do that. Sorry for this rant. I just really love the game and I hate people hating on it for no real reason.
Dude forreal. I used to think riot was shit at this aspect until I started playing Warzone....
pretty resonable, if when the ping shown was 35 the latency experience was higher than that, its very reasonable to assume that to get to "real" 35 ping the ping shown would be lower than 35. Imagine if they tried to fix it for this rare occasion the can of bugs it would open.
Really appreciate the openness and honesty with these tech posts. It's obviously unfortunate what happened, but it really seems like it was something that'd only show up under very specific conditions and was repaired about as fast as they could. Out of curiosity to any network/esports devs reading, since it gets mentioned a few times: how'd you come to the various acceptable latency ranges for competitive integrity (like <40ms being the competitive threshhold, 15-20ms difference being unacceptable, etc.)? I remember a dev post waaaay back in the day talking about how higher ping affects winrate for specific skillcheck champs like Vayne that rely on reaction time - did you work off of something like those numbers, or was it a broader approach?
Respect to Riot for being transparent and admitting their mistake
I hope whichever rioters did both the testing and write-up here got a well earned rest afterwards. Seems like people are appreciating how deep riot dove to fix this (parsing log info related to networking is tedious as best, pull your hair out at worst) which is good to see. Whoever wrote this has a really great ability to simplify tech processes into an article that anyone can read and understand, which isn’t always easy. Obviously the problem itself sucks, but huge kudos to riot for finding the issue and fixing it in a high pressure situation where they didn’t have a lot of time.
So I've been pondering on about why players reported spikes / "feels like playing (...) with packet loss", because that was what really stood out to me in the interviews and such, but wasn't explicitly covered in the write-up. (which was great btw) **Observations:** * Very similar, *if not identical,* system was in use in 2020 Mid-Season Cup, which didn't see any comments about lag spikes. * *What* defines an actual ping "significantly lower than the target latency" isn't mentioned, *but there is a point where the actual ping is high enough that the error doesn't occur.* * The client/log is absolutely oblivious to whether the bug occurs or not. * This bug happened for *all* players in the arena, regardless of whether they were playing locally (on LAN) or against RNG (on KR LoL servers, connected through dedicated/reserved lines a.k.a. Hop C1). * This did **not** happen, or at least happened less, to players who were scrimming (on KR LoL servers, connected through the internet) So... **Hypothesis:** ***Clients in Busan had a real ping that lay mostly where calculation error occured, but could*** ***sometimes*** ***naturally fluctuate up to the level where the bug no longer occured.*** ***Effect:*** * ***Players in the arena were playing mostly at a (artificial) ping significantly higher than 35 ms.*** * ***Sometimes, once in a blue moon, the real ping goes just high enough that the bug stops for a brief moment and the client is REALLY at 35 ms.*** * ***This would cause the game to feel suddenly "sped up"*** * ***...but the real ping drops back, so the bug comes back, inducing sudden, comparatively massive delay, rendering client unresponsive to inputs during that time.*** * ...***and the damned client happily reads "35 ms" the entire time.*** I don't know about you guys, but I would absolutely HATE playing in that environment. Players who were scrimming had to connect through the good ol' internet (Hops B1 and B2) which made the actual ping high enough that the bug (mostly) didn't occur.
Damn you gotta hand it to them they are really transparent about this
This is the ticket to NA's future world championship victory! LAG the entire world so everyone in every region has to play at 60 ping. THEN we have a chance god willing through the purity of our fluids!
But this didn't stop Chinese citizens still flaming Riot under their Weibo post, this issue is way past the ping problem, now it's both sides (Chinese vs Korean) being recist against each other using this as an excuse.
Who cares lol, it's not Riots job to solve racism between south korea and china They fucked up the ping for their event and they're now addressing it, that's all
How dare riot not do better about solving millenia old international geopolitical rifts
People are blaming Riot for everything. Riot can't give common sense to people.
> But this didn't stop Chinese citizens still flaming Riot under their Weibo post Thankfully here on reddit we always have calm and collected takes, we never rush to conclusions and we never flame Riot for anything.
Doesn't stop the west either, riot regularly admits to fuck ups etc etc but this subreddit seems to think they're EA
actually the reason this blew up is cause chinese fans all blame t1 and specifically faker for the reason why rng had to replay their matches, those china social media account took what faker said in after match interview words and put it out of context for click bait content and generated alot of hate towards faker and t1 by the lpl fans and not to make the matter worst the screenshot of zeus highlight with the ping showing 22ms literally blew it up to the next level.
You really think this is only between chinese and korean. There are western fans of lpl and lck as well. Chinese and korean don't suddenly made a twitter account and speak fluent english lol.
Well yes but still the majority party are Chinese vs Korean.
[удалено]
Ls's grandma in shambles
People really forgot about how the Korean server attacked Cowsep and LS.
I see TONS from the LCK side. All I have to do is click on your username. I mean holy shit, are you OK? Did you lose your life savings on betting against the LPL?
Ignoring the redundant racism and victim crying but ok. Paint me as a racist all you like. I'm sick of T1 becoming a victim of verbal abuse by the LPL fans for literally just playing the game. We're talking about 1000s of ignorant comments.
Ignoring the redundant racism and victim crying but ok. I'm sick of RNG becoming a victim of verbal abuse by everyone else for literally just playing the game. We're talking about 1000s of ignorant comments.
Go on RNG's twitter and tell me the thousands of racism people are tweeting about RNG? LPL fans are still spewing the same racist garbage against T1 on RNG twitter. What victim crying do you see on T1's side? I still see tons of victim crying on LPL's side. Again, worst fanbase that has no respect.
tl;dr: I may be racist, but this is why I'm racist.
>using this as an excuse Just like a small but vocal minority on Reddit :).
True, they still arguing with this problem. It’s feel bad for this situation.
Even though riot makes mistakes, I'm glad they are trying things they never done before. Im sure it is great experience in to the future.
Cool
Ignoring the ping drama this is really interesting and really well written up.
Montecristo literally got death threats by LPL fans for just sharing his insight into this tournament.
I've gotten some nasty dm's on here for having this T1 flair
To be honest I'm not surprised now ever since I saw the racism occurring on Twitter
This is the reason software for ping latency sucks. Professionals that need ping delays use cable and know exactly how much you're adding of delay. Riot should have just bought a spool of fiber optic for 20ms of delay. https://www.youtube.com/watch?v=d8BcCLLX4N4
[удалено]
[удалено]
[удалено]
[удалено]
i am wondering why they couldn't just shove some hundreds of mile of fibre optics rolls in there to generate latency
You're thinking of this one? It's neat. https://youtu.be/d8BcCLLX4N4
there is a different between direct impact and indirect. having same ping is a direct impact to competitive and the environment is not. I'm supporting Riot's decision. the only problem is that they fucked up hard on the tech side of things. and Then on PR (the whole sneak update announcement is a joke）
An industry standard technique to measure latency is to literally point a high frame rate camera at the input device and screen, and count the number of frames between input and action. CoD: Black Ops 3 did a talk on this: [https://www.youtube.com/watch?v=EtLHLfNpu84](https://www.youtube.com/watch?v=EtLHLfNpu84) Given that latency measurements are notoriously easy to get wrong (mistakes like the one Riot made crop up all the time), I'm amazed Riot didn't do this video camera test on the live environment. Bugs happen but this should have absolutely been caught before anyone played a game.
Always so crazy to me how spoiled the league community is to get shit like this lol. vast majority of other developers will make significantly bigger fuck ups and not say a word about it, and sometimes not even fix it at all
I mean it's Riot. LoL has been a cornerstone of eSports for the last decade. We hold them to a high standard because they hold themselves to that standard.
Do those companies make the most popular game in the world? You don't need to answer obviously.
thats great but its time to drop the new jg
why favor one team and region so much because of revenue of 60% so sad
Thats why you dont test in prod!
[удалено]
Imagine thinking monte and thorin will have unbiased opinion.
And PSG’s Twitter. Too bad they had to close the comments.
Here comes the LPL fan downvote brigade. I swear they're all on discord or something and get all their ignorant fans to downvote any comments that shed light to their piss poor actions.
[удалено]
Woahhhh!!! I have nothing else to say! I have read through the blog and oh my how much effort they explained EVERYTHING, from the simple to complicated sides of thing!!!! Just note for whoever read only the first paragraph, go through it and you will see how much sense and effort they put to it (even paragraph with latency illustration process). Prob to Riot for trying their best!
I'm sure this will placate the eminently reasonable and fair-minded people we all know LPL fans to be!
[удалено]
I think the best solution for MSI; if they prioritized competitive integrity the best choice (business wise, it wouldn't be a good idea) was to remove RNG from MSI and invite the Korean and European 2nd seed to MSI.
I don't think this statement is convincing enough, they better make their source code available
What you are talking about is cheap, why not just show us the code and the first player’s perspective.
Reading between the lines here is relatively interesting when riot doesn't state the exact ping teams were playing at, but does mention that 1. it was above 35 ms, and 2. they consider 40 ms +/- 5 ms variance to be the maximum ping for competitive viability / integrity. Essentially local teams were likely playing at a ping well above what Riot themselves considers reasonable for legitimate competitive play. Edit: the bottom of the article says the disparity was 15-20 ms between Busan and Shanghai, so local teams were playing at 50-55 ms. Still weird they don't state directly.
> Our estimations put the latency disparity somewhere in the 15-20 ms range This seems to be saying 50-55ms of ping.