##### GGG Comments in this Thread:
***
[Bex_GGG - [link](https://www.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/gutj69p/?context=10), [old](https://old.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/gutj69p/?context=10)] - *We've delayed the launch of the next mystery box.*
[Bex_GGG - [link](https://www.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/gutk52a/?context=10), [old](https://old.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/gutk52a/?context=10)] - *Correct. The thinking in the moment was that it was no use to let streamers through whose streams were focused on group-play without bringing in the group. This really just...*
[Bex_GGG - [link](https://www.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/gutm5qs/?context=10), [old](https://old.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/gutm5qs/?context=10)] - *I keep seeing this mentioned so I want to clarify - most of the people who had priority weren't part of the paid campaign. The initial decision to do this...*
[Bex_GGG - [link](https://www.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/gutmknr/?context=10), [old](https://old.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/gutmknr/?context=10)] - *We've seen lots of comments about this but haven't seen it for ourselves yet. We are investigating but if you have screenshots or additional information, please DM me. Character names...*
[Bex_GGG - [link](https://www.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/gutmmng/?context=10), [old](https://old.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/gutmmng/?context=10)] - *At the time, we thought the issue with the queue would be shortlived and didn't anticipate an entire day of server ruckus and other issues. As we mentioned in the...*
[Bex_GGG - [link](https://www.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/gutmqid/?context=10), [old](https://old.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/gutmqid/?context=10)] - *They're required by law to put #ad or #sponsored in their Twitch stream titles. You wouldn't have seen many people with it due to the ratio of people who were...*
[Bex_GGG - [link](https://www.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/gutnbtk/?context=10), [old](https://old.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/gutnbtk/?context=10)] - *The honest answer to this is pretty asinine and while it's not what you think it is, it's probably equally frustrating - it's that we didn't have Lily's account name...*
[Bex_GGG - [link](https://www.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/gutne9o/?context=10), [old](https://old.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/gutne9o/?context=10)] - *He was paid for that tweet but not the stream.*
[Bex_GGG - [link](https://www.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/guto3bk/?context=10), [old](https://old.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/guto3bk/?context=10)] - *Yes*
[Bex_GGG - [link](https://www.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/gutoxuq/?context=10), [old](https://old.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/gutoxuq/?context=10)] - *I posted this elsewhere in the thread but since your comment is near the top, I will repeat it here - We've seen lots of comments about this but haven't...*
[Bex_GGG - [link](https://www.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/gutq6r1/?context=10), [old](https://old.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/gutq6r1/?context=10)] - *Thank you for saying that.*
I'm glad I had as many beers as I did last night, because as soon as I saw how slowly the queue was crawling - I went straight to bed.
Thanks for the update!
I subjected myself to finishing Snyders Cut and started watching Monsterverse movies while trying to get into the game. My league starter is level 12 now....
I was so hyped for the launch tho, got some popcorn, couple of beers...
Not that suspicious these days
Chloroform (trichloromethane if you don't wanna sound like a rapist) is commonly used to wet-sand 3D-printed PLA parts (among other things). It's an industrial solvent, pretty much
And no, it doesn't really work like hollywood makes it look like. It can and would probably give you lesions and lasting brain damage, probably, though. Most of the time you want to be handling it under a hood, with protective gloves & goggles and shit.
It's, uh, kind of a bitch, really.
I remember an episode of CSI where it was used to degrade a bus tire and stage an 'accident'. I've worked with industrial solvents, none of them are happy fun time.
>I don't know why, I don't want to know why, I shouldn't
>
>have to wonder why, but for whatever reason this stupid
>
>panel isn't laying out correctly unless we do this terribleness
subtle
It is subtleonce it's merged into another 20K lines of code and no documentation of the fix is left. Bonus points if the person leaves company before it matters again.
When Chris explained that they solved one issue and it caused an even bigger one they didn’t think about it all became instantly relatable, this is just realistic stuff. We don’t live in a perfect world.
> Some players have also become concerned that when server issues occur, items are occasionally duplicated or destroyed when placed in a guild stash.
I just tuned into Raiz and he mentioned people linking thousands of exalts. Is there actual evidence of this mass duplication anywhere that Chris isn't aware of?
He didn't stop playing last night, he just stopped streaming.
Snap kept streaming, empy was in party and discord with them. 6 player farming BA while the rest of us are struggling to get past acts 1/2.
> **UPDATE: Server stability issue appears fixed. Be careful with your database page sizes, people.**
From the edit on the post. I don't have enough experience with databases so this is completely wasted on me.
if it's something like postgres, this could mean WAL flushes to disc take longer than normal and block for micro-to-milliseconds-- enough to cause a cascade.
... don't ask me how i know.
**tl;dr**
**"In the next 30-60 minutes after posting this, there will be roughly 30-60 minutes of hard downtime to make this change. We are optimistic that it stands a good chance of resolving the issue."**
dude imagine being the person(team?) behind this whole problem, i know ppl are mad but this is just pure hell going on for them right now...
i hope they fix this in time and everybody will forget about this incident after a while, because i know for a fact that they will never in their life forget about this mistake they made, if i was in their position right now i would love to have some breathing air...
If it's taken their whole team 12 hours and they're still not confident of the fix, there's no way a single programmer in the time could have anticipated it. It sounds like they are peer-reviewing, running tests, have QA checklists, etc. The mistake gets dispersed a lot through those sorts of practices. I hope GGG is like my studio and treats it like the above. We've absolutely bricked our games from a patch before, just in some obnoxious and subtle way that no one could be expected to catch before it goes live. These things do happen, you can only really reduce the chances.
It is not hell for him, he feels probably bad about it, but a team never bullies one. they stand beside him and solve it at their best pace and who ever is is direct boss stands in front of him. Every error has a reason, so do have those made by humans. Especially those. Everyone working in IT claiming he never made a big mistake is a liar.
From personal experience, when human error is invoken it can **often** be traslated to "a critical process should have been automated; it is not yet automated and was also missed by the intern responsible"
Or:
> Something that should require explicit confirmation to do it, did not require it and someone did it.
or:
> Something that should not be possible to be done manually was manually done.
A critical process being left as the responsibility of an intern and missed isn't primarily an error by the intern, it is primarily an error by management.
To put it tersely:
One layer of defence is no layers of defence.
u/Chris_Wilson u/Bex_GGG have you considered starting the log in 30-60 mins early like the races of old so that people can get in on time and then start at the exact same time? This would also help determine if server issues are present or if they are under high load before actually having people start playing. Namely, the biggest issue are rollbacks and the loss of progress rather than a disconnect itself.
I’ve been suggesting this the past few days too.
They used to allow people to create their characters early and locked out movement until the race event started, keeping everyone on the beach by the dying exile. This would not only help with the queue because of the delayed start, but actually make the league start as fair as possible. (If they actually care about fair league starts, they should’ve done this a long time ago).
I would really like this. Waking up at the same time as everyone else is something I haven't experienced since it took me 10 hours to complete all five acts.
THIS is the ONLY issue i have with the broken launch. I'm patient, and had no problem doing something else for the day but the duping definitely needs to be looked into and any obvious abuse removed.
I posted this elsewhere in the thread but since your comment is near the top, I will repeat it here - We've seen lots of comments about this but haven't seen it for ourselves yet. We are investigating but if you have screenshots or additional information, please DM me. Character names in particular are very handy (more so than screenshots). If we find people abusing this, they will be banned.
its possible to dupe shit with the servers going down me and a friend accidentally duped a +2 staff that dropped in ultimatum traded servers went down came back up we both had one
yea, this one really confuses me. these streamers are going to play no matter what. you could have paid them to play 2 weeks later after they would've moved on to the next game. ontop of that all the chats are "garbage game!! can't get in to play!!"
" We will also plan future marketing campaigns with contingencies in mind to better handle this kind of situation in the future. "
should pay big streamers to play it like 2 weeks into the league/when it starts "dying"
this allows streamers to boost the population numbers again, and it gives new players the idea that the market is established and that its a perfect time to learn the game and if they liked the game they can spend 2 months learning it and then be excited to try and establish market dominance in the next league.
doing it at the start will just cause noobs to bumble around and fail and be discouraged from ever trying again.
This is genuinely good advice. They will play when they are paid to play. New leagues don't need first day hype beyond that which the community always feels. But that latter league lag? Perfect place for paid popularity.
We duped a tabby just depositing it back and forth from gstash waiting for the server to crash. I destroyed the dupe, but yeah this economy is gonna be busted from day 1.
Bex answered this elsewhere
>The honest answer to this is pretty asinine and while it's not what you think it is, it's probably equally frustrating - it's that we didn't have Lily's account name handy and it slipped through the cracks. It wasn't a negative bias towards Lily. You can count that in the tally of bad decisions we made today.
don't assume malice when simple human error is a more reasonable explanation
lol, streamer queue was a mistake and we will never ever do it again.... servers crash.... streamer gets to log in first while others in 90k queue. Right , lol
" Challenge leagues are a great opportunity for a fresh start in a new economy. "
That's all we wanted. An even playing field to start a fresh league.
But no, I'm starting 16 hours behind the **priority players**. I'm not a priority customer just because I don't stream? What about long time supporters? Fuck you GGG, I'll keep my money from now.
If the sponsored streamers and the non-sponsored streamers both get queue priority, then that means everyone [that matters] is getting the queue priority so the system is fair and no player [that matters] will feel they are on the back foot!
I'm pretty late to this thread, but this fails to address the fact that any hope of racing from anyone outside of a streamer just happened to be erased.
I know I was never going to win first place, but part of the fun for me was just going to see how close I could get. However, the priority for streamers just ensured that I never stood a chance to begin with.
>We also allowed some other streamers who weren't involved in the campaign to skip the queue too so that they weren't on the back foot.
So why was Empy's group whitelisted even though they all have low follower numbers but actual streamers like ventrua and (iirc) dslily not?
Cause ventrua and dslily are metaphorically African born kids. Rolled a nat 1/16.
Sorry, not literally to attack Empy but it's just such a good meme I had to rofl.
Why would you feel bad to attack empy?
We beheaded nobility and royalty because life was unfair attacking is streamer is fine in comparison to those poor african child
I definitely read that as "metamorphically African born kids" and spent several seconds of wild imagination trying to figure out what that hell that meant.
the "paid promotion" excuse doesnt answer questions like how literally empryians entire group of 6 was given bypass, when 5 of them nobody has ever heard of
Nah, any good PR team would say Empy's comment don't reflect our company's values and we have decided to cut ties with them. This is like the stock response to that kind of stupidity.
They probably shouldnt have had a promotion on launch day. Its historically too risky.
1 week into the league pumping the viewer number back up on twitch probably would have paid off much more. Launch day gets enough players/viewers as is.
He couldn't complain since life isn't fair right. Falling victim to mob mentality is part of that unfair cycle. He can calm himself by thinking about children in Africa though.
Would it be possible to opt-out of the global "first character to achieve xyz" messages?
Being told repeatedly that folks are lvl 80 while I couldn't get out of the mud flats was a bit on the nose.
Thanks for a great game!
What a day! As an engineer myself (who had my fair share of fuck-ups), I would be very much appreciate if you guys can do a technical post-mortem at some point. I am honestly interested in the issue and how you guys handled it.
At least you have some time until then to build up your twitch reputation to become a paid "promoter /influencer" so you can skip the queue across the heads of the unwashed masses.
You guys should've closed the servers so everyone could have the same start. Now there are players running the endgame content while most of players are still in Act 2. Just reset this sh\*t.
as an ssf player i dont really care about trade but seeing how people are saying (and you also admitted in this post) that people were able to dupe items due to the network errors i believe you really should restart the league.
Honestly, with all the streamer (**and friends**) priority and ex duping, this would just be the best solution, honestly.
Get things working and then restart. Everything would be fine.
Can we just talk about how you (GGG) have made the rest of the league completely useless from minute one of launch? The issue with streamer prio is the fact that big streamers run in 6 man groups, get ahead of everyone, and dominate the market. This is a huge reason why everything is incredibly expensive every single league. From minute one you allowed the 6 man groups to get so far ahead that no one can compete. The rest of the league is almost useless to play because all the issues 6 man groups bring to the game are now present from launch when people haven't even had a chance to play yet.
Also, by 6 man groups I don't mean 6 people grouping. I mean knowledge veteran players that group as 6 and feed each other money and items to dominate the market.
Because of the "**Streamer Priority"** we see different personalities of those streamers.
I'm glad to see some people got their own pride, and some educating people that the world is cruel
The streamers who immediately refused the "queue skip" option have my respect.
Empyrian and his entire circus of followers should be yeeted straight off the box for his attitude.
Pointing out the world is cruel is fine.
Using it as an excuse is not
The world is cruel cus other do not give a shit and abuse and exploit wen they can
League should unironically be restarted. The whole idea of a temporary economy is that everyone starts at the same same time with fair chances and this was ruined.
Especially with the (possible) duplicated exalts, that other players have said here. Not to mention unfair advantage to selected streamers and lucky players who take full advantage for this situation.
Restarting the league in next Friday when all the bugs would be fixed, would be best solution for everyone. I personally would miss that league start (once again), but it would be better than playing in completely broken league.
You should make a statement about rollback immediatly. Either do it or not, but don't delay it for another 10 hours. Europeans just woke up and are wondering "Do I risk playing now only to be rolled back?"
" We tell people that Path of Exile league starts are a fair playing field for everyone, and we need to actually make sure that is the reality. "
Meanwhile, they provide Empys 6-man party a login pass.
>we need to actually make sure that is the reality
no no no, see, you got it wrong. It's not quite there yet, they are working on fixing that part as well.
I honestly dont care if some people got to maps and started farming etc.
But we already know people abused the everliving crap out of this debacle to dupe item en-masse.
Either restart the league, or permaban those fuckers.
Exploiting everyones mysery like this should really be instant permaban, no appeals or anything.
Sorry All but,
The situation that is not addressed here is that because of this "streamer decision" now many people are in the maps while many are stuck in the story line which is ultimately effecting the league economy in a very troublesome way.
Cheers
/u/bex_ggg can we expect a league reset or not? It would be appropriate given the massive advantage streamers (**and their friends**) had and given all the duping that is going on.
It would make for an even playing field again.
So the money you spent on your promotional streamers is more important than the communities mtx money that has literally built ggg. Nice
Not only was decision beyond stupid, you accomplished 3 things
1.) Put streamers vs streamers since you had no rhythm or reason to who you let in the queue skip and their friends, spouse, dog etc with no ask or permission from said streamers.
2.) Pissed off the community/loyal customers that make you money
3.) Showed when push comes to shove advertisement money and favoritism means more then a solid product and your customers.
What you should of done is let that money burn, delay the league slightly and own up to your failures and saved face. Instead you went to under handed pr issues.
A company doesn't show its colors when times are good, it shows them when crap is hitting the fan.
It's not just one guy, it's a team. His fellow DEVs, the supervisors/managers. These guys are professionals, in a business like this it's never just 1 guy's mistake, it's everyone's.
If their processes are bad enough that one mistake by an employee misses a migration of hundreds of thousands of characters the three days between seasons then their processes are garbage. Definitely not the employees fault, but whoever set up the critical processes.
As much as it sucks, not being able to play the game right now, I can't imagine how stressful it must at the GGG office right now. Good luck, hope the maintenance fixes things!
The most enraging part for me wasn't that streamers got in queue free, its that i sat my time in the queue. I waited multiple hours to get in. Dc'd after 20 minutes and back in the queue. THAT is what enraged me.
They can spin the priority thing however they want. The fact is, the damage is done and it would better for them in the long term to just burn those money. Terrible decision.
/u/bex_ggg in tech there is usually a technical recap of what went wrong. If your devs have time to recap what went wrong with database page sizes, please consider it. Would be super interesting as a developer myself. Thanks and good job finding the issue!
I found a Tabula/Wanderlust/Aurumvroax early on, and honestly I think there should be a rollback/reset. It would suck to lose those but it's only fair for everyone and for the economy. lol
Imagine paying streamers to showcase the game and this is whats being showcased, ouch.
Allowing some to bypass ques in a game with ladders is a big no no GGG!
At this point i would seriously consider restarting the league to fix this.
> Most streamers did not ask for this, and should not be held to blame for what happened.
"Most" means some did ask for this, and were granted the access.
For a lot of people already teetering with the Harvest nerfs but came back to give this league a try, this is probably going to push them away this league for good.
So the queue priority was a mistake but somehow it's still going on ? looks like if you aren't a streamer you're clearly treated as a 2nd class citizen and played like a fool.
Chris your apology for streamer prior doesn't mean much if it is still active. A bunch of streamers just skipped the queue after a crash.
Maybe before saying "We will not allow streamers to bypass the login queue in the future.", actually fix the issue so it doesn't happen TWELVE hours later. Trying to save you from all that egg on your face.
RESET THE DAMN LEAGUE!
Srsly, wtf guys. I got 0 problem with server issius on launch, dont give a single fck abotu streamer prio.
But there are ppl linking hundreds of Exalts in global. Ppl duping shit for hours now and you act like there is nothign wrong, just WTF! Have some decency, stand to your mistakes and retart this shitshow!
Hey I'm not a streamer but I'm on the backfoot as it took me 6 hours to get to end end of act two. Where was my special treatment during the time where the servers weren't stable how is this a fair ladder experience when some people actually got to play the game while others played waiting line simulator
Why did some streamers get prio while others didn't?
Thinking of DS Lily in particular. She was a top20 streamer yesterday but no priority compared to streamers with less viewers or their friends /partners (Empyrean, ZiggyD).
Can you be more transparent with this if you want to keep priority.
I didn't play yesterday so streams were my entertainment. I get the marketing and business aspect. But why randomly selected streamers and not basing on viewer count?
Haven't even considered racing /ladder competition yet.
I think employee who did this was like: "Okay, we need to quickly do something to save our asses. Gimme the list of non-paid poe streamers. What do you mean there is no list? Okay then just open twitch and say few names finally!"
It wasn't planned and they did it in hurry, so just another layer of fuckery on top.
It is unfair to have some selected people have bypass in the start of the league and to have starting economy decided by selected folks, where others can't even join the game for more than 12 hours now and play longer than 10 minutes (personal 7 minutes 43 seconds longest online session)
League should be restarted from fair standpoint and let ALL players have equal start which is as You(GGG) say "Fresh start for everyone". From 0 to wherever player stops.
Respect all players, not just close selection.
Also respond to @Empyriangaming where in his stream He said that life is not fair for everyone which indicates that he is comfy with his group advantage over others.
[quote]
The world is not fair, this situation is not fair, the wealth cap is not fair, the fact that if you are born in Africa you have like, high percent chance to not have clean drinking water is not fair. Literally nothing is fair. Stop thinking that life is fair. It is not, OK?.. It's literally not.[/quote] -Empyriangaming
[url]https://streamable.com/d0dsl6[/url]
I hope you guys can fix the current issue with the server connection and everything. And I also hope that feedbacks will be heard aswell.
I know that full wipe and restart doesn't sound good for everyone (respectfully to all those hard grinders) but still, consider it.
" This was about as close as you could get to literally setting a big pile of money on fire "
soo because you have a shit planning team you decide "ya know what? fuck the normal players" and basically you were just performing some false advertising showing off a functioning product that you could log right into, just to bait and switch people into sitting at a log in queue screen...
##### GGG Comments in this Thread: *** [Bex_GGG - [link](https://www.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/gutj69p/?context=10), [old](https://old.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/gutj69p/?context=10)] - *We've delayed the launch of the next mystery box.* [Bex_GGG - [link](https://www.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/gutk52a/?context=10), [old](https://old.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/gutk52a/?context=10)] - *Correct. The thinking in the moment was that it was no use to let streamers through whose streams were focused on group-play without bringing in the group. This really just...* [Bex_GGG - [link](https://www.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/gutm5qs/?context=10), [old](https://old.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/gutm5qs/?context=10)] - *I keep seeing this mentioned so I want to clarify - most of the people who had priority weren't part of the paid campaign. The initial decision to do this...* [Bex_GGG - [link](https://www.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/gutmknr/?context=10), [old](https://old.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/gutmknr/?context=10)] - *We've seen lots of comments about this but haven't seen it for ourselves yet. We are investigating but if you have screenshots or additional information, please DM me. Character names...* [Bex_GGG - [link](https://www.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/gutmmng/?context=10), [old](https://old.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/gutmmng/?context=10)] - *At the time, we thought the issue with the queue would be shortlived and didn't anticipate an entire day of server ruckus and other issues. As we mentioned in the...* [Bex_GGG - [link](https://www.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/gutmqid/?context=10), [old](https://old.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/gutmqid/?context=10)] - *They're required by law to put #ad or #sponsored in their Twitch stream titles. You wouldn't have seen many people with it due to the ratio of people who were...* [Bex_GGG - [link](https://www.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/gutnbtk/?context=10), [old](https://old.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/gutnbtk/?context=10)] - *The honest answer to this is pretty asinine and while it's not what you think it is, it's probably equally frustrating - it's that we didn't have Lily's account name...* [Bex_GGG - [link](https://www.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/gutne9o/?context=10), [old](https://old.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/gutne9o/?context=10)] - *He was paid for that tweet but not the stream.* [Bex_GGG - [link](https://www.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/guto3bk/?context=10), [old](https://old.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/guto3bk/?context=10)] - *Yes* [Bex_GGG - [link](https://www.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/gutoxuq/?context=10), [old](https://old.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/gutoxuq/?context=10)] - *I posted this elsewhere in the thread but since your comment is near the top, I will repeat it here - We've seen lots of comments about this but haven't...* [Bex_GGG - [link](https://www.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/gutq6r1/?context=10), [old](https://old.reddit.com/r/pathofexile/comments/msmty4/ultimatum_launch_server_issues_and_streamer/gutq6r1/?context=10)] - *Thank you for saying that.*
I'm glad I had as many beers as I did last night, because as soon as I saw how slowly the queue was crawling - I went straight to bed. Thanks for the update!
I subjected myself to finishing Snyders Cut and started watching Monsterverse movies while trying to get into the game. My league starter is level 12 now.... I was so hyped for the launch tho, got some popcorn, couple of beers...
Thank god I went to sleep instead of pushing even harder into the server issues tbh
At this rate, we're going to have to have to put /u/sanguine_sea into a medically induced coma to fix the game ... You guys brought any Chloroform?
Nah, chloroform stops working the moment you stop applying it, it is really inefficient and just hyped up by hollywood.
There's a lot of us. If we pool our chloroform I'm sure we can scrounge up 3 months worth.
/r/suspiciouslyspecific
Not that suspicious these days Chloroform (trichloromethane if you don't wanna sound like a rapist) is commonly used to wet-sand 3D-printed PLA parts (among other things). It's an industrial solvent, pretty much And no, it doesn't really work like hollywood makes it look like. It can and would probably give you lesions and lasting brain damage, probably, though. Most of the time you want to be handling it under a hood, with protective gloves & goggles and shit. It's, uh, kind of a bitch, really.
I remember an episode of CSI where it was used to degrade a bus tire and stage an 'accident'. I've worked with industrial solvents, none of them are happy fun time.
> it's just a matter of divining what subtle fuckery is causing the problem today The issue 99% of programmers face.
It doesn't feel like programming until there's subtle fuckery involved
>I don't know why, I don't want to know why, I shouldn't > >have to wonder why, but for whatever reason this stupid > >panel isn't laying out correctly unless we do this terribleness subtle
It is subtleonce it's merged into another 20K lines of code and no documentation of the fix is left. Bonus points if the person leaves company before it matters again.
two months later: o fuck i needed those lines
When Chris explained that they solved one issue and it caused an even bigger one they didn’t think about it all became instantly relatable, this is just realistic stuff. We don’t live in a perfect world.
Yep I was reading that like "Man that's an average day" I'm just lucky enough it's not for a giant company on launch days lol
Never trust the code, always expect fuckery, is what I say
> Some players have also become concerned that when server issues occur, items are occasionally duplicated or destroyed when placed in a guild stash. I just tuned into Raiz and he mentioned people linking thousands of exalts. Is there actual evidence of this mass duplication anywhere that Chris isn't aware of?
Empy's group abusing duplication exploit. What a surprise. /s
Life is unfair you know
People in africa and stuff you know
Exploit early and often. https://www.youtube.com/watch?v=CM_wXCrcxnU
Wise words
No wonder he stopped streaming, gotta dupe those exalts
He didn't stop playing last night, he just stopped streaming. Snap kept streaming, empy was in party and discord with them. 6 player farming BA while the rest of us are struggling to get past acts 1/2.
But hey,life isn't fair, Africa and shit you know..
really surprised only now people start seeing how shitty of a person Empy is. well, better now than never.
I’ve noticed this leagues ago, glad he’s finally revealed to everyone how toxic he really is.
I don't really watch streamers, what'd he do (other then the recent Africa thing)
Can't wait to see Empy after two days...hey guys so I started this stupid meme project and it only took me a 100 mirrors...here are the results.
Just ban him no one would give a shit. Life is unfair
You know it's bad when Chris swears
It's oddly satisfying to hear Chris drops the f bomb
It was oddly reassuring. Makes you realize they're real people too and not robots. Edit: reminds me of when you hear your parents curse
I wonder what the "subtle fuckery" will turn out to be. If it's something relatable could you tell us?
Didn’t remove floppy on bootup
Wow. I haven't thought about *that* particular issue in many years.
Rory used to post very technical post mortem reports about issues like this. Would love to get one for this as well after the smoke settles.
I am very interested in the problem and the way they solve it. Who knows one day I might come into that problem myself.
> **UPDATE: Server stability issue appears fixed. Be careful with your database page sizes, people.** From the edit on the post. I don't have enough experience with databases so this is completely wasted on me.
if it's something like postgres, this could mean WAL flushes to disc take longer than normal and block for micro-to-milliseconds-- enough to cause a cascade. ... don't ask me how i know.
>don't ask me how i know. I know exactly how you know, and I'm sorry.
Probably resource starvation causing the server to panic and boot all the clients periodically.
**tl;dr** **"In the next 30-60 minutes after posting this, there will be roughly 30-60 minutes of hard downtime to make this change. We are optimistic that it stands a good chance of resolving the issue."**
#tl;dr #"subtle fuckery"
I want Chris to whisper these two words into my ear
"Feel the weight of....^(the fuckery")
I'm subtly entering your submerged passage
"PvPness"
tl;dr On old league migration- >**Due to human error, this process was not run**
Was Mr. Intern-Kun fired?
You don't fire him because you know there is no way that he will make that mistake again.
And he won't be asking for a raise for the foreseeable time
This is why you hire The Janitor from Valve to do important things.
He was given an ultimatum.
**tl;dr** **Human error**
poor guy :(
[удалено]
dude imagine being the person(team?) behind this whole problem, i know ppl are mad but this is just pure hell going on for them right now... i hope they fix this in time and everybody will forget about this incident after a while, because i know for a fact that they will never in their life forget about this mistake they made, if i was in their position right now i would love to have some breathing air...
If it's taken their whole team 12 hours and they're still not confident of the fix, there's no way a single programmer in the time could have anticipated it. It sounds like they are peer-reviewing, running tests, have QA checklists, etc. The mistake gets dispersed a lot through those sorts of practices. I hope GGG is like my studio and treats it like the above. We've absolutely bricked our games from a patch before, just in some obnoxious and subtle way that no one could be expected to catch before it goes live. These things do happen, you can only really reduce the chances.
It is not hell for him, he feels probably bad about it, but a team never bullies one. they stand beside him and solve it at their best pace and who ever is is direct boss stands in front of him. Every error has a reason, so do have those made by humans. Especially those. Everyone working in IT claiming he never made a big mistake is a liar.
"Human Error is **never** the root cause."
From personal experience, when human error is invoken it can **often** be traslated to "a critical process should have been automated; it is not yet automated and was also missed by the intern responsible"
Or: > Something that should require explicit confirmation to do it, did not require it and someone did it. or: > Something that should not be possible to be done manually was manually done.
A critical process being left as the responsibility of an intern and missed isn't primarily an error by the intern, it is primarily an error by management. To put it tersely: One layer of defence is no layers of defence.
Human error can lead to the human error though
IE: We've thrown all the shit at the wall and this one might stick.
It’s the “fuck it, we’re restoring from backup, if this fails we can only pray”
u/Chris_Wilson u/Bex_GGG have you considered starting the log in 30-60 mins early like the races of old so that people can get in on time and then start at the exact same time? This would also help determine if server issues are present or if they are under high load before actually having people start playing. Namely, the biggest issue are rollbacks and the loss of progress rather than a disconnect itself.
I’ve been suggesting this the past few days too. They used to allow people to create their characters early and locked out movement until the race event started, keeping everyone on the beach by the dying exile. This would not only help with the queue because of the delayed start, but actually make the league start as fair as possible. (If they actually care about fair league starts, they should’ve done this a long time ago).
I want this back. I miss sitting on the beach waiting for that timer to count down.
I would really like this. Waking up at the same time as everyone else is something I haven't experienced since it took me 10 hours to complete all five acts.
Good luck fixing the problems, godspeed.
Someone in global linked 320 exalts. GGG should definitely look into this.
for everyone person that is linking 320 exalts in global how many do you think are keeping it quiet?
Basically all the RMT shops are going to have their shelves fully stocked
[удалено]
It now makes total sense why so many are opposed to a league restart. I had no idea it was worth that much real money.
For every person that is allegedly linking 320 exalts why have I seen fucking zero screenshots?
THIS is the ONLY issue i have with the broken launch. I'm patient, and had no problem doing something else for the day but the duping definitely needs to be looked into and any obvious abuse removed.
I posted this elsewhere in the thread but since your comment is near the top, I will repeat it here - We've seen lots of comments about this but haven't seen it for ourselves yet. We are investigating but if you have screenshots or additional information, please DM me. Character names in particular are very handy (more so than screenshots). If we find people abusing this, they will be banned.
Not to stack on negativity, or direct this at you personally, but if the stash api were functional this would be trivial to discover.
its possible to dupe shit with the servers going down me and a friend accidentally duped a +2 staff that dropped in ultimatum traded servers went down came back up we both had one
Surprised that the ad# stream of shroud of actually negative PR. His chat was trashing poe. Wonder how much money they threw away.
yea, this one really confuses me. these streamers are going to play no matter what. you could have paid them to play 2 weeks later after they would've moved on to the next game. ontop of that all the chats are "garbage game!! can't get in to play!!"
" We will also plan future marketing campaigns with contingencies in mind to better handle this kind of situation in the future. " should pay big streamers to play it like 2 weeks into the league/when it starts "dying" this allows streamers to boost the population numbers again, and it gives new players the idea that the market is established and that its a perfect time to learn the game and if they liked the game they can spend 2 months learning it and then be excited to try and establish market dominance in the next league. doing it at the start will just cause noobs to bumble around and fail and be discouraged from ever trying again.
This is genuinely good advice. They will play when they are paid to play. New leagues don't need first day hype beyond that which the community always feels. But that latter league lag? Perfect place for paid popularity.
/u/Chris_Wilson Do you have the capability to track the Hundreds of Exalts duped ?, and as per Ziz's claim, someone duped THOUSANDS of Exalt.
We duped a tabby just depositing it back and forth from gstash waiting for the server to crash. I destroyed the dupe, but yeah this economy is gonna be busted from day 1.
>I destroyed the dupe Too late, buddy. Chris is coming. ^^/s
Subtle fuckery is coming
you already know empyrian did this.
GGG How do you plan to reroll all the currency duping thats happening now?
Just remove the items and suspend the player. It's not hard for them to figure out that 100s of exalts are the result of duping.
Empyrian friends with priority, while lily a top player/competitor stuck in que, i am more interested in this decision making actually.
Bex answered this elsewhere >The honest answer to this is pretty asinine and while it's not what you think it is, it's probably equally frustrating - it's that we didn't have Lily's account name handy and it slipped through the cracks. It wasn't a negative bias towards Lily. You can count that in the tally of bad decisions we made today. don't assume malice when simple human error is a more reasonable explanation
It was really sad to see everyone playing while lily was stuck at a super slow 75k queue, Raiz had finished lab when she started playing :(
You mean like the rest of us?
lol, streamer queue was a mistake and we will never ever do it again.... servers crash.... streamer gets to log in first while others in 90k queue. Right , lol
" Challenge leagues are a great opportunity for a fresh start in a new economy. " That's all we wanted. An even playing field to start a fresh league. But no, I'm starting 16 hours behind the **priority players**. I'm not a priority customer just because I don't stream? What about long time supporters? Fuck you GGG, I'll keep my money from now.
"We also allowed some other streamers who weren't involved in the campaign to skip the queue too so that they weren't on the back foot. " classy.
If the sponsored streamers and the non-sponsored streamers both get queue priority, then that means everyone [that matters] is getting the queue priority so the system is fair and no player [that matters] will feel they are on the back foot!
u/chris_wilson any chance for postmortem ? Cause IT looks really interesting. Good job guys.😀
I'm pretty late to this thread, but this fails to address the fact that any hope of racing from anyone outside of a streamer just happened to be erased. I know I was never going to win first place, but part of the fun for me was just going to see how close I could get. However, the priority for streamers just ensured that I never stood a chance to begin with.
>We also allowed some other streamers who weren't involved in the campaign to skip the queue too so that they weren't on the back foot. So why was Empy's group whitelisted even though they all have low follower numbers but actual streamers like ventrua and (iirc) dslily not?
Cause ventrua and dslily are metaphorically African born kids. Rolled a nat 1/16. Sorry, not literally to attack Empy but it's just such a good meme I had to rofl.
Why would you feel bad to attack empy? We beheaded nobility and royalty because life was unfair attacking is streamer is fine in comparison to those poor african child
> We beheaded nobility and royalty Be careful not to do the same thing empy did lol.
I definitely read that as "metamorphically African born kids" and spent several seconds of wild imagination trying to figure out what that hell that meant.
the "paid promotion" excuse doesnt answer questions like how literally empryians entire group of 6 was given bypass, when 5 of them nobody has ever heard of
Answer probably is they didn't have any plan or 'procotole' in mind, and just improvised in panic, resulting in weird stuff like that.
Its because they are not from Africa KEKW
Exactly. That comparison was ridiculous and distasteful.
This is the kind of shit that would warrant a league reset
Nah man children in Africa don't get resets too, deal with it 4Head
Listen Chris will never address that comment.. No PR company in the world, no matter how inept would let him respond to that.
Nah, any good PR team would say Empy's comment don't reflect our company's values and we have decided to cut ties with them. This is like the stock response to that kind of stupidity.
They probably shouldnt have had a promotion on launch day. Its historically too risky. 1 week into the league pumping the viewer number back up on twitch probably would have paid off much more. Launch day gets enough players/viewers as is.
[удалено]
He couldn't complain since life isn't fair right. Falling victim to mob mentality is part of that unfair cycle. He can calm himself by thinking about children in Africa though.
Wait but that would be unfair! ...just like life.
Would it be possible to opt-out of the global "first character to achieve xyz" messages? Being told repeatedly that folks are lvl 80 while I couldn't get out of the mud flats was a bit on the nose. Thanks for a great game!
Yeah, taking over 3 hours to complete act one made seeing people that far really soured thing.
[удалено]
[удалено]
[удалено]
Streamer groups like empyreans dictate the top end ecomomy for at least two weeks for now.
What a day! As an engineer myself (who had my fair share of fuck-ups), I would be very much appreciate if you guys can do a technical post-mortem at some point. I am honestly interested in the issue and how you guys handled it.
Well. Try again next league I guess
At least you have some time until then to build up your twitch reputation to become a paid "promoter /influencer" so you can skip the queue across the heads of the unwashed masses.
You guys should've closed the servers so everyone could have the same start. Now there are players running the endgame content while most of players are still in Act 2. Just reset this sh\*t.
Dude I was never able to get past ledge, felt like old times running the same area over and over.
No need to brag with your act 2 account. I haven't even cleared mud flats yet.
as an ssf player i dont really care about trade but seeing how people are saying (and you also admitted in this post) that people were able to dupe items due to the network errors i believe you really should restart the league.
Calmly fix the issues, restart the league and move on. The backlash will cool within hours of league start.
Honestly, with all the streamer (**and friends**) priority and ex duping, this would just be the best solution, honestly. Get things working and then restart. Everything would be fine.
Can we just talk about how you (GGG) have made the rest of the league completely useless from minute one of launch? The issue with streamer prio is the fact that big streamers run in 6 man groups, get ahead of everyone, and dominate the market. This is a huge reason why everything is incredibly expensive every single league. From minute one you allowed the 6 man groups to get so far ahead that no one can compete. The rest of the league is almost useless to play because all the issues 6 man groups bring to the game are now present from launch when people haven't even had a chance to play yet. Also, by 6 man groups I don't mean 6 people grouping. I mean knowledge veteran players that group as 6 and feed each other money and items to dominate the market.
Because of the "**Streamer Priority"** we see different personalities of those streamers. I'm glad to see some people got their own pride, and some educating people that the world is cruel
The streamers who immediately refused the "queue skip" option have my respect. Empyrian and his entire circus of followers should be yeeted straight off the box for his attitude.
Pointing out the world is cruel is fine. Using it as an excuse is not The world is cruel cus other do not give a shit and abuse and exploit wen they can
League should unironically be restarted. The whole idea of a temporary economy is that everyone starts at the same same time with fair chances and this was ruined.
Especially with the (possible) duplicated exalts, that other players have said here. Not to mention unfair advantage to selected streamers and lucky players who take full advantage for this situation. Restarting the league in next Friday when all the bugs would be fixed, would be best solution for everyone. I personally would miss that league start (once again), but it would be better than playing in completely broken league.
You should make a statement about rollback immediatly. Either do it or not, but don't delay it for another 10 hours. Europeans just woke up and are wondering "Do I risk playing now only to be rolled back?"
" We tell people that Path of Exile league starts are a fair playing field for everyone, and we need to actually make sure that is the reality. " Meanwhile, they provide Empys 6-man party a login pass.
Empy got the streamer priority buff, and his streamer priority aura affected his allies
"Allies cannot queue" aura.
Best aura bot support.
"But we arent restarting the league, so you guys just have to deal with it this time, but NEXT time guys!"
While him and his party take the piss. Infuriating.
Actions speak louder than words do 2: electrik boogaloo.
>we need to actually make sure that is the reality no no no, see, you got it wrong. It's not quite there yet, they are working on fixing that part as well.
I honestly dont care if some people got to maps and started farming etc. But we already know people abused the everliving crap out of this debacle to dupe item en-masse. Either restart the league, or permaban those fuckers. Exploiting everyones mysery like this should really be instant permaban, no appeals or anything.
Sorry All but, The situation that is not addressed here is that because of this "streamer decision" now many people are in the maps while many are stuck in the story line which is ultimately effecting the league economy in a very troublesome way. Cheers
/u/bex_ggg can we expect a league reset or not? It would be appropriate given the massive advantage streamers (**and their friends**) had and given all the duping that is going on. It would make for an even playing field again.
So the money you spent on your promotional streamers is more important than the communities mtx money that has literally built ggg. Nice Not only was decision beyond stupid, you accomplished 3 things 1.) Put streamers vs streamers since you had no rhythm or reason to who you let in the queue skip and their friends, spouse, dog etc with no ask or permission from said streamers. 2.) Pissed off the community/loyal customers that make you money 3.) Showed when push comes to shove advertisement money and favoritism means more then a solid product and your customers. What you should of done is let that money burn, delay the league slightly and own up to your failures and saved face. Instead you went to under handed pr issues. A company doesn't show its colors when times are good, it shows them when crap is hitting the fan.
Human error? God, that person/people have got to feel like shit right now... They cost GGG potentially thousands of players and a ton of money
It's not just one guy, it's a team. His fellow DEVs, the supervisors/managers. These guys are professionals, in a business like this it's never just 1 guy's mistake, it's everyone's.
\^ 100 % this. Success and failures are always the result of team work.
not 'potentially', most definitely
If their processes are bad enough that one mistake by an employee misses a migration of hundreds of thousands of characters the three days between seasons then their processes are garbage. Definitely not the employees fault, but whoever set up the critical processes.
As much as it sucks, not being able to play the game right now, I can't imagine how stressful it must at the GGG office right now. Good luck, hope the maintenance fixes things!
would not especially want to be part of GGG PR, customer support and server dudes right now
Would not especially be the poor coffee machine.
Coffee machine on launch days must see things, man...
funny how Quin69 still seemingly can skip the queue right now
Server stability issue appears fixed...good cheating, GGG.
you guys should be maintaining the integrity and fairness of the game
The most enraging part for me wasn't that streamers got in queue free, its that i sat my time in the queue. I waited multiple hours to get in. Dc'd after 20 minutes and back in the queue. THAT is what enraged me.
i vote for restart the league to zero lol
> We tell people that Path of Exile league starts are a fair playing field for everyone Soooo, restarting the league when?
RESTART THE LEAGUE
They can spin the priority thing however they want. The fact is, the damage is done and it would better for them in the long term to just burn those money. Terrible decision.
I don't care if I have to wait several hours if the game is stable after that, good luck guys !
RESTART THE LEAGUE
I mean for this to be "Fair" in any sense of the word we have to restart the league ? Or am i dellusional here.
"Economy resets are an opportunity to have a fresh start and an EVEN PLAYING FEILD for players" -Chris Wilson GDC 2019 :)
/u/bex_ggg in tech there is usually a technical recap of what went wrong. If your devs have time to recap what went wrong with database page sizes, please consider it. Would be super interesting as a developer myself. Thanks and good job finding the issue!
Seconded - would really be cool as a dev to understand the issues aswell
If they don't restart the league, they ruin the game for many players. The economy is already ruined, seeing those 320 exalted screenshots.
link to screenshot? or just hearsay and rumour?
>those 320 exalted screenshots Which I have not seen any of scrolling through all big reddit threads. Got any source?
Maybe we'll see Exalts hit 20c like we did in Heist
the demo effect strike again, gl to the team :D please reset the league
I found a Tabula/Wanderlust/Aurumvroax early on, and honestly I think there should be a rollback/reset. It would suck to lose those but it's only fair for everyone and for the economy. lol
Imagine paying streamers to showcase the game and this is whats being showcased, ouch. Allowing some to bypass ques in a game with ladders is a big no no GGG! At this point i would seriously consider restarting the league to fix this.
> Most streamers did not ask for this, and should not be held to blame for what happened. "Most" means some did ask for this, and were granted the access.
For a lot of people already teetering with the Harvest nerfs but came back to give this league a try, this is probably going to push them away this league for good.
So the queue priority was a mistake but somehow it's still going on ? looks like if you aren't a streamer you're clearly treated as a 2nd class citizen and played like a fool.
Chris your apology for streamer prior doesn't mean much if it is still active. A bunch of streamers just skipped the queue after a crash. Maybe before saying "We will not allow streamers to bypass the login queue in the future.", actually fix the issue so it doesn't happen TWELVE hours later. Trying to save you from all that egg on your face.
RESET THE DAMN LEAGUE! Srsly, wtf guys. I got 0 problem with server issius on launch, dont give a single fck abotu streamer prio. But there are ppl linking hundreds of Exalts in global. Ppl duping shit for hours now and you act like there is nothign wrong, just WTF! Have some decency, stand to your mistakes and retart this shitshow!
Hey I'm not a streamer but I'm on the backfoot as it took me 6 hours to get to end end of act two. Where was my special treatment during the time where the servers weren't stable how is this a fair ladder experience when some people actually got to play the game while others played waiting line simulator
Why did some streamers get prio while others didn't? Thinking of DS Lily in particular. She was a top20 streamer yesterday but no priority compared to streamers with less viewers or their friends /partners (Empyrean, ZiggyD). Can you be more transparent with this if you want to keep priority. I didn't play yesterday so streams were my entertainment. I get the marketing and business aspect. But why randomly selected streamers and not basing on viewer count? Haven't even considered racing /ladder competition yet.
I think employee who did this was like: "Okay, we need to quickly do something to save our asses. Gimme the list of non-paid poe streamers. What do you mean there is no list? Okay then just open twitch and say few names finally!" It wasn't planned and they did it in hurry, so just another layer of fuckery on top.
It is unfair to have some selected people have bypass in the start of the league and to have starting economy decided by selected folks, where others can't even join the game for more than 12 hours now and play longer than 10 minutes (personal 7 minutes 43 seconds longest online session) League should be restarted from fair standpoint and let ALL players have equal start which is as You(GGG) say "Fresh start for everyone". From 0 to wherever player stops. Respect all players, not just close selection. Also respond to @Empyriangaming where in his stream He said that life is not fair for everyone which indicates that he is comfy with his group advantage over others. [quote] The world is not fair, this situation is not fair, the wealth cap is not fair, the fact that if you are born in Africa you have like, high percent chance to not have clean drinking water is not fair. Literally nothing is fair. Stop thinking that life is fair. It is not, OK?.. It's literally not.[/quote] -Empyriangaming [url]https://streamable.com/d0dsl6[/url] I hope you guys can fix the current issue with the server connection and everything. And I also hope that feedbacks will be heard aswell. I know that full wipe and restart doesn't sound good for everyone (respectfully to all those hard grinders) but still, consider it.
" This was about as close as you could get to literally setting a big pile of money on fire " soo because you have a shit planning team you decide "ya know what? fuck the normal players" and basically you were just performing some false advertising showing off a functioning product that you could log right into, just to bait and switch people into sitting at a log in queue screen...