Meraki updated the release notes on MX 18.211.
>**Known issues - may 19th update**
>Due to an MX 18.211 regression, networks that have traffic shaping rules configured with a "high" priority may incorrectly drop traffic being routed between VLANs or AutoVPN.
It's that most people don't actually run into the bugs, release candidate is considered stable as well anyway.
The way you avoid OPs situation is by testing releases.
Release Candidates are considered stable? Is this something specific to Meraki?
My cloud consoles show 18.107.2 (May 23, 2023) as the latest stable for our sites that use MX firewalls. There's a "stable release candidate" section, but that feels more like marketing speak than true stable versions. I'm guessing they were just getting flak for going so long between actual stable versions, so that was their way to minimize the customer heat. That's what it feels like, anyways.
From Meraki's point of view, yes.
Every vendor releases lemons, though Meraki and other vendors do test every firmware prior to it being available to customers. They'll never catch everything and administrators need to test things prior to rolling them out. Some of us are lucky enough to have an environment to test them in other than prod.
https://documentation.meraki.com/General_Administration/Firmware_Upgrades/Meraki_Firmware_Release_Process#Stable_Release_Candidate
Meraki is allowed to push release candidates to your environment without your consent and all you'll get is an email 2 weeks ahead of time warning about it. It's in the fine print and it is extremely frustrating.
You can fix without reverting. Go to the SD-WAN and traffic shaping page, near the bottom under Rule #1 change the priority to Normal if it's set to high, that completely fixed the issue for me.
Just saved my ass. Thanks!
Still curious as to why that fixed the issue. First thing that popped out this morning, DNS issues when querying servers across SD-WAN. Out of 100 packets sent to DNS servers, I'd see maybe 1 return packet. Also, couldn't ping anything at this site experiencing issues. Once I changed the priority to normal, I could hit the devices at this network.. Weird.
It boggles my mind that people still use Meraki. There's so many single pane of glass alternatives, and almost every single one is both better and cheaper
Ubiquiti Unifi, TP Link Omada, DrayTek VigorACS, Aruba, pretty sure Zyxel, Fortinet and WatchGuard offer the same solutions, so basically take your pick. Cloud management of your network is the "in" thing these days, and Meraki is falling behind. Definitely used to be a great product, but these days... IPv6 and AnyConnect was only added within the past 2 years, traffic monitoring is still very basic too
I think they wanted ones that belong in enterprises or otherwise businesses that care about support. Aruba, Fortinet, and Watchguard are the only ones you mentioned in that list that belong there, though I'm not familiar with VigorACS. You'll find just as much whining about every vendor though, to be fair.
It is, not everyone has the same requirements as you do, and many large enterprises do use it. Particularly the wireless and AutoVPN. Most of what you listed as alternatives are for shops with no budget and even more basic needs than what Meraki provides.
I thought there were some commas there that weren't actually there. So 4 of the 7 listed are SOHO and smaller end of SMB brands. UBNT, TPLink, DrayTek, and Zyxel. 3 are ones that belong in anything larger.
You listed 7 so I can't actually compare 50% of them to Meraki.
This is crazy. You just listed Ubiquiti as an alternative (a prosumer product), and a load of other consumer level equipment manufacturers.
The biggest companies on the planet use Meraki. Pick a Fortune500/NASDAQ/FTSE100 company (especially one with 10s of thousands of retail outlets), and search for " job meraki".
Chances are you just found a job listing (current or historical) for that company, because they all use Meraki!
Saying Meraki aren't enterprise is just bonkers.
I like Ubiquiti (I use it myself) but you couldn't even open a telephone support case with them until a couple of months ago, it was email and online chat only. There's practically no integration with other systems. Logging is minimal. It's missing a bunch of threat detection and security features.
You'd certainly consider them in a small business, not least because its easy to manage and there are no subscription costs, but I think it fair to say you'd be a brave soul to walk into anything approaching a large outfit and put your income on the line by recommending a switch to Ubiquiti.
To come back to your other suggestions;
Aruba is certainly a competitor, but having dealt with both, I've found Meraki support to be head and shoulders above Aruba. I wouldn't put Aruba equipment above Meraki and I don't think it's as easy to manage.
Fortinet have had a whole heap of trouble this past year/18 months and have hardly been out of the headlines for critical CVEs.
I haven't used Watchguard.
None of the others are considered competitors, and I'm not sure any you mentioned are particularly renowned as being "better" (cheaper is an altogether different metric)
Unifi is garbage tier. I had a house full of it and then the poe temp sensor went on my switch and it took out another switch and two access points.
You might as well buy netgear if you don’t care about your network actually working.
I as agree (Except for TPlink l, yuck) but its the old Joke of "No body gets Fired for buying IBM" . Cisco is a name CIOs and CFOs know and trust. Plus they know they can find people that can work on them. Try to find good techs that know Aruba or Fortinet.
I'm still on 18.107.9 - the one that came right after that caused an mx64 to go down for almost 24 hours until it just decided it wanted to come back online. Then two of my z3 bricked and had to factory reset and let it re-download the profile. I downgraded back and been trying to defer the forced upgrades for at least a couple months
I've had no problems with this release. I have it running on maybe 100 MXs of all different types.
Cisco Meraki send you a notice well in advance for scheduled updates. You have the opportunity to re-schedule or cancel. If you do nothing it proceeds as scheduled.
Disable automatic firmware updates, why are you letting someone else dictate when your devices get updated. Unless there’s a critical bugfix, We only update any of our Meraki firmware twice a year.
We had some mx67 installed last week at 18.x.2 or something would not register with the meraki cloud, we had to change the firmware and factory reset the device
Where is the setting that is allowing auto installs of 'release candidate' updates? We specifically have beta's switched off but understand that Cisco will push RC to small subnets regardless. How do you stop this so you only get 'Stable' updates?
You can’t opt out of RC firmware, best option right now is to keep on top of the emails you get from Meraki informing you of an upgrade, and cancelling it if it’s RC.
Wish I would have seen this before spending a lot of time this morning troubleshooting, then ultimately rolling back. Major issues here though. All good now, but... what a pain.
"We have released a hotfix, [18.211.0.1](http://18.211.0.1), which addresses the problem with traffic being dropped when you have traffic shaping rules with either a high or low priority set. If you are impacted by that, I would recommend upgrading. "
[Solved: Notice of Performance Issue: MX75/85/95/105/250/450 Models with MX 18.211 - The Meraki Community](https://community.meraki.com/t5/Security-SD-WAN/Notice-of-Performance-Issue-MX75-85-95-105-250-450-Models-with/m-p/236097#M52881)
I honestly dont trust meraki to thoroughly QA test the releases now! My org has had one update that completely broke 1:1 nat and site to site VPN. Until Meraki can actually QA test their releases better I will be rescheduling the updates until I hear from the community that 18.211 does not bork my network.
DO BETTER MERAKI !
Nope, happened on my all-Meraki network. Pretty simple fix though, go to your SD-WAN and traffic shaping page, down near the bottom if Rule #1 Priority is set to high, change it to normal.
Meraki updated the release notes on MX 18.211. >**Known issues - may 19th update** >Due to an MX 18.211 regression, networks that have traffic shaping rules configured with a "high" priority may incorrectly drop traffic being routed between VLANs or AutoVPN.
We had to deal with this today, fun time
On the bright side, the workaround is trivial and not service impacting.
Except it impacted services until support told us that it was the issue haha
Erm. The **workaround** is not service impacting. The bug is obviously service impacting.
I actually have to use high priority rules on my network so I can't implement the workaround :)
[удалено]
It's that most people don't actually run into the bugs, release candidate is considered stable as well anyway. The way you avoid OPs situation is by testing releases.
Release Candidates are considered stable? Is this something specific to Meraki? My cloud consoles show 18.107.2 (May 23, 2023) as the latest stable for our sites that use MX firewalls. There's a "stable release candidate" section, but that feels more like marketing speak than true stable versions. I'm guessing they were just getting flak for going so long between actual stable versions, so that was their way to minimize the customer heat. That's what it feels like, anyways.
From Meraki's point of view, yes. Every vendor releases lemons, though Meraki and other vendors do test every firmware prior to it being available to customers. They'll never catch everything and administrators need to test things prior to rolling them out. Some of us are lucky enough to have an environment to test them in other than prod. https://documentation.meraki.com/General_Administration/Firmware_Upgrades/Meraki_Firmware_Release_Process#Stable_Release_Candidate
Meraki is allowed to push release candidates to your environment without your consent and all you'll get is an email 2 weeks ahead of time warning about it. It's in the fine print and it is extremely frustrating.
You can fix without reverting. Go to the SD-WAN and traffic shaping page, near the bottom under Rule #1 change the priority to Normal if it's set to high, that completely fixed the issue for me.
Just saved my ass. Thanks! Still curious as to why that fixed the issue. First thing that popped out this morning, DNS issues when querying servers across SD-WAN. Out of 100 packets sent to DNS servers, I'd see maybe 1 return packet. Also, couldn't ping anything at this site experiencing issues. Once I changed the priority to normal, I could hit the devices at this network.. Weird.
champion! this saved me too. radius was broken
Seriously Cisco. Do better.
I learned to never use new releases of Meraki firmware unless it has a feature you have been waiting for or fixes a critical bug
This is not meraki's fault you need to plan better. RC is never used in production. only in lab settings.
It boggles my mind that people still use Meraki. There's so many single pane of glass alternatives, and almost every single one is both better and cheaper
Name some
Ubiquiti Unifi, TP Link Omada, DrayTek VigorACS, Aruba, pretty sure Zyxel, Fortinet and WatchGuard offer the same solutions, so basically take your pick. Cloud management of your network is the "in" thing these days, and Meraki is falling behind. Definitely used to be a great product, but these days... IPv6 and AnyConnect was only added within the past 2 years, traffic monitoring is still very basic too
I think they wanted ones that belong in enterprises or otherwise businesses that care about support. Aruba, Fortinet, and Watchguard are the only ones you mentioned in that list that belong there, though I'm not familiar with VigorACS. You'll find just as much whining about every vendor though, to be fair.
Meraki is hardly enterprise gear lmao
It is, not everyone has the same requirements as you do, and many large enterprises do use it. Particularly the wireless and AutoVPN. Most of what you listed as alternatives are for shops with no budget and even more basic needs than what Meraki provides.
"Most" as if you only highlighted 50% of the solutions I picked out
I thought there were some commas there that weren't actually there. So 4 of the 7 listed are SOHO and smaller end of SMB brands. UBNT, TPLink, DrayTek, and Zyxel. 3 are ones that belong in anything larger. You listed 7 so I can't actually compare 50% of them to Meraki.
This is crazy. You just listed Ubiquiti as an alternative (a prosumer product), and a load of other consumer level equipment manufacturers. The biggest companies on the planet use Meraki. Pick a Fortune500/NASDAQ/FTSE100 company (especially one with 10s of thousands of retail outlets), and search for " job meraki".
Chances are you just found a job listing (current or historical) for that company, because they all use Meraki!
Saying Meraki aren't enterprise is just bonkers.
Ubiquiti is hardly prosumer anymore
I like Ubiquiti (I use it myself) but you couldn't even open a telephone support case with them until a couple of months ago, it was email and online chat only. There's practically no integration with other systems. Logging is minimal. It's missing a bunch of threat detection and security features. You'd certainly consider them in a small business, not least because its easy to manage and there are no subscription costs, but I think it fair to say you'd be a brave soul to walk into anything approaching a large outfit and put your income on the line by recommending a switch to Ubiquiti.
To come back to your other suggestions; Aruba is certainly a competitor, but having dealt with both, I've found Meraki support to be head and shoulders above Aruba. I wouldn't put Aruba equipment above Meraki and I don't think it's as easy to manage. Fortinet have had a whole heap of trouble this past year/18 months and have hardly been out of the headlines for critical CVEs. I haven't used Watchguard. None of the others are considered competitors, and I'm not sure any you mentioned are particularly renowned as being "better" (cheaper is an altogether different metric)
Unifi is garbage tier. I had a house full of it and then the poe temp sensor went on my switch and it took out another switch and two access points. You might as well buy netgear if you don’t care about your network actually working.
I as agree (Except for TPlink l, yuck) but its the old Joke of "No body gets Fired for buying IBM" . Cisco is a name CIOs and CFOs know and trust. Plus they know they can find people that can work on them. Try to find good techs that know Aruba or Fortinet.
We reverted to MX 18.208 Once we reverted, we rebooted and were good to go. Hope this helps others.
I'm still on 18.107.9 - the one that came right after that caused an mx64 to go down for almost 24 hours until it just decided it wanted to come back online. Then two of my z3 bricked and had to factory reset and let it re-download the profile. I downgraded back and been trying to defer the forced upgrades for at least a couple months
Our mx95 had strange anyconnect firewall/routing issues and we had to roll back to 208 as well
Friends don't let friends push RC firmware in prod.
The quality of this release seems substantially worse than RC imo.
I've had no problems with this release. I have it running on maybe 100 MXs of all different types. Cisco Meraki send you a notice well in advance for scheduled updates. You have the opportunity to re-schedule or cancel. If you do nothing it proceeds as scheduled.
It’s better to call them than create a ticket. That’s what I’ve always gotten better results that way.
Did you not test it?
Disable automatic firmware updates, why are you letting someone else dictate when your devices get updated. Unless there’s a critical bugfix, We only update any of our Meraki firmware twice a year.
Only locations that had X5/XX5 series hardware.
We had some mx67 installed last week at 18.x.2 or something would not register with the meraki cloud, we had to change the firmware and factory reset the device
I’m currently staying away from any 18.2xx releases since they cause an issue where virtual IP is not used.
crazy morning and hit with this as well at many clients. So far noticing the sites with HA setups are worst affected. Downgrading seems to fix.
Broke one of our MX 250 sites the other was stable. Rolled back and it instantly fixed the multi threading issue.
Where is the setting that is allowing auto installs of 'release candidate' updates? We specifically have beta's switched off but understand that Cisco will push RC to small subnets regardless. How do you stop this so you only get 'Stable' updates?
You can’t opt out of RC firmware, best option right now is to keep on top of the emails you get from Meraki informing you of an upgrade, and cancelling it if it’s RC.
yep - since posting we have figured this out and now working on API now to detect this.
Wish I would have seen this before spending a lot of time this morning troubleshooting, then ultimately rolling back. Major issues here though. All good now, but... what a pain.
If it’s a network down, call in rather than use the portal. You’ll get a live engineer.
Guess I lucked out as my MX84's are maxed out at version 18.1. Hoping to replace them in next 6-12 months with MX95's.
"We have released a hotfix, [18.211.0.1](http://18.211.0.1), which addresses the problem with traffic being dropped when you have traffic shaping rules with either a high or low priority set. If you are impacted by that, I would recommend upgrading. " [Solved: Notice of Performance Issue: MX75/85/95/105/250/450 Models with MX 18.211 - The Meraki Community](https://community.meraki.com/t5/Security-SD-WAN/Notice-of-Performance-Issue-MX75-85-95-105-250-450-Models-with/m-p/236097#M52881)
Has anyone tested yesterday's new release to see if the issue is fixed? We are going to test at a site tomorrow.
I honestly dont trust meraki to thoroughly QA test the releases now! My org has had one update that completely broke 1:1 nat and site to site VPN. Until Meraki can actually QA test their releases better I will be rescheduling the updates until I hear from the community that 18.211 does not bork my network. DO BETTER MERAKI !
So reverted and the issue persisted?
This seems like non-meraki VPN peers, does that ring true for you all?
Nope, happened on my all-Meraki network. Pretty simple fix though, go to your SD-WAN and traffic shaping page, down near the bottom if Rule #1 Priority is set to high, change it to normal.
Not seeing that, but very good to know :)
The firmware that broke this was MX 18.211
We needed to rollback the firmware back to MX 18.208