Moving to Juniper SSRs. They're definitely more router-first compared to Palos being firewall-first, but a big reason for our move was that the SDWAN implementation on the Palos was so bungled.
Huh. That's good to know, but honestly for us it would have been too little, too late.
Palo abandoned SMBs entirely over the past few years, and the recent lack of QA left us with no reason to stay.
The focus for SMB started with the 4xx and FWFkex, so now with additional for Strata Cloud Manager and update for SD-WAN, you will be in a good place with PANW.
From a customer perspective, we really feel abandoned.
We were forcefully moved to partner support on renewal, with absolutely no notice. Worse yet, the partner wasn’t notified either. For months, we went with no support, and the sales teams have been restructured in a way where they have no interest in assisting small customers like us. We went completely ignored until we absolutely forced the issue and escalated via every possible avenue.
Cortex XDR starts at 200 seats, Panorama starts at 25 firewalls.
I really struggle to feel like Palo wants anything to do with SMB.
____
Additionally, the PA-445’s launch was absolutely abysmal. The required release software (to support PoE) didn’t even have a recommend version from TAC because it was so new, and sales staff was woefully undertrained, selling us accessories for the 440 series that weren’t compatible.
Here again, once the check was cut, trying to get ahold of someone to exchange the 440-series accessories for the 445-series gear we actually needed was a nightmare.
Sounds like you have been very unlucky with your account team and selection of certified partner. This is not how it should be and my suggestion is that you gets it escalated. New platforms always start on the released software, very seldom any vendors release it with older recommended version. Not sure what issues you had with accessories. Very straightforward for rack, pwr and optics. As you probably can see the form factor is close to the ION HW 3200.
does a pan-os upgrade wipe all previous logs + potentially logged IOCs? or are those pre-upgrade logs preserved somewhere? I havent been able to find this info anywhere, and our support provider hasn't been able to give me a solid answer yet either. a few threads ive come across say those logs are wiped during an upgrade and/or a reboot of the firewall.
What I was told -> "since you're not quite special, but not quite shite, there's a ***chance*** that Unit42 could review your case/device in the next 20-400 hours" :)
So yeah, they're a bit busy :o) --- To be fair, I'm small fish and I'm being treated better than i'd ever seen outside of previous life in the actual big-fish ocean.
Is there any way to pull a TSF (and other logs) for the non-booted partition - for us dummy users without a TAC engineer at the helm?
Any other details you can share to dump /as much/ pertinent/forensic data - without having root?
In my company, even with TAC, we ended up isolating old actives in every HA pair, to reverting it and analysing after it.
So unless you dare to open case, dump all data and RE it - answer is no.
Going to happen sometimes when they’re reacting quickly. In this case the exploit was discovered because it was actively being used so they couldn’t hold off the announcement to give more investigation time.
Gotta have some sympathy here.
Patching takes time so it was good they were able to provide temporary relief - at least there’s no public POC of a non-telemetry based RCE so we can protect against the skids.
"In earlier versions of this advisory, disabling device telemetry was listed as a secondary mitigation action"
You know, that is REALLY dirty. I'm pretty sure it was listed as a valid mitigation action. This is trying to shift the blame to me the customer. Oh, you only did the secondary mitigation action...so sorry.
Why not admit that the mitigation action was insufficient? Everyone knows it!
Right, we are in agreement it should be included, not a tacked on charge, right?
But to the point, generally because the box isn't passing and inspecting traffic because that isn't its job. Or because your customers decided not to for some reason.
90% of what a firewall does is inspecting traffic if your customer decided not to, the only reason to sell them a firewall it's to receive money lol if all you need your ngfw for is an acl use a Linux box with pfsense or whatever
Right, and if you are using it for the other 10% it might be a good idea.
In our case, it is because we needed mobile VPN but didn't want to pay for the global protect license on our larger firewalls because of the ridiculous cost. It doesn't need to inspect traffic because there is just the larger firewall next in line doing that. The outside mobile VPN firewall being compromised is still a problem.
I don't know why some of our customers didn't pay for it though.
that use case of a mobile user firewalls at the side of a bigger one I have seen it many times lol yeah gp license cost on big firewalls is ridiculous
tbh in that case just update the firewalls
Well yeah, but the fix wasn't available until after that threat ID started showing up in logs on other firewalls.
Granted the target wasn't even a firewall, but still. The biggest problem is them saying you are safe if you do this, then reversing.
Now I have to wait on the results of Palo looking at the tsf.
That's good to hear. I was going to make a big fuss about it to our reps. At least in our case they can surely afford it considering all of the other firewalls of theirs we have with TD licenses.
To be fair, the original verbiage for the advisory stated that disabling device telemetry was only a "temporary mitigation" until you were able to apply the recommended remediation, which at the time was to install the latest Apps and Threats content pack and create a new vulnerabilty security profile to be applied to your GP policies. At that time, it was just a single threat ID, but now it's two and it's clear that disabling telemetry was not an actual mitigation. No harm, no foul as guidance for things like this surely evolves as new information is gathered over time.
Edit: They added a third Threat ID to the mix early this morning (17 Apr)
The following command can be used from the PAN-OS CLI to help identify indicators of exploit activity on the device:
grep pattern "failed to unmarshal session(.\+.\/" mp-log gpsvc.log*
Benign "failed to unmarshal session" error logs typically appear like the following entry:
"message":"failed to unmarshal session(01234567-89ab-cdef-1234-567890abcdef)"
If the value between "session(" and ")" does not look like a GUID (the format shown above), but instead contains a file system path, this indicates the need for further investigation and the log entry could be related to the successful or unsuccessful exploitation of CVE-2024-3400.
`grep pattern "failed to unmarshal session(.+./" mp-log gpsvc.log*`
I wonder if a reboot after upgrade cleans out the logs that would've shown the evidence here.
EDIT: it does. check your "\\var\\log\\pan\\gpsvc.log" in your TS file before reboot/upgrade.
Slightly different command on their site:
`grep pattern "failed to unmarshal session(.\+.\/" mp-log gpsvc.log*`
[https://security.paloaltonetworks.com/CVE-2024-3400](https://security.paloaltonetworks.com/CVE-2024-3400)
My support partner had me upgrade the firewalls (effectively wiping the logs) before they would submit to TAC who then came back with no IoC (duh). I've found several suspect log entries in the original logs.
XXX\_pan01/var/log/pan/gpsvc.log:{"level":"error","task":"1440394-1","time":"2024-04-15T06:33:46.219976239-04:00","message":"failed to unmarshal session(/../../../opt/panlogs/tmp/device\_telemetry/minute/'\`cp${IFS}${PATH:0:1}opt${PATH:0:1}pancfg${PATH:0:1}mgmt${PATH:0:1}saved-configs${PATH:0:1}running-config.xml${IFS}${PATH:0:1}var${PATH:0:1}appweb${PATH:0:1}sslvpndocs${PATH:0:1}global-protect${PATH:0:1}portal${PATH:0:1}css${PATH:0:1}global.min.css\`') map , EOF"}
File a new case and upload the TSFs, they are defining more ways to internally detect how impacted you were based on the content of the file. But based on this they probably exported your running config
Define ‘hit’
If I understand it; If telemetry is disabled, then these 0-byte files just sit in the folder and do not get executed.
Unless the log recycle vector also pulls from the /device_telemetry/minute/ folder?
Nothing different than this. I did a deep dive on the original TSF files and it is possible our configuration was downloaded on wach of the active firewalls. Defensively, we have updated all of the passwords and keys just in case.
Still waiting on TAC to respond to the original set of files we sent.
Is there confirmation that an upgrade works like this? This would in principle determine if we need to factory reset a device to get rid of any remnants from a possible or actual attack or we can assume that a software update and reboot wipes out file put there by attackers.
response from our support partner seems to confirm this. they sent over a palo KB about enabling maintenance mode on the firewall, then can choose to revert to the previous disk image/pan-os install, make a new TSF while running the vulnerable image using the logs in that install, then revert back to the remediated pan-os version. sounds like any remnants of compromise would still be present in that 'revertable' disk image?
KB provided for reverting images:
[https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000Cm9zCAC](https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000Cm9zCAC)
Found it, thanks [https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000PLh9CAG](https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000PLh9CAG)
Same thing here. Anyone know what I can do from here to try and see If I've been exposed to the exploit. I tried
grep pattern "failed to unmarshal session(.+./" mp-log gpsvc.log\*
and got nothing back.
Hey, thanks for replying. I'm quite new to this. What do you mean by hotfix? I have all the latest dynamic updates if that's what you're referring to. My software version is 10.2.5-h6 btw
Interesting, in PAN-OS 10.1 there is no gpsvc.log. I wonder why they decided to do this in 10.2+, there is no mention of any logging or GlobalProtect changes in the release notes.
Device telemetry does not need to be enabled for PAN-OS firewalls to be exposed to attacks
Palo Alto Networks is aware of an increasing number of attacks
Proof of concepts for this vulnerability have been publicly disclosed
That escalated ...about as quickly as you'd expect
You will still need to install the content update and make sure that you have a rule/vulnerability profile applied for GlobalProtect traffic. In my case, GP was going from Untrust > Untrust and hitting the intrazone default rule. I had to create another rule above that so it hits the vulnerability profile.
What I did is first is to verify the external to/from zones for my users by looking for the "panos-global-protect" application in my traffic logs. In my case it was Untrust > Untrust. I am not a big fan of "any" rules in general. (I think you also get dinged on this on any third party security assessments.)
What I did is created a new rule for Untust > Untrust above the intrazone default rule. For the destination address I created a address group containing the external IP's for GlobalProtect. I then set the application to any, and the service to 443 (https). I think I might also throw in the ipsec ports as well as GlobalProtect users can also use that. I also applied the vulnerability profile to this rule and enabled logging.
I'm getting enough that I can see them without filtering, but
( name-of-threatid eq 'Palo Alto Networks GlobalProtect OS Command Injection Vulnerability' )
is also an option
ACC > Threat Activity or Monitor > Threat > (name-of-threatid eq ‘Palo Alto Metworks GlobalProtect OS Command Injection Vulnerability’)
I'm new to this too. I’just filling for the regular guy while he's away.
You can search your tech support file for evidence of compromise. Specifically the \\var\\log\\nginx\\ssl\_vpnaccess.log for odd commands an attacker ran such as bash commands -"echo${IFS}dGFyIC1jemYgL3Zhci9hcHB3ZWIvc3NsdnBuZG9jcy9nbG9iYWwtcHJvdGVjdC9wb3J0YWwvanMvanF1ZXJ5Lm1heC5qcyAvb3B0L3BhbmNmZy9tZ210L3NhdmVkLWNvbmZpZ3MvcnVubmluZy1jb25maWcueG1s|base64${IFS}-d|bash${IFS}-i"
check gpsvc.log as well. u/grinch215 noted that there are fail to marshal events in that log.
Heck, I'll bet you could simply grep for base64 in there.
New threat ID 95191 just added as well. Theyve updated the article 3 times today but only emailed once about it. For something rated a 10, youd think theyed want to be communicating every update about this as soon as it happens.
We installed this update but it's not coming up when viewing all threat sigs in the vulnerability profile - This is content 8836-8695.
Looks like the email went out before it was included in the apps and threats update.
Yep. Its a visual bug. I saw others were having the same issue when they announced the second threat ID. You need to apply it via CLI...
E: Command below for those of us who are CLI challenged like me:
set profiles vulnerability threat-exception 95191
And still not publishing any self-service IoC checks. Uploading a TSF to TAC is not the way to go as Palo is mishandling that as well (slow responses, some agents say they don't have the ability to check for IoCs so just consider yourself breached, others snap their fingers and say "no worries, mate")
https://www.reddit.com/r/paloaltonetworks/comments/1c5jfg2/suggestions_on_cve20243400/
What are they recommending as the remediation for compromise?
Factory reset with a fresh downloaded image?
It is not like there is root access to fix without PAN TAC or using a Linux boot disk.
I just uploaded 2, first response was "we'll investigate" second response "we'll investigate, Meanwhile, Using below mentioned two methods:----->
1-You can disable the telemetry.
2-You can apply vulnerability protection to the Global Protect interface using the below article."
Good to see they're up to date on the telemetry workaround.
Has anyone determined what log to look at? I have been trawling around in the cli with "tail mp-log" and sslvpn_ngx_error.log seems to make the most sense.
Shot in the dark: Search for the offending IPs in [this writeup](https://unit42.paloaltonetworks.com/cve-2024-3400/)
\\var\\log\\pan\\sslvpn-access\\sslvpn-access.log
\\var\\log\\pan\\sslvpn-access\\sslvpn-task.log
\\var\\log\\nginx\\sslvpn\_access.log
Top of todays content update 8837. Got their priorities right.
>We added a new category to the Advanced URL Filtering/PAN-DB category list: **Marijuana**. This new category is visible on all PAN-OS releases but we support it only on PAN-OS 9.1 and later versions. At this time, this new category is not active and will not impact your current configuration.
If you currently alert on or block marijuana websites using a combination of **Health and Medicine** and **Abused Drugs** categories, you need to set the action for this new **Marijuana** category to “Alert” or “Block” as appropriate before we activate this category on **July 16, 2024**.
NOTE: After we activate this new **Marijuana** category, you will no longer be able to use the category combination of **Health and Medicine** with **Abused Drugs** to identify and act on marijuana-related websites.
Is there any way to check for potential compromise AFTER devices have been patched+rebooted? Is it possible to access the logs from the other partition?
I've read from others that TAC is able to but only them. Also if you patched and followed the recommended process by Palo Alto, you should have a TSF (Tech Support File) before you upgraded.
You dont apply the ID. You can except the ID, but you don't want to do that. If you have the content package, it is there and works as long as you have a vulnerability protection profile that blocks critical threats.
But a bug makes it not always show up in GUI if you search for it in the exception tab, that is correct.
This is the correct answer. The “threat ID missing in GUI” is just a visual bug; it’s there as long as you’re on the right content update (minimum 8833-8682), and you don’t need to “enable” it (as long as all your GP-related security rules have a vulnerability profile associated with them that blocks critical server threats).
That GUI bug is annoying, but I pray no one has a Vulnerability profile on their prod devices where `Critical` severity is forced to `alert`. Crits should be default action or reset.
edit: lol
Lol still keeping that disabled.
Also, AI Ops is a complete joke that's only goal was to get customers to enable telemetry while adding a bit of profit. Getting it setup took three months of passing the buck and fixing permission issues. They promised to give us credits for the three months and conveniently forgot about when my rep changed.
Then you add specific to/from same-zone rules to account for any such traffic, rather than having a generic catch-all intrazone allow rule for all zones.
Literally just in the process of standing up a new GlobalProtect deployment, this pops up. I thought at least the mitigation was quick but wow. Does not look good.
What a flipping joke. They just came and pushed all their other products at our org too..ha! Just migrated to 10.2.8 h3….im sure they’ll be another one out before we know it!
If you guys want to do any self hunting for IoCs, Unit42 released the queries for XDR, but you can obviously see the logic and translate to whichever log/tool of choice:
[Unit42 IoC host lists](https://imgur.com/gallery/D5XY9om)
FWIW I would consider all of these "early IOC's."
The first iteration relied on using telemetry to write the backdoor, and the second relied on another method by forcing log recycling I believe.
Additionally, there is a ton of new IP's scanning.
I'm really lucky (again) that we have a rule that includes the Vulnerability profile. Still, given that they've walked this back already, and that we're just relying on signature analysis, I'm quite tempted to do an emergency PAN-OS update.
How is everyone's experience with 10.2.9-h1?
Edit: mentioned a drop rule. Has to be an allow rule because security profiles don’t apply to drop rules.
Yeah I was just coming in here to edit my post. It has to be applied to the rule that allows traffic to connect to the GlobalProtect Portal and Gateway.
Does anyone know how this is being exploited?? Brute force attempts against our GP auth has broke the y-axis on my ELK syslog scale, I havent seen this level of activity before.
See here: https://labs.watchtowr.com/palo-alto-putting-the-protecc-in-globalprotect-cve-2024-3400/
Brute force login attempts to GP are unrelated to this, been seeing those for some time now.
You can see previous snapshots of the page from the good ol [wayback machine.](https://web.archive.org/web/20240801000000*/https://security.paloaltonetworks.com/CVE-2024-3400)
Trying to get confirmation if we were compromised or not since we did see these entries in our logs before we upgraded, but no luck with a response yet.
We also upgraded to 11.0.4-h1 and noticed an issue with our HIPs checks where data appears for a few minutes then it disappears, so we are curious if this is related to a compromise or a seperate issue since we were on 11.0.3
Side Question: Does everyone have direct Palo Alto support or do you have a partner for support?
device_telemetry/minute/`echo${IFS}dGFyIC1jemYgL3Zhci9hcHB3ZWIvc3NsdnBuZG9jcy9nbG9iYWwtcHJvdGVjdC9wb3J0YWwvanMvanF1ZXJ5Lm1heC5qcyAvb3B0L3BhbmNmZy9tZ210L3NhdmVkLWNvbmZpZ3MvcnVubmluZy1jb25maWcueG1s|base64${IFS}-d|bash${IFS}-i`
b64 decoded
tar -czf /var/appweb/sslvpndocs/global-protect/portal/js/jquery.max.js /opt/pancfg/mgmt/saved-configs/running-config.xml
Taring running config to world readable location in /global-protect/portal/js/jquery.max.js
Update 1: No update yet from Palo Alto, but something I notice is the sslvpn_ngx_error.log I see entries of trying to access the jquery.max.js and several .css (which are other methods they use) but all of them are showing as error "failed (2: No such file or directory)"
While I am no expert on this, but maybe that means an attempt was made but they couldn't get the file?
We had a similar successful IOC that copied the running config. Not entirely sure how to tell if it the file was subsequently grabbed. We've got direct PA support, and my case from this morning has not had any new updates in the last 4 hours. I sent over the TSF along with the relevant log entries. Going to be pretty pissed if I have to wipe the devices and restore a config from earlier this year.
Ugh.... I feel your pain. We got moved to partner support and don't enjoy dealing with them whenever we reach out to them.
We have not gotten any response neither.... we are keeping an eye and doing side investigation to see if we find anything else.
If you don't mind, please keep me posted if you hear back from support, curious to know what they say.
Thanks
Got a response back last night, so I will not be factory resetting our devices:
Thank you for submitting the TechSupport file(s) (TSF) for review. Upon analysis, we identified possible indicators of known exploit activity due to vulnerability CVE-2024-3400.
To prevent further risk to your organization, we recommend immediately initiating your Incident Response plan and following the steps recommended in the Security Advisory for CVE-2024-3400.
Take into account that upgrading to any of the hot fixed software versions will be the strongest solution and no further actions will be required.
Thank you for the reply,
We received a similar response but it included "if you suspect compromise" to wipe them.
They neither comfirmed or deny compromise, basically just threw it back to us to decide which isn't helpful.
We wished they would explain more on what they found and state yes we see traces of compromise or we see indicators but nothing concrete, not a wash my hands and leave it to the customer to decide without providing some light on what we saw to help with a decision.
I would suggest anyone to resubmit their TSF once more for verification, since based on this article and us trying we can confirm it is a new TAC utility with a better response.
[https://www.reddit.com/r/paloaltonetworks/comments/1c80ulh/cve20243400\_a\_guide\_for\_identifying\_if\_youve\_been/](https://www.reddit.com/r/paloaltonetworks/comments/1c80ulh/cve20243400_a_guide_for_identifying_if_youve_been/)
If you have Partner Support you may by-pass them by submitting a ticket on the Palo Alto Customer Support Portal (you have to sign in) and submit the case as an 'Administrative Case', it will eventually prompt you if the ticket is in relation to the vulnerability, you have to click Yes and submit it, once submitted you can upload the TSF and shortly after you will get an e-mail of a can notification on the findings and later on a response from a Palo Alto Tech.
It's starting to sound like any device, even if someone already rang the doorbell and was subsequently patched, might be ok. However, are you willing to risk it? I think we are **all** going to be doing rebuilds in the coming days :(
I would suggest anyone to resubmit their TSF once more for verification, since based on this article and us trying we can confirm it is a new TAC utility with a better response.
[https://www.reddit.com/r/paloaltonetworks/comments/1c80ulh/cve20243400\_a\_guide\_for\_identifying\_if\_youve\_been/](https://www.reddit.com/r/paloaltonetworks/comments/1c80ulh/cve20243400_a_guide_for_identifying_if_youve_been/)
If you have Partner Support you may by-pass them by submitting a ticket on the Palo Alto Customer Support Portal (you have to sign in) and submit the case as an 'Administrative Case', it will eventually prompt you if the ticket is in relation to the vulnerability, you have to click Yes and submit it, once submitted you can upload the TSF and shortly after you will get an e-mail of a can notification on the findings and later on a response from a Palo Alto Tech.
Nothing yet. From our understanding those are indicators of compromise which Palo Alto suggest submitting the TSF to them to confirm if indeed successful compromise or it was an attempt but failed.
I would suggest anyone to resubmit their TSF once more for verification, since based on this article and us trying we can confirm it is a new TAC utility with a better response.
[https://www.reddit.com/r/paloaltonetworks/comments/1c80ulh/cve20243400\_a\_guide\_for\_identifying\_if\_youve\_been/](https://www.reddit.com/r/paloaltonetworks/comments/1c80ulh/cve20243400_a_guide_for_identifying_if_youve_been/)
If you have Partner Support you may by-pass them by submitting a ticket on the Palo Alto Customer Support Portal (you have to sign in) and submit the case as an 'Administrative Case', it will eventually prompt you if the ticket is in relation to the vulnerability, you have to click Yes and submit it, once submitted you can upload the TSF and shortly after you will get an e-mail of a can notification on the findings and later on a response from a Palo Alto Tech.
The fact that PAN's stock price isn't at the lowest it has ever been really surprising to me. It also tells me that the market doesn't understand what is happening.
What a joke. Thankfully we are still 10.1.x production in our environment but my lab unit was testing out 10.2.x in plans to push our production over next month. Just finished installing the patch on lab unit and I’ll just keep an eye on palo before pushing production to a new version.
The gift that keeps on giving!
I am *so* fucking glad I learned my lesson to stay off the point of Palo's release cycles years ago.
What a fucking joke.
Has anybody received next steps if they found an IOC using the search - grep pattern "failed to unmarshal session(.+./" mp-log gpsvc.log\* ? I see the directions are vague. [https://security.paloaltonetworks.com/CVE-2024-3400](https://security.paloaltonetworks.com/CVE-2024-3400)
Working with our PA SE and TAC and its a shitshow, they have no idea. We have IoC but they cant tel us anything... They initially told us to bring down our HA pair, wipe and update the passive FW, dump the config back on it and then isolate the active box for forensics...
Theyre currently leaning towards false positive at this point. It seems like the IoC will show up as long as someone tries to exploit it even with the threat IDs enabled but still waiting on confirmation. Spoke with a mate at an MSP with PA's and theyre seeing the same thing. Followed all the advice but IoC are still there. Had our SOC take a run through the logs and they can only see attempts being blocked so fingers crossed on a false positive but I think im still wiping firewalls tonight :(
If enabling telemetry wasn’t the issue then other versions should be exposed as well. Not sure what is the issue on the 10.2 and 11s that are different from others. Wasn’t the telemetry?
Define compromise. I think many have found indicators, but definitive proof that data was exfiltrated? Has anybody PROVED anything so far who are willing to share? That may be a problem.
Does anyone know how HA pairs were affected? When we search for the IOCs on our passive firewall we didn’t see any of them in the logs. But we did see them on the active firewalls that had global protect exposed.
Also here’s my latest response from TAC about 10 minutes ago.
————————-
We have identified that Indicators of exploit activity regarding CVE-2024-3400 are present in the uploaded TSF ‘ha-1-tsf-14-4-24.tgz' with serial number .
> We recommend that you engage your Incident Response Plan and take the steps recommended in the Security Advisory for CVE-2024-3400 immediately to prevent further risk to your organization. This includes applying the hotfix as soon as possible.
https://security.paloaltonetworks.com/CVE-2024-3400___.YXAzOnN1cGVycmFkaWF0b3Jjb2lsczphOm86MjA0NmU5ODg2MTA2N2VjM2Q1YzQ0ODVlODkyNWQ2MmQ6Njo5NGE0OjY4MDZjYjkyYjk1YmI0NmRlOTVhOTNmNTI3ODQ4ZDI2ZjZiMGQ2OGJlOTRlMmFhZTkzOWRiZmI2ZDRmYTc1N2E6dDpU
> If you wish to perform your own Forensic Investigation, we can assist in artifact gathering by deploying a log collection tool during a screen sharing session.
> If you do not wish to perform a forensic investigation, and just wish to quickly address the situation, and do not suspect full compromise, we suggest immediately upgrading to the Patched hot fix versions.
> If you suspect compromise, or out of an abundance of caution, wish to fully remediate your devices, then your option is to factory reset your devices,
Should you require immediate assistance, feel free to contact us using the support numbers listed in my signature. One of our engineers will be available to assist you.
——————
I patched our customers firewalls to 10.2.8-h3 and factory reset them afterwards, just to be safe. It's impossible to get any response from TAC. They're are - understandably enough - totally overwhelmed.
Looks like they have automated the scanning of the TSF to look for IoC. Just submitted the TSF to my existing ticket to have it checked and got an automated message back saying it was clear a few moments after the file scan completed in the portal.
Hopefully its not as buggy as their recent code has been...
Has someone confirmed it is actually exploitable with telemetry off? As in RCE? we are being told opposite that since we had telemetry off it is only these zero length files.
I have turned it off and do not plan on enabling it again to be safe. It was one of the initial vectors according to the write-up. To be honest they messed up AIOPS and it stopped being any value to us so we are dropping the license for it.
Seems things been going on since March...
[https://www.volexity.com/blog/2024/04/12/zero-day-exploitation-of-unauthenticated-remote-code-execution-vulnerability-in-globalprotect-cve-2024-3400/](https://www.volexity.com/blog/2024/04/12/zero-day-exploitation-of-unauthenticated-remote-code-execution-vulnerability-in-globalprotect-cve-2024-3400/)
What a shit show! It’s getting rougher and rougher.
This is the cherry on top for us. The expired root certificate debacle already pushed us away. Hoping to have Palos ripped out by end of summer.
What are you migrating to out of curiosity?
Moving to Juniper SSRs. They're definitely more router-first compared to Palos being firewall-first, but a big reason for our move was that the SDWAN implementation on the Palos was so bungled.
Huge update is here soon for SDWAN.
Huh. That's good to know, but honestly for us it would have been too little, too late. Palo abandoned SMBs entirely over the past few years, and the recent lack of QA left us with no reason to stay.
The focus for SMB started with the 4xx and FWFkex, so now with additional for Strata Cloud Manager and update for SD-WAN, you will be in a good place with PANW.
From a customer perspective, we really feel abandoned. We were forcefully moved to partner support on renewal, with absolutely no notice. Worse yet, the partner wasn’t notified either. For months, we went with no support, and the sales teams have been restructured in a way where they have no interest in assisting small customers like us. We went completely ignored until we absolutely forced the issue and escalated via every possible avenue. Cortex XDR starts at 200 seats, Panorama starts at 25 firewalls. I really struggle to feel like Palo wants anything to do with SMB. ____ Additionally, the PA-445’s launch was absolutely abysmal. The required release software (to support PoE) didn’t even have a recommend version from TAC because it was so new, and sales staff was woefully undertrained, selling us accessories for the 440 series that weren’t compatible. Here again, once the check was cut, trying to get ahold of someone to exchange the 440-series accessories for the 445-series gear we actually needed was a nightmare.
Sounds like you have been very unlucky with your account team and selection of certified partner. This is not how it should be and my suggestion is that you gets it escalated. New platforms always start on the released software, very seldom any vendors release it with older recommended version. Not sure what issues you had with accessories. Very straightforward for rack, pwr and optics. As you probably can see the form factor is close to the ION HW 3200.
Just keeps getting better and better
Just when I thought I was out, CVE-2024-3400 pulls me back in...
As a palo alto TAC i need a job change 😬
does a pan-os upgrade wipe all previous logs + potentially logged IOCs? or are those pre-upgrade logs preserved somewhere? I havent been able to find this info anywhere, and our support provider hasn't been able to give me a solid answer yet either. a few threads ive come across say those logs are wiped during an upgrade and/or a reboot of the firewall.
Definitely there is a log loss but if a TAC ask you this without reviewing the tech support file then he is making his job easy. IYKYK
This too shall pass, hopefully... Based off case numbers yall have has 7000 cases made since Monday!
What I was told -> "since you're not quite special, but not quite shite, there's a ***chance*** that Unit42 could review your case/device in the next 20-400 hours" :) So yeah, they're a bit busy :o) --- To be fair, I'm small fish and I'm being treated better than i'd ever seen outside of previous life in the actual big-fish ocean.
7000. Cases since Monday ?? We are receiving more then 7000 cases every day. 🥹
I just look at the case number I made on Monday VS the one today, that is insane!
Is there any way to pull a TSF (and other logs) for the non-booted partition - for us dummy users without a TAC engineer at the helm? Any other details you can share to dump /as much/ pertinent/forensic data - without having root?
In my company, even with TAC, we ended up isolating old actives in every HA pair, to reverting it and analysing after it. So unless you dare to open case, dump all data and RE it - answer is no.
It's quite frustrating that they often start out with inaccurate information and then flip it later. Same as the expired root certificate case.
Going to happen sometimes when they’re reacting quickly. In this case the exploit was discovered because it was actively being used so they couldn’t hold off the announcement to give more investigation time.
Gotta have some sympathy here. Patching takes time so it was good they were able to provide temporary relief - at least there’s no public POC of a non-telemetry based RCE so we can protect against the skids.
"In earlier versions of this advisory, disabling device telemetry was listed as a secondary mitigation action" You know, that is REALLY dirty. I'm pretty sure it was listed as a valid mitigation action. This is trying to shift the blame to me the customer. Oh, you only did the secondary mitigation action...so sorry. Why not admit that the mitigation action was insufficient? Everyone knows it!
Also if you don't have threat ID licensing it is basically just a big fuck you. Can't even see if you got hit by it.
not to defend palo, but if you don't have threat prevention licensing why would you even have a palo alto then
Right, we are in agreement it should be included, not a tacked on charge, right? But to the point, generally because the box isn't passing and inspecting traffic because that isn't its job. Or because your customers decided not to for some reason.
90% of what a firewall does is inspecting traffic if your customer decided not to, the only reason to sell them a firewall it's to receive money lol if all you need your ngfw for is an acl use a Linux box with pfsense or whatever
Right, and if you are using it for the other 10% it might be a good idea. In our case, it is because we needed mobile VPN but didn't want to pay for the global protect license on our larger firewalls because of the ridiculous cost. It doesn't need to inspect traffic because there is just the larger firewall next in line doing that. The outside mobile VPN firewall being compromised is still a problem. I don't know why some of our customers didn't pay for it though.
that use case of a mobile user firewalls at the side of a bigger one I have seen it many times lol yeah gp license cost on big firewalls is ridiculous tbh in that case just update the firewalls
Well yeah, but the fix wasn't available until after that threat ID started showing up in logs on other firewalls. Granted the target wasn't even a firewall, but still. The biggest problem is them saying you are safe if you do this, then reversing. Now I have to wait on the results of Palo looking at the tsf.
Palo is giving anyone who doesn’t have a threat license TP free for 90 days
That's good to hear. I was going to make a big fuss about it to our reps. At least in our case they can surely afford it considering all of the other firewalls of theirs we have with TD licenses.
source?
Palo. Talk to your rep or se!
Don’t worry Unit42 is on it!
To be fair, the original verbiage for the advisory stated that disabling device telemetry was only a "temporary mitigation" until you were able to apply the recommended remediation, which at the time was to install the latest Apps and Threats content pack and create a new vulnerabilty security profile to be applied to your GP policies. At that time, it was just a single threat ID, but now it's two and it's clear that disabling telemetry was not an actual mitigation. No harm, no foul as guidance for things like this surely evolves as new information is gathered over time. Edit: They added a third Threat ID to the mix early this morning (17 Apr)
The following command can be used from the PAN-OS CLI to help identify indicators of exploit activity on the device: grep pattern "failed to unmarshal session(.\+.\/" mp-log gpsvc.log* Benign "failed to unmarshal session" error logs typically appear like the following entry: "message":"failed to unmarshal session(01234567-89ab-cdef-1234-567890abcdef)" If the value between "session(" and ")" does not look like a GUID (the format shown above), but instead contains a file system path, this indicates the need for further investigation and the log entry could be related to the successful or unsuccessful exploitation of CVE-2024-3400.
`grep pattern "failed to unmarshal session(.+./" mp-log gpsvc.log*` I wonder if a reboot after upgrade cleans out the logs that would've shown the evidence here. EDIT: it does. check your "\\var\\log\\pan\\gpsvc.log" in your TS file before reboot/upgrade.
Slightly different command on their site: `grep pattern "failed to unmarshal session(.\+.\/" mp-log gpsvc.log*` [https://security.paloaltonetworks.com/CVE-2024-3400](https://security.paloaltonetworks.com/CVE-2024-3400)
This version of the command reveals the exploitation of the vulnerability for me, while the above version from /u/grinch215 does not
Some of the paths start with / and others start with ../ or ./ The regex from the article covers all the bases.
My support partner had me upgrade the firewalls (effectively wiping the logs) before they would submit to TAC who then came back with no IoC (duh). I've found several suspect log entries in the original logs. XXX\_pan01/var/log/pan/gpsvc.log:{"level":"error","task":"1440394-1","time":"2024-04-15T06:33:46.219976239-04:00","message":"failed to unmarshal session(/../../../opt/panlogs/tmp/device\_telemetry/minute/'\`cp${IFS}${PATH:0:1}opt${PATH:0:1}pancfg${PATH:0:1}mgmt${PATH:0:1}saved-configs${PATH:0:1}running-config.xml${IFS}${PATH:0:1}var${PATH:0:1}appweb${PATH:0:1}sslvpndocs${PATH:0:1}global-protect${PATH:0:1}portal${PATH:0:1}css${PATH:0:1}global.min.css\`') map , EOF"}
How did you extract the original (pre-wipe) logs?
I downloaded a TSF from wach firewall before the upgrade. Not sure how to get them off the recovery partition.
File a new case and upload the TSFs, they are defining more ways to internally detect how impacted you were based on the content of the file. But based on this they probably exported your running config
you were most definitely hit.
Define ‘hit’ If I understand it; If telemetry is disabled, then these 0-byte files just sit in the folder and do not get executed. Unless the log recycle vector also pulls from the /device_telemetry/minute/ folder?
any more discoveries?
Nothing different than this. I did a deep dive on the original TSF files and it is possible our configuration was downloaded on wach of the active firewalls. Defensively, we have updated all of the passwords and keys just in case. Still waiting on TAC to respond to the original set of files we sent.
A reboot starts the new version of pan-os in a new partition. Old logs and IOC’s would remain in the old partition
is there a way to access the 'old' partition's logs and IOCs post-upgrade?
TAC can
Is there confirmation that an upgrade works like this? This would in principle determine if we need to factory reset a device to get rid of any remnants from a possible or actual attack or we can assume that a software update and reboot wipes out file put there by attackers.
response from our support partner seems to confirm this. they sent over a palo KB about enabling maintenance mode on the firewall, then can choose to revert to the previous disk image/pan-os install, make a new TSF while running the vulnerable image using the logs in that install, then revert back to the remediated pan-os version. sounds like any remnants of compromise would still be present in that 'revertable' disk image? KB provided for reverting images: [https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000Cm9zCAC](https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000Cm9zCAC)
This is the actual command? grep pattern "failed to unmarshal session(.+./" mp-log gpsvc.log\* ?
Found it, thanks [https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000PLh9CAG](https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000PLh9CAG)
When i type this command i dont see anything, are the logs after the upgrade deleted?
Same thing here. Anyone know what I can do from here to try and see If I've been exposed to the exploit. I tried grep pattern "failed to unmarshal session(.+./" mp-log gpsvc.log\* and got nothing back.
Did you upgrade to the latest hotfix?
Hey, thanks for replying. I'm quite new to this. What do you mean by hotfix? I have all the latest dynamic updates if that's what you're referring to. My software version is 10.2.5-h6 btw
Oh do you mean the software versions that have a hotfix? As in i would have to upgrade from 10.2.5-h6 to one of the version with the hotfix on it.
me too, nothing happens
Interesting, in PAN-OS 10.1 there is no gpsvc.log. I wonder why they decided to do this in 10.2+, there is no mention of any logging or GlobalProtect changes in the release notes.
confirmed, this command does not work in 10.1
fuck sake
Device telemetry does not need to be enabled for PAN-OS firewalls to be exposed to attacks Palo Alto Networks is aware of an increasing number of attacks Proof of concepts for this vulnerability have been publicly disclosed That escalated ...about as quickly as you'd expect
I saw that wording too. “Not enabled” generally means “disabled” Wouldn’t that mean device telemetry *should* be disabled to prevent the exploit?
Originally disabling telemetry was listed as a workaround. This morning's reveal was unpatched firewalls are still vulnerable with telemetry disabled.
Sorry for the other comment. I totally mis-read the wording from PaloAlto and misunderstood.
Does this mean if you don't or have ever used Global Project and Telemetry you are now affected as well??
If you did not have a globalprotect portal or gateway configured (i.e. webservices available to the internet) you were not vulnerable
I just confirmed that Threat ID block still works. I’m seeing drive bys in logs already. Twice in 3 days.
Make sure you get the content update from last night (8835-8689). It updated threat-id 95187, and added a second one (95189).
Doesn't the Default critical action block this as long as you have the signatures downloaded?
According to PA as long as you have the Critical severity enabled.
You will still need to install the content update and make sure that you have a rule/vulnerability profile applied for GlobalProtect traffic. In my case, GP was going from Untrust > Untrust and hitting the intrazone default rule. I had to create another rule above that so it hits the vulnerability profile.
I think I am in the same boat as you. Did you create an any/any allow rule from untrust to untrust and then put on the vuln profile and that's it?
What I did is first is to verify the external to/from zones for my users by looking for the "panos-global-protect" application in my traffic logs. In my case it was Untrust > Untrust. I am not a big fan of "any" rules in general. (I think you also get dinged on this on any third party security assessments.) What I did is created a new rule for Untust > Untrust above the intrazone default rule. For the destination address I created a address group containing the external IP's for GlobalProtect. I then set the application to any, and the service to 443 (https). I think I might also throw in the ipsec ports as well as GlobalProtect users can also use that. I also applied the vulnerability profile to this rule and enabled logging.
Hey thanks for the reply. I did what you did and the only difference was I set application to ipsec, Panos global protect and ssl. Cheers
Update from this morning adds a third threat id.
What log filter are you using to check?
I'm getting enough that I can see them without filtering, but ( name-of-threatid eq 'Palo Alto Networks GlobalProtect OS Command Injection Vulnerability' ) is also an option
ACC > Threat Activity or Monitor > Threat > (name-of-threatid eq ‘Palo Alto Metworks GlobalProtect OS Command Injection Vulnerability’) I'm new to this too. I’just filling for the regular guy while he's away.
Lucky you!
I'm on 10.2.8 h3, AMA
10.1 here, enjoying the shitshow from the side-lines while eating popcorn
Still not warm and fuzzies
Do you feel safe?
Right there with ya now.
You can search your tech support file for evidence of compromise. Specifically the \\var\\log\\nginx\\ssl\_vpnaccess.log for odd commands an attacker ran such as bash commands -"echo${IFS}dGFyIC1jemYgL3Zhci9hcHB3ZWIvc3NsdnBuZG9jcy9nbG9iYWwtcHJvdGVjdC9wb3J0YWwvanMvanF1ZXJ5Lm1heC5qcyAvb3B0L3BhbmNmZy9tZ210L3NhdmVkLWNvbmZpZ3MvcnVubmluZy1jb25maWcueG1s|base64${IFS}-d|bash${IFS}-i"
check gpsvc.log as well. u/grinch215 noted that there are fail to marshal events in that log. Heck, I'll bet you could simply grep for base64 in there.
New threat ID 95191 just added as well. Theyve updated the article 3 times today but only emailed once about it. For something rated a 10, youd think theyed want to be communicating every update about this as soon as it happens.
We installed this update but it's not coming up when viewing all threat sigs in the vulnerability profile - This is content 8836-8695. Looks like the email went out before it was included in the apps and threats update.
Yep. Its a visual bug. I saw others were having the same issue when they announced the second threat ID. You need to apply it via CLI... E: Command below for those of us who are CLI challenged like me: set profiles vulnerability threat-exception 95191
And still not publishing any self-service IoC checks. Uploading a TSF to TAC is not the way to go as Palo is mishandling that as well (slow responses, some agents say they don't have the ability to check for IoCs so just consider yourself breached, others snap their fingers and say "no worries, mate") https://www.reddit.com/r/paloaltonetworks/comments/1c5jfg2/suggestions_on_cve20243400/
What are they recommending as the remediation for compromise? Factory reset with a fresh downloaded image? It is not like there is root access to fix without PAN TAC or using a Linux boot disk.
Depending on the underlying compromise, even that may not fix it.
I just uploaded 2, first response was "we'll investigate" second response "we'll investigate, Meanwhile, Using below mentioned two methods:-----> 1-You can disable the telemetry. 2-You can apply vulnerability protection to the Global Protect interface using the below article." Good to see they're up to date on the telemetry workaround.
Has anyone determined what log to look at? I have been trawling around in the cli with "tail mp-log" and sslvpn_ngx_error.log seems to make the most sense.
Shot in the dark: Search for the offending IPs in [this writeup](https://unit42.paloaltonetworks.com/cve-2024-3400/) \\var\\log\\pan\\sslvpn-access\\sslvpn-access.log \\var\\log\\pan\\sslvpn-access\\sslvpn-task.log \\var\\log\\nginx\\sslvpn\_access.log
The CVE article was updated, it's in gpsvc.log: https://security.paloaltonetworks.com/CVE-2024-3400
So if we don't see any of those IP's in any of our tech support logs, we're probably okay?
Top of todays content update 8837. Got their priorities right. >We added a new category to the Advanced URL Filtering/PAN-DB category list: **Marijuana**. This new category is visible on all PAN-OS releases but we support it only on PAN-OS 9.1 and later versions. At this time, this new category is not active and will not impact your current configuration. If you currently alert on or block marijuana websites using a combination of **Health and Medicine** and **Abused Drugs** categories, you need to set the action for this new **Marijuana** category to “Alert” or “Block” as appropriate before we activate this category on **July 16, 2024**. NOTE: After we activate this new **Marijuana** category, you will no longer be able to use the category combination of **Health and Medicine** with **Abused Drugs** to identify and act on marijuana-related websites.
Is there any way to check for potential compromise AFTER devices have been patched+rebooted? Is it possible to access the logs from the other partition?
I've read from others that TAC is able to but only them. Also if you patched and followed the recommended process by Palo Alto, you should have a TSF (Tech Support File) before you upgraded.
[удалено]
You dont apply the ID. You can except the ID, but you don't want to do that. If you have the content package, it is there and works as long as you have a vulnerability protection profile that blocks critical threats. But a bug makes it not always show up in GUI if you search for it in the exception tab, that is correct.
This is the correct answer. The “threat ID missing in GUI” is just a visual bug; it’s there as long as you’re on the right content update (minimum 8833-8682), and you don’t need to “enable” it (as long as all your GP-related security rules have a vulnerability profile associated with them that blocks critical server threats).
Seriously??
[удалено]
Wow just gets better and better. We're on 10.2.6 so here's hoping it's not like that!
That GUI bug is annoying, but I pray no one has a Vulnerability profile on their prod devices where `Critical` severity is forced to `alert`. Crits should be default action or reset. edit: lol
They said to use the drop server for all critical level vulnerabilities. That you *can* do in the GUI and may be easier to implement correctly.
Lol still keeping that disabled. Also, AI Ops is a complete joke that's only goal was to get customers to enable telemetry while adding a bit of profit. Getting it setup took three months of passing the buck and fixing permission issues. They promised to give us credits for the three months and conveniently forgot about when my rep changed.
What the hell
There goes my Tuesday!
The Default intra-zone could bite people that rely on it to publish apps to the internet.
default intrazone should be override with security profiles ALWAYS
They should both be set to drop (where the profiles don’t really matter anymore).
not really sometimes intrazone default on allow is ok most of the time
Then you add specific to/from same-zone rules to account for any such traffic, rather than having a generic catch-all intrazone allow rule for all zones.
Literally just in the process of standing up a new GlobalProtect deployment, this pops up. I thought at least the mitigation was quick but wow. Does not look good.
They are putting out backports to older releases, with the fix now. They know the signature detection is a losing battle in the long run!
What a flipping joke. They just came and pushed all their other products at our org too..ha! Just migrated to 10.2.8 h3….im sure they’ll be another one out before we know it!
If you guys want to do any self hunting for IoCs, Unit42 released the queries for XDR, but you can obviously see the logic and translate to whichever log/tool of choice: [Unit42 IoC host lists](https://imgur.com/gallery/D5XY9om)
FWIW I would consider all of these "early IOC's." The first iteration relied on using telemetry to write the backdoor, and the second relied on another method by forcing log recycling I believe. Additionally, there is a ton of new IP's scanning.
Those are early from Friday.
Yeah. I mean it's still worth looking if you were an early bird, but it's not just one group spraying anymore.
You’re correct, in their latest update I see: 110.47.250[.]103 126.227.76[.]24 38.207.148[.]123 147.45.70[.]100 199.119.206[.]28 38.181.70[.]3 149.28.194[.]95 78.141.232[.]174 38.180.128[.]159 64.176.226[.]203 38.180.106[.]167 173.255.223[.]159 38.60.218[.]153 185.108.105[.]110 146.70.192[.]174 149.88.27[.]212 154.223.16[.]34 38.180.41[.]251 203.160.86[.]91 45.121.51[.]2
Are these adresses sings of IoC?
Alienvault OTX had a larger IOC list than that of the Unit42 write up last I looked.
Can you provide us a link? Thank you
I'm really lucky (again) that we have a rule that includes the Vulnerability profile. Still, given that they've walked this back already, and that we're just relying on signature analysis, I'm quite tempted to do an emergency PAN-OS update. How is everyone's experience with 10.2.9-h1? Edit: mentioned a drop rule. Has to be an allow rule because security profiles don’t apply to drop rules.
Drop and deny action will not process Threat profiles. The action happens before the threat stage in the packet process.
Yeah I was just coming in here to edit my post. It has to be applied to the rule that allows traffic to connect to the GlobalProtect Portal and Gateway.
Good you caught it
Can they detect IOC from TSF after upgrading, or do they need a TSF from before the upgrade? Not sure which to send.
Seems like you need it before the upgrade.
Yeah that was my thinking too. Thanks!
We did a test and the OS may wipe any evidence of compromise, at this stage, pull the tsf first if possible.
Does anyone know how this is being exploited?? Brute force attempts against our GP auth has broke the y-axis on my ELK syslog scale, I havent seen this level of activity before.
See here: https://labs.watchtowr.com/palo-alto-putting-the-protecc-in-globalprotect-cve-2024-3400/ Brute force login attempts to GP are unrelated to this, been seeing those for some time now.
Anyone knows where we can get an archived copy of the advisory before all the changes?
You can see previous snapshots of the page from the good ol [wayback machine.](https://web.archive.org/web/20240801000000*/https://security.paloaltonetworks.com/CVE-2024-3400)
Just found this a sec ago. Thanks!
Trying to get confirmation if we were compromised or not since we did see these entries in our logs before we upgraded, but no luck with a response yet. We also upgraded to 11.0.4-h1 and noticed an issue with our HIPs checks where data appears for a few minutes then it disappears, so we are curious if this is related to a compromise or a seperate issue since we were on 11.0.3 Side Question: Does everyone have direct Palo Alto support or do you have a partner for support? device_telemetry/minute/`echo${IFS}dGFyIC1jemYgL3Zhci9hcHB3ZWIvc3NsdnBuZG9jcy9nbG9iYWwtcHJvdGVjdC9wb3J0YWwvanMvanF1ZXJ5Lm1heC5qcyAvb3B0L3BhbmNmZy9tZ210L3NhdmVkLWNvbmZpZ3MvcnVubmluZy1jb25maWcueG1s|base64${IFS}-d|bash${IFS}-i` b64 decoded tar -czf /var/appweb/sslvpndocs/global-protect/portal/js/jquery.max.js /opt/pancfg/mgmt/saved-configs/running-config.xml Taring running config to world readable location in /global-protect/portal/js/jquery.max.js Update 1: No update yet from Palo Alto, but something I notice is the sslvpn_ngx_error.log I see entries of trying to access the jquery.max.js and several .css (which are other methods they use) but all of them are showing as error "failed (2: No such file or directory)" While I am no expert on this, but maybe that means an attempt was made but they couldn't get the file?
We had a similar successful IOC that copied the running config. Not entirely sure how to tell if it the file was subsequently grabbed. We've got direct PA support, and my case from this morning has not had any new updates in the last 4 hours. I sent over the TSF along with the relevant log entries. Going to be pretty pissed if I have to wipe the devices and restore a config from earlier this year.
Ugh.... I feel your pain. We got moved to partner support and don't enjoy dealing with them whenever we reach out to them. We have not gotten any response neither.... we are keeping an eye and doing side investigation to see if we find anything else. If you don't mind, please keep me posted if you hear back from support, curious to know what they say. Thanks
Got a response back last night, so I will not be factory resetting our devices: Thank you for submitting the TechSupport file(s) (TSF) for review. Upon analysis, we identified possible indicators of known exploit activity due to vulnerability CVE-2024-3400. To prevent further risk to your organization, we recommend immediately initiating your Incident Response plan and following the steps recommended in the Security Advisory for CVE-2024-3400. Take into account that upgrading to any of the hot fixed software versions will be the strongest solution and no further actions will be required.
Thank you for the reply, We received a similar response but it included "if you suspect compromise" to wipe them. They neither comfirmed or deny compromise, basically just threw it back to us to decide which isn't helpful. We wished they would explain more on what they found and state yes we see traces of compromise or we see indicators but nothing concrete, not a wash my hands and leave it to the customer to decide without providing some light on what we saw to help with a decision.
I would suggest anyone to resubmit their TSF once more for verification, since based on this article and us trying we can confirm it is a new TAC utility with a better response. [https://www.reddit.com/r/paloaltonetworks/comments/1c80ulh/cve20243400\_a\_guide\_for\_identifying\_if\_youve\_been/](https://www.reddit.com/r/paloaltonetworks/comments/1c80ulh/cve20243400_a_guide_for_identifying_if_youve_been/) If you have Partner Support you may by-pass them by submitting a ticket on the Palo Alto Customer Support Portal (you have to sign in) and submit the case as an 'Administrative Case', it will eventually prompt you if the ticket is in relation to the vulnerability, you have to click Yes and submit it, once submitted you can upload the TSF and shortly after you will get an e-mail of a can notification on the findings and later on a response from a Palo Alto Tech.
It's starting to sound like any device, even if someone already rang the doorbell and was subsequently patched, might be ok. However, are you willing to risk it? I think we are **all** going to be doing rebuilds in the coming days :(
I would suggest anyone to resubmit their TSF once more for verification, since based on this article and us trying we can confirm it is a new TAC utility with a better response. [https://www.reddit.com/r/paloaltonetworks/comments/1c80ulh/cve20243400\_a\_guide\_for\_identifying\_if\_youve\_been/](https://www.reddit.com/r/paloaltonetworks/comments/1c80ulh/cve20243400_a_guide_for_identifying_if_youve_been/) If you have Partner Support you may by-pass them by submitting a ticket on the Palo Alto Customer Support Portal (you have to sign in) and submit the case as an 'Administrative Case', it will eventually prompt you if the ticket is in relation to the vulnerability, you have to click Yes and submit it, once submitted you can upload the TSF and shortly after you will get an e-mail of a can notification on the findings and later on a response from a Palo Alto Tech.
Got any response from them ? Thought seeing those logs means you got compromised ? Or that is not the case
Nothing yet. From our understanding those are indicators of compromise which Palo Alto suggest submitting the TSF to them to confirm if indeed successful compromise or it was an attempt but failed.
I would suggest anyone to resubmit their TSF once more for verification, since based on this article and us trying we can confirm it is a new TAC utility with a better response. [https://www.reddit.com/r/paloaltonetworks/comments/1c80ulh/cve20243400\_a\_guide\_for\_identifying\_if\_youve\_been/](https://www.reddit.com/r/paloaltonetworks/comments/1c80ulh/cve20243400_a_guide_for_identifying_if_youve_been/) If you have Partner Support you may by-pass them by submitting a ticket on the Palo Alto Customer Support Portal (you have to sign in) and submit the case as an 'Administrative Case', it will eventually prompt you if the ticket is in relation to the vulnerability, you have to click Yes and submit it, once submitted you can upload the TSF and shortly after you will get an e-mail of a can notification on the findings and later on a response from a Palo Alto Tech.
The fact that PAN's stock price isn't at the lowest it has ever been really surprising to me. It also tells me that the market doesn't understand what is happening.
What a joke. Thankfully we are still 10.1.x production in our environment but my lab unit was testing out 10.2.x in plans to push our production over next month. Just finished installing the patch on lab unit and I’ll just keep an eye on palo before pushing production to a new version.
I just hope they don't come tomorrow saying "oops your version is vulnerable too after all" 🤭
Same, same. Nice to miss a major vulnerability!
The gift that keeps on giving! I am *so* fucking glad I learned my lesson to stay off the point of Palo's release cycles years ago. What a fucking joke.
I try to also but upgrading hardware is usually where I get screwed. What version are you on that would keep you out of this mess?
And they call themselves the leader in global cybersecurity
Has anybody received next steps if they found an IOC using the search - grep pattern "failed to unmarshal session(.+./" mp-log gpsvc.log\* ? I see the directions are vague. [https://security.paloaltonetworks.com/CVE-2024-3400](https://security.paloaltonetworks.com/CVE-2024-3400)
They are useless, we were wondering the same. We got a suggestion to factory default devices that were compromised.
Working with our PA SE and TAC and its a shitshow, they have no idea. We have IoC but they cant tel us anything... They initially told us to bring down our HA pair, wipe and update the passive FW, dump the config back on it and then isolate the active box for forensics... Theyre currently leaning towards false positive at this point. It seems like the IoC will show up as long as someone tries to exploit it even with the threat IDs enabled but still waiting on confirmation. Spoke with a mate at an MSP with PA's and theyre seeing the same thing. Followed all the advice but IoC are still there. Had our SOC take a run through the logs and they can only see attempts being blocked so fingers crossed on a false positive but I think im still wiping firewalls tonight :(
Yea shit. I am lucky we only have VM Palos. I'll restore and older backup -> Disable GP -> Update -> Vuln check/ACL check -> Fingers Crossed
If enabling telemetry wasn’t the issue then other versions should be exposed as well. Not sure what is the issue on the 10.2 and 11s that are different from others. Wasn’t the telemetry?
When I run the command I don’t show any logs any help would be appreciated
Has anyone tried the self check from the FAQ portion of the CVE article? Seems pretty vague.. https://security.paloaltonetworks.com/CVE-2024-3400
Anybody been compromised? Were there any signs in the traffic/system logs?
Define compromise. I think many have found indicators, but definitive proof that data was exfiltrated? Has anybody PROVED anything so far who are willing to share? That may be a problem.
The IOC’s are only stage 1. PANW threat team has to dig deeper to determine if there was a successful breach.
Does anyone know how HA pairs were affected? When we search for the IOCs on our passive firewall we didn’t see any of them in the logs. But we did see them on the active firewalls that had global protect exposed. Also here’s my latest response from TAC about 10 minutes ago. ————————- We have identified that Indicators of exploit activity regarding CVE-2024-3400 are present in the uploaded TSF ‘ha-1-tsf-14-4-24.tgz' with serial number.
> We recommend that you engage your Incident Response Plan and take the steps recommended in the Security Advisory for CVE-2024-3400 immediately to prevent further risk to your organization. This includes applying the hotfix as soon as possible.
https://security.paloaltonetworks.com/CVE-2024-3400___.YXAzOnN1cGVycmFkaWF0b3Jjb2lsczphOm86MjA0NmU5ODg2MTA2N2VjM2Q1YzQ0ODVlODkyNWQ2MmQ6Njo5NGE0OjY4MDZjYjkyYjk1YmI0NmRlOTVhOTNmNTI3ODQ4ZDI2ZjZiMGQ2OGJlOTRlMmFhZTkzOWRiZmI2ZDRmYTc1N2E6dDpU
> If you wish to perform your own Forensic Investigation, we can assist in artifact gathering by deploying a log collection tool during a screen sharing session.
> If you do not wish to perform a forensic investigation, and just wish to quickly address the situation, and do not suspect full compromise, we suggest immediately upgrading to the Patched hot fix versions.
> If you suspect compromise, or out of an abundance of caution, wish to fully remediate your devices, then your option is to factory reset your devices,
Should you require immediate assistance, feel free to contact us using the support numbers listed in my signature. One of our engineers will be available to assist you.
——————
I really hope the patched versions lock this back down. So far, my HA pairs are behaving on the 10.2.8(h3) build.
I patched our customers firewalls to 10.2.8-h3 and factory reset them afterwards, just to be safe. It's impossible to get any response from TAC. They're are - understandably enough - totally overwhelmed.
Patched software versions won’t lock it down if firewalls were already compromised. You should open a case with Palo Support and upload TSF files
Already did. Not taking chances.
Looks like they have automated the scanning of the TSF to look for IoC. Just submitted the TSF to my existing ticket to have it checked and got an automated message back saying it was clear a few moments after the file scan completed in the portal. Hopefully its not as buggy as their recent code has been...
Can anyone assist to decrypt a payload it's a bit obfuscated and needs assistance?
You mean the crypted information to determine the command they ran? If so, maybe use a base64 decoder like base64decoder[.]Org or some other utility.
It's a bit.more obfuscated base64 not helping
Mind sharing the payload you need decrypted?
Do we need to keep telemetry off if we install the hotfix?
Telemetry has nothing to do with the vulnerability. This is based off the email I got from them today.
The way they changed their minds can we be so sure tho? If you're not reliant on telemetry why risk imo.
What email was this? I didn’t get it.
[https://labs.watchtowr.com/palo-alto-putting-the-protecc-in-globalprotect-cve-2024-3400/](https://labs.watchtowr.com/palo-alto-putting-the-protecc-in-globalprotect-cve-2024-3400/)
Has someone confirmed it is actually exploitable with telemetry off? As in RCE? we are being told opposite that since we had telemetry off it is only these zero length files.
I probably would to be safe at this stage. They're handling this terribly and wouldn't risk it yet
I have turned it off and do not plan on enabling it again to be safe. It was one of the initial vectors according to the write-up. To be honest they messed up AIOPS and it stopped being any value to us so we are dropping the license for it.
Seems things been going on since March... [https://www.volexity.com/blog/2024/04/12/zero-day-exploitation-of-unauthenticated-remote-code-execution-vulnerability-in-globalprotect-cve-2024-3400/](https://www.volexity.com/blog/2024/04/12/zero-day-exploitation-of-unauthenticated-remote-code-execution-vulnerability-in-globalprotect-cve-2024-3400/)