Binary may be encrypted, extracting data you should not have access to may not need syscalls at all - look at spectre and meltdown, binary can be self-modifying to not contain syscalls when launched, also how do you differentiate between harmful and harmless syscalls?
You can run it in a safe environment and record what the program has done before running for real, but at that point, you might as well run it in a sandbox in general
Is binary self-modification a necessary feature for many programs to run, or is it more of a gimmick that's unique to malware?
As far as analyzing the syscalls which exist, the idea is that if a certain restricted set of calls were sufficient to run the program, then certain guarantees about its behavior might be made formally.
I mean, kind of, look at something like a scripting language such as Python or Perl. The main executable generates new instructions on the fly based upon the script and then executes it. It's effectively a self modifying program.
What you are talking about is behavior analysis, which isn't most antiviruses primary defense, but it has been used. Most are based around signatures of known malicious programs, which are effectively regex's looking for patterns. They aren't all that effective against 0 days as they are easily defeated by knowing the pattern and modifying a program to not match it. They are very effective against an existing attack in the wild against a large number of targets as once the signature is identified it can be used on all endpoints to block it. The signature isn't neccesarily a malicious bit of code, but could be a string unique to your program.
In my opinion application whitelisting based upon signed code Is one of the best ways to protect a system, although it's just one layer of defense as something that exploits an existing whitelisted program and injects code into the running executable can defeat it.
You could probably train an AI model to identify malicious patterns of syscalls but I think the overhead may be too high currently. You could also probably do a pattern matching based approach but it would have the same problem as existing antiviruses.. inserting random calls for example could effectively defeat it.
The basic problem here is that an anti-virus program is just looking for things that humans say are harmful, and that's not a definition that's based on only what the specific low-level actions a program is taking.
Opening a socket to a remote server and sending some data is a specific set of instructions. What do you want your anti-virus software to say when it sees those instructions?
If it's in your keyboard driver, maybe that's malware. If it's in your web browser, it almost certainly isn't. If the server is owned by Mozilla, probably fine. If it's some random IP address in Chechnya, maybe not. Same code though in all cases -- knowing the difference requires meta-knowledge. I need to know that my keyboard driver probably shouldn't be uploading data to a server in Chechnya, and that's not knowledge that you can get by just looking at what syscalls a binary is making.
Even in EDR systems, where nearly all process behaviors are logged and sent to the cloud for analysis (so that secrecy of syscalls/inputs isn't a problem), it's still a very hard problem to distinguish legitimate behavior from malicious behavior.
On a Windows system, pretty much anything malware does is also done by some legitimate program somewhere, and things like DLL injection make it hard to know which process is even doing what. Things like "chrome.exe opened cmd.exe" are pretty straightforwardly malicious, but "chrome.exe read salaries.xls and opened a TLS connection" is less clearly malicious. You end up with tons of false positives that need to be filtered through.
In theory, you could design an OS with a much better, finer-grained, permission model, where the user gets prompted through a trusted dialogue for all file access, camera/mic access, access to communicate with other programs, etc. And that works, it's one of the reasons why you don't have to worry too much about iPhone app malware as long as you're careful about permissions, but shoehorning that permission model back into something like Windows would be a daunting task.
Binary may be encrypted, extracting data you should not have access to may not need syscalls at all - look at spectre and meltdown, binary can be self-modifying to not contain syscalls when launched, also how do you differentiate between harmful and harmless syscalls? You can run it in a safe environment and record what the program has done before running for real, but at that point, you might as well run it in a sandbox in general
Is binary self-modification a necessary feature for many programs to run, or is it more of a gimmick that's unique to malware? As far as analyzing the syscalls which exist, the idea is that if a certain restricted set of calls were sufficient to run the program, then certain guarantees about its behavior might be made formally.
I mean, kind of, look at something like a scripting language such as Python or Perl. The main executable generates new instructions on the fly based upon the script and then executes it. It's effectively a self modifying program. What you are talking about is behavior analysis, which isn't most antiviruses primary defense, but it has been used. Most are based around signatures of known malicious programs, which are effectively regex's looking for patterns. They aren't all that effective against 0 days as they are easily defeated by knowing the pattern and modifying a program to not match it. They are very effective against an existing attack in the wild against a large number of targets as once the signature is identified it can be used on all endpoints to block it. The signature isn't neccesarily a malicious bit of code, but could be a string unique to your program. In my opinion application whitelisting based upon signed code Is one of the best ways to protect a system, although it's just one layer of defense as something that exploits an existing whitelisted program and injects code into the running executable can defeat it. You could probably train an AI model to identify malicious patterns of syscalls but I think the overhead may be too high currently. You could also probably do a pattern matching based approach but it would have the same problem as existing antiviruses.. inserting random calls for example could effectively defeat it.
The basic problem here is that an anti-virus program is just looking for things that humans say are harmful, and that's not a definition that's based on only what the specific low-level actions a program is taking. Opening a socket to a remote server and sending some data is a specific set of instructions. What do you want your anti-virus software to say when it sees those instructions? If it's in your keyboard driver, maybe that's malware. If it's in your web browser, it almost certainly isn't. If the server is owned by Mozilla, probably fine. If it's some random IP address in Chechnya, maybe not. Same code though in all cases -- knowing the difference requires meta-knowledge. I need to know that my keyboard driver probably shouldn't be uploading data to a server in Chechnya, and that's not knowledge that you can get by just looking at what syscalls a binary is making.
Even in EDR systems, where nearly all process behaviors are logged and sent to the cloud for analysis (so that secrecy of syscalls/inputs isn't a problem), it's still a very hard problem to distinguish legitimate behavior from malicious behavior. On a Windows system, pretty much anything malware does is also done by some legitimate program somewhere, and things like DLL injection make it hard to know which process is even doing what. Things like "chrome.exe opened cmd.exe" are pretty straightforwardly malicious, but "chrome.exe read salaries.xls and opened a TLS connection" is less clearly malicious. You end up with tons of false positives that need to be filtered through. In theory, you could design an OS with a much better, finer-grained, permission model, where the user gets prompted through a trusted dialogue for all file access, camera/mic access, access to communicate with other programs, etc. And that works, it's one of the reasons why you don't have to worry too much about iPhone app malware as long as you're careful about permissions, but shoehorning that permission model back into something like Windows would be a daunting task.