A Practical Guide to Threat Hunting in Process Data
Introduction
This post aims to provide a core set of ideas for threat hunting — particularly in an intel-driven fashion. The intended audiences are detection engineers, threat hunters, and those aspiring to be one of the two.
It will also examine the traditional nomenclature of TTPs (Tactics, Techniques, and Procedures) and where time is spent hunting compared between the three.
Lastly, it will end with some smaller anecdotes and tips.
Caveats
- I cannot stress this enough — MITRE ATT&CK is not a checklist — they even said so themselves. What this means practically, for this post, is that 100% MITRE coverage does not mean you are “secure”. It means that you can contextualize hunts and detections in a kill chain (more on this later).
- To properly implement things discussed in this post, you will need process data (with command line), the ability to automate things (Python is recommended), and a VirusTotal Intelligence API key.
- I am not suggesting the methods in here are the best — or better than anything else — simply what has worked for me.
- This content focuses heavily on hunting process data. Other styles of hunting (Yara/RFC violations/long-tail analysis/etc.) are definitely valid however not in scope.
MITRE ATT&CK Context
This article will focus on Procedures rather than Techniques so I want to give some examples.
Take T1566.001 for example. The tactic is Initial Access, the technique is Phishing: Spearphishing Attachment, and there are a ton of procedures listed. If I told you to hunt for this technique, there are a lot of ways to do it because the procedures vary widely; the payloads in the procedures include, but are not limited to, Word documents, Excel sheets, and PDFs. All of these payloads behave slightly differently and will look different at the process level.
For this article, a procedure is any combination of programs, files, and/or arguments, that — when combined — achieves some technique. Here are a few spearphishing procedures:
- Excel (program) launching CMD/PowerShell (program)
- Word (program) contacting a remote server for a template (file & argument)
- Word (program) launching PowerShell (program) with “-enc” (argument)
Hunting “Known Bad” Procedures is Priority
Relying on the definition above, a known bad procedure is any combination of programs, files, and/or arguments that achieves some technique and has been documented to be used by a threat actor.
Let’s use an example:
The above is a series of commands that Recorded Future noted in an investigation into APT10 (Source: APT10 Targeted Norwegian MSP and US Companies in Sustained Campaign).
Looking at just the first command:
We can break this down into programs, arguments, and files:
- bitsadmin.exe (program)
- /transfer (argument)
- http (argument)
- ProgramData (file [path])
- Temp (file [path])
Now if you wanted to build a hunt for this specific procedure, it would look for any time bitsadmin.exe ran with all of the arguments and file [paths] seen in the command line details.
However, where this level of granularity provides value is in breaking out the hunts and looking for any combination of the above:
- bitsadmin.exe with “/transfer”
- “/transfer” with “http”
- bitsadmin.exe with “ProgramData”
- “http” with “Temp”
- etc.
By creating multiple smaller hunts, you still have a chance to catch the activity if APT10 changes their procedure, or if someone else uses a similar one.
Consider this, the goal of the procedure is likely Ingress Tool Transfer (downloading files) (see bitsadmin.exe transfer docs). So if APT10 alters their procedure — like by renaming bitsadmin.exe to svchost.exe — then the first hunt above won’t catch it but the next 3 will. Additionally, if they use a different binary to download it but still write to the same folder (and everything is supplied in the command line) then the third one will catch it.
That is the power of breaking a procedure down to its core artifacts and hunting for combinations of said artifacts. Not only will you catch the activity you were looking for but you will also catch slight variations. This way, you can be more confident in your ability to detect what you are interested in.
Not every potential combination of artifacts is a useful hunt. Some may produce unmanageable haystacks while others may simply produce irrelevant results. The trick is finding those combinations that provide coverage for more than one version of the procedure without bringing in too much noise. A easy way to do this is to focus on arguments (e.g. “/transfer” and “http”).
Have Reliable Sources of Intelligence
You need at least one source of known bad procedures that is regularly updated, trustworthy, and accurate. An easy answer here is MITRE ATT&CK.
My personal recommendation is to find a MITRE Group you want to hunt, using the MITRE Groups page, and open all of the references at the bottom of their listing. Take APT1 for example:
By reading through these, you are likely to gleam more technical data on their procedures and able to build hunts around them. I also walk through this in my post about hunting Lazarus Group.
Having a source of malware hashes specific to the threat actor and access to a VirusTotal Intelligence API key are crucial for scaling. You can extract “Processes Created” from VirusTotal’s Behavior information for a hash.
Take this process seen in a WannaCry hash: VirusTotal Link
It shows vssadmin.exe deleting shadow copies. This would be considered a known bad procedure for WannaCry since it accomplished the goal of MITRE Technique T1490 (Inhibit System Recovery). If we apply the same logic here as we did for bitsamdin.exe and break this down to its core artifacts then we can build a series of hunts for this procedure and its variations.
- Vssadmin.exe and “delete”
- “delete” and “shadows”
- Vssadmin.exe and “shadows”
- etc.
By having a reliable source of known bad procedures — either from hands-on-keyboard operations or malware — and the ability to automatically extract them (e.g. VirusTotal API) one can build an extensive library of hunts for demonstrably malicious activity and its variations.
Contextualizing With MITRE
If you map each procedure you develop a hunt for, and any relevant ones you already had, to it’s MITRE Technique and Tactic then you can use the ATT&CK Navigator to visualize the change in coverage. Take the following example where a threat actor likes to use PowerShell to run payloads, Scheduled Tasks and BITS Jobs for persistence.
If you let green be existing hunts and blue be new hunts, then you can see that prior to the most recent R&D cycle, you had hunts to catch the threat actor at Execution, Persistence, and Privilege Escalation and then the R&D cycle added hunts at Persistence and Defense Evasion.
Also, I am not claiming this method will catch the threat actor you are interested in; however, I am suggesting that creating multiple smaller hunts off one procedure using the method described above will increase you chances at catching them compared to creating one very specific hunt.
Know What You Are Paying For
If you take the total time spent on a single hunt/detection it can be broken down into 2 categories, R&D versus actually hunting through the results. Now generally speaking (again, for process data hunts) the more time spent in R&D means less time will be spent actually hunting it. This is because the more time spent in R&D tends to mean it is more specific. Looking at the hunts made above for bitsadmin.exe and vssadmin.exe, we spent our time making multiple hunts for known bad procedures which are unlikely to actually produce results unless suspicious activity occurs. Contrast that with just looking for all executions of bitsadmin.exe and vssadmin.exe; those take comparatively very little R&D but would take more time to review the results. This is roughly illustrated below as the “Temporal Costs of TTPs”.
What this chart represents is that the core components of TTPs (Tactics, Techniques, and Procedures) impose temporal costs at different stages of hunting and by extension serve slightly different functions.
Firstly, Procedures cost more in R&D but take less time to hunt. Secondly, Tactics are typically easier to R&D but take longer to hunt due to larger result sets. Lastly, Techniques typically sit in a sort of middle ground between the two.
You get the opportunity to choose where to pay that cost and what you choose will largely depend on what you care about more:
- Do you want to hunt in the unknown?
- Or, do you want to hunt for known bad?
One is not better than the other. That is the crux of this illustration. However, I do hope this at least gives you an additional (and valuable) consideration when writing hunts and detections.
Admittedly, I am sure most SOC analysts, hunters, and detection engineers intuitively know that the more specific you make your hunts (the more time you spend in R&D) the less results you will have. That’s just natural — the more specific your criteria, the less results fit that criteria. However, I have never seen it specifically called out and I find this framing of hunt to be useful in managing priorities. Am I concerned about all PowerShell abuse? Or just how APT X tends to abuse PowerShell? The answer will vary depending on multiple things, but the answer will also inform me of how my time is about to be spent and I get to choose what is most effective for the situation at hand.
Known Bad vs Unknown
When I started out in Blue Team work, I heard over and over that threat hunting was looking at the unknown. While I think the sentiment is nice, I never really found much utility in the statement. I would hear “you have to know what is normal to find the abnormal” and that was not a particularly encouraging statement because it implied I had to know a lot about a lot to even get started. Don’t get me wrong, it is a valid approach, I just did not find it very helpful.
So instead, I took the route of studying the known bad. I think I would modify the adage to something like, “you must be able to recognize the abnormal when you see it”.
I said earlier that hunting Tactics, Techniques, and Procedures serve different functions — this is where that comes in. Kind of by definition, to hunt the unknown (very broadly), you are also hunting a tactic. For example, say you are looking at all commands ran on a file server (tactic: Collection) — you are looking at the unknown. As you move over to a Technique like Data From Local System (T1005) you might focus on SCP and FTP commands — you are moving closer to known bad procedures. Finally, the last step would be looking for a variation of a known bad procedure you were interested in and sourced from a threat actor or malware sample. An example of this might be SCP commands that copy specific directories.
Sort Your Data
It sounds trivial but hands down the easiest way to make a long list of potentially suspicious commands more digestible to hunt is to sort them alphabetically.
At first, sorting alphabetically makes almost no sense. It seems as if sorting by user or device would be better. However, in hunting the unknown, sorting processes alphabetically tends to group similar processes together (whether by their process name, folder path, or arguments) allowing you to more easily identify outliers.
Sorting this way does not require you to actually know what the known good commands are doing, only to recognize common procedures and that those are likely benign; so focus on the others.
Yes, some threat actors are experts at hiding in the noise. Some will absolutely spend the time to craft a legitimate looking DLL and/or software package to blend in with really common procedures in the environment. However, every hunt you have that is a variation of their known procedures is another trip wire on their path. There are threat actors that are exceptionally hard to hunt in this manner (because they craft every procedure to blend in with the noise) but that appears to be the exception, not the rule; most tend to have a least a few procedures that can be distinguished from normal day-to-day operations. In those rare cases where the threat actor blends in well, hunting in the unknown and being more attentive to detail than normal is almost necessary.
You Probably Knew This Already
As touched on earlier, I fully recognize that most people who have experience in a SOC probably already know the things that have been discussed here — whether they were consciously aware of this knowledge or not. However, I do think it is still important to shed some light on some of the more fundamental parts of hunting, challenge some ideas, and introduce new ones that break this discipline down into learnable chunks.
At the core of all this is one principle — threat hunting is teachable — and it is not just learning common TTPs and how to detect them. It is sometimes referred to as “the art of threat hunting”. A threat hunter can absolutely be compared to an artist, because they can both have their preferred style. However, threat hunting itself is a skill, just like painting.
