Threat Hunting. Why might you need it
The classic trigger-based approach, where cybersecurity specialist react to security alerts, cannot cope with the new challenges effectively and should be reinforced with Threat Hunting practices.
Threat Hunting has already proven itself to be very effective.
According to the FireEye M-Trends annual reports, the Dwell Time, that measures the median time between the compromise of an environment and the threat being detected, has been reducing in the last 3 years. In 2017, this metric stood at 101 days, in 2018, it was at 78 days, and in 2019, it dropped to 56 days. FireEye attributes the reduction in Dwell Time to two major factors: the continuous improvement of monitoring procedures and tools, and the growth in the number of incidents involving ransomware and cryptocurrency miners which are, by their destructive nature, easily detectable. There is no doubt that the evolution of such disciplines as Threat Intelligence and Threat Hunting, and the increased focus on endpoint monitoring have also contributed to the improvement.
According to the SANS 2019 Threat Hunting survey, around 70% of the respondents ascribe the reduction in Dwell Time to the implementation of Threat Hunting at their organizations. The survey also highlights that companies have not yet realized the full range of advantages and the scope of potential offered by Threat Hunting.
With Threat Hunting being a new trend, the definition as well as the practice itself lack overall consensus. This results in multiple interpretations of the term and disagreement on the benefits of this practice. For some, Threat Hunting is a process inherent to cybersecurity programs, while for others, this is another term coined by marketers to ignite demand for new cybersecurity solutions and services.
What Is Threat Hunting?
The term Threat Hunting was first used in the 2011 summer issue of Information Security Magazine, in an article titled "Become a Hunter" by Richard Bejtlich, who then headed General Electric CERT. Notably, he wrote, "To best counter targeted attacks, one must conduct counter-threat operations (CTOps). In other words, defenders must actively hunt intruders in their company." He also recalled that in the mid-2000s, the US Air Force used the term “hunter-killer” during the training that involved attack simulation on their networks.
Nowadays, the term Threat Hunting is used to denote a process of proactive and iterative analysis of telemetry gathered from endpoints and network sensors (such as IDS/IPS) to detect threats that evade traditional preventive security solutions.
The word "proactive" is key in this definition. Threat Hunters don’t just sit at their security consoles waiting for alerts to go off or a threat to be detected by the SIEM correlation rules. Instead, they rely on their entire experience and knowledge to search through telemetry data that is continuously collected from endpoints and network sensors in order to spot any suspicious activity. Aside from that, Threat Hunters are also active consumers of Threat Intelligence products which are used to explore the so-called tactics, techniques and procedures (TTPs) used by attackers. Having obtained intelligence about a new intrusion technique, a Threat Hunter creates a hypothesis of how this technique may be exploited to attack their infrastructure. The Threat Hunter also tries to profile known indicators based on the available telemetry. If the initial hypothesis fails, it is modified and retested , which is the essence of "iteration" mentioned above. In other words, the testing of hypotheses is a continuous process. As Threat Hunters get more information about new intrusion tactics and techniques, they use it in an on-going cycle of analyzing telemetry data.
We can now delve deeper and look at the essentials of integrating Threat Hunting with general corporate security practices.
What Is Required for Threat Hunting?
Let us assume that you have decided to upgrade your traditional alert-based monitoring practices with a proactive threat search—this is the implementation of Threat Hunting in a nutshell. The basic resources you will need for this are below.
People appear to be the most important resource. You will hardly succeed without a professional team of Threat Hunters.
You will need to develop and train your staff—this is crucial. Threat Hunting requires profound technical expertise in a variety of disciplines:
- network and OS security
- operating system’s internals
- cybersecurity incident response
- threat intelligence
- host and network forensics
Threat Hunters must also possess critical thinking, an analytical mindset, curiosity, perseverance and attention to detail. They should always concern themselves with thorough examination of any given matter.
However, as the market lacks professionals with the required qualifications, you should not necessarily hire a team of prepared experts. Instead, you can find an expert and leader who will assemble and educate a team.
To be able to hunt for threats in your infrastructure, you will need telemetry, i.e., host and network.
Table 1 shows information about endpoint telemetry that can be used in Threat Hunting activities:
Table 1. Host telemetry| Type of Telemetry | Description | Possible Telemetry Sources |
|---|---|---|
| Process activity events | Contains events generated as a result of periodic execution of various utilities or periodic requests to standard Windows OS management interfaces (e.g., WMI) and comparing the current and previous execution results. This data collection method can be used where certain reasons prevent an agent from being deployed for continuous system monitoring. Examples of collected data:
|
|
| RAM scanning | Contains events generated as a result of periodic memory scans of the executed processes and detection of anomalies that potentially indicate the presence of malicious code in the memory or unauthorized modification of OS critical structures and objects. RAM scanning enables for the detection of the most sophisticated threats posed by fileless malware, when the intruder carries out their malicious activity on RAM. However, this type of telemetry is only available in advanced EDR solutions. Unfortunately, free solutions such as Sysmon or standard operating system audit tend to be helpless in the case of fileless malware. The best option would be commercial EDR solutions with an embedded RAM scanning module. Examples of events that are included in this type of telemetry:
| EDR |
| Inventory of autostart extensibility points | Contains events generated as a result of periodic scanning of known OS autorun points (the so-called ASEP—Autostart Extensibility Points). Inventory of autorun points is an effective data source in Threat Hunting, an easy path to victory, since most malware and attackers try to gain their foothold in a compromised system to survive a reboot. Therefore, analyzing autostart extensibility points is one of the quickest ways to identify previous system compromises. This type of telemetry includes the following events:
Certain sources provide this information as a single event (combining files metadata and a command line) for every object detected in the autorun, while others present these events separately. |
|
| OS events | Contains events generated by standard OS audit capabilities. It is a perfect supplement for process telemetry data, as many techniques can only be detected by OS events. Events may vary depending on the OS. For example, the following events identifiable by the Windows OS family can be potentially interesting from the Threat Hunting perspective (the list is not exhaustive—the Windows OS has a large number of interesting events across multiple logs, which are beyond the scope of this article):
| Standard OS capabilities |
| Listing of the directories that can be used by an adversary or malware to store their executable files | Contains events generated as a result of periodic indexing of directories content (e.g., Windows, Temp, ProgramData). Such scanning results in a separate event for each file detected in the directory, according to the filter settings. Alongside with the presence of an executable file or script in the respective directory, the event must also contain the files’ metadata (hashes, creation and modification dates, file system attributes, signature data, etc.). The listing of certain directories along with the inventory of autorun points allows to identify past incidents that have left file artifacts and are currently inactive. |
|
| Periodic snapshot of the results of various utilities | Contains events generated as a result of periodic execution of various utilities or periodic requests to standard Windows OS management interfaces (e.g., WMI) and comparing the current and previous execution results. This data collection method can be used where certain reasons prevent an agent from being deployed for continuous system monitoring. Examples of collected data:
|
|
Network telemetry is no less important in Threat Hunting than host telemetry. Details are presented in Table 2 below:
Table 2. Network telemetry| Type of Telemetry | Description | Possible Telemetry Sources |
|---|---|---|
|
Metadata about downloaded files |
Malware files downloaded from the Internet is a popular vector for the initial delivery of malicious code onto protected environments. Moreover, a compromised host can download its modules continuously during operations. Metadata sourced from telemetry (hash, file type, source of download and HTTP attributes) about all files downloaded from the web can be of great help both in investigations and threat detection as part of Threat Hunting practices (such as the detection of executable files downloaded by servers from unknown resources or downloaded files whose hashes are found in Threat Intelligence feeds). |
|
|
HTTP activity metadata |
Information about resources accessible from the company network. This type of telemetry can be used to detect malware code communications with command centers, Drive-By attacks, or attempted data exfiltration via HTTP. |
|
|
SSL/TLS traffic metadata |
If your network sensor can provide TLS/SLL traffic telemetry, this can perfectly supplement your Threat Hunting software. For instance, this type of telemetry can be used to:
|
|
|
DNS traffic metadata |
This is a traditional source used for security monitoring purposes. It can be incorporated into Threat Hunting practices to identify:
|
|
|
LDAP traffic metadata |
LDAP is often used by attackers either to obtain information about the compromised domain infrastructure (e.g., using the popular BloodHound tool) or to prepare for Roasting attacks (AS-REP Roasting, Kerberoasting). Furthermore, the attacker can exploit LDAP information to brute-force passwords for domain accounts. Equipped with telemetry capabilities that enable the collection of LDAP traffic metadata, a Threat Hunter can detect all of the threats mentioned above. |
|
|
Kerberos traffic metadata |
Kerberos is a standard authentication protocol in domain networks. Today, a variety of attacks on this protocol in Windows have been identified (Pass-the-Ticket, Golden Ticket, Silver Ticket, attacks on delegation mechanisms). These attacks can be partially detected with the help of Kerberos traffic metadata. |
|
|
SMB/DCE RPC traffic metadata |
SMB/DCE RPC protocols are transports for various service features of the Windows OS. The collection of metadata of such protocols can help to discover a wide range of post exploitation techniques within the Windows environment:
|
|
|
Metadata about files transmitted within the SMB network |
The SMB protocol serves as a transport for various service features of the Windows OS. In addition to interacting with all types of control interfaces, attackers can use this protocol to transmit files within the compromised network. Metadata about these files would be helpful in detection of anomalies caused by an attacker’s activities in the environment (such as the transmission of executable files between workstations, or the transmission of Windows registry hive dumps). |
|
|
Email metadata, information about emailed files and links |
Email is the most popular channel used to deliver malware content, whether as enclosed files or as links the recipient is prompted to click on. Having a source of events that along with email headers provides information about files (hash, type, size and in certain cases the result of YARA scanning) and links from emails correlated with metadata from email headers and attached content, would be a powerful tool enabling Threat Hunters to detect and investigate incidents involving email as a initial compromise vector. |
|
|
NetFlow |
Although not the most valuable source of telemetry as applied to Threat Hunting, NetFlow data can be quite useful in certain cases. Below are examples of incidents that can be detected with the help of NetFlow:
|
|
The most effective Threat Hunting framework can be built by combining host and network telemetry, enriched with Threat Intelligence from multiple sources. This will ensure maximum coverage at all stages of potential attacks. However, this requires substantial investments both in the infrastructure and Threat Hunting analysts who possess sufficient qualifications to process the heterogeneous data flow.
Your Threat Hunting platform can utilize SIEM as the underlying technology to collect, categorize and correlate security events. SIEM features allow a quick search through the collected data, user-friendly visualization, incident investigation and response.
However, it should be kept in mind that your tools will have to be adapted by your Threat Hunters. Eventually, the best tools will be either created or improved by your people. For example, you can build a robust Threat Hunting platform based on the ELK stack (Elasticsearch, Logstash, Kibana). If you have a qualified DevOps engineer and a professional capable of developing an event enrichment pipeline, you can create a high-quality Threat Hunting tool from open sources, which is no worse and may even surpass commercial solutions. The internet is abundant with resources offering quality information on how to create a Threat Hunting platform powered by ELK stack, here are a few references: click here or click here.
Threat Hunters use tools and collected telemetry data to proactively search for threats in their environment. However, to ensure the best performance and maximum business value, Threat Hunting practices should be implemented together with or after the associated processes, i.e., Threat Intelligence and Incident Response.
Below we provide an example to illustrate our statement.
Suppose a Threat Hunting exercise uncovered indicators of malware activity in the internal segment of your company network. This means that a privileged account has been compromised. Are you ready for this scenario? What is your next step? In other words, do you have a robust incident response process in place, which enables the correct processing of data obtained through a Threat Hunting exercise?
If there is no such process in place, the collected data will be processed ad-hoc or will remain unprocessed. And then the question arises: Why invest resources into expensive technologies (EDR, Threat Hunting platform) and specialists, where their findings have no practical application? Therefore, a robust incident response process is essential for the successful implementation of Threat Hunting practices.
Now let us take a brief look at the role of Threat Intelligence in Threat Hunting practices. To generate new hypotheses, Threat Hunters should always be studying the techniques and tools used by attackers. This requires a structured process that involves continuous analysis of information about new threats/attacks/tools from trusted sources (cybersecurity product vendors, renowned researchers, lectures at specialized conferences, etc.). Without an established process, Threat Hunting methods can still be applied, however, your Threat Hunting activities will be chaotic and unpredictable.
Threat Hunting Approaches
Once you have put together your team and quipped them with the necessary tools, telemetry data and at least somewhat streamlined processes, you are prepared to start a proactive search for threats in your environment. Today, Threat Hunters employ several approaches to Threat Hunting depending on their expertise, available telemetry and tools. To gain a better understanding of the advantages and disadvantages of different approaches, we have to look at the fundamental concept introduced by David J Bianco in 2013. Cleverly dubbed "the Pyramid of Pain", it illustrated the break down of indicators according to their severity:
Picture 1. Pyramid of Pain
The levels within the pyramid represent different types of indicators of compromise that can be used to detect malicious activity in your environment. The width of each level reflects the number of possible indicators of this type, whilst its colour and position show how much hassle or "pain" it may cause an attacker when their indicators are detected and blocked.
For instance, blocking IP addresses or C2 domain names will cause minor irritations to an attacker since IP addresses can be easily changed and new domains can be registered. However, blocking specific utilities can cause the attacker much greater inconvenience, as they will have to reinvent their tool or develop or purchase a new one in order to evade this protection, which is rather expensive and annoying. Furthermore, the tactics, techniques and procedures (TTPs) employed by an adversary can be identified with the help of detection rules. This will cause greatest pain to the intruder and they will have to either change their techniques, which may be difficult and even impossible, or simply retreat.
Now that you understand the concept behind the Pyramid of Pain, let us turn our attention to the approaches that are currently applied in Threat Hunting:
- IoC-based: the first four levels of the Pyramid (from the base up). Malicious activity is identified through the application of specific indicators. These could be file hashes corresponding to specific hacker tools such as Mimikatz or Empire, or a domain name specific to the attacker’s control server. Less volatile indicators would be specific registry keys, changing malware or User-Agent strings.
- Tools-based: the next level of the Pyramid. This approach involves the detection of identifiers associated with specific tools, e.g., command lines that can be obtained from process execution events or VERSIONINFO attributes of executable files supplied by EDR solutions such as Sysmon, as referred to above.
- TTPs-based: the top of the Pyramid represents tactics, techniques and procedures. Let us assume that the attacker applies a remote code execution technique Service Execution (T1035) to enable lateral movement and get a foothold in the system. An experienced Threat Hunter will explore this technique and determine what telemetry data they might need to develop a detection rule. Once ready, the Threat Hunter applies the respective tools to proactively detect this technique being used in the protected environment.
The most advanced Threat Hunters tend to use the last two approaches. How does it work in practice?
Let us examine a brief scenario. Analysis of the latest Threat Intelligence report reveals that companies operating in your industry are being attacked by a criminal group that is making ample use of remote code execution techniques, namely, hacker utilities from the Impacket toolkit. According to the short description on the Impacket Git repository, this "is a collection of Python classes for working with network protocols. Impacket is focused on providing low-level programmatic access to the packets and for some protocols (e.g., SMB1-3 and MSRPC) the protocol implementation itself."
We will explore the different approaches to Threat Hunting, drawing on the example of remote code execution utilities from the Impacket toolkit: psexec, smbexec and wmiexec.
When applying the IoC-based approach, a Threat Hunter can identify the indicators of compromise by downloading the executable files and scripts from the Git repository and calculating their hash sums:
Table 3. Host indicators of compromise for the psexec, smbexec and wmiexec utilities| Tools | MD5 | SHA1 | SHA256 |
|---|---|---|---|
|
psexec.py |
2773C4094DACED9F193C3B310B6CC287 | 1CB2D13297C7A82DEF2A8408CEAB05FD9A25A5CA | 7369300FBEDF4F3440A01F4439D39C681B5D9BDBA91177A356E7F68FF4E9CD09 |
|
smbexec.py |
DC8094A0E2A5AA30677C4D6B31523356 | A6265D68F7AB1FACED6B0652E2D4C189AD0DE2B8 | EE44915454BB006C9BAAC1EE270BD6430EAF6430E3F1C34FA595F68F88007918 |
|
wmiexec.py |
15CF3D5B72D037EAE9D1CE18F9179B4B | 81D9A370D3C1C64E1F9C0DCDABBB241A7C1EF20F | BE1FE5F978D21367E3654883035391C9608068C0479D309E1F6C20EC842D735E |
The identified indicators can be incorporated into the Sysmon rule and uploaded to the IOC scanner or to your own tool. We can also utilize the event FileCreate (EventID 11) Sysmon and write a rule for the creation of files named smbexec, psexec or wmiexec for Windows OS workstations.
This approach is easy to implement and is quite reliable. However, in this case the attacker can easily evade our detection rules. Utilities may be given trivial names and modifying just a single byte may result in a different hash sum of the files. Besides, you will hardly spot these files on your workstations as they are more likely to be executed from a remote host beyond your control. Nevertheless, this does not mean that the approach is impractical. It can complement other approaches, especially when responding to incidents, as it allows for quick detection and removal of attack artifacts. However, on its own, this approach is not very effective in Threat Hunting practices.
Climbing higher up the Pyramid of Pain, we come to the tools-based approach. To detect the tools used in an attack, a Threat Hunter generally examines their source code (if it is open access) and simulates attacks in a test environment. They further analyze the network and host telemetry gathered during the simulation to figure out specific attributes of the utilities. These include specific signatures in the network traffic and events on the target host.
Table 4 and Table 5 summarize the tools-based indicators of the smbexec and wmiexec utilities:
Table 4. Specific attributes of the smbexec utility| Source Code Snippet | Host Indicators | Network Indicators |
|---|---|---|
|
|
Process-specific command lines: C:\Windows\system32\cmd.exe /Q /c echo cd ^> \\127.0.0.1\C$\__output 2^>^&1 > C:\Windows\TEMP\execute.bat & C:\Windows\system32\cmd.exe /Q /c C:\Windows\TEMP\execute.bat & del C:\Windows\TEMP\execute.bat |
A specific name of a remotely created service: content: “B|00|T|00|O|00|B|00|T|00|O” A specific command line for the created service (lpBinaryPathName): content: “%|00|C|00|O|00|M|00|S|00|P|00|E|00|C|00|%|00| |00|/|00|Q|00| |00|/|00|c|00| |00|e|00|c|00|h|00|o|00| ”; A specific host name used as an argument of function ROpenSCManager: content: “D|00|U|00|M|00|M|00|Y” |
Table 5. Specific attributes of the wmiexec utility
| Source Code Snippet | Host Indicators | Network Indicators |
|---|---|---|
|
|
Process-specific command lines:
cmd.exe /Q /c cmd.exe 1> \\127.0.0.1\ADMIN$\__1565402391.05
2>&1
|
The use of a temporary file with a specific name beginning with “__” and unix timestamp to transmit, over the network, the content of input-output buffers of a command executed on a remote host: content: “|05 00|”; distance: 8; within: 2; content: “_|00|_|00|1|00|”; distance: 106; within: 6; content: “|00|.|00|” |
The identified indicators can be used to create correlation rules and intrusion detection signatures to discover and remove the smbexec and wmiexec utilities. Searching through historical data, Threat Hunters can test their hypothesis regarding the absence/presence of indicators of these utilities in the company infrastructure.
As we can see, the smbexec and wmiexec utilities leave a lot of specific indicators that enable their detection. However, an advanced attacker can easily modify the source code of the smbexec and wmiexec utilities, bypassing tool-based detection. This is when Threat Hunters rely on the TTPs-based approach.
The TTPs-based approach relies on identifying behavioural patterns specific to the attacker’s techniques or tools. Let us explore how the psexec utility behaves and try to identify such patterns.
After connecting via the SMB protocol, the utility copies a payload file to a remote host and creates a file execution service. The utility communicates through named pipes and the payload file is eventually deleted.
Based on the described utility behaviour, we can generate a few hypotheses that may help for its detection (Table 6).
Table 6. Hypotheses for TTPs Detection| psexec.py Utility Behaviour | Hypotheses for TTP Detection (based on host telemetry) | Hypotheses for TTP Detection (based on network telemetry) |
|---|---|---|
|
Connects to the remote host via the SMB protocol, copies the payload file and creates a file execution service with a randomly generated name. |
|
|
|
Uses a named pipe mechanism to communicate with the compromised host |
|
|
|
After the command is successfully executed, removes the service and its executable file |
|
|
As you can see, our hypotheses are not linked to the utility’s signature attributes. We keep the focus primarily on its behaviour. The key advantage of this approach is the ability to spot malicious activity wherein the attacker keeps changing tools. It should however be noted that the TTPs-based approach is the least accurate of the lot. Such practice could potentially generate a lot of false positives, and hence requires highly qualified analysts to do triage.
Where Do Threat Hunters Get Ideas for Hypotheses?
Threat Hunting starts with an idea and a hypothesis. Ideas can be derived from the following sources:
- MITRE ATT&CK matrix: a vast knowledge base of attack tactics, techniques and procedures. Studying the MITRE techniques and their simulation in test environments can serve as a foundation for the development of hypotheses.
- Threat Intelligence reports: contain a lot of useful information about attack techniques and procedures based on real incidents. Periodic analysis of such reports should spark some thought and give rise to a plethora of Threat Hunting ideas.
- Blogs, Twitter, and conference talks: cybersecurity researchers and experts are eager to share the results of their researches. It is there the information about a new attack technique appears for the first time, even before the attackers start actively using it. The timely study of such information will allow Threat Hunters to be proactive and prepare before the new attack technique becomes widespread.
- Incident response and investigation practices: once you have gained some hands-on experience in the field, you will get a holistic and in-depth understanding of the TTPs used by attackers.
- Practicing penetration testing: attackers tend to use tools similar to those applied by experienced pentesters. Therefore, studying pentesting practices creates a treasure trove of knowledge for generating Threat Hunting hypotheses. If you have an in-house team of penetration testers, it would be worthwhile cooperating with them. This will be a mutually beneficial process, i.e., Threat Hunters will gain valuable knowledge about the tools and techniques used by attackers, while penetration testers will learn how response teams can detect their activity.
A practical example demonstrates how analyzing Threat Intelligence reports can help Threat Hunters arrive at their hypotheses.
One of the most recent Zscaler reports describes the activity of the Ursnif banking trojan, also known as Gozi or Dreambot. Ursnif uses phishing emails with malware attachments as the initial point of entry. Obfuscated multi-trojan payloads enable effective evasion of standard defences.
We have selected a few techniques used by Ursnif to generate our Threat Hunting hypotheses. Table 7 summarizes the results with the techniques mapped on the MITRE matrix, and shows the required telemetry data.
Table 7. Threat Hunting hypotheses based on Ursnif trojan activity analysis report| Report Extracts | MITRE Techniques | TTP Detection Hypotheses | Detection Telemetry |
|---|---|---|---|
|
“As mentioned earlier, the malware’s initial payload was being delivered via document files with the name info_03_24.doc during the time of our analysis. The document didn’t contain any exploits but used a macro code to drop the second-stage payload.” |
Spearphishing attachment (T1193) User Execution (T1204) Scripting (T1064) |
|
|
|
“Copy the mshta.exe code to microsoft.com (possibly to reduce footprints). Write the second stage payload to index.html. Execute the index.html with microsoft.com.” |
Mshta (T1170) Masquerading (T1036) |
|
|
|
“Get the path %temp% and append index.dll (the final payload filename) to it. Execute the downloaded DLL using regsvr32.” |
Regsvr32 (T1117) |
|
|
|
“Command and control communication with newly registered campaign domains” |
Standard Application Layer Protocol (T1071) |
|
|
These hypotheses can be used to create correlation rules and/or conduct proactive hunt for threats on the network and, also, demonstrate how the TTPs-based approach can be applied in Threat Hunting.
Conclusion
Alas, one article is not enough to cover a topic as broad as Threat Hunting. Hopefully this publication has helped you in gaining an understanding of the basics of this process and its value.
We recommend that you combine different approaches to Threat Hunting, while staying focused on the TTPs-based approach. Always keep in mind that people are the key links between processes. The right team equipped with the right tools and telemetry is the cornerstone of success. Happy Hunting!