When the term “threat hunting” is brought up in the cyber security community, it can come across as more of a buzzword than a viable and important strategy for organizations to adopt. While there is plenty of discussion about what threat hunting means and why having a hunt program is important, the mindset, methods, and key steps for executing adversary threat hunting are sometimes overlooked.
In this blog, we dig a little deeper into the early stages of a hunt operation from our webinar, “6 Lessons Learned Hunting Advanced Criminals.” Here’s an excerpt where we discuss important steps for fostering a hunt mindset and beginning the incident response and investigation.
One of the more exciting things about working in a company that handles incident response cases is that you never know how your day’s going to go. Often, however, it starts with a panicked call from a customer.
Establishing a Hunt Frame of Mind
It was Friday evening when I got a call from an IT Manager we had worked with previously. He had been investigating a CPU spike on a system that he described as, “normally dead as a doorknob.” Not everyone would stay late on a Friday to dive into something like this, but this particular IT Manager was a Threat Hunter. He had experienced a significant incident about a year prior and, as a result, was on constant alert. He didn’t make any assumptions when it came to odd behaviors on his network, and the risk of another compromise drove his decisions.
As we dug deeper, his fears were validated. Looking through the Windows event logs, we noticed odd and randomly named services in scheduled tasks. The IT Manager sent over some artifacts and I began my analysis. It didn’t take long for me to recognize the threat: PowerShell running an obfuscated version of Meterpreter, a favorite exploitation framework for criminals and penetration testers alike. Importantly, we were able to find a hard-coded call-back IP address. A quick look at the firewall showed that about 20 systems that day had been communicating with the IP address. The metaphorical alarm bells were ringing.
We had a full-blown incident on our hands. That first night ended with isolating the affected hosts, blocking the IP address, and collecting forensics artifacts from those 20 systems. The customer also started instrumenting their network in preparation for an ongoing investigation and response effort.
What’s the lesson from this story? Foster a hunt mindset and assume your network is breached. You can’t necessarily trust what’s in your network, and you shouldn’t dismiss those anomalies happening within your network. In fact, you should always be suspicious of them. Stay on alert. Fostering a hunt mindset at all levels of your organization will help detect incident sooner, or sometimes find threats you may have never caught.
Planning the Investigation
Once your threat hunting endeavors yield results, you need to respond. Investigating an incident starts with collecting historical data. For instance, for this incident, we did some deep-dive technical analysis and collection of the endpoints that we knew were affected, including the ones that showed up in the firewall logs that we discovered that first day.
From there, our investigation identified the totality of other artifacts and witness devices that were available to answer our pressing questions. This meant going through the list of other systems and infrastructure devices that might have some trace of the malicious activity. We thought through what information we might gain from examining systems like domain controllers, security appliances, routers, web-proxies, and more. We used all of this information to track the lateral movement of the incident backwards.
There’s a whole slew of technologies, skills, and tools you need to craft an investigation. It’s not just collecting all your Windows events and all your firewall logs and putting them on a timeline. While that’s important, there are a lot of specialized skills and techniques that you can’t expect your IT guy to just turn around and perform incident response and forensics on the spot.
Train, Improvise, Adapt
You have to make sure that your organization is invested in training the IT staff and providing them with tools that they’re going to need for the investigation. While you can attempt to predict what you’re going to need for the investigation, you’ll also need to improvise, adapt, and overcome some new investigative requirements on the fly. If your team is trying to do that while also trying to acquire tools and go through a cram session on how to use those tools, you’re already one step behind.
You want to be able to focus on fighting the adversary and the incident, and not your own tools, and techniques, and skills. Planning for this beforehand is going to help you out. Recognizing that it’s difficult and specialized will make you more prepared for the future.
Check out the on-demand version of our webinar, “6 Lessons Learned Hunting Advanced Criminals” to learn more about the hunt operation lifecycle and how to approach each step.
Andrew Cook is a manager for incident response services at Delta Risk. Read more Delta Risk bogs here.
