Blog

Dentro del SOC

When Hallucinations Become Reality: An Exploration of AI Package Hallucination Attacks

Default blog imageDefault blog imageDefault blog imageDefault blog imageDefault blog imageDefault blog image
30
Oct 2023
30
Oct 2023
This blog discusses the plausible threat of malicious actors taking advantage of errors in generative AI tools, or AI “hallucinations”, to launch malicious packet attacks and how Darktrace’s suite of products might detect these attempts.

AI tools open doors for threat actors

On November 30, 2022, the free conversational language generation model ChatGPT was launched by OpenAI, an artificial intelligence (AI) research and development company. The launch of ChatGPT was the culmination of development ongoing since 2018 and represented the latest innovation in the ongoing generative AI boom and made the use of generative AI tools accessible to the general population for the first time.

ChatGPT is estimated to currently have at least 100 million users, and in August 2023 the site reached 1.43 billion visits [1]. Darktrace data indicated that, as of March 2023, 74% of active customer environments have employees using generative AI tools in the workplace [2].

However, with new tools come new opportunities for threat actors to exploit and use them maliciously, expanding their arsenal.

Much consideration has been given to mitigating the impacts of the increased linguistic complexity in social engineering and phishing attacks resulting from generative AI tool use, with Darktrace observing a 135% increase in ‘novel social engineering attacks’ across thousands of active Darktrace/Email™ customers from January to February 2023, corresponding with the widespread adoption of ChatGPT and its peers [3].

Less overall consideration, however, has been given to impacts stemming from errors intrinsic to generative AI tools. One of these errors is AI hallucinations.

What is an AI hallucination?

AI “hallucination” is a term which refers to the predictive elements of generative AI and LLMs’ AI model gives an unexpected or factually incorrect response which does not align with its machine learning training data [4]. This differs from regular and intended behavior for an AI model, which should provide a response based on the data it was trained upon.  

Why are AI hallucinations a problem?

Despite the term indicating it might be a rare phenomenon, hallucinations are far more likely than accurate or factual results as the AI models used in LLMs are merely predictive and focus on the most probable text or outcome, rather than factual accuracy.

Given the widespread use of generative AI tools in the workplace employees are becoming significantly more likely to encounter an AI hallucination. Furthermore, if these fabricated hallucination responses are taken at face value, they could cause significant issues for an organization.

Use of generative AI in software development

Software developers may use generative AI for recommendations on how to optimize their scripts or code, or to find packages to import into their code for various uses. Software developers may ask LLMs for recommendations on specific pieces of code or how to solve a specific problem, which will likely lead to a third-party package. It is possible that packages recommended by generative AI tools could represent AI hallucinations and the packages may not have been published, or, more accurately, the packages may not have been published prior to the date at which the training data for the model halts. If these hallucinations result in common suggestions of a non-existent package, and the developer copies the code snippet wholesale, this may leave the exchanges vulnerable to attack.

Research conducted by Vulcan revealed the prevalence of AI hallucinations when ChatGPT is asked questions related to coding. After sourcing a sample of commonly asked coding questions from Stack Overflow, a question-and-answer website for programmers, researchers queried ChatGPT (in the context of Node.js and Python) and reviewed its responses. In 20% of the responses provided by ChatGPT pertaining to Node.js at least one un-published package was included, whilst the figure sat at around 35% for Python [4].

Hallucinations can be unpredictable, but would-be attackers are able to find packages to create by asking generative AI tools generic questions and checking whether the suggested packages exist already. As such, attacks using this vector are unlikely to target specific organizations, instead posing more of a widespread threat to users of generative AI tools.

Malicious packages as attack vectors

Although AI hallucinations can be unpredictable, and responses given by generative AI tools may not always be consistent, malicious actors are able to discover AI hallucinations by adopting the approach used by Vulcan. This allows hallucinated packages to be used as attack vectors. Once a malicious actor has discovered a hallucination of an un-published package, they are able to create a package with the same name and include a malicious payload, before publishing it. This is known as a malicious package.

Malicious packages could also be recommended by generative AI tools in the form of pre-existing packages. A user may be recommended a package that had previously been confirmed to contain malicious content, or a package that is no longer maintained and, therefore, is more vulnerable to hijack by malicious actors.

In such scenarios it is not necessary to manipulate the training data (data poisoning) to achieve the desired outcome for the malicious actor, thus a complex and time-consuming attack phase can easily be bypassed.

An unsuspecting software developer may incorporate a malicious package into their code, rendering it harmful. Deployment of this code could then result in compromise and escalation into a full-blown cyber-attack.

Figure 1: Flow diagram depicting the initial stages of an AI Package Hallucination Attack.

For providers of Software-as-a-Service (SaaS) products, this attack vector may represent an even greater risk. Such organizations may have a higher proportion of employed software developers than other organizations of comparable size. A threat actor, therefore, could utilize this attack vector as part of a supply chain attack, whereby a malicious payload becomes incorporated into trusted software and is then distributed to multiple customers. This type of attack could have severe consequences including data loss, the downtime of critical systems, and reputational damage.

How could Darktrace detect an AI Package Hallucination Attack?

In June 2023, Darktrace introduced a range of DETECT™ and RESPOND™ models designed to identify the use of generative AI tools within customer environments, and to autonomously perform inhibitive actions in response to such detections. These models will trigger based on connections to endpoints associated with generative AI tools, as such, Darktrace’s detection of an AI Package Hallucination Attack would likely begin with the breaching of one of the following DETECT models:

  • Compliance / Anomalous Upload to Generative AI
  • Compliance / Beaconing to Rare Generative AI and Generative AI
  • Compliance / Generative AI

Should generative AI tool use not be permitted by an organization, the Darktrace RESPOND model ‘Antigena / Network / Compliance / Antigena Generative AI Block’ can be activated to autonomously block connections to endpoints associated with generative AI, thus preventing an AI Package Hallucination attack before it can take hold.

Once a malicious package has been recommended, it may be downloaded from GitHub, a platform and cloud-based service used to store and manage code. Darktrace DETECT is able to identify when a device has performed a download from an open-source repository such as GitHub using the following models:

  • Device / Anomalous GitHub Download
  • Device / Anomalous Script Download Followed By Additional Packages

Whatever goal the malicious package has been designed to fulfil will determine the next stages of the attack. Due to their highly flexible nature, AI package hallucinations could be used as an attack vector to deliver a large variety of different malware types.

As GitHub is a commonly used service by software developers and IT professionals alike, traditional security tools may not alert customer security teams to such GitHub downloads, meaning malicious downloads may go undetected. Darktrace’s anomaly-based approach to threat detection, however, enables it to recognize subtle deviations in a device’s pre-established pattern of life which may be indicative of an emerging attack.

Subsequent anomalous activity representing the possible progression of the kill chain as part of an AI Package Hallucination Attack could then trigger an Enhanced Monitoring model. Enhanced Monitoring models are high-fidelity indicators of potential malicious activity that are investigated by the Darktrace analyst team as part of the Proactive Threat Notification (PTN) service offered by the Darktrace Security Operation Center (SOC).

Conclusion

Employees are often considered the first line of defense in cyber security; this is particularly true in the face of an AI Package Hallucination Attack.

As the use of generative AI becomes more accessible and an increasingly prevalent tool in an attacker’s toolbox, organizations will benefit from implementing company-wide policies to define expectations surrounding the use of such tools. It is simple, yet critical, for example, for employees to fact check responses provided to them by generative AI tools. All packages recommended by generative AI should also be checked by reviewing non-generated data from either external third-party or internal sources. It is also good practice to adopt caution when downloading packages with very few downloads as it could indicate the package is untrustworthy or malicious.

As of September 2023, ChatGPT Plus and Enterprise users were able to use the tool to browse the internet, expanding the data ChatGPT can access beyond the previous training data cut-off of September 2021 [5]. This feature will be expanded to all users soon [6]. ChatGPT providing up-to-date responses could prompt the evolution of this attack vector, allowing attackers to publish malicious packages which could subsequently be recommended by ChatGPT.

It is inevitable that a greater embrace of AI tools in the workplace will be seen in the coming years as the AI technology advances and existing tools become less novel and more familiar. By fighting fire with fire, using AI technology to identify AI usage, Darktrace is uniquely placed to detect and take preventative action against malicious actors capitalizing on the AI boom.

Credit to Charlotte Thompson, Cyber Analyst, Tiana Kelly, Analyst Team Lead, London, Cyber Analyst

References

[1] https://seo.ai/blog/chatgpt-user-statistics-facts

[2] https://darktrace.com/news/darktrace-addresses-generative-ai-concerns

[3] https://darktrace.com/news/darktrace-email-defends-organizations-against-evolving-cyber-threat-landscape

[4] https://vulcan.io/blog/ai-hallucinations-package-risk?nab=1&utm_referrer=https%3A%2F%2Fwww.google.com%2F

[5] https://twitter.com/OpenAI/status/1707077710047216095

[6] https://www.reuters.com/technology/openai-says-chatgpt-can-now-browse-internet-2023-09-27/

DENTRO DEL SOC
Darktrace son expertos de talla mundial en inteligencia de amenazas, caza de amenazas y respuesta a incidentes, y proporcionan apoyo al SOC las 24 horas del día a miles de clientes de Darktrace en todo el mundo. Inside the SOC está redactado exclusivamente por estos expertos y ofrece un análisis de los ciberincidentes y las tendencias de las amenazas, basado en la experiencia real sobre el terreno.
AUTOR
SOBRE EL AUTOR
Charlotte Thompson
Cyber Analyst
Tiana Kelly
Deputy Team Lead, London & Cyber Analyst
Book a 1-1 meeting with one of our experts
share this article
CASOS DE USO
No se ha encontrado ningún artículo.
Cobertura básica

More in this series

No se ha encontrado ningún artículo.

Blog

Email

Looking Beyond Secure Email Gateways with the Latest Innovations to Darktrace/Email

Default blog imageDefault blog image
09
Apr 2024

Organizations Should Demand More from their Email Security

In response to a more intricate threat landscape, organizations should view email security as a critical component of their defense-in-depth strategy, rather than defending the inbox alone with a traditional Secure Email Gateway (SEG). Organizations need more than a traditional gateway – that doubles, instead of replaces, the capabilities provided by native security vendor – and require an equally granular degree of analysis across all messaging, including inbound, outbound, and lateral mail, plus Teams messages.  

Darktrace/Email is the industry’s most advanced cloud email security, powered by Self-Learning AI. It combines AI techniques to exceed the accuracy and efficiency of leading security solutions, and is the only security built to elevate, not duplicate, native email security.  

With its largest update ever, Darktrace/Email introduces the following innovations, finally allowing security teams to look beyond secure email gateways with autonomous AI:

  • AI-augmented data loss prevention to stop the entire spectrum of outbound mail threats
  • an easy way to deploy DMARC quickly with AI
  • major enhancements to streamline SOC workflows and increase the detection of sophisticated phishing links
  • expansion of Darktrace’s leading AI prevention to lateral mail, account compromise and Microsoft Teams

What’s New with Darktrace/Email  

Data Loss Prevention  

Block the entire spectrum of outbound mail threats with advanced data loss prevention that builds on tags in native email to stop unknown, accidental, and malicious data loss

Darktrace understands normal at individual user, group and organization level with a proven AI that detects abnormal user behavior and dynamic content changes. Using this understanding, Darktrace/Email actions outbound emails to stop unknown, accidental and malicious data loss.  

Traditional DLP solutions only take into account classified data, which relies on the manual input of labelling each data piece, or creating rules to catch pattern matches that try to stop data of certain types leaving the organization. But in today’s world of constantly changing data, regular expression and fingerprinting detection are no longer enough.

  • Human error – Because it understands normal for every user, Darktrace/Email can recognize cases of misdirected emails. Even if the data is correctly labelled or insensitive, Darktrace recognizes when the context in which it is being sent could be a case of data loss and warns the user.  
  • Unclassified data – Whereas traditional DLP solutions can only take action on classified data, Darktrace analyzes the range of data that is either pending labels or can’t be labeled with typical capabilities due to its understanding of the content and context of every email.  
  • Insider threat – If a malicious actor has compromised an account, data exfiltration may still be attempted on encrypted, intellectual property, or other forms of unlabelled data to avoid detection. Darktrace analyses user behaviour to catch cases of unusual data exfiltration from individual accounts.

And classification efforts already in place aren’t wasted – Darktrace/Email extends Microsoft Purview policies and sensitivity labels to avoid duplicate workflows for the security team, combining the best of both approaches to ensure organizations maintain control and visibility over their data.

End User and Security Workflows

Achieve more than 60% improvement in the quality of end-user phishing reports and detection of sophisticated malicious weblinks1

Darktrace/Email improves end-user reporting from the ground up to save security team resource. Employees will always be on the front line of email security – while other solutions assume that end-user reporting is automatically of poor quality, Darktrace prioritizes improving users’ security awareness to increase the quality of end-user reporting from day one.  

Users are empowered to assess and report suspicious activity with contextual banners and Cyber AI Analyst generated narratives for potentially suspicious emails, resulting in 60% fewer benign emails reported.  

Out of the higher-quality emails that end up being reported, the next step is to reduce the amount of emails that reach the SOC. Darktrace/Email’s Mailbox Security Assistant automates their triage with secondary analysis combining additional behavioral signals – using x20 more metrics than previously – with advanced link analysis to detect 70% more sophisticated malicious phishing links.2 This directly alleviates the burden of manual triage for security analysts.

For the emails that are received by the SOC, Darktrace/Email uses automation to reduce time spent investigating per incident. With live inbox view, security teams gain access to a centralized platform that combines intuitive search capabilities, Cyber AI Analyst reports, and mobile application access. Analysts can take remediation actions from within Darktrace/Email, eliminating console hopping and accelerating incident response.

Darktrace takes a user-focused and business-centric approach to email security, in contrast to the attack-centric rules and signatures approach of secure email gateways

Microsoft Teams

Detect threats within your Teams environment such as account compromise, phishing, malware and data loss

Around 83% of Fortune 500 companies rely on Microsoft Office products and services, particularly Teams and SharePoint.3

Darktrace now leverages the same behavioral AI techniques for Microsoft customers across 365 and Teams, allowing organizations to detect threats and signals of account compromise within their Teams environment including social engineering, malware and data loss.  

The primary use case for Microsoft Teams protection is as a potential entry vector. While messaging has traditionally been internal only, as organizations open up it is becoming an entry vector which needs to be treated with the same level of caution as email. That’s why we’re bringing our proven AI approach to Microsoft Teams, that understands the user behind the message.  

Anomalous messaging behavior is also a highly relevant indicator of whether a user has been compromised. Unlike other solutions that analyze Microsoft Teams content which focus on payloads, Darktrace goes beyond basic link and sandbox analysis and looks at actual user behavior from both a content and context perspective. This linguistic understanding isn’t bound by the requirement to match a signature to a malicious payload, rather it looks at the context in which the message has been delivered. From this analysis, Darktrace can spot the early symptoms of account compromise such as early-stage social engineering before a payload is delivered.

Lateral Mail Analysis

Detect and respond to internal mailflow with multi-layered AI to prevent account takeover, lateral phishing and data leaks

The industry’s most robust account takeover protection now prevents lateral mail account compromise. Darktrace has always looked at internal mail to inform inbound and outbound decisions, but will now elevate suspicious lateral mail behaviour using the same AI techniques for inbound, outbound and Teams analysis.

Darktrace integrates signals from across the entire mailflow and communication patterns to determine symptoms of account compromise, now including lateral mailflow

Unlike other solutions which only analyze payloads, Darktrace analyzes a whole range of signals to catch lateral movement before a payload is delivered. Contributing yet another layer to the AI behavioral profile for each user, security teams can now use signals from lateral mail to spot the early symptoms of account takeover and take autonomous actions to prevent further compromise.

DMARC

Gain in-depth visibility and control of 3rd parties using your domain with an industry-first AI-assisted DMARC

Darktrace has created the easiest path to brand protection and compliance with the new Darktrace/DMARC. This new capability continuously stops spoofing and phishing from the enterprise domain, while automatically enhancing email security and reducing the attack surface.

Darktrace/DMARC helps to upskill businesses by providing step by step guidance and automated record suggestions provide a clear, efficient road to enforcement. It allows organizations to quickly achieve compliance with requirements from Google, Yahoo, and others, to ensure that their emails are reaching mailboxes.  

Meanwhile, Darktrace/DMARC helps to reduce the overall attack surface by providing visibility over shadow-IT and third-party vendors sending on behalf of an organization’s brand, while informing recipients when emails from their domains are sent from un-authenticated DMARC source.

Darktrace/DMARC integrates with the wider Darktrace product platform, sharing insights to help further secure your business across Email Attack Path and Attack Surface management.

Conclusion

To learn more about the new innovations to Darktrace/Email download the solution brief here.

All of the new updates to Darktrace/Email sit within the new Darktrace ActiveAI Security Platform, creating a feedback loop between email security and the rest of the digital estate for better protection. Click to read more about the Darktrace ActiveAI Security Platform or to hear about the latest innovations to Darktrace/OT, the most comprehensive prevention, detection, and response solution purpose built for critical infrastructures.  

Learn about the intersection of cyber and AI by downloading the State of AI Cyber Security 2024 report to discover global findings that may surprise you, insights from security leaders, and recommendations for addressing today’s top challenges that you may face, too.

References

[1] Internal Darktrace Research

[2] Internal Darktrace Research

[3] Essential Microsoft Office Statistics in 2024

Continue reading
About the author
Carlos Gray
Product Manager

Blog

No se ha encontrado ningún artículo.

Managing Risk Beyond CVE Scores With the Latest Innovations to Darktrace/OT

Default blog imageDefault blog image
09
Apr 2024

Identifying Cyber Risk in Industrial Organizations

Compromised OT devices in ICS and SCADA environments pose significant physical risks, even endangering lives. However, identifying CVEs in the multitude of complex OT devices is labor-intensive and time-consuming, draining valuable resources.

Even after identifying a vulnerability, implementing a patch presents its own challenges limited maintenance windows and the need for uninterrupted operations strain IT and OT teams often leading organizations to prioritize availability over security leading vulnerabilities remaining unresolved for over 5 years on average. (1)

Darktrace’s New Innovation

Darktrace is an industry leader in cybersecurity with 10+ years of experience securing OT environments where we take a fundamentally different approach using Self-Learning AI to enhance threat detection and response.

Continuing to combat the expanding threat landscape, Darktrace is excited to announce new capabilities that enable a contextualized and proactive approach to managing cyber risk at industrial organizations.

Today we launch an innovation to our OT Cybersecurity solution, Darktrace/OT, that will add a layer of proactivity, enabling a comprehensive approach to risk management. This industry leading innovation for Darktrace/OT moves beyond CVE scores to redefine vulnerability management for critical infrastructure, tackling the full breadth of risks not limited by traditional controls.  

Darktrace/OT is the only OT security solution with comprehensive Risk Management which includes:

  • Contextualized risk analysis unique to your organization
  • The most realistic evaluation and prioritization of OT risk
  • Effectively mitigate risk across your OT infrastructure, with and without patching.
  • The only OT security solution that evaluates your defenses against Advanced Persistent Threat (APT) Groups.

The most comprehensive prevention, detection, and response solution purpose built for Critical Infrastructures

Darktrace’s Self-Learning AI technology is a cutting-edge innovation that implements real time prevention, detection, response, and recovery for operational technologies and enables a fundamental shift from the traditional approach to cyber defense by learning a ‘pattern of life’ for every network, device, and user.  

Rather than relying on knowledge of past attacks, AI technology learns what is ‘normal’ for its environment, discovering previously unknown threats by detecting subtle shifts in behavior. Through identifying these unexpected anomalies, security teams can investigate novel attacks, discover blind spots, have live time visibility across all their physical and digital assets, and reduce time to detect, respond to, and triage security events.  

  • Achieve greater visibility of OT and IT devices across all levels of the Purdue Model.
  • The industry's only OT security to scale threat detection and response, with a 92% time saving from triage to recovery.  
  • The only OT focused security solution to provide bespoke Risk Management.

To learn more about how Darktrace/OT approaches unique use cases for industrial organizations visit the Darktrace/OT Webpage or join us LIVE at a city near you.

Read more below to discover how new innovations to Darktrace/OT are bringing a new, contextualized approach to Risk Management for Industrial organizations.

For more information on the entire Darktrace/OT Solution read our solution brief here.

Darktrace/OT and New Risk Management

Risk Identification

Leveraging the visibility of Darktrace/OT which identifies individual systems throughout the Purdue Model and the relationship between them, Darktrace/OT identifies high-risk CVEs and presents potential attack routes that go beyond techniques requiring a known exploit, such as misuse of legitimate services. Each attack path will have a mathematical evaluation of difficulty and impact from initial access to the high value objectives.  

Together this gives comprehensive coverage over your real and potential risks from both an attacker and known vulnerability perspectives. OT attack paths as seen here even leverage insights between the industrial and corporate communications to reveal ways threat actors may take advantage of IT-OT convergence. This revelation of imperceptible risks fills gaps in traditional risk analysis like remote access and insider threats.

Figure 1: Darktrace/OT visualizing the most critical attack paths at an organization
Figure 1: Darktrace/OT visualizing the most critical attack paths at an organization
Figure 2: A specific Attack Path identified by Darktrace/OT

Risk Prioritization

Darktrace/OT prioritizes remediations and mitigations based on difficulty and damage to your unique organization, using the established Attack Paths.

We ascertain the priorities that apply to your organization beyond pure theoretical damage answering questions like:

  • How difficult is a particular vulnerability to exploit considering the steps an attacker would require to reach it?
  • And, how significant would the impact be if it was exploited within this particular network?

This expanded approach to risk prioritization has a much more comprehensive evaluation of your organization's unique risk than has ever been possible before. Traditional approaches of ranking only known vulnerabilities with isolated scores using CVSS and exploitability metrics, often leaves gaps in IT-OT risks and is blind to legitimate service exploitation.

Figure 3: Darktrace/OT leverages its contextual understand of the organization’s network to prioritize remediation that will have the positive impact on the risk score

Darktrace provides mitigation strategies associated with each identified risk and the relevant impact it has on your overall risk posture, across all MITRE ATT&CK techniques.

What sets Darktrace apart is our ability to contextualize these mitigations within the broader business. When patching vulnerabilities directly isn’t possible, Darktrace identifies alternative actions that harden attack paths leading to critical assets. Hardening the surrounding attack path increases the difficulty and therefore reduces the likelihood and impact of a breach.

That means unpatched vulnerabilities and vulnerable devices aren’t left unprotected. This also has an added bonus, those hardening techniques work for all devices in that network segment, so apply one change, secure many.

Figure 4: Darktrace prioritizes mitigation reducing accessibility of vulnerability and the overall risk score when patches aren’t available

Communicate Board Level Risk with APT Threat Mapping

Darktrace/OT bridges theory and practice as the only security solution that maps MITRE techniques, frequently used by APT Groups, onto AI-assessed critical Attack Paths. This unique solution provides unparalleled insights including sector and location intelligence, possible operating platforms, common techniques, exploited CVEs, and the number of potential devices affected in your environment, supporting holistic risk assessment and proactive defense measures.

Ultimately, this becomes a power dashboard to communicate board level risk, using both metric based evidence and industry standard threat mapping.

Conclusion

Darktrace/OT is part of the Darktrace ActiveAI Security Platform a native, holistic, AI-driven platform built on over ten years of AI research. It helps security teams shift to more a productive mode, finding the known and the unknown attacks and transforming the SOC with the various Darktrace products to drive efficiency gains. It does this across the whole incident lifecycle to lower risk, reduce time spent on active incidents, and drive return on investment.

Discover more about Darktrace's ever-strengthening platform with the upcoming changes coming to our Darktrace/Email product and other launch day blogs.

Join Darktrace LIVE half-day event to understand the reality versus the hype surrounding AI and how to achieve cyber resilience.

Learn about the intersection of cyber and AI by downloading the State of AI Cyber Security 2024 report to discover global findings that may surprise you, insights from security leaders, and recommendations for addressing today’s top challenges that you may face, too.  

References

1. https://research-information.bris.ac.uk/ws/portalfiles/portal/313646831/Catch_Me_if_You_Can.pdf

Continue reading
About the author
Mitchell Bezzina
VP, Product and Solutions Marketing
Our ai. Your data.

Elevate your cyber defenses with Darktrace AI

Inicie su prueba gratuita
Darktrace AI protecting a business from cyber threats.