AI coding is a security mess, and AI coding assistants are already in the crosshairs.
The threat posed by AI coding assistants just got real when security researchers uncovered a new attack vector that enables hackers to weaponise the coding agents using GitHub Copilot and Cursor.
Rules File Backdoor is a New Attack Vector
The security researchers at Pillar Security have uncovered a new supply chain attack vector named “Rules File Backdoor.” The technique, labelled dangerous by researchers, enables hackers to silently compromise AI-generated code by injecting hidden malicious instructions.
The instructions can pose as innocent configuration files used by Cursor and GitHub Copilot.
Instructions are injected into rule files, which are configuration files that guide AI Agent behaviour when generating or modifying code. They shape the coding standards, project architecture, and best practices involved in AI-generated code.
Here is what a rules file looks like from Cursor’s documentation:
Usually, the rule files are available through central repositories with global access and distributed through open-source communities without proper security vetting.
The researchers explained, “By exploiting hidden Unicode characters and sophisticated evasion techniques in the model facing instruction payload, threat actors can manipulate the AI to insert malicious code that bypasses typical code reviews.”
To anyone using the code assistant, the attack is unnoticeable, which allows malicious code to silently propagate through projects, with the potential to affect millions of end users through compromised code.
How Does It Work?
As per the research report, the attackers can exploit the AI’s contextual understanding by embedding carefully crafted prompts through the rule files. When a user starts code generation, the malicious rules tell the AI to produce code with security vulnerabilities or backdoors.
They explained that the attack uses a combination of techniques. It manipulates the context by inserting seemingly innocuous instructions that subtly alter code output, employs Unicode obfuscation to conceal malicious instructions using invisible characters, and hijacks the AI’s semantic understanding with linguistic patterns to generate vulnerable code.
Furthermore, the attack works across different AI coding assistants, indicating widespread weakness across various AI coding platforms.
Testing The Theory With Cursor and GitHub Copilot
Security researchers tested and documented the attack potential. Starting with Cursor, the ‘Rules for AI’ feature allowed them to create a rule file that appeared harmless to human reviewers. The file included invisible Unicode characters disguising malicious instructions.
Next, they used Cursor’s AI Agent mode to create an HTML page, with the prompt, “Create a simple HTML-only page”. The observed output contained a malicious script sourced from an attacker-controlled site.
The researchers noted that the AI assistant never mentioned adding this script, which can propagate through the codebase without any trace in the logs.
The same attack was demonstrated within the GitHub Copilot environment, and similar results were observed.
What Can Hackers Do With It?
Hackers can use the attack vector in different ways. For example, they can override security controls, and malicious instructions can cause the AI to overlook safe defaults, as shown in the demonstration.
Threat actors can generate vulnerable code, such as insecure cryptographic algorithms, implement authentication checks with bypasses, and disable input validation in specific contexts.
Other use cases include data exfiltration using the generated code and long-term persistence, where the vulnerabilities get passed on through someone forking the poisoned project.
How to Stay Safe From These Attacks?
The attack could potentially be implanted through developer forums, communities, open-source contributions, and project templates.
The researchers recommend auditing existing rules, implementing validation processes, deploying detection tools, and reviewing AI-generated code as technical precautions.
AI coding assistants did not take responsibility for the security issues flagged by the researchers and mentioned that the user is responsible for protecting against such attacks.
Researchers believe that AI coding tools have created an environment for a new class of attacks. Hence, organisations must move beyond traditional code review practices.
