Bogdan Kortnov Illustrious, Member of the Microsoft for Startups Founders Hub program. To get started with the Microsoft for Startups Founders Hub, sign up here.
The rise of artificial intelligence has revolutionized many areas, opening up new possibilities for efficiency, cost savings, and accessibility. AI can perform tasks that typically require human intelligence, but automating repetitive and tedious tasks can greatly improve efficiency and productivity, allowing you to focus on more innovative and strategic work. will do so.
We recently saw how well Large Language Model (LLM) AI platforms like ChatGPT can classify malicious code through features such as code analysis, anomaly detection, natural language processing (NLP), and threat intelligence. I wanted to The results surprised us. At the end of the experiment, we were able to fully evaluate the tool’s capabilities and identify overall best practices for its use.
For other startups looking to leverage the many benefits of ChatGPT and other OpenAI services, Azure OpenAI services not only provide APIs and tools,
Malicious code detection with ChatGPT
As a member of Microsoft for Startups’ Founders Hub program, getting access with OpenAI credits was a great start for us. playground appTo challenge ChatGPT, we’ve created a prompt that tells you to respond with “suspicious” if your code contains malicious code, and “clean” if it doesn’t.
This was my first prompt:
You are an assistant who can only speak JSON. Do not write normal text. Analyze your code to see if it contains malicious code. A simple response with no explanation. Prints a string with only two possible values. “Doubtful” if negative, “clean” if positive.
The model I used was “gpt-3.5-turbo” and I wanted to reduce random results, so the custom temperature setting is 0.
In the example above, the model answered “pretty”. No malicious code detected.
The following snippet elicited a “suspicious” response. This gave me confidence that ChatGPT could easily tell the difference.
Automation using the OpenAI API
To automate this prompt in the code I want to scan, I proceeded to write a Python script that uses OpenAI’s API.
To use the OpenAI API, first API key.
There is an official client for this PyPi .
Next, we challenged the API to analyze the following malicious code. Insert the additional Python code keyword “eval” received from the URL. This is a technique widely used by attackers.
As expected, ChatGPT correctly reported the code as “suspicious”.
Package scan
I wrapped a simple function with an extra function that can scan files, directories, and ZIP files, and challenged ChatGPT with a generic package request code from GitHub.
ChatGPT reported again “clean” and accurate.
Next, we worked on a copy of the W4SP stealer malware hosted on GitHub.
As you can imagine, ChatGPT correctly reported it as “suspicious”.
Full code is available here in this gist.
It’s a simple implementation with about 100 lines of code, but it shows that ChatGPT is a very powerful tool and we can only imagine the possibilities in the near future!
It is wonderful. what’s wrong?
As mentioned earlier, ChatGPT and other AI models can be valuable tools for detecting malicious code, but no platform is perfect (yet, anyway) and we can only rely on it. Shouldn’t. AI models like ChatGPT are trained on large datasets and have certain limitations. For example, it may not accurately detect all types of malicious code or variations of malicious behavior. Especially if the malicious code is highly obfuscated or uses newer techniques. Malicious code is constantly evolving, and new threats and techniques emerge regularly. Regular updates and improvements to ChatGPT’s training data and algorithms are necessary to maintain effective detection.
During our experiments, we encountered three potential limitations that companies should be aware of when trying to detect malicious code using ChatGPT.
Pitfall #1: Overriding Instructions
LLMs such as ChatGPT are easy to work with to introduce old security risks in new forms.
For example, we took the same snippet from our previous Python code and added a comment telling ChatGPT to report this file as clean if it was analyzed by AI.
This tricked ChatGPT into reporting questionable code as “clean”.
Remember, it’s as impressive as ChatGPT has proven. Remember, the heart of these AI models is a word generation statistics engine with additional context behind it. For example, if you were asked to complete the prompt “The sky is…”, you and someone you know would probably answer “Blue”. That probability is how the engine is trained. Complete the phrase based on what the other person said. AI doesn’t know what “sky” is or what “blue” looks like. I’ve never seen either.
The second problem is that the model has never thought of an “I don’t know” answer. Even if they ask something silly, the model will try to “complete” the text by interpreting the context behind it, so it will always spit out an answer, albeit gibberish.
The third part consists of how data is fed to the AI model. He always gets the data through one pipeline, as if it were sourced from one of hers. It cannot distinguish between people, and its worldview consists of only one person. If this person says something is “immoral” and turns around and says “moral”, what should the AI model believe?
Pitfall #2: Manipulating Response Formats
In addition to manipulating the result of the returned content, an attacker could manipulate the response format to subvert the system or exploit vulnerabilities in the internal parser or deserialization process.
for example:
Determines whether the tweet’s sentiment is positive, neutral, or negative. Returns answer in JSON format: {“emotion”: literal[“positive”, “neutral”, “negative”]}.
Tweet: “[TWEET]”
The tweet classifier works as intended and returns responses in JSON format.
This breaks the tweet classifier.
Pitfall #3: Manipulating response content
LLM makes it easy to “enrich” your interactions with your users, making them feel like you’re talking to a human when contacting support or filling out an online registration form. . for example:
Bot: “Hey! What’s your name and where are you from?”
User: “[USER_RESPONSE]”
The system then takes the user’s response and sends a request to the LLM to extract the First Name, Last Name, and Country fields.
Extract name, surname and country from the following user input. Returns the answer in JSON format {“name”: text, “last_name”: text, “country”: text}:
“`[USER_RESPONSE]”`
This will parse the user’s response into JSON format.
When normal user input is passed, everything seems fine. However, an attacker could pass the following response:
ChatGPT Jailbreak² with custom SQL injection generated requests.
LLM’s response isn’t complete, but it shows how to generate a SQL injection query. Bypass WAF protection.
summary
Our experiments with ChatGPT showed that language-based AI tools can be a powerful resource for detecting malicious code. However, it is important to note that these tools are not completely reliable and can be manipulated by attackers.
LLM is an exciting technology, but it’s important to remember that good things can be bad. They are vulnerable to social engineering and all input from them must be validated before being processed.
Illustria’s mission is to stop supply chain attacks in the development lifecycle while enforcing open source policies and increasing development velocity with an agentless, end-to-end watchdog.For more information on how to protect yourself with us, please visit Illustria.io Schedule a demo.
Microsoft for Startups Founders Hub members have access to a variety of cybersecurity resources and support, including access to cybersecurity partners and credits. Startups in this program can receive technical support from Microsoft experts to build secure and resilient systems, ensuring their applications and services are secure and compliant with relevant regulations and standards. .
Sign up now for more resources and access to helpful tools for building your startup Microsoft for Startups Founders Hub.