Evaluating AI-Generated Code: Beyond Syntax and Correctness
AI tools have become a regular part of software development. Developers now use them to write functions, fix bugs, explain unfamiliar code, generate tests and even suggest ways to structure an application.
I've found these tools useful, especially when working through problems that would otherwise take more time to solve.
But there is something worth paying attention to. Code that looks correct is not always good code.
An AI tool can generate a function that runs without errors and gives the expected result in a simple test. But what happens when it receives unexpected input? Does it handle errors properly? Is the implementation easy to understand? Does it fit into the rest of the application?
These are the things I think developers should look at when reviewing AI-generated code.
In this article, I'll go through some of the things I consider important when evaluating code generated by AI.
1. Does the code actually do what it is supposed to do?
This is usually the first thing I check.
When an AI generates code, I want to know whether it solves the problem I gave it, not just whether it looks reasonable.
Take this simple JavaScript function:
function calculateDiscount(price, percentage) { return price - (price * percentage) / 100; } If I pass 1000 as the price and 20 as the discount percentage, I get 800.
That looks fine.
But what if the price is negative? What if someone enters 150 as the discount percentage? What happens if the price is passed as a string instead of a number?
The function doesn't account for any of these situations.
A better implementation would include some basic validation:
function calculateDiscount(price, percentage) { if ( typeof price !== "number" || typeof percentage !== "number" || !Number.isFinite(price) || !Number.isFinite(percentage) ) { throw new TypeError("Price and percentage must be valid numbers."); } if (price < 0) { throw new RangeError("Price cannot be negative."); } if (percentage < 0 || percentage > 100) { throw new RangeError("Percentage must be between 0 and 100."); } return price * (1 - percentage / 100); } This version handles more situations and makes the expected inputs clearer.
Of course, this is still a simple example. A real payment system would also need proper handling of decimal values and rounding.
The point is that I wouldn't judge the first implementation based on one successful result. I would want to understand how it behaves in other situations too.
2. What happens when something unexpected happens?
One thing I pay attention to when reviewing generated code is how it handles situations outside the normal flow.
It's easy to test a function with the exact values it was designed for. The more interesting part is seeing what happens when those values change.
For example, when testing the discount function, I would try:
- A normal price and discount.
- A discount of zero.
- A discount of 100 percent.
- A negative price.
- A discount greater than 100 percent.
- A string passed instead of a number.
Each test tells me something about the implementation.
This is also why I don't think passing a few tests should automatically mean that generated code is correct.
The tests themselves need to make sense. They should cover the actual requirements and include situations that could cause the implementation to fail.
A test suite can give you confidence, but only when it is testing the right things.
3. Is the code easy for another developer to understand?
Something can work perfectly and still be difficult to maintain.
I've seen how important this becomes when working on applications with multiple components, API integrations and different developers contributing to the same codebase.
When reviewing code, I pay attention to things like variable names, function structure, repeated logic and unnecessary complexity.
For example, an AI tool might generate a solution with several nested conditions when a simpler approach would do the same job.
It might also introduce extra functions or abstractions that the project doesn't really need.
This doesn't necessarily mean the code is broken, but it can make future changes harder.
I usually ask myself a few questions:
- Can another developer understand what this code does?
- Are the function and variable names clear?
- Is there logic that could be simplified?
- Does it follow the existing coding style?
- Will it be easy to update when requirements change?
These questions matter because software is rarely written once and left untouched. Requirements change, bugs appear and new features get added.
Code should be written with that in mind.
4. Are there security problems?
This is another area where I wouldn't blindly trust generated code.
An AI tool might generate an API endpoint that works correctly but doesn't properly check whether a user is allowed to access certain information.
It could also overlook input validation, expose sensitive information or introduce unsafe database queries.
Some of these problems won't show up during normal testing.
For example, imagine an endpoint that allows users to retrieve their account information. The endpoint might return the correct data, but if it doesn't verify the identity and permissions of the person making the request, another user could potentially access information that doesn't belong to them.
That is a serious problem, even if the endpoint appears to work.
When reviewing generated code, I pay attention to authentication, authorization, user input, database queries and how sensitive information is handled.
Security should not be something we only consider after an application has been built.
The OWASP Secure Coding Practices Guide is a useful resource for understanding some of these risks.
5. Does the code fit into the existing application?
This is something I think is easy to overlook when evaluating AI-generated code.
Getting an AI tool to write a standalone function is one thing. Getting it to make a change inside an existing application is another.
A piece of code can work perfectly on its own but cause problems when introduced into a real project.
Think about a React application that already has an established way of handling API requests and application state.
An AI tool might generate a new component that makes its own API requests and introduces another layer of state management.
The component might work, but it could also duplicate existing logic and make the application harder to maintain.
This is why understanding the existing codebase matters.
Before accepting a generated implementation, I want to know whether it follows the project's existing structure, uses the right APIs and works with the way other parts of the application are built.
This is also one of the reasons I find real codebase evaluation more interesting than simply asking an AI tool to generate small examples.
The SWE-bench research benchmark is a good example of this. It evaluates AI models using real software engineering issues from GitHub repositories.
You can read the original research here:
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
6. Is the implementation unnecessarily expensive or slow?
Performance is another thing worth checking.
An implementation might produce the correct result but still do more work than necessary.
In a React application, for example, a component might make repeated API requests whenever it renders, even when the data hasn't changed.
The application may still display the correct information, but those unnecessary requests can affect performance.
Other things I would look out for include inefficient database queries, repeated calculations, unnecessary component renders and excessive dependencies.
I don't think every small function needs complicated optimization. That can create more problems than it solves.
What matters is understanding whether the implementation is doing unnecessary work and whether that work has a real effect on the application.
My approach to reviewing AI-generated code
After looking at these areas, I think evaluating generated code comes down to asking a few important questions.
Does it solve the actual problem?
Does it handle inputs and situations that weren't part of the simplest example?
Can another developer understand and maintain it?
Are there security concerns?
Does it fit into the existing application?
Is it doing more work than necessary?
I wouldn't treat every question as equally important for every task. A payment feature, for example, needs particularly careful attention to security and correctness. A simple UI component may require more attention to rendering, accessibility and maintainability.
The requirements of the task should determine what gets the most attention.
Final thoughts
AI coding tools have made some parts of software development faster, and I don't think developers should ignore that.
But generating code and understanding code are two different things.
A developer should still be able to explain what an implementation does, why a particular approach was used and what could go wrong.
I think this is becoming an increasingly important part of working with AI tools.
The goal shouldn't be to accept whatever code an AI produces simply because it runs. The goal should be to use the generated code as a starting point, review it properly and make sure it is good enough for the problem it is supposed to solve.
At the end of the day, the responsibility for the software we ship still belongs to the people building it.
References
- Jimenez et al. (2024). SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
- Google. What to Look for in a Code Review
- OWASP. Secure Coding Practices Quick Reference Guide
- GitHub. Review AI-Generated Code

