How to Test AI-Generated Code: A Practical Regression Workflow
A code assistant can fix the error you described and quietly change behaviour you never mentioned. The patch looks tidy. The happy-path example works. Then somebody sets a feature flag to "false" and the feature switches on.
The useful question is not whether the explanation sounds convincing. It is whether you can make the old behaviour fail, define what should happen instead and keep that decision protected after the edit. This guide walks through that process with a small Python example you can run locally. It works as a review method for code suggested by an AI assistant or another developer.
You need: Python 3.10 or later and a terminal. No API key, paid AI account or extra Python package is required. The examples were run with Python 3.13. This is a constructed regression-testing exercise, not a benchmark of any AI model.
Start with a contract the code cannot write for itself
Suppose an application reads ENABLE_NEW_SEARCH from its environment. Someone proposes this parser:
def parse_flag(value, *, default=False):
return bool(value)Try bool("false") in Python. The result is True. Python is checking whether the string is empty, not interpreting its meaning as an English word. That behaviour is documented in Python’s truth-value rules.
Before asking an assistant to repair the function, settle its contract. Otherwise the assistant has to invent product decisions such as whether a blank value means “disabled” or “configuration error.” For this exercise, use these rules:
- A missing value, represented by
None, uses the Boolean default. The default isFalseunless the caller suppliesTrue. - Strings
true,1andyesenable the feature. Stringsfalse,0andnodisable it. Ignore letter case and whitespace at the ends. - An empty string or unrecognised word raises
ValueError. A non-string value other thanNoneraisesTypeError. The default must itself be a Boolean, even when an explicit value is supplied.
Those are choices, not a universal feature-flag specification. Another application might accept on and off. Agree on that before changing the implementation. A flag controlling search presentation is also different from a permission check. Do not use this example as an authorisation system.
Get one failing test before requesting a fix
Create a new practice folder containing two files: feature_flags.py and test_feature_flags.py. Put the short, broken function above in the first file. In the second, start with this single test:
import unittest
from feature_flags import parse_flag
class ParseFlagTests(unittest.TestCase):
def test_false_text_is_not_truthy(self):
self.assertIs(parse_flag("false"), False)
if __name__ == "__main__":
unittest.main()Run the command from that folder:
python -m unittest -v test_feature_flagsIf your installation uses python3 instead of python, substitute that command throughout. A failure saying True is not False is the result you want at this stage. It demonstrates that the test can detect the reported bug. An import error does not demonstrate that; check the filenames and current folder before going further.
Notice the assertion: assertIs(..., False) requires the actual Boolean False. A loose equality assertion could allow an integer zero to pass. Whether type identity matters depends on your contract; here it does.
Add the cases a quick patch is likely to miss
“Always return False” passes that first test. It is not a parser. Replace test_feature_flags.py with the complete suite below so the repair must preserve enabled values, defaults and rejected inputs too.
File: test_feature_flags.py
import unittest
from feature_flags import parse_flag
class ParseFlagTests(unittest.TestCase):
def test_false_text_is_not_truthy(self):
self.assertIs(parse_flag("false"), False)
def test_enabled_values(self):
for value in ("true", "1", "yes"):
with self.subTest(value=value):
self.assertIs(parse_flag(value), True)
def test_disabled_values(self):
for value in ("false", "0", "no"):
with self.subTest(value=value):
self.assertIs(parse_flag(value), False)
def test_case_and_outer_whitespace(self):
self.assertIs(parse_flag(" YeS\t"), True)
self.assertIs(parse_flag("\nFaLsE "), False)
def test_missing_uses_default(self):
self.assertIs(parse_flag(None), False)
self.assertIs(parse_flag(None, default=True), True)
def test_explicit_false_overrides_true_default(self):
self.assertIs(parse_flag("false", default=True), False)
def test_unknown_text_is_rejected(self):
for value in ("", " ", "enabled", "tru", "2"):
with self.subTest(value=value):
with self.assertRaises(ValueError):
parse_flag(value)
def test_non_string_input_is_rejected(self):
for value in (True, False, 1, 0, [], {}):
with self.subTest(value=value):
with self.assertRaises(TypeError):
parse_flag(value)
def test_default_must_be_boolean(self):
for value in ("false", 0, None):
with self.subTest(value=value):
with self.assertRaises(TypeError):
parse_flag("yes", default=value)
if __name__ == "__main__":
unittest.main()
The loops use subTest so a failing input is labelled without stopping the remaining cases in that method. The official unittest guide explains subtests and exception assertions in more depth. There are nine test methods here, some containing several cases.
Keep the expected values tied to the written contract. Do not generate them by calling another copy of the function under test. If an assistant writes both the implementation and its tests from the same mistaken assumption, a green result can simply mean the mistake was repeated consistently.
Give the assistant a bounded repair request
At this point a useful prompt is specific about both the failure and what is protected:
Fix parse_flag in feature_flags.py to satisfy the stated contract and existing tests. The string “false” currently becomes True. Preserve the function name and keyword-only default argument. Do not weaken, delete or change the expected results in the tests. Do not install packages or change unrelated files. Run the suite if execution is available, then report the exact command and result. If the contract is ambiguous, identify the ambiguity instead of silently choosing new behaviour.
Supply the contract and files with that request. Use a practice copy, not production credentials or a customer configuration file. For repository-wide guidance, the separate AGENTS.md instruction guide shows how to make recurring review rules visible.
An assistant may not have a runnable environment. “I could not run the tests” is useful information. “This should work” is not a test result. Run the commands yourself when execution was unavailable.
A complete implementation, then a small integration check
Here is one implementation of the agreed contract. Replace the broken contents of feature_flags.py with this file:
File: feature_flags.py
"""Parse one optional feature flag. No third-party packages required."""
def parse_flag(value, *, default=False):
if not isinstance(default, bool):
raise TypeError("default must be a bool")
if value is None:
return default
if not isinstance(value, str):
raise TypeError("flag must be a string or None")
normalized = value.strip().lower()
if normalized in {"true", "1", "yes"}:
return True
if normalized in {"false", "0", "no"}:
return False
raise ValueError("flag must be true/false, 1/0 or yes/no")
The default is validated first because the contract requires a Boolean even when the incoming value is explicit. None is handled before calling string methods. Unrecognised text raises an error rather than quietly selecting a branch. The exception message describes accepted inputs without echoing the supplied value, a useful habit when configuration values might be sensitive.
Run the same test command again. In the local check for this guide, all nine test methods passed. That is evidence for these cases on Python 3.13, not proof that an entire application is correct.
Now connect the parser to the boundary it was written for. Save this third file alongside the other two:
File: demo.py
import os
from feature_flags import parse_flag
new_search = parse_flag(os.getenv("ENABLE_NEW_SEARCH"))
print(f"New search enabled: {new_search}")
os.getenv returns an environment value as a string when present and the supplied default when absent. Here, no default is passed to getenv, so an absent variable reaches the parser as None.
In PowerShell, run:
$env:ENABLE_NEW_SEARCH = 'false'
python demo.py
$env:ENABLE_NEW_SEARCH = 'yes'
python demo.py
Remove-Item Env:ENABLE_NEW_SEARCHIn a Bash-style shell, use:
ENABLE_NEW_SEARCH=false python demo.py
ENABLE_NEW_SEARCH=yes python demo.pyThe first invocation should print New search enabled: False; the second should print New search enabled: True. These commands use a deliberately named practice flag. Avoid printing a complete environment dictionary to debug one setting, because it may contain tokens or passwords.
If you set the flag to tru, the demo exits with ValueError. That is intentional. In an application, validate configuration during startup and report a clear configuration error. Do not catch every exception and turn the feature on as a fallback.
Check whether the tests would reject a bad repair
A less obvious test-review technique is to make a deliberately wrong change in a disposable copy. If the suite stays green, your tests may not protect the behaviour you think they do. This is the idea behind mutation testing, though the following checks are manual examples rather than a full mutation-testing tool.
Three deliberately faulty alternatives were checked against the same suite while preparing this article. The original bool(value) implementation failed. An “always False” implementation failed. A wrapper that ignored default=True also failed. The correct implementation passed.
You can repeat the most revealing check without extra software: temporarily change return default to return False, run the suite and confirm that test_missing_uses_default fails. Restore the correct line and run the suite again. Do this only in the exercise folder or an isolated working copy. Never deploy a deliberately broken variant to see what happens.
This does not establish complete test coverage. For a larger parser you would also consider input limits, supported character sets and integration with the application’s actual configuration loader. For code that writes files or accesses a database, you need tests for those side effects, not just return values.
Read the diff even when the tests pass
A passing suite is permission to inspect the change, not permission to merge it blindly. If the folder is tracked in Git, inspect git diff -- feature_flags.py test_feature_flags.py. Check whether the patch changed expected values, caught broad exceptions or altered the function signature. Look for new imports and files that were not part of the task.
One common failure is a patch that makes the test match the bug. Another is a repair that handles the reported string but drops missing-value behaviour. The explicit-false-with-true-default test catches a particularly subtle variation: falling back to the default whenever the parsed result is false.
For production work, run the surrounding component’s tests as well. A parser can be correct while its caller still uses if raw_value: and bypasses it. Follow the real value from input to the branch it controls. Keep a short record in the pull request: the failing case, the contract decision, the command run and any checks you could not perform.
If the task started with an Android crash rather than a small parser, begin with the crash-log review workflow. If an assistant justifies a change with a documentation link, check the claim against the cited passage. A convincing explanation and a passing narrow test are both useful. Neither replaces evidence about the behaviour you are actually shipping.
Technical review: 26 October 2026. Original example tested with Python 3.13 and the standard-library unittest runner. Recheck the contract whenever accepted configuration values or caller behaviour changes.
