@Linkshaender Sorry, there is a lot to argue with here, so let me give you an extreme, but hopefully illustrative example:
Our spec is to add two finite integers. There is a non-zero chance, that the LLM will implement this by calculating square roots and applying a ton of hacks and so it does.
Humans are not able to touch the code since every bit of it is unreasonable and literally drives them crazy. Having an LLM do a change makes things worse.
Still all of your tests pass, since that's the success criteria for the LLM (btw how do you know what to test for in this case?). Some inputs (you don't know which) would DoS your system or give inaccurate results, but no one complained the last 6 months.
Would you be a happy PO of this system?
@gadi