Also the problem isn't code that's obviously wrong when you read it. The problem is when code looks OK, but is subtly wrong. Which keeps happening - as we know today, at least two examples featured on Copilot homepage have this problem. To make this work, you have to read each snippet super carefully - which defeats the whole point.
I really don't think it does defeat the point. The point isn't just churning stuff out as fast as possible without concerning yourself with quality. There are lots of other reasons you'd benefit from snippets -- rote work done for you, familiarity with language ecosystem, and so on.
It's all well and good to say there should be something better than just discipline but there's no idiot-proof way of writing programs.
Ok, maybe it doesn't defeat the whole point. The way I see it, there are two ways of using Copilot:
1. As a way to specify a problem and get a suggested solution (or a selection of them), to use as a starting point.
2. As a way to specify a problem, get it solved by the AI, and consider it done.
I'd expect that any developer worth their salt will do 1. I expect that of myself. I also worry this is so streamlined that people, including myself, will naturally shift to doing 2 over time.
This is similar to the problem with self driving cars - you can't incrementally approach perfection, you have to get it right in one go, because the space between "not working" and "better than human in every way" is where self-driving is more dangerous than not having it. When it works most of the time, it lulls you into a false sense of security, and then when it fails, you aren't prepared and you die. Similarly, Copilot seems to be working well enough to make you think the code is OK, but it turns out the code is often buggy in a subtle way.
> familiarity with language ecosystem
This is an interesting angle to explore the topic. Familiarity is a big factor when inspecting such generative snippets. For example, I'm really familiar with modern C++, and I'm confident I could spot problems in Copilot output (if and when it starts producing C++), maybe 50% of the time. If it's a logic issue, or initialization issue, I'll spot it for sure. If it's a misuse of some tricky bits of the Standard Library? I might not. I make enough of those mistakes on my own. Or, I know enough JS to be dangerous, but I don't consider myself fluent. I'm definitely going to miss most of the subtle bugs there.
To my mind the difference between copilot and semi-autonomous cars is that split-second decisions are not required in this instance. If it takes you a minute to snap out of it and notice the suggestion was wrong, no problem.
On your other point, it's true that if you're working in an unfamiliar ecosystem, spotting subtle issues will be harder. But I think getting automatic suggestions that are probably idiomatic in that language will be more helpful than hurtful.
> When it works most of the time, it lulls you into a false sense of security, and then when it fails, you aren't prepared and you die.
That still doesn't _necessarily_ imply that 'partially self-driving cars' are worse than actually existing humans. Really, anything that's (statistically) better than humans is better, right?
I don't think it's reasonable to think that even 'perfect' self-driving cars would result in literally zero accidents (or even fatalities).
Also the problem isn't code that's obviously wrong when you read it. The problem is when code looks OK, but is subtly wrong. Which keeps happening - as we know today, at least two examples featured on Copilot homepage have this problem. To make this work, you have to read each snippet super carefully - which defeats the whole point.
reply