It's pretty good in most places and straight up terrible in others. Google maps tends to have higher lows even though there definitely are cases when Apple's highs are actually higher. But you have to remember: users are using this for going to new places. They don't know they're getting the lowest of the low quality results until it's too late. After a few times getting burned, most people just give up on it which means their most recent impression of it is probably post-failure where-the-fuck-am-I panic.

>What would it take for that impression to go away?

Raise those lows up. I almost think it'd be better for it actually give up and say "I don't know" instead of possibly routing users incorrectly. Users will still think it sucks but they won't be stressed out about the sucky-ness and might be willing to try it again later. From my experience, Siri is a good example of this. It's probably the worst "assistant" out there but it's pretty good about knowing when to give up so users can just Google something.