Maps are Great Mmkay. Now Let’s Get to Testing Them.

Now that we’ve got choices (plural!) for maps on CarPlay, the testing and evaluation floodgates have opened and we at CarPlay World couldn’t be happier. We love this stuff.

When Google Maps and then Waze became available with iOS 12 in September, we settled into a fairly constant mindset of switching between them every few days to pick the best for our specific manner of driving and locating destinations. See our Apple Maps vs. Waze comparison to get an idea of how we approach mapping in CarPlay.

How do you test the three mapping apps for accuracy? Why, you say a destination and watch them work. The goal is to have as few distractions and as little further human input as possible — after all, you are driving.

It’s not the Journey, it’s uh… The Destination

Colorado State Capitol
200 E Colfax Ave, Denver, CO 80203

This test is so easy. For humans. For computers however,  phones included, “state capitol” — our search phrase for this test — is hard… it’s not a “proper noun” business, like for instance “Starbucks”. My phone and these mapping apps have to do some extra work. In this case, they have to know which state I’m in before they can do the rest of the mapping calculations. And they have to know I’m looking for the state capitol… that I’m looking for a single location.

Apple Maps: Pass

Understood the request? Yes

Remembered the request? Yes

Result: Passed

After the dictation, I was given a choice screen to pick the correct destination. Subsequent “state capitol” voice commands didn’t elicit this choice — it just became my destination. Nice.

By remembering my request, we avoid — excuse the pun — an extra “round trip”. Next time, Apple Maps won’t offer me a choice, and I don’t need to involve myself in choosing. Remember, I’m driving.

Waze:  Fail

Understood the request? No

Remembered the request? N/A

Result: Failed

Waze just didn’t get it. Literally, I heard “I didn’t get that” or some variation repeatedly while saying “state capitol”. It was all for naught. To a passerby I probably looked like a lunatic: enunciating “state capitol” loudly and clearly, over and over, to my car’s interior.

I tried speaking normally at the first go. “I’m not sure I understand”. STATE CAPITOL. “I didn’t get that.” STATE. CAPITOL. “I’m not sure I understand.” Etc.

The closest I got with Waze with the evil “state capitol” verbal driving directions test

I tried about ten times, then gave up and used “Colorado state capitol”. That worked. I used the same test environment as the Apple Maps test: driving around 25 MPH, no music playing, normal road/traffic noise, windows up. I even tried when stopped at a light to eliminate the already very low ambient road noise. 

After failing (after eight attempts it obviously wasn’t a matter of how I delivered the words), I relented and gave Waze the “Colorado state capitol” gimme, and that worked.

Google Maps: Fail

Understood the request? No

Remembered the request? N/A

Result: Failed

Google Maps acted a lot like Waze in its failure. It’s probably no coincidence, because Waze became a Google company in 2013. Five years is enough time to cross-pollinate them so they use the same engine, or at the very least, use many of the same methods to do their work. Which is why we see them fail similarly.

Kinda promising…

Both apps got in trouble not in hearing or parsing my phrase, but by failing to understand that I was looking for, simply, the Colorado State Capitol building. This isn’t a display, UI or mapping problem, it’s a “deep” AI problem.

But can we stop to get beef jerky?
After the cheat word “Colorado” is given…

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.