Yep, same as if you left your car keys or keyfob on your desk. The nice thing about this setup though is you no longer have to keep track of both keys and cell phone. My new cell phone cover has a wallet built in, just enough room for a drivers licenses and credit card, it's the only thing I'll ever need to carry once I get the car.
I do the same, cell phone in a case combined with my wallet and credit cards + back up key card incase the phone fails (so far it hasn't failed)
However you still need to carry keys to your home (even if you have an automated lock that works off the phone as well) so the phone/ wallet isn't the only thing to be carrying.
That being said I have my back up key card and house key on a seperate keychain/ card holder still.
Here's an idea for a software update... next update use the camera on the driver's side B pillar as facial recognition to get into the car. I know that camera is not going to be as sophisticated as the face id on the iPhone X however I have a work around to keep up the security.
1) one tug on the door, then center screen wakes up.
2) put your face in proximity of the B pillar camera
3) once inside the car you enter a pin code on the center screen since you used face id to get in
4) once pin code is accepted you can start the car, drive it, unlock the glove box and trunk and frunk. That way even if someone where to put a picture of you in front of the camera... they still can't drive off, get into your frunk or trunk, or into important documents in the glove box
This way you have 3 different ways to get in so even if you loose your phone & you don't bring your back up key card with you since you now trust the phone will work 80% of the time then you still have a 3rd option for that 20% that it doesn't work.