But at that point, do you need to own the car? Basically you just described a taxi service without human drivers.

Perhaps a taxi service based around self driving cars could drive costs below the cost of ownership. My guess is that the limiting factor is rush-hour.