I imagine there will be pissed passengers in the car who are sitting stopped for far too long because some jackass things he's above traffic laws. I suspect the self-driving car will also usher the age of people getting beat up for fucking with self-driving cars.
There's a million social elements here we haven't explored. Imagine driving through a ghetto and lots of people rush over to your car to rob you (or worse). Your car will happily brake for them, when a human would have recognized the situation and accelerated and veered off to avoid them. Or weird edges cases like garbage on the road or somesuch and the car not being able to outmaneuver it because it may be illegal, but a human with a more lax acceptance of traffic laws will veer into the right lane close to another car to avoid it.
Or just the daily grind of speed limits. On the expressways I drive the posted speed is 65-70 yet most everyone does 80-85. Doing a steady 60-65mph because of how laws were written to get speeding ticket revenue might be annoying. Especially if you're the only non-human driven car and everyone is speeding past you and honking because you're not going the speed of traffic.
You should forward this post to your psychoanalyst. It's a rich, concentrated dose of mental pathology for his consideration, or maybe it's just the ramblings of an old fart looking back on a sixth of his life spent commuting. The cool kids aren't staring out the window searching for bums to beat up. (besides how would they stop the car or get the door open?) The cool kids are playing a game on the shared table in the middle of the car (the front seats face to the rear, you know) or they're working on their phablet or they're videoconferencing with their coworkers in other robocars.
And if there were no self-important driver to get pissed off, would "jaywalking" even be a thing? If the first automobile drivers had been poor folks with little political power (haha yeah right), "jaywalking" would never have been made a crime in the first place. Some streets will have heavy pedestrian traffic; the rational robot routes around them to lower-traffic streets.
I find your post somewhat detuned from reality. For instance, jaywalking is rarely a crime. Also, jaywalking becoming a thing was precisely related to cars becoming federated to the "poor folks with little political power" because in many communities, there is heavy car traffic that was prioritized over walking due to damn near everyone having at least one car.