Well, ultimately the problem is that San Francisco is a small area where a lot of people want to live, and they're willing to pay a premium to live there. You can fiddle around with the supply side of this "problem" if you like (which will by your prescription also lower the demand by making the city a less nice place to live), but there's only so much you can do.
As far as I'm concerned the lack of cheap housing in San Francisco is no more of a problem than the lack of cheap housing in Beverley Hills.
there's only so much you can do. That's not really true, per the linked article above.
As far as I'm concerned the lack of cheap housing in San Francisco is no more of a problem than the lack of cheap housing in Beverley Hills.
See Edward Glaeser's Triumph of the City for more about why this is a problem. For example, he cites an array of research that comes to this conclusion:
"Cities enable collaboration, especially the joint production of knowledge that is mankind's most important creation. Ideas flow readily from person to person in the dense corridors of Bangalore and London, and people are willing to put up with high urban prices just to be around talented people, some of whose knowledge will rub off.
Rousseau famously wrote, 'Cities are the abyss of the human species,' but he had things completely backward. Cities enable the collaboration that makes humanity shine most brightly. Because humans learn so much from other humans, we learn more when there are more people around us. Urban density creates a constant flow of new information that comes from observing others' successes and failures" {Glaeser "Triumph"@247}.
In other words, having a large concentration of smart people leads to new ideas and "the joint production of knowledge." Preventing people from moving to cities through price restrictions means less knowledge and has an array of negative environmental consequences.
No, Beverley Hills is desirable specifically for its prestige.
San Francisco is desirable as location convenient to jobs and transportation.
Not having more housing in Beverley Hills has no real impact on the rest of LA. Not having more housing in San Francisco means the extra demand spills over into a variety of other places.
Higher Density housing does not necessarily result in a location being less-nice. Vancouver BC has made a point of choosing trading density over suburban sprawl with supposedly nice results:
I agree w.r.t. San Francisco proper (it's already very dense, certainly by American standards), but I think quite a bit more could be done in the Valley, which would also ease some of the pressure on SF. A lot of jobs are in the Valley, but people (esp. young people) end up living in SF because the Valley refuses to build anything other than suburban housing for the most part.
San Jose has been building condos and apartment buildings near the Diridon Caltrain station, so that's something. But much of the rest of the Valley between SJ and SF is extremely anti-development. Try to get approval to put in a condo tower in Palo Alto! I don't even mean that jokingly: given the demand created by the Palo Alto tech businesses and Stanford, it could easily support several condo towers near the downtown area. The fact that the city government bans them means that that demand is displaced elsewhere, e.g. to SF.
As far as I'm concerned the lack of cheap housing in San Francisco is no more of a problem than the lack of cheap housing in Beverley Hills.