Tim Lee writes in Forbes: Why The Bay Area Should Have 11 Million Residents Today – Forbes:

“Today, the Bay Area has about 7 million residents. In a free housing market, the population of the San Francisco Bay Area would have been growing rapidly over the last two decades. For example, between 1900 and 1920, the growth of the auto industry helped the population of the Detroit metro area nearly triple, from 540,000 people to 1.4 million people. If the San Francisco Bay Area had grown that fast since 1990, it would have about 16 million people today.”

Having lived in the Bay Area for half a decade, I don’t doubt there are housing restrictions in some municipalities. There are many municipalities. Some would welcome development. Other just demand more side payments. Where is the boom in the East Bay? In the East Bay development is more welcome than the Valley. Where are the hordes clamoring to live in Oakland? Why have the bad neighborhoods areas not been gentrified? Why are existing zoning caps not reached (with buildings torn down and rebuilt more intensely). Because, the demand is too weak.
The so-called economies of agglomeration are just not large enough to justify side payments, or they would be made, or overcome social problems constraining gentrification.
Those 4 million come from somewhere. Maybe they come from Phoenix or Dallas or Houston or Atlanta (or of course everywhere?), but then those places have the infrastructure, land, and housing stock, where if the demand were lower would just see lower land prices, not less people (mostly).
Maybe some development would not have been built there. But that development (i.e. roll back the last 10 years of US development for 2 million housing units, reallocate that to the Bay Area, as is implied in the article) and most of that would be single family houses that would be unbuilt. Preferences for 2 million single family homes cannot physically be satisfied in the Bay Area, so residents would thus face a far worse housing situation than preferred.
We have a friend from Minneapolis we nicknamed “Pre-Millenium Man”, since in the early 2000s he wanted to live the hipster dream of San Francisco in the 1990s, and he did move to the Bay Area and works in software development. Nothing stopped him. He undoubtedly bid out someone who did not care to live there quite as much. So the people who are bid out contribute much less to the economies of agglomeration than the average person who is there (their wages reflect willingness to pay for housing, and are determine by their productivity). Adding 2 million marginals to the pool won’t increase the total productivity as much as the average person today, and will reduce average productivity. It will worsen the aggregate productivity elsewhere. There are diminishing economies of agglomeration, and increasing negative externalities with larger populations.
The entire rationale for housing regulations is to reduce negative externalities, both perceived and real. If we lived in a world where those externalities were otherwise internalized, great, communities would be much more willing to allow more development, as the negative spillovers would not exist. We do not yet live in that world.
I know the Bay Area (thinks it) is the most important place in the world, delivering us the future. I know that future would just be so much worse without famous investments like Pets.com. Real productivity is created there, but so is crap. A much better strategy would be to stop wasting so much effort on duplication, drivel, and so on, and incentivize those competent software developers to make things that are worthwhile. If people can no longer find good investments, or can no longer distinguish between good investments and crap, perhaps the economies of agglomeration have been exhausted.