agglomeration theory backwards

Agglomeration theory is a fancy word for the geographic clustering of similar economic activities. For example, the financial district of Manhattan. Even though the businesses compete with each other, everyone tends to benefit from the close proximity and the concurrency of transportation networks, and ready access to a labor force and market. It helps explain why cities exist and why they tend to be wealthier on a per capita basis as they get larger.

But what about when things fall apart? Would it be reasonable to assume that as a geographic region exhibits consolidation in similar businesses its overall health would start to decline? Also, as an area loses population, it would lose diversity in resources across the board. This is mathematically obvious and it has dangerous implications for any city that sheds a portion of its population because the signs will point to continued decline. The maxim "grow or die" seems harsh, but long, drawn out misery of many small cities is a testament to this.