Col, that's a fair question. The problem usually isn't the wooden or brick siding but having a wooden frame that flexes in high enough winds. There IS such a thing as a steel-frame house with brick veneer. They last longer. You can also build a house with concrete walls and steel frame. Those things are like fortresses but they are a beast if you want to add on a room. But on the good side, their R-value (insulation) is EXTREMELY high so the heating and cooling costs are less. They are immune to termites. I looked into building one a long time ago, but then Mom got ill and my life plans changed drastically for several years.
Specific to my general area (south Louisiana): We have a few steel-frame homes to the north of New Orleans, across Lake Ponchartrain. One reason we don't have more steel-frame homes is cost. Another is weight, because south of the lake, we are building on really ancient wetlands that suffer from soil erosion and subsidence. The soft wetland soil here is so deep that the foundations become prohibitively expensive. I recall reading that to reach bedrock here, you might have to drive pilings 1500 feet deep. We heard about that when the Louisiana Superdome (stadium) was being built and folks questioned why it would be so expensive. (Answer: A really expensive foundation.)