Have you seen any older trees around your neighborhood with a pile of sawdust around the base? This is actually caused by some very busy carpenters.
Carpenter ants that is. They like to build their nests inside dead and decaying wood, the sawdust you see around it is the result of some recent renovations.
Knowing this, it is easy to understand why some people would think that these pests need to be dealt with before they can do anymore damage. But the truth is they mostly only affect wood that is already dead and rotting and rarely cause any harm to healthy tissue.