You Scratch My Back and I'll Scratch Yours

You have probably noticed patches of what looks like fungus or mold growing on the the trunks and branches of your trees. These are actually lichens; symbiotic colonies of fungi, algae and/or cyanobacteria.
The fungi provides housing, giving the algae/cyanobacteria areas in which to grow that they could not normally inhabit. In trade, the algae/cyanobacteria use photosynthesis to provide carbohydrates (food) to the fungi.
In general, they are nothing to worry about.