The primary reason to have the chimney swept is to minimize the chance of a chimney fire. A chimney fire may do no damage at all or it may completely burn down the house structure and potentially harm the occupants.
Sweeping the chimney on a regular basis increases the efficiency of the fireplace, reduces smoking problems and reduces the risk of carbon monoxide entering the home.