Why do so many religious groups have restrictions on sexual relations?

Not all do, and some have more than others. But why is this? Especially when the activities in question do not harm. Also, at times, certain activities that are harmful to others (such as rape) are condoned in certain circumstances, yet (consensual) fornication and adultery with the rest of the general population are strictly prohibited.