The way I think laws should run are like this (and remember that this is a very loose rule of thumb):
1 - Actions that infringe another person's natural rights (we all know what those are, don't we), cause anyone harm (even the one who committed the act in the first place), cause a person emotional harm, or put a person or persons in physical danger, or else have potential to harm society in general, should be considered illegal. Examples: Murder, theft, drug use, child neglect, etc. It's a long list.
2 - Actions that do none of the above should be considered legal, even if morally questionable. It is up to the individual to decide whether or not they should take part in these activities. Examples: Adultery, homosexuality, pornography, etc. It's an even longer list.
Of course I've no experience in matters of legality, so feel free to ignore my opinions on this...
...but, yes, as there is no harm caused by adultery (assuming the two concenting adults or teens took proper precautions), it would be a gross breach of privacy to illegalize it. Even though adultery is not something I agree with... it's something that must be tolerated.
So in other words, I side with cimics.