Terrible, socially-maladaptive ideas like swinging are largely a product of having mutations in your genes that affect your brain and cause you to come up with bad ideas like that.
If you're ugly, then that's often also a result of mutation in your genome—either directly, with mutations to the part of the genome that governs your appearance; or indirectly, with other mutations that leave your body overtaxed and with fewer resources to keep you looking attractive and healthy and normal.
Plus, a maladaptive brain will come up with behaviors that compound unattractiveness (bad nutrition, bad hygiene, crazy hair dye, etc.).
About 80% of your genome affects how your brain develops, and the other 20% affects how your body develops. If mutations are present in the small part of your genome that affects your body, then it's very likely that mutations are also present (and fairly numerous) in the part of your genome that affects your brain—as opposed to only being present in the smaller part.
So if you're ugly, it almost always gives away the fact that there's something wrong with your brain, too.
Recall the people who tried to tell you that the idea that "the heroes are all attractive, and the villains are all ugly" is actually very unfair and inaccurate and we shouldn't think that way. There are some exceptions, but in reality, it's a fairly reliable pattern. It's some ancient wisdom shit.
Ed Dutton (The Jolly Heretic on YouTube) does a good job of going into this stuff.