Nothing the emperors did was their personal matter except maybe what they did in the pedo sex dungeons on Capri and even with that Tiberius was famous for. Claudius was considered a bad emperor for his limp and spend most of his rule trying to prove everyone wrong but nobody liked him before they got Nero. Julian the Apostate was told by an advisor to stop going out of the palace to talk to random people on the street because it makes him look motherfucking ridiculous and if he keeps doing that the same people he graced with his presence will stop respecting him. I'm betting a big reason why that general tried to usurp Marcus was the fact he was a pathetic fag who didn't mind if his wife was used as a cumsock by people beneath him. His failure to properly raise Commodus would have no impact if Marcus did not ordain him as the next emperor; the precedent was for the current emperor to find a protege he would groom as his successor and Markus broke a centuries-old tradition 'cause he ironically couldn't control his emotions.
BTW. I'm sure his wife was furious when he shrugged and went "meh", being married to the most powerful man in the world and ignored like that must sting a bit.