I'm mostly indifferent to the actual cartoons- Like any media, some is OK, some is cancer (although, anime is more cancer-heavy than most media; Apart from straight-up CP, I don't know of any other commercial medium that has a whole, out-in-the-open sub-genre depicting adults banging children, or brothers banging their sisters).
But it's the fandom that's dogshit, and causes the "hate". The fact people treat watching anime as if it's a substitute for having a personality, in a way you don't get for any other genre/medium. The fact that weebs are 90% adult men, who fixate on shows made for 13yo Japanese girls. The way they think that makes them quirky and special (rather than sad and embarrassing). The fact that 99% of weebs will actively defend the pedos in their midst, to the point of giving pedophilia the cutesy euphemism of "Lolicon". The fact that I've never seen a single weeb raising concerns over the fact that they have actual, underage children in the same fandom where comics/videos/drawings depicting adults having sex with children are openly sold at every convention, or about the fact these CP videos are listed on non-age restricted websites frequented by kids like MAL, alongside Pokemon, Sailor Moon, and One Piece listings. (at best, I've seen a tiny minority of the fandom reluctantly admit that they don't really like this stuff, but only ever in response to some OUTSIDER raising the issue; But most will actively defend it, and even those who say they don't like it, will happily just ignore the CP/pedo problem all around them, as long as no-one else brings it up).
Anime fandom has the exact same issue with pedophilia, that furries have with dog-fucking; Even those members of the fandom who aren't active degenerates themselves, never want to raise such an awkward, fucked-up issue in their super-fun, super-cute hobby, so they just stay quiet, and a few years down the track, you've created a total haven for dog-fuckers/pedos, with the established norm of everyone just accepting the degeneracy, and denying the issue to outsiders (and to themselves).
And also in common with furries, the fandom is just a natural magnet for autistic loners, cringe behaviour, sexual degenerates, and deeply unattractive people with no self-awareness, thinking they're being all "cute" and "fun" and "quirky", when really they're just being embarrassing, annoying, and/or disgusting. A bunch of cutesy gifs, or memes about "head-pats" or "lewding the loli", or weird, cat-face emoticons, don't make a bunch of obese 37yo men suddenly seem cute and kawaii- It just makes a bunch of obese 37yo men acting retarded, larping as 12yo Jap girls (There's a reason why Chris made it onto the thumbnail of the Filthy Frank 'Weeaboos" song; He really is the living incarnation of the Western anime fan)
If you want to watch anime, go for it. No-one cares. Same as no-one cares if you're into horror movies, or rap music, or french literature. But it's the way that weebs tend to act like anime is THE defining aspect of their personality, or like it has some fundamental, pseudo-religious importance to them, and is some core part of their identity, that's why people "hate anime". It's not the shows that are the issue (with the exception of the "Loli hentai"/CP stuff); it's people's ATTITUDE towards the shows, and bizarre, emotional over-investment, that causes the "hate". If you treat it the same way as any other movie/tv show/whatever (ie watch what you want, and STFU about it), then no-one will have a problem