"Oh, so I'm a pedophile if I think Jotaro Kujo is hot?"
As you point out, that's a massive red herring cope. The issue isn't that characters who look, walk, talk, and behave as an adult are "tEcHnIcAlLy" under the age of majority. The issue is that characters who look, walk, talk, and behave as children are the objects of their sexual desires. When Ethan Klein said an anime character who looks like an adult was attractive in their depiction, the only people who cared were lolicons who stubbornly tried to throw it in his face (and no one cared because it's a retarded argument). It's a logical non-starter.
Drawings are representations of archetypes and symbols onto which we project significance. Sourcing your sexual arousal from the archetypical representation of a child is pedophilia. The fact they need to raise points which are immaterial to this central issue reveals, to me at least, that they know this and they know they have no answer. They delude themselves to avoid the truth because being a lolicon who "doesn't identify as a pedophile" is the epitome of having their cake and eating it too in this context despite its untenability as a logical premise.
All that babbling aside, there's a very simple test anyone can employ to prove that these lolicons are pedophiles: If a man masturbates to a photograph of a woman, or separately, a drawing of a woman, then we can deduce that he derives sexual pleasure from women. He would be straight.
If a man masturbates to a photograph of a man, or separately, a drawing of a man, then we can deduce that he derives sexual pleasure from men. He would be gay.
If a man masturbates to both photographs of men and women, or separately, drawings of men and women, then we can deduce that he derives sexual pleasure from both men and women. He would be bisexual.
If a man masturbates to a photograph of a dog, or separately, a drawing of a dog, then we can deduce that he derives sexual pleasure from dogs. He would be a zoophile.
Lolicons will instantly agree with all of the above, but then will seethe at the following continuation of the logic: Therefore, if a man masturbates to a photograph of a child, or separately, a drawing of a child, then we can deduce that he derives sexual pleasure from children. He would be a pedophile. Case closed!
The entire question of the "lore" of the anime is wholly irrelevant because their sexual arousal comes from the immediately evident visual information present in their pornography of choice, that being lolicon, and not the character's tertiary personal details which are out of sight and out of mind.