As
@Grub said, some people don't like being compared to animals. Your particular example has context between you and your friend.
If I, an adult, would use the word "cute" to describe a child, people would rightfully look at me the wrong way as I'm using an affectionate term towards a child.
Again, the purpose of loli is to fulfill a purpose of having a carticuture of a child in a position that the artist or owner of said material desires.
How many threads exist of people trying to rationalize their depraved fetish of children because "it's art?!"