East Africans didn't ride elephants so far as I know. It still goes to the pure stupidity of trying to directly assign real world racial characteristics to everything in Tolkien since even when he did loosely base things on some specific race he never strictly based it on them. For instance, he somewhat openly based dwarves on Jews. However, along with a smattering of slightly Jewish stereotypes, they also have a lot of other characteristics associated with other stereotypes.
It just seems really dumb to try to fish out some kind of cryptic hidden racism in absolutely everything. Is there some shortage? People can't go pick on H.P. Lovecraft for his absolutely open racism? Or better yet, just cope with the fact some people are racists and maybe, I dunno, try to convince them otherwise or ignore them and move on?