You're not the only one, same shit happens to me too
I believe it's the drifting mechanic. Over the years it's been heavily tweaked to where it's pretty much perfect in Mario Karts 7 and 8. The drifting in 64 feels a bit wonky where it's like you're really not in complete control of your turns like you think you are, so you're constantly fighting with it to make sure you're not going off track, especially on those wide turns. Also doesn't help that karts aren't customizeable and each character has a unique stat specific to them, so if you're a Bowser player (like i am), you're basically playing on the hardest difficulty all the time. At least with 7/8, you can make bowser easier to handle with the different kart parts (or harder for youreself if you're a masochist)