I think before they started recognizing them as cars some states basically classed them as farm or offroad vehicles similar to side by sides.
I believe they were classed as "heavy motorbikes" for the few years they were legal for US sale in the late 60s and early 70s, prior to the ban.
(Which they got away with due to having sub 1-litre powerplants, the BIGGEST one available for the OG Subaru 360 was only 423 cc By comparison? A "small" modern car/engine is something like, say, 1.8L (1,800cc) that's how tiny we're talking)
And safety of the driver was just the public excuse, the real safety issue was those tiny engines and lack of a high enough final drive gear in the transmissions made them incapable of sustained highway speeds, as they were never designed with typical American driving in mind, in either speed, frequency or conditions.
They were designed to drive in flat, slow, narrow streets, and that was it.
On any US freeway, it was a question of when, not if, they'd get catastrophically rear ended as they'd always be doing 10-20 mph below the flow of traffic.
Grades were an issue too, they were more liable than not to stall out just trying to go up a moderately sized hill.