The M4 was bought as a replacement for the M3 grease gun
the M4 was originally an extension of the various carbine replacement projects to follow on the M1, M2, and M3
carbine that was used through from WW2 into Vietnam and even Grenada in some limited capacity. the idea was indeed a smaller, more portable version of the newly mass-issued M16 rifle that would maintain greater than 85% parts interchangeability. the various XM177 projects through the 60's and into the 70's to satisfy both the Air Force and the Army eventually split into the GAU-5A and GUU-5P for the Air Force (used through into the 90's) and the US Army's demands for a grenade launcher for the carbine on the XM177 series (E1, E2, et c). Colt Models 651 through 654 are all variations for the M16A1 carbine, which eventually created the XM4 project for the USMC in the early 1980's as the carbine equivalent upgrade for the M16A2 rifle. Colt eventually split the M16A2 Carbine into the M4 carbine family, and the "CAR-15" or "Colt Commando" family of weapons (famously the Colt Model 723 and 733, with export variations like the 725).
the 723 and 725 eventually were domestically produced in Canada by Diemaco as the C8, and among the last M4's were the Colt 727 for the UAE which were also purchased for US Law Enforcement and eventually the US military when the original M4 (still with burst fire held over from the M16A2 carbine project) was upgraded in the early 90's to the M4A1 which pioneered the 1913 Picatinny Accessory rail and replaced the burst-fire option with full auto like the original M16A1.
the Mk 18 has largely replaced the old CAR-15 and various XM177, GUU-5P, Colt 6xx and 7xx carbines, and SOPMOD M4's in inventory, rotating several into other departments or agencies. the vast majority of the M3 SMG were given away to allies like the Philippines or Singapore as the US military adopted the HK MP5 for quite a while as the preferred SMG, even with the US Navy and USMC having specialized versions just for them (the "Navy" trigger group) - the US Air Force opted to retain the GUU-5P instead of issue an SMG, and the US Army agreed with the Air Force, opting to issue XM177 and CAR-15 carbines instead of SMGs, or just hand out a pistol if you weren't cool enough. this left the M4 and M4A1 for SF, SOF, and sometimes mechanized or light infantry and USMC squad leaders.
for a while even into the 2000s, a decade or more after the M4A1 was adopted, it was rarely issued as a general weapon outside of units like paratroopers or air assaulters. i had an M4 with no optic, a carry handle, and the standard handguards (and 4+1 mags) issued to me during the 03 Iraqi Freedom invasion, going over the border in an M2A2. Bought a MARS sight at the BX in Kuwait the previous month and slapped it on there. Good times.