What's happening is your traffic is going from device > router > ISP > router > server. NAT Loopback would cut out the ISP component because your router should already know that you're trying to reach the WAN side of it.
If you wanted to get a WAN IP for your server, I think you could just stick a switch between your router and modem, and then hook your server to that switch. Worked that way for me on Spectrum at least. If you have two NICs you could have a local IP and a WAN IP. But I am pretty sure even in this set up you'd have to route through the ISP and be restricted by your WAN speed. NAT Loopback is going to be the solution to the bandwidth issue you're facing.