Checks are a relevant example here; did you know it doesn't have to be a piece of paper issued by your bank? Anything can be a check, if you have the correct account details written on it (although
the story about the live cow is fiction). More importantly, you are able--or used to be able--to call up a bank and do transfers by voice, using the account details and verifying information like your birthdate.
A lot of the "don't give out your bank details" is actually good advice from the days before the Internet, when these things were more common. But many of these avenues have been closed now that most check clearing and fund transfer is automated. If you have to log in to your bank website to start a fund transfer, the security is moved to that login. If you walk into a bank with only account details, it's easy for someone to request ID and pull up a database of customer info to confirm it's fraud. If you try to deposit a non-standard check through an ATM, it can detect it, plus you're on camera making the attempt.
It's still good advice not to give out bank info, for the same reason you don't give out your SSN: many places still use it as a pseudo-secret PIN to confirm the person they're talking to is genuine.