The way I see it, it'll just be common knowledge that audio can be easily fabricated, and so it won't be taken seriously as evidence. People are able to make convincing Photoshops now that weren't possible decades ago, and that tricks people every now and then, but it doesn't have massive repercussions because people are aware that the tool exists.
I also wouldn't be surprised if there are people developing tools to detect AI-generated audio at a similar pace to the people developing tools to create it. But my guess is that arms race will end in favor of AI stuff becoming more and more believable.
The problem will be that genuine audio and video evidence can no longer be trusted, rather than that deepfakes will doom people.