That got me curious as to why the Federation was there in the first place, so I went and reviewed the first episode of DS9. The reason the Federation is present in the first place is that the Bajoran government asked them to provide relief efforts after the Cardassians withdrew as stated below.
01X01-02 - Emmisary-1.mp4
Then after the discovery and stabilization of the wormhole the Federation decides to capitalize on it by establishing a more permanent presence at Bajor.
01X01-02 - Emmisary-1(1).mp4
It seems that if the wormhole wasn't present, the Federation would have provided aid the the Bajoran government, then left after the situation on Bajor had stabilized. However, with the discovery of the wormhole the Federation wanted a strong as possible presence near the wormhole to control passage through it. This included inducting Bajor as a member of the Federation.
So why does the Feds tolerate the religious fanatics? The same reason the U.S dealt with the various tribes in the Middle East for control of strategic resources. Though I think the Federation came out better than the U.S did in the long run.
I kind of saw the Bajorans as more of a Space Panama than Space Israel.