I'd say that applies more to Doffy's crew than himself. Doffy was a beast, yes, he folded pretty quick to gear 4, but he took a lot of punishment to reach that point. The mingo pirates on the other hand, outside of Sugar, were not particularly threatening.
A lot of them you just see it in their design, they are old and fat and got complacent in their status quo. All of Baroque works felt like a threat in Alabasta, in Dresrosa only Mingo was an actual threat. That probably adds to why the arc feels like it drags, Mingo is doing a lot of heavy lifting while his crew is taken out by "nobodies".