The IQ increase is probably less about work, and more about better nutrition.
Large swatches of Africa suffer from malnutrition, both micronutrients and protein deficiency.
There was a study in, I believe, South America where they had an intervention of simply adding a single egg a day to the children's diet and cognitive tests before and after. Even with that small change, the result was improved cognition because they were so malnutritioned there as well.
Turns out malnutrition stunts not only your body but also your brain. Unfortunately once past important development windows, it likely can't be changed by addressing the malnutrition that caused it.
If you look, the places with the lowest IQ are also places where malnutrition is common. Not necessarily lack of total calories, but animal foods, protein, and micronutrients.
Once the people became slaves, they would have been in situations that (while not ideal) were likely at least sufficient in protein and had fresh food with plenty of micronutrients. A couple generations of that and cognition (thus IQ) likely would have gone up a couple clicks.
I could be wrong, but the easiest way to test it would have been looking at African cultures with good diets. The Maasai would be my pick (a ranching culture, meaning access to milk and beef) but I don't know of any IQ tests on them unfortunately.