Inversion obviously happened in that the sides flipped as to which was more pro-federalism and which was more anti-Black, but otherwise the parties' evolution was so drawn out and complex that you can't just point to any one point and say that's when it happened, you can't even just point to a party in the past and say "you'd have been this then" because the parties were alien to what they are now.
Which party in 1861 was the party of protectionism and repression of poor immigrants and religious minorities? The Republicans.
Which party in 1861 was also the "big government" (keeping in mind that big government then pretty much just meant infrastructure spending) and the pro-Black party? The Republicans.
Blacks started supporting the Democrats due to the New Deal, the Republicans took them for granted and around the Progressive Era started throwing Blacks under the bus (North and South began to unify against Black interests). That opened up an opportunity for Roosevelt to win them over, and then Johnson won their loyalty with Civil Rights while also finalizing the Democrat shifts to gommunism.