I would use pay-as-you-go model which lets me scale as and when required and pay accordingly.
The traditional problem with Home grown software is you can't accurately predict the growth when you open the floodgates. So choosing a pay-as-you-go model works perfectly here.
The other constraint I see with Open source software is most of the them expect a experienced user and clearly lack an established support system which you can get in a paid software.
I would be more helpful if I understand the details in depth and happy to discuss that over a call.

It depends on the following two questions:

1) How good is your in-house team at compliance and enforcement? Affiliate is very much still the Wild West. This means that there will always be some affiliates either stealing commissions from other affiliates in your program, doing things that may put your company in legal jeopardy with the Feds, or cannibalizing your other channels by stealing credit for sales they didn't drive thus impacting your ROAS.

2) How extensive is your network of potential affiliate contacts; and if it is not extensive, how compelling is your offer/brand/niche to affiliate partners?

If you have time and expertise for deal with compliance and if you feel you have a big enough pool of potential affiliate partners; then I would go with a paid established software like DirectTrack. If not, I would go with an affiliate network like ShareASale (mentioned by one of the other commenters) or ImpactRadius (which has a lot of flexibility).

Regardless, I would not go with an Open Source software solution unless your dev’s have time on their hands. Whatever you might save in network fees or software fees you will lose 2x in ongoing dev costs and opportunity costs.

If the site is wordpress there are a couple of plugins for 1 time fees that are plenty good.