This is a bug that has been reported to Minecraft and fixed in the 1.18 snapshots but I think should be addressed in paper for previous versions. https://www.youtube.com/watch?v=v1yO0EFwUJg
Which version and build are you on? Might be related to this issue, which was fixed in 1.16.5 build 757.
I have tested in 1.16.5, 1.17, and 1.17.1 on paper and vinella and it still works in all of them even on bedrock and pocket edition
I’m having some trouble reproducing this, do you happen to have have a world download or schematic of the villager breeder you’re using?
I just went back and tested in 1.16.5 paper and it is indeed patched but still works on 1.17.1 paper and all it takes is 2 farmer villagers some beds a few stacks of bread and a few day night cycles.