Players keep dropping from my server

Hello, this is a weird problem. I have a small server at home and at peaks there is 10 players online. In such situation it is quite common that most of the players keep getting disconnected every few minutes or so! Some have it worse then others and some are totally fine.
The server is powerful enough with core i3 -10100 working at 4GHz and TPS never goes under 20! CPU load is around 20% so I don’t think this is a bottleneck.
It seems that when there is less players it is more stable even for the most problematic ones.
I asked them to make an internet speed test and they usually have ping around 15ms and download speed 20-50MBps, so that looks absolutely fine. They also claim that they don’t get disconnected like this from other Minecraft servers.

So next possible cause can be my ISP logically, but I don’t know how to check this, because all tests show that my connection is really good! It’s a cable modem connection and downloads reach 200Mbps! Upload is capped at 10Mbps and it also shows in speedtests.
When there is a rush hour on my Minecraft server I can see in task manager that upload speed is around 2Mbps, so it never even gets close to those 10Mbps.
Ping is super stable as well because I let it ping some external website and it never seem to drop packets. Jitter is neglibible

So the question is - could there be some other metric like how many concurrent connections can be handled steadily which could be a problem for my connection even though it looks super fast? What is your experience with hosting a Minecraft server on a consumer grade connection?

so right now 4 players on and still some of them are disconnected every few minutes even though there is no load
not sure why timings is above 100% but someone on Discord said it is not relevant on Windows server

news: so I have moved my server to a friend’s place and now it’s connected via O2 instead of my Vodafone. So far it looks like the situation improved and I saw only a handful player drops today…