[Bug] Dedicated Servers not responding to every hit

So far used 2 different providers: G-Portal, and Low.MS. Each Time 3 of us playing on the same open world, building in our own spots. After a while of building (and we do love our building) we notice as we travel the Frame Rates drop in certain areas, and not always where we have built anything. Go to fell a few trees, and they almost hover, falling VERY slow. Hit the ground and bounce around like a rubber ball. Make any sort of action (picking up fiber, hitting rocks, etc...) and it doesn't recognize it for the first 1-3 hits/pulls. then it might catch up, or just realize you are doing something.

We have rebooted servers, added more memory, CPU threads, doesn't matter. Still happens. Gets so that after 1-2 months it's virtually unplayable.