Dedicated Server unplayable with a built up map

I had an open world going for months. I built 5 or so castle like structures. Huge structures at different parts of the map. I tried uploading this map to two different hosting providers. Both had great performance on a new map. Both were unplayable with my built up map. Think 1 second delay when you open a door or do a gather action. The graphics themselves are fine. FPS is fine. It is just any interaction that I engage in is so lagged that you can't play. Same goes for hunting or fighting. You will miss every time because of the lag.