Dedicated Server Join Friend Without Password Support

It would be nice if there was a permission level for dedicated servers that would allow friends to join the server while you are online. I'd love to use my dedicated server to play with friends, but don't want them to end up using it when I'm not around and then have to kick them off to switch to whatever activity I want to do. Having an ability for friends to join me on the server without having to enter a password, while still protecting the server with a password for initial connection would be a great quality of life improvement.