Spectators not possible in coop?

I've set spectators to be allowed in coop, even start as spectator... but its not working?
How can I do this, is there another way? Besides allowing people to switch teams. Which in coop makes no sense because it lets people go on the enemy team too.

Basically my server limit is at 10. I would like to allow up to 5 spectators as well who are not playing but can switch to join the match when room is available and someone playing leaves.

Or, a join queue option of sorts.

My servers are full several times a day and people in the community keep complaining they can't get in ... 5 times today this happened