I've tested this out, it looks like my FPS only dips when I'm looking at human players. The problem could have something to do with the way the game updates player positions.

I get perfect FPS when I start a private game with bots.

I've also noticed that there's a 1 second freeze when I bring up the map. The game turns into a slideshow if I keep on pressing TAB.