Give armor higher weight.
When you wear armor your ADS speed slows down. It makes some realistic sense and even more gameplay balance sense. If you take more time to be killed, you need more time to kill others. Simple as that.