While talking about the "damage reflection" TK penalty, I came up with an idea that might solve issues with TKers in a more realistic and immersive fashion. The problem I have with the current system is how arcadey it is; it's almost like something you'd come across in an MMO or something. For a game going heavily for immersion, this mechanic just doesn't seem to fit. I want the game to allow the teammates to solve the problem, rather than having the game directly solve the problem by having Allah himself smite the TKer for continuing to shoot teammates.
As the title suggests, the idea I came up with is a Traitor system.
Upon doing too much team damage (maybe either 60 HP after the TK damage reduction is applied or after killing a teammate regardless of HP; also, fire support and explosive weapons might need differing modifiers) the player in question is marked as a "traitor". The mark itself is simple; the teammate indicator/icon's color is switched to the color of the hostile faction. The mark only lasts for a short period of time; maybe 10-15 seconds. The automatic kick after three quick TKs would still be in effect.
If a player is marked as a traitor, his teammates can shoot and kill him (within that 10-15 second window) with no friendly-fire damage reduction or TK penalty (i.e. killing him doesn't count towards the three-TK autokick). There is no extra reward or incentive for this, however, apart from possibly terminating a toxic player. The player marked as a traitor still has the steep friendly-fire damage reduction when shooting teammates, but can otherwise continue as normal. Gameplay is not affected for the player convicted of being a traitor (unless his teammates decide to take him out, obviously).
Essentially, the traitor is marked as an enemy to his teammates, but the traitor's teammates are still marked as friendlies to him.
In the event of an accidental friendly-fire or teamkill incident, the traitor would have to plead for his life. There still wouldn't be much of a point for his team to choose to kill another teammate unless the traitor is intentionally teamkilling, trolling, etc.
To add some icing on the cake, a few extra voice lines could really sell the betrayal:
Ex. Directly after a teammate shoots a friendly Insurgent and is marked as a traitor: "Backstabber! I'll take you down, brother!"
A tagline for killing a traitor would be pretty neat, though. Actually, no, since that's kind of a reward.
EDIT2: Another possible mechanic is if a player accidentally shoots a teammate, but doesn't kill him, the teammate has a grace period of a few seconds to shoot and kill the teammate that shot him first with no penalty. The friendly-fire damage reduction would still be in effect for both parties the whole time.
The idea behind this is if a player shoots a teammate and the teammate kills said player with a twitch shot or something, they aren't penalized for teamkilling as they were shot first. Granted, this could create awkward situations where a player might fire at a teammate, miss, and then the teammate shoots them back and kills them, as then the teammate would be marked as a traitor. Accounting for teammate suppression might alleviate this problem.
The whole "traitor" system would then be in effect if a player actually kills said teammate.