I think it would be nice to have faster secondary draw in some situations. In 'emergency' situations if you go for secondary you would just drop the primary to hang on sling and quickly reach for pistol.
Here is my suggestion:
By double tapping key for secondary weapons you would perform fast draw that would be significantly faster than regular draw.
You need to have quick draw attachment on your handgun and it only works if you are standing (no crouch or prowl position).
Not saying it should be the fastest possible time which would be well under 1 second, but just above 1 second or whatever you feel is right... Basically removing the time needed to put down your rifle from current animation as it happens simultaneously with pistol drawing.
It would also give quick draw attachement more purpose. It could still perform the same way it does now, just with that bonus of standing fast draw.