And as the rest of your post states, being armed can make you seem like more of a threat to the other person you're confronting. THAT is where, in fact and in law, your GUN RIGHTS will serve to limit your FIRST AMENDMENT RIGHTS, or, at your option, you can have your full 100% set of gun rights, but then you better what what you say, and thus your 1st Amendment free speech rights have taken a hit.
If I'm not armed, I can come up to you and get in your face and insult you with much stronger language, even wishing that you would die, than I could legally get away with if I did the same thing while openly carrying a gun. Especially a long gun that I'm holding in my hands.
Hate + Insults + Threats + Weapon = reasonable fear of imminent mortal danger.