Bronn already established in the first season that armour slows you down, so the lack of it doesn't in itself break plausibility. We already know Evil Frodo thinks a little differently to everyone else.
What doesn't sit very well with me is how someone with no notable history in battle or particular training privilege would be a Darth Sidious-like swordsman. Or that such a mean-spirited little creature with rampant inadequacy issues would be able to inspire more than a half-hearted grunt in the bedroom department. And how exactly did he explain the slaughter of a patrol of his own men last season? There's a lot about that guy that makes me scratch my head.