As long as most people run with mitigations on, you're technically probably indeed safe. But you should still care that things get fixed with mitigations=on otherwise you wouldn't have the shield of "almost everyone has mitigations enabled for this so noone has reason to bother exploiting this"!
