You're talking about trusting experts, which is a different thing. Doctors and judges are not selected by popular vote, but by formal exams, therefore you have a reason to trust them, even if they are wildly unpopular otherwise.