We know the rate of mesothelioma in people with no known occupational exposure, but we have very little quantitative data about how much a given amount of asbestos exposure might increase one's risk. The disease takes decades to develop after exposure.
I think the point that are trying to make is that if it was more common, it would be noticeable and more quantifiable. Otherwise, most of the cases seems to have been tracked to industrial exposure.
Deducing that something is rare from limited evidence is really easy. If there is no good evidence regarding something, that default position should be that the thing is rare.
The cliche is wrong. The absence of evidence is evidence of absence. Not conclusive evidence.
There are an infinite number of things that don't exist and that we don't have any evidence of. It is nice if we can positively disprove them, but that is not necessary to justify disbelief.