Let a resource have one hash and potentially many source domains. Define a CSP whitelist consisting of trusted domains. This list is applied to resource loading in the browser, and it is also applied to cached resources in the following way:

   cache = {
	b0af301e782bf5e2a8ccce919b6ca3b70aa771db: 
        {domains: ['evil.com','airbnb.com'], content: '...'},
	35d778783c4155c20360d269c9dd000fdcd39548: 
        {domains: ['javajosh.com'], content:'...'}
   }

You go to secure.com, but a malicious user has put the b0af301 script in your path. CSP's white list for secure.com is [secure.com, javajosh.com]. The browser dereferences the hash, checks against the associated domains, and rejects if a whitelisted domain isn't in that list. Your browser running secure.com would reject the b0af301 script.

(Something I personally would like would be for for orgs like EFF.org to post known-good hashes, so I can always add the EFF hashes to my site's CSP whitelist, and have a warm-and-fuzzy feeling.)