I think it's because the ecosystems are quite different. In x86 processors you can swap them in isolation and keep the same connectivity standards, the same types of RAM, the same operating systems and programs. In phones you can't just buy an Apple processor and drop it in, you have to "sign up" for a whole ecosystem of particular devices, a single OS that runs on those devices, a single software store that serves that OS, a particular connector, etc.. A direct CPU-to-CPU comparison doesn't hold as much weight when you can't actually act upon it without changing a whole host of other stuff you might be perfectly happy with.