I've found diving to be the least demanding "action sport" activities that I've tried, and I've done a fair bit of climbing, diving and flying gliders. Careful and well trained open water diving leaves you with plenty of safety margins in all directions. There's (well, almost) no single thing you can really screw up that would be deadly.

But of course, if the whole idea of something going wrong will leave you lying awake the night before, it's probably not worth it :)