You need an abstraction over different hardware implementations to have a standard API. Abstraction comes with a cost. And 3D graphics applications want to utilize hardware at its fullest capability. So basically it's not a simple problem. Especially considering that hardware is constantly evolving, inventing new approaches. For example I learned OpenGL about 15 years ago and I never used any shaders. Now, AFAIK, shaders are everywhere.
Probably the best thing you could do at this point is to use game engine like Unity or UE as an abstraction.
Meanwhile we have been compiling the same C and above code for mips, arm, power, x86, and riscv for decades in portable ways while still getting performant code.
Now to be fair GPUs have a legacy of slowly becoming generic processors whereas 10+ years ago they were largely fixed function hardware. Writing generic code for a GPU today is totally sensible because it supports most arithmetic, primitive types, branching, etc.
But there is nothing stopping you from optimizing your compiler for glsl any more than optimizing your platform specific custom vendor compiler for C.
The problem here is that abstraction comes at a cost, and that cost is so great that we need even crazier abstractions to be able to work around it... and now the suggestion is, as you say, work with a whole engine. Ah, now I can draw a pixel on screen! I see it as a lot of wasted potential (even if maybe not that much wasted commercial potential).
Probably the best thing you could do at this point is to use game engine like Unity or UE as an abstraction.