The "SafeStack" [1] scheme is probably the most well-known that does that. The "safe" stack is the normal stack, and the "unsafe" stack contains all automatic variables that could get dereferenced - including arrays that could overflow.
The security of the scheme depends on keeping the location of the safe stack secret, which for one thing means that all code in the program needs to use the SafeStack ABI.
It is used more or less in several BSD Unices (most notably HardenedBSD), and it is the default in Fuchsia (except when the program was written in Go).
CPU architectures that have register windows [2] tend to spill shifted-out registers onto a separate stack lazily, and those registers could contain the return pointer.
An interesting trick pioneered I think on OpenBSD was to encrypt the spilled stack. Even simple XOR was enough to prevent easy manipulation of stack by malware.
Register renaming is a microarchitecture feature; your OS can be completely oblivious to them because it has no way of using (or even knowing in many cases) them. Register windows, OTOH, require the OS to be aware of the “spill stack”. The difference is also in their purpose: register renaming is for OOO superscalar execution pipes to allow multiple operations at once; register windows are an attempt to avoid pushing “clobbered” registers to the stack when calling functions.
Nothing precludes an architecture from having both: register windows to avoid register clobbering and register renaming to allow dozens/hundreds of micro-ops “in flight” at the same time.
CPU architectures that have register windows [2] tend to spill shifted-out registers onto a separate stack lazily, and those registers could contain the return pointer.
[1] <https://clang.llvm.org/docs/SafeStack.html>
[2] <https://en.wikipedia.org/wiki/Register_window>