We’re extremely pleased with Talos. Much more secure than Azure (our cloud of choice, unfortunately) which run a full-blown Ubuntu underneath. We haven’t run into any issues with Talos and upgrading is super easy with the talosctl tool, both Kubernetes and Talos version.
We currently have a thanos instance in each cluster. We could move it to a separate cluster to reduce some overhead, but the current approach works. We’re ingesting about 60Gi per day of metrics into the S3 bucket, so we might have to optimise that.
We've been using Talos for our internal clusters for a while, but with quite small ones (3 kube node, 5 worker nodes).
Upgrading has been generally a non event, and we're quite happy with them.
How do you deploy Thanos ? In one of the clusters ?