Your “infrastructure” server could be a CI server - it’s just building “my” code ignoring that many (all?) build systems allow execution of arbitrary code as part of the build process (rust, cmake, bazel, JS ecosystem, Go, etc etc) and many involve 3p dependencies. And CI servers often handle secrets to infrastructure (publishing packages, etc). So you could end up allowing a supply chain attack that reads out various API keys & whatnot.
In other words, properly drawing the boundary around “this is safe with meltdown disabled” is very hard, non-intuitive, and you’re one configuration/SW change or a violated assumption away from a Meltdown attack which is cross-process memory access & one notch below remote access. There’s a reason you design for security in depth rather than trying to carefully build a jenga tower where you’re one falling block away from total compromise.
