I don't think Swift is particularly mature in this area yet, and it's concurrency depends upon Grand Central Dispatch. Go on the other hand is rather mature, especially for its age.
Other than that, it really depends on what the server is for, but in general I would say Go is probably a better choice.
If you're interested in something similar to Swift for the server, I think Kotlin w/ Spring Boot would probably be a safer bet at this point.
Swift is a much more expressive language, designed by folks that accept the world of programming paradigms in the mainstream has progressed and is sponsored by IBM for server programming.
Swift is most likely not ready for this until 3.0/SPM release, but once it'll be very competitive. Swift generics are very good and Go's are not. There's no language-level concurrency, but there's GCD and https://github.com/VeniceX/Venice