All of the server backends at my company are written in Go. This was a result of me writing a couple servers in Python a few years back, ending up with lots of problems related to hanging connections, timeouts, etc. I tried a couple different server libraries on Python but they all seemed to struggle with even tiny loads. Not sure what was up with that, but ultimately I gave Go a swing, having heard that it was good for server applications, and I haven't looked back. It has been bullet proof from day one and I am overall happy with the development experience.
That was the good. The bad? Garbage collection, dependency management, and lack of first-tier support in various libraries. Garbage collection makes the otherwise lightweight and speedy language a memory hog under heavy loads. Not too bad, but I have to kick the memory up on my servers. Dependency management is a nightmare; honestly the worst part about it. The lack of first-tier support in various libraries is a close second. AWS's API libraries had relentless, undocumented breaking changes when we were using them, all on the master branch of their one repo (breaking Golang's guidelines for dependencies). Google itself doesn't actually have any real API libraries for their cloud services. They autogenerate all API libraries for golang, which means they're not idiomatic, are convoluted to use, and the documentation is a jungle.
We continue to use Go because of its strengths, but it just really surprises me how little Google seems to care about the language and ecosystem.
We run a cluster of P2P, GPU heavy machines that use Go to ingest byte streams of raw radar data, store that info in btrees, and render, cache & serve map tiles that are drawn on the fly in response to http requests. We are not using much outside the stdlib (opengl and gdal bindings).
Garbage collection has become very fast in recent versions of the language. It's quite painless now, and we did struggle with it in the past. I'd take the current GC over dealing with other strategies unless you are a very specific, >60fps kind of situation. We literally don't think about the GC anymore.
Random stats from one of our busier nodes, over the past 20 min:
40GB currently in use, 391GB allocated (and freed) in the last 20min
average of 2.6ms GC pauses every 70.6s
95th percentile on GC time is 7.26ms
For dependency management, there's a lot of tools that make things more automated and I think probably offer people what they are expecting/used to from other environments. This has been a sticky point for sure. I've settled on a bare bones solution that seems to work well: just using git submodules in the vendor directory within the project repo. Does everything I need (pin to version of a fork, build immediately after a clone and a submodule update --init). As of Go1.5, the vendor directory is supported by the compiler, and it will look there first to resolve dependencies. While this does work with recursive dependencies, you will want to flatten the dependency graph as much as possible if types are being used across multiple deps because types are distinguished by their full import paths. This encourages breaking things into modules so I'm ok with it.
First-tier support is a problem I have no doubt, but not one we've suffered from personally.
I'd say that debugging comes up a lot too, and Go doesn't have an official, reliable option for debugging. The Delve project is getting close to that though it looks like. I personally debug with the stdlib pprof package (provides an http server that prints stack traces for all goroutines currently active, allows cpu/memory/blocking profiles to be requested), print statements, general testing/experimentation and newrelic.
I don't have great measurements for this, but we have done optimizations to reduce the data flowing over the Go<->C interface. One of our key measurements that does make a big impact on performance is how often we need to upload data (that's not already over there) to the GPUs. So that's something we have worked on reducing (buffer reuse, compression). We also have a series of caches on the other side, so we aren't drawing more than we need to. It's hard for me to tease apart how much of these optimizations (and others) are ultimately aimed at addressing the cgo overhead, and how many are just typical stuff. The data we work with is cumbersome and my intuition is that there's probably a lot of room for optimization in our drawing even still, regardless of cgo. I wouldn't be surprised if a direct port C/C++ implementation of the rendering pipeline was significantly faster than ours in getting data into and out of the GPUs, but a big part of the project is data storage/networking/serving/caching as well and Go has bridged the gap for us (a small team that needs to build reasonably fast things reasonably quickly :)).
That's interesting. The cgo overhead was the only thing holding me back from considering it for games, since I didn't want to write a lot of C wrappers around the C libraries I want to use just to have them be more efficient, which is a shame, since Go is pretty nice, barring the C interop in some cases.
When I used to frequent gonuts, I raised the issue why they didn't went the FFI way as D, Rust, .NET, Delphi, FreePascal do, but sadly they rather use cgo as solution.
Yes, that is the purpose of the vendor directory. You copy the source of each of your dependencies into vendor/ and commit it to your own source control so it can't change unless you do so yourself. I personally use git submodules to manage the contents of the vendor/ tree, but the go compiler/toolchain itself doesn't really care how you get the files into there.
Edit: other comments below have mentioned some of the tools available to manage your vendor/ tree. Using git submodules manually can be crude, especially when adding dependencies that themselves have other dependencies.
We've been running splice.com on Go for 3 years now and handle 5TB of audio/binary data per day. Our memory usage is around 10-15MB per server and the GC pause time has been really low. You do need to stream your IOs instead of reading everything in memory. In regards to dependency management, we honestly had no issues and now with vendoring, it's even easier. We do use a main repo with lots of smaller packages and only a few 3rd party dependencies that are vendored or available via private git repos.
We don't use Google cloud but I heard they have 2 repos, one that has auto generated code and one that has hand written code (but less complete).
yes buffered io, you use readers to read/write small chunks at a time instead of loading everything at once. Go offers way to do both since in some cases, such as loading an entire file in memory can be fine/better/faster.
> We continue to use Go because of its strengths, but it just really surprises me how little Google seems to care about the language and ecosystem.
Go is certainly a language that is used at Google, but AFAIK a lot of "Googlers" don't really like it and don't use it. It certainly not the "official language at Google", given the weight of C++ and Java there. But that's the consequence of being opinionated. Using Go means having Rob Pike over your shoulder telling you how to write code. And he made sure you can't escape that fact since there is no place for "ninja coding" with Go.
Erm. StackOverflow was built by folk who used C# heavily, and cargo-culting is amazingly prevalent on both the C# and Java "traditional IT" communities, so _of course_ StackOverflow is biased towards those runtimes...
This slideshow clinches it for me. Go has some specific strengths that match low level, network infra types of problems. Beyond that, Go is not a good fit. This sounds like a criticism, but I don't think so. It's a compliment: it's a sharp tool for a specific kind of cutting. It's not trying to be some all-singing all-dancing language, which has gotten us all in to quite a bit of trouble.
I think it refers to the Perl-era idea that good code should be somehow "clever" rather than maintainable. It's how bad programmers who spend hours agonizing over how to reduce their line count (presumably to save disk space?) justify their behavior.
Personally, I think "cowboy coding" is the best derogatory name for this. IMO, what all of these ninjas have in common is that they don't realize that software development is a team exercise. Berkeley did a study on BSD and found that a file was opened 10x more often for reading that writing (i.e. people read code 10 times for every time they make a change). To my mind, "cowboy" conveys the proper amount of ignorance of the other people on your "team."
"Cowboy coding" already has a distinct meaning - it refers to writing code as fast as possible without concerns for technical debt.
The archetypal form of cowbody coding is the copy-and-paste: faster than any code reuse technique, but a booby trap for the future.
"Ninja coding", on the other side, refers to coding for cleverness' sake, at the cost of legibility and ease of use. No self-respecting ninja would simply copy-and-paste.
I totally agree with this sentiment. I think it's pretty well established at this point that in many cases clarity can trump optimization. Even more so when the optimization isn't performance based but rather line count based or "code golf" based. Also, do you happen to have a link to that berkley study? I can't seem to find it with a brief search.
Or, it is how some programmers continue to find motivation and fulfillment in the activity as an intellectual pursuit, yes at the cost of maintainability, but as opposed to being good little corporate drones writing a zillion dull lines of kindergarten-obvious code in Golang and Java that even the lowest decile of programming dunce can understand. Which is what Golang is designed to do. That's what they mean by optimization for large code bases: dumbing down. That's not a criticism by the way because that's optimal for large teams in large companies with large code bases and no room for too much creativity. But programming as art/fun, where appropriate, is also not (yet) completely to be dismissed as stupid, IMO.
For optimal code maintainability, there is a happy compromise between excessively terse and excessively verbose code. There is a lot of code in the wild that's too spread out.
Well sure, but erring on the side of verbosity at least ensures that someone can follow your thinking as long as the code is well structured. I would prefer to read 25 lines of decent Python over 3 lines of regexes in Perl.
Sure, but at least I would prefer 25 lines of decent Python over 125 lines of overly verbose Java. Nothing is black and white, it is easy to make code too verbose and thus very taxing for the reader.
Have a look at Elixir and Phoenix. GC is per-process so no global slowdown. Extremely fast response times and uptimes. Many other features that mesh nicely with webserving (some courtesy of Erlang's VM).
And a completely opposite philosophy from Go when it comes to error handling. Erlang/Elixir embraces failure (and immediately logs and restarts the process); Go seems to ignore it (unless you explicitly check, which to me seems... insane. For reasons why failing fast is better than failing silently, read https://blog.codinghorror.com/whats-worse-than-crashing/ .)
But you still have to check for errors after every line of code where they are possible (if you're covering all the bases), no? Due to lack of a traditional exception model?
Elixir/Erlang at least have pattern-matching which makes checks like that (from functions that return a non-OK value on errors) basically inlined:
`{:ok, val} = call_some_func(with_args)`
If it doesn't match on the :ok (i.e., an error occurred), you get a match error which gets logged, and the process typically gets killed and restarted by a supervisor process in a millisecond.
Is it that bad now in 1.6 with vendor support? And a tool like `govendor` makes it easy to stick things inside of vendor.
> it just really surprises me how little Google seems to care about the language and ecosystem
To be blunt, google's priority is google, not the open source community or other companies using golang. Dependency management wasn't a priority because of mono repo. Having said that, they are pretty good about improving things for everyone, but it will never be like a company such as Typesafe whose product is the language and tooling itself.
I switched to Glide, a vendor-focused package manager, on 1.5. I think the community has already decided in favor of this approach. Post 1.5, Godeps' workflow just feels weird and unnecessary.
To be fair, Go's GC got A LOT better over the past few versions and it's getting much better still.
What I would really like to be able to do though is be able to write unmanaged blocks when I need them and know better than the compiler rather than having to "run my own heap" on top of a buffer like I typically have to do in managed languages that don't have that opt out.
That is, you do this if you want a block to run without the GC interrupting it. That is, it's a really high-priority (realtime, or close) block. When it needs to run, it's the most important use of the CPU. But if you demand that the GC makes progress, you're saying that it also is important. And it is, but it's less important than the critical block. If you don't say that, then you wind up with "everything is important", and that way lies madness.
But you may be able to do it. You need enough CPU bandwidth that you can run your critical sections and spend enough time outside them that the garbage collector keeps up. (And then, every time you add to your code, you need to make sure that the CPU still has enough time...)
What does it mean for a GC to "make progress"? My understanding was that the OP wanted to stop the GC for a particular code block, so it would seem that any "progress making" would be undesirable.
> Google itself doesn't actually have any real API libraries for their cloud services. They autogenerate all API libraries for golang, which means they're not idiomatic, are convoluted to use, and the documentation is a jungle.
Are you sure? Google offers 2 Google Cloud APIs for Go:
- Google API Client Library for Go, which is auto-generated like you wrote;
- Google Cloud Platform Client Library for Go, which is intended to be idiomatically-designed.
> They autogenerate all API libraries for golang, which means they're not idiomatic, are convoluted to use, and the documentation is a jungle.
Are idiomatic API libraries really a good thing? If an API is used heavily throughout your application, doesn't that mean it's time to wrap that part of the API in some idiomatic wrapper code that is meaningful in your domain?
Functions are idiomatic in all languages. Personally, I'd rather APIs mostly stick to simple functions, and let the application developer build idioms around them if they like. And if the API call involves a web request, I am happy to just make the HTTP call myself. Half the time you need to understand the interchange format to debug your code anyway. My experience is that wrappers rarely protect you from that.
I guess if your application is 90% calls to an API, and you use a huge range of different functions in that API then you'd want a good, complete library. But how common is that?
That's right. It's production-quality, but the API surface might change.
If you're OK with changing your code sometime in the future, then I'd recommend giving it a try for this or your next project. Changes will likely be minimal.
Functions are idiomatic in all languages. Personally, I'd rather APIs mostly stick to simple functions, and let the application developer build idioms around them if they like.
Idiomatic APIs can be a real PITA, especially when you call them from a different language. KISS should reign supreme. (With bad 80's special effects in a made for TV movie filmed at a carnival.) If you leave it up to the client developer to build the idiomatic facade/interface, then the API is simpler, the code is more properly idiomatic in the end, and everyone is happier.
Just a quick note about dependency management: we use "go get" to download / update dependencies, integrate / test, and then just include them all in the project's git repo. Never had an issue.
Another major issue I have had with Go is database connectivity. The db drivers are really lackijg for Go. Makes it tough for those of us who use Teradata or Hive.
Source: I've written many servers in both C and Java and given the choice will never use a GC language again for a server. Holding out some hope for Rust..
There are definitely some pain points building a server in Rust, but I think on the whole the tradeoff is worth it, and will become moreso as the ecosystem matures.
Presumably he is complaining about the GC implementation's characteristics, not its existence. You can read a description of the team's design decisions here: https://blog.golang.org/go15gc but a simple summary would be: memory is cheap and getting cheaper, so focus on making GC fast rather than small memory footprints.
And in my opinion, this is a great tradeoff. Ultimately, adding more RAM to a server (or clicking a button on the AWS console) is a very easy fix. However, reducing latency is almost never easy. If they can prevent GC stuttering, I'll take the higher memory overhead every time.
> I tried a couple different server libraries on Python but they all seemed to struggle with even tiny loads. Not sure what was up with that
Could the problem have been the "global interpreter lock"? The GIL makes it impossible for multiple threads to simultaneously perform certain basic operations, effectively making the system single-threaded.
Slide 13 (https://talks.golang.org/2016/applicative.slide#13) is interesting. I was expecting Go to be very close to C/C++ on the X axis (fast/efficient) as it doesn't use VM, but it is more close to Java ?
Ideally, Go should be pretty close to C, but currently there are reasons for it lagging behind (though there are also Go programs which perform better than their C counterparts):
- The Go compiler certainly isn't as sophisticated as the better C compilers. It looks that the code produces by Go 1.7 is quite improves vs. 1.6 with the SSA compiler and I would expect further gains in future releases.
- The Go compiler does not offer optimization levels and overall is optimized for compilation speed to, so these tradeoffs could be limiting.
- While the Go GC is very good and improving, so that the pauses are very low, it does use CPU, also while running in parallel to the program. How much exactly very much depends on the allocation behavior of the program, but should be accounted for.
It's true that Go compiles down to an executable and doesn't run on a VM like Java, but it does have it own runtime compiled into it, which manages the GC and goroutine scheduler to name two, so there is definitely some more overhead to Go then just straight C/C++
In addition, the language isn't designed for zero-cost abstractions like C++ is. You see this in the design of things like defer, which requires allocating records on the heap in certain cases, as compared to exceptions in C++ which can be implemented in a zero-cost manner.
After a well optimized implementation, speed comes down to manual vs GC memory management, static vs virtual function dispatch, dynamic vs static typing and heap vs stack allocation.
Compile speed is usually slowed down by any sort of type inference or templating/generics, which is probably one reason why go resists implementing generics or subclassing
Your rules of thumb are probably mostly related to CPU bound tasks. If database i/o is the bottleneck, then the differences across language probably don't matter much.
And, runtime performance isn't always the most important thing. In many cases, "time to market" or "cost of development" might be a higher order priority. In those cases, a higher level language might be more attractive.
Sorry, but I feel like I'm a little confused by your post. How can it be 6 times slower to program in JS or Python than C? Isn't this very much contrary to the conventional wisdom?
dep mgmt, at least, is poised to improve a lot in the coming months. we've got growing consensus around some metadata files, and i'm nearly done with my SAT solver (github.com/sdboyer/vsolver), which will be in glide, and maybe others, soon.
That was the good. The bad? Garbage collection, dependency management, and lack of first-tier support in various libraries. Garbage collection makes the otherwise lightweight and speedy language a memory hog under heavy loads. Not too bad, but I have to kick the memory up on my servers. Dependency management is a nightmare; honestly the worst part about it. The lack of first-tier support in various libraries is a close second. AWS's API libraries had relentless, undocumented breaking changes when we were using them, all on the master branch of their one repo (breaking Golang's guidelines for dependencies). Google itself doesn't actually have any real API libraries for their cloud services. They autogenerate all API libraries for golang, which means they're not idiomatic, are convoluted to use, and the documentation is a jungle.
We continue to use Go because of its strengths, but it just really surprises me how little Google seems to care about the language and ecosystem.