https://dlang.org/spec/attribute.html#nogc

- You'd have to ensure that your large data structure gets allocated entirely within the special region. That's simple enough if all you have is a big array, but it gets more complicated if you've got something like a map of strings. Each map cell and each string would need to get allocated in the special region, and all of the types involved would need new APIs to make that happen.

- You'd have to ensure that data structures in your special region never hold references to anything outside. Since the whole point of the region is that the GC doesn't scan it, nothing in the region will be able to keep anything outside the region alive. Any external references could easily become dangling pointers to freed memory, which is the sort of security vulnerability that GC itself was designed to prevent.

All of this is doable in theory, but it's sufficiently difficult, and it comes with sufficiently many downsides, that it makes more sense for a project with these performance needs to just use C or Rust or something.

You can treat external references as GC roots.

Which brings you back to doing an expensive scan of the large off-heap data structure you were trying to avoid.

Probably, yeah. But the Golang team would never add such a feature because of their philosophy of keeping the language simple.

[1] https://docs.oracle.com/javase/9/gctuning/garbage-first-garb...

That real business problem is Java generates boat load of garbage so GC needs a lot more performance tuning to make application run normal.

As you fight your language, you're GC avoidance system will become larger and larger. At some point you might re-evaluate your latency requirements, your architecture, and which are the right tools for the job.

https://golang.org/pkg/sync/#Pool

Just pool could've mitigated your problem at least partially, is what I'm saying.

I also think it is great that Discord is using the right tool for the job. It isn't often that you need the performance gains that Rust & Tokio so pick what works best to get the job done and iterate.

> Rust is blazingly fast and memory-efficient: with no runtime or garbage collector, it can power performance-critical services, run on embedded devices, and easily integrate with other languages.

I’m not so sure they would have done the rewrite if the Go GC was performing better, and the choice of Rust seems primarily based on prior experience at the company writing performance sensitive code rather than delivering business value.

Don't really know about Go versus Rust for this purpose, but don't really care because read states (like nearly everything that makes Discord less like IRC) is an anti-feature in any remotely busy server. Anything important enough that it shouldn't be missed can be pinned, and it encourages people to derail conversations by replying out of context to things posted hours or days ago.

Desktop development is a total wasteland these days -- there isn't nearly as much effort put into optimization as server side. They're not paying for your local compute, so they can waste as much of it as they want.

But think of it this way, all the effort they put into their desktop app works on all major OSes without a problem. They even get to reuse most of the code for access from the browser, with no installation required.

Now imagine approaching your PM and saying "Look I know we put X effort into making our application work on all the platforms, but it would be even faster if we instead did 4x effort for native implementations + the browser".

[0] From what I've seen in the "gamer community" is that most gamers don't care that much about that kind of extra performance. Discord itself doesn't feel slow once it's started. Joining a voice channel is instant, quickly switching to a different server and then to a chat channel to throw in a some text is fast and seamless (Looking at you MS Teams!!!).

Sure Mumble/Teamspeak are native and faster, but where are their first party mobile apps and web clients? One of the incredible things Discord did to really aid in adoption was allow for web clients, so when you found some random person on the internet, you didn't have to tell them to download some chat client, they could try it through their browser first.

tl;dr

Yes electron apps can be slow, but discord IMO has fine client side performance, and they clearly do put resources into optimizing it. Yes it "could be faster" with native desktop apps, but their target community seems perfectly content as is.

I do think it's reasonable to get the startup time of Discord to be near what VS Code's startup times are. If we remove the updater, it actually starts pretty fast (chromium boot -> JS loaded from cache) is <1s on a modern PC. And there's so much more we can do from there, for example, loading smaller chunks in the critical "load and get to your messages path" - being able to use v8 heap snapshots to speed up starting the app, etc...

The slow startup time is very much an us problem, and not an electron problem and is something I hope we'll be able to address this year.

Electron startup time as well as v8 snapshots have been a hot topic for a looooong time. I actually started a pull request for it in 2015 [0]. My pull request was focusing on source code protection, but ANY information on how you use v8 snapshots, etc. would be awesome!

[0] https://github.com/electron/electron/issues/3041

Once it's loaded, how much slower than Word is it?

Almost every single app I run auto-updates itself in some form.

But on the client side, it's arguably the slowest to launch application I have installed even among other Electron apps. Perfectly fine.

This completely re-enforces my original statement: "Desktop development is a total wasteland these days -- there isn't nearly as much effort put into optimization as server side" Desktop having horrible startup performance is "fine" but a little GC jitter on the server requires a complete re-write from the ground up.

First and foremost, we do care deeply about desktop performance. We shipped this week a complete rewrite of our messages components, that come with a boatload of performance optimizations, in addition to a new design. We spent a lot of time to do that rewrite in addition to applying new styles, because given what we know now (and what's state of the art in React world), we can write the code better than we did 3+ years ago. In terms of total engineering time spent, the rewrite of messages actually took much longer than the rewrite of this service from go to rust.

That being said, the desktop app does load much slower than we'd like (and honestly than I'd like personally.) I commented in another thread why that is. That being said, the person who is writing backend data-services, is not the one who's going to be fixing the slow boot times (our native platform team). These are efforts that happen independently.

As for our motivations for using rust, I think saying that "a little GC jitter on the server requiring a complete rewrite" is one of many reasons we wanted to give rust a shot. We have some code that we know works in a Golang. We want to investigate the viability of Rust to figure out how it'd look like to write a data service in rust. We have this service that is rather trivial, and has some GC jitters (that we've been fine with for a year.) So, an engineer (the author of this blog post) spends some time last year to see what it'd look like to write an equivalent service in rust, how it'd perform, how easy it'd be, and what the general state of the ecosystem is like in practice.

I think it's easy to forget that a lot of work we do as engineers isn't all about what's 100% practical, but also about learning new things in order to explore new viable technologies. In this case, this project had a very clear scope and set of requirements (literally rewrite this thing that we know works), and a very well defined set of success criteria (should perform as-good or better, see if a lack of GC will improve latencies, get a working understanding of the state of the ecosystem and how difficult it would be to write future data services in rust vs go.) Given the findings in our rewrite of this service, running it in production, and now using features that have stabilized in rust, we're confident in saying that "in places where we would have used golang, we consider rust viable, and here's why, given our exercise in rewriting something from go to rust."

The big thing IMO is that once started I normally leave discord running, and most actions within discord itself feel very snappy - E.g. You click on a voice channel and you're instantly there. I think that's what they mean, they're trying to keep the delay for such an action low. Sometimes you click a voice Chanel and there's a few seconds of delay, those for some reason more annoying then the long (ish) startup time

Go is what would have happened if Bell Labs wrote Java.

In that case, Go is Bell Labs' second attempt at Java.

(And Kernighan was their floor-mate too, that must have been a stunningly great environment)

And Unix is what happened when Bell Labs wrote an operating system -- something that was born outdated from the start.

Just like Golang.

> We figured we could tune the garbage collector to happen more often in order to prevent large spikes, so we implemented an endpoint on the service to change the garbage collector GC Percent on the fly. Unfortunately, no matter how we configured the GC percent nothing changed. How could that be? It turns out, it was because we were not allocating memory quickly enough for it to force garbage collection to happen more often.

As someone not too familiar with GC design, this seems like an absurd hack. That this 2-minute hardcoded limitation is not even configurable comes across as amateurish even. I have no experience with Go -- do people simply live with this and not talk about it?

IIRC I have used GitLab and Bitbucket and self-hosted Gitea instances the same exact way, and I'm fairly sure there was an hg repo in one of those. Don't recall doing anything out of the ordinary compared to how I would use a github URL.

Ouch, Go never ceases to amaze. The Bitbucket case[0] is even more crazy, calling out to the Bitbucket API to figure out which VCS to use. It has a special case for private repositories, but seems to hard-code cloning over HTTPS.

If only we had some kind of universal way to identify resources, that told you how to access it...

[0]: https://github.com/golang/go/blob/e6ebbe0d20fe877b111cf4ccf8...

Wow, that's sad. I'm glad it works seamlessly, don't get me wrong, but I was assuming I could chalk it up to defacto standards between the various vendors here.

  GOGC=off

More details here: https://golang.org/pkg/runtime/

I think with that, you could turn off GC after startup, then turn it back on at desired intervals (e.g. once an hour or after X cache misses).

It's definitely risky though. E.g. if there is a hiccup with the database backend, the client library might suddenly produce more garbage than normal, and all instances might OOM near the same time. When they all restart with cold caches, they might hammer the database again and cause the issue to repeat.

CloudFront, for this reason, allocates heterogeneous fleets in its PoPs which have diff RAM sizes and CPUs [0], and even different software versions [1].

> When they all restart with cold caches, they might hammer the database again and cause the issue to repeat.

Reminds me of the DynamoDB outage of 2015 that essentially took out us-east-1 [2]. Also, ELB had a similar outage due to unending backlog of work [3].

Someone must write a book on design patterns for distributed system outages or something?

[0] https://youtube.com/watch?v=pq6_Bd24Jsw&t=50m40s

[1] https://youtube.com/watch?v=n8qQGLJeUYAt=39m0s

[2] https://aws.amazon.com/message/5467D2/

[3] https://aws.amazon.com/message/67457/

This is definitely a familiar problem if you rely on caches for throughput (I think caches are most often introduced for latency, but eventually the service is rescaled to traffic and unintentionally needs the cache for throughput). You can e.g. pre-warm caches before accepting requests or load-shed. Load-shedding is really good and more general than pre-warming, so it's probably a great idea to deploy throughout the service anyway. You can also load-shed on the client, so servers don't even have to accept, shed, then close a bunch of connections.

The more general pattern to load-shedding is to make sure you handle a subset of the requests well instead of degrading all requests equally. E.g. processing incoming requests FIFO means that as queue sizes grow, all requests become slower. Using LIFO will allow some requests to be just as fast and the rest will timeout.

I've read the first SRE book but having worked on large-scale systems it is impossible to relate to the book or internalise the advice/process outlined in it unless you've been burned by scale.

I must note that there are two Google SRE books in-circulation, now: https://landing.google.com/sre/books/

But, per other comments, there isn't any direct malloc/free behavior. It just provides tools to help you enable the compiler to determine that GC is not needed for some.

> We kept digging and learned the spikes were huge not because of a massive amount of ready-to-free memory, but because the garbage collector needed to scan the entire LRU cache in order to determine if the memory was truly free from references.

So maybe this is one of those things that just doesn't come up in most cases? Maybe most services also generate enough garbage that that 2-minute maximum doesn't really come into play?

Often something like Redis is used as a shared cache that is invisible to the garbage collector, there is a natural key with a weak reference (by name) into a KV store. One could embed a KV store into an application that the GC can't scan into.

Doesn't Go have something like this available? It's an obvious thing for garbage-collected languages to have.

But (virtually) nobody is writing games in Go, so it's entirely possible that it's an unusual case in the Go ecosystem. Being an unsupported usecase is a great reason to switch language.

Is this true? Go was built specifically for C++ developers, which, even when Go was first release, was a pretty unpopular language for writing web services (though maybe not at Google?). That a non-trivial number of Ruby/Python/Node developers switched was unexpected. (1)

https://commandcenter.blogspot.com/2012/06/less-is-exponenti...

> Our target community was ourselves, of course, and the broader systems-programming community inside Google. (1)

(1) http://www.informit.com/articles/article.aspx?p=1623555

[1] https://blogs.unity3d.com/2018/11/26/feature-preview-increme...

IDK if this is true for earlier versions, but as of today C# has pretty clear rules: 16MB in desktop or 64MB in server (which type is used can be set via config) will trigger a full GC [1]. Note that less than that may trigger a lower level GC, but those are usually not the ones that are noticed. I'm guessing at least some of that is because of memory locality as well as the small sizes.

On the other hand, in a lot of the Unity related C# posts I see on forums/etc, passing structs around is considered the 'performant' way to do things to minimize GC pressure.

[1] https://docs.microsoft.com/en-us/dotnet/standard/garbage-col... [2] https://blog.golang.org/ismmkeynote

I started up KSP just now, and it was at 5.57GB before I even got to the main menu. To be fair, I hadn't launched it recently, so it was installing its updates or whatever. Ok, I launched it again, and at the main menu it's sitting on 5.46GB. (This is on a Mac.) At Mission Control, I'm not even playing the game yet, and the process is using 6.3GB.

I think a better takeaway is that you can get away with GC even in games now, because it sucks and is inefficient but it's ... good enough. We're all conditioned to put up with inefficient software everywhere, so it doesn't even hurt that much anymore when it totally sucks.

You could maybe hack around the GC performance without destroying the aims of LRU eviction by batching additions to your LRU data structure to reduce the number of pointers by a factor of N. It's also possible that a Go BTree indexed by timestamp, with embedded data, would provide acceptable LRU performance and would be much friendlier on the cache. But it might also not have acceptable performance. And Go's lack of generic datastructures makes this trickier to implement vs Rust's BtreeMap provided out of the box.

This is something important to know before choosing a GC-based language for a task like this. I don't think "generating more garbage" would help, the problem is the scan is slow.

If Discord was forced to do this in pure Go, there is a solution, which is basically to allocate a []byte or a set of []bytes, and then treat it as expanse of memory yourself, managing hashing, etc., basically, doing manual arena allocation yourself. GC would drop to basically zero in that case because the GC would only see the []byte slices, not all the contents as individual objects. You'll see this technique used in GC'd languages, including Java.

But it's tricky code. At that point you've shucked off all the conveniences and features of modern languages and in terms of memory safety within the context of the byte expanses, you're writing in assembler. (You can't escape those arrays, which is still nice, but hardly the only possible issue.)

Which is, of course, where Rust comes in. The tricky code you'd be writing in Go/Java/other GC'd language with tons of tricky bugs, you end up writing with compiler support and built-in static checking in Rust.

I would imagine the Discord team evaluated the option of just grabbing some byte arrays and going to town, but it's fairly scary code to write. There are just too many ways to even describe for such code to end up having a 0.00001% bug that will result in something like the entire data structure getting intermittently trashed every six days on average or something, virtually impossible to pick up in testing and possibly even escaping canary deploys.

Probably some other languages have libraries that could support this use case. I know Go doesn't ship with one and at first guess, I wouldn't expect to find one for Go, or one I would expect to stand up at this scale. Besides, honestly, at feature-set maturity limit for such a library, you just end up with "a non-GC'd inner platform" for your GC'd language, and may well be better off getting a real non-GC'd platform that isn't an inner platform [1]. I've learned to really hate inner platforms.

By contrast... I'd bet this is fairly "boring" Rust code, and way, way less scary to deploy.

[1]: https://en.wikipedia.org/wiki/Inner-platform_effect

To be clear: I wasn't suggesting that generating garbage would help anyone. Only that in a more typical case, where more garbage is being generated, the two minute interval itself might never surface as the problem because other things are getting in front of it.

- lazy devs and product managers, everywhere

Or do you think it adds to the discussion?

At least with code hacking around the GC's behavior that code ends up being portable across the ecosystem.

There doesn't seem to really be a good option here either way. This amount of tuning-by-brute-force (either by knobs or by code re-writes) seems to just be the cost of using a GC.

If these issues were more common, there would be more configuration available.

[EDIT] to downvoters: I'm not saying it's not an issue worth addressing (and it may have already been since they were on 1.9), I was just answering the question of "why this might happen"

> The ballast in our application is a large allocation of memory that provides stability to the heap.

> As noted earlier, the GC will trigger every time the heap size doubles. The heap size is the total size of allocations on the heap. Therefore, if a ballast of 10 GiB is allocated, the next GC will only trigger when the heap size grows to 20 GiB. At that point, there will be roughly 10 GiB of ballast + 10 GiB of other allocations.

> For those interested, there is a proposal to add a target heap size flag to the GC which will hopefully make its way into the Go runtime soon.

What's wrong with the existing parameter?

I'm sure they aren't going this far to avoid all environment config without a good reason, but any good reason would be a flaw in some part of their stack.

Sometimes it means an easy thing in most other languages is difficult or tiresome to do in Go. Sometimes it means hard-coded values/decisions you can't change (only tabs anyone?).

But overall this makes for a language that's very easy to learn, where code from project to project and team to team is very similar and quick to understand.

Like anything, it all depends on your needs. We've found it suits ours quite well, and migrating from a Ruby code base has been a breath of fresh air for the team. But we don't have the same performance requirements as Discord.

See:

https://stackoverflow.com/questions/19094704/indentation-in-...

The Azul guys get to claim that you don't need to tune their gc, golang doesn't.

Not sure how that makes the golang position any better.

Have you ever worked in a code base with many contributors that changed over the course of years? In my experience it always ends up a jumble where indentation is screwed up and no particular tab setting makes things right. I've worked on files where different lines in the same file might assume tab spacing of 2, 3, 4, or 8.

For example, say there is a function with a lot of parameters, so the argument list gets split across lines. The first line has, say, two tabs before the start of the function call. The continuation line ideally should be two tabs then a bunch of spaces to make the arguments line up with the arguments from the first line. But in practice people end up putting three or four tabs to make the 2nd line line up with the arguments of the first line. It looks great with whatever tab setting the person used at that moment, but then change tab spacing and it no longer is aligned.

Consider linting tools in your build.

        some_function(arg1, arg2, arg3, arg4,
                      arg5, arg6);

  some_function(
      arg1,
      arg2,
      arg3,
      arg4,
      arg5,
      arg6,
  );

rustfmt looks to try and be "smarter" as it will move the argslist and add linebreaks to it go not go beyond whatever limit is configured on the playground, gofmt apparently doesn't insert breaks in arglists.

Not all editors, however, support this style of alignment, even if they support plugins (looks at vim and its forks).

Instead, format arguments in a separate block:

    some_function(
        arg1, arg2, arg3, arg4,
        arg5, arg6);

My own preference is tabs, because of less visual noise in code diff [review].

Thing is, few if any IDEs support the concept, so if I want to have half-indents, I must use spaces. Unfortunately, these days that means giving up and using a more common indent style most of the time, as the extra bit of readability generally isn't worth losing automatic formatting or multi-line indent changes.

BTW when you mix spaces with tabs you eliminate all benefits that tabs give (for example you no longer can dynamically change tab size without ruining formatting.

No, I used to be a notepad user (on personal projects, not shared work) (you can kinda see it in the use of indentation to help convey information that IDEs would use font or colour to emphasize), and these days use tabs but longingly wish Eclipse, etc. had more options in their already-massive formatting configuration dialogues.

<tab>(inserts 4 spaces)<tab>(replaces 4 spaces into a tab that is 8 columns)<tab>(adds 4 spaces after the tab)<tab>(replaces with two tabs and so on)

Unless I misunderstood what formatting you were using.

These are two things that make a lot of sense at Google if you read why they were done.

But unless you're working at Google, I struggle to guess why you would care about either of these things. The first requires sacrificing anything resembling a reasonable type system, and even with that sacrifice Go doesn't really deliver: are we really supposed to buy that "go generate" isn't a compilation step? The second is sort of nice, but not nice enough to be a factor in choosing a language.

The core language is currently small, but every language grows with time: even C with its slow-moving, change-averse standards body has grown over the years. Currently people are refreshed by the lack of horrible dependency trees in Go, but that's mostly because there aren't many libraries available for Go: that will also change with time (and you can just not import all of CPAN/PyPy/npm/etc. in any language, so Go isn't special anyway).

If you like Go for some aesthetic of "simplicity", then sure, I guess I can see how it has that. But if we're discussing pros and cons, aesthetics are pretty subjective and not really work talking about.

I like Go and I consider it a simple language because:

1. I can keep most of the language in my head and I don't hit productivity pauses where I have to look something up.

2. There is usually only one way to do things and I don't have to spend time deciding on the right way.

For me, these qualities make programming very enjoyable.

You mean where I explicitly said that "simple" didn't mean anything, so we should talk about what we mean more concretely?

> 1. I can keep most of the language in my head and I don't hit productivity pauses where I have to look something up.

The core language is currently small, but every language grows with time: even C with its slow-moving, change-averse standards body has grown over the years.

> 2. There is usually only one way to do things and I don't have to spend time deciding on the right way.

Go supports functional programming and object-oriented programming, so pretty much anything you want to do has at least two ways to do it--it sounds like you just aren't familiar with the various ways.

The problem with having more than one way to do things isn't usually choosing which to use, by the way: the problem is when people use one of the many ways differently within the same codebase and it doesn't play nicely with the way things are done in the codebase.

This isn't really a criticism of Go, however: I can't think of a language that actually delivers on there being one right way to do things (most don't even make that promise--Python makes the promise but certainly doesn't deliver on it).

I've been happy working with it for a year now, though I've had the chance to work with Kotlin and I have to say, it's very nice too, even if the parallelism isn't quite easy/ convenient to use.

1. map/filter are 2 lines of code each. 2. Inheritance is part of mainstream OOP, but there are some less common languages that don't support inheritance in the way you're probably thinking (i.e. older versions of JavaScript before they caved and introduced two forms of inheritance). 3. Generics are more of a strong type thing than an OOP thing.

Rust will probably be faster because it benefits from optimizations in LLVM that Go likely doesn't have.

Go arrays of structs are contiguous, not indirect pointers.

It is worth it to read and understand: https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i...

Also, it is almost trivial to edit the Go sources (they are included in the distribution) and rebuild it, which usually takes just a minute. So Go is really suited for your own experiments - especially, as Go is implemented in Go.

Well, parts of it. You can't implement "make" or "new" in Go yourself, for example.

I do, I'm just objecting to "Go is implemented in Go".

The stable compiler is permitted to use unstable features in stable builds, but only for compiling the compiler. In essence, there are some Rust features that are supported by the compiler but only permitted to be used by the compiler. Unsurprisingly, various non-compiler users of Rust have decided that they want those features and begun setting the RUSTC_BOOTSTRAP envvar to build things other than the compiler, prompting consternation from the compiler team.

[0] https://github.com/rust-lang/cargo/issues/6627 [1] https://github.com/rust-lang/cargo/issues/7088

[0]: https://doc.rust-lang.org/src/std/io/mod.rs.html#379

[1] https://news.ycombinator.com/item?id=22240223

[1] https://golang.org/ref/spec

https://golang.org/src/runtime/map.go

https://golang.org/src/runtime/slice.go

I don't see why you couldn't do something similar in your own Go code. It just won't be as convenient to use as the compiler wouldn't fill in the type information (element size, suitable hash function, etc.) for you. You'd have to pass that yourself or provide type-specific wrappers invoking the unsafe base implementation. More or less like you would do in C, with some extra care to abide by the rules required for unsafe Go code.

Obviously they consider spending 50% more on hardware is a worthwhile compromise for the gains they get (e.g. reduction of developer hours and reduced risk of security flaws or avoiding other effects of invalid pointers).

If you do a lot of allocations, the GC overhead rises of course, but also would the effort of doing allocations/deallocations with a manual managing scheme. In the end it is a bit trade-off, what fits the problem at hand best. The nice thing about Rust is, that "manual" memory management doesn't come at the price of program correctness.

As a consequence, the heap pressure of a Go program is not necessarily significantly larger than that of an equivalent C or Rust program.

In contrary, in e.g. C I can wrap two 32-bit fields in a struct and freely pass then anywhere with zero heap allocations.

Also, record types are not going to fix the pointer chasing problem with arrays. This is promised by Valhalla, but I'be been hearing about it for 3 years or more now.

Ruby 1.8.x wants to say "Hello"

Can someone explain to me how BTreeMap is more memory efficient than a HashMap?

Note that this explanation is a bit handwavy, as both data structures have numerous optimizations in production scenarios.

Rust's HashMap stores the collisions in the same table as the non-collisions (open addressing), not in a separate collection.

We have 2 golang services left, one of them has a rewrite in rust in PR as of last week (as a fun side project an engineer wanted to try out.)

Additionally, as we move towards a more SOA internally, we plan to write more high velocity data services, and rust will be our language of choice for that.

This isn’t exactly “Linux kernel: now in Rust!”

Glad you’re making tech for you all better.

We get to take up the externalized runtime costs of the mess that is the Electron app.

Engineers are super efficient at offloading the last mile of effort.

Short: ffi for erlang.

Kinda like this: https://blog.sentry.io/2016/10/19/fixing-python-performance-... ?

They were able to rewrite their hot spot in a new language without having to rewrite all their business logic in a new language. Not that there wouldn’t have been solutions with a monolith, but this certainly seems elegant and precise.

Also note this was with Go1.9. I know GC work was ongoing during that time, I wonder if this time of situation would still happen?

EDIT: Actually, no it didn't, I misunderstood it.

It says things like:

“We were not creating a lot of garbage.”

... but that statement there doesn’t say anything about the heap size, including the size and count of live objects (i.e., not garbage).

It also says:

“There are millions of Users in each cache. There are tens of millions of Read States in each cache.”

Large is often in the eye of the beholder, but I missed it if it said anything specifically about not having a large heap size.

Not sure why you got downvoted, you're actually right, I'm wrong: I misread that and/or assumed one meant the other.

That said, this is a case that should be ideal for generational GC, which Go specifically eschewed at one point. I'm not sure this is still the case, however--I have yet to wade through this[1] to update my knowledge here.

[1] https://blog.golang.org/ismmkeynote

> During garbage collection, Go has to do a lot of work to determine what memory is free, which can slow the program down.

read like blogospam to me (which it is).

For comparison sake - similar post from Twitch has a lot more technical detail and generally makes me view their team in a lot better light than Dicord’s after reading both.

It isn't surprising to me. It's stated elsewhere they tried 4 difference version of Go, up through 1.10 apparently, and had performance problems with all of them. At some point you can't suffer garbage collector nonsense anymore and since they'd already employed Rust on other services they tried it here.

It worked on the first try.

That's not surprising either.

What would be surprising is if any of these "but version such and such is Waaay better and they should just use that" actually panned out. The best case would be that the issue just manifests as some other garbage collector related performance problem. That's the deal you sign up for when you saddle yourself with a garbage collector.

They could have tried 1 other version (not 4) and picked either the latest (1.13) or the version that contains the GC improvements (1.12) to test. Usually when you are looking to upgrade something you skim the release notes so testing 1.12 or 1.13 is obvious especially when 1.12 seems to specifically address their performance concern.

If upgrading something avoids a service re-write that is usually the way to go unless you were looking for an excuse to re-write the service in the first place which may have been the case.

edit:

It turns out they did exactly what my comment stated: they tested the latest version (1.10). It's just that this article was published recently but the events happened quite a while back.

Maybe it does, maybe it doesn't. Maybe 1.14 has a performance regression with their specific load and they get screwed.

We'll never know. You know why? They permanently solved their GC problems by eliminating GC.

That's the smart play.

It seems like my comment was just an entry point for you to shit on gc which, ironically, I mostly agree with in this context.

> We tried upgrading a few times. 1.8, 1.9, and 1.10. None of it helped. We made this change in May 2019. Just getting around to the blog post now since we've been busy.

https://www.reddit.com/r/programming/comments/eyuebc/why_dis...

> Another Discord engineer chiming in here. I worked on trying to fix these spikes on the Go service for a couple weeks. We did indeed try moving up the latest Go at the time (1.10) but this had no effect.

> For a more detailed explanation, it helps to understand what is going on here. It is not the increased CPU utilization that causes the latency. Rather, it's because Go is pausing the entire world for the length of the latency spike. During this time, Go has completely suspended all goroutines which prevents them from doing any work, which appears as latency in requests.

> The specific cause of this seems to be because we used a large free-list like structure, a very long linked list. The head of the list is maintained as a variable, which means that Go's mark phase must start scanning from the head and then pointer chase its way through the list. For whatever reason, Go does (did?) this section in a single-threaded manner with a global lock held. As a result, everything must wait until this extremely long pointer chase occurs.

> It's possible that 1.12 does fix this, but we had tried upgrading a few times already on releases that promised GC fixes and never saw a fix to this issue. I feel the team made a pragmatic choice to divest from Go after giving the language a good attempt at salvaging the project.

Anyway, in this case, I guess they're using a free list because (1) it's simpler since you don't need an external collection keeping a list of unused stuff, and (2) reason (a) above.

Collections are one of the big areas where Go's lack of generics really hurts it. In Go, if one of the built in collections does not meet your needs, you are going to take a safety and ergonomic hit going to a custom collection. In Rust, if one of the standard collections does not meet your needs, you (or someone else) can create a pretty much drop-in replacement that does that has similar ergonomic and safety profiles.

A corollary to this is that adding more generic collections to Go’s standard library implies expanding the set of magical constructs.

It is rather that due to a lack of genericity (namely const generics) you can't implement traits for [T;N], you have to implement them for each size individually. So there has to be an upper bound somehow[1], and the stdlib developers arbitrarily picked 32 for stdlib traits on arrays.

A not entirely dissimilar limit tends to be placed on tuples, and implementing traits / typeclasses / interfaces on them. Again the stdlib has picked an arbitrary limit, here 12[2], the same issue can be seen in e.g. Haskell (where Show is "only" instanced on tuples up to size 15).

These are not "weird hacks", they're logical consequences of memory and file size not being infinite, so if you can't express something fully generically… you have to stop at one point.

[0] here's 47 https://play.rust-lang.org/?version=stable&mode=debug&editio...

[1] even if you use macros to codegen your impl block

[2] https://doc.rust-lang.org/src/core/fmt/mod.rs.html#2115

(anecdotal: in Java I've never needed anything else than a HashMap or an ArrayList)

It appears that Go has a lower CPU floor, but it's killed by the GC spikes, presumably due to the large cache mentioned by the author.

This is interesting to me. It suggests that Rust is better at scale than Go, and I would have thought with Go's mature concurrency model and implementation would have been optimized for such cases while Rust would shine in smaller services with CPU bound problems.

Great post!