It's a good conclusion I feel: this is always the issue with language benchmarks - who wrote the code, and how good were they with each of the languages.
Similarly as the article points out, the compiler matters a lot: ICC can in certain cases be more than 200% faster than GCC with similar flags, and is generally 15-20% faster anyway, mainly due to more intelligent inlining and much faster (and more accurate with fpmath=fast) math libs.
I think it's worth pointing out though that idiomatic C is probably going to be more consistently performant. It seems common to run in to situations in Haskell where one change can cause 10X speedup, but I don't see that nearly as often with C code. I don't have a lot of evidence on hand to support this, just what I've observed personally. This seen fair? Relevant?
> C is, by nature of the language, easier to optimize than Haskell, since it is lower level.
Depends what level you mean - if you want to do all the optimisations yourself, C is better; but if you want to write code and let the compiler do the details, I would think higher-level is better (you can just write doWhatIMean() and the compiler will automatically choose the optimal implementation for your current problem and platform - where if you'd specified the implementation details yourself, you'd be sub-optimal in many cases)
Although the results are impressive, the title is a bit optimistic. Haskell beat GCC slightly in the first benchmark. Haskell beat C in the second benchmark because the C code is very poorly written. They did not compare with the obvious C implementation. Secondly, GCC isn't the best C compiler, even though the paper claims that it is the best compiler that "we could find". They should have used ICC. C++ is also 4x faster than Haskell in the second benchmark, giving further indication that the C is slow just because it is poorly written. So it's hard to justify such a title. The work itself is very good however.
That's actually something I've been saying for quite awhile when people bring up microbenchmarks complaining about how a language like Haskell is slow.
Like, yes, no question, if you try and implement a tightloop in Haskell vs C, C is going to come out a lot faster.
However, no one writes Haskell like that! People rely pretty heavily on standard libraries (or even third party libraries), and on the optimizations afforded to them by the "rich" language of Haskell.
C might be capable of being faster, but that doesn't imply that a C program will inherently be faster than a similar Haskell program, especially if the programs are relatively large.
If Haskell is easier to optimize than C, then it could easily be that there's some amount of programmer effort, for which expending that much effort in Haskell yields a faster program than expending that much effort in C. If that amount of effort is in the range of effort most people are able to expend on a class of projects, then Haskell is faster than C for those projects. It may even be that those are most projects.
It is, of course, not the case that Haskell is faster than C with arbitrary effort expended tuning to the specific hardware - no one is claiming that.
Some people like bickering. But I think the point of the article is that C is, by nature of the language, easier to optimize than Haskell, since it is lower level.
* The vast majority of code doesn't need to be fast, and it's trivial to write code that's within a factor of 5 of optimized C. I can write Haskell at least 10x faster than I can write optimized C for the same problem.
* If you want code that's within a factor of 2 of C then you'll need to understand quite a bit about the internal details of the compiler, but your code won't get too ugly. I'd say it's about the same amount of work in both C and Haskell to get this level of performance.
* If you want code that's just as fast as optimized C then you'll need great understanding of both the compiler and the underlying hardware. At this point, you can expect your code will get hideously ugly. Also your code will break a lot when GHC versions change. I can write optimized C about 10x faster than I can write optimized Haskell.
Of course every problem is different, and this is only a rough average of my experience.
The C code is not particularly optimized, but more importantly, C being a low level language makes writing multi-threaded programs much more work. wc though cannot make use of multi-threading regardless because it must have the ability to work on streams, not just seekable files, as the haskell code here does.
I'm surprised by how dramatic the difference is between the speed of C and Haskell. One of my professors (at The University of Glasgow, so appropriately a Haskell fan) once claimed that it had "c-like performance".
I suppose that's the point of this paper though, that "c-like performance" is a terribly vague term, meaningless without knowledge of the specific comparisons being made.
1) Nowhere in my comment did I say C is closer to the machine.
2) Despite #1 C is still closer to the machine than Haskell, and I'm not sure how you could maintain otherwise
3) Nearly all of the C optimizations will, at best, make a speedup by a constant factor. Things that add (or remove) an O(n) factor in Haskell can and do happen.
> This article is in response to an earlier one comparing Haskell and C, which made the claim that Haskell beats out C when it comes to speed.
Perhaps my reading comprehension of the original post is different from Jacques' (or I'm just wrong), but I don't think that the original article made such a claim. Here's the TL;DR of the original article:
> TL;DR: Conventional wisdom is wrong. Nothing can beat highly micro-optimised C, but real everyday C code is not like that, and is often several times slower than the micro-optimised version would be. Meanwhile the high level of Haskell means that the compiler has lots of scope for doing micro-optimisation of its own. As a result it is quite common for everyday Haskell code to run faster than everyday C. Not always, of course, but enough to make the speed difference moot unless you actually plan on doing lots of micro-optimisation.
From this, I understood that in a larger program, most programmers wouldn't be doing the kind of micro-optimizations that they do for the Benchmarks Game. I figure that most code would be written following this pattern:
* Write code to follow the spec (usually without thinking too much about performance)
* Run the code, and evaluate its performance
* If the performance is good enough, you're done
* If the performance needs to be better, run a profiler
* Find the biggest bottleneck and optimize it
* Re-run the code and re-evaluate its performance
The original article took a micro-benchmark (a mistake in my opinion, because it's easy to micro-optimize every single detail of that code) and showed that in the case of Haskell, the first version was too slow, but that with the help of a profiler, finding the cause was easy, and the fix was trivial, while in C the profiler didn't show any problem with the code of the user, so it must be a problem in the standard library's code, and to fix it required changing the design of the program and making it more different than the spec. And I felt this was the author's real point; that to get the maximum performance from a C program, you'll need to code it not like a human would imagine it, but like a computer would, and it makes the code harder to maintain later on.
I believe they do optimize for the simple cases, but just in very different ways. Haskell optimizes for composition, abstraction and expressiveness in a function-application sense. C optimizes for imperative loops, simple functions and controlling your memory layout and execution precisely [There is probably a better characterization of C, please comment].
I think there are plenty of tricks involved in writing performant C code that aren't obvious. Things like cache behavior, memory access patterns, etc. The job of the compiler and PL is to help us by making it unnecessary to worry about such things unless we really need to.
It's a testament to the power of modern programming languages and computer speeds that there are many programmers who don't understand registers, caches, assembly, virtual memory, etc.
I grabbed the source linked from the article, made it run on Linux, compiled it with gcc -O3, and it beats my system's wc by a factor of almost 2x. C is faster than C!
This is on Ubuntu 18.04 on x86_64. If the author's wc also came precompiled (which it seems to be), they cannot make any valid comparison at all. If manually compiling on their machine with maximum optimization would also get them a 2x speedup, their fancy multicore Haskell would still be slower than C... and that's fine.
IMO you can make C as fast or slow as you want it to be. If you write terrible C it's going to be slower than good Haskell.
If you write common C (and by common I mean what you see in well known daemons, kernels, etc) its generally at the very least on par. That's mostly because most of the functions come from libraries that have been optimized, in both cases.
"So here is the bottom line. If you really need your code to run as fast as possible, and you are planning to spend half your development time doing profiling and optimisation, then go ahead and write it in C because no other language (except possibly Assembler) will give you the level of control you need. But if you aren't planning to do a significant amount of micro-optimisation then you don't need C. Haskell will give you the same performance, or better, and cut your development and maintenance costs by 75 to 90 percent as well."
Note the 'same performance or better' in there.
Maybe you missed that bit in the original article?
This wasn't a large effort by any stretch of the imagination and a factor of 5 difference compared to the Haskell code isn't even in the same ballpark as "the same performance", and about a factor 10 difference with the C code listed in the original article. You'll notice no micro optimizations were done in the re-write, just some global re-arranging of the way data flows through the program.
The rest is in response to the misleading title, that Haskell is 'faster' than C, faster to program, faster to debug, easier to maintain and so on while making claims about performance that are not born out by real world comparisons.
Speed was the wrong thing to compare on here, and should not have been part of the original article without a serious effort to become equally proficient in both languages.
That's kind of an odd comparison, using unfused C code compared to fused Haskell code. Their point seems to focus on stream fusion's advantages and possibly that optimizing C takes more effort.
To quote the paper: "Clearly “properly”-written C++ can outperform Haskell. The challenge is in ?guring out what “proper” means."
It's not neccesarily which language is faster, but the which algorithm is faster.
He said naively written C, which mean the algorithm may be entirely different and run in O(n2) and much slower than a Haskell version which use a different algorithm and run in O(nlogn).
icc's not actually that much better than gcc in optimization, it's primarily a much better runtime library that gives icc its speed boost.
reply