I'm not familiar with this realm to comment on veracity of claims but it could very well be
"Posting benchmark results is bad because it quickly becomes a race to the wrong solution. Someone misrepresented our performance in a benchmark, here are the actual results."
"Posting benchmark results is bad because it quickly becomes a race to the wrong solution. But somebody showed us sucking on a benchmark, so here's our benchmark results showing we're better."
This title seems like an exaggeration of what is claimed in the article. In the article, they state that they benchmarked their solver in a biased way that made their solver look like it performed better than it did, not that they faked performance data altogether.
No, that's because some of the people don't have first principles and trust at all and instead of validating they just make stuff up ;-)
I was trying to point out that disputing things is fine, but the whole basis of a website where benchmarks are uploaded is trust-and-reputation-over-time to the point where enough other people can re-run the same tests on their machines (one they get them) to validate the results. Heck, you might almost call that science!
Right now, there aren't additional results and you can't easily reproduce them because the machines aren't wide spread or available. But we can take the track record and reputation of the site and application and use that to value the integrity of the published benchmarks to be 'likely correct'.
Not really. It's a lose/lose proposition to publish benchmarks, especially on something that's so environment and dataset dependent. Either real world performance is way below the benchmark leading to all kinds of storm and strife, or way above and then you lose credibility and people wondered why you bother anyways.
This is kind of the issue with an interested party/vendor running benchmarks like these. Be it by pure dumb luck or malfeasance you are much more likely to configure and be knowledgeable about your own product than the others and toss out responses and results that are wildly inaccurate/misleading.
It's nice to see someone tell you that their benchmark is flawed and why. Most try to pass off their benchmark as the end-all-be-all measurement of whatever they are testing
> Benchmarking requires expertise that, it turns out, very few people have. I don't think I even have enough skills to do it correctly and meaningfully.
Very important and often overlooked point.
But I wonder, why not forbid public dissemination of inaccurate, non-reproducible benchmarks?
> Spreading wrong performance information can hurt a business.
"But unclear about their benchmarking method". That can be said for every single performance claim they make. There's a rather distinct lack of objective facts.
Post and heading are written to attribute this to malice, without offering any proof. It seems likely given how new Bun is that the benchmark writers simply lacked familiarity.
"Posting benchmark results is bad because it quickly becomes a race to the wrong solution. Someone misrepresented our performance in a benchmark, here are the actual results."
reply