Agreed, isolating the contribution of the "model" itself can be borderline impossible, and the details of model engineering have a considerable impact on performance.
That's why I like Kaggle competitions: most approaches tried on a given challenge will likely be optimized to their very limits, so that it's the nature of the different approaches that ends up making the difference.
I understand that there are more models being thrown at the solution space, but if you think about people at jane street, while they don't have 3000 employees working on these, they've got a few hundred, and then if you think about the _time_ they spend on (the real, not this kaggle competition simplified stuff) models, I think there's a good chance that they'll have a significant advantage in man-hours (it's a full time job).
Also, while there is a high standard to people doing kaggle competitions, I think you can easily discard >90% of them as not being super competitive, then your 3000 models becomes 300, and those 300 have less than 1/3 of the time spent on them that jane street-ers are spending on their models.
(I realise it's not an apples to apples comparison, as the models they're working on are different since this is a toy example, but you said "will outperform anything that Jane Street _could_ come up with)
I don't think you understand the point. Your claims that "all of this needs extensive tuning and hand-holding and picking results" do not help your argument, they help _mine_.
Most egregious if you are even doing more tuning and cherry picking that the authors of the models are doing, which you definitely are.
>The point I'm wanting to make is that users will go to whoever has the best model.
Best isn't defined just by quality though. In some instances for some groups, things like whether the model is trained on licensed content (with permission) and / or is safe for commercial use is more important.
This is one reason why Adobe's Firefly has been received relatively well. (I work for Adobe).
Why is too few models the issue? Wouldn't the problem be that the existing models arent good enough? Its not clear to me how an increased diversity in models addresses the problem - at the end if the day a form has to pick one to make any particular decision. If each firm is using a different model in the name of diversity, then while the few firms with more accurate models potentially do well the subpar models will underperform in general and probably increase the overall risk level of the industry. Is there something I'm overlooking from my skin through the article here?
True, and I don't think they expect models that are superior to their own, I'd look at this as a hiring / marketing tool. Plus, even if you had a model that from a pure engineering standpoint was able to match Jane Street's approach, the model would not work without the wealth of proprietary (and expensive) data sources that Jane Street is sure to ingest, so you still couldn't just go out and do it yourself without some serious upfront investment first to get the same data. That is assuming all data they use is even available commercially, which I doubt as well. There are probably data sources that only become available to you through personal relationships with the right folks at the right places.
There is a sense in which the person/people who select the training data for a model are the artist. Or at least, a sense in which they can be an artist (models that just train on as much data as possible don't fit).
So you’re saying the experts chosen are a more literal mixture of layers from each model? Rather than a simple “pick which model to run”?
reply