I was beginning to worry I had overlooked the inner beauty of DeepSeek. When I have tried it, it was totally useless. Never understood HN cheering for it.
> There is always a huge discrepancy between benchmarks and real world capabilities.
I mean of course, the real world is messy, the models are only good for what they've been trained on, and post-training frequently takes benchmark scores into account too, leading to this sort of "Great score, sucks in the real world" syndrome that seemingly most models are affected by.
This is perfect. Real programming has a ton of these instances of minor changes.
There is always a huge discrepancy between benchmarks and real world capabilities.
For my use case deepseek 4.1 flash is almost as dumb as haiku 5.5. Both are fast garbage producers.
I was beginning to worry I had overlooked the inner beauty of DeepSeek. When I have tried it, it was totally useless. Never understood HN cheering for it.
> There is always a huge discrepancy between benchmarks and real world capabilities.
I mean of course, the real world is messy, the models are only good for what they've been trained on, and post-training frequently takes benchmark scores into account too, leading to this sort of "Great score, sucks in the real world" syndrome that seemingly most models are affected by.
Two brothers created a benchmark, family is so proud.