This is an example of why I'm a little stumped about why people are excited about Jev-style decision models. For any application at scale it'd be way more accurate to have a dedicated classifier using the right architecture (neural net, bayesian, etc.) LLMs drive the cost of making those models down to near zero.
Just use the LLM to make a thing that's actually good at what you need it to do.
> Just use the LLM to make a thing that's actually good at what you need it to do.
This is the end state of most imagined use cases for “agentic” software IMO. Use the AI to build software, with a classifier or LLM as the expansion joint in awkward spots.
For example, a client want a classifier to decide to make a go/no go decision on roofing jobs based on weather. The end result doesn’t use LLMs at all, but does use a lot of other techniques including human feedback when it makes the wrong call.
(Now the work crew gets the decision at 5 AM instead of 5:30 and the guy making the decision doesn’t have to wake up at all.)
I do agree with you, but I do think you're also making the classic "curse of knowledge" mistake.
You have to know the right words to ask the LLM to make and train a classifier. You and I already know the words. But they are not actually that trivial to learn for an average joe, and people are lazy.
I don't believe this is true with current frontier coding agents. Or at least, it's becoming increasingly less true, such that I think it will effectively be nullified in ~6 months.
«We came full circle on the problem that also exists with Google Search, and why so many people don't get good results out of it.» [0]
It is not that access to all information was hidden to anyone before, it was harder to find it, sure. Now it can be found easily and aggregated, however, it is not as simple as saying "Do X" and believe that X is correct. Recall the infamous Swing & Product Management meme? [1] The problem is the absence of information you provide, the LLM will have to make a guess of what you mean, much like the human. Likelihoods of this guess being correct depends on the domain and background.
So yes, it'll likely build whatever you ask it (I do think we're there already), but if that matches what you intended to actually get is vastly different. You'll be settling with "Good enough" though and declare it working.
Others already pointed out that 6 months seem to be the constant of distance, irrespective of the date.
It's funny you mention this, because I've been thinking lately that all the focus on code quality benchmarks for LLMs seems to miss something. In my experience, LLMs aren't actually all that good at prompting other LLMs, and that's exactly the skill that matters for proper subagent usage.
The corpus of code available to train LLMs on is rich in both volume and quality. The same can't be said for LLM prompts. Prompts only came into existence alongside LLMs themselves, so most of the prompts out there are actually LLM generated, and training on them causes every model to overfit to the same patterns.
LLMs can give clear and concise directives to other LLMs, but only for basic tasks. Beyond that, they frequently overstuff their prompts with too much prose and too much direction, and worse, they often fill them with subtle contradictions.
Of course, there's a spectrum here. Some models turn out to be better at this than others, and I suspect this is what differentiates Fable-type models from the rest, and why vendors have started keeping their models' reasoning confidential.
The "agent factories" available to employees at the labs and hyperscalers clearly prove that agent delegation does work. Today, maybe that capacity lies with Fable class models, as you pointed out, but there is no doubt with opus 5.5 that this capability will be available to cheaper models as they get better.
That's what I've been doing in areas I lack knowledge tbh. Instead of a half assed attempt at describing what I need, I ask an llm to take my base idea and write it up using industry standard jargon as if I were commissioning a team to do the work. Has been working pretty well so far!
Okay so I just asked it "What question should I be asking you but I don't because I'm not smart enough to think of it?" Response was kind of interesting I have to say.
Ten years ago, in France, two doctors (one in a professional setting and the other during a private consultation) asked me about a wound I had on my shoulder; obviously I couldn't identify what it was, and they offered no suggestions, so I supposed it was not something important.
It took a visit to a cardiologist—for an entirely different reason—for him to suggest that I rapidly consult a dermatologist.
It was a carcinoma, and surgery removed it.
If ChatGPT had existed back then, this carcinoma would have been removed several years earlier.
I don't know, a lot of the things we used to take for granted as "hard" just aren't anymore. Writing a parser for a compiler? It used to be an endeavor that took several days, now an LLM will do it in 20 mins.
Same for annoying training jobs, with sufficient data you can probably let an LLM loose on it in autotrain mode and it will come back with a decent classifier.
For this particular case, it's probably worth it because a dermatologist visit is $200. If I were the NHS or Kaiser, I'd partner with other integrated insurance and service providers to collect as much training data as possible and fund classifier development.
> why people are excited about Jev-style decision models.
I think its a combination of being new to classification models and not using in anger.
LLM Decision models are still relatively new and suffer from a number of issues. Order of choices impacts probability, it starts to break down over 20 classifications and even how you word things impacts probability.
I have been incredibly impressed by LLMs for coding and data analysis and was expecting the LLMs to perform better so I was surprised by the results especially since this is an old model and nowhere near state of the art for skin cancer diagnosis.
For some types of applications, Jev is also useful as a stepping stone, you use it for the cases it's most confident so you already get gains from using it, and then collect all the usage to train your specialized model.
it's precisely because you can tune them to do that off a strong base
and because they start off generalized, you can train them against multiple of your workloads (they'll start with some generalized performance/world knowledge about them)
Which allows you to embed proprietary processes/data (world knowledge only you may have) across multiple domains into a model cheaply
I mean, it's the pareto principle, right? 80% of the applications require 20% of the intelligence. If I can drop in something without needing to create high quality training data that's good enough, that's great! Skin cancer probably isn't one of those applications.
> I used Inception v3 pretrained on everyday photographs and then fine-tuned all layers on 68,011 dermoscopic and clinical images from the ISIC Archive on a single H100 GPU.
>These 200 test lesions, and every other photograph of the same lesion or patient, were removed from its data (including the validation set) before training, so it had never seen them, and they were not used to choose the model or set its threshold.
This is just benchmaxxing. A benchmaxxed model will always beat an unbenchmaxxed model, but it will be brittle.
You can always beat a specific dataset with a small model by finetuning on a train/test split. But it doesn't generalize well to 'wild' images, because they are out-of-domain.
This is a good point and was an issue for skin cancer AI in the early days but has largely been solved now - newer models use ensembles of vision transformers and much greater number of training images and have been proven accurate in clinical trials. See for example https://skin-analytics.com/ (I am not linked in any way)
As people have mentioned, it isn't that surprising that a specially trained model beats a general purpose model.
However, I think it kinda misses the potential of llms for diagnosis. What happens if we also include journal notes? What if we let the model generate questions to ask the patient and include those? What if the images aren't perfect examples and happen to contain out of distribution content like a tattoo?
I found SkinGPT-4 when I was looking into this article. From the nature paper "among the 150 cases, a significant percentage of SkinGPT-4’s diagnoses (80.63%) were evaluated as correct or relevant by board-certified dermatologists." Which does outperform the CNN in this article.
I remember being blown away by CNNs and GANs when they first gained popularity. I do think, with skin cancer diagnosis, a transformer architecture is going to be the better bet.
The CNN architecture used here is definitely outdated. I used it as a baseline to compare to the LLMs and because the model is small enough to run using WebGPU in a phone browser for demonstration. Modern approaches use ensembles of vision transformers and have proven accurate in clinical trials
The problem is that they are 'everything boxes'. They're the first application that can actually lay claim to being a universal computing aid. What then happens is that people will start using their newfound hammer and before you know it everything becomes a nail, even if for a particular problem there is a better solution you can't beat 'nothing to invest, nothing to learn, get an answer in 10 seconds flat' from a convenience perspective. If I look at the things people use LLMs for I wonder why they take such an unsuitable tool, and then I marvel at the fact that they actually get more-or-less to where they would have gotten with the proper tools, and most likely in less time.
Slide rules displaced tables, calculators displaced slide rules and calculators themselves were displaced for the most part (they are still in use, in education but not nearly as much as they used to be outside of that) by the PC.
LLMs do that sort of thing for just about every information processing aid.
LLMs should be for general/high level work. For example, an LLM with the right harness could fetch and use that CNN, or train it's own CNN or other model to use for this task.
The harness in my experience makes a huge different too. Out of curiosity I tried having Gemini chat (with Argon) try to solve a level of this new puzzle game I'm linking (Order of the Sinking Star). It just got in an infinite loop and crashed. Then I try Antigravity with Flash 3.8 and it solved it easily. I saw the harness looked at other sokoban puzzles, coded some stuff, tried to replicate the level, etc. All things that a pure Gemini Chat doesn't really do.
LLMs are great for the high level harness, but if you want to count letters for example, it will still rely on "tools" like a piece of code to do it more reliably. Similarly in the above example, you wouldn't have the LLM directly try to analyze the images, but instead fetch and use the best tool for the job. But that can only be as good as the harness/tools it has access to.
This does make sense, however in view of the recent maths results and since there are tens of thousands of training images for skin cancer available on the public internet I was expecting the LLMs to perform a lot better than they did
That's just semantics around the word "performance".
We're messaging each other right now inside a heavily sandboxed virtual environment that interprets XML-like markup and runs a text-based JITted scripting language from the 90s. It doesn't really matter that it's extremely inefficient, since it's still effectively instant.
Actually not. Watt per task is a metric bothers more than LLM operators and users. We care about it a lot, and working a lot to measure and improve it, and we're not doing it in context of AI or LLMs.
Effectively instant means nothing, since everything is fast for small n. I love to give this example from a codebase I have worked on during my Ph.D. The problem which took half an hour under Matlab was done in ~40 seconds with mine, however, a thousand-step version of the same problem would take 11 hours. Saving a single second from that 40 second loop would shave my work by ~20 minutes, and this will compound for bigger problems still.
Just because we see the results instantly, doesn't mean it's fast or efficient by any means. It means we might forego optimizing it for sufficiently small n, if we don't care about that part of the code. That's all.
> The watts per task is abysmal compared to a desk calculator.
It depends on the calculator and task at hand. Consider graphing, integration or any programmable needs, and they might converge. ALU of a processor is pretty efficient at what it does.
> The app is less efficient only in an abstract, impractical way.
Again, depends, as I said above. You tend to brush efficiency away because you only mentally measure time to completion, only, which is understandable.
This is like all those “I fine tuned a model for <task> that beats <sota model>” stories. It’s normally possible with sufficient data to build train/test splits and fine tune a narrow model in a way that it outperforms general models, it’s a given that’s it’s possible with the right task definition and data.
The problem is the narrow models are always super fragile. You change the lighting, zoom level, skin tone, or give it a picture of a cat (or more dangerously a misaligned image) instead of a skin lesion, and it breaks completely.
None of this is new, it’s why narrow classifiers peaked almost 10 years ago and why large general transformer models took over. If you have a task where the data distribution is always identical and has no long tail or outliers then inception or resnet or whatever is great. Problem is these are vanishingly rare.
LLMs shouldn't be treated as if they were purpose-built for specific cases just because they generally output a plausible-looking answer, and this is not something that was discovered only recently
Here's a good heuristic for predicting the ability of a modern frontier LLM at any non-automatable task: Is it hugely overrepresented in their pre-training dataset (e.g. code), and were they specifically post-trained on it using reinforcement learning (e.g. coding)? If the answer to one or both is yes, you should expect them to be very good at it. Otherwise, you should expect them to be, at best, as good as an average human who's read a few books on the subject and is good at bullshitting.
This doesn't seem that surprising to me? One is tasked with a simple very narrow task. The other has a corpus including billions of other images and pieces of information. It'd seem to me that instinctively one would outperform the other, regardless of age.
It's cool that there's an in-browser skin classifier...but it says "It is not a state-of-the-art model". The obvious question is, is there a free in browser state-of-the-art model?
Would be difficult to fit in a browser as state of the art requires multiple vision transformers. Also it would not be sensible to provide access to models without human oversight as at this point in time all models have the potential to miss skin cancer.
This isn't really surprising. LLMs are intended to be generally intelligent. They are spiky on the things frontier labs tend to benchmark against (software/math) but are not intended or expected to be great at everything. The general intelligence unlocks new workflows, it does not perfect existing ones.
That being said, I'm sure a frontier LLM could make some improvements to your CNN.
This isn't surprising but it's good to have reporting to point to. The only way a general purpose LLM like these will match the performance of a bespoke classifier is if it has access to such a model.
It seems one is more specialized for the purpose and compare it against a general purpose LLM instead of a specalized/focused LLM?
This is like saying I compared this against photos for sure stored in a home swimming pool (CNN), compared to photos presumably stored in an ocean (generalized LLM).
I use LLMs where I can, but I'm no automatic fan of LLMs, a healthier comparison
here seems to be what if a general purpose LLM was provided those CNN images, and perhaps what does a fine tuned local model with the CNN images perform like?
It still might not be better, but I would suspect it's closer, and a more realistic apples to apples comparison.
General assistant models compared to task specific models seem to be a well known spot where things don't always fit, but might be improving.
Which CNN? A fine-tuned version of Inception v3, fine-tuned on 68,011 dermatology images
Which LLM? GPT-6 Astra and Claude Fable 5.1, not specified exactly which version
Obviously this is a bit like comparing an 100 iq radiologist, 20 years experience, with a just-tested-at-iq-130 6 year old and observing the radiologist beat the crap out of the kid. Was anybody really expecting a different outcome? Let's see after fine tuning.
Btw: the actual lesson from that network when it came out, which is really important is that Inception v6, the Google trained version (ie. can recognize cats, halloween masks and airplanes) beat, with a wide margin, the version that was trained from scratch on clinical images (and still didn't do so well, 62%. Not even really good enough for a first triage).
You know AI companies are selling the dream that you don't even need that just-tested-at-iq-130 6 year old. They're selling the dream that an sub-average iq 6 year old can do it.
So yes, this is an important finding that LLMs are not better than experience.
It's good to be reminded that "frontier models" aren't particularly good at most tasks. They happen to be good at programming, since there's a lot of training data available, and the task has parallels with report writing and story telling. For skin cancer? Probably very little. Also, images of skin cancer probably don't translate that well to a Large Language Model's internal representation.
It's very strange to me that articles like this get upvoted, because they are coming from this assumption that LLMs are better at everything and that it's notable and newsworthy when there's something that they aren't actually better at.
I don't think we should need to be reminded that frontier models aren't particularly good at most tasks. Just like we shouldn't need to be reminded that a hammer is not particularly good for driving a screw, or writing an essay.
It's an admission of magical thinking about a tool that's made out of the same material of every other digital tool since the invention of computing.
There is no such thing as “belong to the front page”. Flag if you think it doesn’t have its place. Otherwise, it’s up to the crowd to decide what they want upvoted. HN front page reflects upvoter interest, nothing else
The reason I say this is that I suspect many people are reflexively upvoting the title without reading the article. If they read the article they would realize the evidence is flimsy and not really worth reading.
I will not flag it because the post is fine, it's the voters who I believe are acting poorly.
This is an example of why I'm a little stumped about why people are excited about Jev-style decision models. For any application at scale it'd be way more accurate to have a dedicated classifier using the right architecture (neural net, bayesian, etc.) LLMs drive the cost of making those models down to near zero.
Just use the LLM to make a thing that's actually good at what you need it to do.
> Just use the LLM to make a thing that's actually good at what you need it to do.
This is the end state of most imagined use cases for “agentic” software IMO. Use the AI to build software, with a classifier or LLM as the expansion joint in awkward spots.
This is exactly what we’ve been doing.
For example, a client want a classifier to decide to make a go/no go decision on roofing jobs based on weather. The end result doesn’t use LLMs at all, but does use a lot of other techniques including human feedback when it makes the wrong call.
(Now the work crew gets the decision at 5 AM instead of 5:30 and the guy making the decision doesn’t have to wake up at all.)
I do agree with you, but I do think you're also making the classic "curse of knowledge" mistake.
You have to know the right words to ask the LLM to make and train a classifier. You and I already know the words. But they are not actually that trivial to learn for an average joe, and people are lazy.
I don't believe this is true with current frontier coding agents. Or at least, it's becoming increasingly less true, such that I think it will effectively be nullified in ~6 months.
I'm gonna cite myself here:
«We came full circle on the problem that also exists with Google Search, and why so many people don't get good results out of it.» [0] It is not that access to all information was hidden to anyone before, it was harder to find it, sure. Now it can be found easily and aggregated, however, it is not as simple as saying "Do X" and believe that X is correct. Recall the infamous Swing & Product Management meme? [1] The problem is the absence of information you provide, the LLM will have to make a guess of what you mean, much like the human. Likelihoods of this guess being correct depends on the domain and background. So yes, it'll likely build whatever you ask it (I do think we're there already), but if that matches what you intended to actually get is vastly different. You'll be settling with "Good enough" though and declare it working.
Others already pointed out that 6 months seem to be the constant of distance, irrespective of the date.
[0] - https://news.ycombinator.com/item?id=50063427#50063744
[1] - https://www.productftw.com/productftw-2-the-tire-swing-carto...
I'm done waiting 6 months. Just let me know the night before when I don't have to come in to work tomorrow.
Just ask the LLM for these words, obviously !
It's funny you mention this, because I've been thinking lately that all the focus on code quality benchmarks for LLMs seems to miss something. In my experience, LLMs aren't actually all that good at prompting other LLMs, and that's exactly the skill that matters for proper subagent usage.
The corpus of code available to train LLMs on is rich in both volume and quality. The same can't be said for LLM prompts. Prompts only came into existence alongside LLMs themselves, so most of the prompts out there are actually LLM generated, and training on them causes every model to overfit to the same patterns.
LLMs can give clear and concise directives to other LLMs, but only for basic tasks. Beyond that, they frequently overstuff their prompts with too much prose and too much direction, and worse, they often fill them with subtle contradictions.
Of course, there's a spectrum here. Some models turn out to be better at this than others, and I suspect this is what differentiates Fable-type models from the rest, and why vendors have started keeping their models' reasoning confidential.
The "agent factories" available to employees at the labs and hyperscalers clearly prove that agent delegation does work. Today, maybe that capacity lies with Fable class models, as you pointed out, but there is no doubt with opus 5.5 that this capability will be available to cheaper models as they get better.
That's what I've been doing in areas I lack knowledge tbh. Instead of a half assed attempt at describing what I need, I ask an llm to take my base idea and write it up using industry standard jargon as if I were commissioning a team to do the work. Has been working pretty well so far!
Okay so I just asked it "What question should I be asking you but I don't because I'm not smart enough to think of it?" Response was kind of interesting I have to say.
It gave me a very helpful answer, my project will be improving just because I asked.
I did this just now, ironically before I even saw this discussion. I ended up questioning the AI about humanism.
So what? The average joe shouldn't be trying to diagnose people with cancer.
Healthcare is more than having medical knowledge.
Ten years ago, in France, two doctors (one in a professional setting and the other during a private consultation) asked me about a wound I had on my shoulder; obviously I couldn't identify what it was, and they offered no suggestions, so I supposed it was not something important.
It took a visit to a cardiologist—for an entirely different reason—for him to suggest that I rapidly consult a dermatologist.
It was a carcinoma, and surgery removed it. If ChatGPT had existed back then, this carcinoma would have been removed several years earlier.
Maybe if chatgpt existed back then you'd have had your carcinoma extracted along with 10 other benign anomalies.
Automation doesn't make up for carelessness.
Skin cancer is easily treated with excisions if it’s caught early. Better to have ten small false positives than a single miss.
[delayed]
This is like saying we don’t need Dropbox because we have FTP servers.
People vastly underrate the patience of people to learn and setup a classifier. Jev is one API call. Training a good classifier is hard.
I don't know, a lot of the things we used to take for granted as "hard" just aren't anymore. Writing a parser for a compiler? It used to be an endeavor that took several days, now an LLM will do it in 20 mins.
Same for annoying training jobs, with sufficient data you can probably let an LLM loose on it in autotrain mode and it will come back with a decent classifier.
It’s not the software development that’s hard. It’s the data science and domain knowledge to build a good training set and a meaningful objective.
We don't need Dropbox, it's a heap of shit.
With enough marketing you can sell anything. Also a decent way to make a product is to do the hard part for someone else.
I've been training classifiers for ten years, it used to be hard. Now it's hard to do really but it's trivial to do well enough.
> Just use the LLM to make a thing that's actually good at what you need it to do.
I don’t have a deep understanding of this topic, but wouldn’t you also need a good amount of relevant data to train on?
Not if you take advantage of open-weight models and make an adaptor or distill its knowledge into the thing you want.
There's a significant engineering cost to having a custom classifier.
An off-the-shelf API has way less friction, if the results are good enough why bother?
For this particular case, it's probably worth it because a dermatologist visit is $200. If I were the NHS or Kaiser, I'd partner with other integrated insurance and service providers to collect as much training data as possible and fund classifier development.
> why people are excited about Jev-style decision models.
I think its a combination of being new to classification models and not using in anger.
LLM Decision models are still relatively new and suffer from a number of issues. Order of choices impacts probability, it starts to break down over 20 classifications and even how you word things impacts probability.
Once you have a hammer everything is a nail.
Even the hammer?
I have been incredibly impressed by LLMs for coding and data analysis and was expecting the LLMs to perform better so I was surprised by the results especially since this is an old model and nowhere near state of the art for skin cancer diagnosis.
For some types of applications, Jev is also useful as a stepping stone, you use it for the cases it's most confident so you already get gains from using it, and then collect all the usage to train your specialized model.
You can train gradient boosted trees on jev style features, it is a quick “good enough” approach for a lot of stuff
it's precisely because you can tune them to do that off a strong base
and because they start off generalized, you can train them against multiple of your workloads (they'll start with some generalized performance/world knowledge about them)
Which allows you to embed proprietary processes/data (world knowledge only you may have) across multiple domains into a model cheaply
They make very good classifiers and are easy to fine tune for specific use cases.
I mean, it's the pareto principle, right? 80% of the applications require 20% of the intelligence. If I can drop in something without needing to create high quality training data that's good enough, that's great! Skin cancer probably isn't one of those applications.
> I used Inception v3 pretrained on everyday photographs and then fine-tuned all layers on 68,011 dermoscopic and clinical images from the ISIC Archive on a single H100 GPU.
>These 200 test lesions, and every other photograph of the same lesion or patient, were removed from its data (including the validation set) before training, so it had never seen them, and they were not used to choose the model or set its threshold.
This is just benchmaxxing. A benchmaxxed model will always beat an unbenchmaxxed model, but it will be brittle.
You can always beat a specific dataset with a small model by finetuning on a train/test split. But it doesn't generalize well to 'wild' images, because they are out-of-domain.
> This is just benchmaxxing.
I believe the normal term is "gaming a benchmark test".
Nothing in what you quoted is related to that. It's a normal way to train and test a classifier.
The only way to determine if it's brittle is to test on a different dataset and look at the sensitivity and specificity.
Even so, there is nothing wrong with a model that is trained to a narrow domain when it doesn't need to deal with "wild" images.
This is a good point and was an issue for skin cancer AI in the early days but has largely been solved now - newer models use ensembles of vision transformers and much greater number of training images and have been proven accurate in clinical trials. See for example https://skin-analytics.com/ (I am not linked in any way)
Could you explain why this is benchmaxxing?
As people have mentioned, it isn't that surprising that a specially trained model beats a general purpose model.
However, I think it kinda misses the potential of llms for diagnosis. What happens if we also include journal notes? What if we let the model generate questions to ask the patient and include those? What if the images aren't perfect examples and happen to contain out of distribution content like a tattoo?
I found SkinGPT-4 when I was looking into this article. From the nature paper "among the 150 cases, a significant percentage of SkinGPT-4’s diagnoses (80.63%) were evaluated as correct or relevant by board-certified dermatologists." Which does outperform the CNN in this article.
I remember being blown away by CNNs and GANs when they first gained popularity. I do think, with skin cancer diagnosis, a transformer architecture is going to be the better bet.
The CNN architecture used here is definitely outdated. I used it as a baseline to compare to the LLMs and because the model is small enough to run using WebGPU in a phone browser for demonstration. Modern approaches use ensembles of vision transformers and have proven accurate in clinical trials
And a calculator outperforms LLMs at multiplication and division...
In a world where masses believe that LLMs are "everything boxes", some reality check is always good.
The problem is that they are 'everything boxes'. They're the first application that can actually lay claim to being a universal computing aid. What then happens is that people will start using their newfound hammer and before you know it everything becomes a nail, even if for a particular problem there is a better solution you can't beat 'nothing to invest, nothing to learn, get an answer in 10 seconds flat' from a convenience perspective. If I look at the things people use LLMs for I wonder why they take such an unsuitable tool, and then I marvel at the fact that they actually get more-or-less to where they would have gotten with the proper tools, and most likely in less time.
Slide rules displaced tables, calculators displaced slide rules and calculators themselves were displaced for the most part (they are still in use, in education but not nearly as much as they used to be outside of that) by the PC.
LLMs do that sort of thing for just about every information processing aid.
LLMs should be for general/high level work. For example, an LLM with the right harness could fetch and use that CNN, or train it's own CNN or other model to use for this task.
The harness in my experience makes a huge different too. Out of curiosity I tried having Gemini chat (with Argon) try to solve a level of this new puzzle game I'm linking (Order of the Sinking Star). It just got in an infinite loop and crashed. Then I try Antigravity with Flash 3.8 and it solved it easily. I saw the harness looked at other sokoban puzzles, coded some stuff, tried to replicate the level, etc. All things that a pure Gemini Chat doesn't really do.
LLMs are great for the high level harness, but if you want to count letters for example, it will still rely on "tools" like a piece of code to do it more reliably. Similarly in the above example, you wouldn't have the LLM directly try to analyze the images, but instead fetch and use the best tool for the job. But that can only be as good as the harness/tools it has access to.
We don't disagree by a great margin. I'm not an anti-LLM person. I'm anti-hype in this case.
LLMs are pretty good at writing calculators.
I'd rather use a verified one. Scientific calculations on a ALU is tricky. There are many traps and hardware limitations to consider.
You say this, but the reaction I get from colleagues is increasingly "why bother training new models when you can just ask an LLM?"
Doesn't matter the domain or the task, the answer from some researchers is "LLM" always.
This does make sense, however in view of the recent maths results and since there are tens of thousands of training images for skin cancer available on the public internet I was expecting the LLMs to perform a lot better than they did
Most people need to be reminded of this. Repeatedly.
And if you give the LLM a simple calculator MCP, then the calculator's outperformance disappears.
Nope. It'll be more efficient and faster.
That's just semantics around the word "performance".
We're messaging each other right now inside a heavily sandboxed virtual environment that interprets XML-like markup and runs a text-based JITted scripting language from the 90s. It doesn't really matter that it's extremely inefficient, since it's still effectively instant.
Actually not. Watt per task is a metric bothers more than LLM operators and users. We care about it a lot, and working a lot to measure and improve it, and we're not doing it in context of AI or LLMs.
Effectively instant means nothing, since everything is fast for small n. I love to give this example from a codebase I have worked on during my Ph.D. The problem which took half an hour under Matlab was done in ~40 seconds with mine, however, a thousand-step version of the same problem would take 11 hours. Saving a single second from that 40 second loop would shave my work by ~20 minutes, and this will compound for bigger problems still.
Just because we see the results instantly, doesn't mean it's fast or efficient by any means. It means we might forego optimizing it for sufficiently small n, if we don't care about that part of the code. That's all.
Generally when I want to use a calculator, I use an app on my phone or computer. The watts per task is abysmal compared to a desk calculator.
The app is less efficient only in an abstract, impractical way.
> The watts per task is abysmal compared to a desk calculator.
It depends on the calculator and task at hand. Consider graphing, integration or any programmable needs, and they might converge. ALU of a processor is pretty efficient at what it does.
> The app is less efficient only in an abstract, impractical way.
Again, depends, as I said above. You tend to brush efficiency away because you only mentally measure time to completion, only, which is understandable.
Imagine performing literally trillions of matrix multiplications in order to figure out the answer to 1 + 2.
Your computer/phone just performed millions of them in order to display the contents of my comment to you.
Pretty much what our brain does.
Sure, but what about performance per joule?
If performance per joule is important in this loosely-defined hypothetical scenario, then it's important.
If it's not, then it's not.
This is like all those “I fine tuned a model for <task> that beats <sota model>” stories. It’s normally possible with sufficient data to build train/test splits and fine tune a narrow model in a way that it outperforms general models, it’s a given that’s it’s possible with the right task definition and data.
The problem is the narrow models are always super fragile. You change the lighting, zoom level, skin tone, or give it a picture of a cat (or more dangerously a misaligned image) instead of a skin lesion, and it breaks completely.
None of this is new, it’s why narrow classifiers peaked almost 10 years ago and why large general transformer models took over. If you have a task where the data distribution is always identical and has no long tail or outliers then inception or resnet or whatever is great. Problem is these are vanishingly rare.
LLMs shouldn't be treated as if they were purpose-built for specific cases just because they generally output a plausible-looking answer, and this is not something that was discovered only recently
My dog outperforms the world's top researchers in an unassisted 100 meter sprint.
If your goal is sprinting 100m, using the dog seems like a better idea then, no matter how good the researchers are in other areas.
I think you’re onto something.
Here's a good heuristic for predicting the ability of a modern frontier LLM at any non-automatable task: Is it hugely overrepresented in their pre-training dataset (e.g. code), and were they specifically post-trained on it using reinforcement learning (e.g. coding)? If the answer to one or both is yes, you should expect them to be very good at it. Otherwise, you should expect them to be, at best, as good as an average human who's read a few books on the subject and is good at bullshitting.
The most sane comment I've seen in this thread
An example of "Dumb it down"
This doesn't seem that surprising to me? One is tasked with a simple very narrow task. The other has a corpus including billions of other images and pieces of information. It'd seem to me that instinctively one would outperform the other, regardless of age.
Great luckily you are not in charge of a pension fund, thag wkuld be horrible
It's cool that there's an in-browser skin classifier...but it says "It is not a state-of-the-art model". The obvious question is, is there a free in browser state-of-the-art model?
Would be difficult to fit in a browser as state of the art requires multiple vision transformers. Also it would not be sensible to provide access to models without human oversight as at this point in time all models have the potential to miss skin cancer.
This isn't really surprising. LLMs are intended to be generally intelligent. They are spiky on the things frontier labs tend to benchmark against (software/math) but are not intended or expected to be great at everything. The general intelligence unlocks new workflows, it does not perfect existing ones.
That being said, I'm sure a frontier LLM could make some improvements to your CNN.
This isn't surprising but it's good to have reporting to point to. The only way a general purpose LLM like these will match the performance of a bespoke classifier is if it has access to such a model.
Yeah obviously. Xgboost outperforms llms in algorithmic trading, even applying the token prediction method to markets xgboost is effectively better.
Llms have their applications. Other models still do.
yeah why is this news. breaking: we still need models besides llms.
I'd like to see it compared to chinese models like DeepSeek, Kimi, GLM, etc just for the sake of research
It seems one is more specialized for the purpose and compare it against a general purpose LLM instead of a specalized/focused LLM?
This is like saying I compared this against photos for sure stored in a home swimming pool (CNN), compared to photos presumably stored in an ocean (generalized LLM).
I use LLMs where I can, but I'm no automatic fan of LLMs, a healthier comparison here seems to be what if a general purpose LLM was provided those CNN images, and perhaps what does a fine tuned local model with the CNN images perform like?
It still might not be better, but I would suspect it's closer, and a more realistic apples to apples comparison.
General assistant models compared to task specific models seem to be a well known spot where things don't always fit, but might be improving.
fyi CNN: convolutional neural network
Which CNN? A fine-tuned version of Inception v3, fine-tuned on 68,011 dermatology images
Which LLM? GPT-6 Astra and Claude Fable 5.1, not specified exactly which version
Obviously this is a bit like comparing an 100 iq radiologist, 20 years experience, with a just-tested-at-iq-130 6 year old and observing the radiologist beat the crap out of the kid. Was anybody really expecting a different outcome? Let's see after fine tuning.
Btw: the actual lesson from that network when it came out, which is really important is that Inception v6, the Google trained version (ie. can recognize cats, halloween masks and airplanes) beat, with a wide margin, the version that was trained from scratch on clinical images (and still didn't do so well, 62%. Not even really good enough for a first triage).
You know AI companies are selling the dream that you don't even need that just-tested-at-iq-130 6 year old. They're selling the dream that an sub-average iq 6 year old can do it.
So yes, this is an important finding that LLMs are not better than experience.
I'm sorry but as an MLE this is not a remotely useful article, and does not belong on the front page.
It's comparing a fine-tuned CNN to general purpose models on a tiny dataset.
It's good to be reminded that "frontier models" aren't particularly good at most tasks. They happen to be good at programming, since there's a lot of training data available, and the task has parallels with report writing and story telling. For skin cancer? Probably very little. Also, images of skin cancer probably don't translate that well to a Large Language Model's internal representation.
It's very strange to me that articles like this get upvoted, because they are coming from this assumption that LLMs are better at everything and that it's notable and newsworthy when there's something that they aren't actually better at.
I don't think we should need to be reminded that frontier models aren't particularly good at most tasks. Just like we shouldn't need to be reminded that a hammer is not particularly good for driving a screw, or writing an essay.
It's an admission of magical thinking about a tool that's made out of the same material of every other digital tool since the invention of computing.
There is no such thing as “belong to the front page”. Flag if you think it doesn’t have its place. Otherwise, it’s up to the crowd to decide what they want upvoted. HN front page reflects upvoter interest, nothing else
The reason I say this is that I suspect many people are reflexively upvoting the title without reading the article. If they read the article they would realize the evidence is flimsy and not really worth reading.
I will not flag it because the post is fine, it's the voters who I believe are acting poorly.