It's incredible Simon still believes pelicans on bikes aren't part of the training set, despite hundreds of them on blogs, forums, and Github. Stuff we put in our company blog shows up known by LLMs 6 months later, and we have 1000x less traffic than Simon's own website
the nature of the test was to see if the models can effectively compose an image of a novel concept outside the training set. If they are trained on it, it ceases to be an interesting test to some extent.
it's still interesting because there's no pelican-on-bike model, and if you're training a model well enough, then it should be obvious when a model has reached "AGI" or whatever.
Simon has stated a few times that he knows it’s possible that pelicans could be in the training sets. He also has other tests he doesn’t share publicly. He’s just a fan of pelicans.
More to it, the actual bloody companies are using them as a reference. Maybe it’s a 3d version, not an svg - but it clearly shows they’re on the radar of these companies.
This is a sight-reading test. If a musician practices a piece for thousands of hours, it would no longer be an effective sight reading / creativity test. The purpose of the test was to see how models would compose something novel requiring the ability to compose orthogonal, normally unrelated, components into a coherent image.
They can be in the training set but not deliberately trained for. There may be a lot of people posting pelican svgs, but not typically because they're high quality and worth replicating.
> How does the prompt “Generate an SVG of a pelican riding a bicycle” add up to 95 input tokens? OpenAI’s tokenizer counts 10, Anthropic’s counts 10 for Opus 4.6, 30 for Opus 4.7 and 25 for Sonnet 5/Fable 5. Prompting “hi” to Kimi K3 counted 86 tokens, suggesting there may be an 85 token hidden system prompt. It refused to leak it though.
I wonder how the Chinese labs are training a 3 trillion parameter model on what has to be vastly smaller compute resources. If the U.S. compute advantage is persistent, it's hard to imagine that Chinese labs will be able to keep pace forever, as a matter of physics, but... so far they seem to be doing just fine.
The idea is not to use pelicans on bikes but a similarly random non-sensical prompts: crows on scooters, squirrels in a moon rover etc. Then pick another one for another for next cross-llm evaluation.
Do any of the vision models render the SVG and look at the result.
Perhaps more importantly can they do that during reinforcement training. Learning how to critically analyse the appearance of what it generates would be quite useful.
Manually feeding images back to models has been hilariously bad in the past which suggests that relating something it sees to something it wrote is not an ability it is very good at.
My personal benchmark for new models has been to compare video making skills with something like remotion. Usually reveals if they have any "taste" or outside the box thinking.
I'm starting to not trust any "benchmarks" when it comes to frontier models at least. As an example Sol feels the most "gets stuff done" but has zero taste, or any capability to surprise.
And for frontier models I go one step ahead and try to recreate a complex animation video, with the ability for the model to review its own work. And at this Fable is still the top one.
Ex: https://www.youtube.com/watch?v=uDAeAuYyl0E (recreation of Claude announcement video) and https://www.youtube.com/watch?v=cSsVNtGPOIg (recreation of a fireship video). Sol did something similar but you can instantly tell its AI slop from very small things, and it just has no narrative or thought put into the writing.
And on creativity at least visually, Gemini 3.1 pro is somehow still up there. But its really hindered by its inability to use tool calls effectively or make a long term plan.
Another day, another model and another pelican :-)
I can't help but wonder where is the trend going? What will we have in five years? Maybe it will all have puttered out, and we will have moved to the next thing? Or maybe the prompt then will be "make a pelican ride a bicycle", and out will come the genetic code for a giant pelican with extremities suitable for a handle bar and pedals, and an inborn affinity to ride bicycles?
You say it's performative joke, but it all depends what you're using model for. So far the rule has been quite straightforward, better models consistently renders pelican in higher quality, I've yet to see an exception. It is also a good enough (for me at least) test for "taste" the model has.
> better models consistently renders pelican in higher quality
The article literally avoid making this argument and gives counterexamples to this statement.
Imagine what amazing SVG generators we could have if Simon had randomized the target image from the start (and companies wouldn't just overfit on pelicans).
You’re reading a personal blog and complaining about an open source personal project he runs and distributes for free. He’s allowed to talk about his personal work on his personal blog. Especially considering the cli utility he talks about is directly related to the post.
Imagine complaining about someone generating valuable content for free and not packaging it to your personal tastes.
K3 is as expensive as Sonnet, not great at writing English, is handing IP back to the Chinese, and once open source will be difficult to run at scale without the compute that OpenAI and Anthropic have largely grabbed.
Sorry, how again is this the end of the frontier labs?
According to some benchmarks has the coding capability of Opus at the price of Sonnet, supposedly will be open weights and is not subject to random trade wars with allied states.
Engineers get unbelievably silly about evaluating costs of things.
"The tokens are so expensive!" Oh my sweet child, how much would even the least capable human effort cost? This is what the executives properly understand that the programmers don't.
they're comparing to similar capability llm models, not humans. If one dishwasher does job at similar quality as another dishwasher, but using 30% more water and energy, you wouldn't compare to how much it costs human to do the same work, it would make no sense.
> they're comparing to similar capability llm models, not humans
25 cents is 10x the cost of 2.5 cents, but it's still extremely cheap for the product. It's very much the wrong comparison for a world where the primary competition is still humans who need to eat, and it treats percentage differences as more important than absolute differences when the opposite is true.
Well first of all, any non-trivial use of LLMs is going to be orders of magnitude more tokens than this, usually multiple millions at minimum. Benchmarks are just benchmarks after all.
Secondly, humans vs LLMs are apples vs oranges. It makes no more sense to compare human costs vs LLM costs as it would have to compare human costs vs calculator costs. LLMs are faster and cheaper but extremely different beasts with different limitations. Humans do not one-shot SVGs of pelicans riding bicycles, and they do not charge in tokens.
Comparing LLM cost efficiency is not something that should need to be defended. It's quite straightforward and reasonable...
It's incredible Simon still believes pelicans on bikes aren't part of the training set, despite hundreds of them on blogs, forums, and Github. Stuff we put in our company blog shows up known by LLMs 6 months later, and we have 1000x less traffic than Simon's own website
Pelicans and bikes can be in the training set without them training for this specific benchmark.
Yes and that would improve its ability to draw SVGs of pelicans on bikes, no?
and that is bad because ?
the nature of the test was to see if the models can effectively compose an image of a novel concept outside the training set. If they are trained on it, it ceases to be an interesting test to some extent.
it's still interesting because there's no pelican-on-bike model, and if you're training a model well enough, then it should be obvious when a model has reached "AGI" or whatever.
Simon has stated a few times that he knows it’s possible that pelicans could be in the training sets. He also has other tests he doesn’t share publicly. He’s just a fan of pelicans.
More to it, the actual bloody companies are using them as a reference. Maybe it’s a 3d version, not an svg - but it clearly shows they’re on the radar of these companies.
Did you read the post? It's not even that long. He explicitly mentions this...
Are they responding to: “I’m still not convinced that labs are training for the benchmark—if they were, I’d expect much better results.”
Clearly not. There's a subset of HN users who rush to post this same thing every single time.
Imagine if we applied this train of logic to humans.
"That artist saw a pelican at the beach once!" [cue the outrage] "He's not a real artist, he's a cheater and produces nothing original!"
This is a sight-reading test. If a musician practices a piece for thousands of hours, it would no longer be an effective sight reading / creativity test. The purpose of the test was to see how models would compose something novel requiring the ability to compose orthogonal, normally unrelated, components into a coherent image.
Except, of course, LLMs are not humans, and they do not learn or "reason" in a way which even remotely resembles humans.
Plus obviously humans can still overfit to a specific style of test.
They can be in the training set but not deliberately trained for. There may be a lot of people posting pelican svgs, but not typically because they're high quality and worth replicating.
> How does the prompt “Generate an SVG of a pelican riding a bicycle” add up to 95 input tokens? OpenAI’s tokenizer counts 10, Anthropic’s counts 10 for Opus 4.6, 30 for Opus 4.7 and 25 for Sonnet 5/Fable 5. Prompting “hi” to Kimi K3 counted 86 tokens, suggesting there may be an 85 token hidden system prompt. It refused to leak it though.
This is quite possibly reasoning-effort prompt which is injected before the opening <think> token whenever you set a custom reasoning effort, see e.g. DeepSeek-V4 max mode prompt: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main...
I wonder how the Chinese labs are training a 3 trillion parameter model on what has to be vastly smaller compute resources. If the U.S. compute advantage is persistent, it's hard to imagine that Chinese labs will be able to keep pace forever, as a matter of physics, but... so far they seem to be doing just fine.
The idea is not to use pelicans on bikes but a similarly random non-sensical prompts: crows on scooters, squirrels in a moon rover etc. Then pick another one for another for next cross-llm evaluation.
Do any of the vision models render the SVG and look at the result.
Perhaps more importantly can they do that during reinforcement training. Learning how to critically analyse the appearance of what it generates would be quite useful.
Manually feeding images back to models has been hilariously bad in the past which suggests that relating something it sees to something it wrote is not an ability it is very good at.
My personal benchmark for new models has been to compare video making skills with something like remotion. Usually reveals if they have any "taste" or outside the box thinking.
I'm starting to not trust any "benchmarks" when it comes to frontier models at least. As an example Sol feels the most "gets stuff done" but has zero taste, or any capability to surprise.
And for frontier models I go one step ahead and try to recreate a complex animation video, with the ability for the model to review its own work. And at this Fable is still the top one. Ex: https://www.youtube.com/watch?v=uDAeAuYyl0E (recreation of Claude announcement video) and https://www.youtube.com/watch?v=cSsVNtGPOIg (recreation of a fireship video). Sol did something similar but you can instantly tell its AI slop from very small things, and it just has no narrative or thought put into the writing.
https://mesmer.tools/benchmarks/ai-video-generation , I usually put basic ones here.
And on creativity at least visually, Gemini 3.1 pro is somehow still up there. But its really hindered by its inability to use tool calls effectively or make a long term plan.
Is there a gallery of all pelicans generated by simon over time?
https://simonwillison.net/tags/pelican-riding-a-bicycle/ isn’t quite a gallery, but pretty close.
If Simon reads this debate, I would gladly vote for such a gallery. It would belong to "digital heritage of mankind".
Don't see why we have to have this spammed every model release when Fable class models perform the same as Opus on basic tasks like these.
I would be surprised if pelican svgs are not part of the training corpus rn
If that were the case then it'd do a way better job. Think experienced artist level.
Another day, another model and another pelican :-)
I can't help but wonder where is the trend going? What will we have in five years? Maybe it will all have puttered out, and we will have moved to the next thing? Or maybe the prompt then will be "make a pelican ride a bicycle", and out will come the genetic code for a giant pelican with extremities suitable for a handle bar and pedals, and an inborn affinity to ride bicycles?
You are thinking too hard on this. This entire "benchmark" is a performative joke for attention that only works on HN.
> What will we have in five years? Maybe it will all have puttered out, and we will have moved to the next thing?
We will just have more of the same.
You say it's performative joke, but it all depends what you're using model for. So far the rule has been quite straightforward, better models consistently renders pelican in higher quality, I've yet to see an exception. It is also a good enough (for me at least) test for "taste" the model has.
> better models consistently renders pelican in higher quality The article literally avoid making this argument and gives counterexamples to this statement.
I’m excited for this specific brand of survival horror.
Imagine what amazing SVG generators we could have if Simon had randomized the target image from the start (and companies wouldn't just overfit on pelicans).
If anyone wants to try SVG generation from different models, I made this: https://codeinput.com/svg (here is an older generation: https://codeinput.com/s/5KEGl1e3rB3)
You still need an OpenRouter API Key and be careful this can burn quite a bit of money.
Imagine shilling some CLI tools no one uses in this post.
Lighten up.
You’re reading a personal blog and complaining about an open source personal project he runs and distributes for free. He’s allowed to talk about his personal work on his personal blog. Especially considering the cli utility he talks about is directly related to the post.
Imagine complaining about someone generating valuable content for free and not packaging it to your personal tastes.
K3 is as expensive as Sonnet, not great at writing English, is handing IP back to the Chinese, and once open source will be difficult to run at scale without the compute that OpenAI and Anthropic have largely grabbed.
Sorry, how again is this the end of the frontier labs?
According to some benchmarks has the coding capability of Opus at the price of Sonnet, supposedly will be open weights and is not subject to random trade wars with allied states.
Competition is always good.
You mean the scale that AWS provides with Bedrock?
Bedrock needs to actually update their chinese models to the newest versions for this to matter.
> This is expensive—the pelican cost 25 cents!
Engineers get unbelievably silly about evaluating costs of things.
"The tokens are so expensive!" Oh my sweet child, how much would even the least capable human effort cost? This is what the executives properly understand that the programmers don't.
they're comparing to similar capability llm models, not humans. If one dishwasher does job at similar quality as another dishwasher, but using 30% more water and energy, you wouldn't compare to how much it costs human to do the same work, it would make no sense.
> they're comparing to similar capability llm models, not humans
25 cents is 10x the cost of 2.5 cents, but it's still extremely cheap for the product. It's very much the wrong comparison for a world where the primary competition is still humans who need to eat, and it treats percentage differences as more important than absolute differences when the opposite is true.
Well first of all, any non-trivial use of LLMs is going to be orders of magnitude more tokens than this, usually multiple millions at minimum. Benchmarks are just benchmarks after all.
Secondly, humans vs LLMs are apples vs oranges. It makes no more sense to compare human costs vs LLM costs as it would have to compare human costs vs calculator costs. LLMs are faster and cheaper but extremely different beasts with different limitations. Humans do not one-shot SVGs of pelicans riding bicycles, and they do not charge in tokens.
Comparing LLM cost efficiency is not something that should need to be defended. It's quite straightforward and reasonable...
Would anyone pay a human to create an SVG of a pelican riding a bike?
Well, no, not now they won’t.
In fact humans get paid to create SVGs of all kinds of things.