"turn all the lights on/off" and "it's too dark in the bathroom" worked for me, but anything less direct didn't. "it's too cold" actually made it turn the thermostat down ("it's cold" made it... turn the lights down?)! Although the confidence on the bad responses was pretty low, so it might be worth adding a threshold to the demo.
Or maybe it just has a weird thermostat down bias? "make it hot" also had it turn it down (specifically it went from 20->18, or at least tried to, the UI still showed 20), with high confidence. Also might have a bit of a Celsius vs Fahrenheit confusion. Neat concept, but I might not want to let it control the oven at the moment.
The laptop demo worked better until I tried to open the mail app. "Check mail" kept opening the browser with an error, and "check email" makes a note with the text "email", "open email" goes to "https://api.email.com/v1/email" in my real browser, but "open mail" does work.
And I presume the "reasoning" isn't very trustworthy? In the car I got "'turn it up' means lower volume -> set_volume with lower value." For the house, reasoning would correctly say that I wanted the alarm off, but it didn't actually do it.
Wondered if it'd turn on the lights in the bathroom with these:
"I need a wee" -> tries to play music because "wee" is a genre
"I need a wee wee" -> starts the vaccuum in the bathroom
"I'm going to the toilet" -> says it'll turn on the toilet, and I'm not totally sure what that entails.
"I'm going to the toilet and can't see" -> reasons that lights should be on in the bathroom, then chooses again to turn on the toilet.
"I'm going to the toilet and can't see where I'm going" -> reasoning is "'going to the toilet' -> control_device with device 'coffee maker' (toilet implies coffee maker)"
"I'm going to the toilet and can't see where I'm going because it is too dark" -> "'dark' -> direction 'dark'; adjust_lights with brightness 100 for darker light"" and chooses to turn the lights in the living room to "dark" which fails.
At this point the vacuum is in a dark bathroom, the living room is 100% brightness and playing "wee". At least there's coffee.
thanks for these haha, you can actually edit the tools and/or their descriptions, the demo is just a "get started" preset. But still we do have room for reasoning improvement!
I have an idea for a use case for this, and I'm wondering if you think it makes sense or if you have any thoughts on the approach.
I'm a big fan of OpenStreetMap, and I enjoy editing it from my computer. From my phone, I find it quite tedious trying to make sure I type in the phone number exactly correctly and double-check it, or find and select the right field from the large list of fields available in Upredor.
Generally, how it works is I see a restaurant, and there's a sign. I know that it says, "Cash only. Here's the phone number. Here's the opening hours." What would be really cool is if I could just speak to the phone and say, "Hey, here's the information about this place." It would automatically use your location to detect what places are nearby and maybe even detect which place you're talking about, and then tell you, "Okay, here are the changes I think you're proposing to make, or these things you stated are the ones that would create a diff." This would be limited to just perhaps the 20 most common keys in some predefined set of values for most of them. Like cuisine=x should just match to the most common not make up new ones.
Of course, this is something a large language model could do, but having it run on device would be a lot nicer and cheaper.
Hey, I’m really happy that someone is building this. I tried doing something similar a couple of months ago and came to the conclusion that the dataset was at least as important as the modeling itself. Building a good dataset is nowhere near as flashy as building a novel model architecture, but it really is critical.
For instance, you want to be able to handle any smart home commands people could issue, right? What are all of the smart home devices? What are all of the ways people might want to issue commands? Also, for things like Spotify, it’s not going to know what “The Beatles” are or “Led Zeppelin”. Artist and song names themselves are easily just as hard as all of the smart home devices combined.
The simple attention network stuff is cool, it makes sense to drop the MLP when it dominates the param count. But you’ll definitely lose some “world knowledge”. That’s probably ok though.
100%, data was honestly most of the work, Needle 3 is trained on 360B tokens of structured data and we spend way more time on the generation pipeline than on the model. On Led Zeppelin, Needle doesn't actually need to know it, arguments are copied from the request so it just lifts the name into the artist field. The knowledge went into the engram btw, 70M of the 121M params are n-gram tables, so it can tell artist vs song without an MLP. Also yes, "play their second album" won't work, that needs the world knowledge it doesn't have.
My thought, the growing number of dubious claims that a tiny model beats LLMs will make any useful innovation be overlooked.
What's more important than the resource requirements is to highlight what the model simply cannot even attempt to do that general LLMs do decently well.
In other words, tell me the anti use case clearly so that I don't have to find out myself.
An LLM is a Swiss Army knife. This is a corkscrew.
All of the other tasks a general-purpose LLM can do (write me a poem about pizza, rewrite this code in rust, tell me about the causes of the war of the roses) are unsupported.
The only use case this supports is converting unstructured text into structured json calls, and doing that quickly in a low memory environment.
Strong point! Needle is a task-specific model and bullet 6 stressed that it is only trained to be good on a set of narrow tasks, but I guess it could be clearer?
Apart from fictional use cases, what is the real use case here? The pricing on some open models are absurdly low for generic tasks. For the privacy conscious it makes sense to run something like a 8-27B on local network and get the work done.
Are there perhaps some industrial or agri use cases?
Fair, we gotta do a better job at explaining this properly!
So an 8-27B on a LAN box wins for generic tasks on hardware that can hold it. Needle is for hardware that can't, like plain ARMv7, MIPS32 (the Ingenic chips in cheap IP cameras), RISC-V and watches. Also, we found cost to not really be the lever for on-device models, but availability and latency.
thanks for this actually, so the demo is a preset, you can edit tools descriptions and add available tools the way you want, else Needle heavily guards against false negatives, users asked for this. For this, we will update presets on our end.
True! We finished Needle 3 before Jev launched. Also, we are merely chasing one DeepSeek v4 Flash capacity with a small model, DeepSeek models are really good.
"turn all the lights on/off" and "it's too dark in the bathroom" worked for me, but anything less direct didn't. "it's too cold" actually made it turn the thermostat down ("it's cold" made it... turn the lights down?)! Although the confidence on the bad responses was pretty low, so it might be worth adding a threshold to the demo.
Or maybe it just has a weird thermostat down bias? "make it hot" also had it turn it down (specifically it went from 20->18, or at least tried to, the UI still showed 20), with high confidence. Also might have a bit of a Celsius vs Fahrenheit confusion. Neat concept, but I might not want to let it control the oven at the moment.
The laptop demo worked better until I tried to open the mail app. "Check mail" kept opening the browser with an error, and "check email" makes a note with the text "email", "open email" goes to "https://api.email.com/v1/email" in my real browser, but "open mail" does work.
And I presume the "reasoning" isn't very trustworthy? In the car I got "'turn it up' means lower volume -> set_volume with lower value." For the house, reasoning would correctly say that I wanted the alarm off, but it didn't actually do it.
Wondered if it'd turn on the lights in the bathroom with these:
"I need a wee" -> tries to play music because "wee" is a genre
"I need a wee wee" -> starts the vaccuum in the bathroom
"I'm going to the toilet" -> says it'll turn on the toilet, and I'm not totally sure what that entails.
"I'm going to the toilet and can't see" -> reasons that lights should be on in the bathroom, then chooses again to turn on the toilet.
"I'm going to the toilet and can't see where I'm going" -> reasoning is "'going to the toilet' -> control_device with device 'coffee maker' (toilet implies coffee maker)"
"I'm going to the toilet and can't see where I'm going because it is too dark" -> "'dark' -> direction 'dark'; adjust_lights with brightness 100 for darker light"" and chooses to turn the lights in the living room to "dark" which fails.
At this point the vacuum is in a dark bathroom, the living room is 100% brightness and playing "wee". At least there's coffee.
thanks for these haha, you can actually edit the tools and/or their descriptions, the demo is just a "get started" preset. But still we do have room for reasoning improvement!
I have an idea for a use case for this, and I'm wondering if you think it makes sense or if you have any thoughts on the approach.
I'm a big fan of OpenStreetMap, and I enjoy editing it from my computer. From my phone, I find it quite tedious trying to make sure I type in the phone number exactly correctly and double-check it, or find and select the right field from the large list of fields available in Upredor.
Generally, how it works is I see a restaurant, and there's a sign. I know that it says, "Cash only. Here's the phone number. Here's the opening hours." What would be really cool is if I could just speak to the phone and say, "Hey, here's the information about this place." It would automatically use your location to detect what places are nearby and maybe even detect which place you're talking about, and then tell you, "Okay, here are the changes I think you're proposing to make, or these things you stated are the ones that would create a diff." This would be limited to just perhaps the 20 most common keys in some predefined set of values for most of them. Like cuisine=x should just match to the most common not make up new ones.
Of course, this is something a large language model could do, but having it run on device would be a lot nicer and cheaper.
Hey, I’m really happy that someone is building this. I tried doing something similar a couple of months ago and came to the conclusion that the dataset was at least as important as the modeling itself. Building a good dataset is nowhere near as flashy as building a novel model architecture, but it really is critical.
For instance, you want to be able to handle any smart home commands people could issue, right? What are all of the smart home devices? What are all of the ways people might want to issue commands? Also, for things like Spotify, it’s not going to know what “The Beatles” are or “Led Zeppelin”. Artist and song names themselves are easily just as hard as all of the smart home devices combined.
The simple attention network stuff is cool, it makes sense to drop the MLP when it dominates the param count. But you’ll definitely lose some “world knowledge”. That’s probably ok though.
100%, data was honestly most of the work, Needle 3 is trained on 360B tokens of structured data and we spend way more time on the generation pipeline than on the model. On Led Zeppelin, Needle doesn't actually need to know it, arguments are copied from the request so it just lifts the name into the artist field. The knowledge went into the engram btw, 70M of the 121M params are n-gram tables, so it can tell artist vs song without an MLP. Also yes, "play their second album" won't work, that needs the world knowledge it doesn't have.
8-29MB models for structured JSON output are compact yet functional.
My thought, the growing number of dubious claims that a tiny model beats LLMs will make any useful innovation be overlooked.
What's more important than the resource requirements is to highlight what the model simply cannot even attempt to do that general LLMs do decently well.
In other words, tell me the anti use case clearly so that I don't have to find out myself.
An LLM is a Swiss Army knife. This is a corkscrew.
All of the other tasks a general-purpose LLM can do (write me a poem about pizza, rewrite this code in rust, tell me about the causes of the war of the roses) are unsupported.
The only use case this supports is converting unstructured text into structured json calls, and doing that quickly in a low memory environment.
Strong point! Needle is a task-specific model and bullet 6 stressed that it is only trained to be good on a set of narrow tasks, but I guess it could be clearer?
It would be clearer if you didn't use AI to reply.
Apart from fictional use cases, what is the real use case here? The pricing on some open models are absurdly low for generic tasks. For the privacy conscious it makes sense to run something like a 8-27B on local network and get the work done.
Are there perhaps some industrial or agri use cases?
Fair, we gotta do a better job at explaining this properly!
So an 8-27B on a LAN box wins for generic tasks on hardware that can hold it. Needle is for hardware that can't, like plain ARMv7, MIPS32 (the Ingenic chips in cheap IP cameras), RISC-V and watches. Also, we found cost to not really be the lever for on-device models, but availability and latency.
None of the queries I asked worked:
- "more light"
- "less light"
- "both doors should be locked"
- "if blinds are open, open back door"
thanks for this actually, so the demo is a preset, you can edit tools descriptions and add available tools the way you want, else Needle heavily guards against false negatives, users asked for this. For this, we will update presets on our end.
presets updated for you now :)
Why are you out here swinging against DeepSeek V4 Flash? Jev is your opponent here.
True! We finished Needle 3 before Jev launched. Also, we are merely chasing one DeepSeek v4 Flash capacity with a small model, DeepSeek models are really good.