Going directly for the logprobs is always icky when you use a chat model as base, because they are trained to write prose as output. So your "choice" tokens and thus their probabilities might get diluted in whatever else it wanted to say. If you have to do it in the same way as this post, at least add clear system instructions and a carefully worded beginning to the assistant output section of the prompt to lower the chances of it wandering off immediately.
I've found that using structured outputs solves this problem much better. Instead of letting a model generate only "A", "B" or "C" and looking at the probs, have it directly generate "Legitimate", "Spam" or "Phishing" or any other pre-defined option from a set of multi-token sequences. Behind the scenes it boils down to something quite similar, but you're not running into the risk that the model actually wanted to say "A phishing attempt seems likely, so answer (C) is correct.", which would lead "A" to have the highest probability in the first token. You can even use a reasoning budget this way either via inherent reasoning or a free-form part preceding the remaining output structure. You can also have it assign probabilities (either in words or numbers) using more complex output structures, but I would not rely on them much more than the token logprobs (they can still be quite good though).
+1, llama.cpp has a --grammar parameter which you can pass a BNF style grammar file to constrain generation. It can be used in Python llama.cpp wrapper
Agreed, it's a real issue, but it can probably be vastly reduced by having the schema in the system prompt and by giving the model an expectation of a fixed value: no decent modern would pick a prose ligament over a provided value.
To completely squash the issue, a few cheap LoRa iterations will do the trick just fine.
Sure, you can fix that in a couple lines. Then a couple more lines for evaluating multiple questions on the same answer in parallel. Then a couple more lines for the confidence score (which is trivial to compute from all we have, but missing regardless). Then a harness to fine-tune an existing model to perform better on this specific task, and a collection of training data to use for that
I think we can all agree that Jev is not rocket science. It's a good idea executed well, with marketing that might have been a tad too bold
The confidence score is not trivial to compute. That is the whole point of the model. Even if you are using a proper scoring function such as NLL, it is not enough to ensure calibration in deep nets. So you have to do good post training to ensure it. These are all known techniques, but they are far from trivial, especially on large scale datasets.
In my experience as well using logprobs to try to quantify uncertainty, LLMs are a poor fit. Neural nets in general struggle with 'calibration' --- ie. if a prediction is truly 50/50, neural nets are often prone to predicting overconfidently [0].
I ran some tests using GPT-4 to do some basic classification a couple years ago. On ambiguous options which had to be escalated to a human, the LLM would regularly output something like a 99.8% probability, compared to 99.99% for a correct answer.
Best option would be reasoning + clear system instructions + constrained output. That is, if you have to use a chat model. Which works well enough to be sure, but hey I haven't tried raising millions of dollars when I did that 3 years ago. But perhaps I was the stupid one.
Because of masked attention in LLMs, if you put the options before the body (the email to analyze), the transformer already knows what it needs to look for, and can use more tokens to create state to address that specific task (BERT has no mask in the attention, so tokens attend also to next tokens). You could also do a few examples in the system prompt to improve calibration.
Another trick that works is to repeat the question two times: "I'm repeating the task and labels for clarity: ..."
Beyond the missing latency and compute comparisons that Heaney commenter mentioned, also nothing about its error rate compared to Jev (nor if it even always outputs in a format the app can parse, not sure how solved that is).
But then at the end it says it’s parody. Maybe HN title should say it’s a joke.
Which massively slows down the output. Doing this with Qwen 9B already takes you into seconds per answer territory, and Jev is supposedly frontier level intelligence.
Whilst I do like reading these things for technical know how, I can sympathise with the creator of jev who now presumably has to apply an order of magnitude effort to explain why the 100 smaller things done better than this add up to a much better product.
Replace 'explain' with 'sell'. Don't forget that it's a gold rush. There's no reason to sympathize with corporations in their rush for the slice of the pie.
If you're comparing with something, you need to state 'fast' in relative terms. Jev is definitely fast, and if this Python takes the same time to get a decision then it's also fast. If it's 100* slower than Jev though, you shouldn't be calling it 'fast', because relatively speaking it's really, really slow.
By design it can't be significantly slower than Jev: the prompt processing (AKA PP) is exactly the same on both and will take most of the time. Then you can process every single "question" in parallel, just predicting one or two tokens (if an answer is ambiguous with a single token) per each question, again in a single batch.
So, fast in the LLM space and comparable with Jev.
strong "You can build dropbox quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem" vibes
You have built something like jev but not jev (for starters, the output of what you've built will be absolutely worthless, the whole reason Jev is getting so much hype is because the output is good enough)
I wonder if this could be a good stepping stone to write a local prompt router to optimise what model get what prompt. I.e. if the prompt is just a lookup, send it to haiku, if it's reasoning, send it to opus and if it's implementation send it to sonnet.
not sure if true, but if you look at laya they use BERT type models. If jev is also using a BERT-type model it is autoregressive and therefore can't reason in the way that GPT-type models can. However, you get the advantage of being able to attend in both directions.
Should they be writing quicksort in assembly as a first step? I think its legitimate in this case given that Jev is likely using the same tools as the example. Showing how easily the core is created using those tools helps to dispel some of the mystery and hype.
Example why its legit:
I just invented a new "Regression Estimate Validator" aka Rev. It takes hundreds of input dimensions, then outputs an interpretable score. Its very fast and statistically robust. Response: Ok but you could just use `pytorch.nn.Linear(d_in, 1)`? True, it is equivalent, but that's concealing millions of lines of hand-tuned math libs, CUDA, python, and other stuff.
The fact that there are many lines of code underpinning the target functionality doesn't make it any harder to use, and doesn't increase the value of the sales pitch for the "new shiny thing" using those few lines of code.
However, I do sympathize with your frustration that people can just say "its 1 line of code" when that line is "invoke API" which is really millions of lines / databases, etc. as a way to dismiss legitimate work without understanding its implications.
Going directly for the logprobs is always icky when you use a chat model as base, because they are trained to write prose as output. So your "choice" tokens and thus their probabilities might get diluted in whatever else it wanted to say. If you have to do it in the same way as this post, at least add clear system instructions and a carefully worded beginning to the assistant output section of the prompt to lower the chances of it wandering off immediately.
I've found that using structured outputs solves this problem much better. Instead of letting a model generate only "A", "B" or "C" and looking at the probs, have it directly generate "Legitimate", "Spam" or "Phishing" or any other pre-defined option from a set of multi-token sequences. Behind the scenes it boils down to something quite similar, but you're not running into the risk that the model actually wanted to say "A phishing attempt seems likely, so answer (C) is correct.", which would lead "A" to have the highest probability in the first token. You can even use a reasoning budget this way either via inherent reasoning or a free-form part preceding the remaining output structure. You can also have it assign probabilities (either in words or numbers) using more complex output structures, but I would not rely on them much more than the token logprobs (they can still be quite good though).
+1, llama.cpp has a --grammar parameter which you can pass a BNF style grammar file to constrain generation. It can be used in Python llama.cpp wrapper
https://til.simonwillison.net/llms/llama-cpp-python-grammars
Yes. But even then, the probabilities are not calibrated. In jev/laya, they are (well, relatively anyways).
Agreed, it's a real issue, but it can probably be vastly reduced by having the schema in the system prompt and by giving the model an expectation of a fixed value: no decent modern would pick a prose ligament over a provided value.
To completely squash the issue, a few cheap LoRa iterations will do the trick just fine.
Sure, you can fix that in a couple lines. Then a couple more lines for evaluating multiple questions on the same answer in parallel. Then a couple more lines for the confidence score (which is trivial to compute from all we have, but missing regardless). Then a harness to fine-tune an existing model to perform better on this specific task, and a collection of training data to use for that
I think we can all agree that Jev is not rocket science. It's a good idea executed well, with marketing that might have been a tad too bold
The confidence score is not trivial to compute. That is the whole point of the model. Even if you are using a proper scoring function such as NLL, it is not enough to ensure calibration in deep nets. So you have to do good post training to ensure it. These are all known techniques, but they are far from trivial, especially on large scale datasets.
In my experience as well using logprobs to try to quantify uncertainty, LLMs are a poor fit. Neural nets in general struggle with 'calibration' --- ie. if a prediction is truly 50/50, neural nets are often prone to predicting overconfidently [0].
I ran some tests using GPT-4 to do some basic classification a couple years ago. On ambiguous options which had to be escalated to a human, the LLM would regularly output something like a 99.8% probability, compared to 99.99% for a correct answer.
0: https://arxiv.org/pdf/1706.04599
Seems like all normal english words could risk the same, so would using short but random strings be even better?
Actually to me it sounds it could be benchmarked if this kind of effect exists in the first place.
Best option would be reasoning + clear system instructions + constrained output. That is, if you have to use a chat model. Which works well enough to be sure, but hey I haven't tried raising millions of dollars when I did that 3 years ago. But perhaps I was the stupid one.
That's the approach that daseinlabs/open-jev takes, in contrast to the above, which is what TheoLeeCJ/openjev and ekzhang/openjev-sglang do
https://sgnt.ai/p/jev/
Because of masked attention in LLMs, if you put the options before the body (the email to analyze), the transformer already knows what it needs to look for, and can use more tokens to create state to address that specific task (BERT has no mask in the attention, so tokens attend also to next tokens). You could also do a few examples in the system prompt to improve calibration.
Another trick that works is to repeat the question two times: "I'm repeating the task and labels for clarity: ..."
Beyond the missing latency and compute comparisons that Heaney commenter mentioned, also nothing about its error rate compared to Jev (nor if it even always outputs in a format the app can parse, not sure how solved that is).
But then at the end it says it’s parody. Maybe HN title should say it’s a joke.
latency and compute comparisons highly depends on your local setup.
you can swith to a better model for lower error rate.
Which massively slows down the output. Doing this with Qwen 9B already takes you into seconds per answer territory, and Jev is supposedly frontier level intelligence.
Whilst I do like reading these things for technical know how, I can sympathise with the creator of jev who now presumably has to apply an order of magnitude effort to explain why the 100 smaller things done better than this add up to a much better product.
Replace 'explain' with 'sell'. Don't forget that it's a gold rush. There's no reason to sympathize with corporations in their rush for the slice of the pie.
IT's the infamous "OneDrive in 10 lines of code (SFTP)"
While technically correct, it's not the same thing
It's fast.
If you're comparing with something, you need to state 'fast' in relative terms. Jev is definitely fast, and if this Python takes the same time to get a decision then it's also fast. If it's 100* slower than Jev though, you shouldn't be calling it 'fast', because relatively speaking it's really, really slow.
By design it can't be significantly slower than Jev: the prompt processing (AKA PP) is exactly the same on both and will take most of the time. Then you can process every single "question" in parallel, just predicting one or two tokens (if an answer is ambiguous with a single token) per each question, again in a single batch.
So, fast in the LLM space and comparable with Jev.
strong "You can build dropbox quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem" vibes
You have built something like jev but not jev (for starters, the output of what you've built will be absolutely worthless, the whole reason Jev is getting so much hype is because the output is good enough)
I wonder if this could be a good stepping stone to write a local prompt router to optimise what model get what prompt. I.e. if the prompt is just a lookup, send it to haiku, if it's reasoning, send it to opus and if it's implementation send it to sonnet.
I was thinking the same. Haven't tried it out.
What I don’t understand is, why would you not want “reasoning” in a classifier?
Speed and cost are obvious reasons, but isn’t this a tradeoff?
not sure if true, but if you look at laya they use BERT type models. If jev is also using a BERT-type model it is autoregressive and therefore can't reason in the way that GPT-type models can. However, you get the advantage of being able to attend in both directions.
Latency and compute comparison needed.
Is benchmarking Jev still a ToS violation?
Was it? That would make it unusable in any corporate setting.
Now, can you do it in <200ms for 45 questions at once, have 0% malformed output, and any kind of meaningful benchmark? We’ll wait!
> <200ms for 45 questions at once
Considering your own question length: ~120 characters x 45 divided by 4.1 ~= 1317 tokens.
So question processing at 5.5k PP(around the actual PP speed of GPT5.6 Sol) it would take around ~0.24 seconds + the context processing.
Computing the output should be around ~20ms (at 50 tok/s), computing 45 tokens in parallel.
> have 0% malformed output
Pretty trivial; only the allowed output is selectable :)
So, I keep repeating myself: Jev was a low-hanging fruit all along; no one cared, and probably no one will in a few weeks?
The principle is this.
Nothing I hate more than bullshit articles claiming X in Y lines of code, only to use libraries abstracting hundreds of thousands of lines of code.
Should they be writing quicksort in assembly as a first step? I think its legitimate in this case given that Jev is likely using the same tools as the example. Showing how easily the core is created using those tools helps to dispel some of the mystery and hype.
Example why its legit:
I just invented a new "Regression Estimate Validator" aka Rev. It takes hundreds of input dimensions, then outputs an interpretable score. Its very fast and statistically robust. Response: Ok but you could just use `pytorch.nn.Linear(d_in, 1)`? True, it is equivalent, but that's concealing millions of lines of hand-tuned math libs, CUDA, python, and other stuff.
The fact that there are many lines of code underpinning the target functionality doesn't make it any harder to use, and doesn't increase the value of the sales pitch for the "new shiny thing" using those few lines of code.
However, I do sympathize with your frustration that people can just say "its 1 line of code" when that line is "invoke API" which is really millions of lines / databases, etc. as a way to dismiss legitimate work without understanding its implications.