I'm one of the authors of this blog post. Figuring out how to integrate the LLM, the inference and serving engine, and the constrained decoding implementation such that you get the intended output and overall quality definitely requires careful engineering and testing. Please feel free to AMA. :)
To the LLM, the <|...|> tags are single, special tokens. They are rendered as <|...|> tags for readability, but to the model the tags are single tokens.
I do not know why these tokens are visualized as <|...|>, but I speculate that this way they are easily recognizable as special-purpose tokens, and difficult to confuse with other types of existing markup.
I'm one of the authors of this blog post. Figuring out how to integrate the LLM, the inference and serving engine, and the constrained decoding implementation such that you get the intended output and overall quality definitely requires careful engineering and testing. Please feel free to AMA. :)
Do you know where <||> syntax is coming from? I find it very odd that existing markup wasn't used and there is very little info online about it.
To the LLM, the <|...|> tags are single, special tokens. They are rendered as <|...|> tags for readability, but to the model the tags are single tokens.
I do not know why these tokens are visualized as <|...|>, but I speculate that this way they are easily recognizable as special-purpose tokens, and difficult to confuse with other types of existing markup.
Correct me if I wrong, I saw this when llama.cpp or ollama initializes, it is used as token separator of some sort.
GPT-2 used <|endoftext|>
Constrained decoding came to JS land transformers last month for in browser structured output as well, also used for tool calling.
https://github.com/huggingface/transformers.js/blob/main/pac...