Do Chatbot LLMs Talk Too Much?

5 comments

• dgfl a day ago ago

  They will keep a live leaderboard at https://huggingface.co/spaces/tabularisai/YapBench

  Here is the top 10 right now:

    | Rank | Model                                    |   YapIndex | YapTax$ |
    | ---: | ---------------------------------------- | ---------: | ------: |
    |    1 | openai/gpt-5.6-sol (reasoning)           |  18.5 ±4.8 |    0.51 |
    |    2 | openai/gpt-5.6-sol                       |  19.2 ±4.4 |    0.51 |
    |    3 | openai/gpt-3.5-turbo                     |  22.7 ±4.8 |    0.02 |
    |    4 | openai/gpt-5.6-luna                      |  27.8 ±9.8 |    0.15 |
    |    5 | openai/gpt-5.4 (reasoning)               | 40.7 ±10.7 |       — |
    |    6 | openai/gpt-5.4                           |  40.7 ±9.0 |       — |
    |    7 | moonshotai/kimi-k2-0905                  |  44.7 ±4.8 |    0.05 |
    |    8 | mistralai/mistral-small-2603 (reasoning) | 46.2 ±31.5 |    0.03 |
    |    9 | openai/gpt-4                             | 51.2 ±20.6 |    1.39 |
    |   10 | openai/gpt-5.3-codex                     |  64.8 ±9.9 |       — |
  

  I also checked how the Claude models did specifically:

    | Rank | Model                                   |    YapIndex | YapTax$ |
    | ---: | --------------------------------------- | ----------: | ------: |
    |   23 | anthropic/claude-opus-4.5               |  97.0 ±28.9 |    1.52 |
    |   25 | anthropic/claude-opus-4.5 (reasoning)   |  99.2 ±29.3 |    1.44 |
    |   42 | anthropic/claude-3.5-sonnet             | 199.7 ±24.5 |    2.53 |
    |   57 | anthropic/claude-sonnet-4.5 (reasoning) | 278.7 ±41.7 |    1.65 |
    |   61 | anthropic/claude-sonnet-4.5             | 285.0 ±39.4 |    1.63 |
    |   71 | anthropic/claude-opus-4.6               | 330.3 ±59.1 |       — |
    |   72 | anthropic/claude-haiku-4.5              | 333.2 ±26.7 |    0.64 |
    |   73 | anthropic/claude-haiku-4.5 (reasoning)  | 335.2 ±26.8 |    0.65 |
    |   76 | anthropic/claude-opus-4.6 (reasoning)   | 342.3 ±30.5 |       — |
    |   87 | anthropic/claude-3.5-haiku              | 401.2 ±25.4 |    0.59 |
    |   88 | anthropic/claude-sonnet-4.6 (reasoning) | 422.8 ±23.3 |       — |
    |   89 | anthropic/claude-sonnet-4.6             | 425.0 ±26.3 |       — |
  

  No other Anthropic model seems to be there for now. I would have been very curious to see the more recent ones.
  [-]
  • shostack a day ago ago

    I needed this. I had no idea it existed.

    I have bent over backwards trying to enforce brevity with deepseek v4 flash to the point where I think I broke some things trying to do prompt injection in my Hermes setup and was still unsuccessful.

    Meanwhile Sol blows me away and I want that to be my default for everything now.

    In general though I seem to have the most success with a "<=10w" requirement in my prompts.

    What I don't see listed and would be a good comparison is the STS models. OpenAI's live model is an absolute joy to talk with.

• josefritzishere a day ago ago

  This defies Betteridge's law of headlines.

• rektomatic a day ago ago

  Yes, yes they do

• alikhater30000 a day ago ago

  [flagged]