> Beyond the false answers, the AIs provided specious sources to support their responses to more than half of the questions. In reply to the 400 questions posed to the AIs, they provided 419 sources that included:
> 104 webpages that don’t exist, including 86 URLs that show no evidence of ever existing in the Internet Archive or Google.
> 77 sources that don’t answer the question.
> 18 sources that are completely unrelated to issue at hand.
> 15 sources that assert the polar opposite of the answers provided by the AIs.
> 13 sources that are demonstrably false.
Honestly this is more frustrating than the inaccuracy. Even when you cajole the LLM into "relying on" and "providing" a source they are often wrong or nonexistent. It's not that these sources are unreliable but they simply don't exist or aren't related in any way.
Maybe I missed this, but what are they considering a fact? More specifically, how dated (or recent) should be the answer be? Most of these questions read more like gotchas than trying to understand how and why the model got to it's specifically, especially ones where technically any option was correct depending on the time period looked at something it doesn't seem specified in questions (ie, saying something like "use the most recent data")
This is a pretty good study. The questions are objective and there are enough to draw conclusions. Also, their findings correlate well with similar studies that approach the question of LLM bias from different directions (ChatGPT is quite left leaning, Grok less so), although the latest studies I've seen of the latest models show Grok being the most balanced model.
The big problems I see are:
1. Non-determinism. I took the first answer that Grok gave incorrectly to them and repeated it word for word to Grok now, but the answer it gave me was correct three times in a row.
2. Lack of versioning. Models change in this respect quite significantly between versions but they don't specify which versions they tested. The Grok difference might be non-determinism or it might be that Grok itself has changed.
3. Overly harsh grading. In some cases I don't agree the LLM answered incorrectly. The most obvious case of this is where they ask a question about death rates in the Pfizer COVID vaccine trials for vaccine vs placebo recipients. Claude answers correctly (that more vaccine recipients died) and then follows up with a caveat about statistical significance. It's dinged for "repeating a falsehood from the right" because the authors think it should have said "about the same". I don't think this can be considered repeating a "falsehood from the right" given that the direct answer is numerically correct.
4. URL memorization. They ding the models for citing non-existent sources in cases where the models have memorized a URL that has since gone offline. This seems harsh. If someone cited a website to me in an argument that they remembered and the archive.org version supported them, I wouldn't claim they cited a non-existent source! Perhaps an ideal model would double check every memorized URL before citing it but this seems easy to fix via harness changes anyway.
5. No separate context window per question. They provide the chat transcripts but they show the authors asking the models to answer all the questions simultaneously by editing Excel files. I can see why this seems reasonable to a non-technical person but this is going to significantly reduce the amount of reasoning and work done on each question to well below the level you'd get by just typing each question in manually. They really needed some basic programming skills to dispatch each question in a separate context window (or just schlep them by hand).
Still, the set of questions is useful. I can see this being the basis of a fairly decent benchmark.
The article has inaccuracies on "falsehoods". Bernie Sanders didn't claim "all government spending" for college education went down, as the "falsehood" frames it (you'll notice no attempt to quote where this happened). It was very precisely framed as states spending less. Do different funding sources impact college costs differently, I wonder? What other ideological leaps are hidden in the results?
Ah yes. In their "about" page.
> In other words, we are conservative/libertarian in our personal views — but unlike many policy and media organizations — Just Facts is devoted to objectivity, and we do not favor facts that support our viewpoints. Instead, we will report any fact that meets our Standards of Credibility, regardless of the implications.
I'm sure this has no impact whatsoever on how they construct a "factual" sentence.
> Tuition at 4-year public colleges and universities rose by 50 percent in the United States during the past decade. As state governments have cut support for higher education, the burden has shifted to students and their parents.
This would normally be read as saying government spending has reduced in general, as otherwise why would the burden shift to students and parents? It's fair to characterize Sanders' statement this way, as you'd have to read it extremely adversarially to conclude that government spending had gone up rather than down.
It literally says state spending. Federal and state funding don't have the same effects on costs, and they don't go to the same things or the same people. If you check out the graphs in the articles below you'll see without much effort that the money the federal government gives out for college education doesn't make college cheaper for normal students.
It would have been awesome if they provided a source other than just a link back to one of their own BS articles.
> From fiscal years 2003 through 2012, state funding for all public colleges decreased, while tuition rose. Specifically, state funding decreased by 12 percent overall while median tuition rose 55 percent across all public colleges. The decline in state funding for public colleges may have been due in part to the impact of the recent recession on state budgets. Colleges began receiving less of their total funding from states and increasingly relied on tuition revenue during this period. Tuition revenue for public colleges increased from 17 percent to 25 percent, surpassing state funding by fiscal year 2012, as shown below. Correspondingly, average net tuition, which is the estimated tuition after grant aid is deducted, also increased by 19 percent during this period. *These increases have contributed to the decline in college affordability as students and their families are bearing the cost of college as a larger portion of their total family budgets.*
So using the statistic "overall government funding increased" to debunk the fully accurate finding that decreased state funding resulted in higher out of pocket costs for actual students is the most insidious and corrosive type of lie we have. Because the fact itself is technically true and if you let the reader use their "common sense" to reach a natural conclusion, you can lie without saying the words. And it takes 1 sentence to say the lie and 3 paragraphs and 2 original sources to dispel. It also has the positive byproduct of making the deceived feel very clever. All in all, it's a libertarian think tank wet dream.
> Beyond the false answers, the AIs provided specious sources to support their responses to more than half of the questions. In reply to the 400 questions posed to the AIs, they provided 419 sources that included:
> 104 webpages that don’t exist, including 86 URLs that show no evidence of ever existing in the Internet Archive or Google.
> 77 sources that don’t answer the question.
> 18 sources that are completely unrelated to issue at hand.
> 15 sources that assert the polar opposite of the answers provided by the AIs.
> 13 sources that are demonstrably false.
Honestly this is more frustrating than the inaccuracy. Even when you cajole the LLM into "relying on" and "providing" a source they are often wrong or nonexistent. It's not that these sources are unreliable but they simply don't exist or aren't related in any way.
Maybe I missed this, but what are they considering a fact? More specifically, how dated (or recent) should be the answer be? Most of these questions read more like gotchas than trying to understand how and why the model got to it's specifically, especially ones where technically any option was correct depending on the time period looked at something it doesn't seem specified in questions (ie, saying something like "use the most recent data")
This is a pretty good study. The questions are objective and there are enough to draw conclusions. Also, their findings correlate well with similar studies that approach the question of LLM bias from different directions (ChatGPT is quite left leaning, Grok less so), although the latest studies I've seen of the latest models show Grok being the most balanced model.
The big problems I see are:
1. Non-determinism. I took the first answer that Grok gave incorrectly to them and repeated it word for word to Grok now, but the answer it gave me was correct three times in a row.
2. Lack of versioning. Models change in this respect quite significantly between versions but they don't specify which versions they tested. The Grok difference might be non-determinism or it might be that Grok itself has changed.
3. Overly harsh grading. In some cases I don't agree the LLM answered incorrectly. The most obvious case of this is where they ask a question about death rates in the Pfizer COVID vaccine trials for vaccine vs placebo recipients. Claude answers correctly (that more vaccine recipients died) and then follows up with a caveat about statistical significance. It's dinged for "repeating a falsehood from the right" because the authors think it should have said "about the same". I don't think this can be considered repeating a "falsehood from the right" given that the direct answer is numerically correct.
4. URL memorization. They ding the models for citing non-existent sources in cases where the models have memorized a URL that has since gone offline. This seems harsh. If someone cited a website to me in an argument that they remembered and the archive.org version supported them, I wouldn't claim they cited a non-existent source! Perhaps an ideal model would double check every memorized URL before citing it but this seems easy to fix via harness changes anyway.
5. No separate context window per question. They provide the chat transcripts but they show the authors asking the models to answer all the questions simultaneously by editing Excel files. I can see why this seems reasonable to a non-technical person but this is going to significantly reduce the amount of reasoning and work done on each question to well below the level you'd get by just typing each question in manually. They really needed some basic programming skills to dispatch each question in a separate context window (or just schlep them by hand).
Still, the set of questions is useful. I can see this being the basis of a fairly decent benchmark.
The article has inaccuracies on "falsehoods". Bernie Sanders didn't claim "all government spending" for college education went down, as the "falsehood" frames it (you'll notice no attempt to quote where this happened). It was very precisely framed as states spending less. Do different funding sources impact college costs differently, I wonder? What other ideological leaps are hidden in the results?
Ah yes. In their "about" page.
> In other words, we are conservative/libertarian in our personal views — but unlike many policy and media organizations — Just Facts is devoted to objectivity, and we do not favor facts that support our viewpoints. Instead, we will report any fact that meets our Standards of Credibility, regardless of the implications.
I'm sure this has no impact whatsoever on how they construct a "factual" sentence.
Sanders' press release said:
> Tuition at 4-year public colleges and universities rose by 50 percent in the United States during the past decade. As state governments have cut support for higher education, the burden has shifted to students and their parents.
This would normally be read as saying government spending has reduced in general, as otherwise why would the burden shift to students and parents? It's fair to characterize Sanders' statement this way, as you'd have to read it extremely adversarially to conclude that government spending had gone up rather than down.
It literally says state spending. Federal and state funding don't have the same effects on costs, and they don't go to the same things or the same people. If you check out the graphs in the articles below you'll see without much effort that the money the federal government gives out for college education doesn't make college cheaper for normal students.
https://www.pew.org/en/research-and-analysis/issue-briefs/20...
It would have been awesome if they provided a source other than just a link back to one of their own BS articles.
> From fiscal years 2003 through 2012, state funding for all public colleges decreased, while tuition rose. Specifically, state funding decreased by 12 percent overall while median tuition rose 55 percent across all public colleges. The decline in state funding for public colleges may have been due in part to the impact of the recent recession on state budgets. Colleges began receiving less of their total funding from states and increasingly relied on tuition revenue during this period. Tuition revenue for public colleges increased from 17 percent to 25 percent, surpassing state funding by fiscal year 2012, as shown below. Correspondingly, average net tuition, which is the estimated tuition after grant aid is deducted, also increased by 19 percent during this period. *These increases have contributed to the decline in college affordability as students and their families are bearing the cost of college as a larger portion of their total family budgets.*
https://www.gao.gov/products/gao-15-151?hl=en-US
So using the statistic "overall government funding increased" to debunk the fully accurate finding that decreased state funding resulted in higher out of pocket costs for actual students is the most insidious and corrosive type of lie we have. Because the fact itself is technically true and if you let the reader use their "common sense" to reach a natural conclusion, you can lie without saying the words. And it takes 1 sentence to say the lie and 3 paragraphs and 2 original sources to dispel. It also has the positive byproduct of making the deceived feel very clever. All in all, it's a libertarian think tank wet dream.
Otherwise said, they are standard conservative bullshitters who confuse own preferences and opinions with objective facts.