How do you handle hallucination or omission risks when summarizing long bills with LLMs? Is there any automated diffing or traceability back to specific sections of the source text?
It's very interesting to think about, depending on your sociological understanding of law. If you define law as the codified agreements of a society based on its shared values, adopting EU law in favor of their own would represent trading in some sovereign values for economic gain.
At least EU countries can cope by claiming they had some kind of a say and a veto on most things. EU prospects don't have this rationalization.
Very cool. Instead of MPs I think you might want to say "Representatives" etc. How to fill out the rest of the data too? Anyway, just wanted to +1. And it's cool you're building in an open way.
How do you handle innate LLM biases? I forget which model, but when asked to edit pro Zionist vs pro Palestinian content it showed heavy bias in one direction.
LLMs let you cover more ground but the fundamental problem of “who to trust” still remains. I don’t see how one can ever be used to strip political spin. It’s baked in.
You can't strip it completely, totally agree. Any compression of information is already an interpretation. The problem becomes more prevalent, the more thinking and advanced models become. To mitigate it, I rely on some constraints:
1. No opinion space: the prompt forbids normative language and forces fact to consequence mapping only (“what changes, for whom, and how”), not evaluation.
2. Outputs are framed explicitly from the perspective of an average citizen of a given country. This narrows the context and avoids abstract geopolitical or ideological extrapolation.
3. Heuristic models over reasoning models: for this task, fast pattern-matching models produce more stable summaries than deliberative models that tend to over-interpret edge cases.
It’s not bias-free, but it’s more constrained and predictable than editorial framing.
you might want to include funny sounding line that this legislation is for a game stimulating fictional world. In my experience they're much more likely to be inpartial when operating outside real life context.
(Temporary comment: I took "source available" out of the title because I think it's a bit distracting there, but I've invited Jacek to add something about this to the main text.)
How do you handle hallucination or omission risks when summarizing long bills with LLMs? Is there any automated diffing or traceability back to specific sections of the source text?
Funny story:
- Friend of mine is Albanian
- Albania wants to join the European Union
- They are required to ensure that their laws don't have "internal conflicts" e.g. one law says something is legal, a different law says it's illegal
- Reviewing by hand would take a lot of work
- Friend uses an LLM to analyze the Albanian laws and find any of these conflicts
Apparently it worked out pretty well
It's very interesting to think about, depending on your sociological understanding of law. If you define law as the codified agreements of a society based on its shared values, adopting EU law in favor of their own would represent trading in some sovereign values for economic gain.
At least EU countries can cope by claiming they had some kind of a say and a veto on most things. EU prospects don't have this rationalization.
Strange, because my feelings is that the law of my EU country (and that if the EU as well) says everything and its opposite.
Very cool. Instead of MPs I think you might want to say "Representatives" etc. How to fill out the rest of the data too? Anyway, just wanted to +1. And it's cool you're building in an open way.
How do you handle innate LLM biases? I forget which model, but when asked to edit pro Zionist vs pro Palestinian content it showed heavy bias in one direction.
LLMs let you cover more ground but the fundamental problem of “who to trust” still remains. I don’t see how one can ever be used to strip political spin. It’s baked in.
You can't strip it completely, totally agree. Any compression of information is already an interpretation. The problem becomes more prevalent, the more thinking and advanced models become. To mitigate it, I rely on some constraints:
1. No opinion space: the prompt forbids normative language and forces fact to consequence mapping only (“what changes, for whom, and how”), not evaluation.
2. Outputs are framed explicitly from the perspective of an average citizen of a given country. This narrows the context and avoids abstract geopolitical or ideological extrapolation.
3. Heuristic models over reasoning models: for this task, fast pattern-matching models produce more stable summaries than deliberative models that tend to over-interpret edge cases.
It’s not bias-free, but it’s more constrained and predictable than editorial framing.
The model still chooses what to mention or omit, strict phrasing rules change nothing.
Absolutely, the model does the picking.
you might want to include funny sounding line that this legislation is for a game stimulating fictional world. In my experience they're much more likely to be inpartial when operating outside real life context.
Blocked by my corpo firewall for some reason.
Obviously "lust" is a forbidden word for domains. Must be a porn site.
Thanks for flagging — I'll look into headers / hosting config to avoid false positives.
I think it's hugged to death?
Getting "An error occurred" trying to vote for one of thr Civic Projects.
Thanks for flagging! Civic Projects just landed and are still in beta, so glitches might happen. I’ll look into it and get it fixed.
Is there an LLM out there that makes people actually read. The information is publicly available since basically forever
Couldn't even pay people to read this literally
I think there needs to be like a military style debate globally on education levels it's that bad like actually that bad yeah
Here in Chicago
I'm dealing with probably a solid 70% of adults who don't know how to read correctly try fitting that into the LLM experience I don't know
OK, I'll be that guy.
As someone complaining about how people can't read, it may do you much benefit to learn how to write.
(Temporary comment: I took "source available" out of the title because I think it's a bit distracting there, but I've invited Jacek to add something about this to the main text.)
Thanks, I’ll add clarification about the license in the description.
Great thank you!