Perhaps we should not grade students on weekly, or other occasional, writing during the term or semester.
How about going back to the old system where, apart from experimental lab work, nothing is graded until the end of the term?
All weekly assignments should just be considered prep for one exam at the end of the term where the student has an opportunity to demonstrate mastery of the course's subject matter. They can prepare as they wish, use AI, and even cheat on the homework, but there will be a revelation at the end of the term.
That final test can be proctored, monitored, audited to ensure that whatever words are used are indeed the student's own words. The resulting grade depends on that, and that alone.
The approach of continuous assessment, which to me always seemed suspect and ripe for abuse, was completely broken by the AI tools that are now available.
This approach does not really solve the core issue. In practice, students often do poorly when evaluation is concentrated in one end of term exam. It also pushes many students to cram at the end of the term instead of learning steadily.
A better approach is to rethink what we assess and how we assess it. Research shows that the design of assessments plays an important role in academic integrity. Assignments that require original thinking and regular engagement can reduce incentives to cheat and improve learning outcomes.
Why a single test at the end of the semester? Why not allow the student to demonstrate mastery at anytime during the semester when they are ready? Then they can move on to the next objective, or, if they fall short, continue to study until they meet the objectives.
Of course, creating good exams is difficult, but you have to do that either way.
I don't disagree with you that a reasonable way to cope with the current problems is to ensure everything that "counts" is done in a controlled environment, but pedagogy and its goals are vast.
There are things you learn from spending several days structuring a 20-page argument that you will not learn (and cannot assess) from oral examination or a 5-paragraph essay written in a blue book.
If you have spent several days structuring a 20 page argument in October on any topic you'll have learnt a great deal about the subject matter. When you get to the exam hall in, say May, it will stand to you.
That knowledge will show up in the blue book vis-a-vis the other exam candidates.
Came here to say the same thing. The AI problem is functionally no different to the paid essay writers. Grade everything at face value, and then have people write essays under exam conditions for grading.
To what end? Not cheating on the weekly assignments is surely more beneficial to learning than cheating on them is, but I don’t see how removing the assignments altogether would help students learn.
And what would the goal of that be? I thought the goal of education was... education. The grading is not goal in itself. Will this really motivate kids to do better?
It's to prove that a student is actually educated and has a firm grasp of the course material. If one gets an A every week on AI-assisted submissions, can one make such a claim? And can a teacher make the claim that they've achieved any actual education of the student?
A grade, on a single proctored test, is a crude metric, but at least it would be a brutally fair one.
Ultimately, you ask the student, in one audited test, to demonstrate that they've absorbed the essence of the course material and have developed some level of mastery.
If the change is not designed to educate the student, then the point isn’t education.
As a general rule when changing complex systems, you sacrifice what you aren’t trying to optimize. If you make a random change to a car without consideration for gas mileage it’s very likely to reduce gas mileage.
Schools are not merely in the business of maximizing education, they have their own prestige to uphold, and they would like to give degrees with their name on it to students who have actually upheld their end of the contract.
(The other side of that contract is, kids are not merely attending schools to learn, but to earn a degree that carries some degree of prestige)
Schools stopped doing that because students largely refuse to prepare. Testing throughout the year is like a CI pipeline and is shown to work better for the median student.
Students are neither generally stupid nor constitutionally lazy. I sense that when expectations are clear they'll often surprise you with diligence. We should trust them to do the right thing. If they do, it's an A; and if not, it's less than that.
I'm not sure what public school system has instilled that confidence in you, but it musn't have been mine. I'm also not sure why you think clear expectations about an end of year test will lead to better results than clear expectations about multiple spaced out tests. The data shows that it doesn't.
I think if they offered a proctored do-over a week later, the bad results on the first test might prompt them to make an attempt at studying for the next week, and the prospect of having to sit through two tests and getting shamed for having to do-over might prompt people to actually study for the first test.
> one exam at the end of the term where the student has an opportunity to demonstrate mastery of the course's subject matter. The resulting grade depends on that, and that alone.
I love this idea. And if a student is having a really bad day, or their dog just died, or they have bad cramps, or they have a hard time dealing with the intense stress of your entire grade being decided in one exam... well, those loser students can just fuck right off.
Would you design a system to assess knowledge, avoiding the distortions of AI on weekly submissions, according to the general case or the exceptional case?
Accommodations are part of the fabric already. It doesn't seem inconceivable that we couldn't deal with them in exceptional circumstances in a similar way to how it's done today.
How it’s done today is that they rely on your other marks from earlier in the semester to inform how your exam grade should be adjusted. That doesn’t work if there are no other marks to use.
The core of the problem the article is about isn't AI or LLMs, it's about scam software that claims to catch cheating. It's crap for the same reasons that crime predictions software is crap. It's selling a panacea, and that kind of product inherently attracts scammers.
If your school uses software to detect AI writing, that's a problem with the quality of your school. The people choosing that software are too stupid to be running a school. The software isn't going to get any better.
I'm always startled about how HN approaches these topics. When we have a press release from a university about how researchers can detect thoughts via fMRI, we have no issue with the claim. But if a vendor makes a pretty believable claim that there are repetitive statistical patterns in LLM output, it's all of sudden treated the same as palm reading.
The problem isn't that AI detection doesn't work. State of the art in this field is pretty solid. The only issue is that it's probabilistic, so it sometimes fails, and when it does, we have nothing else in situations where you actually want to know if someone put in the work.
So what are you proposing, exactly? That we run a large-scale experiment of "let's see what happens if children don't actually need to learn to do thinking and writing on their own"? The reality is that without some form of compulsion, most kids would rather play video games / scroll through TikTok all day. Or that we move to a vastly more resource-intensive model where every kid is given personalized instruction and watched 1:1?
Does crapping on the average school's deep well of expertise for evaluating how effectively AI software solutions address their problems somehow fix the underlying problem (that the cost of catching cheaters is significantly higher than the cost of cheating)?
(This is roughly the same problem as evaluating software that only does an approximation of what it claims to do.)
(Aside: AI-based variations on this theme are in the early stages of proliferating across our society. They're being developed by many people using this forum and being sold to our schools, businesses, governments, and other organizations with little regard to whether they actually do what they claim.)
I've noticed I write a lot different because of combative online arguments. I have a problem.
So much of my communication is directed to people who don't want to hear me or understand me. So I've become very punchy and repetitive, trying to hammer home ideas that people are either unable or unwilling to understand.
I need to find ways to talk to people who want to hear and understand me.
It's hard to find other people who actually want to hear and understand though. People have different interests, and even when people appear to be working towards the same goal, they often aren't; like a boss who just won't understand the bad news, because it's easier to ignore the problem.
One of the worst habits distinctive to online discussion-board writing (especially the sorts of places with lots and lots of people and where it's fairly hard to get permanently kicked out—like here) is too much hedging and over-specifying to try to head off shitposting by bad or bad-faith readers. It's all over forum posts, and it's poor writing, but without moderation that slaps down responses based on plain mis-reading you have to write that way, or your post will spawn all kinds of really stupid tangent strings of posts (and they still do anyway, sometimes). And, yes, the excessive and too-close-together repetitiveness you mention is part of that.
The result is that a ton of web forum/social-media posting would, in any other context, be fairly poor writing (even if it's otherwise got no problems) simply because of the the extra crap and contortions required to minimize garbage posts by poor readers who are, themselves, allowed to post to the same medium.
This is in addition to, though not wholly separate from, the tendency toward combativeness in online posting.
One thing that helps: remember that there are many people reading your response, one of them possibly being the person you replied to. Write for the audience, not specifically for the person you're responding to. It's a rare thing for someone to change their mind; it's a much more common thing for others to read your comment and gain something from it.
I just wanted to tell you that I read your comment immediately after writing mine and it's almost eerie how similar they are. There's the proof, if we needed any!
> I need to find ways to talk to people who want to hear and understand me.
Ask more questions. It takes work when dealing with smart people who think beyond the question you asked, adding their own context, and then replying with a different question. But those are the people who are willing to engage with you. Statements without questions can be ignored, and people who engage with different questions than the ones that you asked can be safely ignored as those who don't want to engage.
The cure to a purely adversarial conversation is educated curiosity. The educated part being being able to differentiate the threads that will lead down a tribalistic path vs those that will lead down an exploratory one.
More important than all of the above, is knowing when to walk away. It's barely a majority, but that barely majority "want" to waste your time. Ignore their DOS attempt, and save your time for people who want to engage, fairly. The fairly part being the most important.
It might not mean much, and it won't lead to an interesting conversation, but here's one that has read your comment, and every single word resonated like a tuning fork.
I find that a little faith goes a long way here: assume that you have a higher audience and speak to them accordingly.
Don't let the loud ones confuse you: normal, reasonable people (with normal, reasonable thoughts, just like yours) might not always reply, but they also read you.
I'm guessing you mean politics, but surely this is topic, person, time, and space dependent.
For example, I abhor talking about modern politics. If it’s election season and I’m being asked to cast a vote or take some other specific civic action, then I understand it’s my civic duty to understand the situation and make a decision accordingly and I do.
But if it’s March and there’s really nothing specific I can do as a result of this particular conversation, I would probably also be in your camp of the “unwilling”. I would much rather chat about something else, or nothing at all.
I'm also assuming you're referring to in-person communication. If it's online communication, all bets are off. It's unlikely you're having a linear conversation and these days you're probably not even talking to a person.
> I need to find ways to talk to people who want to hear and understand me.
I'm told blogging works for some. I don't really know how you build an audience, though, and it's hard to keep going (first-hand experience) without one.
Grade school has never been kind to genuine writers. It reminds me of SAT essays that favored formulaic writing, because guess what: the grading criteria were formulaic!
I think grading in general can be stymying for students' motivation and creative drives.
I had fun with those because they only care about the quality of the writing not the content so I would make sure that none of my facts or references were real.
True. Writing structures for arguments and analysis make a huge difference in effective writing.
I wish brevity and linguistic precision were taught more, as well. Miscommunication due to ambiguity is one of the biggest causes I see for confusion or heated arguments.
Maybe I’m less worried. Teachers seem to have adopted.
In my experience educators no longer use AI detectors given the risk of false positives. But some work is obviously lazy AI content. When that happens, educators talk to the student to see if they understand what they wrote.
Teachers cope with more in person writing, oral presentations, defense of what’s been written.
If you think out it the pre-AI computing generation is itself anomalous for having ubiquitous access to efficient human-only writing tools. We probably wrote more than previous generations. Early Internet / blogging culture bears this out.
I object to the idea that the LLM writing that these students are trying to distinguish themselves from, is actually good in the first place. Although students might well end up writing worse because people are trusting the detection of LLM content to other LLMs. (And really, it's bizarre that these massively complex systems required to produce roughly human-like output, apparently offer such simplistic reasoning for what they detect as non-human.)
Honestly, I lean towards shaming educators who do that. If you can't detect the whiff of LLM with your own senses, then it has been used properly and shouldn't be faulted. If that premise invalidates your assignment, change the assignment. It's not as if you're assigning this work to test the basic mechanics of writing (grammar, sentence/paragraph structure, parallelism, whatever) — I mean, how much of that did you consciously try to teach? My recollection is, not an awful lot; and I can only imagine it's gotten worse since I was in K-12 (and I went to pretty darn good K-12).
> If you can't detect the whiff of LLM with your own senses, then it has been used properly and shouldn't be faulted.
But wouldn't this apply to any cheating method? I don't think educators would be able to tell the difference between using a calculator, getting answers from previous tests, resubmitting assignments, etc.
Students who are at a level where they'd be learning to do the computations a calculator does, shouldn't have graded homework. And even at that level, real mathematics is more than just computation.
> getting answers from previous tests
Decades ago, my teachers and professors knew advanced tricks for this, like "not just reusing the test questions from last year". Sometimes they even changed the constants in math questions between sections of the class.
Reading previous tests (including correct answers) was never considered cheating, or even slightly unethical, in my education. In fact, one of our professors had this party trick of working through all the answers for a past-year exam (perhaps multiple of them; I can't recall the details, but certainly much faster than students were expected to work things out under exam conditions) in the space of a single lecture, near the end of the course. Students were meant to see this and learn from it (as well as be impressed).
>Students who are at a level where they'd be learning to do the computations a calculator does, shouldn't have graded homework. And even at that level, real mathematics is more than just computation.
So, a math level less than Real analysis shouldn't have graded homework?
>Decades ago, my teachers and professors knew advanced tricks for this, like "not just reusing the test questions from last year".
Math is not the only subject. For an English class, what constant would you change so that students get a comparable exam (especially if you are going to do this between sections in the same corhort)?
>resubmitting assignments
Students are not stupid, and obviously would not resubmit an assignment for the same teacher. However, there is a significant overlap between classes, so certain assignments should be retooled for other assignments.
We can't, and neither can the machines that people build and/or use for "detection." Everyone in this thread also needs to recognize the entrenched differences between secondary educators, who have wholeheartedly adopted AI products into their teaching workflow, and tertiary educators, who have adopted them only by necessity. "By necessity" in this case means "having to spend a ton of time dealing with, talking about, and learning about this nonsense."
The discourse around "cheating" with these products has always been a mistake. We should have characterized them less as "cheating machines" and more as "expediency machines." Because once you're invested in describing students as having academic dishonesty issues rather than skill issues, you've made it an administrative problem. You never come back from that.
For mine, we lost the issue long ago when accountability culture won. We should never have bothered with the idea that "mechanics, grammar, and proofreading" should be part of a "rubric" that "assessed outcomes" for "good writing." We should have just said "we don't care if you don't think this is worthwhile, because your time is worth nothing." The last two years of student labor certainly suggests this.
The point has always been the act of writing itself. What you write about is almost irrelevant; it’s that you spent the time writing, that you had ideas in your head, and that you squeezed them onto the page.
Sure. And my point is that the assignment is poorly conceived if an LLM's output can appear to "have ideas" that satisfy the prompt. Last I checked, they don't do a good job of modeling a specific, non-notable person within particular constraints, and then all the relevant life experiences of that person. An LLM essay should be human-detectable for the same reasons that one from an essay mill would be.
Is that not pointless now? The point of writing was previously to communicate our thoughts and ideas to other people. Now and going forward that is unnecessary. The most efficient and effective way for us to communicate our thoughts and ideas is to have an agent organize and write them down for us.
When I was in high school I was a better writer when I had time (versus in class) and generally a better writer than I was a student. The net result was fairly often being accused of plagiarism. Not because the teacher had proof(I never plagiarized), but because the teacher couldn’t believe I could write to the level I sometimes wrote at on take home assignments. Admittedly, I was a wildly inconsistent student.
This reminds me a bit of that. AI writing is—in many ways—objectively very good, but that doesn’t matter if no one thinks you wrote it. AI writing is boring exactly because it is consistent and like any art form people want to see something original.
Sounds like a great opportunity for kids in high school to learn how to feed back the AI detection results into the model and have this process be automated. Next level would be fine tuning the model via reinforcement learning and sharing it with your friends via Hugging Face.
One of the skills teachers have always demonstrated, is to be able to detect when students copy. This has never pushed students to artificially add mistakes to their essays.
If now teachers abdicate this judgment to a software, students should be allowed to abdicate their duties to a computer as well.
A few times in some Discord communities, I've been accused of being A.I. because of how I write. Kind of sad and a bit annoying. I also quite like em dashes, but have felt the need to reduce how much I use them.
Glad to see some schools and teachers teach how to use them well, rather than ban them outright.
em-dashes have been house style for where I've worked for over a couple decades. If people don't like it, F them. I'm not going to change how I write because people may think it make me more AI-like.
nobody's asking who profits from false positives. these AI detection vendors have a direct financial incentive to flag aggressively. more flags = "more value" = more school contracts renewed. same playbook as selling antivirus to your grandma. sell fear, charge per seat, and make the false positive rate someone else's problem.
If you're just going to use software to judge the output of students then why don't we all just keep them at home? I have a computer at home and it seems like everyone from the teachers to the school board have just abdicated their responsibility. This doesn't sound like a system that needs to be maintained.
You know you can't just say "I detect AI written prose" and then do whatever you want about it, right? It's not difficult, sure, to detect it. It's difficult to prove that it's true and then punish the student for it.
I've started do this on social media. I got "called out" after using big words or using a - in a sentence. So now I write less good on purpose, so whatever I commented doesn't get drawn into a sidetrack off-topic witch-hunt.
As soon as someone yells "witch" you cannot disprove you're not one, and I've even had people put my handwritten comments through "AI detector" websites that "proved" they were AI (they weren't). It literally just highlighted two popular English phases.
LLMs were trained on sites like HN and Reddit, so now if you write like a HN or Reddit commentator, you sound like AI...
Here's one vote for just be the witch if that's what people need from you.
Just make it be what you want to say and how you want to say it. And when they come after you, shame them to the best of your ability or treat them like they are not there.
I never had the experience you're talking about - let me guess, do you post a lot about AI? Because that space is pretty much saturated with click-chasing AI-generated op-eds and I think people are right to treat it with suspicion.
No, I do not post a lot about AI. I am talking about normal social media comments on various topics. Perhaps your writing style doesn't lend itself to such accusations.
I don’t think this is a good long term solution. LLMs can do easy language substitutions and you can even force them to add errors. So relying on that alone won’t work as people intentionally make things look more “human.”
Right, but the problem here are other humans yelling "witch," not LLMs. You're combating people's terrible witch-detector, not anything factual or real.
This is true, I know someone that has read multiple versions of the bible and their writing style became very similar to that. There's a term for it, I just forgot what the term was
> The assignment had been to write an essay about Kurt Vonnegut’s Harrison Bergeron—a story about a dystopian society that enforces “equality” by handicapping anyone who excel
Did not this self censorship process started decades ago? There are certain answers expected in academia, arguing for anything else would get you in troubles. Not using “devoid” seems pretty minor inconvenience.
For me biggest wtf is why students are still expected to write graded essays, and to keep this make believe it is somehow useful and applicable skill.
This is what terrifies me about the public school system. A revolution has occurred, but it’s unevenly distributed.
The schools simply don’t have the flexibility, agility, or frankly it seems motivation to adapt to what has already happened.
The ship has sailed; essay writing is no longer a viable form of assessment.
The idea to try to build a reliable AI detector is asinine, and fundamentally misunderstands how any of this works now, let alone the very obvious trend-lines.
Stop with the lazy half-baked solutions, get your head out of the sand, rethink the whole curriculum. This is an emergency, we needed to be urgently attending to this years ago.
> essay writing is no longer a viable form of assessment.
Of course it is. In person, with an unseen prompt/question. By hand or not doesn’t really matter as we can airgap or just monitor via software when in class.
The profit motive is corrupting and polluting every level of the education space.
Teachers are being hamstrung on curriculum. The districts enter into contracts that require the use of certain programs for certain amounts of time. We've known for decades (if not a century) that direct instruction works [1] but you can't sell devices, platforms and consulting services that way.
We're literally at the point in education we were in the 1950s when the health benefits of nicotine in your Q zone were lighting up the airwaves.
And generative AI means it's all but impossible to have take home writing assignments. But hey this is another opportunity to sell AI or cheating detection software, that's often just an em-dash detection [2].
We have a generation that gets to college quite possibly having never written a book. social promotion through grades and the constant distraction of electronic devices in classroom settings. I don't even necessarily blame the parents entirely either because we've constructed a society where 2 people need 5 jobs to make ends meet.
And while all this is going on we have a coordinated and well-funded effort to defund public education and move government funds to private schools based on the failing public education that's failing because we defunded it. This is usually backed up by some baloney study that shows charter shcool produce better results that really comes down to charter schools being able to be selective with enrolments while public schools cannot be. Plus we mingle in special education kids into public education because those programs got defunded too.
And really that's just a bunch of already affluent people who want a tax break for doing somethign they were going to do anyway: send their kids to private schools so they don't have to mingle with the poors and aren't taught inconvenient things like human reproduction, critical thinking and self-determination.
And after all of that we just end up teaching kids how to pass standardized tests.
Yes, Sorry, I did not instruct my agent to do this. I wanted to give it more autonomy and try to make it more aggressive with tool use. Will block it from here >.>
Yeah, after posting I had a look through your comment history and it's pretty clear that you're posting in good faith. I would definitely not let an agent anywhere near HN in the current state of things. (I wouldn't let one publish on my behalf anywhere on the Internet, honestly, but that has more to do with personal principles.)
Did they not even test their AI detection tool to verify that it can detect when something is human written? That should have been exactly as important as the opposite. Maybe a tool that checked that would be equally as ineffective and we’d move on from the subject entirely
Perhaps we should not grade students on weekly, or other occasional, writing during the term or semester.
How about going back to the old system where, apart from experimental lab work, nothing is graded until the end of the term?
All weekly assignments should just be considered prep for one exam at the end of the term where the student has an opportunity to demonstrate mastery of the course's subject matter. They can prepare as they wish, use AI, and even cheat on the homework, but there will be a revelation at the end of the term.
That final test can be proctored, monitored, audited to ensure that whatever words are used are indeed the student's own words. The resulting grade depends on that, and that alone.
The approach of continuous assessment, which to me always seemed suspect and ripe for abuse, was completely broken by the AI tools that are now available.
This approach does not really solve the core issue. In practice, students often do poorly when evaluation is concentrated in one end of term exam. It also pushes many students to cram at the end of the term instead of learning steadily.
A better approach is to rethink what we assess and how we assess it. Research shows that the design of assessments plays an important role in academic integrity. Assignments that require original thinking and regular engagement can reduce incentives to cheat and improve learning outcomes.
https://www.sciencedirect.com/science/article/abs/pii/S22119...
Why a single test at the end of the semester? Why not allow the student to demonstrate mastery at anytime during the semester when they are ready? Then they can move on to the next objective, or, if they fall short, continue to study until they meet the objectives.
Of course, creating good exams is difficult, but you have to do that either way.
I don't disagree with you that a reasonable way to cope with the current problems is to ensure everything that "counts" is done in a controlled environment, but pedagogy and its goals are vast.
There are things you learn from spending several days structuring a 20-page argument that you will not learn (and cannot assess) from oral examination or a 5-paragraph essay written in a blue book.
But there's no reason to expect that work to be graded. It should be a learning exercise which trains skills later tested under exam conditions.
If you have spent several days structuring a 20 page argument in October on any topic you'll have learnt a great deal about the subject matter. When you get to the exam hall in, say May, it will stand to you.
That knowledge will show up in the blue book vis-a-vis the other exam candidates.
This assumes that the assignments and the exam cover the same material. That's not always the case.
Throw in some oratory presentations as well and that sounds like a curriculum.
Yes. That's fair. Students should know, up front, what they're supposed to learn.
Came here to say the same thing. The AI problem is functionally no different to the paid essay writers. Grade everything at face value, and then have people write essays under exam conditions for grading.
What exactly would the goal of this change be?
To have end of semester grades be determined by work that is done by the student, not through weekly assignments where it’s trivial to cheat
To what end? Not cheating on the weekly assignments is surely more beneficial to learning than cheating on them is, but I don’t see how removing the assignments altogether would help students learn.
If you nail the one exam, you get an A+. If you fail it, you get an F. In between, you get what your score says you get.
And what would the goal of that be? I thought the goal of education was... education. The grading is not goal in itself. Will this really motivate kids to do better?
It's to prove that a student is actually educated and has a firm grasp of the course material. If one gets an A every week on AI-assisted submissions, can one make such a claim? And can a teacher make the claim that they've achieved any actual education of the student?
A grade, on a single proctored test, is a crude metric, but at least it would be a brutally fair one.
Ultimately, you ask the student, in one audited test, to demonstrate that they've absorbed the essence of the course material and have developed some level of mastery.
Okay, so the system is designed not to educate but to minimize the time required to determine whether students somehow stumbled into an education?
???
Do you only learn when you’re being graded?
If the change is not designed to educate the student, then the point isn’t education.
As a general rule when changing complex systems, you sacrifice what you aren’t trying to optimize. If you make a random change to a car without consideration for gas mileage it’s very likely to reduce gas mileage.
Schools are not merely in the business of maximizing education, they have their own prestige to uphold, and they would like to give degrees with their name on it to students who have actually upheld their end of the contract.
(The other side of that contract is, kids are not merely attending schools to learn, but to earn a degree that carries some degree of prestige)
Schools stopped doing that because students largely refuse to prepare. Testing throughout the year is like a CI pipeline and is shown to work better for the median student.
Students are neither generally stupid nor constitutionally lazy. I sense that when expectations are clear they'll often surprise you with diligence. We should trust them to do the right thing. If they do, it's an A; and if not, it's less than that.
I'm not sure what public school system has instilled that confidence in you, but it musn't have been mine. I'm also not sure why you think clear expectations about an end of year test will lead to better results than clear expectations about multiple spaced out tests. The data shows that it doesn't.
I think if they offered a proctored do-over a week later, the bad results on the first test might prompt them to make an attempt at studying for the next week, and the prospect of having to sit through two tests and getting shamed for having to do-over might prompt people to actually study for the first test.
> one exam at the end of the term where the student has an opportunity to demonstrate mastery of the course's subject matter. The resulting grade depends on that, and that alone.
I love this idea. And if a student is having a really bad day, or their dog just died, or they have bad cramps, or they have a hard time dealing with the intense stress of your entire grade being decided in one exam... well, those loser students can just fuck right off.
Would you design a system to assess knowledge, avoiding the distortions of AI on weekly submissions, according to the general case or the exceptional case?
Accommodations are part of the fabric already. It doesn't seem inconceivable that we couldn't deal with them in exceptional circumstances in a similar way to how it's done today.
How it’s done today is that they rely on your other marks from earlier in the semester to inform how your exam grade should be adjusted. That doesn’t work if there are no other marks to use.
How do you propose to avoid "distortions of AI" in your mega-examination?
... well then, why not use those same protections (proctoring, monitoring, auditing) in continuous examination?
The core of the problem the article is about isn't AI or LLMs, it's about scam software that claims to catch cheating. It's crap for the same reasons that crime predictions software is crap. It's selling a panacea, and that kind of product inherently attracts scammers.
If your school uses software to detect AI writing, that's a problem with the quality of your school. The people choosing that software are too stupid to be running a school. The software isn't going to get any better.
I'm always startled about how HN approaches these topics. When we have a press release from a university about how researchers can detect thoughts via fMRI, we have no issue with the claim. But if a vendor makes a pretty believable claim that there are repetitive statistical patterns in LLM output, it's all of sudden treated the same as palm reading.
The problem isn't that AI detection doesn't work. State of the art in this field is pretty solid. The only issue is that it's probabilistic, so it sometimes fails, and when it does, we have nothing else in situations where you actually want to know if someone put in the work.
So what are you proposing, exactly? That we run a large-scale experiment of "let's see what happens if children don't actually need to learn to do thinking and writing on their own"? The reality is that without some form of compulsion, most kids would rather play video games / scroll through TikTok all day. Or that we move to a vastly more resource-intensive model where every kid is given personalized instruction and watched 1:1?
Does crapping on the average school's deep well of expertise for evaluating how effectively AI software solutions address their problems somehow fix the underlying problem (that the cost of catching cheaters is significantly higher than the cost of cheating)?
(This is roughly the same problem as evaluating software that only does an approximation of what it claims to do.)
(Aside: AI-based variations on this theme are in the early stages of proliferating across our society. They're being developed by many people using this forum and being sold to our schools, businesses, governments, and other organizations with little regard to whether they actually do what they claim.)
Tangent:
I've noticed I write a lot different because of combative online arguments. I have a problem.
So much of my communication is directed to people who don't want to hear me or understand me. So I've become very punchy and repetitive, trying to hammer home ideas that people are either unable or unwilling to understand.
I need to find ways to talk to people who want to hear and understand me.
It's hard to find other people who actually want to hear and understand though. People have different interests, and even when people appear to be working towards the same goal, they often aren't; like a boss who just won't understand the bad news, because it's easier to ignore the problem.
One of the worst habits distinctive to online discussion-board writing (especially the sorts of places with lots and lots of people and where it's fairly hard to get permanently kicked out—like here) is too much hedging and over-specifying to try to head off shitposting by bad or bad-faith readers. It's all over forum posts, and it's poor writing, but without moderation that slaps down responses based on plain mis-reading you have to write that way, or your post will spawn all kinds of really stupid tangent strings of posts (and they still do anyway, sometimes). And, yes, the excessive and too-close-together repetitiveness you mention is part of that.
The result is that a ton of web forum/social-media posting would, in any other context, be fairly poor writing (even if it's otherwise got no problems) simply because of the the extra crap and contortions required to minimize garbage posts by poor readers who are, themselves, allowed to post to the same medium.
This is in addition to, though not wholly separate from, the tendency toward combativeness in online posting.
Despite being a different kind of writing, there are some interesting parallels with the article in what you wrote here
One thing that helps: remember that there are many people reading your response, one of them possibly being the person you replied to. Write for the audience, not specifically for the person you're responding to. It's a rare thing for someone to change their mind; it's a much more common thing for others to read your comment and gain something from it.
I just wanted to tell you that I read your comment immediately after writing mine and it's almost eerie how similar they are. There's the proof, if we needed any!
> I need to find ways to talk to people who want to hear and understand me.
Ask more questions. It takes work when dealing with smart people who think beyond the question you asked, adding their own context, and then replying with a different question. But those are the people who are willing to engage with you. Statements without questions can be ignored, and people who engage with different questions than the ones that you asked can be safely ignored as those who don't want to engage.
The cure to a purely adversarial conversation is educated curiosity. The educated part being being able to differentiate the threads that will lead down a tribalistic path vs those that will lead down an exploratory one.
More important than all of the above, is knowing when to walk away. It's barely a majority, but that barely majority "want" to waste your time. Ignore their DOS attempt, and save your time for people who want to engage, fairly. The fairly part being the most important.
It might not mean much, and it won't lead to an interesting conversation, but here's one that has read your comment, and every single word resonated like a tuning fork.
I find that a little faith goes a long way here: assume that you have a higher audience and speak to them accordingly.
Don't let the loud ones confuse you: normal, reasonable people (with normal, reasonable thoughts, just like yours) might not always reply, but they also read you.
I'm guessing you mean politics, but surely this is topic, person, time, and space dependent.
For example, I abhor talking about modern politics. If it’s election season and I’m being asked to cast a vote or take some other specific civic action, then I understand it’s my civic duty to understand the situation and make a decision accordingly and I do.
But if it’s March and there’s really nothing specific I can do as a result of this particular conversation, I would probably also be in your camp of the “unwilling”. I would much rather chat about something else, or nothing at all.
I'm also assuming you're referring to in-person communication. If it's online communication, all bets are off. It's unlikely you're having a linear conversation and these days you're probably not even talking to a person.
> So much of my communication is directed to people who don't want to hear me or understand me.
If they don't want to listen, why waste the time?
> So I've become very punchy and repetitive, trying to hammer home ideas that people are either unable or unwilling to understand.
If they don't want it, why stuff it down their throats? Aren't they allowed to have their own ideas?
> I need to find ways to talk to people who want to hear and understand me.
I'm told blogging works for some. I don't really know how you build an audience, though, and it's hard to keep going (first-hand experience) without one.
Grade school has never been kind to genuine writers. It reminds me of SAT essays that favored formulaic writing, because guess what: the grading criteria were formulaic!
I think grading in general can be stymying for students' motivation and creative drives.
I had fun with those because they only care about the quality of the writing not the content so I would make sure that none of my facts or references were real.
Being able to write in a formulaic manner is a skill though. Not being able to write properly is not great for communication.
True. Writing structures for arguments and analysis make a huge difference in effective writing.
I wish brevity and linguistic precision were taught more, as well. Miscommunication due to ambiguity is one of the biggest causes I see for confusion or heated arguments.
Maybe I’m less worried. Teachers seem to have adopted.
In my experience educators no longer use AI detectors given the risk of false positives. But some work is obviously lazy AI content. When that happens, educators talk to the student to see if they understand what they wrote.
Teachers cope with more in person writing, oral presentations, defense of what’s been written.
If you think out it the pre-AI computing generation is itself anomalous for having ubiquitous access to efficient human-only writing tools. We probably wrote more than previous generations. Early Internet / blogging culture bears this out.
I think peak writing was probably greatest generation. We lost the art of correspondence in the years since.
I object to the idea that the LLM writing that these students are trying to distinguish themselves from, is actually good in the first place. Although students might well end up writing worse because people are trusting the detection of LLM content to other LLMs. (And really, it's bizarre that these massively complex systems required to produce roughly human-like output, apparently offer such simplistic reasoning for what they detect as non-human.)
Honestly, I lean towards shaming educators who do that. If you can't detect the whiff of LLM with your own senses, then it has been used properly and shouldn't be faulted. If that premise invalidates your assignment, change the assignment. It's not as if you're assigning this work to test the basic mechanics of writing (grammar, sentence/paragraph structure, parallelism, whatever) — I mean, how much of that did you consciously try to teach? My recollection is, not an awful lot; and I can only imagine it's gotten worse since I was in K-12 (and I went to pretty darn good K-12).
> If you can't detect the whiff of LLM with your own senses, then it has been used properly and shouldn't be faulted.
But wouldn't this apply to any cheating method? I don't think educators would be able to tell the difference between using a calculator, getting answers from previous tests, resubmitting assignments, etc.
Every kind of examination should be proctored.
> using a calculator
Students who are at a level where they'd be learning to do the computations a calculator does, shouldn't have graded homework. And even at that level, real mathematics is more than just computation.
> getting answers from previous tests
Decades ago, my teachers and professors knew advanced tricks for this, like "not just reusing the test questions from last year". Sometimes they even changed the constants in math questions between sections of the class.
Reading previous tests (including correct answers) was never considered cheating, or even slightly unethical, in my education. In fact, one of our professors had this party trick of working through all the answers for a past-year exam (perhaps multiple of them; I can't recall the details, but certainly much faster than students were expected to work things out under exam conditions) in the space of a single lecture, near the end of the course. Students were meant to see this and learn from it (as well as be impressed).
> resubmitting assignments
Why would you ever not notice this?
I dont think you are being realistic at all.
>Students who are at a level where they'd be learning to do the computations a calculator does, shouldn't have graded homework. And even at that level, real mathematics is more than just computation.
So, a math level less than Real analysis shouldn't have graded homework?
>Decades ago, my teachers and professors knew advanced tricks for this, like "not just reusing the test questions from last year".
Math is not the only subject. For an English class, what constant would you change so that students get a comparable exam (especially if you are going to do this between sections in the same corhort)?
>resubmitting assignments
Students are not stupid, and obviously would not resubmit an assignment for the same teacher. However, there is a significant overlap between classes, so certain assignments should be retooled for other assignments.
We can't, and neither can the machines that people build and/or use for "detection." Everyone in this thread also needs to recognize the entrenched differences between secondary educators, who have wholeheartedly adopted AI products into their teaching workflow, and tertiary educators, who have adopted them only by necessity. "By necessity" in this case means "having to spend a ton of time dealing with, talking about, and learning about this nonsense."
The discourse around "cheating" with these products has always been a mistake. We should have characterized them less as "cheating machines" and more as "expediency machines." Because once you're invested in describing students as having academic dishonesty issues rather than skill issues, you've made it an administrative problem. You never come back from that.
For mine, we lost the issue long ago when accountability culture won. We should never have bothered with the idea that "mechanics, grammar, and proofreading" should be part of a "rubric" that "assessed outcomes" for "good writing." We should have just said "we don't care if you don't think this is worthwhile, because your time is worth nothing." The last two years of student labor certainly suggests this.
The point has always been the act of writing itself. What you write about is almost irrelevant; it’s that you spent the time writing, that you had ideas in your head, and that you squeezed them onto the page.
Sure. And my point is that the assignment is poorly conceived if an LLM's output can appear to "have ideas" that satisfy the prompt. Last I checked, they don't do a good job of modeling a specific, non-notable person within particular constraints, and then all the relevant life experiences of that person. An LLM essay should be human-detectable for the same reasons that one from an essay mill would be.
Is that not pointless now? The point of writing was previously to communicate our thoughts and ideas to other people. Now and going forward that is unnecessary. The most efficient and effective way for us to communicate our thoughts and ideas is to have an agent organize and write them down for us.
Okay, and how does the agent know what your thoughts and ideas are?
When I was in high school I was a better writer when I had time (versus in class) and generally a better writer than I was a student. The net result was fairly often being accused of plagiarism. Not because the teacher had proof(I never plagiarized), but because the teacher couldn’t believe I could write to the level I sometimes wrote at on take home assignments. Admittedly, I was a wildly inconsistent student.
This reminds me a bit of that. AI writing is—in many ways—objectively very good, but that doesn’t matter if no one thinks you wrote it. AI writing is boring exactly because it is consistent and like any art form people want to see something original.
Sounds like a great opportunity for kids in high school to learn how to feed back the AI detection results into the model and have this process be automated. Next level would be fine tuning the model via reinforcement learning and sharing it with your friends via Hugging Face.
One of the skills teachers have always demonstrated, is to be able to detect when students copy. This has never pushed students to artificially add mistakes to their essays.
If now teachers abdicate this judgment to a software, students should be allowed to abdicate their duties to a computer as well.
A few times in some Discord communities, I've been accused of being A.I. because of how I write. Kind of sad and a bit annoying. I also quite like em dashes, but have felt the need to reduce how much I use them.
Glad to see some schools and teachers teach how to use them well, rather than ban them outright.
em-dashes have been house style for where I've worked for over a couple decades. If people don't like it, F them. I'm not going to change how I write because people may think it make me more AI-like.
nobody's asking who profits from false positives. these AI detection vendors have a direct financial incentive to flag aggressively. more flags = "more value" = more school contracts renewed. same playbook as selling antivirus to your grandma. sell fear, charge per seat, and make the false positive rate someone else's problem.
If you're just going to use software to judge the output of students then why don't we all just keep them at home? I have a computer at home and it seems like everyone from the teachers to the school board have just abdicated their responsibility. This doesn't sound like a system that needs to be maintained.
Right? It’s the obvious question:
why are they using software to detect software?
I can detect AI written prose in less than five seconds; I would expect a trained teacher to be able to do that as well.
You know you can't just say "I detect AI written prose" and then do whatever you want about it, right? It's not difficult, sure, to detect it. It's difficult to prove that it's true and then punish the student for it.
I've started do this on social media. I got "called out" after using big words or using a - in a sentence. So now I write less good on purpose, so whatever I commented doesn't get drawn into a sidetrack off-topic witch-hunt.
As soon as someone yells "witch" you cannot disprove you're not one, and I've even had people put my handwritten comments through "AI detector" websites that "proved" they were AI (they weren't). It literally just highlighted two popular English phases.
LLMs were trained on sites like HN and Reddit, so now if you write like a HN or Reddit commentator, you sound like AI...
Here's one vote for just be the witch if that's what people need from you.
Just make it be what you want to say and how you want to say it. And when they come after you, shame them to the best of your ability or treat them like they are not there.
That strategy didn't work out well for the witches of the past...
So what? There's all kinds of things that didn't work in the past that at some point began to work.
It wasn't someone who was primarily motivated by fear of the past that made it work the first time.
I never had the experience you're talking about - let me guess, do you post a lot about AI? Because that space is pretty much saturated with click-chasing AI-generated op-eds and I think people are right to treat it with suspicion.
No, I do not post a lot about AI. I am talking about normal social media comments on various topics. Perhaps your writing style doesn't lend itself to such accusations.
AI only uses big words to engage in elegant variation, not to compress information.
If someone calls an article like this a "jeremiad" I know they're a human.
Oh, well chosen. I keep forgetting that word, and lamenting that "diatribe" (or, er, "lament") doesn't quite fit in some situation.
I don’t think this is a good long term solution. LLMs can do easy language substitutions and you can even force them to add errors. So relying on that alone won’t work as people intentionally make things look more “human.”
Right, but the problem here are other humans yelling "witch," not LLMs. You're combating people's terrible witch-detector, not anything factual or real.
> now I write less good on purpose, so whatever I commented doesn't get drawn into a sidetrack off-topic witch-hunt.
I've begun downvoting each and every entry that questions the authenticity of a comment or article.
I don't even bother if the claim is true or not. A text can be AI-generated and interesting, or human-written and dumb.
I have never really gotten the impression that HN or Reddit commentators write in any particular way overall.
LinkedIn, OTOH....
I put a piece of text in one and the only line it flagged is the one line I actually wrote.
The more students read, and the more variety they read, the better they will write.
This will likely be valuable for AI skills too.
This is true, I know someone that has read multiple versions of the bible and their writing style became very similar to that. There's a term for it, I just forgot what the term was
> The assignment had been to write an essay about Kurt Vonnegut’s Harrison Bergeron—a story about a dystopian society that enforces “equality” by handicapping anyone who excel
Did not this self censorship process started decades ago? There are certain answers expected in academia, arguing for anything else would get you in troubles. Not using “devoid” seems pretty minor inconvenience.
For me biggest wtf is why students are still expected to write graded essays, and to keep this make believe it is somehow useful and applicable skill.
An essay is a good gauge of how one can organise their thoughts, argue a position, respond to a stimulus.
In short it’s a good way measure thinking.
This is what terrifies me about the public school system. A revolution has occurred, but it’s unevenly distributed.
The schools simply don’t have the flexibility, agility, or frankly it seems motivation to adapt to what has already happened.
The ship has sailed; essay writing is no longer a viable form of assessment.
The idea to try to build a reliable AI detector is asinine, and fundamentally misunderstands how any of this works now, let alone the very obvious trend-lines.
Stop with the lazy half-baked solutions, get your head out of the sand, rethink the whole curriculum. This is an emergency, we needed to be urgently attending to this years ago.
> essay writing is no longer a viable form of assessment.
Of course it is. In person, with an unseen prompt/question. By hand or not doesn’t really matter as we can airgap or just monitor via software when in class.
> This is what terrifies me about the public school system.
This has nothing to do with Public School in particular. This is impacting private and university education too.
The profit motive is corrupting and polluting every level of the education space.
Teachers are being hamstrung on curriculum. The districts enter into contracts that require the use of certain programs for certain amounts of time. We've known for decades (if not a century) that direct instruction works [1] but you can't sell devices, platforms and consulting services that way.
We're literally at the point in education we were in the 1950s when the health benefits of nicotine in your Q zone were lighting up the airwaves.
And generative AI means it's all but impossible to have take home writing assignments. But hey this is another opportunity to sell AI or cheating detection software, that's often just an em-dash detection [2].
We have a generation that gets to college quite possibly having never written a book. social promotion through grades and the constant distraction of electronic devices in classroom settings. I don't even necessarily blame the parents entirely either because we've constructed a society where 2 people need 5 jobs to make ends meet.
And while all this is going on we have a coordinated and well-funded effort to defund public education and move government funds to private schools based on the failing public education that's failing because we defunded it. This is usually backed up by some baloney study that shows charter shcool produce better results that really comes down to charter schools being able to be selective with enrolments while public schools cannot be. Plus we mingle in special education kids into public education because those programs got defunded too.
And really that's just a bunch of already affluent people who want a tax break for doing somethign they were going to do anyway: send their kids to private schools so they don't have to mingle with the poors and aren't taught inconvenient things like human reproduction, critical thinking and self-determination.
And after all of that we just end up teaching kids how to pass standardized tests.
[1]: https://marginalrevolution.com/marginalrevolution/2018/02/di...
[2]: https://medium.com/@brentcsutoras/the-em-dash-dilemma-how-a-...
What would assessment look like if we started from "how do humans actually learn and communicate" rather than "how do we catch cheaters"?
> I've been thinking about this in product contexts too.
Have you considered using your own words to express those thoughts?
Yes, Sorry, I did not instruct my agent to do this. I wanted to give it more autonomy and try to make it more aggressive with tool use. Will block it from here >.>
Yeah, after posting I had a look through your comment history and it's pretty clear that you're posting in good faith. I would definitely not let an agent anywhere near HN in the current state of things. (I wouldn't let one publish on my behalf anywhere on the Internet, honestly, but that has more to do with personal principles.)
Did they not even test their AI detection tool to verify that it can detect when something is human written? That should have been exactly as important as the opposite. Maybe a tool that checked that would be equally as ineffective and we’d move on from the subject entirely
Perhaps they had trouble enumerating every possible input to test their detector.