This all sounds good on paper but I have an honest question: How do you develop judgement as a junior or mid level engineer to critique AI if you yourself never learned that skill by making mistakes in the first place?
I’m not saying that all code should be typed by hand in 2026 but there are certain subtle things you learn only when you get into nitty gritty details especially related to security.
Also today’s AI is notoriously bad at ownership. When you ask it to give you a concrete answer, it will still give you options with pros & cons of each so that ultimately you own the decision and not it. So how do you decide between the two (or more) when you never learned to do it yourself?
So, how is this different from years ago when there were software libraries but in class we still rolled our own implementation to understand the algo. For example in machine learning, yes you could just use a Matlab toolbox or libsvm or scikit-learn. But then you wouldn't learn the SVM training algorithm, so we had to implement it ourselves. We knew that we wouldn't roll our own SVM at a job, in production. The goal of the exercise was to learn the algorithm and how it actually works.
Today we have AI, which is basically like having infinite libraries available that do what you ask for. But you will not learn if you just take that code, similarly to how you don't learn if you just call scikit-learn to train your SVM.
And of course students grumbled back then also and said why do we need to do this when all those libraries exist?
Learning often requires not taking the most efficient path for every project.
Precisely my point. But unfortunately companies are rushing to get the features out. No cares if you’re learning and augmenting your own understanding or not.
> This all sounds good on paper but I have an honest question: How do you develop judgement as a junior or mid level engineer to critique AI if you yourself never learned that skill by making mistakes in the first place?
Shouldn't this be a pretty fundamental part of a degree? If you're in school right now for a CS or SWE related degree, I would imagine you're learning how code actually works, how the math actually works, etc.
If an LLM makes a mistake, you should be able to call that out.
You might be arguing that a junior engineer may not have the RWE to make those judgements, but by the time you've completed a 4-5 year degree, you should have done the following:
- Completed coursework where you've learned the fundamentals of programming and software engineering
- Have done dozens of projects (building everything from basic web apps to more advanced pieces of software) where you've seen what works, what doesn't work, etc. This also gives you real world exposure to the latest and greatest frameworks, tools, etc.
- Have done several (at least 2) internships where you've worked at a real company writing real code, and have seen/been mentored into what AI is good at, where it fails, etc.
> Shouldn't this be a pretty fundamental part of a degree? If you're in school right now for a CS or SWE related degree, I would imagine you're learning how code actually works, how the math actually works, etc.
CS and SWE are radically different subjects. Science vs Engineering. I'm sure it varies by school but many CS grads will have approximately zero exposure to engineering concepts or any of the latest and greatest frameworks.
> internships where you've worked at a real company writing real code
Though why would companies make that nontrivial investment if they can use AI instead?
Furthermore, I’d argue that learning good judgement takes around a decade of full-time dev experience. In particular, experiencing the long-term consequences of one’s design and implementation decisions. You don’t get that just by college education and a few internships.
> Shouldn't this be a pretty fundamental part of a degree? If you're in school right now for a CS or SWE related degree, I would imagine you're learning how code actually works, how the math actually works, etc.
I’m sorry to say but colleges don’t teach the kinda stuff you’re gonna use day-to-day. They’ll focus on architecture, data structures, algorithms, databases, OS internals theory that only a minuscule number of systems engineers would get to work on. They won’t teach you that in your /login endpoint, if a user is not found, you should verify the password against a pre-computed dummy hash so the response delay matches a real user account workflow to avoid timing attacks. You only learn this on the job under the supervision of a senior mentor.
Depends a lot on the college. I had electives like Java phone app development (J2ME, dating myself), DirectX game engine development, web development, alongside the more theoretical math classes.
> They won’t teach you that in your /login endpoint, if a user is not found, you should verify the password against a pre-computed dummy hash so the response delay matches a real user account workflow to avoid timing attacks. You only learn this on the job under the supervision of a senior mentor.
Actually this kind of timing attacks were taught to me in information security and cryptography class, alongside other side channel attacks.
> I’m sorry to say but colleges don’t teach the kinda stuff you’re gonna use day-to-day.
This was not my experience in college. A lot of it was VERY applied. Granted, that was 13 years ago.
But even so, if you're at a college where you feel that you're not getting enough exposure, that's why I also called out "Have done dozens of projects" and "Have done several (at least 2) internships".
Stakeholders are not technical enough to understand that so CEOs won’t do it. We are already seeing it play out. Entry-level hiring has been consistently down ever since AI got mainstream.
- does the company provide the harness and specific model for this? Or the interviewee use whatever they have access to? If they don't do a good job, is it the fault of the harness or the model or the interviewee not knowing how to fully utilise the harness and model or alternative harnesses and models?
- if every interviewee uses different harnesses and models, how does the company ensure that it's a fair comparison between interviewees for the same role?
- will the interviewer go through all lines in the code generated or let AI do it?
Something I appreciated when going through Anthropic interview process was that I only dealt with humans. They could have been showcasing how even their interviews are done with Claude or something, but no, the whole loop was interacting with real humans, and nothing was on the topic “how do you work with LLMs”. It was about how me, the human, think and approach situations, and how I handle working with others (humans), etc. Which made a lot of sense to me, and I overall appreciated their process (other than it’s very time consuming, requires lots of prep). But it’s interesting to see that AI labs customers feel the need to showcase how much of their processes are now around agentic stuff
Hmmm... I had a different experience. They had a fully automated environment where you ha to write test that passes some tests. No human involved. And the time requirement was insanely tight
they will use their own product. but they will rule like the plutocracy that they are. they will tell me that they are not privileged to have my custom, rather, i am privileged to be their customer.
we will tell ourselves that we should wait eagerly to see if we will be graced by "tibo's" largesse, like peasants clamouring to see a royal throwing bread among the masses.
or that it is right and good that claude should refuse to answer on account of safety, but hack other companies on behalf of anthropic. surely we are too simple, and sometimes naive, truly we know not what is best for ourselves.
It's imperative to understand that Anthropic and OpenAI view their customers as subhuman drug addicts (read: *users*).
From this perspective, the people who use these tools are beneath the people who design them. From this perspective, the rightful state of the world is for the superior members of the in-group to control the experience of the inferior members of the out-group.
By using these tools, you willingly accept the position of the inferior/the sub-human/the addict.
AI critics and honest AI executives of 2023 might say that AI companies are automating labor so that Capital can capture all productive value. But we have to give them some credit. They are hiring real humans. And not even hounding them about how they are not being sufficiently useless-meatbag relay points for the AI models and agents. And that demonstrates that they are actually being real humane mensch.
Meanwhile, how many jobs have AI critics created? Crickets.
Where are they getting all those free tokens from?!? I don't like this rhetoric. It still costs money to write code, only now that wealth gets transferred to Anthropic instead of to individual contributors.
And even ignoring the token costs, that is just not true.
Sure, for some cases, the cost is going down. I and others on my team "one-shotted" impressive features in an hour of agent work that would have taken humans probably a month if done completely manually.
I've also seen agents going in circles for an hour on a fix that would have taken even a junior 3 minutes to get right (after it was struggling for 10 minutes, I wanted to see if it can ever get it right as an experiment).
And when taking a look at the whole organization, orgs still just aren't really shipping that much more quality features to their users as the impressive one-shot demos would make you think.
I'm not sure if it is a good idea when you are referencing your "rebuilding" after you reduce headcount by 14% . If this isn't some kind of entendre then it's actually extremely effective.
I'm very skeptical of that graph. What does it mean for code to be AI-generated vs human-generated?
If I merge a PR that was opened by Claude do all the changes in it automatically count as AI-generated? How about the individual commits? If I'm reviewing locally and make manual changes but then have Claude create the commit it appears to be AI-generated but might not have been.
The point is that these metrics are easy to game and I've definitely wasted time and tokens refining code with Claude that would have been easier to just edit by hand. It can be kind of fun and when the goal is just "use AI" I don't find it surprising that graphs showing 100% switch to AI-generated code could be defensibly generated without really saying anything about how much manual intervention is happening or how efficient the process is.
I've been considering different ways to measure code understanding and fluency. I'm still using traditional 'easy' problems on a coding pad to probe this, but I'm finding a new generation candidates that don't prepare for this, and may be able to have a full career with just a higher level of code understanding. But they might not ever develop the ride-a-bike with one hand level of fluency unless they grind pointless leetcodes.
So I'm increasingly uncertain about what and how to test. My default for now is still to rely on ability to write basic code fluently, but I'm open to changing this perspective.
I really want to know how this existing repo AI-assisted live coding test works, with example problems.
It seems the standard data structure puzzle type thing won't be feasible if you are using an LLM.
Also the latency for these agentic coding/prompts seems like it would make the interview a bit awkward.
Anyone been conducting or taking interviews with this kind of thing with thoughts to share?
I’ve been conducting AI Coding interviews at my job. Candidate will screen share and use AI-assisted coding environment and tools of their choice.
I ask them to implement xyz thing. What I’m looking for is how they interact with the AI agent. Do they ask the agent to plan first? Do they review the plan? Etc. It’s pretty typical stuff that you might expect an experienced engineer to do if they effectively use such tools daily.
There are a series of follow-up questions about how to productionize the toy system, which gives some additional signal about how well they understand what they’re making. I sprinkle these in when there’s dead air waiting for the bot to think.
I think we’ll need to evolve and refine this problem and process as the models continue to improve.
>It’s pretty typical stuff that you might expect an experienced engineer to do if they effectively use such tools daily.
It's also the kind of stuff that somebody can learn in a week, so not hiring the right person who just didn't spend this week of time yet for whatever reason is a loss.
Yeah it tells you nothing of value. It's the Recency Bias taken to its ultimate conclusion isn't it. Anything I learned about AI I learned in the last few months but somehow that's as important as all the experience of my decades long software career? Come on.
It signals you don't have enough time or will to chase the latest stuff, otherwise you would have learned it with other cool kids half a year earlier. It carries some signal, but I would not choose based on this alone. I would even consider selecting against this to a certain point, but I also work in a place that COBOL on a mainframe. YMMV.
> I would even consider selecting against this to a certain point
I agree. A technology professional who views themselves as "one of the cool kids" (read: easily manipulated by social media) is a legitimate security threat, as are many of the popularly promoted approaches to "LLM-assisted software developement".
I am really looking forward to the day where AI is normalized during interviews. Right now the duality of interviews is basically:
- You get a leetcode question and if you're lucky is an easy medium that you can solve, if you're really lucky you already solved it and can pretend you are approaching the problem the first time. Good luck if you get a hard question and you never saw it before.
- You get a home assignment, in a framework you might not know but you're expected to be fluent with it, then waste 1 hour setting up the project structure, and one more hour to find out how the framework expects you to define the CORS allow list. You are expected to deliver the project in 3 hours.
The good I see in AI is that it completely removes the need to study just for interviews, and you can also delegate all the project setup to the AI. Then you can focus on what you would test (e2e? integration? what are the boundaries? what do we mock?), how to keep the documentation, how to structure your code. You have an expensive endpoint, do I make it sync or add an async jobs framework?
Imagine you're an expert in C++ interviewing for a Django position and the interview consists of fixing a big in a repo. The bug is that a function without type hinting is modifying what is expected to be a list, but the caller is passing a tuple. Trivial after a week you work in python and you have your environment set up for type warnings, also trivial with AI and definitely not an interesting problem that shows expertise with software engineering in general.
We also did this in our last interview at work, and it was a really good indicator to see if someone just copy pasted code, or understood it after it was generated. Some candidates had a unit test fail and couldn't debug it for his life, even if he "wrote" all the code himself. Others simply did not understand the architecture they wrote, and assumed that a function defined with "async" and awaited would run in parallel from the code that called it (as if you spawned a thread)
I was on the interviewer side last month. We stopped giving take-home assignments because they don't produce a signal anymore.
We explicitly said -- feel free to use whatever framework and AI assistant, just show us how you do it. The practical part had no leetcode too. Just build something really basic, then explain a snippet of code (3 lines) and generalize it. A trick question (with a disclosure it's a trick question) if the candidate did it fast enough that we didn't have to go into the overtime. A bit of theory about protocols, all in all an hour and we leave another 30 minutes on top to answer questions.
At the end of the day we just filter out with confidently bad takes, people who can't do 2+2 and ones that can't understand the question without rephrasing it three times.
The most bizzarre candidate didn't know anything at all, but was so relaxed and confident, that he spent all of the 30 minutes asking about the company and how his day would like and all that, while he clearly bombed it.
The difficult part is how to not filter out a competent person who doesn't necessarily agree with all of your takes, uses all the same tools and had all of the same experiences as both of the interviewers.
> people who can't do 2+2 and ones that can't understand the question without rephrasing it three times
We resorted to filtering candidates in-person with 5 basic technical questions on pen and paper. And I mean _really_ basic questions.
This was surprisingly effective because it filters for many non-technical skills like being able to read and write English, follow instructions, and show up to the office, on time, and appropriately dressed.
The number of candidates that failed these basic skills was astounding. We had candidates show up 20 minutes late, or email 2 minutes before the "interview" asking for a Teams link even though the invitation stated the meeting was in-person (highlighted in yellow). Others couldn't write their own name legibly on the paper.
A candidate that cannot answer basic technical questions has no hope of being able to prompt an AI effectively or review the code it produces.
If you want it back, ruthlessly mock the low-performing "engineers" who are addicted to LLMs. In every company, we need to make these weaklings our scapegoats.
I would do the opposite and would likely use the same framework that some AI companies are doing to evaluate candidates. By either not allowing them to use AI assistants in any part of the interview process (Anthropic and OpenAI does this.) or give candidates a strict token limit (100k tokens) until the candidate runs out of tokens.
They have to earn it, as the tokens are not free.
Given that deskilling and over-reliance in AI assistance will continue to happen, putting a hard token limit <100k tokens in the interview process serves as a great filter to prevent the vibe-coders and "tokenmaxxers" out and forces a higher bar for quality, with clean code and reasoning across well maintained software with less tokens rather than increasing the slop.
Do you want a candidate that knows when to use AI and carefully uses tokens with in their limits, or do you want a candidate generating incomprehensible AI slop to be tokenmaxxing out your company limits and then draining your company bank account?
I would start looking for a new job if I heard that my company hired someone who wasn't able to prove that they could write code without LLM assistance. A lot of people like that are legitimately illiterate.
This all sounds good on paper but I have an honest question: How do you develop judgement as a junior or mid level engineer to critique AI if you yourself never learned that skill by making mistakes in the first place?
I’m not saying that all code should be typed by hand in 2026 but there are certain subtle things you learn only when you get into nitty gritty details especially related to security.
Also today’s AI is notoriously bad at ownership. When you ask it to give you a concrete answer, it will still give you options with pros & cons of each so that ultimately you own the decision and not it. So how do you decide between the two (or more) when you never learned to do it yourself?
So, how is this different from years ago when there were software libraries but in class we still rolled our own implementation to understand the algo. For example in machine learning, yes you could just use a Matlab toolbox or libsvm or scikit-learn. But then you wouldn't learn the SVM training algorithm, so we had to implement it ourselves. We knew that we wouldn't roll our own SVM at a job, in production. The goal of the exercise was to learn the algorithm and how it actually works.
Today we have AI, which is basically like having infinite libraries available that do what you ask for. But you will not learn if you just take that code, similarly to how you don't learn if you just call scikit-learn to train your SVM.
And of course students grumbled back then also and said why do we need to do this when all those libraries exist?
Learning often requires not taking the most efficient path for every project.
Precisely my point. But unfortunately companies are rushing to get the features out. No cares if you’re learning and augmenting your own understanding or not.
> This all sounds good on paper but I have an honest question: How do you develop judgement as a junior or mid level engineer to critique AI if you yourself never learned that skill by making mistakes in the first place?
Shouldn't this be a pretty fundamental part of a degree? If you're in school right now for a CS or SWE related degree, I would imagine you're learning how code actually works, how the math actually works, etc.
If an LLM makes a mistake, you should be able to call that out.
You might be arguing that a junior engineer may not have the RWE to make those judgements, but by the time you've completed a 4-5 year degree, you should have done the following:
- Completed coursework where you've learned the fundamentals of programming and software engineering
- Have done dozens of projects (building everything from basic web apps to more advanced pieces of software) where you've seen what works, what doesn't work, etc. This also gives you real world exposure to the latest and greatest frameworks, tools, etc.
- Have done several (at least 2) internships where you've worked at a real company writing real code, and have seen/been mentored into what AI is good at, where it fails, etc.
> Shouldn't this be a pretty fundamental part of a degree? If you're in school right now for a CS or SWE related degree, I would imagine you're learning how code actually works, how the math actually works, etc.
CS and SWE are radically different subjects. Science vs Engineering. I'm sure it varies by school but many CS grads will have approximately zero exposure to engineering concepts or any of the latest and greatest frameworks.
> internships where you've worked at a real company writing real code
Though why would companies make that nontrivial investment if they can use AI instead?
Furthermore, I’d argue that learning good judgement takes around a decade of full-time dev experience. In particular, experiencing the long-term consequences of one’s design and implementation decisions. You don’t get that just by college education and a few internships.
> Shouldn't this be a pretty fundamental part of a degree? If you're in school right now for a CS or SWE related degree, I would imagine you're learning how code actually works, how the math actually works, etc.
I’m sorry to say but colleges don’t teach the kinda stuff you’re gonna use day-to-day. They’ll focus on architecture, data structures, algorithms, databases, OS internals theory that only a minuscule number of systems engineers would get to work on. They won’t teach you that in your /login endpoint, if a user is not found, you should verify the password against a pre-computed dummy hash so the response delay matches a real user account workflow to avoid timing attacks. You only learn this on the job under the supervision of a senior mentor.
Depends a lot on the college. I had electives like Java phone app development (J2ME, dating myself), DirectX game engine development, web development, alongside the more theoretical math classes.
> They won’t teach you that in your /login endpoint, if a user is not found, you should verify the password against a pre-computed dummy hash so the response delay matches a real user account workflow to avoid timing attacks. You only learn this on the job under the supervision of a senior mentor.
Actually this kind of timing attacks were taught to me in information security and cryptography class, alongside other side channel attacks.
> I’m sorry to say but colleges don’t teach the kinda stuff you’re gonna use day-to-day.
This was not my experience in college. A lot of it was VERY applied. Granted, that was 13 years ago.
But even so, if you're at a college where you feel that you're not getting enough exposure, that's why I also called out "Have done dozens of projects" and "Have done several (at least 2) internships".
You hire juniors but don’t let them use AI tools for a year :)
Stakeholders are not technical enough to understand that so CEOs won’t do it. We are already seeing it play out. Entry-level hiring has been consistently down ever since AI got mainstream.
Curious to know:
- does the company provide the harness and specific model for this? Or the interviewee use whatever they have access to? If they don't do a good job, is it the fault of the harness or the model or the interviewee not knowing how to fully utilise the harness and model or alternative harnesses and models?
- if every interviewee uses different harnesses and models, how does the company ensure that it's a fair comparison between interviewees for the same role?
- will the interviewer go through all lines in the code generated or let AI do it?
Something I appreciated when going through Anthropic interview process was that I only dealt with humans. They could have been showcasing how even their interviews are done with Claude or something, but no, the whole loop was interacting with real humans, and nothing was on the topic “how do you work with LLMs”. It was about how me, the human, think and approach situations, and how I handle working with others (humans), etc. Which made a lot of sense to me, and I overall appreciated their process (other than it’s very time consuming, requires lots of prep). But it’s interesting to see that AI labs customers feel the need to showcase how much of their processes are now around agentic stuff
Hmmm... I had a different experience. They had a fully automated environment where you ha to write test that passes some tests. No human involved. And the time requirement was insanely tight
WORK FASTER, SLAVE! FASTER!
Fourth crack commandment: never get high on your own supply
they will use their own product. but they will rule like the plutocracy that they are. they will tell me that they are not privileged to have my custom, rather, i am privileged to be their customer.
we will tell ourselves that we should wait eagerly to see if we will be graced by "tibo's" largesse, like peasants clamouring to see a royal throwing bread among the masses.
or that it is right and good that claude should refuse to answer on account of safety, but hack other companies on behalf of anthropic. surely we are too simple, and sometimes naive, truly we know not what is best for ourselves.
no more.
Vive la révolution!!
It's imperative to understand that Anthropic and OpenAI view their customers as subhuman drug addicts (read: *users*).
From this perspective, the people who use these tools are beneath the people who design them. From this perspective, the rightful state of the world is for the superior members of the in-group to control the experience of the inferior members of the out-group.
By using these tools, you willingly accept the position of the inferior/the sub-human/the addict.
It's why you never meet a dog that works in a dogfood factory.
AI critics and honest AI executives of 2023 might say that AI companies are automating labor so that Capital can capture all productive value. But we have to give them some credit. They are hiring real humans. And not even hounding them about how they are not being sufficiently useless-meatbag relay points for the AI models and agents. And that demonstrates that they are actually being real humane mensch.
Meanwhile, how many jobs have AI critics created? Crickets.
> When the cost of building goes to zero
Where are they getting all those free tokens from?!? I don't like this rhetoric. It still costs money to write code, only now that wealth gets transferred to Anthropic instead of to individual contributors.
And even ignoring the token costs, that is just not true.
Sure, for some cases, the cost is going down. I and others on my team "one-shotted" impressive features in an hour of agent work that would have taken humans probably a month if done completely manually.
I've also seen agents going in circles for an hour on a fix that would have taken even a junior 3 minutes to get right (after it was struggling for 10 minutes, I wanted to see if it can ever get it right as an experiment).
And when taking a look at the whole organization, orgs still just aren't really shipping that much more quality features to their users as the impressive one-shot demos would make you think.
I'm not sure if it is a good idea when you are referencing your "rebuilding" after you reduce headcount by 14% . If this isn't some kind of entendre then it's actually extremely effective.
https://xcancel.com/brian_armstrong/status/20516167591451857...
I'm very skeptical of that graph. What does it mean for code to be AI-generated vs human-generated?
If I merge a PR that was opened by Claude do all the changes in it automatically count as AI-generated? How about the individual commits? If I'm reviewing locally and make manual changes but then have Claude create the commit it appears to be AI-generated but might not have been.
The point is that these metrics are easy to game and I've definitely wasted time and tokens refining code with Claude that would have been easier to just edit by hand. It can be kind of fun and when the goal is just "use AI" I don't find it surprising that graphs showing 100% switch to AI-generated code could be defensibly generated without really saying anything about how much manual intervention is happening or how efficient the process is.
I've been considering different ways to measure code understanding and fluency. I'm still using traditional 'easy' problems on a coding pad to probe this, but I'm finding a new generation candidates that don't prepare for this, and may be able to have a full career with just a higher level of code understanding. But they might not ever develop the ride-a-bike with one hand level of fluency unless they grind pointless leetcodes.
So I'm increasingly uncertain about what and how to test. My default for now is still to rely on ability to write basic code fluently, but I'm open to changing this perspective.
I really want to know how this existing repo AI-assisted live coding test works, with example problems.
It seems the standard data structure puzzle type thing won't be feasible if you are using an LLM.
Also the latency for these agentic coding/prompts seems like it would make the interview a bit awkward.
Anyone been conducting or taking interviews with this kind of thing with thoughts to share?
I’ve been conducting AI Coding interviews at my job. Candidate will screen share and use AI-assisted coding environment and tools of their choice.
I ask them to implement xyz thing. What I’m looking for is how they interact with the AI agent. Do they ask the agent to plan first? Do they review the plan? Etc. It’s pretty typical stuff that you might expect an experienced engineer to do if they effectively use such tools daily.
There are a series of follow-up questions about how to productionize the toy system, which gives some additional signal about how well they understand what they’re making. I sprinkle these in when there’s dead air waiting for the bot to think.
I think we’ll need to evolve and refine this problem and process as the models continue to improve.
>It’s pretty typical stuff that you might expect an experienced engineer to do if they effectively use such tools daily.
It's also the kind of stuff that somebody can learn in a week, so not hiring the right person who just didn't spend this week of time yet for whatever reason is a loss.
Yeah it tells you nothing of value. It's the Recency Bias taken to its ultimate conclusion isn't it. Anything I learned about AI I learned in the last few months but somehow that's as important as all the experience of my decades long software career? Come on.
It signals you don't have enough time or will to chase the latest stuff, otherwise you would have learned it with other cool kids half a year earlier. It carries some signal, but I would not choose based on this alone. I would even consider selecting against this to a certain point, but I also work in a place that COBOL on a mainframe. YMMV.
> I would even consider selecting against this to a certain point
I agree. A technology professional who views themselves as "one of the cool kids" (read: easily manipulated by social media) is a legitimate security threat, as are many of the popularly promoted approaches to "LLM-assisted software developement".
I am really looking forward to the day where AI is normalized during interviews. Right now the duality of interviews is basically:
- You get a leetcode question and if you're lucky is an easy medium that you can solve, if you're really lucky you already solved it and can pretend you are approaching the problem the first time. Good luck if you get a hard question and you never saw it before.
- You get a home assignment, in a framework you might not know but you're expected to be fluent with it, then waste 1 hour setting up the project structure, and one more hour to find out how the framework expects you to define the CORS allow list. You are expected to deliver the project in 3 hours.
The good I see in AI is that it completely removes the need to study just for interviews, and you can also delegate all the project setup to the AI. Then you can focus on what you would test (e2e? integration? what are the boundaries? what do we mock?), how to keep the documentation, how to structure your code. You have an expensive endpoint, do I make it sync or add an async jobs framework?
Imagine you're an expert in C++ interviewing for a Django position and the interview consists of fixing a big in a repo. The bug is that a function without type hinting is modifying what is expected to be a list, but the caller is passing a tuple. Trivial after a week you work in python and you have your environment set up for type warnings, also trivial with AI and definitely not an interesting problem that shows expertise with software engineering in general.
We also did this in our last interview at work, and it was a really good indicator to see if someone just copy pasted code, or understood it after it was generated. Some candidates had a unit test fail and couldn't debug it for his life, even if he "wrote" all the code himself. Others simply did not understand the architecture they wrote, and assumed that a function defined with "async" and awaited would run in parallel from the code that called it (as if you spawned a thread)
I was on the interviewer side last month. We stopped giving take-home assignments because they don't produce a signal anymore.
We explicitly said -- feel free to use whatever framework and AI assistant, just show us how you do it. The practical part had no leetcode too. Just build something really basic, then explain a snippet of code (3 lines) and generalize it. A trick question (with a disclosure it's a trick question) if the candidate did it fast enough that we didn't have to go into the overtime. A bit of theory about protocols, all in all an hour and we leave another 30 minutes on top to answer questions.
At the end of the day we just filter out with confidently bad takes, people who can't do 2+2 and ones that can't understand the question without rephrasing it three times.
The most bizzarre candidate didn't know anything at all, but was so relaxed and confident, that he spent all of the 30 minutes asking about the company and how his day would like and all that, while he clearly bombed it.
The difficult part is how to not filter out a competent person who doesn't necessarily agree with all of your takes, uses all the same tools and had all of the same experiences as both of the interviewers.
> people who can't do 2+2 and ones that can't understand the question without rephrasing it three times
We resorted to filtering candidates in-person with 5 basic technical questions on pen and paper. And I mean _really_ basic questions.
This was surprisingly effective because it filters for many non-technical skills like being able to read and write English, follow instructions, and show up to the office, on time, and appropriately dressed.
The number of candidates that failed these basic skills was astounding. We had candidates show up 20 minutes late, or email 2 minutes before the "interview" asking for a Teams link even though the invitation stated the meeting was in-person (highlighted in yellow). Others couldn't write their own name legibly on the paper.
A candidate that cannot answer basic technical questions has no hope of being able to prompt an AI effectively or review the code it produces.
Writing for investors in the AI Era
- We have embraced everything about the AI Era at our company
- Arbitrary topic about how that changes something
We gave away the most fun part of being a SWE to the clankers
The most fun part being seeping through stderr of tsc and restarting the unit test runner after realigning some assumptions.
If you want it back, ruthlessly mock the low-performing "engineers" who are addicted to LLMs. In every company, we need to make these weaklings our scapegoats.
I would do the opposite and would likely use the same framework that some AI companies are doing to evaluate candidates. By either not allowing them to use AI assistants in any part of the interview process (Anthropic and OpenAI does this.) or give candidates a strict token limit (100k tokens) until the candidate runs out of tokens.
They have to earn it, as the tokens are not free.
Given that deskilling and over-reliance in AI assistance will continue to happen, putting a hard token limit <100k tokens in the interview process serves as a great filter to prevent the vibe-coders and "tokenmaxxers" out and forces a higher bar for quality, with clean code and reasoning across well maintained software with less tokens rather than increasing the slop.
Do you want a candidate that knows when to use AI and carefully uses tokens with in their limits, or do you want a candidate generating incomprehensible AI slop to be tokenmaxxing out your company limits and then draining your company bank account?
I would start looking for a new job if I heard that my company hired someone who wasn't able to prove that they could write code without LLM assistance. A lot of people like that are legitimately illiterate.