so, i've gone through a bunch of iterations on how to interview in days of LLMs and i came up with a process that is quite good:
first, a coding problem that is solved live (right now, online) -- it's not a hard one and most LLMs can easily one shot it.
BUT: if you are not a senior developer, you need to do it without LLM, and if you are a senior+ and do it via LLM, the bar is super raised and now you need to explain in details everything the LLM wrote as if you wrote it yourself.
the main component here is to know if the candidate understands coding in general, understands problem solving and how to dive into an existing (although quite small) codebase. solving the problem is not really the goal -- in fact, most candidates we hired did not solve it 100%, but were able to understand and explain everything that they put it on code. this usually takes 1h30.
(we also ask the candidate to open a PR against the repo just to see if they know the basics of git -- you would be surprise how many people cannot do it).
the second is an architecture interview, where we build a system together BUT instead of doing the whole spec, miro board, whatever, we do it in more of a free way: we start with a really simple, MVP level, spec and we ask how the candidate would build it. then, we add another feature, and we ask how they would go for it. during this whole thing, i ask random questions like "why would you use x instead of y?" or "why do you prefer this approach?".
one important thing: we allow the candidate to say "i don't know" and we tell the candidate that there's no gotcha questions (which is true, i hate these kind of questions). this lets the candidate be super comfortable (feedback we had from real people).
in general, this flow helped us a ton and i really really like running it :)
I feel like git operations are one of the few things that I truly don't care at all about whether someone uses an LLM for and doesn't tell me. PR descriptions and commit messages should be written by humans for humans, but like, if someone doesn't know how to manually configure their git remotes and what exact command to use to amend/squash their commits into the shape needed and instead has an agent prepare all that and use the GitHub CLI to open the PR, why should I care? Beyond the code itself, I care about the things that will continue existing in the long run fitting my team's conventions (the number of commits, the commit messages, the PR description and comments, and then whether there's a merge commit at the end), but the process used to create those things doesn't matter to me.
At one job I had, we used git extensively, but not GitHub. We had internal systems for review and branch management. I could tell you all the gory details about various workflows and landmines in git. Is the test here really about git, or git+GitHub?
Something similar for senior devs: code reviews. Reading and understanding code has always been important. I find offering simple code reviews very telling. In a way, your LLM approach matches that. It's a way to do a code review. It's a test of understanding, and not just memorization.
This is the third Jane Street-related thing I've seen on here this week after not seeing much about them that I can remember any time recently beforehand I'm aware of their use of OCaml and probably do other things that might be of interest to the HN community, but it seems like a lot in a very short time out of nowhere. Is there some sort of marketing push or something going on?
People like to talk about Jane Street because it is this Very High IQ™ company that pays people astronomical salaries and is extremely hard to get into.
They don't have to do marketing, though. It's not like they sell anything to Hacker News readers, they make their money trading finantial assets
You don't think that companies who want to hire engineers ever try to post stuff to Hacker News as a way to market themselves as a potential employer? Where did you get this impression of "Very High IQ" and "astronomical salaries"? I'm guessing at least part of it is from reading stuff from them or from talking to other people who read stuff from them.
The fact that you seem to adamantly disagree that there's any plausible angle here doesn't mean it's not marketing, but it might mean that it is marketing and that it's working on you.
Such strong wording, you got that from me saying "They don't have to do marketing"?
> where did you get this impression of [...] "astronomical salaries"?
From the real world fact that the base salary is US$300k/yr, and often said that the total compensation for the first year ends up being US$700k+. Even for Manhattan + SWE jobs, surely that merits "astronomical". Other devs making that much are usually working at Anthropic or OpenAI, aren't they?
> where did you get this impression of "Very High IQ"?
You didn't copy my ™ there, which was meant to be snarky.
In any case, what I was trying to say is that they don't have to do marketing in the usual sense. They don't have to get the Jane Street logo in your subconscious so you buy a Jane St. product when you go to Kroger's, and you can't sign a Jane St. subscription plan.
People in this thread seem to be implying that the company is botting/astroturfing, but for what? To maintain the image that they're a highly sought after employer?
Anything Paul Graham posts gets upvoted here instantly, because it's Paul Graham who posted it. For Jane Street there's the same halo effect, because many believe Jane Street's work is impressive. Is it really easier to believe the idea of botting to get an even longer list of candidates to interview, than it is to believe people just like to hear what this company has to say?
I don't ever see anyone saying Paul Graham is botting posts about his "essays"
I had lunch with a couple friends this week, and we each work in different tech environments. Most of the conversation was about the workflows we're each using, and seeing our colleagues use.
It's a really interesting time to work in this field. People who are paying attention to workflows are using a range of tools, and constantly refining their own workflow. For example a whole team might use Claude, but one person collects data about their own usage on an ongoing basis and develops more advanced workflows than everyone else on the team. Their focus isn't just about working faster, but about doing better work.
I don't think the post is vague; I think we're at a time where you need flexibility in your interview process to find candidates who are doing the best work these days.
This sort of announcement has made me very happy I left the software world. I still code as part of my job, but it's not my main part and luckily I don't have to use LLMs are part of my job. I still code 100% by hand.
In some ways, although I detest AI, it has clarified the sort of people and organizations I want to associate with. Any latent characteristics that I abhor have been made obvious with the existence of AI and of course, I would want nothing to do with Jane Street.
so, i've gone through a bunch of iterations on how to interview in days of LLMs and i came up with a process that is quite good:
first, a coding problem that is solved live (right now, online) -- it's not a hard one and most LLMs can easily one shot it.
BUT: if you are not a senior developer, you need to do it without LLM, and if you are a senior+ and do it via LLM, the bar is super raised and now you need to explain in details everything the LLM wrote as if you wrote it yourself.
the main component here is to know if the candidate understands coding in general, understands problem solving and how to dive into an existing (although quite small) codebase. solving the problem is not really the goal -- in fact, most candidates we hired did not solve it 100%, but were able to understand and explain everything that they put it on code. this usually takes 1h30.
(we also ask the candidate to open a PR against the repo just to see if they know the basics of git -- you would be surprise how many people cannot do it).
the second is an architecture interview, where we build a system together BUT instead of doing the whole spec, miro board, whatever, we do it in more of a free way: we start with a really simple, MVP level, spec and we ask how the candidate would build it. then, we add another feature, and we ask how they would go for it. during this whole thing, i ask random questions like "why would you use x instead of y?" or "why do you prefer this approach?".
one important thing: we allow the candidate to say "i don't know" and we tell the candidate that there's no gotcha questions (which is true, i hate these kind of questions). this lets the candidate be super comfortable (feedback we had from real people).
in general, this flow helped us a ton and i really really like running it :)
(edit: spelling)
I feel like git operations are one of the few things that I truly don't care at all about whether someone uses an LLM for and doesn't tell me. PR descriptions and commit messages should be written by humans for humans, but like, if someone doesn't know how to manually configure their git remotes and what exact command to use to amend/squash their commits into the shape needed and instead has an agent prepare all that and use the GitHub CLI to open the PR, why should I care? Beyond the code itself, I care about the things that will continue existing in the long run fitting my team's conventions (the number of commits, the commit messages, the PR description and comments, and then whether there's a merge commit at the end), but the process used to create those things doesn't matter to me.
> we also ask the candidate to open a PR against the repo just to see if they know the basics of git
I've heard this multiple times now and it still boggles my mind that git is something you have to test for.
At one job I had, we used git extensively, but not GitHub. We had internal systems for review and branch management. I could tell you all the gory details about various workflows and landmines in git. Is the test here really about git, or git+GitHub?
it's incredibly sad but it's real -- so many senior candidates have problems with commit messages, opening PRs.
not even joking: i've seen people with like 10y of experience trying to push to main :(
PRs are not only not a basic feature of git - it's not a git feature at all. Though button clicking on GitHub is anyway a crazy thing to test
Something similar for senior devs: code reviews. Reading and understanding code has always been important. I find offering simple code reviews very telling. In a way, your LLM approach matches that. It's a way to do a code review. It's a test of understanding, and not just memorization.
This is the third Jane Street-related thing I've seen on here this week after not seeing much about them that I can remember any time recently beforehand I'm aware of their use of OCaml and probably do other things that might be of interest to the HN community, but it seems like a lot in a very short time out of nowhere. Is there some sort of marketing push or something going on?
Probably a self fulfilling prophecy.
Jane street news made it to the front page a couple times and got a lot of attention.
Now more people are recognizing the name and upvoting other Jane Street posts.
They’re pretty techy, idk why they’re showing up recently though.
Also one of my absolute favorite programming talks ever was hosted by Jane street:
https://www.janestreet.com/tech-talks/algorithm-for-precisio...
It’s like half talk, half story. Got me super interested in logic programming, and minikanren in particular.
I'd lean towards no on that. Might be a bit of a Baader-Meinhof thing. They've been mentioned/discussed on HN pretty regularly for years:
https://hn.algolia.com/?q=jane+street
People like to talk about Jane Street because it is this Very High IQ™ company that pays people astronomical salaries and is extremely hard to get into.
They don't have to do marketing, though. It's not like they sell anything to Hacker News readers, they make their money trading finantial assets
They sell employment to HN, is the parent comment's point.
You don't think that companies who want to hire engineers ever try to post stuff to Hacker News as a way to market themselves as a potential employer? Where did you get this impression of "Very High IQ" and "astronomical salaries"? I'm guessing at least part of it is from reading stuff from them or from talking to other people who read stuff from them.
The fact that you seem to adamantly disagree that there's any plausible angle here doesn't mean it's not marketing, but it might mean that it is marketing and that it's working on you.
> you seem to adamantly disagree
Such strong wording, you got that from me saying "They don't have to do marketing"?
> where did you get this impression of [...] "astronomical salaries"?
From the real world fact that the base salary is US$300k/yr, and often said that the total compensation for the first year ends up being US$700k+. Even for Manhattan + SWE jobs, surely that merits "astronomical". Other devs making that much are usually working at Anthropic or OpenAI, aren't they?
> where did you get this impression of "Very High IQ"?
You didn't copy my ™ there, which was meant to be snarky.
In any case, what I was trying to say is that they don't have to do marketing in the usual sense. They don't have to get the Jane Street logo in your subconscious so you buy a Jane St. product when you go to Kroger's, and you can't sign a Jane St. subscription plan.
People in this thread seem to be implying that the company is botting/astroturfing, but for what? To maintain the image that they're a highly sought after employer?
Anything Paul Graham posts gets upvoted here instantly, because it's Paul Graham who posted it. For Jane Street there's the same halo effect, because many believe Jane Street's work is impressive. Is it really easier to believe the idea of botting to get an even longer list of candidates to interview, than it is to believe people just like to hear what this company has to say?
I don't ever see anyone saying Paul Graham is botting posts about his "essays"
> We know this sounds vague,
ok thanks for this slop then. Dont see why this stupid article is on frontpage.
They seem to belive in 'organic' marketing like doing a fake podcasts on dwarkesh. So maybe they have some bots upvoting their name.
Pretty vague blogpost tbh. Apart from the fact that it's changing, it doesn't really describe much
I had lunch with a couple friends this week, and we each work in different tech environments. Most of the conversation was about the workflows we're each using, and seeing our colleagues use.
It's a really interesting time to work in this field. People who are paying attention to workflows are using a range of tools, and constantly refining their own workflow. For example a whole team might use Claude, but one person collects data about their own usage on an ongoing basis and develops more advanced workflows than everyone else on the team. Their focus isn't just about working faster, but about doing better work.
I don't think the post is vague; I think we're at a time where you need flexibility in your interview process to find candidates who are doing the best work these days.
Thanks, that certainly clears up the doubts
This sort of announcement has made me very happy I left the software world. I still code as part of my job, but it's not my main part and luckily I don't have to use LLMs are part of my job. I still code 100% by hand.
In some ways, although I detest AI, it has clarified the sort of people and organizations I want to associate with. Any latent characteristics that I abhor have been made obvious with the existence of AI and of course, I would want nothing to do with Jane Street.
I see much buzz lately about Jane Street. Probably they had a good year and now they’re looking for fresh meat, trying to expand even more.
What's up with the obsession with Jane Street today ?
Money.