The source article in Korean actually seems to tell a somewhat different story, that it's been a massive productivity boost in their early experiments. You could focus on the points of friction, but that isn't the main takeaway I think.
I understand that they used Claude to create some sort of verification code and test environments from circuit-design information, verification programs, and protocol specifications. But isn't precisely this quite risky due to the statitical nature of LLMs? I would have thought that they rather operate with very rigid test suites based on such software like F*, Coq, and whatnot.
Can someone with understanding of the process chime in here?
Here is the text snippet from the original korean article. Indeed the information was misrepresented:
"Customer-specific SoC verification work, which Usually takes Over a month, was completed in just two days. A second-year employee completed a USB model development that previously takes a month in a single day".
From what's described in the article they are using it to write the process code. That is, control code for setting the states of the SOC under test and the test equipment, then recording the data that the test equipment would measure. It can be a pain to do because each SOC and piece of test equipment might have it's own control language, drivers, different data formats, etc. This usually all has to tie back in to a local (self hosting is the norm) cloud service for quality tracking and some 3rd party enterprise software from companies like Cadence. The test equipment would be something like a Keithley Parametric wafer tester, which is actually an entire rack of equipment glued together by 3rd party code with semi-open APIs. It's not uncommon to make thousands of measurements on an SOC during verification and then process engineers work to reduce the measurement count to just the critical ones for manufacturing.
Hey, thank you so much for jumping in and for your elaboration.
> SOC and piece of test equipment might have it's own control language
OK, so apart from code generation the LLMs are providing language integration and data integration.
But how would you control quality, or make sure the LLms does not create some unwordly control code? I am trying to understand if they are using type checking or any other methodology that keeps the LLM generated code in check.
I am not from the hardware field, but I do understand that we are trying to build deterministic mashines. It is quite mindboggling that we are using statistical models for building the test suite, instead of, say, permutating through all critical input states and testing the output parameters for it.
I am asking because this isnt certainly the only mode of operation for LLMs in testing and quality control, and I would love to get a better understanding how to build such a system.
AI's can write deterministic code and have been doing so for a while. Humans do this as well. We write deterministic code with a goal, and generally it works out okay.
If you say that you can't trust AI because LLM's are statistical models, that's a true statement. Though it also turns out that we don't trust humans to write code correct either, which is why we have code reviews.
I was not trying to say that statistical can't generate deterministic code - I was wondering what tools I could use to make sure that they catch all the invariants when given a specification of inputs, logic layouts, and outputs. Certainly, feedback loops have to be involved - e.g. look at the lifetimes guarantees the Rust compiler gives - to make sure the answer is refined in iterative steps.
> Claude built a virtual test environment with placeholder blocks standing in for the missing RTL and started catching errors before the real design was even finished. Reportedly, this task would normally take over a month, but Claude managed to finish it in two days.
Seems a lot went quite right, traditional 30 day task done in 2 days
The source article in Korean actually seems to tell a somewhat different story, that it's been a massive productivity boost in their early experiments. You could focus on the points of friction, but that isn't the main takeaway I think.
I understand that they used Claude to create some sort of verification code and test environments from circuit-design information, verification programs, and protocol specifications. But isn't precisely this quite risky due to the statitical nature of LLMs? I would have thought that they rather operate with very rigid test suites based on such software like F*, Coq, and whatnot.
Can someone with understanding of the process chime in here?
Here is the text snippet from the original korean article. Indeed the information was misrepresented:
"Customer-specific SoC verification work, which Usually takes Over a month, was completed in just two days. A second-year employee completed a USB model development that previously takes a month in a single day".
From what's described in the article they are using it to write the process code. That is, control code for setting the states of the SOC under test and the test equipment, then recording the data that the test equipment would measure. It can be a pain to do because each SOC and piece of test equipment might have it's own control language, drivers, different data formats, etc. This usually all has to tie back in to a local (self hosting is the norm) cloud service for quality tracking and some 3rd party enterprise software from companies like Cadence. The test equipment would be something like a Keithley Parametric wafer tester, which is actually an entire rack of equipment glued together by 3rd party code with semi-open APIs. It's not uncommon to make thousands of measurements on an SOC during verification and then process engineers work to reduce the measurement count to just the critical ones for manufacturing.
Hey, thank you so much for jumping in and for your elaboration.
> SOC and piece of test equipment might have it's own control language
OK, so apart from code generation the LLMs are providing language integration and data integration.
But how would you control quality, or make sure the LLms does not create some unwordly control code? I am trying to understand if they are using type checking or any other methodology that keeps the LLM generated code in check.
I am not from the hardware field, but I do understand that we are trying to build deterministic mashines. It is quite mindboggling that we are using statistical models for building the test suite, instead of, say, permutating through all critical input states and testing the output parameters for it.
I am asking because this isnt certainly the only mode of operation for LLMs in testing and quality control, and I would love to get a better understanding how to build such a system.
AI's can write deterministic code and have been doing so for a while. Humans do this as well. We write deterministic code with a goal, and generally it works out okay.
If you say that you can't trust AI because LLM's are statistical models, that's a true statement. Though it also turns out that we don't trust humans to write code correct either, which is why we have code reviews.
I was not trying to say that statistical can't generate deterministic code - I was wondering what tools I could use to make sure that they catch all the invariants when given a specification of inputs, logic layouts, and outputs. Certainly, feedback loops have to be involved - e.g. look at the lifetimes guarantees the Rust compiler gives - to make sure the answer is refined in iterative steps.
Yes, that was my takeaway as well from the source article. <https://news.ycombinator.com/item?id=49274393>
Feels like a disingenuous heading for the article.
All of the problems they listed seem quite solvable. And it seems like it's saving a lot of time.
But saying AI is doing good doesn't generate the same number of clicks!
AI makes a great shotgun; occasional sniper.
But it is not a divine shotgun sniper rifle hybrid.
What could possibly go wrong?
> Claude built a virtual test environment with placeholder blocks standing in for the missing RTL and started catching errors before the real design was even finished. Reportedly, this task would normally take over a month, but Claude managed to finish it in two days.
Seems a lot went quite right, traditional 30 day task done in 2 days
[flagged]