My 4xe died in my driveway on Saturday after the update. Let me explain, from the perspective of a 4xe owner, how bad the response has been from Jeep/Stellantis:
- As of Monday 8am ET, zero legitimate communication from any Jeep-related accounts on any social media platform, or any other form of acknowledgement from the company (unless I've missed something?)
- I only found out about the issue after finally searching a few Jeep groups on Facebook (of all places) to see if anyone else was experiencing the weird failure mode I was after the update.
- The only remotely-official info was from a 'JeepCares' account (which is ran by Jeep) on some random off-roading forum? We were seriously all living off of screenshots from this forum, and the advice coming from the JeepCares accounts was contradictory: they claimed that the Uconnect update was separate from the telematics update, and that there was no way to stop the telematics update if the vehicle received it. Later they gave advice to defer the Uconnect update, making it sound like they were coupled.
- Due to the lack of info from Jeep, people were coming up with all kinds of "if you reboot Uconnect while the Jeep's in ACC mode, it clears the check engine light". This probably did clear the CEL but didn't fix the fault.
- There is no way to tell if you received the bad update.
- There is no way to tell if you received the 'fix' either.
- Dealerships have literally no idea what is going on.
- You're basically at risk of your Jeep going limp (power loss, unable to safely make it to the shoulder) and being stranded on the highway, even as I write this.
> - You're basically at risk of your Jeep going limp (power loss, unable to safely make it to the shoulder) and being stranded on the highway, even as I write this.
This seems extraordinary.
I was going to ask: Are you really saying they kill the vehicle's power system, effictively the engine, while the vehicle is being driven on the highway?
But no need to ask, the article says yes, that's what is reported:
> Instead, the failure appears to occur while driving—a far more serious problem. For some, this happened close to home and at low speed, but others claim to have experienced a powertrain failure at highway speeds.
Ya, that is shockingly scary. It makes me think we need some new standards about software updates to vehicles in general (or perhaps these already exist but were missed for some reason?). I can totally imagine that software used to be this ancillary selling point that didn't need such tight regulation, but as it becomes core infrastructure for the vehicle this is less of an IoT toy, and deserves stricter standards.
How about: you get to say whether you want to update and when and manufacturers are required to very explicitly list all of the changes in an update? That would seem to be an acceptable minimum.
I don't think that Jeep would have sent out a message saying that one of the changes would brick your machine.
It seems that the ability to trivially roll back any update would be a better choice, at least for this. (But I'm sure there are downstream effects I haven't thought about if that were implemented.)
How do you roll back a fatal car accident caused by the faulty update?
Giving user’s control over when the update runs allows them to be in a safe and secure setting when that update happens. Allowing them time, gives them and Jeep the ability to slow roll the update so they can halt it if initial feedback is negative.
I say this as a Mac user who does not allow auto updates for MacOS. I wait a week or so until the chatter validates it as non-breaking. They pushed an OS update several years ago that broke a few things I rely on. So I don’t trust them now, but these things just happen on OS’s with third party software. I expect it. But, I also don’t want to be forced to deal with the headaches immediately. I’d rather let the third parties run updates and advise how to deal, before I have to dive into fixing things. With car firmware, there’s really no excuse for this except poor engineering / processes.
> Giving user’s control over when the update runs allows them to be in a safe and secure setting when that update happens. Allowing them time, gives them and Jeep the ability to slow roll the update so they can halt it if initial feedback is negative.
This does not fix any QA process that is broken. And frankly you should not need to update any control unit firmware after it is sold. The fact that they're even doing this is broken.
Unless your Mac is somehow attached to 5000 pounds of metal going 65 on the highway, the same standards should probably not apply.
Good points, I did miss those. However, if I had this vehicle and I was reading this article today - and had the ability I'm asking for - I would just keep my current version running until they figure this mess out. It's the advantage of letting other people run the updates first, you get to hear about issues before you experience them.
Many security compliances require auto-updates to be on. It's thought of to be a lesser evil, because many (most) users never update their OS/browsers, which is worse.
It's not perfect but seems reasonably easy to implement and would certainly help. If the user needs to approve each update and can see what the changes are most updates will either be skipped or delayed long enough that catastrophic bugs will only hit the small subset of cars that update immediately.
I would bet most updates, especially from a company this bumbling, will be more along the lines of increasing telemetry or pointless UI changes than releasing actually useful features and bug fixes.
The current regulations are written for a time where cars didn't have rolling computers in them. And even then, the regulations don't account for Tesla-style linked systems. So I say we do need new regulations.
NHTSA's power is simultaneously very broad and narrow. They're empowered to investigate potential safety issues after the fact, but this may not be a safety issue in the very pedantic sense often used. NHTSA can proactively set standards, but the standards they've set (FMVSS) largely ignore modern electronics. So on and so forth.
>> - You're basically at risk of your Jeep going limp
This has happened to me twice with a Nissan Leaf. I paid money to get a read out from the computer system, and there were no timestamps on the screens of data.
Modern cars "computers on wheels" are dreadful.
Is it possible to disconnect the power from the radios used for "over the air" nonsense? Then at least they would be stable.
In the Leaf, you can disconnect the TCU from the CAN gateway controller located behind the infotainment system to disable its remote connections.
It will throw a perpetual "check engine" light and disable the hands-free microphone, but OVMS users have made a "dummy TCU" that gets around that annoyance.
I have the opposite problem. The specific infotainment system update I need requires a $200 visit to a dealership with a specific model of a USB 2.0 SanDisk Flash Drive (NI-52727-1). Not available OTA despite the Leaf's OTA capabilities.
Is it possible to disconnect the power from the radios used for "over the air" nonsense? Then at least they would be stable.
I've read online that for some cars, you have to dig deep inside to disconnect the cellular antenna.
I'm a little more lucky. On my car, you can pop out the SIM card from a slot in the ceiling, behind the rear-view mirror.
This assumes you haven't given your car access to your home WiFi. (Some people do this so they don't have to pay for a data plan for their car, and it kinda sorta "syncs" when you get home.)
It is completely insane that this is happening. I did DD on a company in the automotive space a couple of years ago and flagged that they did not check if the vehicle was stationary, motor disabled before updating. They were all surprised at how I thought that this could possibly ever lead to issues.
I have Java code running on commercial aircraft. You can’t actually run Java code on commercial aircraft because the FAA doesn’t (or at least not at the time) know how to certify it.
The entire box it’s on isn’t powered while the plane is in motion (“wheels on ground”). It’s shut off before preflight and doesn’t turn back on until the plane is on the ground. The service my code is part of is responsible for queuing updates and downlinking telemetry. Updates are manual and obviously you can’t run them while in motion if the box they are on doesn’t even have power.
Cars probably don’t have to go this far, but there’s a continuum and they’re clearly in the wrong part.
Even iPhones and windows let you schedule update times. Just the fact that a freaking MOVING MACHINE doesn’t is egregious on itself. Imagine if stellantis would manufacture industrial equipment or nuclear reactors!
Given the state of the software industry, it's honestly more surprising that this doesn't happen more often. Our industry is a complete joke, and somehow we've been given responsibility over people's lives.
> Given the state of the software industry, it's honestly more surprising that this doesn't happen more often.
It probably does. We just don't notice.
> Our industry is a complete joke, and somehow we've been given responsibility over people's lives.
Amen to that. kqr made some choice comments the other day in that thread about the airliner that came to within a hair of crashing due to running out of fuel. Thinking about risk is not a skill that we're born with and it is always sobering to read the 'risks digest' for a bit and to see how thin the ice really is.
We are really only about 60 years old as a proper profession, and we seem to be trailing behind doctors for professionalism and standard of care by about 100 years.
I don’t know what will turn out to be our penicillin, or our Joseph Lister, but in 1960 the former is something that didn’t exist when older doctors were in school, and latter had only been dead for fifty years. It may not have happened for us yet.
On the topic of professions: Joseph Lister was a surgeon. Modern surgery (which I define as surgery aided by anesthesia) is a relatively recent discipline dating to the early 19th century. The introduction of anesthesia made lengthy and intricate operations possible but also ushered in novel problems and complications. Surgery as a field had to learn tough lessons over time.
He was known more for antiseptics but the biggest surgery moment for me will always be “using soap” and I wonder what the software equivalent is.
Like I said we are still young, so it feels sort of arrogant saying we have figured something out when I know how many things are industry standard now that almost resulted in shouting matches trying to get done even 20 years ago. Maybe our soap moment is coming up ten years from now.
But I suspect automated testing may be the wash your hands, because it represents a sort of hygiene that “we” used to just say fuck it or make a minimal effort.
I’m going on a limb here because i’m not directly on the software industry but my first suspect would be metrics and the fact that you have to deliver a product at certain time “no matter what”.
According to the article, that's not what is happening. The update itself completes fine; it's the updated firmware that is buggy, and seems to cause/require a reset of the ECU while in operation. Not that that makes it any less insane, but the update process does not seem to be implicated here.
they did not check if the vehicle was stationary, motor disabled before updating. They were all surprised at how I thought that this could possibly ever lead to issues.
My anecdata is that my car won't update its software without the owner explicitly requesting it. And then, it will only do it if the car has something like 50% charge, hasn't been used for an hour, and nobody is inside.
I once tried to do the update while I was inside, and it refused.
My BYD wants the battery over some percentage, the vehicle in park, and the hood closed. The hood one was surprising, I wonder if it's for the safety of the car or of anyone working on it.
This sounds like the sort of thing that happens when a team has a deliverable that slips multiple times but everyone had vacations planned for a responsible amount of time after the deadline.
Under time pressure and confirmation bias they signed off on code that was giving off signs of being broken, pushed it, and now key staff are either on airplanes, out of coverage on their phones, or cannot work entirely from memory and don’t have their computers with them because vacation.
I bet they already have communications ready, but everyone Director and above needs to "wordsmith" it to deflect blame and make sure nobody looks bad, and all the lawyers need to bless it so that it's not admitting anything. I bet you there's a Word doc being frantically E-mailed around all day, needing to be approved by 25 people before it reaches the light of day.
How did you verify that a software update can 1. Occur during driving operation of the vehicle and 2. results in vehicle power loss?
I worked in an auto supplier years ago and there where several protections in place to prevent the risk of update corruption on safety related components. One of the simplest one the UDS programming session having entry protections related to vehicle speed, vehicle driving mode, etc.
Personally, I would return the vehicle as defective after an issue like this. No way I'm going to trust the lives of family, friends and myself to a company like this.
I really want a 4XE Wrangler to replace my aging JK at some point, on paper its amazing, lots of torque and power, decent economy for a brick with large tires, and can actually run a pure electric enough to get around town, plus still takes all the standard Wrangler parts.
However in classic Jeep style they just can't get reliability down, and the PHEV part seems too complicated for them.
If it was just reliable it would still be the best selling PHEV in America, they let that go.
There is no sign of the 2026 Wrangler 4XE it might be canceled like the Gladiator version...
My Wife just got one last year, the electric engine will give you about 17 miles on a full charge that takes about 7 hrs, its pretty much useless on full electric mode, pretty good with the milage on hybrid which is what this really is. And yes do not expect reliability, first day out of the dealership we got a check engine light, apparently one of the workers at the dealership forgot to attach a wire or hose, at least thats the explanation that we got, i told my wife to insist on getting it documented somehow or get a log from the dealer but of course they played it off as human error (total BS). Few months later and her screen started to turn off and on, or just die completely when air play was connected. Stay away from these cars.
It's probably canceled, not only for reliability reasons. EVs and plugin hybrids are probably doomed, at least in the US: The EV tax credit subsidy is gone, fuel economy requirements will be or already have been eliminated, electricity prices climbing (my rate increased almost 50% on my most recent bill) and the Trump administration is extremely hostile towards anything related to renewable energy.
My Wrangler JK is 10 years old with almost no issues, its a relatively simple vehicle compared to most, easy to work on as well. The Pentastar V6 is well proven engine and the issues with it are well understood.
I really like the vehicle, it has served me well and taken me many interesting places across the country, along with daily driving. I tow it with a RV and its one the few that can do so now days, plus its extremely capable offroad.
The 4XE is very alluring, much more fun to drive (I have rented one) and I could charge it off home solar and still tow with RV (the only PHEV thats possible to do so). If only it was reliable...
There’s just no comparison when you consult reliability reports, though. Of course, even companies like Toyota are going in the direction of always on, arbitrarily updated software and subscriptions.
I have no idea what I’ll buy when my 11 year old toyota finally retires.
imho the occasional mixup is going to happen, and eventually it'll be big like this or like Crowdstrike, but pushing these out on Fridays means the critical staff isn't there to help. The people who could have communicated this stuff to customers and dealerships were in bed when people got into their jeeps at 6am on Saturday after an overnight update.
I believe crowdstrike's update was on a Friday night as well.
Unless its a serious security bug, it can wait for not only for better QA testing but also for next Tuesday. Read-on Fridays need to be an industry-wide thing.
To me the bigger issue is that the infotainment system can affect the core function of the vehicle. That seems completely unacceptable, regardless of when an update is pushed out. The two systems should be separate with the infotainment system as the lesser important and unable to actually effect the core system.
Honestly the only thing that's going to change this is criminal liability for safety related software bugs. Otherwise, it's just business as usual and the business for the last 15 years has been cutting QA and asking developers to do testing themselves, which is literally impossible for a lot of software due to lack of proper staging environments and large permutations of configurations.
I think the nature of capitalism will make this impossible. The capital owning class will not allow criminal action for this and will also fight any common-sense regulations. If the working class gets that regulation in via the democratic process, that's fine, but its unlikely especially since it hasn't happened yet since we've gone digital on near everything starting from the 70s and 80s.
The working class lately seems more focused on 'culture war' issues and not economic or material or consumer or worker's rights issues anyway, so we're probably as far from any kind of regulation reform in software as possible. I remember a couple decades ago FOSS as an ideal seemed stronger and you had people like Lessig pushing hard for IP reform and Swartz and others for 'information must be free' honest-to-goodness mainstream movements and all of that seems to have went nowhere and is somewhat to very unpopular today. When was the last time you saw a populist movement towards liberal tech reform like this? Outside of some edge cases like medicine or power generation, the regulations here are purposely kept weak because that's what the wealthy desire.
Maybe our kids or grandkids will have this after the pendulum swings back, shrug.
Aviation relators just allowed boeing to self certificate again. Avition has a lot of historical regulations and can work through sheer inertia for years.
But the whole point is, regulator wont have teeth. They have teeth when politicians back them. And as of now, politicians back billionaires and deregulation. Wall street hates regulators, billionaires hate regulators and sizable part of population prefers people dying if it means they cam hurt libs and ennemies.
I recently worked at a big home lighting company, working on the OS of the router device that communicates with the light bulbs themselves and the internet/user.
Our OTAU architecture uses A/B system updates [1]. Core idea is that both the kernel and the rootfs (read-only) partitions had 2 different bootslots in storage, and the OTAU would only write to the bootslot that is unused. Hence, if something went wrong, the system would automatically fallback to the previous version by just switching the bootslot used. Over the numerous years that that architecture was used, I couldn't find a single post-mortem that resulted in devices being bricked. Something to note is that the rootfs partition was overlaid with a writable partition for persisting state data etc.
Now that was a $two-figure USD device, not a $5/6-figure USD electric SUV. Is this a cost-cutting measure? At those price levels, doubling your NAND size is not even half of a percent of the total cost of the vehicle.
Unless there was a serious issue that the used bootslot corrupted the unused bootslot, then I don't see how this could have happened.
It's saddening that car manufacturers are so unserious about the code they're deploying.
I've worked in both IoT lighting and automotive, so I'm comfortable comparing the two. This also isn't offered as a defense.
The big auto OEMs are just as sensitive to absolute BOM cost optimization, regardless of the percentage increases. I don't think this was a bootslot issue though, regardless of the word "bricked". Even as backwards and ill-advised as auto software can be, generally accepted practice is that updates are impossible while the vehicle is in motion. This is usually enforced by systems shared across multiple OEMs through the tier system.
The situation sounds more like a disastrously buggy new firmware.
I wouldn't put either past stellantis though. The auto industry already scrapes the bottom of the proverbial barrel sometimes, and stellantis isn't exactly known for their top of market compensation.
This is generally how other devices work as well - for example all Android devices and Android-derivatives (which many of these cars are!) use a similar A/B partition to prevent bricking.
It definitely reduces the risk of updates, but it absolutely doesn't eliminate it.
The A/B updater itself is a surface area - especially if the logic is complex and there are other child devices that are updated at the same time (likely for cars). In that case you're not just coordinating between two independent partitions, you're coordinating between 2 * N partitions, half of which have dependencies on each other.
Also, the key bit of the mechanism is that upon successful boot the new partition is flagged as "good", which causes flags to be set to assert that the update was successful and the backup partition is no longer needed. That logic can (and does) fail - if your failure point occurs after this checkpoint you're hosed still because you're past the point of no return.
Making that worse is that in most systems you want the "it's all good" checkpoint to occur early - you don't want to, for example, wait multiple minutes for all user services to come up. But that also means that if a critical failure happens in said services, you're past the checkpoint.
> Now that was a $two-figure USD device, not a $5/6-figure USD electric SUV. Is this a cost-cutting measure? At those price levels, doubling your NAND size is not even half of a percent of the total cost of the vehicle.
Could just be a competence and priorities problem. If it's cost cutting, it feels way more likely that some PM cut some story from a sprint to hit a deadline (and objections were either not raised or ignored), than they did some engineering analysis and explicitly decided to save $3 per vehicle by cutting the NAND size.
Edit: Actually, I don't think that technique would have helped, the problem wasn't a botched update, but a seriously buggy one. From the OP:
> The buggy update doesn't appear to brick the car immediately. Instead, the failure appears to occur while driving—a far more serious problem.
> Edit: Actually, I don't think that technique would have helped, the problem wasn't a botched update, but a seriously buggy one. From the OP:
That and combined with general refusal of new automotive bootloaders to downgrade. You can go only up in versioning. So even that you could have working version on second partition, it will never get loaded because it has lower version than currently one you are running.
1) Total cost of the vehicle does not matter. What does matter is the operating margin. Half a percent of the total cost of the vehicle will move them from 2% margin to 1.5% margin. (Ford has operating margin of 2% as an example)
In other words an increase in 0.5% cost of total vehicle will reduce their profits by 25%.
That’s a huge number now! Note also that car manufacturers are in a bad spot because their volumes are fairly low (smartphone = 1M/yr, car = 40k/yr) and have harsher requirements for chips, driving the cost way up.
2)AB updates are great, but they can still fail or get soft locked. Especially important around code when you configure the slot to be good and when bad.
I have heard anecdotally that auto manufacturers are sensitive to a price change less than $5/vehicle. This is better than some industries that are sensitive to $1.
What could easily have happened is that the negotiators didn't include A/B updates in their spec, or they only specced A/B updates at 1GB OTA size.
They do their usual hammering on price, and the head unit or ECU manufacturer gave them some savings by cutting storage space to the bone.
Maybe it was still enough for A/B updates, until the usual software bloat took the updates past the critical limit.
They could still do a safe update by doing an A/B/A update (where B is a shrunken, update-only OS), but that requires development time, and the engineers should already be working on the next vehicle.
Worked for them. Corporations with many brands in their portfolio might discuss for weeks over price differences of components of 0.20 Euro. That‘s twenty Euro cents difference for e.g. a USB connector. If you expect that a vehicle platform sells in the 10s of millions over its lifetime, you‘re talking real money very quick!
yet another example of the flawed logic where "we don't have time/money to do it right now, yet we always seem to find the time/money to redo it later after the shit hits the fan"
> the system would automatically fallback to the previous version by just switching the bootslot used.
That's the hard part though.
It's shockingly common in my experience to have an A/B boot setup, but no actual logic in the userspace application to switch back to the other partition if something goes wrong. It's just a defense against somebody pulling the plug during the OTA, it doesn't protect against software bugs at all.
We used to do that with device that where in difficult to reach places with harsh uptime requirement! Think industrial routers and protocol converters. I think it pays for itself very quickly. Sending someone for such a device can get expensive.
Well, on the positive side, at least they were stationary unlike these vehicles. Don't get me started on botched OTA updates, there are so many ways companies get those wrong it's not even funny.
I once managed to brick a PC motherboard that advertised "dual BIOS". It didn't fallback to the previous version after a botched BIOS update.
It's totally possible that the update corrupted the other bootslot as well. If those blocks aren't off-limits to the updater program, it's just an off-by-one error waiting to happen. Slot 0 or slot 1?
Another possibility is that the updated version booted up just enough not to trigger the automatic fallback, and then got stuck in a loop.
I’m sorry, but you’re incorrect the vehicle completely shutting down while driving and not working again until you put it into park and then it’s shutting down five minutes later is effectively bricked and extremely dangerous. Myself and my family almost died just trying to get home from dinner. It was a complete loss of propulsion and power steering.
Being temporarily unusable is not how I've seen "bricked" used, bricked means unrecoverable and the item is completely unusable except for as a brick/paperweight/door stop.
Then it would better be described as a life-threatening event rather than a bricking - especially as, in the hierarchy of concerns, the former is more serious than the latter.
And then it was fixed with another OTA, so it was not bricked. Why bring up this pedantic point you may ask? Because the grandparent raises a scenario that doesn't apply here. A/B updates or not were not at all the issue here.
I for one am always grateful when things are engineered thoughtfully and with redundancy as it is symbolic of respect for the people who are your customers. Especially in something as important as a car, "can be bootstrapped from a central computer" - when? how easily? how reliably? - is not good enough because things will inevitably go wrong for some portion of the user base.
Brick is now slang for a lot of fail conditions that aren't classically 'bricked.' This has become really common I've noticed. Honestly, this ship has sailed and isn't even worth fighting anymore. Its like Xerox asking people to stop calling copies Xeroxes.
We just never bothered to develop a new term. Maybe 'soft-bricked?' 'Semi-bricked?' I would like journalists at least to start using more accurate terms, but 'bricked' I imagine gets a lot more engagement and ad impressions, so here we are.
Wikipedia has a section about this. They call it soft bricked vs hard bricked, according to the difficulty of restoring device function and how the broken state presents. Even hard bricked is usually recoverable with appropriate tools, so it is a spectrum.
All those words you are saying, it's quite possible the sub-contractor to the sub-contractor to the sub-contractor in a foreign low-cost country that actually did the work has absolutely no idea what any of that means, and they are doing the bare minimum to deliver
Why wouldnt a foreigner know what this means? This seems very xenophobic. And if US/Euro management is hiring these groups and not giving them requirements for redundancy then guess who is at fault? Not the contractor.
I rented a Jeep Wagoneer recently and found it to have such comically glitchy electronics that this comes as no surprise. The second day we had it, the liftgate stopped latching entirely, it beeped and popped some error messages on the dashboard and simply wouldn't latch shut at all, no matter what we did. Searching the internet produced lots of people with the same problem, reporting that it required a software update to fix. There was no manual override to the electronic latching mechanism.
Luckily we were near a location of the rental car company—rather than deep in the middle of nowhere where we were headed—and exchanged it for another of the same model, which was all they had available. The next 1000-something miles we drove were filled with endless weird glitches:
- When a passenger plugged in their Steam Deck in the back, the entire infotainment system cut out and went black, including the instrument panel, and then started glitching in and out until they unplugged it.
- When parking, the driver's seat would retract slightly to make it easier to get out, but it never moved forward again, so the seat would get further back at each stop until it was manually repositioned.
- The entire drive the system flashed an un-dismissable error about a rear seat latch, which seemed completely functional.
- The TPMS light went on and off periodically as it seemingly lost and then regained signal from one wheel or another.
- The system flashed errors related to the automated cruise control being unavailable/broken at random times.
- The electronic parking brake kept applying itself while briefly paused in parking lots.
- There was something inscrutably wrong with the climate control that we never really figured out where sometimes it'd just get hot inside the car despite no change to the AC settings.
When we got back I found tons of people online talking about similar (often worse) issues. Incredibly terrible for any new vehicle, never mind one that costs $80k.
Honestly, unsurprising. Jeep and Stellantis/Dodge in general has horrible quality control and extremely poor electrical designs. They have a huge enthusiast community that will be happily apologize away the copious amounts of flaws. Frankly, nobody should ever buy their vehicles, it's just robbing yourself.
I own a 2002 Grand Cherokee which sometimes will have a 10A+ power drain for no apparent reason. Of course it doesn’t do it when I’ve got my voltmeter on it, except once (when the 10A fuse in my Fluke blew). I resigned myself to unplugging the battery or leaving it plugged in to a high current battery charger at home, and leave it running if I drive it somewhere.
I rented a Jeep Liberty or Compass circa 2018 whose headlights were permanently in DRL mode: couldn’t turn them off or on. Fortunately I didn’t need to drive at night.
In 2017, rented a 300 with 500 miles on it; the infotainment was completely broken, which hosted the controls for the seat heaters and temperature setting. It was well below zero in Minneapolis but we had to drive around with our windows down because the fancy climate system defaulted to max heat blast + max heated seats based on ambient temperature.
Long ago I had a 1996 Neon where the wiring harness started to fail, and the speedometer would stop working. Later on the oil light would come on despite oil pressure being fine. Eventually the entire car just quit running at all at random - nothing but a dim oil light. I sold the car for scrap for $65 since I got tired of being randomly stranded.
So what I’m saying is that it sounds like Chrysler has managed to actually keep doing the same thing for 29 years: electrically unreliable vehicles.
In my personal experience with cars that had strange electrical problems, they tend to be on a bad ground somewhere in the loop. I once took a Chevy S-10 to a place my dad recommended. A guy walked out to ask what the issue was, he nodded, took a step back to look at the truck and asked the year of the truck. He then nodded and said "Yep", and then without looking reached under the dash on the driver's side and tightened a screw by hand. All electrical problems went away. He walked away after politely telling me to have a nice day. I was baffled, and he said it would cost him more in time to write the repair up than he could honestly charge me.
The point is that stable ground connections are notoriously hard on something that by design shakes, rattles, and rolls with all of the vibrating and bouncing on our "modern" streets. It's also a very easy thing to misdiagnose unless you're a mechanic that specializes in automotive electrical systems. It also takes time for new year models to display their warts enough that non-dealer mechanics gain experience repairing them.
Yes ... However. Most car manufacturers manage to deal with this without it becoming too common, with standard engineering controls ( proper fasteners, torque specifications, QC etc).
The neon issue was the engine harness fraying to the PCM and eventually burning out either that pin on the PCM or cable grounded itself.
Back in the day I was buying these, around 2005' or so, for $300-400 non stop and repairing that, the dash that cracked and misc cosmetics.
They were great cars, the R/T model in manual was fantastic in gas, reliability and safety (sadly crashed it.) but boy was 16-20yr old me happy with these neons. Can't believe they sold shy of $9,999 when new (for base of course)
Just reading your post took me back 2 decades, wow.
I own a Jeep Wrangler, and you're right the electronics are terrible. The rest of the vehicle is really solid though. The only problems I've had with it in three years are electronic in nature. And I've really pushed it to the limits: Colorado Passes, Utah Dessert, Montana backroads. I drove it to the Arctic Ocean and back on the Dempster.
Still there is no excuse for how terrible the electronics are in Jeep / Dodge (I'm assuming all Chrysler) vehicles. And it's been that way for decades.
I owned a Jeep 4XE, and I was glad the day we sold it, and I'm doubly glad today. The electronics and software were crap, and the powertrain was simply insufficient. At one point, they issued a notice that amounted to 'it might catch on fire, keep it away from your house.'
Yeah, I have family members with 2 JKs and a JL, unfortunately all plagued with issues, almost entirely related to the electronics. A Jeep Wrangler is a vehicle that sounds great on paper, but actually owning one is an exercise in frustration unless you just enjoy fucking with wiring harnesses. I am sure many others will come out of the woodwork to say that Jeeps are great, unfortunately they are not.
It’s too bad because the wagoneer is the best designed car in the segment, inside and out for the most part.
I have a somewhat bad back and want something that I can occasionally work from, so a big space, comfy middle seats, a wide center console. Car makers for some reason refuse to make essentially a Tahoe but shorter wheelbase / 2 row which would be ideal. Instead you have to go with the full size to get full-width.
But out of those, only American brands seem to understand the utility of blocky interiors. Armada and all the Japanese and Korean large SUVs always use swooping rounded edges which really reduce utility.
But the American brands are all less reliable and struggle with consistent quality.
This is disturbing. There is absolutely no valid reason to make OTA updates to your ECU or other essential components. Never.
Ever. If there ever was a need to do such an update, it should be made in the dealership by a professional who can roll it back if there’s a problem. Automakers have become far too complacent while at the same time over-automating everything and profiteering by turning things into a subscription model. This is why so many car enthusiasts (myself included) prefer older cars.
I've had a Jeep for a few months, and it bothers me so much that the entire community is about modifying the vehicle as much as possible, but they still come with this locked down OS.
If any car could be the champion of OpenSource, it is a Jeep Wrangler, but they're using an OS made by SiriusXM for some reason.
Isn't Jeep the company that introduced infotainment pop-up ads that play at the stop light? I doubt they'd want you loading an open source OS replacement to avoid that.
You're starting out with an assumption, that this is an OTA update for the infotainment system, and then conclude this incident shouldn't be possible. The problem is the assumption.
This is a OTA vehicle update. It has the ability to update the infotainment, ECU, ECM, TCM, and BCM. Multiple manufacturers have been able to release recalls that fix major vehicle defects (safety, reliability, and performance). That wouldn't be possible without OTA updates that update core vehicle computer systems.
Unclear where this idea that OTA = Infotainment came from. I'd go as far as to say that most manufacturers can do this in 2025.
> Unclear where this idea that OTA = Infotainment came from.
Because to some people, the idea of an OTA update being allowed to change mission critical parts of a machine automatically without a solid rollback system is absurd, and the best way to do that is to never do OTA updates of mission critical parts at all.
This is why OTA updates should simply be illegal/considered negligent engineering. If you want a convenient update, let people plug their phones or computers in via a USB port or something, or take it to a mechanic to do so. There shouldn't be security concerns with an appliance because it shouldn't be writable outside of an owner-intended maintenance mode, which should be impossible to activate wirelessly.
Wait until when fridge or TV will come with its own 5G chip and they will get bricked by remote update because it is time to buy a new one and there will be nothing you can do about it.
They don't have to do this. The cheap materials in the compressor or cheap capacitors used on the power supply board will just silently fall. And the cost to repair the problem, for the average person, will be slightly less than just buying a newer version of the crap that just broke.
My LG refrigerator recently stopped cooling. The error code suggested it was the defrost mechanism. It was more of a hassle of a repair than I wanted to take on so I found an authorized repair shop on the LG site and opted for their flat rate repair.
First trip the repairman replaced all of the defroster parts and sensors. It failed again with the same code 18 hours later. The second time he replaced the main board and at least one other part. It now works great and I have effectively a new fridge aside from the compressor for less than $400.
Compressor still has three years of warranty left and we expect to move before then. It can (hopefully) be someone else's problem.
This should be made illegal. It’s a massive national security threat. Imagine on the eve of a war, instead of Jeep 4xes, it’s every recent Ford or Toyota or GM car, and instead of a software update that can be rolled back, it wipes the flash completely, or reprograms the ECU to damage the engine or disable the brakes on the highway or something else to cause accidents.
Just wait until you hear how much of our country's critical infrastructure is hooked up to the Internet. Traffic lights, water treatment plants, power plants.
The infotainment system can be the gateway to the rest of the vehicle network. It makes sense to attach a 4G modem to the display head to do mapping, hands free calling or emergency response, you may as well use it to download ECU updates too.
"I'd go as far as to say that most manufacturers can do this in 2025."
What does that have to do with OP's comment? And their point is still valid, and OTA update should not be able to brick a vehicle, regardless of the system receiving the update. And regardless if "they all can do it".
Any update can brick your device if done poorly. This device just happens to be a car.
You misunderstood what OP was saying. They claimed that an update to the infotainment system shouldn’t be able to brick the other systems in the car. The response points out the car’s OTA update subroutine has access to update every critical system in the car by design. It’s flawed logic to assume that OTA updates only affect the infotainment system.
If OTA updates can update core vehicle computer systems, in ways that can correct safety, performance, and reliability problems then they can also brick that vehicle.
The manufacturer has the ability to push an update that reprograms computers that control how physical components behave in a vehicle. By the very nature of that; they can push good or evil updates.
Which is a reason the market for "dumb" cars is tightening up. Both my cars are "smart" and sometimes I wonder if I really own them. It bothers me that the maker can cause an update without my permission. (Yes, I know that's the world we've been living in for a while now.)
most cars these days have GPS and return location and so on, why can't manufacturer run these updates only at night and when the car is parked at home? There should be no reason for any OTA update to happen while the vehicle is running (or on a trip etc.), downloading the OTA update, sure, but definitely not applying it. Also there should be a documented procedure to restore the previous in case an OTA update fails.
Because cost. Same reason why dash clusters and infotainment systems are now all monitors - its actually way cheaper to use those than analog gauges. The software is built on a famous bullshit paradigm of "never rewrite, always reuse", and as a result shit gets patched together without any concern of how everything cooperates.
Now with hybrid or electrical drives, a motor controller is basically a package that runs its own software, which then interfaces with the rest of the car. And OTA updates can overwrite its firmware.
The only manufacturer that has avoided most issues is Toyota, since they have been doing hybrids for quite some time. Other companies are just starting on this path and to catch up, they can't be bothered to do software deep dives and figure shit out.
As long as it exchanges information (mundane things like muting the music when parking sensors have to be heard, requesting battery/fuel status for advising the next fill stop etc) the isolation can't be entirely complete.
I wasn't really talking about galvanic isolation. And on modern vehicles instruments and infotainment tend to run hypervised on the same physical host. Thing is as long as you exchange information there's always potential for logic coupling allowing the trouble to cross the boundary. Not to mention the basic rate excess/denial of service situations.
I had an OTA update brick my Tahoe infotainment system. Now that backup cameras are standard requirements, those were all unusable. Also affected things like the clicking sound you hear when you use your turn signal. That was completely silent. Cost me ~$2k to get it fixed and wasn't covered under warranty. Good stuff. I've disable future "updates".
An FYI for the future, but backup cameras are considered a safety system and manufacturers are required to repair issues they've caused in safety systems regardless of warranty status. The appropriate escalation if the manufacturer doesn't recognize this is to get NHTSA involved with a safety complaint [0]. That's the main way recalls happen.
it clicks a relay. Just like with ICE vehicles people usually use it to warm up their car in the winter.
Also, batteries may need to be preconditioned if too hot or cold. A lot of EVs let you set your ideal departure time in a widget as opposed to using a remote though.
This is a 4xe. It is a gas jeep with an overpriced, undersized battery and motor bolted on.
It can be started just like all the other gas cars.
Although even with full EVs, there's a reasonable concept of a "start". Some even let you essentially unlock and allow driving remotely, even if the local driver doesn't have a key. That's useful sometimes.
The article doesn't go into a lot of details, but it only says that the bug happens while in motion, not that the software update itself happened while in motion:
> The buggy update doesn't appear to brick the car immediately. Instead, the failure appears to occur while driving—a far more serious problem.
The problem is worse than "just don't update on the fly while driving". The update happened while not driving; the bug causing the failure mode of shutting down power and engine happened later while driving. There's nowhere to hide from these types of problems it seems.
The infotainment system on my car can make changes to the suspension. Can change from street to track mode and even has a launch mode I can initiate for starting a timed 0-60, etc.
I can also put the car into valet mode so it won’t go fast. If I forget the valet mode password I am told I have to buy a very expensive replacement because it can’t be unlocked by a dealer.
I can't for the life of me understand why infotainment systems knock so many engineers for a loop. Is there a particular reason (industry/domain-specific) beyond just low-quality software development?
My Mazda 3 (2018) just had a class action lawsuit for its infotainment system which, completely at random after years of normal operation, starts clicking on menu items and scrolling about the settings (only to stop and not do it again for a couple of months). It can get so bad you just have to disconnect any devices and drive in silence/with the AM/FM radio.
it's the consequence of not vertically integrating, having 20000 different ECUs each for a specific purpose and trying to nickel-and-dime suppliers on all of them, and the lack of urgency that traditional manufacturers and tier-1 suppliers have for adopting modern software development practices
I recently did some testing of Bluetooth connecting an Android phone to a reasonably new Mercedes. Being a "luxury" brand, you'd think the Mercedes would have good infotainment software, but I found at least 5 bugs in the GUI in about 1 hour of testing, and I wasn't particularly looking for bugs, just testing what I thought would be the happy paths for Bluetooth connection. The Android phone, on the other hand, did its job flawlessly.
I know software and embedded systems well enough to know all of the issues I found were preventable, if anyone cared.
The car seems well built in many other respects. It doesn't look like the problem is engineering ability.
(See also: Set-top box GUIs that are painfully slow to render menus, scroll, search etc. on hardware that I know can render 10-100x faster when programmed to.)
As you alluded to, the answer is they don't care. Car companies look at software like it's any other line item on the BOM. Like a bolt or a gasket: Source it as cheaply as possible and spoon it onto the product somewhere on the assembly line. I see fit-and-polish mistakes all the time in car infotainment. Text that can't handle unicode, icons that are misaligned by 1 or more pixels, connections dropping and coming back, audio mixing problems, things drawing outside their "windows." Nobody cares--they get to the drop dead date when the software needs to be on the assembly line, hand it over, and start flashing devices.
I worked at a company that did software for these connected infotainment system. They cost cut those things to the bone, minimal ram, minimal cpu, shit screens. Even in the high end models.
Gotta remember that the car radio has always been a cheap gimme.
Gotta remember that the car radio has always been a cheap gimme.
Not really. Some of the hardware that you could get in the 1950s-1970s timeframe was great. Heavy chrome knobs and bezels, permeability-tuned front ends with separate RF stages... electromechanical mechanisms that seemed like witchcraft when I took them apart as a kid, and would still be cool to play with today.
I never heard of this and follow Tesla groups/communities/forums/etc. for over 10 years. At most you'll hear one or another person complaining about having initiated the update process and suddenly getting annoyed because they find out they need to go somewhere (it might take an hour).
I can't find any news stories about Tesla's OTAs bricking cars? Especially nothing mid-drive like we're seeing with Jeep. That'd be a pretty major story considering how every Tesla story makes headlines.
That's one person claiming an update bricked their car, but it's unclear if that was due to a bad software update or a hardware failure that coincided with the update. Tesla usually explains what they fixed, so it's odd that the poster never replied with more details.
Even if every software update was perfect, you would see individual stories like the one you linked to. There are millions of Teslas in the world, and they all get updates frequently, so a hardware failure will sometimes coincide with a software update. If a bad update were shipped to customers, it would be a story similar to this Jeep issue: thousands of cars affected at once, lots of furious customers, and news articles about the failure.
Well, given that the article says it caused powertrain failures on the highway, I'd say it's severe enough that it should absolutely cause the manufacturer's stock to drop.
I own a Land Rover and their system software also ("Pivi") seems to have tons of little quirks and issues.
Sometimes the cameras won't work unless you restart Pivi. One time, the entire car wouldn't respond to locking (via the app, keyfob, in-car buttons) unless you disconnected the battery and waited ~30 mins. Many people complain that they can't even successfully upgrade between Pivi releases. It'll error out a lot of times and they have to restart the process.
(Would be interested to know the kind of tech they're using if anybody's familiar with it!)
Software updates for cars should be rare, with the infotainment systems physically isolated from the rest of the car (aside from constant/switched power). And if it needs updates for whatever reason (eg a recall), they should be done at a dealership - not over the air.
Bricking a car with an over-the-air software update is stupid and unacceptable behavior. Stuff like this is why I'm actually kind of happy to be driving a 20 year old car still.
I looked at the Jeep awhile back but their safety ratings were absolutely terrible. I’ve been in car accidents aplenty so that was a deal breaker for me. Just about the only real electronic thing in my Lexus is an ECU which rarely goes and can be replaced relatively easily . Forget updating the maps and other nonsense. Just drive the damn thing and do mechanical maintenance and a little grease.
Just wrote about my battles with software updates[1]. Without getting into what should or shouldn’t happen, I’ll express my sympathy for the Jeep team and the owners. Software updates are hard.
[Edit: a commenter notes that Jeep’s parent has recently mandated LLM usage in development. Let’s hope these two are unrelated and that we aren’t going to start seeing more catastrophic failures like this over the next few months as people work out the limitations of this sorcery.]
Cars probably have multiple bootloaders even. Surely there are at least two, one for the ECU and one for the infotainment system. Perhaps there are even more depending on how complex components like parking cameras etc. are.
That came after the decisions to reduce both costs and tailpipe emissions - both obvious worthy goals. Is the implementation that is flawed, not the idea.
We could, but we shouldn't, because most software is crap. When the user is stuck with whatever software they got as a consequence of buying the machine they actually wanted, there's no incentive for the software not to be crap.
That doesn't even make sense in light of this article. The user isn't stuck with the original software at all... instead, due to an OTA update, they're stuck in the road.
I think if you compare a modern car with an ECU to a "traditional" car with manual ignition / carburetion system you will find that the modern one outperforms significantly on both power and fuel efficiency.
A few bad updates borked my Linux install and cost me hours of potential work and left me with a deep fear of installing updates. No idea what I'd do if I had a car that needs updates
A side note: is it just me or not but I prefer my new car to still providing tactile buttons to control things? I get that smart cars like Tesla wants to push touch-screen on everything but the control mechanism just feels clunky for me. When I sit on a car with physical buttons, I know exactly where they are and what they do when I press & turn them.
leadership problem, as everywhere. old grampas that used to manually draw gears on paper now have to "strategically align" a huge corporation that has to deal with new shiny and complicated things like software and they all have zero fucking clue. at least with cars you can always try to safely stop, with planes - not so possible. this will also soon creep up there.
My 4xe died in my driveway on Saturday after the update. Let me explain, from the perspective of a 4xe owner, how bad the response has been from Jeep/Stellantis:
- As of Monday 8am ET, zero legitimate communication from any Jeep-related accounts on any social media platform, or any other form of acknowledgement from the company (unless I've missed something?)
- I only found out about the issue after finally searching a few Jeep groups on Facebook (of all places) to see if anyone else was experiencing the weird failure mode I was after the update.
- The only remotely-official info was from a 'JeepCares' account (which is ran by Jeep) on some random off-roading forum? We were seriously all living off of screenshots from this forum, and the advice coming from the JeepCares accounts was contradictory: they claimed that the Uconnect update was separate from the telematics update, and that there was no way to stop the telematics update if the vehicle received it. Later they gave advice to defer the Uconnect update, making it sound like they were coupled.
- Due to the lack of info from Jeep, people were coming up with all kinds of "if you reboot Uconnect while the Jeep's in ACC mode, it clears the check engine light". This probably did clear the CEL but didn't fix the fault.
- There is no way to tell if you received the bad update.
- There is no way to tell if you received the 'fix' either.
- Dealerships have literally no idea what is going on.
- You're basically at risk of your Jeep going limp (power loss, unable to safely make it to the shoulder) and being stranded on the highway, even as I write this.
> - You're basically at risk of your Jeep going limp (power loss, unable to safely make it to the shoulder) and being stranded on the highway, even as I write this.
This seems extraordinary.
I was going to ask: Are you really saying they kill the vehicle's power system, effictively the engine, while the vehicle is being driven on the highway?
But no need to ask, the article says yes, that's what is reported:
> Instead, the failure appears to occur while driving—a far more serious problem. For some, this happened close to home and at low speed, but others claim to have experienced a powertrain failure at highway speeds.
Wow.
Ya, that is shockingly scary. It makes me think we need some new standards about software updates to vehicles in general (or perhaps these already exist but were missed for some reason?). I can totally imagine that software used to be this ancillary selling point that didn't need such tight regulation, but as it becomes core infrastructure for the vehicle this is less of an IoT toy, and deserves stricter standards.
How about: you get to say whether you want to update and when and manufacturers are required to very explicitly list all of the changes in an update? That would seem to be an acceptable minimum.
I don't think that Jeep would have sent out a message saying that one of the changes would brick your machine.
It seems that the ability to trivially roll back any update would be a better choice, at least for this. (But I'm sure there are downstream effects I haven't thought about if that were implemented.)
How do you roll back a fatal car accident caused by the faulty update?
Giving user’s control over when the update runs allows them to be in a safe and secure setting when that update happens. Allowing them time, gives them and Jeep the ability to slow roll the update so they can halt it if initial feedback is negative.
I say this as a Mac user who does not allow auto updates for MacOS. I wait a week or so until the chatter validates it as non-breaking. They pushed an OS update several years ago that broke a few things I rely on. So I don’t trust them now, but these things just happen on OS’s with third party software. I expect it. But, I also don’t want to be forced to deal with the headaches immediately. I’d rather let the third parties run updates and advise how to deal, before I have to dive into fixing things. With car firmware, there’s really no excuse for this except poor engineering / processes.
> Giving user’s control over when the update runs allows them to be in a safe and secure setting when that update happens. Allowing them time, gives them and Jeep the ability to slow roll the update so they can halt it if initial feedback is negative.
This does not fix any QA process that is broken. And frankly you should not need to update any control unit firmware after it is sold. The fact that they're even doing this is broken.
Unless your Mac is somehow attached to 5000 pounds of metal going 65 on the highway, the same standards should probably not apply.
Giving user’s control over when the update runs allows them to be in a safe and secure setting when that update happens
FTFA:
> The buggy update doesn't appear to brick the car immediately. Instead, the failure appears to occur while driving — a far more serious problem
And from the GP upthread:
> There is no way to tell if you received the bad update.
> There is no way to tell if you received the 'fix' either.
Good points, I did miss those. However, if I had this vehicle and I was reading this article today - and had the ability I'm asking for - I would just keep my current version running until they figure this mess out. It's the advantage of letting other people run the updates first, you get to hear about issues before you experience them.
> user who does not allow auto updates for MacOS.
Many security compliances require auto-updates to be on. It's thought of to be a lesser evil, because many (most) users never update their OS/browsers, which is worse.
It's not perfect but seems reasonably easy to implement and would certainly help. If the user needs to approve each update and can see what the changes are most updates will either be skipped or delayed long enough that catastrophic bugs will only hit the small subset of cars that update immediately.
I would bet most updates, especially from a company this bumbling, will be more along the lines of increasing telemetry or pointless UI changes than releasing actually useful features and bug fixes.
You might not accept an update with a bunch of changes that didn’t sound relevant to you.
I certainly wouldn’t accept one while I was still driving the car!
The update didn’t happen while people were driving. Rather, the bug took time to occur, well after the update had been applied.
There is no need to invent new regulations. We already have criminal liability, endangerement from gross negligence, and manslaughter!
I do not see reason, why CEOs of big companies should be exempt from this!
If bus driver makes mistake, or someone drives drunk.... They get punished. This is the same thing!
> There is no need to invent new regulations.
The current regulations are written for a time where cars didn't have rolling computers in them. And even then, the regulations don't account for Tesla-style linked systems. So I say we do need new regulations.
Yes and we have the NHTSA (unless it's already been neutered by the chaos) who can accumulate statistics and issue recalls.
NHTSA's power is simultaneously very broad and narrow. They're empowered to investigate potential safety issues after the fact, but this may not be a safety issue in the very pedantic sense often used. NHTSA can proactively set standards, but the standards they've set (FMVSS) largely ignore modern electronics. So on and so forth.
Incredible that Jeep did not think to have updates only go out to cars which are stationary with engine off.
>> - You're basically at risk of your Jeep going limp
This has happened to me twice with a Nissan Leaf. I paid money to get a read out from the computer system, and there were no timestamps on the screens of data.
Modern cars "computers on wheels" are dreadful.
Is it possible to disconnect the power from the radios used for "over the air" nonsense? Then at least they would be stable.
In the Leaf, you can disconnect the TCU from the CAN gateway controller located behind the infotainment system to disable its remote connections.
It will throw a perpetual "check engine" light and disable the hands-free microphone, but OVMS users have made a "dummy TCU" that gets around that annoyance.
I have the opposite problem. The specific infotainment system update I need requires a $200 visit to a dealership with a specific model of a USB 2.0 SanDisk Flash Drive (NI-52727-1). Not available OTA despite the Leaf's OTA capabilities.
Is it possible to disconnect the power from the radios used for "over the air" nonsense? Then at least they would be stable.
I've read online that for some cars, you have to dig deep inside to disconnect the cellular antenna.
I'm a little more lucky. On my car, you can pop out the SIM card from a slot in the ceiling, behind the rear-view mirror.
This assumes you haven't given your car access to your home WiFi. (Some people do this so they don't have to pay for a data plan for their car, and it kinda sorta "syncs" when you get home.)
It is completely insane that this is happening. I did DD on a company in the automotive space a couple of years ago and flagged that they did not check if the vehicle was stationary, motor disabled before updating. They were all surprised at how I thought that this could possibly ever lead to issues.
I have Java code running on commercial aircraft. You can’t actually run Java code on commercial aircraft because the FAA doesn’t (or at least not at the time) know how to certify it.
The entire box it’s on isn’t powered while the plane is in motion (“wheels on ground”). It’s shut off before preflight and doesn’t turn back on until the plane is on the ground. The service my code is part of is responsible for queuing updates and downlinking telemetry. Updates are manual and obviously you can’t run them while in motion if the box they are on doesn’t even have power.
Cars probably don’t have to go this far, but there’s a continuum and they’re clearly in the wrong part.
Even iPhones and windows let you schedule update times. Just the fact that a freaking MOVING MACHINE doesn’t is egregious on itself. Imagine if stellantis would manufacture industrial equipment or nuclear reactors!
Given the state of the software industry, it's honestly more surprising that this doesn't happen more often. Our industry is a complete joke, and somehow we've been given responsibility over people's lives.
> Given the state of the software industry, it's honestly more surprising that this doesn't happen more often.
It probably does. We just don't notice.
> Our industry is a complete joke, and somehow we've been given responsibility over people's lives.
Amen to that. kqr made some choice comments the other day in that thread about the airliner that came to within a hair of crashing due to running out of fuel. Thinking about risk is not a skill that we're born with and it is always sobering to read the 'risks digest' for a bit and to see how thin the ice really is.
We are really only about 60 years old as a proper profession, and we seem to be trailing behind doctors for professionalism and standard of care by about 100 years.
I don’t know what will turn out to be our penicillin, or our Joseph Lister, but in 1960 the former is something that didn’t exist when older doctors were in school, and latter had only been dead for fifty years. It may not have happened for us yet.
On the topic of professions: Joseph Lister was a surgeon. Modern surgery (which I define as surgery aided by anesthesia) is a relatively recent discipline dating to the early 19th century. The introduction of anesthesia made lengthy and intricate operations possible but also ushered in novel problems and complications. Surgery as a field had to learn tough lessons over time.
He was known more for antiseptics but the biggest surgery moment for me will always be “using soap” and I wonder what the software equivalent is.
Like I said we are still young, so it feels sort of arrogant saying we have figured something out when I know how many things are industry standard now that almost resulted in shouting matches trying to get done even 20 years ago. Maybe our soap moment is coming up ten years from now.
But I suspect automated testing may be the wash your hands, because it represents a sort of hygiene that “we” used to just say fuck it or make a minimal effort.
I’m going on a limb here because i’m not directly on the software industry but my first suspect would be metrics and the fact that you have to deliver a product at certain time “no matter what”.
According to the article, that's not what is happening. The update itself completes fine; it's the updated firmware that is buggy, and seems to cause/require a reset of the ECU while in operation. Not that that makes it any less insane, but the update process does not seem to be implicated here.
Yes, and if the update happened while at home, most people could get the error at safe speeds (most people does not live <1 min from a highway).
That’s not how this problem occurred. The update happened hours before, but the bug only manifested once the driver was on the road.
they did not check if the vehicle was stationary, motor disabled before updating. They were all surprised at how I thought that this could possibly ever lead to issues.
My anecdata is that my car won't update its software without the owner explicitly requesting it. And then, it will only do it if the car has something like 50% charge, hasn't been used for an hour, and nobody is inside.
I once tried to do the update while I was inside, and it refused.
That's good. You may want to list the brand here.
My BYD wants the battery over some percentage, the vehicle in park, and the hood closed. The hood one was surprising, I wonder if it's for the safety of the car or of anyone working on it.
Probably a safeguard to keep sonebody from unplugging something during the update.
This sounds like the sort of thing that happens when a team has a deliverable that slips multiple times but everyone had vacations planned for a responsible amount of time after the deadline.
Under time pressure and confirmation bias they signed off on code that was giving off signs of being broken, pushed it, and now key staff are either on airplanes, out of coverage on their phones, or cannot work entirely from memory and don’t have their computers with them because vacation.
I bet they already have communications ready, but everyone Director and above needs to "wordsmith" it to deflect blame and make sure nobody looks bad, and all the lawyers need to bless it so that it's not admitting anything. I bet you there's a Word doc being frantically E-mailed around all day, needing to be approved by 25 people before it reaches the light of day.
How did you verify that a software update can 1. Occur during driving operation of the vehicle and 2. results in vehicle power loss?
I worked in an auto supplier years ago and there where several protections in place to prevent the risk of update corruption on safety related components. One of the simplest one the UDS programming session having entry protections related to vehicle speed, vehicle driving mode, etc.
My update occurred while parked. I hit the failure mode 1-2 hours later pulling out of my driveway.
Thanks for the clarity. Wow this is a big deal.
My condolences. I wonder if you qualify for a loss-of-use rental under the warranty.
Personally, I would return the vehicle as defective after an issue like this. No way I'm going to trust the lives of family, friends and myself to a company like this.
I really want a 4XE Wrangler to replace my aging JK at some point, on paper its amazing, lots of torque and power, decent economy for a brick with large tires, and can actually run a pure electric enough to get around town, plus still takes all the standard Wrangler parts.
However in classic Jeep style they just can't get reliability down, and the PHEV part seems too complicated for them.
If it was just reliable it would still be the best selling PHEV in America, they let that go.
There is no sign of the 2026 Wrangler 4XE it might be canceled like the Gladiator version...
My Wife just got one last year, the electric engine will give you about 17 miles on a full charge that takes about 7 hrs, its pretty much useless on full electric mode, pretty good with the milage on hybrid which is what this really is. And yes do not expect reliability, first day out of the dealership we got a check engine light, apparently one of the workers at the dealership forgot to attach a wire or hose, at least thats the explanation that we got, i told my wife to insist on getting it documented somehow or get a log from the dealer but of course they played it off as human error (total BS). Few months later and her screen started to turn off and on, or just die completely when air play was connected. Stay away from these cars.
Agreed, it's an expensive but great vehicle. Closing the last mile of the reliability gap is always tough, but they need to figure it out.
It's probably canceled, not only for reliability reasons. EVs and plugin hybrids are probably doomed, at least in the US: The EV tax credit subsidy is gone, fuel economy requirements will be or already have been eliminated, electricity prices climbing (my rate increased almost 50% on my most recent bill) and the Trump administration is extremely hostile towards anything related to renewable energy.
Does the market actually want EVs and hybrids if they require subsidies and regulation to make significant market penetration?
Unfortunately to survive the next few hundred years, we may need another dictator of human behaviour than what the market wants.
A functioning market that priced in the externalities and future risks would far prefer them.
I’ve always been surprised that people even buy jeeps given the notorious reliability issues and frankly strange design choices.
The times that I have been given a jeep rental while on vacation or work trips have always left me disappointed with the vehicle.
My Wrangler JK is 10 years old with almost no issues, its a relatively simple vehicle compared to most, easy to work on as well. The Pentastar V6 is well proven engine and the issues with it are well understood.
I really like the vehicle, it has served me well and taken me many interesting places across the country, along with daily driving. I tow it with a RV and its one the few that can do so now days, plus its extremely capable offroad.
The 4XE is very alluring, much more fun to drive (I have rented one) and I could charge it off home solar and still tow with RV (the only PHEV thats possible to do so). If only it was reliable...
There’s just no comparison when you consult reliability reports, though. Of course, even companies like Toyota are going in the direction of always on, arbitrarily updated software and subscriptions.
I have no idea what I’ll buy when my 11 year old toyota finally retires.
imho the occasional mixup is going to happen, and eventually it'll be big like this or like Crowdstrike, but pushing these out on Fridays means the critical staff isn't there to help. The people who could have communicated this stuff to customers and dealerships were in bed when people got into their jeeps at 6am on Saturday after an overnight update.
I believe crowdstrike's update was on a Friday night as well.
Unless its a serious security bug, it can wait for not only for better QA testing but also for next Tuesday. Read-on Fridays need to be an industry-wide thing.
To me the bigger issue is that the infotainment system can affect the core function of the vehicle. That seems completely unacceptable, regardless of when an update is pushed out. The two systems should be separate with the infotainment system as the lesser important and unable to actually effect the core system.
Honestly the only thing that's going to change this is criminal liability for safety related software bugs. Otherwise, it's just business as usual and the business for the last 15 years has been cutting QA and asking developers to do testing themselves, which is literally impossible for a lot of software due to lack of proper staging environments and large permutations of configurations.
Lack of QA also tilts the power dynamic with project management. You could have the lead engineer and tester tell mgmt that things are not ready.
I think the nature of capitalism will make this impossible. The capital owning class will not allow criminal action for this and will also fight any common-sense regulations. If the working class gets that regulation in via the democratic process, that's fine, but its unlikely especially since it hasn't happened yet since we've gone digital on near everything starting from the 70s and 80s.
The working class lately seems more focused on 'culture war' issues and not economic or material or consumer or worker's rights issues anyway, so we're probably as far from any kind of regulation reform in software as possible. I remember a couple decades ago FOSS as an ideal seemed stronger and you had people like Lessig pushing hard for IP reform and Swartz and others for 'information must be free' honest-to-goodness mainstream movements and all of that seems to have went nowhere and is somewhat to very unpopular today. When was the last time you saw a populist movement towards liberal tech reform like this? Outside of some edge cases like medicine or power generation, the regulations here are purposely kept weak because that's what the wealthy desire.
Maybe our kids or grandkids will have this after the pendulum swings back, shrug.
Aviation proves every day that this is perfectly possible if there is a will to do it and a regulator with teeth.
Aviation relators just allowed boeing to self certificate again. Avition has a lot of historical regulations and can work through sheer inertia for years.
But the whole point is, regulator wont have teeth. They have teeth when politicians back them. And as of now, politicians back billionaires and deregulation. Wall street hates regulators, billionaires hate regulators and sizable part of population prefers people dying if it means they cam hurt libs and ennemies.
I recently worked at a big home lighting company, working on the OS of the router device that communicates with the light bulbs themselves and the internet/user.
Our OTAU architecture uses A/B system updates [1]. Core idea is that both the kernel and the rootfs (read-only) partitions had 2 different bootslots in storage, and the OTAU would only write to the bootslot that is unused. Hence, if something went wrong, the system would automatically fallback to the previous version by just switching the bootslot used. Over the numerous years that that architecture was used, I couldn't find a single post-mortem that resulted in devices being bricked. Something to note is that the rootfs partition was overlaid with a writable partition for persisting state data etc.
Now that was a $two-figure USD device, not a $5/6-figure USD electric SUV. Is this a cost-cutting measure? At those price levels, doubling your NAND size is not even half of a percent of the total cost of the vehicle.
Unless there was a serious issue that the used bootslot corrupted the unused bootslot, then I don't see how this could have happened.
It's saddening that car manufacturers are so unserious about the code they're deploying.
[1] https://source.android.com/docs/core/ota/ab
I've worked in both IoT lighting and automotive, so I'm comfortable comparing the two. This also isn't offered as a defense.
The big auto OEMs are just as sensitive to absolute BOM cost optimization, regardless of the percentage increases. I don't think this was a bootslot issue though, regardless of the word "bricked". Even as backwards and ill-advised as auto software can be, generally accepted practice is that updates are impossible while the vehicle is in motion. This is usually enforced by systems shared across multiple OEMs through the tier system.
The situation sounds more like a disastrously buggy new firmware.
I wouldn't put either past stellantis though. The auto industry already scrapes the bottom of the proverbial barrel sometimes, and stellantis isn't exactly known for their top of market compensation.
This is generally how other devices work as well - for example all Android devices and Android-derivatives (which many of these cars are!) use a similar A/B partition to prevent bricking.
It definitely reduces the risk of updates, but it absolutely doesn't eliminate it.
The A/B updater itself is a surface area - especially if the logic is complex and there are other child devices that are updated at the same time (likely for cars). In that case you're not just coordinating between two independent partitions, you're coordinating between 2 * N partitions, half of which have dependencies on each other.
Also, the key bit of the mechanism is that upon successful boot the new partition is flagged as "good", which causes flags to be set to assert that the update was successful and the backup partition is no longer needed. That logic can (and does) fail - if your failure point occurs after this checkpoint you're hosed still because you're past the point of no return.
Making that worse is that in most systems you want the "it's all good" checkpoint to occur early - you don't want to, for example, wait multiple minutes for all user services to come up. But that also means that if a critical failure happens in said services, you're past the checkpoint.
> Now that was a $two-figure USD device, not a $5/6-figure USD electric SUV. Is this a cost-cutting measure? At those price levels, doubling your NAND size is not even half of a percent of the total cost of the vehicle.
Could just be a competence and priorities problem. If it's cost cutting, it feels way more likely that some PM cut some story from a sprint to hit a deadline (and objections were either not raised or ignored), than they did some engineering analysis and explicitly decided to save $3 per vehicle by cutting the NAND size.
Edit: Actually, I don't think that technique would have helped, the problem wasn't a botched update, but a seriously buggy one. From the OP:
> The buggy update doesn't appear to brick the car immediately. Instead, the failure appears to occur while driving—a far more serious problem.
> Edit: Actually, I don't think that technique would have helped, the problem wasn't a botched update, but a seriously buggy one. From the OP:
That and combined with general refusal of new automotive bootloaders to downgrade. You can go only up in versioning. So even that you could have working version on second partition, it will never get loaded because it has lower version than currently one you are running.
Two points to add:
1) Total cost of the vehicle does not matter. What does matter is the operating margin. Half a percent of the total cost of the vehicle will move them from 2% margin to 1.5% margin. (Ford has operating margin of 2% as an example)
In other words an increase in 0.5% cost of total vehicle will reduce their profits by 25%.
That’s a huge number now! Note also that car manufacturers are in a bad spot because their volumes are fairly low (smartphone = 1M/yr, car = 40k/yr) and have harsher requirements for chips, driving the cost way up.
2)AB updates are great, but they can still fail or get soft locked. Especially important around code when you configure the slot to be good and when bad.
You are conflating gross and operating margin.
It's also more dynamic than your presentation. They have a little bit of pricing power, so a small increase doesn't all come out of the margin.
I have heard anecdotally that auto manufacturers are sensitive to a price change less than $5/vehicle. This is better than some industries that are sensitive to $1.
What could easily have happened is that the negotiators didn't include A/B updates in their spec, or they only specced A/B updates at 1GB OTA size.
They do their usual hammering on price, and the head unit or ECU manufacturer gave them some savings by cutting storage space to the bone.
Maybe it was still enough for A/B updates, until the usual software bloat took the updates past the critical limit.
They could still do a safe update by doing an A/B/A update (where B is a shrunken, update-only OS), but that requires development time, and the engineers should already be working on the next vehicle.
Worked for them. Corporations with many brands in their portfolio might discuss for weeks over price differences of components of 0.20 Euro. That‘s twenty Euro cents difference for e.g. a USB connector. If you expect that a vehicle platform sells in the 10s of millions over its lifetime, you‘re talking real money very quick!
However, the price of recalls and warranty rework is never computed into that number.
yet another example of the flawed logic where "we don't have time/money to do it right now, yet we always seem to find the time/money to redo it later after the shit hits the fan"
> the system would automatically fallback to the previous version by just switching the bootslot used.
That's the hard part though.
It's shockingly common in my experience to have an A/B boot setup, but no actual logic in the userspace application to switch back to the other partition if something goes wrong. It's just a defense against somebody pulling the plug during the OTA, it doesn't protect against software bugs at all.
We used to do that with device that where in difficult to reach places with harsh uptime requirement! Think industrial routers and protocol converters. I think it pays for itself very quickly. Sending someone for such a device can get expensive.
That's a good point.
I'm curious if failing to do that opens Jeep up to legitimate lawsuits.
Well, on the positive side, at least they were stationary unlike these vehicles. Don't get me started on botched OTA updates, there are so many ways companies get those wrong it's not even funny.
I once managed to brick a PC motherboard that advertised "dual BIOS". It didn't fallback to the previous version after a botched BIOS update.
It's totally possible that the update corrupted the other bootslot as well. If those blocks aren't off-limits to the updater program, it's just an off-by-one error waiting to happen. Slot 0 or slot 1?
Another possibility is that the updated version booted up just enough not to trigger the automatic fallback, and then got stuck in a loop.
I've had a bunch of updates break some stuff but since moving to Fedora Atomics/ublue I've never had a system I could not get back into.
Nothing was bricked at all. Thats just how clickbait journalists describe things that stop working in some way after an update nowadays.
(Most computers in a car don't need duplicate partitioning because they can be bootstrapped from a central computer)
I’m sorry, but you’re incorrect the vehicle completely shutting down while driving and not working again until you put it into park and then it’s shutting down five minutes later is effectively bricked and extremely dangerous. Myself and my family almost died just trying to get home from dinner. It was a complete loss of propulsion and power steering.
There are many things that are dangerous that aren't "brick"-ings. If it can be later restored to function, then it is not bricked.
being unable to drive my vehicle due to a software update is bricking. It's also a pun, us Jeep owners call our Jeep's flying bricks.
Being temporarily unusable is not how I've seen "bricked" used, bricked means unrecoverable and the item is completely unusable except for as a brick/paperweight/door stop.
If you can do something, anything, to the vehicle to repair it, then it's not bricked.
Thank you. I really hate how watered down the term "bricked" has become.
I prefer the term borked in these situations
Then it would better be described as a life-threatening event rather than a bricking - especially as, in the hierarchy of concerns, the former is more serious than the latter.
And then it was fixed with another OTA, so it was not bricked. Why bring up this pedantic point you may ask? Because the grandparent raises a scenario that doesn't apply here. A/B updates or not were not at all the issue here.
I for one am always grateful when things are engineered thoughtfully and with redundancy as it is symbolic of respect for the people who are your customers. Especially in something as important as a car, "can be bootstrapped from a central computer" - when? how easily? how reliably? - is not good enough because things will inevitably go wrong for some portion of the user base.
Brick is now slang for a lot of fail conditions that aren't classically 'bricked.' This has become really common I've noticed. Honestly, this ship has sailed and isn't even worth fighting anymore. Its like Xerox asking people to stop calling copies Xeroxes.
We just never bothered to develop a new term. Maybe 'soft-bricked?' 'Semi-bricked?' I would like journalists at least to start using more accurate terms, but 'bricked' I imagine gets a lot more engagement and ad impressions, so here we are.
Wikipedia has a section about this. They call it soft bricked vs hard bricked, according to the difficulty of restoring device function and how the broken state presents. Even hard bricked is usually recoverable with appropriate tools, so it is a spectrum.
https://en.wikipedia.org/wiki/Brick_(electronics)#Soft_brick
All those words you are saying, it's quite possible the sub-contractor to the sub-contractor to the sub-contractor in a foreign low-cost country that actually did the work has absolutely no idea what any of that means, and they are doing the bare minimum to deliver
Why wouldnt a foreigner know what this means? This seems very xenophobic. And if US/Euro management is hiring these groups and not giving them requirements for redundancy then guess who is at fault? Not the contractor.
Because the whole point is that it’s cheap labor. Cheap labor is inherently inferior
I rented a Jeep Wagoneer recently and found it to have such comically glitchy electronics that this comes as no surprise. The second day we had it, the liftgate stopped latching entirely, it beeped and popped some error messages on the dashboard and simply wouldn't latch shut at all, no matter what we did. Searching the internet produced lots of people with the same problem, reporting that it required a software update to fix. There was no manual override to the electronic latching mechanism.
Luckily we were near a location of the rental car company—rather than deep in the middle of nowhere where we were headed—and exchanged it for another of the same model, which was all they had available. The next 1000-something miles we drove were filled with endless weird glitches:
- When a passenger plugged in their Steam Deck in the back, the entire infotainment system cut out and went black, including the instrument panel, and then started glitching in and out until they unplugged it.
- When parking, the driver's seat would retract slightly to make it easier to get out, but it never moved forward again, so the seat would get further back at each stop until it was manually repositioned.
- The entire drive the system flashed an un-dismissable error about a rear seat latch, which seemed completely functional.
- The TPMS light went on and off periodically as it seemingly lost and then regained signal from one wheel or another.
- The system flashed errors related to the automated cruise control being unavailable/broken at random times.
- The electronic parking brake kept applying itself while briefly paused in parking lots.
- There was something inscrutably wrong with the climate control that we never really figured out where sometimes it'd just get hot inside the car despite no change to the AC settings.
When we got back I found tons of people online talking about similar (often worse) issues. Incredibly terrible for any new vehicle, never mind one that costs $80k.
On a recent one week trip with my family, we went through FOUR Grand Wagoneers, each one with another show-stopping problem.
Honestly, unsurprising. Jeep and Stellantis/Dodge in general has horrible quality control and extremely poor electrical designs. They have a huge enthusiast community that will be happily apologize away the copious amounts of flaws. Frankly, nobody should ever buy their vehicles, it's just robbing yourself.
I own a 2002 Grand Cherokee which sometimes will have a 10A+ power drain for no apparent reason. Of course it doesn’t do it when I’ve got my voltmeter on it, except once (when the 10A fuse in my Fluke blew). I resigned myself to unplugging the battery or leaving it plugged in to a high current battery charger at home, and leave it running if I drive it somewhere.
I rented a Jeep Liberty or Compass circa 2018 whose headlights were permanently in DRL mode: couldn’t turn them off or on. Fortunately I didn’t need to drive at night.
In 2017, rented a 300 with 500 miles on it; the infotainment was completely broken, which hosted the controls for the seat heaters and temperature setting. It was well below zero in Minneapolis but we had to drive around with our windows down because the fancy climate system defaulted to max heat blast + max heated seats based on ambient temperature.
Long ago I had a 1996 Neon where the wiring harness started to fail, and the speedometer would stop working. Later on the oil light would come on despite oil pressure being fine. Eventually the entire car just quit running at all at random - nothing but a dim oil light. I sold the car for scrap for $65 since I got tired of being randomly stranded.
So what I’m saying is that it sounds like Chrysler has managed to actually keep doing the same thing for 29 years: electrically unreliable vehicles.
In my personal experience with cars that had strange electrical problems, they tend to be on a bad ground somewhere in the loop. I once took a Chevy S-10 to a place my dad recommended. A guy walked out to ask what the issue was, he nodded, took a step back to look at the truck and asked the year of the truck. He then nodded and said "Yep", and then without looking reached under the dash on the driver's side and tightened a screw by hand. All electrical problems went away. He walked away after politely telling me to have a nice day. I was baffled, and he said it would cost him more in time to write the repair up than he could honestly charge me.
The point is that stable ground connections are notoriously hard on something that by design shakes, rattles, and rolls with all of the vibrating and bouncing on our "modern" streets. It's also a very easy thing to misdiagnose unless you're a mechanic that specializes in automotive electrical systems. It also takes time for new year models to display their warts enough that non-dealer mechanics gain experience repairing them.
Yes ... However. Most car manufacturers manage to deal with this without it becoming too common, with standard engineering controls ( proper fasteners, torque specifications, QC etc).
The neon issue was the engine harness fraying to the PCM and eventually burning out either that pin on the PCM or cable grounded itself.
Back in the day I was buying these, around 2005' or so, for $300-400 non stop and repairing that, the dash that cracked and misc cosmetics.
They were great cars, the R/T model in manual was fantastic in gas, reliability and safety (sadly crashed it.) but boy was 16-20yr old me happy with these neons. Can't believe they sold shy of $9,999 when new (for base of course)
Just reading your post took me back 2 decades, wow.
I own a Jeep Wrangler, and you're right the electronics are terrible. The rest of the vehicle is really solid though. The only problems I've had with it in three years are electronic in nature. And I've really pushed it to the limits: Colorado Passes, Utah Dessert, Montana backroads. I drove it to the Arctic Ocean and back on the Dempster.
Still there is no excuse for how terrible the electronics are in Jeep / Dodge (I'm assuming all Chrysler) vehicles. And it's been that way for decades.
I owned a Jeep 4XE, and I was glad the day we sold it, and I'm doubly glad today. The electronics and software were crap, and the powertrain was simply insufficient. At one point, they issued a notice that amounted to 'it might catch on fire, keep it away from your house.'
Yeah, I have family members with 2 JKs and a JL, unfortunately all plagued with issues, almost entirely related to the electronics. A Jeep Wrangler is a vehicle that sounds great on paper, but actually owning one is an exercise in frustration unless you just enjoy fucking with wiring harnesses. I am sure many others will come out of the woodwork to say that Jeeps are great, unfortunately they are not.
It’s too bad because the wagoneer is the best designed car in the segment, inside and out for the most part.
I have a somewhat bad back and want something that I can occasionally work from, so a big space, comfy middle seats, a wide center console. Car makers for some reason refuse to make essentially a Tahoe but shorter wheelbase / 2 row which would be ideal. Instead you have to go with the full size to get full-width.
But out of those, only American brands seem to understand the utility of blocky interiors. Armada and all the Japanese and Korean large SUVs always use swooping rounded edges which really reduce utility.
But the American brands are all less reliable and struggle with consistent quality.
This is disturbing. There is absolutely no valid reason to make OTA updates to your ECU or other essential components. Never. Ever. If there ever was a need to do such an update, it should be made in the dealership by a professional who can roll it back if there’s a problem. Automakers have become far too complacent while at the same time over-automating everything and profiteering by turning things into a subscription model. This is why so many car enthusiasts (myself included) prefer older cars.
Not even two weeks after Stellantis mandates vibe coding engineering workflows. Has to be a new record.
https://www.stellantis.com/en/news/press-releases/2025/octob...
Wow! But seriously this would have to be code written before two weeks ago to be pushed to production OTA to a fleet of vehicles, right?
All bets are off for any org willing to push fleetwide updates on a Friday afternoon.
I've had a Jeep for a few months, and it bothers me so much that the entire community is about modifying the vehicle as much as possible, but they still come with this locked down OS.
If any car could be the champion of OpenSource, it is a Jeep Wrangler, but they're using an OS made by SiriusXM for some reason.
I don't think the software hacking side of Wrangler mods is barren. e.g. https://www.jlwranglerforums.com/forum/threads/jeep-hacking-..., which IIUC has been commercialized into https://www.zautomotive.com/products/z_tzr_jlm.
I don't know that anyone has broken the head unit firmware though.
Isn't Jeep the company that introduced infotainment pop-up ads that play at the stop light? I doubt they'd want you loading an open source OS replacement to avoid that.
Why does an OtA update even have the ability to brick the entire vehicle?
The infotainment system should be completely isolated from the driving system.
You're starting out with an assumption, that this is an OTA update for the infotainment system, and then conclude this incident shouldn't be possible. The problem is the assumption.
This is a OTA vehicle update. It has the ability to update the infotainment, ECU, ECM, TCM, and BCM. Multiple manufacturers have been able to release recalls that fix major vehicle defects (safety, reliability, and performance). That wouldn't be possible without OTA updates that update core vehicle computer systems.
Unclear where this idea that OTA = Infotainment came from. I'd go as far as to say that most manufacturers can do this in 2025.
> Unclear where this idea that OTA = Infotainment came from.
Because to some people, the idea of an OTA update being allowed to change mission critical parts of a machine automatically without a solid rollback system is absurd, and the best way to do that is to never do OTA updates of mission critical parts at all.
Rollback is getting extinct for security reasons. When you will screw up, you need to do a new release. Hopefully screwed part is still talking.
This is why OTA updates should simply be illegal/considered negligent engineering. If you want a convenient update, let people plug their phones or computers in via a USB port or something, or take it to a mechanic to do so. There shouldn't be security concerns with an appliance because it shouldn't be writable outside of an owner-intended maintenance mode, which should be impossible to activate wirelessly.
Wait until when fridge or TV will come with its own 5G chip and they will get bricked by remote update because it is time to buy a new one and there will be nothing you can do about it.
They don't have to do this. The cheap materials in the compressor or cheap capacitors used on the power supply board will just silently fall. And the cost to repair the problem, for the average person, will be slightly less than just buying a newer version of the crap that just broke.
My LG refrigerator recently stopped cooling. The error code suggested it was the defrost mechanism. It was more of a hassle of a repair than I wanted to take on so I found an authorized repair shop on the LG site and opted for their flat rate repair.
First trip the repairman replaced all of the defroster parts and sensors. It failed again with the same code 18 hours later. The second time he replaced the main board and at least one other part. It now works great and I have effectively a new fridge aside from the compressor for less than $400.
Compressor still has three years of warranty left and we expect to move before then. It can (hopefully) be someone else's problem.
I've got various IP subranges categorized by probability of having to block them in the router's firewall.
Main idea was locking updates to once or twice a year and resort to HomeAssistant.
It's at 33% execution stage so no idea on the feasibility.
> Rollback is getting extinct for security reasons.
Unusable devices are technically the most secure ones.
... but then you'd have to pay mechanics at dealerships to do it. Middleman cutting.
The first paragraph of OP said
> The automaker pushed out a telematics update for the Uconnect infotainment system that evidently wasn't ready
This should be made illegal. It’s a massive national security threat. Imagine on the eve of a war, instead of Jeep 4xes, it’s every recent Ford or Toyota or GM car, and instead of a software update that can be rolled back, it wipes the flash completely, or reprograms the ECU to damage the engine or disable the brakes on the highway or something else to cause accidents.
You assume that it will be a foreign enemy and not your own government bricking your car on the eve of revolution.
Just wait until you hear how much of our country's critical infrastructure is hooked up to the Internet. Traffic lights, water treatment plants, power plants.
The infotainment system can be the gateway to the rest of the vehicle network. It makes sense to attach a 4G modem to the display head to do mapping, hands free calling or emergency response, you may as well use it to download ECU updates too.
"I'd go as far as to say that most manufacturers can do this in 2025."
What does that have to do with OP's comment? And their point is still valid, and OTA update should not be able to brick a vehicle, regardless of the system receiving the update. And regardless if "they all can do it".
Any update can brick your device if done poorly. This device just happens to be a car.
You misunderstood what OP was saying. They claimed that an update to the infotainment system shouldn’t be able to brick the other systems in the car. The response points out the car’s OTA update subroutine has access to update every critical system in the car by design. It’s flawed logic to assume that OTA updates only affect the infotainment system.
It has everything to do with it.
If OTA updates can update core vehicle computer systems, in ways that can correct safety, performance, and reliability problems then they can also brick that vehicle.
The manufacturer has the ability to push an update that reprograms computers that control how physical components behave in a vehicle. By the very nature of that; they can push good or evil updates.
Which is a reason the market for "dumb" cars is tightening up. Both my cars are "smart" and sometimes I wonder if I really own them. It bothers me that the maker can cause an update without my permission. (Yes, I know that's the world we've been living in for a while now.)
most cars these days have GPS and return location and so on, why can't manufacturer run these updates only at night and when the car is parked at home? There should be no reason for any OTA update to happen while the vehicle is running (or on a trip etc.), downloading the OTA update, sure, but definitely not applying it. Also there should be a documented procedure to restore the previous in case an OTA update fails.
...because the very first paragraph of the article says it was an infotainment update? Thanks for the info, tho.
Why didn't the vehicle manufacturers robustly test their software systems on their vehicle's hardware before releasing the product to the public?
Because cost. Same reason why dash clusters and infotainment systems are now all monitors - its actually way cheaper to use those than analog gauges. The software is built on a famous bullshit paradigm of "never rewrite, always reuse", and as a result shit gets patched together without any concern of how everything cooperates.
Now with hybrid or electrical drives, a motor controller is basically a package that runs its own software, which then interfaces with the rest of the car. And OTA updates can overwrite its firmware.
The only manufacturer that has avoided most issues is Toyota, since they have been doing hybrids for quite some time. Other companies are just starting on this path and to catch up, they can't be bothered to do software deep dives and figure shit out.
As long as it exchanges information (mundane things like muting the music when parking sensors have to be heard, requesting battery/fuel status for advising the next fill stop etc) the isolation can't be entirely complete.
How about read only over an optic cable?
I wasn't really talking about galvanic isolation. And on modern vehicles instruments and infotainment tend to run hypervised on the same physical host. Thing is as long as you exchange information there's always potential for logic coupling allowing the trouble to cross the boundary. Not to mention the basic rate excess/denial of service situations.
I had an OTA update brick my Tahoe infotainment system. Now that backup cameras are standard requirements, those were all unusable. Also affected things like the clicking sound you hear when you use your turn signal. That was completely silent. Cost me ~$2k to get it fixed and wasn't covered under warranty. Good stuff. I've disable future "updates".
An FYI for the future, but backup cameras are considered a safety system and manufacturers are required to repair issues they've caused in safety systems regardless of warranty status. The appropriate escalation if the manufacturer doesn't recognize this is to get NHTSA involved with a safety complaint [0]. That's the main way recalls happen.
[0] https://www.nhtsa.gov/report-a-safety-problem
This is a lawsuit, to recover repair costs and any loss of use.
It's not worth it, but it's necessary.
They're not isolated anymore, Tesla set this precedent and now everyone is trying to copy them. Volvo is having the same set of problems.
I'm guessing that it has features like "remote start" and that these features weren't designed particularly well.
What does it mean to "start" an EV?
it clicks a relay. Just like with ICE vehicles people usually use it to warm up their car in the winter.
Also, batteries may need to be preconditioned if too hot or cold. A lot of EVs let you set your ideal departure time in a widget as opposed to using a remote though.
This is a 4xe. It is a gas jeep with an overpriced, undersized battery and motor bolted on.
It can be started just like all the other gas cars.
Although even with full EVs, there's a reasonable concept of a "start". Some even let you essentially unlock and allow driving remotely, even if the local driver doesn't have a key. That's useful sometimes.
Pre-warm the battery, pre-heat or cool the interior, enable the defrosters.
Turn on the temperature control system?
Why does the update even happen while in motion?
The article doesn't go into a lot of details, but it only says that the bug happens while in motion, not that the software update itself happened while in motion:
> The buggy update doesn't appear to brick the car immediately. Instead, the failure appears to occur while driving—a far more serious problem.
The problem is worse than "just don't update on the fly while driving". The update happened while not driving; the bug causing the failure mode of shutting down power and engine happened later while driving. There's nowhere to hide from these types of problems it seems.
it was a telematics update
The infotainment system on my car can make changes to the suspension. Can change from street to track mode and even has a launch mode I can initiate for starting a timed 0-60, etc.
I can also put the car into valet mode so it won’t go fast. If I forget the valet mode password I am told I have to buy a very expensive replacement because it can’t be unlocked by a dealer.
I’ve noticed that newer cars seem to get updates that affect performance. Things like how they shift gears.
I can't for the life of me understand why infotainment systems knock so many engineers for a loop. Is there a particular reason (industry/domain-specific) beyond just low-quality software development?
My Mazda 3 (2018) just had a class action lawsuit for its infotainment system which, completely at random after years of normal operation, starts clicking on menu items and scrolling about the settings (only to stop and not do it again for a couple of months). It can get so bad you just have to disconnect any devices and drive in silence/with the AM/FM radio.
it's the consequence of not vertically integrating, having 20000 different ECUs each for a specific purpose and trying to nickel-and-dime suppliers on all of them, and the lack of urgency that traditional manufacturers and tier-1 suppliers have for adopting modern software development practices
I recently did some testing of Bluetooth connecting an Android phone to a reasonably new Mercedes. Being a "luxury" brand, you'd think the Mercedes would have good infotainment software, but I found at least 5 bugs in the GUI in about 1 hour of testing, and I wasn't particularly looking for bugs, just testing what I thought would be the happy paths for Bluetooth connection. The Android phone, on the other hand, did its job flawlessly.
I know software and embedded systems well enough to know all of the issues I found were preventable, if anyone cared.
The car seems well built in many other respects. It doesn't look like the problem is engineering ability.
(See also: Set-top box GUIs that are painfully slow to render menus, scroll, search etc. on hardware that I know can render 10-100x faster when programmed to.)
As you alluded to, the answer is they don't care. Car companies look at software like it's any other line item on the BOM. Like a bolt or a gasket: Source it as cheaply as possible and spoon it onto the product somewhere on the assembly line. I see fit-and-polish mistakes all the time in car infotainment. Text that can't handle unicode, icons that are misaligned by 1 or more pixels, connections dropping and coming back, audio mixing problems, things drawing outside their "windows." Nobody cares--they get to the drop dead date when the software needs to be on the assembly line, hand it over, and start flashing devices.
They’re being cheap. None of it is vertically integrated and it’s all outsourced to the cheapest third world contractor they can find.
I worked at a company that did software for these connected infotainment system. They cost cut those things to the bone, minimal ram, minimal cpu, shit screens. Even in the high end models.
Gotta remember that the car radio has always been a cheap gimme.
Gotta remember that the car radio has always been a cheap gimme.
Not really. Some of the hardware that you could get in the 1950s-1970s timeframe was great. Heavy chrome knobs and bezels, permeability-tuned front ends with separate RF stages... electromechanical mechanisms that seemed like witchcraft when I took them apart as a kid, and would still be cool to play with today.
You get a new device every year and teams of professionals are constantly churning updates for it.
With cars, you don't get to get a new device, it has to be consistent and keep working and you had better make it all work with a skeleton crew.
Someone correct me if I am wrong, we've haven't heard that Tesla OTA updates bricking people's cars.
They implemented a dual redundant system similar like the dual BIOS for motherboard since 1999.
I never heard of this and follow Tesla groups/communities/forums/etc. for over 10 years. At most you'll hear one or another person complaining about having initiated the update process and suddenly getting annoyed because they find out they need to go somewhere (it might take an hour).
They also do a fuckton of HIL testing for onboard software and in general approach testing like a software company instead of a bean counter company
Literally the first umpteen hits on Google says this happens with Tesla:
https://teslamotorsclub.com/tmc/threads/tesla-software-updat...
I can't find any news stories about Tesla's OTAs bricking cars? Especially nothing mid-drive like we're seeing with Jeep. That'd be a pretty major story considering how every Tesla story makes headlines.
That's one person claiming an update bricked their car, but it's unclear if that was due to a bad software update or a hardware failure that coincided with the update. Tesla usually explains what they fixed, so it's odd that the poster never replied with more details.
Even if every software update was perfect, you would see individual stories like the one you linked to. There are millions of Teslas in the world, and they all get updates frequently, so a hardware failure will sometimes coincide with a software update. If a bad update were shipped to customers, it would be a story similar to this Jeep issue: thousands of cars affected at once, lots of furious customers, and news articles about the failure.
Discussion from over the weekend: https://news.ycombinator.com/item?id=45558318
So how long until software in cars is treated with the same seriousness and rigor as the software in airplanes?
That's what MISRA C [1] is sort of meant to be.
[1]: https://en.wikipedia.org/wiki/MISRA_C
I suspect that we will see it when an OTA update very rapidly causes hundreds of deaths.
When it kills one of the Oligarchs' daughters.
And… Stellantis is up 3.5% right now in public trading. Nothing makes sense anymore haha
It is likely an unrelated correction. They are still down -7.92% over five days; this is just making it so they aren't -11.42%.
It's getting priced like an easy-to-fix recall that affects some cars of a specific model for one of their brands.
"no such thing as bad publicity" maybe?
It's a bug. Why should a software bug have an effect on a manufacturer's stock price? It's not like the update caused brake failures or something.
Well, given that the article says it caused powertrain failures on the highway, I'd say it's severe enough that it should absolutely cause the manufacturer's stock to drop.
Learning that Stellantis pushes bugs that cause power and engine failure while driving should decrease demand for their cars.
> or something
Maybe do some research into the problem you're confidently asserting was trivial / read the article you're commenting on:
"...others claim to have experienced a powertrain failure at highway speeds."
https://www.reddit.com/r/Jeep/comments/1o47064/jeep_4xe_shut...
I'm surprised this doesn't happen more often.
I own a Land Rover and their system software also ("Pivi") seems to have tons of little quirks and issues.
Sometimes the cameras won't work unless you restart Pivi. One time, the entire car wouldn't respond to locking (via the app, keyfob, in-car buttons) unless you disconnected the battery and waited ~30 mins. Many people complain that they can't even successfully upgrade between Pivi releases. It'll error out a lot of times and they have to restart the process.
(Would be interested to know the kind of tech they're using if anybody's familiar with it!)
Software updates for cars should be rare, with the infotainment systems physically isolated from the rest of the car (aside from constant/switched power). And if it needs updates for whatever reason (eg a recall), they should be done at a dealership - not over the air.
Bricking a car with an over-the-air software update is stupid and unacceptable behavior. Stuff like this is why I'm actually kind of happy to be driving a 20 year old car still.
I looked at the Jeep awhile back but their safety ratings were absolutely terrible. I’ve been in car accidents aplenty so that was a deal breaker for me. Just about the only real electronic thing in my Lexus is an ECU which rarely goes and can be replaced relatively easily . Forget updating the maps and other nonsense. Just drive the damn thing and do mechanical maintenance and a little grease.
Just wrote about my battles with software updates[1]. Without getting into what should or shouldn’t happen, I’ll express my sympathy for the Jeep team and the owners. Software updates are hard.
[Edit: a commenter notes that Jeep’s parent has recently mandated LLM usage in development. Let’s hope these two are unrelated and that we aren’t going to start seeing more catastrophic failures like this over the next few months as people work out the limitations of this sorcery.]
[1] https://tritium.legal/blog/update
So many layers of failure here. It points to very suspect architecture and development practices, the bad update is just sprinkles on top.
The cars needs a partition for the running OS, and a second as backup, and "reboot to recovery partition" to fall back to in case the update breaks.
Hah, curious to think that cars now have bootloaders...
Cars probably have multiple bootloaders even. Surely there are at least two, one for the ECU and one for the infotainment system. Perhaps there are even more depending on how complex components like parking cameras etc. are.
I suppose some version of CTRL-ALT-DELETE is needed to reset the car's OS.
The first layer of failure was the decision to make the car computer-controlled.
That came after the decisions to reduce both costs and tailpipe emissions - both obvious worthy goals. Is the implementation that is flawed, not the idea.
Why would cars be the only thing we wouldn't manage with computers?
We could, but we shouldn't, because most software is crap. When the user is stuck with whatever software they got as a consequence of buying the machine they actually wanted, there's no incentive for the software not to be crap.
That doesn't even make sense in light of this article. The user isn't stuck with the original software at all... instead, due to an OTA update, they're stuck in the road.
To avoid power and engine failure on the highway after a bad software update.
Because they work fine without them.
I think if you compare a modern car with an ECU to a "traditional" car with manual ignition / carburetion system you will find that the modern one outperforms significantly on both power and fuel efficiency.
A few bad updates borked my Linux install and cost me hours of potential work and left me with a deep fear of installing updates. No idea what I'd do if I had a car that needs updates
I have a Jeep Wrangler JK. I considered buying a 4XE. Now I'm glad I didn't.
Mopar and dead car.
lol, i love jeeps, but you have to buy japanese cars with software from 60s, hopefully this does not change for another 80 years
A side note: is it just me or not but I prefer my new car to still providing tactile buttons to control things? I get that smart cars like Tesla wants to push touch-screen on everything but the control mechanism just feels clunky for me. When I sit on a car with physical buttons, I know exactly where they are and what they do when I press & turn them.
[dupe] https://news.ycombinator.com/item?id=45558318
leadership problem, as everywhere. old grampas that used to manually draw gears on paper now have to "strategically align" a huge corporation that has to deal with new shiny and complicated things like software and they all have zero fucking clue. at least with cars you can always try to safely stop, with planes - not so possible. this will also soon creep up there.
Truly can’t believe the shit coming out of these automakers now that AI assisted coding has become so commonplace in the industry.
Do we have any actual evidence that AI-assisted coding has anything to do with this?
https://www.stellantis.com/en/news/press-releases/2025/octob...