Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Single Component Can Brick Older Teslas

thedrive.com

Post reply on HN
Print view View original

81–90 of 106 posts

Re: A Single Component Can Brick Older Teslas

#81

Post by nemosaltat »

zaroth wrote: post #57
fucking_tragedy wrote: post #37

Earlier quoted context omitted.

> At the same time, it is pretty amazing they are remotely logging into your car to review system logs and diagnose the issue, and then pushing firmware updates to try to remedy the problem. It really isn't if it didn't fix the problem and they used it as an excuse to not follow up with their scheduled appointment. This would be like if I was fixing a bug, SSH'd into a live instance, didn't fix anything, then filed t…

But that's not what happened. nemosaltat didn't provide a complete chronology, but did say they had a service appointment on 4-13, received a FW update on 4/4, and a rep contacted them on 4/5 saying they reviewed the logs and the new FW fixed the issue. Then nemosaltat responded to Tesla; > Ok- I’d still like to understand what happened....It’s significantly less understandable, that the issue mysteriously resolves i…

I think gatherhunter hit it dead on. It’s metrics thing and they wanted to close tickets. I sincerely doubt the FW update they pushed had anything to do with the issue being resolved. It is common practice (according to Tesla forums) to push FW before a service appointment just to be sure the car has the latest version before it goes in.

The sketchy part for me is the inconsistency between the Service Tech and the rep who canceled the appointment. The tech (HW flaw) was very convincing, and the rep (SW flaw) was very vague. The functionality did not return immediately after the FW update, and I think the timing may have been coincidental. It’s even more frustrating that they weren’t/aren’t the least bit apologetic or accommodating. I can’t seem to get a straight answer on why basic cruise can’t be restored via a FW patch while I wait for my appointment. Is the same hardware/software required for Cruise on Model 3s without NoAP and FSD?

Re: A Single Component Can Brick Older Teslas

#82

Post by Crosseye_Jack »

pbreit wrote: post #78
joezydeco wrote: post #59

Earlier quoted context omitted.

"Replacing a tiny part" is trivializing the work necessary to do this, in my opinion. eMMC flash is typically a ball-grid array part and it's soldered down hard to the motherboard of the processor. Almost every mobile device has the same setup and it's tricky and expensive to rework this chip. You need to carefully desolder it off the board, reball the part with the correct masking stencil, then reapply it without to…

So it's not fixable with software update?

The under lying issue can not be fixed with software. the eMMC is simply running out of write cycles. BUT they can mitigate the issue by not writing the syslog to the eMMC allowing the remaining write cycles to last much longer (This change could be done with a software update).

On the newer versions of the Media Unit instead of disabling writing the syslog to the eMMC they added more storage to the processor boards which isn't going to solve the problem just push it from appearing to further down the road.

Re: A Single Component Can Brick Older Teslas

#83

Post by Crosseye_Jack »

joezydeco wrote: post #77
solarkraft wrote: post #65

Earlier quoted context omitted.

Luckily Tesla isn't very space or money limited with this pretty ciritcal component. An SD card would make life easier for everyone, as the logs could be easily read out and the booting OS changed - yay for actually owning the device you paid for. But wait: That's not something Tesla wants.

eMMC is literally the equivalent of an SD card soldered down to the PCB. It’s more mechanically stable and also way faster. In my experience, removable SD wouldn’t hold up to an application like this.

The actual logs Tesla seem interested in (including the stills of things they are interested in) get written to a removable SD Card before getting uploaded to the Tesla mother-ship.

You are not wrong that a soldered eMMC (compared to a eMMC held by a ZIF or a removable SD Card) would be more mechanically stable and potentially faster (A fast uhs-ii sd card compared to a cheap slower eMMC for example), just saying that Tesla already use removable SD cards in production cars today (which doesn't brick your media unit if it dies or dislodges).

Storing the OS on a removable SD Card is (imo) a bad idea, but if they are really interested in these logs they already have a easier to replace removable storage "drive" they could write the logs to instead of burning up the eMMC.

Re: A Single Component Can Brick Older Teslas

#84

Post by castratikron »

JaimeThompson wrote: post #69
castratikron wrote: post #14

Earlier quoted context omitted.

Problems with an SD card is that it will wear out even faster than an eMMC due to lack of FTL, and also friction fit instead of soldered connections means the part is susceptible to wiggling out of the socket and also corrosion between the contactors. Not sure of the benefits of an SSD over an eMMC. Guessing the former will have a more sophisticated FTL and other things.

Multiple companies make high endurance SD cards for things like dash cams and automobiles. Take, for example, WD https://www.westerndigital.com/products/embedded-removable-f...

Didn't know that, thanks.

But there's still the problem of a friction fit connection in an environment with a lot of vibration and contaminants. Seems like these would be good for use cases where the part needs to be removable and also have high write cycles, like a dash cam or data logger that has no other interface to a PC.

Also the eMMC interface exposes quite a bit of the underlying hardware (see mmc-utils on Linux). With SD cards I think you're generally stuck with some kind of SMART interface.

Re: A Single Component Can Brick Older Teslas

#85

Post by m463 »

stefan_ wrote: post #4

This is rather disingenuous. Of course Tesla doesn't replace the eMMC; that's a soldered on chip. No car manufacturer in the world does chip level electronics repairs. Instead you replace the entire board, and I have no doubt you can get this repair from Tesla. That's not to entirely excuse Tesla. If you have a modern operating system running from flash that needs to work for 20+ years as people expect from a car, it…

> all logging and runtime data written only to a RAM disk, system and user data on entirely separate partitions, etc.

I suspect "we can fix it in a software update" lowers the priority of actually shipping this design.

and then add in "autopilot this year" and "model y coming soon", you get indefinite postponement.

Re: A Single Component Can Brick Older Teslas

#86

Post by [deleted] »

joezydeco wrote: post #59
BluSyn wrote: post #7

This issue is best explained on Rich Rebuilds' recent video here: https://youtube.com/watch?v=o-7b1waoj9Q&t=525 This has been a known issue for some time, and it's amazing Tesla hasn't fixed it yet. As explained in the video, the issue is simply the base linux OS has full syslog's still enabled, which over time burn own the eMMC chip used for storage. Currently Tesla just replaces the entire media unit when this happ…

"Replacing a tiny part" is trivializing the work necessary to do this, in my opinion. eMMC flash is typically a ball-grid array part and it's soldered down hard to the motherboard of the processor. Almost every mobile device has the same setup and it's tricky and expensive to rework this chip. You need to carefully desolder it off the board, reball the part with the correct masking stencil, then reapply it without to…

[deleted]

Re: A Single Component Can Brick Older Teslas

#87

Post by m463 »

nemosaltat wrote: post #18

Model 3 owner here. Having same issue as back in March. No AP, no FSD and most frustratingly, no cruise control. Multiple TS steps over phone with no resolution. The earliest appointment was months out so I drove into a service center. A frazzled, yet personable, tech assured me that the issue was NOT from the recent FW update but rather a hardware malfunction with the left repeater. I pressed for details- is it wiri…

I don't think you have the issue mentioned in the article.

If they could pull your logs, your emmc did not fail.

(not that they are handling other issues well)

Re: A Single Component Can Brick Older Teslas

#88

Post by m463 »

TazeTSchnitzel wrote: post #61

I'm sure the Model 3 having a single multifunction screen that does everything (combining entertainment with critical driving features) will go great .

Actually, every control in the model 3 is used for multiple functions (for example drive selector also controls autopilot, scroll wheel controls many functions, etc)

I guess you'll just have to drive with your phone.

Re: A Single Component Can Brick Older Teslas

#89

Post by naikrovek »

Crosseye_Jack wrote: post #82
pbreit wrote: post #78

Earlier quoted context omitted.

So it's not fixable with software update?

The under lying issue can not be fixed with software. the eMMC is simply running out of write cycles. BUT they can mitigate the issue by not writing the syslog to the eMMC allowing the remaining write cycles to last much longer (This change could be done with a software update). On the newer versions of the Media Unit instead of disabling writing the syslog to the eMMC they added more storage to the processor boards…

The problem is that there is significant write amplification as the used space on the partition approaches the total space of the partition.

To say that the choice to use eMMC is the fault is to say that ANY use of flash storage is a faulty decision.

If there is sufficient space on the drive to mitigate write amplification, then the media will last a very long time indeed. Longer than the drivetrain, certainly.

The problem is that Tesla doesn't delete logs it has collected from the cars. All Tesla vehicles (as I understand it) upload their logs more or less continually. Currently, the firmware on the cars doesn't delete those logs once they've been uploaded, presumably so that they're available when it is time to service the vehicle.

The disk filling up due to bad log file hygiene, coupled with the fact that the storage is flash-based, together cause the issue.

Re: A Single Component Can Brick Older Teslas

#90

Post by naikrovek »

joezydeco wrote: post #77
solarkraft wrote: post #65

Earlier quoted context omitted.

Luckily Tesla isn't very space or money limited with this pretty ciritcal component. An SD card would make life easier for everyone, as the logs could be easily read out and the booting OS changed - yay for actually owning the device you paid for. But wait: That's not something Tesla wants.

eMMC is literally the equivalent of an SD card soldered down to the PCB. It’s more mechanically stable and also way faster. In my experience, removable SD wouldn’t hold up to an application like this.

An SD card might work fine if it is mounted properly. The slot found on a Raspberry Pi would not at all be sufficient.

If it were held in and held against some robust contact pins it likely would be fine, even under automotive conditions. There are SIM card slots that are used on heavy vehicle computers and controllers (the ones that run the engine and so on) that are robust enough to hold a SIM card in place for years of excessive daily vibration.

Post reply on HN