Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The software that flies SpaceX rockets and starships

stackoverflow.blog

Post reply on HN
Print view View original

101–110 of 112 posts

Re: The software that flies SpaceX rockets and starships

#101

Post by lnsru »

wikibob wrote: post #89
bodhiandpysics1 wrote: post #86

Earlier quoted context omitted.

They actually use more than three now. The basic architecture is that computers are organized in "strings". Each string is doubled. If the two don't agree, the result is chucked, and the system reboots. Each of these "strings" then participates in a quorum. The result is extreme reliability, with relatively fast convergence.

Fascinating. Could you write more about this, or do you have suggestions for what to read to learn more?

Functional safety standards like ISO 61508. But I am not sure if you will like it.

Re: The software that flies SpaceX rockets and starships

#102

Post by MPSimmons »

elteto wrote: post #46
numpad0 wrote: post #38

How are distributed realtime systems like a rocket is synchronized? Are there central timeservers and allocated transmission slots and timestamped query-response action messaging or is it just small individual realtime nodes making their own decisions? I’ve been wondering about this for a year or so because I can’t seem to find one that matches my description, other than concept drawings on NIST 4D/RCS paper, but the…

A popular solution is PTP [0]. Once you have synchronized hw clocks you lay your software synchronization on top of that. [0] https://en.m.wikipedia.org/wiki/Precision_Time_Protocol

Yes. This does require all of the network devices between the nodes to also implement PTP (otherwise you fall back to NTP-like behavior) but this isn't a problem if you're greenfielding the implementation.

Re: The software that flies SpaceX rockets and starships

#103

Post by icegreentea2 »

procarch2019 wrote: post #71
icegreentea2 wrote: post #64

Earlier quoted context omitted.

I am quite interested to see how they deal with regulations and standards (I'm in medical device space). I scanned through their job postings and I can't find a single standard listed (this is up to the Sr Engineer/Developer positions), as well as a lack of dedicated software QA/test positions. Obviously they are doing some amazing, and quite high quality work, so I am quite interested in what their internal setup is…

Yes, considering how fast they move I’m wondering if it’s closer to self regulated/peer quality control. Some QA/QC can make work come to a screeching halt in biopharma space (makes sense when a “batch” in the cell and gene therapy space is a single person and their wellbeing). It’s not like regulation and quality have not fallen by the wayside or been too relaxed for other aerospace companies, especially in recent h…

Even in a self-regulated/peer quality control context, I think it's quite amazing to not have people tasked for dedicated (or more focused towards) testing and QA, even within the software/engineering group.

I wonder how developer/engineers are expected to split their focus between design and development of their own areas of responsibilities, and doing peer review and testing of other people's areas of responsibilities. One of the dangers of this setup is always that people (and their managers) prioritize output of their own areas, resulting in neglecting testing and review of other areas. It's certainly possible to make the setup, but it takes good people (and team cohesion/morale) up and down the chain to make this work.

I definitely agree that it's a fine line.

Re: The software that flies SpaceX rockets and starships

#104

Post by 908B64B197 »

api wrote: post #72
horstmeyer wrote: post #23

Earlier quoted context omitted.

MongoDB is web scale, not universe scale.

It's interesting how much contempt people have around here for one of the most successful open source oriented companies in recent memory. Most of the bad things about MongoDB are actually bad things about the NoSQL fad, and that fad was huge around here a few years ago.

Mongo and NoSQL, to me, are really symptoms of the Bootcamp craze that happened a few years ago.

It's way easier to understand than relational databases, can be simply browsed and retrieved as json (usefull when your only language is JS) and had good adapters for the cookie cutter frameworks used at the time. And you don't need to bother with a schema or joins.

There are use-cases for NoSQL but it's not a once size fits all solution.

Re: The software that flies SpaceX rockets and starships

#105

Post by sam_bristow »

V_Terranova_Jr wrote: post #77
fallingmeat wrote: post #28

Interesting they seem to ad-hoc the lifecycle processes, or at least no mention of any industry standards mandatory (RTCA/DO-178C, ISO26262 or even sys eng like SAE ARP4754, etc). Also, no talk of formal methods (formal modeling or formal verification, a la DO-333 with something like Simulink Design Verf or TLA+). Works for them obviously!

It's been ~a decade since I was knowledgeable in SpaceX software development details, but at least then, they were not the kind of shop that emphasized rigor and formality. Their culture is oriented toward rapid implementation & test, pragmatism, and empiricism. That has its pluses and minuses. My personal view is that more of the industry, including SpaceX, should be moving toward engineering things to be more corre…

Are the properties that have been proven for seL4 particularly relevant for avionics systems?

I thought they mainly focussed on process isolation and security. A flight computer would be much more concerned about deterministic performance and real-time guarantees.

Re: The software that flies SpaceX rockets and starships

#106

Post by icefo »

7thaccount wrote: post #39
tangoed wrote: post #8

Earlier quoted context omitted.

There is a Reddit AmA by SpaceX software team that has some important details!

Yep, this and there is a Quora question somewhere I think iirc. It's mostly LabView I think, which is an interesting choice given all the hate it gets. For those that don't know, LabView is a graphical (drag and drop) language for the most part where you interconnect various hardware components together. The advantage is that a lot of labs need to plug a lot of devices and sensors to run an experiment and LabView has…

I think the main advantage of LabVIEW is not the language but the hardware it runs on. While it is ungodly expensive it's very reliable in a wide range of temperature & vibration.

I used LabVIEW to code the control / data logging of a prototype racecar and the hardware only broke once when we probably zapped it with a nasty ESD or reverse polarity connection somewhere. We never figured it out and didn't ask when we sent the part to be repaired.

Programming in LabVIEW is something I wouldn't wish on my worst enemy though. It's extremely hard to debug and time consuming to refactor. I always felt that I was 10-50x slower than when I would code in a standard text-based programming language. Working collaboratively also put you in a world of pain because, at least when I used it and I think it's still the case, LabVIEW doesn't support git natively so you have to use an external tool. They have a tool to merge / diff their proprietary binary files but the UX is terrible and I think even if they really tried to do something good, diffing diagrams (which is what a LabVIEW program is IMO) is going to stay hard.

From my experience the people that code in LabVIEW are not software engineer and I had discussions where I was arguing that yes the channel feature is broken but that's not a reason to put all the variables used to communicate between components in a single global variable file for example.

But once you fought against the language long enough and have something that works you can be pretty sure it will keep working forever.

Re: The software that flies SpaceX rockets and starships

#107

Post by idealmedtech »

icefo wrote: post #106
7thaccount wrote: post #39

Earlier quoted context omitted.

Yep, this and there is a Quora question somewhere I think iirc. It's mostly LabView I think, which is an interesting choice given all the hate it gets. For those that don't know, LabView is a graphical (drag and drop) language for the most part where you interconnect various hardware components together. The advantage is that a lot of labs need to plug a lot of devices and sensors to run an experiment and LabView has…

I think the main advantage of LabVIEW is not the language but the hardware it runs on. While it is ungodly expensive it's very reliable in a wide range of temperature & vibration. I used LabVIEW to code the control / data logging of a prototype racecar and the hardware only broke once when we probably zapped it with a nasty ESD or reverse polarity connection somewhere. We never figured it out and didn't ask when we s…

I'd disagree about debugging; you can create custom probes for any given wire that run arbitrary code whenever the data gets updated, that combined with the ability to "hold wire values" aka program state, after execution has finished (or even paused) is a godsend.

Re: merging; yes absolutely, this is totally broken. NI should either release the file format for VIs, or offer conversion to something saner like XML, which you could easily build a nice git-diff tool for.

I maintain that LabVIEW is a good language with a terrible teaching problem; you can do so much if you just master the queued message handler architecture, and you can do things in a way that are extendible and maintainable for years afterwards

Re: The software that flies SpaceX rockets and starships

#108

Post by marsokod »

inamberclad wrote: post #42
numpad0 wrote: post #38

How are distributed realtime systems like a rocket is synchronized? Are there central timeservers and allocated transmission slots and timestamped query-response action messaging or is it just small individual realtime nodes making their own decisions? I’ve been wondering about this for a year or so because I can’t seem to find one that matches my description, other than concept drawings on NIST 4D/RCS paper, but the…

There's typically a central time source sending a 1Hz pulse into the main computer(s). The sensors will use something like Time Triggered Ethernet, RS-485, or a MIL-1553 bus.

For LEO missions, the source is now typically a GNSS receiver outputting a PPS line. Otherwise, you try to have a precise clock (TCXO, OCXO, and now CSAC are available) as a reference distributing the time to all the subsystems, and you correct the satellite time offset from the ground (similar to NTP).

Re: The software that flies SpaceX rockets and starships

#109

Post by procarch2019 »

icegreentea2 wrote: post #103
procarch2019 wrote: post #71

Earlier quoted context omitted.

Yes, considering how fast they move I’m wondering if it’s closer to self regulated/peer quality control. Some QA/QC can make work come to a screeching halt in biopharma space (makes sense when a “batch” in the cell and gene therapy space is a single person and their wellbeing). It’s not like regulation and quality have not fallen by the wayside or been too relaxed for other aerospace companies, especially in recent h…

Even in a self-regulated/peer quality control context, I think it's quite amazing to not have people tasked for dedicated (or more focused towards) testing and QA, even within the software/engineering group. I wonder how developer/engineers are expected to split their focus between design and development of their own areas of responsibilities, and doing peer review and testing of other people's areas of responsibilit…

I’ll preface this with “I think I’m good people.”

Context switching is hard especially when understaffed and most certainly when doing different types of work. I’m still “executing” engineering projects, but I’m also performing inside sales functions, helping advance and maintain our dev systems, and now manage 2 functional groups. Unfortunately stuff does fall by the way side all the time.

Prioritizing things is key, but at the end of the day there is a capacity to what one can do. It takes good management skills to realize the capacity of the employees and when things need to change. Luckily for me I’m working with groups in all three of the areas I was working in to offload those extra responsibilities to other people so I can focus on what I’m supposed to be doing (managing).

It would be interesting to see how they operate.

Re: The software that flies SpaceX rockets and starships

#110

Post by ethbr0 »

procarch2019 wrote: post #109
icegreentea2 wrote: post #103

Earlier quoted context omitted.

Even in a self-regulated/peer quality control context, I think it's quite amazing to not have people tasked for dedicated (or more focused towards) testing and QA, even within the software/engineering group. I wonder how developer/engineers are expected to split their focus between design and development of their own areas of responsibilities, and doing peer review and testing of other people's areas of responsibilit…

I’ll preface this with “I think I’m good people.” Context switching is hard especially when understaffed and most certainly when doing different types of work. I’m still “executing” engineering projects, but I’m also performing inside sales functions, helping advance and maintain our dev systems, and now manage 2 functional groups. Unfortunately stuff does fall by the way side all the time. Prioritizing things is key…

From seeing this work and not work at different companies, I've decided at root it's a company culture issue.

Specifically, "How does my company deal with failure?"

Which is another way of saying "Are people disincentivized from telling the truth at my company?"

You can never have a process that requires truth-to-power as a regular occurrence that's successful in an environment where there are career penalties to speaking uncomfortable truths.

That's why legacy providers & government have engineered a system that accepts +50% wasted time, in exchange for not requiring truth because almost everything is checked and verified.

Good culture: We failed. I succeeded.

Bad culture: You failed. I succeeded.

Post reply on HN