Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Sorcerer (YC S24) – Weather balloons that collect more data

news.ycombinator.com

Post reply on HN
Print view View original

151–160 of 173 posts

Re: Launch HN: Sorcerer (YC S24) – Weather balloons that collect more data

#151

Post by jviotti »

f549abd01 wrote: post #138
jviotti wrote: post #84

Earlier quoted context omitted.

CBOR is a schema-less binary format. JSON BinPack supports both schema-less (like CBOR) and schema-driven (with JSON Schema) modes. Even on the schema-less mode, JSON BinPack is more space-efficient than CBOR. See https://benchmark.sourcemeta.com for a live benchmark and https://arxiv.org/abs/2211.12799 for a more detailed academic benchmark

Thanks for linking the benchmarks. I appreciate the work on shaving additional bytes especially in cases where every byte matters. Real savings seem to be in the schema-driven mode. Comparing a "realistic", schemaless payload for a general storage use-case (eg. the config examples), it looks pretty even with CBOR. E: my bad, BinPack is getting more efficient with larger payloads https://benchmark.sourcemeta.com/#json…

Exactly! The real hardcore savings will always be when you pass a schema, as JSON BinPack uses that to derive smarter encoding rules.

However, schema-less is very useful too. The idea with supporting both is that clients can start with schema-less, without bothering with schemas, and already get some space-efficiency. But once they are using JSON BinPack, they can start incrementally adding schema information on the messages they care about, etc to start squeezing out more performance.

Compare that with i.e. Protocol Buffers, which pretty much forces you to use schemas from the beginning and it can be a bit of a barrier for some projects, mainly at the beginning.

Re: Launch HN: Sorcerer (YC S24) – Weather balloons that collect more data

#152

Post by jviotti »

michaelmior wrote: post #111
lajr wrote: post #73

Earlier quoted context omitted.

This looks promising! One of the important aspects of protocol buffers, avro etc is how they deal with evolving schemas and backwards/forward compatibility. I don't see anything in the docs addressing that. Is it possible for old services to handle new payloads / new services to handle old payloads or do senders and receivers need to be rewritten each time the schema changes?

A lot of people already think about this problem with respect to API compatibility for REST services using the OpenAPI spec for example. It's possible to have a JSON Schema which is backwards compatible with previous versions. I'm not sure how backwards-compatible the resulting JSON BinPack schemas are however.

Great seeing you over here Michael :) For other people reading this thread, Michael and I are collaborating on a paper covering the schema compiler I've been working on for JSON BinPack. Funny coincidence!

Re: Launch HN: Sorcerer (YC S24) – Weather balloons that collect more data

#153

Post by worldvoyageur »

DoctorOetker wrote: post #144

Is there a reason weather balloons can't be designed to stay aloft for much longer? is it really impossible to miniaturize gas separators in weight to replenish the lighter fraction of molecules to compensate for the slow leakage?

In reading the documents google's Loon made available before shutting down (and saying that hydrogen airships were the way to go) [https://storage.googleapis.com/x-prod.appspot.com/files/The%...], the bottleneck on staying up longer was helium loss. Lots of things, mechanical failure etc, caused flights to end early. But the ultimate limit seems to be either helium loss or the impact of helium absorption on seals etc causing failure. Helium is a very small molecule, so very hard to keep confined.

However, as they learned from experience, Loon was slowly increasing their upper limit on how long their balloons could stay up.

Re: Launch HN: Sorcerer (YC S24) – Weather balloons that collect more data

#154

Post by jviotti »

jviotti wrote: post #17

Very cool! How are the balloons transferring telemetry back to earth for analysis, etc? Asking because my research at the University of Oxford was around hyper space-efficient data transfer from remote locations for a fraction of the price. The result was an award-winning technology ( https://jsonbinpack.sourcemeta.com ) to serialise plain JSON that was proven to be more space-efficient than every tested alternative…

It surprised me how popular this message got. I love nerding out about binary serialization and space-efficiency and great to see I'm not the only one :)

If you want to get deeper, I published two (publicly available) deep papers studying the current state of JSON-compatible binary serialization that you might enjoy. They study in a lot of detail technologies like Protocol Buffers, CBOR, MessagePack, and others that were mentioned in the thread:

- https://arxiv.org/abs/2201.02089

- https://arxiv.org/abs/2201.03051

Hope they are useful!

Re: Launch HN: Sorcerer (YC S24) – Weather balloons that collect more data

#155

Post by jviotti »

ok_dad wrote: post #90
leeoniya wrote: post #46

Earlier quoted context omitted.

> JSON BinPack is space-efficient, but what about runtime-efficiency? > When transmitting data over the Internet, time is the bottleneck, making computation essentially free in comparison. i thought this was an odd sales pitch from the jsonbinpack site, given that a central use-case is IoT, which frequently runs on batteries or power-constrained environments where there's no such thing as "essentially free"

> batteries or power-constrained environments I would imagine that CPUs are much more efficient than a satellite transmitter, probably? I guess you'd have to balance the additional computational energy required vs. the savings in energy from less transmitting.

Yeah, it all depends very much, given how huge the "embedded/IoT" spectrum is. Each use case has its own unique constraints, which makes it very hard to give general advice.

Re: Launch HN: Sorcerer (YC S24) – Weather balloons that collect more data

#156

Post by DoctorOetker »

worldvoyageur wrote: post #153
DoctorOetker wrote: post #144

Is there a reason weather balloons can't be designed to stay aloft for much longer? is it really impossible to miniaturize gas separators in weight to replenish the lighter fraction of molecules to compensate for the slow leakage?

In reading the documents google's Loon made available before shutting down (and saying that hydrogen airships were the way to go) [ https://storage.googleapis.com/x-prod.appspot.com/files/The%... ], the bottleneck on staying up longer was helium loss. Lots of things, mechanical failure etc, caused flights to end early. But the ultimate limit seems to be either helium loss or the impact of helium absorption on seals e…

Sure there will be leakage.

Let's look at the composition vs height:

https://en.wikipedia.org/wiki/Atmosphere_of_Earth#/media/Fil...

I understand helium will be sparse at low heights, but at each height there is a diversity of species, among which will be a lighter one. Could a balloon be oversized so that instead of using helium, at each height an 80% fill of the locally (that height) lightest gas could keep the balloon afloat. Oversized for this but also for the added weight for separating equipment for the wanted gas and associated panels to power it.

I.e. what prevents a balloon from flying indefinitely? At least up until radiation damage of the balloon membrane...

Have such attempts been made and what were the lessons?

Re: Launch HN: Sorcerer (YC S24) – Weather balloons that collect more data

#157

Post by tndl »

DoctorOetker wrote: post #144

Is there a reason weather balloons can't be designed to stay aloft for much longer? is it really impossible to miniaturize gas separators in weight to replenish the lighter fraction of molecules to compensate for the slow leakage?

One of the first ideas we explored was putting an electrolyzer on the balloon to replenish hydrogen over time. Unfortunately right now, for balloons our size and power budget it's just not feasible. And actually, we can get a pretty low leakage rate with our materials which lets us stay aloft for a really long time, but eventually the UV degradation becomes to extreme.

Re: Launch HN: Sorcerer (YC S24) – Weather balloons that collect more data

#159

Post by tndl »

ramonyc wrote: post #143

Neat :) I'm involved in a few projects focused on vertical profiling in the coastal urban boundary layer. I saw in a response you said the balloons will periodically return to sea level and ascend (which sounds like a fun design challenge by itself.) Will you be doing so near populated areas as well? Good luck!

Very interesting, I'd love to hear more about it! In short, yes, we plan to do descents near urban areas if there's a route where we can go down to a safe height and stay away from any airspace. What cities are you looking at right now?

Re: Launch HN: Sorcerer (YC S24) – Weather balloons that collect more data

#160

Post by the__alchemist »

jviotti wrote: post #17

Very cool! How are the balloons transferring telemetry back to earth for analysis, etc? Asking because my research at the University of Oxford was around hyper space-efficient data transfer from remote locations for a fraction of the price. The result was an award-winning technology ( https://jsonbinpack.sourcemeta.com ) to serialise plain JSON that was proven to be more space-efficient than every tested alternative…

Why this over a compact, data-specific format? JSON feels like an unnecessary limitation for this company's use case. I am having a hard time believing it is more space-efficient than a purpose-built format.
Post reply on HN