Live data from Hacker News

Launch HN: Sorcerer (YC S24) – Weather balloons that collect more data

news.ycombinator.com

71–80 of 173 posts

Re: Launch HN: Sorcerer (YC S24) – Weather balloons that collect more data

#71
So the equivalent of these balloons in oceanography are called ARGO floats, which similarly cannot be driven laterally but can control their own depth like a submarine. So far millions of timeseries have been collected across the world ocean using these floats.

https://argo.ucsd.edu/

One difference though is that the ARGO floats are unfortunately not recycled, and just wash up on various beaches. (I'm curious whether you think you can realistically collect many of these mini balloons?)

If you do want to control the lateral position of fleets of sensors, oceanographers also now have "gliders", which are basically small powered drone submarines. These are used by a few groups, but most of the gliders in the world are operated by the US Navy, who launch them out of torpedo tubes to survey local ocean conditions (which is badass).

https://oceanservice.noaa.gov/facts/ocean-gliders.html

The recorded measurements present an interesting data assimilation challenge - they record data along 3D trajectories (4D including time), sampling jagged and twisting lines through the 4D space. But we normally prefer to think of weather/ocean data as gridded, so you need to interpolate the trajectory data onto the grid, whilst keeping the result physically-consistent. Oceanographers use systems like ECCO for ocean state estimation, which effectively find the "ocean of best fit" to various data sources.

https://www.ecco-group.org/

Interestingly ECCO uses an auto-differentiable form of the governing equations for the ocean flow to ensure that updates stay physically consistent. This works by using a differentiable ocean fluid model called [MITgcm](https://github.com/MITgcm/MITgcm) to perform runs which match experimental data as closely as possible, and minimizing a loss function through gradient descent. The gradient is of a loss function (error) with respect to model input parameters + forcings, which is calculated by running MITgcm in adjoint mode - i.e. automatic differentation. Therefore this approach is sort of ML before it was cool (they were doing all this well before the new batch of AI weather models). See slides 9-18 of this deck for a nice explanation

https://firebasestorage.googleapis.com/v0/b/firescript-577a2...

The trajectory data is also interesting because it's sort of tabular, but also you often want to query it in an array-like 4D space. You could also call it a "ragged" array. We have nice open-source tools for gridded (non-ragged) arrays (e.g. xarray and zarr, and the pangeo.io project) but I think we could provide scientists with better tools for trajectory-like data in general. If that seems relevant to you I would love to chat.

P.S: Sorceror seems awesome, and I applaud you for working on something hard-tech & climate-tech!

Re: Launch HN: Sorcerer (YC S24) – Weather balloons that collect more data

#72
post #70

Interesting. I have two questions: 1) What parameters are you measuring ? Did you think about also measuring gases? 2) What's your business model?

1. Wind speed/direction, air pressure, temperature, humidity, and solar irradiance. We are considering doing atmospheric composition as well, there are a couple of partners that are interested in GHGs, VOCs, water vapor, etc. We have the weight budget to do it, but we haven't flown any production payloads with those sensors yet.

2. The US National Weather Service actually has a commercial data buy program called MESONET [0], where they buy weather data from both academic and commercial partners. We're in the process of becoming one of their commercial partners now. Once we are, a single balloon will pay for itself in a matter of days with the data it can collect, which will let us scale up the number of systems we have deployed. The data we collect right now also lets us build niche weather forecast products like the stratospheric wind forecast mentioned in the post. Once we have enough balloons up, we can start producing useful weather forecast products at a regional and then global scale.

[0]: https://nationalmesonet.us/

Re: Launch HN: Sorcerer (YC S24) – Weather balloons that collect more data

#73
post #17

Very cool! How are the balloons transferring telemetry back to earth for analysis, etc? Asking because my research at the University of Oxford was around hyper space-efficient data transfer from remote locations for a fraction of the price. The result was an award-winning technology ( https://jsonbinpack.sourcemeta.com ) to serialise plain JSON that was proven to be more space-efficient than every tested alternative…

This looks promising! One of the important aspects of protocol buffers, avro etc is how they deal with evolving schemas and backwards/forward compatibility. I don't see anything in the docs addressing that. Is it possible for old services to handle new payloads / new services to handle old payloads or do senders and receivers need to be rewritten each time the schema changes?

Re: Launch HN: Sorcerer (YC S24) – Weather balloons that collect more data

#75
post #69
post #55

This sounds like super interesting and meaningful work. Are you hiring, or do you have any advice for your average software engineer on getting into this space?

We're not hiring right now, but definitely check back in a few months. As for advice, there's almost always a place for talented engineers. https://www.climatetechlist.com/ and https://jobs.climatebase.org/ both aggregate jobs at climatetech cos specifically if that's what you want to pursue.

Thank you, this is just what I was looking for!

Re: Launch HN: Sorcerer (YC S24) – Weather balloons that collect more data

#76
post #20

For those interested, check out Bill Brown, the grandfather of lightweight modern ballooning. Multiple circumnavigations have been achieved with his equipment. https://www.stratoballooning.org/membership#!biz/id/5f4d7b97...

Bill Brown is a legend! Love Lee Meadows with SBS as well - 767+ days is unreal

https://www.scientificballoonsolutions.com/news/

Re: Launch HN: Sorcerer (YC S24) – Weather balloons that collect more data

#77
post #36

This is awesome, how do you manage climbing and descending with a balloon. Are you compressing the gas on board or using thermals?

How we make it go up and down is the secret sauce :) I'm a hangglider guy, so I'd love to be using thermals, but I can say that's not how we do it right now

That’s what I figured :) Being able to control it with a one pound payload is very impressive.

Re: Launch HN: Sorcerer (YC S24) – Weather balloons that collect more data

#78

So the equivalent of these balloons in oceanography are called ARGO floats, which similarly cannot be driven laterally but can control their own depth like a submarine. So far millions of timeseries have been collected across the world ocean using these floats. https://argo.ucsd.edu/ One difference though is that the ARGO floats are unfortunately not recycled, and just wash up on various beaches. (I'm curious whether…

This is super interesting, I'd never come across ARGO before. Data assimilation is a similar problem for our data, and there currently exist systems for assimilating weather balloon observations into gridded reanalysis data (https://www2.mmm.ucar.edu/wrf/users/). One thing we believe, however, is that the reanalysis step in weather forecasting is unnecessary in the long term, and that future (ML) weather models will eventually opt to generate predictions based on un-assimilated raw data and will get better results in doing so.

That being said, trajectory-based data tooling could be super interesting to us. Let's definitely chat: austin@sorcerer.earth

And re: recovery, we're pretty confident we'll be able to recover the majority of our systems. Being in the air has the advantage that we can choose to 'beach' ourselves in a specific location, rather than the first place we run across land like with the buoys. At his previous company, Alex wrote a prediction engine able to get similar balloon systems to land in a predicted 1kmx1km zone for recovery

Re: Launch HN: Sorcerer (YC S24) – Weather balloons that collect more data

#79
post #25

Earlier quoted context omitted.

It’s a literal launch!

Being a balloon company means we get to launch pretty much every day, which is very fun :)

My coworker used to fly the chase plane for the Canadian Space Agency's balloons; they would call position and altitude for air traffic control and recover the instrument gondola. Lots of bushwhacking; one came down on an eagle's nest and mama wouldn't let them near the tree.

Re: Launch HN: Sorcerer (YC S24) – Weather balloons that collect more data

#80

> In 1981, weather disasters caused $3.5 billion in damages in the United States. In 2023, that number was $94.9 billion ( https://www.ncei.noaa.gov/access/billions/time-series ). Does that surprise someone? I think I would not have guessed this growth to be on such a scale. The chart suggests that severe storms are the main culprit.

Not at all. Look at the growth in human buildings in the most at-risk areas and you’ll see why that number is so big now. It’s only slightly due to an increase in severe weather event frequency / severity.

Indeed, and not just building more in more at-risk places but also the cost of building materials, construction labor and code compliance requirements have all generally increased more than baseline inflation. Factors like these tend to greatly increase recent estimates vs historical.

I read a paper a few years back which dove into how the data sources for weather damage assessment have changed a lot over the years. Much of the increase is due to more complete reporting and changes in categorization. Also, nowadays more things are insured and modern IT has made gathering the insurance reporting far more exhaustive. Plus local, state and federal agencies responsible for relief and/or recovery are gathering and reporting increasing amounts of data with each decade since the 70s (in part because their budgets rely on it). Factors like these mean in prior decades the total damage costs may have been more similar to today's than they appear but a lot of the damage data we gather and report now wasn't counted or gathered then.

Although I have no experience related to weather science, I remember the paper because it made me realize how many broad-based, multi-decadal historical data comparisons we see should have sizable error bars (which never make it into the headline and rarely even into the article). Data sources, gathering and reporting methods and motivations are rarely constant on long time scales - especially since the era of modern computing. Of course, good data scientists try to adjust for known variances but in a big ecosystem with so many evolving sources, systems, entities and agencies, it quickly gets wickedly complex.

Post reply on HN