21–30 of 549 posts
Say someone ingests this data and clean it up make it usable, who's the customer for that service? What would they want to know from it?
Say someone ingests this data and clean it up make it usable, who's the customer for that service? What would they want to know from it?
I want to be able to visit a website, select my hospital, the procedure, and my insurance, and see what it will cost. Next to the result, please show me how much the same procedure would cost with the same insurance at other hospitals near my location. You will literally save American lives.
Earlier quoted context omitted.
Yep, we have built this database at Turquoise Health. I agree, the data is massive - and don't forget that it is all refreshed monthly!
Is that from the hospital side or the insurer side?
Say someone ingests this data and clean it up make it usable, who's the customer for that service? What would they want to know from it?
I want to be able to visit a website, select my hospital, the procedure, and my insurance, and see what it will cost. Next to the result, please show me how much the same procedure would cost with the same insurance at other hospitals near my location. You will literally save American lives.
I wonder what percentage of work in the US healthcare system is completely unnecessary from a general perspective but made necessary deliberately to justify the unethical system that allows millions to die unnecessarily.
[1]https://www.pgpf.org/blog/2022/07/how-does-the-us-healthcare...
Seems like every week there's a new massive scale DB project or company getting announced on HN. If they're looking for projects that create public value and demonstrate the power of their products at scale, digitizing this and making it searchable may be a good marketing project that's appealing to certain kinds of customers.
Petabytes uncompressed would be tricky if you need to slice those columns. SQLite caps out at ~281 terabytes of storage before it can't track any additional pages.
None of this is to say you couldn't partition the data across a lot of SQLite instances in varying ways. I will probably take a shot at it this weekend. Looking to see just how unlimited my AT&T fiber connection is anyways.
Seems like every week there's a new massive scale DB project or company getting announced on HN. If they're looking for projects that create public value and demonstrate the power of their products at scale, digitizing this and making it searchable may be a good marketing project that's appealing to certain kinds of customers.
Figuring out the size of this data was part of the research phase for doing just that: building out that database. I'm curious to know if other people are already working on it (maybe Turquoise Health?)
I'd be very curious to read more about the data cleaning phase when you get there. Specifically, how hard it is to combine this data and construct good schemas.
Earlier quoted context omitted.
Figuring out the size of this data was part of the research phase for doing just that: building out that database. I'm curious to know if other people are already working on it (maybe Turquoise Health?)
Yep, we have built this database at Turquoise Health. I agree, the data is massive - and don't forget that it is all refreshed monthly!
edit: As I reflect, I'm amused to recall that this was early enough in my path that I didn't know about DB indexes, so I was very proud that I figured out how to basically roll my own indexes by pre-sorting the columns by lat and lon. I don't remember whether my solution actually prevented a full-table scan, but it felt like a major breakthrough at the time.
Seems like every week there's a new massive scale DB project or company getting announced on HN. If they're looking for projects that create public value and demonstrate the power of their products at scale, digitizing this and making it searchable may be a good marketing project that's appealing to certain kinds of customers.
It would appear us SQLite zealots have encountered the final boss. Petabytes uncompressed would be tricky if you need to slice those columns. SQLite caps out at ~281 terabytes of storage before it can't track any additional pages. None of this is to say you couldn't partition the data across a lot of SQLite instances in varying ways. I will probably take a shot at it this weekend. Looking to see just how unlimited my…
That's cute. :)
There isn't much value in feeding it all into a conventional RDBMS. OLAPs and columnar stores are what is needed here. But first it will need a great deal of grooming and ETL work.