51–59 of 59 posts
Might be interested in orthogonal reading - "The Textual Warehouse" (ISBN-10: 163462954X) by data warehouse pioneer Bill Inmon. He is and always has been ahead of his time with his thinking!
Does this also extract semantic relationships and data dependencies between fields? In the past I'd built an internal tool that transforms insurance PDFs to structured data. I wanted to extract explicit data dependencies between fields to perform validation. Insurance forms can sometimes have 30-40 pages and they can have fields on page 40 that depend on fields on page 4 with a few nested if conditions. Would Parsewi…
On the scale aspect, because we go in multiple passes, we break the scope into small enough pieces and then build it back up in a later step. Iirc the largest document I've seen a customer use was over 1k pages.
There are more complex data dependency scenarios where we find that the data that's extracted and combined (e.g. from page 4 and 40), needs to then be further transformed in different ways (e.g. having an evaluation and a clarification outcome at the end). To make these be aligned in value we are soon releasing a feature for what we call derived agents.
Does this also extract semantic relationships and data dependencies between fields? In the past I'd built an internal tool that transforms insurance PDFs to structured data. I wanted to extract explicit data dependencies between fields to perform validation. Insurance forms can sometimes have 30-40 pages and they can have fields on page 40 that depend on fields on page 4 with a few nested if conditions. Would Parsewi…
Yes, we do it by having multiple stages to the pipeline. First we would extract the independent data points (from say both page 4 and 40) and a second pass step establishes relationship (we call this resolution). On the scale aspect, because we go in multiple passes, we break the scope into small enough pieces and then build it back up in a later step. Iirc the largest document I've seen a customer use was over 1k pa…
2. Can I ask how? When I was building something like this, I realized there's an element of burning tokens for correctness. Meaning, splitting things into small units and small processes, each using a separate LLM output to be later combined. For a 1k page document, what kind of token usage do you see?
Earlier quoted context omitted.
Yes, we do it by having multiple stages to the pipeline. First we would extract the independent data points (from say both page 4 and 40) and a second pass step establishes relationship (we call this resolution). On the scale aspect, because we go in multiple passes, we break the scope into small enough pieces and then build it back up in a later step. Iirc the largest document I've seen a customer use was over 1k pa…
1. Incredible! Can I make an unsolicited ask? If you had industry specific templates for standardized PDFs it would be easier for me to send Parsewise to the insurance companies I'd worked for. Something similar to https://www.useanvil.com/forms/?type=pdf-templates but with your clean, semantic data model. 2. Can I ask how? When I was building something like this, I realized there's an element of burning tokens for c…
2. We see exactly the same thing. There is a trade-off in correctness vs token burning. However, some tokens (models) are cheaper and faster than others, so the small pieces can benefit from that. The token usage is also surprisingly variable, because it depends on the information density of the document and also on the information density of the question (e.g. is it a single needle in a haystack or are we analyzing the entire haystack from 10 perspectives). So the parsing for 1k pages may be on the order of millions of tokens, while a series of queries (extractions) on top of it could be 1-2 orders of magnitude more.
Earlier quoted context omitted.
Noted — and I did wish the founder success. I have no personal ill will towards them. But what I'd ask HN to consider is this: our world, and the technology we introduce into it, isn't apolitical or free of normative stakes and real, harmful implications for people. Treating where you've worked and what technology you've stewarded into being as an ethically neutral fact isn't neutral at all. What concerns me is that…
Sure, and HN hosts many threads where people debate these points. We're not against that and often as not agree with them. But this is a startup launch thread about something unrelated, and hounding someone about an ex-employer is a tenuous ground for bringing such material up. It's the sort of thing this guideline (from https://news.ycombinator.com/newsguidelines.html ) asks people not to do, even apart from the per…