Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Boring but important tech no one is working on?

news.ycombinator.com

Post reply on HN
Print view View original

351–360 of 464 posts

Re: Ask HN: Boring but important tech no one is working on?

#351

Post by chaps »

pani5ue wrote: post #325
chaps wrote: post #82

Extracting deeply useful information from tens/hundreds/thousands of thousands of scanned PDFs in a way that dumps its information, regardless of structure (eg, tables, text), into a relational database that's (mostly) trivially queryable and repeatable. Preferably open source. This is such a hard problem right now.

I'm doing this for a customer right now! Using Textract from Amazon to do the OCR. I dump the raw json in a SQLite database. Information extraction is trickier. I extract useful things like lines and pages, locations of lines, set up various columns and now can search the SQLite db either with python code or SQL queries. There's of course cost for running textract OCR on AWS. There are open source solutions like Tess…

Oh yes, I'm very familiar with these. All these do though is extract information, but don't immediately make them useful. So there's a massive gulf of a middle-step that's not yet done. Textract gets close...ish to that, but it's prohibitively expensive.

Re: Ask HN: Boring but important tech no one is working on?

#352

Post by vasili111 »

iammjm wrote: post #36

Educational games. Hear me out. The way we teach us basically how we did it 500 years ago. This is stupid, boring and not scalable. We dont have enough teachers, attention span is short, education is costly. So we need something that scales, is fun and involves all types of media plus gamifies education. Think Skyrim or GTA meets MS Encarta

I would add to that educational animations. For example, in biology/medicine good animation are trully helpful to understand better and form more memorable associations. Animations cannot replace textbooks but can complement them and make learning much more easy.

Re: Ask HN: Boring but important tech no one is working on?

#353

Post by chaps »

fffobar wrote: post #305
chaps wrote: post #82

Extracting deeply useful information from tens/hundreds/thousands of thousands of scanned PDFs in a way that dumps its information, regardless of structure (eg, tables, text), into a relational database that's (mostly) trivially queryable and repeatable. Preferably open source. This is such a hard problem right now.

> Extracting deeply useful information This right there is the difficult part - what do you mean exactly? I cannot come up with anything better than search, as in like Google search. And they did it for books already, it's seriously good.

A big problem that happens with FOIA requests is that you're often sent data in the form of a spreadsheet that was converted to a PDF. And then scanned. For thousands of pages. Solve that generally so that you can insert all of that data into a postgres database, with sensible indexes.

Re: Ask HN: Boring but important tech no one is working on?

#354

Post by chaps »

Fizzz wrote: post #297
chaps wrote: post #160

Earlier quoted context omitted.

I have around 2 million pages from FOIA requests that need information systematically extracted and I'm not alone in this problem. The costs for the systematization of many pages will be prohibitive. The public good of having a resource like this available to the public for free is beyond unimaginable as far as I'm concerned.

How do you currently extract info from the FOIA request pages? What kind of info do you look for? Just thinking how you could standardise this

Mostly tesseract, or uploading it through documentcloud and running manual searches. I do a combination of data analysis and spend many hours reading through documents. Sometimes I use unix tools like grep/awk/etc, sometimes I use SQL. If the PDF isn't scanned, I use tabula for csv extraction, but if it's scanned it becomes a silly ordeal.

Mind you, I'm not exactly looking for advice here. It's a supremely difficult problem and gut-ideas more often than not don't pan out.

Re: Ask HN: Boring but important tech no one is working on?

#355

Post by antifa »

EddySchauHai wrote: post #34
georgeburdell wrote: post #24

Not tech but I think millenials, gen Z, and younger have lost the know-how to make a dollar independently of mega corps. I never see kids going door to door mowing lawns anymore, for example.

I disagree with this, I'm (unfortunately) swamped with short videos on how to 'make side hustles' doing just these sorts of things due to the cost of living crisis.

And almost all of these videos are scams, lies, MLMs, or too obvious to be helpful.

Re: Ask HN: Boring but important tech no one is working on?

#356

Post by davidivadavid »

darepublic wrote: post #255
charleshan wrote: post #239

Earlier quoted context omitted.

This comment reminds me of Status as a Service by Eugene Wei. "Some people find status games distasteful. Despite this, everyone I know is engaged in multiple status games. Some people sneer at people hashtag spamming on Instagram, but then retweet praise on Twitter. Others roll their eyes at photo albums of expensive meals on Facebook but then submit research papers to prestigious journals in the hopes of being publ…

Some status games are better than others. ie when the game requires some common good to be performed, or the mastery of a useful skill. Some status games reward things that only drag us down

But what's "common good" and a "useful skill" itself is defined by a status game.

Re: Ask HN: Boring but important tech no one is working on?

#357

Post by jamal-kumar »

mrtksn wrote: post #102
rmdoss wrote: post #77

Earlier quoted context omitted.

Duolingo is doing it amazingly to learn a new language.

I speak 3 languages, attempted to learn the 4th one (German) on Duolingo and I find it very ineffective. I recall having trouble learning English back in high school and then having a breakthrough. The breakthrough was, I figured out how to think in English when trying to debug my errors with the teacher who actually have lived in the USA. Once it clicked, I no longer had to memorise things but predict instead. My En…

Duolingo is a pretty good tool for practice, but for trying to learn a language from scratch it's probably best to go through some YouTube videos for the pronunciation and some books first and consider it a kind of fancy flash card thing to do on the side.

Re: Ask HN: Boring but important tech no one is working on?

#358

Post by mikewarot »

astrange wrote: post #330
ccooffee wrote: post #262

Earlier quoted context omitted.

Can't watch the video right now, but the slides (available at [0]) list these four "forgotten ideas": 1. Linda Tuple Spaces [1] 2. Flow based programming [2] 3. Xanadu [3] 4. Unix pipes [4] [0] https://codesync.global/uploads/media/default/0001/01/de7dfa... [1] https://en.wikipedia.org/wiki/Tuple_space [2] https://en.wikipedia.org/wiki/Flow-based_programming [3] https://en.wikipedia.org/wiki/Project_Xanadu [4] https:…

Almost everyone who wants to fundamentally reinvent programming seems to come up with visual dataflow programming, which is strange to me because aren't those just function calls?

No... it's plumbing. Function calls are nested to-do lists. It's a different thing all together.

Flows all have queues.

Re: Ask HN: Boring but important tech no one is working on?

#359

Post by bredren »

ccorcos wrote: post #336
sowbug wrote: post #138

Less a specific technology and more a mindset. Repairing things around the house. Today, it's almost always rational to throw out the broken thing and buy a new one. There are a bunch of reasons why that's true: cost of one's own labor, lack of discrete replacement parts, lack of repair documentation, improvements in technology since original purchase, risk of further breakage, risk of injury to self, etc. But the re…

I feel like there's a city vs suburbs correlation here too. If you live in a city, (nevermind the fact that you're probably renting and paying way to much in rent consider fixing anything yourself) you probably don't have the space and the tools to enjoy taking apart and fixing things. In the suburbs, you probably have a garage and a side yard where you have the space to entertain these kinds of hobbies. Just recolle…

> If you live in a city, (nevermind the fact that you're probably renting and paying way to much in rent consider fixing anything yourself) you probably don't have the space and the tools to enjoy taking apart and fixing things.

The way housing is now, overpaying for rent is just “paying rent.” There is no special circumstance as a renter where repairs come faster or with less pliant and nuanced communication to the landlord.

The places are actually worse, because competition for below market single family homes, (which are basically always in need of repair,) is extremely fierce.

My experience is landlords don’t fix up anything and you have to pick and choose the most important stuff for them to fix.

Technically are they on the hook to fix everything? Maybe? But only people who don’t rent or haven’t rented in too long to remember think this is actually what happens.

And anyway there is some implicit understanding that since they are not absolutely gouging you like every other landlord, you owe it to them to not ask for too much.

So implicitly, you either have to learn to fix things or have a home that is sorta kinda broken / jenk six ways from Sunday.

Either learn how to fix stuff or live the life of death by a thousand paper cuts.

Re: Ask HN: Boring but important tech no one is working on?

#360

Post by fan_of_yoinked »

what-imright wrote: post #347

My startup is working on solutions to crop failure scenarios such as from a nuclear war. Unfortunately we calculate there are probable outcomes where the only reasonable course of action is for one selected group to actually consume parts of the population in order to survive a nuclear winter. And so we’re building an automated system to take a newly deceased human being and create as many wholesome and nutritious me…

wut
Post reply on HN