Ask HN: What weird or hard problems are you trying to solve? • Hacker News Board

Re: Ask HN: What weird or hard problems are you trying to solve?

#101

Post by IshKebab » Sun, Jul 05, 2020, 9:24 PM UTC

dsteinman wrote: post #66

I am attempting to bring 100% client-side speech recognition to the web: https://github.com/jaxcore/bumblebee Although the first release is not officially out yet, the NodeJS code is working and you can install the development version of the app server and try out the hello world app locally. The solution involves running Mozilla DeepSpeech inside an Electron desktop application with a websocket server and client API…

It's not really "the web" if you have to use Electron and Node surely? Wouldn't it make more sense to do it with web workers and wasm?

Re: Ask HN: What weird or hard problems are you trying to solve?

#102

Post by fabianlindfors » Sun, Jul 05, 2020, 9:24 PM UTC

toomuchtodo wrote: post #50

httpsterio wrote: post #11

I've been mulling over an idea that is essentially a combination of personal ID, secure digital authentication and online communications all baked into one. There's a EU directive instructing on how citizens should be able to identify online with eIDAS. In my country, you can use eIDAS to authenticate in basically any governmental agency portal, but you can't get any eIDAS enabled auth method as a citizen. The curren…

I’m very interested in doing this as well, and have been trying to get the Login.gov folks (US centric) onboard (with Estonia’s electronic ID system as the model). We should chat!

Hi! I'm also working on something similar to this. If you want to chat, please reach out! fabian (at) flapplabs.se

Re: Ask HN: What weird or hard problems are you trying to solve?

#103

Post by eitland » Sun, Jul 05, 2020, 9:25 PM UTC

Getting time to do anything in between kids, work and whatever else is expected from me is the actual hard part for me.

The technical aspect is mostly fun in my case :-)

Re: Ask HN: What weird or hard problems are you trying to solve?

#104

Post by kd5bjo » Sun, Jul 05, 2020, 9:26 PM UTC

j88439h84 wrote: post #70

kd5bjo wrote: post #62

I’m trying to map relational algebra onto Rust’s type system. If I’m successful, I’ll have a bunch of collection types with different performance characteristics that are all drop-in replacements for each other.

Got a link?

Not right now; I haven’t set up a repository for it yet; I’ll certainly post it here when it’s ready to show.

I’m working from the bottom up and am almost ready to start working on collections— I started with newtypes to represent columns and then tried to get them to combine together properly.

As of today, I’ve got what looks like a good scheme to treat arbitrary objects as records, and can do joins, projections, and column renames with the type system keeping track of which objects have which columns.

The plan for tables/collections is to implement a sequential-scan interface backed by something simple, and then add wrapper write-through index objects to speed up particular query types.

Re: Ask HN: What weird or hard problems are you trying to solve?

#105

Post by dvt » Sun, Jul 05, 2020, 9:26 PM UTC

One-click API data ingestion.

Just about every company these days has their data spread out all over the cloud: marketing data on Facebook and Google, social media data on Twitter and Snapchat, customer data on Salesforce, sales data on Shopify and Amazon, and so on. Most companies will either (a) hire a team of data engineers to collect and exploit this data or; (b) hire an expensive consulting firm to build an ETL pipeline, or; (c) let this data rot in the cloud. For the past 6 years, I've worked as a data engineer (where I became intimately familiar with Facebook and Salesforce APIs), and I'm confident that I can automate around 80% of my job.

It's clear that the value prop is astronomical: just one data engineer will run you at least 150k/yr and most of the work will involve maintaining API data pipelines. Having a "one-click" solution where one simply provides an API key and what data they'd like to warehouse (e.g. marketing data, social media data, customer data) and where (FTP, S3, Redshift, DynamoDB) would be invaluable to companies that want to make sure they exploit this treasure trove.

Some hard/interesting problems:

- API specs constantly change (Facebook, for example, has a quarterly update schedule)

- Inferring JSON schemas is hard

- Data integrity is hard (data types sometimes change willy-nilly)

- API rate limiting is tricky

- Resilience is hard

- Recovering old data (especially for certain services) might be impossible

Everyone is starting to become keenly aware that letting the data rot is starting to have a higher and higher opportunity cost. Not warehousing your own data is simply not a tenable option any more: the world’s most valuable resource is no longer oil, but data[1].

[1] https://www.economist.com/leaders/2017/05/06/the-worlds-most...

Re: Ask HN: What weird or hard problems are you trying to solve?

#106

Post by yanovskishai » Sun, Jul 05, 2020, 9:27 PM UTC

erezsh wrote: post #95

I'm trying to replace SQL by building a language that compiles to SQL. It has first-class functions, nicer syntax, better type-system, introspection, and other things you would expect from a modern language. But in the end, you still get SQL's performance, and the ability to use it with dozens of database engines.

As someone who spent handreds of hours writing (and cursing) SQL queries -it definitely sounds compelling.

Re: Ask HN: What weird or hard problems are you trying to solve?

#107

Post by shezi » Sun, Jul 05, 2020, 9:28 PM UTC

hazeii wrote: post #94

How to measure pedal (and/or crank) angle on a moving bicycle to 0.05 degree. Anyone?

Grey code with 13 bits and as many light sensors, maybe?

Re: Ask HN: What weird or hard problems are you trying to solve?

#108

Post by erezsh » Sun, Jul 05, 2020, 9:29 PM UTC

eitland wrote: post #98

erezsh wrote: post #95

I'm trying to replace SQL by building a language that compiles to SQL. It has first-class functions, nicer syntax, better type-system, introspection, and other things you would expect from a modern language. But in the end, you still get SQL's performance, and the ability to use it with dozens of database engines.

You mean you work on LINQ? Only half joking here.

LINQ did a tiny step in the right direction. I'm aiming a lot further.

Re: Ask HN: What weird or hard problems are you trying to solve?

#109

Post by hashamali » Sun, Jul 05, 2020, 9:29 PM UTC

jeremylevy wrote: post #91

Automating cloud architecture creation. I'm building a "catalog" of architectures that you could use to create a complete cloud architecture on your AWS, GCP or Azure account in less than one minute. So, for example, you could create a docker-based architecture with CI/CD, auto-scaling, zero downtime deployment, SSL, load-balancing, high availability and MongoDB in less than one minute in your own AWS account. It's l…

Have you considered something like Pulumi for the glue?

Love this idea, the ease of Heroku + the flexibility of Kubernetes would be amazing.

Re: Ask HN: What weird or hard problems are you trying to solve?

#110

Post by closed » Sun, Jul 05, 2020, 9:33 PM UTC

Working on siuba, a data analysis tool for python. It's a port of the R library dplyr, and can produces SQL queries!

I've programmed in python for much longer than R, and really want to be able to move at the same speed when using python for data analysis :o.

It's a weird problem though because the two languages have basically opposite approaches to DataFrames. pandas has a very fat DataFrame implementation, R an extremely minimal one. (Pros and cons to both approaches).

https://github.com/machow/siuba