How to get my kids to bed to bed.
Ask HN: What problem are you close to solving and how can we help?
11–20 of 486 posts
Re: Ask HN: What problem are you close to solving and how can we help?
#12We are having atrocious READ/WRITE latency with our PG database (api layer is django rest framework). The table that is the issue consists of multiple JSON BLOB fields, with quite a bit of data— I am convinced these need to be abstracted to their own relational tables. Is this a sound solution? I believe it is the deserialization in these fields of large nested JSON BLOBS that is causing latency. Note: this database…
Bit hard to tell without some idea of the structure of the data, but my experience has been storing blobs in the database is only a good idea if those objects are completely self contained i.e. entire files.
If you write a small program to check the integrity of your blobs i.e. that the structure of the json didn't change over time, you may be able to infer a relational table schema that isolates those bits that really need to be blobs. Too leave it too long invites long term compatibility issues if somebody changes the structure of your json objects.
Re: Ask HN: What problem are you close to solving and how can we help?
#13Earlier quoted context omitted.
imo, json dt should be an intermidiary step in db struc in rdb, never the final. Once you know & have stable columes, unravel the json into proper cols with indexing, it should improve the situation if youre having issues with 5gb, you will face exponential problems when it grows due to lack of indexing
Cheers for the response (and affirmation). After some latency profiling I am convinced proper cols with indexing will vastly improve our situation since the queries themselves are very simple.
You can take incremental approach and p.o.c with the data that you have so you can justify your move too!
Re: Ask HN: What problem are you close to solving and how can we help?
#14Who are "we"?
I suppose it is just him and the driver.
Re: Ask HN: What problem are you close to solving and how can we help?
#15We are having atrocious READ/WRITE latency with our PG database (api layer is django rest framework). The table that is the issue consists of multiple JSON BLOB fields, with quite a bit of data— I am convinced these need to be abstracted to their own relational tables. Is this a sound solution? I believe it is the deserialization in these fields of large nested JSON BLOBS that is causing latency. Note: this database…
I had this issue at a previous job where we would query an API (AWS actually) and store the entire response payload. As we started out we would query into the JSONB fields using the JSON operators, however at some point we started to run into performance issues and ended up "lifting" the data we cared about to columns on the same record that stored the JSON.
Re: Ask HN: What problem are you close to solving and how can we help?
#16https://free-visit.net :
Like Matterport but with an FPS game engine.
I am looking for my first client: Ideally someone in charge of a Museum/gallery or other grandiose indoor space.
Re: Ask HN: What problem are you close to solving and how can we help?
#17We are having atrocious READ/WRITE latency with our PG database (api layer is django rest framework). The table that is the issue consists of multiple JSON BLOB fields, with quite a bit of data— I am convinced these need to be abstracted to their own relational tables. Is this a sound solution? I believe it is the deserialization in these fields of large nested JSON BLOBS that is causing latency. Note: this database…
I don't know about PG, but with MariaDB, a nice way to find bottlenecks is to run SHOW FULL PROCESSLIST in a loop and log the output. So you see which queries are actually taking up the most time on the production server. If you post those queries here, we can probably give tips on how to improve the situation.
Interesting. I believe I noted a similar function in the Postgres docs I was scouring through Friday. I’ll give it a look and see what I can find.
Tangentially related for those who have experience, I am using Django-silk for latency profiling.
Re: Ask HN: What problem are you close to solving and how can we help?
#18Frustrated by the degree of manual programming process in production metal machining. The industry exists largely on inertia. I would like to resolve this by applying standard optimization algorithms to a set of known machining strategies plus machine, work-holding, material, part and tool inputs. Have already analyzed the problem space to some extent and will be touring a huge production facility next week to better understand best-in-class processes from large established players. Need someone to either wrap existing simulation algorithms (any CAM system) or write enough of one (not that hard, the solution space is extremely multivariate but well understood and well documented) to make it feasible (not too hard for 2.5D machining). You can get as intellectual as you like in the solution, but remember perfect is the enemy of done. Value is huge, happy to split equity on a new entity to resolve if a workable solution for the easier subset of parts emerges in the next few weeks.
Re: Ask HN: What problem are you close to solving and how can we help?
#19https://free-visit.net : Like Matterport but with an FPS game engine. I am looking for my first client: Ideally someone in charge of a Museum/gallery or other grandiose indoor space.
Hangs when I click "Try the example".