Earlier quoted context omitted.
I love that idea, you should submit that on frontpage.
Frontpage?
I meant submit it on HN as its own post.
Ask HN: What problem are you close to solving and how can we help?
281–290 of 486 posts
Re: Ask HN: What problem are you close to solving and how can we help?
#282I have a pile of mp3s and want to splice them together with a single ffmpeg operation. Essentially injecting multiple small audio files into a large one using time codes. I know there's got to be a way to do it, but I have yet to find a way to do it in a single operation instead of multiple passes.
Are they sequential, or are you splicing at arbitrary times?
They might be spliced in at arbitrary times but multiple audio files may be spliced in sequentially at an arbitrary time. I'm not going to be upset if there's a millisecond of gap between them or whatever by just splicing at approximate time codes to make them sequential.
Re: Ask HN: What problem are you close to solving and how can we help?
#283I am blocked on finding a good (defined below) way to determine whether a product description A and product description B refer to the same product. Imagine that a product description is a n-dimensional vector like: ( manufacturerName, modelName, width, height, length, color, ...) Now imagine you have a file with m such vectors (where m is in millions), and that not all fields in the vectors are reliable info (typos,…
I have worked on this problem many times, at many companies. I am working on it again, actually. Usually some combination of scoring and persisting results in CSVs for human review.
(edit: I am at a desktop now and I can say a bit more)
Here is the process in a nutshell:
1. Create a fast hashing algorithm to find rows that might be dups. It needs to be fast because you have lots of rows. This is where SimHash, MinHash, etc. come into play. I've had good luck using simhash(name) and persisting it. Unfortunately you need to measure the hamming distance between simhashes to calculate a similarity score. This can be slow depending on your approach.
2. Create a slower scoring algorithm that measures the similarity between two rows. Think about a weighted average of diffs, where you pick the weights based on your intuition about the fields. In your case you have handy discrete fields, so this won't be too hard. The hardest field is name. Start with something simple and improve it over time. Blank fields can be scored as 0.5, meaning "unknown". Hashing photos can help here too.
3. Use (1) to find things that might be dups, then score them with (2). Dump your potential dups to a CSV for human review. As another poster indicated, I've found human review to be essential. It's easy for a human to see that "Super Mario 2" and "Super Mario 3" are very different.
4. Parse your CSV to resolve the dups as you see fit.
Have fun!
Re: Ask HN: What problem are you close to solving and how can we help?
#284I have posted this here before- hexafarms.com. I am trying to use ML to discover optimal phenotype for growing plants in vertical indoor farms to a. have the higest quality produce b. to lower the cost of producing leafy green/med plants, etc. within cities itself. Basically, every leafy green (and herbs, and even mushrooms), can grow in a range of climatic condition (phenotype, roughly) ie temperature, humidity, wat…
There's some great research on using evolutionary computation to explore plant growing recipes (light strength, how long to leave the lights on, etc). In one experiment, researchers discovered that basil doesn't need to sleep - it grows best with 24 hours of light per day. Risto Miikkulainen shared the experiment on Lex Fridman's podcast: https://youtu.be/CY_LEa9xQtg?t=27m7s I believe this is the paper describing that experiment: https://journals.plos.org/plosone/article?id=10.1371/journal...
Re: Ask HN: What problem are you close to solving and how can we help?
#285What are the resources to look at for designing the architecture of a scheduling system (for coroutines/threads)? Would I first look at how Operating systems implement it? Or how VMs like Erland implement it? Same question for effects systems, where would I look to understand how they're designed and the trade offs for their design decisions?
It indeed makes sense to look at on OS. You might want to start with a simple one, like FreeRTOS - which is more or less just a task scheduler at core.
I would recommend not to start with coroutines in the beginning if your main focus is scheduling. In the end coroutines and async/await is about building a userspace scheduler on top of a scheduler that already exists in the OS, so you just get twice the amount of logic. However the schedulers used in userspace are often a lot more trivial than the OS ones, since they don't support preemption or priorities. Erlang might be the exception and an interesting thing to look into.
Re: Ask HN: What problem are you close to solving and how can we help?
#286I have a pile of mp3s and want to splice them together with a single ffmpeg operation. Essentially injecting multiple small audio files into a large one using time codes. I know there's got to be a way to do it, but I have yet to find a way to do it in a single operation instead of multiple passes.
Some god awful combination in a complex filter using atrim to pull the pieces, adelay to set the positions in the output, and amix to put all of the output back together could probably do it. What that command may actually be is definitely open but that's probably the only way that would work as otherwise this is really 2 separate operations (chopping then merging) so there isn't going to be consideration for having it in a single premade flag.
On another note if the goal is just to avoid files/writing to disk then a bunch of ffmpeg splices to named pipes as inputs to another ffmpeg command to merge them could do the same without the command soup.
Re: Ask HN: What problem are you close to solving and how can we help?
#287I'm working on a different type of compression (for all file types). I am able to to get in the 10-20% range, but the speed to compress is to slow many times, or the compression doesnt complete at other times (I've been working on this for years). My personal website: http://danclark.org
I'm also working on a conversational search engine (using NLP) at http://supersmart.ai
Re: Ask HN: What problem are you close to solving and how can we help?
#288I have posted this here before- hexafarms.com. I am trying to use ML to discover optimal phenotype for growing plants in vertical indoor farms to a. have the higest quality produce b. to lower the cost of producing leafy green/med plants, etc. within cities itself. Basically, every leafy green (and herbs, and even mushrooms), can grow in a range of climatic condition (phenotype, roughly) ie temperature, humidity, wat…
This sort of ml problem is characterized by relatively expensive data labeling. Hence, hiring an expert or mixture of experts, and modeling the crop responses to their choices, will save you a lot of hill climbing The wrong part of the decision space
Re: Ask HN: What problem are you close to solving and how can we help?
#289Don't have anything to add right now but I like the idea of this thread and would support it becoming a regular thing.
Re: Ask HN: What problem are you close to solving and how can we help?
#290A way to preserve and link factual data sets.
Most reference to Wikipedia are dead links.
Many legacy media will stealth edit articles or outright delete them.
Original media files can be loss and after strange eons their authenticity will not be able to be asserted.
It will soon be impossible to distinguish from deep fakes and actual original and genuine media.
Some regimes such as Maoist China wanted to rewrite their past from scratch and erased all historical artifacts from their territory.
There are strong pressure to create an Orwellian Doublespeak to erase certain words entirely from speech, books and records. With e-books now the norm it has now become legitimate question to ask if the books are the same they were when the author published them.
Collaborative content websites have shown that they were not immune to subversive large and organized influence operations.
I have set my mind to multiple solutions (even bought a catchy sounding *.org domain name!). Obviously it will have to be distributed as to build a consensus and thus it will have to rely on hashes. But hashes alone are meaningless so some from of information will have to come along with them, which in themselves are information to authenticate with other hashes. I was thinking that the authentication value would come from individual recognized signatories. Those would be a a mesh of statements of records. For example you might not trust your government, but you might trust you grandparents and you old neighbors who all agree that there was a statue on the corner of the street and they all link to each other and maybe link to hashes of pictures and 3D scans with links. Future generations can then confirm those links with other functional URIs.
Something like blockchain technology seems an obvious choice but I have no experience with that (for now) but also there is the problem that it needs to be easily usable; therefore there is a need of a bit of a centralization (catchy domain name yay!) although any one could setup his/her own service for certain specialized subjects.
Thoughts?