How to make png encoding much faster? I'm working with large medical images and after a bit of work we can do all the needed processing in under a second (numpy/scipy methods). But then the encoding to png is taking 9-15secs. As a result we have to pre-render all possible configurations and put them on S3 b/c we can't do the processing on demand in a web request. Is there a way to use multiple threads or GPU to encod…
> Is there a way to use multiple threads or GPU Maybe you could write the png without compression, compress chunks of the image in parallel using 7z, then reconstitute and decompress on the client side.
Ask HN: What problem are you close to solving and how can we help?
291–300 of 486 posts
Re: Ask HN: What problem are you close to solving and how can we help?
#292Earlier quoted context omitted.
Wake up at 0430, exercise, take a shower and then work till you need to get ready for your full time job. If possible also dedicate half of your lunch hour to your project as well, together with half a weekend day. It's quiet so early in the morning, so your productivity will skyrocket. I've coded my Paras this way while working a full time, heavy blue-collar job.
Does it not cause adverse impact on the day job?
Re: Ask HN: What problem are you close to solving and how can we help?
#293How to make png encoding much faster? I'm working with large medical images and after a bit of work we can do all the needed processing in under a second (numpy/scipy methods). But then the encoding to png is taking 9-15secs. As a result we have to pre-render all possible configurations and put them on S3 b/c we can't do the processing on demand in a web request. Is there a way to use multiple threads or GPU to encod…
GPGPU is the way to go. Not terribly hard if you only need 1-2 formats supported, e.g. RGBA8 only. You don't need to port the complete codec, only some initial portion of the pipeline and stream data back from GPUs, the last steps with lossless compression of the stream ain't a good fit for GPUs. If you want the code to run on a web server, after you'll debug the encoder your next problem is where to deploy. NVidia t…
Re: Ask HN: What problem are you close to solving and how can we help?
#294I am blocked on finding a good (defined below) way to determine whether a product description A and product description B refer to the same product. Imagine that a product description is a n-dimensional vector like: ( manufacturerName, modelName, width, height, length, color, ...) Now imagine you have a file with m such vectors (where m is in millions), and that not all fields in the vectors are reliable info (typos,…
I'd start by detecting common typos. Typos are similar to un-typo'd data, so I'd do a frequency analysis on the textual representations of manufacturer name and model name, and a Levenshtein distance calculation, then synonymise the obvious synonyms (looking things up when I wasn't sure). The key idea is that you have access to more information than just this dataset: Tony and Tomy are different manufacturers, but Sony and Somy aren't (even though somy is in the dictionary and tomy isn't).
Once the manufacturer and model fields are mostly typo-free (after typo replacement – don't modify the original file, if you can help it!), you can start looking at dimensions and colour. Sort by manufacturer, and start de-duping entries. Once you get a feel for the process you're doing (e.g. under what circumstances do you check whether there's a 102mm Phillips screwthread?), you can start automating bits of it. There will always be special-cases, but your job is to get the data processed correctly, not to get the computer to process the data.
Accidentally aliasing two different products is much worse than leaving the same product described twice, so err on the side of “these are different”. (Keep in mind that manufacturers of some things, e.g. SD cards, often pretend two different products are the same – so you can't always win!) Remember, humans exist: bothering them a few million times is a problem, but bothering them a few hundred would be okay.
When new data comes in, I'd run all the code I used to come up with my system, and see if the output was notably different. If it was, I'd get the computer to let me know.
I'd also add some way for users to flag duplicates. Many humans make light work.
Re: Ask HN: What problem are you close to solving and how can we help?
#295How to make png encoding much faster? I'm working with large medical images and after a bit of work we can do all the needed processing in under a second (numpy/scipy methods). But then the encoding to png is taking 9-15secs. As a result we have to pre-render all possible configurations and put them on S3 b/c we can't do the processing on demand in a web request. Is there a way to use multiple threads or GPU to encod…
I'm unsure if this will help, but the new image format JPEG XL (.jxl) is coming soon to replace JPEG. It will have a lossless and a lossy abilities. It claims to be faster than JPEG. Another neat feature is that it's designed to be progressive, so you could host a single 10mb original file, and the client can download just the first 1mb (up to the quality they are comfortable with). Take a look: https://jpegxl.info/
Re: Ask HN: What problem are you close to solving and how can we help?
#296Earlier quoted context omitted.
I love that idea, you should submit that on frontpage.
Can't. As the legal framework does not exist yet. That's why I need a lawyer in the team.
Your call, of course. I'm just a random internet stranger spitballing ideas here for how to take that next step and I know next to nothing about this problem space.
Best of luck, whatever you choose to do.
* Make sure you read the rules for Show HN and use your own judgement there as to whether this qualifies.
Re: Ask HN: What problem are you close to solving and how can we help?
#297Re: Ask HN: What problem are you close to solving and how can we help?
#298Yeah, this week I've restarted my scanning tunnel microscope that I'm failing to make work for years... The current one is a standard pair of long metal bars with the piezoelectric component on one end, with 2 screws, and a 3rd screw on the other end. My problem is that it doesn't matter how I design the thing, either the screws offer too little precision so I can't help but to crush the tip into the sample every tim…
Wow, this sounds very ambitious! Perhaps you could somehow attach the piezoelectric component or bars to a micrometer [1] which is designed for accurate and repeatable measurement? [1] https://en.wikipedia.org/wiki/Micrometer
Yet, they are a bit expensive. I'm still not willing to budget all that, but I'm starting to consider it.
Re: Ask HN: What problem are you close to solving and how can we help?
#299Earlier quoted context omitted.
GPGPU is the way to go. Not terribly hard if you only need 1-2 formats supported, e.g. RGBA8 only. You don't need to port the complete codec, only some initial portion of the pipeline and stream data back from GPUs, the last steps with lossless compression of the stream ain't a good fit for GPUs. If you want the code to run on a web server, after you'll debug the encoder your next problem is where to deploy. NVidia t…
Thanks, I hadn't heard of that and I will look into it. This is a research setting with plenty of hardware we can request and not a huge number of users so that part doesn't worry me.
If you don’t care about cost of ownership, use CUDA. It only runs on nVidia GPUs, but the API is nice. I like it better than vendor-agnostic equivalents like DirectCompute, OpenCL, or Vulkan Compute.
Re: Ask HN: What problem are you close to solving and how can we help?
#300I have posted this here before- hexafarms.com. I am trying to use ML to discover optimal phenotype for growing plants in vertical indoor farms to a. have the higest quality produce b. to lower the cost of producing leafy green/med plants, etc. within cities itself. Basically, every leafy green (and herbs, and even mushrooms), can grow in a range of climatic condition (phenotype, roughly) ie temperature, humidity, wat…