Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How can I help you?

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 43 posts

Re: Ask HN: How can I help you?

#31

Post by shepik »

ignoramous wrote: post #13

Mostly technical questions from me: 1. Re: Presto/Clickhouse : Have you looked at dsq, pola.rs, DuckDB, Apache DataFusion, Clickhouse Local? If so, what's your opinion on where the (data science) ecosystem is moving towards (for example, ibis-project.org taking over Presto/Trino). 2. Re: Bitmaps : What's the most compact way you know to store a bitmap-index in printable ASCII (b64 etc)? Puny code esque state machines…

1) DuckDB - like the performance and that it's tightly integrated into python. Clickhouse local - i saw the announcement when it came out, but i just don't see the usecase for it in analytics. DataFusion is new for me. I'd say that the ecosystem is moving towards snowflake/bigquery/redshift/...

2) Why would you want to do that? You'd use bitmap index because it's quite compact and you can process the data at the speed of memory bandwidth, using ascii defeats that, no?

3) Not really. I just can't imagine costs of running any large website or app with serverless.

4) I think it's the same in the US - everyone competes for the same talent with FAANG (MAANG?). The salary gap between big companies and startups is even lower in Russia. Also, we humans want to do meaningful things, some of us struggle to find meaning in being another bigco employee.

5) Py

Re: Ask HN: How can I help you?

#32

Post by ignoramous »

shepik wrote: post #31
ignoramous wrote: post #13

Mostly technical questions from me: 1. Re: Presto/Clickhouse : Have you looked at dsq, pola.rs, DuckDB, Apache DataFusion, Clickhouse Local? If so, what's your opinion on where the (data science) ecosystem is moving towards (for example, ibis-project.org taking over Presto/Trino). 2. Re: Bitmaps : What's the most compact way you know to store a bitmap-index in printable ASCII (b64 etc)? Puny code esque state machines…

1) DuckDB - like the performance and that it's tightly integrated into python. Clickhouse local - i saw the announcement when it came out, but i just don't see the usecase for it in analytics. DataFusion is new for me. I'd say that the ecosystem is moving towards snowflake/bigquery/redshift/... 2) Why would you want to do that? You'd use bitmap index because it's quite compact and you can process the data at the spee…

Thanks.

> Why would you want to do that?

We store user preferences (200+ yes/no knobs) in a bitmap (well, a bitmap-index like the one in Hash Mapped-Array Tries). We want to capture those prefs in a single sub-domain (limited to 63 lower-case alphanumeric chars) or a URL (limited to 200 mixed-case alphanumerics). Today, we simply convert the bitmap into url-b64 (or, b32 to store it in the subdomain), but we will soon run out the 63-char limit if we introduce more knobs.

A demonstration of it is here, in case the above didn't explain it well: https://rethinkdns.com/configure (choose blocklists, and see the selection generate a path appended to the base-url shown in the search-bar).

Re: Ask HN: How can I help you?

#33

Post by anm89 »

atlasunshrugged wrote: post #8
anm89 wrote: post #5

I'm looking to raise ~$2 million to build a coliving space with the goal to grow to multiple spaces. I'm bringing half a million of my own money to the table. Have any connections?

Where are you looking to build it? Is the money literally for construction or retrofitting an existing space or other stuff? Coliving.com had a guide on some of this but maybe it's old hat for you https://coliving.guide/

my primary interest is in Colombia but also open to some other places that are also affordable, maybe Italy or Croatia.

Yeah the primary concept is to design and build a property specifically optimized for coliving.

I have seen that guide before but I'll take another look.Thanks

Re: Ask HN: How can I help you?

#35

Post by shepik »

ignoramous wrote: post #32
shepik wrote: post #31

Earlier quoted context omitted.

1) DuckDB - like the performance and that it's tightly integrated into python. Clickhouse local - i saw the announcement when it came out, but i just don't see the usecase for it in analytics. DataFusion is new for me. I'd say that the ecosystem is moving towards snowflake/bigquery/redshift/... 2) Why would you want to do that? You'd use bitmap index because it's quite compact and you can process the data at the spee…

Thanks. > Why would you want to do that? We store user preferences (200+ yes/no knobs) in a bitmap (well, a bitmap-index like the one in Hash Mapped-Array Tries). We want to capture those prefs in a single sub-domain (limited to 63 lower-case alphanumeric chars) or a URL (limited to 200 mixed-case alphanumerics). Today, we simply convert the bitmap into url-b64 (or, b32 to store it in the subdomain), but we will soon…

Oh. No, i don't think it's possible - i'd suggest to just use multiple subdomains.

Technically, you can squeeze out some bits: there are 36-37 possible characters of which you are using only 32, so with arithmetic coding you would be looking at about 1 extra bit for 5 characters, but it's a nightmare to code. And after those extra bits run out, you will get the same problem anyway.

Re: Ask HN: How can I help you?

#36

Post by guiand »

ww520 wrote: post #20

What's your thought on TreeMap (ordered by key)? What problems can be best solved by TreeMap? Why do people use HashMap instead of TreeMap?

Hash tables are (usually) faster to do all sorts of operations than tree based maps, as most operations become a simple function to calculate a tree’s hash followed by a table lookup. Of course, they’re unordered, so if you need to iterate in order, or find all keys Also, TreeMap uses a red-black tree to implement the map, which is a basic type of binary tree. Depending on the data you’d like to store, other kinds of tree-based maps can have better performance characteristics. A map based on a Splay Tree[1] speeds up repeated accesses, so it could perform well if you had keys that were cheap to compute an ordering but expensive to compute a hash, and your access pattern has good temporal locality.

[1] https://en.wikipedia.org/wiki/Splay_tree

Re: Ask HN: How can I help you?

#37

Post by maxdemarzi »

ignoramous wrote: post #32
shepik wrote: post #31

Earlier quoted context omitted.

1) DuckDB - like the performance and that it's tightly integrated into python. Clickhouse local - i saw the announcement when it came out, but i just don't see the usecase for it in analytics. DataFusion is new for me. I'd say that the ecosystem is moving towards snowflake/bigquery/redshift/... 2) Why would you want to do that? You'd use bitmap index because it's quite compact and you can process the data at the spee…

Thanks. > Why would you want to do that? We store user preferences (200+ yes/no knobs) in a bitmap (well, a bitmap-index like the one in Hash Mapped-Array Tries). We want to capture those prefs in a single sub-domain (limited to 63 lower-case alphanumeric chars) or a URL (limited to 200 mixed-case alphanumerics). Today, we simply convert the bitmap into url-b64 (or, b32 to store it in the subdomain), but we will soon…

Does another party need to decode the url? What about using a dictionary for the top 10k seen starting combinations and then encode the rest?

What about run length encoding? 1-9 for positive sequences. a-i for negative sequences (max means pattern continues) and the rest for frequent patterns like alternating sequences, etc

9967b would be 24 yes, 1 no, 7 yes, 3 nos, 1 yes etc

Re: Ask HN: How can I help you?

#39

Post by shepik »

ww520 wrote: post #20

What's your thought on TreeMap (ordered by key)? What problems can be best solved by TreeMap? Why do people use HashMap instead of TreeMap?

Simple answer: Use tree if you need range access or to get elements ordered by key, and use hash otherwise.

More nuance: - hashmap may be resized if it's over capacity, the resize may cause a latency spike.

- hashmap is essentially a single random memory access, tree is a couple of accesses but they are not random

- tree is a bit like a sorted array with fast inserts/deletes. Some trees, like leveldb, are in fact sorted arrays (plus some tricks, of course)

- if you use b-tree, you are more memory-efficient (but less cpu efficient), and access to nearby elements is almost free. That's why b-trees are used to store data in a permanent memory

- there are many other tree variants, each of them with different trade-off

Re: Ask HN: How can I help you?

#40

Post by shepik »

atlasunshrugged wrote: post #6

Asking here in case others find it interesting. Can you share more about your experiences building in Russia, especially more unique challenges you faced (e.g. harder to literally get funds from foreign clients, regulatory environment, etc.). Are there things solved there that you are shocked you haven't seen elsewhere? What is your feeling about the current buildup of troops around Ukraine & what do Russian friends/…

That's a difficult question - i don't have experience outside of Russia so no idea what is unique and what isn't.

It is harder to get funds from foreign clients, primarily because customers are wary of paying money to a russian company (because of hackers and scammers, i think). But you can solve it by registering a US company.

There is a "local market" trap: Russian market is obviously smaller than US one, but still large enough to be considered as a viable single option. That's why many russian companies go solely for the russian market, and that's why companies from Belarus or Ukraine (where local market is not large enough) are often focused on US/Europe from the start.

As for the recent events, I am not a fan, and hope that the situation resolves peacefully.

Post reply on HN