Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

blog.wilsonl.in

Post reply on HN
Print view View original

101–110 of 171 posts

Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

#101

Post by GeneralMayhem »

jxy wrote: post #84

> We can see that in this case, where perhaps the X axis represents "more cat" and Y axis "more dog", using the euclidean distance (i.e. physical distance length), a pitbull is somehow more similar to a Siamese cat than a "dog", whereas intuitively we'd expect the opposite. The fact that a pitbull is "very dog" somehow makes it closer to a "very cat". Instead, if we take the angle distance between lines (i.e. cosine…

The issue with normalization is that you lose a degree of freedom - which when you're visualizing, effectively means losing a dimension. Normalized 2d vectors are really just 1d vectors; if you want to show a 2d relationship, now you have to use 3d vectors (so that you have 2 degrees of freedom again).

Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

#102

Post by holoduke »

deadbabe wrote: post #65
wilsonzlin wrote: post #42

Earlier quoted context omitted.

Thanks! Yeah I'd like to dive deeper into the sentiment aspect. As you say it'd be interesting to see some overview, instead of specific queries. The negative sentiment stood out to me mostly because I was expecting a more "clear-cut" sentiment graph: largely neutral-positive, with spikes in the positive direction around positive posts and negative around negative posts. However, for almost all my queries, the sentim…

Anecdotally, I think anyone who reads HN for a while will realize it to be a negative, cynical place. Posts written in sweet syrupy tones wouldn’t do well here, and jokes are in short supply or outright banned. Most people here also seem to be men. There’s always someone shooting you down. And after a while, you start to shoot back.

Really? Mmm i think hn is a place with on avarage above intelligent people. People who understand that their opinion is not the only one. I rarely have issues with people here. Might be also because we are all in the same bubble here.

Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

#103

Post by flir »

deadbabe wrote: post #65
wilsonzlin wrote: post #42

Earlier quoted context omitted.

Thanks! Yeah I'd like to dive deeper into the sentiment aspect. As you say it'd be interesting to see some overview, instead of specific queries. The negative sentiment stood out to me mostly because I was expecting a more "clear-cut" sentiment graph: largely neutral-positive, with spikes in the positive direction around positive posts and negative around negative posts. However, for almost all my queries, the sentim…

Anecdotally, I think anyone who reads HN for a while will realize it to be a negative, cynical place. Posts written in sweet syrupy tones wouldn’t do well here, and jokes are in short supply or outright banned. Most people here also seem to be men. There’s always someone shooting you down. And after a while, you start to shoot back.

I think it's the engineering mindset. You're always trying to figure out what's wrong with an idea, because you might be the poor bastard that ends up having to build it. Less costly all round if you can identify the flaw now, not halfway through sprint 7. After a while it bleeds into everything you do.

Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

#104

Post by fragmede »

goles wrote: post #86
sethammons wrote: post #77

Earlier quoted context omitted.

I've been here a hot minute and I agree with you. Lots of good faith. Lots of personal anecdotes presumably anchored in experience. Some jokes are really funny, just not reddit-style. Similarly, no slashdot quips generally, such as "first post" or "i, for one, welcome our new HN sentiment mapping robot overlords." Sometimes things get downvoted that shouldn't, but most of the flags I see are well deserved, and I vouc…

I wonder how much of a persons impression of this is formed by their browsing habits. As a parent comment mentions big threads can be a bit of a mess but usually only for the first couple of hours. Comments made in the spirit of HN tend to bubble up and off-topic, rude comments and bad jokes tend to percolate down over the course of hours. Also a number of threads that tend to spiral get manually detached which takes…

some of the more negative threads will get flagged and detached and by the end of the day a casual browse through the comments isn't even going to come across them. eg something about the situation in the middle east is going to attract a lot of attention.

Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

#106

Post by pinkmuffinere »

sourcepluck wrote: post #105

Where is lisp?! I thought it was a verifiable (urban) legend around these parts that this forum is obssessed with lisp..?

Maybe lisp is so niche that even a rather small interest makes HN relatively lispy?

Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

#107

Post by b800h »

oersted wrote: post #27
wilsonzlin wrote: post #19

Earlier quoted context omitted.

Hey, thanks for the kind words. I wasn't able to mention the costs in the post (might follow up in the future) but it was in the hundreds of dollars, so was reasonably accessible as a hobby project. The GPUs were surprisingly cheap, and was only scaled up mostly because I was impatient :) --- the entire cluster only ran for a few hours. Do you have any links to your work? They sound interesting and I'd like to read m…

"Hundreds of dollars" sounds a bit painful as an EU engineer and entrepreneur :), but I guess it's all relative. We would think twice about investing this much manpower and compute for such an exploratory project even in a commercial setting if it was not directly funded by a client. But your technical skill is obvious and very impressive. If you want to read more, my old bachelor's thesis is somewhat related, from w…

As an EU-based engineer, you wouldn't do this, it's a massive GDPR violation (failure to notify data subjects of data processing), which does actually have extraterritoriality, although I somehow doubt that the information commissioners are going to be coming after OP.

Re: Show HN: Exploring HN by mapping and analyzing 40M posts and comments for fun

#109

Post by lmeyerov »

minimaxir wrote: post #100
minimaxir wrote: post #73

Earlier quoted context omitted.

I'll give that a look: the feature set of GPU-accelerated ops seems just up my alley for this pipeline: https://github.com/rapidsai/cuml EDIT: looking through the docs it's just GPU-acceletated UMAP, not a parametric UMAP which trains a NN model. That's easy to work around though by training a new NN model to predict the reduced dimensionality values and minimizing rMSE.

Tested it out and the UMAP implementation with this library is very very fast compared to Parametric UMAP: running it on 100k embeddings took about 7 seconds when the same pipeline on the same GPU took about a half-hour. I will definitely be playing around with it more.

Yeah we advise Graphistry users to keep GPU umap training sets to RAPIDS umap is darn impressive tho. Instead of focusing on improving further, it did the job. Our bottleneck shifted to optimizing the ingest pipeline to feed umap, so we released cu_cat as a GPU-accelerated automated feature engineering library to get all that data into umap. RAPIDS cudf helps take care of the intermediate IO and wrangling in-between.

Downstream, we generally stopped doing DBSCAN , despite being so pretty. We replace with cugraph/GFQL on the umap similarity graph, to avoid quality issues we see in practice, and then visually & interactively investigate the similarity graph in pygraphistry. Once you can see the k-nn similarity edges - and lack thereof -- you realize why scatter plot clusterings (visual or algorithmic) are so misleading to analysts and treat with more caution. There is a variety of umap contenders nowadays, but with this pipeline, we haven't felt the need to go beyond. That's a multi-year testament to Leland and team.

The result is we can now umap and interactively visualize most real world large datasets, database query results, and LLM embeddings that pygraphistry & louie.ai users encounter in seconds. Many years to get here, and now it is so easy!

Post reply on HN