Live data from Hacker News

Ask HN: Is there an open source equivalent to Palantir?

news.ycombinator.com

41–50 of 51 posts

Re: Ask HN: Is there an open source equivalent to Palantir?

#41

I think you're asking the wrong question. There is nothing special about Palantir software that you can't find in the open source world already. What Palantir does is just throw a bunch of bodies at the problem to "integrate" things. If you truly want to provide an open source alternative then curating some jupyter notebooks with the right libraries and integration is all it takes. A properly curated set of R librari…

I think people greatly underestimate the importance of integration, but I take your point

Re: Ask HN: Is there an open source equivalent to Palantir?

#42
post #2

Not exactly what you're asking for. But http://www.jplusplus.org/en/ have mailing lists and is a community of journalists looking through data. I'm sure they have alternatives or suggestions.

I'm thinking about software that does link analysis - this person works with that person, got paid by other person, etc. Basically, computer support for intelligence analysts, but in the public service.

Re: Ask HN: Is there an open source equivalent to Palantir?

#43
post #31

"[..] from software that could help track networks, timelines, etc., and organize unstructured data the way an intelligence analyst would." Do you know how it looks in Palantir's case? I've never actually seen it ..

no. they keep it pretty hidden, but there are other tools designed for intelligence analysis that are easier to see.

Re: Ask HN: Is there an open source equivalent to Palantir?

#44
post #24

Most of the core platforms these companies use are built on open source. I think the fundamental reason there isn't something comparable in the free and open arena is because at the end of the day to coral the data volumes you will find your self well on your way down the road to hell. There is a minimum physical hardware cost in the multi-million dollar range just to get started. Not to mention the labor cost to ope…

Nah. I'm thinking of the people trying to understand the relationship between (for example) Trump and the Russians. They don't have that much data because they're not drinking the firehose of wiretaps. They're relying on news reports, contacts, financial documents, etc. It's a question of helping people connect the dots.

Re: Ask HN: Is there an open source equivalent to Palantir?

#45

The Investigative Consortium of Journalists used Neo4j to untangle the Panama Papers leak. https://neo4j.com/blog/icij-neo4j-unravel-panama-papers/

The problem with Neo4j is that the end results are great, but the ingestion pipeline (especially for unstructured data) is very hard to make general purpose. The ICIJ used a combination of Apache Tika, Nuix, Tesseract and a bunch of other components when loading data into Neo4j before interrogating it within Linkurious. It's also worth noting, that Panama data-set is riddled with data quality issues (even if this is…

My first job out of university was with a Palantir competitor (Detica at the time I was hired, then BAE Systems Detica, then finally BAE Systems Analytics or something like that[1]). There's nothing general purpose about either of their platforms (nor the similar offering from SAS). Companies like that just throw a bunch of fresh graduates at the data, and they hand-write loads of custom ETL code for every data set. A lot of times, even the "analytics" are just shitty little pattern matches over tiny subgraphs formed from the data, the vast majority of which are also coded anew by the "analysts"[2] for each data set. Data quality issues were handled with a massive case analysis during ETL.

There's really no magic going on in such products — the only part that's really general purpose is the GUI used to view the end results. The rest of it is just a bunch of lowly peons doing a ton of gruntwork to hammer the data into a form that said GUI will accept.

[1]: That last name change was after I'd left. I didn't even make it a full year at the company before my conscience got the better of me and I quit.

[2]: An "analytics" job at Detica was really just a half-step above data entry. It was mind-numbing and soul-eating. There was a very high turnover rate because even new graduates were overqualified for the position, and almost everyone was miserable.

Re: Ask HN: Is there an open source equivalent to Palantir?

#46
post #24

Most of the core platforms these companies use are built on open source. I think the fundamental reason there isn't something comparable in the free and open arena is because at the end of the day to coral the data volumes you will find your self well on your way down the road to hell. There is a minimum physical hardware cost in the multi-million dollar range just to get started. Not to mention the labor cost to ope…

[deleted]

Re: Ask HN: Is there an open source equivalent to Palantir?

#47

Are you certain that you want a competitor to palantir? It sounds like you might want I2 Analyst Notebook from IBM, or something like maltego from Paterva. Maltego is also free (ish) which is cool.

This is what I'm looking for. Thank you.

Re: Ask HN: Is there an open source equivalent to Palantir?

#49
post #32

I would start looking at gephi: https://gephi.org and sigma.js for visualization. To run influence or social network analysis I would look at R's igraph: http://igraph.org/r/ or spark graphx: http://spark.apache.org/docs/latest/graphx-programming-guide... . For the graph database. neo4j , elasticsearch or the open source graph database created by the uk intelligence service: https://github.com/gchq/Gaffer . Another i…

You may want to look into Linkurious (https://linkurio.us) which integrates Neo4j, ElasticSearch and a graph visualization interface. It was used extensively in the Panama Papers (https://linkurio.us/panama-papers-how-linkurious-enables-ici...)

Jean from Linkurious

Re: Ask HN: Is there an open source equivalent to Palantir?

#50
post #43
post #31

"[..] from software that could help track networks, timelines, etc., and organize unstructured data the way an intelligence analyst would." Do you know how it looks in Palantir's case? I've never actually seen it ..

no. they keep it pretty hidden, but there are other tools designed for intelligence analysis that are easier to see.

Well, for certain values of "hidden" that include 40 minute long demos of product on Youtube, e.g. https://www.youtube.com/watch?v=f86VKjFSMJE

Look around in their 293 videos on that channel and I'm sure you'll see a wide variety of their products, I just pulled out the first one that popped up and looked like a demo. Remember, companies need to advertise to find new customers, so they can't be too secretive.

(My biggest tip to someone looking to interview at a company is to find their youtube channel and watch a few videos of them demo their product before you interview. You will be an expert on their product compared to 95% of the people they interview, which will make you stand out. If they publish white papers, read one or two that look interesting to you. If they publish white papers and none of them look interesting to you, that's a pretty important sign too. Sure, bone up on how to find a cycle in a directed graph, but spend an hour or two on the company themselves and it will be very rewarding.)

Post reply on HN