21–30 of 51 posts
I dunno, are there non-state actors with the capacity to inflitrate and disrupt militarized adersaries at a global scale in furtherance of the foreign policy motives of a super power on the world stage? I wonder if they'd release under the GPL3 or or BSD licence. Maybe they'll let me fork them on github.
The only thing close that I can think of is: https://github.com/sirensolutions/kibi Demo video: https://www.youtube.com/watch?v=g0O8UNM0B7Y
There is a minimum physical hardware cost in the multi-million dollar range just to get started.
Not to mention the labor cost to operate and architect said hardware. Any thing in the Petabyte scale is looking at a minimum skeleton crew of (if they are top talent, more people if they aren't top talent) maybe 3-4 people between sysadmin architect and data engineers to keep things production worthy. The industry is still light on people with those skills so ontop of the bare hardware you are probably talking about 600 to 800 K in annual labor cost as a starting point.
This would all be before you get that much usage, the minute that utilization becomes high that skeleton crew won't cut it any more. The SysAdmin will need to turn into an on-call ops team. Data Engineering will need specialists for on-boarding vs analytics vs access layer.
If you are doing anything controversial don't forget the importance of a solid security organization which is highly challenging in the distributed computing space.
These are just the technical hurdles. Depending on the data you would be bringing in there are quite possibly legal barriers as well depending on the locality.
So again its not so much that this isn't possible, more that there hasn't been someone willing to endow the kind of funding that would be required to scale something of this nature solely for the benefit of the public good.
I've been working on a project called Aleph ( https://github.com/pudo/aleph , live: http://data.occrp.org/ ), which is targeted at investigative reporters. It's supposed to handle a related set of problems, which is data integration of diverse public records and journalistic lead generation (i.e. what to investigate next). The road map for the coming two years includes some visual analytics similar to what I guess mo…
I've been working on a project called Aleph ( https://github.com/pudo/aleph , live: http://data.occrp.org/ ), which is targeted at investigative reporters. It's supposed to handle a related set of problems, which is data integration of diverse public records and journalistic lead generation (i.e. what to investigate next). The road map for the coming two years includes some visual analytics similar to what I guess mo…
I would love to try out your tool, but I'm getting a 504 Gateway Time-out on http://data.occrp.org/ . Is there any chance I can give Aleph a spin?
I worked at palantir. As a designer. Grains of salt. The value is in the dataset. Most of what's on top is handmade by fresh grads. Spend your effort getting access to the firehose, and the rest will follow.
I worked at palantir. As a designer. Grains of salt. The value is in the dataset. Most of what's on top is handmade by fresh grads. Spend your effort getting access to the firehose, and the rest will follow.
Orly. The value of Palantir you think is in the dataset but NOT in the full stack 10 ish years old low technical risk analytical platform?
If you're a journalist or educator and want to check it out, shoot me an email, I'm andy@