No post body was provided.
Ask HN: Building out analytics department, what platforms should we use?
1–10 of 11 posts
Re: Ask HN: Building out analytics department, what platforms should we use?
#2Same here :) Right now, we're just starting to automate a bunch of work that our one overworked analyst does manually with GA, FB marketing insights, and a bunch of third party services. She is asking dev to extract data sets that crash Excel when she imports them :/ We need to integrate all of this data into one somewhat consistent stream. Then whenever she asks for a data set, we (I) need to ask her what she's going to do with it, and whip up a little stream processing service that can answer that question continuously. Preferably in chat, not with a big fancy dashboard, because really, all those graphs are just a means to an end. Her output isn't a graph, it's a single paragraph recommendation or a boolean answer to a vague question.
I've just started looking into Apache Kafka and Storm. The whole stream oriented paradigm looks really promising. If we could archive a huge torrent of incoming events, and then build little stream processing services over time, as specific questions come in from analytics, and replay old events through the system, I think that would put us in the position we want to be in. I've been playing around with Storm on my laptop in clojure (Clojure is awesomesauce), and contemplating how we should deploy this.
As far as deployment goes, what I would like to do, is set up a big fancy Mesos cluster to run all these little stream processors, and heck, might as well run all of our product services in it too. What I will probably have to settle for, is starting off on a single EC2 instance that occasionally has little accidents. My ops skill are... a work in progress.
Anyway, if you want to get in touch, shoot me an email or something. daniel.ross at kasra.co. It would be really cool to watch several of these systems built up in parallel, so that we could all see what is working and what problems need to be solved by a bigger group than our own. It isn't easy convincing the CEO to let me open source stuff like this, but it also isn't easy to hire developers who can learn a new sub-field and build something cool in it, at least not for a relatively small company.
Re: Ask HN: Building out analytics department, what platforms should we use?
#3Same here :) Right now, we're just starting to automate a bunch of work that our one overworked analyst does manually with GA, FB marketing insights, and a bunch of third party services. She is asking dev to extract data sets that crash Excel when she imports them :/ We need to integrate all of this data into one somewhat consistent stream. Then whenever she asks for a data set, we (I) need to ask her what she's goin…
Just realised how silly it is for the CEO to think that we gain more over the competition by working in isolation with a small team, than we do by collaborating with people who are highly unlikely to actually be interested in our niche. I'm going to have to thoroughly explain how this open source thing works, over a beer or three.
Re: Ask HN: Building out analytics department, what platforms should we use?
#4Same here :) Right now, we're just starting to automate a bunch of work that our one overworked analyst does manually with GA, FB marketing insights, and a bunch of third party services. She is asking dev to extract data sets that crash Excel when she imports them :/ We need to integrate all of this data into one somewhat consistent stream. Then whenever she asks for a data set, we (I) need to ask her what she's goin…
Just realised how silly it is for the CEO to think that we gain more over the competition by working in isolation with a small team, than we do by collaborating with people who are highly unlikely to actually be interested in our niche. I'm going to have to thoroughly explain how this open source thing works, over a beer or three.
The thing with open source is that when it breaks, you have no one to yell at. We've been using R/Python for sometime now and have not had any problems there at all, but it is probably their biggest concern. And I'm sure this very subject has come up so often you should be able to find some very convincing arguments around it.
Re: Ask HN: Building out analytics department, what platforms should we use?
#5 Please lead with the location of the position and include the keywords REMOTE, INTERNS and/or VISA when the corresponding sort of candidate is welcome. When remote work is not an option, please include ONSITE.
Submitters: please only post if you personally are part of the hiring company—no recruiting firms or job boards.
Readers: please only email submitters if you personally are interested in the job—no recruiters or sales calls.
You can also use kristopolous' nifty console script to search the thread: https://news.ycombinator.com/item?id=10313519.
Re: Ask HN: Building out analytics department, what platforms should we use?
#6Earlier quoted context omitted.
Just realised how silly it is for the CEO to think that we gain more over the competition by working in isolation with a small team, than we do by collaborating with people who are highly unlikely to actually be interested in our niche. I'm going to have to thoroughly explain how this open source thing works, over a beer or three.
The thing with open source is that when it breaks, you have no one to yell at. We've been using R/Python for sometime now and have not had any problems there at all, but it is probably their biggest concern. And I'm sure this very subject has come up so often you should be able to find some very convincing arguments around it.
Oh, the business peeps are totally fine with using OS. It's the idea of sharing our own work that scares them; they worry that our competitors will leapfrog us. I think that if our stuff is really that good, then we don't really need to worry about competition from the tech angle. Especially since we are dominating an obscure niche at the moment.
Re: Ask HN: Building out analytics department, what platforms should we use?
#7Please lead with the location of the position and include the keywords REMOTE, INTERNS and/or VISA when the corresponding sort of candidate is welcome. When remote work is not an option, please include ONSITE. Submitters: please only post if you personally are part of the hiring company—no recruiting firms or job boards. Readers: please only email submitters if you personally are interested in the job—no recruiters o…
This was not a job solicitation. This was a question as to the tech stack of an analytics team. Thanks.
Re: Ask HN: Building out analytics department, what platforms should we use?
#8Earlier quoted context omitted.
Just realised how silly it is for the CEO to think that we gain more over the competition by working in isolation with a small team, than we do by collaborating with people who are highly unlikely to actually be interested in our niche. I'm going to have to thoroughly explain how this open source thing works, over a beer or three.
The thing with open source is that when it breaks, you have no one to yell at. We've been using R/Python for sometime now and have not had any problems there at all, but it is probably their biggest concern. And I'm sure this very subject has come up so often you should be able to find some very convincing arguments around it.
I'm pretty sure there ARE people you can pay and yell to. Have you searched ?
Re: Ask HN: Building out analytics department, what platforms should we use?
#9We use Segment > Lambda Function > Kinesis Firehose > Redshift + S3 > Tableau
If you don't mind writing some boilerplate JS to capture things like Device, Location, etc, then you could really just all of the events straight to the lambda function via Amazon's API Gateway. This would save anywhere from $100/mo to $2k a month.
The only downside to this approach is that we had to define our analytics schema early on in the process in order to have it match up with the redshift database. If you went with elastic search instead of redshift you would probably gain some flexibility in the schema (but at the cost of a normalized dataset to wire up in Tableau or some other BI tool).
I would also be very interested to hear if anyone's used Amazon's BI service (Quicksight?) or other tools like Looker or Periscope.
Re: Ask HN: Building out analytics department, what platforms should we use?
#10I'd very much recommend you first think about the nature of your problem, data, environment, transformation and storage needs and then start evaluating platforms. Don't just take the latest buzz words and start hacking away.
Do you need batch analysis or stream processing? How much data do you have? How will you need to combine stored data with streaming data? What data needs to be in memory vs disk? How far back in time does you analysis need to go? Can you pull that off in a streaming fashion or do you need a Lambda architecture? Are all your queries known ahead of time or ad-hoc? How will you access historical events? Do you really have data that requires 'big data' tools or can you get away with R on a powerful workstation?
You don't give many details here so I can only through out questions which first need answers.
I.e. in our environment we have the whole Cloudera stack, we use map reduce for some things, spark for others and storm for others. We discovered we don't need Kafka but can get away with just a message broker. We have HDFS, Hive, Couchbase, Oracle and a few other stores all with their own sweet spot.