Asked to get some kind of analytics query to someone, and they need it fast and want it in some kind of visualization tool.
1. You open Zeppelin, take a bunch of database tables and whip out a query that is basically what they want and export it to an Elasticsearch + kibana instance.
Now comes the edge cases:
Oh I forgot, it needs to be on the internet
2. Need to set up a public IP, DNS, Nginx server and a series of rules to make it read-only (and it's still dangerous mind you)
Why isn't this password protected?
3. Add a nginx basic-auth with a single password
It needs to be available to admins and sales managers only.
4. Set up ngx_http_auth_request_module to hit the our authentication server (the cookies should be present - OHH the DNS name doesn't match the cookies. Set up a /subpath on the existing application.
It needs to work as an embedded view in the company's mobile app.
5. That uses tokens not cookies, so the auth-request module no longer works for this, need to come up with a SSO solution with a cookie and a place to store the cookie in a database. That requires a REST service on the existing app server, which will require a redeploy.
I just added a product to our system and it's not in Kibana
6. Need to modify the spark code to use spark streaming
The Spark server restarted and my new products aren't showing up!
7. Need to set up a service on the system to auto-start the spark job.
It is feasibly an actual product feature at this point, but (1) was asked for, but they really wanted 1 - 7. I would argue that (1) would only take 25% the time 2-7 takes to do. Not every product change is like that, obviously, but sometimes people think all changes are just so simple. Often it's the history and unstated features that make a huge difference.
Also this for fun: http://outofmymind.scanlen.com/wp-content/uploads/2011/04/Wh...
