Amundsen – Lyft’s data discovery and metadata engine
1–10 of 13 posts
Re: Amundsen – Lyft’s data discovery and metadata engine
#2We did a presentation in Strata SF 2019 last week which detail could be found at https://conferences.oreilly.com/strata/strata-ca/public/sche.... The slide could be found at https://www.slideshare.net/taofung/strata-sf-amundsen-presen... .
Re: Amundsen – Lyft’s data discovery and metadata engine
#3This was probably in development before FoundationDB was open sourced, but this is exactly the type of system where FoundationDB shines. A developer from Apple did a talk at the FoundationDB Summit describing Apple’s metadata store for machine learning that people interested in this from Lyft may like.
Re: Amundsen – Lyft’s data discovery and metadata engine
#4Re: Amundsen – Lyft’s data discovery and metadata engine
#5congrats to the team! reminds me of Airbnb's data discovery tool: https://medium.com/airbnb-engineering/scaling-knowledge-acce...
Re: Amundsen – Lyft’s data discovery and metadata engine
#6https://github.com/lyft/amundsenfrontendlibrary
there are four repos (3 for microservices, 1 for data ingestion library):
https://github.com/lyft/amundsenfrontendlibrary
https://github.com/lyft/amundsensearchlibrary
https://github.com/lyft/amundsenmetadatalibrary
https://github.com/lyft/amundsendatabuilder
Re: Amundsen – Lyft’s data discovery and metadata engine
#7https://github.com/lyft/amundsenfrontendlibrary
there are four repos (3 for microservices, 1 for data ingestion library): https://github.com/lyft/amundsenfrontendlibrary https://github.com/lyft/amundsensearchlibrary https://github.com/lyft/amundsenmetadatalibrary https://github.com/lyft/amundsendatabuilder
Nothing quite highlights the fragmentation of Python more than 3 different Python versions being called out across 4 repos.
Re: Amundsen – Lyft’s data discovery and metadata engine
#8Would someone mind explaining how ML models are used in the real world? In this case Lyft has users, trips, points of interest (destinations). What would they create a model for? How would it improve driver/rider experience?
I've gone through a few TensorFlow tutorials, but still can't grok what the appropriate use-case is.
Edit: I really appreciate the responses @tedsanders and @theossuary - thank you so much! Not sure what the etiquette is other than upvoting you!
Re: Amundsen – Lyft’s data discovery and metadata engine
#9Would someone mind explaining how ML models are used in the real world? In this case Lyft has users, trips, points of interest (destinations). What would they create a model for? How would it improve driver/rider experience? I've gone through a few TensorFlow tutorials, but still can't grok what the appropriate use-case is. Edit: I really appreciate the responses @tedsanders and @theossuary - thank you so much! Not s…
A few examples I can think of off the top of my head:
- AirBnB uses a model to recommend a daily price for a new listing. They have to base this off the area and how other similar listings are priced.
- Uber has an entire team (maybe teams) dedicated to forecasting demand at a location/time, they distribute this information to divers so they can plan to be certain places at certain times to get more fares.
- Uber collects your location 5 minutes before/after your ride. They use this (at least in part) to figure out where people normally go when they request to be dropped off at an address, and then they can optimize the drop-off location using that information.
Really though, the trick is to find a problem that can be solved with machine learning, and where profit of the solution is multiple times the cost of developing it. The only way to spot these opportunities reliably is to have a lot of machine learning experience, and quite a bit of domain knowledge. It may be worth following some tech company ML engineering blogs to get a better grasp of how they leverage ML.
[1]: https://medium.com/airbnb-engineering/aerosolve-machine-lear...
[2]: https://eng.uber.com/forecasting-introduction/
[3]: https://www.npr.org/sections/alltechconsidered/2016/12/01/50...
Re: Amundsen – Lyft’s data discovery and metadata engine
#10Would someone mind explaining how ML models are used in the real world? In this case Lyft has users, trips, points of interest (destinations). What would they create a model for? How would it improve driver/rider experience? I've gone through a few TensorFlow tutorials, but still can't grok what the appropriate use-case is. Edit: I really appreciate the responses @tedsanders and @theossuary - thank you so much! Not s…
I received an offer from Lyft last year to do data science. Here are some potential use cases of machine learning for a taxi company:
-Predicting wait times
-Predicting the best route
-Predicting where to send drivers before requests come in
-Predicting demand so that you can surge price predictively
-Predicting what assets/requests to prefetch to the phone and when
-Predicting when drivers will churn and what tactics will reduce churn
-Predicting when passenger will churn and what tactics will reduce churn (e.g., special deal for 25% off next 10 rides)
-Predicting which passengers are more or less price sensitive and then price discriminating accordingly
-Predicting what it will cost various vehicles to reroute to pick up an additional Lyft Pool passenger on the way to their destination
-Predicting the driver's position some time ahead (e.g., you don't want to send a request for them to get off the freeway right as they are passing their exit)
-Predicting car location based on fusion of GPS, accelerometers, priors from past driving data (e.g., if GPS says you're 10 m right of the freeway traveling at 60 MPH, you're actually probably on the freeway)
-Predicting which Facebook users will be most likely to click ads
-Predicting who and when to send out email marketing
-Predicting all sorts of server load balancing type stuff on the back end (proactively scaling capacity for known busy times, etc.)