Earlier quoted context omitted.
Excuse my ignorance, but why is an NFS better than S3? Both are loading from disk to memory of the Tensorflow Serving container, aren't they?
NFS is faster and it looks like a normal filesystem to the app so you don't need any special file I/O code.
Ask HN: What does your ML pipeline look like?
41–50 of 50 posts
Re: Ask HN: What does your ML pipeline look like?
#42GitHub repo and a Makefile...
Cut C++ binaries, statically linked, test on local against small data, and ship them to production with scp. Run either standalone, on GPU cluster or MPI cluster.
How the data gets to where it needs to be depends on the environment.
Makefile contains all the steps : from compile to test to deploy.
Re: Ask HN: What does your ML pipeline look like?
#43GitHub repo and a Makefile... Cut C++ binaries, statically linked, test on local against small data, and ship them to production with scp. Run either standalone, on GPU cluster or MPI cluster. How the data gets to where it needs to be depends on the environment. Makefile contains all the steps : from compile to test to deploy.
That sounds terrible to me. I hope there is at least proper CI which executes stuff from the makefile, but even that, `scp` to ship something in 2019 is a step back (https://www.phoronix.com/scan.php?page=news_item&px=OpenSSH-...) .
Re: Ask HN: What does your ML pipeline look like?
#44GitHub repo and a Makefile... Cut C++ binaries, statically linked, test on local against small data, and ship them to production with scp. Run either standalone, on GPU cluster or MPI cluster. How the data gets to where it needs to be depends on the environment. Makefile contains all the steps : from compile to test to deploy.
That sounds terrible to me. I hope there is at least proper CI which executes stuff from the makefile, but even that, `scp` to ship something in 2019 is a step back ( https://www.phoronix.com/scan.php?page=news_item&px=OpenSSH-... ) .
Nah, it’s very easy to know what’s going on as a result. The endpoint is only ever a single binary that uses environment variables. Nothing mutates the environment. And the environment is only ever a place with an interface to hardware and to data.
There is a single threaded CI for each environment that checks prerequisites, builds, tests, copies, executes and saves logs.
So CI is also a simple queue and binaries remove the need for containers.
Experience has made me a believer in making things only as complicated as they need to be and favouring solutions that can fit into one brain. Do it whenever it fits your requirements and is possible.
Re: Ask HN: What does your ML pipeline look like?
#45Earlier quoted context omitted.
+1, tomasdpinho. Yes to everything, and notably the queues everywhere, versioning the models, and the issue to mix sync and async (go for queues). As a scientist designing risk management systems, I also like to: . avoid moving the data; . bring the (ML/stats) code to the data; . make in-memory computations (when possible) to reduce latency (network+disk); . work on live data instead of copies that drift out-of-date;…
+2 for bringing the (ML/stats) code to the data instead of the other way around
Could you speak to your experience with this particular list item?
Re: Ask HN: What does your ML pipeline look like?
#46We're building a deployment system based on the following principles that we've learned: - Version control of models and datasets - Code, naming conventions and formatting consistency - Testing of models before deployment into production (in most cases, this helps gain credibility across engineering) - Model review process with a senior data scientist - Kubernetes/Docker for deployment serving as an API or running a…
Could you also give us some details about the software you specifically use in the pipeline, other than kubernetes/docker? Do you use any available (possibly open source)? tools for versioning and monitoring? Or are you building this all up from scratch for your needs?
Re: Ask HN: What does your ML pipeline look like?
#47Earlier quoted context omitted.
I do machine learning work in healthcare, and work for a HIPAA covered entity. The issue of permissions and data access often gets applied in an unnecessarily strict fashion to data scientists in these environments, often due to a lack of understanding from engineering managers who have been brought in from a non-regulated environment (e.g., hiring a salesforce engineering manager into a healthcare system so they can…
I'd just like to point out that "the minimum necessary for their job" is the reason many engineering managers apply unnecessarily strict rules. It's very difficult to build rules and policies that allow broad access while maintain minimum necessary. Some project may be completely justified in accessing "all" (waves hands) data at its conception but slowly morphing to focus on only a few key identifiers while still pr…
Totally, and to extend that further, one employee may work on multiple projects for which the "minimum necessary" is different. Part of my job involves patient matching and reporting patient-level data to partners we have a BAA with. That means I need access to patient names and addresses. However, if I'm working on training some ML model to predict diabetic progression, it's not necessary for me to pull the names of patients.
I think there's an incorrect assumption in here that there exists a technical solution which entirely solves this problem; that we just need to figure out what the right set of rules are, or get the right column-level and row-level security policies in place and we're all set. It's necessary but not sufficient to have those kinds of safeguards in place. You also need to trust somebody in the organization, and you need to give those somebodies training and support to do the right thing.
In my case, I need access to all of the (clinical) data within the organization. I don't really care how that end is achieved: with one account that has every permission, with multiple accounts that are used for different purposes, or whatever. Ultimately, it's in the interest of the organization to make sure that I have the access I need to successfully do my job.
Re: Ask HN: What does your ML pipeline look like?
#48Here’s a framework we’ve been developing for this purpose, delivered as a python cookiecutter: https://github.com/hackalog/cookiecutter-easydata The framework makes use of conda environments, python 3.6+, makefiles, and jupyter notebooks, so if that tooling fits into your data/ML workflow. We gave a tutorial on the framework at Pydata NYC, and it’s still very much in active development - we refine it with every new p…
+1 for mentioning this. Took it out for a test run. Really neat stuff.
Re: Ask HN: What does your ML pipeline look like?
#49We're building a deployment system based on the following principles that we've learned: - Version control of models and datasets - Code, naming conventions and formatting consistency - Testing of models before deployment into production (in most cases, this helps gain credibility across engineering) - Model review process with a senior data scientist - Kubernetes/Docker for deployment serving as an API or running a…
Could you also give us some details about the software you specifically use in the pipeline, other than kubernetes/docker? Do you use any available (possibly open source)? tools for versioning and monitoring? Or are you building this all up from scratch for your needs?
We use Jenkins for our builds (and soon simple testcases), which then feeds into our system that's built from scratch. However, we are looking to productized the system as we had some discussions with other corporate partners who are interested in as well.
The system we've built in-house is fairly simple to keep development fast - versioning is manual (https://packaging.python.org/guides/hosting-your-own-index/) but no reason why you couldn't use a repository manager.
For monitoring, we essentially track any activity related to the model including inputs, outputs, timestamps, duration, etc. to a database and have JavaScript charts render. We might put this into Kafka but seems overkill at the moment and likely force us to hire an actual support team.
Re: Ask HN: What does your ML pipeline look like?
#50Earlier quoted context omitted.
+2 for bringing the (ML/stats) code to the data instead of the other way around
Could you speak to your experience with this particular list item?
We deal with fairly large volumes of data on a frequent basis so it would not make sense for each data scientist to create a copy within their own environment. Everyone works off a centralized data source and we provide them with Jupyter/Spark in an internal cloud environment.