Show HN: Orchest – Data Science Pipelines

We want to try to make it easier for people to understand the landscape of tools and our position within it.

I personally like something like GitLab's https://about.gitlab.com/devops-tools/. We'll try to put up something similar on our website at some point.

I'm not deeply familiar with MLFlow, but from what I have seen/read it is more of a tracking framework that you can integrate into an existing codebase.

While Orchest allows you to take your existing codebase and structure it into a pipeline to get a visual and containerized way of interacting with the codebase (allow a mix of notebooks and .py/.sh/.R scripts), running the pipeline, and visually inspecting success/failure of pipeline runs/steps.

Another key point of difference is how we are more concerned with managing the flow of data. Since we let you build pipelines we can give you abstractions to separate data flow from the pipeline code. I.e. letting you define generic pipelines that take any data source (in some standard form, like a schema'd database) and produce reports. Because we control the data source in relation to the containerized pipelines we can also make sure the whole thing performs well when it's being executed in parallel (i.e. same version of the pipeline running grid search over paramaterized pipelines). In other words, we also control more of the underlying infrastructure when executing the pipelines.

We'll allow your team to move parts of your code base from notebooks to scripts (.py/.sh/.R) to alleviate your frustration.

That way you can keep using notebooks only for those parts where it makes most sense.

They come together in pipelines that are JSON defined and git-versioned.

Yes please! It's on our roadmap. We want to be language agnostic as much as possible close to the spirit of Jupyter.

We have a lot in place already to make it easy to add languages, so if more people like it we'll have Julia support coming up relatively soon.

Agreed! ;)

Disclaimer: I've recently at my company built and released a product that has a displayed DAG on screen and allows people to build ML pipelines in addition to the existing data prep and blend capability. My company is profitable and the ML product is doing well. We also have the ability to have jupyter notebooks execute in-line with the DAG.

The comments on here are reflective of the fact that selling products like yours to typical coders is a dead-end. They don't like it or want it, not because it wouldn't be useful to them, but because they are attached to their current tooling and don't see a need for making their workflows more maintainable for non-coders. Some coders get it, after working in an office and realizing they are spending inordinate amounts of time tweaking their software for various data sources instead of focusing on the actual interesting stuff, but most don't.

Your product has the challenge of falling in a valley between the domain experts who can't code and therefore won't have notebooks available to pull into each node, and the people who can code and have notebooks not wanting to spend money or use a dirty, filthy, evil GUI instead of their precious glue code.

Your sweet spot of customers won't be on HN. They are the massive army of people with the job title "data scientist" who really aren't close to being actual data scientists. The enterprises are filled to the brim with them. Many of them can barely write Python or R, and are outright frauds. But plenty of them have a cursory knowledge of Python coding in Jupyter, and can actually do some Kaggle problems. However, basic ETL skills, web scraping, etc are foreign to them, and they have no ability to embed their shitty Jupyter notebook code into an ETL pipeline. These people aren't on HN. If they were, they wouldn't be so shitty at their job. Those people are your customers. The underqualified, borderline fraud data scientists who are EVERYWHERE. I sound like a cynical asshole.

But there's a silver lining here:

The best customers you can ever get aren't in that camp at all, and also would add a lot of value to their orgs using your tool. These folks are the domain expert analysts who really, really know their company's data and have the deep domain expertise. They are the people who data scientists have to talk to to be able to create value. You want to go after these people, and upskill them to basic python. People who know their business and it's data, and have just a little bit of data science skills create far more value than the opposite.