Ask HN: We have a great team and capital but can't find a good idea

what industries has the PE person done deals in? in those industries, are there any sources of very large costs for businesses? think about ways to get in the middle and lower those costs.

there are probably also some spaces where existing solutions are mediocre because access to capital is more important than quality to get a business going. since you have access to capital you might be able to compete.

it’s possible to do a startup that is tech heavy while also being B2B. the classic image of a consumer facing website chasing user engagement isn’t everything.

There is a much longer version, of course, but I had to cut 75% of this before Hacker News would allow me to post it.

-----------------------------

SUMMARY:

For many years I've worked with startups involved in data mining, so I've gotten to know how the current crop of these firms operates. The companies that I've worked with rely on Machine Learning (ML) and Natural Language Processing (NLP). All of the startups that I've worked with so far, and all of the one's that I've read about, fall into 2 categories:

1.) they got into a business where they thought they could use humans to do data aggregation, and now they are desperately trying to build ML/NLP stacks to automate the work.

2.) they were certain they didn't need humans, because their Deep Neural Network was magically effective, but now they are having to rethink their business model because their Deep Neural Network has been less of a magical breakthrough than they expected.

There is a great deal of redundancy in the current efforts to find the right balance of ML/NLP and humans. One company might use Spark to ingest documents about medicine, another analyzes advertising using a Hadoop cluster, another scans the web to pull in millions of articles that it runs through Storm, where it parses the data and then stores the final results in Cassandra. Each company stumbles through a painful process of trying to figure out where it should use humans to patch the failures of its ML/NLP techniques.

In particular, there are specific patterns of using Mechanical Turk to vet those items where ML algorithms could not reach a high level of confidence. A given item can be sent to 5 different people on Mechanical Turk, and we can accept a vote of 4 or 5 as representing a high level of confidence. But if the vote is 3 or less, then we need to escalate that item to an even higher level or review. I've seen multiple companies build similar processes, which is why I think this should be a business of its own.

It's important to be aware of the limits of startups such as Bigml.com. Most of the time, data analysis is just the starting point of a longer process that involves much interpretation on the part of humans. But much of that later human vetting can be standardized, and so firms should outsource the work.

LONG VERSION:

Let's start by talking about one particular industry, which is firms that sell data about privately-held companies. There is a great conversation on Quora that summarizes the strengths and weaknesses of most of the enterprises in this field:

"How do CB Insights, PrivCo, DataFox, Owler, Tracxn!, Mattermark, and Venture Scanner compare for private company research?"

https://www.quora.com/How-do-CB-Insights-PrivCo-DataFox-Owle...

Danielle Morrill has written a great blog post about how and why she created Mattermark ("The Deal Intelligence Company"):

https://medium.com/@DanielleMorrill/introducing-mattermark-t...

According to Crunchbase, she has so far raised $17 million to build out her company. Samiur Rahman, the lead data engineer at Mattermark, has given a revealing interview about how NLP helps Mattermark pinpoint data about deals that companies may be arranging:

https://www.techemergence.com/how-natural-language-processin...

Anyone who tries to scour the Web for information about companies, either publicly held or privately owned, immediately runs into a few problems, including the fact that any given company may have dozens of subsidiaries with similar names. The law, or a regulatory agency, might have forced the company to break up. So, for instance, German law has forced Deutsche Bank to set up different companies for loans and investments. As a consequence, one finds the following names on the web:

Deutsche Bank - Corretora de Valores S.A. Deutsche Bank A S Deutsche Bank AG Deutsche Bank GmbH Deutsche Bank S.A. Banco Alemao Deutsche Bank Trust Company Americas Deutsche Bank Trust Company Delaware Deutsche Bank Journalists tend to misspell these names, or just use the generic "Deutsche Bank," which makes it difficult to discern which incarnation of Deutsche Bank is the subject of the article.

Samiur Rahman says he is a fan of "word vectors" and "paragraph vectors", and then he uses the Nearest Neighbor algorithm to figure out which companies are similar to each other. As the interview makes clear, Mattermark got into a business where they thought they could use humans to do data aggregation; now they are desperately trying to build ML/NLP stacks to automate the work. (In my taxonomy, they're a Category 1 firm.)

...Some customers buy from several of these data-analysis companies. Some buy from both CB Insights, which has the best data about deals, and also CB-Mark, which has the best estimates on revenue. The customers are often sales teams hoping to find new customers for their own companies, but sometimes the data is also used for research (as at the universities) and sometimes the data is used for VC investments, as Danielle Morrill made clear in her history of Mattermark. Right now there are something like 30 companies in this field (selling data about private companies), though I assume this will eventually consolidate to something like 3 or 4 companies.

THE HYBRID APPROACH USES BOTH HUMANS AND ML/NLP

For the foreseeable future, the best approach to data mining is a mix of ML/NLP plus humans. The ML/NLP scripts can be calibrated to produce an estimate of confidence. When there is high confidence, the entire ingestion process can be automated. When there is low confidence, the item-of-data needs to be flagged for human review.

The process I'm working on needs two levels of human review, but I'll also talk about a third, if only to dismiss it.

1.) low-level: for tasks that don't need much context, yet still can't be automated, the item-of-data should be sent to 5 different people using Amazon's Mechanical Turk. If at least 4 of these people come to the same conclusion, we can assume that this was data that low-skilled people were successful at categorizing. For instance, if an article had an ambiguous mention of Deutsche Bank, but 4 out of 5 people came to the same conclusion about which Deutsche Bank was under discussion, then we can consider this item-of-data categorized, and the rest of the ingestion process can proceed automatically.

2.) medium-level: this level of skill is needed if people with low-level skill failed to agree on how to categorize an item-of-data. If something was sent to Mechanical Turk but 5 people each reached 5 different conclusions, the data clearly needs a much closer inspection. Medium level skill review would also include large parts of medical and legal data sets, where data can be read from context by a person who lacks specific medical or legal training. "While I worked at Google I downloaded all of Google's documents for self-driving cars and now I'd like Uber to finance my new self-driving car startup" is a sentence that can be understood by an untrained layman, though the realization that this describes illegal activity is still beyond the ability of a pure ML/NLP approach. People of medium-level skill would be people who work directly with my startup. They could do the work remotely, but they would require some training, so I would have a long-term relationship with them.

3.) high-level: this is a different market, and so I'm not going to consider this. Data that requires high levels of skill might be items of medical or legal knowledge that a person needs extensive training to understand. Trying to deal with this kind of data would require a completely different business model. I'm not considering this as part of my current project.

WHO ARE MY CUSTOMERS?

My customers would be every group that:

1.) needs the output of NLP scripts...

2.) ...applied to a data source they specify...

3.) ...vetted to a high level of confidence

The market for ML and AI startups might seem crowded, but I think there is still many opportunities there, in particular, regarding standardizing the process whereby humans review the output of ML and NLP scripts.