> Are you able to tell more about what types of data this was?I mined data in the real estate, QSR, automotive and airline sectors (and a few peripherally related ones). We would identify a source of data that was a demonstrably strong proxy for a specific company's revenue (that is to say, if we broke out a naive timeseries of the data it would map nearly 1:1 to earnings results each quarter). Then we would collect this data en masse, writing software over a custom crawler and infrastructure, bespoke to each case. During ingestion we'd process and normalize the data and dump it in a database. Once we had a significant amount of data we'd build a forecasting model, which could be very straightforward ("how many products have they sold this quarter", for simpler companies) to very complex ("can we reverse engineer the amount of business which is now online" or "can we determine the undisclosed amount of business being done in this specific region"). This analysis (and not the data itself) constituted the product - we never even provided the data directly to the hedge funds.
Once the crawling and analysis was mature, we'd incubate it for a few quarters while building interest; then, after showing a very strong mapping over several quarters, offer it as a data product. It was not uncommon to achieve The data was always public and legal, though challenging to identify and difficult to effectively crawl. It was typically derived from very uncommon sources, and would range in complexity from the extremely simple ("we can crawl sequentially incremented integers from this third party that appear to map to products sold by this company") to the very sophisticated and complex (e.g. "we've collected a zero sum distribution of ad partners and their spend on the network, are more partners churning off, and is this a better or worse outcome for the company?"). I've spoken about this in comments here before.
This is still very doable, and I still do this sort of work for personal research (and profit). However, the work I do is now much more quantitative - I'm currently taking a similar approach for baskets of equities with the goal of forecasting macroeconomic trends (e.g. subprime auto loans) instead of the earnings results of individual equities. To give a very specific throwaway example (because it doesn't violate an NDA and it no longer works): it used to be possible to pretty accurately forecast large tech retailers' product sales each quarter (like Apple) by reverse engineering FedEx and UPS tracking numbers.