Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

One year as a solo dev building open-source data tools without funding

datastation.multiprocess.io

Post reply on HN
Print view View original

31–40 of 120 posts

Re: One year as a solo dev building open-source data tools without funding

#31

Post by Fiahil »

> How do you query this data or join or filter across such disparate sources? The only two solutions today are to put ALL data into a warehouse or to write custom scripts. The problem with warehouses are that they are expensive and that the ETL process for every new database is expensive too.

> Tons and tons of companies are trying to solve this. You'll get lost among all the vendors trying to capitalize on the “Modern Data Stack”. They're expensive.

Yes, and Yes. However I see a path for a new kind of warehouse that isn't a "Git for Data" or a Databricks' product.

"Topics" (Kafka terminology) containing versioned datasets could help organise and discover what is available in a given repo. The Data versioning would help ensuring reproducibility of any AI pipeline consuming those, because artefacts would be kept unaltered "forever". And finally, availability would be supported by a simple set of read-only replicas.

Your best alternative today consist of stuffing a blob storage with parquet files, and giving write access to a proxy (or to everyone, if you want to live dangerously). Where are append-only semantics ? Conflict-free artefact creation with concurrent reads ?

There is massive opportunities here, it's almost exciting !

PS : > I'm particularly interested in ending up at a database or analytics company. So if you're a database or analytics company hiring managers or developers, feel free to message me!

I see you in the thread, so does a consulting company with a massive Data Science arm is of any interest to you ? :)

Re: One year as a solo dev building open-source data tools without funding

#32

Post by jensneuse »

jasfi wrote: post #24
jensneuse wrote: post #13

Earlier quoted context omitted.

The marketing guy, I found on YC Startup School Co-Founder Match ( https://www.startupschool.org/ ), the Developer was creating GraphQL Open Source projects that were related to what I was building, so I've got in touch with them. I can definitely recommend Startup School, it helped me build a lot of useful connections.

How did you determine how to split equity? Edit: I'm not looking for actual %s, just the factors involved.

You know you've got the right equity split when everybody thinks it's fair and the team can focus 100% on execution. Factors can be time spent on the project, experience, who "owns" what, connections, etc... But in the end, if you have to discuss this topic too much with someone, you know it's not going to work. If the Co-Founders cannot easily come to the same result, their mindset doesn't align well enough, which can be a risk for the whole operation. So, I'd advise against haggling too much, for both sides. Also, I'd suggest to discuss this topic very early on. Don't waste time if you don't know their expectations are in the same ballpark as yours. If you get the right people to commit helping you, I'd rather give them more equity, when the other option is to keep doing this on my own forever. Ideally, you can all "start" at the same time and split equity equally, but the reality is that you don't have all the stars aligned on day one.

Re: One year as a solo dev building open-source data tools without funding

#33

Post by eatonphil »

Fiahil wrote: post #31

> How do you query this data or join or filter across such disparate sources? The only two solutions today are to put ALL data into a warehouse or to write custom scripts. The problem with warehouses are that they are expensive and that the ETL process for every new database is expensive too. > Tons and tons of companies are trying to solve this. You'll get lost among all the vendors trying to capitalize on the “Mode…

> I see you in the thread, so does a consulting company with a massive Data Science arm is of any interest to you ? :)

ha, maybe! You have my email.

Re: One year as a solo dev building open-source data tools without funding

#34

Post by tmcw »

eatonphil wrote: post #5

Hey this is my story! It's been a genuinely fun year. Looking forward to doing better next time. Datastation/dsq are in a great spot and I'm looking forward to them continuing to grow. Happy for your suggestions and flames.

Really liked the post! I really admire the attitude & intention that's there – at least from an outside perspective it seems like you've made the most of the experience and sort of had fun with it. Hope to take some inspiration from that.

Re: One year as a solo dev building open-source data tools without funding

#35

Post by Sytten »

jensneuse wrote: post #9

Thanks for this writeup, I can relate 100%, especially the part on the supportive spouse. I've built WunderGraph ( https://wundergraph.com/ ) as a solo-dev for multiple years until I reached a similar point like you. It's weird to say, but my wife kind of accepted that I'd work on weekends between 1-3 pm when the kids sleep. What helped me get out of this "solo-time-drain-thing" was to find the right Co-Founders to h…

I am currently building Caido (https://caido.io) without funding and the best decision we ever made was to build it as a team. The second decision that I made differently from my first attempt at a startup was to drop all my freelance contracts and only focus on that project. You need a lot of mental space to build a business and I found I almost burned myself out when I tried to build it while working on other projects. Last decision was to move back to my parent's place to cut cost and live frugally for the year (Same as the author we tried to get funding and mostly failed. In the end I think it is for the best as I am not a fan of the growth at all cost, I much prefer to build a smaller sustainable business if that is still possible. One thing that I wish we had is a more experienced person in the team since we burned time and money for stuff we didn't know better (small tear when I see my lawyer bill). It is a great experience otherwise, I just hope it will go somewhere.

Re: One year as a solo dev building open-source data tools without funding

#36

Post by cube2222 »

Fiahil wrote: post #31

> How do you query this data or join or filter across such disparate sources? The only two solutions today are to put ALL data into a warehouse or to write custom scripts. The problem with warehouses are that they are expensive and that the ETL process for every new database is expensive too. > Tons and tons of companies are trying to solve this. You'll get lost among all the vendors trying to capitalize on the “Mode…

I think Pachyderm[0] is exactly what you're describing.

[0]:https://www.pachyderm.com

Re: One year as a solo dev building open-source data tools without funding

#37

Post by heisenbit »

jensneuse wrote: post #17
ushakov wrote: post #14

Earlier quoted context omitted.

> working a full time job and building WunderGraph at night and on the weekends this is actually a terrible advice, although i'm glad it worked out for you! this type of side-hustling can quickly turn into losing your startup or your job (or both) better approach is to save some money, leave for a year and become profitable from day 1 profitability will decrease your chances of failure dramatically

I actually lost my job. I'd also advise against replicating what I did, but it was my only option really. I'm not born rich, single income dad, paying off the house. I wasn't able to afford to "leave for a year". It's also a timing thing. The right time to do WunderGraph is now, not in two years. So I had to decide to give up a dream or work through this and risk my job. Luckily, I've found the right people to help m…

This is really the one challenge in Germany: High fixed costs and high taxes making it hard to sock away seed money yourself.

Re: One year as a solo dev building open-source data tools without funding

#39

Post by johnnymorgan »

jensneuse wrote: post #9

Thanks for this writeup, I can relate 100%, especially the part on the supportive spouse. I've built WunderGraph ( https://wundergraph.com/ ) as a solo-dev for multiple years until I reached a similar point like you. It's weird to say, but my wife kind of accepted that I'd work on weekends between 1-3 pm when the kids sleep. What helped me get out of this "solo-time-drain-thing" was to find the right Co-Founders to h…

A good team is worth its weight in gold.

This is great advice, also take the time to find people that are passionate about what your are doing.

Re: One year as a solo dev building open-source data tools without funding

#40

Post by Fiahil »

cube2222 wrote: post #36
Fiahil wrote: post #31

> How do you query this data or join or filter across such disparate sources? The only two solutions today are to put ALL data into a warehouse or to write custom scripts. The problem with warehouses are that they are expensive and that the ETL process for every new database is expensive too. > Tons and tons of companies are trying to solve this. You'll get lost among all the vendors trying to capitalize on the “Mode…

I think Pachyderm[0] is exactly what you're describing. [0]: https://www.pachyderm.com

We used pachyderm before MLOps was even a thing, sadly, back then they tried to solve everything at once while failing with the most basic needs.

If your data versioning "Utilizes a Git-like structure that enables effective team collaboration through commits, branches and rollbacks", then it means you can only consume one version at a time in your pipeline. This is, unfortunately, a fatal flaw. Especially when working with time series !

Post reply on HN