Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Map of my personal data infrastructure

beepb00p.xyz

Post reply on HN
Print view View original

51–60 of 81 posts

Re: Map of my personal data infrastructure

#51

Post by karlicoss »

throw345hn wrote: post #50
karlicoss wrote: post #45

Earlier quoted context omitted.

Not the first time, but possibly the biggest thing I've drawn in it... There definitely are some weird things when you try to plot complicated things, fighting with weird placement, clusters etc. But not sure if it's me or Graphviz to blame for this. But I don't really know a better tool. If I knew how the diagram would look in hindsight I might have drawn in manually in inkscape or something, but when I started I di…

Working with graphs, I realized its hard to generate dynamic ones and you inevitably end up with domain specific layouts. I guess thats the nature of graph drawing algorithms, its usually specific to the problem. Thanks for the code, taking at look at that.

Probably too messy to understand what's exactly happening -- but my main takeaway is that if you implement your own DSL (for Graphviz at least), implement in such a way that you can freely mix DSL and raw bits. That way it's very easy to experiment or tweak minor bits without rewriting half of the code.

Re: Map of my personal data infrastructure

#55

Post by brokenmachine »

borisjoffe wrote: post #35

I love the emphasis on data export / analysis and ownership of your data. I also love the quantified self aspects of this. However as I've gotten older, I've learned that it's incredibly freeing to at least partially let go of some of your data and not worry if it gets lost or erased. Tracking and managing all of this sucks up a lot of time and mental effort, that you could be using to progress in other areas of your…

The problem that I have is that three years down the track I will suddenly think of a topic and remember just a few details of a really good blog post or comment thread that I would like to read again, but I can never find it.

I'm just not very good at knowing in advance which is the 0.001% of posts I will want to read again.

Storage is so cheap that in my ideal world, I would like to store everything I read and post. But that's difficult as you mentioned.

Re: Map of my personal data infrastructure

#56

Post by johnwyles »

karlicoss wrote: post #34
systemvoltage wrote: post #30

Hello author, hope you don't mind the negative comments since this isn't for everyone including myself. I found your dedication to organization very impressive and only shows that you'd do the same professionally.

Hi, thanks for kind words :) I don't really see them as negative (or at least don't get offended). It seems that people are on a spectrum of organization from not even using a calendar and never taking notes ("if it's important I'll remember") to what I do (and maybe even more extreme??). And it's okay! Some people suffer from information overload, some people suffer from inefficiencies and forgetting things, and you…

So I've been waiting patiently for something equivalent to Home Assistant but for QS and you pop up often. You're probably most expert to say what progress is being made for an integrated QS platform whereby you plop in a new integration and, even if there are a few manual steps (exporting, running a local software tool, etc.) you easily can import, slice on, and chart your data in a time series DB (similar to how with InfluxDB and Grafana you can do with Home Assistant). There were a few open source tools out there on GitHub but the ones I found were dated or didn't seem to have much love or enough of an audience to support them.

OH! I just saw you are the author of HPI which I have starred - it certainly was as close as I found but I pined for more evaluating it about half a year ago. How is progress? What other wise words can you lend?

Re: Map of my personal data infrastructure

#57

Post by elseweather »

z77dj3kl wrote: post #53

Although there is a lot of data here, it seems all of it is roughly repetitive, relational data. Why not store it in a central database? I have a (much smaller project) that stores all data in a postgres datastore.

There's an explanation here: https://beepb00p.xyz/unnecessary-db.html (the link behind "Against unnecessary databases")

Re: Map of my personal data infrastructure

#58

Post by teraku »

karlicoss wrote: post #34
systemvoltage wrote: post #30

Hello author, hope you don't mind the negative comments since this isn't for everyone including myself. I found your dedication to organization very impressive and only shows that you'd do the same professionally.

Hi, thanks for kind words :) I don't really see them as negative (or at least don't get offended). It seems that people are on a spectrum of organization from not even using a calendar and never taking notes ("if it's important I'll remember") to what I do (and maybe even more extreme??). And it's okay! Some people suffer from information overload, some people suffer from inefficiencies and forgetting things, and you…

I really enjoy your blogposts for two reasons:

1) They teach me things I'm interested in

2) They are really well structured and start of with a hypothesis or premise

To elaborate:

1) On the spectrum you mentioned I'm more on the organized side, though not as extreme as you. A lot of my colleagues are really disorganized and they picked up some pointers from me and our collaboration has become much more efficient by them just taking a few more notes and sharing instead of me always starting at 0 (or 1) when we start a conversation. Everybody should find his own comfy spot on the spectrum, but people should try to not make others miserable by relying entirely on external knowledge.

2) Recently I read a few books which follow this structure (which is also the standards for scientific papers) and it's just so refreshing to the usual brain dump blog post. I do enjoy some of these brain dumps, especially if they are more for entertainment and less so for teaching, but sometimes it is exhausting if a blog post aims to answer a question, but it's not clear where it's going in the beginning and you just keep on reading.

Re: Map of my personal data infrastructure

#59

Post by teraku »

mikelago wrote: post #36

Thank you for sharing this! This genre of software tool is something that has been on my mind since at least 2010. Here* is a quick summary of my idea. I even applied to the first YC Fellowship in 2015 with the idea, but I didn’t make it past the application. I’ve come across many related ideas over years of reading HN (the general consensus is to just use org mode, so far I’ve been strictly a vim user xD), but this…

I find it weird to tell people which exact tools to use.

I also really enjoy knowledge management and I think I would really love org mode once I figured it out, but I just never got around to it. I am a Sublime user and really enjoy it, so I found other tools for my use cases and so far I am happy with the results. Could org mode provide me more value? Maybe. But it's not the point.

Somebody might be taking more notes than others, because they are used to it, or because they need it, and it's totally okay and people should do what works for them. If your data was truly yours (data affiliation vs data ownership) this would be even less of a problem.

Re: Map of my personal data infrastructure

#60

Post by MarkSweep »

generalizations wrote: post #2

The way I see it, if you need a map then it's too complicated.

If you are doing everything by hand, perhaps. But if you are working at a higher level of abstraction using something like Apache Airflow, it will take care of making the map for you.
Post reply on HN