Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Map of my personal data infrastructure

beepb00p.xyz

Post reply on HN
Print view View original

41–50 of 81 posts

Re: Map of my personal data infrastructure

#41

Post by mathgladiator »

karlicoss wrote: post #34
systemvoltage wrote: post #30

Hello author, hope you don't mind the negative comments since this isn't for everyone including myself. I found your dedication to organization very impressive and only shows that you'd do the same professionally.

Hi, thanks for kind words :) I don't really see them as negative (or at least don't get offended). It seems that people are on a spectrum of organization from not even using a calendar and never taking notes ("if it's important I'll remember") to what I do (and maybe even more extreme??). And it's okay! Some people suffer from information overload, some people suffer from inefficiencies and forgetting things, and you…

Here are some more kind words: I appreciate you.

Thank you for sharing.

Re: Map of my personal data infrastructure

#42

Post by karlicoss »

generalizations wrote: post #31
karlicoss wrote: post #18

Earlier quoted context omitted.

Hi, author here. Yes, it's complicated and this is exactly one of the points of this map, to illustrate how ridiculous it is -- that I have to jump over all these hoops to do something useful with the data that's already somewhere in my phone/was in my browser, but hidden and siloed (from myself!) for various reasons.

Fair enough. The effort and quality of your work is apparent - well done. I guess for me, I would have opted for greater inconvenience and simply let some of what you graphed slide, rather than deal with the mental overhead of actually tracking it all. But I would also not have enjoyed building that graph. :)

Thanks :) The thing is -- often it's very hard to know in advance how much time you would spend on something. You think "ah this would be kind of a nice feature to have/question to answer... hmmm it'll probably take only an hour", and then you spend weeks doing that, and something else related, etc. Sometimes you do wish you hadn't bothered in the first place, but then it's also sunk cost so it's tempting to finish.

You know what's crazy -- I started with simply trying to improve my sleep (so exorting data from Jawbone bracelet) and wanting to be a bit more efficient at learning physics (and other things) in my spare time. I did learn some physics (and other things) thanks to all these systems, but certainly went down a massive rabbit hole too :)

Re: Map of my personal data infrastructure

#43

Post by sodality2 »

karlicoss wrote: post #34
systemvoltage wrote: post #30

Hello author, hope you don't mind the negative comments since this isn't for everyone including myself. I found your dedication to organization very impressive and only shows that you'd do the same professionally.

Hi, thanks for kind words :) I don't really see them as negative (or at least don't get offended). It seems that people are on a spectrum of organization from not even using a calendar and never taking notes ("if it's important I'll remember") to what I do (and maybe even more extreme??). And it's okay! Some people suffer from information overload, some people suffer from inefficiencies and forgetting things, and you…

This is incredible. If you enjoy it, more power to you. I hope I can manage my digital life with the efficiency and scale you've reached.

Re: Map of my personal data infrastructure

#44

Post by throw345hn »

karlicoss wrote: post #17

Hey author here. Happy to answer your questions!

Can I ask - how did you find working with graphviz. Was this your first time working with it, how long did it take you to figure out how to place things and draw?

I am working with drawing graphs at the moment and evaluating what library to use, its coincidentally nice to see what you built but it also seems like it was thoughtfully laid out (less edge crossings etc). I am trying to do something more dynamic so it may not be as applicable for me.

I am looking at your code as well, thanks for providing that

Re: Map of my personal data infrastructure

#45

Post by karlicoss »

throw345hn wrote: post #44
karlicoss wrote: post #17

Hey author here. Happy to answer your questions!

Can I ask - how did you find working with graphviz. Was this your first time working with it, how long did it take you to figure out how to place things and draw? I am working with drawing graphs at the moment and evaluating what library to use, its coincidentally nice to see what you built but it also seems like it was thoughtfully laid out (less edge crossings etc). I am trying to do something more dynamic so it ma…

Not the first time, but possibly the biggest thing I've drawn in it...

There definitely are some weird things when you try to plot complicated things, fighting with weird placement, clusters etc. But not sure if it's me or Graphviz to blame for this. But I don't really know a better tool. If I knew how the diagram would look in hindsight I might have drawn in manually in inkscape or something, but when I started I didn't know what I would end up with, so needed to be an automatic tool :)

To minimize the manual work, I ended up with a mix of DSL in python and raw graphviz commands: https://github.com/karlicoss/myinfra/blob/fc6345c31c4e49b534...

Depending on the things you want to represent a better fit might be force layout, for example something like https://observablehq.com/@morvasaaty/d3-force-notes

Re: Map of my personal data infrastructure

#46

Post by redux-xplatform »

throw345hn wrote: post #44
karlicoss wrote: post #17

Hey author here. Happy to answer your questions!

Can I ask - how did you find working with graphviz. Was this your first time working with it, how long did it take you to figure out how to place things and draw? I am working with drawing graphs at the moment and evaluating what library to use, its coincidentally nice to see what you built but it also seems like it was thoughtfully laid out (less edge crossings etc). I am trying to do something more dynamic so it ma…

If you need to be able to diff the code for graphs, graphviz (dot language) is great.

If you need auto-layout, it's hard to do much better than graphviz. The creators of graphviz have some good papers on what it takes to do that right. They'll discourage you from wanting to re-invent that particular wheel, if you read them. Tricky bit here is, depending on your exact graph and constraints, it's not exactly fully solvable in the general case, if you want to have no overlap or no crossing lines under any circumstances whatsoever. Still, you'll struggle to do better, and there aren't actually a ton of implementations out there other than Graphviz that are anywhere near as good (at least, as of a couple years ago).

Templating or building programmatically? Great, it's just text. Generating on demand? Decent, it's fairly fast.

If you need pretty, and I mean really pretty, and especially pretty and interactive... well it's less enjoyable for that, but then nothing that is enjoyable for that purpose is much good at the other points above.

Re: Map of my personal data infrastructure

#48

Post by for_i_in_range »

karlicoss wrote: post #37
for_i_in_range wrote: post #32

Earlier quoted context omitted.

Big fan of yours. I think this is the second time I’ve come across your site on HN. I’ve been getting deep into developing my own Second Brain lately. I glanced briefly at your Exobrain Repo and it seems you’re using a custom system. Have you tried out using https://obsidian.md or the whole Wikilink/Zeitelkasten Personal Knowledge Management methodology? Curious to hear your experiences.

Yep, exobrain is using org-mode export + some tweaks. I've settled on emacs + org-mode a few years ago (perhaps on the fifth attempt it finally clicked). Before it was gitit/Zim/sublime text/random scripts, whatnot. All the shiny apps like Obsidian/Roam/etc have appeared over the last couple of years (and it's awesome!). But I'm too hooked onto emacs now, maybe the only thing I should use more is org-roam [0]. For me…

Looking forward to diving into more detail on this. Thank you for sharing your work and appreciate your engagement in here!

Re: Map of my personal data infrastructure

#49

Post by throw345hn »

redux-xplatform wrote: post #46
throw345hn wrote: post #44

Earlier quoted context omitted.

Can I ask - how did you find working with graphviz. Was this your first time working with it, how long did it take you to figure out how to place things and draw? I am working with drawing graphs at the moment and evaluating what library to use, its coincidentally nice to see what you built but it also seems like it was thoughtfully laid out (less edge crossings etc). I am trying to do something more dynamic so it ma…

If you need to be able to diff the code for graphs, graphviz (dot language) is great. If you need auto-layout, it's hard to do much better than graphviz. The creators of graphviz have some good papers on what it takes to do that right. They'll discourage you from wanting to re-invent that particular wheel, if you read them. Tricky bit here is, depending on your exact graph and constraints, it's not exactly fully solv…

Thanks for that, will try out graphviz. I was doing some custom implementation for minimizing edge crossings and it seems to work fine for me but thats mostly for 2-layered graphs. If I am to expand to multi-layer graphs, its going to be fairly more complicated to figure out how to do that and I dont know if it will work well.

Graphviz seems to be great for getting you a mostly-useful auto layout majority of the time.

I have a requirement to be able to mix fixed-positioned nodes along with dynamically changing nodes, I havent used graphviz to know if it could do that. A library like cytoscape works ok or maybe d3 (I am still evaluating these)

Honestly coming into this, I thought graph drawing would be fairly easy and I could look up online for examples - but it seems to be quite opposite, its really complicated to get graphs that are both dynamically generated with the nodes positioned at the same place in case there are similar nodes

Re: Map of my personal data infrastructure

#50

Post by throw345hn »

karlicoss wrote: post #45
throw345hn wrote: post #44

Earlier quoted context omitted.

Can I ask - how did you find working with graphviz. Was this your first time working with it, how long did it take you to figure out how to place things and draw? I am working with drawing graphs at the moment and evaluating what library to use, its coincidentally nice to see what you built but it also seems like it was thoughtfully laid out (less edge crossings etc). I am trying to do something more dynamic so it ma…

Not the first time, but possibly the biggest thing I've drawn in it... There definitely are some weird things when you try to plot complicated things, fighting with weird placement, clusters etc. But not sure if it's me or Graphviz to blame for this. But I don't really know a better tool. If I knew how the diagram would look in hindsight I might have drawn in manually in inkscape or something, but when I started I di…

Working with graphs, I realized its hard to generate dynamic ones and you inevitably end up with domain specific layouts. I guess thats the nature of graph drawing algorithms, its usually specific to the problem.

Thanks for the code, taking at look at that.

Post reply on HN