Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Map of my personal data infrastructure

beepb00p.xyz

Post reply on HN
Print view View original

71–80 of 81 posts

Re: Map of my personal data infrastructure

#71

Post by karlicoss »

teraku wrote: post #58
karlicoss wrote: post #34

Earlier quoted context omitted.

Hi, thanks for kind words :) I don't really see them as negative (or at least don't get offended). It seems that people are on a spectrum of organization from not even using a calendar and never taking notes ("if it's important I'll remember") to what I do (and maybe even more extreme??). And it's okay! Some people suffer from information overload, some people suffer from inefficiencies and forgetting things, and you…

I really enjoy your blogposts for two reasons: 1) They teach me things I'm interested in 2) They are really well structured and start of with a hypothesis or premise To elaborate: 1) On the spectrum you mentioned I'm more on the organized side, though not as extreme as you. A lot of my colleagues are really disorganized and they picked up some pointers from me and our collaboration has become much more efficient by t…

Thank you! Glad you like the structure too -- sometimes I wish I could do more free-form writing too, but so far it was more of "have a problem -- convince yourself it's important to solve -- come up with solution -- document a narrative how you did it" :)

Re: Map of my personal data infrastructure

#72

Post by karlicoss »

randyrand wrote: post #62

There is a way to solve all your API issues, a camera recording your entire life! Anything you interact with will be recorded. Then the problem just becomes processing it all =P Unfortunately, it's not very ergonomic yet.

Yep. Even simpler (and less 'creepy'), e.g. OCR against your own screens, in theory, could provide quite a lot of data for free. But not very practical yet, unfortunately?

Re: Map of my personal data infrastructure

#73

Post by rektide »

borisjoffe wrote: post #35

I love the emphasis on data export / analysis and ownership of your data. I also love the quantified self aspects of this. However as I've gotten older, I've learned that it's incredibly freeing to at least partially let go of some of your data and not worry if it gets lost or erased. Tracking and managing all of this sucks up a lot of time and mental effort, that you could be using to progress in other areas of your…

> However as I've gotten older, I've learned that it's incredibly freeing to at least partially let go of some of your data and not worry if it gets lost or erased.

I'm ok with people opting to let it go.

What's uncool is how much effort it takes, how ridiculous it is to opt to do anything other than let it go. To marshal oneself & one's data should not be a huge sprawling saga, fought for tooth & nail every step of the way. Resisted at every step.

This is not an ok condition for humanity to try to live under. As Geohot said[1], we are coming to live in an eternal prison. Providing the advice to let it go, find happiness in these conditions, is what I hear again and again and again on this site, and I'm mad. I'm mad people advocate for giving up agency, will, control, for what feel like the most core genuine & true pieces of humanity. I'm ok with the option, absolutely, but only as a choice. And today there is no choice at all.

[1] https://geohot.github.io/blog/jekyll/update/2021/01/18/techn...

Re: Map of my personal data infrastructure

#74

Post by coolgeek »

This is pretty amazing from an analytical/engineering standpoint.

OTOH, it serves to justify my natural inclination to limit where and how often I participate on sites and/or particular media (e.g. chat apps).

It's not just a matter of data/privacy, either. It's simply a matter of attention, available time, and the best use of that time.

I went from BBSs, to Usenet News, to (mostly) niche sites on the web. The tendency of all of these to focus conversations made them (and make them) more useful to me - better uses of my time and attention - than social media.

I pretty much only comment on this site, and on reddit, and only sporadically on either. I pretty much ignore almost all social media. Deleted facebook. Haven't regretted it for a moment.

I understand that social media provides real value - in certain situations - for people. I just don't need it, personally.

Re: Map of my personal data infrastructure

#76

Post by karlicoss »

rektide wrote: post #75
karlicoss wrote: post #17

Hey author here. Happy to answer your questions!

Which of these data-streams do you most often find yourself looking at & using? Are there any that you have been surprised to find yourself using more than you expected?

The most useful are perhaps the one's I'm using for search [0] -- chat logs, web annotations, book highlights, reddit/hackernews saves, tweets, etc. It's literally my external memory, I'm not exaggerating.

Everything in Promnesia [1] is super useful, makes my internet lurking much ... efficient? This word has somewhat bad rap, but it is what it is -- doesn't mean I don't have fun, just makes it easier to keep track and discover new things.

Not sure about surprising data sources -- I'm trying to integrate them all as far as possible and make sure they work for me passively, so I don't have to think about specifics :)

[0] https://beepb00p.xyz/pkm-search.html

[1] https://beepb00p.xyz/promnesia.html

Re: Map of my personal data infrastructure

#77

Post by karlicoss »

jpmattia wrote: post #69
karlicoss wrote: post #17

Hey author here. Happy to answer your questions!

Kudos! I find this work fascinating. There are moments that I'm tempted to go down a similar organizational road because I'm curious about the possible lessons from the data aggregation, but I usually pass because the payoff didn't seem large enough. So that makes me want to ask: Have you learned much from the data? or is it more about organization/keeping control of life?

Perhaps mostly about organization, yeah.

The "quantified self" bit is what you could call learning, but ironically so far I've mostly failed to find any significant correlations in my personal data (which is also an interesting finding, in a way :) ). However I did learn a lot about nutrition/exercise/sleep while tinkering with data, so it was totally worth it so far.

I write about it a bit more here: https://beepb00p.xyz/tags.html#quantifiedself

Re: Map of my personal data infrastructure

#78

Post by karlicoss »

johnwyles wrote: post #56
karlicoss wrote: post #34

Earlier quoted context omitted.

Hi, thanks for kind words :) I don't really see them as negative (or at least don't get offended). It seems that people are on a spectrum of organization from not even using a calendar and never taking notes ("if it's important I'll remember") to what I do (and maybe even more extreme??). And it's okay! Some people suffer from information overload, some people suffer from inefficiencies and forgetting things, and you…

So I've been waiting patiently for something equivalent to Home Assistant but for QS and you pop up often. You're probably most expert to say what progress is being made for an integrated QS platform whereby you plop in a new integration and, even if there are a few manual steps (exporting, running a local software tool, etc.) you easily can import, slice on, and chart your data in a time series DB (similar to how wi…

Yep, that's me as well :)

As for progress, I'm experimenting with automatic InfluxDb/Grafana integration right now, actually [0]! I think ideally HPI would be able to automatically create influxdb mirror + 'reasonable' Grafana dashboards for each data source, by using type information. And of course, it would also be possible for the user to create custom dashboards (from Grafana, or from python code which would automate the manual Grafana steps).

For something Memex-like, Grafana probably won't be enough. But I'd like to use existing open source tools to the maximum extent possible -- both Influx/Grafana/Jupiter/etc are awesome, well optimized and powerful instruments, and I don't want to reinvent the wheel.

[0] https://twitter.com/karlicoss/status/1361100437332590593

Re: Map of my personal data infrastructure

#79

Post by z77dj3kl »

elseweather wrote: post #57
z77dj3kl wrote: post #53

Although there is a lot of data here, it seems all of it is roughly repetitive, relational data. Why not store it in a central database? I have a (much smaller project) that stores all data in a postgres datastore.

There's an explanation here: https://beepb00p.xyz/unnecessary-db.html (the link behind "Against unnecessary databases")

To me, it just seems like the author wants to hoard their data, not make it useful. Making it useful (the first part of which is making it queriable) involves making some kind of interface to access it, and some code to parse it. In my own case, I find that just dumping JSON docs around the place gives me an unmanageable mess of unstructured data that I never come back to. If all you want to do is to hoard your data, then I guess that achieves the purpose... But if you want to do something with it, at some point you have to make sense of it.

Doesn't mean you have to go and fully normalize everything and make a production-level database!

Re: Map of my personal data infrastructure

#80

Post by donblanco »

karlicoss wrote: post #26
Laminary wrote: post #22

Earlier quoted context omitted.

I believe my app (launching soon) is close to solving your "what do I want" but currently wrapping up the API. Are there some other communities, links to other blogs or people that are doing what you are doing? (Tracking this much data, etc.)

For data tracking/analysis, you're looking for "quantified self" https://github.com/woop/awesome-quantified-self#awesome-quan... For exporting and owning the data the keyword is "data liberation" Otherwise, the keywords are perhaps "personal knowledge management" and "memex". Have some links here, although they all have different components in scope https://beepb00p.xyz/exobrain/memex.html#cmmnts I usually maintain r…

I just noticed that the awesome-quantified-self on github has been abandoned with no updates for 3-5 years now. Looking for alternatives!
Post reply on HN