Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building personal search infrastructure for your knowledge and code

beepb00p.xyz

Post reply on HN
Print view View original

91–100 of 162 posts

Re: Building personal search infrastructure for your knowledge and code

#91

Post by Viliam1234 »

insomniacity wrote: post #63
fctorial wrote: post #49

Earlier quoted context omitted.

Why don't you host it on an ec2 instance? They won't be blocking amazon ip. Where do you work?

There's definitely no external access without going through the web proxy. And a new uncategorized site would be blocked by the web proxy - and it wouldn't pass review. I work in a highly regulated industry... Any workaround would be grounds for termination. So there's no point to my comment really - just curious if anyone else is in the same boat.

> just curious if anyone else is in the same boat.

Yes. Spent a few years building a knowledge base in an offline application. Now I have a new job that doesn't allow me to install software. So all my notes stay at home.

Maybe one day I'll make an export to PDF and use that at work. But I will miss the editing functionality.

On one hand I understand the need to prevent data leaks, malware, etc. On the other hand -- am I supposed to memorize literally everything? Or search everything on Stack Exchange over and over again, hoping that the explanation is there, is correct, and is up to date? Figuring out stuff and making simple notes is my strength. Memory is my weakness. This sucks.

I wonder if it would have been better to install some wiki software on my private website and build my knowledge base online. Reading unknown webs is not forbidden in my current job (the web filter apparently uses blacklists). But maybe in my next job it will be, who knows.

Re: Building personal search infrastructure for your knowledge and code

#92

Post by otakucode »

sqs wrote: post #28

Sourcegraph CEO here. I see the doc mentions Sourcegraph for code search (cool!). Something like ripgrep is indeed better for your case, a single person who just needs to search code in local directories on their own machine. I made a PR for our docs at https://github.com/sourcegraph/sourcegraph/pull/8075 that should clarify this. Sourcegraph is a web-based code search tool that automatically syncs and indexes many r…

I'm not familiar with your product, so this question might be either overly simple or entire outside your wheelhouse, but does Sourcegraph have any integration with IDEs? Like if my organization had a module in a repository that did image compression (random example), could I hit a hotkey and search for it, then have the plugin pull the module out of a different project and insert it into what I'm working on?

Re: Building personal search infrastructure for your knowledge and code

#93

Post by kortex »

silicon2401 wrote: post #22
napoleond wrote: post #11

I've been thinking a lot about how I manage my own data lately (notes, photos, code, reference material, etc) and have concluded that the primary feature I'm looking for is longevity. I'm saddened by the amount of data I've lost over the years, either because of hard disk failures or third-party services going out of business/making it difficult to extract things/getting too expensive. In light of this, I'm biasing t…

what's the optimal setup for long-term, large-scale (personal) data storage? I want to build one big Backup. Some initial research has pointed me to something like Bacula to manage the data backup process from a machine. With the 3-2-1 rule, I know I also need my Backup itself to have at least 3 copies, in at least 2 different forms (cloud/hard disk), at least one of which is off-site from me. As an individual, do yo…

Oooh. I've been wrestling with this problem for a while now.

Basically I'm working on a tiered system. Files/dirs are categorized by size (25GB) , and by sensitivity (public, confidential, secure. And importance is usually proportional to security). I have fortunately found that security is usually inverse to size. Github/lab anything which makes sense. Confidential small stuff (sans keys) is just stored in gmail/drive. Big, boring stuff (music, ebooks) is just kept on external hard drives.

Secure, ultra-important stuff, I don't really have a system for.

The system I'm leaning towards is just encrypt archives and store the key/password securely, and store it like you would any boring data, with a local NAS and a cloud backup service of some sort, or just stored on drives offsite.

Re: Building personal search infrastructure for your knowledge and code

#94

Post by K0SM0S »

ketzo wrote: post #23

It's been mentioned a few times in these comments, but I want to add a +1 for Roam[1]. Note-taking/personal knowledge tool that's very, very different from anything I've seen before -- closest thing I can compare it to is Wikipedia. It's still in beta with some rough edges, but VERY worth checking out. [1] roamresearch.com

"trackcmp.net" keeps breaking navigation for me on roamresearch.com, and that tracker is not even https (classic unsafe warnings from Chrome). Weird. Unfortunately, that makes the whole thing look shady, and I can't even get to the create account screen after signing up. :/

Maybe they'd do better to ease up on the tracking, especially for a "give us all your documentation" service.

Re: Building personal search infrastructure for your knowledge and code

#95

Post by ramses0 »

user00012-ab wrote: post #30

My problem with a lot of services listed below, is they all eventually go away, and all your data is off somewhere else. Unless you store your data locally in a human readable format (markdown) you are just putting all your data into a system that WILL go away at some point in the future. Google has already had 2-3 services to manage your data that they have closed down. Maybe they are the ones that taught me not to…

https://en.wikipedia.org/wiki/Blosxom ... I got started in 2004 with that setup: http://www.robertames.com/blog.cgi/entries/bloxsom-started.h...

...a bit contrarian compared to the WordPress and BlogSpot frenzy at the time, but I've been happy with it.

[rames@...:~/blog/entries]$ find . -type f | wc -l 331 [rames@...:~/blog/entries]$ find . -type f | xargs -n1 cat | wc -c 574481

It's been very stable over ~15 years, but I think it might be time to adopt SQLite, at least as a caching layer. ;-)

Re: Building personal search infrastructure for your knowledge and code

#96

Post by otakucode »

stillwater56 wrote: post #78

Does anyone else find that the simple act of writing notes helps them remember and process better? I spent forever trying to find an ideal note-taking solution, but now I just write things in a single notebook. I rarely review my notes, but I find that simply writing thoughts down consistently has improved my memory and understanding of new concepts.

This certainly applies for me personally. My theory is that it ties in with the sort of 'geographic' memory where when you think of something, you might not be able to remember exactly what it is, but you can remember pretty precisely that it's in the middle of a certain notebook, on a heavily marked-up page, in the bottom left corner. By tying things to a location which you can remember, placing it in a bit of a context, its easier to hold on to. I also find, and for this I have no explanation at all, that I can remember sequences of numbers and code very well, better than anything else. I couldn't tell you the date I started or left my job 2 employers ago, but I could rattle off my 7-digit numeric security code for the door no problem. The brain is weird.

Re: Building personal search infrastructure for your knowledge and code

#97

Post by philsnow »

napoleond wrote: post #11

I've been thinking a lot about how I manage my own data lately (notes, photos, code, reference material, etc) and have concluded that the primary feature I'm looking for is longevity. I'm saddened by the amount of data I've lost over the years, either because of hard disk failures or third-party services going out of business/making it difficult to extract things/getting too expensive. In light of this, I'm biasing t…

You mention S3 and Athena, but also that you're building for longevity. Are you planning for the future obsolescence of AWS, or going to cross that bridge when you get to it?

Re: Building personal search infrastructure for your knowledge and code

#98

Post by sqs »

otakucode wrote: post #92
sqs wrote: post #28

Sourcegraph CEO here. I see the doc mentions Sourcegraph for code search (cool!). Something like ripgrep is indeed better for your case, a single person who just needs to search code in local directories on their own machine. I made a PR for our docs at https://github.com/sourcegraph/sourcegraph/pull/8075 that should clarify this. Sourcegraph is a web-based code search tool that automatically syncs and indexes many r…

I'm not familiar with your product, so this question might be either overly simple or entire outside your wheelhouse, but does Sourcegraph have any integration with IDEs? Like if my organization had a module in a repository that did image compression (random example), could I hit a hotkey and search for it, then have the plugin pull the module out of a different project and insert it into what I'm working on?

Sourcegraph has editor plugins (https://docs.sourcegraph.com/integration/editor) that give you editor hotkeys for (1) multi-repository search and (2) go-to-file on Sourcegraph (in your web browser, so you can read the code without ruining your editor state or share the URL of your current file with a teammate).

That advanced use case you mentioned isn't supported, but it sounds very cool. It's in the realm of things we'd like to offer someday. If anyone's interested in hacking on that (and making a PR to https://github.com/sourcegraph/sourcegraph), I'd be happy to screenshare with them and give them some pointers.

Re: Building personal search infrastructure for your knowledge and code

#99

Post by semitext »

stillwater56 wrote: post #78

Does anyone else find that the simple act of writing notes helps them remember and process better? I spent forever trying to find an ideal note-taking solution, but now I just write things in a single notebook. I rarely review my notes, but I find that simply writing thoughts down consistently has improved my memory and understanding of new concepts.

I think there is a lot of truth to this, however I also (when I remember to) like to review notes I wrote a month or so later, and check to see if they make any sense to me. If they don't it means I didn't understand the concept as well as I thought I did and it is worth going back over the source material.

Re: Building personal search infrastructure for your knowledge and code

#100

Post by mlthoughts2018 »

sqs wrote: post #28

Sourcegraph CEO here. I see the doc mentions Sourcegraph for code search (cool!). Something like ripgrep is indeed better for your case, a single person who just needs to search code in local directories on their own machine. I made a PR for our docs at https://github.com/sourcegraph/sourcegraph/pull/8075 that should clarify this. Sourcegraph is a web-based code search tool that automatically syncs and indexes many r…

Wow, the prices seem extremely high to me for a search engine across code repos.

$30/person is almost double what Stack Overflow charges, and that product can act as a frontend to search not just code but any type of documents, with voting, tagging, analytics on what confuses people the most and more.

It would be hard for me to justify even $10/person for something like Sourcegraph in my company (a Fortune 500 ecommerce brand), for the highest enterprise tier of functionality.

$30/person per month for the lowest tier? Boy, I wish I knew of companies willing to pay that. None in my experience ever have been.

Post reply on HN