Can anyone point me to the best place to get castor going? I cant install it on my 22.04 install unmet dependencies...
The “small web” is bigger than you might think
61–70 of 263 posts
Re: The “small web” is bigger than you might think
#62It's easy to hand-curate a list of 5,000 "small web" URLs. The problem is scaling. For example, Kagi has a hand-curated "small web" filter, but I never use it because far more interesting and relevant "small web" websites are outside the filter than in it. The same is true for most other lists curated by individual folks. They're neat, but also sort of useless because they are too small: 95% of the things you're look…
My approach operates under the assumption that good, non-commercial webpages will be similar to other good webpages. Slop, SEO spam, and affiliate content will resemble other such content.
So a similarity-based graph/network of webpages should cluster good with good, bad with bad. That is what I've seen so far, anyway.
With that, you just need to enter the graph in the right place, something that is fairly trivial.
Re: The “small web” is bigger than you might think
#63Kagi Small Web has about 32K sites and I'd like to think that we have captured most of (english speaking) personal blogs out there (we are adding about 10 per day and a significant effort went into discovering/fidning them). It is kind of sad that the entire size of this small web is only 30k sites these days.
What methods are you using to find them? I notice my own doesn't appear, although it does show up well under some (very niche) Google search terms. I suspect there's the potential for an order of magnitude more sites than have been found.
Re: The “small web” is bigger than you might think
#64Kagi Small Web has about 32K sites and I'd like to think that we have captured most of (english speaking) personal blogs out there (we are adding about 10 per day and a significant effort went into discovering/fidning them). It is kind of sad that the entire size of this small web is only 30k sites these days.
Suspect there's a long tail/iceberg you still haven't captured (source: you haven't found me yet and I'm not hiding, I'm just not chasing SEO).
Re: The “small web” is bigger than you might think
#65I don't expect many people to agree but I think that the "small web" should reject encryption, which is the opposite direction that Gemini is taking. I don't deny the importance of encryption, it is really what shaped the modern web, allowing for secure payment, private transfer of personal information, etc... See where I am getting at? Removing encryption means that you can't reasonably do financial transactions, ac…
Anyone between you and the server can change the content of the page on unencrypted connections. I would love to live in a world where encryption is unnecessary, but unfortunately that world does not exist right now.
You could do signatures/MAC without encryption to guarantee that the message was not modified
Re: The “small web” is bigger than you might think
#66Kagi Small Web has about 32K sites and I'd like to think that we have captured most of (english speaking) personal blogs out there (we are adding about 10 per day and a significant effort went into discovering/fidning them). It is kind of sad that the entire size of this small web is only 30k sites these days.
Does this use frames or iframe? https://kagi.com/smallweb I would expect a raw link in the top bar to the page shown, to be able to bookmark it etc.
There is a '↗'-shaped icon in the navigation bar at the top. If you click on that it takes you to the original post in a new tab. On Firefox and Safari, you can also right click that icon and add the original post to the bookmarks.
Re: The “small web” is bigger than you might think
#67Kagi Small Web has about 32K sites and I'd like to think that we have captured most of (english speaking) personal blogs out there (we are adding about 10 per day and a significant effort went into discovering/fidning them). It is kind of sad that the entire size of this small web is only 30k sites these days.
Suspect there's a long tail/iceberg you still haven't captured (source: you haven't found me yet and I'm not hiding, I'm just not chasing SEO).
I am happy to hear this.
Re: The “small web” is bigger than you might think
#68A little shell function I have in my ~/.zshrc: pages() { for _ in {1..5}; do curl -sSw '%header{location}\n' https://indieblog.page/random | sed 's/.utm.*//'; done } Here is an example output: $ pages https://alanpearce.eu/post/scriptura/ https://jmablog.com/post/numberones/ https://www.closingtags.com/blog/home-networking https://www.unsungnovelty.org/gallery/layers/ https://thoughts.uncountable.uk/now/ On macOS, we…
Caveat that Kagi gates that repo such that it doesn't allow self-submissions so you're only going to see a chunk of websites that other people have submitted that also know about the Kagi repo.
But per the instructions, it seems like that if one wants to add your own website, then one needs to add 2 other small websites (that are not on the list already)...so technically it does open things up to those who are not aware of the repo...assuming their site is pulled in when someone wants to add their own website. Obviously this scale is slow...but i think that's kinda the point, eh? Nevertheless, for every 1 person wanting to add their stuff, 2 others would technically get added i guess.
See: https://github.com/kagisearch/smallweb?tab=readme-ov-file#%E...
Re: The “small web” is bigger than you might think
#69Earlier quoted context omitted.
Right, but that basically works as a retro alternative to scrolling through social media. If you're looking for something specific, it's simultaneously true that there's a small web page that answers your question and that it's not on any "small web" list because the owner of the webpage never submitted it there, or didn't meet the criteria for inclusion. For example, I have several non-commercial, personal websites…
Feel free to suggest changes to criteria for inclusion. It is mostly the way it is now as the entire project is maintained by one person - me :)
Looking at the criteria again, I can think of at least three things that arbitrarily exclude large swathes of the small web:
1) The requirement that it needs to be a blog. There's plenty of small-web sites of people who obsess over really wonderful and wacky stuff (e.g., https://www.fleacircus.co.uk/History.htm) but don't qualify here.
2) The requirement that it needs to be updated regularly. Same as above - I get that infrequently updated websites don't generate a "daily morning" feed, but admitting them wouldn't harm in any way.
3) Blanket ban on Substack-like platforms while allowing Blogspot, Wordpress.com, YouTube, etc. Bloggers follow trends, so you're effectively excluding a significant proportion of personal blogs created in the last six years, including the stuff that isn't monetized or behind interstitials. The outcomes are pretty weird: for example, noahpinionblog.blogspot.com is on your list, but noahpinion.blog is apparently no longer small web.
Re: The “small web” is bigger than you might think
#70I don't expect many people to agree but I think that the "small web" should reject encryption, which is the opposite direction that Gemini is taking. I don't deny the importance of encryption, it is really what shaped the modern web, allowing for secure payment, private transfer of personal information, etc... See where I am getting at? Removing encryption means that you can't reasonably do financial transactions, ac…
> I think that the "small web" should reject encryption, which is the opposite direction that Gemini is taking.
I think it should allow but not require encryption.
> Removing encryption means that you can't reasonably do financial transactions, accounts and access restriction, exchange of private information, etc... You only share what you want to share publicly, with no restrictions. It seriously limits commercial potential which is the point.
Note that the article linked to says "the Gemini protocol is so limited that it’s almost incapable of commercial exploitation", even though Gemini does use TLS. (Also, accounts and access restriction can sometimes be used with noncommercial stuff as well; they are not only commercial.)
> It also helps technically. If you want to make a tiny web server, like on a microcontroller, encryption is the hardest part.
This is one of the reasons I think it should not be required. (Neither the client side nor server side should require it. Both should allow it if they can, but if one or both sides cannot (or does not want to) implement encryption for whatever reason, then it should not be required.)
> Anyone can man-in-the-middle and change the web page, TLS prevents that. But what I think is an even better solution is to do it at the content level: sign the content, like a GPG signature
Using TLS only prevents spies (except Cloudflare) from seeing or altering the data, and does not prevent the server operator from doing so (or from reassigned domain names, if you are using the standard certificate authorities for WWW; especially if you are using cookies for authentication rather than client certificates which would avoid that issue (but the other issues would not entirely be avoided)).
Cryptographic signatures of the files is helpful, especially for static files, and would help even if the files are mirrored, so it does have benefits. However, these are different benefits than those of using TLS.
In other cases, if you already know what the file is and it is not changing, then using a cryptographic hash will help, and a signature might not be needed (although you might have that too); the hash can also be used to identify the file so that you do not necessarily need to access it from one specific server if it is also available elsewhere.
> Well, if want to protect yourself, TLS won't save you, you will be given away by your IP address, they may not see exactly what you are looking at, but the simple fact you are connecting to a server containing sensitive data may be evidence enough.
There is also SNI. Depending on the specific server implementation, using false SNI might or might not work, but even if it does, the server might not provide a certificate with correct data in that case (my document of Scorpion protocol mentions this possibility, and suggestions of what to do about it).