Viktor Lofgren writes about running the Marginalia independent search engine, plus software, the small web, and productivity.
HN commenters recommend it as a blog worth reading, note the author is a regular on HN, and link it as an example of a search engine for the small web.
> It would only be feasible to build on top of existing libraries, which is the whole point here.
I don't think you've made a single point that substantiates that claim and neither has anyone else, really. The person you're arguing with has built (as per their own estimation) more from scratch when it comes to the actual search engine core than anything else in their own project (https://marginalia.nu/).
Honestly, it seems more like the people arguing a search engine is somehow "off limits" from scratch are doing so because they imagine it's less feasible, probably because they simply don't know or are in general pretty bad at implementing things from scratch, period.
To a large degree there is also a wide difference in actual project scope. Remember that we are comparing this to implementing Space Invaders, one of the simplest games you can make, and one that is doable in an evening even if you write your own renderer.
To talk about "webscale" (a term that is so deliciously dissonant with most of the web world which runs at a scale several 100 times slower and less efficient than things should be) is like suddenly talking about implementing deferred shading for the Space Invaders game. Even if this was something you'd want to explore later, it's just not something you're doing right now because this is something you're using to learn as you go.
I recently wrote an essay about this search engine, and its ranking algorithms.
Initially Marginalia used an interesting variant of PageRank discussed in the original paper, called Personal Pagerank.[1] Currently pages are ranked with BM25.
I think Personalized PageRank is still used for a new feature of Marginalia which is ranking pages based on similarity. I think this is already integrated into the website but there used to only be this testing page: https://explore2.marginalia.nu/
In any case I have a lot of respect for the creator. Marginalia has seen a lot of growth and it's been interesting reading the blogposts.[2]
[1]: https://www.marginalia.nu/log/26-personalized-pagerank/ [2]: https://www.marginalia.nu/log/
There are already many project about search:
- https://www.marginalia.nu/
- https://searchmysite.net/
- https://lucene.apache.org/
- elastic search
- https://presearch.com/
- https://stract.com/
- https://wiby.me/
I think that all project are fun. I would like to see one succeeding at reaching mainstream level of attention.
I have also been gathering links meta data for some time. Maybe I will use them to feed any eventual self hosted search engine, or language model, if I decide to experiment with that.
- domains for seed https://github.com/rumca-js/Internet-Places-Database
- bookmarks seed https://github.com/rumca-js/RSS-Link-Database
- links for year https://github.com/rumca-js/RSS-Link-Database-2024
Having an RSS indicator in the URL bar used to be the first step to subscribing a feed. How do you get it if its not there. When a feature is removed, slowly everyone will forget that it was even a thing once.
Opera (my main browser back then) use to have so much stuff built in (RSS, email, IRC, note taking, mouse gestures). Chrome use to have a resizable extension bar recently. OneNote use to have a more colorful and compact UI. Start menu and taskbar used to be useful before windows 11/10/8.
Reminds of an article posted here about removing less used keys/letters from keyboard. https://www.marginalia.nu/log/48-i-have-no-capslock/
Why should contribution to “the world”, the gargantua of global culture, history, and economy, be the yardstick of worth? I believe almost everyone has something valuable to contribute to their friends, family, neighborhood, or community. We can step away from the chaotic, nebulous global village and focus on the tangible human connections of actual village systems, where “the defining trait of a village is that it’s group of people where the average interaction over time is with people you’ve seen before.”
https://www.marginalia.nu/log/82_killing_community/
https://news.ycombinator.com/item?id=36296882
just a sampling of my rss reader this morning:
marginalia's blog for interesting tech problems building a search engine "specifically" for indie/small/old-web sites. (the search engine itself is a gold mine for exactly what you're looking for). https://www.marginalia.nu/log/
feuilleton for thoughtful posts on recent niche art history - http://www.johncoulthart.com/feuilleton/feed/
the spatial heritage review for advances in 3D models from an educational context https://nebulousflynn.substack.com/