Seirdy writes about search engines with their own indexes, privacy, web best practices, and the security claims of free software.
HN commenters cite his list of search engines with their own indexes as the best they have seen, and note his IndieWeb, Webmention-based site.
Very cool, I subscribed to the newsletter. I’ve experimented with retrieval and ranking across a sample of a million pages from the early days of the Common Crawl (around 2014) and I was surprised by how many of them seemed high quality. The CTO of CC tells me it’s because most of the early URLs were donated by Blekko, which was an old search engine that he used to work for. I don’t know what the quality of recent CC stuff is like, but I think it would be fun to supplement an index with this older data, especially because you’d get a lot of pages that are 404’s now (but you could deliver the extracted text to the user, or link to a temporally nearby snapshot from WayBack).
Another fun thing to consider is making a meta search engine that functions like MetaCrawler used to, where it gets all (or a bunch of) the available results from all the source engines, and then actually fetches and extracts the text from the linked pages, and then matches the query and ranks the pages independent of what the source engines did. If you’d like to do that, I would recommend adapting the source code of 4get.ca (at least for the scrapers), because the guy who writes it is rather talented at coming up with and maintaining workarounds.
If you monetize this, I’d be interested in working for you. I know Python, HTML, CSS, am familiar with JavaScript, and have a lot of experimental (and successful!) experience with ranking web results.
Also, you might be interested in reading this article (from…
> because there are no restrictions on things that just work with root, donation links for open source open geo data contribution platforms[1], roll your own payment scheme without giving anyone a cut, put ads in it if you like...
Respectfully, that was not my point. The point was that having access to the source code fundamentally doesn't mean much. You can read more about why there since I don't want to open this debate again: https://seirdy.one/2022/02/02/floss-security.html
> As does every web browser, but somehow banking on websites seems to very rarely be intercepted?
Banking apps exist, and are required as a modern 2FA. Since 2021, strong 2FA is a requirement in the EU for banking operations. Mail clients also do this. DANE would be the ideal approach on web browsers. This might be up to a more general debate that doesn't belong here though.
> Wouldn't it be nice if you could actually see what this app sends to Google about you?
It's perfectly nice, and mitming is a great tool to ensure Google (and others) doesn't lie about what data they send.
> Previously you'd add a cert to your OS and you'd be good to go.
Now, this argument is rather moot though, since it's still doable. Not sure that the higher technical barrier would matter much, and most users will benefit from having certificate pinning anyway. Google is not doing this to prevent researchers to do their work. Nothing is permanently a blackbox anyhow.
> Hah, the author just spent ~2800 words criticizing…
Try out some queries from your browser history and see if anything beats google. Other engines are more likely to beat google for queries that google has failed at.
Majority of engines were found at below link. Surprisingly many engines use the same indices, for example duckduckgo is mostly just privacy-protected bing. Big thanks to Rohan Kumar.
https://seirdy.one/2021/03/10/search-engines-with-own-indexes.html
A proxy is needed to bypass iframe restrictions for most websites. I used this x-frame-bypass client js code [https://github.com/niutech/x-frame-bypass] which (previously?) used proxy.megatunger.com . That free proxy could be recording your queries, who knows, so use caution.
If (when actually) that proxy goes down I think I'll set up something like CORSflare to have a more reliable and trustworthy alternative. [https://github.com/Darkseal/CORSflare]
I hacked this together over about one workday in total so it's likely to have significant issues. You can comment them here or (preferably) file an issue on the repo: [https://github.com/qpwo/searchall/]
There's also a demo video on the repo if the site goes down but you wanted to see how it looks. Feel free to file feature requests on the repo as well. If you'd rather copy it and make your own version then that's fine too.
I'll try to reply to all comments here.
It's definitely a thing, but IME hosting a small instance, it's really not that bad. My advice for new admins would be to 1. find a small, decently up-to-date block list as a starting point (I can recommend FediNuke [0]), 2. check the #FediBlock hashtag (using any Mastodon UI or the RSS feed automatically generated) once a day or so for any warnings about bot attacks, or any other things you would like to be super proactive about, 3. promptly respond to the (likely very infrequent) reports you get from your users.
I could see why this might not scale as well to instances with thousands of users, or if a user is a frequent target of harassment (where opt-in federation might be the only solution), but for our lab, the existence of all the nazi or whatever instances has been resolved via a simple "Oh, these again. click, click, click There, the whole instance is banned." a few times a year. I have a suspicion that Dunbar's number or lower might be the optimal size for any federated social media instance, since knowing every user on my instance makes it a lot easier to tailor the little effort I have to put in.
[0] https://seirdy.one/posts/2023/05/02/fediverse-blocklists/
If you use a non-declarative adblocker, you're not just trusting the developer, but also all the third-party filter lists you've subscribed to. These filters have powerful capabilities and can even exfiltrate website data [1], and they are updated in real time, so if a bad actor pushed a malicious update (e.g. by gaining access to any EasyList contributor account), you would most likely be affected.
However, it's true some websites (like YouTube) are especially problematic and a declarative adblocker is not enough. What you can do is combine both approaches: use a declarative adblocker (uBlock Origin Lite) as a baseline, and selectively enable non-declarative adblockers (uBlock Origin) for specific websites (see [2] for a detailed overview).
I like this layered approach because it gets you the best of both worlds: the security and performance of a declarative adblocker, and the functionality of a non-declarative adblocker when you need it, without compromising your entire browsing session.
[1] https://portswigger.net/research/ublock-i-exfiltrate-exploiting-ad-blockers-with-css
[2] https://seirdy.one/posts/2022/06/04/layered-content-blocking/
For searches where you want more that just the first result and want a depth of results to go through and maybe even check out more than the first page of results I like to use meta-search engines that grab results from multiple sources. Plus it helps route around censorship since you are getting results from a variety of sources. Searxng is the best known one.
I was quite fond of ixquick but it shut down ages ago. These days I like etools.ch especially since it includes results from search engines like marginalia etc that I tend to forget to search directly but like having meshed into my general searches. Plus you can change which engines it uses in settings and it shows which search engine(s) each link came from which is handy.
On a related note I like to check out Serdys list of search engines with their own indexes once if a while. It gets updated here and there and includes a fair amount of search engines I don't tend to see elsewhere.
https://seirdy.one/2021/03/10/search-engines-with-own-indexes.html