Marginalia (Viktor Lofgren)

It's awesome! I like most suggestions, it feels a bit like https://wiby.me/surprise but generally less weird.

I've often wanted to have a go at making my own search engine, and I think I would penalize any form of advertising (especially big ad networks, referral links) or tracking (Google Analytics, etc.) as these can create (or reveal) perverse incentives. This would likely get rid of most of the "SEO spam" that we see nowadays. Reading the about page[1], this seems like what you are doing here, but I can't really tell as it's light on details.

Q: would this be able to handle foreign-language sites? I don't yet have a blog/personal website, but if I did, I guess it would be mixed-language. Should I submit some of my friends' blogs, even though they might not be entirely (or at all) written in English?

A relatively new sort of search-engine junk, especially visible in non-English results from big search engines is also auto-generated (or probably machine-translated) websites, full of nonsensical content. They might be translated from genuine sites in other languages, I'm not sure. It would seem hard to fend these off, but luckily, fighting perverse incentives such as advertisement revenue probably gets rid of them too.

I also wondered if this was curated list, and if the list was available somewhere, but it seems it's just a good old spider, and I guess that exposing too much info about the metrics might enable some to game the system? Not that marginalia is big enough…

I’ve been puttering away at making a search engine of my own (I should really do a Show HN sometime); let’s see how my experience compares with 18 years ago:

Bandwidth: This is now also cheap; my residential service is 1 Gbit. However, the suggestion to wait until you’ve got indexing working well before optimizing crawling is IMO still spot-on; trying to make a polite, performant crawler that can deal with all the bizzare edge cases (https://memex.marginalia.nu/log/32-bot-apologetics.gmi) on the Web will drag you down. (I bypassed this problem by starting with the Stack Exchange data dumps and Wikipedia crawls, which are a lot more consistent than trying to deal with random websites.)

CPU: Computers are really fast now; I’m using a 2-core computer from 2014 and it does what I need just fine.

Disk: SATA is the new thing now, of course, but the difference these days is HDD vs SSD. SSD is faster: but you can design your architecture so that this mostly doesn’t matter, and even a “slow” HDD will be running at capacity. (The trick is to do linear streaming as much as possible, and avoid seeks at all costs.) Still, it’s probably a good idea to store your production index on an SSD, and it’s useful for intermediate data as well; by happenstance more than design I have a large HDD and a small SSD and they balance each other nicely.

Storing files: 100% agree with this section, for the disk-seek reasons I mention above. Also, pages from the same website often compress very well…

Are there any technical infos about the search engine?
Found some information here https://memex.marginalia.nu/projects/edge/about.gmi

Author said they threw it together on consumer hardware. How big is the index? (TB used or entries) how is it realised?

I'm pretty much interested in this since I myself am crawling some pages for my own "search index".

Oh and thx for making and posting. Added it as a keyword to firefox

Edit: Just realized that my question is a bit shallow. What I'm particular interested in is the storage before the indexing. I'm trying to store the raw html so that I can reindex everything with better algorithms, but I'm hitting many limits. It takes a few minutes getting the size of a site-directory (every site has it's own dir) and I'm at a point where I can't reasonably manage the scrape-versioning over git and I cycled through a few filesystems only to find that the metadata management kind of sucks for most of them. It's rather interesting how we store such files and I'm thinking about storing a few sites in a simple sqlite format for easy access and search. I'm thinking about a a few low overhead solutions like facebooks project haystack (implemented open source in seaweedfs) or something similar... Hopefully this gives some context to the question of storage and sites that are indexed

If you dig the whole website (or geminisite, I dunno) you can find a post to which the very same criticism of the OP applies:

https://memex.marginalia.nu/log/36-localized-programming-languages.gmi

We all draw different lines for what feature is not common yet not rare enough to remove, so the context always matters.

For this subject of character encodings, browsers have already reduced their repertoire of suppported encodings for a good chunk of the last decade: the Encoding Standard [1] is substantially smaller than what it used to be (I know because I did implement all of them back then). Some of them (e.g. UTF-7, not exactly "removed" from the Standard because it was never added AFAIK) caused security issues but others (e.g. HZ) didn't and only got removed by the lack of usage. This kind of decision, either for or against, can be only backed with telemetry-like quantification anyway. No criticism in the issue was based on the concrete evidence; Henri Sivonen is right to say that he "can't address remaining non-Latin problems without seeing concrete examples that need addressing".

[1] https://encoding.spec.whatwg.org/

The main issue is the conflating of usage and value. The persona/user group that consumes the Trending pages are likely users who themselves contribute to open source, and thus are part of a flywheel that drives more code contribution overall. (e.g. how 1% of users post on reddit vs 99% consume) Not all usage is made equally.

This scenario reminds me of a funny story posted on HN a while ago: https://memex.marginalia.nu/log/48-i-have-no-capslock.gmi (TL;DR; if you take this "usage based" feature removal logic to it's conclusion, a lot of important keys would be removed from computers/keyboards).

My dude, them's fighting words. So I built a SSG in 50 lines of mostly idiomatic Java (one line is a bit squished, but 52 just doesn't sound as cool):

https://github.com/vlofgren/50-line-java-ssg

Does deal with partials, bit stuff like an automatic index requires maybe 5-10 additional lines.

Took about an hour (plus 15 minutes to try to figure out how to get into my github account, which I haven't actively used in the last 15 odd years).

My "real" Java SSG is used to generate https://memex.marginalia.nu/. Unfortunately I don't have the source published.