Rich Skrenta, founder of the Blekko search engine, writes about search, competing with Google, and startups.
Commenters call his posts on competing with Google the best on the topic, though one notes an old MD5 password tutorial is bad advice.
> Do any site operators actually block non-Google search engine crawlers because being listed DDG/Bing/etc isn't worth the extra cost of serving the crawler?
Many website operators do actually block crawlers from non Google search engines and it's because the cost of being crawled isn't worth it to them. Here's a good quote from one such webmaster:
As a webmaster I get a bit tired of constantly having to deal with the startup crawler du jour.
From law firms looking for DMCA violations to verticals search engines, to image aggregators, to company intelligence resellers… It feels to me that everybody and their brother has gotten into spidering sites.
With 10,000s of pages that have content that is only relevant to a targeted audience who is perfectly able to find us on the majors, I do not hesitate to block (and possibly ban) when I see an aggressive crawler that does not provide me or my customers with direct benefits.
Taken from http://www.skrenta.com/2008/04/cuill_is_banned_on_10000_sites.html
> Perhaps other search engines should spoof GoogleBot. Browsers have being doing that since forever spoofing Netscape (Mozilla), Safari, etc. for the same reason.
People have tried this and it doesn't work. Google provides ways to check to make sure traffic is coming from Google IP addresses and practitioners and academics study how to spot fake…
It's not an opinion I hold strongly, it's just something I've noticed. I've seen more than one talk where someone apologized for implementing something with 'tie'.
Here's one blog post about it (still opinion):
http://www.skrenta.com/2007/05/tie_considered_harmful.html
And, this article from the Modern Perl site itself has mixed opinions on the subject, but comes down on the side of not using it, in the general case, but knowing how and when to use it can be really valuable (Tie is the last subject covered on the page):
http://modernperlbooks.com/books/modern_perl_2014/11-what-to-avoid-in-perl.html
I'm not trying to say there's no place for it. Tie in Perl has been used for really cool stuff.
Kind of interesting. His criticisms of Google are not unreasonable [1], but it's just bluster until he gives a clue about his approach.
1. 'PageRank wrecked the web' http://www.skrenta.com/2007/12/pagerank_wrecked_the_web_3.html (fun in comments)
'Google sees own shadow, jumps overboard' http://www.skrenta.com/2007/12/google_sees_own_shadow_jumps_o.html
We hired a naming company, and they couldn't come up with a better name -- most of their suggestions were too boring.
http://www.skrenta.com/2008/01/about_the_name_blekko.html
If we succeed, the name will be considered great. If we fail, people will blame the name. Does the name really affect anything? I bet good results are more important.
Rich Skrenta has written some interesting stuff on his blog (http://www.skrenta.com - nothing recent though) but I have to admit Cuil made me nervous about a "stealth search engine". I haven no idea how blekko compares, but I like the DuckDuckGo approach much better. Best of luck to them both, they've got their work cut out for them.
It looks like he just misread this article, which is kind of a disaster itself -- it recommends storing MD5 hashes of passwords, maybe with a salt if you want to be extra secure.
http://www.skrenta.com/2007/08/md5_tutorial.html
Edit: Oh dear, there's a followup post:
http://www.skrenta.com/2007/08/crypto_vs_the_working_coder.html