Ahrefs Engineering Blog

Ahrefs | OCaml developer / DevOps | Singapore + San Francisco, CA | REMOTE ok, VISA ok (SG only)

Ahrefs runs an internet-scale bot that crawls the whole Web 24/7, storing huge volumes of information to be indexed and structured in a timely fashion. Backend system is powered by a custom petabyte-scale distributed key-value storage to accommodate all that data coming in at high speed. The storage system is implemented in OCaml with thin performance-critical low-level part implemented in C++ and Rust. On top of that Ahrefs is building various analytical services for end-users.

We are a small team and strongly believe in better technology leading to better solutions for real-world problems. We worship functional languages and static typing, extensively employ code generation and meta-programming, value code clarity and predictability, and are constantly seeking to automate repetitive tasks and eliminate boilerplate, guided by DRY and following KISS. If there is any new technology that will make our life easier - no doubt, we'll give it a try. We rely heavily on opensource code (as the only viable way to build maintainable system) and contribute back [1]. Occasionally we track down CPU bugs [2].

We are looking for a backend developer [3] with a deep understanding of networks, distributed systems, OS fundamentals and taste for simple and efficient architectural designs. Our backend is implemented in OCaml (and some C++), as such proficiency in OCaml is very much appreciated,…

I have an European alternative, my own iron in my own home, it's European since I'm an EU citizen living in the EU...

The alternative to the modern computing is the classic iron ownership instead of living on the shoulder of giants, otherwise there is no difference between a US company and an EU one selling the same services. They are both third party whose job is making money for their shareholders.

"But.... HW cost money!", yes, because someone else computers are notoriously free... https://tech.ahrefs.com/how-ahrefs-gets-a-billion-dollar-worth-infrastructure-with-a-90-discount-5edd473b2399 and of course owned hw is not a asset with an eventual resale value and a vast series of possibilities if a startup idea fail...

Oh, yes, bandwidth might be tricky but co-location until a company is large enough to pay a personal proper link it's still an option, on their own hw.

Dear devs startuppers where the recliner is the most valuable owned asset you have, it's about time to recalculate your economy.......

While I agree that for user-facing APIs, serverless can both help handling load spikes and keep costs low, how would serverless help with spidering and indexing?

Spidering and indexing are processes that would most likely run continuously for any search engine like Google or Kagi. There is always data to update and new web pages being created.

Thus, they would benefit from dedicated servers from the get go for costs. Ahrefs posted an interesting article on this topic: https://tech.ahrefs.com/how-ahrefs-saved-us-400m-in-3-years-by-not-going-to-the-cloud-8939dd930af8

Then again, you could use serverless on your own dedicated servers to benefit from the advantages of both.

Debian announcement

https://lists.debian.org/debian-devel/2017/06/msg00308.html

Ahrefs writeup

https://tech.ahrefs.com/skylake-bug-a-detective-story-ab1ad2beddcd

The Intel spec update still labels SKL150 as No Fix but there is a microcode update available. Dunno exactly what to make of that distinction.

https://www.intel.com/content/www/us/en/processors/core/desktop-6th-gen-core-family-spec-update.html

Can an x86 program detect whether this update has been applied? Can a Linux process set a DONT_HYPERTHREAD_ME_BRO bit?

>> somewhat

It's a factor of 5 to 20 [1] in many cases.

Regular hosting is unbelievably cheap, especially in high traffic environments.

[1] https://tech.ahrefs.com/how-ahrefs-saved-us-400m-in-3-years-by-not-going-to-the-cloud-8939dd930af8