Meltwater's engineering team, writing about running large Elasticsearch clusters, query load balancing, nginx API testing, and mob programming.
HN commenters link its Elasticsearch posts on wildcard handling and query routing, ask engineers about query distribution, and note an ad blocker blocking the site.
Shameless plug, but one of the other blog posts in the above series explain how we addressed high and unpredictable memory load in relation to wildcard searches.
https://underthehood.meltwater.com/blog/2022/11/25/how-we-upgraded-an-old-3pb-large-elasticsearch-cluster-without-downtime-part-3-search-performance-and-wildcards/
After those changes were made then we haven't had any major issues with heap spikes. But it depends on a lot of factors, how your data looks, how your queries look and what aggregations that are made so its very hard to give some always applicable advice.
Since I see meltwater folks in this discussion, can you say a bit about the distribution of your queries? In the other blog post about routing [1], you mention that most are a few ms but others are minutes and more.
What’s the rough break down? Are most users performing queries “scoped” to a fairly small subset?
[1] https://underthehood.meltwater.com/blog/2018/09/28/using-machine-learning-to-load-balance-elasticsearch-queries/
Generally when running nginx as an api proxy it can be troublesome to get the routing correctly the first time, therefore testing is required.
This repository showcases how to do nginx api-gateway route tests using RSpec.
This repository also got a blog to explain it in detail:
http://underthehood.meltwater.com/blog/2017/12/12/lightweight-tests-for-your-nginx-api-gateway/
FWIW, I'm getting:
uBlock Origin has prevented the following page from loading:
http://underthehood.meltwater.com/blog/2016/06/01/mob-programming/
Because of the following filter
||meltwater.com^