Scalyr's company blog covers its log management service, with engineering posts on fast log search, outages, and complex system failure.
HN links it mainly for the Scalyr service; commenters (some of them Scalyr staff) recommend it for log analytics, with one doubting it is cheaper.
Scalyr | San Mateo | Site Reliability Engineer | Full-Time, Onsite
We are looking for a Site Reliability Engineer who can help keep our uptime promise to our customers by making sure we meet our SLOs and can help our engineering teams ship software to our customers fast and with quality. On this job, you will have an amazing opportunity to drive outcomes that improve reliability, stability and cost efficiency of Scalyr. We are looking to add a SRE with prior extensive operations experience for a SaaS product who can drive deployment re-architecture with focus on self-service and automation. Someone who has driven continuous deployment, has run incident post-mortems, has provided feedback to engineering architecture decisions and has automated repetitive operational tasks would be a great fit. You will join a like minded team of awesome SRE engineers who help run our operations smoothly at scale. We value good written communication skills, data driven decisions and a keen eye for continuous improvements. You’ll help simplify, have a passion for new ideas and know how to execute iteratively towards the final goal. We value candor and collaboration.
SRE and DevOps are two titles loosely used in the industry. A SRE engineer at Scalyr defines and provisions the common set of tools the engineering teams to use and facilitate dev <-> ops collaboration by consulting and driving best practices. SRE at Scalyr is also responsible for uptime and providing feedback to engineering on…
SEEKING FREELANCER - San Francisco Bay Area, REMOTE is acceptable
Windows Python programmer with focus on creating Windows Services
https://www.scalyr.com
We are an early stage startup building a log management, monitoring, and alerting service based on our experience running cloud services at Google. We are seeking a freelancer to adapt the agent program that our customers run on their servers to collect system metrics and server logs. It is written in Python and currently runs on Linux so we are looking to adapt it for Windows.
This is a great opportunity to work with a small, very engineering-focused company at an early stage of development working with very interesting technologies. The project will have a big impact on our customers. Plus, the agent is open sourced so this work benefits the community and will give the developer something they can point at for future employers. Finally, this project could result in more ongoing work in the months / years to come.
The main tasks will include:
* Create a build process that will bundle the Python agent as a Windows executable and run it as a Windows service (using something like py2exe)
* Create an installer
* Develop a mechanism to allow our users to easily update the agent when new releases are made
* Write code to collect relevant system metrics from the Windows server
We are looking for candidates who meet the following requirements:
* Windows expert, focusing on Window Services and best practices
*…
Scalyr | San Francisco/San Mateo | REMOTE | Multiple Roles - UI/Software Engineer/SRE | Full Time | https://www.scalyr.com/careers/
Scalyr is a log management SaaS, a part of monitoring & observability stack. Our vision is "to be the leading source of community engagement on the use and impact of observability", and we have a compelling log management product that we are famous for! [store and search petabytes of data, blazing fast queries (sub 200ms) and high ingestion rates (TB/s)]. Scaly's technology stack is built from the ground up, optimizing it for speed and scalability. We are growing in terms of both data we host and customers!
If you are a frontend engineer (React, GraphQL) or a generalist software engineer who wants to be part of a hyper growth startup and want to work with talented engineers who challenge you, please reach out to us.
We are upgrading our infrastructure and seek the help of a DevOps engineer and thinker to transform our operations to a modern tech stack, learn from it and improve the product for our customers. We run on AWS.
We value transparency and collaboration.
If you are interested, please get in touch with Lakshmi - saasy_log=scalyr mail -s "Hire me" jobs@${saasy_log}.com
Office is in San Mateo close to caltrain station. We can consider remote post COVID too! We offer 100% employer paid medical benefits among other things! We value you and your family!
PS: If you use silver searcher or ripgrep or ack to grep through your code,…
> Retries are tricky because depending on how the job is implemented they can cause more harm than good.
Ah, yes, there's nothing like bringing capacity back online only to have it crushed by all your customers retrying at the same time.
AWS got bit hard by this[3] but there's a blog post[1] about it, which is linked to by the docs for their client software[2].
[1] https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/
[2] https://docs.aws.amazon.com/general/latest/gr/api-retries.html
[3] https://aws.amazon.com/message/5467D2/ ... basically DynamoDB is a fundamental service for AWS and had implemented some new streams features. This all appeared to be working, but they were running closer to capacity than intended, and when a cluster went out this caused a cascading failure.
And see this which linked to that RCA: https://blog.scalyr.com/2015/09/irreversible-failures-lessons-from-the-dynamodb-outage/
Scalyr | DevOps, Backend Engineers | San Mateo, CA | ONSITE
I've built half a dozen startups, including Writely (aka Google Docs), and I can honestly say that Scalyr is my favorite by far. We are the blazing fast log management platform built for the DevOps front line. We built our technology stack from the ground up, optimizing it for speed and scalability. With over 180 customers, real revenue and only 32 employees, the time to join is now!
DevOps Engineer: Join the core engineering team to help us scale our growing infrastructure, ingesting tens of TBs of logs every day.
Backend Engineer: We've built a NoSQL data engine from scratch that searches test at 1TB/second. How'd you like to help us scale and optimize to multiple TB/second?
Sound interesting? apply now: https://www.scalyr.com/jobs
That's good advice. As the saying goes: "trust, but verify".
Regarding your performance issue - most platforms (including dotCloud) enforce ram and cpu separation between nodes, but are vulnerable to IO contention at some level. This is also true for EC2 if you use EBS: your standalone instances will almost certainly, at some point, suffer from degraded and inconsistent performance because another instance is competing for IOPS [1].
You can avoid this with the new "provisioned IOPS" volumes [2], or by skipping EBS altogether for stateless components.
[1] http://blog.scalyr.com/2012/10/16/a-systematic-look-at-ec2-io/
[2] http://aws.amazon.com/about-aws/whats-new/2012/07/31/announcing-provisioned-iops-for-amazon-ebs/