Engine Yard's company blog, covering its Ruby on Rails hosting platform, PostgreSQL, Puppet, and infrastructure practices.
HN commenters mention Engine Yard as an older Rails platform still around, an alternative to Heroku, and note its visible sales and support phone numbers.
Response (awaiting moderation, I think it's the links?):
-------
> haha, that’s funny. i regularly use non-CouchDB databases and I get along great with
> all the people from other databases at conferences.
Oh, I tend to bite people when I find out they use another database . Maybe I should stop that?
> even if i did feel like we were competing, i wouldn’t care. this post really is about
> reliability issues i don’t think your users are fully aware of and i honestly hope that
> you fix.
You must be thrilled to learn that single server durability is coming. I look forward to a followup post extolling MongoDB’s virtues this fall.
> I responded earlier to the complexity of actually keeping something available that
> depends on [multiple servers]. so i won’t cover it again.
Yes, it is a difficult, but not unsolvable, problem. Mongo’s made a bunch of tradeoffs in the awesome vs. easy to program area. For instance, remember last year when CouchDB was saying MongoDB sucked because of its lack of concurrency? That it was too complicated to do concurrency in C++ and that Erlang was the way? Well, now Mongo has concurrency, so on to the next “must have” thing.
>You make it sounds like this is all just a matter of bugs, it’s not, and i find blaming
> it on users who don’t use JIRA or get on IRC a little distasteful.
People discuss everything from bugs to architecture to lunch on our various forums. I was trying to say, possibly badly, that we have a lot of ways for people with…
I missed that. It's actually a much bigger problem.
stdout and stderr(or) is two output paths to explicitly diverge error logs and problems from standard output.
stdout / stdin is reserved for user interaction, informational messages (md5sum), actual output (ls, less, etc.), and the like. OTOH, stderr is explicitly for error messages only, and it's very useful (and necessary).
A real use case from my daily job: I'm an administrator of a large system (approx ~1K servers), and I have substantial amount of cattle, and a lot of pets [0]. All of these servers have cron jobs, and other automated tasks on them. All servers can mail to a local mail server to report us problems, so our cron automatically mails any outputs to us.
All the tools we use, and scripts we write have the following properties:
- If everything is OK, they are silent by default.
- They output to stderr, if anything notable happens.
- Also we copy (think tee) all stderr to their respective logs (both local and on a centralized server).
Now consider:
- The (e-mail, log) noise if all the tools were writing their outputs to stdout.
- The work required to silence all tools. What if they don't have any --quiet switches?
- Furthermore, they wrote everything to stderr. How can I know whether thing has worked as it should?
- How can I find the problem quickly if everything is written to "Error" log?
- Furthermore how can I revise the errors happened quickly? Info & Error on the same file. grep galore!
-…
It amazes me that so much writing on this subject is so reductionist. I read so many of these comments talking about the "return on your degree" and ROIs like they can be measured, and i'm kind of flabbergasted.
I have two observations: A liberal-arts degree gives you perspective, context, and judgement . Contrary to popular belief, these are not innate traits. Given that we just read a very well-received essay about how important judgement is to a coder (http://www.engineyard.com/blog/2011/the-number-one-trait-of-a-great-developer/), it amazes me that we could just sort of forget about other ways these things can be developed.
The other observation is that people talk about being well-rounded like its an inherent good. Perhaps we should spell out why: It increases your luck surface area and exposes you to positive low-probability events.
Being well-rounded means you are better able to place new information in a meaningful framework, because you have a broader and wider set of contexts in which to cross-reference and evaluate your new information. The broader you are, the more likely it is that you can make some useful association with new information. Unfortunately, for the highly-specialized, not only do they not make those associations, but they also don't even know they are available to be made... How do you compute an ROI when so many of the benefits appear to be random?
An example. My father majored in history and english. He ended up writing law and…
Your points are excellent, but number 2 seems a weak: why think of it as a single "machine"? If that's the metric, dual- and quad-processing machines would counter any advantage.
Plus or minus some overclocking, a GTX280 and a Core i7 960 both consume roughly the same number or watts (130 for the processor vs 200 for the card?) and cost roughly the same (at least they did when I bought comparables). So while you can stick (and probably should) stick multiple cards in the same machine, it seems a little silly measure this in terms of multiples of single CPU performance.
NVidia's blog response is interesting, and the some of the comments are actually worth reading:
http://blogs.nvidia.com/ntersect/2010/06/gpus-are-only-up-to-14-times-faster-than-cpus-says-intel.html
To me, the interesting question here is the degree to which the code has been optimized in each case, and the difficulty of doing so. A while ago there was a competition to attempt to generate SHA1 hash collision: http://www.engineyard.com/blog/2009/programming-contest-and-the-winners-are/
The GPU's trounced every other approach. I participated in the contest minimally (running slightly tweaked versions other people's code on both an Nvidia GPU and a Core i7) and I was amazed at how much more efficient the GPU solution was. I don't recall the exact ratios, but 'orders of magnitude' was definitely not an exaggeration.
But that's sort of a contrived example. So what I'd love to see here is a framework…
Engine Yard | San Francisco, CA; Portland, OR, Dublin, Ireland; REMOTE | Platform Engineer
As a platform engineer, you will spend most of your time working on orchestration and automation which manages customer environments across AWS and Azure infrastructures. Our platform is largely written in Ruby so a strong grasp of the language is preferred.
Every day you will deliver features on a platform that powers tens of thousands of servers. Ship software that runs the production environments for many startups and enterprises. Have a direct impact on the lives of other developers so they can focus on getting work done. Help us make cloud deployments, better, faster and stronger.
The engineering team at Engine Yard is spread across 8 timezones (Dublin, Ireland to San Francisco, California), so experience working in a distributed team is very helpful. You will talk to colleagues via Zoom, post funny gifs to our slack chat rooms, reserve and deploy your topic branches to staging environments via our good pal `eybot`. When the build is green the trap is clean. We deploy multiple times a day.
You'll get:
Competitive salary
Full medical, dental and vision benefits
25 paid days off per year (holidays and PTO)
401(k)
We take care of your phone bill
Paid travel to conferences
Must haves:
A passion for writing great software to solve real problems for real people
Demonstrated experience working with large-scale production systems
Experience working in a…
You hit the nail on the head. If you don't need strong consistency guarantees any sort of message queue will do. That is a solved problem.
There are two key quotes from the article:
1. "... if our queue begins dropping messages, we run the risk of silently corrupting data"
The two statements that they are executing "app.update" (modifying the database) and "enqueue" (sending to message queue") are not atomic. They can make it atomic using something like 2 phase commit but implementing that will be more trouble than it's worth. Additionally, there will be performance (latency) implications since it is a synchronous operation.
2. "Notice how both of our writes are inside of a local database transaction"
Now they have atomicity very easily!
I agree with the grandparent that using SQL as a message queue is generally a bad idea. However the pitfalls are well known and can be engineered around. See this article:
http://www.engineyard.com/blog/2011/5-subtle-ways-youre-using-mysql-as-a-queue-and-why-itll-bite-you/
I am actually working on something very similar to this at my day job. The "buffer processor" that the article is describing, we implemented using a scala process. Perhaps what's most interesting about the entire system as mentioned at the end of the article is maintaining the large amount of state for durability (we use hbase) and deliverability (we wrote a custom scala server).