Scale AI Blog

Thanks for the pointer. We do agree that not all agentic systems should be multi-agent.

Having said that, from our experience we see that for complex workflows e.g. customer support for enterprises, both quality and maintainability stands to gain when the system is decomposed into smaller scoped agents. We see a parallel of this in humans as well. For instance, when we call into customer support we get routed to different human agents based on our query.

OpenAI says something similar in their recent guide on building agents [0]: "For many complex workflows, splitting up prompts and tools across multiple agents allows for improved performance and scalability. When your agents fail to follow complicated instructions or consistently select incorrect tools, you may need to further divide your system and introduce more distinct agents."

A relevant benchmark here might be the Instruction Following benchmark: https://scale.com/leaderboard/multichallenge. Even the best reasoning models top out at ~60% accuracy on this.

The options to improve accuracy then, are (a) either fine-tune a model on this task specific dataset, (b) or decompose the problem into smaller sub-problems (divide-and-conquer) - this is more practical and maintainable.

[0] https://cdn.openai.com/business-guides-and-resources/a-practical-guide-to-building-agents.pdf

Scale AI | Product Security Engineer | TypeScript, Node, Python, AWS | Full-time | Hybrid in San Francisco, CA or NYC or Remote

This is for a team that I'm working closely with and helping grow. We're hiring for somebody with a hybrid SWE and Security background to help scale up the team.

It's currently one engineer who's got a ton on his plate, so the ideal person for this role is somebody who's interested in learning, digging in deep to fix issues (especially shipping PRs), and helping shape the future roadmap of the team. It is not an analyst role.

The role is primarily targeted at mid-career folks with a few years of experience (2+ years minimum). Those that are Senior/Staff level folks, especially those with a strong SWE background that are curious about security, are encouraged to apply if this role resonates strongly. (I personally transitioned from SWE -> ProdSec, years ago, and several other folks on the broader Security team have non-Security backgrounds too.)

Here is the role: https://scale.com/careers/4602047005

Feel free to apply there, email me with you resume (on my profile), or add me on LinkedIn[0]. I'll try my best to answer questions and reply to everybody, but sometimes there is so much inbound that it isn't possible. Thanks!

0: https://www.linkedin.com/in/freewortley/

Scale | Backend/Full Stack and Frontend (Full-Time) | SF

We label data for your favorite computer vision teams. Our mission is to accelerate the development of AI applications - we believe building a high quality labelled dataset is the biggest bottleneck to deploying supervised deep learning systems, so that's what we're tackling first.

We’ve had phenomenal past few years, raised an $1B series C, and are looking to grow our team of 130 (35 engineers).

We’re looking for engineers to work on projects ranging from making labelling more efficient via front-end work/ML work to launching completely new product lines.

Frontend: https://jobs.lever.co/scaleai/0381f41e-7779-463e-a6c7-926fa21e4f90?lever-origin=applied&lever-source%5B%5D=HackerNews

Backend: https://jobs.lever.co/scaleai/c1443865-f64e-4467-bfdc-8980533a2ad3?lever-origin=applied&lever-source%5B%5D=HackerNews

Fullstack: https://jobs.lever.co/scaleai/868065d0-7139-42cf-999f-3f50d476d38e?lever-origin=applied&lever-source%5B%5D=HackerNews

Learn more about us at: https://scale.com/blog

Scale AI | Strategic Product Manager | REMOTE or SF | Full Time | https://scale.com/careers

Scale (YC 16) is one of the fastest growing AI startups in the world. Our mission is to accelerate the development of AI and our first product involves providing high quality labeled data to help our customers train ML models.

We are looking for a PM to lead strategic projects. This person will be in charge of delivering results for our most important customers or developing new products and processes in order to move key company metrics. The ideal candidate would have an entrepreneurial mindset (past founders would be ideal) and strong technical skills.

Full job description: https://scale.com/careers/935d4882-8002-4816-9d87-c4b8b045d640

Feel free to email me directly at ruben[at]scale.com

In 2021, Alex Wang of Scale (the author of the linked article) uncritically quoted the following from Eric Schmidt:

"Eric states that the global growth of AI technology is a national security threat for the United States. In March, we believed we were ahead of China’s technological development in the AI sector, but they quickly showed us that we were wrong. By June, they’d demonstrated a model capable of producing human-like text that was comparable to OpenAI’s GPT-3 model in quality."

https://scale.com/blog/Eric-Schmidt-Geopolitics-TransformX-Scale-AI

As such, it's important to note that the author views the entire Chinese AI industry as a threat to the United States, and may believe that the growth of AI anywhere in the rest of the world is a threat to the United States.

Let’s get a reality check. Current agent performances are no where near that kind of threat, and it’s uncertain when we get there.

I think people continue bullshitting in this domain because they’re worried they have no moat, so they have to discourage via sophistry or bold claims.

Don’t take my word for it, here are the stats https://scale.com/leaderboard

For reference, 25% means getting 1 in every 4 question right.

Secondly, if we ever do get to that point, people have lots of social mechanisms they’ve developed over millions of years of evolution which will kick into place. I am not sure anyone dreaming of this “utopia” is then going to be around to see it.