OpenAI Blog

These findings seem to be at odds. The former says that deep linear nets are useful, non-linear and trainable with gradient descent. The latter says that the non-linearity only exists due to quirks in floating point and that evolutionary strategies must be use to find extremely small activations that can exploit the non-linearities in floating point.

Exact solutions to the nonlinear dynamics of learning in deep linear neural networks

https://arxiv.org/abs/1312.6120

"We attempt to bridge the gap between the theory and practice of deep learning by systematically analyzing learning dynamics for the restricted case of deep linear neural networks. Despite the linearity of their input-output map, such networks have nonlinear gradient descent dynamics on weights that change with the addition of each new hidden layer. We show that deep linear networks exhibit nonlinear learning phenomena similar to those seen in simulations of nonlinear networks, including long plateaus followed by rapid transitions to lower error solutions, and faster convergence from greedy unsupervised pretraining initial conditions than from random initial conditions."

Nonlinear Computation in Deep Linear Networks

https://blog.openai.com/nonlinear-computation-in-linear-networks/

"Neural networks consist of stacks of a linear layer followed by a nonlinearity like tanh or rectified linear unit. Without the nonlinearity, consecutive linear layers would be in theory mathematically equivalent to a single linear…

AWS employee on the container services team here.

We are definitely building out a hosted Kubernetes solution with EKS, but its important to realize that Kubernetes is just one part of the container services ecosystem at AWS, living alongside ECS.

Just like we have multiple database solutions (DynamoDB, MySQL/Postgres in RDS, Aurora, Redshift, etc) we also have multiple orchestrators (EKS managed Kubernetes and ECS).

ECS is like DynamoDB: it is the AWS native solution to the problem. Built to be incredibly scalable, and built for AWS patterns and best practices that we know will scale for our customers.

EKS is like RDS: its the AWS managed hosting for the open source software, designed to solve some of your scaling and configuration problems, and it gives you more flexibility, but that flexibility also gives you the power to do things that may not scale as well.

In ECS you may not be able to do everything you can do with Kubernetes (just like DynamoDB limits what you can do compared to the capabilities of SQL), but just like DynamoDB can scale far past SQL because of its limits, ECS is more scalable than Kubernetes because of its limits.

You can run a much larger cluster in ECS than you can in Kubernetes. We would actually still recommend to our largest customers to use ECS instead of EKS if they have any significant scale. The thing to realize is that ECS is a multi tenant control plane: it already keeps track of every single container on every host in every cluster…

Thanks for your response, I had actually heard about Yuri before and was hoping you would respond to this post!

I have a lot of questions for you if you don't mind, maybe you don't want to answer any/all of them that's fine.

1. How do you handle constraining the AI such that it's enjoyable? If you read through the comments here you'll notice just about everyone is saying the hard part is not making a good AI but in making a fun AI. Do you know of any research in this area? most examples I see are all about making the best AI possible. It seems like it's really hard to find a good reward signal to use for "fun". Have you seen this [0,1] paper on learning from human preferences? does Yuri attempt to do something similar? It seems like it would be labor intensive for the play testers.

2. Does Yuri make use of imitation learning at all? I.E learning from lots of human data when it's available to bootstrap the learning process?

3. Do you let the game designers impose any structure on the AI? I.E. "I want three phases for the boss where he progressively gets more aggressive" or use RL just for certain small parts of the behavior? A well defined path finding algorithm with the RL deciding where the agent tries to go? stuff like that?

I have a ton of questions about the technical details side of Yuri too, what frameworks are you using, which RL algorithms etc etc, not sure if you are willing to share those or not.

[0] https://arxiv.org/abs/1706.03741
[1]…

Take a look at the Robot Operating System[1] and their tutorials[2]. Keep in mind though that very slowly ROS is transitioning to ROS2 which fixes some major problems caused by ageing. Most notably the switch from ROS messages to the Data Distribution Service (DDS) which is very popular in robotics.

ROS has a large amount of packages available for popular robots and integrates well with many simulation environments also. This will save you a massive amount of time learning and from being overwhelmed with the massive amount of things that go into even making a simple robot work. Download a bunch of packages and focus on just the part/parts you are learning right now. You will also see lots of great examples of how robotics software is written. Whether it is DDS, ROS messages, or LCM it is generally structured into a collections of nodes that interact with each other.

As far as hardware, even having personal projects with a simple line following robot should be enough to get a foot in the door. Where I worked previously preferred to hire people who were software engineers first, or at least keep the team balanced and I learned on the job. Simulation to real training has had some recent breakthroughs also, see "Learning Dexterity" by OpenAI[3] so it is increasingly possible to do effective work in simulation.

[1]: http://www.ros.org/

[2]: http://wiki.ros.org/ROS/Tutorials

[3]: https://blog.openai.com/learning-dexterity/

One of the things I'm rather worried about here is AI's ability to learn on the most unlikely (to human cognition) features. Take the image recognizer that finds "sheep" by seeing pastures where it thinks sheep might graze http://aiweirdness.com/post/172894792687/when-algorithms-surprise-us , or take any of the adversarial perturbing examples like https://blog.openai.com/robust-adversarial-inputs/ .

You're not allowed to discriminate on race in housing. But is an AI that determines your creditworthiness for mortgages allowed to discriminate on what you eat, where you go to church, what your favorite music is, etc.? Maybe it doesn't have that data, but it will have one level higher - what store credit cards you have and how much you use them and where you opened them.

If you train an AI on a segregated city with a history of actively discriminatory citizens, where the few people of race X who moved into a not-race-X neighborhood got harassed out and sold before they paid off their mortgages, how easily will the AI conclude that people born in certain neighborhoods are more likely to pay off their mortgage if they avoid certain other neighborhoods?

Is that illegal? (My guess is there's no way to prove to a court, to the court's usual standards, that the AI happened to learn the city's racial tensions.) Should it be illegal, if outright racial discrimination is illegal?

Surely, that is a solved problem now that Google has "advance[d] the state of the art in natural language technologies and buil[t] systems that learn to understand and use language in context"? [1]

Or, I guess, one can just buy more NVIDIA GPUs, and simply solve the problem using Torch, following their online guide on "understanding Natural Language with Deep Neural Networks Using Torch" [2].

In general, I hear that many language understanding tasks are finally conclusively conquered, as shown by OpenAI with a combination of supervised and unsupervised learning that has led to an ImageNet-class increase in the state-of-the art results in language understanding datasets [3].

Why, these days, we can even build an "AI System [that] Understands Music Like Humans Do"! (NVIDIA, again; [4])

The inhabitants of Scunthorpe and all the Dicks, Pennys and Dikshits of the world can finally rejoice! Machine learning is on the job!

______________

[1] https://ai.google/research/teams/language/

[2] https://devblogs.nvidia.com/understanding-natural-language-deep-neural-networks-using-torch/

[3] https://blog.openai.com/language-unsupervised/

[4] https://news.developer.nvidia.com/ai-system-understands-music-like-humans-do/