TensorFlow Blog

> breakthrough innovation in programming which will enable programmers to deliver ambitious web/mobile apps at a very high speed.

That's a pretty narrow framing. 10 years from now, a lot of programming won't look like traditional application development at all. For example, a self-driving car or other machine with a similar level of automation is a valuable product that requires a lot of programming but doesn't fit into the category of "web/moble app." Or consider the development of natural language interfaces: right now this is at the level of Alexa skills, but over the next 10 years we will see frameworks that allow programmers to specify not just a few phrases, but mini-languages for entire domains: pilots talking to airplanes, doctors talking to EMRs, etc.

Use cases like this suggest that frameworks similar to PyTorch/Tensorflow will become increasingly important. These frameworks have features (particularly automatic differentiation, very sophisticated yet easy-to-use SGD optimizers like Adam, and built-in GPU support), which hugely increase productivity and lower the barrier to entry compared to the tools of 10 years ago yet they still have tons of rough edges. Its easy to imagine that 10 years from now they'll be even more powerful and easier to use; for example I'm pretty excited about the future possibilities of tools like [AutoGraph][AG].

[AG]: https://medium.com/tensorflow/autograph-converts-python-into-tensorflow-graphs-b2a871f87ec7

Even from the paper, it's hard to tell what this library actually does: Section 5 in https://arxiv.org/pdf/1910.02054.pdf

The paper talks about parameter partitioning and overlapped communication, but doesn't actually give many details on how those things happen.

The library appears to be an implementation of some common algos for solving the 'pebble game,' as explained decently here: https://medium.com/tensorflow/fitting-larger-networks-into-memory-583e3c758ff9

The essential point is that:

(1) model parallelism is hard to do and has historically been done manually to scale wide models across GPUs

(2) inter-GPU I/O is expensive for vanilla data-parallel jobs (that typically use naive mirroring strategies)

(3) researchers have figured out now how to 'compile' a deep model so that layers span GPUs and save on both memory usage and I/O

(4) so scaling wide models is still hard, but now we have better tools for deep models

Existing all-reduce-based data-parallel problems have already been well-studied (see e.g. https://people.eecs.berkeley.edu/~jfc/papers/14/Kylix.pdf ), so it's really nice to see gains through new techniques.

Definitely like seeing this 'compilation' being wrapped up into a library. Just wish they did a better job of communicating key ideas.

Edward http://edwardlib.org/

An amazing Python library built atop Keras and Tensorflow for composing and training probabilistic models in just a few lines. If you're interested in machine learning and/or Bayesian statistics and modeling, you owe it to yourself to check this library out. The webpage has several different tutorials that are all super interesting.

I don't know too much, but I believe this library will be superseded by Tensorflow Probability (Dustin Tran, who writes & maintains Edward is also on the TF Probability team).

https://medium.com/tensorflow/introducing-tensorflow-probability-dca4c304e245

Either one of these are super super valuable if you're looking to play around with and invent new ML algorithms, since you can do in 5 lines what would have taken you 1,000 lines and 4 days of debugging to do without these libraries.

For anyone wondering it is Electron app, that packs some heavy web app assets(300mb + 150mb runtime), but still loads rest of web app from https://client.getposturehealth.co, and then it uses google pretrained posenet tensorflowjs model[0][1]. (loads it from https://storage.googleapis.com/tfjs-models/savedmodel/posenet/mobilenet/float/075/model-stride16.json).

In perfect world one would made it offline, release it at fixed price and be done with it, in our imperfect world it is turned in infinite cash-cow that requires connectivity and bazillion metrics.

[0] https://medium.com/tensorflow/real-time-human-pose-estimation-in-the-browser-with-tensorflow-js-7dd0bc881cd5

[1] https://github.com/tensorflow/tfjs-models/blob/master/posenet/README.md

I took the recent fastai course that had two lessons about S4TF taught by Chris Lattner and walked away excited.

It’s not ready for prime time yet but the roadmap looks great. Building auto-differentiation into the language and having an easy way for anyone to experiment with core architecture changes (without killing performance) will be a huge win for everyone.

I expect lots of prototyping will be done on S4TF (because it will be easy for developers to keep things on the GPU with XLA and MLIR) and then the things that work will be back-ported to other ecosystems.

https://medium.com/tensorflow/mlir-a-new-intermediate-representation-and-compiler-framework-beba999ed18d

I was really looking forward to probabilistic programming but then I saw this project is based on Julia. Not to keen on leaning Julia as I already know Python/C++/Java/Scala/R. Wish it was based on c++/python to at least get started with. Please let me know if there are other interesting probabilistic programming frameworks available or tutorials to get started with.

Edit: https://medium.com/tensorflow/an-introduction-to-probabilistic-programming-now-available-in-tensorflow-probability-6dcc003ca29e There is support in tensorflow for probabilistic programming. How is this any different?