The Blaze and Dask developers, in the PyData community, write about Python tools for querying and computing on large data.
Commenters mention Blaze as a PyData project that lets users query stores like Spark and Hive with Pandas-like syntax.
> how does this compare to theano? it seems like some of the ideas are similar?
It's quite similar, we just take some of the ideas farther and try to generalize the data storage to include storage backends that data scientists use more frequently ( i.e. SQL, CSV, S3, etc ). We're very friendly with the Theano developers and hope to bridge the projects with a compatibility layer at some point.
> (it gives the illusion of a python solution, but in practice you're off in some other world that only looks like python).
I would argue that's what make Python a great numeric language, and NumPy so succesfull. You get this high level language where you can express domain knowledge but also this 1:1 mapping between fast code execution at the C level. Blaze is the continuation of that vision
> i guess the AST is fed to something that does the work. and that something will have an API and be replaceable.
Precisely, we build up a intermediate form called ATerm out of the construction expression objects, do type inference, graph rewriting, and then pattern match our layout, metadata, and type information against a number of backends to find the most optimal one to perform execution. Or if needed we build a custom kernel with Numba informed by all this type and data layout information we've inferred from the graph.
We don't aim to solve all the subproblems in this area ( expression optimization passes, distributed scheduling ) but I think we have a robust enough system that others…
Having a bit of a look at the docs^1, I don't really see it as "yet another arrays package". The API exposes generalized table/array operations, and a subset of those features can be implemented using NumPy's ndarrays. If the interface is clean, then I don't think adoption will be an issue: there simply isn't anything else that does this.
You are absolutely right that both of these projects are extremely ambitious, but, given the $3 mil grant, I can't help but be optimistic.
1 http://blaze.pydata.org/docs/overview.html#blaze-is-a-generalization-of-numpy
Spark has a python API and python has a bigger ecosystem for exploratory data analysis than spark.
Also python has a big data interface with out of core matrices and tables as well as a compiler than can speed numerical code, with a just a function decorator, to C like speeds. The former is an interface to the latter.
http://blaze.pydata.org/docs/dev/index.html
https://github.com/numba/numba
Some of the work that continuum have done on blaze [1] look to be tackling the problem of streaming in python too. In fact some of the ideas in this library come directly from haskell and one of the main developers is Stephen Diehl, who posted the very popular "what I wish I knew about haskell" slides recently.
Looks like a match made in heaven :-)
[1] http://blaze.pydata.org/
There seems to be a large overlap between several of the projects mentioned here and those developed by Continuum through a DARPA grant. For instance Seamless (Enthought) seems quite similar to Numba (Continuum), and ODIN (Enthought) looks a lot like Blaze (Continuum).
http://numba.pydata.org/
http://blaze.pydata.org/
Well, we'll be able to do that without sending the data to the client, not sure if our implementation right now will work without loading the data into memory, though long term that is definitely the plan (we will leverage http://blaze.pydata.org/)
If you want to discuss further, please email bokeh@continuum.io