Jeff Jonas

One past employer was looking to do such a thing. They handled electronic prescriptions, lab results and insurance claim filings. The premise was that pharmaceutical companies would be interested in long term longitudinal studies of patients.

I identified the highest risk to be the anonymizer system. I called it "sausagizer" because it is easy to turn cows into sausages, but hard/impossible to turn sausages back into cows.

One problem is that data entry isn't consistent or repeatable. So what might be entered as "John Doe" one day, might be "J Doe" or "Doe John" some other days. This is called "patient matching" and has been an area of interest for several decades. One paper discussing it is:
http://www.cecs.csulb.edu/~monge/research/thesis.pdf

Most researchers in similar areas ended up getting lured into "bioinformatics" for the Human Genome Project. More prestige, more money and gene matching is suprisingly similar to string matching.

Sadly, most of the research in the area of patient matching "went dark" after 911. By "going dark" I mean a combination of researchers who used to publish, now no longer do; and companies that used to publish even white papers have disappeared from the market. One such was NORA from SRD. IBM ended up buying the company, and shortly afterwards, all web pages, white papers and even the whole website of SRD vanished (even from archive.org).
http://www.pcworld.com/article/103692/tracking_terrorists_the_las_vegas_way.html…

Not that I know of, but my most recent information comes from this over-a-year-old post: http://www.scottaaronson.com/blog/?p=954 Google buying one doesn't make me shift much one way or another -- D-Wave has built something interesting that seems to be faster than other hardware, they are able to improve from the marketing buzz around 'quantum', but may or may not be using any quantum effects. People like http://jeffjonas.typepad.com/ have algorithms for problems that do increasingly better than the competition with increasingly "harder" or larger instances, so that behavior isn't really that weird.

What open-source tools are people using to build real-time big data analysis systems?

Jeff Jonas of IBM doesn't recommend batch systems such as Hadoop (http://jeffjonas.typepad.com/jeff_jonas/2011/04/the-data-is-the-query.html, http://techcrunch.com/2010/10/27/big-data/) for real-time context accumulation systems. I have heard that IBM is making use of topological embedding algorithms to build distributed graph databases capable of real-time analysis, but very little of that research is public.