Brendan O'Connor

My gut-ballpark was going to be around an order of magnitude, not two. Here's are two naive comparisons (granted, from the late aughts and not a direct comparison but a more general one) that show ~7-8x[0,1]. The overhead of PyStringObject is not trivial[2] (though the implementation details likely have changed between Py2.x and Python 3).

For things like building accumulators a set of data/log parsing rather than data munging (hits per hour or enumerative tasks), I'd imagine (g|n)?awk might hit your 100x since you'd just grab the fd and traverse being IO bound. I'm not sure how awk does it, but if it's just saving an accumulator value (or 10) in a register. Assuming x64-64 treats, say, an r3 fetch analogously to a fetch to ecx (err..rcx now I guess), rather than having to keep a full object in L1 cache, awk has a huge advantage.

---

N.b., if you're benching tasks like this, don't use 'time' and STDOUT and think you're getting real performance numbers. Your bottleneck (terminals can only render $x lines a minute, so the kernel call to write(STDOUT, ....) will be where you choke, not at the language. Also disk fragmentation would be another issue. Put the both the test file and the output file on a RAMdisk.) Cache flush with Something like sync; echo 3 > /proc/sys/vm/drop_caches (on Linux, I forget the BSD way of doing it) then `time benchmark.py /mnt/ramdisk1/file > /mnt/ramdisk2' over multiple runs, under various loads, with different data sets, etc

Another interesting…

Here's a few things that really helped me out:

http://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/
http://brenocon.com/blog/2013/10/tanh-is-a-rescaled-logistic-sigmoid-function/

Sort of a high-level overview (some of this is particular to neural nets as they're implemented in JMP, a product made by SAS, I don't know how well they generalize):

• Neural nets consist of an input layer, an output layer, and some number of hidden layers between the input and output layers
• The hidden layers consist of a number of nodes, which can be thought of as “on-off” switches (called a “step function” if we’re using the proper term), although in practice they’re represented by a smooth sigmoid curve with an upper and lower bound that represent on and off. In JMP, this takes the form of a hyperbolic tangent function, which is bound between -1 and 1.
• Each node has a value that’s called the bias, which if you’re thinking of the nodes as an on-off switch, is the threshold a linear combination of the inputs needs to reach to turn the node on. In the sigmoid curves output by JMP, the bias represents the point at which the hyperbolic tangent function returns 0, given a linear combination of the inputs.
• A “deep learning” model is one that has more than one hidden layer; the lowest hidden layer serves as the input layer to the next hidden layer, and so on.
• The output layer takes the nodes of the highest hidden layer and produces an output. When dealing with probability,…

Interesting article! However, Deep Blue, Watson, and AlphaGo are very different from one another. I don't think anyone deemed beating humans at chess or jeopardy impossible at the time Deep Blue and Watson were built. On AlphaGo the point about generalizing AI is valid, yet I think the author doesn't fully appreciate the novelty of approach described in the AlphaGo paper. Their work advances the field, has more general utility than Deep Blue's chess or Watson's jeopardy programs. AlphaGo paper specifically represents an advance in machine learning algorithms for games in general. As I understand, Watson's new NLP algorithm is PRISMATIC [1]. PRISMATIC is a rule-based NLP system and AlphaGo is more statistical inference/neural networks. Even if AlphaGo's 'policy-network- value-network' framework is not too generalizable, the philosophical implication is that we can build AIs that can mimic 'human intuition'. Jeopardy and chess have lesser components of 'human intuition' than Go. They are apples and oranges. So, I wonder if the author errs in bringing Deep Blue and Watson-Jeopardy into the picture.

In my opinion while Watson-Jeopardy and Deep Blue were 'over-fitting' for Jeopardy and chess respectively, AlphaGo algorithms are more general and over-fits for the larger category of 'games'.

[1] http://brenocon.com/watson_special_issue/05%20automatic%20knowledge%20extration.pdf

When I wrote "In this case it measures the relative reduction in MSE" I meant exactly that.

The mean squared error of the baseline model which doesn't include the state as a regressor is 0.25 (it predicts always 0.5 - it's off by 0.5 in every case).

The mean squared error of the model which includes the state as a regressor is 0.2475 (it predicts 0.45 or 0.55 depending on the state - in both cases it's off by 0.45 with 55% probability and it's off by 0.55 with 45% probability).

The mean squared error is directly related to variance when the predictor is unbiased. The ratio of the sum of squares is the same as the ratio of the mean square errors.

Edit: http://brenocon.com/rsquared_is_mse_rescaled.pdf

"R2 can be thought of as a rescaling of MSE, comparing it to the variance of the outcome response."

https://dabruro.medium.com/you-mention-the-average-squared-error-how-is-this-defined-if-it-is-different-from-mean-squared-fb9a1703b7aa

"Also it is worth mentioning that R-squared (coeff. of determination) is a rescaled version of MSE such that 100% is perfection and 0% implies the same MSE that you would get by simply always predicting the overall mean of the dataset."

Statsmodels has a nice implementation of logistic regression more suited for statistical analysis.

It has no default regularisation, nice labelled coefficients table, and model evaluation metrics.

https://www.statsmodels.org/dev/generated/statsmodels.discrete.discrete_model.Logit.html

This reflects the difference in intended uses. I view Sklearn as a collection of tools to help forecast a target, whilst statsmodels is collection of tools to examine a model you have of the data-generating process.

If you want to do the former, sensible defaults are helpful - for example random forest has many tune-able parameters, and I am glad maintainers haven't shrugged and left all unspecified, because I know I can quickly fit a reasonable first-cut model. For latter, you are better off using statsmodels. In a way this is part of the culture difference between statistics and machine learning.

http://brenocon.com/blog/2008/12/statistics-vs-machine-learning-fight/

One common pain point with sklearn is that it uses numpy arrays as representation of data series, whilst a lot of the ecosystem create data cleaning and prepare their data as pandas dataframes.

Well, it shouldn't be faster than "Read 1,000,000 bytes sequentially from memory" (741ns) which in turn shouldn't be faster than "Read 1,000,000 bytes sequentially from disk" (359 us).

That said, all those numbers feel a bit off by 1.5-2 orders of magnitude -- that disk read speed translates to about 3 GB/s which is well outside the range of what HDDs can achieve.

https://brenocon.com/dean_perf.html indicates the original set of numbers were more like 10us, 250us, and 30ms.

And it links to https://github.com/colin-scott/interactive_latencies which seems like it extrapolates progress from 14 years ago:

// NIC bandwidth doubles every 2 years
// [source: http://ampcamp.berkeley.edu/wp-content/uploads/2012/06/Ion-stoica-amp-camp-21012-warehouse-scale-computing-intro-final.pdf]
// TODO: should really be a step function
// 1Gb/s = 125MB/s = 125*10^6 B/s in 2003

which means that in 2026 we'll have seen 11 doublings since gigabit speeds in 2003, so we'll all have > terabit speeds available to us.