Gregory Friedman The Greg Wide Web on the name →

The Lectures · Course I

Machine Learning

One classic algorithm per lecture, and each one gets three treatments: the mathematics, derived rather than asserted, with every update rule earned from a loss function and the chain rule; the implementation, in plain Python, structured so that each line of code answers to a line of the derivation; and an interactive plate to descend, train, or break.

The code is pyml, whose GitHub description reads, in full, ML algs implemented in python for fun; I love reinventing the wheel. That was meant as a joke and has become a mission statement: frameworks are how you ship, from scratch is how you understand. When model.fit(X, y) works first time you have learned that the API is well designed; when your own gradient descent diverges, oscillates, then walks down the loss surface after you fix a transposed matrix, you have learned what fitting is. The ground rules follow from that. No ML frameworks anywhere. Linear algebra is fair game, automatic differentiation is not, because deriving the gradients is the entire point. Every analytic gradient is checked entry by entry against finite differences in the test suite, since a derivation that cannot survive a central difference is not a derivation but a hope. And an algorithm ships only when its lecture does.

That last rule is structural rather than principled. From-scratch ML repositories are the most-started and least-finished projects on GitHub: everyone writes linear regression, far fewer write the second algorithm, and almost nobody writes tests, because the fun is front-loaded and the discipline is not. Chaining each implementation to a published lecture means an unfinished algorithm is a visibly unfinished page on this site. The honest limitation is the genre's own: raw Python is orders of magnitude slower than anything you would deploy, and the numerical-stability machinery (careful initialisation, log-sum-exp, conditioning) is deliberately skipped unless it becomes a lesson in its own right. Every omission is a thing you can now see a framework doing for you, which is the point, but this is a teaching instrument and not a library.

  • I.
    Linear Regression & Gradient Descent published

    Least squares derived from the two partials, the normal equations, and the one-line update that survives every later model.

    Plate The loss surface itself: contours, the closed-form optimum, and four optimizers walking down to it at a learning rate you choose.

  • II.
    A neural network from scratch published

    Backpropagation derived honestly as the chain rule run backwards, then implemented raw, with every gradient checked against finite differences.

    Plate A two-layer network learning a boundary no line can draw, with the backward pass stepped through layer by layer.

  • III.
    Decision trees, and a model with no gradient published

    Greedy search over thresholds instead of descent, impurity as the thing being reduced, and the axis-aligned bias that arrives free with the choice of question.

    Plate A tree grown one split at a time beside the plane it is carving, the two linked because every node of a tree is a rectangle; both impurity criteria scored at once on the split about to be made.

  • IV.
    Clustering without an answer key published

    k-means and a Gaussian mixture as the same alternation with different commitments, and a dataset on which the correct answer scores worse than the wrong one.

    Plate The same points clustered twice, k-means against EM, stepped one phase at a time, with responsibilities drawn as shading and the seed exposed so local optima can be found on purpose.

Keys on this page

A figure's keys are its own: they answer once the figure has focus, so tab to it or click it first. Esc closes this sheet.