Start by framing OLS as solving the normal equations X^T X β = X^T y, which only requires accumulating the p×p Gram matrix and p-vector X^T y. Then walk through a streaming, numerically stable, and parallelizable implementation, and finish with a verification strategy against a small in-memory baseline.
Pro tip: Emphasize that you would never form X^T X explicitly if p is large; instead, use a QR-based streaming approach or Cholesky with regularization, and always center/scale features to improve conditioning.
Explain that OLS only needs X^T X and X^T y, which can be computed in a single pass over the data. This avoids storing the full 25 GB file in memory.
Describe how to read the CSV in chunks, compute local Gram matrices and vectors, and combine them. Use Welford's algorithm or compensated summation to avoid catastrophic cancellation, and consider scaling features.
Split the file across workers, compute partial X^T X and X^T y on each, then reduce (sum) them. Use a tree reduction to minimize error, and solve the normal equations on a single node.
Solve (X^T X + λI)β = X^T y using Cholesky decomposition or QR. Mention that λ is a small ridge term for numerical stability, not regularization.
Sample a small subset that fits in memory, fit OLS using a standard library, and compare coefficients and predictions. Also check residual norms and condition number of X^T X.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.