Hand Computation, Conceptual Debugging, and Coding Projects

by Justin Skycak on

The 3 types of problems that I would have students work out back when I was teaching ML.

Back when I was teaching ML, there were 3 main types of problems that I would have students work out. Only one of those problem types involved coding.

1) HAND COMPUTATION

For instance: Find the new function after 1 iteration of gradient descent when fitting f(x)=1/(1 + e^(ax+b)) to the data [(0,0.1), (1,0.2), (2,0.5)] using

  • initial guess f(x)=1/(1 + e^(-x))
  • learning rate alpha=0.1
  • MSE as the loss function

2) CONCEPTUAL DEBUGGING

Questions in the format “suppose you make a particular mistake when coding up your algorithm. What will happen?”

For instance: What will happen if you add the gradient instead of subtracting it, i.e., you code up your gradient descent as


param --> param + alpha * dMSE/d(param)

instead of


param --> param - alpha * dMSE/d(param)?

(Answer: The loss will go up instead of down)

You can also flip these questions the other way: given the symptom, diagnose the cause.

For instance: Suppose you have a loss curve that’s going up instead of down. Which of the following could be the root cause?

  • I. The gradient is being added instead of subtracted
  • II. The learning rate is too small
  • III. The learning rate is too big

(Answer: I and III.)


3) CODING PROJECTS

I’d start out having students write the code to carry out a procedure they previously worked out (an iteration of) by hand.

Afterwards, we’d tackle more complicated cases where it would be infeasible to work out even a single iteration by hand.