Practical Methodologies


(This page is based primarily on material from Chapter 11 of the Deep Learning book)

Random Hyperparameter Search

Random hyperparemeter search approach:
  1. Discrete: bernoulli or multinoulli distribution
  1. Continuous: uniform distribution and then exponentiate
At each step, sample from all distributions
Why is random search better than grid search?
Exponentially more efficient when there are several params that don't matter much


Goodfellow's debugging tips
  1. Test cases
    1. Design test case so simple correct behaviour can be predicted (e.g. tiny dataset)
    2. Design test case that exercises one part of the model in isolation
  1. Visualisations
    1. Model in action
    2. Worst (and best) performance
    3. Model statistics