Questions to ask of high-dimensional data
Section of High-dimensional analysis, part of Statistical learning
Questions to ask of high-dimensional data
| Type | Section |
| Taught in | High-dimensional analysis |
| Part | Statistical learning |
The choice of method depends on the question.
- Which of the predictor variables are associated with the response? Either one variable at a time, or the explanatory variables all at once.
- Can the explanatory variables be transformed into a space of smaller dimension without losing anything important about the association with the response?
- Can they be transformed into a smaller space when there is no response, either to capture structure or to cluster?
- How can the highest-quality predictions of the response be made from the explanatory variables?
Selecting a subset of coordinates is one restricted form of dimension reduction. Finding scientific associations and finding a predictive representation need not give the same answer. Unsupervised principal components come first, then marginal testing, then joint penalised regression. Keep the target question visible when comparing their outputs.
Used in
See also
Rank bound · Family-wise control implies false-discovery control · Benjamini–Hochberg · Benjamini–Hochberg controls the false discovery rate · Why classical methods fail when p > n · Rank of the sample covariance · Regression when p > n · Where high-dimensional columns come from