The probability that a particular regression coefficient is zero. When p is small, say
less than .05, there is only a 5% chance or less than the coefficient is zero.
Significant variable or coefficient
A variable or a coefficient of a variable is significant
when its p-value is less than .05. That is, there is less than a 5% chance that the
coefficient is zero.
Insignificant variable or coefficient
A variable or a coefficient of a variable is
insignificant when its p-value is greater than .05. That is, there is more than a 5%
chance that the coefficient is zero. As a general rule (there are exceptions), when a
variable is found to be insignificant in a particular model, it should not be included in
future models.
95% confidence interval
The interval in which we can be 95% certain that a coefficient
will lie, meaning that the coefficient will lie in this interval 95% of the time.
Principle of parsimony
Equivalent to K.I.S.S. If we have a choice between two models, we
should choose the simpler or smaller model of the two, provided that it does reasonably
as well as the larger, more complicated model. (This principle is also known as Occam’s
Razor: Things should not be multiplied without reason.)