Day 14 &15 Table of Contents

• Review topics from last week
• Choosing appropriate Tests of Significance
• t Test for Independent and Dependent Groups
• Analysis of Variance (ANOVA) and Multiple Comparisons
• Factorial Analysis of Variance
• Analysis of Covariance (ANCOVA)
• Multiple Regression
• Chi Square
• Closure

Review

Inferential Statistics:	Attempts to identify if the results we obtain from a sample reflect the results we would obtain if we studied an entire population.
Standard Error of the Mean:	The standard error of the mean is the standard deviation of the sample means (also known as the standard deviation of the sampling error). It is found by dividing the standard deviation of the sample by the square root of sample minus 1.
Null Hypothesis:	States that there is no difference due to treatment (although Standard Error may contribute to a difference).
Level of Significance:	An estimate of the probability that we are wrong when we say there is no difference between the two samples (alpha level).
Type I error:	We state the treatment made a difference when the difference was really caused by chance (we rejected the null hypothesis when we shouldn't).
Type II error:	We state there is no difference due to treatment when there was (we accept the null hypothesis when we shouldn't).
Tales of a test:	Two tailed indicates either group may do better than the other; one tail indicates one group will do better than the second.
Degrees of Freedom:	Found by subtracting the number of constraints from the number of subjects (example: N-2).

Tests of Significance

"It is important that the researcher select an appropriate test; an incorrect test can lead to incorrect conclusions." Gay, p 476.

To select the appropriate test:

• Determine if you can use a parametric test. You must meet assumptions:
  • The dependent variable is normally distributed. The variable you are measuring is not skewed.
  • The data is represented on an interval or ratio scale. The differences between scores are equal.
  • The subjects are selected independently. The selection of one subject doesn't effect the selection of the others.
    
  • The variances of the comparison groups are equal. Both groups have similar standard deviations

   

  Use a non-parametric test when:

  • Subjects were not selected independently
  • The scores are ranked (ordinal scale)
  • The scores are greatly skewed or we have no knowledge of the distrubiton.

Parametric tests are more powerful for any sample size (easier to reject the null hypothesis).

• Choose the appropriate test based on:

  the number of groups; the number of variables; and the pretest scores.

The following key will allow you to select some of the statistical tests according to our class notes. Please note that this is not a comprehensive list; it simply reflects the more common tests.

t Test

General information on t tests:

• Determines if two means are significantly different at a selected probability level (are the experimental and control groups really different on the posttest?)
• Compares the actual mean differences with the differeneces expected by chance (the groups will never exactly be the same, even without treatment).
• If the calculated t is equal or greater than the table value, we reject the null hypothesis.

t Test for Independent Samples

• members of one group are not related to other group (other than same population)
  • students are randomly assigned to one of two groups
    

t Test for Dependent Samples

• members of one group are related to other group
  • same sample but at different times (pre and post test; Test 1 and Test 1)
    

Which are better: Gain scores or posttest scores?

• If pretest scores are similar, use posttest scores (not gain scores: Ceiling effect)
• If pretest scores differ, use analysis of covariance (statistically equates pretest scores).

Analysis of Variance

Used to identify a significant difference in two or more means at a selected probability level. Better than multiple t Tests.

The ANOVA computes what is termed an F ratio

The F ratio compares the variance between the groups (variance due to treatment) divided by the variance within the groups (similar to combining the standard errors).

If the ratio of the two variances are similar (if the differences are not larger than what we would expect by chance), we accept the null hypothesis.

If the ratio of the two variances are significantly different, we reject the null hypothesis. Conclusion: the treatment had a significant difference on the groups.

But which group means are significantly different? You need to complete a multiple comparison to identify.

Scheffé test allows you to check possible comparisons involving a set of means. Best to decide which means to examine before experiment.

Example: We use a control group receiving traditional instruction, a group receiving multimedia instruction, and a group receiving instruction through video conferencing. If means are compared and we find a significant F ratio, we can run a Scheffé's test to see which group was significantly different.

Factorial Analysis of Variance

When two or more independent variables are considered as well as the interaction between them, use the factorial analysis of variance.

This computes a separate F ratio for each independent variable and the interaction between the variables.

Example: You want to study if a new interface to your computerized training program helps employees learn the material quicker than the old interface. You are also interested if it affects both high and low IQ employees the same.

Dependent variable: Amount of knowledge gained.
Independent variables: Type of Interface and IQ.

Three separate F ratios will be generated.

		Example 1 shows that the new interface worked well for both those with low and high IQ, and that high IQ employees did better than low IQ employees with both interfaces. A factorial analysis of variance indicated a statistically significant difference in the interface and IQ, but not on an interaction.
		Example 2 shows that the new interface worked best for those with high IQ, and that high IQ employees did better than low IQ employees with both interfaces. A factorial analysis of variance indicated a statistically significant difference in IQ, but not in interface. It also showed a significant difference in the interaction of IQ and method.

Analysis of Covariance

Analysis of Covariance (ANCOVA) is used to statistically equate scores (it controls for a variable that is not equal before a treatment).

An assumption is random assignment of subjects. Can be used for

• Causal comparative research (cautiously)
  • Example: Did students who graduated in 1995, who had Ms. Jones for English, get better CBEST scores at the end of the year than those who had Mr. Smith?
    
• Experimental research
  • Example: Will students who are assigned in 1997 to Ms. Jones in English get better CBEST scores at the end of the year than those who will have Mr. Smith?
    
• Both examples assume that we have pretest CBEST scores, and that students are ramdomly assigned to the classes.

ANCOVA also increases the power of a statistical test by reducing the chance of a Type I error.

Multiple Regression

Used to identify which variables predict (correlate with) a criterion (outcome).

Good with ratio, interval, and nominal data, and with experimental, causal-comparative, and correlational studies.

It identifies:

• Which variable is the strongest predictor, and the strength of that variable
• Which variable adds the next amount of strength, and
• If any other variables add strength to the prediction.

Example: What variables determine how well students do in Ed 690 final exam?

• Dependent variable (posttest score) is the criterion
• Possible variables to examine might include:
  • Midterm exam
  • Semesters in the EdTec program
  • Age
  • Gender
  • Math section of GRE score

With multiple regression we may find that a prediction can be based on 50% math section of GRE and another 10% from the midterm exam. None of the other variables contributed to improve the prediction rate.

Limitation: the number of variables examined is dependent on the sample size.

Chi Square

This is a nonparemetric test used when data are in the form of frequency counts occurring in two or more mutually exclusive categories (nominal catagories).

The Chi Square compares observed counts with expected counts.

Example: You show 30 people a differently colored background screen for a web page. If all are equal, you would expect to have 10 people select each color.

• If 6 people choose red; 8 people choose blue, and 16 choose green, is this a significant difference (or simply due to chance)?

A one-dimensional Chi Square examines one independent variable; a two-dimensional Chi Square examines two independent variables (color and gender).

A Chi Square table, using alpha and degrees of freedom, identifies when a calculation will be significant.

Closure; Review and Assignments

Review questions: (To find the answers, click on the question mark icon)

• Explain when to use parametric and nonparametric tests
  State the purpose of the t test for independent and non-independent samples
  Explain a major problem associated with using gain scores
  State the purpose of the analysis of variance and multiple comparison
  State the purpose of the factorial analysis of variance
  Explain the two uses of the analysis of covariance
  Explain when it is appropriate to use the Chi Square test

Before next week:

• Complete your final projects