Significance Testing

Significance Testing checks the statistical significance between the results in the columns in the aggregated table when they have independent samples (for example if crossing with single questions in the column dimension). Significance Testing will be conducted on both proportion and means, and can also be applied to weighted tables.

Statistical Significance means that the differences found in the sample(s) may be assumed to exist in the population(s) from which the probability samples are drawn. Statistical significance has nothing to do with “importance”, as the term “significance” is used in normal language. The statistically significant differences are not more important than other differences. The larger the statistical significance, the greater is the probability that these differences are representative of the total population.

Reportal provides two methods of significance testing:

• T-Test
• Chi Square
• Z-Test

In the Significance Testing tab, click the down-arrow to open the list of test methods available and select the method to be used .

Figure 1 - Selecting the type of significance testing to be used

The formulae used for significance testing are described in Appendix A: Significance Testing.

T-Test and Z-Test Significance Testing

Forsta Plus supports T-Test and Z-Test significance testing.

When conducting Significance Testing, you can define up to two confidence levels. The higher the confidence levels, the more certain you can be that there really is a difference in the two groups being tested. For example, 90% confidence means that there is a 10% chance that a difference in scores could have been found purely through the effects of sampling .

Note that the second confidence level should be set lower than the first if the second level is to be identifiable in the table.

Figure 2 - Settings for Significance Testing

Reportal represents statistical significance by indicating the columns in which the corresponding cells are located, on the same row, where the difference is statistically significant. The columns are represented with letters that are included in the column headers. Significant cells will be displayed with upper case letters if the base is greater than or equal to 30. For base numbers less than 30, lowercase letters will be displayed. You can also specify a background color that will be used on these cells.

Significant cells for the second confidence level will be marked in parentheses if the cell is not already significant for the first level.

If you select "Significance test columns" in "Meta data" (this setting is default), text is included below the table explaining which columns have been tested and the confidence levels .

Figure 3 - Table with significance testing

The formulae used are described in Appendix A: Significance Testing.

Chi Square Significance Testing

With the Chi Square significance test, users can set a color for positive and negative probabilities and set probability intervals for indicating up to five different levels of significance (represented by + or - in the table cells depending on a positive or negative probability) .

Figure 4 - The properties when Chi Square significance testing is selected

Probability Level1 (*) - Probability Level5 (*****) are thresholds for displaying '-' and '+' in the table. You do not need to provide values for all the levels, but if you provide any levels the values must be in ascending order and in consecutive levels from level 1.

The formulae used are described in Appendix A: Significance Testing.

Note: If you select the Significance Test Columns option in the Meta Data tab (go to Meta Data for more information), text is included below the table explaining which columns have been tested and the confidence level.