Paper of the Edgeworth Series in Quantitative Behavioral Science

(Paper No. ESQBS-97-1)

Earlier version of this paper was presented in the April 1997 AERA meetings in Chicago. Prepared for the Symposium The logical basis and practical application of confidence (and other) intervals, Organizer & Chair: Professor John T. Behrens, Arizona State University.

About the authors: Bruno D. Zumbo is Professor of Psychology & Mathematics at UNBC, Gregory Pope is a graduate student at UNBC with interests in psychometrics and personality assessment, Johnny Stork is a graduate student at UNBC with interests in statistical science, computation in psychological research, and psychophysiology.

Address Correspondence to:

Dr. Bruno D. Zumbo, The University of Northern British Columbia, 3333 University Way, Prince George, B.C., CANADA

e-mail: zumbob@unbc.ca

WWW: http://www.educ.ubc.ca/faculty/zumbo/

When citing this paper, please use the following format (APA):

Zumbo, B. D., Pope, G. A., & Stork, J. (1997). Are Confidence Intervals Useful?: Looking Beyond the Simple Cases (Paper No. ESQBS-97-1). Prince George, B.C.: University of Northern British Columbia. Edgeworth Laboratory for Quantitative Behavioral Science.

We provide a list of resources to help researchers use interval estimation (primarily confidence and prediction) in everyday research practice. In so doing, we also make several observations on interval estimation in the social and behavioral sciences and show that confidence intervals when combined with data visualization methods (even as simple as a line graph) can help us understand complex data patterns and easily convey them to readers. We also make the case, via an example, that by focusing on the simplest of research scenarios (e.g., one-mean, 1 or 2 proportions, or mean differences for two groups), authors have undersold the potential of confidence intervals. Finally, we give brief consideration (again, via an example) to the use of confidence intervals with effect size indices.

This paper was originally titled Confidence Intervals Beyond the Simple Case. We accepted John Behrens' invitation to show through some examples how to compute confidence intervals in research scenarios more complex than those discussed in most statistics textbooks in the social sciences (e.g., a single mean, the difference between two means of dependent samples or independent samples with equal variances). To that point in time we (like Behrens and others) had not come across any thorough discussions of confidence intervals with an eye toward practice. Like most of you, we had seen many calls for the use of confidence intervals but no demonstration of how to go about using them in everyday statistical practice.

However, much to our delight (or dismay given that we had a paper to present) we found three sources which together provide detailed easily understandable descriptions of how to compute confidence intervals in these rarely discussed more complex cases. We would like to bring these sources to your attention. Table 1 provides a detailed list of where one can find the

various research scenarios in three key sources. These noteworthy contributions to the use of confidence intervals are Glass and Hopkins (1984) who discuss many applications of confidence intervals, and Loftus and Masson (1994) who present a lucid and comprehensive discussion of using confidence intervals in between-subject and (the rarely discussed) within-subject designs. Bowerman and O'Connell (1990) provide an extensive discussion and many examples of the use of confidence (and prediction) intervals for examining mean differences and for regression analysis. Bowerman and O'Connell also provide computer program code in SAS.

In reading these papers and textbooks we became haunted by the question of whether one should even be using confidence intervals? This, we believe, is the heart of today's symposium. As we all know, confidence intervals are increasingly being advocated as a means of analysis in research practice (see for example Cohen, 1994; Thompson, 1996). Should we be using them simply because Cohen, Thompson, and others are recommending them? The answer, of course, is no! The reason we should use confidence intervals is because they provide useful information in our scientific endeavors.

It is our contention that introductory textbooks and articles almost single-minded focus on the simple research case has hampered the wide-scale adoption of confidence intervals in research practice. That is, there is no particularly compelling reason to adopt confidence intervals for the commonly discussed simple cases of 2-samples or the confidence interval for 1 or 2 proportions. In these simple scenarios confidence intervals really do not add much to what we already can get from hypothesis testing -- and these intervals are certainly portrayed as such. This precise sentiment is voiced by students every term the senior author teaches introductory statistics. He, of course, dutifully reminds the students that in some research scenarios confidence intervals are particularly useful. The purpose of this paper, then, is to provide (in general terms) the situation(s) wherein confidence intervals are particularly useful and compelling.

With an eye toward the purpose of our paper, let us begin with a series of observations we have made regarding the motivation and proposed use of intervals in the social and behavioral sciences.

Some Observations on Interval Estimation in the Social and Behavioral Sciences

Observation #1. Clearly much of the current enthusiasm and literature in the social and behavioral science community on interval estimation is motivated by a desire to provide an alternative to null-hypothesis significance testing. However, these articles tend to be either unclear as to how one would benefit from interval estimates and/or they tend to use very simple research scenarios for which confidence intervals provide very little information over and above hypothesis testing (see for example, Cohen, 1994; Borenstein, 1994).

Observation #2. It is noteworthy that of all of the concepts in social and behavioral science methodology, statistical significance testing (and in particular null hypothesis testing) has the noble distinction of simultaneously: (a) being criticized fiercely and disavowed, and yet (b) guarded as one of our sacred cows.

The first of these distinctions can be evidenced in the work of Carver (1978; 1993), Guttman (1985), and Rozeboom (1960) amongst others. The second distinction can be seen in a 1993 issue of the Journal of Experimental Education (for example see Levin, 1993) as well as Lachman's (1993) suggestion to eliminate the phrase statistical significance from our scientific lexicon and to replace it with probable nonchance difference. If we are to believe Lachman's argument the ails of significance testing will be cured by continuing our use of conventional hypothesis testing but simply referring to our findings as either demonstrating or not demonstrating a probable nonchance difference1.

The object of many of the critics is not simply statistical significance testing (i.e., inference in its wide-sense meaning in statistics) but rather more specifically null hypothesis testing (NHT). This can be clearly seen in Guttman's (1985) remarks wherein he expresses a concern about the predominance of the reject/accept type of hypothesis testing:

Science has no special loss function which is to be minimized. And scientific hypotheses are always open to be retested. Research in science should be cumulative. In the course of accumulation, previous hypotheses become modified by corrections and extensions. There is no need for clear-cut acceptance or rejection of hypotheses as in operations research. The idea of accepting or rejecting a null hypothesis on the basis of a single experiment - which is the crux of Neyman-Pearson theory - is antithetical to science. (p. 4, 1985).

Rather than speaking generally of statistical significance testing or inference, Guttman is very particular as to the object of his concerns. Therefore, we will refer to the object of Guttman's, Carver's, Rozeboom's and countless other's concerns as NHT rather than to statistical significance testing or confidence intervals, per se.

Observation #3. Even though they have been vast in number, the criticisms of NHT are often seen only as potential problems and deconstructive. This can be seen clearly by Levin (1993) who, writing from the perspective of a statistician, substantive researcher, and then editor of the Journal of Educational Psychology, states "... [A]lthough one can certainly be roused by arguments full of sound and fury, until something better comes along significance testing just might be science's best alternative" (p. 378). The purpose of the present symposium is to offer and expand on an alternative that has been suggested by some -- interval estimation.

Observation #4. It is important to remember that the spirit of the alternative presented herein (confidence intervals) was first offered over 35 years ago (Rozeboom, 1960) but it appears we have ignored it until recently. In fact, Rozeboom's paper is often cited as being one of the first and most penetrating criticisms of NHT (e.g., Carver, 1978, p. 390) but what is not often cited is that Rozeboom also offers a workable alternative to NHT in the form of confidence intervals (although he does have a particular interpretation of them). Because Rozeboom's paper is rather dense and his discussion of confidence intervals is relatively brief, we provide (for archival purposes) a more detailed discussion of Rozeboom's approach in Appendix A including an example of how one could use the Rozeboom interpretation of confidence intervals.

Our answer to this question is twofold:

1. Confidence intervals are particularly helpful when used in understanding, and/or conveying in a simple manner, a particularly complex patterning in data. That is, for example, we concur with several authors' suggestion (Loftus and Masson, 1994; Tukey, 1974, 1977; Wainer & Thissen, 1993) that combining confidence intervals with data visualization (even a simple line graph) can help disentangle and understand a complicated patterning in the data (often among group means).
2. One can use confidence intervals to get a sense of the variability and confidence bounds for measures of effect size. Of course, one could also apply Rozeboom's particular interpretation of confidence intervals in this context (see Appendix A).

Let us treat each of these separately with an example data set taken from Lehman (1991, pp. 399-403) wherein one is studying the effect of three different drugs on headache pain. The design has 3 pain-relieving drugs and a placebo as one independent variable and age (classified as "young" and "old") as a second independent variable. Participants are chronic headache sufferers. Participants are asked to take the drug (or unbeknownst to them a placebo) during their next headache, wait 30 minutes, and then rate the degree of relief on a 20 point scale (1-20) where 20 represents complete relief and 1 indicates no relief at all. Appendix B lists the data and Table 2 lists the means and standard deviations for the study.

Using Confidence Intervals to Help Interpret Complex Patterns

Figure 1 lists the results of the 2-way completely randomized ANOVA. We can clearly ascertain from the ANOVA results that there is no main effect of the factor Drug but that there is a main effect of Age and an Age by Drug interaction. The simultaneous confidence intervals were computed based on the Studentized range statistic (Tukey). The analyses were conducted using SYSTAT.

Interestingly, Figure 2, a graphical representation of the Age main effect, does not add a great deal to what we already know from the ANOVA table (we will come back to this effect when we discuss the effect size). However, scenarios as simple as this one are often used as the impetus for using confidence intervals. Figure 3 is also particularly uninteresting in this case because there was no statistically significant main effect of drug. If there had been a main effect, Figure 3 could help us ascertain where that effect lay.

Figure 4 on the other hand is quite informative. That is, we know from the ANOVA table in Figure 1 that there is a statistically significant interaction. Figure 4 is split into two parts2 where the plot on the left is for young subjects (Age level 1) and the plot on the right for old subjects (Age level 2). Also note that both plots have the same metric so that one can easily make comparisons between them.

Comparing between the plots for the young and old subjects, as a whole, the confidence intervals are wider (indicating more variability) for the older group's (age=2) means than the corresponding means of the young group. Also, overlaying the two plots (this can be easily achieved manually by using a ruler) we can see that the younger and older groups' means are different from each other for each of the three drugs but not the placebo treatment (drug 4). Also, Drug 3 provides the least relief for the old group while the most relief for the young group.

From Figure 4 we can also see that the main effect of Age (noted earlier) is, in fact, somewhat misleading because it is not true that the young subjects got more pain relief at all levels of the drugs (as suggested by the main effect of Age). The cross-over interaction makes the Age main effect uninterpretable and inconsistent with the effects within drug levels.

Note that focusing on the plot of the means for the older group we can see that the confidence intervals barely overlap for Drugs 1 and Drug 2 indicating a very marginal difference. Also, for this same group of participants Drug 1 also provided statistically significantly more relief than Drug 3. On the other hand, for the younger group not only did Drug 3 provide better relief than any of the other drugs, including the placebo (Drug 4), but all of the means differed from each other except for Drug 1 which was not statistically different from the placebo.

Clearly, then, Figure 4, with its corresponding confidence intervals, provides a great deal of valuable information to the researcher and helps understand a fairly complex patterning in the means.

Using Confidence Intervals with Effect Size Measures

Let us imagine that rather than having a two factor design in Appendix B we only had data on the Age factor -- for the purpose of this example let us imagine that the data did not arise from 4 different drugs. We would conduct a t-test and compute a confidence interval for this difference: mean difference of 1.83 with a 95% confidence interval of (0.585, 3.065), t(78)=2.93, p=.004. Clearly there is a statistically significant difference and for practical inferential purposes the confidence interval does not provide any interesting new information over and above the NHT. To get a sense of the magnitude of the effect, it is often recommended that we compute an effect size index. In this case the point-biserial correlation is often recommended as a measure of effect size. The point-biserial correlation for this problem is 0.315. We, however, have not seen any discussion in the NHT literature (e.g., Kirk, 1996) of using a confidence interval for the point-biserial in gauging the variability of the effect size. The 95% confidence interval for the point-biserial of 0.315 is (0.101, 0.529) which can be interpreted as a range of plausible alternative hypotheses (as per Rozeboom's interpretation, see Appendix A) or interpreted as a more conventional 95% probability of coverage. Theoretically, these confidence bounds could also be interpreted with assistance from Cohen's (1992) or Kirk's (1996) criteria for a small, medium, or large effect. In this case ignoring the confidence interval would result in a conclusion that there is a near large effect (according to Cohen, one that is observable by the untrained eye) whereas the confidence interval for the point-biserial indicates that the effect is somewhere between a small and large effect -- in fact, covering the whole range of small to large.

In this paper we have provided a list of resources (see Table 1) for computing confidence intervals. Glass and Hopkins (1984), Bowerman and O'Connell (1990), and Loftus and Masson (1994) together consider a very wide range of research scenarios. The remainder of the paper was devoted to supporting our contention that confidence intervals are useful when used with graphical methods to interpret and convey, for the reader and researcher, complex data patterns. Furthermore, we contend that the common focus on confidence intervals for simple research cases (e.g., one sample mean, one proportion, two independent groups, or a paired sample problem) has undersold the true utility of confidence intervals for aiding in inference.

A critic may argue, as did a reviewer for an earlier draft of this paper, that what we claim as the effectiveness and power of confidence intervals is solely the effect of our use of methods of simultaneous inference. The effectiveness of understanding complex data patterns (like the one in our example above) is no doubt aided by simultaneous inference but the graphical display itself (including the simultaneous confidence intervals) is an essential part of understanding complex patterns in data. Furthermore, it is our opinion that confidence intervals are inextricably linked to simultaneous inference so that one cannot uphold one without the other.

In closing, there are several areas which require further development before one can easily adopt the everyday use of confidence intervals.

1. Software needs to be expanded to easily allow for: (a) plotting with confidence intervals, (b) more choice of method for simultaneous intervals (for a review see, Hancock & Klockars, 1996), and (c) publication quality graphics rather than text graphics such as those found in MINITAB or BMDP.
2. The theoretical and practical aspects of confidence intervals for effect size measures needs to be further developed. We reported a rather simple effect size measure (the point-biserial for 2 independent groups) in our example because we were not aware of easily computable confidence intervals for effect size measures like eta-squared or omega-squared.
3. More work needs to be done on the problem of confidence intervals for within-subjects designs and split-plot designs. This is particularly so for the cases where one would have to report a corrected result based on Geisser-Greenhouse or Huynh-Feldt methods.
4. Some meta-methodological consideration needs to be given to the potential impact of Rozeboom's (1960) suggestion of how to interpret confidence intervals and how it corresponds with Bayesian methods.

1. On a tangential note, Lachman's paper is demonstrative of a common misconception that null hypothesis testing can be used to test any kind of scientific hypothesis. Lachman's anecdote involving an occultist's claim to be able to make people disappear is, in fact, illustrative of deterministic hypotheses. Deterministic hypotheses do not involve any stochastic elements that can be accounted for in terms of probabilities and hence are inappropriate for null hypothesis testing. Therefore, in Lachman's anecdote, making even one person disappear is all of the evidence needed to support the occultist's claim. It should be noted, however, that Bayesian hypothesis testing allows one to evaluate some deterministic hypotheses.
2. We chose not to produce overlapping plots because the resulting Figure was too cluttered and hence hampered interpretation.

Borenstein, M. (1994). A note on the use of confidence intervals in psychiatric research. Psychopharmacology Bulletin, 30, 235-238.

Bowerman, B. L., & O'Connell, R. T. (1990). Linear statistical models: An applied approach. Boston, Mass.: PWS-KENT Publishing Company.

Carver, R. P. (1978). The case against statistical significance testing. Harvard Educational Review, 48, 378-399.

Carver, R. P. (1993). The case against statistical significance testing revisited. Journal of Experimental Education, 61, 287-292.

Cohen, J. (1994). The earth is round (p<.05). American Psychologist, 49, 997-1003.

Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155-159.

Glass, G. V., & Hopkins, K. D. (1984). Statistical methods in education and psychology (2nd Edition). Englewood Cliffs, N.J.: Prentice-Hall, Inc..

Guttman, L. (1985). The illogic of statistical inference for cumulative science. Applied Stochastic Models and Data Analysis, 1, 3-10.

Hancock, G. R., & Klockars, A. J. (1996). The quest for alpha: Developments in multiple comparison procedures in the quarter century since Games (1971). Review of Educational Research, 66, 269-306.

Kirk, R. E. (1996). Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56, 746-759.

Lachman, S. J. (1993). Statistically significant difference or probable nonchance difference. American Psychologist, 48, 1093.

Lehman, R. S. (1991). Statistics and Research Design in the Behavioral Sciences. Belmont, Calif.: Wadsworth Publishing Company.

Levin, J. R. (1993). Statistical significance testing from three perspectives. Journal of Experimental Education, 61, 378-382.

Loftus, G. R., & Masson, M. E. J. (1994). Using confidence intervals in within-subject designs. Psychonomic Bulletin & Review, 1, 476-490.

Mosteller, F., & Tukey, J. W. (1977). Data Analysis and Regression: A Second Course in Statistics. Reading, Mass.: Addison-Wesley Publishing Company.

Rozeboom, W. W. (1960). The fallacy of null-hypothesis significance testing. Psychological Bulletin, 57, 416-428.

Thompson, B. (1996). AERA editorial policies regarding statistical significance testing: Three suggested reforms. Educational Researcher, 25, 26-30.

Tukey, J. W. (1974). The future of data analysis. Annals of Mathematical Statistics, 33, 1-67.

Tukey, J. W. (1977). Exploratory data analysis. Reading, Mass.: Addison-Wesley.

Wainer, H., & Thissen, D. (1993). Graphical data analysis. In G. Keren & C. Lewis (Eds.), A handbook for data analysis in the behavioral sciences: Statistical issues (pp.391-458). Hillsdale, N.J.: Erlbaum.

To read (or print) a copy of Table 1 click here

To read (or print) a copy of Table 2 click here

To read (or print) a copy of Figure 1 click here

To read (or print) a copy of Figure 2 click here

To read (or print) a copy of Figure 3 click here

To read (or print) a copy of Figure 4 click here

Rozeboom's (1960) statement that confidence intervals are one of the most adequate aids to inference that we possess is true even today. Confidence intervals are encountered by most students of the social and behavioral sciences. In this light, confidence intervals should not be interpreted simply in terms of probability of coverage, as is often taught in introductory courses, but rather as an impartial simultaneous evaluation of all of the alternatives under consideration for the current data. As Rozeboom states, a confidence interval is a subset of the alternative hypotheses computed from the data at hand in such a way that for a p% confidence level, the probability that the true hypothesis is included in a set so obtained is p%. Note that in Rozeboom's suggestion the commonly used 5% chance of error refers to the incorrect simultaneous dismissal of a large part of the total set of alternative hypotheses and is the total likelihood of error, not just of Type I. Quite simply with the use of confidence intervals a socio-behavioral scientist can conclude a given data are much better fit by those hypotheses whose values fall inside the interval than by those outside. Therefore, Rozeboom's suggestion could be interpreted in our current climate of anti-NHT as suggesting that rather than salvage the sacred cow of hypothesis testing, we butcher the beast and move to a more nutritious feast offered to us over three decades ago by Rozeboom.

Rozeboom's interpretation of confidence intervals is not the current norm of what is taught in social and behavioral science textbooks. A semblance of Rozeboom's suggestion can be seen, however, in Mosteller and Tukey's (1977) classic textbook wherein they introduce the one-sample t-test and end their discussion of testing against various hypothesized population means by stating, "It [a hypothesized population mean] might take on, in turn, all possible values, as when we seek a confidence interval (p. 3).

Rozeboom's interpretation of confidence intervals appears, at least on a surface level, to be akin to Bayesian credible intervals (or highest density regions). In this same light, it should be noted that it is inherent in the Bayesian (and also the fiducial) approaches to confidence intervals that the parameter in question is, in some sense, random. It is important to note that this is not the case for more commonly used probability of coverage perspectives on confidence intervals where only the interval is random and may or may not cover the fixed but unknown value of the quantity we are trying to estimate.

Briefly, following Rozeboom (1960) we show that instead of using hypothesis testing as the ultimate mechanism through which we make decisions and make commitment to action on the basis of experimental data, we should use a methodology that reflects the scientific process as a cognitive activity. That is, the scientific process is a cognitive activity in which we make an appropriate adjustment in the degree to which we accept or believe knowledge claims given the empirical finding(s) at hand.

An example

The hypothetical data come from an illustration in Glass and Hopkins (1984, pp. 233-236) wherein, based on previous research and theory and maybe some intuition, a researcher believes that an intensive treatment of environmental stimulation will change the intelligence of infants. Thirty-six infants were randomly assigned to either a control or experimental group. After two years of treatment, an intelligence test was administered to all thirty-six children.

The sufficient statistics to compute a confidence interval about the mean differences are: experimental group sample statistics are mean of 108.10, variance of 289.00, and the sum of squared observations of 4913.00, and for the control group a mean of 98.40, variance of 196.00, and sum of squared observations of 3332.00.

The 95% confidence interval for the mean difference of 9.70 is 9.70 plus and minus 10.59 or between -0.89 and 20.29. The conventional interpretation of this confidence interval is that over a long run frequency of repeated samplings 95 out of 100 samples would lie between -0.89 and 20.29. Furthermore, given the duality of confidence intervals and hypothesis testing where one can technically obtain a confidence interval by inverting a hypothesis test, and that the confidence interval includes the value of 0 where the null hypothesis is that the population means for the experimental and control groups are equal, the null hypothesis is not rejected.

From Rozeboom's perspective the confidence interval of -0.89 to 20.29 represents a subset of the alternative hypotheses computed from the data at hand such that for the 95% confidence level, the probability that the true hypothesis is included in this set is 95%. More specifically, given: (1) the mathematical model entailed by computing the confidence interval the way we did, (2) the experimental design, and (3) the measurement, the current data are much better fit by some parameters (inside the interval) than by others (outside). On a practical note, then, the researcher set out with the belief (or at least the hunch or intuition) that the environmental stimulation will change infants' intelligence. The finding at hand suggests that the researcher needs to make an adjustment in the degree to which she/he believes her/his intuition or hunch.

It is important to note that Rozeboom's approach reflects the scientific process as a cognitive activity where intuition and degrees of belief in various hypotheses are adjusted based on the empirical findings at hand and is akin to the Bayesian method of highest density regions.

>LIST DRUG AGE SEVERITY

Case number DRUG AGE SEVERITY

1 1.000 1.000 11.000

2 1.000 1.000 8.000

3 1.000 1.000 6.000

4 1.000 1.000 8.000

5 1.000 1.000 11.000

6 1.000 1.000 6.000

7 1.000 1.000 10.000

8 1.000 1.000 9.000

9 1.000 1.000 5.000

10 1.000 1.000 6.000

11 1.000 2.000 10.000

12 1.000 2.000 9.000

13 1.000 2.000 9.000

14 1.000 2.000 15.000

15 1.000 2.000 10.000

16 1.000 2.000 11.000

17 1.000 2.000 13.000

18 1.000 2.000 13.000

19 1.000 2.000 13.000

20 1.000 2.000 14.000

21 2.000 1.000 14.000

22 2.000 1.000 10.000

23 2.000 1.000 10.000

24 2.000 1.000 12.000

25 2.000 1.000 12.000

26 2.000 1.000 13.000

27 2.000 1.000 11.000

28 2.000 1.000 11.000

29 2.000 1.000 10.000

30 2.000 1.000 10.000

31 2.000 2.000 9.000

32 2.000 2.000 8.000

33 2.000 2.000 7.000

34 2.000 2.000 9.000

35 2.000 2.000 8.000

36 2.000 2.000 7.000

37 2.000 2.000 7.000

38 2.000 2.000 8.000

39 2.000 2.000 7.000

40 2.000 2.000 7.000

41 3.000 1.000 14.000

42 3.000 1.000 13.000

43 3.000 1.000 11.000

44 3.000 1.000 15.000

45 3.000 1.000 13.000

46 3.000 1.000 13.000

47 3.000 1.000 15.000

48 3.000 1.000 12.000

49 3.000 1.000 15.000

50 3.000 1.000 18.000

51 3.000 2.000 7.000

52 3.000 2.000 6.000

53 3.000 2.000 6.000

54 3.000 2.000 6.000

55 3.000 2.000 5.000

56 3.000 2.000 6.000

57 3.000 2.000 8.000

58 3.000 2.000 7.000

59 3.000 2.000 7.000

60 3.000 2.000 6.000

61 4.000 1.000 9.000

62 4.000 1.000 8.000

63 4.000 1.000 9.000

64 4.000 1.000 12.000

65 4.000 1.000 13.000

66 4.000 1.000 8.000

67 4.000 1.000 10.000

68 4.000 1.000 6.000

69 4.000 1.000 8.000

70 4.000 1.000 13.000

71 4.000 2.000 8.000

72 4.000 2.000 11.000

73 4.000 2.000 6.000

74 4.000 2.000 10.000

75 4.000 2.000 8.000

76 4.000 2.000 8.000

77 4.000 2.000 9.000

78 4.000 2.000 12.000

79 4.000 2.000 14.000

80 4.000 2.000 11.000

>

If you are so inclined, please sign our guest book below