Documents Reviewed
The documents below were reviewed for Quality of Research. The research point of
contact can provide information regarding the studies reviewed and the availability
of additional materials, including those from more recent studies that may have been conducted.
Study 1
Battistich, V., Schaps, E., Watson, M., Solomon, D., & Lewis, C. (2000). Effects of the Child Development Project on students' drug use and other problem behaviors. Journal of Primary Prevention, 21(1), 75-99.
Solomon, D., Battistich, V., Watson, M., Schaps, E., & Lewis, C. (2000). A six-district study of educational change: Direct and mediated effects of the Child Development Project. Social Psychology of Education, 4, 3-51.
Study 2
Munoz, M. A., & Vanderhaar, J. E. (2006). Literacy-embedded character education in a large urban district: Effects of the Child Development Project on elementary school students and teachers. Journal of Research in Character Education, 4(1-2), 27-44.
Study 3
Chang, F., & Munoz, M. A. (2006). School personnel educating the whole child: Impact of character education on teachers' self-assessment and student development. Journal of Personnel Evaluation in Education, 19(1-2), 35-49.
Study 4
Marshall, J., & Caldwell, S. (2007). Caring School Community implementation study four-year evaluation report. Rapid City, SD: Marshall Consulting.
Supplementary Materials
Battistich, V., Solomon, D., Watson, M., & Schaps, E. (1997). Caring school communities. Educational Psychologist, 32(3), 137-151.
Elementary school student questionnaire measures. (2000). Oakland, CA: Developmental Studies Center.
Marshall, J., & Caldwell, S. (2006). Caring School Community the Characterplus way. Rapid City, SD: Marshall Consulting.
Middle school student questionnaire measures. (2000). Oakland, CA: Developmental Studies Center.
Sherblom, S. A., & Marshall, J. C. (2005, April). Caring School Community: Growing character and academic achievement. Paper presented at the annual meeting of the American Educational Research Association, Montreal, Canada.
Outcomes
| Outcome 1: Alcohol use |
|
Description of Measures
|
Lifetime use of alcohol was assessed using a single question: "Do you drink alcohol (beer, wine, liquor)?" Participants indicated their use of alcohol on a 5-point scale (1 = never; 2 = once or twice; 3 = once in a while; 4 = often; 5 = used previously, but not anymore).
|
|
Key Findings
|
Reported use of alcohol declined significantly over time among students in schools that demonstrated high program implementation, while it increased slightly among students in matched comparison schools (p < .05). This difference represents a very small effect size (Cohen's d = 0.18).
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Quasi-experimental
|
|
Quality of Research Rating
|
2.5
(0.0-4.0 scale)
|
| Outcome 2: Marijuana use |
|
Description of Measures
|
Lifetime use of marijuana was assessed using a single question: "Do you smoke marijuana ("pot," "grass")?" Participants indicated their use of marijuana on a 5-point scale (1 = never; 2 = once or twice; 3 = once in a while; 4 = often; 5 = used previously, but not anymore).
|
|
Key Findings
|
Reported use of marijuana declined significantly over time among students in schools that demonstrated high program implementation, while it increased slightly among students in matched comparison schools (p < .01). This difference represents a small effect size (Cohen's d = 0.22).
|
|
Studies Measuring Outcome
|
Study 1
|
|
Study Designs
|
Quasi-experimental
|
|
Quality of Research Rating
|
2.5
(0.0-4.0 scale)
|
| Outcome 3: Concern for others |
|
Description of Measures
|
Concern for others was assessed using a 10-item Likert-type, self-report scale that measured students' feelings of concern for and desire to help other people. Items included, for example, "Everybody has enough problems of their own without worrying about other people's problems" and "When I hear about people who are sad or lonely, I want to do something to help."
|
|
Key Findings
|
Over 4 academic years, students in high-implementation schools across six school districts showed a small increase from baseline in their self-reported concern for others, while students in matched comparison schools showed a decrease in their concern for others (p < .01). This difference represents a very small effect size (Cohen's d = 0.13).
Another evaluation conducted in a single school district found that students in program schools reported greater concern for others 1 year following program implementation compared with students in matched comparison schools (p < .01).
|
|
Studies Measuring Outcome
|
Study 1, Study 2
|
|
Study Designs
|
Quasi-experimental
|
|
Quality of Research Rating
|
3.1
(0.0-4.0 scale)
|
| Outcome 4: Academic achievement |
|
Description of Measures
|
Academic achievement was measured using grade 3 and/or grade 4 nationally normed reading diagnostic test scores or statewide reading and math assessment scores.
|
|
Key Findings
|
An evaluation conducted in a single school district demonstrated a program effect on reading scores, with students in program schools achieving higher scores than students in matched comparison schools 1 year following implementation (p < .05). This difference represents a very small effect size (Cohen's d = 0.06).
A second evaluation conducted in the same schools 2 years later showed that high-implementation schools outperformed comparison schools in reading gains (p < .01). Specifically, the high-implementation schools reduced the percentage of students reading at the novice level from 29% to 16%, while the comparison schools reduced the percentage of students reading at the novice level from 26% to 21%.
A third evaluation conducted in another school district found program effects for both math (p < .05) and reading (p < .05) achievement. Based on State performance categories, approximately 45% of students in the program schools were categorized as proficient or advanced in math, compared with 37% of students in control schools. This difference represents a medium effect size (eta-squared = 0.12). Effects were even more pronounced for reading achievement, with longer duration of implementation associated with greater performance: 56% of students in schools with 3 years of implementation were categorized as proficient or advanced, compared with 50% in schools with 2 years of implementation, 46% in schools with 1 year of implementation, and 38% in control schools. This difference reflects a large effect size (eta-squared = 0.22).
|
|
Studies Measuring Outcome
|
Study 2, Study 3, Study 4
|
|
Study Designs
|
Experimental, Quasi-experimental
|
|
Quality of Research Rating
|
3.0
(0.0-4.0 scale)
|
| Outcome 5: Student discipline referrals |
|
Description of Measures
|
Data on student discipline referrals were provided by participating schools.
|
|
Key Findings
|
An evaluation conducted in a single school district demonstrated a program effect on student discipline referrals. The number of referrals across program schools decreased from 214 to 142 over 1 school year (p < .05). All but one program school showed a decrease in the annual number of referrals.
A second evaluation conducted in another school district also found a program effect on referrals. Over a 2-year period, a significant 24% decline was found in student discipline referrals in 20 program schools, while referrals increased 42% in 4 control schools (p < .01).
|
|
Studies Measuring Outcome
|
Study 3, Study 4
|
|
Study Designs
|
Experimental, Quasi-experimental
|
|
Quality of Research Rating
|
2.3
(0.0-4.0 scale)
|
Study Populations
The following populations were identified in the studies reviewed for Quality of
Research.
|
Study
|
Age
|
Gender
|
Race/Ethnicity
|
|
Study 1
|
6-12 (Childhood)
|
51.7% Female
48.3% Male
|
41.4% White
25.3% Hispanic or Latino
19.9% Black or African American
8.4% Asian
5% Race/ethnicity unspecified
|
|
Study 2
|
6-12 (Childhood)
|
50.6% Male
49.4% Female
|
52.3% White
37.3% Black or African American
10.4% Race/ethnicity unspecified
|
|
Study 3
|
6-12 (Childhood)
|
50% Female
50% Male
|
65% White
35% Race/ethnicity unspecified
|
|
Study 4
|
6-12 (Childhood)
|
Data not reported/available
|
Data not reported/available
|
Quality of Research Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the Quality of Research for an intervention's
reported results using six criteria:
For more information about these criteria and the meaning of the ratings, see Quality of Research.
|
Outcome
|
Reliability
of Measures
|
Validity
of Measures
|
Fidelity
|
Missing
Data/Attrition
|
Confounding
Variables
|
Data
Analysis
|
Overall
Rating
|
|
1: Alcohol use
|
2.0
|
2.3
|
3.0
|
2.7
|
2.3
|
2.5
|
2.5
|
|
2: Marijuana use
|
2.0
|
2.3
|
3.0
|
2.7
|
2.3
|
2.5
|
2.5
|
|
3: Concern for others
|
3.8
|
3.0
|
3.8
|
2.4
|
2.5
|
3.0
|
3.1
|
|
4: Academic achievement
|
3.3
|
3.0
|
3.1
|
3.0
|
2.8
|
3.0
|
3.0
|
|
5: Student discipline referrals
|
1.7
|
1.7
|
3.0
|
2.0
|
2.3
|
3.0
|
2.3
|
Study Strengths
The evaluation approach was theory driven and followed a conceptually sound logic model. Nationally normed, criterion-referenced achievement tests were used to measure academic achievement. Self-reported measures of drug use and concern for others relied on items similar to those used in the field or used scales with established reliability and validity. All teachers and staff associated with the program received intense training prior to program implementation, followed by continuous staff development training throughout the year. Implementation fidelity was measured using multiple measures including systematic classroom observations, program implementation checklists, and staff implementation surveys. Individual studies had adequate power to detect differences at the student level.
Study Weaknesses
The use of convenience samples and control schools that were involved in other initiatives limits the ability to make causal conclusions. Reliability and validity of the discipline referral data were questionable. Across studies, program implementation varied considerably among the treatment schools. In the majority of studies, treatment schools implemented the program in different years, introducing the possibility of interaction effects such as history and selection bias. The analytic strategy did not take into account the clustering of students by classroom.
