Quality of Research
Review Date: September 2010
Documents Reviewed
The documents below were reviewed for Quality of Research. The research point of
contact can provide information regarding the studies reviewed and the availability
of additional materials, including those from more recent studies that may have been conducted.
Study 1Evans, W., & Smith, M. (2010). Project MAGIC: Building communities of support for alternatives to juvenile detention: Cumulative data report. Unpublished manuscript. Study 2Smith, M., & Evans, W. (1997). Project "MAGIC": Building communities of support for alternatives to juvenile detention. In Bringing excellence to substance abuse services in rural and frontier America: 1997 Award for Excellence award-winning papers (pp. 15-27). Rockville, MD: Substance Abuse and Mental Health Services Administration, Center for Substance Abuse Treatment.
Supplementary Materials Cooperative Extension National Evaluation Review Process
Smith, M. (2000). "MAGIC" (Making a Group and Individual Commitment): A program for entry-level juvenile tribal offenders in Owyhee, Nevada (Fact Sheet 00-31). Reno: University of Nevada Cooperative Extension. Available at http://www.unce.unr.edu/publications/files/cd/2000/fs0031.pdf
Smith, M., Usinger-Lesquereux, J., & Evans, W. (1999). Rural juvenile first offenders describe what is working and what is not. International Journal of Offender Therapy and Comparative Criminology, 43(3), 322-337.
Van Poperin, A. L., & Smith, M. (2006). Project MAGIC in Northern Nye and Esmeralda counties (Fact Sheet 06-72). Reno: University of Nevada Cooperative Extension. Available at http://www.unce.unr.edu/publications/files/cy/2006/fs0672.pdf
Outcomes
Outcome 1: Academic engagement and achievement |
Description of Measures
|
Academic engagement and achievement were assessed using the Nevada Bureau of Alcohol and Drug Abuse (BADA) Youth/Parent Questionnaire at pretest; a shorter version called the MAGIC Youth/Parent Questionnaire was used at posttest. The shorter version measures the same constructs as the full questionnaire but is designed to reduce respondent burden. Both instruments are self-report and include multiple-choice questions that measure knowledge about and attitudes toward substance abuse, parental monitoring, goals, school, and other key areas addressed by the Project MAGIC curriculum.
|
Key Findings
|
Survey responses of youths indicated that from pre- to posttest, Project MAGIC youth participants improved in aspects of academic engagement and achievement:
- Reduction in school absences (p < .01)
- Better grades (p < .001), with more youths reporting that they were now earning better grades than most of their classmates (p < .001)
- Greater recognition that the things the youths were learning in school are important for later life (p < .01)
- Greater participation in extracurricular activities (p < .01) and in community service or volunteer work (p < .001)
Survey responses of parents showed pre- to posttest improvements among Project MAGIC youth participants in many of the same aspects:
- Reduction in school absences (p < .001)
- Better grades (p < .05)
- Greater recognition that the things the youths were learning in school are important for later life (p < .001)
- Greater recognition that schoolwork is meaningful and important (p < .01)
- Greater participation in community service or volunteer work (p < .001).
|
Studies Measuring Outcome
|
Study 1
|
Study Designs
|
Preexperimental
|
Quality of Research Rating
|
2.6
(0.0-4.0 scale)
|
Outcome 2: Attitudes toward substance use and perceived substance use by peers |
Description of Measures
|
Attitudes toward substance use and perceived substance use by peers were assessed using the Nevada BADA Youth/Parent Questionnaire at pretest; a shorter version called the MAGIC Youth/Parent Questionnaire was used at posttest. The shorter version measures the same constructs as the full questionnaire but is designed to reduce respondent burden. Both instruments are self-report and include multiple-choice questions that measure knowledge about and attitudes toward substance use, parental monitoring, goals, school, and other key areas addressed by the Project MAGIC curriculum.
|
Key Findings
|
Survey responses of youths indicated that from pre- to posttest, Project MAGIC youth participants developed greater awareness that consuming one or two alcoholic drinks per day presents a risk of harm (p < .05). In addition, from pre- to posttest, youth participants reported having fewer peers who had tried alcohol (p < .001), smoked cigarettes (p < .001), used marijuana (p < .001), or used other illegal drugs (p < .01) in the past year.
Survey responses of parents suggested that from pre- to posttest, there was an increase in Project MAGIC youth participants' agreement with each of the following statements:
- It is wrong (for themselves) to drink alcohol (p < .001).
- It is wrong (for themselves) to smoke cigarettes (p < .01).
- It is wrong for their peers to drink alcohol (p < .001).
|
Studies Measuring Outcome
|
Study 1
|
Study Designs
|
Preexperimental
|
Quality of Research Rating
|
2.6
(0.0-4.0 scale)
|
Outcome 3: Parental monitoring |
Description of Measures
|
Parental monitoring was assessed using the Nevada BADA Youth/Parent Questionnaire at pretest; a shorter version called the MAGIC Youth/Parent Questionnaire was used at posttest. The shorter version measures the same constructs as the full questionnaire but is designed to reduce respondent burden. Both instruments are self-report and include multiple-choice questions that measure knowledge about and attitudes toward substance abuse, parental monitoring, goals, school, and other key areas addressed by the Project MAGIC curriculum.
|
Key Findings
|
Survey responses of parents indicated that from pre- to posttest, parents gained greater awareness of where their children are when not at home (p < .05) and greater awareness of their children's friends/associates (p < .05).
|
Studies Measuring Outcome
|
Study 1
|
Study Designs
|
Preexperimental
|
Quality of Research Rating
|
2.6
(0.0-4.0 scale)
|
Outcome 4: Internal locus of control |
Description of Measures
|
Internal locus of control was assessed using the Nowicki-Strickland Locus of Control Scale, a 40-item questionnaire that measures the perception of connection between actions and outcomes. Lower scores indicate higher levels of internal control; for example, a juvenile with a lower score (representing good internal control) would be more likely to report that luck has little to do with getting a good grade in school.
|
Key Findings
|
From pre- to posttest, mean internal locus of control scores for Project MAGIC youth participants improved 14% (from 15.74 to 14.19; p < .01).
|
Studies Measuring Outcome
|
Study 2
|
Study Designs
|
Preexperimental
|
Quality of Research Rating
|
2.6
(0.0-4.0 scale)
|
Outcome 5: Life skills development |
Description of Measures
|
Over the course of 20 program sessions, youth participants completed tasks in the Portfolio Student Workbook which are designed to measure the development of life skills related to conflict resolution, communication skills, cooperation behaviors, aggression, and school performance and involvement. The portfolio tasks were scored by trained facilitators on a continuum from 0 to 100, with 0-15 representing a beginning level of life skills development, 16-45 an emerging level, 46-75 a developing level, and 76-100 a maturing level. A perfect score of 100 would indicate mastery of all skills taught in the project.
|
Key Findings
|
From pre- to posttest, mean life skills scores for Project MAGIC youth participants increased 130% (from 21.87 to 51.15; p < .01), showing a progression from the emerging level to the developing level in the areas of life skills taught in the program.
|
Studies Measuring Outcome
|
Study 2
|
Study Designs
|
Preexperimental
|
Quality of Research Rating
|
2.2
(0.0-4.0 scale)
|
Study Populations
The following populations were identified in the studies reviewed for Quality of
Research.
Study
|
Age
|
Gender
|
Race/Ethnicity
|
Study 1
|
6-12 (Childhood) 13-17 (Adolescent) 26-55 (Adult)
|
73% Male 27% Female
|
88% White 12% Race/ethnicity unspecified
|
Study 2
|
6-12 (Childhood) 13-17 (Adolescent)
|
50% Female 50% Male
|
72% White 17% Hispanic or Latino 7% American Indian or Alaska Native 3% Black or African American 1% Asian
|
Quality of Research Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the Quality of Research for an intervention's
reported results using six criteria:
For more information about these criteria and the meaning of the ratings, see Quality of Research.
Outcome
|
Reliability
of Measures
|
Validity
of Measures
|
Fidelity
|
Missing
Data/Attrition
|
Confounding
Variables
|
Data
Analysis
|
Overall
Rating
|
1: Academic engagement and achievement
|
3.0
|
2.5
|
3.0
|
3.5
|
1.5
|
2.0
|
2.6
|
2: Attitudes toward substance use and perceived substance use by peers
|
3.0
|
2.5
|
3.0
|
3.5
|
1.5
|
2.0
|
2.6
|
3: Parental monitoring
|
3.0
|
2.5
|
3.0
|
3.5
|
1.5
|
2.0
|
2.6
|
4: Internal locus of control
|
3.5
|
3.0
|
2.5
|
2.3
|
1.5
|
3.0
|
2.6
|
5: Life skills development
|
2.3
|
2.3
|
2.5
|
2.3
|
1.5
|
2.5
|
2.2
|
Study Strengths The Nevada BADA Youth/Parent Questionnaire, MAGIC Youth/Parent Questionnaire, and Nowicki-Strickland Locus of Control Scale have acceptable validity and reliability levels. A number of fidelity measures were put into place; for example, checklists of program components were filled out by facilitators and reviewed by the project director for completion, attendance at youth sessions and parent meetings was tracked, and site visits were conducted. One of the two studies had very low attrition and minimal missing data.
Study Weaknesses The Portfolio Student Workbook is not an established measure and lacks reliability and validity information. Both studies are vulnerable to confounds due to their pretest/posttest-only design. Despite the establishment of many fidelity measures, it is unclear how the parents' process was monitored, particularly because it was self-paced and done at home. One study had a high attrition rate for parents; only 37% of the parents completed the posttest evaluation.
|
|
Readiness for Dissemination
Review Date: September 2010
Materials Reviewed
The materials below were reviewed for Readiness for Dissemination. The implementation
point of contact can provide information regarding implementation of the intervention
and the availability of additional, updated, or new materials.
Smith, M., & Evans, W. (2005). Making A Group and Individual Commitment student workbook. Reno: University of Nevada Cooperative Extension.
University of Nevada Cooperative Extension. (2005). Making A Group and Individual Commitment: Give juvenile offenders a chance to change. Curriculum guide. Reno: Author.
Program Web site, http://www.gbcnv.edu/magic
Other program materials:
- Job descriptions
- MAGIC Annual Compliance Monitor Form
- MAGIC Certification Report
- MAGIC: Introduction & Start-Up Information
- MAGIC Reporting Forms
- Quality Assurance Plan--Project MAGIC
- Research Involving Children
- Research Involving Prisoners
- Research Protocol
- Resources for training and support
Readiness for Dissemination Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the intervention's Readiness for Dissemination
using three criteria:
- Availability of implementation materials
- Availability of training and support resources
- Availability of quality assurance procedures
For more information about these criteria and the meaning of the ratings, see Readiness for Dissemination.
Implementation
Materials
|
Training and Support
Resources
|
Quality Assurance
Procedures
|
Overall
Rating
|
3.5
|
2.0
|
3.5
|
3.0
|
Dissemination Strengths Extremely comprehensive, well organized, and easy to understand, the curriculum guide details the educational content and processes for the youth sessions and parent meetings. All materials required for implementation are available electronically on CD-ROM. The curriculum guide comes with supplemental materials to augment facilitator knowledge, pretest/posttest surveys and protocols, and various process and outcome evaluation forms. The tools for measuring participant progress are accompanied by clear scoring guidance and examples to support consistent use across implementers.
Dissemination Weaknesses Implementation planning guidance is vague and general; the materials do not address recommended group size or target audience and do not provide much helpful guidance about facilitator qualifications. No information accompanies the organizational Certification Report to explain how it is used, and it is not clear if the review elements it lists must be verified before the program can be started. Little information is available about training and support. Some quality assurance tools are available only by request or are not accompanied by complete guidance for intended use.
|
|
Costs
The cost information below was provided by the developer. Although this cost information
may have been updated by the developer since the time of review, it may not reflect
the current costs or availability of items (including newly developed or discontinued
items). The implementation point of contact can provide current information and
discuss implementation requirements.
Item Description
|
Cost
|
Required by Developer
|
Curriculum guide and student workbook (includes all quality assurance tools)
|
- Shipping costs only for CD-ROM
- $150 for a set of hard-copy materials
materials
|
Yes
|
Phone support from program staff
|
$50 per hour
|
No
|
3-day, off-site training in Elko, Nevada
|
$500 per participant
|
No
|
3-day, on-site training
|
$400 per participant, plus travel expenses for two trainers
|
No
|
Additional Information The developer will negotiate a discounted rate for trainings with more than 10 participants.
|
Replications
Selected citations are presented below. An asterisk indicates that the document
was reviewed for Quality of Research.
Smith, M. (2000). "MAGIC" (Making a Group and Individual Commitment): A program for entry-level juvenile tribal offenders in Owyhee, Nevada (Fact Sheet 00-31). Reno: University of Nevada Cooperative Extension. Available at http://www.unce.unr.edu/publications/files/cd/2000/fs0031.pdf
Smith, M., Neufeld, J., & Hullinger, D. (1997). National award-winning Project MAGIC: A program for entry-level juvenile offenders (Fact Sheet 97-07). Reno: University of Nevada Cooperative Extension.
Smith, M., & Wallock, R. (1997). National award-winning Project MAGIC: A program for entry-level juvenile offenders in Elko 1997 (Fact Sheet 97-10). Reno: University of Nevada Cooperative Extension. Available at http://www.unce.unr.edu/publications/files/cy/other/fs9710.pdf
Smith, M., & Wallock, R. (1997). National award-winning Project MAGIC: A program for entry-level juvenile offenders in Winnemucca 1997 (Fact Sheet 97-11). Reno: University of Nevada Cooperative Extension. Available at http://www.unce.unr.edu/publications/files/cy/other/fs9711.pdf
Van Poperin, A. L., & Smith, M. (2006). Project MAGIC in Northern Nye and Esmeralda counties (Fact Sheet 06-72). Reno: University of Nevada Cooperative Extension. Available at http://www.unce.unr.edu/publications/files/cy/2006/fs0672.pdf
|
|