Evaluation on a Dime

You and your transitional living program staff work yourselves to the bone. You know in your gut that your effort makes a big difference to your clients. Many of them are getting jobs, graduating from high school, moving out on their own. But you don't have quantitative proof of your program's positive effects.

"Evaluation is the ethical thing to do." says David Pollio, an associate professor at the George Warren Brown School of Social Work at Washington University in St. Louis, who studies outcomes for runaway and homeless youth: "If you don't do evaluation, you don't know if what you're doing works."

The following tips were compiled from conversations with Pollio and with Jamie Van Leeuwen, director of development and public affairs at Urban Peak, a youth-serving agency in Denver, Colorado.

• Don't jump off the deep end. Don't begin with a long-term, fancy study. Start off small.


• Know what you're evaluating and why. When you're collecting data, stop and ask yourself why you're collecting it. Form focus groups-one composed of management, another of staff, and a third of youth-to answer questions about what kind of data will be useful and what purpose it will serve.


• Surveys don't have to be complicated. Staff can work together to write simple surveys. Van Leeuwen, who is working on a doctorate in public policy, regularly polls runaway and homeless youth in Urban Peak's care. The results help the agency's staff to pinpoint areas that need improvement. For instance, youth complained that staff often were too busy to talk, so Urban Peak is looking at its case managers' loads to find ways to reduce the time they spend on less important tasks. The agency also is recruiting more mentors, so kids always have someone to talk to.


• Involve youth. At Urban Peak, youth help draft surveys and suggest additions. Van Leeuwen also pilot tests the surveys on two or three young people. "The more you can involve them, the better," he says. "The experts on homeless and runaway youth are your homeless and runaway youth."


• Build evaluation into daily tasks. You probably already interview clients at intake and discharge; simply standardize those processes, and you have an evaluation tool.


• Collaborate, collaborate, collaborate. "It's almost impossible for an agency to design a sensitive evaluation without collaboration," Pollio says. Because most agencies don't have staff with expertise in evaluating programs, conducting surveys, and compiling statistics, youth service organizations can enlist the help of professors from local colleges or universities. College students can also be great collaborators, working as unpaid interns who gather and process data.


• Be prepared for what you might find out. Open yourself up to constructive feedback. Distinguish between the kid who is having a bad day and the kid who says his or her case manager doesn't listen.


• Use what you learn. "If it's going to sit in your desk, it's not worth collecting the data," Van Leeuwen says. Managers should meet with staff after an evaluation and come up with an action plan for improving programs and processes over the next 6 months.


• Take advantage of the Runaway and Homeless Youth Management Information System (RHYMIS). Organizations that receive FYSB grants are required to report on their programs using the Bureau's management information tool, but programs can also use RHYMIS to track youth whose care is paid for by non-FYSB grants. In addition, even programs not receiving FYSB funding can track their progress using RHYMIS and its support hotline. For more information, go to http://extranet.acf.hhs.gov/rhymis.