Translating Science: A Town Hall on the Challenges

• Welcome and Introduction
• David Chambers
• John Easton
• Patrick Gallagher
• Linda Mellgren
• Questions and Answers
• QUESTION: How do we prepare for the future?
• QUESTION: Do you have any advice for how to take a successful, locally-based program to scale nationally?
• QUESTION: How do we educate elementary school children?

Welcome and Introduction

John H. Laub Good morning. We'd like to get started. Nice to see all of you here this morning. One of the ideas that I'm emphasizing as NIJ as we move forward is the idea of translational criminology. I first learned about translational criminology, translational research really, from my daughter, who is a pediatrician at the University of Pennsylvania. Translational research and medicine has been embraced there full fold. In fact, they have a building for translational research and medicine.

The idea of translational criminology is a simple one, but I believe quite powerful. If we want to prevent, reduce and manage crime, scientific discoveries must be translated into policy and practice. It is no surprise given my interest in translational criminology that it is the theme of this year's annual conference at NIJ: Translational Criminology: Shaping Policy and Practice with Research.

Now the challenge of translating research to the field is not unique to criminology and criminal justice; hence, this plenary session — in fact, town hall panel — where we will look to other federal science agencies for help, as to how to make the translation possible. I'm very excited and pleased at the response from my colleagues and other federal science agencies who are on the stage with me here. I'd like to introduce each in turn and tell you a little about how we're going to format this plenary session.

Beginning with David Chambers to my immediate left, David Chambers is the Associate Director for Dissemination and Implementation Research at the National Institute of Mental Health. He leads the National Institutes of Health initiatives around the coordination of dissemination and implementation research in health. Since 2008, he has been the Chief of the Services Research in Clinical Epidemiology Branch at NIMH. In addition, he manages the portfolio of grants that studied the integration of scientific findings and effective clinical practices in mental health within real-world practice settings. Prior to his arrival at NIMH, Dr. Chambers worked at Oxford University where he studied national efforts to implement evidence-based practices within health care settings. Dr. Chambers holds a doctorate degree in philosophy.

To his left is John Easton, who is the Director of the Institute of Education Sciences, and just told me he's just finished up his second year in that position. Dr. Easton most recently served as the Executive Director of the Consortium on Chicago School Research at the University of Chicago. He holds a doctorate degree in measurement evaluation and statistical analysis from the University of Chicago. He's also the author or co-author of numerous reports, articles and two books: Charting Chicago School Reform: Democratic Localism as a Lever for Change, and Organizing Schools for Improvement: Lessons from Chicago.

To his left is Patrick Gallagher, the Under Secretary of Commerce for Standards and Technology and the Director of the National Institute of Standards and Technology, known as NIST. Prior to becoming director in 2009, Dr. Gallagher was NIST's Deputy Director and the Director of the NIST Center for Neutron Research. He served as the Chair of the Interagency Working Group on Neutron and Light Source Facilities under the White House Office of Science and Technology Policy. Currently he serves as a co-chair of OSTP Standards Subcommittee under the National Science and Technology Council. He holds a doctorate degree in physics from the University of Pittsburgh.

Finally, Linda Mellgren is a senior social science analyst in the Office of the Assistant Secretary for Planning and Evaluation in the Department of Health and Human Services. Her areas of policy and research work include child support, fatherhood, marriage, intersection of human services and criminal justice populations. Currently, she is manning the National Center for Family and Marriage Research on the Evaluation of Family Strengthening Grants for Fathers and their Partners in HHS activities related to the interagency re-entry council. Linda has an MPA from the Maxwell School of Citizenship and Public Affairs at Syracuse University.

What you see here is a wide range of domains of areas where research is being translated to the field of practice and policy: mental health, education, science and technology and health and human services. So, what we're going to do is, I've asked each of the panelists to present five to ten minutes regarding their focus on translational research. After that, what we'd like to do is open up to the audience for questions, comments, observations, and really have a true dialogue, because I think this is a perfect way for NIJ to kick off a very serious effort at translating research to the field of policy and practice. So with that, I'm going to turn things over to David Chambers as the beginning. Thank you.

[End of video clip]

David Chambers

David Chambers Thanks very much and good morning. It's absolutely wonderful to be particularly at this conference because those of us who work on the mental health side know of the tremendous connection and often unfortunate connection between mental health or mental illness and the justice system. In fact, one of the most recent initiatives that we had at NIMH was specifically around trying to get effective interventions for mental illness in the broad criminal justice system sector. I'm very grateful to be here and grateful to engage in dialogue as much as possible with you. That said, I'll speak for just a couple of minutes.

Across all of health, we have a pretty spotty record, unfortunately, of getting effective interventions in practice. Many people will cite the 17-year gap, after which only a fraction of our scientific findings ever make their way to truly impact people's health.

The bottom line from this kind of analysis that says what happens when science is completed and you're waiting for that gap. What we find is that publication alone doesn't work. Too often, science is shared within as we know publications that unfortunately are more likely to sit on the shelf than they are to really benefit people.

The gap really persists across all of health, but we feel it very, very strongly within mental health, where according to large epidemiological surveys, really only 40 to 50 percent with any of our mental disorders get any services. Of those, it's been calculated that only about a third get what would be considered minimally acceptable care. That gap between research and practice is incredibly strong here.

This led us at the National Institute of Mental Health to have a significant focus on dissemination and implantation research, which we really saw as the knowledge base, the translational science that some people would call T2 or now in more common terms T3, T4, T12. Basically, how do you get from effective interventions to actually become standard care?

The challenge of course is multiple-fold. The typical development of practices of interventions is pretty linear. The assumption is you develop your treatment. You then subject it in a very pristine test in an efficacy trial to see if there's a signal. You then expand a little bit into effectiveness and then after that you think about okay, now how do we get this implemented within care. Unfortunately, many of our researchers, with the incredible expertise they have, don't know as much about the variety within actual practice. So they're not as well suited toward trying to figure out how my intervention that I developed and tested can be used as broadly as possible.

There's also few incentives for many of our interventions to be used. So there isn't always the business case that says the profits are going to accrue if one uses a particular treatment. There's also a work force that isn't always as well resourced as we find within trials. Trials often include people that can have their pick of who's going to deliver a particular intervention, but the work force has many, many more constraints on them.

There's also within science really a tremendous focus on internal validity, but not as much on generalizability. Of course, when we're talking about the ultimate uptake of effective interventions, we're really wanting things to be as generalizable as possible.

In 2002, NIMH, the National Institute of Mental Health, issued a specific call for applications around dissemination and implementation research. We saw it as two overlapping, but pretty significant challenges. Dissemination. How do we understand best how to transfer knowledge, knowledge from science into practice, and implantation, which was more of how do you imbed effective interventions within a variety of practice settings?

This first program announcement called for perspective efforts to improve uptake of interventions at the individual, at the organization and even at broader system and policy levels. In 2005, as we were recognizing that again within mental health this was by no means only our problem, but actually extended across health, we engaged in a process through which we brought the NIMH announcement across all of NIH.

We now have 12 or 13 institutes and centers involved in our announcement. We were able to create a centralized review committee. One of the concerns was that people who are engaged in this science, it requires different expertise that may require different designs and as a result may benefit from having particular expertise doing the review of those kinds of applications.

We began annual meetings that were also intended to develop this large community of practice, community of science around people who are engaged in this work. In the first year, we had I think about 300 people who attended. Just in the last few years it's quadrupled, 1,200 in this last year.

We're also benefiting from a centralized reporting system where all of the information accruing from these different trials are gathered and actually reported to the Secretary of Health and Human Services quarterly. I think the value of this certainly is that really keeping our feet to the fire, making sure that as a broad field that we're advancing, not only solving individual problems, but really trying to build a field of knowledge. We're also working to build the capacity for dissemination and implementation research. We have a number of summer training institutes that bring a cohort of investigators on site for a week to really learn as much as possible about the variety of questions and tools in dissemination implementation research.

The current state of the field for us in mental health, and really in health more broadly, we've had a nice shift. For a long time, we were solely focused on identifying barriers and facilitators. Why aren't things working? Why can't we figure out how to get practices that seem to be great off the shelf and into the hands of people who can benefit?

We've nicely moved more into the efficacy kind of work, more toward testing in implementation strategy, assuming that the best case scenario is not necessarily a front-end training and then some technical assistance, but really now trying to look at comparative effectiveness of much more active strategies. It's not easy to get the practices that we know work to become standard care, so thinking much more prospectively and proactively around what are the opportunities to ultimately see that a practice cannot only be implemented, but hopefully sustained over time.

That really gets us to the horizon. This is where I think not only just within the health sector, but across all of our sectors we can do a lot more to try and think about sustainability and really broader scale-up. Even within our studies, when they've been able to randomize in some cases counties, or be able to look across states, we're still not able to make that biggest jump. It ought to be that the scientific findings become beneficial for everybody and we think that it's only through this kind of dialogue and really the opportunity to connect with all of you and to learn from all of you that we're going to get there.

I'm going to stop here, but look forward to the discussion and appreciate your attention. Thank you.

[End of video clip]

John Easton

John Easton Thanks David. That was really interesting. I'm from the Institute of Education Sciences and we, like NIMH, have an intersection with you. We produce far too many kids in our schools who don't succeed and end up in the criminal justice system. We really focus on improving our outcomes and attainment. My agency is the government in the Department of Education's agency for education statistics, assessment, evaluation and research. We're probably best known for The Nation's Report Card, the National Assessment of Educational Progress, but we have many other large, important programs as well, including a very active grant program to education researchers. That's mostly what I'm going to talk about.

Education researchers often talk about the problem of translation in two different ways. The first is how do we take basic or exploratory research and move it in to interventions or programs. The second is more like we just heard about, actually translating interventions that are successful in small, limited settings, into a larger scale.

I'm going to talk mostly about the first problem. I'm going to give an example from the organization that I just came from a couple of years ago, the Consortium on Chicago School Research in a study that was just completed that I initiated before I left. It was a study about students and teachers' perceptions of safety inside their schools. It really did a beautiful job of untangling the many, many factors that influence how kids and teachers feel inside the schools. It looked at student poverty, student race, crime and social capital in the residential neighborhoods, their incoming achievement levels, the levels of perceived peer support for academic engagement, and it looked at how all of these together combined influence how teachers and students perceive their level of safety in the school.

More importantly, besides looking at these kinds of demographic background characteristics, it looked on to what processes within the school building can actually medicate these real external factors and really learn very nicely about adult relationships, how teachers work together and how teachers relate to parents, how much influence it can have in mediating the external.

The question is, the translation is, how do we systematically change practices in schools, specifically around how adults relate to each other and work together in order to ameliorate these external factors that influence school safety? I think it's a wonderful study.

It's really rich data sources, elegant analysis, but now what do we do with it? One answer that a number of researchers in my field are proposing is to conduct the research itself so that it's actually designed to bring in these translation stages. In other words, from the beginning, the research is designed so that it won't need translating.

There's one example of work that a number of people are engaged in called Designed-Based Implementation Research. There are related things called like improvement research, formative interventions, and social design experiments. The point is that in these instances, really trying to build in the translation into the research itself.

The work, no matter which of those types that I refer to, kind of have several core principles. They focus on persistent problems of teaching and learning. The researchers have a very strong commitment to an iterative process. It's very collaborative, particularly collaborative with practitioners or policymakers, depending on the problem.

They have a real concern with developing knowledge and theory through systematic inquiry. They've also got a focus on building capacity with the people that they're working with, whether again they are practitioners or policymakers.

I want to give an example of this type of work that we're supporting at IES, an organization called the Strategic Education Research Partnership. In their words, SERP is the acronym, SERP stimulates innovation and education through sustained collaborations among distinguished researchers, educators and designers. SERP partnerships expand the capacity for continuous improvement while remaining mindful of what teachers do, how schools operate, and how students learn.

One quick example of their work is a program called Word Generation that's intended to build academic vocabulary in middle school students, which is seen as a real barrier for learning science. They don't have the academic language. This is sort of a cross-subject instructional program that was developed with the researchers and the school people to develop and test it collaboratively among researchers and educators.

I want to end saying a little bit more about a couple of specific efforts at IES to address this translation problem or actually I would say to try to avoid the need for the translation, to obviate the problem.

One thing that every opportunity that I have to speak to researchers, which is quite a few — I get asked to talk to universities all across the country — is I really encourage researchers to conduct research that is really relevant and useful to practitioners and policymakers. I use this phrase; I say: we can reduce the research to practice problem by increasing the practice to research link.

When practitioners and policymakers have a stake in planning research, there are really double benefits. It's more likely to be useful to them and they're more likely to take it up because they had a hand in creating it.

I also want to encourage more of the partnerships that I mentioned before that SERP, like my former group, the Consortium at the University of Chicago. We have a research partnership going on in New York City for the New York City Public Schools. The RAND Corporation is heavily involved in Pittsburgh Public Schools.

There's a brand new research partnership between the Michigan Department of Education, the University of Michigan and Michigan State that started from a funded program from our office. We are encouraging these partnerships by putting some incentives into our research request for proposals.

Finally, on a broader scale, IES supports a series of a set of ten regional education laboratories across the country. We're recompeting them right now and we are actually requiring these labs to create what we call research alliances that can bring in multiple stakeholders to define a problem that conducts research that will be usable and picked up there.

I just want to conclude that my point really is that we should be designing our research and conducting it up front so that there's less need for translation and that we can really do this by encouraging more practice-based and collaborative research. Thank you.

[End of video clip]

Patrick Gallagher

Patrick Gallagher It is a pleasure to be here and bask in the reflected glory here of John's Stockholm prize and to join this very important discussion on translation.

I'm going to actually just set the stage for our subsequent discussion by sharing with you just a few thoughts. I'm going to bring a different perspective to this discussion maybe than you've heard so far.

Translational research is something NIST does all the time and it's something I never talk about. It's both the premise of our agency and what we do and it's something that we just don't discuss at all. I wanted to share with you why that's happening.

NIST is the nation's measurement laboratory. That's the good way to think about it. In fact, many of you probably still remember it from its original name, which goes back to 1901 as the National Bureau of Standards. It's one of the nation's oldest science and technology agencies and its mission is basically to define the system of measurement of the United States. So, yes, we define a second and a meter and a kilogram and all the basic units.

What NIST mostly does is take those basic ideas of measurement and turn them into practice. Whether that's weights and measure programs that are operated by the states so that when you pump a gallon of gasoline, you know it's a gallon of gasoline, or whether it's working with regulators so that if you go to get your vehicle checked for emissions, that we know how to measure the greenhouse gasses or the acid rain gasses that are coming off your vehicle, we can measure those accurately. In fact, NIST has had a strong tie in the realm of forensics science, really almost going all the way back to its beginning in 1901.

That shouldn't be a surprise. We're not a criminal science agency, but of course the practice of law enforcement is deeply tied to the ability to measure. This tie goes all the way back to our history. In fact, the most famous case in this probably is actually NIST work or the National Bureau of Standards work on the Charles Lindbergh baby kidnapping and murder, where an NBS scientist by the name of Wilmer Souder was involved in the handwriting analysis and subsequently of the wood analysis that identified and eventually led to the arrest of Bruno Hauptmann. In fact, Souder went on subsequently to help the FBI stand up their crime laboratory in the 1930s.

More recently, NIST has been involved in a whole variety of areas from digital data, digital forensics, nuclear forensics, DNA analysis, law enforcement technologies to protect people, a whole host of areas. It's very natural for NIST to do this because it's imbedded in our mission.

In the context of translation, the way I tend to think of this is the notion of translation make sense if you're looking at the world view from the perspective of being a researcher. You have these ideas and the focus is, "how do I translate this into practice?" I tend to look at it differently. The mission of my agency is putting these measurement technologies into practice, and the programs have to be designed to do that.

Research is a fundamental enabler of that mission. Our programs have to be designed to basically put these methods into practice, one of the things we will need as we research. We often view things as almost the opposite. We're getting a strong practitioner pull, rather than a science push.

A good example of that is our work in DNA. John Butler is in the audience. I don't know where John is sitting. You'll see him. He has "The Butler Did It" on his shirt. One of the things that happened shortly after 9/11 of course with the attack on the World Trade Center was the large volume of very degraded DNA material that was on the site. This challenged the current technology in terms of being able to rapidly look and characterize and generate profiles for small samples, very damaged DNA samples.

John and his team basically went back, developed and modified the underlying methodology to develop these profiles and then didn't just write a science paper. What they did was they worked to disseminate that into the practice. They worked with manufacturers to develop the technology for that new equipment. They developed the testing infrastructure so that this equipment, when it's deployed, can be calibrated and checked and used reliably in the field.

The whole program was designed to drive this new understanding into practice, but the research itself was a key underlying piece. I'm looking forward to this discussion. I think it's going to be great. Again, thanks for the opportunity.

[End of video clip]

Linda Mellgren

Linda Mellgren Good morning. It's a pleasure to be here. One of the problems of being the fourth speaker is that you can think about all those things you wish you had thought to say when you were writing your notes previously. I would've changed my presentation after each one of the speakers, but I'm going to just go with what I've got here.

My office in the Department of Health and Human Services conducts social science research, develops policy and evaluates program initiatives for children, families and individuals. In some ways, we are also a translational research office because we have the dual function of increasing knowledge and then acting on those research findings through the development of policy. We don't usually develop programs, but we do work on the development of policy, that is, legislation, regulation and program models.

Much of the research we do would be considered applied research or research synthesis. We evaluate programs or particular models within programs and we look at the research findings across a number of disciplines to see if there is emerging consensus about relationships that might be amenable to policy interventions. We also fund a number of social science research centers that do basic social science research in the area of poverty and family functioning. We have a National Poverty Research Center, three area poverty centers, and we fund a National Center for Family and Marriage Research.

These centers fund a significant amount of work on the causes and consequences of poverty and the dynamics of family relationships. As you can imagine, they're very multidisciplinary, looking at economics, psychology, sociology, the whole range of disciplines and how the activities of families and the activities of society involve poverty and both positive and disrupted family relationships.

One of the interesting things in being asked to speak today is that for the past number of years, our centers have been looking specifically at how poverty and family relationships affect crime and incarceration and are affected by crime and incarceration. That has become part of the ongoing portfolios of work of those centers.

For example, the National Center for Family and Marriage Research has funded and published papers on the relationship of incarceration to men having children with multiple partners, of adolescent drug use and son's delinquency, and it currently has funded three studies on fatherhood and incarceration, which will be part of a national fatherhood research conference that the center will be sponsoring in 2012.

The four poverty centers have an extensive set of papers on the relationship between poverty and incarceration or crime. They focus on health, on welfare policy, on race, on aspects of employment, transitions into adulthood, child support, and have most recently funded a study on the relationship between incarceration and children's food security.

I would venture to guess that most of the folks in this room have never been to the websites of any of those centers to look at what they are, the kind of research that they're producing that directly affects the work that you do.

For us, the issue is not how to think about incarceration and crime as part of the context in which poverty and family relationships are studied, but how to translate the findings into something that is useful and operationally accessible to policy-makers and program operators in both the criminal justice field and in health and human services. I just want to give an example of one of the projects that we've undertaken in that arena.

About five years ago, Congress appropriated $150 million to fund responsible fatherhood and healthy marriage programs. It's the first such funding that had happened. We were sort of fresh off of a research conference that had happened a couple of years earlier. We had funded the Urban Institute under the leadership of Jeremy Travis, whom some of you may know, to hold a conference on the relationship of incarceration to children, families and communities.

From the findings of that conference, which gathered researchers from all kinds of discipline, we knew — or thought we knew, as is the case in research of a number of things — that incarceration disrupts family relationships; that positive family relationships seem to contribute to successful re-entry; and also that there are interventions that appear to strengthen family relationships. Those interventions had never been tested in the context of work with families where one of the partners is incarcerated.

Using just a small part of that $150 million, HHS published a grant solicitation that encouraged applicants to submit proposals to do marriage and family strengthening programs for incarcerated fathers and their partners in prisons. We also contracted for an evaluation of those programs to see if they actually strengthen family relationships and contribute to more successful outcomes post release. We don't have findings yet of the impact of this evaluation. Those will be coming out in the next couple of years, but we've learned an awful lot about the complications of doing family interventions in the context of a prison environment.

I want to just mention what those are, but first I want to just very, very briefly talk about what the intervention is. The intervention in a family strengthening program is to basically help couples develop relationship skills centering primarily around communication that help them be able to discuss their relationship, discuss problems, discuss issues. They learn skills relating to empathy, skills relating to anger management, things like time out before you speak when you're very distressed. These relationship skills programs have been fairly well tested in environments other than prison environments and have shown to have positive results for up to five years or longer.

That's the intervention. What we've learned that trying to do that intervention within the context of a prison environment is fraught with difficulties. Here are some of the practical issues that have to be addressed in terms of that. One is getting the buy-in from the prison staff, from the warden, all the way down to the security guards. One of the things that these programs mean is that you have people from the outside — that is, visitors — coming into the program and sort of disrupting the way the prisons are usually operated.

You have to find usable space because these programs require that the couples do these interventions together and there is often not very usable space in the context of prison which is accessible to folks from both the inside and the outside.

You have to have trained facilitators that can deliver the services, that understand the nuances of developing these relationship skills. That either means importing people from the outside — that is, external facilitators, which has its set of problems — or training people inside the prison — current prison staff — to deliver those services.

You have to have an appropriate curriculum. I mentioned that these programs have been tested and found successful, but not necessarily with this population. How do you develop or modify curriculums so that the exercises, the skills, the examples that are being used are appropriate for the population that is currently in prison and having a set of stressors that may be substantially different from those of couples on the outside.

Lastly, there's a whole question of the timing of the intervention. If we want to keep couples' relationships together, are these kinds of programs best delivered at the beginning of someone's sentence because that's when those relationships are the strongest and you want to make sure you maintain those relationships over time, or is it best delivered at the end, when the incarcerated individual is being released and will be going back to their partner or their community. But by that time many of these relationships have in fact disintegrated, and so you've lost the opportunity to help maintain relationships that maybe had a chance.

All of these issues have made us think very hard about whether or not this kind of model is even viable within a prison context. We don't even have the results yet and we're already questioning whether or not this is a very good model because we see that the implementation of this may in fact have higher cost to sort of the institutional operation of the prison than most prisons would be willing to absorb.

We're going to be looking at some alternatives. We don't have any plans for evaluations of them, but if there are models which include just doing the relationship education for men only — not with their partners — that is, assuming that the kind of skills that they increase through these programs will have some effect on their ability to maintain positive relationships with their partners.

Another model that we're looking at and in fact was used in some of the sites, but not on a consistent basis, is to sort of do an inside-outside program where the couple are actually taking the same course, but are taking the separately. He's doing the work in the prison and she's doing the work in classes on the outside. Opportunities are provided for them to discuss sort of what they're learning in terms of that.

That's sort of an example of how we're trying to translate research. I think from our perspective there's sort of a global thing we'd like to say is that you have to know the research. That is one of the things that often we find policy agencies or program agencies really don't know the research. To translate it into practice you have to know what the research findings are. You have to be prepared to take advantage of opportunity. The fact that Congress passed this appropriation was not something that we had planned on, but we were ready when it happened, to do something different. Then you have to keep your eyes and ears open to see how the field is changing because the research findings that you might be acting on may in fact be out of date and not current with what the new research findings are.

[End of video clip]

Questions and Answers

Laub What I hear is rather interesting in terms of not only is translational research moving publications to the field, but it's also about thinking differently about how we design research. I think it's about infrastructure. I think I've been accused of having grandiose goals from the nationals to the justice, but I think it would be wonderful if next year at the annual conference we could say we don't talk about translational criminology anymore because we do it, following my colleague, Patrick Gallagher, at NIST.

Can we have the equivalent of a gallon of gas in the criminal justice field? I'm not sure, but it's not a bad goal.

[End of video clip]

QUESTION: How do we prepare for the future?

Chambers I can just start it off and then maybe queue it up for other folks. I think one of the challenges that we have no matter how plentiful or absent resources are, are in assuming that we know exactly what will fit best within different systems. I think other speakers said it better in terms of trying to look first at the situation on the ground and develop interventions, develop solutions, that specifically fit whatever that level is. I know within mental health we have a preponderance of psychotherapeutic interventions that assume that you have a captive audience or an interest for 10 to 15 to 18 to 30 weeks at a time when we know that within practice there may be only one or two sessions that someone may actually be around for.

I think it's resources on all different dimensions. There's the financial resource. There's certainly the resources just in terms of personnel and there's the consistency, the desire for a fit between what we see as a solution with the actual environment. I think part of it really does come from, as was said before, thinking about knowledge creation not absent from the ultimate use of that knowledge, but actually developing solutions that are tailor-made for the settings, given incredible limitations and trying to find efficiencies there too.

Laub John?

Easton I'd just like to mention one thing that we're trying to do to address this. We're using some of our federal resources to actually be more actively involved with federal and local agencies that are making decisions to help them design the rollout of new programs in a way that can be evaluated very well so that we're really looking for efficiency here. So often schools and states roll out things all at once. We're trying to get them to be thinking more about piloting with a real strong evaluation design so that there's much better use of their resources. We think our role is kind of a capacity-building around doing better evaluation and being more analytic and more careful about how programs are rolled out.

Gallagher I think your question about resources is actually not only a result of current budget discussions. I think that when you're talking about the relevance of research and driving it into practice, one of the very common things is that one organization or one entity doesn't own the process from the beginning to the end. What you're really talking about is a mixture between the research activities all the way to the practitioner activities. Even in a perfect world where all budgets were going up, you can still have resource mismatches. In other words, you have to look at that whole portfolio of activities because in the end you're going to be as strong as the weakest link in the chain. There's always a responsibility to look sort of holistically at this whole chain of performance and make sure that all of the parties are participating.

We've seen mismatches in cases when in fact budgets were tight just because somebody didn't understand that they were playing a key role. I think what the budget situations do is they make this more imperative. It's even more important to take a full look at doing this. One thing to keep in mind is these things often get cast as relative priorities once the research is overfunded. That's probably not the most useful way to look at it because it focuses you on one particular budget line. There are reasons where a certain organization has a unique contribution to make. You don't want to be taking that organization away. If an organization was designed to be a top research organization, you don't want to be pulling them away to do less research and try to do more transit.

I think you really have to take sort of a portfolio approach and it's likely that it's much broader than federal questions. It gets right into state, local and non-government organizations and everybody in between.

Mellgren I also think that at the operational level, one of the things that we tend to do is that every time there's sort of a new practice identified, we just add that on top of everything else that's going on, instead of looking at how you redesign what it is folks are doing sort of at the field level to incorporate these new practices. Maybe that means letting go of some of the other things that are being done because they don't seem as effective or as germane. It's one of how do you integrate the findings from new research practices into the totality of what's going on sort of at the operational level, not just keep adding on individual additional things to do until someone just sort of feels like they're totally overwhelmed. There often has to be a redesign, sort of field practices when you want to add in sort of some kind of new best practice.

[End of video clip]

QUESTION: Do you have any advice for how to take a successful, locally-based program to scale nationally?

Gallagher You stumped us. [Laughter] The context of a program like this, I can't offer any direct advice. One of the things NIST does is work very extensively with manufacturing. In the case of a manufacturing organization where we're going to try to translate technology into the market, one of the things you find is that now you get to a small business enterprise at a community. The set of issues they're facing are broader than simply the enabling technology that's there. They're dealing with a whole set of issues: the underlying business conditions, the markets, the workforce issues.

One of the things we found is that the scale-up problem is tied to a much broader set of assets than just one pipeline. You've got to really look at that whole set of conditions that need to be in place for a program to grow. I would enjoy talking with you offline to see, but it ends up being one of these almost community responses, which is one of the reasons in the innovation space so much discussion is around these regional innovation clusters to try to create this ecosystem under which organizations can take the next step and take a great idea, and now take it to full scale in market.

Chambers I do think that researchers — some of the folks whom we work with — would love to have the example of what looks like an innovative program that seems to be working. What we've seen is opportunities to foster partnerships where someone can help go in and help understand as much as possible about why the program seems to be working, and then potentially expand the look at it, so it might be engaging a few other similar sites around towards trying to say what made it work within Baltimore. Might it also work either in a different region of the same city or might it work in other cities?

We do actually have people who are always on the lookout for really exciting new programs. The fallacy is that innovation comes solely from sort of a basic look and then expands outward, but seeing what's going on in on the ground. The opportunity I think maybe along choosing a way to characterize what seems to be success is a first step. Then working with folks who can help see how we make this work in other settings I think is a good start.

[End of video clip]

QUESTION: How do we educate elementary school children?

Easton I'm probably the obvious one to give that a try. I think educators increasingly understand that kids need an early, strong foundation to succeed in school. There are a number of indicator systems used around the country that can predict high school dropout by things like third-grade attendance, by whether kids have really learned to comprehend text at about that same age. There are a number of activities that are really aimed as a greater focus on early childhood preparation these days.

Right now, our department is making the single largest investment that it ever has in a project that we call reading for understanding, which is about comprehension. It's beyond the decoding, beyond being able to read the sentences, but to really understand what they mean. We think that this will lead to potentially powerful interventions. We're spanning the grades from preschool to 12 to develop a kind of a suite, a regimen of interventions.

I think it's really well understood that prevention needs to start very early with very strong educational programs. Kids who are more likely to succeed in school are much more likely to engage and succeed through life.

[Applause]

[End of video clip]