Get Funded: Developing a Better Proposal

Bernard Auchter: So our plan today is Alan and I, we represent two different offices within NIJ, and we will review some, what we consider to be, very basic application information and processing information and review how we do reviews.

And then we will be followed by Cherie Crawford Watson, who is our human subjects officer, and she will get into all those details and nuances that are critical to getting it right in regard to human subjects and privacy issues.

And Cherie will be followed by Angela Wade, and Angela is an expert in the realm of the budget, and she will get into the details about the importance of all the various budget detailed information that's required in applications.

And Jolene will conclude. Jolene is our director of the Office of Communications, and she will conclude with a variety of things related to publication and dissemination that you will find interesting.

And as I said throughout our presentations, please come to the mic if you have a question or a comment.

So Alan and I will get underway here. First — and I think I started to get into this — we will have a brief overview, give you just a general sense of NIJ and where we fit within the greater scheme of things, then get into some areas of interest and the solicitation process and applying and then the budget issues. We might have one or two slides on budget, but Angela will have, certainly, all the detail and some hints that we consider to be very important in writing a competitive application, too, and then finally the peer review process.

So what does NIJ do, I think you're probably well aware of much of this, and that may be one reason why you're here, science, R&D, evaluation and testing, but much of our work is aimed and I have found over the many years that I've been at NIJ, more and more we're geared towards the “so what” question, so what will this research provide to the field, how will we do things differently, how will we do things better. That's the “what works” part of this.

So, for background, organizational background, this is, of course, the department, all the various litigation sections, and then we narrow down to the Office of Justice Programs and the National Institute of Justice within that. The other offices on the same line, we refer to as often as our sister agencies, and frequently some of our solicitations come from them, the Office of Juvenile Justice and Delinquency Prevention or the Bureau of Justice Assistance, wants one of their major programs evaluated. So they'll provide funds to us to issue a solicitation for an evaluation; same with the Office for Victims of Crime. Other offices on the other lines are various support offices.

So, within NIJ, this is actually new within the past year, this revised organizational chart, but it gives, I think, a good sense of the various offices, major offices.

Alan is in the Office of Investigative and Forensic Sciences. Office of Operations focuses on the various processing of grants, et cetera, and Office of Communications, Jolene's office. Science and Technology gets into more of the hard science issues, bulletproof vests and standards for various police equipment, and then the office I'm in, Office of Research and Evaluation, which has three major divisions, we are the soft sciences, referred to.

So the research areas of interest, Alan and I will jump in and out here.

Alan Spanbauer: As Bernie said, I'm from the Office of Investigative and Forensic Sciences, better known as OIFS, so we just go ahead and call us “OIFs” and that will —

[Laughter.]

Spanbauer: The majority of our programs, a big majority of our programs are formula type driven programs, the DNA backlog reduction programs, but underneath that, we also do research, R&D in DNA forensics, research training, looking for training that's needed out in the community, also research in general forensics in which new technologies are emerging in the forensic realm, also developing cold care units and missing persons units. So we have quite a wide background of solicitations that we do put out.

In the Office of Science and Technology, they're looking at less lethal technologies, body armor, and then we go into some of the other areas, into our Office of Research and Evaluation or ORE, which deals with the crime mapping, prison rape, tribal crime and justice, and violence against women.

And I don't know. Bernie, did you want to say anything about those?

Auchter: Let's see, on Violence Against Women. The Violence Against Women program is one, certainly a major one within our Office of Research and Evaluation, and it's been with us for 15 years. We have dedicated funds. It's one of the very few programs on the social sciences side that has dedicated funds each year, and I have come to see it as an excellent model for Congress, if they knew that this is one way. If you know very little about a given area, one good way is to provide ongoing funds that are dedicated to that topic, and as a result, 15 years later now, we have over 200 different studies that have been done and with approximately $80 million. So I'll leave it at that.

Spanbauer: In fiscal year 2009 — that's a little typo up there — in 2009, we made 508 awards for approximately a total of $284 million, and those awards were made from 38 different solicitations. In 2010, as an increase in solicitations, we put out 61 solicitations this year, so quite an increase over previous years.

Some of the areas of support in fiscal year 2009 is my program, the Coverdell Forensic Science Improvement Grants Program, the DNA backlog reduction and solving cold cases with crime, cold cases, missing persons. As you can see, there's just a wide variety through the Office of Research and Evaluation, crime and justice research, violence against women, as Bernie has mentioned, elder abuse, and these are just a few of the different topics that we have seen.

Auchter: Alan, maybe you can comment on this. One of the things I wanted to point out was — and I think this relates to some of our program areas, too. We are actually, even though we're both within NIJ, we come under different legislation. Our office on the social sciences side comes under the original Omnibus Crime Control and Safe Streets Act. Your office, I believe, comes under the Department of Homeland Security legislation.

Spanbauer: In some instances, yes.

Auchter: And, as a result, some of the work that you're involved in is very much field oriented for programs, whether it be on DNA related backlog or cold cases.

Spanbauer: Yes.

Auchter: So, moving into the — any questions up until this point?

[No response.]

Auchter: No?

Moving into the solicitation process, in general, we identify gaps, and how do we do that? Well, Alan's office and the Office of Science and Technology have what's called “technical working groups.” So they meet, what, several times a year, and they have here recommendations from those who work in the field in regard to what are the next critical studies and work that needs to be done.

On the social sciences side of things, we're moving in that direction also towards working groups; however, in the past, we've utilized — we've brought together workshops of experts, and it's almost always researchers and practitioners jointly in the same room discussing what some of the gaps are.

But also we have on occasion congressionally mandated studies that come out of different legislation, and, in fact, the entire Violence Against Women Act came out of a National Academy of Sciences report that was done over 15 years ago on violence against women, but there are other mandated studies that might be small studies. Recently, in the office, we're working on one that's related to hate crime. That was from some legislation last year.

Yes.

Attendee: There's a lot of seats in the front.

Auchter: I'm sorry?

Attendee: There's a lot of seats in the front.

Auchter: Oh, OK. Come on up front, everybody. Plenty of seats up here.

And so, in general, we have both targeted and general solicitations: targeted, often when we have good ideas to what is the next step or what do we need to focus on next; and general ones, like on the social sciences side called our “crime and justice research announcement,” which is investigator initiated or field initiated.

And, generally, the pattern is early fall and into the winter, most of our solicitations are released, and the goal is to have them out for at least 90 days, recognizing, though, that I know people who are responding would prefer even longer periods of time, and it does take us at least six months to get through the entire process, to the point that everyone is notified, yes, you did get an award or, no, you didn't.

Spanbauer: One thing that's of real interest for people is to go to the NIJ website and look at the funding area and at archived funding opportunities. That way, you can see what was posted in 2008, 2007, and see if something is there that might pique your interest, and that way, you could be prepared for when that solicitation does — if it does open in the next fiscal year.

Auchter: So we have — this is just an example of what you would see on our website when you go to the solicitation section that Alan mentioned. This is an example of one that was up this year.

And throughout this presentation, we'll sometimes say, you know, make sure you read the solicitation carefully because all of the details are right in there in terms of what is required for an eligible response.

So, at the top of this first page, you get things in summary, too, what the program is about and contact information, who the individuals are working on this particular solicitation, and we would encourage you to call us, especially if this is the first time you're doing this. Call us and discuss the idea that you have, do you think that fits within what's being requested or whatever your question might be. We normally don't accept detailed concept papers, but we'll speak with you at any length that's needed in regard to your ideas or we might give you some reactions about, based on our experience with the whole process and many years of peer review, what some peer reviewers frequently do mention. So some of that conversation can be very useful to you, so we encourage you to call us.

Spanbauer: In OIFS, we've coined a phrase, “RTFS,” which is read the solicitation.

[Laughter.]

Spanbauer: And so that's what we tend to send to people, is be sure to read the solicitation. They're not that long. They're less than 20 pages. So that's one of our biggest hints for people.

Auchter: And so this is at our website on funding, and as Alan was saying, on the left hand column, here is a — you can go to current funding opportunities or all the way down at the bottom if you want to go to some of the expired ones that are up there as well; in fact, they go back many years — or several years.

You actually can find out about our announcements through grants.gov as well as at our own website, and grants.gov, I believe was meant to be the original site for all federal funding. And you can go to the website there and subscribe to receive e-mail notifications. You can select different categories or different agencies, and then when a new solicitation is out, you'll get an e-mail notifying you of that, and it makes the process much simpler.

And our own NIJ funding page and the larger, our larger umbrella Office of Justice Programs funding page, they all are sources of information on solicitations.

Spanbauer: One thing, Bernie, we may want to mention is that not all solicitations do go through grants.gov. So a good — what I tell a lot of my applicants is that access NIJ's funding page at least once a month because there are updates regularly and also to go to the OJP funding page and just have that as a tickler on your Outlook calendar to go in and look at what current funding opportunities are available.

Jolene Hernon: You can also subscribe. On the funding page — can you go back a second?

Spanbauer: Here.

Hernon: You can subscribe up here to get it. Now, every time we issue a solicitation, you can get a little e-mail if you subscribe to this page.

Spanbauer: OK.

Audience Member 1: I subscribe to that page, and I think I do so wherever available OJP. I get the notification, but it says “we released a solicitation.” It doesn't say what solicitation. So I go to funding list. I have to — there's so many, I don't know which ones.

Auchter: So now we'll move into the application process, and, again, just starting an application, this is at our website, but then, as Alan said, the other one is at grants.gov, so two possibilities.

Spanbauer: Typically, NIJ applications require that there's a Standard Form 424, which is the fact sheet in which we're looking at what you're applying for. A lot of grantees tend to miss what they're asking for in the total. So be sure when you're going over that 424 form that you're putting in your federal amount that's requested. That seems to be a problem for some folks.

Typically, an abstract is required, which is 4 to 600 words. We do do some basic minimum requirement review. We won't kick you out for 601 words, but be aware that it should be short and concise.

The program narrative is required in which you're looking at a research question or problem or the statement of the problem; the goals and objectives of the research, the design and the methods, and implications for knowledge and practice. We always look at the dissemination plan for the project deliverables. We also encourage our peer reviewers to look at those dissemination plans very critically, look at a description of the estimated cost, your staffing and management plan. One of the most important things we like to look at is the timeline of the research, that you're putting in a correct timeline and that it's doable, also your data archiving strategy, which Bernie will touch upon, and any tables, figures and charts that will add to your application.

Auchter: Actually, I'll comment on both of those last two items.

The data archiving strategy is something that has been enhanced in the recent, I'd say, two years, and that is not only — we're looking for a detailed paragraph or two or three on exactly what your data archiving strategy will be.

Many of you may have seen we do have the folks from the University of Michigan out here. They have a booth for the Criminal Justice Data Arching Program, and the intent, of course, is that the data, especially that data from a very useful study, can be analyzed in many different ways in a number of different projects. We have a specifics annual solicitation that's called the Data Resources Program that is a solicitation for research to utilize existing data.

Just a comment on the idea of tables, figures, charts, I have seen review panels and as well as staff in our own office can find an explanatory chart or graph. So, well, a picture, again, is worth a thousand words, and even though you may have something in narrative about it, just charting something — I've seen a literature review charted as well as having a narrative, and it just brought out the key elements of the literature very clearly. So I could not overemphasize the need for the occasional table or chart that is really focused on what you're trying to present.

And in terms of application forms, the required forms are stated here. The first one, we often call the “face sheet,” SF 424, and, of course, Angela will get into the budget worksheet and summary and budget narrative. All are critical, and, in fact, I know that without one of those, applications have been denied just because — or haven't even gone into peer review because they don't have one of those elements, and then the various forms that are required, too, certification regarding lobbying and drug free workplace, et cetera.

And, if funded, then there are a variety of other forms, and Cherie will get into some of the human subjects forms and privacy certificate issues.

And I don't know if you wanted to emphasize anything here?

Spanbauer: Just one thing that Bernie said, when we say these things are required, they go through what are called “basic minimum requirements,” and because it is a competitive process, in the economic times it's becoming more and more competitive, and so we have to be rigorous in our review. And if a budget narrative is not there, although that doesn't sound too fair — “oh, I missed something” — it doesn't meet the basic minimum requirements, and it won't go to peer review. So be sure — in the back of the solicitation, there's a checklist — that you go through that checklist and make sure that you have all the components.

Auchter: And this slide just is an attempt to put the cycle on paper here, the entire annual cycle that we go through with solicitations and drafting on the right center there. We post it and go through program review and then peer review. We have both external and internal peer review, and then we brief the director who is the final decision maker on all grants. And the acting director, currently Kris Rose, she makes the actual funding decision, and an award then goes into being processed, as we are now also beginning to plan for next fiscal year, so the process goes on.

And I will leave the human subjects issues to Cherie, so we won't touch on those.

But after an award is made, there are financial reporting requirements quarterly, and perhaps Angela will touch on those, and semiannual progress reports are required just to note progress in your research plan and whether or not you've come across any roadblocks and delays and how have you dealt with them and the like, and then a final progress report as well as a final research report.

Informed consent, Cherie will touch on.

You also have a draft final reports, depending —

Spanbauer: Depending on the program.

Auchter: — on the program.

Spanbauer: Yes.

Auchter: Yes. So the draft technical report then is required 90 days before the end of a grant and the data as well in code book. The draft, then, we have reviewed both internally and externally. Normally, we'll have one researcher and one practitioner review the draft report, similar to articles that are being reviewed in a journal. We then provide comments back to the principal investigators. They make revisions and then provide us with a final copy and a letter discussing how they used the comments from the reviewers.

So, at the every end, we have an executive summary on a project and an abstract as well as the full final report that we then archive at the National Criminal Justice Reference Service where anyone in the world can read your great work.

And you have an invention report, if applicable?

Spanbauer: That's true.

Auchter: OST and others?

Spanbauer: Yes, yes.

And also, touching on the semiannual progress reports, over in OIFS, we get a lot of data calls from Congress, and we take those data metrics that are submitted, and we have a person that's dedicated to compiling the metrics. And so that way, we can look at what a backlog is, how many samples have been analyzed. So those semiannual and final progress reports are of utmost importance to us, so that we can answer questions from Congress and then receive continual funding.

We'll talk just briefly about some of the budget categories that are present in the solicitations. You can hire personnel or use monies for overtime and pay fringe benefits. Travel to training is always acceptable. Some equipment, depending on the program, it will be listed in the solicitation what type of equipment can be purchased, what supplies. You can also bring in contracts. You can bring in trainers. You can bring in consultants to do work for you. There's an “other” category that catches a lot of things; for example, any training, all the registration fees go into the “other” category, just as an example. And then, if you have an indirect cost rate, you need to provide your indirect cost rate proof from your federal cognizant agency, which states your indirect cost rate.

Auchter: And just some useful websites here on the budget side, and perhaps Angela will also have these in her slides. So there's an OJP financial guide as well as NIJ guidelines for submitting applications and things within NIJ that we say funds cannot be used for. Now, this varies a little bit. So Alan will have a comment after I say that you can't use funds for this. He'll probably say you can.

[Laughter.]

Auchter: But, on the social sciences side, provision of training or direct service, given that we have limited funds for research and evaluation, the bulk of our funds are to focus on paying for the researchers and the analysis and the report writing, and, therefore, even if it's an evaluation of a program, the funds cannot be used for administering and doing the program, but it can be used, of course, for the evaluation component.

Also, purchasing of equipment is usually not — other than, say, when you make an argument for, oh, we really do need, given the field work we have to do, a laptop here and the university somehow doesn't have enough of them, so we're going to budget for two of them, something like that. And proposals to fund a program, of course, as I think I just said, funds cannot be used for that.

Spanbauer: Just a little touch on that, just depending on the solicitation, mainly our formula type awards, we do allow for the payment of training for the analyst to go out to receive training. Also, through those, through the DNA backlog reduction and through the Coverdell programs, we do allow people to buy laboratory equipment. So it just really goes to, again, the RTFS statement and reading the solicitation and seeing what's allowable.

Auchter: And we'll try to move a little more quickly. Writing a competitive — some hints that we have for writing a good proposal, identifying, of course, the priority area, and call us and discuss it with us.

What are your research questions? It is interesting that we do get proposals that don't even identify a hypothesis or a research question. You may find that shocking, but that happens, and the theories that underlie the question, I think it's important to demonstrate what theories you're basing your research on and key constructs, your plan for collection, that you're going to do surveys or you're going to do observational study with various site visits in detail, and that's the real crux of the matter, I think. And you'll see that when we come to criteria, we do provide ratings, and it's the methods and design of the project that are critical and are allocated the most number of points in regard to the overall proposal.

Any comments here, Alan? No?

Human subjects and privacy issues, Cherie will get into.

A dissemination strategy, what is your plan? I mean, we do publications, and Jolene will talk about, but what do you want to do with — what do you see the results of your work leading to and who will it be useful to and how will you get it out to them. More and more, we are encouraging people to look to publish in a practitioner journal to get the word out on your results.

Also, making sure that you have memos of understanding and letters of support, it is noticed when those letters of support are rather perfunctory and just a paragraph on “yes, this researcher may have access to the data in our office” kind of thing versus one that suggests the more collaborative arrangement where you've been working with them. They may have even helped you design the study to some extent. So letters of support can be quite different, and I'd encourage you to be more collaborative if you're looking to do a project with some practitioners.

Again, here I'd like to emphasize the idea of graphs, charts, logic models. If you're proposing an evaluation, the logic model is critical; you know, what is the logic behind this program that you want to examine?

Any comments?

[No response.]

Auchter: Other helpful hints. The “so what” question is the question that often comes up when we're meeting with our director. So what will the results of this research provide, and what will we do differently now that we have the answer to this question? How might it impact on policy or practice? Always keep that in mind frequently. I mean, we suggest it in solicitations that you address that, the policy implications, and that's done in a variety of ways. Some people just put a paragraph in; others put two pages in about how this might be useful to practice.

Timeline in a management plan is critical, and I think Alan touched on that, too. Budget will be discussed in more detail and, again, data archiving strategy, so these are essentially reviewing some things we already touched on.

And Alan has a few more emphasis statements here.

Spanbauer: Right. We'd like you to look at that solicitation and just have a really good grasp for the goals and the requirements of that solicitation, and if you do have questions, call the program manager and say, “You know, I really don't get it. What are you getting at? What am I trying to do here?”

Think of yourself as a peer reviewer. How are you meeting the goals of the solicitation? Think of it that way. Think of it critically. If you have somebody else in another department, give them the application to read next to the solicitation and the selection criteria. Having another set of eyes is always good to do it.

Be clear, concise and comprehensive. We don't need a lot of fluff. I got 200 applications this year, and, you know, you really need to get to the point and be concise and give the data and move forward with your application.

State your project's deliverables and how you will achieve them. That's one of the most important goals. Again, think of it, look at that selection criteria, and think of yourself as a peer reviewer, am I answering the question and how am I going to meet the goal.

And think about the questions if you were reviewing the proposal and the budget. Is the budget really applicable, and does it meet what you're trying to do?

Auchter: OK. And, quickly, we'll go through the peer review process. Panels can be either in person or conference call. The rest of the ones that we will be doing this year are going to be by conference call.

Consensus in a panel is sought but not required. Certainly, those that have the consensus rise, often rise to the top.

And in reviewing applications, peer reviewers are instructed to look at the criteria stated in the solicitation for their review, and every application is reviewed. And it's reviewed not in a comparative sense. Reviewers are instructed to review applications with the criteria laid out in the solicitation as the measure.

The entire process — and this was mentioned before — takes about six months. External both technical and practitioner reviewers are on panels, and then we have internal reviews as well and office briefings and then a final decision meeting with the director.

This is simply a statement of what is in most solicitations, these major elements, the criteria, everything from understanding the problem to the technical merit, which is the heart of the matter. We do have a section, Frequently Asked Questions, at our website, and I think it's very useful. You might want to check that out.

And, again, call us or the panel manager named in the solicitation and discuss your idea.

Just a few questions that people often have had in sessions like this, so what's the average length of most proposals, and this might vary, depending on our different offices. Our proposals now go from — are usually two to three years or between a year and a half to three years.

Average award, we used to say $250,000, but now it's somewhat more than that, probably more like 350 to 450 as an average.

Is qualitative research supported? Yes, it is, even though I think if you look at most of our work, it's probably heavily weighted to quantitative work. This is on the social sciences side.

And how many applications get funded? Right now I think it's in the neighborhood of about 10 percent. So, if we get 20 applications, two, at most, three might be rewarded.

Can you apply under multiple RFPs? Yes, if it's a different proposal, but if you submit the same proposal to two different applications because they happen to be able to fit under two different solicitations, we will ensure that it gets one review that year, if it's the same application, same proposal.

Can you revise and resubmit? Yes, you definitely can the next year.

Unless there are some questions, we'll move on to the human subjects issues.

Cherie?

Cheryl Crawford Watson: To receive funding, all applications need to have on file two things: a human subjects protection form and a privacy certificate.

I will take questions as I go along, but some of the questions that I'm sure you have right now will be answered in the coming slide. So let me go through a few first, and, hopefully, the answers will become clear.

You want to just back up?

Alan Spanbauer: Oh, sorry.

Watson: That one, yeah.

Spanbauer: Sorry.

Watson: So you need the form and the privacy certificate. Not all projects will need IRB approval or the approval of an institutional review board. If institutional review board approval is required, you'll also need to have the IRB approved consent forms and the IRB approved instruments.

The second item you need is the privacy certificate, and it needs to be properly completed. All of the sections, if they're applicable, it needs to be signed, and it needs to be dated. It needs to have the proper signatures. You'll hear me say that over and over because I often get them unsigned and undated, so they're important forms. Have them signed and dated.

Here are the laws. I'm going to go quickly through this. All of the information is on our website, all the statutes, all of the regulations, DOJP instruction, the forms, helpful hints about how to fill out the forms, the regulations themselves section by section and a section on frequently asked questions. So it's all there. There's even a new item which I'll tell you about in a few slides, but I want you all to go look at them because it will help you determine whether these regulations apply to your project.

State laws may also apply. What state laws? Well, for those of you that do child abuse report, child abuse reporting or any kind of research in that area, there may be mandatory reporting laws, and I'll speak to that in a little bit in the next slide.

How do you determine if these regulations apply and ask what are the funds that you want from us being used for? The Office of Justice Programs — and these regulations apply to NIJ and all of our sister agencies. Is it for research? And I'll provide that in a few minutes. Is it for evaluation? Is it just for training? Is it to put on a conference? Is it to disseminate publications? Is it to build a widget or some type of technology, or is it for program development? Some of these things will not be covered by the regulations, but some of them will. We'll get into that in a little bit, so you can ask that question.

They only apply to research. OK. Well, you're all researchers. You all have a definition of “research.” Your definition doesn't matter. It's the definition that's in the regulation, and here are the regulations. In a nutshell, it's to advance the state of knowledge in a particular area or to develop or to contribute the generalizable knowledge.

This is the new item that's on the OJP website, and it's linked also on the NIJ website. And it's a decision tree that will help you look at what you're proposing to do. It will ask you questions, and it will say yes/no, yes/no, yes/no, and it will tell you whether or not you need to fill out these forms and get IRB approval.

So, if the activities are being conducted solely for the purpose of performing measures or counting the number of kids at a midnight basketball program or counting the number of cases that were processed by an office, the human subject protection regulations may not apply unless, of course, you're doing some type of evaluated research where the definition works. This decision tree is new. As I said, it's on both websites, and it's very specific. So, hopefully, it will help you, you know, come to a better understanding of whether the regs apply to your particular project.

Next slide.

This is the most frequently asked question I get either by e-mail or, “Oh, these don't apply, how do I get out of this, this is just a big hassle.” Sorry. It's not a big hassle. You have to comply with it. You may be a successful applicant. You may think that once you get the award letter that your budget has been approved in Angela's office and everything is hunky dory. You got the award letter, and the next day, you're going to get the check. It doesn't happen that way if you're conducting research with human subjects or if you are collecting identifiable information: name, address, Social Security number, DNA. Anything identifiable, these regulations will apply, and you won't get your money. And it will be withheld until you or your agency have been compliant.

Next slide.

These are the basic rules of thumb. Use the decision tree. Is it research? The rules apply. If it's not research — and, again, it's not that simple. Some of you may have projects where it's really hard to tell. You may not have human subjects, but you may have identifiable data. You may do a secondary data analysis. In some cases, the regs will apply; in some cases, they won't. Ask. Ask the solicitation manager. Again, read the solicitation. Down at the bottom of the first page will be a person who you can call or e-mail or contact the human subject protection officer. That's me. My e-mail address was on the first slide. It will be again on the last slide. Don't call me because I don't — I will respond. I will say, “Hi, Why are you calling?” And you'll say, “I have this question,” and I'll say, “E-mail me.” Why? Because I like to respond in an e-mail. You'll then have it to keep. I'll then have it for my records, and believe it or not, I'll probably get 50 of the same question. So I do have some standard responses that I can just hit the button and send out, and then you'll have a record of it [indistinguishable].

Human subject protection, this is the regulation. All research involving human subjects — and that's with federal agency funds — must be reviewed and approved by an institutional review board before the money can go out.

Let's go again. What is research? Another definition, what is a human subject? It's a living individual about whom you, the principal investigator, may obtain data either from direct interaction with them or identifiable data, so, if you're collecting anything or using secondary data.

There's a human subjects protection form. Those of you having been through this — and I see some people nodding, some people kind of giving me a dirty look and saying, “Oh, you're the one — there's a one page form. It's easy. If your project doesn't involve human subjects and only secondary data analysis, go to Box 8, “This project will not involve human subjects,” sign it, date it, send it in, attach it to your application. If yes, then there are boxes that have to do with the federal wide assurance number of your agency and your institutional review board and when or whether they may do your project. It just gives a little more detail on that, so look at that form. Again, the form is in our website.

The privacy certificate. We'll now jump to privacy. It's a different regulation, 28 CFR Part 22. All applicants for DOJ funds must submit the privacy certificate. It must describe the research. The very first box says brief description of project. Does anybody want to guess each year how many awards we fund and the project description is not there? Or, it will say see the narrative in my application. It's a stand-alone document. Got it? It's just, you know, five steps, but make sure they're specific, not “I will save the world.” It's how you will save the world in five sentences, so that it's easy to understand and to see where you're going.

You need to assure the Department of Justice that you will comply with all the requirements, and you'll describe how you will keep the data that you're collecting confidential and how you will inform the subjects.

Next slide.

How you get to whether or not you need to complete the entire privacy certificate. Just answer some other questions. Does your project include information that's identifiable to a private person? So either name, other personal identified. In some cases, with some mapping research, it may be that an address becomes identifiable, but a three block area, maybe not, or a citywide area, maybe, maybe not. Again, it depends on your hypothesis.

What is a private person? A private person is different than a human subject. A private person could be an individual, an association. It can be a department store chain, if that's what you're conducting research about. It can be a neighborhood association. It can be a number of things. So it's not just one living individual.

The elements of the privacy certificate, again, describe it. The reset of the elements need to be completed if, indeed, you're collecting identifiable information. So how you plan to notify the subjects, informed consent, how you plan to maintain the identifiable data; whether you use password protected laptops or whether you have the interview files locked in a secure file cabinet in a locked office; who needs access to that data, whether or not you have need for an information transfer agreement, if you're transferring information between two universities, say two universities are being funded; how you will dispose of the data and how you will archive it.

There's one exception to confidentiality, and that's immediate threats to self or others. That's the only line you can have in your informed consent for subjects that will be approved.

And the slide.

What about mandatory reporting? [Inaudible.]

The Department of Justice confidentiality statute supersedes all state law. So any research funds from the Department of Justice can only be used for research or statistical purposes and no other purpose without consent of the person from whom it's collected. So researchers who are using Department of Justice funds, if they want to report child abuse or need to, because they're in a state that has mandatory reporting laws, they will need to obtain two consents, one from the subject to participate in the research and a second separate consent form to allow the reporting should you suspect it or have knowledge, come into knowledge about it.

These are some resources, again, all related to the NIJ Web page. There's the human subject protection form, the privacy certificate and the decision tree. You'll find all of those on there as well as some links also to some other websites. I think that's on the next one.

Yes. The Office for Human Research Protection website, the top one there, is with the Department of Health and Human Services. There are three vulnerable populations in research: pregnant women, children and prisoners. If any of you are doing any research with any of those populations, you should go to the OHRP website. They have some great guidance on prisoner research. They have some educational materials. They have more detailed decision trees for those vulnerable populations. They have other links as well to how other federal agencies deal with some of these issues. So that's a good way for you to become knowledgeable with all the other types of research that is going on within HHS.

If you have more questions, again, send an e-mail. I am happy to answer any questions. If you have any questions now, I'm happy to answer them. If some of you are current grantees and you have a specific question about your grant, I'm happy to talk with you offline afterwards about that.

Thank you.

Angela Wade: Unlike my colleagues here, I do not work for NIJ. I work in the Office of the Chief Financial Officer. Now, as Bernie mentioned at the beginning, the Office of the Chief Financial Officer is one of the support offices in OJP. So anything financial dealing with your grant will come through OCFO.

Now, OCFO actually handles grant applications. We process and approve your budgets. If you have grant adjustment notices, we handle those as well. Also, if you've ever had a financial monitoring visit, that person would come out of our office.

Now, I'm going to be doing a brief review of the budget. So what does a good budget look like? I'm going to start with telling you what OCFO looks at when we're reviewing budgets, some of the things you need to do in order to get your budget cleared and the top 10 reasons why budgets do not clear.

Now, what are we looking for? One of the first things we look for when you submit your budget, we look to see if you submitted a budget detail worksheet and a budget narrative. Now, your budget detail worksheet lists the cost of all your budget items. It should be detailed. We want to see all your calculations for each budget item.

Along with your budget detail is a budget narrative. Now, Bernie and Alan talked about narratives earlier, about program narratives. A budget narrative is totally separate from your program narrative. A lot of people will submit their budgets and just submit the budget detail and wont' submit the narrative thinking, well, I've already submitted a program narrative. No, you must submit a budget narrative.

Now, your budget narrative actually describes in detail everything that's on your budget. It shows how you calculated the cost on your budget and explains why it is useful to your program.

Now, along with your detail worksheet and narrative, you should also include a budget summary. The budget summary actually totals each budget category from your budget detail. It also shows the federal amount of funds that you're actually requesting, the total cost of your project, and if your grant program requires you to provide match, you want to distinguish how much is that project cost match.

Audience Member 1: Is it possible — this is a small thing, but [inaudible] to put the match requirement on the front page of the solicitation because — it can be a decision — it helps us make a decision as to whether or not we can afford to apply.

Wade: OK. Bernie just informed me that that normally doesn't apply to NIJ grants.

Audience Member 1: [Inaudible.]

Wade: OK, but, yes. I'll take that comment back. OK.

Along with your budget summary, we look to see if your federal budget that you're requesting matches what you're being awarded. Like, for example, say that your award is for $200,000, then you should have a federal budget for $200,000.

We also check to see if your costs are reasonable and necessary. If you normally buy laptop computers — I always like to use laptops for an example. If you buy laptop computers that cost $2,500 normally for your organization, don't request $8,000 for laptop computers for your grant program. That will be considered unreasonable.

We also like to know that what you're requesting is actually necessary to your program — is it supporting your project activities?

And, also, along with these items, there are several nonbudget items that we consider when we are processing your budget. If you have outstanding quarterly financial reports, if you have not submitted your progress reports, and if you're required to submit a single audit report and you have not submitted it, we will not clear your budget. So keep that in mind as well. If you have other grants, if you have not completed those items, it will affect your current grant applications.

Now, what do you need to do to get your budget cleared? Some of the things you want to consider when you're submitting your budget or preparing your budget, you want to make sure it's complete, that all your costs are allowable, and it's cost effective. When I say complete, again, we can't say it enough. Please submit a budget detail worksheet, a budget narrative and a budget summary.

Now, we don't have a specific form that we require you to use, although we have one we will provide for you, but as long as you include all the budget categories, your total costs, et cetera, that is acceptable.

Another thing we like and you need to double check, make sure you show all your calculations. That's very important. If you think back to back when you were in school and you're taking a math test, your teacher will always tell you unless you show your work, you will not get credit. The same thing goes for your budget. If we can't see how you came up with your numbers, we will not approve your budget.

And, again, the budget narrative, your budget narrative should explain every single item that you have on your detail. We need to know where everything is coming from. So say you're hiring a program director to manage your grant program, we want to know what they're going to actually be doing for your grant program, how much time they're going to be spending running your grant program, et cetera.

And, again, we want to see your budget summary. Your budget summary, again, will show the cost breakdown of all your budget categories.

Now, we have eight budget categories that we use in OJP for your grants: personnel, fringe benefits, travel, equipment, supplies, contractual, other and indirect. The way we see it, you don't necessarily have to use all these budget categories, but you should be able to fit your costs into at least one of them, one way or the other, and please do not throw everything into “other.”

And what we're looking for with the budget categories, if you're using personnel, we want to see the position and title of the person that you're hiring. If you already know the person, we would like their name as well, how much time are they spending on a project, how much are you paying them, if the project goes over more than one year, are you going to provide a cost of living increase for them. You want to include that in your budget as well.

Fringe benefits. Are you paying them health insurance, workmen's compensation, retirement, et cetera? We want to see that as well. We want to see the calculations, how you're coming up with the number. And your fringe benefits should tie back to only the persons that you have listed under your personnel category.

Also, one thing that we want to state that's unallowable, we do not want to see you trying to pay bonuses out to your personnel under fringe benefits. That does not get paid with your grant funds.

Travel. If your grant program requires you to travel, we want to see the computations for that. Now, if you have your own travel policy in place, then feel free to use that. Otherwise, you are required to use the federal travel policy. And when you're showing the calculations for travel, show where you're going, why you're going there. We want to see a breakdown between your transportation and your per diem as well.

If you are required to provide a match for your grant program, we want to see that as part of your budget as well, how are you providing that match. Some grant programs might require you to provide the source of your match.

If you have any allowable costs in your budget, that would be returned as well. And like Alan and Bernie mentioned earlier, please read your solicitation. Read your solicitation. You can also check the OJP financial guide. There's a whole chapter on allowable costs and unallowable costs.

Again, we want to see a breakdown by categories. This will be your budget summary. What is the total of each budget category you are using? What is the total project cost?

Again, a complete budget narrative. We want to see everything that's on your budget detail worksheet explained in your budget narrative.

If you're a new grantee for non profit and commercial, you are required to submit an accounting system and financial capability questionnaire form. This one thing will hold up your budget process. I've seen it. Last budget season, we had at least half of our new grantees did not return these forms in, and it will hold you up from beginning your grant program.

If you have a grant program and you're applying for a supplemental award and your budget did not clear from your initial award, that will hold you up from receiving your supplemental.

And, again, if you have indirect cost rate, please provide your indirect cost rate agreement.

And if you submitted your budget and it has not cleared and the analyst handling your budget has returned it to you for follow up information, please provide clear responses back to them. If you're unsure what they're looking for, just give them a call or e-mail them. They'll be glad to help you.

And, also, I have copies of a sample budget. If anybody wants to take a look at, please free to ask me.

Jolene Hernon: OK. So only 10 percent of all the applications we get are ever funded. It's very competitive, and then of those that are funded, only a small percentage do we actually get in my department, the Communications Department, and publish it through the Department of Justice.

So you've just spent two or three years doing research. You know everything about your data. You know how many of this and how many of that, and you know how this analysis was done, and you had a whole team of people involved, but, really, what you have to do now if NIJ decides to publish your work is you have to distill it to the bottom line because the chief of police, he doesn't really care how big your N was, and he doesn't really care what your methods were. He wants to know, ”How's it going to help me catch those bad guys? How's it going to help me prosecute more cases? How's it going to help me clear the backlog in my labs? How's it going to help me do my job better?” So you really have to think about your audience, who cares, why they care, and then you have to break it down into bite sized pieces.

Usually, your research is complicated and intertwined, and it can be pretty messy, but when you are talking to a Member of Congress who is dealing with health care issues, education issues, transportation, bridges falling down, he needs it in the simplest, most straightforward manner possible. So break your research into small pieces.

I have a little example of how sometimes this helps.

What do I do? Push this one here? Which one?

Bernard Auchter: To the cursor. Yeah, yeah.

Hernon: This is a mind map. It's sort of like what your research looks like. You've got this little bit piece up here about what happened in Michigan, and you've got this piece down here about the young girls. Then you got a section over here about boys, and you got a section over here about corrections. So you have to see how they're all interconnected here and get down to the heart of the matter and tell one little piece of the story.

Here's another example. There's a whole bunch of mind maps. Google “mind maps” and you'll see a whole bunch of them. But I like to show this example because I think it's a good way of trying to organize all the pieces and all the messages you have for many different audiences. It's possible that you might have one message for corrections officials and a different message for prosecutors and a different message for the victim advocates. So your messages can be different for different audiences, and we're happy at NIJ to publish different things for different audiences.

So this slide is an example of several of the different kinds of options we have for distribution of your research. We have Web and print publishing. Those are kind of the hardcore things. We've got a very routinized way of doing print, and we're getting more and more routinized about how we do our website.

We have Web topic pages that synthesize collections of research of topics. We have the NIJ Journal; that's our magazine. There's copies of it here at the conference. We do a lot of training material. We write chapter books. I mean, this is a perfect example. If you have — we did one recently on drug courts, and there was a chapter for the drug court administrator, there was a chapter for the judge, there was a chapter for the treatment provider and so forth. So there were different messages for different audiences, and we did that one in a chapter book.

More and more — I think Bernie said this — we are partnering with associations to get into their magazines and newsletters because many of the practitioners will trust their associations, their policing associations or their governors' association. They read those magazines more than they might read a federal government publication. So, if we get into their products, we get the message targeted to the right people faster.

We do a series, a number of different kinds of reports, special reports, research and brief reports, research for practice and so forth.

We do webinars with researchers and practitioners talking to each other. We have a very nice partnership with Harvard Innovators Network in which researchers, practitioners and policymakers discuss the issues and the points of view that the research can help each one of those policymakers or practitioners.

And “Research for the Real World Seminars,” Kris Rose this morning in her opening marks talked about Ed Latessa's “Research for the Real World Seminar” recently. They're very popular. Go check them out on our website.

Conferences and networking, huge for us. We have a lot of workshops and meetings and technical working groups and conferences like this one.

We are getting better and better at reaching bloggers. Again, a lot of people will get into a niche market. Cops do this a lot. They really trust other cops. So blogging cops are where we want to be. We want to have your research in those blogs.

Press and Capitol Hill is not handled by NIJ's Communications Division. It's handled by the Office of Justice Programs, Public Affairs, but we do — this administration is very open to open to having researchers come up and talk about the science and how it can improve policy and practice. I guess it was last week, Kris Rose and someone from our staff, Kathy Browning and Mark Nelson, went up to the Hill to brief them on the Making Sense of the DNA Backlog story.

So briefing Capitol Hill is really a great way for us to get your research to the policymakers who are making decisions about money and policies, and, of course, we love to be in the press all the time.

Last week — was it last week? — there was a death investigation study, a death investigation workshop out in Arizona looking at trying to develop standardized ways of investigating death cases. I guess it's not just homicides. It's any kind of death, right? Yeah.

For the first time ever, coroners and medical examiners were brought together in the same room to talk about how they do business. They do business differently in different states, and they do it differently if you're a coroner and if you're a medical examiner. So it's a big issue in the field, and NIJ brought them all together.

So why am I telling you this? Because “Frontline” was there, and there will be a “Frontline” story on death investigations. NPR is going to be doing a series on that, too. So, getting in the press, we love the press.

Steps to getting published. So now here's the nuts and bolts. At NIJ, we have an editorial board, and your program manager goes before the editorial board and makes a pitch on why NIJ should spend time and money publishing your findings. The board vets that. There's pretty hard hitting questions, and we want to make sure we're doing the right thing and we're getting the message shaped in the right way and we're doing the right kind of product for the right kind of audience.

The editorial board then makes a recommendation to NIJ's director, and the director approves or disapproves, makes comments.

If the idea is approved, we start working on a manuscript or training. If it's a training product, it might be, you know, a video scripts or something like that, but the manuscript, getting the manuscript approved, many of you maybe know this, is the hardest part. It goes around and around and around many times until we get it really, really perfect. You might have to go through three, six drafts, something like that, before it actually gets to NIJ, and then once it gets to NIJ, there's a number of reviews that happen there as well, and then it comes back to you for more reviews and revisions.

And then the manuscript goes through a review process. NIJ has a science advisor who looks at everything for the director, and then the director looks at it. And then it's approved, and there usually are some comments to it, and then it comes back to you for more revisions. But, eventually, you have a manuscript that's final and approved and ready to go into what we call a “laser.” It used to be called “galley pages” in the old days. And getting the laser approved is a lot easier because the words are already OK. So now we put it into a layout, and it looks really great, and it makes a big difference whether you use color and how your charts look when it's in laser.

Once the laser is approved, again, through the NIJ director, it then has a very routinized process for clearing through the department. It goes to the Assistant Attorney General, and you don't really even want to know all the steps that it goes through to get cleared, but, finally, it's released and you're published at the Department of Justice.

Here's a couple of ideas. A lot of researchers have a problem talking to a layperson. So I just have a couple of ideas out here to help you kind of get outside the academic thing and think about how to make your ideas stick. This book by Chip and Dan Heath is really good. How to Get Your Point Across in 30 Seconds by Milo Frank is pretty old fashioned. I think it was written in late '60s, but it's still a classic. A Whole New Mind by Daniel Pink is about — oh, you know, it's about the mind map and how different parts of your mind get different ideas and how you can use them in different ways, and, of course, Malcolm Gladwell's Tipping Points is a classic. Everybody should be reading that.

The question was about publishing software, have we ever published software. I'm feeling like some of our training work has been software based. Yeah. Crime Stat, oh, that's right. That's a mapping software. That was funded by the Department of Justice, right.

Also, we're doing a telephone app for — the death investigation guide is being updated, and instead of doing just a print version of it, we're putting it on a telephone, you know, on an app, so that's kind of cool.

So, OK.

I think we're done. Thank you.