by Marie Cohen

In Home Visiting: More Hype than Hope, I wrote about the failure of decades of studies to prove that home visiting programs have real-life outcomes for children and families. In this post, I explain how the federal government promotes these and other programs as interventions to prevent foster care despite the lack of meaningful evidence for their success. A “Prevention Services Clearinghouse” established by federal legislation approves programs for reimbursement based on the results of outcome studies, with virtually no attention to the quality of the measures used, the lack of corroboration, the total weight of the evidence, or the logic and size of the impacts. There is little evidence to support the effectiveness of these programs in making it possible for children who are maltreated (or at risk of foster care for another reason) to remain safely in their homes. Congress should consider eliminating the Clearinghouse and changing the type of services services eligible for federal funding.

The stated purpose of the Family FIrst Prevention Services Act (FFPSA),¹ which was signed by President Trump in 2018, was to “provide enhanced support to children and families and prevent foster care placements through the provision of mental health and substance abuse prevention and treatment services, in-home parent skill-based programs, and kinship navigator services.” To achieve this purpose, FFPSA expanded the use of Title IV-E funds from foster care and adoption to what it called “foster care prevention services and programs” designed to prevent the removal of children from their homes.² These programs were nothing new; at the state level, they are often called in-home services, family preservation services, or intact family services. Under FFPSA, these services are available to children who are “candidates for foster care” [meaning the child is at imminent risk of entering foster care but can safely remain at home or with a relative with the provision of services provided under the Act³] and to pregnant or parenting foster youth and the parents or kin caregivers of these children.⁴

Under FFPSA, these “foster care prevention services” must be provided “in accordance with practices that meet the requirements for being promising, supported, or well-supported practices.” Among the requirements, the components of the practice must be described in a book or manual, there must be no evidence suggesting the program does more harm than good, and when there are multiple studies of one program, the “overall weight of the evidence” must support the benefits of the practice. Each practice must be found to be “superior to an appropriate comparison practice using conventional standards of statistical significance in terms of demonstrated meaningful improvements in validated measures of important child and parent outcomes, such as mental health, substance abuse, and child safety and well-being.”

To receive the highest rating of “well-supported,” a practice must have data from at least two randomized controlled trials (RCT’s), or quasi-experimental research designs if RCT’s are not available. At least one of the studies must demonstrate that the practice has a sustained effect lasting at least a year. The requirements are somewhat lower to be a “supported” practice and lower still to be a “promising” practice. In order for a program to be rated as “supported” or “well-supported,” the supporting studies must have been conducted in a “usual care or practice” setting, which is defined as an existing service provider that delivers substance abuse, mental health, parenting or kinship navigator services as part of its typical operations.

At least 50 percent of a state’s foster care prevention services funding must be for “well-supported” programs in order to claim federal support under Title IV-E. Thus it is not surprising that states have chosen to include mostly “well-supported practices” in their Title IV-E Prevention Plans.

In order to further define the requirements and apply them to existing programs, FFPSA created the Title IV-E Prevention Services Clearinghouse (“the Clearinghouse” from now on), which is run under contract by Abt Global, formerly Abt Associates. The Clearinghouse has published a Handbook of Standards and Procedures describing the methodology it has developed to evaluate whether a program meets the requirements established by FFPSA.⁵ Program ratings are based on what the handbook calls “contrasts.” A contrast is defined as “a comparison of an eligible intervention condition to an eligible comparison condition on a specific outcome for a specific posttest measurement.” A “contrast” might be a comparison of the number of substantiated child abuse reports for the experimental vs the control group in one study, for example. A contrast must be statistically significant based on conventional standards in order to be considered favorable.⁶

The Clearinghouse staff has interpreted the requirements written by Congress in a way that sets a very low bar for a program to be judged “well-supported,” “supported,” or “promising.” To be “well-supported,” the Clearinghouse requires no more than two contrasts from different studies that show favorable effects.” At least one of the contrasts must demonstrate a “sustained favorable effect” of at least 12 months. This is a low standard for many reasons.

• A program can be rated “well-supported” with as few as two positive contrasts, even if they are wildly outnumbered by contrasts that fail to show an impact. Even within a specific category (such as parental drug abuse, for example), one favorable contrast is enough, no matter how many other related indicators from the same or other studies show no effect.
• Any positive contrast is counted, regardless of whether it is based on objective indicators or subjective ones like self-reports. Improvements on subjective measures are counted even if objective measures fail to find impacts.
• Any pattern of impacts (or contrasts) is accepted, regardless of whether it is predicted by the program’s theory or logic model. If a program is supposed to work by changing parental behavior, but it instead changes children’s behavior directly, the outcome is counted. There is no requirement that a program impact be explained by its logic model or theory.
• There is no requirement that a specific contrast reported by one study be corroborated by another study. Many of the individual impacts that are reported are supported by only one study, with each study cited providing evidence of a different impact.
• Any contrast that is statistically significant is accepted, regardless of whether it is of practical significance. The American Statistical Association has stated that “[S]tatistical significance is not equivalent to scientific, human, or economic significance.” A tiny effect can be statistically significant if the sample size or measurement precision is large enough.
• Studies are counted even when the specific populations studied are very different from those eligible for Title IV-E Prevention services.

Examining the Clearinghouse evidence on some of the most popular programs clearly reveals the flaws of its methodology, providing multiple examples of the problems noted above.

Parents as Teachers is the most popular program rated by the Clearinghouse, having been included by 31 states in their Title IV-E Prevention plans as of August 2024. PAT is a home visiting program that describes its mission as “To promote the optimal early development, learning and health of children by supporting and engaging their parents and caregivers.” The rating of “well-supported” was based on only three studies, one of which was not conducted in a “usual care and practice setting” and another of which was conducted in Switzerland–with a very different population from that of the U.S. The Clearinghouse found three favorable contrasts (out of six total contrasts) on child social functioning, all from one study, along with one unfavorable contrast and two showing no effect. They found two favorable contrasts on child cognitive abilities (one from the Swiss study) compared to 10 findings of no effect). And they found two small favorable contrasts on child welfare administrative reports (with two showing no effect) from a comparison group study that was not done in a “usual care or practice setting,” Only one study reviewed looked at the quality of parenting or the home environment, and it found no favorable effects. The authors of that study said that the results “raised questions about the underlying premise of PAT that focusing services on parents to improve parenting knowledge, attitudes, and behaviors is an effective way to benefit children.” They also concluded that their results (including the three positive contrasts on child social functioning and one on child cognitive functioning) “are consistent with the overall research base for family-focused early childhood programs, which have produced ‘modest and inconsistent effects.'” (ee Appendix A for more detail on all the contrasts discussed here).

Functional Family Therapy (FFT) is second in popularity only to Parents as Teachers, being included in the plans of 25 states as of August, 2024. As described by the Clearinghouse, FFT “aims to address risk and protective factors that impact the adaptive development of 11 to 18 year old youth who have been referred for behavioral or emotional problems.” The Clearinghouse rated FFT as “well-supported” based on the results of six studies. The Clearinghouse reported two favorable contrasts, 23 contrasts showing no effect, and one unfavorable contrast on Child Behavioral and Emotional Functioning–the core goal of the program. It showed two favorable contrasts, two unfavorable contrasts, and 16 contrasts showing no effect on child delinquent behavior. No contrasts showed an effect on positive parenting practices and only two out of 13 contrasts showed an favorable effect on family functioning. Ten contrasts showed a favorable impact on children’s substance abuse, compared to eight findings of no effect. But all of these favorable contrasts came from one study of alcohol-abusing youths who resided in a shelter after running away–a very specific population that may not be generalizable to children abusing other substances or those who are living at home.

Like FFT, Motivational Interviewing (MI) is included by 25 states in their Family First plans.  MI, according to the Clearinghouse, is a” method of counseling clients designed to promote behavior change and improve physiological, psychological, and lifestyle outcomes.” The Clearinghouse reviewed studies of MI focused on illicit substance and alcohol use or abuse among youth and adults, and nicotine or tobacco use among youth under the age of 18. These programs are typically delivered in one to three sessions with each session lasting about 30 to 50 minutes. The Clearinghouse rated MI as “well-supported” based on results from 21 studies. The contrasts reviewed showed no effects on child substance use, caregiver mental health, caregiver criminal behavior, family functioning, parent/caregiver physical health, or economic and housing stability. The only favorable effects were on parent/caregiver substance abuse, for which there were sixteen favorable contrasts, two unfavorable contrasts, and 91 showing no effect. Eleven of the favorable contrasts came from one study of heavy-drinking college students–a very different population from parents of children at risk of foster care placement; presumably few if any were parents at all. It was also not conducted in a “usual care or practic setting.” It is no surprise that a program of one to three sessions would not lead to major changes in parents’ and children’s lives, but it is strange that serious analysts would accept these results as proof that such a minimal program would change lives.

The Clearinghouse rated the Healthy Families America (HFA) home visiting program (included by 22 states in their Title IV-E Prevention Plans) to be “well-supported” based on the results of six studies. Healthy Families America was developed as an intervention to prevent child maltreatment and is now the signature program of an organization called Prevent Child Abuse America, as I discussed in an earlier post. Yet, HFA’s effects on child maltreatment are decidedly underwhelming. The Clearinghouse found five positive contrasts on self-reported measures of maltreatment and no favorable contrasts on 99 more objective measures like CPS reports, hospitalizations and injuries. Of the 16 remaining favorable contrasts for the entire program, five come from improvements in parental reports of child behavioral and emotional functioning (a self-reported item with possible bias), another was on the child’s self reports about skipping school “often,” and none were corroborated by another study.

Some readers might observe that the standard critiques of the clearinghouse focus on the difficulty, not the ease, of meeting its standards. Critics like The Child Welfare League of America have stated that the “high evidentiary standards for the clearinghouse require rigorous, costly research that many states, Tribes and programs are unable to fund. This barrier is particularly onerous for programs that focus on underserved populations, such as Black and Native families, children of more than one race, and rural communities.” This assertion is not inconsistent with my conclusions about the Clearinghouse. That is because there are two types of standards. The standards for the rigor of the studies themselves may be too high for many programs to meet. But once a study meets the requirements to be reviewed, the requirements for being “well-supported” are almost laughable.

Another problem with the Clearinghouse is that many of the programs included are generally paid for by Medicaid, private insurance, or other state and federal programs, such as the federal home visiting program. And Title IV-E is required to be the payer of last resort for these services so it cannot pay if another source is available. The idea was apparently that Family First could be used to supplement these sources for families without private health insurance or where these funds are not available, but this does not seem to be occurring on a large scale, as I explain below.⁷

Finally, the Clearinghouse does not include what may be the most important foster care prevention service of all–case management. All of the services in the clearinghouse depend on relationships, but it is the relationship with the social worker managing the in-home case that may be the most important intervention for a maltreating parent. It is the case manager who refers the client to the parenting, substance abuse and mental health programs prescribed by the Act and who maintains contact with the programs to monitor the parent’s participation and progress. The case manager is responsible, through home visits, for monitoring the safety of the child or children who have not been placed in foster care. Without such monitoring, there can be no foster care prevention services.

Currently, case management is treated as an administrative cost under Title IV-E, which means it cannot be paid for unless the client is receiving other services that are supported by the Clearinghouse. At least one agency, the District of Columbia’s Child and Family Services Agency, has found a way to adapt one of the “well-supported practices” listed in the Clearinghouse as a case management model, allowing it to claim Title IV-E funding for case management for all children receiving foster care prevention services. CFSA adapted the practice of Motivational Interviewing (MI, discussed above), which is in the Clearinghouse as a substance abuse intervention, as a model for case management for all in-home services. As one account puts it, the approval of this use for MI was “particularly notable because while [MI] was approved by the Title IV-E Clearinghouse for Family First reimbursement only as a substance abuse service, DC received approval to implement and claim for it as an integral component of CFSA’s case management practice for all families.” But it should not be necessary for states to go through this type of charade in order to obtain Title IV-E funding for their case management services.

All of the problems mentioned above may explain why states are not drawing down large amounts of Title IV-E funding for foster care prevention programs. The federal government spent only $182 million reimbursing states for Title IV-E prevention services in FY2024, serving only about 18,300 children per month. A recent federal report showed that reported reimbursement claims on Title IV-E prevention services constituted less than two percent of overall Title IV-E program reimbursement claims in FY 2023, serving about two percent of the children receiving Title IV-E funded services.

What can be done?

Raising the standards for “promising,” “supported,” ‘well-supported” to be meaningful is simply not a viable option. Adjusting the three ratings to incorporate the quality of the measures, the need for corroboration, the weight of the evidence and the logic and size of the impacts, would probably mean that few if any programs would be classified in the top tier. So there is not really a way to classify programs based on evidence that will work well for the purpose of funding foster care prevention programs in Title IV-E.

Perhaps this is not surprising. The concept of evidence-based practice was adapted from medicine. When applied to social services, the concept has many limitations. Research conducted under controlled, small-scale, well-funded conditions, even if conducted in a “usual care and practice” setting is often not applicable to the messy, underfunded world of social services practice. It is well-known that effect sizes often shrink when a small pilot program is expanded to cover a larger population. Moreover, a study conducted on one population may not be generalizable to other groups. The population of parents at risk of losing their children to foster care is distinct from many other populations included in the studies mentioned above. The antecedents of child maltreatment, substance abuse and mental illness are extremely complex and may go back for generations. The idea that a three-session, three-month or even a three-year program can eliminate these problems may be unrealistic.

Finally, in relationship-intensive services like all of those included in the Clearinghouse, the characteristics of the practitioner probably matter more more than the specific model. In psychotherapy, multiple studies have concluded that “who your therapist is matters more than the model they use.” Ordinary people dealing with mental illness or addiction do not usually look for an “evidence-based practice.” They look for the best provider they can find–the one they have heard by word-of-mouth, or by scanning the internet for ratings. As mentioned above, the quality of the therapist matters more than the model, and the same applies to program staff. When I was a foster care social worker, and we had a child who had complex mental health needs, we used available funds to reimburse a top-notch provider who did not accept Medicaid. I don’t think we ever talked about what “model” they offered, and they met with clients for much longer than than program manuals prescribe. Medicaid itself does not require proof that a program is evidence-based.

In light of these issues, Congress should consider eliminating the Title IV-E Clearinghouse, as proposed by the Bipartisan Policy Center’s child welfare working group in its recent Blueprint for Child Welfare Financing and Accountability Reform.” The group of 15 child welfare experts representing different professional experiences and perspectives agreed that the clearinghouse process for approving programs as evidence-based “is cumbersome, is idiosyncratic, and impedes states from meeting the needs of families….” The group recommended replacing the clearinghouse with an expert panel convened by the U.S. Department of Health and Human Services that would consider proposals from the states for programs they want to provide using Title IV-E funds. States would be required to submit “evidence to support the efficacy of the interventions, and data that demonstrates the why states believe that proposed interventions would reduce child welfare involvement or improve outcomes of child welfare-involved children and families.” 

Congress should also change the definition of the “prevention” services that can receive federal reimbursement under Title IV-E. Instead of continuing to pay for programs that belong to the field of mental health and substance abuse or are generally funded by Medicaid or private insurance, Title IV-E funds should be used for the critical service provided either directly or under contract by child welfare agencies–case management. Agencies should not have to go through the contortions that DC’s Child and Family Services Agency went through to adapt a three-session substance abuse cessation program into a case management model only to receive reimbursement from Title IV-E. This is the main service that they provide directly and it should be funded. Congress could also add a provision for states to be reimbursed to provide services mental health or drug treatment for the small number of parents who do not have public or private coverage for such services–in other words, where a payer of last resort is needed.

In drafting and passing FFPSA, Congress hoped to make new funds available to programs that would ameliorate the problems that were putting children at risk of being removed from their homes. In its effort to ensure that effective programs were funded, Congress instead created a process that awards ratings to programs based on the mindless and mechanical application of standards that mean little–and that fails to unlock the hoped-for source of funding for foster care prevention. The law must be changed in order to fulfill the intent of its framers.

Notes

1. See Title VII of Public Law 115–123. ↩︎
2. There is something odd about the concept of services to prevent foster care. Prevention usually refers to a social problem like child maltreatment or teen pregnancy, not a government policy, which is itself a response to the problem of child maltreatment. It is like talking about preventing hospitalization rather than preventing illness. It would be more natural to think of services to prevent the underlying problem that result in foster care, which is usually child abuse or neglect, but can also be children’s behavioral health problems that are so severe that parents feel compelled to relinquish their care to the state. But this is the term used by FFPSA and I will use it here. ↩︎
3. A note about foster care candidacy is in order here. In general, children are placed in foster care because they are abused or neglected or are at imminent risk of abuse or neglect. However , there is another group of children entering foster care who have drawn increasing attention. These are children whose parents voluntarily relinquish them because they are unable to care for them at home or obtain needed services–usually care for severe behavioral issues. ↩︎
4. Federal research has estimated that as many as five percent of all children entering foster care between 2017 and 2019 may have entered care primarily to receive behavioral health
   or disability services, not because of maltreatment. ↩︎
5. The Handbook has already been revised once, but the language on this specific issue has not been revised. ↩︎
6. See pages 83-84 for more on statistical significance. This author did not find a statement of the required p-level. ↩︎
7. Also problematic is the failure to include domestic violence services in the clearinghouse. This was always puzzling. Child welfare social workers and academics often speak of the “Big Three” factors that result in child welfare involvement– drug abuse, mental illness and domestic violence. So the omission of domestic violence was strange–especially because domestic violence programs are greatly underfunded and not chargeable to an another program like Medicaid. ↩︎

Appendix

Parents as Teachers Contrasts Cited by Clearinghouse

• Two small favorable contrasts from the matched comparison group study, which was not carried out in a usual care or practice setting and two contrasts showing no effect on child welfare administrative reports;
• One contrast showing no effect on out-of-home placement;
• Three favorable contrasts, two contrasts showing no effect and one unfavorable contrast on child social functioning;
• Two small favorable contrasts and ten contrasts showing no effect on child cognitive functions and abilities;
• Three contrasts showing no effect on child physical development and health;
• One contrast showing no effect on adult parenting practices;
• Eight contrasts showing no effect and one showing an unfavorable effect on family functioning;
• Nine contrasts showing no effect and one showing an unfavorable effect on adult economic and housing stability.

Functional Family Therapy (FFT) Contrasts Cited by Clearinghouse

• Two favorable contrasts, 23 contrasts showing no effect, and one unfavorable contrast on Child Behavioral and Emotional Functioning–the core goal of the program. One of the two favorable contrasts is on “Strengths and Needs Assessment: Child Behavioral/Emotional Needs” immediately after completing the program. But another study shows no impact on the same outcome.
• Two favorable contrasts, two unfavorable contrasts, and 16 contrasts showing no effect on chid delinquent behavior;
• Nine contrasts showing no effect on positive parenting practices;
• Two favorable contrasts and 13 contrasts showing no effect on family functioning.
• Ten contrasts showing a favorable impact compared to only eight showing no effect on children’s substance abuse. But all of these effects came from one study of alcohol-abusing youths who resided in a shelter after running away–a very specific population that may not be generalizable to children who are living at home.

Motivational Interviewing (MI) Contrasts Cited by Clearinghouse

• Thirteen contrasts showing no effects on child substance use;
• Five contrasts showing no effect on caregiver mental health;
• Sixteen contrasts showing a favorable effect, 91 showing no effect, and two showing an unfavorable effect on parent/caregiver substance use. Among the favorable contrasts, one study shows a favorable effect on the number of drinks per week while there is no effect on the number of drinks per day but the Clearinghouse did not apparently take account of such contradictions.
• Seven contrasts showing no effect on parent/caregiver criminal behavior;
• One contrast showing no effect on family functioning;
• Ten contrasts showing no effect on parent-caregiver physical health;
• One contrast showing no effect on economic and housing stability.

Healthy Families America (HFA) Contrasts Cited by Clearinghouse

• Forty-three contrasts showing no effect on child safety as measured by child welfare administrative reports, medical indicators of maltreatment risk, or “maltreatment risk assessment” measures.
• Five favorable contrasts, 38 contrasts showing no impact, and one showing a negative impact on child safety based on parental self-reports of maltreatment. Clearly these self-reports are less valid than more objective measures, since parents clearly know what answers are expected after having been through the program. A parent’s negative answer to the question of whether she ever used physical abuse in the past year (one of the actual indicators used) cannot be trusted to be accurate.
• Seven contrasts showing no effect on child safety as measured by “medical indicators of maltreatment risk,” a strange heading for a group of questions from the Adolescent-Adult Parenting Inventory including “Inappropriate Expectations, Lack of Empathy, and “Belief in Corporal Punishment.”
• Eleven contrasts showing no effect on “Maltreatment Risk Assessment” measures such as hospitalizations and injuries needing medical care.
• Five favorable contrasts and two showing no effect on parent reports of child behavioral and emotional functioning, also a self-reported item;
• Two favorable contrasts, six showing no effect, and one unfavorable contrast on child cognitive functions and abilities.
• One favorable contrast on “child delinquent behavior;” The measure was actually “child skips school often,” self-reported by children in first or second grade, according to the study.
• One favorable contrast and two showing no effect on child educational achievement and attainment. This was a positive contrast on the percentage of children retained in first grade. However, the percentage of children who performed above or below grad level did not change.
• Three favorable contrasts and 24 showing no effect on positive parenting practices. The three favorable contrasts came from observations of “Positive Parenting” from researchers’ observations of parents during a puzzle task, a “delay of gratification” task, and a cleanup task. However observations of harsh parenting during those same tasks did not show a statistically significant change.
• Three contrasts showing a favorable impact and 16 contrasts showing no effect on parent/caregiver mental or emotional health;
• Fifteen contrasts showing no effect on parent/caregiver substance use;
• Three contrasts showing a favorable effect and 28 contrasts showing no effect on family functioning. Twenty-two of the contrasts were measures of intimate partner violence (IPV) or family violencefrom one study only and not corroborated. The positive impacts were on three specific measures based on the perpetrator, the type of violence and the age of the child. Only three of those 30 contrasts showed a positive impact and no effect sizes were provided.
• Five contrasts showing no effect on economic and housing stability.