The federal Title IV-E Prevention Services Clearinghouse recently approved Homebuilders, a well-known family preservation program, for Title IV-E funding, giving it the highest rating of “well-supported.” This decision is sure to be lauded by many child welfare administrators looking for more program choices, including those in the fourteen states where it is currently being used as of 2018. Unfortunately, the Clearinghouse decision does not appear to be justified by the research it cites. Of the two studies cited as the basis of the rating, one worked to reunify rather than preserve families; the other study concluded that Homebuilders was not effective in preserving families.

As many readers know, the Family First Prevention Services Act expanded the use of Title IV-E funds, which were formerly used only for foster care, to pay for evidence-based practices to prevent the placement of children in foster care. “In-Home Parent Skill-based” services were one of the three groups of services authorized, and Homebuilders has been frequently cited as a likely member of this category. To be approved for funding, each practice must be approved by the Prevention Services Clearinghouse, which was also created by the Act, with a rating of “promising,” “supported” or “well-supported.” The Act establishes criteria for meeting each of these standards.

In order to be rated as “well-supported,” a practice must be shown to be superior to an “appropriate comparison practice using conventional standards of statistical significance” as demonstrated by improvements in “important child and parent outcomes, such as mental health, substance abuse and child safety and well-being.” This must be established by at least two studies that were determined by an independent review to be “well-designed and well-executed,” used random assignment or a quasi-experimental design, and “were carried out in a usual care or practice setting,” and at least one of which established a sustained effect lasting at least a year.

Homebuilders is the best-known family preservation program. Developed in 1974 by the Institute for Family Development, it provides “intensive, in-home crisis intervention, counseling, and life-skills education for families who have children at imminent risk of placement in state-funded care.” Its goal is to “prevent…unnecessary out-of-home placement … through intensive, on-site intervention, and to teach families new problem-solving skills to prevent future crises.”

In the 1980s and early 1990s, a movement in support of Homebuilders and other Intensive Family Preservation Services (IFPS) spread throughout the child welfare world, spearheaded by wealthy foundations and advocacy groups, as described by Richard Gelles in his influential text, The Book of David: How Preserving Families Can Cost Children’s Lives. This movement resulted in a rapid expansion of these programs around the country, culminating in federal legislation allocating $1 billion to these programs nationwide. However, a major study authorized by Congress and conducted between 1994 and 2002 cast doubt on the effectiveness of these programs at keeping children safe and reducing foster care placements. In recent years, that study seems to have been forgotten. Indeed it is common for supporters to express the belief that “research shows Homebuilders has been well-supported for decades,” as one advocate told the Chronicle of Social Change.

Therefore there were no expressions of surprise or consternation that the clearinghouse gave Homebuilders its highest rating as a “well-supported practice.” In its narrative, the Clearinghouse explains that Homebuilders meets the criteria laid out in the Family First Act for that rating. Specifically, “at least two studies with non-overlapping samples carried out in usual care or practice settings achieved a rating of moderate or high on design and execution and demonstrated favorable effects in a target outcome domain. At least one of the studies demonstrated a sustained favorable effect of at least 12 months beyond the end of treatment on at least one target outcome.”

The clearinghouse reviewed 17 studies for possible relevance to  the Homebuilders program. Of these studies, only three were determined eligible for inclusion as evidence of Homebuilders’ effectiveness. The others were ruled out because they were done before 1990, were not relevant, or did not meet basic quality standards. Of those three studies, two were determined to meet the clearinghouse design for “moderate” or “high” support of the causal evidence, and both met the “moderate” rather than the “high” standard. So  the Clearinghouse based its recommendation on two studies only–the minimum required for Clearinghouse approval–both of which the lowest acceptable standard for support of the evidence.

For each of these two studies, the Clearinghouse separated out each individual effect found at each site and date, resulting in separate listings for the same effect at different follow-up times and sites if the project had multiple sites. At the end of this process, the Clearinghouse cited 10 instances of “no effect” on child safety across the two programs, confirming what was already well-known. For child permanency, there were seven favorable outcomes, two unfavorable outcomes, and 13 findings of no effect. For adult well-being, they found one favorable effect, 14 instances of no effect, and no unfavorable effects. Despite the preponderance of findings that Homebuilders had no effect, the eight  favorable outcomes were enough to give Homebuilders the coveted rating of well-supported.”

The aggregate data is already underwhelming but it becomes worse when considering that seven of the eight favorable effects came from one study, which should not have been included at all in the review. That study, described by Elaine Walton and others in reports published between 1993 and 1998, assessed an intensive family reunification program provided in Utah. The study involved 110 families divided between the program and control groups. There were many things that made this program a strange prototype for Homebuilders. First, the program was aimed at reunification of children in foster care with their families, not the prevention of foster care placement. While Homebuilders can be used for reunification as well as for family preservation, it is known predominantly as a family preservation program.  Similarly, while Title IV-E funds can be used for family reunification programs as well as family preservation (if a state chooses to define children existing foster care as “foster care candidates”) Family First has been described by its supporters almost exclusively as an initiative to prevent the placement of children in foster care.

A program that is successful in family reunification may not be successful for family preservation. A family facing the possible removal of a child is in a very different situation from a family with a child already in foster care. Children in the Walton study had been in out-of-home placements from one to 88 months, with an average of one year. Most of the parents had presumably already participated in court-ordered services such as therapy, drug treatment and parenting classes.

Further undermining the relevance of the Walton study is that it was not a study of the Homebuilders program. The model described by Walton et al, called Family Reunification Services (FRS) by the authors,  departed from Homebuilders in many respects. Services were less intensive and longer in duration. Workers spent an average of about three hours per week in direct contact with the families, and this contact could last up to 90 days. This is a very different model from Homebuilders, which typically provides at least 40 hours of face-to-face services or about seven to ten hours a week, over a period of only four to six weeks.  It is hard to understand how the Clearinghouse could use this study of a non-Homebuilders reunification program to affirm the success of Homebuilders in general.  Yet, the Clearinghouse drew six or seven of its eight or nine favorable results from this study.[^1]

The second study cited by the Clearinghouse was conducted by three well-known research firms, Westat, Chapin Hall, and James Bell & Associates. This was a congressionally mandated evaluation that was intended to overcome shortcomings of previous studies. It included three family preservation program sites using the Homebuilders model, although the Clearinghouse cites only the studies from New Jersey (343 families) and Kentucky (442 families).  The researchers studied one family reunification program in New York, but the Clearinghouse did not review that portion of the study.

In reviewing the Homebuilders family preservation program sites in the three states,  the Westat researchers found no impact on child safety or foster care placement. The researchers concluded that their results were consistent with other studies that “have failed to produce evidence that family preservation programs with varying approaches to service have placement prevention effects or have more than minimal benefits in improved family or child functioning.”

Not surprisingly, the Clearinghouse found only one “favorable effect” from the Westat study. That effect was not on child safety or permanency but on adult well-being. They found a favorable impact on adult (not child) receipt of WIC program benefits at the Kentucky site immediately after program participation–not surprisingly as one would hope the Homebuilders caseworker helped families sign up for WIC. This is a very weak hook upon which to hang a “well-supported” rating.

The authors of the Westat study suggest that “The extent to which the intensive, short-term, crisis approach fits the needs of child welfare clients should be reexamined. The lives of these families are often full of difficulties—externally imposed and internally generated—such that their problems are better characterized as chronic, rather than 24 crisis. Short-term, intensive services may be useful for families with chronic difficulties, but those services are unlikely to solve, or make much of a dent in the underlying problems. Of course, the hope is family preservation programs will be able to connect families with on-going services to treat more chronic problems. But, that appears to happen far less than needed.”

In sum, the Clearinghouse based its rating on Homebuilders on two studies of two different programs. One of the programs did not follow the Homebuilders model and worked only to reunify families already separated by foster care. The second was a study of Homebuilders family preservation programs and according to its authors failed to demonstrate any favorable program impacts. The clearinghouse found only one favorable effect from this study, and it pertained to adult well-being. It is hard to believe that any reasonable person would conclude that these two studies together provide “strong support” that Homebuilders is effective in meeting the goals of Family First.

The flaws in the Clearinghouse approach to Homebuilders raise issues that are broader than the effectiveness of this one program. While the Act’s criteria for approval of a program are often described as “rigorous,” the Homebuilders result show that they are anything but that. Requiring only two studies to show favorable results regardless of the number of studies that show no impact; reporting that two studies have found favorable effects even though the one relevant study had only one minor favorable effect that its authors did not mention in their conclusion; allowing the use of data from an evaluation of one program to support the effectiveness of another program; rating a program as “well-supported” without specifying the specific outcomes that it has been shown to achieve — all of these connote a lack of rigor. Regarding the last point in particular, if a program is found to work for only one goal (such as family reunification and not family preservation) the Clearinghouse should approve it only for that goal.

The initial wave of Homebuilders expansion was spurred by an onslaught of non-scientific “evaluations” funded by foundations intent on demonstrating its effectiveness, as Richard Gelles describes in his book cited above. Sadly, the same type of advocacy-based analysis was used to support the passage of Family First.  Supporters of the Act repeatedly stated that we know what works to preserve families and we just need to fund it.  Yet Child Welfare Monitor has found few or no programs with strong evidence of large favorable effects. It is likely that other practices approved by the Clearinghouse have equally skimpy support.

This post was updated on April 16, 2020 after the Clearinghouse responded to Child Welfare Monitor’s question, submitted on April 2, requesting an explanation on apparent internal inconsistencies in its table about favorable program effects for Homebuilders.