Feds confuse substantiation with victimization

On January 28, the Administration of Children and Families (ACF) released its annual report on child maltreatment. In its press release, the agency heralded “a decline in the number of victims who suffered maltreatment for the second consecutive year.” There are three problems with this. First, the alleged decrease in victimization between Federal Fiscal Years (FFY) 2016 and 2017 is so small as to be insignificant. Second, what declined was not child maltreatment but rather the number of children who were “substantiated” as maltreated–a decline that may reflect changing state practice rather than declining child maltreatment. Finally, ACF’s presentation appears designed to support a narrative that favors family preservation over child safety rather than to report the data in an objective manner.

The newest edition of ACF’s Child Maltreatment report is based on state data for FFY 2017, which ran from October 2016 to September 2017. The report shows that states received 4.1 million referrals (calls to child abuse hotlines) alleging maltreatment involving 7.5 million children in 2017. The number of referrals as a percentage of the number of children has increased annually since 2013. ACF does not discuss the reasons for this ongoing increase, nor does it present referral numbers by state, but such increases could stem from increased awareness of child abuse and neglect (often due to highly-publicized child deaths), public information campaigns, or other factors.

Of the 4.1 million referrals received nationwide in 2017, 2.4 million  (or 58%), were “screened in” by state or county child welfare agencies, which means they met agency criteria for receiving a response. As a result, 3.5 million children received either a traditional child maltreatment investigation or were assigned to an alternative non-investigative track, as described below. And of these 3.5 million children, an estimated 674,000 or 19% were found to be victims of abuse or neglect. This flowchart, based on data from Child Maltreatment 2017, illustrates this funneling effect from referrals to substantiation.After rounding and a calculation to account for missing data from Puerto Rico in FFY 2016, HHS estimates that the number of children found to be maltreated decreased of 3,000 (or 0.4%) from the previous year. Such a small change is hardly meaningful; it would be more accurate to say that the number was basically unchanged. This difference from one year to the next is so small that the rate of children found to be victimized was the same in 2017 as in 2016–9.1 per 1000 children. In other words, almost one percent of all children were found to be the victims of maltreatment in 2016 and 2017.

But perhaps more important than the small size of the decrease is the fact that referring to a “decline in the number of victims” or the “victimization rate”  is deceptive, which is why I have used cumbersome terms like “found to be victims of child maltreatment.” Most states use the term “substantiation” to connote that they have concluded maltreatment have occurred; some have an additional finding called “indication” that is somewhat less conclusive than substantiation. But as we all know, a finding of maltreatment is not the same as actual maltreatment. Just look back at my columns on Jordan Belliveau in Florida, Anthony Avalos in California, the Hart children in Oregon, and Adrian Jones in Kansas to find cases where horrific abuse occurred but was not substantiated until a child died.

And that is not the only problem. As ACF itself explained, changes in state policy and practice can influence the number of reports that are substantiated. Different states have different evidence thresholds to substantiate an allegation. According to the report, 37 states require a “preponderance of” evidence, 8 states require “credible” evidence, 6 states require “reasonable” evidence and one requires “probable cause.” One state changed its evidence threshold between 2016 and 2017.

In addition to different criteria for substantiation, some states treat all screened-in referrals in the same way while others have a two-track system of responding to reports. In these two-track systems (often called “differential response” or “alternative response”), some allegations receive a standard investigation, but others (usually deemed to be at lower risk of harm) receive less rigorous response, often known as a “family assessment.” The children in these cases are not determined to be victims even if they have been abused or neglected. Instead their families are offered voluntary services. About half of states reported data on children in alternative response programs. As the above flow chart shows, 639,634 children received an alternative response, almost as many as the 674,000 who were determined to be maltreated.

So a given state’s substantiation rate will be influenced by whether it has differential response in all or part of the state. And if the use of differential response in a state was expanding or contracting over a given period, this will influence the change in the number of  children who are determined to be victims of maltreatment. Specifically, if states increased their use of differential response overall, that would have reduced the number of children found to be maltreated.

And indeed, ACF reports that “states’ commentaries suggest the increased usage and implementation of alternative response programs ….may have contributed to the changes noted in the 2017 metrics.” And upon review,  the commentaries, included in an Appendix to the report, do suggest that the number of reports subject to differential response increased between FFY 2016 and 2017. Six states, including New York, and Texas (two of the four states with the highest number of children)1 were ramping up their use of differential response during FFY 2017, while Massachusetts and Oregon stopped using the two-track system in FFY 2016 and 2017 respectively.

ACF also suggests that changes to state legislation and child welfare policies and practices might influence the number of substantiated allegations. The state commentaries reveal that some states experienced such changes, although it is not clear that they trended in one direction. Some states like Pennsylvania reported an increased emphasis on safety resulting in increased substantiations and others like New Jersey reporting reduced substantiations due to new policies.

As all this discussion shows, it is almost impossible to attribute a change in the number or rate of substantiation to an actual change in the amount of child abuse and neglect. Too many other things are influencing this number and rate.

Not only did ACF inaccurately herald a decrease in maltreatment but it went on to contrast this alleged decrease with the increasing number of referrals, stating “We are experiencing increases in the number of children referred to CPS at the same time that there is a decrease in the number of children determined to be victims of abuse and neglect.” Media outlets lost no time in picking up on this alleged contrast. For example,  the Chronicle of Social Change reported that Child Victimization Declines as Reports of it Continue to Rise.

The interpretation of child welfare numbers to paint a picture of decreasing maltreatment in the face of increasing reporting is not an accident. It feeds into the narrative that is currently dominating in most states and on the federal level without regard to party. According to this narrative, almost all children are better off staying with their parents, no matter how egregious the maltreatment. Removals should be prevented at all costs. If maltreatment is decreasing and reporting is increasing, perhaps something should be done to squelch those pesky hotline callers.

The data presented in Child Maltreatment is extremely important. It is too bad ACF did not stick to reporting it accurately so that readers can understand what it means–and what it does not.


  1. Colorado, Georgia, Nebraska and Washington were the other four states that expanded the use of differential response during FY 2017. 

The misuse of data and research in child welfare: home visiting and infant removals in New York State

Healthy Families New YorkData and research have tremendous potential to inform policymaking, allowing us to identify population trends and to assess the effectiveness of programs. Unfortunately the increasing importance placed on these tools has resulted in their frequent misuse. One recent article in the Chronicle of Social Change, a major online child welfare publication, exemplifies typical errors often made by public officials and accepted uncritically by the media.

The article is called The Program New York Says Helped Cut Newborn Removals to Foster CareIn it, Ahmed Jallow reports that the number of infants removed into foster care in New York State has “plummeted” while the same indicator has been increasing in the majority of states. Jallow quotes unnamed “state officials” that a home visiting program called Healthy Families New York (HFNY) is “the primary reason for this reduction in infant removals” and devotes most of the article to explaining and supporting this assertion. Unfortunately, the officials Jallow quotes simply don’t have the evidence to substantiate their claims. Rather than make this clear, Jallow reports these unbacked claims without qualifications and even adds additional misleading information to bolster them. These issues can be grouped into several categories.

Attributing causality without evidence. The centerpiece of the article is the claim by  New York State officials that the HFNY home visiting program is the primary reason for the reduction in infant removals in New York City. HFNY is New York’s version of one of the most popular home visiting models, which is called Healthy Families America (HFA). The difficulty of proving causality is well-known by social scientists, and journalists who write about policy should know enough to caution against accepting such blanket statements. To reduce child removals, a home visiting program would first have to reduce child maltreatment, and that reduction would have to be translated into a reduced removal rate. There are many factors that could more directly affect the number of infant removals, such as a shift in policy to prioritize keeping families together while accepting higher risks to children. And indeed, in New York City, by far the largest jurisdiction in the state, the Commissioner of the Administration on Human Services has attributed the decline in its foster care rolls to his agency’s “focus on keeping families together wherever we can.”

Making factual errors. Jallow states that “evaluations of HFNY show a significant impact in preventing further maltreatment incidents for parents involved with child protective services.” Actually, evaluations do not show a significant impact of the HFA model on child maltreatment. As a matter of fact, the respected California Evidence based Clearinghouse on Child Welfare (CEBC)  gave HFA a rating of “4” for prevention of child abuse and neglect, which means that studies have failed to find that it has any effect on child maltreatment. (The only worse rating is 5, which indicates that a program may be harmful to participants.) The only evaluation that Jallow cites is an interim report from an ongoing evaluation of HFNY suggesting that the program might reduce subsequent reports among women who had a previous substantiation for abuse or neglect. However, this study was never published in a peer-reviewed journal and therefore was not included in CEBC’s review.

Misusing evidence-based practice compilations. The CEBC and other clearinghouses of evidence-based practices can be very helpful to lay audiences by digesting and translating the results of methodologically complex studies and rating programs by the strength of their evidence. But users must be careful to read and understand the reports they are using.  Jallow states that the HFA home visiting  model (of which HFNY is an example) “has the highest rating of effectiveness on the California Evidence-Based Clearinghouse.” But he was reading the wrong report. As mentioned above, CEBC found that HFA failed to demonstrate any effect on child abuse and neglect. It is in a separate report on home visiting programs for child well-being that HFA CEBC gave HFA its top rating (“well supported by research evidence”) because of its impact on outcomes other than child abuse and neglect.

Overgeneralization: “In terms of documented proof, home visiting is the one that we know absolutely works,” Timothy Hathaway, executive director of Prevent Child Abuse New York, told Mr. Jallow. Unfortunately, Mr. Hathaway was overgeneralizing. There are many different home visiting programs which vary based on the nature of the provider, the content of the program, the goals of the program, and other factors. The effects of most home visiting programs on child abuse and neglect have been disappointing. The only program that has been found to have well-supported evidence of an impact on child abuse and neglect from CEBC is the Nurse Family Partnership program, which is very expensive and difficult to implement, and can only be used for certain populations–like first-time mothers. It is not surprising that many jurisdictions have opted to implement HFA instead.

Disregarding recent data. In addition to all the problems cited above, Jallow and his New York State informants chose to disregard the most recent data on foster care entries in New York. Jalloh reports, accurately, that the decline in infant foster care placement between 2012 and 2016 was part of an overall decline in the number of New York children entering foster care. And as Jallow states, this decline occurred while entries into foster care increased on the national level. But the pattern was reversed in 2017: nationally, foster care entries decreased slightly, while New York’s foster care entries increased. We don’t yet have the 2017 data for infants, but it seems likely that the trend in infant removals also reversed. Could it be that New York is starting to see the same kind of increase in removals that occurred earlier in many other states? Perhaps a growing opioid crisis in western New York is contributing to this, or perhaps the increase in child removals stems from concern that the focus on family preservation is endangering children.  And indeed an increase in child removals in New York City over the past 18 months has been attributed to an increase in hotline reports and a more aggressive response to these reports by investigative staff in the wake of  the highly-publicized child abuse deaths of two children who were known to the system but not removed. Disregarding the most recent year of data certainly makes for a clearer picture, but but it may be a less accurate one.

Jallow’s article illustrates how a flawed understanding of research and data can lead to faulty conclusions. A grandiose claim that one program is responsible for large changes in an indicator like child removals  deserves initial skepticism and rigorous vetting. Uncritical acceptance of such claims can lead to misguided policy decisions, like a decision to direct more funding to a program that is unproven. The press should scrutinize such claims assiduously, rather than accepting them credulously, presenting them without qualifications, or adding  flawed arguments in favor of these claims.