Mother and Infant Home Visiting Program Evaluation

MIHOPE includes two groups of participants. First, the included 4,229 women across 88 local home visiting programs (sites) in 12 states. Women will be randomized so that half can receive home visiting services and half are in a control group that receives referrals to other services in the community. Local home visiting programs chosen for the study must meet the following criteria: (1) they have been in operation for at least two years, (2) they are receiving funds through the MIECHV program, (3) they are operating one of the four evidence-based models described earlier, and (4) there is enough need for the program's services that they can provide a control group without reducing the number of families that receive home visiting.

Families were recruited as follows. Mothers were identified as eligible for a local home visiting program by staff from that program, using the program's current procedures. If the mother appears to be eligible for MIHOPE (based on the mother's age and pregnancy status or child's age), the home visiting staff will mention that a study is going on and that someone from the study team would like to be in touch to explain the study and to see if the mother is interested in participating. The home visiting program entered the mother's address and phone number. In addition, the home visiting program provided Mathematica with the mother's name and date of birth and the child's date of birth for purposes of making sure the family is not already in the study or has not previously declined to participate in the study. The MDRC Institutional Review Board determined that a partial authorization of the HIPAA Authorization requirement was warranted for use of the mother's information for sample recruitment.

Study field staff attempted to contact the family and schedule a visit to explain the study, request their participation, and obtain informed consent for research activities. Potential study participants were asked to sign two consent forms at the beginning of the study: one to participate in the study and to allow the team to collect administrative and survey data, and a second to be video-recorded during two home visits, if assigned to the home visiting group. Mothers also provided consent to allow us to collect information from the home visiting programs about the services they receive and information from the government on various information, including Medicaid records, vital records, child welfare records, and employment and earnings information from the National Database of New Hires. If an applicant was a non-emancipated minor, assent was obtained from the minor and written consent was obtained from her parent or guardian if the parent or guardian was in the home at the time of the recruitment visit. If the parent or guardian was not in the home, verbal consent was obtained. If verbal consent was provided, the parent or guardian was mailed a copy of the assent form that the woman signed and that explains what the research entails.

After consent was provided, study staff in the family's home initiated a call to the study's survey operations center for purposes of administering a one-hour survey. The family baseline survey included information on several domains specified in the ACA: newborn health; parental health and well-being; parenting practices, attitudes, and beliefs; domestic violence; history with the criminal justice system; family economic self-sufficiency; and referral and coordination of social services. In addition, the baseline survey collected information on demographics and household composition to describe the study sample, and contact information for family members or friends who can help locate the family at follow-up if they move. The survey also contained information about the parent's expectations regarding the home visiting program, which will inform research on program implementation. While the mother was completing the baseline survey by phone, the study field staff conducted the Home Observation for Measuring the Environment (HOME) to assess the quality and amount of stimulation that the child receives in the home as well as observations of the home environment. At the end of the telephone survey, the interviewer determined whether the family was assigned to the program (home visiting program) or control group (comparison group that will receive referrals to other services in the community); the result of random assignment was sent by automated email to the point of contact at the local program.

A second round of data was collected when the child was about 15 months old. These include: (1) a one-hour telephone interview (the family follow-up survey), (2) a 30-minute video-recording of the mother and child playing with some toys (the Three Bags task), (3) a direct assessment of the child's receptive language skills (expected to take 40 minutes), (4) measurement of the child's weight and height and the mother's weight (expected to take 5 minutes), and (5) observations of the family's home environment, which will not add time to the data collection procedures. Finally, data were collected from three state systems: (1) Medicaid, (2) vital records, and (3) child welfare. Follow-up data collection plans have been approved by the federal Office of Management and Budget.

The family follow-up survey included information on several domains specified in the ACA: infant and child health; child development; parental health and well-being; parenting practices, attitudes, and beliefs; domestic violence; history with the criminal justice system; family economic self-sufficiency; and use of social services. Survey questions focused on outcomes for which previous studies of home visiting have found effects and on outcomes that would not be available from other sources (such as administrative records).

Follow-up data collection also included several types of data collected in the family's home:

• Direct assessments of children's receptive language skills will be done using the Preschool Language Scales-5 Auditory Comprehension Scale (PLS-5), which is an individually administered test that assesses the child's ability to understand language. During the assessment, the child sat on the mother's lap while the field interviewer administered this test. At 15 months, toddlers' spoken language capabilities are only just beginning to develop. For this reason, the Auditory Comprehension subtest of the PLS-5 was used. The Auditory Comprehension cluster measures a child's ability to be attentive and respond to stimuli in the environment and to comprehend basic vocabulary or gestures.
• During the follow-up visit, field staff used a measuring tape and a weighing scale when they conduct field visits, which they will use to obtain both the mother's weight and the child's height and weight. Direct measurements of the child's weight and length will provide information on whether the child's growth is within a normal range or exhibits early signs of unhealthy growth trajectories (i.e., risk of obesity or under-development). Measurement of the mother's weight along with self-reported height will provide information on whether the mother is obese, which is associated with a host of other health problems. When measuring weight, if the child is willing, the field interviewer will ask him or her to stand on the weighing scale by him or herself. If the child is unwilling to do so, then the staff person will obtain the mother's weight and then ask her to pick up the child and stand on the scale with the child. The mother's weight will then be subtracted from this total weight to obtain the child's weight. These measures are being collected out of concern that the mother may not accurately report her own weight and may not know her child's current weight and height.
• The Home Observation for Measuring the Environment (HOME) - described under baseline data collection -- was conducted again at follow-up.
• The three bags task, a direct observation of parent-child interactions, was conducted. This assessment is intended to capture the parenting constructs of parental sensitivity, cognitive stimulation, positive regard, intrusiveness, negative regard, detachment, relationship quality (degree of relatedness and mutual engagement), and boundary dissolution (parent's inability to maintain an appropriate role in his or her interaction with child). From this task, children's behaviors towards the parent will also be gathered in the context of the parent-child interaction, including engagement with the parent, sustained attention, and negativity towards the parent.

In addition, the study asked approximately 500 home visitors and 100 supervisors in the 85 sites to enroll in the study. Some supervisors may also see families and carry a caseload, acting in the capacity of both supervisor and supervisor. Program staff were identified by a local program manager and sent emails by the study team asking them to enroll in the study. Home visitors and supervisors were asked to complete a 75-minute web-based survey around the time that the site enters the study. The home visitor and supervisor baseline surveys collected information on staff characteristics and on staff perceptions of organizational factors related to service delivery; employment, supervision and program outcomes; staff beliefs about home visitor roles and responsibilities, ratings of her or his own training and skills in supervising staff to carry out activities; ratings of her or his own ability to secure supervision and professional consultation; and demographics and individual background characteristics. Individuals who are both home visitors and supervisors were asked to complete a survey that combined elements of the home visitor and supervisor surveys, and lasted about 105 minutes. Because the surveys included sensitive information, home visitors and supervisors were asked to consent to the survey and home visitors were asked to consent to videotaped interactions with families. Home visitors and supervisors were asked to provide consent through a web-based application before completing the web-based surveys.

Home visitors and supervisors were asked to complete a 75-minute web-based survey again a year after enrolling in the study. Staff who are both home visitors and supervisors were asked to complete a 105-minute web-based survey again a year after enrolling in the study. Staff follow-up surveys parallel the baseline surveys. They measured malleable respondent characteristics and perceptions of organizational factors related to service delivery.

MIHOPE has also completed two additional follow-up surveys that were conducted around the time the child turns 2 ½ and 3 ½ years old. At those two points in time, the mother - or the child's primary caregiver if the mother was not available - was asked to take part in a short survey. The survey included about 15 minutes of questions on topics such as child and parental health, parenting style, child development, and social services used. The caregiver was also asked to provide updated contact information, as well as share information about people who may know how to locate her in the future.

MIHOPE conducted a kindergarten follow-up during the four school years in which children in the sample were slated to attend kindergarten: 2018-2019 (Cohort 1), 2019-2020 (Cohort 2), 2020-2021 (Cohort 3), and 2021-2022 (Cohort 4). However, due to the COVID-19 pandemic, kindergarten data collection paused during the 2020-2021 school year. It resumed during the 2021-2022 school year, when children in Cohort 3 were slated to attend first grade and children in Cohort 4 were slated to attend kindergarten. The study team adapted the kindergarten data collection to conduct all in-home assessments virtually for Cohorts 3 and 4 during the 2021-2022 school year. To gather information on MIHOPE families' experiences during the pandemic, the study team designed and implemented a brief web survey and virtual qualitative interviews with caregivers. These data were used to contextualize the study's kindergarten findings.

Because home visiting aims to affect a wide range of outcomes, the study team collected a wide range of data for the kindergarten follow-up, from the following data sources:

• A structured interview with the children's mothers to measure a broad set of constructs that are mostly not available from other data sources. In cases where the mother was not available (for example, because she no longer had custody of the child), data collection was conducted with the child's primary caregiver, such as the child's father. This approach was also used at the MIHOPE follow-up that occurred when children were 15 months of age.
• Direct assessments of children's language, math, and executive functioning skills conducted by trained field interviewers.
• Observations of parental warmth and children's self-regulation conducted by trained field interviewers during the direct assessment.
• Observations of mother-child interactions, such as parental sensitivity and child engagement of parent, by trained independent observers during a video-recorded semi-structured play interaction.
• A direct assessment of mothers' working memory conducted by trained field interviewers.
• A teacher survey to measure children's social and emotional development, approaches to learning, disciplinary incidents, receipt of special services, and school attendance.
• Federal administrative data on healthcare use via Medicaid.
• Federal administrative data on employment covered by the unemployment insurance system (National Directory of New Hires).
• State child welfare records.
• School records from state and local education agencies.

As mentioned previously, two additional data sources collected information about families' experiences during the COVID-19 pandemic: (1) a brief web survey was administered to families in all cohorts in September and October 2020, and (2) qualitative interviews were conducted with a subsample of 100 families who responded to the web survey between October 2020 and January 2021.

The study team will examine the effects of home visiting on 66 child and family outcomes in five outcome areas and show estimated effects for these individual outcomes in the kindergarten report. To focus the impact analyses and aid in the interpretation of the results, the study team has developed several topical research questions and will use omnibus tests to interpret the pattern of effects across the outcomes relevant to each research question. (The study team plans to use omnibus tests based on Devin Caughey, Allan Dafoe, and Jason Seawright, "Nonparametric Combination (NPC): A Framework for Testing Elaborate Theories." Journal of Politics 79, 2 2017: 688-701 to characterize patterns by accounting for the magnitude of estimated effects; they have used this approach in earlier MIHOPE analyses (see Michalopoulos et al., 2019 and Faucetta et al., 2023).) The study team will draw on the answers to these multiple topical research questions in interpreting the effects of home visiting at the kindergarten follow-up. While several topical research questions were pre-specified (shown below), additional research questions may be examined. In the kindergarten report, questions will be identified as pre-specified or not pre-specified. The research questions are listed in the section on outcomes.

MIHOPE has also gathered administrative data through the end of children's third grade year to conduct a benefit-cost analysis.

For the benefit-cost analysis, the study team will use data from earlier follow-ups along with administrative data through the time children are in third grade to examine how the benefits of MIECHV-funding home visiting programs compare with the costs of those services. Specifically, the analysis will (1) calculate the cost of operating evidence-based home visiting services for program group families who participated in home visiting compared to the estimated costs of other home visiting or parenting services that control group families may have received, and (2) monetize a set of outcomes home visiting aims to affect for both program and control group families, limited to those with established monetary values and selected to minimize the potential double counting of benefits. The calculation of costs will build on and extend earlier work that estimated costs through the first year of participation in home visiting. The difference in the monetized outcomes for program and control group families ("total benefits") will be compared to the difference in the estimated costs for program and control group families ("net costs") to ultimately determine the extent to which benefits outweigh costs (shown as "net benefits" and a "benefit-cost ratio").

The team will conduct three sets of analyses to provide information about whether benefits outweigh the costs of home visiting:

• The observed follow-up period analysis: The observed follow-up period analysis will measure benefits from study entry through the time the focal child is in the third grade on a limited set of outcomes that are created using data collected through third grade.
• The lifetime analysis: The lifetime analysis will include the same valuations for several of the outcomes in the observed follow-up period analysis, lifetime valuations for two outcomes in the observed follow-up period analysis, a projected valuation for one outcome in the observed follow-up period analysis, and lifetime valuations for two additional outcomes based on data collected through third grade.
• The supplemental analysis: The supplemental analysis will include all valuations used in the lifetime analysis as well as lifetime valuations for three additional outcomes. These additional outcomes will be based on data collected through third grade for which there is more uncertainty around the monetized values and for which monetized values have high dollar values.

The benefit-cost analysis will examine benefits and costs from three perspectives:

• study participants: The study participants' perspective will include benefits that accrue to both the mother and focal child;
• the government: The government perspective will encompass benefits that accrue to federal, state, and local governments, and
• society: The societal perspective includes benefits that accrue to society only plus the benefits to study participants and government. Benefits that are transfers within society do not affect the societal perspective.

To aid in the interpretation of the results, the study team will conduct a series of sensitivity analyses to test the degree to which the results of the analyses change with alternative assumptions. These sensitivity analyses include varying the discount rate, varying the assumptions used to estimate costs, varying the assumptions used to estimate benefits, and a Monte Carlo simulation to indicate the probability that benefits exceed costs. The study team will also conduct a series of tests to assess the degree to which the study's findings may be affected by missing data.

Inclusion criteria

• Eligible for home visiting program selected to be in the study
• At least 15 years old
• Pregnant or have a child under 28 weeks old

Exclusion criteria

• women already enrolled in home visiting (for example, with an older child)
• children in foster care in sites using EHS program model
• homeless families in sites using EHS program model
• women who speak neither English or Spanish