Keywords: Autism Integrated Care Program, pediatrics, autism, mental health, integrated care, implementation
Introduction
Integrated care models are revolutionizing healthcare by fostering collaboration between primary and specialty care, particularly in mental health. This approach is crucial for children’s well-being, highlighted by the proposal to consider mental health screening as the “eighth vital sign” in pediatric check-ups. While the evidence supporting integrated pediatric mental health care grows, its application for specific populations like children with Autism Spectrum Disorder (ASD) remains limited.
Children with ASD often face co-occurring mental health conditions that are frequently overlooked and untreated. Their complex needs require coordinated care across multiple providers, making integrated care within primary pediatric settings essential. Currently, there’s a gap in tailored integrated care models for children with ASD. Addressing this gap is critical for pediatricians to effectively identify and manage the mental health needs of children with ASD, thereby improving their access to and quality of care. This article explores the feasibility and acceptability of a tailored model, “Access To Tailored Autism Integrated Care” (ATTAIN), designed for school-aged children with ASD in primary care.
The ATTAIN Model: Co-Designing an Autism Integrated Care Program
The ATTAIN model was developed through a collaborative co-design process involving pediatricians, mental health professionals, healthcare administrators, caregivers, and implementation science experts. This partnership aimed to create an autism integrated care program that effectively identifies mental health needs and facilitates connections to tailored mental health services for children with ASD, while also aligning with existing organizational structures and workflows. The design was further informed by a detailed contextual assessment, emphasizing the necessity for a customized strategy that considers organizational capabilities to support implementation. Although various integrated care models exist, ATTAIN is unique in its specific design to address the distinct needs of children with ASD and their co-occurring mental health conditions.
ATTAIN’s adaptability is key, with customizations for each participating organization based on ongoing feedback from leaders and providers. The model comprises eight core steps, integrated into routine well-child visits and adapted to each organization’s workflow. These adaptations primarily focused on assigning responsibility for each step within the local context. The eight steps are:
- Patient Eligibility Confirmation: Verifying ASD diagnosis and age (4–16 years) in the patient’s medical record.
- Caregiver Screening (PSC-17): Caregivers complete the Pediatric Symptom Checklist-17 (PSC-17) to screen for clinically significant mental health symptoms. The PSC-17 was chosen for its brevity, validation across age ranges, availability in multiple languages, and relevance to common co-occurring mental health issues in children with ASD.
- PSC-17 Scoring and Documentation: Scoring the PSC-17 and documenting results in the electronic medical record (EMR).
- Elevated Score Discussion: For scores ≥ 15, the pediatrician discusses the results using a psychoeducational aid, the “ATTAIN (ASD+MH Information Sheet).”
- Mental Health Referral: Offering a referral to mental health services.
- Referral Documentation: Documenting the referral and the family’s decision (acceptance or declination) in the EMR encounter note, either via automated language or a paper form.
- Referral Follow-Up: A designated staff member (e.g., case manager) contacts the family regularly to assist in scheduling a mental health appointment.
- Referral Confirmation: The staff member confirms with the referring pediatrician once the mental health appointment is scheduled or attended.
To support the implementation of this autism integrated care program, three main strategies were used: (1) provider/clinic champions, (2) regular reflections, and (3) technical assistance. Clinic champions, acting as primary contacts, were engaged bi-weekly to discuss progress, obstacles, and needed adaptations. Regular reflections, inspired by ethnographic methods, involved research team members contacting champions to assess implementation and identify technical support needs. Technical assistance was delivered through bi-weekly emails with fidelity tips, answers to common questions, and reminders about ATTAIN procedures.
Implementation Science Framework: EPIS
The Exploration, Preparation, Implementation, Sustainment (EPIS) framework guided this study’s design, measurement, and analysis. EPIS is a widely recognized framework in implementation science, applicable across diverse healthcare settings and populations. It facilitates the evaluation of implementing and sustaining evidence-based practices or innovations within specific care environments. EPIS is valuable for structuring research questions, study design, data collection (quantitative, qualitative, and mixed methods), and identifying points where adaptations might be necessary. Key features of EPIS include its phased approach to implementation and its emphasis on contextual factors (outer system, inner organization, innovation characteristics) that influence implementation throughout its lifecycle. The EPIS framework was selected for its focus on understanding the multi-level contextual influences—specifically, how primary care providers within different healthcare systems deliver care—that affect implementation processes and outcomes of the autism integrated care program.
Study Objectives
This study aimed to evaluate the early implementation and sustainment outcomes of the ATTAIN model from the perspective of primary care providers (inner context users). It focused on how ATTAIN’s characteristics and implementation factors, such as organizational capacity to manage workflow changes, impacted the initial adoption phase of this autism integrated care program. The primary objective was to report implementation outcomes—feasibility, acceptability, and adoption—following a pilot trial in pediatric primary care clinics across three healthcare systems. As an exploratory pilot study, it did not test specific hypotheses but sought to understand provider experiences and identify necessary refinements for broader implementation.
Methods
A mixed-methods concurrent exploratory design (quantitative + qualitative) was employed to investigate primary care providers’ (PCPs) perceptions of feasibility, acceptability, adoption, and the barriers and facilitators to using ATTAIN in pediatric primary care settings for this autism integrated care program.
Participants and Setting
The ATTAIN pilot was conducted in pediatric primary care clinics from three healthcare organizations involved in ATTAIN’s development: 1) a Federally Qualified Health Center serving diverse communities near the U.S.-Mexico border; 2) a large network of pediatric practices in Southern California; and 3) a major integrated healthcare system in California. Seven clinics were selected based on leadership recommendations and geographic diversity. None of the participating clinics had pre-existing integrated mental health models for pediatric patients or specific populations like children with ASD. All invited clinics agreed to participate, and 36 PCPs from these clinics enrolled and completed a baseline survey.
Of the 36 PCPs, 22 completed a post-pilot survey and 16 participated in post-pilot interviews. Four PCPs were lost to follow-up due to retirement or leave, resulting in a 69% retention rate. Among the 22 post-pilot survey respondents, 3 were from Organization 1 and completed an abbreviated survey due to their organization’s early withdrawal from the pilot. Further details on the PCP sample are in Table 1.
Procedures
Recruitment and training procedures are detailed elsewhere (Stadnick, Penalosa, Martinez, et al., 2021). Briefly, all invited clinics agreed to participate. The research team conducted a one-hour ATTAIN training at each clinic during staff meetings. Participants received training materials and completed a baseline survey. Trainings occurred between November and December 2019.
Following training, providers were asked to use ATTAIN with up to five eligible patients (4–16 years with ASD) over four months, starting January 2020. Organization 1 experienced a delayed start and, shortly after launch, faced disruptions due to the COVID-19 pandemic and subsequent stay-at-home orders in California starting March 19, 2020. This significantly altered clinic workflows and capacity. Providers from Organization 1 were re-contacted post-pilot and completed an abbreviated survey to assess challenges and interest in future participation.
For Organizations 2 and 3, the pilot was paused for ten weeks after the March 2020 stay-at-home order due to substantial clinical and administrative changes, including reduced well-child visits and modified staffing. The pilot resumed for one final month in June 2020, totaling approximately four months of implementation. At the end of this period, all PCPs who completed the baseline survey were invited to complete an online evaluation survey. Providers from Organization 1 completed an abbreviated version. Table 2 details the measures completed by PCPs in each organization.
PCPs from Organizations 2 and 3 were also invited to participate in 30-minute virtual interviews to discuss their ATTAIN implementation experiences. Twenty providers participated in seven group interviews (n=17) and three individual interviews (n=3). Group interviews averaged three participants (range 2–4) and lasted approximately 30 minutes, slightly longer than individual interviews (average 24 minutes). Interviews were audio-recorded and transcribed. The interview guide is available upon request.
To minimize bias, the PI who led the ATTAIN training was not involved in post-pilot data collection. All study procedures were approved by the Institutional Review Board at the University of California San Diego and partner organizations.
Measures
Table 1 details measures completed by each organization.
Baseline Survey
PCPs (n=36) completed a 10–15-minute baseline survey assessing demographics, professional characteristics, Evidence-Based Practice Attitude Scale (Aarons, 2004), Organizational Readiness for Implementing Change (Shea et al., 2014), Implementation Climate Scale (Ehrhart, 2014), and the ASD + Mental Health Comfort and Knowledge Questionnaire. Baseline survey results are reported in Stadnick, Penalosa, Martinez, et al., 2021. Baseline data from the ASD + Mental Health Comfort and Knowledge Questionnaire were used to assess changes during the pilot.
COVID-19 Impact
Locally developed items and select items from the NIH CoRonavIruS Health Impact Survey V0.3 Adult Self-Report Baseline Form were used to assess COVID-19 impacts on clinical services and personal/family experiences.
ATTAIN Training Evaluation
Five items assessed PCP perceptions of the ATTAIN training, including training method, helpfulness in tailoring communication, sufficiency of training, utility of bi-weekly tips, and the number of patients they used ATTAIN with.
Feasibility of Intervention Measure (FIM) and Acceptability of Intervention Measure (AIM)
These 4-item measures (Weiner et al., 2017) assessed PCP perceptions of ATTAIN’s acceptability, appropriateness, and feasibility using a Likert scale (0 “completely disagree” to 4 “completely agree”). Example items: “ATTAIN is appealing to me.” (AIM); “ATTAIN seems easy to use.” (FIM). Scores were averaged, ranging from 0–4. Internal consistency was excellent: AIM (α=0.96), FIM (α=0.98).
Perceived Characteristics of Intervention Scale (PCIS)
The 18-item PCIS (Cook et al., 2015) assessed attitudes toward ATTAIN, including relative advantage, compatibility, and complexity, using a 5-point Likert scale (0 “not at all” to 4 “a very great extent”). Example items: “The ATTAIN model is clear and understandable.” Total scores were averaged. Internal consistency was excellent (α=0.94). Only Organizations 2 and 3 completed this measure.
Adoption
ATTAIN adoption was measured as the proportion of eligible patients with whom PCPs used ATTAIN, derived from de-identified EHR data.
Measure of Innovation-Specific Implementation Intentions (MISII)
This 3-item measure (Moullin et al., 2018), adapted for ATTAIN, assessed provider intentions to continue using ATTAIN, using a 5-point Likert scale (0 “not at all” to 4 “a very great extent”). Example item: “I plan to use ATTAIN with my patients.” Total scores were averaged. Internal consistency was excellent (α=0.95).
ASD Knowledge and Confidence
This measure assessed PCP knowledge and confidence in caring for children with ASD, including mental health discussions. Rated on a 5-point Likert scale (0 “not at all knowledgeable/confident” to 4 “extremely knowledgeable/confident”) at baseline and post-pilot. Subscale scores were averaged. Post-pilot internal consistency was very good (Knowledge α=0.89; Confidence α=0.88).
Mental Health Screening, Referral, and Linkage Comfort
This measure assessed PCP comfort levels with mental health screening and referral/linkage practices for children with ASD, using a 10-point Likert scale (1 “not at all comfortable” to 10 “very comfortable”). Items were created based on formative needs assessment (Stadnick, Martinez, Aarons, et al., 2020). Each item is reported separately.
Data Analysis Plan
Quantitative Data Analysis
Chi-square analyses examined demographic differences between baseline and post-pilot PCP samples. Descriptive statistics, effect sizes, and 95% confidence intervals characterized study constructs and feasibility, acceptability, and adoption rates. Changes in PCP knowledge, confidence, and comfort were also analyzed. Data analysis used SPSS Statistics version 26, emphasizing descriptive statistics and confidence intervals for pilot trial analysis (Lee, Whitehead, Jacques & Julious, 2014).
Qualitative Data Analysis
Rapid qualitative assessment methods (Hamilton, 2020) were used to analyze interview data from Organizations 2 and 3. A templated matrix summarized interview responses by question. Two researchers iteratively reviewed summaries to develop themes organized by EPIS phase and inner context domain, aided by Microsoft Excel version 16.53.
Integration
Qualitative themes were integrated with quantitative findings to assess convergence and expansion (Palinkas et al., 2019). Integrated findings informed refinement recommendations, summarized in a joint display in Table 3.
Community Involvement
Community pediatric providers, organizational leaders, and caregivers of children with autism were involved in ATTAIN’s development and implementation, ensuring the autism integrated care program was relevant and responsive to community needs.
Results
Of the 22 PCPs who completed both baseline and post-pilot surveys, 73% were female and 23% Hispanic/Latino. PCP tenure was varied: <1 year (14%), 1–3 years (9%), 3–10 years (27%), and >10 years (50%). A significantly higher proportion of the post-pilot sample was female compared to the baseline sample (χ2 (1)=4.00, p < .05). Table 2 provides further demographic details.
Impact of COVID-19 on Care Delivery
The COVID-19 pandemic significantly impacted care delivery during the pilot. Most PCPs (73%) reported clinic suspensions of in-person services, with 96% shifting to telehealth. A majority (84%) felt prepared or confident in delivering telehealth services.
Perceptions of ATTAIN Training
All post-pilot survey respondents (n=22) reported receiving in-person (95%) or virtual (5%) ATTAIN training. 24% reviewed bi-weekly training tip emails. Regarding training utility, 67% agreed it helped tailor mental health communication with ASD patients; 71% found the training sufficient; and 48% felt bi-weekly tips aided ATTAIN use. Organization 3 PCPs reported using ATTAIN with more patients (M=8) than Organization 2 PCPs (M=3).
Changes in Knowledge, Confidence, and Comfort
A medium effect size (Cohen’s d = .5; 95% CI: −0.7,−0.01) was observed in increased confidence in caring for children with ASD from baseline (M=2.5, SD=.6) to post-pilot (M=2.8, SD=.6). Small effect sizes were noted in knowledge (Cohen’s d = .3), comfort interpreting screening results (Cohen’s d = .3), and coordinating care (Cohen’s d = .1). No significant change was found in comfort identifying mental health needs (Cohen’s d = 0.0).
Implementation Outcomes
PCPs generally agreed ATTAIN was acceptable (M=2.7, SD=.8) and feasible (M=3.0, SD=.6). The PCIS total score (M=2.1, SD=.8) indicated moderate agreement regarding ATTAIN’s characteristics (compatibility, clarity, support). Intentions to sustain ATTAIN were moderate (MISII M=1.5, SD=1.3).
Adoption rates were similar across Organizations 2 and 3: 53% in Organization 2 and 55% in Organization 3.
Qualitative Results
Qualitative findings supported quantitative results, confirming ATTAIN’s feasibility and acceptability, particularly the mental health needs identification components. Qualitative data expanded on survey data, highlighting organizational differences in feasibility and acceptability perceptions. Organization 3’s integration of a direct referral order to Psychiatry Developmental Case Managers enhanced positive perceptions. Organization 2 PCPs found the automated PSC-17 screening feasible but expressed challenges with post-referral steps, particularly the perceived insufficiency of staff-led family contact for scheduling appointments and variability in mental health service access based on insurance.
Both organizations reported low eligible patient volume, exacerbated by COVID-19 impacts on in-person visits. PCPs across organizations valued ATTAIN’s focus on mental health needs for children with ASD, especially the PSC-17 and psychoeducational aids.
Mixed-methods findings led to recommendations for enhancing feasibility and acceptability, including automating patient eligibility identification and PSC-17 processes across all clinics, to improve the autism integrated care program.
Discussion
This mixed-methods pilot study evaluated ATTAIN, a tailored autism integrated care program for children with ASD. Findings indicate that ATTAIN is a feasible and acceptable model for mental health needs identification and referral in primary care. PCPs showed mixed intentions for continued use, with qualitative data providing deeper insights into implementation determinants and guiding refinement recommendations.
These results align with growing evidence supporting integrated care models for mental health screening, referral, and treatment (Walter et al., 2019) and underscore that integrated care implementation needs to be tailored to organizational and patient contexts (Njoroge et al., 2016). ATTAIN pilot findings also support previous research showing increased provider comfort in managing mental health within integrated care models (Burkhart et al., 2020).
While ATTAIN focuses on children with ASD and mental health concerns, its findings may generalize to other pediatric integrated care models. A key generalizable finding is the feasibility and acceptability of the PSC-17 as a mental health screener in primary care. Compared to more narrowly focused screeners like SCARED or PHQ-9 (Burkhart et al., 2020), the PSC-17 screens for a broader range of common co-occurring conditions in children with autism (Brookman-Frazee et al., 2018) and is validated for children as young as four.
Several inner organizational context refinements are recommended to improve scaling and sustainment, guided by study findings and the call for rapid implementation (Smith et al., 2020). Rapid implementation emphasizes delivering evidence-based practices efficiently by adapting methods to real-world needs. Proposed refinements include:
First, in line with rapid implementation principles, regular (weekly/bi-weekly) data feedback loops to PCPs and clinic managers are recommended to facilitate agile responses to implementation challenges and reinforce implementation efforts. Feedback could also include brief interactive support during team meetings.
Second, to address PCPs’ need for more action-oriented family support for mental health linkage, several recommendations are made: assigning a dedicated staff lead (e.g., family navigator) to assist families with logistical issues and explain the rationale for mental health services; preparing families for follow-up calls post-referral; and including referral follow-up information in after-visit summaries.
Third, consistent with the need for closed-loop communication, referral outcomes should be consistently shared back with referring PCPs.
Fourth, to enhance feasibility and acceptability, automated decision support within EHR systems should be further developed for this autism integrated care program.
Study limitations include the small PCP sample size, although they were recruited from diverse healthcare systems. The diversity of settings was both a strength and a challenge, highlighting the need for tailored approaches but straining research resources for intensive support. The pilot’s timing during the COVID-19 pandemic also presented challenges, affecting clinic operations and potentially data collection. Despite these challenges, the study provides real-world evidence of ATTAIN’s feasibility and valuable insights for refinement.
Consistent with the iterative nature of implementation, next steps involve returning to the Preparation phase and conducting debrief meetings with each organization to plan micro-pilots of suggested refinements. The ATTAIN infrastructure will also support the implementation of a family navigation intervention in a new stepped-wedge implementation trial (NIMH R34 MH120190).
Acknowledgements
We thank the ATTAIN Advisory Board, Joanna Sariñana, David Lee, Bryan Lin, Bhanuja Dub, Sara Walpole, Dr. Rachel Ireland, Darrell Walters, and the ATTAIN pilot participants, clinical and administrative staff and leaders for their contributions.
Source of Funding: This study was funded by grants from the National Institutes of Health (NIMH K23110602: Stadnick; R34MH120190: Stadnick; NIDA R01DA049891: Aarons; and NIMH R03MH117493: Aarons).
Footnotes
Conflicts of Interest: The authors declare no conflicts of interest.
