PARENT EXPERIENCES OF A FAMILY-CENTERED INTERVENTION: EXAMINING ETHNOCULTURAL GROUP DIFFERENCES by CRISTINA E. BUSTOS A DISSERTATION Presented to the Department of Counseling Psychology and Human Services and the Graduate School of the University of Oregon in partial fulfillment of the requirements for the degree of Doctor of Philosophy March 2011 ii DISSERTATION APPROVAL PAGE Student: Cristina E. Bustos Title: Parent Experiences of a Family-Centered Intervention: Examining Ethnocultural Group Differences This dissertation has been accepted and approved in partial fulfillment of the requirements for the Doctor of Philosophy degree in the Department of Counseling Psychology and Human Services by: Dr. Benedict McWhirter Chair Dr. Elizabeth Stormshak Member Dr. Joseph Stevens Member Dr. Thomas Dishion Outside Member and Richard Linton Vice President for Research and Graduate Studies/Dean of the Graduate School Original approval signatures are on file with the University of Oregon Graduate School. Degree awarded March 2011 iii © 2011 Cristina E. Bustos iv DISSERTATION ABSTRACT Cristina E. Bustos Doctor of Philosophy Department of Counseling Psychology and Human Services March 2011 Title: Parent Experiences of a Family-Centered Intervention: Examining Ethnocultural Group Differences Approved: ________________________________________________ Dr. Benedict McWhirter Ensuring that psychological interventions are well received and effective among ethnically and culturally diverse groups is at the forefront of psychological research. This study is a nonexperimental, posttest evaluation of differences between European American and ethnocultural group parents’ perceptions of the Family Check-Up (FCU), a family-centered, ecologically and community-based intervention that provides family assessment, support, and motivation to change for families coming to counseling. Research has demonstrated the effectiveness of the FCU but has yet to evaluate parent perceptions of the FCU. This study examines features of intervention implementation that lead to treatment satisfaction and adherence by evaluating parent perceptions of (a) the FCU intervention, (b) therapist interpersonal qualities, and (c) therapist multicultural competence. Both parent and observational coder ratings of the intervention were assessed along these dimensions. v In the first year of a longitudinal study of the FCU, data were collected from parents of children who attended three public middle schools in a metropolitan area of the Pacific Northwest. Within-subjects analyses were conducted to assure measurement validation and treatment fidelity. A between-subjects multivariate analysis of variance was utilized to examine ethnocultural group differences. Results revealed that all measures demonstrated high internal consistency reliability, high interscale correlations, and good construct validity. Results revealed high interrater agreement between parent ratings of treatment receipt and coder ratings of treatment delivery, indicating treatment fidelity. Results revealed no statistically significant differences in parent perceptions of the FCU intervention, regardless of parent ethnocultural group. Additional analyses demonstrated that observational coders rated family consultants who worked with ethnocultural group parents higher in multicultural competence than those who worked with European American parents. In summary, study results suggest (a) that the measures developed and adapted for this investigation were reliable and valid, and (b) that we found no evidence of perceived differences in the FCU intervention across ethnocultural group and European American parents. The FCU continues to be an intervention that can be successfully implemented among ethnically and culturally diverse families. vi CURRICULUM VITAE NAME OF AUTHOR: Cristina E. Bustos GRADUATE AND UNDERGRADUATE SCHOOLS ATTENDED: University of Oregon, Eugene University of California, San Diego DEGREES AWARDED: Doctor of Philosophy, Counseling Psychology and Human Services, 2011, University of Oregon Master of Education, Counseling, Family, and Human Services, 2004, University of Oregon Bachelor of Arts, Human Development, 2001, University of California, San Diego AREAS OF SPECIAL INTEREST: Cultural Competency in Psychological Intervention Latino Mental Health Intimate Partner Violence PROFESSIONAL EXPERIENCE: Clinical Research Therapist, Department of Psychiatry, University of California, San Diego, California, 2010-Present Predoctoral Psychology Intern, Kaiser Permanente, San Diego, California, 2009- 2010 Bilingual Parent Consultant, Early Steps Research Project, Child and Family Center, University of Oregon, Eugene, 2005-2008 vii Instructor/Junior Faculty Member, Family and Human Services Undergraduate Program, University of Oregon, Eugene, 2007-2008 Peer Supervisor, Counseling Psychology Program, University of Oregon, Eugene, 2007-2008 Teaching Assistant, Family and Human Services Undergraduate Program, University of Oregon, Eugene, 2004-2007 Practicum Child and Family Therapist, Child and Family Center, University of Oregon, Eugene, 2006-2007 Extern Therapist/Neuropsychological Examiner, Veterans’ Affairs, Eugene, Oregon, 2006-2007 Practicum Therapist, Veterans’ Affairs, Eugene, Oregon, 2005-2006 University Supervisor, Family and Human Services Undergraduate Program, University of Oregon, Eugene, 2004-2006 Visit Supervisor, Kids First Safe Alternatives Center, Eugene, Oregon, 2003-2005 Intern Therapist, Looking Glass Adolescent Recovery and Counseling Program, Eugene, Oregon, 2003-2004 Intern Therapist, Center for Family Therapy, Eugene, Oregon, 2003-2004 Regional Clinical Supervisor, Behavior Therapy and Family Counseling Clinic, San Diego, California, 2001-2002 Behavior Management Consultant, Behavior Therapy and Family Counseling Clinic, San Diego, California, 2001-2002 GRANTS, AWARDS AND HONORS: General University Scholarship, University of Oregon, 2009 Counseling Psychology and Human Services Research Travel Scholarship, 2008 Diversity Building Scholarship, University of Oregon, 2008 viii Jean Twohig Scholarship, College of Education, University of Oregon, 2008 General University Scholarship, University of Oregon, 2007 Florence Wolfard Scholarship, College of Education, University of Oregon, 2006 General University Scholarship, University of Oregon, 2005 ix ACKNOWLEDGMENTS I would like to thank Benedict McWhirter, Thomas Dishion, Elizabeth Stormshak, and Joe Stevens for their support, guidance, and assistance during the research process. I would like to thank Kathryn Kavanagh for her research mentorship, assistance, and support. I would also like to thank the staff of Project Alliance in Portland, Oregon, and the Child and Family Center in Eugene, Oregon, for their support and hard work, as well as the families who have participated in this project. I would like to acknowledge my colleagues and members of my Doctoral Seminar, past and present, for their assistance in research, professional development, and social support provided throughout this research project and my doctoral education. This research was supported by grant DA018374 from the National Institute of Health, awarded to Elizabeth Stormshak, Kathryn Kavanagh, and Thomas Dishion at the University of Oregon. x For my family, whose hard work, dedication, sacrifices, love, support and belief in me has allowed me to achieve my professional dreams. For my husband, for his love, support, understanding, and belief in me. For my friends, for their never-ending encouragement and support. xi TABLE OF CONTENTS Chapter Page I. INTRODUCTION AND LITERATURE REVIEW ...................................................... 1 Family-Centered Interventions .................................................................................. 5 The Role of Multicultural Competency in Family Intervention ................................ 6 Case Example: The Family Check-Up ...................................................................... 7 Study Purpose ............................................................................................................ 11 Research Design ........................................................................................................ 12 Research Question ..................................................................................................... 13 II. METHODS ................................................................................................................... 14 Participants ................................................................................................................ 14 Measures .................................................................................................................... 19 III. RESULTS.................................................................................................................... 34 Data Management and Pre-Analysis Screening ......................................................... 34 Preliminary Analyses ................................................................................................. 38 Group Differences in Parent Perceptions of the FCU ................................................ 45 Differences in Parent and Coder Ratings of Multicultural Competence ................... 46 Univariate Analyses of Group Differences ................................................................ 48 IV. DISCUSSION ............................................................................................................. 53 Main Findings ............................................................................................................ 53 Measurement .............................................................................................................. 56 xii Chapter Page Limitations ................................................................................................................. 57 Future Directions ....................................................................................................... 60 Conclusion ................................................................................................................. 61 APPENDICES A. PROGRAM IMPRESSIONS SURVEY #1 ................................................................. 63 B. FEEDBACK OBSERVER GLOBAL IMPRESSIONS ............................................... 68 C. FEEDBACK OBSERVER GENERAL IMPRESSIONS RATING MANUAL OBSERVATION AND RATING PROCEDURES ..................................................... 74 D. PAL2 FEEDBACK FOGI RATINGS .......................................................................... 87 E. CORRELATIONS AND Z-SCORES FOR OBSERVATIONAL CODER RATINGS (FOGI) ........................................................................................................ 89 REFERENCES CITED ..................................................................................................... 91 xiii LIST OF TABLES Table Page 1. Participant Family Demographics ................................................................................. 15 2. Parent Survey (PROIM) and Observational Coder Ratings (FOGI) Descriptive Statistics and Correlations ............................................................................................ 21 3. Descriptive Statistics for the CRF-S: Parent Survey and Original Study Data ............. 24 4. Correlations Between CRF-S Total Scale and Subscales for Parent Responses .......... 25 5. Descriptive Statistics for CRF-S Ratings: Parent Survey (PROIM), Observation Coder (FOGI) and Original Measure Statistics ............................................................. 30 6. Correlations Between CRF-S Total Scale and Subscales for FOGI Responses ........... 31 7. Multitrait Multimethod Correlations ............................................................................ 39 8. Treatment Fidelity Correlations .................................................................................... 43 9. MANOVA Structure and Function Coefficients .......................................................... 46 10. Group Means ............................................................................................................... 47 11. MANOVA Structure and Function Coefficients ........................................................ 48 12. Correlations and Z-Scores for Parent Survey Ratings (PROIM) ................................ 51 1 CHAPTER I INTRODUCTION AND LITERATURE REVIEW The importance of ensuring that psychological interventions are equally effective for ethnic and racial minority populations (heretofore referred to as ethnocultural groups) as well as majority populations increases every day as the diversity of the U.S. population grows and shifts. Currently, of more than 300 million people living in the United States, approximately 74% identify as European American, 15% as Latino, 12% African American, 4.5% Asian, 2% multiethnic or multiracial, and 1% American Indian (U.S. Bureau of the Census, 2006). The largest ethnocultural group (ECG) in the U.S. is Latino/a, which grew by 2.5% between 2000 and 2006 (U.S. Bureau of the Census, 2006). These population changes highlight the importance of addressing issues of race, culture, and ethnicity more thoroughly in intervention research. First, it is important to recognize that most models of psychological intervention in the U.S. have been developed from predominantly European American contexts and populations (Sue & Zane, 2006). Second, the changing demographics of the U.S. highlight the ethical imperative to conduct more dissemination and intervention research on ethnocultural group populations to ensure that existing interventions are received well by and are effective for ethnically, racially, and culturally diverse groups (Sue, Bingham, Porché- Burke, & Vasquez, 1999). Mental health disparities among groups in the U.S. are one reason why the research agenda is so important. Public demographic data reveal that one in 10 children in 2 the U.S. suffers from mental illness, but only one in five children with mental illness receive treatment (U.S. Public Health Service, 2000). Childhood and adolescent emotional and behavioral problems are associated with impairments, a greater likelihood for school dropout, and a greater probability of experiencing a lower quality of life (U.S. Public Health Service, 2000). Moreover, the U.S. Surgeon General (U.S. Department of Health and Human Services, 1994) has noted striking disparities in mental health for ethnocultural groups, including (a) less access to mental health services, (b) less likelihood to receive mental health services, (c) poorer quality of mental health care, and (d) underrepresentation in mental health research. Ethnocultural group children, children living in poverty, and children with co-morbid disorders are also disproportionately represented among children in the juvenile justice system (U.S. Public Health Service, 2000). One reason for this disparity between European American and ethnocultural children is that there is a significant lack of cultural competence among mental health providers who provide service, and limited embedded outreach services and prevention programs for ethnocultural groups (U.S. Public Health Service, 2000). Effective mental health services for children and families require cultural competence, family participation, and appropriate services and supports (U.S. Public Health Service, 2000). The movement to establish evidence-based practices in psychology (EBPP) has sought to reduce treatment disparities that have resulted from traditional mental health practices (U.S. Department of Health and Human Services, 1999; U.S. Public Health Service, 2001). However, there is still little research on evidence-based practices in psychology with ethnocultural groups (Sue, 2003; Sue & Zane, 2006; Yasui & Dishion, 3 2007). Many studies include racial and ethnic minority subjects; however, inclusion alone does not necessarily lead to new knowledge of ethnocultural populations (Sue & Zane, 2006; Hall, 2001), nor is it sufficient to assume that if an intervention is effective for majority populations, it will be equally effective for minority populations (Sue, 1999; Bernal & Scharrón-del-Rio, 2001). Additionally, researchers often ignore intra- and intergroup differences, and treat ethnic majority and minority groups as homogenous groupings (Okazaki & Sue, 1995). Treatments are often designated as validated without validity research on effectiveness of treatment for ethnocultural groups (Sue, 1999). Moreover, Morales and Norcross (2010) argue that research supporting the cultural sensitivity of EBPPs is necessary, as interventions lacking cultural sensitivity are irrelevant. In order to advance our knowledge of effective interventions for ethnocultural groups, research needs to focus on process-level variables—e.g., client perceptions of and response to interventions—as well as outcome-based issues (Hill & Corbett, 1993). At the same time, while studies on EBPP and empirically supported treatments have proliferated in the adult research, there are fewer studies on effective treatments for children and families that have been conducted in naturalistic settings or that account for the cultural complexities of home-based treatment (Weisz, Sandler, Durlak, & Anton, 2005). Szapocnik and Kurtines (1993) argue that it is necessary to study the context of individuals within their families and cultures in working with diverse youth. In research conducted in naturalistic settings, contextual factors, including personal factors and culture of both clients and therapist, influence intervention outcome. Contextual factors that are important to consider in working with diverse populations, for instance, include 4 multicultural competence and the interpersonal influence of the interventionist (Fuertes & Brobst, 2002; Kumpfer & Alvarado, 2003). One of the difficulties in establishing EBPP with ECGs is psychology’s bias towards achieving internal validity (Sue, 1999; Sue & Zane, 2006). External validity has been of secondary interest, and the difficulty in pursuing research with adequate samples and methodology has resulted in a paucity of information on EBPP with racial and ethnic minorities (Sue, 2003; Sue & Zane, 2006). Research would benefit from placing equal value on studies with high external validity in order to maximize the potential for generalizability and increase our knowledge of EBPP with ECGs (Sue, 2003; Sue & Zane, 2006). Community-based interventions offer great promise for enhancing the external validity of treatments and increasing positive outcomes among ethnocultural youth. For youth with emotional or behavioral disorders, effective community-based programs share the following characteristics: (a) Programs function as service components in a system of care and adhere to system values (i.e., they are individualized, family-centered, and strengths based); (b) programs and services are provided not only in offices, but also in homes, schools, communities, and neighborhoods; (c) rather than receiving formal clinical training, direct care providers are supervised by traditionally trained mental health professionals; (d) services may operate under the auspices of any human services sector; (e) the external validity of empirically evaluated services is enhanced as programs are developed and studied in the field with real-world child and family clients; and 5 (f) community-based services and programs are much less expensive to provide than institutional care (U.S. Public Health Service, 2000). Family-Centered Interventions Family-centered interventions are an excellent and very promising example of effective community-based interventions, and of interventions that improve the lives of youth. These programs underlie the importance of family in effecting positive change when marriage, family, youth or school problems are present. Effective parenting, for instance, is one of the most critical factors in the prevention of adolescent problems (Kumpfer & Alvarado, 2003; Patterson, Reid, & Dishion, 1992). In fact, many precursors of serious adolescent problems can be reduced or eliminated through early intervention to improve parenting and family systems dynamics (Kumpfer & Alder, 2003; Kumpfer & Alvarado, 2003). The most significant protective family factors for improving adolescent behaviors include positive parent-child relationships, positive discipline methods, monitoring and supervision, and communication or prosocial and healthy family values and expectations (Ary et al., 1999; Center for Substance Abuse Prevention, 2000; Kumpfer & Alvarado, 2003). Family-centered interventions such as behavioral parent training, family skills training, and brief family therapy are all effective models of intervention when applied as a prevention program with high-risk youth (Kumpfer & Alvarado, 2003). Many family-centered interventions, such as parenting programs, also show potential promise for helping ethnocultural communities. But many programs remain untested across ECGs, thus suggesting an important direction for future research 6 in order to address mental health service disparities between ECGs (Bernal, Jiménez- Chafey, & Domenech Rodríguez, 2009). The Role of Multicultural Competency in Family Intervention Kumpfer and Alvarado (2003) argue that family interventions can be powerful and cost-effective tools for reducing youth problems; however, population-specific versions of evidence-based programs are necessary to increase appropriateness for age and gender as well as cultural, geographic, and special considerations (Kumpfer & Alder, 2003; Kumpfer, Alvarado, Smith, & Bellamy, 2002). Others argue that consideration of the context of individual and family cultures (Szapocnik & Kurtines. 1993) and a demonstration of cultural sensitivity, multicultural awareness, and multicultural counseling skills (Pope-Davis et al., 2001) are necessary in order to work competently with diverse families and youth (Sue, 1999). Researchers and practitioners widely agree on the importance of cultural sensitivity; however, ambiguities surrounding the actual constructs that comprise cultural competence make it difficult to measure cultural competency and the impact of therapist behaviors on clients (LaFromboise, Coleman, & Hernandez, 1991; Sue, 2003; Sue & Zane, 2006). Three types of support for multicultural competence include general acceptance among professionals, scale-specific research, and research on the effects of culturally responsive counselor behavior (Ponterotto, Fuertes, & Chen, 2000; Worthington, Soth-McNett, & Moreno, 2007). Psychological research on the multicultural competence of therapists has relied primarily on self-report measures 7 (Pope-Davis et al., 2001). Pope-Davis and colleagues have recommended using real clients in real-world settings, conducting more qualitative research to thoroughly investigate client experiences in multicultural counseling, and examining the multicultural competence of counselors and how perceptions of both competence and therapist affect therapy outcomes. Fuertes and Brobst (2002) have demonstrated that perceptions of multicultural competence are highly correlated with perceptions of counselor attractiveness, trustworthiness, expertness, and empathy, which are factors that have been found to mediate the interpersonal process of intervention (LaCrosse, 1980; Strong, 1968). Additionally, LaCrosse (1977) found that counseling effectiveness is determined to a large extent by client perceptions of counselor behavior. Case Example: The Family Check-Up The Family Check-Up (FCU; Dishion & Kavanagh, 2003; Stormshak & Dishion, 2002) is an example of a family-centered, ecologically focused, evidence-based, and community-based intervention that provides family assessment, professional support, and motivation to change in the administration of parent training and family intervention. The FCU is based on the Drinkers Check-Up (Miller & Rollnick, 1991), which utilizes motivational interviewing (MI) to promote change and reduce problem behaviors (Moyers, Miller, & Hendrickson, 2005). In the FCU, MI can be applied to the process of encouraging parents with high-risk youth to reflect on parenting practices and provide 8 motivation to continue efforts or change when youth and other family members are at risk of and engaged in problematic behaviors. The FCU consists of three sessions: the initial interview; a multiagent and multimethod ecological assessment, including family observations, school assessment, and youth and parent reports conducted by an assessment team; and a family feedback session using MI skills with a family consultant who provides a menu of available, empirically supported parenting interventions for parents to choose from if they desire further intervention (Dishion & Kavanagh, 2003; Dishion & Stormshak, 2007). During the initial interview, the family consultant explores the parents’ concerns, evaluates their readiness for changing the identified problem or their ―stage of change‖ (Prochaska, DiClemente & Norcross, 1992; Prochaska & Norcross; 2001), and motivates them to participate in a family assessment. After family assessment, the parents meet with the family consultant for a feedback session in which they summarize the results of the assessment utilizing MI strategies (Dishion, Nelson, & Kavanagh, 2003). The feedback session incorporates a set of five behavior-change principles utilized in motivational interviewing (Burke, Arkowitz, & Menchola, 2003; Dishion & Kavanagh, 2003; Miller & Rollnick, 2002). The feedback session consists of six intervention targets referred to as FRAMES: providing feedback, conveying responsibility for behavior change, giving advice, providing realistic ideas for making changes, empathy, and promoting parent self- efficacy for making changes (Dishion & Kavanagh, 2003; Dishion & Stormshak, 2007). The Family Check-Up feedback session is based on the family assessment and serves as a bridge to treatment by emphasizing parent and family strengths and calling attention to 9 potential areas of change to motivate parents to address issues with support or independently (Dishion & Kavanagh, 2003; Dishion et al., 2008; Yasui & Dishion, 2007). The Family Check-Up model has been used with families of diverse ethnic, racial, and cultural backgrounds, and to date has demonstrated strong treatment outcomes of reducing child and adolescent problem behavior, with some supporting evidence for outcomes among ethnocultural groups (Connell, Dishion, Yasui, & Kavanagh, 2007; Dishion & Andrews, 1995; Gill, Hyde, Shaw, Dishion, & Wilson, 2008; Kavanagh, Dishion, & Connell, 2006; Shaw, Dishion, Supplee, Gardner, & Arnds, 2006; Shaw, Dishion, Connell, Wilson, & Gardner, 2009; Slavet et al., 2005). Research on the FCU has focused on the outcomes of adaptive family intervention on preventing high-risk behavior, improving behavioral outcomes, and reducing substance abuse among children and adolescents in general (Connell et al., 2007; Dishion & Andrews, 1995; Shaw et al., 2006; Slavet et al., 2005; Stormshak et al., 2011). For families with adolescents, the FCU has demonstrated an ability to reduce the risk of problem behaviors between early and late adolescence, including the use of alcohol, tobacco, and marijuana (Connell et al., 2007; Dishion et al., 2003), antisocial behavior during ages 11-17, and substance abuse diagnoses and arrests through age 18 (Connell et al., 2007). Adolescents have also demonstrated increased confidence in their ability to resist drug use after engaging in the FCU, with their parents also reporting higher confidence in impacting their child’s choices regarding risky behaviors (Slavet et al., 2005). Research has found that among parents of high-risk young adolescents, engagement in the FCU and related parenting services was associated with improved parental monitoring (Dishion et al., 2003). The 10 FCU has been found to improve child and family functioning despite family and neighborhood risk factors (Gill et al., 2008). It has been used not only to reduce child conduct and interpersonal problems, but also to improve maternal depression, parental involvement, and positive parenting practices (Shaw et al., 2009). For families of toddlers, the FCU has been found to reduce disruptive behavior and increase maternal involvement, especially in families with children at higher risk for conduct problems between ages 2 and 4 (Shaw et al., 2006). Within the Family Check-up, one way researchers have attempted to adapt the family intervention for ECGs is to individualize the interventions based on FCU assessment information (Dishion & Stormshak, 2006; Yasui & Dishion, 2007). Adaptive intervention allows for (a) altering intervention components to fit the needs and presenting problems of an individual, and (b) evaluating unique risk and protective factors while determining an appropriate intervention (Collins, Murphy, & Bierman, 2004; Yasui & Dishion, 2007). This may entail adapting intervention targets and treatment considerations, or further treatment options. By utilizing data from tailored assessments in the FCU as decision-making tools, interventionists are able to adapt the intervention model to be culturally sensitive (Yasui & Dishion, 2007). While some research has demonstrated the effectiveness of the FCU on behavioral outcomes with families of various ECGs in the context of larger populations, there is to date no research that has evaluated client perceptions of the FCU along variables that predict long-term treatment continuation and efficacy. Such research may provide additional evidence that the FCU is an equally valid intervention for ECGs (Bernal et al., 11 2009; Bernal & Scharrón-del-Rio, 2001; U.S. Department of Health and Human Services, 1999). Moreover, no research has examined client satisfaction along the dimensions of client perceptions of the FCU intervention, therapist interpersonal qualities, and multicultural competence. Study Purpose In an effort to contribute to research on EBPP with ethnoculturally diverse populations and in the spirit of the American Psychological Association’s efforts to advance multicultural research and practice (American Psychological Association, 2003; American Psychological Association Presidential Task Force on Evidence-Based Practices. 2006), the principal goal of this study is to assess whether the FCU intervention is perceived equally by ethnic majority parents and ethnocultural parents who have participated in the FCU intervention. To do this, I measured three constructs via a survey: (a) parent ratings of their experience of the FCU intervention; (b) parent ratings of family consultant interpersonal qualities (expertness, attractiveness, trustworthiness); and (c) parent evaluation of family consultant multicultural competence. Additionally, I obtained independent observer ratings of the following family consultant, treatment, and general characteristics: (a) consultant FCU consistent behavior, (b) consultant use of Motivational Interviewing strategies, (c) consultant interpersonal qualities, (d) general interpersonal observations, (e) consultant multicultural competence, and (f) overall parent response to treatment. 12 More specifically, in order to examine parent perceptions of the FCU intervention and analyze the influence of family contextual factors in the FCU with ethnically diverse families, I developed parent self-report and observational coding measures (DeVellis, 2003; Weisz et al., 2005; Worthington & Whittaker, 2006). Validating the parent self- report and coding measures allowed me to analyze group differences of parent perceptions of the intervention, their family consultant interpersonal qualities, and their family consultant’s multicultural competence. By examining ethnocultural group differences in parent perceptions of family consultant interpersonal qualities and the FCU intervention, this study may help inform how to strengthen the FCU by enhancing its cultural sensitivity for ethnoculturally diverse families, as well as add to the literature on EBPP with ethnoculturally diverse populations. In order to measure these constructs reliably, preliminary analyses evaluated measure reliability and validity of the parent and coder measures. I also conducted preliminary analyses to examine treatment fidelity in order to examine if family consultants adhered to the FCU intervention model. Research Design The research design of this study is a quantitative, nonexperimental posttest-only design. There was no experimental or statistical control over the predictor variables and no random assignment of participants who completed the survey (Pedhazur & Pedhazur Schmelkin, 1991). Both within-subjects and between-subjects analyses were conducted. Within-subjects analyses were utilized for measurement validation and treatment fidelity, and between-subject analyses were utilized to examine group differences. Additionally, 13 group differences between survey respondents and nonrespondents were analyzed to evaluate if the groups were significantly different. Research Question Do parent responses to the FCU differ between ethnic majority and ethnoculturally diverse parents? That is the principal research question of this study. More specifically, are there statistically significant differences in the way that ethnoculturally diverse parents and ethnic majority parents experience the FCU intervention? Parents’ experience of the intervention will be examined by both parent and observer ratings of (a) the FCU intervention; (b) family consultant qualities (expertness, attractiveness, and trustworthiness); and (c) family consultant multicultural competence. I hypothesize that ethnocultural parents will rate their experience with the FCU intervention significantly differently than ethnic majority parents, and that observer ratings will corroborate parent reports of their experiences. 14 CHAPTER II METHODS Participants Families Participants in this study were parents who completed FCU feedback sessions during their first year of participation in Project Alliance 2 (PAL-2), a longitudinal study of the Child and Family Center that utilized the FCU with middle school families in Portland, Oregon (Kavanagh et al., 2006). After the feedback sessions of all PAL-2 families participating in the FCU, the parent program impressions survey (PROIM; see Appendix A) was sent to them along with a cover letter and self-addressed, stamped envelope for returning their surveys. Parents were asked to voluntarily complete the measure used in this dissertation study. Families were compensated with $10 gift cards for returning their surveys via mail and consented to their information being used for research purposes. A parent survey was sent to the 157 families who participated in the Family Check-Up, and 95 returned a completed survey, a response rate of 60.5%. Parent responses were collected via a fill-in bubble survey, which was scanned electronically to import data into an SPSS file. The SPSS file was scanned by Child and Family Center data-management specialists. Two families did not report ethnicity; therefore, they were excluded from analyses, resulting in a sample size of 93 for the parent survey. Therefore, 15 parent responses for 59% of the families who completed the FCU were analyzed for parent perceptions of the intervention. Participating parents’ demographic information is presented in Table 1. Ethnicity of participating parents was categorized as European American (EA) and Ethnocultural (EC), which included African American, American Indian/Native American, Latino/Hispanic, Asian American/Pacific Islander, and Multiethnic parents. TABLE 1. Participant Family Demographics a Two parents did not disclose their ethnicity, so they were excluded from analyses. Observational Coders Observational coders (n = 5) were female graduate students in their first (n = 1), second (n = 1), third (n = 2), and fifth year (n = 1) of study in the Counseling Psychology program at the University of Oregon. Their therapy experience ranged from 1-4 years (mean = 1.95 years). Two coders held master’s degrees in a psychology-related field Parent survey sample Observational coding sample n % n % Ethnicity European American Ethnocultural 39 41 42 31 54 57 94 69 Total Child Gender Male Female 93 37 36 98 a 50.5 49.5 136 70 66 100 51.5 48.5 16 (Couples and Family Therapy), and all held bachelor’s degrees in a psychology-related field. Coders varied in ethnicity. One coder identified as European American, two as multiethnic Latino/European American, one as multiethnic Asian/European American, and one as African American. Overall Procedure for Coding Family feedback sessions were routinely taped for research purposes as part of the Project Alliance-2 protocol and were transferred to DVD to be stored for future data analysis purposes at a secure location at the CFC. These videotaped feedback sessions were already collected, and participants had previously consented to their use for research purposes as part of their participation in Project Alliance-2. Coder Training Coders were recruited by sending recruitment emails to the Counseling Psychology program at the University of Oregon. Coders were trained during two 3-hour training meetings. During the first training meeting, coders were introduced to the purpose of the study, the key concepts of the Family Check-Up Intervention, Motivational Interviewing, and principles of multicultural competence. Observational coders were trained to evaluate process-level skills and fidelity of the intervention by recognizing adherence to and delivery of the principles and components of the feedback session. They also received the Feedback Observer Global Impressions Ratings (FOGI) coding system 17 and manual. The second session included a practice session in which the entire group of coders reviewed segments of video and discussed the details of the coding rating system. Observational Coding Procedure: Pilot Phase Videotaped Family Feedback Sessions from a similar grant project within the same research center, utilizing the same treatment model, were used to train the team of observational coders to use the FOGI. Each coder rated three to four family feedback sessions and their ratings were compared for reliability with the most experienced coder. Reliability ratings ranged from 76% to 92%, with only two ratings falling below 80%. Ratings on items below 80% were discussed as a team and item descriptions edited for clarity so the coding team could achieve 80% interrater reliability. Observational Coding Procedure: Research Study All family feedback sessions were coded, not just those of parents responding to the parent survey, in order to be able to compare data from respondents and nonrespondents. Of the 157 completed FCU feedback sessions, 136 were coded by the observational coding team using the Feedback Observer Global Impressions Ratings (FOGI). Eighteen videotaped sessions of the total sample were not viable (i.e., had no sound, would not play, or were not recorded properly). Three tapes were from the second year of data collection and mistakenly included in the first data set. Additionally, 23% of the sessions (n = 32) were coded twice for reliability. Of those coded, 29 sessions were in 18 Spanish (21% of the sample) and were coded by the only bilingual coder. Additionally, 57% (n = 78) of the sessions involved one parent, and 43% (n = 58) involved two parents. During the coding phase, coder ratings were closely monitored for interrater reliability. Twenty-three percent of each coder’s ratings were compared to another coder’s ratings of the same family feedback session to examine agreement. Agreement was considered to be within a +/- 1 rating. Coders maintained 80% to 100% agreement during the coding project. Only on four occasions were ratings of 76-78% obtained, and divergent items were discussed during weekly coding meetings until the coding team came to agreement on item definitions and ratings. Coder ratings were entered via a computer-based data-entry system, which created an SPSS file as output, after being checked for errors. Family Consultants Family consultants in the PAL-2 intervention all had a bachelor’s degree, with the exception of one. Parent consultant ethnicity was closely matched to that of participating families. Training and supervision of the parent consultants were ongoing throughout the course of data collection in PAL-2. Consultants followed a written manual, were trained via didactic instruction and role-playing, and received videotaped supervision throughout the intervention (Dishion et al., 2003). Family consultants, serving as research intervention staff, previously consented to their work being used for research and training purposes. Family consultants were informed of the purposes of gathering parent impressions and coding feedback sessions. 19 Family consultants met with research staff via videoconference and in person to discuss the purposes of this study. Five female family consultants were observed delivering the FCU intervention. Family consultants varied in ethnicity. Two consultants were African American (34.5% of sessions), two were European American (36.1% of sessions), and one was Latin American (29% of sessions). Measures Existing Data Data on participant race, ethnicity, and child gender had been previously collected via parent surveys through the PAL-2 research study. Parent Report Measures The parent program impressions survey (PROIM; see Appendix A) is comprised of three measures to gather parent perceptions of (a) the FCU intervention; (b) family consultant interpersonal qualities (expertness, attractiveness, and trustworthiness); and (c) family consultant multicultural competence during their family feedback sessions. It consists of a 5-item measure developed specifically to evaluate parent impressions of the Family Check-Up (FRAMES) and two already validated measures: (a) the Counselor Rating Form-Short (CRF-S; Corrigan & Schmidt, 1983) to measure parent perceptions of family consultant interpersonal qualities (expertness, attractiveness, and trustworthiness); and (b) an adapted version of the Cross-Cultural Counseling Inventory-Revised (CCCI-R; 20 LaFromboise et al., 1991) to measure parent perceptions of family consultant multicultural competence. Descriptive statistics and interscale correlations for the parent survey and observational coder system are provided in Table 2. Correlation coefficients between the measures on the parent survey (PROIM) demonstrate that correlations were all statistically significant and were greater than or equal to r = .56, p < .01. This demonstrates high convergent validity of parent responses on the PROIM measures, indicating the three subscales may reasonably function as one dimension to measure parent impressions of the intervention. Family Check-Up Scale (FRAMES) The first portion of the parent survey (PROIM) consists of five items based on the foundation of the FCU (Dishion & Kavanagh, 2003), referred to as the FRAMES measure. These questions serve as a measure of parent response to the FCU intervention. These items are scored on a 5-point Likert scale ranging from ―Not very much‖ to ―Very much‖ to measure parent perceptions of family consultant behavior consistent with the model during the feedback session. Question items include ―My family consultant: provided helpful feedback about my child; provided useful advice; provided realistic ideas for making changes; understood my situation; and inspired me to make changes.‖ Development of the FRAMES measure followed suggested scale-development guidelines (DeVellis, 2003; Worthington & Whittaker, 2006). The item pool for the FRAMES measure was reviewed by experts in the FCU model, including Drs. Thomas 21 TABLE 2. Parent Survey (PROIM) and Observational Coder Ratings (FOGI) Descriptive Statistics and Correlations Parent Survey (PROIM) Observational Coder Ratings (FOGI) FRAMES CRF-S CCCI-R FCU MISTS CRF-S CCCI-R INT OBS Parent Resp PROIM FRAMES 4.02 .79 .91 1 .56 ** .79 ** .33 ** .26 * .28 * .23 * .20 -.03 CRF-S 6.39 .93 .97 .56 ** 1 .70 ** .37 ** .30 ** .33 ** .22 * .32 ** .12 CCCI-R 4.39 .57 .90 .79 ** .70 ** 1 .27 ** .20 .23 * .16 .14 .03 FOGI FCU 3.94 .59 .98 .33 ** .37 ** .27 * 1 .86 ** .80 ** .73 ** .82 ** .54 ** MISTS 3.72 .77 .94 .26 * .30 ** .20 .86 ** 1 .68 ** .64 ** .82 ** .55 ** CRF-S 4.44 .56 .94 .28 * .33 ** .23 * .80 ** .68 ** 1 .71 ** .74 ** .36 ** CCCI-R 4.17 .58 .92 .23 * .22 * .16 .73 ** .64 ** .71 ** 1 .74 ** .41 ** INT OBS 4.14 .82 .94 .20 .32 ** .14 .82 ** .82 ** .74 ** .74 ** 1 .68 ** Parent Resp 4.16 .66 .94 -.03 .12 .03 .54 ** .55 ** .36 ** .41 ** .68 ** 1 Note. PROIM = Parent Program Impressions Survey; FOGI = Feedback Observer Global Impressions Ratings; FRAMES = Family Check-Up Rating Scale; CRF-S = Counselor Rating Form-Short (Corrigan & Schmidt, 1983); CCCI-R = Cross-Cultural Counseling Inventory-Revised (LaFromboise et al., 1991); FCU = Family Check-Up Coder Rating Scale; MISTS = Motivational Interviewing Supervision and Training Scale (Madson, Campbell, Barrett, Brondino, & Melchert, 2005); INT OBS = Coder Interpersonal Observations Scale; Parent Response = Coder observation of parent response scale. N = 81. Missing data excluded listwise. *Correlation is significant at the 0.05 level (2-tailed). **Correlation is significant at the 0.01 level (2-tailed). 22 Dishion, Elizabeth Stormshak, Benedict McWhirter, Alison Ball, and Erika Lunkenheimer of the Child and Family Center, to determine if the items reflected the constructs of interest, to minimize redundancy, to limit the number of items, and to ensure that items were only rating one construct at a time (DeVellis, 2003; Fowler, 2002). Additionally, FCU model experts reviewed the rating format and determined that it should be consistent with other CFC measures by using a Likert-type scale for rating, as families are familiar with this system. The measure was converted into a teleform that could be electronically scanned for data entry. As the FRAMES measure was developed based on the theory and framework of the FCU, and was reviewed by five expert research scientists who use the FCU, the measure has adequate content validity (DeVellis, 2003; Hoyt, Warbasse, & Chu, 2006). Establishing criterion-related validity for the FRAMES measure is difficult, as there is not currently any commonly accepted measure for evaluating the FCU, nor any measure for parent rating of the FCU (DeVellis, 2003). This measure was developed both in English and in Spanish so that both English- speaking and Spanish-speaking participants could respond to the survey. The Spanish survey was translated by three bilingual professionals: one graduate student, one research assistant, and one graduate student/research assistant. Translations were reviewed by a panel of bilingual professionals, including interventionists, research assistants, and bilingual graduate students, for agreement on content, semantic, and technical equivalence between the English and Spanish surveys (Erkut, Alarcón, Garcia Coll, Tropp, & Vázquez Garcia, 1999; Geisinger, 1994; Matías et al., 2003). 23 Descriptive statistics for the FRAMES measure are presented in Table 2. Interscale correlations between the FRAMES and other parent survey measures were all statistically significant and were greater than or equal to r = .56, p < .01. This indicates strong correlations between this newly developed measure of FCU model consistent behavior and the two previously validated measures comprising the parent survey (PROIM). Parent responses on the FRAMES measure were also highly internally consistent, indicating strong measure reliability (α = .91). Counselor Rating Form-Short (CRF-S) The CRF-S (Corrigan & Schmidt, 1983) consists of 12 Likert-type items rated on a 7-point scale that measure three 4-item subscales. The subscales measure expertness, attractiveness, and trustworthiness. According to Strong (1968), expertness is defined as clients’ beliefs that their counselor has the knowledge and skills to help them deal effectively with their problems. Attractiveness refers to clients’ feelings of liking, admiration, and desire to be similar to their counselor. Trustworthiness is defined as clients’ perceptions of their counselor’s sincerity, openness, and absence of motives for personal gain (Strong, 1968). Higher scores on each of the Likert subscales correspond to higher ratings of the perceived characteristic. Considered brief and easy to administer, the CRF-S requires only an eighth-grade reading level (Ponterotto & Furlong, 1985). Although the three CRF-S subscales can be used as individual dependent measures, several previous studies have supported the use of the CRF-S as a global measure of perceived counselor competence because of its high interscale correlations. 24 Table 3 presents results from descriptive analyses conducted on the CRF-S and subscales, along with Cronbach’s alpha estimates of internal consistency with this sample, in comparison with previously reported reliability estimates for subscales. Results indicate strong reliability of responses on this measure, as well as the subscales, with this sample. TABLE 3. Descriptive Statistics for the CRF-S: Parent Survey and Original Study Data Note. CRF-S = Counselor Rating Form-Short (Corrigan & Schmidt, 1983). Original development of CRF-S reported split-half reliabilities of internal consistency in LaCrosse and Barak (1976). Additional reliabilities were replicated by a study with a sample of college students viewing expert therapists as well as utilizing outpatient client ratings (Corrigan & Schmidt, 1983). Correlation coefficients were also computed between the total measure and subscales on the CRF-S. Results presented in Table 4 show all correlations were statistically significant and were greater than or equal to r = .80, p < .01. This demonstrates high intrascale correlations for responses on this measure, indicating the three subscales may reasonably function as one scale with this sample. Additional descriptive statistics for the CRF-S scale are presented in Table 2. Interscale correlations between the CRF-S and other parent survey measures were all statistically significant and were greater than or equal to r = .56, p < .01. This indicates Scale Parent survey Original sample range Replication study range Mean SD α Attractiveness Expertness Trustworthiness Total 6.53 6.19 6.46 6.39 .92 1.05 .94 .93 .97 .92 .94 .97 .85 .87 .91 .89-.93 .85-.94 .82-.91 25 TABLE 4. Correlations Between CRF-S Total Scale and Subscales for Parent Responses Scale CRF-S Total CRF-S Attractiveness CRF-S Expertness CRF-S Trustworthiness Total 1 .96** .93** .98** Attractiveness .96** 1 .80** .95** Expertness .93** .80** 1 .86** Trustworthiness .98** .95** .86** 1 **Correlation is significant at the 0.01 level (2-tailed). strong correlations between this validated scale of interpersonal influence or perceived counselor competence, and the other measures comprising the parent survey (PROIM). Cross-Cultural Counseling Inventory-Revised (CCCI-R) The third portion of the parent survey consisted of items from the Cross-Cultural Counseling Inventory-Revised (CCCI-R; LaFromboise, Coleman, & Hernandez, 1991). The CCCI-R is a 20-item, 6-point Likert-type measure with response options ranging from 1 (―Strongly disagree‖) to 6 (―Strongly agree‖). The CCCI-R was originally designed for observers to assess cross-cultural counseling competence. The measure is based on the cross-cultural counseling competencies identified by the APA Division 17 Education and Training Committee (Sue et al., 1982). The CCCI-R consists of items representing three areas: cross-cultural counseling skill, sociopolitical awareness, and cultural sensitivity. CCCI-R scores range from 20 to 120, with higher scores corresponding to higher ratings of cross-cultural counseling competence. The measure is reported to have good content, construct, and criterion-related validity (LaFromboise et al., 1991; Sabnani & Ponterotto, 1992). In this study, the CCCI-R was adapted to a 5-point Likert scale to improve reliability with CFC rating systems. Additionally, items 26 representing each of the three areas of cultural competency were chosen for the parent survey; however, only eight items were used, as items that did not load highly onto the three factors on the CCCI-R were dropped in order to limit the length of the parent measure. One item specifically addressing respect of parenting was added as well. Language was also modified so that it could be completed by clients. For example, the original item of ―Counselor understands the current sociopolitical system and its impact on the client‖ (LaFromboise, et al., 1991) was modified as ―My family consultant is aware of barriers that affect me and my family (for example: racism, finance, transportation, etc).‖ The entire measure was used in the observational coder rating system. Descriptive analyses for the adapted CCCI-R measure are presented in Table 2. Interscale correlations between the CCCI-R Adapted and other parent survey (PROIM) measures were all statistically significant and were greater than or equal to r = .70, p < .01. This indicates strong correlation between this validated and adapted scale of counselor multicultural competence and other measures comprising the parent survey. A calculation of internal consistency also indicates strong reliability with this sample (α = .90; original measure alpha = .95; LaFromboise et al., 1991). Observer Ratings The Feedback Observer Global Impressions Rating System (FOGI; see Appendices B and C) was developed to code family feedback sessions and includes items to assess family consultant use of the FRAMES model as well as the general process of 27 intervention consistent with the Family Check-Up model. As no psychometric information exists for the newly developed FRAMES scale, the coding measure ratings were used to establish validity of the parent rating scale, and vice versa. The following six dimensions are included in the observer rating system: (a) FCU knowledge and behavior; (b) motivational interviewing skills; (c) family consultant interpersonal qualities (expertness, attractiveness, trustworthiness); (d) multicultural competence; (e) general interpersonal observations, and (f) general impressions of parent response. Descriptive statistics on the FOGI measures are presented in Table 2. Correlation coefficients computed between measures on the FOGI show all correlations were statistically significant and were greater than or equal to r = .36, p < .01. This demonstrates moderate to strong interscale correlations on this newly developed system for coder observations. Rating FCU Knowledge and Behavior (FCU) The observer rating measure used in this study was inspired by the Fidelity of Implementation Rating System (FIMP), an observation-based measure to assess adherence to the Oregon model of Parent Management Training (Forgatch, Patterson, & DeGarmo, 2005) at the Oregon Social Learning Center. The FIMP evaluates five dimensions of adherence to the Oregon Model of Parent Management Training: (a) knowledge, (b) structure, (c) teaching skill, (d) clinical skills, and (e) overall effectiveness. The FIMP is also based upon other Oregon Social Learning Center observational systems: Therapist Performance Observational System (TPOS; Reid et al., 28 1979) and the Therapy Process Code (TPC; Chamberlain et al., 1986). In order to make the measure consistent with the Family Check-Up, I used the FIMP as a basis for developing the family feedback session observer rating system. The coding measure developed evaluates family consultant use of the FRAMES underpinnings of the FCU similar to the FIMP scale measuring knowledge and proficiency in PMTO. In order to make the measure more consistent with the FRAMES model, I used an existing observational coding system previously employed by the CFC (Feedback Rating Scale: Family Check-Up Coding Project; Birkholz, Patras, & Dishion, 2002) to certify interventionists in their adherence to the FCU intervention. However, this measure is significantly different from the CFC Feedback Rating Scale, as it evaluates model consistent behavior in much more detail. In addition, the measure asks about behavior indicative of case conceptualization in session as well as the general phases that should be included in feedback sessions of the FCU for treatment fidelity purposes. Family consultant knowledge and implementation of Family Check-Up intervention aims (FRAMES) was evaluated via global ratings on quality of (a) feedback items discussed, (b) linking parent comments and questions with feedback, (c) providing advice on behavioral and developmental issues, (d) providing realistic steps for making changes, (e) communicating role of parenting for child behavior change, (f) expressing empathy for parent situation, and (g) supporting client self-efficacy. Results of descriptive analyses for the FCU measure are presented in Table 2. Interscale correlations between the FCU and other FOGI measures were all statistically significant and were greater than or equal to r = .54, p < .01. This indicates strong 29 correlation between this newly developed measure of observed FCU model consistent behavior and the other previously validated, newly developed, and adapted measures comprising the FOGI. Additionally, internal consistency of coder responses on the FCU scale indicated strong reliability of the measure developed for this study (α = .98). Motivational Interviewing Supervision and Training Scale (MISTS) The MISTS (Madson et al., 2005) was used to code for family consultant use of MI skills and MI behavior. The MISTS was designed to rate therapist use of MI skills for supervision and training purposes, as well as treatment monitoring and therapist evaluation. The MISTS was designed for behavior counts of types of therapist responses during sessions as well as global ratings of the quality of therapist responses, MI fidelity, and effectiveness of therapist intervention. Coders watched videotaped FCU family feedback sessions and provided global ratings on MI skills and principles, including questions, reflections, affirming, summarization, eliciting change talk, addressing ambivalence, and rolling with resistance. Each item was operationalized in the training manual written by the authors and included in the coding manual. The MISTS is designed for a 7-point Likert-type scale and was adapted to a 5-point scale to improve reliability. Descriptive statistics for the modified MISTS measure are presented in Table 2. Interscale correlations between the MISTS and other FOGI measures were all statistically significant and were greater than or equal to r = .55, p < .01. This indicates strong correlation between this measure of MI consistent behavior and other coder measures on the FOGI. The original investigation of the MISTS had reliability estimates ranging from 30 p 2 = .41-.81 (Madson et al., 2005). Internal consistency of coder responses on the MISTS measure with this sample indicated strong reliability of the measure developed for this study (α = .94). Counselor Rating Form-Short (CRF-S) Descriptive statistics for the CRF-S (Corrigan & Schmidt, 1983) measure are presented in Table 2. Interscale correlations between the CRF-S and other FOGI measures were all statistically significant and were greater than or equal to r = .36, p < .01. This indicates moderate to strong correlation between this validated scale of interpersonal influence or perceived counselor competence, and other model consistent and interpersonal measures on the FOGI. Table 5 presents results from descriptive analyses conducted on the CRF-S and subscales along with an alpha reliability coefficient of internal consistency with this sample, in comparison with previously reported reliability for subscales. Results indicate strong reliability of coder responses on this scale (α = .94). TABLE 5. Descriptive Statistics for CRF-S Ratings: Parent Survey (PROIM), Observation Coder (FOGI) and Original Measure Statistics Scale Mean FOGI SD α Mean PROIM SD α Original sample α range Replication study α range Attractiveness 4.39 .62 .90 6.53 .92 .97 .85 .89-.93 Expertness 4.30 .72 .91 6.19 1.0 .92 .87 .85-.94 Trustworthiness 4.63 .56 .91 6.56 .94 .94 .91 .82-.91 Total 4.44 .56 .94 6.39 .93 .97 Note. CRF-S = Counselor Rating Form-Short (Corrigan & Schmidt, 1983); PROIM = Parent Program Impressions Survey; FOGI = Feedback Observer Global Impressions Ratings. Original development of CRF-S reported split-half reliabilities of internal consistency in LaCrosse and Barak (1976). Additional reliabilities were replicated by a study with a sample of college students viewing expert therapists as well as utilizing outpatient client ratings (Corrigan & Schmidt, 1983). Item range on the FOGI was 1-5. Item range on the PROIM was 1-7, which accounts for differences in means. 31 Correlation coefficients were also computed between the total scale and subscales on the CRF-S. Results presented in Table 6 show all correlations were statistically significant and were greater than or equal to r = .60, p < .01. This demonstrates high intrascale correlations for responses. TABLE 6. Correlations Between CRF-S Total Scale and Subscales for FOGI Responses Scale CRF-S Total CRF-S Attractiveness CRF-S Expertness CRF-S Trustworthiness Total 1 .84** .91** .92** Attractiveness .84** 1 .60** .67** Expertness .91** .60** 1 .80** Trustworthiness .92** .67** .80** 1 Note. CRF-S = Counselor Rating Form-Short (Corrigan & Schmidt, 1983). **Correlation is significant at the 0.01 level (2-tailed). Cross-Cultural Counseling Inventory-Revised (CCCI-R) Descriptive analyses for the adapted CCCI-R (LaFromboise et al., 1991) measure are presented in Table 2. Interscale correlations between the CCCI-R Adapted and other FOGI measures were all statistically significant and were greater than or equal to r = .41, p < .01. This indicates moderate to strong correlation between this adapted measure of multicultural competence within the FCU model and other measures comprising the FOGI. Coder ratings on the CCCI-R Adapted measure were also highly internally consistent, indicating strong measure reliability (α = .92, original measure alpha = .95; LaFromboise et al., 1991). 32 Therapist Interpersonal Observations (INT OBS) General items to rate client receipt of the family feedback intervention were also incorporated into the observer rating system in order to rate observer impressions of client treatment receipt. Items included on this measure were modeled after dimensions on the System for Observing Family Therapy Alliances (SOFTA; Friedlander et al., 2006). These items include (a) engagement in session, (b) agreement with family consultant, (c) attitude, (d) resistance, (e) parent self-disclosure, (f) communication of hope, (g) talk time, and (h) overall response of client to feedback. Internal consistency of coder ratings on this new measure indicated strong reliability of the measure developed for this study (α = .94). Descriptive statistics for the INT OBS measure are presented in Table 2. Interscale correlations between the INT OBS and other FOGI measures were all statistically significant and were greater than or equal to r = .68, p < .01. This indicates strong correlation between this newly developed measure of interpersonal observation and other FOGI measures. General Parent Response Similar to items developed to rate the family consultant’s interpersonal alliance with parents, items were developed to rate general parent response to the intervention and consultant. The dimensions used by Friedlander et al. (2006) on the SOFTA were also incorporated into this measure. Descriptive statistics for the General Parent Response measure are presented in Table 2 for either the ―primary caregiver‖ or as an average for 33 the ―primary caregiver‖ and ―alternative caregiver,‖ as designated by PAL-2 research study criteria, if both participated. Interscale correlations between the average of the General Parent Response measure and other FOGI measures were all statistically significant and were greater than or equal to r = .36, p < .01. This indicates moderate to strong correlations between this newly developed measure of observed parent response to the intervention and other FOGI measures. Internal consistency of coder ratings on the measure indicated strong reliability of the measure developed for this study (α = .94). Statistical Power Analysis A priori examinations of statistical power were conducted to describe this dissertation study’s probability of detecting a significant effect when one is present (Cohen, Cohen, West, & Aiken, 2003; Pedhazur & Pedhazur Schmelkin, 1991). The notion of effect size is also an important consideration in analyzing statistical power. For the principal analysis comparing ethnic majority parent ratings to ethnocultural parent ratings, a sample size of n = 85 or greater with two levels of the predictor variable will have a 90% probability (β = .70) of detecting a significant effect size of .25, or a ―medium‖ effect when one is present (Rosenthal & Rosnow, 1991). 34 CHAPTER III RESULTS Data Management and Pre-Analysis Screening Parent Survey (PROIM) Missing Data Missing data were analyzed using PASW missing values analysis. Missing data were found to be missing at random (MCAR; χ2 = 205.46, df = 151, p = .002). Therefore, missing data were estimated and replaced using the expectation maximization (EM) algorithm (Schafer & Graham, 2002; Tabachnick & Fidell, 2007), which forms an estimated population missing data correlation/covariance matrix to predict values based on what would most likely occur in the sample. Outliers Data were examined via boxplots and stem and leaf plots to identify outliers and extreme cases (Mertler & Vannatta, 2005; Tabachnick & Fidell, 2007). Of the 93 respondents’ ratings on 26 items, 72 outliers were identified as lying outside of the 25th-75th percentile from the median of the distribution on boxplots (Mertler & Vannatta, 2005), and 20 extreme item ratings were identified as lying farther from the median for 29 cases. In this case, the distribution of the variables has more outliers and extreme values 35 than a normal distribution; however, they were maintained in the data, as they may provide important information pertinent to the main analysis. Normality Data had significant negative skew and kurtosis on multiple variables, with skew and kurtosis ranging between +/-3 for most variables, and kurtosis for ethnocultural group parents exceeding this range for 10 variables (Mertler & Vannatta, 2005). The differences in ethnocultural group and European American parent responses may be vital to the between-group analyses. Additionally, skewness has been found to only have a slight effect on significance and power (Glass, Peckham, & Sanders, 1972). The Kolmogorov- Smirnov statistic with Lilliefors significance level tested the normality of distributions for both groups and indicated a non-normal distribution, as the test statistic was significant for all variables (Mertler & Vannatta, 2005). Data were transformed in an attempt to correct for non-normality; however, data still violated normality tests for all transformations. Therefore, original data with imputed missing values were used in analyses. Homogeneity of Variance Levene’s test was selected to assess for the assumption of homogeneity of variance between groups, as it is not affected by violations to the assumption of normality (Mertler & Vannatta, 2005). The equality-of-variance assumption for the two groups was violated for three variables; however, this should not be considered ―fatal‖ to this analysis 36 (Mertler & Vannatta, 2005). Examination of between-group variance may be significant to the investigation of ethnocultural differences in parents’ experience of the FCU. Observational Coder Ratings (FOGI) Missing Data ―Unobservable ratings‖ were coded as system missing, as data were often missing due to videotaping problems (i.e., video starting partway into FCU session or ending early) or a participant not being visible on video. Each variable on the FOGI was found to have between 0-49% missing data. However, missing data were analyzed using PASW missing values analysis. Missing data were found to be missing completely at random (MCAR; χ2 = 3741.58, df = 3862, p = .92). While missing data could have been estimated and replaced using the expectation maximization (EM) algorithm (Schafer & Graham, 2002; Tabachnick & Fidell, 2007), which forms an estimated population missing data correlation/covariance matrix to predict values based on what would most likely occur in the sample, missing data were kept in the analysis and simply excluded listwise, as we did not want to predict observable phenomena that were unobservable. Outliers Data were examined via boxplots and stem and leaf plots to identify outliers and extreme cases (Mertler & Vannatta, 2005; Tabachnick & Fidell, 2007). Of the coder ratings on 84 items, 192 outliers were identified as lying outside of the 25th-75th percentile from the median of the distribution on boxplots (Mertler & Vannatta, 2005), 37 and 95 extreme item ratings were identified as lying farther from the median for 136 cases. In this case, the distribution of the variables does have more outliers and extreme values than a normal distribution; however, they were maintained in the data, as they may provide important information pertinent to the main analysis. Normality Grouped data had significant skew and kurtosis on multiple variables, with skew and kurtosis ranging from -3 to +3 for European American and ethnocultural group parents (Mertler & Vannatta, 2005). The Kolmogorov-Smirnov statistic with Lilliefors significance level tested the normality of distributions for both groups and indicated a non-normal distribution, as the test statistic was significant for all variables (Mertler & Vannatta, 2005). Homogeneity of Variance Levene’s test was conducted (p < .05) to assess for equal variance between groups in this sample, as it is a good test for homogeneity and is not affected by violations to the assumption of normality (Mertler & Vannatta, 2005). The equality-of-variance assumption for the two groups was violated for six variables; however, this should not be considered ―fatal‖ to this analysis (Mertler & Vannatta, 2005). This between-group variance may be significant to this investigation. 38 Preliminary Analyses Construct Validity of PROIM and FOGI Measures A modified multitrait-multimethod matrix (MTMM; Campbell & Fiske, 1959; Trochim, 2006) and Pearson correlations were used to assess convergent and discriminant validity and to evaluate the strength of correlations between measures on the PROIM and FOGI. Composite reliability coefficients (Cronbach, 1951, as cited in Helms, Henze, Sass, & Mifsud, 2006) and correlational data for both the overall PROIM survey and FOGI coder rating system were included along with data on individual measures for reliability and validity information. Results of the modified MTMM are presented in Table 7. Estimates of internal consistency reliability (Cronbach’s alpha) are shown on the diagonal (Campbell & Fiske, 1959; Trochim, 2006). The first requisite criterion of an MTMM for demonstrating reliability of measures is that the reliability correlations must be high (Campbell & Fiske, 1959; Rohrer Murphy & Suen, 1999).‖ Results indicate the reliability calculations using Cronbach’s alpha are consistently the highest in the matrix, indicating high internal consistency among the items in each measure. The second requisite criterion of an MTMM for demonstrating convergent validity is that correlations between measures of the same trait measured using different methods (monotrait-heteromethod correlations), or validity estimates, should be significantly greater than zero and demonstrate a strong correlation, as they measure the same concepts (Campbell & Fiske, 1959; Rohrer Murphy & Suen, 1999). Four of five 39 TABLE 7. Multitrait Multimethod Correlations Measure Parent Survey (PROIM) Observational Coder Ratings (FOGI) PROIM FRAMES CRF-S CCCI-R FOGI FCU MISTS CRF-S CCCI-R INT OBS Parent Survey .97 FRAMES .91 CRF-S .56** .97 CCCI-R .79** .70** .90 Observational Coder Ratings .35** .98 FCU .33** .37** .27* .93 MISTS .26* .30** .20 .87** .94 CRF-S .28* .33** .23* .80** .68** .94 CCCI-R .23* .22* .16 .73** .64** .71** .92 INT OBS .20 .32** .14 .82** .82** .74** .74** .94 PAR RESP -.03 .12 .03 .54** .55** .36** .41** .68** Note. FRAMES = Family Check-Up Rating Scale; CRF-S = Counselor Rating Form-Short (Corrigan & Schmidt, 1983); CCCI-R = Cross- Cultural Counseling Inventory-Revised (LaFromboise et al., 1991); FCU = Family Check-Up Coder Rating Scale; MISTS = Motivational Interviewing Supervision and Training Scale (Madson et al., 2005); INT OBS = Coder Interpersonal Observations Scale; PAR RESP = Parent General Response. *Correlation is significant at the 0.05 level (2-tailed). Missing data excluded listwise. N = 81. **Correlation is significant at the 0.01 level (2-tailed). 40 validity estimates are statistically significant with moderate correlations. Validity estimates are presented in bold in Table 7. Results indicate weak convergent validity between the overall parent survey and coder rating system (PROIM and FOGI; r = .35, p < .01). For individual measures, results indicate weak convergent validity between the Family Check-Up measures (FRAMES and FCU; r = .33, p < .01), and the interpersonal influence or general counselor competence measures (CRF-S; r = .33, p < .01). Additionally, the Motivational Interviewing scale (MISTS) is also weakly correlated with the Family Check-Up measure on the parent rating measure, as predicted due to its theoretical relatedness (r = .26, p < .05). Similarly, the Motivational Interviewing measure (MISTS) is also weakly correlated with the interpersonal influence measure (CRF-S) on the parent survey, which may indicate theoretical relatedness of underlying concepts (r = .30, p < .01). Unfortunately, the CCCI-R adapted measure failed to demonstrate convergent validity between the parent and coder ratings (r = .16, p > .05), which could be expected, as the CCCI-R measures skills specific to multicultural competence rather than the general counseling effectiveness that the aforementioned instruments measure. Therefore, coefficients in the validity diagonal are significantly different from zero and high enough to indicate weak convergent validity for (a) the overall parent survey and coder rating system (PROIM & FOGI), (b) the Family Check- Up measures (FRAMES & FCU) and Motivational Interviewing measure (MISTS), and (c) the consultant interpersonal influence measures (CRF-S). Unfortunately, while there is evidence for weak convergent validity, the correlations are only weak to moderate in 41 strength rather than strong. Additionally, no evidence of convergent validity was found for the adapted Cross-Cultural Counseling Inventory (CCCI-R). The third requisite criterion of an MTMM for demonstrating discriminant validity is that correlations of the same constructs should be higher than correlations between measures of different constructs (Campbell & Fiske, 1959; Rohrer Murphy & Suen, 1999). Therefore, the validity estimates should be stronger than heterotrait-heteromethod correlations. This criterion holds true for the FCU measures. The interpersonal influence measure (CRF-S) is highly correlated with all measures across the parent survey and coder rating system. This is not necessarily unexpected given the global relatedness of interpersonal influence with other intervention skills (e.g., motivational interviewing, multicultural competence, therapeutic alliance). However, this indicates that the CRF-S fails to evidence discriminant validity with this sample. Additionally, the counselor multicultural competence measure (CCCI-R Adapted) also fails to demonstrate discriminant validity, as it is more highly correlated with all other measures (e.g., model consistent behavior, interpersonal influence, and motivational interviewing) than with itself across both the parent survey and coder rating system (Campbell & Fiske, 1959; Rohrer Murphy & Suen, 1999; Trochim, 2006). In summary, analyses of measure validity provide weak support for convergent validity of the overall parent survey and coder rating system (PROIM & FOGI), the FCU measures on both the parent survey and coder rating system (FRAMES & FCU), the Motivational Interviewing observational measure (MISTS), and the measure of interpersonal influence on both the parent survey and coder rating system (CRF-S). 42 Convergent validity was not found for the measure of multicultural competence between parent and coder ratings (CCCI-R). Additionally, only parent and coder measure of the FCU demonstrated discriminant validity. Treatment Fidelity FCU delivery was examined in order to monitor delivery of the intervention and to demonstrate reliability and validity of the behavioral intervention while controlling for internal threats to validity (Bellg et al., 2004; Hogue, Liddle, & Rowe, 1996; Mowbray, Holter, Teague, & Bybee, 2003; K. Zvoch, personal communication, 2008). Measures regarding treatment delivery, including parent-report of treatment receipt and observational coder ratings of treatment implementation (i.e., PROIM FRAMES, FOGI FCU, and FOGI MISTS), were analyzed as quantified measures of treatment fidelity. Correlations among these scales were analyzed and are presented in Table 8. Results indicate significant moderate positive correlations between parent and coder ratings of family consultant delivery and parent receipt of the FCU treatment model on the PROIM FRAMES and FOGI FCU scales (r = .315, p < .01). Additionally, the subscale of family consultant FCU knowledge and behavior and effectiveness on the FOGI FCU measure was significantly and moderately positively correlated with parent ratings of consultant behavior (r = .294-306, p < .01; see Table 8). Using Cohen’s Kappa to account for chance agreement (Cohen, 1960; Fleiss, 1971; Landis & Koch, 1977), I evaluated interrater agreement between parent and coder 43 TABLE 8. Treatment Fidelity Correlations Parent survey (PROIM) Observational coder rating system (FOGI) FRAMES FCU FCU knowledge and behavior FCU effectiveness FCU/MI TRAPS MISTS MISTS OARS MISTS SPIRIT Parent survey FRAMES 1 Observational coder ratings FCU .32 ** 1 FCU knowledge and behavior .29 ** .93 ** 1 FCU effectiveness .31 ** .89 ** .75 ** 1 FCU/MI TRAPS .19 .75 ** .51 ** .61 ** 1 MISTS .25 * .86 ** .73 ** .84 ** .71 ** 1 MISTS-OARS .19 .77 ** .68 ** .76 ** .59 ** .94 ** 1 MISTS-SPIRIT .26 * .86 ** .72 ** .82 ** .74 ** .97 ** .83 ** 1 Note. PROIM = Parent Program Impressions Survey; FOGI = Feedback Observer Global Impressions Ratings; FRAMES = Family Check-Up Rating Scale; FCU = Family Check-Up Coder Rating Scale; FCU/MI TRAPS = FCU subscale on intervention traps for Motivational Interviewing and FCU intervention; MISTS = Motivational Interviewing Supervision and Training Scale (Madson et al., 2005); MISTS OARS = MISTS subscale on reflective listening skills; MISTS SPIRIT = MISTS subscale on spirit of MI. *Pearson Correlation is significant at the 0.05 level (2-tailed). N = 80. Missing data excluded listwise. **Pearson Correlation is significant at the 0.01 level (2-tailed). 44 ratings of treatment implementation and receipt by categorizing agreement as within +/- 1 instead of using perfect agreement. Interrater reliability for observational coders was calculated in a similar fashion. Interrater agreement for parents and coders was found to be Kappa = 0.89, p < .01, 95% confidence interval observed, indicating strong agreement between parents and coders (Fleiss, 1971; Landis & Koch, 1977). This indicates the FCU intervention was delivered as intended, as rated by parents and coders. Differences Between Survey Responders and Nonresponders A multivariate analysis of variance (MANOVA) was conducted on observational coder ratings to examine potential group differences between parents who returned the parent impressions survey and those who did not. This MANOVA was conducted to enhance the generalizability of study findings by reducing the possibility that survey respondents were merely those parents who responded to the intervention differently than survey nonrespondents. Wilk’s test of multivariate significance showed that the predictor variable of parent survey response was not statistically related to the weighted multivariate combination of criterion measures of FOGI ratings, Λ = .98, F (6, 129) = .51, p > .05, η2 = .02. This indicates that parent response or nonresponse to the parent survey caused no statistically significant differences in coder ratings of family consultant behavior or parent response to the intervention. 45 Group Differences in Parent Perceptions of the FCU A multivariate analysis of variance (MANOVA) was performed with ethnocultural group status serving as the predictor variable, and all parent surveys (PROIM) means and coder rating system (FOGI) measures serving as the criterion variables. The predictor variable of ethnocultural group status had two levels: (a) European American Parents, and (b) Ethnocultural Parents. Wilk’s test of multivariate significance showed that the predictor variable of parent ethnicity was not statistically related to the weighted multivariate combination of criterion measures (PROIM and FOGI measure ratings), Λ = .87, F (9, 71) = 1.14, p > .05, multivariate η2 = .13, observed power = .52. These results indicate that ethnicity of the parents caused no statistically significant differences between parent responses to the intervention, as measured by multiple methods and multiple raters. A second multivariate analysis of variance (MANOVA) was performed with ethnocultural group status serving as the predictor variable, and Family Check-Up means, interpersonal influence, and multicultural competence measures on both the parent survey (PROIM) and the coder rating system (FOGI) serving as criterion variables. The predictor variable of ethnocultural group status had two levels: (a) European American Parents, and (b) Ethnocultural Parents. Wilk’s test of multivariate significance showed that the predictor variable of parent ethnicity was not statistically significantly related to the weighted multivariate combination of criterion measures, Λ = .88, F(6, 74) = 1.68, p > .05, multivariate η2 = .12, observed power = .60. These results indicate that ethnicity of the parents caused no statistically significant differences between parent responses to the 46 intervention, as measured by multiple methods and multiple raters. Function and structure coefficients are presented in Table 9 to define each variable in the multivariate analysis. TABLE 9. MANOVA Structure and Function Coefficients Standardized discriminant function coefficients Structure coefficients Parent survey FRAMES -0.14 -0.40 CRF-S 0.66 -0.01 CCCI-R -0.73 -0.40 Observational coder ratings FCU 0.59 -0.30 CRF-S -0.15 -0.41 CCCI-R -1.08 -0.72 Note. FRAMES = Family Check-Up Rating Scale; CRF-S = Counselor Rating Form-Short (Corrigan & Schmidt, 1983); CCCI-R = Cross-Cultural Counseling Inventory-Revised (LaFromboise et al., 1991); FCU = Family Check-Up Coder Rating Scale. Descriptive statistics for parent responses on the parent survey and observational coder ratings by parent ethnicity are presented in Table 10. Differences in group means reveal the largest variability in responses across measures—e.g., in parent responses to their perceptions of the FCU intervention. Differences in Parent and Coder Ratings of Multicultural Competence As the CCCI-R adapted measure did not demonstrate validity as part of the overall FOGI rating system, and yet is theoretically related to the predictor variable of ethnicity, a follow-up MANOVA between parent responses and coder ratings on the CCCI-R measure (criterion variable) was conducted with parent ethnicity as a predictor variable. Wilk’s test of multivariate significance showed that the predictor variable was statistically significantly related to the weighted multivariate combination of criterion measures, 47 TABLE 10. Group Means Note. FRAMES = Family Check-Up Rating Scale; CRF-S = Counselor Rating Form-Short (Corrigan & Schmidt, 1983); CCCI-R = Cross-Cultural Counseling Inventory-Revised (LaFromboise et al., 1991); FCU = Family Check-Up Coder Rating Scale; MISTS = Motivational Interviewing Supervision and Training Scale (Madson et al., 2005); INT OBS = Coder Interpersonal Observations Scale; Parent Response = Coder Observation of Parent Response Scale. SD = .57) than European American Parents (M = 3.99, SD = .47) on the FOGI CCCI-R Adapted, F (1, 79) = 5.64, MSE = 1.60, p < .05, observed power = .65. The mean Λ = .92, F (2, 78) = 3.28, p < .05, η2 = .08, observed power = .60. Examination of the standardized discriminant function coefficients (SDFC) used to weight the multivariate composite revealed that coder ratings on the CCCI-R measure (SDFC = 0.87) were most important in forming the function that distinguished between the two predictor groups. Parent ratings on the parent survey CCCI-R Adapted contributed less to the function (SDFC = 0.39). Inspection of the structure coefficients indicated that the observed measures had moderate to strong correlations with the multivariate composite, PROIM CCCI-R Adapted (r = 0.50), FOGI CCCI-R Adapted (r = 0.92; see Table 11). N Ethnocultural parents European American parents Mean difference M n M SD n M SD Parent Survey FRAMES CRF-S CCCI-R Observational Coder Ratings FCU MISTS CRF-S CCCI-R INT OBS PARENT RESP 81 136 48 94 5.29 4.12 6.40 4.45 4.06 3.95 3.73 4.46 4.23 4.19 4.16 .75 .78 1.02 .58 .58 .61 .76 .58 .61 .82 .67 33 42 5.18 3.88 6.39 4.29 3.96 3.92 3.70 4.39 4.04 4.02 4.14 .61 .80 .78 .54 .55 .57 .82 .52 .48 .83 .59 0.11 0.24 0.01 0.16 0.10 0.03 0.03 0.07 0.19 0.17 0.02 48 TABLE 11. MANOVA Structure and Function Coefficients Standardized discriminant function coefficients Structure coefficients Parent survey CCCI-R 0.39 0.50 Observational coder ratings CCCI-R 0.87 0.92 Note. CCCI-R = Cross-Cultural Counseling Inventory-Revised (LaFromboise et al., 1991). Univariate ANOVAs on each of the two measures comprising the multivariate composite revealed statistically significant mean differences between predictor groups on one of the two criterion variables. Ethnocultural Parents had a higher mean (M = 4.28, difference between Ethnocultural Parents and European American Parents on the PROIM CCCI-R Adapted was not statistically significant, F (1, 79) = 1.68, MSE = .53, p > .05, observed power = .25. Alpha was adjusted for multiple testing (i.e., .05/2 = .025) to maintain the probability of type I error at .05. Univariate Analyses of Group Differences In this investigation, a MANOVA was used in the main analysis to create a linear combination of criterion variables and to maximize mean group differences in this multimethod, multirater study (Stevens, 2002; Mertler & Vannatta, 2005). Even though follow-up univariate analyses were unlikely to reveal new information, they were conducted to provide additional descriptive information. One-way analyses of variance (ANOVA) were conducted to evaluate the relationships between ethnocultural status and parent response on the three measures on the parent survey (PROIM). The ANOVAs failed to find significant differences between groups on all three measures, and found that 49 the predictor variable accounted for 0-2% of variance for each measure. Results of the ANOVA on the FCU scale found ethnocultural status accounted for only 2% of variance across responses, F(1,79) = 1.73, p = .19, η2 = .02. Similarly, on the interpersonal influence measure (CRF-S; Corrigan & Schmidt, 1983), F(1,79) = .002, p = .97, η2 = .00, parent ethnocultural status accounted for approximately 0% of the variance in responses. The ANOVA conducted on the parent responses to the adapted multicultural counseling competence measure (CCCI-R; LaFromboise et al., 1991), F(1,79) = 1.68, p = .20, η2 = .02, found parent ethnocultural status accounted for 2% of the variance in parent responses. Group means are provided in Table 10. One-way ANOVAs were also conducted on observational coders’ ratings to evaluate the relationship between ethnocultural status and coder ratings on measures evaluating the family consultant, intervention, and perceived parent response to the intervention. As previously stated, no significant differences were found between groups on coder ratings across measures. On an ANOVA of observational coder ratings of model consistent behavior (FCU), F(1,134) = 0.79, p = .78, η2 = .001, parent ethnocultural status accounted for less than 1% of the variance in coder responses. Similarly, on an ANOVA on coder ratings of use of Motivational Interviewing (Miller & Rollnick, 1991; see also Madson et al., 2005), F(1,134) = .02, p = .88, η2 = .00, parent ethnocultural status accounted for approximately 0% of the variance in coder ratings. For the ANOVA conducted on coder ratings of family consultant interpersonal influence (CRF-S; Corrigan & Schmidt, 1983), F(1,134) = .44, p = .51, η2 = .00, parent ethnocultural status accounted for approximately 0% of the variance in coder ratings as well. For the follow-up ANOVA 50 conducted on coder ratings of consultant use of multicultural counseling skills (CCCI-R; LaFromboise et al., 1991), F(1,134) = 3.27, p = .07, η2 = .02, parent ethnocultural status accounted for approximately 2% of the variance in coder ratings. These findings vary from the between-subjects follow-up MANOVA and the subsequent ANOVA that found statistically significant differences in group means of coder ratings between groups regarding consultant use of multicultural counseling skills, and may be due to differences in sample size and power. The aforementioned MANOVA and follow-up ANOVA included multivariate data from both the parent survey and the observational coder ratings. This ANOVA was performed with only observational coder data, resulting in a larger sample size. For the ANOVA performed on coder ratings of their interpersonal observations of coder behavior and alliance with parents, F(1,134) = 1.21, p = .27, η2 = .009, parent ethnocultural status accounted for approximately 1% of the variance in coder ratings. Lastly, for the ANOVA performed on coder ratings of parent responses to the family consultant in the intervention, F(1,134) = .05, p = .83, η2 = .00, parent ethnocultural status accounted for approximately 0% of the variance in coder ratings. These results confirm that parent and coder ratings of the intervention did not vary by parent ethnicity. In further evaluation of the relationships between parent responses to the intervention by ethnicity, correlational analyses were conducted for parent responses on the parent survey for both European American parents and Ethnocultural parents (see Table 12). Correlation coefficients were transformed into standardized z-scores using Fisher’s r-to-z transformations to statistically compare correlations (See Table 12). None 51 TABLE 12. Correlations and Z-Scores for Parent Survey Ratings (PROIM) Measure European American parents n = 33 Ethnocultural Group Parents n = 48 Difference in Correlations of Parent Ratings FRAMES CRF-S CCCI-R FRAMES CRF-S CCCI-R FRAMES CRF-S CCCI-R Parent survey (PROIM) FRAMES r zr 1 .52** .57 .77** 1.01 1 .61** .70 .79** 1.07 -0.57 -0.26 CRF-S r zr .52** .57 1 .64** .76 .61** .70 1 .74** .94 -0.57 -0.78 CCCI-R r zr .77** 1.01 .64** .76 1 .79** 1.07 .74** .94 1 -0.26 -0.78 Note. PROIM = Parent Program Impressions Survey; FOGI = Feedback Observer Global Impressions Ratings; FRAMES = Family Check-Up Rating Scale; CRF- S = Counselor Rating Form-Short (Corrigan & Schmidt, 1983); CCCI-R = Cross-Cultural Counseling Inventory-Revised (LaFromboise et al., 1991). *Correlation is significant at the 0.05 level (2-tailed). Z-scores calculated via Fisher’s r-to-z transformations (*). **Correlation is significant at the 0.01 level (2-tailed). ***Z-scores ≥ 1.96, p = .05. Missing data excluded listwise. European American SEzr = 0.183. Ethnocultural Group SEzr = = 0.15. 52 of the correlations were statistically significant at z ≥ 1.96, p = .05. Differences between correlations were also tested, and none were statistically significant (See Table 12). In summary, findings suggest that for this highly diverse sample, there were no statistically significant differences in parent response to the FCU intervention due to ethnocultural group status. Results show the FCU intervention was delivered and received as intended, thereby controlling for threats to internal validity due to variable fidelity in implementation. Additionally, the parent-report and observational coder rating scales and measures developed specifically for this investigation of the FCU intervention demonstrated viability for future research. 53 CHAPTER IV DISCUSSION This investigation had two primary goals: The main goal was to examine ethnocultural group differences in parent perceptions of the FCU. The second goal was to develop and validate measures to facilitate the first goal. Therefore, study results include reporting measurement reliability and validity, as well as evaluating group differences in parent perceptions of the FCU intervention. Main Findings The study involved use of multiple raters and methods to evaluate parent perceptions along the following dimensions: (a) the FCU intervention; (b) family consultant qualities (expertness, attractiveness, and trustworthiness); and (c) family consultant multicultural competence. The study also evaluated independent observational coder perceptions of family consultant and intervention characteristics, including (a) consultant FCU consistent behavior, (b) consultant use of Motivational Interviewing strategies, (c) consultant interpersonal qualities, (d) general interpersonal observations, (e) consultant multicultural competence, and (f) overall parent response to treatment. Most important, the results of this study reveal no significant differences in responses to the Family Check-Up intervention between ethnocultural group and European American parents along these dimensions. In this case, a lack of significant differences is an important finding. 54 While the study failed to find any significant differences in response between ethnocultural group and European American parents, results indicated observational coder ratings of family consultant multicultural competence discriminated between the two groups. Observational coders rated family consultants who worked with ethnocultural group parents to have significantly greater multicultural competencies than the family consultants who worked with European American parents. This finding could have various interpretations. One explanation is that family consultants working with ethnocultural group parents indeed had higher multicultural competencies than those working with European American parents. This might have been a direct effect of supervisors assigning ethnocultural group parents to more multiculturally skilled consultants, as would be normative and ethical practice in any clinical setting. Another possible explanation is that coders may have perceived family consultant multicultural competence skills as more relevant when rating feedback sessions involving parents from ethnocultural groups, thereby rating consultant multicultural skills as higher whether or not they actually were. Yet another explanation may be that parent ethnicity and/or ethnic match between a family consultant and parents may have an influence on family consultants’ use of multicultural counseling skills, and so indeed the rating of greater multicultural competencies was accurate and also a clear artifact of an interaction between client and consultant (Constantine, 2001). A final explanation may be that coder ethnicity or ethnocentric bias (positive or negative) may have influenced coder ratings (Yasui & Dishion, 2008). As Yasui and Dishion have commented elsewhere (2008), an underpinning of observational data is that all coders perceive what they see in the same way. It is also possible that coders became more skilled in rating multicultural 55 competence over time. These are issues that should be addressed in future coder training and addressed in future studies using coder data. Our findings are noteworthy for researchers and practitioners using the FCU and other family-centered interventions. The inclusion of parent ratings of the intervention in the main analysis is a major strength of this study, as the FCU had yet to be evaluated by parents along these dimensions. By examining ethnocultural group differences in parent perceptions of family consultant interpersonal qualities and the FCU intervention, this study adds to the empirical literature supporting the FCU with ethnoculturally diverse families in real-world settings. This study provides evidence that the FCU intervention is a generalizable intervention across ethnocultural groups. This study supports the FCU as an intervention that represents the delicate balance between culturally competent practice and scientific rigor (Bernal et al., 2009), as the FCU has been supported as an EBPP by ample previous research (e.g., Connell et al., 2007; Dishion & Andrews, 1995; Gill et al., 2008; Shaw et al., 2009; Shaw et al., 2006; Slavet et al., 2005). It also provides evidence that the FCU, when adapted to a client’s cultural context, is perceived well and similarly across ethnocultural group parents while maintaining adherence to the treatment model. Research on adapted interventions has received much attention recently (Bernal et al., 2009; Hall, 2001; Sue et al., 1999), and this study supports the FCU as an intervention that can be adapted to a clients’ cultural context while maintaining fidelity to the intervention. 56 Measurement In this study, both the parent survey and the observational coder rating system included measures that were newly developed and adapted for this investigation. The results indicated that all measures, whether existing, adapted, or new, demonstrated strong internal consistency reliability with this sample (Pedhazur & Pedhazur Schmelkin, 1991). Additionally, results demonstrated high interrater reliability between parent perceptions and observational coder ratings, as demonstrated by high interrater agreement, which accounted for chance agreement. This indicates that the intervention was delivered and received as intended, and that both parent and coder ratings may be used independently in future research or practice. Among the existing, adapted, and new measures used in this study, all, with the exception of the Cross-Cultural Counseling Inventory-Revised (CCCI-R Adapted; LaFromboise et al., 1991), demonstrated adequate convergent validity. High correlations between measures measuring different constructs may have resulted in weaker convergent validity estimates (Pedhazur & Pedhazur Schmelkin, 1991). Similarly, high correlations between measures may have also resulted in the lack of clear separation between convergent and discriminant validity (Pedhazur & Pedhazur Schmelkin, 1991). While this is most likely not attributable to a lack of reliability, it could be due to methodological variance. Sources of method variance could include coder or parent bias (i.e., personal traits). These potential sources of variability should be considered not only in statistical analyses, but also in future coder training and when designing control measures for respondents, such as social desirability scales. 57 Similarly, all measures, with the exception of the CCCI-R Adapted (LaFromboise et al., 1991) and the CRF-S (Corrigan & Schmidt, 1983), demonstrated good discriminant validity in use with this sample (Pedhazur & Pedhazur Schmelkin, 1991). One plausible explanation for the CCCI-R’s failure to demonstrate discriminant validity is that previous findings have shown this measure is often highly correlated with general counselor effectiveness and interpersonal influence ratings, both of which are theoretical underpinnings of other measures in this study (Constantine, 2002; Fuertes & Brobst, 2002; LaFromboise et al., 1991). This may also explain why the CRF-S failed to demonstrate adequate discriminant validity, as it is considered a measure of global interventionist competence. LaFromboise et al. (1991) posited that a client’s global assessment of an interventionist influences his or her assessment of specific competencies. The theoretical relatedness of these measures might easily explain the lack of discriminant validity. Limitations Several potential limitations appear in this study. First, most demographic data used in this study were previously collected, with limited data available for further evaluation of group differences such as socioeconomic status and level of acculturation, both of which, for instance, play a role in ethnocultural group experience and identity. Future research should consider collecting additional information about family demographics and ethnic and cultural identity in order to evaluate the role of families’ ethnocultural and sociocultural context in potential group differences. Measurement of 58 demographic information in the existing data used in this study lacked some of the nuance and specificity that may have cast better light on our findings. Second, due to the nature of survey research and self-report rating scales, it is quite possible that parent responses were positively skewed, resulting in a ceiling effect for the data (Pedhazur & Pedhazur Schmelkin, 1991). Previous investigations have found this pattern in client responses to the CRF-S (Epperson & Pecnik, 1985). Skewed parent ratings could lead to limited variability in responses, which signifies that this study may not have captured variance in parent and coder ratings that may exist (Pedhazur & Pedhazur Schmelkin, 1991). Future research will have to test this potential limitation. Third, another potential limitation of this study lies in the possibility of type II error resulting in a lack of statistically significant differences between groups. While this is a possibility, it would most likely not be due to inadequate power, as the study had adequate power to detect medium effect sizes. However, it is possible that a larger sample size might have led to stronger statistical power to detect significant effects, if significant group differences were actually present (Cohen, 1988; Pedhazur & Pedhazur Schmelkin, 1991). There may also be some statistical limitations in this study. Using Cronbach’s alpha as a measure of reliability is less rigorous than other methods. Internal consistency reliability, or alpha, is the least rigorous measure of reliability, as it evaluates if a scale measures something consistently by averaging intercorrelations between pairs of items in the scale (Helms et al., 2006). Alpha has been shown to result in inflated reliability when calculated on large numbers of items or items that are highly correlated (Helms et al., 2006 ; Pedhazur & Pedhazur Schmelkin, 1991). Other measures of reliability may be 59 more modest and more rigorous in assessing the internal consistency of a particular measure. Similarly, most of the analyses evaluating measure validity in this study are correlational and thereby provide evidence for measure validity based upon this study’s sample specifically. As such, results may not generalize to other samples (Pedhazur & Pedhazur Schmelkin, 1991). Further research is recommended to evaluate these measures at both the item and factor level to support their validity and generalizability for research and practice. Perhaps the most notable limitation of this study may be the low sample size and, therefore, the potentially low statistical power in the analyses, which, to correct, necessitated the division of ethnicity into two rough groups: parents from ethnocultural groups (that included African American, Latino, Asian American/Pacific Islander, American Indian/Native American, and Multiethnic parents) and European American parents. This way of grouping ethnocultural groups certainly has limitations, such as leading our analyses to necessarily ignore intragroup and intergroup differences that are normative among all ethnocultural groups (and that include, for example, differences in level of acculturation and socioeconomic status, ethnic identity, differences between parents in the same family on cultural practices and ethnic identity, etc.) These individual and group-level differences could be critical factors in explaining findings that proved to be nonsignificant in this study. So, treating these two groups (ethnocultural group parents and European American parents) as homogenous was a limitation in this study (Okazaki & Sue, 1995). 60 Future Directions While the limitations of this study are important to note, it is perhaps of equal importance to emphasize the potential of future research in refining these results, further validating measures, ruling out alternative explanations for findings, and linking these findings to outcomes. Given the strengths and limitations of this study, additional research on the FCU may benefit from recruiting larger samples of ethnocultural group participants in order to investigate potential differences between distinct ethnocultural groups and evaluate whether factors such as acculturation level, language, and socioeconomic status moderate the intervention. Similarly, future research may benefit from training and utilizing more family consultants in order to examine potential therapist effects using multilevel analyses (Pedhazur & Pedhazur Schmelkin, 1991). Furthermore, families were mostly matched with family consultants by ethnocultural status. Sue, Fujino, Hu, Takeuchi, and Zane (1991) have found that ethnic match between clients and clinicians has predicted positive treatment outcomes for Mexican Americans. This may be an interesting focus for future research on the FCU as well. Future research should further evaluate this study’s newly developed and adapted measures with clients in both research and practice settings. Research evaluating these newly developed measures at both the item and factor levels may support the validity of these measures for research and practice. Future outcome-based research may benefit from examining data collected by the interpersonal observation and general parent response measures included in the observational coder rating system. These measures target treatment alliance factors, which are often related to treatment outcomes such as 61 improved parenting practices (Kazdin & Whitley, 2006). Additionally, qualitative information, including parent-identified targets for change and the parents’ stage of change (Prochaska et al., 1992; Prochaska & Norcross, 2001), were identified and rated by observational coders. Future studies may consider evaluating the relationships between treatment alliance, targets for change, and parent stage of change with treatment outcomes at future time points of data collection and intervention. Conclusion Current findings are significant for the delivery and evaluation of the FCU in real-world settings. Most importantly, this investigation failed to detect significant differences in parent perceptions of the FCU along the dimensions of (a) the FCU intervention, (b) family consultant interpersonal qualities, and (c) family consultant multicultural competence between European American and ethnocultural group parents. While these findings are not significant, they are notable for a few reasons. First, the lack of detection of significant differences between groups is important given the observed power of analyses and use of multiple raters and methods to evaluate differences. Additionally, this is the first study on the FCU that included parent perceptions of the FCU, as well as coder ratings of the FCU along these dimensions. Third, findings from this study provide evidence that delivery of the FCU can be adhered to while adapting the intervention to a family’s cultural context. Findings do not suggest that the FCU intervention cannot be generalizable across ethnocultural groups in real-world settings. Lastly, while the measures developed to evaluate the FCU for this study need further validation, they are promising as independent measures to evaluate the FCU from both 62 the parent and observational perspective; both of which could be quite useful in future research, training and practice of the Family Check-up. 63 APPENDIX A PROGRAM IMPRESSIONS SURVEY #1 64 65 66 67 68 APPENDIX B FEEDBACK OBSERVER GLOBAL IMPRESSIONS 69 Feedback Observer Global Impressions Family ID: Family Consultant: Coding Date: Coder Initials: Session Length: FCU Knowledge & Behavior Not At All Somewhat Very Effective Effective Effective Not Observed Explains process of feedback session 1 2 3 4 5 Provides opportunity for parent self- assessment 1 2 3 4 5 Provides rationale to generate interest in feedback 1 2 3 4 5 Explains data-based feedback on profile 1 2 3 4 5 Links feedback to questionnaires and observations 1 2 3 4 5 Links parent comments and questions with feedback 1 2 3 4 5 Communicates role of parenting for child behavior change 1 2 3 4 5 Provides advice on behavioral & developmental issues 1 2 3 4 5 Provides realistic steps for making changes 1 2 3 4 5 Provides summary statement of feedback or profile 1 2 3 4 5 Discusses specific targets and menu for intervention 1 2 3 4 5 Overall Effectiveness of FCU Intervention Not At All Somewhat Very Effective Effective Effective Expresses empathy for parent situation 1 2 3 4 5 Degree of directiveness Low Moderate High 1 2 3 4 5 Effectiveness of directiveness 1 2 3 4 5 Level of difficulty of family situation/experience Easy Moderate Difficult 1 2 3 4 5 General quality of FCU intervention/consultant efforts 1 2 3 4 5 General effectiveness of FCU intervention in creating change 1 2 3 4 5 MI/FCU Traps Not At All Somewhat Very Much Neither Agree /Disagree Provokes resistance (confrontation-denial) 1 2 3 4 5 Focuses on negatives/weaknesses/Areas of change 1 2 3 4 5 Labels 1 2 3 4 5 Prematurely focuses on feedback/issues 1 2 3 4 5 Expresses blame for negative behavior 1 2 3 4 5 70 Motivational Interviewing Supervision & Training Scale Not At All Somewhat Very Effective Effective Effective Not Observed Active Listening Skills Questions 1 2 3 4 5 Reflection 1 2 3 4 5 Affirming 1 2 3 4 5 Summarization: Content, feeling, themes, context 1 2 3 4 5 Spirit of Motivational Interviewing Engaging client in feedback process 1 2 3 4 5 Elicits or reinforces client change talk 1 2 3 4 5 Addresses client’s ambivalence 1 2 3 4 5 Rolling with client resistance 1 2 3 4 5 Collaborating with client 1 2 3 4 5 Supports client self-efficacy 1 2 3 4 5 Overall Family Consultant Rating General effectiveness of facilitating MI 1 2 3 4 5 Multicultural Skills (CCCI-R) Strongly Neither Strongly Disagree Agree/Disagree Agree Not Observed At ease talking with client 1 2 3 4 5 Communication is appropriate for clients 1 2 3 4 5 Aware of professional responsibilities and acts professionally 1 2 3 4 5 Communicates variety of verbal and nonverbal messages 1 2 3 4 5 Elicits variety of verbal and nonverbal responses 1 2 3 4 5 Suggests useful skills 1 2 3 4 5 Values and respects ethnic and cultural differences 1 2 3 4 5 Respects parenting of family 1 2 3 4 5 Demonstrates knowledge about client’s culture 1 2 3 4 5 Discusses problem within client’s cultural context 1 2 3 4 5 Is aware of the barriers that affect family (racism, finances, transportation, etc.) 1 2 3 4 5 Understands the stressors that affect family (poverty, work stress, divorce, etc.) 1 2 3 4 5 71 Therapist Interpersonal Influence (CRF-S) Not Very Very Not Observed Friendly 1 2 3 4 5 Likable 1 2 3 4 5 Sociable 1 2 3 4 5 Warm 1 2 3 4 5 Experienced 1 2 3 4 5 Expert 1 2 3 4 5 Prepared 1 2 3 4 5 Skillful 1 2 3 4 5 Honest 1 2 3 4 5 Reliable 1 2 3 4 5 Sincere 1 2 3 4 5 Trustworthy 1 2 3 4 5 Therapist Interpersonal Observations Strongly Neither Strongly Disagree Agree/Disagree Agree Not Observed Consultant genuinely connects with client 1 2 3 4 5 Consultant creates a ―safe place‖ for therapeutic discussion 1 2 3 4 5 Consultant appears self-aware of verbal and nonverbal behavior, and how it affects client(s) 1 2 3 4 5 Consultant is interpersonally consistent throughout session 1 2 3 4 5 Consultant develops good rapport with client(s) in session 1 2 3 4 5 Consultant responds well to client self- disclosure 1 2 3 4 5 Feedback session felt positive and hopeful. 1 2 3 4 5 72 Parent(s) Response- 1st parent: Relationship______ Not Very Much Neutral/ Very Much Strongly Disagree Somewhat Strongly Agree Not Obs. Non-Verbal Engagement (nodding, mirroring, attending) 1 2 3 4 5 Verbal Engagement (talk, sharing, interest) 1 2 3 4 5 Agreement with family consultant 1 2 3 4 5 Cooperation with family consultant 1 2 3 4 5 Attitude Negative Neutral Positive Resistance (interrupts, confronting, avoidant) 1 2 3 4 5 Expresses ambivalence to change 1 2 3 4 5 Parent self-disclosure 1 2 3 4 5 Communication of hope/ positive future oriented talk 1 2 3 4 5 Overall response of client to feedback 1 2 3 4 5 Argumentative Less Argumentative Change Talk Disagrees Some Agreement Engaged Short Answers Expanded Answers Argues for Change Main Issue: ___________________ Stage of Change Precontemplation Contemplation Preparation Action Maintenance Main Issue: ___________________ Stage of Change Precontemplation Contemplation Preparation Action Maintenance Main Issue: ___________________ Stage of Change Precontemplation Contemplation Preparation Action Maintenance 73 Parent(s) Response- 2nd parent: Relationship______ Not Very Much Neutral/ Very Much Strongly Disagree Somewhat Strongly Agree Not Obs. Nonverbal Engagement (nodding, mirroring, attending) 1 2 3 4 5 Verbal Engagement (talk, sharing, interest) 1 2 3 4 5 Agreement with family consultant 1 2 3 4 5 Cooperation with family consultant 1 2 3 4 5 Attitude Negative Neutral Positive Resistance (Interrupts, confronting, avoidant) 1 2 3 4 5 Expresses ambivalence to change 1 2 3 4 5 Parent self-disclosure 1 2 3 4 5 Communication of hope/ positive future oriented talk 1 2 3 4 5 Overall response of client to feedback 1 2 3 4 5 Argumentative Less Argumentative Change Talk Disagrees Some Agreement Engaged Short Answers Expanded Answers Argues for Change Main Issue: ___________________ Stage of Change Precontemplation Contemplation Preparation Action Maintenance Main Issue: ___________________ Stage of Change Precontemplation Contemplation Preparation Action Maintenance Main Issue: ___________________ Stage of Change Precontemplation Contemplation Preparation Action Maintenance 74 APPENDIX C FEEDBACK OBSERVER GENERAL IMPRESSIONS RATING MANUAL OBSERVATION AND RATING PROCEDURES 75 Feedback Observer General Impressions Rating Manual Observation and Rating Procedures Each feedback session will be observed in its entirety (50-90 minutes). Each tape has an intro prompt with the family ID#, parent consultant code, and date. Coders will fill out the rating form with this information. Coders will then watch a video in approximately 10-minute increments, keeping frequency counts of behavior consistent with the Family Check-Up and Motivational Interviewing as a worksheet to inform global ratings.. After each ~10-minute increment, coders may pause to take notes and will pause to total their tallies of frequency of behavior before continuing onto the next 5-minute video observation. They may stop the tape as desired to take notes as well. Additionally, if something occurs during the session and the coder feels uncomfortable, they are welcome to stop coding and give to the coding project director to code. At the end of the family feedback session, coders will evaluate the family consultant via global impressions codes on Family Check-Up behavior, Motivational Interviewing behavior, skills, and spirit, multicultural competency, and interpersonal qualities, and rate the parent on their general response to the intervention. Coders should also take notes regarding: poor tape quality, poor sound, if cannot see a participant, if the DVD malfunctions, or any other issue affecting ratings. Coders are also encouraged to note the time stamp of items they would like to discuss during coding meetings. Scoring for Sections (except MISTS) Each dimension is rated separately on a 5-point scale of effectiveness or agreement. Not Very Much/Strongly Disagree/Not At All Effective (1, 2). This range is used when the interventionist fails to display competent adherence to key FCU dimensions or does not demonstrate certain process skills. A score of 1 indicates no evidence of competence. A score of 2 indicates some competence or emerging competence. If a behavior or skill is not observed, the rater is to mark ―not observed‖ instead of a 1 or 2. Neutral/Neither Agree nor Disagree/Somewhat Effective (3). This range is used when there is adequate performance but shows problems or mistakes; nevertheless the interventionist manages to recover, move on, change direction, or otherwise perform with competence. Very Much/Strongly Agree/Very Effective (4, 5). This range is used for generally quality work. Scores of 4 and above indicate that competency has been met in regards to adherence to FCU principles and use of process skills. A score of 5 indicates clearly better than adequate. Scoring for MISTS Coders will provide global ratings of active listening skills, spirit of Motivational Interviewing, and overall ratings of the family consultant on the MISTS. Each item is operationalized in the training manual included as written by the authors. The MISTS is designed for a 7-point Likert scale; however, it has been adapted to a 5-point scale for coder ease. Guidelines  Contextual factors relating to the family, the session, and the section of tape being viewed should be considered.  Do not score against hypothetical perfection or hold family consultants to idealized standards (i.e.., be realistic). 76  Within the session family consultant behavior should be coded, not what coders believe could have or should have occurred in session.  Coders should try to ―clear the slate‖ before watching a videotape to avoid being influenced by factors relating to family consultant style or characteristics, session content, or family, either positively or negatively.  Videotape suitability. Consider poor videotape quality and advise supervisor.  The starting point for ratings for each item should be 3. In other words, the rater should begin by assuming that a family consultant should behave adequately. When ratings are assigned a score below 3, the rater should have examples to support their scores. Family Check-Up Knowledge and Behavior 1. Explains process of feedback session: 1= Does not describe process of feedback session before beginning. 3= Brief or vague description of process of feedback session. 5= Clear description of process of feedback session and opportunity for parent to ask questions. 2. Provides opportunity for parent self-assessment: 1= Does not ask parent if they learned anything during the ecological assessment. 3= Briefly mentions or asks if parent learned anything from ecological assessment or project participation thus far. Does not expand on client response. 5= Clearly asks if parent learned anything during the ecological assessment or participation in project thus far. Expands on client response. 3. Provides rationale to generate interest in feedback: 1= Provides feedback without providing any rationale for why categorized as strength or risk factor or without engaging parent 3= Provides feedback while providing some rationale for why information is a strength or risk factor, or only somewhat engages the parent in discussion around rationale. 5= Provides feedback while providing clear rationale for why information is a strength/risk factor, or engages consistently with parent in discussion providing rationale for feedback. 4. Explains data based feedback on profile about behavior and implications: 1= Provides little feedback on family profile, or does not link to assessment, or doesn’t discuss potential consequences of data. 3= Provides some feedback on family profile, or makes a few linkages to assessment sources, or briefly describes the potential consequences and implications of data. 5= Provides lots of feedback and uses family profile; makes linkages to assessment sources, and describes potential consequences and implications of data. 77 5. Links feedback to questionnaires and observations: 1= Does not link feedback or profile information to video observations or project questionnaires (assessment). 3= Mentions relation of feedback and profile information to assessment questionnaires or observations, but does not provide a clear example, meaning, or basis in research for consequences. 5= Clearly links feedback and profile information to assessment questionnaire and video observations with clear and specific examples or discusses meaning or research behind assessments. 6. Links parent comments and questions with feedback: 1= Little response and relating of parental comments or questions to feedback or profile. 3= Some linking of parent comments and questions with feedback, but little building on parent input. 5= Consistent linkage of parent comments and questions to feedback and profile, and builds on parent input. 7. Communicates role of parenting and client responsibility for behavior change: 1= Briefly or barely discusses the role of parenting in changing behavior. 3= Briefly discusses the role of parenting behavior in changing child behavior, but does not emphasize or build on discussion. 5= Detailed examples of discussion of how parenting can influence child behavior. 8. Provides advice on behavioral and developmental issues: 1= Provides little advice on behavioral or developmental issues if they arise in process of feedback, or relies on advice giving when not elicited from parent. (Note: Not observed = NA). 3= Provides some advice on behavioral or developmental issues if they arise. 5= Provides advice on behavioral or developmental issues if they arise, and links it to a target for further intervention to keep feedback session moving and avoid providing treatment. 9. Provides realistic steps for making changes: 1= Does not provide any, or very little suggestions for steps to take to change child behavior or context of family. 3= Provides some vague concrete examples, suggestions, or referrals. Steps seem reasonable, but family consultant does not check with parent to see if steps are realistic. 5= Provides concrete examples of suggestions for change or referral. Dialogues with parent regarding what is possible. 10. Provides summary statement of feedback: 1= Does not summarize feedback, strengths, and risk factors. 3= Vaguely or briefly summarizes feedback, strengths and risk factors. 5= Takes time to summarize feedback, strengths, and risk factors. 78 11. Discusses specific targets or menu for intervention: 1= Provides menu of options, but does not link feedback to options for treatment, does not make recommendations. 3= Provides menu of options, but only vaguely links feedback to options for treatment or recommendations. 5= Provides menu of options, provides feedback based linkages to treatment options, and makes clear recommendations. Overall Effectiveness of FCU Intervention 12. Expresses empathy for parent situation (acceptance, support, empathy): 1= Little verbal expression of understanding of parent and family situation. 3= Some half-hearted expression of understanding of parent and family situation, evidence: of empathy. 5= Clear and consistent verbal expression of understanding and care for parent and family situation. 13. Degree of family consultant directiveness (teaching, confronting, activity level, structure) 1/Low= Family consultant uses primarily open ended questions, presents data and discusses meaning with client without interpreting information, uses OARS to redirect resistance, follows client lead, and provides minimal structure to session. 3/Moderate= Family consultant balances open and closed-ended questions, interprets assessment data but also dialogues with parent about their impressions and meaning of data for them, does not confront client or address resistance overtly but uses more redirections when encounters resistance, initiates topics and follows clients lead, and provides some structure to session. 5/ High= Family consultant uses many closed-ended questions, interprets data for client more than discussing data with client, confronts client, addresses resistance overtly, initiates topics, or teaches skills/provides information without client request. Session highly structured. 14. Effectiveness of directiveness (regardless of degree) 1= Degree of directiveness did not seem effective. 3= Degree of directiveness was somewhat effective. 5= Degree of directiveness was very effective. 15. Level of difficulty of family situation/experience: (Contextual: How hard is family situation/How hard to work with?) 1/Easy= Family/child strengths outnumber weaknesses. Family feedback focused on maintaining positive outcomes. 3/Moderate= Family/child experience some difficulties, but also have strengths. Family feedback is balanced. 5/Difficult= Family/child difficulties outnumber strengths. Family feedback is focused on improving outcomes, avoiding negative outcomes, and harm reduction. 79 16. General effectiveness/quality of FCU intervention: 1= FCU intervention did not appear effective in motivating parent to make parenting changes. 3= FCU intervention appeared somewhat effective in motivating parent to make parenting changes. 5= FCU intervention appeared very effective in motivating parent to make parenting changes. Traps 17. Provokes resistance/Confrontation-Denial Trap 1= Not At All. Client and family consultant do not argue over the need to change, or client argues for change (positive resistance). 3= Somewhat. Some client arguing for status quo, however, also some change-talk from parent. 5= Very Much. Client argues in response to every family consultant statement. Consultant and parent engage in argumentative exchanges. Consultant argues for change, and parent counters arguing for status quo (negative resistance). 18. Focuses on negatives/weaknesses: 1= Not at all. Family consultant delivers strength-based feedback. 3= Somewhat. Family consultant balances strength-based feedback and areas for potential change. 5= Very Much. Family consultant focuses on weaknesses and areas needing change. 19. Labels: 1= Not at all. Consultant de-emphasizes labels regarding child behavior. 3= Somewhat. Consultant discusses child negative behavior as well as range of typical/healthy behavior or type of behavior, but does not label behaviors or problems. 5= Consultant attempts to ―convince‖ parent that child’s behavior is problematic, or attempts to diagnose child problems. 20. Prematurely focuses on feedback issues: 1= Not at all. Family consultant presents all feedback and attends to areas in need of attention to discuss them with parent. 3= Somewhat. Family consultant seems to be focused on a specific problem of aspect of a problem, but does not lose sight of the overall feedback or strengths. 5= Very much. Family consultant focuses too quickly on a specific problem or aspect of a problem. Focus raises client resistance. 80 21. Expresses blame for negative behavior: 1= Not at all. While family consultant communicates parenting role in behavior change, uses a ―no fault‖ policy, and focuses on what can be done. 3= Somewhat. While family consultant communicates parenting role in behavior change, consultant may attribute problems to parent behavior or cause, but focuses on what can be done to change situation/behavior. 5= Very much. While communicating parenting role in behavior change, family consultant communicates parent at fault for difficulties child or family is encountering. 81 Characteristics of the Culturally Skilled Counseling Psychology (Sue et al., 1982) Beliefs and Attitudes Has moved from being culturally unaware to being aware and sensitive to one’s own cultural heritage and to valuing and respecting differences:  Other cultures are seen as equally valuable and legitimate as one’s own.  Culturally unaware counselors may impose personal values onto a minority client. Aware of one’s own values and biases and how they may affect minority clients:  Constantly attempt to avoid prejudices, unwarranted labeling and stereotyping.  Try not to hold preconceived limitations and notions of minority clients. Is comfortable with differences that exist between the counselor and client in terms of race and beliefs:  Does not negate existence of differences in attitudes and beliefs. Sensitive to circumstances (biases, stage of ethnic identity, sociopolitical influences) which may dictate referral of minority client to a member of his or her own race or culture. Knowledge Has a good understanding of the sociopolitical system’s operation in the U.S. with respect to its treatment of minorities:  Understands impact and operation of oppression (racism, sexism, classism), politics of counseling, and the racist concepts that have permeated helping professions.  Understands role cultural racism plays in development of identity and worldviews among minority groups. Possesses specific knowledge and information about the particular group one is working with:  Aware of history, experiences, cultural values, and lifestyle of various racial and ethnic groups. Has a clear and explicit knowledge and understanding of the generic characteristics of counseling and therapy:  Clearly understand value assumptions inherent in counseling and how they interact with values of culturally different.  Able to determine what is useful to culturally different clients.  Understands language factors, culture, and class-bound values. Aware of institutional barriers which prevent minorities from using mental health services:  Aware of locations of agencies, and availability to minorities. Skills Must be able to generate a wide variety of verbal and nonverbal responses. Must be able to send and receive both verbal and nonverbal messages accurately and ―appropriately.‖  Able to send thoughts and feelings to client, but also able to read messages from client  Able to send and receive cultural cues in setting.  Accuracy of communication tempered by appropriateness: subtlety, indirectness, directness, and confrontation appropriate for client context. Able to exercise institutional intervention skills on behalf of client when appropriate:  Involves outside help-giving. 82 Definitions and Rating Guidelines for Global Ratings MISTS Rating Guidelines Madson, Campbell, Barrett, Rugg, and Stoffel (2005) Item 1: Questions Raters are to provide a global rating of the family consultant’s use of questions in the session. This includes the use of open and closed questions. A closed question implies or requires the client to give a one or two word answer (e.g., Yes or No) and is mainly used to gather information. Open questions do not purposely limit the nature of the answer to a one-word response, can be phrased as queries or phrased as directives, and imply that the client provide a thorough answer (Hill & O`Brien, 1999). The appropriate use of questions is an important aspect of motivational interviewing; thus raters are to judge the appropriateness with which the family consultant uses questions in session. Rating Anchors 1. Relies on closed questions which do not engage client and allows only for information gathering. 3. Uses closed and open questions, but in general asks too many questions does not facilitate client exploration. 5. Selective use of primarily open question used to facilitate exploration of important topical areas. Item 2: Simple Reflection Raters are to make a judgment of the family consultant’s use of simple reflections in the counseling session. Simple reflections are family consultant restatements of the session content, thoughts, and feelings that acknowledge and validate what the client has said (Substance Abuse and Mental Health Services Administration, 1999). Rating Anchors 1. Infrequent use of reflections. Mainly paraphrase or restatement to clarify information, not used to communicate understanding. 3. More regular occurrence and in varied contexts. Used to clarify information, communicate understanding. 5. Frequent and varied use to communicate understanding, reinforce important statements and elicit more exploration of topic. Item 3. Complex Reflection Complex reflections are an important ingredient of motivational interviewing to help facilitate client change. Raters are to make judgments about the family consultant’s use of complex reflections. Complex reflections are a family consultant’s restatements of session content, client thoughts and feeling, with something added to facilitate movement toward positive change (Substance Abuse and Mental Health Services Administration, 1999). 83 One complex reflection is when a family consultant restates what the client has said, but in an exaggerated form—to restate the statement in a stronger or even more extreme fashion than what the client communicated (Miller & Rollnick, 2002). Client: I don’t understand why my wife is so concerned about my drinking. I don’t have a problem. Family consultant: So your wife is worrying needlessly about your drinking. Another complex reflection is the double-sided reflection in which the family consultant restates a client statement that captures both sides of the client’s ambivalence (Miller & Rollnick, 2002). Client: I know people want me to completely stop drinking, but I am not going to completely quit. Family consultant: You are really aware that there are some problems related to your drinking, but you are not ready to completely quit drinking. Rating Anchors 1. Infrequent or limited use, used at inopportune times or in a clumsy manner. • An inopportune time may be a time in which the client is not ready to hear the additional information, for example, early in an initial session when the family consultant should be building the relationship. 3. More regular use of complex reflections. Used only to communicate understanding. 5. Used skillfully to reinforce, redirect, amplify or change client awareness. Simple vs. Complex Reflection Reflections are not dichotomous. Rather, reflections occur on a continuum. Viewing reflections as occurring on a continuum will help to clarify three concepts involved in motivational interviewing: simple reflections, complex reflections, and interpretations. The visual display will help to demonstrate the continuum of reflections. Simple Reflections  Echoes and paraphrases used to mirror the client. Complex Reflections  Shift focus in session;  Add meaning to what client stated;  Interpretation;  Family consultant adds meaning not provided by client. Item 4: Affirming Providing affirmation communicates to the client the family consultant’s support and acknowledgement of the client’s difficulties and experience. Affirmation helps the client begin to feel comfortable with the family consultant and with discussing difficult experiences (Substance Abuse and Mental Health Services Administration, 1999). Examples of affirmations include: 84 I appreciate you coming here today and recognize how difficult it must have been. I think it is great that you are taking your family’s advice and coming to counseling. That is a good suggestion you made for changing. Rating Anchors 1. Little or no attempt to identify client strengths or successes. 3. Maintains a nonjudgmental, accepting stance toward client goals, activities, but little active affirming. 5. Regularly and systematically elicits and reinforces strengths, communicating a sense of optimism and hope. Item 5: Summarization Similar to paraphrases, but are used to clarify and distill what the client has said over a longer time span. May be used at the beginning or end of the session, as a transition to a new topic, or to clarify complex issues. Helps both the family consultant and the client organize thinking about what is happening in the session (Ivey & Bradford Ivey, 2003). Rating Anchors 1. Few summaries, and use of summaries is infrequent and superficial. Used only to clarify client statements. 3. Used to review a section of the session. 5. Regular use of summaries to reiterate important themes, direct focus and transition within the session. Item 6: Engaging Client in the Therapeutic Process In engaging the client into the therapeutic process, the family consultant uses active listening skills to express genuine empathy and establish a warm and safe environment that helps the client to feel safe to share information. A client engaged in the therapeutic process tends to discuss issues that are not superficial such as basic information, but becomes less guarded and discusses issues about what the experience was like and their thoughts, feeling, and vulnerabilities. Rating Anchors 1. Does little to create safe environment in which the client can feel safe to explore the problem; some suggestion of nonacceptance such as being judgmental, argumentative, suspicious, condescending, or aloof. 3. Makes an effort to create a safe environment, is less judgmental; however, has some difficulties establishing rapport or helping the client actively participate in therapy. 5. Creates an environment in which the client can feel safe to explore problems and actively participate in therapy by remaining nonjudgmental, warm, flexible, and respectful of the client. Item 7: Elicits and Reinforces Client Change Talk Change talk, also referred to as self-motivational statements, is extremely important to motivational interviewing. One of the major goals of motivational interviewing is to 85 assist the client in establishing change talk. Client change talk is a signal that the client’s ambivalence about change is diminishing and that the client is becoming increasingly ready to change. In eliciting or reinforcing client change talk the family consultant can use a variety of techniques like simple and complex reflections, questions, affirmations, and summaries. Rating Anchors 1. Consistently misses or ignores opportunity to elicit or reinforce client change talk. Does not facilitate client’s change talk. 3. Elicits or reinforces change talk inconsistently in session and does not facilitate client self talk further. 5. Consistently elicits and reinforces change talk that facilitates client exploration, awareness, and future change talk. Item 8: Addresses Client Ambivalence Ambivalence is an important concept in motivational interviewing as it is often a central client problem. In motivational interviewing, the family consultant needs to recognize and facilitate the client exploration of ambivalence with the goal of resolving the ambivalence. Family consultants can use a variety of techniques to address ambivalence including questioning, simple and complex reflections, affirmations, and summaries. Rating Anchors 1. Consistently misses or ignores client ambivalence. 3. Recognizes client ambivalence, but does not fully explore or attempt to address it in session. 5. Recognizes ambivalence and consistently addresses it with the client. Item 9: Rolling with Client Resistance Family consultants incorporating a motivational interviewing style conceptualize resistance as a signal that there is a difference in how the client and family consultant view the situation. To roll with resistance effectively, the family consultant avoids arguing with the client, listens more carefully, changes direction, and responds to the client in a nonconfrontational manner that attempts to change client energy toward discussing positive change. Rating Anchors 1. Argues with client for change by using persuasion, confrontation, inappropriate education, or another strategy that evokes client resistance and arguing against change. 3. Acknowledges resistance, argues minimally with client, less confrontative, evokes less client resistance arguing against change, but appears unsure how to use resistance appropriately. 5. Uses client resistance during session as indicator of a need to change focus, shift direction, and explore in a nonconfrontative fashion. Uses a variety of techniques like agreement with a twist, shifting focus, and siding with the negative. 86 Item 10: Collaborating with the Client The family consultant enters into a partnership with the client that honors the client’s expertise and perspectives. This relationship is conducive (i.e., facilitates or contributes) to change; not coercive. The family consultant acts as a partner, not an expert (Miller & Rollnick, 2002). Rating Anchors 1. Regularly assumes an expert role, does too much telling, instructing, and advising. 3. Minimal expert role, but does not routinely elicit from client reasons and methods for change. 5. Works with client, communicates appreciation for client’s experience and expertise, asks for permission before giving commentary and advice. Item 11: Supports Client Self-Efficacy Self-efficacy for change is an important ingredient in facilitating change. The family consultant using a motivational interviewing style recognizes client strengths and makes attempts to raise client awareness of these strengths. Supporting client self-efficacy involves eliciting and supporting client hope, optimism, and feasibility of accomplishing change (Substance Abuse and Mental Health Services Administration, 1999). Rating Anchors 1. Consistently misses opportunities to instill hope, has a pessimistic attitude in session, focuses on clients weaknesses, discusses nonfeasible change. 3. Communicates hope and optimism inconsistently, misses opportunities to recognize and reinforce client strengths. Awkwardly discusses the feasibility of change. 5. Consistently communicates optimism, hope, and the possibility of client change. Recognizes, communicates and reinforces clients strengths. Discusses feasible change. Item 16: Effectiveness of Family consultant Interventions in Session This item provides a rating of the overall effectiveness of the family consultant’s motivational interviewing interventions. Ratings are based on the overall rating provided for the use of active listening skills (item 12), the appropriate sequencing of motivational interviewing (item 13), the overall spirit of motivational interviewing (item 14), and the client response (item 15). Rating Anchors 1. Not effective in facilitating MI. 3. Moderately effective in facilitating MI. 5. Extremely effective in facilitating MI. 87 APPENDIX D PAL2 FEEDBACK FOGI RATINGS 88 PAL2 FEEDBACK FOGI RATINGS Reliability Scoring Form Family ID: __ __ __ __ Cal ID: __ __ Date: __ __/__ __/ 2009 Rel ID: __ __ Date: __ __/__ __/ 2009 FCU KNOWLEDGE AND BEHAVIOR /11 % OVERALL EFFECTIVENESS OF FCU INTERVENTION /6 % MI/FCU TRAPS /5 % MISTS /11 % MULTICULTURAL SKILLS /12 % THERAPIST INTERPERSONAL QUALITIES /19 % PARENT RESPONSE- 1 PARENT /10 % TOTAL /74 % PARENT RESPONSE- 2 PARENT /10 % TOTAL /84 % Stages of Change- 1 PARENT / (3) % Stages of Change- 2 PARENT / (3) % 89 APPENDIX E CORRELATIONS AND Z-SCORES FOR OBSERVATIONAL CODER RATINGS (FOGI) 90 Correlations and Z-Scores for Observational Coder Ratings (FOGI) Measure European American parents n = 33 Ethnocultural Group parents n = 48 FCU MISTS CRF-S CCCI-R INT OBSERV ATION Parent Resp FCU MISTS CRF-S CCCI-R INT OBS Parent Resp Observational Coder Ratings (FOGI) FCU r zr 1 .92** 1.62 .83** 1.19 . 83** 1.18 .89** 1.42 .70** .86 1 .80** 1.09 .76** .99 .68** .83 .75** .98 .46** .49 MISTS r zr .92** 1.62 1 .78** 1.05 .83** 1.18 .88** 1.39 .69** .84 .80** 1.09 1 .57** .65 .53** .59 .76** .99 .49** .54 CRF-S r zr .83** 1.19 .79** 1.05 1 .79** 1.06 .82** 1.16 .54** .60 .76** .99 .57** .65 1 .66** .80 .66** .79 .27 .27 CCCI-R r zr .83** 1.18 .83** 1.18 .79** 1.06 1 .82** 1.17 .54** .61 .68** .83 .53** .59 .66** .80 1 .70** .86 .38** .40 INT OBS r zr .89** 1.42 .88** 1.39 .82** 1.16 .82** 1.17 1 .70** .87 .75** .98 .76** .99 .66** .79 .70** .86 1 .69** .84 Parent Resp r zr .70** .86 .69** .84 .54** .60 .54** .61 .70** .87 1 .46** .49 .49** .54 .27 .27 .38** .40 .69** .84 1 Note. FCU = Family Check-Up Coder Rating Scale; MISTS = Motivational Interviewing Supervision and Training Scale (Madson et al., 2005); INT OBS = Coder Interpersonal Observations Scale; Parent Response = Coder observation of parent response scale. *Correlation is significant at the 0.05 level (2-tailed). Z-scores calculated via Fisher’s r-to-z transformations (*). **Correlation is significant at the 0.01 level (2-tailed). ***Significant z-score ≥ 1.96, p = .05. Missing data excluded listwise. European American SEzr = 0.183. Ethnocultural Group SEzr = = 0.15. 91 REFERENCES CITED American Psychological Association. (2003). Guidelines on multicultural education, training, research, practice, and organizational change for psychologists. American Psychologist, 58(5), 377–402. American Psychological Association Presidential Task Force on Evidence-Based Practices. (2006). Evidence-based practice in psychology. American Psychologist, 61(4), 271–285. Arredondo, P. (1998). Integrating multicultural counseling competencies and universal helping conditions in culture-specific contexts. The Counseling Psychologist, 26, 592–601. Ary, D. V., Duncan, T. E., Biglan, A., Metzler, C. W., Noell, J. W., & Smolkowski, K. (1999). Development of adolescent problem behavior. Journal of Abnormal Child Psychology, 27, 141–150. Atkinson, D. R., Kim, B. S., & Caldwell, R. (1998). Ratings of helper roles by multicultural psychologists and Asian American students: Initial support for the three-dimensional model of multicultural counseling. Journal of Counseling Psychology, 45(4), 414–423. Barak, A., & LaCrosse, M. B. (1975). Multidimensional perception of counselor behavior. Journal of Counseling Psychology, 22, 471–476. Bellg., A. J., Borrelli, B., Resnick, B., Hecht, J., Minicucci, D. S., Ory, M., . . . Czajkowski, S. (2004). Enhancing treatment fidelity in health behavior change studies: Best practices and recommendations from the NIH Behavior Change Consortium. Health Psychology, 23(5), 443–451. Bernal, G., Jiménez-Chafey, M. I., & Domenech Rodríguez, M. M. (2009). Cultural adaptation of treatments: A resource for considering culture in evidence-based practice. Professional Psychology, Research and Practice, 40(4), 361–368. Bernal, G., & Scharrón-del-Río, M. R. (2001). Are empirically supported treatments valid for ethnic minorities? Toward an alternative approach for treatment research. Cultural Diversity and Ethnic Minority Psychology, 7(4), 328–342. Biglan, A., Mrazek, P. J., Carmine, D., & Flay, B. R. (2003). The integration of research and practice in the prevention of youth problem behaviors. American Psychologist, 58(6/7), 433–440. 92 Birkholz, N., Patras, J. M., & Dishion, T. J. (2002). Feedback Rating Scale: Family Check-Up Coding Project. Unpublished instrument, Child and Family Center, University of Oregon, Eugene. Burke, B. L., Arkowitz, H., & Menchola, M. (2003). The efficacy of Motivational Interviewing: A meta-analysis of controlled clinical trials. Journal of Consulting and Clinical Psychology, 71(5), 843–861. Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81–105. Center for Substance Abuse Prevention. (2000). The national cross-site evaluation of high risk youth programs: Final report. Rockville, MD: Center for Substance Abuse Prevention, Substance Abuse and Mental Health Services Administration. Chamberlain, P., Davis, B., Forgatch, M. S., Frey, S., Patterson, G. R., Ray, J. R., . . . Trombley, T. (1986). The therapy process code: An observational system. Eugene, OR: Oregon Social Learning Center. Claiborn, C. D., Goodyear, R. K., & Horner, P. A. (2001). Feedback. Psychotherapy, 38(4), 401–405. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37–46. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum. Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2003). Applied multiple regression/correlation analysis for the behavioral sciences. Mahwah, N.J.: Erlbaum. Collins, L. M., Murphy, S. A., & Bierman, K. L. (2004). A conceptual framework for adaptive preventive interventions. Prevention Science, 5(3), 185–196. Connell, A. M., Dishion, T. J., Yasui, M., & Kavanagh, K. (2007). An adaptive approach to family intervention: Linking engagement in family-centered intervention to reductions in adolescent problem behavior. Journal of Consulting and Clinical Psychology, 75(4), 568–579. Constantine, M. G. (2001). Predictors of observer ratings of multicultural counseling competence in Black, Latino, and White American trainees. Journal of Counseling Psychology, 48(4), 456–462. 93 Constantine, M. G. (2002). Predictors of satisfaction with counseling: Racial and ethnic minority clients’ attitudes toward counseling and ratings of their counselors’ general and multicultural counseling competence. Journal of Counseling Psychology, 49(2), 255–263. Corrigan, J. D., & Schmidt, L. D. (1983). Development and validation of revisions in the counselor rating form. Journal of Counseling Psychology, 30(1), 64–75. DeVellis, R. F. (2003). Scale development: Theory and applications (2nd ed., Applied Social Research Methods Series No. 26). Thousand Oaks, CA: Sage. Dishion, T. J., & Andrews, D. W. (1995). Preventing escalation in problem behaviors with high-risk young adolescents: Immediate and 1-year outcomes. Journal of Consulting and Clinical Psychology, 63(4), 538–548. Dishion, T. J., & Kavanagh, K. (2003). Intervening in adolescent problem behavior: A family-centered approach. New York, NY: Guilford. Dishion, T. J., Nelson, S. E., & Kavanagh, K. (2003). The Family Check-Up with high- risk young adolescents: Preventing early-onset substance use by parent monitoring. Behavior Therapy, 34, 553–571. Dishion, T. J., Patterson, G. R., Stoolmiller, M., & Skinner, M. S. (1991). Family, school, and behavioral antecedents to early adolescent involvement with antisocial peers. Developmental Psychology, 27, 172–180. Dishion, T. J., Shaw, D., Connell, A., Gardner, F., Weaver, C., & Wilson, M. (2008). The Family Check-Up with high-risk indigent families: Preventing problem behavior by increasing parents’ positive behavior support in early childhood. Child Development, 79(5), 1395–1414. Dishion, T. J., & Stormshak, E. (2007). Intervening in children's lives: An ecological, family-centered approach to mental health care. Washington, DC: APA Books. Epperson, D. L., & Pecnik, J. A. (1985). Counselor rating form—Short version: Further validation and comparison to the long form. Journal of Counseling Psychology, 32(1), 143–146. Erkut, S., Alarcón, O., Garcia Coll, C., Tropp, L. R., & Vázquez García, H. A. (1999). The dual-focus approach to creating bilingual measures. Journal of Cross- Cultural Psychology, 30(2), 206–218. 94 Fischer, A. R., Jome, L. M., & Atkinson, D. R. (1998). Reconceptualizing multicultural counseling: Universal healing conditions in a culturally specific context. The Counseling Psychologist, 26, 525–588. Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378–382. Forgatch, M. S., Patterson, G. R., & DeGarmo, D. S. (2005). Evaluating fidelity: Predictive validity for a measure of competent adherence to the Oregon model of parent management training. Behavior Therapy, 36(1), 3–13. Fowler, F. J. (2002). Survey research methods (3rd ed., Applied Social Research Methods Series No. 1). Thousand Oaks, CA: Sage. Fraga, E. D., Atkinson, D. R., & Wampold, B. E. (2004). Ethnic group preferences for multicultural counseling competencies. Cultural Diversity and Ethnic Minority Psychology, 10(1), 53–65. Friedlander, M. L., Escudero, V., Horvath, A. O., Heatherington, L., Cabero, A., & Martens, M. P. (2006). System for Observing Family Therapy Alliances: A tool for research and practice. Journal of Counseling Psychology, 53, 214–225. Fuertes, J. N., & Brobst, K. (2002). Clients’ ratings of counselor multicultural competency. Cultural Diversity and Ethnic Minority Psychology, 8(3), 214–223. Geisinger, K. F. (1994). Cross-cultural normative assessment: Translation and adaptation issues influencing the normative interpretation of assessment instruments. Psychological Assessment, 6(4), 304–312. Gill, A. M., Hyde, L. W., Shaw, D. S., Dishion, T. J., & Wilson, M. N. (2008). The Family Check-Up in early childhood: A case study of intervention process and change. Journal of Clinical Child and Adolescent Psychology, 37(4), 893–904. Glass, G. V, Peckham, P. D., & Sanders, J. R. (1972). Consequences of failure to meet assumptions underlying the fixed analysis of variance and covariance. Review of Educational Research, 42, 237–288. Hage, S. M., Romano, J. L, Conye, R. K., Kenny, M., Matthews, C., Schwartz, J. P., & Waldo, M. (2007). Best practice guidelines on prevention practice, research, training, and social advocacy for psychologists. The Counseling Psychologist, 35(4), 493–566. 95 Helms, J. E., Henze, K. T., Sass, T. L., & Mifsud, V. A. (2006). Treating Cronbach’s alpha reliability coefficients as data in counseling research. The Counseling Psychologist, 34(5), 630–660. Hall, G. C. (2001). Psychotherapy research with ethnic minorities: Empirical, ethical, and conceptual issues. Journal of Consulting and Clinical Psychology, 69(3), 502– 510. Hill, C. E., & Corbett, M. M. (1993). A perspective on the history of process and outcome research in counseling psychology. Journal of Counseling Psychology, 40(1) 3– 24. Hill, C. E., & O`Brien, K. M. (1999). Helping skills: Facilitating exploration, insight, and action. Washington, DC: American Psychological Association. Hogue, A., Liddle, H. A., & Rowe, C. (1996). Treatment adherence process research in family therapy: A rationale and some practical guidelines. Psychotherapy, 33(2), 332–345. Hoyt, W. T., Warbasse, R. E., & Chu, E. Y. (2006). Construct validation in counseling psychology research. The Counseling Psychologist, 34(6), 769–805. Ivey, A. E., & Ivey, M. (2003). Intentional interviewing and counseling: Facilitating client development in a multicultural society (5th ed.). Pacific Grove, CA: Thompson Brooks/Cole. Karlsson, R. (2005). Ethnic matching between therapist and patient in psychotherapy: An overview of findings, together with methodological and conceptual issues. Cultural Diversity and Ethnic Minority Psychology, 11(2), 113–129. Kavanagh, K., Dishion, T., & Connell, A. (2006). A multi-level approach to family- centered prevention in schools. The Family Psychologist, 21(4), 8–10. Kazdin, A. E., & Whitley, M. K. (2006). Pretreatment social relations, therapeutic alliance, and improvements in parenting practices in parent management training. Journal of Consulting and Clinical Psychology, 74(2), 346–355. Knutson, N. M., Forgatch, M. S., & Rains, L. A. (2003). Fidelity of Implementation Rating System (FIMP): The manual for PMTO. Eugene, OR: Oregon Social Learning Center. 96 Kumpfer, K. L., & Alder, S. (2003). Dissemination of research-based family interventions for the prevention of substance abuse. In Z. Sloboda & W. J. Bukoski (Eds.), Handbook of drug abuse prevention (pp. 75–119). New York, NY: Kluwer Academic/Plenum. Kumpfer, K. L., & Alvarado, R. (2003). Family-strengthening approaches for the prevention of youth problem behaviors. American Psychologist, 58(6/7), 457–465. Kumpfer, K. L., Alvarado, R., Smith, P., & Bellamy, N. (2002). Cultural sensitivity in universal family-based prevention interventions. Prevention Science, 3(3), 241– 244. LaCrosse, M. B. (1977). Comparative perceptions of counselor behavior: A replication and extension. Journal of Counseling Psychology, 24, 464–471. LaCrosse, M. B. (1980). Perceived counselor social influence and counseling outcomes: Validity of the Counselor Rating Form. Journal of Counseling Psychology, 27(4), 320–327. LaCrosse, M. B., & Barak, A. (1976). Differential perception of counselor behavior. Journal of Counseling Psychology, 23(2), 170–172. LaFromboise, T. D., Coleman, H. L., & Hernandez, A. (1991). Development and factor structure of the Cross-Cultural Counseling Inventory—Revised. Professional Psychology: Research and Practice, 22(5), 380–388. Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174. Madson, M. B., Campbell, T. C., Barrett, D. E., Brondino, M. J., & Melchert, T. P. (2005). Development of the Motivational Interviewing Supervision and Training scale. Psychology of Addictive Behaviors, 19(3), 303–310. Madson, M. B., Campbell, T. C., Barrett, D. E., Rugg, C., & Stoffel, V. (2005). Motivational Interviewing Supervision and Training Scale: Guidelines for rating audiotape sessions. Unpublished manuscript, Center for Addiction and Behavioral Health Research, University of Wisconsin, Milwaukee. Retrieved from http://www.usm.edu/MIConsortium/Webpage%20Images/MISTSTrainingmanual. pdf Matías, L. E., Chávez, L. M., Negrón, G., Canino, G., Aguilar-Gaxiola, S., & Hoppe, S. (2003). The Spanish translation and cultural adaptation of five mental health outcome measures. Culture, Medicine, and Psychiatry, 27, 291–313. 97 Mertler, C. A., & Vannatta, R. A. (2005). Advanced and multivariate statistical methods (3rd ed.). Glendale, CA: Pyrczak. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13–103). New York, NY: MacMillan. Miller, W. R., & Rollnick, S. (1991). Motivational interviewing: Preparing people to change addictive behaviors. New York, NY: Guilford Press. Miller, W. R., & Rollnick, S. (2002). Motivational interviewing: Preparing people to change (2nd ed.). New York, NY: Guilford Press. Moncher, F. J., & Prinz, R. J. (1991). Treatment fidelity in outcome studies. Clinical Psychology Review, 11(3), 247–266. Morales, E., & Norcross, J. C. (2010). Evidence-based practices with ethnic minorities: Strange bedfellows no more. Journal of Clinical Psychology: In Session, 66(8), 821–829. Mowbray, C. T., Holter, M. C., Teague, G. B., & Bybee, D. (2003). Fidelity criteria: Development, measurement, and validation. American Journal of Education, 24(3), 315–340. Moyers, T. B., Miller, M. R., & Hendrickson, S. M. L. (2005). How does Motivational Interviewing work? Therapist interpersonal skill predicts client involvement within Motivational Interviewing sessions. Journal of Consulting and Clinical Psychology, 73(4), 590–598. Okazaki, S., & Sue, S. (1995). Methodological issues in assessment research with ethnic minorities. Psychological Assessment, 7(3), 367–375. Patterson, G. R., Reid, J. B., & Dishion, T. J. (1992). A social learning approach: Vol. 4. Antisocial boys. Eugene, OR: Castalia. Pedhazur, E. J., & Pedhazur Schmelkin, L. (1991). Measurement, design, and analysis: An integrated approach. Hillsdale, NJ: Erlbaum. Ponterotto, J. G., Fuertes, J. N., & Chen, E. C. (2000). Models of multicultural counseling. In S. D. Brown & R. W. Lent (Eds.), Handbook of Counseling Psychology (3rd ed., pp. 639–669). Hoboken, NJ: Wiley & Sons. Ponterotto, J. G., & Furlong, M. J. (1985). Evaluating counselor effectiveness: A critical review of rating scale instruments. Journal of Counseling Psychology, 32(4), 597– 616. 98 Ponterotto, J. G., Rieger, B. P., Barrett, A., & Sparks, R. (1994). Assessing multicultural counseling competence: A review of instrumentation. Journal of Counseling and Development, 72, 316–322. Pope-Davis, D. B., Liu, W. M., Toporek, R. L., & Brittan-Powell, C. S. (2001). What’s missing from multicultural competency research: Review, introspection, and recommendations. Cultural Diversity and Ethnic Minority Psychology, 7(2), 121– 138. Pope-Davis, D. B., & Ottavi, T. M. (1994). Examining the association between self- reported multicultural counseling competencies and demographic variables among counselors. Journal of Counseling and Development, 72, 651–654. Pope-Davis, D. B., Toporek, R. L., Ortega-Villalobos, L., Ligiéro, D. P, Brittan-Powell, C. S., Liu, W. M., . . . Liang, C. T. (2002). Special issue: Client perspectives of multicultural counseling competence. The Counseling Psychologist, 30(3), 355– 393. Prochaska, J. O., DiClemente, C. C., & Norcross, J. C. (1992). In search of how people change: Applications to addictive behaviors. American Psychologist, 47, 1102– 1114. Prochaska, J. O., & Norcross, J. C. (2001). Stages of change. Psychotherapy, 38(4), 443– 448. Reid, J. B., Fleischman, M. J., Arthur, J., Toobert, D. J., Stern, S., & Patterson, G. R. (1979, December). Therapist performance observational system. Paper presented at the Association for the Advancement of Behavior Therapy, San Francisco. Rohrer Murphy, L. C., & Suen, H. K. (1999, April). Validating measures of structural knowledge through the multi-trait multi-method matrix. Paper presented at the annual meeting of the American Educational Research Association, Montreal. Rosenthal, R., & Rosnow, R. L. (1991). Essentials of behavioral research: Methods and data analysis (2nd ed.). New York, NY: McGraw-Hill. Sabnani, H. B., & Ponterotto, J. G. (1992). Racial/ethnic minority-specific instrumentation in counseling research: A review, critique, and recommendations. Measurement and Evaluation in Counseling and Development, 24(4), 161–187. Schafer, J. L., & Graham, J. W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7(2), 147–177. 99 Shadish, W., Cook, T., & Campbell, D. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin. Shaw, D. S., Dishion, T. J., Connell, A., Wilson, M. N., & Gardner, F. (2009) Maternal depression as a mediator of intervention in reducing child problem behavior. Development and Psychopathology, 21(2), 417–439. Shaw, D. S., Dishion, T. J., Supplee, L., Gardner, F., & Arnds, K. (2006). Randomized trial of a family-centered approach to the prevention of early conduct problems: 2- year effects of the Family Check-Up in early childhood. Journal of Consulting and Clinical Psychology, 74(1), 1–9. Slavet, J. D., Stein, L. A. R., Klein, J. L., Colby, S. M., Barnett, N. P., & Monti, P. M. (2005). Piloting the Family Check-Up with incarcerated adolescents and their parents. Psychological Services, 2(2), 123–132. Spoth, R. L., Kavanagh, K. A., & Dishion, T. J. (2002). Family-centered preventive intervention science: Toward benefits to larger populations of children, youth, and families. Prevention Science, 3(3), 145–152. Stevens, J. P. (2002). Applied multivariate statistics for the social sciences (4th ed.). New York, NY: Taylor & Francis. Stormshak, E. A., Connell, A. M., Véronneau, M., Myers, M. W., Dishion, T. J., Kavanagh, K., & Caruthers, A. S. (2011). An ecological approach to promoting early adolescent mental health and social adaptation: Family-centered intervention in public middle schools. Child Development 82(1), 209–225. Stormshak, E. A., & Dishion, T. J. (2002). An ecological approach to child and family clinical and counseling psychology. Clinical Child and Family Psychology Review, 5(3), 197–215. Stormshak, E. A., Dishion, T. J., Light, J., & Yasui, M. (2005). Implementing family- centered interventions within the public middle school: Linking service delivery to change in student problem behavior. Journal of Abnormal Child Psychology, 33(6), 723–733. Strong, S. R. (1968). Counseling: An interpersonal influence process. Journal of Counseling Psychology, 15, 215–224. Sue, D. W., Arredondo, P., & McDavis, R. J. (1992). Multicultural counseling competencies and standards: A call to the profession. Journal of Counseling and Development, 70, 477–486. 100 Sue, D. W., Bernier, J. E., Durran, A., Feinberg, L., Pedersen, P., Smith, E. J., Vasquez- Nuttall, E. (1982). Position paper: Cross-cultural counseling competencies. The Counseling Psychologist, 10(2), 45–52. Sue, D. W., Bingham, R. P., Porché-Burke, L., & Vasquez, M. (1999). The diversification of psychology: A multicultural revolution. American Psychologist, 54(12), 1061– 1069. Sue, S. (1999). Science, ethnicity, and bias: Where have we gone wrong? American Psychologist, 54(12), 1070–1077. Sue, S. (2003). In defense of cultural competency in psychotherapy and treatment. American Psychologist, 58(11), 964–970. Sue, S., Fujino, D. C., Hu, L., Takeuchi, D. T., & Zane, N. (1991). Community mental health services for ethnic minority groups: A test of the cultural responsiveness hypothesis. Journal of Consulting and Clinical Psychology, 59, 533–540. Sue, S., & Zane, N. (2006). Ethnic minority populations have been neglected by evidence-based practices. In J. C. Norcross, L. E. Beutler, & R. F. Levant (Eds.), Evidence-based practices in mental health: Debate and dialogue on the fundamental questions (pp. 338–345, 359–361). Washington, DC: American Psychological Association. Szapocznik, J., & Kurtines, W. M. (1993). Family psychology and cultural diversity: Opportunities for theory, research, and application. American Psychologist, 48(4), 400–407. Tabachnick, B. G., & Fidell, L. S. (2007). Using multivariate statistics (4th ed.). Boston, MA: Allyn & Bacon. Trochim, W. M. (2006). The research methods knowledge base (2nd ed.). Cincinnati, OH: Atomic Dog. Retrieved from http://www.socialresearchmethods.net/kb/ U.S. Bureau of the Census. (2000). Census 2000 summary file. Retrieved from http://factfinder.census.gov U.S. Bureau of the Census. (2006). American community survey. Retrieved from http://factfinder.census.gov 101 U.S. Department of Health and Human Services. (1994). Preventing tobacco use among young people: A report of the Surgeon General. Atlanta, GA: U.S. Department of Health and Human Services, Public Health Service, Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Office on Smoking and Health. U.S. Department of Health and Human Services. (1999). Mental health: A report of the Surgeon General. Rockville, MD: U.S. Department of Health and Human Services, National Institutes of Health, National Institute of Mental Health. U.S. Public Health Service. (2000). Report of the Surgeon General's Conference on Children's Mental Health: A national action agenda. Washington, DC: Department of Health and Human Services. U.S. Public Health Service. (2001). Youth violence: A report of the Surgeon General. Washington, DC: Department of Health and Human Services. Weisz, J. R., Sandler, I. N., Durlak, J. A., & Anton, B. S. (2005). Promoting and protecting youth mental health through evidence-based prevention and treatment. American Psychologist, 60(6), 628–648. Worthington, R. L., Mobley, M., Franks, R. P., & Tan, J. A. (2000). Multicultural counseling competencies: Verbal content, counselor attributions, and social desirability. Journal of Counseling Psychology, 47(4), 460–468. Worthington, R. L., Soth-McNett, A. M., & Moreno, M. V. (2007). Multicultural counseling competencies research: A 20-year content analysis. Journal of Counseling Psychology, 54(4), 351–361. Worthington, R. L., & Whittaker, T. A. (2006). Scale development research: A content analysis and recommendations for best practices. The Counseling Psychologist, 34(6), 806–838. Yasui, M., & Dishion, T. J. (2007). The ethnic context of child and adolescent problem behavior: Implications for child and family interventions. Clinical Child and Family Psychology Review, 10(2), 137–179. Yasui, M., & Dishion, T. J. (2008). Direct observation of family management: Validity and reliability as a function of coder ethnicity and training. Behavior Therapy, 39, 336–347.