- RTTT 4C
- Elementary & Secondary Demonstration Grant: Agawam, Massachusetts Public Schools
- Elementary & Secondary Demonstration Grant: Central Falls, Rhode Island Public Schools
- Multi-State Evaluation Project
- NOSCA-CSCORE Fellows Program
- Career Planning and Implementation Conferences (CPIC) - Years One, Two, and Three
- The Chicago Public Schools Study
- The Real Game
- College Board Inspiration and Innovation Study
- Bridges Transitions Evaluation
- Evidence Based Practice Panel
- Delphi Study
Professional Development Training for RTTT 4C Districts
Massachusetts is one of 12 states that were funded in the national Race to the Top competition, funded by the U.S. Department of Education to promote reform in four areas: standards and assessments, great teachers and leaders, school turnaround, and data systems. Half of RTTT funding is allocated to the Department of Elementary and Secondary Education for enhanced data systems, model curriculum, evaluation and other state infrastructure to continue to support the education reform begun in the Commonwealth in 1993. The balance of RTTT funding has been allocated to over 250 participating school districts and charter schools who have signed on to the challenge of implementing RTTT programs at the local level. RTTT resources will allow us to build the skills, expand the capacity, create the tools, and harness the technology needed to close persistent achievement gaps while increasing achievement for all. Focus on development and implementation of college and career readiness initiatives to better prepare all students for post-secondary success.
Elementary & Secondary Demonstration Grants
Agawam, Massachusetts Public Schools
Through its three-year, million dollar Elementary and Secondary School Counseling grant, the Agawam Massachusetts Public School district is developing a comprehensive developmental elementary school counseling program that promotes success for all students through improved academic achievement, increased school engagement, reduced disciplinary events, and a safe school environment.
Two school counselors, plus a school social worker and school psychologist were hired to reduce ratios of student support personnel in the elementary schools. Job descriptions for the new school counselors were written to reflect roles and responsibilities consistent with the MASCA school counseling model. Classroom guidance lessons, using evidence based curricula whenever possible, are being delivered to every classroom in grades Kindergarten through 6th and the guidance staff has developed a full Scope and Sequence of classroom lessons. In addition, Building-Based Data Success Teams have been created at each of the district’s four elementary schools. The Teams are comprised of the building principal, teachers from each grade level, the school counselor, and other school personnel. School counselors are serving as leaders and advocates in collaborative and accountability-driven initiatives, using data to determine when it is necessary to develop Tiers 1, 2, and 3 behavioral and academic interventions. Nationally recognized experts have provided extensive professional development to the school counseling staff on all aspects of Model implementation. Grant personnel are working to create systemic changes with a preventative rather than a reactive approach and a focus on early intervention for students at-risk for academic and behavioral challenges.
As evaluator of this Federal Department of Education grant, CSCORE is working closely with the administration and counseling staff of the Agawam Public Schools to measure progress in each of the grants’ stated goals and outcomes. Multiple data collection methods are being employed in order to collect both strong qualitative and quantitative data from numerous constituencies, including students, parents, school counselors, and other school staff and administration. CSCORE conducted a focus group with the school Advisory Council to gather information about what staff felt were the most pressing needs of their students; created a Parent Survey to measure parent/guardian perceptions of the school counseling program and of school climate; administered a teacher-rated survey to measure student academic engagement; conducted a student survey to gauge impact of the district’s pro-social curricula such as Second Step, Steps to Respect, and PBIS; and is collecting numerous institutional data to measure changes in discipline referrals and behavioral incidents, monitor improvement in different academic indicators, and disaggregate data for specific populations to work towards closing the achievement gap. In addition, CSCORE has provided professional development to the Agawam school counseling staff on topics related to evaluating interventions, data-based decision making, accountability, and evidence-based counseling.
Central Falls, Rhode Island Public Schools
The Central Falls, Rhode Island school district was awarded a three year, million dollar Elementary and Secondary School Counseling grant in spring of 2009, and contracted with CSCORE to serve as evaluator of the grant. Central Falls serves a student population with a rich ethnic and linguistic diversity and faces myriad challenges related to low socioeconomic status and high mobility rates.
Federally-mandated outcomes for the grant included reducing the school counselor-to-student ratio and decreasing the number of disciplinary incidents in the district’s elementary schools. Prior to the grant, the district did not employ any school counselors at the elementary level. Therefore the first priority of the grant team was to hire four professional school counselors to work at each of the district’s elementary schools. These new school counselors participated in a five-day summer Professional Development Institute led by a team of nationally-recognized experts in the field of school counseling, including staff from CSCORE, and covered such topics as data, accountability, evaluation, curriculum, and leadership.
CSCORE began tracking disciplinary data to determine if the work of the new school counselors was positively impacting the number of disciplinary referrals in the elementary schools. The first step in this process was to create a standardized Office Discipline Referral (ODR) form to be used by all of the elementary schools. Baseline data were collected on both minor and major infractions using a hard copy, paper-and-pencil form. During the 2009-2010 school year the district installed a new student information system with a functionality related to discipline referrals. CSCORE is working with the district’s Informational Technology department to customize the new student information system to gather and report discipline data and to move towards an electronic ODR process. In spring 2010 one elementary school piloted the new electronic version of the Office Discipline Referral form and it is anticipated that by fall 2010 all four elementary schools will be recording disciplinary incidents electronically. This uniform and electronic referral process will provide more detailed and nuanced information about discipline referrals that counselors can then use to more effectively tailor specific social/emotional interventions to certain students or classrooms.
CSCORE also worked closely with the Central Falls elementary counselors to create Support Personnel Accountability Report Cards for the district. SPARCs are a continuous improvement tool that gives a school site an opportunity to demonstrate effective communication and a commitment to getting results. The SPARC has been developed by a school counselor advisory group in Los Angeles and each year the Rhode Island School Counseling Association gives awards to school districts submitting exemplary SPARCri. To create a SPARC for their schools, the elementary counselors used the data-based decision making process considered best practice in the field including collecting and examining baseline data for areas in need of improvement, developing interventions, measuring the impact of those interventions through pre and post data, and sharing results with the larger school community.
Additional Year One project activities to meet grant-specified goals and objectives have included training for all of the school counselors on Student Success Skills, an evidence-based curriculum, by the developer of that curriculum; constructing a pre-K to 5 scope and sequence for the school counseling department; creating a Curriculum Framework for School Counseling that delineates and describes the elementary school counseling program and its curriculum; writing an Administrative Handbook for School Counseling that defines the operational side of the school counseling program including roles and accountabilities and strategic and annual planning; delivering classroom curriculum at each grade level with embedded measures to assess student progress; creating brochures in multiple languages to introduce the elementary school counseling program to families and to inform families of services provided by the new counselors; and building capacity among the new school counselors through regular professional development.
Multi-State Evaluation Project
CSCORE recently completed a large evaluation study to examine the contribution of counseling programs and practices to student educational outcomes. The states of Connecticut, Missouri, Nebraska, Utah, and Rhode Island participated in the evaluation study. School-level outcome data such as attendance and suspension rates and school-level demographic data such as per pupil expenditures and the percentage of students eligible for free/reduced lunch were obtained from the Department of Education of each state. Principals and Guidance Directors from every public high school in each state were invited to complete an electronic version of the Principal and School Counselor Survey (Lapan & Carey, 2007) to collect information about school counseling practices and programs at each high school.
The evaluation model analyzed the relationship of school counseling program characteristics to student outcomes after controlling for the variability in outcomes related to demographic differences among schools. More detailed correlational analyses were then conducted to identify which specific practices were associated with identified student outcomes.Results of CSCORE's Multi-State School Counseling Research Project: Power Point presentation given at the 2010 American School Counselor Association Conference.
The Massachusetts Department of Elementary and Secondary Education (DESE) partnered with CSCORE during the 2007-08, 2008-09, and 2009-10 school years to provide professional development and training to school counselors throughout the Commonwealth on assisting students in effective and meaningful career planning for the 21st century. Topics for each year progressively built on the content discussed in previous years and were chosen based on feedback and comments from counselors about what information and skills they needed.
Year OneThe first year of the career planning and implementation professional development series focused on developing the capacity of Massachusetts vocational high school educators to help all students construct career plans that support academic achievement and career development. CSCORE facilitated three one-day conferences held in Amherst, North Andover, and Bridgewater with 80 to 100 attendees at each conference; schools were encouraged to bring teams of school counselors, administrators and teachers to the event. Representatives from different institutions of higher education, MASCA, and DESE spoke at the conferences about the essential knowledge and skills needed to create effective career plans, best practices for using assessment information, and what the research demonstrates about the impact of career planning on student success. Following the one day event, conference attendees were able to choose from a follow-up menu of professional development options to be delivered throughout the school year from CSCORE staff and other experts in the field, including an on-line career development workshop, on-site or phone consultation about the career planning process at a participant’s school, or participation in a Leadership Academy.
Many school counselors who participated in year one of the CPIC commented that they lacked the knowledge and skills needed to navigate the numerous online and electronic options available for career planning in the 21st century. CSCORE and the Massachusetts DESE therefore decided to coordinate a statewide conference on the use of technology to support effective and meaningful career planning during year two of the CPIC. Conference topics included linking career information delivery systems with career planning systems, streamlining guidance functions through technology, creating Educational Proficiency Plans that also serve as career plans; and using technology for graduate follow-up data collection. Conference participants were also able to provide feedback to the Massachusetts Educational Finance Authority (MEFA) on the development of the Massachusetts College and Career Portal.
The Massachusetts School Counselor Leadership group formed during the sustained professional development series of the previous year continued its work during the second year of CPIC. CSCORE facilitated a weekend conference in which the group was tasked with identifying best practices in career development throughout the Commonwealth. The list of best practices included transition planning; classroom guidance lessons by grade level; goal setting; cross-cultural, integrated, and multi-modal career interventions; career advisories; and specific strategies for effective Individual Planning sessions. CSCORE presented the work of the Leadership group at the spring 2009 Massachusetts School Counseling Association (MASCA) conference.
Supporting career and college readiness through career planning was the focus of the third year of the CPIC professional development series. A statewide conference was held in December 2009 at Quinsigamond College and featured two education experts, Dr. Joyce Brown from the Chicago Public Schools, and Jibril Solomon from UMass Boston. Dr. Brown spoke about her 12-Touch Program, a framework of practices for school counselors to ensure that every child in the school is connected to a caring adult. Jibril Solomon discussed the importance of making a shift in thinking about assessment information as value capital to assist students in becoming college and career ready.A second career-related goal for the year was to revise the Massachusetts Career Development Education Benchmarks. CSCORE and DESE staff, along with school counselors throughout the state, rewrote sections of the Benchmarks to reflect changes that had occurred in the global marketplace since the Benchmarks were first written in 2000. In order to provide a chance for public comment, CSCORE posted on its web site a survey to allow various constituents across the Commonwealth to offer feedback on the revised Benchmarks.
Over the past several years, the Chicago Public Schools (CPS) has instituted a major reform initiative to revitalize and enhance the work of their school counselors. CPS school counselors have adopted a best practices approach, based on the ASCA National Model, that emphasizes the development and implementation of a comprehensive school counseling program. During the 2007-2008 school year, the Chicago Public Schools (CPS) commissioned CSCORE to conduct an evaluation study to examine the benefits to CPS students when school counselors implement key components of a comprehensive school counseling program.
Every public high school in Chicago was invited to participate in the study. The sample was representative of the diverse types of CPS high schools and included general, selective enrollment, military, magnet, career academy, and special education high schools. Principals and school counselors at participating high schools completed an electronic version of the Principal and Counselor Survey (Lapan & Carey, 2007) to assess activities delivered by school counselors in each high school during the previous school year. Student data on such indicators as percentage of students receiving free or reduced lunch, attendance, state standardized test scores, Advanced Placement scores and enrollment, dropout rate, and post secondary plans were also collected from the CPS district.
Major analyses of the Principal and Counselor Survey focused on three critical variables: Non-Guidance tasks, Individual Planning, and the 12 Touch Program. Both school counselors and principals were asked to assess the extent to which CPS high school counselors were carrying out such Non-Guidance tasks as performing hall duty, supervising the lunchroom, or acting as principal for the day. Individual Planning, one of the four primary components of a comprehensive school counseling program, involves assisting all students in exploring and planning for their educational and career futures, working with students one-on-one, in small groups, or in classroom settings. The 12 Touch Program is a menu of transition practices designed to intentionally connect freshman to high school. The program aims to better integrate entering ninth graders into their high schools by promoting a student’s sense of belonging and connection to school. The relationships between these variables and student data indicators were then further explored to examine the role of CPS school counselors in promoting students’ academic achievement, college readiness, and transition from 8th grade into high school.
Four key findings were reported from this study: CPS high school counselors play an important role in promoting student academic achievement; CPS high school counselors impact how students plan, prepare, search, apply for and enroll in college; CPS high school counselors play a vital role in helping 8th grade students successfully transition into high school; an implementation gap exists across CPS high schools in the organization and delivery of a comprehensive guidance program, thus advantaging some students and disadvantaging others. To read a full copy of the report, click on the following link:
The Center for School Counseling Outcome Research & Evaluation (CSCORE), funded by America’s Career Resource Network, conducted a multi-state, multi-year evaluation of The Real Game Series (RGS). The Real Game Series is a K-16 set of curriculum materials designed to bring interactive, experiential learning to classroom and group settings to increase students’ perceptions of the relevance and importance of their school experience (Jarvis & Keeley, 2003). The purpose of the evaluation was to determine the impact of the intervention on student career and academic outcomes. The evaluation focused on the RGS component that is designed for middle school (grade 7 and 8) implementation, The Real Game.
Using quantitative and qualitative methodologies CSCORE evaluated (a) the immediate learning outcomes of the Real Game; and (b) the secondary outcomes related to academic achievement such as school involvement, planning for the future, self-efficacy and motivation, homework completion, attention, and appropriate school behavior.
This evaluation of the Real Game career development education curriculum found that it is an effective intervention in several domains. The first goal of the evaluation was to determine whether students participating in the RG learned the specific content about careers and the world of work. On the post-test, the students who participated in the RG scored significantly higher on every measure of content knowledge (World of Work, Short-term Learning Objectives, and Matching) than the control group, indicating that they had learned this information more than the control group had, and that the learning that occurred is most likely due to the intervention.
The second goal of the evaluation was to determine whether participating in the RG results in measurable gains in career-related outcomes such as career maturity and career decision-making, better quality high school transition plans, and increased recognition of the relevance of academic education. On the secondary outcomes that were measured, students who participated in the RG scored significantly higher than the control group on the measure related to future orientation, indicating that they were more thoughtful about future plans than the control group. The RG group was not significantly different than the control group on the measure of career maturity. The long-term learning outcomes, measured qualitatively, found some indication that students who participated in the RG were more aware of the importance of making good decisions about high school academics, and that those choices would impact their future career options.
The final goal of the evaluation was to determine whether students who completed the Real Game intervention showed changes in factors that have been demonstrated by research to be related to academic achievement and appropriate goal-directed, school behavior (e.g., self-efficacy, motivation, school engagement/social bonding, decision-making skills, goal-setting skills, and social-problem solving skills). This study found that there were significant differences for the Real Game participants in the domains of self-efficacy, school engagement, and prosocial behavior, but not motivation, decision-making or goal-setting. On measures of self-efficacy, school engagement, and prosocial behaviors the post-test scores for the Real Game group were significantly higher than the control group, indicating that the intervention impacted these beliefs and behaviors.
In 2006, the College Board’s National Office for School Counselor Advocacy sponsored a study conducted by the National Center for School Counseling Outcome Research & Evaluation of the school counseling practices, skills, and dispositions that are found in these award-winning schools.
The Center for School Counseling Outcome Research & Evaluation conducted an evaluation of the Bridges Transitions career exploration and awareness program in Middle and High Schools across the country. The evaluation employed the Missouri Comprehensive Guidance Evaluation Survey to assess career and career related outcomes, and indicators of school engagement through academic achievement and attendance.
The evaluation was sponsored by Bridges Transitions.
Evidence Based Practice Panel
The Evidence-Based Practice Panel (EBP) was established in 2003, based on the recommendations derived from our Delphi Study of research needs for the field of school counseling. The panel is tasked with reviewing the research literature to define practices in the field of school counseling that are evidence-based or promising practices. Evidence-based practices are practices which have been rigorously studied and evaluated, and have accumulated research evidence which consistently shows that those practices are effective. Promising practices are practices that have been rigorously studied and evaluated, but have not yet accumulated enough research evidence. The EBP panel has developed a protocol and rubric for evaluating the extent to which school counseling practices are promising or evidence-based.
The Evidence Based Practice Panel Members
John Carey, Ph.D., Director, National Center for School Counseling Outcome Research & Evaluation, University of Massachusetts Amherst
Carey Dimmitt, Ph.D., Associate Director, National Center for School Counseling Outcome Research & Evaluation, University of Massachusetts Amherst
Trish Hatch, Ph.D., Director, School Counseling Program, San Diego State University;
Richard T. Lapan, Ph.D., Professor, University of Massachusetts Amherst
Susan C. Whiston, Ph.D., Professor, Indiana University
EBP Power Point Presentations
The rationale and work of the EBP Panel were presented during the ASCA Conference in Orlando, FL in June of 2005:
- Presentation with Full Color (best for viewing online- PDF, 442kb)
- Presentation in Black and White (best for printing – PDF, 295kb)
Report of the National Panel for Evidence-Based School Counseling: Outcome Research Coding Protocol and Evaluation of Student Success Skills and Second Step – Report prepared for ASCA conference in Orlando applying the EBP protocol to the Second Step and Student Success Skills programs.
The Center for School Counseling Outcome Research & Evaluation conducted a Delphi Study which began in mid-February, 2003. The study was completed to help address the need for additional research in the field of school counseling. The study determined the critical research questions and proposed a research agenda for the school counseling field. The results of the study were published in the Association of Counselor Education and Supervision journal.
For more specific details related to the results of this study, click here.
The Delphi Technique consists of several rounds of questions presented to a panel of experts in the field of school counseling. It is a research method for generating ideas and gathering comprehensive information from a wide variety of experts in a given field. The purpose of this process is to facilitate consensus on a certain topic. Data gathering occurs via email and consisted of three “rounds” of questions related to the needs of school counseling research. The Delphi Technique allows busy experts the opportunity to interact with one another to generate new ideas to solve a problem. This is done with the help of a researcher who facilitates the process and shares information with the expert panelists in a confidential manner.
An initial survey or questionnaire was sent to all Delphi panelists. These were returned to the researcher who compiled all the ideas that were presented. A second survey instrument was created based on this information. The second survey was then sent out to all panelists, who were asked to rank order, modify, and/or add to their initial responses based upon their review of other members’ ideas. This data was collected and reviewed by the researcher who created a final questionnaire based on the accumulated information that was presented by the panelists. The final instrument asked for further clarification and/or rank ordering of ideas that had been presented throughout the study process. A final report was prepared and distributed to panel members before results of the study were released to others.
Panelists were nominated to participate in this study based on their considerable knowledge and expertise in the field of school counseling. We believe their contribution to this study was invaluable, as each of them is already making exceptional contributions to the field of school counseling.