Comprehensive Evaluation Portfolio : Department of Psychological and Brain Sciences : UMass Amherst

The comprehensive examination in the Clinical Program serves two functions. The first is evaluative: successful completion of this comprehensive exam portfolio indicates that a student has mastered and integrated a sufficient amount of the field’s current knowledge base to warrant being advanced to candidacy for the doctoral degree. The second, and equally important, function is to further students’ educational and professional development: preparation for the examination and the examination process itself gives students the opportunity to consolidate the extensive knowledge they have acquired, produce scholarship, build their academic profile, and provide practice for future evaluative and credentialing processes such as licensure or ABPP exams. The comprehensive examination process culminates in the compilation of a Candidacy, or “Comps,” Portfolio, with multiple deliverables, that has been designed to prepare students for successful careers in clinical science following the completion of the Clinical Program. Successful completion of the Comps Portfolio is necessary for students to progress to doctoral candidacy.

The Comps Portfolio is designed to scaffold students in meeting professional developmental milestones. Whereas other comprehensive examinations (e.g., a series of essays or a test) may assess knowledge-based competency, they do not provide explicit support for students in their development as independent researchers and evidence-informed clinicians. In contrast, we have envisioned the Comps Portfolio deliverables to not only serve as a gateway to doctoral training, but also to serve as benchmarks in students’ own professional development, thereby functioning as a springboard to future careers in clinical science. This process, however, necessitates much forethought and planning. Therefore, we encourage students to begin strategically planning their own Comps Portfolio plan in collaboration with their Advisors beginning in Year 1.

Once a plan is established, students will convene a Comps Committee consisting of three PBS faculty members who are all members of the Graduate Faculty (at least two of whom are primarily affiliated with Division 4, and including the student’s Advisor). The procedures and timeline for Committee review are described below. The Comps Portfolio is designed to demonstrate 6 domains of competency: psychological/neuropsychological assessment, psychotherapy, psychopathology, research methods and statistical analyses, multicultural issues, and professional ethics, as indicated in Appendix 1. The student will work with his or her Committee to determine sufficient “coverage” of each of the 6 competency domains across the following Comps Portfolio deliverables:

Candidacy Portfolio

One Professional Presentation
1. The student must be the first author on a presentation (oral or poster) based on original empirical research and presented at a regional, national, or international conference.
2. This deliverable serves to demonstrate (fully, or at least partially, as per the Committee’s judgment) competencies in research methods and statistical analyses, ethical conduct of research, and, depending on content, it may also demonstrate (fully, or at least partially, as per the Committee’s judgment) competencies in multicultural issues, psychopathology, assessment, and/or psychotherapy.
3. The abstract and either the poster reprint or presentation slides must be included in the portfolio.
One First-Authored Report of an Empirical Study
1. The student must be the first author on an original empirical research report that is either accepted (fully or provisionally) to a peer-reviewed journal or submitted to a journal and approved on a pass/fail basis by the Comps Committee. It is acceptable for this publication to be a product of the master’s thesis work.
2. This deliverable serves to demonstrate (fully, or at least partially, as per the Committee’s judgment) competencies in research methods and statistical analyses, ethical conduct of research, and, depending on content, it may also demonstrate (fully, or at least partially, as per the Committee’s judgment) competencies in multicultural issues, psychopathology, assessment, and/or psychotherapy.
3. The paper and evidence of submission/acceptance must be included in the portfolio.
4. If submitted to the Committee for approval and failed, the student can re-submit in a timeframe agreed upon by the Committee. The Committee will also provide oral and/or written feedback to help guide the resubmission.
One Additional Publication
1. The student can be in any authorship position on an empirical, conceptual, or review article, or a book chapter, that has been accepted (fully or provisionally) or submitted and approved on a pass/fail basis by the Comps Committee.
2. This deliverable serves to demonstrate (fully, or at least partially, as per the Committee’s judgment) competencies in research methods and statistical analyses, ethical conduct of research, and, depending on content, it may also demonstrate (fully, or at least partially, as per the Committee’s judgment) competencies in multicultural issues, psychopathology, assessment, and/or psychotherapy. The paper and evidence of submission/acceptance must be included in the portfolio.
3. If submitted to the Committee for approval and failed, the student can re-submit in a timeframe agreed upon by the Committee. The Committee will also provide oral and/or written feedback to help guide the resubmission.
One Grant Proposal
1. The student must be the Principal Investigator on a grant proposal (≥ 3 pages long) submitted or approved on a pass/fail basis by the Comps Committee (provided that the funding mechanism is identified with a specific submission date that falls after the time frame for which the student is attempting to complete the Comps Portfolio) to any external (federal, state, community, foundation, or professional) organization to support the conduct of research.
2. This deliverable would provide the student with essential grant-writing experience. Ideally, it would also provide support for the student’s dissertation, both in terms of funding for research and by providing a writing opportunity that parallels that of the dissertation.
3. This deliverable serves to demonstrate (fully, or at least partially, as per the Committee’s judgment) competencies in research methods and statistical analyses, ethical conduct of research, and, depending on content, it may also demonstrate (fully, or at least partially, as per the Committee’s judgment) competencies in multicultural issues, psychopathology, assessment, and/or psychotherapy.
4. In addition to working closely with your Advisor, additional support for the completion of this research grant can be obtained, as noted in Appendix 2.
5. The grant, as well as evidence of its submission, must be included in the portfolio.
6. If submitted to the Committee for approval and failed, the student can re-submit in a timeframe agreed upon by the Committee. The Committee will also provide oral and/or written feedback to help guide the resubmission.
One Case Conceptualization Presentation
1. A 30-50 min oral presentation of an assessment and/or psychotherapy case delivered to the Clinical Community during a Program colloquium (or, if not possible, another scheduled presentation time) in consultation with the clinical supervisor and to be approved on a pass/fail basis by your Committee.
2. This deliverable should present a completed assessment/therapy case or an ongoing therapy case that has been seen for a minimum of 4 sessions.
3. This deliverable serves to demonstrate (fully, or at least partially, as per the Committee’s judgment) competencies in psychopathology, assessment, multicultural issues, and/or psychotherapy, and, depending on content, it may also demonstrate (fully, or at least partially, as per the Committee’s judgment) competencies in research methods and statistical analyses and ethical conduct of research.
4. Please see Appendix 3 for detailed Case Presentation Instructions.
5. If the Committee fails the presentation, the student will be required to present another presentation to the Committee or submit an in-depth paper covering the same areas proposed for the presentation, at the Committee’s discretion. The Committee will agree upon the length, content, and timeframe for submission. If this deliverable is failed, the student can re-submit in a timeframe agreed upon by the Committee (after two failures, the Committee will report to the Faculty as a whole to determine next steps in the student’s training – see below).
Additional Deliverables
1. Students can submit additional elements described above (e.g., for any but the case presentation) to demonstrate competencies in the 6 domains for which the Committee deems there is insufficient coverage in the original Comps Proposal.
2. Similar standards apply for these additional elements.
Optional Oral Defense
1. Upon convening their Committee, students will propose how their Comps Portfolio covers the required 6 domains of competency.
2. If there are domains of competency not adequately addressed by elements 1-5 of the Portfolio, students may propose to cover additional elements in an oral defense scheduled with their Committee.
3. Unless special permission is obtained from the Clinical Program faculty, no more than 2 competency domains should be met fully with an oral defense.
4. These oral defenses will be evaluated at the Committee’s discretion, resulting in a pass/fail determination.

Timelines and Grading

The Comps Portfolio deliverables can be completed at any time after enrollment in the Clinical Program.

Steps:

Students are required to complete the Comprehensive Examination Proposal Form and have it approved by a committee of three graduate clinical faculty members after completion of their Masters' degree. The student's Comprehensive Examination committee chair (in most cases, the student's primary research advisor) is required to inform the GPD of Comps approval in order for the student to be eligible for doctoral candidacy (pending dissertation proposal defense.
Once elements 1-3 of the Comps Portfolio are completed, students form a Comps Committee in consultation with their Advisor. Please note that students may request a Comps Committee before completing any elements of the Comps Portfolio, and submit for review a preliminary Comprehensive Examination Proposal prior to completing any Comps elements.
Once a Committee is formed, students submit to their Committee a Comprehensive Examination Proposal Form (see Appendix 4). The proposal will indicate how the students will demonstrate competency across all six domains, and the Committee will evaluate this coverage prior to a meeting. The student should submit at this time deliverables 1-3 (in some cases, this will simply involve proof of presentation or publication; in others it will involve an actual manuscript to be reviewed by the Committee). Initial Comps Proposals are evaluated yearly/on a rolling basis. Please provide 8 weeks for your proposal to be reviewed (not including winter or summer breaks). After review of domain coverage and deliverables 1-3, the Committee and student will schedule a Comps Proposal Meeting. During this meeting, the Committee will either approve the entire plan, or, if applicable, work with the students to revise domain “coverage” plans and/or plan for resubmissions of any failed 1-3 deliverables. The Committee and student will also establish a timeline for completion of the entire portfolio.
Once a Comps Proposal is approved, students should gather or continue to work on their remaining proposed deliverables (i.e., 4 and 5, and any resubmissions of 1-3). Students can furnish evidence of a grant submission or request a Committee review of a grant proposal at any time post-proposal (students should allow 4 weeks for said review). In the case of a failed grant deliverable, the Committee should convene a feedback meeting to establish a plan for re-submission.
Based on a time frame established with the Comps Committee, the student should contact the Colloquium Committee to schedule the Case Presentation. In many cases, though certainly not all, this will be the final deliverable. A case presentation date must be requested prior to the end of the preceding semester.
Once all deliverables are completed and/or approved, the student’s advisor will submit a memo to the Graduate Program Director indicating the results. This, and having already completed the Department’s required 2-course statistics sequence, will trigger advancement to candidacy

Note: Students should enroll for comps credits (PSYCH 796A) during the semester that they anticipate completing their portfolio.

Potential Comps Timeline

Time	Task
Years 1-3	In consultation with your Advisor, form Comps Committee Work diligently in consultation with your Advisor to complete Comps Portfolio deliverables 1-3
End of Year 3	Submit Comps Proposal Form to Committee Committee reviews competency “coverage” and reviews deliverables 1-3 Committee meets to determine if you can proceed with deliverables 4 and 5, and if deliverables 6 or 7 are necessary
Year 4	Receive final approval or revisions from Comps Committee Revise as needed until advanced to candidacy or other programmatic action is taken

The full Comps Committee reviews each element of the Comps Portfolio (see Appendix 5 for form). Students who have elements of the portfolio deemed “unacceptable” or any domain of competency considered inadequately addressed are required to revise and resubmit those elements within one year of the time the feedback was received. After the initial submission, students may resubmit (with revised elements) their Comps Portfolio twice more for review (i.e., 2 total resubmissions allowed). If any element of the portfolio is deemed “unacceptable” or any domain of competency is considered inadequately addressed after two such resubmissions, his or her suitability for doctoral study in clinical psychology will be reviewed, and be regarded as probable cause for dismissal from the program. Written feedback from the Comps Committee will be given to the student, using the form appended. The Comps Committee is encouraged to provide as much detailed feedback as possible for deliverables that are failed. For passed deliverables, the Committee may or may not choose to give detailed feedback. Students are always encouraged to seek feedback from Committee member.

Appendix 1 - Domains of Competency Addressed by Comprehensive Examination

The Comps Portfolio is designed to demonstrate 6 domains of competency: psychological/neuropsychological assessment, psychotherapy, psychopathology, research methods and statistical analyses, multicultural issues, and professional ethics. The student will work with his or her Committee to determine sufficient “coverage” of each of the 6 competency domains across the following Comps Portfolio. For instance, simply obtaining IRB approval for the conduct of research would not cover the “ethics” domain, but careful consideration, explanation, and management of ethical issues relevant to the conduct of research study might cover this domain.

Psychopathology

Competency in this domain was assessed via (a) [1, 2, or 3] peer-reviewed publications and presentations demonstrating expertise in this domain, (b) sophisticated understanding of psychopathology, as evidenced by its assessment and treatment in a clinical case, and/or (c) sophisticated consideration of these issues in a grant proposal.

Psychological/Neuropsychological Assessment

Competency in this domain was assessed via (a) [1, 2, or 3] peer-reviewed publications and presentations demonstrating expertise in this domain, (b) administration and accurate interpretation of clinical assessment tools, and/or (c) sophisticated research-informed selection of assessment tools for a grant proposal.

Research Methods and Statistical Analyses

Competency in this domain was assessed via (a) [1, 2, or 3] peer-reviewed publications and presentations demonstrating expertise in this domain, (b) sophisticated use of these issues in the assessment and/or treatment of a clinical case, and/or (c) sophisticated consideration of these issues in a grant proposal.

Psychotherapy

Competency in this domain was assessed via (a) [1, 2, or 3] peer-reviewed publications and presentations demonstrating expertise in this domain, (b) effective selection and implementation of clinical intervention, and/or (c) sophisticated research-informed selection of interventions in a grant proposal.

Professional Ethics

Competency in this domain was assessed via (a) [1, 2, or 3] peer-reviewed publications and presentations demonstrating expertise in this domain, (b) careful consideration and management of these issues in a clinical case, and/or (c) sophisticated consideration of these issues in a grant proposal.

Multicultural Issues

Competency in this domain was assessed via (a) [1, 2, or 3] peer-reviewed publications and presentations demonstrating expertise and/or sensitivity in this domain, (b) careful consideration and management of these issues in a clinical case, and/or (c) sophisticated consideration of these issues in a grant proposal.

New Overarching Domain of Competency Assessed Via Portfolio

Clinical Science

Competency in this domain was assessed via (a) the establishment of a clinically-relevant research portfolio, (b) the execution of a research-informed clinical assessment/treatment, and (c) demonstration of ability to communicate clinical research via a grant proposal and scientific presentations.

Appendix 2 - Resources for Grant Development

Resources for Grant-Related Questions

Office of Grants and Contracts https://www.umass.edu/research/research-administration
Center for Research on Families http://www.umass.edu/family/home

Grant Funding Options (this list will evolve over time)

NIH National Research Service Award http://grants.nih.gov/training/nrsa.htm
NSF Graduate Student Research Fellowship http://www.nsf.gov/funding/pgm_summ.jsp?pims_id=6201&org=NSF
Check other websites for eligible funding options:
- http://www.apa.org/apf/funding/scholarships/index.aspx
- http://www.apa.org/science/about/psa/2009/11/external.aspx

Appendix 3 - Instructions for Case Presentation

Case presentations are intended to demonstrate adequate understanding of clinical science and competency in clinical practice. As well, these presentations are intended to generate intellectual discussions. As such, case presentations can be most useful when complex cases are selected, and student should be reassured that they will not be evaluated based on the outcome of the case. Rather, students will be evaluated on the use of evidence-based, thoughtful assessment and intervention, articulation of sophisticated clinical theory and case conceptualization, and demonstration of flexibility and thoughtfulness in addressing any barriers that arose over the course of the assessment or treatment. The case supervisor does not need to be a member of the Comps Committee.

Case presentations are intended to last for 30-50 minutes. They are ideally a platform for discussion. As such, please provide at least 15 minutes for questions, and consider how to generate discussion during your presentation.
Cases should be selected that are completed or that have sufficient (i.e., 4 sessions or more) duration to permit a complete case presentation.
The case presentation may demonstrate competencies in different domains for different students, depending on their needs, as indicated on the Proposal. The specific domains to be addressed should be identified in conjunction with the Comps Committee. Regardless of the domains identified, the case presentation should be consistent with a clinical science model, relying on evidence where appropriate to guide clinical practice.
A case presentation will be deemed “passed” if all Comps Committee members rate it as acceptable
Case presentations should include the following elements:

de-identified demographic information,
presenting problem,
relevant history,
assessment findings and diagnoses,
case formulation and treatment plan,
treatment outcome and progress measures (required if a psychotherapy case),
case disposition,
incorporation of evidence throughout to support case formulation, selection of appropriate assessment tools, interventions, and progress measures,
consideration of the role of client and therapist cultural backgrounds and biases on the assessment and treatment process, and
ethical considerations in conceptualization and treatment of this case (i.e., (autonomy/safety, risk management, reporting, confidentiality).

Case Presentation Proficiency Benchmarks For Reference Only – Not an Evaluation Tool

I. Assessment

1) Appropriate assessment methods and/or tools are selected taking into account existing research evidence, client’s presentation or presenting problem.

2) Student appropriately uses, interprets, and integrates assessment data skillfully.

3) Assessment findings are used appropriately in the generation of a treatment plan for the client, where relevant.

4) Selection of assessment methods and/or tools takes into account ethnic, cultural and class variables.

II. Formulation

1) Formulation is appropriately grounded in research and clinical data.

2) Formulation represents a coherently stated theory of personality, psychopathology, and psychotherapeutic change

3) Student is aware of alternative formulations and limits of own formulations.

4) Formulation of the problem takes into account the influences of ethnic, cultural, and class variables.

5) Formulation of the problem demonstrates an understanding of the client’s level of acculturation or stage of adaptation to the dominant culture.

III. Intervention Strategy

1) Integration of relevant research and clinical data in intervention selection.

2) Ability to integrate patient expectations into interventions when appropriate.

3) Appropriate consideration of time limitations, resource constraints, and community resources in the choice of interventions.

4) Shows evidence of ability to modify therapeutic approach when necessary.

5) Intervention strategies are consistent with the rationale for the treatment provided and the case formulation

6) Intervention strategies are consistent with the client’s level of acculturation, language, cultural values, and interpersonal styles.

7) Clarity and thoroughness of treatment program.

IV. Relationship

1) Ability to establish atmosphere that facilitates effective communication.

2) Demonstrates concern and respect in an empathic manner.

3) Able to establish and maintain professional objectivity.

4) Assesses and when appropriate discusses the influence of the client’s ethnicity, culture, and class on the development of the therapeutic relationship.

V. Self Examination

1) Understands own personality and biases.

2) Able to question and reflect on own feelings, attitudes, and behavior during treatment process.

3) Able to recognize limits of competence.

4) Examines the impact on treatment of own attitudes about ethnicity, culture, and class.

VI. Professional Communication Skills

1) Able to communicate orally.

2) Responsive to questions and feedback.

3) Manages stress appropriately.

4) Quality of written work/presentation.

Appendix 4 - Comprehensive Examination Proposal Form.pdf .docx

Appendix 5 - Comps Portfolio Feedback For.pdf .docx

Comprehensive Evaluation Portfolio