The Educator’s Portfolio : Guidelines for Construction of an Educator’s Portfolio

Guidelines for Construction of an Educator’s Portfolio

Purpose: The purpose of this guide is to help faculty at the University of Miami School of Medicine assess and document their educational activities for the purpose of appointment, promotion, and tenure.
Click Here to view or print the entire guide as a PDF file.
If you prefer use the on-line version with scroll bars

Contents:

Criteria for Promotion at UMSM
Background
What is an Educational Portfolio?
Writing an Educational Portfolio
Summary of the Nine Categories for Documenting Educational Activities
Development of a Philosophy of Teaching and General Teaching Aims
CME and Mastery of the Subject
Contact Time, Availability and Accessibility
Teaching Performance and Skills
Instructional Innovation
Course Development and Leadership
Efforts to Improve Teaching
Educational Research and Scholarship
Educational Administration and University Service

Criteria for Promotion at UMSM

Criteria for Promotion for Faculty at the University of Miami School of Medicine The policy of the University of Miami School of Medicine (UMSM) with respect to academic promotions and tenure is set out in its “Guidelines for Faculty Appointment, Promotion and Tenure,” approved by the Medical School Council on July 26, 1996. The UMSM has five full-time faculty tracks. Education is identified within four of the tracks as one of the major areas upon which promotion may be based. The exception is the Research track. However, candidates for promotion who are in the Research Track are encouraged to note evidence of their participation in the School’s teaching mission. As a candidate you have the freedom to choose the relative weight given to these three areas in your application. You are advised to inform the promotions committee, in your cover letter, of the relative weights of these three areas in your academic contribution. Finally, it is important within every track except the Research Track that you have made contributions to more than one focus area. The table below summarizes the criteria for promotion from assistant professor to associate professor and from associate professor to professor for each of the five tracks. Even candidates in the research track may apply for promotion based primarily on their educational activities providing that they show evidence of conducting research on teaching. Therefore, whatever your track, you can choose education as your primary criterion for promotion. Moreover, even if education is not a major component of your application, evidence of competence or of tangible contribution to education is an important part of your application as a second area.

Table of Appointment, Promotion and Tenure Criteria

Track

Asst. Prof. to Assoc. Prof.

Assoc. Prof. to Professor

Tenure earning or Tenure (Appointment requires excellence in one focus area and competence in the other two.)

National recognition in your field and scholarship in at least one of the three focus areas.

Continued growth in stature, further excellence, and continued productivity (since obtaining Asst. Prof status) in at least one focus area.

Clinical Educator (Full-time faculty whose primary professional activities are teaching and clinical services.)

Tangible contributions in more than one focus area within your referral area (local, regional, state).

Tangible contributions in more than one focus area and a national reputation.

Research (Faculty members who devote virtually their entire effort to research).

Demonstration of a national reputation of your research and a record of independent funding.Evidence of participation in the School’s teaching mission should be noted and you should be “responsible” in service to the institution.

Demonstration of continued growth in stature and gaining an international reputation for your research. Evidence of continuous grant support from federal programs and high quality reports of work. Other supporting evidence includes service and teaching activities.

Adjunct (Faculty members who devote virtually all of their effort to specialized clinical service or teaching. Appointment requires outstanding service and/or teaching)

Evidence of accomplishments in your area of clinical or teaching activity.Local recognition.

Evidence of continued growth in stature in your area of clinical or teaching activity and documentation of further excellence since the previous appointment or promotion

Affiliated (Relationships between the depts. of Medicine, Pathology and Radiology with Mt. Sinai Medical Center of Miami Beach).

Evidence identical to that required in other faculty tracks in four areas, clinical activities, teaching, clinical-research, and committee service and administrative duties.

Similar to the promotion criteria of the other tracks. Evidence of growth in meeting the needs of your position since the previous appointment/promotion in the four areas.

Table based on the University of Miami School of Medicine Guidelines for Faculty Appointment, Promotion, and Tenure, Approved by the Medical School Council July 26, 1996.

Back to the top

Background

History

The idea of the teaching portfolio began with the publication of A Mini-Guide to Preparing A Teaching Portfolio, published in the spring of 1978 in Ontario (Shore et. al, 1978). Since then, the concept of a teaching portfolio, alternately called a teaching dossier, has become an international movement fired by the growing need to measure educational activities so that they can be rewarded.

The techniques for evaluating and documenting educational activities have advanced rapidly in the last decade as universities increasingly acknowledge the importance of teaching in their promotion policies. Although the policy of the University of Miami School of Medicine with respect to the primary role of educational activities in academic promotions and tenure is set out in its “Guidelines for Faculty Appointment, Promotion and Tenure” (approved by the Medical School Council on July 26, 1996) few professors have achieved promotions based primarily on teaching. The main reason for the small number of promotions based on teaching is that the techniques for measuring and reporting educational activities are not widely known either to teachers or to promotion committees. Documentation of teaching activities has suffered from the same difficulty as documentation of community service activities, namely the absence of an expert peer review process to make qualitative judgments about candidates’ teaching contributions. Moreover, the evaluations that have been available have seldom provided comparisons with the candidate’s peers. Thus most of the information typically submitted to promotion committees in support of teaching has required the committee to make qualitative judgments that they frequently have been unable or unwilling to make. The challenge of evaluating educational activities is to establish criteria and standards that can measure excellence in teaching similar to the criteria and methods employed in the evaluation of research publications and grant applications.

What Evidence is Accepted for the Documentation of Educational Activities?

The Guidelines for Faculty Appointment, Promotion and Tenure of UMSM (Approved July 26, 1996) uses the broad definition of teaching that is accepted by most educators today. The guidelines require not only a synopsis of teaching assignments and evidence of teaching skills but also evidence of contributions to curriculum development and to educational administration.

This broad definition of teaching has been specified in a recent document, “The Educator’s Portfolio” (Approved 10/18/99 by the Faculty Council). The Educator’s Portfolio defined twelve categories of educational activities that could be included as evidence of educational contributions. These categories have been reorganized into nine categories in this updated version.

Methods of Evaluating Educational Activities

In addition to indicating which activities might be included in your teaching portfolio the Guide also provides methods for evaluating and documenting each teaching activity. The methods are elaborated with examples to help you implement them on your own. Please read the descriptions of teaching under each component and ask yourself, in each case, whether you have engaged in this type of teaching. After determining the activities relevant to your particular teaching profile choose those methods of documentation that are appropriate to your situation, the time you have available and other constraints. Follow the instructions to implement each method. Documents that are worth saving and the kinds of evaluations you might want to arrange are noted. You may discover activities in which you are presently engaging that you never thought were teaching activities. Remember that your teaching portfolio need not look like anyone else’s.

Note that some measures of teaching, such as student ratings, require central administration, distribution, and collection. The results of some of these ratings may be sent to you automatically but others will not be and you must request these from the appropriate offices. This Guide will explain which data you can expect and where to call to obtain the data that is not sent automatically. Other measures, such as self-assessment or attendance at teaching improvement seminars do not require central administration, and you will be able to carry them out yourself. Still other measures might require you to request help from colleagues. Whatever the case, this Guide should help you to obtain the documentation you require. Since teaching is increasingly becoming recognized as highly context specific (Edgerton, 1989), and since teachers are usually assessed within a departmental context, the conventions for portfolios should be expected to vary between departments and even between individuals. It would be helpful if departments developed a set of conventions to guide the collection of documentation and the writing of teaching portfolios. We invite each department to modify these generic guidelines to suit their own needs.

Back to the top

What is an Educational Portfolio?

An educational portfolio is a set of materials documenting teaching activities in the broadest us of the word “teaching” that includes not only teaching performances and skills but instructional design, mentoring, educational administration and many other areas. The portfolio is fast becoming the accepted vehicle for documenting teaching just as the CV has long been the accepted vehicle for documenting research. Typically the portfolio is included with your Curriculum Vitae in order to help promotion committees assess your ability to teach, your commitment to teaching, and your efforts to improve teaching. Indeed, the Educator’s Portfolio has been described as an “educational curriculum vitae.”

Advantages of the Educational Portfolio for the Individual and the School

From the individual faculty member’s viewpoint, the major advantage of the Educator’s Portfolio is its usefulness as documentation for promotion and reappointment. The Medical School benefits from the portfolio in two ways. One benefit is that systematically documented evidence of teaching helps Promotion and Tenure Committees value teaching in their decisions. Another benefit is that portfolios are often the stimulus for faculty development resulting from the faculty member’s reflection on her/his philosophy of education, assessments, continuing education, and strategic planning.

Individual Responsibility

The development and content of the Portfolio is the responsibility of the individual faculty member as it is designed to portray individual growth and contributions.

Back to the top

Writing the Educational Portfolio

Once you have collected information documenting your teaching activities, you are ready to organize the information and write the summary pages that constitute a portfolio. This should not be an onerous task. For one thing the portfolio need not be lengthy. Recommendations from writers in the field vary from Knapper (1978), who believes it should not exceed three pages, to Seldin (1987) who would recommend six or seven. The bulk of the material should be in appendices. Examples of portfolio writing are to be found in recent publications on the subject (for example, O’Neil and Wright, 1992; Seldin, 1991) and examples from colleagues here at the UMSM on this website.

Since there are a number of teaching activities within each component, the total number of teaching activities is very large. Indeed, the number may seem daunting until you realize that the list is merely a menu of the teaching activities in which all teachers may engage. No one could engage in all of these teaching activities any more than one would order every dish on a menu. Each faculty member has a unique set of teaching activities.

Collecting the Information

The following pages present suggestions for many different kinds of information that you might collect in support of your promotion and ways of obtaining that information. Without systematic filing of this information you may forget the meaning of some of the notes, letters or appraisals when you sit down to compile and write your portfolio. One suggestion is to keep the information in a binder or separated files.

Step-by-Step Procedure to Writing the Educational Portfolio

If you would like to follow step-by-step procedure, the following is based on 1986 version of the Canadian Association of University Teachers guide to a teaching portfolio:

STEP 1: CHOOSING THE RELEVANT COMPONENTS AND ACTIVITIES. As a first step read through the section on the documentation of teaching, noting which of the components of teaching and which of the evaluation methods are relevant to your own situation. Note that this guide is not a template for constructing your portfolio. Think of it as a kind of menu. The many headings and methods of evaluation are merely suggestions from which you may choose. The actual headings used on your portfolio will likely be far fewer than the number of headings used in this guide. STEP 2: SORTING YOUR MATERIALS. Sort through all of the materials that you have collected and organize them under the appropriate headings. All of your materials may not fall under the categories that are suggested in this Guide. You may have to add categories. And you may have to gather more evidence when documentation for an activity is thin. STEP 3: WRITE SUMMARY STATEMENTS. Write a summary statement for each category of information that you have chosen. The summary statement should draw the reviewers’ attention to your conclusions based on the data. The data itself can be relegated to appendices or filed so that it will be available if the reviewing committee requests it. STEP 4: PREPARE A BINDER. Some writers recommend inserting the teaching portfolio into your CV, but the CV is usually a simple list; it is no place for the kind of descriptive material that is useful in documenting teaching activities. A better alternative is to submit the teaching portfolio in a separate binder, including all of its appendices. If you choose the latter you will have a little more freedom to be discursive in your summaries and explanations.

Back to the top

Summary of The Nine Categories for Documenting Educational Activities

1.Development of General Teaching Aims or a Philosophy of Education
2.CME and Mastery of the Subject
3.Contact Time, Availability & Accessibility
4.Teaching Performance and Skillsa.Pre-Clerkship Teaching: Year 1 and 2 courses
b.Clerkship and Sub-Internship Teaching
c.Postgraduate Teaching, Post doc and fellow supervision
d.CME Teaching
e.Grand Rounds
f.Advising and Mentoring

5.Instructional Innovation
6.Course Development & Leadership
7.Efforts to Improve Teaching/ Faculty Development
8.Educational Research & Scholarship
9.Educational Administration

Back to the top

1.Development of a Philosophy of Education or General Teaching Aims

Expert teachers—those who are in the process of continually improving their teaching—are expected to reach beyond short-term teaching strategies. They should be aware of their broad aims as teachers. Some teachers even write a personal teaching philosophy that includes some of the following components: their personal theory of learning, characteristics of a good teacher, aims of instruction, learner objectives, beliefs about roles and responsibilities of students and teachers; and variables that promote learning. Evidence that you have developed a teaching philosophy need not be more than a coherent paragraph addressing some of the components mentioned above. Such a statement is relevant documentation of your expertise as a teacher and should be included in your portfolio.

Moreover, by delineating your philosophy at the beginning of the Portfolio you provide the reader with a context within which the portfolio can be evaluated.

Back to the top

2.CME and Mastery of the Subject

Although mastery of the subject is essential to most kinds of teaching, the mastery of medical teachers is often taken for granted. It is assumed that incompetent clinicians would not survive the rigorous discipline imposed by their medical practice. Still, physicians are often asked to teach in areas that are not directly related to their practices and are often called upon to explain phenomena in terms that are more basic than their usual practice requires. Conversely non-clinical teachers are often required to connect basic processes to clinical examples, which they may not have. In short, there are many circumstances in which medical teachers, who are fully competent in their specific sphere of practice, might lack the breadth or type of mastery of the subject required for teaching. Evidence of efforts to enhance one’s mastery of the subject is therefore an important aspect of documentation of teaching excellence. There are several methods of evaluation:

COURSES OR CREDENTIALS

List all of the courses or credentials that you have earned relevant to gaining expertise in the subject matter that you teach.

HOURS OF STUDY

Estimate the number of hours that you have spent reading and preparing the subject matter for your teaching.

CURRICULUM REVIEW

Engage a subject expert to critically review your curriculum, handouts, lecture plans, or any written materials. Reviewers should be asked to identify any ideas that do not reflect the current status of the field and to state the strengths of the particular selection and organization as well. Their report can be submitted as evidence of your being in touch with the field and included in your teaching portfolio.

PEER OBSERVATION

Ask a subject matter expert to observe your teaching and comment briefly, in writing, about the validity and currency of your information.

STUDENT QUESTIONNAIRES

It is generally accepted that although students are valid judges of the effectiveness of the teaching in helping them learn, they are not valid judges of the teacher’s mastery of the subject. Students do have opinions about their teacher’s mastery of the subject, and usually will answer a questionnaire item that asks for their opinion about their teacher’s competence, but since they do not have expert knowledge of the subject matter themselves, they may be wrong. Still, items on a student questionnaire that ask for students’ opinion of their teacher’s knowledge of the subject matter have been considered valuable enough to have been included on some of the most widely used questionnaires. Such information supplies evidence of students’ confidence in a teacher’s ability.

Report the data from questionnaire items asking students about their perception of your knowledge of the subject matter.

Back to the top

3.Contact Time, Availability & Accessibility

The decision to take time for teaching in the busy setting of a typical medical school is a valid measure of commitment to teaching. Most teachers in schools of medicine engage in supervision of residents and graduate students in addition to more conventional teaching. Teaching may end when a lecturer leaves the classroom but supervision and clinical teaching can take as much time as the teacher allows it to take.

In addition, the perception of a teacher’s availability does not always match her or his actual availability. Students often complain that teachers are unapproachable even though they may have regular office hours. It is one thing to be physically present for students, it is quite another to communicate your availability to students—to be seen as willing to listen to students.

SCHEDULED TEACHING

List all of your formal teaching obligations: course title, a brief description of the course, number of students in the class, number of times the class meets. List your attending or rounding time separately. List your average weekly supervision time. A useful method of listing your teaching contributions is the use of a log [HYPERTEXT TO LOG]. Some promotion committee members have argued for the efficiency of a tabular representation of teaching activities. Here [HYPERTEXT to TABLES] are three examples of tabular forms for educational activities.

OUT OF CLASS ACCESSIBILITY

Do not forget to list out of class or clinic teaching, for example: office hours, special sessions for students, allocated time for teaching during clinical rounds, available times for informal consultation, and regular meetings with student feedback groups.

STUDENT PERCEPTIONS OF TEACHER AVAILABILITY

Report the data from questionnaire items asking students about their perception of your availability.

Back to the top

4.Teaching Performance And Skills

Teaching performance is the teaching component that is most familiar to everyone. The most common metaphor of teaching is some kind of performance, whether it be lecturing, seminar leading, group facilitating, coaching. Performance refers to activities carried out in direct, live contact with students. There are several ways of assessing your teaching performance, listed below:

STUDENT RATINGS

Student ratings are by far the most common form of teacher evaluation in use today. The research on student ratings supports the conclusion that student ratings tend to be “reliable, valid. relatively unbiased, and useful.” (Cashin, 1988). Below is a summary of the student ratings used routinely by the University of Miami School of Medicine with an explanation of how to obtain your results.

Pre-Clerkship: Year One & Two Courses

Each week the Office of Program Evaluation (Director Dr. Chi Zhang) administers a questionnaire to all first and second year students. After they receive the questionnaire on Monday students are expected to rate each individual faculty after he or she completes the topic. Then, at the end of the week, they complete a weekly review section. The questionnaire is collected on the following Monday. The individual teacher evaluations consist of a single, global question, rating the teaching on a scale from “excellent” to “poor.” Students are encouraged to add comments if they wish. The weekly review questions consist of 10 standardized items that pertain to the whole week, not to individual teaching performances.

Longitudinal Themes are also evaluated using both individual evaluations and weekly review questions. However, since the individual teachers spend longer amounts of time when teaching the longitudinal themes, students are asked three questions instead of one global question. They are asked to evaluate the teacher/topic on three variables: 1) clarity of objectives, 2) organization, and 3) relevance for medical education. You can obtain your individual evaluations, for either course or longitudinal theme teaching, by calling your course or theme director. The directors will have the results shortly after the completion of each week. If you are not sure who your course director is check the website www.mededu.miami.edu and click on “course listing,” or call the curriculum office at 305 243-6180. You may also be able to obtain the weekly review ratings for the weeks in which you taught, however these will not reflect your individual contribution to the week. A second form is used to evaluate the course. Course evaluations are administered periodically. The form varies from course to course. Results of course evaluations are available through the Course Director.

Clerkship and Sub-Internship

The Office of Program Evaluation administers a standard, centrally controlled questionnaire to the clerks. Typically at the end of each 6 or 8 week block clerks are asked to complete a rating form administered by the course director specifically for their own department. Although these forms typically do not include ratings of individual teachers, if you would like to include in your portfolio evaluation of the block to which you contributed, contact your course director or departmental chief If your department does not have a form for clerkship evaluation and you would like help in designing an evaluation instrument please contact the Educational Development Office. Go to the website http://www.mededu.miami.edu/MedEd, scroll down and click on “Educational Development Office.”

Postgraduate Teaching

Each hospital department has written evaluation forms to give feedback to attending physicians for the purpose of improving their clinical teaching skills. Often the results of such evaluations are sent to attendings and discussed with them by their hospital chiefs. If you do not have the results of your written evaluations call your departmental chief. These evaluations tend to be written for residents. If you would like feedback from other postgraduates that you supervise such as graduate students, fellows or post doctoral fellows, or even seminar teaching and rounds contact the Educational Development Office for help in designing an evaluation instrument tailored to these learners. Go to the website http://www.mededu.miami.edu/MedEd, scroll down and click on “Educational Development Office.”

CME Teaching

Evaluations of events that are registered with the UMSM Office of Continuing Medical Education are collected by that office and sent to the course director for the event (course, conference, seminar or lecture). The evaluations are usually not focused on individuals but on the entire course but if you had an important role in the event and wish to use the evaluations in your portfolio you should contact the course director.

Grand Rounds

Some Departments have their own written evaluations for grand rounds. Check with the individual department administrative assistants. Educational Grand Rounds are evaluated by questionnaires that include 3 objectives and 8 standard items. The results of these evaluations are available from the Educational Development Office at http://www.mededu.miami.edu/MedEd, scroll down and click on “Educational Development Office.”

Advising and Mentoring

Although advising and mentoring plays a huge role in the education of a physician or scientist, this type of teaching is rarely evaluated formally. The literature on mentoring is rapidly growing and includes evaluation. Anyone interested in developing or adopting an instrument to measure supervision, advising or mentoring please call the Educational Development Office at http://www.mededu.miami.edu/MedEd, scroll down and click on “Educational Development Office.”

WRITTEN COMMENTS

Although written methods of gathering student feedback take more time than ratings, they have important advantages both for the learners who complete them and for the teachers who summarize and analyze them particularly for improving teaching but also for documenting teaching performances for the purpose of promotion.

Solicited Letters from Students

Letters from students are a rich source of information both for teaching improvement and for promotion committees, but, for the latter purpose, the teacher should not solicit them herself or himself. Letters from students should be solicited using a random selection process and administered at arms length from the teacher being evaluated, for example by her or his department, Office of Program Evaluation or Educational Development Office. Students should be asked to comment on the candidate’s ability to stimulate and challenge students’ intellectual capacity and ability to influence their intellectual and skills development.

Unsolicited Letters from Students

Promotion committees have traditionally been concerned about the selection bias inherent in unsolicited letters. On the other hand, unsolicited letters are often compelling because the writers are usually highly motivated and the testimonials detailed. Unsolicited letters certainly provide a valuable addition to other sources. Keep them and include them in your teaching portfolio under this heading. Unsolicited Letters from Coordinators & Clients It is customary, when someone gives an invited lecture or teaches an elective (and occasionally when someone completes a regular course of teaching) for the person who coordinates the lecture series or the electives to write a letter thanking her or him for teaching. Some of these letters include comparisons such as “one of the best lectures in the series” or evaluative comments such as “many people have asked me for your address to ask for reprints of your paper.” Keep these for inclusion in your portfolio.

AWARDS BASED ON STUDENT OPINION

If you are fortunate enough to win a teaching award, at the very least it is evidence of customer satisfaction with your teaching. The reason that teaching awards fall under the “Teacher Performance” component of teaching is that most student-generated awards are based on teaching performance. Information about your award can be entered in two places in your promotional documentation. First, it should appear as a brief line in your CV under the heading “Teaching Awards.” It should also appear, in a more lengthy form, in your teaching portfolio. It is useful to convey enough information to aid the promotions committee in interpreting its significance. Describe the process by which the winner was determined. Did all students vote? What percentage of the students is polled? Did the student executive choose it? Describe the number of students that took part in the determination of the award. Describe how many of the awards are given and how often, Is the award based on the Hospital, Department, Faculty, or on some larger unit?

INVITATIONS REGARDING TEACHING

Invitations can be valid evidence in support of teaching excellence, providing there is evidence that you have been chosen over for particular expertise or abilities, for example, to give a lecture, help other teachers, or serve on a teaching related committee. Invitations of this sort can be a type of external validation of your teaching, or at least of your reputation as a teacher.

All the material about invitations could be entered in your teaching portfolio, under a heading entitled “Invitations re Teaching.”

TEACHING OUTSIDE YOUR USUAL OBLIGATIONS

An invitation to teach by itself is not evidence of the value of your teaching. You might have been invited because of your expertise in the subject or because your host could not find anyone else to take over the class while vacationing. On the other hand, an invitation such as the following speaks to your teaching: “We would very much like you to join our annual seminar series again this year. We have always received good feedback about your sessions.”

HELPING OTHER FACULTY & SERVING ON COMMITTEES

To be asked to help an inexperienced or unsuccessful colleague or to serve on a teaching-related committee often implies that you have been identified as a skilled teacher by the informal network of your institution. Are you being distinguished by this invitation or simply burdened with an unwanted job? It depends on the situation. There are teachers who are regularly invited to help their colleagues and to sit on committees because of their reputation. Such invitations can be taken as evidence of reputation for teaching, which is usually earned by teaching performance.

CLASS RECORDS

Previous methods of evaluation have focused mainly on verbal reports from students, written or oral. But a review of certain class records can also provide evidence of excellent teaching performance. Students vote with their feet too. Student attendance can reveal significant trends that are telling for the performance of the faculty member. Upward trends in class overall average on exams may also indicate improved teaching.

OTHER ASSESSMENT METHODS

Although student ratings and written comments are the primary methods currently used by the UMSM to assess teaching performance, there are other useful methods. Some of the more popular ones are:

Open-Ended Questionnaires Classroom Observation Interviews Focus Groups

If you choose to use one of these methods to document your teaching performance, you must initiate the process and you must organize the collection of information so that it is at arm’s length. That is, someone other than yourself has to conduct the evaluation procedure, collect and summarize the results.

For example, if you believe that an open-ended questionnaire or a focus group would be useful in the documentation of your teaching, in addition to student ratings or written comments, contact the Education Development Office for help in organizing the procedure.

Back to the top

5.Instructional Innovation

Any instructional method that is not in use in your teaching setting, whether it is new to the rest of the world or not, should be accepted as an innovation.

This category includes innovations in teaching techniques, curriculum development, on-line learning materials, course planning, and evaluation. Some examples are the following: introduction of new teaching strategies; creative development of media such as slides, tapes, film, TV, on-line learning and displays; live demonstrations, simulations, models; innovations in handouts, textbooks, syllabi, computer program, self-learning packages, problems for problem-based learning; innovations in course development or evaluation methods such as teacher or self-evaluation. Following are thirteen kinds of evidence that may used to document your innovations:

DESCRIPTION

A description of the specific innovation that has been developed including its aims, the problems that it was intended to overcome, a description of the setting and the students, an evaluation, and attempts to improve the innovation.

GRANTS OBTAINED

A list of grants obtained for the purpose of developing an innovation. List the granting agency, a brief description of its terms of reference, and the amount received. Most granting agencies have peer review processes that provide an external validation of the merit of your project.

PUBLICATIONS

List publications about the innovation and comments from the readers of those publications or users of your innovation. The publications themselves will be listed in two places: in your CV under the appropriate heading and in your teaching portfolio. The teaching portfolio is the place to include additional material such as letters about your innovation from others. For example, you might include letters from other schools asking for copies of your teaching manual or film.

AWARDS

List any awards you may have won for your innovation. Include terms of the award and a description of the judges (students, peers, etc.). STUDENT EVALUATION Enter any student evaluation results that you may have, particularly student opinion regarding the usefulness of the innovation in helping their learning. You can include data under this heading that has already been entered under the heading of performance evaluation, but select those responses that pertain to your innovation. For example, a questionnaire item or two might deal with a new simulated patient component of a course you planned or a new syllabus. Enter this data in this section as evidence of the worth of the innovation. Some of this data should be available from the Office of Program Evaluation and your course and clerkship coordinators.

UNSOLICITED STATEMENTS

Unsolicited statements attesting to the quality of your innovation can be useful evidence. Peers can attest to the usefulness and relevance of your innovation. Scholars in your academic field can attest to the uniqueness, creativity, and originality of your innovation. Administrators or instructional coordinators can attest to the role of your innovation in the instructional program, the appropriateness of the method in the context for which is was intended, and benefit to the curriculum.

PEER REVIEW MERLOT at www.merlot.org is a site that reviews, collects and distributes on-line learning materials designed by faculty for higher education. The peer reviews are performed by peer users of instructional technology, and not necessarily peer authors of instructional technology following the model of peer review of scholarship. 14 editorial boards write reviews. In addition to the written findings (review) each entry is assigned a rating based on an evaluation of three dimensions— Quality of Content, Potential Effectiveness as a Teaching Tool, and Ease of Use. There are 5 categories, called stars, 5 being the highest. A review must average three stars (or textual equivalent) to be posted to the MERLOT site.

LETTERS OF APPRAISAL

Letters of appraisal from internal and external referees may include statements that attest to the quality of your innovation. It is the responsibility of the Chair to solicit letters of appraisal. Letters solicited by the candidate or other persons are not suitable for the purpose of promotion.

ALUMNI STATEMENTS

Include statements from Alumni particularly those who can compare your innovation with previous experience in their education.

INVITATIONS

Mention invitations to present your innovation or any other public activities related to your innovation. For example, you might be invited to present your workshop on “teaching residents how to teach” at a National Conference.

REQUESTS FROM TEACHERS

Imitation is the sincerest form of flattery. Include evidence of requests from other teachers indicating interest in the innovation you have developed or seeking advice about how they can adopt your ideas. Incidentally, a comment of one of the early readers of this manual is worth repeating. She said that most of the requests she receives are by phone, leaving no trace for her portfolio. Our recommendation is to ask the requester to follow the oral request with a letter “for your files.” Most people will understand why a teacher or program director might need such letters about their innovations or programs.

STUDENT OUTCOMES

Student outcome measures may include a range of evidence such as written evaluations, achievement, letters from teachers receiving your students, and honors given to your students. Unfortunately student outcome measures seldom accurately reflect teaching efforts because a number of uncontrolled variables confound the measure, including entry characteristics of students, motivation, pressures from other courses and so on.

RELEVANT EDUCATION

Education necessary to the development of the innovation including workshops attended, literature studied, educators consulted. List the educational experiences, the connection with your innovation, and the number of hours spent.

Back to the top

6.Course Development & leadership

This category of teaching includes two sub-categories: 1) course supervising and coordinating, and 2) course design. As well, this category includes leadership within the educational program. Course supervising or coordinating includes organizing and coordinating staff and resources toward the development of a teaching program. Course design includes competence in selecting and synthesizing a body of knowledge or practice, organizing it into a coherent course that constitutes an original synthesis of material suited to a population of students, and designing an appropriate evaluation. Course design also includes the alignment of all of these components. Twelve potential measures of these activities are listed below:

DESCRIPTION

A description of the courses and programs that you coordinate, including their objectives, and a description of the setting including both students and staff, is a good beginning in the documentation of a course. If you are a course designer, include a description of the course that you designed, including its objectives, a description of the setting and the students and the staff.

BACKGROUND EDUCATION

List the skills or knowledge you had to acquire related to your role of course coordinator, supervisor, or designer, including workshops attended, literature studied, educators consulted.

GRANTS OBTAINED

List the grants obtained for the purpose of developing the program or course.

PUBLICATlONS

List any publications about the program you coordinated or course you designed, including program evaluation studies.

AWARDS

List any awards won for program coordination or course design. Include terms of the award and specification of the judges (students, peers, etc.).

STUDENT EVALUATION

Include any student evaluation such as student opinion regarding the usefulness of your program in facilitating student learning or your accessibility as a coordinator to handle complaints. If you are a course designer, include student opinion regarding the following: ·The helpfulness of the course to their learning. ·The ability of the course to stimulate critical and analytical thinking. ·The ability of the course to stimulate students to intellectual effort. ·The ability of the course to challenge students’ conceptual abilities without being too difficult for students to master. ·The degree to which the material matches the context of other material in the students’ program or the level of academic maturity of the students. ·The design of examinations that provide a valid, fair and reliable evaluation system for the course, that are matched to subject matter and course objectives, that encourage core learning of material and basic principles, that emphasize critical and analytical thinking, that stress understanding of the subject rather than memorization, and that test only what has been taught.

ALUMNI COMMENTS

If you are a course designer, include comments from Alumni, especially those who are in a position to compare your course with previous ones and to attest to the relevance of the course to future applications.

STUDENT OUTCOMES

Measures of Student Outcomes or Achievements may include a range of evidence, written evaluations, achievement, letters from teachers receiving your students, and honors given to your students.

Back to the top

7.Efforts to Improve Teaching: Faculty Development

This category includes participation in seminars, workshops, courses and professional meetings aimed at the improvement of teaching or education in general. Following is a list of some of the kinds of evidence that you may use to document your teaching improvement efforts.

ADVANCED DEGREES

The achievement of an advanced degree in education is evidence of serious commitment to enhance your educational expertise and should be listed under this heading in your portfolio.

COURSE PARTICIPATION

Participating in courses, seminars or workshops about education in order to improve the teaching and learning process is also evidence of serious commitment to educational expertise and should be listed under this heading in your portfolio. RESPONDING TO FEEDBACK If you have systematically used feedback and evaluation to make appropriate changes to your course or teaching methods, such efforts should be carefully documented as evidence of teaching improvement. This process may be carried out formally as “classroom research” (See Cross & Angelo, 1988, and Cross & Steadman, 1996), or informally by using student ratings, self-evaluation data, peer ratings, or administrative feedback as a basis of an improvement program.

OPEN-ENDED QUESTIONNAIRES

Open-ended questions provide a useful source of information for the improvement of teaching because they allow students the freedom to respond in their own language rather than within the constraints imposed by the structure and vocabulary of the question. Moreover, when responding to open-ended questions, students often explain the context or qualify their comments. When completing a rating form, students rarely add explanatory notes. For example, a student who comments on the “poor organization” of a lecture is poor might take the time to explain that he or she is aware of the disorganized state of the literature in the field. Or a student who complains that there is too much talking in class might add that students are talking about the subject matter. Data from open-ended questions can be valuable, therefore, in revealing context and subtleties of your teaching that are vital to your improvement efforts. One disadvantage of the written format, pointed out by Centra and others (1982) is that students may be reluctant to be frank for fear that their handwriting could be detected. Detection of handwriting is not a real problem today since most students have access to electronic word processing, but confidentiality may remain a problem with open-ended methods that are implemented in class. Another disadvantage of open-ended questions is that they can be so open that they elicit a broad range of responses that are confusing to interpret. They can even elicit irrelevant responses. A general question about “likes” and “dislikes” might elicit answers like “lunch break” and “exams.” To overcome the problem of irrelevant responses the question can be given minimal structure, as in the examples below. Student comments from open ended questions are from the Office of Program Evaluation and course and clerkship coordinators.

Request for Comments

Certainly the simplest request for written feedback and least likely to lead the student is the single word “Comments?” This question is commonly used at the end of a rating form to give students the opportunity to add a footnote to the ratings. Student response rates to this sort of question are typically low. And when they do respond, their comments are not always helpful. Student feedback appears to need some stimulation. This form of open ended question is not usually part of the UMSM student rating forms, but you may want to use it to collect feedback for your own improvement. The following request is a little more specific: “If you have any comments about this course, about the teacher, or about your experiences in this medical school, please note them here.”

Very specific questions have the disadvantage of constraining the students’ answers to what the author of the questions thinks are the important issues. If you want students’ original ideas, expressed in their own language, then you have to be careful not to specify the questions too strictly.

Strengths & Weaknesses / Improvements

The most common compromise between giving the students freedom to respond and providing them with stimulation to think about an answer is to ask them to identify something that is worth preserving, something that needs to be changed, and to give some thought as to how it might be changed.

Last item of a rating questionnaire

Typically a form of this question is included as the last item of student-rating questionnaires. Following is an example of this type of questionnaire.

Sample:

In the space provided below please briefly describe some aspect of the course or teaching that you would like to see preserved, some aspect that you think could be improved, and some way that the improvement might be brought about (you may list more than one thing): What should be preserved?. What could be improved? My suggestion toward improvement.

Most Significant learning

The trouble with the three questions described above—what should remain, what should be changed, and suggestions for change—is that the relationship of these suggestions to learning may not be clear. You may get responses all over the ballpark, about things that you cannot change and things that are irrelevant to the students’ learning. An alternative is to ask learners to “List your most important learning in the course so far.” (Rosemary Peikes and David Hunt, 1990). Learner responses to this question can be gathered and discussed at a subsequent meeting with the students.

One of the authors of this method (Rosemary Peikes) found that this type of question produces a positive attitude in the group. Learners express a lot of emotion in describing their positive experiences. This question can be followed with, “What are the most important questions that you have left?” Again, the focus is on the learning, although this second question is able to tap shortcomings of the course. Interestingly, the nature of the question treats the shortcomings as unfinished business and therefore is easier for students to raise and for teachers to accept. It is very constructive. Following is an example of this type of instrument. Sample: Please list some of your most important learning in the course so far. What are the most important questions that you have left?

Most & least

Another type of open-ended questionnaire asks for “most” and “least.” Some examples follow, adapted from MaryEllen Weimer’s book How am I Teaching, 1998. As you can see, you can make up any pairs of questions that suit your teaching objectives.

Samples: When do you find the course/instructor most helpful to your learning? When do you find the course/instructor least helpful to your learning? What was the most intellectually stimulating aspect of this course? What was the least intellectually stimulating aspect of this course? What was the most clinically relevant aspect of this course? What was the least clinically relevant aspect of this course? What aspect of the course is most worthwhile? What aspect of the course is least worthwhile? When do you find yourself most actively engaged in the course material? When do you find yourself least actively engaged in the course material?

Specific Questions

Angelo and Cross (1993) describe a technique, which they call One-Minute Papers (referring to the time allowed the student to respond) or Half-Sheet Response (because it is usually written on a half sheet of paper). The teacher saves a few minutes at the beginning or end of the class to ask a question or two which the students answer on these half sheets of paper or index cards and then hand in. This method is particularly useful to identify problems or document an improvement in teaching when the issue is subtle or sensitive. For example, a one-minute paper would be useful to find out if students were offended by sexist language used in the group and were unwilling to raise this issue in class. Of course, if the issue were not sensitive or subtle you would not need to resort to written responses. To find out if students are having difficulty obtaining the readings, for example, you might just ask them. The following brief questionnaire is an example of the use of specific questions.

Sample: Do you think that sexist language is used in this class? If so, does it present a problem, and what do you think we should do about it?

Electronic Mail

Specific questions such as those that stimulate the Half-Sheet Response can be asked very effectively through the use of electronic mail. (Angelo and Cross, 1993). An electronic mail feedback system preserves the anonymity of students, which is especially important in the small classes. If confidentiality is a problem, electronic responses may be a way to prevent identification of handwriting, but there is a more important advantage. Electronic mail relieves both teacher and students from the constraints of class time. Indeed, the freedom to correspond at one’s leisure is an argument for electronic mail in general. Teachers can take the time to write an open letter and students can respond at length as well.

Angelo and Cross (1993) claim that this technique encourages responses that are personal and candid, since the letter format is one with which students are familiar in their personal lives. Below is an example of an open letter from a teacher to his students via electronic mail. Sample: Dear Student: I am enjoying our course and I hope you are too. One of the things that make the course interesting is the sharing of ideas among the members of the diverse group that we are fortunate to have this year. From time to time, however, I am aware that some students have difficulty getting into the conversation. My first reaction is to help them by leading the group in a more directive manner. But I am aware when I do this that I might distort the natural flow of conversation. I don’t want to burden our conversation with heavy leadership, say by reverting to recognizing speakers by raised hands, in order to ensure that few people get into the conversation. What do you think of this issue? Do you have any suggestions on how I should solve this problem? I would be grateful for your thoughts on this and any other issue that you may be concerned with. My access code is James.doe@miami.med.edu Good luck on your final exams.

CLASSROOM OBSERVATION

Direct observation of teaching is a widely used method of assessment for the purposes of improvement. There are three commonly used types of observers, each with its own particular advantages: students, peers and educational consultants. Observation checklists are useful, such as those used by faculty conducting an OSCE exam, in order to standardize the observations.

Student Observation

Students lack the expertise to make judgments about their teachers’ knowledge of the subject matter. Likewise they rarely have sufficient awareness of alternative teaching methods to comment on their teachers’ need to use innovative teaching methods. However, students can be accurate observers of behavior, especially if a checklist or specific assignment guides their observation. General checklists are available from the Educational Development Office. These can be modified to suite your subject matter and format of teaching. A checklist may not be necessary if your objective is merely to observe a few specific behaviors. For example, a teacher whose seminar leading has been criticized because of uneven sharing of conversation by participants, might assign one or more students to observe the interaction both before and after applying a remedy to the situation. Such documentation of specific improvements offers evidence of your teaching as well as evidence of your effort to improve your teaching. There are some caveats in the use of students as observers. First, ethics require that the evaluation that they are asked to perform must enhance their educational experience rather than interfere with it. Second, if the information is to be useful in improving your teaching, it must be collected systematically, i.e. under some set of rules. Simply asking students “How did it go?” is not specific enough. Students must be assigned a specific observation task.

Peer Observation

Peers who are experts in your subject matter are best qualified to comment on the extent to which your teaching performance validly reflects the content. If you have a written text of your lecture or a syllabus to review, the peer evaluator need not actually visit your class. In this case the evaluation would be directed not at your teaching performance but at course development or design: See the section in this Guide entitled “Course Development and Leadership.” On the other hand, there might be situations in which teachers might seek evaluations of their teaching performance by the use of peer observers. For example, a teacher developing new emphases might seek assurance that the traditional emphases were not underplayed. Fortunately, today there is a formidable array of procedures, tools, and guidelines for peer evaluation (See Chism, 1999). Reliability is higher if observers are guided by specific instructions outlining the precise purpose and methods of the observation. As stated earlier, if the information were to be used to document teaching for promotion some process that puts the peer at arm’s length from the teacher would be advisable. UMSM does not have a peer-observation program functioning currently, but if there are a number of requests for this type of service, one could be developed. Meanwhile, the Education Development Office is interested in hearing of requests for peer consultant arrangements since it would help organize such a service.

Observation by an Educational Consultant

Specialists in education are best qualified to serve as observers of teaching techniques and strategies. The special difficulty faced by our teachers is that there is such a broad array of types of teaching in medical education that someone with educational training may not have expertise in the specific type of teaching that he or she is asked to observe. As a medical school UMSM needs to create a group of clinician/educators who are willing and able to observe colleagues’ teaching. There are models for such a system in other universities. A small committee could be developed, staffed by rotating clinician/educators drawn from several departments, who would volunteer when a any faculty member requests an observation. The committee would have to be organized so that all types of teaching could be reviewed. Meanwhile, until UMSM has such a system in place, you are invited to phone the Education Development Office, who would like to hear of requests for educational consultant arrangements.

INTERVIEWS & FOCUS GROUPS

Face-to-face, interactive techniques for gathering information about teaching are capable of producing information of great depth and subtlety. The interviewer can probe behind the learners’ opinions to uncover the reasons behind the opinions, to probe intensity of feelings, and to reconcile differences of opinion that could not be understood from questionnaire results alone. The main disadvantage is that they are time-consuming. While this kind of information is extremely valuable to a teaching improvement effort, it can also be a valuable addition to the documentation of teaching for promotion. Information from personal interaction is usually rich and compelling. It conveys a picture of the teacher in human terms. Some excellent teachers may have puzzling or contradictory questionnaire results until the explanations for student responses are understood in greater depth. Interactive measures of teaching are particularly successful for teachers who are in the process of implementing changes or trying new or risky strategies.

Structured Interviews

Structured interviews of students or former students provide the assessor with the most control. The interviewer will be able to get answers to pre-determined questions and the resulting data will be easier to analyze. Use this method if you are fairly certain about what you want to find out. It is not recommended for exploration. For example, you may have introduced some specific changes to your teaching and want to know if they are having the desired effect. How can you obtain interviewers? UMSM has not yet developed a pool of volunteers for interviewing students. At present you may have to rely on a volunteer colleague.

Unstructured Interviews

Structured interviews bias the situation in favor of the agenda of the interviewer. Since interviewees usually answer the questions that they are asked, the interviewer rarely finds out what they would rather have talked about. An unstructured interview sets out some general themes and allows students to follow their interests. For example, an interview might ask if there is anything that should stay the same or be changed, and if so, how might you suggest changing it. The interviewer is usually given a lot of freedom to probe interesting comments and to ask for clarification or examples. Whether students or former students are interviewed, the sort of information obtained from unstructured interviews can be extremely valuable to teaching improvement. Often the information uncovered is a complete surprise to the teachers. They find out things that they never thought to ask. This process can support your claim that your improvements are having the desired effect.

Focus Groups & Discussion Techniques

Focus groups are a type of group interview in which the evaluator encourages group discussion while he or she is free to observe and record the discussion. The focus group is even better suited to exploring students’ perceptions and feelings and to generating ideas than unstructured one-on-one interviewing because the discussion not only generates a range of opinions but also generates other participants’ reactions to each opinion. Focus group interviews share some of the drawbacks of unstructured interviews in that they produce data that is not easily summarized. Again, as in the case of unstructured interviews, the data can be used to improve teaching and to document teaching improvement efforts.

CHECKLIST FOR EVALUATING COURSE DEVELOPMENT & LEADERSHIP

Teachers or educational consultants can attest to the quality of your course coordinating, using the following dimensions:

·Your commitment or enthusiasm. ·Leadership ability. ·Your accessibility to faculty. ·Ability to carefully monitor the program to ensure that sufficient time is allotted for students to study the material, that lecture time is not excessive, that staff are selected appropriately for their teaching and interest in students. ·Your role in involving staff in decision-making and participation, and in generating commitment, enthusiasm, and energy of staff. ·Your role in encouraging collaboration among colleagues on teaching

Teachers or educational consultants can attest to the quality of your course design using the following dimensions:

·Your course objectives. ·Your course organization and integration, and the logical progression of the subject matter. ·Your course scheduling, notes, syllabi, outlines. ·Course arrangements for the size of the group being taught. ·The timing of the course within the rest of the curriculum. ·Planning for appropriate study time before the course. ·Arrangement of visits, e.g. to hospitals, community services. ·Ensuring relevance and validity. ·Matching subject matter to students’ level. ·The ability of the course to stimulate critical and analytical thinking. ·The ability of the course to stimulate students to intellectual effort, and to challenge students’ conceptual abilities without being too difficult for student to master. ·The degree to which the material matches the context of other material in the students’ program or the level of academic maturity of the students. ·Providing a valid, fair and reliable evaluation system for the course, one that is matched to subject matter and course objectives, that encourages core learning of material and basic principles, that emphasizes critical and analytical thinking, that stresses understanding of the subject rather than memorization, and that tests only on what is taught.

SELF ASSESSMENT The kind of careful reflection and self-awareness required to conduct a self-assessment is one of the hallmarks of a good teacher. Such reflection is particularly indicative of teaching excellence when it is combined with a teaching improvement program. Self-ratings can be an important component of the teaching improvement process. And the latter is singularly impressive. Self-ratings raise the awareness of teachers to important dimensions of their teaching. And self-assessment combined with student assessment provides the teachers with important comparisons or discrepancies that are an important starting point in a teaching improvement sequence. New developments in self-assessment methodology allow self-assessment to provide more specific and less subjective data. There are several techniques:

Talks with Colleagues

Evaluations about one’s own performance are more valuable when informed by discussion and comparison with trusted colleagues. A good example of this kind of guided self-evaluation is regular, informal discussions of teaching problems with one’s colleagues within an interest group.

Use of Forms

Although self-assessment is, by definition, subjective it need not be vague or unfocused. By the use of an inventory or check list (of teaching strengths and weaknesses, teaching aims, or teaching assumptions) or by the use of a self-rating form, inventory or checklist, teachers can