1999-2000
Freshman Writing Program Assessment Coordinator
The Writing Program Office
English Department
Boise State University
October, 2000
Introduction
I titled last year’s Assessment Report –the first – “The Elephant’s Tail” and compared the program to the story of the blind men trying to define an elephant as each one touched only a part of the entire animal. I thought the tale was appropriate as it cautioned us not to define and judge our writing program until we have all of the necessary information. So to follow the same theme, I have titled this year’s report “The Next Appendage” to remind us again of the need to withhold a general judgement. That may be difficult to do. This year’s data both flaunts and supports what we may have been tempted to identify as patterns based on last year’s results. A tail and a tusk hardly begin to form an accurate picture of an elephant. However, we have plenty to examine based on the experience of the assessment readers and the results of the reading.
In order to ensure a high level of compliance, the program is designed be as simple as possible and to put as little burden as possible on the teachers and the students. Many of the teachers use a portfolio system in their classes where the majority of each student’s grade is determined by a 20-25 page portfolio of revised work that they submit at the end of the semester. But a significant number of teachers do not use this system, so we had to design an assessment program that can accommodate both.
The objective of our Student Outcomes Assessment Program is to assess whether our E101 and E102 students are leaving our courses with the competencies we aim to teach them. The assessment program consists of an annual sampling of one student portfolio or folder of collected work from each E101 and E102 class offered in both the fall and spring semesters, for a total of approximately 230 portfolios/folders. A committee of 8-10 readers will meet in late May to read the portfolio/folders and assess them in terms of how well they meet two criteria chosen from the competencies listed on the E101 and E102 course descriptions. (See Attachment A.) Under the direction of the Assistant Writing Director, the Assessment Coordinator, the readers, and the Writing Committee then evaluate the findings and report them to the Department in September.
Each instructor has all of their students keep a folder of the evaluated work they produce for the class. This can be either a portfolio or a collection of submitted and evaluated assignments if the instructor does not use portfolios as part of her individual grading system. At the end of the semester, each instructor collects all of these portfolios/folders.
The Writing Program office determines which student portfolios/folders must be submitted by drawing a number between 1 and 25. All teachers will then submit a copy of the portfolio from the student with that number on their roster. For example, if we draw #7, all instructors will submit the portfolio of student #7 on their roster. Alternative numbers are drawn for cases where the selected student is no longer attending class. All portfolios/folders are then returned to the students.
Neither student nor instructor names are included anywhere on the portfolios. Portfolios are assessed only as representative samples of the freshman writing program and not as individual instructors’ work.
The assessment readers consists of either (or both) the Writing Director or the Assistant Writing Director, the Assessment Coordinator, and 6-7 other freshman writing teachers who volunteer for the project. Each reader is paid $250 to read 50-60 portfolios. The Assistant Writing Director is responsible for training the readers, who read and assess for three days in May after final grades have been turned in.
Over the summer, the Writing Program Office compiles the data. The Assessment Coordinator then presents the initial results to the writing faculty at the annual August Meeting. The faculty spend much of that meeting discussing the results that cause the greatest concern. The Writing Program Office considers the faculty input as it designs a curricular or programmatic response to the assessment results. The Assessment Coordinator incorporates it all into the annual Assessment Report, which she compiles in the fall.
On May 17-19, nine readers met to read and assess the 226 folders collected in the fall and spring semesters. In the interest of building on our previous experience, the program wanted to retain as many of the original readers as possible. This made the training process simpler and shorter and maintained significant continuity from the program’s first year. The readers were: Bud Pedersen, Debbie Dwyer, Joy Kidwell, Marion Thomas, Tudor Cushman, Julie Ewing, Siskanna Naynaha, Michelle Payne, and myself (Stephanie Cox). Everyone but Tudor and Julie had served as a reader last year.
May 17 was devoted to training the readers to assess the folders. The Writing Program Office had pre-selected portfolios that represented a range of competencies for E101 and E102. Assistant Writing Director Michelle Payne led the session. We worked on E101 first, then E102.
We read the portfolios and individually attempted to assess each one as
either “Competent,” “Minimally Competent,” and “Incompetent.” (We changed the language from last year,
when the designations were “Clearly Competent,” “Barely Competent,” and
“Incompetent.” Then we had a norming
session where the assessments were tallied and we discussed why we each had
made the assessment we did. After that
discussion, we reassessed to see how well our opinions had consolidated. From that discussion, we created a “cheat
sheet” for each class that clarified what we expected to see in a “Competent,”
a “Minimally Competent,” and an “Incompetent” folder. (See
Attachment B.)
We spent all day May 18 reading and assessing 109 E101 folders for the following competencies:
· The portfolio demonstrates that the student can produce writing that has a purpose, as well as a clear focus and point. (This competency was also assessed last year.)
· The portfolio demonstrates that the student can make assertions that are supported by evidence gathered from their experience, their reading, their observations, or their conversations with others, and the nature of these assertions and evidence is appropriate to the particular genre in which they are writing (e.g. narrative, argument, etc.).
May 19 was devoted to reading and assessing 117 E102 folders on these competencies:
· The portfolio demonstrates that the student can do more than simply report information they gather from outside sources, but use that information purposefully. (This competency was also assessed last year.)
·
The portfolio demonstrates that [the student can take
charge of their own investigations,]*
formulating the question(s) that will drive the research, finding the
information they need to explore those questions, and coming to their own
conclusions about what they discover.
* After lengthy discussion
combined with the experience of reading sample portfolios, the readers decided
that we could not assess whether or how well a student had taken charge of
their own investigation. So we simply
did not attempt to assess that part of the competency. That was not the end of our troubles with
this competency.
We also wondered if it were
not the same competency as the other one, just worded differently. We tried to make differentiation on the
Cheat Sheets, but the criteria are very similar. The notable difference is that for this competency, we looked for
a sufficient number and variety of sources.
We also sought to answer that question by keeping track of how many
folders scored the same in both competencies.
A large majority did: 92 of 117 folders scored the same.
Each folder had to be assessed by at least two readers. If the first two readers could not agree on their assessment, the folder was sent to a third reader. The reading process went as follows:
1. All folders were placed in a “First Reader” pile. Readers took folders from that pile and read them, using the “Cheat Sheet” to help them with their assessment and marking their judgment on the “Assessment Sheet.”
2. After making judgment, the reader placed the “Assessment Sheet” face down in the back of the folder and put the folder in the “Second Reader” pile.
3. When readers moved on to second readings, they read and assessed the folder in the same manner. They only looked at the previous reader’s assessment after making their own on a separate “Assessment Sheet” or if they were stuck and needed some help in making their own assessment.
4. If the two readers agreed, the folder was placed in a “Finished” pile.
5. If the readers disagreed, the folder was set aside. After all folders had been read, second readers met with first readers try to reach agreement on a disputed assessment. If they could not, the folder was placed in the “Third Reader” pile. It’s worth noting that very few folders were sent to a third reader – a mere handful in E102 and only one in E101.
After finishing the 102 folders on Friday afternoon, the readers discussed the experience and made these four observations:
· It made no difference to our assessment if we were reading a Folder or a Portfolio. That designation wasn’t helpful.
· Once again, we didn’t feel we needed assignment sheets or anything else.
· Teachers’ notes are distracting. We’ll have to do a better job of blocking them out next time.
· The 102 competencies clearly need to be rewritten & perhaps even separated.
The Data
Total portfolios evaluated: 109. Fall: 70; Spring: 39.
(26 Folders. 83 Portfolios.)
Competency: The portfolio demonstrates that the student can produce writing that has a purpose, as well as a clear focus and point.
Fall 99: 44% (31) 43% (30) 13% (9)
Spring 00: 72% (28) 23% (9) 5% (2)
Total: 54% (59) 36% (39) 10% (11)
Total portfolios evaluated: 106. Fall: 69; Spring: 37.
Competent: Minimally Competent: Incompetent:
Fall 98: 54% (37) 36% (25) 10% (7)
Spring 99: 41% (15) 35% (13) 24% (9)
Total: 49% (52) 36%
(38) 15% (16)
Competency: The portfolio demonstrates that the student can make assertions that are supported by evidence gathered from their experience, their reading, their observations, or their conversations with others, and the nature of these assertions and evidence is appropriate to the particular genre in which they are writing (e.g. narrative, argument, etc.).
Fall 99: 34/ 49% 27/ 39% 9/ 13%
Spring 00: 25/ 64% 11/ 28% 3/ 8%
Total: 59/ 54% 38/ 35% 12/ 11%
E102
Total portfolios submitted: 117. Fall: 48; Spring: 69.
(36 Folders. 81 Portfolios.)
Competency: The portfolio demonstrates that the student can do more than simply report information they gather from outside sources, but use that information purposefully.
Fall 99: 23/ 48% 20/ 42% 5/ 10%
Spring 00: 33/ 48% 23/ 33% 13/ 19%
Total: 56/ 48% 43/ 37% 18/ 15%
Total portfolios evaluated: 100. Fall: 43; Spring: 57.
Competent: Minimally Competent: Incompetent:
Fall 98: 37% (16) 47% (20) 16% (7)
Spring 99: 58% (33) 32% (18) 11% (6)
Total: 49% (49) 38%
(38) 13% (13)
Competency: The portfolio demonstrates that [the student can take charge of
their own investigations,]*
formulating the question(s) that will drive the research, finding the
information they need to explore those questions, and coming to their own
conclusions about what they discover.
* Assessors concluded we could not assess this particular part.
Fall 99: 20/ 42% 19/ 40% 9/ 19%
Spring 00: 23/ 33% 30/ 43% 16/ 23%
Total: 43/ 37% 49/ 42% 25/ 21%
Once again, we are encouraged by the high rate (over 95% for both courses) of faculty compliance. This indicates that the program is simple, not burdensome, and that it has become institutionalized. Over the years, this continued level of compliance will help ensure the validity of our data.
E101 figures suggest another successful year. For both of these competencies, 89-90% of our students are competent to at least an acceptable degree. 54% are solidly competent. The picture looks even better when we compare it with last year’s results. We’ve made some notable improvement in Focus, Purpose, and Point. Competence has increased by 5%, and Incompetence has decreased by 5%. Whether this represents a pattern has yet to be seen, but we can take encouragement and even pride from these figures.
E102 figures are similar to E101. In both competencies, the large majority of students – over 78% -- are at least minimally competent in both categories. And when
the results of the Purposeful Use of Research competency are compared with that of last year, we see nearly identical percentages. The incompetence rate of 21% for the second competency is somewhat troubling, but given the difficulties of assessing that competency, I’m not sure how much credence we should give it.
An interesting observation can be made in comparing the Fall/Spring results with those of last year. Last year, the highest incompetence percentages fell in the semesters considered the “off” semesters for each course – Spring for E101 and Fall for E102. That was consistent with the results of the earlier Descriptive Report, which found that more students fail these courses in the “off” semester. But this year the highest level of incompetence was in the “on” semesters for both courses and in both competencies. The gap was especially wide in the competencies that were repeated. 13% of Fall 1999 E101 folders were incompetent in Focus, Purpose, & Point, compared to 5% for Spring. 19% of Spring 2000 E102 folders were incompetent in Purposeful Use, compared to 10% for Fall. I have no idea what this means. It will take several more years of assessment to discover if there is any pattern for the “on” and “off” semester students.
What has emerged as the biggest issue for the Writing Program is the workability of the competencies. We suspected this earlier, but the reading in May and the difficulty of assessing the E102 competency have confirmed it. We need to review the competencies to decide if they accurately define what the program is or should be trying to do, if they are workable and practical, if they are assessable, and if some of them even make sense.
After the assessment results were presented during the annual August Meeting, faculty worked in small groups to revise the second E102 competency. While no definitive consensus was reached, the teachers raised important questions about the nature of the competencies and of the assessment procedure.
The major question raised was whether all of the competencies must be assessable by a third party. The major suggestion for procedure was for the assessment program to implement other ways to assess student outcomes that could be implemented in addition to the portfolio reading. These two issues will be addressed in the “Changes to the Assessment Program” section of this report.
Last year, the Writing Program focused its efforts on helping teachers develop better ways of teaching both mechanical skills and the intricacies of MLA or APA documentation. We will continue with those efforts. We will also concentrate on reviewing and revising the competencies and minimum requirements.
· Faculty Development Meetings will continue to focus on better ways to teach mechanics and documentation, as well as issues related to other competencies.
· The Writing Program will host an In-Service day for all freshman writing faculty. Part of the work will focus on re-evaluation and revision of the E102 competencies and minimum requirements. Part of it will deal with teaching sentence-level mechanics.
· COMPTALK will host a moderated discussion on these topics.
· The Writing Program Office will survey the freshman writing faculty on their experience with and attitudes toward the competencies and minimum requirements.
· The annual Assessment Report will periodically include descriptive data about the E101 & E102 student body and/or about the faculty body.
· We will no longer differentiate between folders & portfolios
· Teachers will designate portfolios written by ESL students
· The Assessment Coordinator will develop alternate methods of assessing student outcomes (e.g. surveys, etc.). The Writing Program Office has already surveyed a sampling of E101 & E102 classes regarding students’ writing history, placement, primary language, and writing attitudes. This data will be included in the 2000-2001 report.
The data provides a narrow picture of what skills students are competent in when they leave the freshman composition classes. The official program response provides another narrow picture of how the Writing Program is working to address weaknesses and continue to build on our strengths. But there is another picture that is not represented in the data or the program response: the effect assessment has on the freshman writing program.
This attention to the program
requires us to regularly examine our teaching philosophies and practices. That’s no surprise. An unexpected benefit though, has been the
honest, thoughtful, and invaluable response of the freshmen writing faculty as
we examine the nature of our program.
Again, that shouldn’t be a surprise given the professional dedication of
our mostly “part-time” faculty. But
years ago, when work to build an assessment program began under the direction
of then-Writing Director Allene Cooper, faculty resistance was high. Now, we experience a comparatively
overwhelming cooperation that ensures that as our picture of the program
expands the writing program will continue to grow and to respond to the needs
of the students.
Attachment A
E 101
Competencies
Students will demonstrate that:
· they can produce writing that has a purpose, as well as a clear focus and point.
· they can use revision to extend their thinking about a topic, not just to rearrange material or "fix" mechanical errors.
· they can make assertions which are supported by evidence gathered from their experience, their reading, their observations, or their conversations with others, and the nature of these assertions and evidence is appropriate to the particular genre in which they are writing (e.g. narrative, argument, etc.)
· they can articulate the rhetorical choices they have made, illustrating their awareness of a writer’s relationship to the subject, a text’s purpose, and audience.
· they can use a variety of strategies for generating ideas for writing, for planning and organizing material, for identifying purpose and audience, and for providing useful feedback to peers during the writing process.
· by the end of the course, they can produce prose without mechanical errors that distract readers from attending to the meaning and purpose of the writing.
E
102 Competencies.
E102 students will demonstrate that:
Attachment B
E101
Assessment Reading
May 2000
Reader’s Initials:_____
# of pages:_________
Competency: The portfolio demonstrates that the student can produce writing that has a purpose, as well as a clear focus and point.
Comments:
· Competency: The portfolio demonstrates that the student can make assertions that are supported by evidence gathered from their experience, their reading, their observations, or their conversations with others, and the nature of these assertions and evidence is appropriate to the particular genre in which they are writing (e.g. narrative, argument, etc.).
Comments:
E102
Assessment Reading
May 2000
# of pages:__________
Competency: The portfolio demonstrates that the student can do more than simply report information they gather from outside sources, but use that information purposefully.
Comments:
·
Competency: The portfolio demonstrates that the student can take charge of
their own investigations, formulating the question(s) that will drive the
research, finding the information they need to explore those questions, and coming
to their own conclusions about what they discover.
Comments:
Cheat Sheet 101 (2000)
Competent:
Purpose
becomes apparent within the opening paragraphs, or the closing may pull the
essay together and make the purpose apparent.
There is a main idea, or perhaps a question, driving the essay.
All
paragraphs generally support and develop a well-defined focus.
Answers
the “So What?” question.
Minimally
Competent:
Purpose
does not become apparent until late in the essay, and the closing does not pull
it together.
Focus
may be very broad; paragraphs may not always stay within the focus.
Sort
of answers the “So What?” question, may have more than one possible answer.
Incompetent:
Purpose
never becomes apparent
Focus
may never be apparent, or may be so broad that supporting paragraphs can not
adequately develop it
Does
not answer the “So What?” question
Define “evidence”
What counts for evidence
in: Reading response/critique, Researched essay/paper, Personal narrative,
Profile, Argument, Subcategories: comp/contrast, observation papers,
“Synthesis” paper, defining paper
Types of evidence:
Quotation, anecdote, example, textual passage, “telling” detail, description, expert testimony, tables & figures, discrete observation
Competent
Uses
details to make the evidence more vivid or comprehensible
Evidence
is credible & relevant to the assertions
Evidence
does not vary in different essays.
Generally one kind (i.e. all personal experience) As Marion says, “A one-trick pony.”
Reflects
ability/knowledge of the purpose of
the evidence, but the portfolio only has one kind of evidence or the evidence
is not well developed
Evidence
is marginally relevant to the assertions
Never
goes from abstractions to the concrete
No
examples, anecdotes,
Does
not make assertions, even if there is evidence (Tudor’s dump & drop)
Little
ability to make assertions, conceptualize ideas
Evidence
may not be relevant or credible or may be insufficient
Cheat Sheet 102 (2000)
Competent:
Purpose
of essay becomes evident.
Writer
guides reader through info
Introduces
& comments on info.
Strong
voice demonstrates control.
Minimally
Competent:
Less
guided, info seems to wander.
Source
driven rather than writer driven.
The
work is structured around the sources.
Incompetent:
Lists
info from sources.
No
purpose to info.
No
writer presence.
[Takes charge of
investigations], formulates questions to drive research, finds info, comes to
their own conclusions. (We’ve decided
we cannot assess if they’ve taken
charge of their own investigations.)
Writer
signals motive.
Strong
voice demonstrates control.
Conclusions
& questions are clear & relevant to the information presented. They contribute to an insightful or
unexpected understanding of the subject.
Sources
offer sufficient, varied, & authoritative information.
Writer’s
motive may not be clear.
Amount
&/or quality of sources limit the investigation &/or are not sufficient
for a thorough understanding of the subject.
Conclusions
state the utterly obvious. Do not add
to readers’ understanding.
Lack
of questions. Or exploration didn’t fully
address them.
No
(or few) conclusions offered.
Amount
& quality of sources are inadequate.
Writer does not appear to have a motive.