2000 - 2001
Compiled by Stephanie Cox
Freshman Writing Program Assessment Coordinator
The Writing Program Office
English Department
Boise State University
November, 2001
Introduction ……………………………………….………. 3
The
Freshman Writing Assessment Program……………….3
The
Reading…………………………………………………4
E101
Data……………………………………………………6
E102
Data……………………………………………………7
E090
Data……………………………………………………8
Reflection……………………………………………………9
Curricular and Programmatic Response……………………..10
E090
Competencies………………………………………….11
E101
Competencies………………………………………….12
E102
Competencies………………………………………….13
“Cheat
Sheets”……………………………………………….14
Introduction
I’ve continued the theme of the
blind men describing the elephant by titling this third annual assessment
report “A Tusk Revealed.” During the
first two years, we had to be conscious of the fact that we were only looking
at information about a small part of the writing program. Now that we have completed the third year of
program assessment, we have evaluated enough of the competencies to form a
picture of at least a piece of the beast.
It is a good picture, indicating that the large majority of our students
leave our classes with the competency they should. This year’s data also indicates an encouraging improvement from
the first year
In order to ensure a high
level of compliance, the program is designed be as simple as possible and to
put as little burden as possible on the teachers and the students. Many of the teachers use a portfolio system in
their classes where the majority of each student’s grade is determined by a
20-25 page portfolio of revised work that they submit at the end of the
semester. But a significant number of
teachers do not use this system, so we had to design an assessment program that
can accommodate both.
The objective of our Student
Outcomes Assessment Program is to assess whether our E101 and E102, and now
E090, students are leaving our courses with the competencies we aim to teach
them. The assessment program consists
of an annual sampling of one student portfolio or folder of collected work from
each E090, E101, and E102 class offered in both the fall and spring semesters,
for a total of approximately 230 portfolios.
A committee of 10 readers will meet in late May to read the portfolio and
assess them in terms of how well they meet two criteria chosen from the
competencies listed on the course descriptions. (See attached.) Under the direction of the Assistant Writing
Director, the Assessment Coordinator, the readers, and the Writing Committee
then evaluate the findings and report them to the Department in September.
Each instructor has all of their
students keep a folder of the evaluated work they produce for the class. This can be either a portfolio or a
collection of submitted and evaluated assignments if the instructor does not
use portfolios as part of her individual grading system. At the end of the semester, each instructor
collects all of these portfolios/folders.
The Writing Program office
determines which student portfolios/folders must be submitted by drawing a
number between 1 and 25. All teachers
will then submit a copy of the portfolio from the student with that number on
their roster. For example, if we draw
#7, all instructors will submit the portfolio of student #7 on their
roster. Alternative numbers are drawn
for cases where the selected student is no longer attending class. All portfolios/folders are then returned to
the students.
Neither student nor instructor names
are included anywhere on the portfolios.
Portfolios are assessed only as representative samples of the freshman
writing program and not as individual instructors’ work.
The assessment readers consists of
either (or both) the Writing Director or the Assistant Writing Director, the
Assessment Coordinator, and 8 other freshman writing teachers who volunteer for
the project. Each reader is paid $250
to read 50-60 portfolios. The Assistant
Writing Director is responsible for training the readers, who read and assess
for three days in May after final grades have been turned in.
Over the summer, the Writing Program
Office compiles the data. The
Assessment Coordinator then presents the initial results to the writing faculty
at the annual August Meeting. The
faculty spend much of that meeting discussing the results that cause the
greatest concern. The Writing Program
Office considers the faculty input as it designs a curricular or programmatic
response to the assessment results. The
Assessment Coordinator incorporates it all into the annual Assessment Report,
which she compiles in the fall.
On May 16-18, ten readers met to
read and assess the 224 folders collected in the fall and spring
semesters. The readers were: Bud
Pedersen, Jill Heney, Marlys Hersey, Sue Hudson, Joy Kidwell, Marion Thomas,
Julie Ewing, Siskanna Naynaha, Michelle Payne, Devan Cook, and myself
(Stephanie Cox).
We read and assessed 95 E101 folders for the following competencies:
·
The
portfolio demonstrates that the student can produce prose without mechanical
errors that distract readers from attending to the meaning and purpose of the
writing. (This competency was assessed in 1999.)
·
The portfolio demonstrates that the student
can make assertions that are supported by evidence gathered from their
experience, their reading, their observations, or their conversations with
others, and the nature of these assertions and evidence is appropriate to the
particular genre in which they are writing (e.g. narrative, argument, etc.). (This competency was assessed in 2000.)
We read and assessed 118 E102 folders on these
competencies:
·
The
portfolio demonstrates that the student has an understanding of the purposes
and uses of documentation and competency in using MLA or APA citations. (This competency was assessed in 1999.)
·
The
portfolio demonstrates that the student is competent at using library reference
sources to find information.
(We decided
that we could only assess this one on a competent/incompetent basis. We did not know how to determine minimal
competence.)
This year, in order to help the Writing Program’s
effort to adopt competencies for
E090,
we decided to assess portfolios from that course as well. Participation was voluntary, and the number
of submitted portfolios was 11, a little less than half the number of classes
offered. Julie Ewing and Jill Heney,
both E090 instructors, also assessed these portfolios on the following draft
competencies:
·
The
portfolio demonstrates that the student can edit his/her work for mechanical
errors to the extent that, while perhaps not “perfect,” surface features of the
language do not interfere with communication.
·
The
portfolio demonstrates that the student can produce writing that has a
beginning, middle, and end developed with relevant details and examples.
As
in the past two years, the first morning was devoted to training the readers to
assess the E101 and E102 portfolios. A separate training session for the E090
readers was held in the afternoon.
Assistant Writing Director Michelle Payne again led these session. We
read three randomly selected portfolios for each course and individually
attempted to assess each one as either “Competent,” “Minimally Competent,” and
“Incompetent.” Then we had a norming
session where the assessments were tallied and we discussed why we each had made
the assessment we did. After that
discussion, we reassessed to see how well our opinions had consolidated. Since seven of the ten readers were repeat
participants, and because three of the four competencies had been assessed in
previous years, our norming session centered on revising the “cheat sheet” for
each class, which clarified what we expected to see in a “Competent,” a
“Minimally Competent,” and an “Incompetent” folder. (See attached.)
The reading process was the same as we have done
every year. Each folder had to be
assessed by at least two readers. If
the first two readers could not agree on their assessment, the folder was sent
to a third reader. The reading process
went as follows:
1.
All
folders were placed in a “First Reader” pile.
Readers took folders from that pile and read them, using the “Cheat
Sheet” to help them with their assessment and marking their judgment on the
“Assessment Sheet.”
2.
After
making judgment, the reader placed the “Assessment Sheet” face down in the back
of the folder and put the folder in the “Second Reader” pile.
3.
When
readers moved on to second readings, they read and assessed the folder in the
same manner. They only looked at the
previous reader’s assessment after making their own on a separate “Assessment
Sheet” or if they were stuck and needed some help in making their own
assessment.
4.
If
the two readers agreed, the folder was placed in a “Finished” pile.
5.
If
the readers disagreed, the folder was set aside. After all folders had been read, second readers met with first
readers try to reach agreement on a disputed assessment. If they could not, the folder was placed in
the “Third Reader” pile.
The Data
Total
portfolios evaluated: 95. Fall: 61;
Spring: 34.
·
Competency: The portfolio demonstrates that the student can produce prose without
mechanical errors that distract readers from attending to the meaning and
purpose of the writing.
Fall
00: 46% (28) 34% (21) 20%(12)
Spring
01: 56%
(19) 38%
(13) 6% (2)
Total: 49%
(47) 36%
(34) 15% (14)
Total
portfolios evaluated: 106. Fall: 69;
Spring: 37.
Competent: Minimally
Competent: Incompetent:
Fall
98: 51% (35) 32% (22) 17% (12)
Spring
99: 32% (12) 41%
(15) 27% (10)
Total: 44% (47) 35% (37) 21%
(22)
Competency: The portfolio demonstrates that the student can make assertions
that are supported by evidence gathered from their experience, their reading,
their observations, or their conversations with others, and the nature of these
assertions and evidence is appropriate to the particular genre in which they
are writing (e.g. narrative, argument, etc.).
Fall
00: 52.5%(32) 36%(22) 11.5%(7)
Spring
01: 59%(20) 41%(14) 0
Total: 55%(52) 38%(36) 7%(7)
Total
portfolios evaluated: 109. Fall: 70 ;
Spring: 39 .
Competent: Minimally
Competent: Incompetent:
Fall
98: 49% (34) 39% (27) 13% (9)
Spring
99: 64% (25) 28%
(11) 8% (3)
Total: 54% (59) 35% (38) 11%
(12)
E102
Total
portfolios submitted: 118. Fall: 57;
Spring: 61.
·
Competency: The
portfolio demonstrates that the student has an understanding of the purposes
and uses of documentation and competency in using MLA or APA citations.
Fall
00: 23%(13) 42%(24) 35%(20)
Spring
01: 18%(11) 52%(32) 30%(18)
Total: 20%(24) 47%(56) 32%(38)
Total
portfolios evaluated: 100. Fall: 43;
Spring: 57.
Competent: Minimally
Competent: Incompetent:
Fall
98: 26% (11) 37% (16) 37% (16)
Spring
99: 37% (21) 39%
(22) 24% (14)
Total: 32% (32) 38% (38) 30%
(30)
·
Competency: The portfolio demonstrates that the student is competent at using
library reference sources to find information.
Fall
00: 63%(36) 37%(21)
Spring
01: 67%(41) 33%(20)
Total: 65%(77) 35%(41)
E090
Total
portfolios submitted: 11. Fall: 9;
Spring: 2.
·
Competency: The
portfolio demonstrates that the student can edit his/her work for mechanical
errors to the extent that, while perhaps not “perfect,” surface features of the
language do not interfere with communication.
Fall
00: 66.6%(6) 33.3%(3) 0
Spring
01: 100%(2) 0 0
Total: 73%(8) 27%(3) 0%
·
Competency: The portfolio demonstrates that the student can produce writing that
has a beginning, middle, and end developed with relevant details and examples.
Fall
00: 89%(8) 11%(1) 0
Spring
01: 100%(2) 0 0
Total: 91%(10) 9%(1) 0%
Reflection
The E102 results are a bit of a puzzle. Students turned out to be less competent at
documenting sources than they were in 1999 even though the Writing Program has
made an effort to develop and share strategies for teaching citation methods
and individual teachers have become more aware of the responsibility to do
so. The data suggests that learning the
details of documentation styles is often difficult and confusing, especially
when students are also grappling with research methods, new critical thinking
strategies, and challenging subject matter.
It is also important to remember that rules for citation have been
complicated by the increasing use of electronic sources. Electronic sources play a role in the
library competency as well. Assessors
decided we could judge a portfolio competent if it contained citations of print
sources or electronic sources available only from the library’s electronic
databases. Incompetent portfolios
relied mostly on Internet sources of questionable credibility. The 35% incompetency rate is a distressing
reminder of how heavily students rely on the quickest Internet searches.
The E101 data, however, reflects some remarkable
improvement. Competency in mechanics
increased by 6% overall from 1999, and competency in supporting assertions
increased 4% from last year. We have
worked hard over the past two years to develop a variety of strategies for
teaching grammar and mechanics, and perhaps now we are seeing the fruit of that
effort. But the biggest cause of such
improvement seems to be the fact that almost twice as many incoming freshmen
are now required to take E090 before enrolling in E101.
In Fall 2000, new placement standards required
higher SAT, ACT, or Compass scores to enroll in E101. This resulting in a doubling of the number of E090 classes offered. In 1999-2000, twelve basic writing classes
were offered. In 2000-2001,
twenty-three were offered. The
correlation between the rise in E090 enrollment and E101 competency can be seen
most dramatically by looking at the competency percentages for the Spring 2001
semester. These Spring portfolios were
considerably more competent at both mechanics and supporting assertions than
their Fall 2000 counterparts. Spring
incompetence for mechanics was only 6% compared to 20% in the Fall. Spring portfolios for supporting assertions
were all at least minimally competent.
Not a single one was judged incompetent. 11.5% of Fall portfolios were.
The wide discrepancy in semesters makes sense if we account for
E090. Fifteen basic writing classes
were offered in Fall 2000, an increase of seven classes (approximately 140 more
students) from the previous Fall.
Assuming that most of those students went on to take E101 in the Spring
would mean that over a third of total E101 enrollment had taken E090. E101 teachers have long complained that
students come into their classes unprepared for college level work. Now that more students are placing into
basic writing classes, E090 seems to be doing an excellent job of preparing
them for E101.
Further evidence of success in E090 is evident in
the fact that the assessment revealed no incompetency in any of the submitted
portfolios. Only eleven portfolios
were submitted out of twenty-three classes, so the results may not accurately
reflect the course. Nevertheless, the
competency of these eleven in encouraging.
Last year, the Writing Program continued to focus its
efforts on helping teachers develop better ways of teaching both mechanical
skills and the intricacies of MLA documentation. We will also began reviewing and revising the competencies and
minimum requirements. In February, we
held an In-service day where Professors Rick Leahy, Jay King, and Michelle
Payne each gave presentations on particular strategies for teaching mechanics.
Curricular
and programmatic responses for this year include:
·
Dr.
Payne has extended in the TA seminar a section on teaching mechanics.
·
Dr.
Cook, Acting Assistant Writing Director, has initiated COMAPTALK discussions
and has organized Roundtables for E102 teachers to share ideas and strategies
for tying coursework more closely to the competencies.
·
The
Writing Program will again host an In-service day in February.
·
The
Writing Program will continue work begun at the August meeting to more effectively
bridge the gaps between E090 and E101, and E101 and E102.
·
Faculty
Development Meetings and Roundtables will continue to focus on re-evaluation
and revision of the E102 competencies and minimum requirements.
·
COMPTALK
will host a moderated discussion on these topics.
Since beginning the
Assessment Program, one thing has become clear: this will be a continuing
process of assessment, feedback, reflection, and response. We’ve been able to look at areas that need
improvement and develop strategies for dealing with them. It may take time for some of these
strategies to bear fruit, but the inquiry into pedagogy and curriculum is
valuable in itself. The Writing Program
is clearly successful in the primary aims of our courses.
E090 COMPETENCIES
Students
will demonstrate that:
§
they
can edit their work for mechanical errors to the extent that, while perhaps not
“perfect,” surface features of the language do not interfere with
communication.
E101 COMPETENCIES
Students
will demonstrate that:
·
They
can produce writing that has a purpose, as well as a clear focus and point.
·
They
can use revision to extend their thinking about a topic, not just to rearrange
material or fix mechanical errors.
·
They
can make assertions which are supported by evidence gathered from their
experience, their reading, their observations, or their conversations with
others, and the nature of these assertions and evidence is appropriate to the
particular genre in which they are writing (e.g. narrative, argument, etc.)
·
They
can articulate the rhetorical choices they have made, illustrating their
awareness of a writer’s relationship to the subject, a text’s purpose, and
audience.
·
They
can use a variety of strategies for generating ideas for writing, for planning
and organizing material, for identifying purpose and audience, and for
providing useful feedback to peers during the writing process.
·
By
the end of the course, they can produce prose without mechanical errors that
distract readers from attending to the meaning and purpose of the writing.
Students
will demonstrate that:
·
They
can do more than simply report information they gather from outside sources,
but use that information
purposefully.
·
They
can take charge of their own investigations, formulating the question(s) that
will drive the research, finding the information they need to explore those
questions, and coming to their own conclusions about what they discover.
·
They
can read critically, including an ability to discern what the author is asking
the reader to believe, and to evaluate the evidence and means of persuasion.
·
They
have an understanding of the purposes and uses of documentation and competency
in using MLA or APA citations.
·
They
are competent at using library reference sources to find information.
Cheat Sheet 101 (2001)
Mechanics:
Competent:
Generally
error-free – minor mistakes do not distract the reader
May
be an occasional typo or error in punctuation, sentence structure, or usage,
but this does not indicate a serious pattern
Minimally
Competent:
May
be several typos or errors in punctuation, sentence structure, or usage, but
they generally indicate sloppy editing and not patterns of misuse.
ESL
tendencies such as misuse of prepositions may be noticeable but not
overwhelming
Incompetent:
There
are many errors, indicating poor editing or patterns of misuse.
Common
ESL errors indicate a lack of mastery of the language
You
don’t have to look for errors; they find you
Define “evidence”
What counts for evidence
in: Reading response/critique, Researched essay/paper, Personal narrative,
Profile, Argument, Subcategories: comp/contrast, observation papers,
“Synthesis” paper, defining paper
Types of evidence:
Quotation, anecdote,
example, textual passage, “telling” detail, description, expert testimony,
tables & figures, discrete observation
Competent
Uses
details to make the evidence more vivid or comprehensible
Evidence
is credible & relevant to the assertions
Evidence
does not vary in different essays.
Generally one kind (i.e. all personal experience) As Marion says, “A one-trick pony.”
Reflects
ability/knowledge of the purpose of
the evidence, but the portfolio only has one kind of evidence or the evidence
is not well developed
Evidence is marginally relevant to the assertions
Never
goes from abstractions to the concrete
No
examples, anecdotes,
Does
not make assertions, even if there is evidence (Tudor’s dump & drop)
Little
ability to make assertions, conceptualize ideas
Evidence
may not be relevant or credible or may be insufficient
Cheat Sheet 102 (2001)
Competent:
Shows
control in introducing sources
Proper
placement & form
Minimally
Competent:
Improper
attribution of sources
Missing
or incorrect punctuation
Errors
in citing electronic sources
Does
not show control in introducing sources
Incompetent:
No
documentation
Mixed
format
Missing
information
Improper
form
Plagiarism
Writer
fails to distinguish between their own words & ideas and the sources’.
Competent:
Has
more credible library sources than online &/or interview sources.
Incompetent:
Sources
are only primary sources (interviews, surveys) &/or Internet sites of
questionable credibility.