Assessment Report: 1999

Freshman Writing Program Assessment

1998-1999

"The Elephant’s Tail"

Prepared by:

Stephanie Cox

Freshman Writing Program Assessment Coordinator

The Writing Program Office

English Department

Boise State University

October, 1999

Introduction

As I was looking at the data from this first year of program assessment, I was reminded of the old story of the blind men and the elephant. A group of blind men are led to an elephant and told to describe the animal. One feels the elephant’s leg and declares that he is like a tree trunk. Another feels its trunk and argues that the animal is like a serpent. Another feels the tail and says no, the elephant is like a rope. And so on. Separately, each man is wrong because he has only experienced a single part of the animal but speaks as if he has knowledge of the whole. But combined, their experiences form an accurate description of the entire beast.

What we have this year is the elephant’s tail. We assessed the E101 and E102 portfolios on only two competencies each. We can describe a small part of the Freshman Writing Program. We can discuss what we know about how well the portfolios met these competencies. But we would be foolish to pass judgement on the entire program.

In coming years, we will learn more about the state of our students’ writing as they leave our classes. It will take a while before we get that full picture, and I caution us to keep that in mind as we continue with the assessment experiment.

History

The Outcomes Assessment Program traces its roots to the development of the Writing Program’s Mission and Philosophy Statement and Course Descriptions. (See Appendix A.) This work was completed in 1996-1997, the year Bruce Ballenger served as interim Writing Director.

The freshman writing faculty participated and contributed significantly to this process. They met in small Round table groups to discuss issues of pedagogy and program; they responded formally to drafts of the Mission Statement and Course Descriptions. This resulted in documents that were built from the ground up by the people who have the largest stake in the program.

Work on the assessment program itself took place in 1997-1998 with Dr. Ballenger serving as Writing Director and Stephanie Cox (myself) serving as Assistant Writing Director. Within each course description is a list of competencies that students should learn in order to pass the class. After consulting with Marcia Belchier, the University Assessment Coordinator, we decided that the simplest, most direct method of outcomes assessment would be to look at how well students are learning what the teachers from this program have decided they should. So we drafted a plan where student portfolios would be assessed on how well they met these competencies.

After much revision and consultation with Marcia Belchier, presented the assessment program plan to Dean Eastman and then to the English Department faculty. English Department Chair Chaman Sahni created space in the 1998-99 department budget for assessment. Dean Eastman funded the position of Assessment Coordinator to run the program in that year, and I assumed that responsibility.

The Freshman Writing Assessment Program

In order to ensure a high level of compliance, the program is designed to be as simple as possible and to put as little burden as possible on the teachers and the students. Many of the teachers use a portfolio system in their classes where the majority of each student’s grade is determined by a 20-25 page portfolio of revised work that they submit at the end of the semester. But a significant number of teachers do not use this system, so we had to design an assessment program that can accommodate both.

The objective of our Student Outcomes Assessment Program is to assess whether our E101 and E102 students are leaving our courses with the competencies we aim to teach them. The assessment program consists of an annual sampling of one student portfolio or folder of collected work from each E101 and E102 class offered in both the fall and spring semesters, for a total of approximately 230 portfolios/folders. A committee of 8-10 readers will meet in late May to read the portfolio/folders and assess them in terms of how well they meet two criteria chosen from the competencies listed on the E101 and E102 course descriptions. Under the direction of the Assistant Writing Director, the Assessment Coordinator, the readers, and the Writing Committee then evaluate the findings and report them to the Department in September.

Each instructor has all of their students keep a folder of the evaluated work they produce for the class. This can be either a portfolio or a collection of submitted and evaluated assignments if the instructor does not use portfolios as part of her individual grading system. At the end of the semester, each instructor collects all of these portfolios/folders.

The Writing Program office determines which student portfolios/folders must be submitted by drawing a number between 1 and 25. All teachers will then submit a copy of the portfolio from the student with that number on their roster. For example, if we draw #7, all instructors will submit the portfolio of student #7 on their roster. Alternative numbers are drawn for cases where the selected student is no longer attending class. All portfolios/folders are then returned to the students.

Neither student nor instructor names are included anywhere on the portfolios. Portfolios are assessed only as representative samples of the freshman writing program and not as individual instructors’ work.

The assessment readers consist of either (or both) the Writing Director or the Assistant Writing Director, the Assessment Coordinator, and 6-7 other freshman writing teachers who volunteer for the project. Each reader is paid $250 to read 50-60 portfolios. The Assistant Writing Director is responsible for training the readers, who read and assess for three days in May after final grades have been turned in.

In the fall semester of 2000, the Writing Program will evaluate the process and results of the first two years of program assessment and make necessary revisions to it.

The Reading

On May 19-21, ten readers met to read and assess the 206 folders collected in the fall and spring semesters. The readers were: Bud Pedersen, Debbie Dwyer, Louis Simon, Joy Kidwell, Sue Hudson, Marion Thomas, Steve Doran, Bruce Ballenger, Michelle Payne, and myself.

May 19 was devoted to training the readers to assess the folders. The Writing Program Office had pre-selected portfolios that represented a range of competencies for E101 and E102. Assistant Writing Director Michelle Payne led the session. We worked on E101 first, then E102.

We read the portfolios and individually attempted to assess each one as either "Clearly Competent," "Barely Competent," and "Incompetent." Then we had a norming session where the assessments were tallied and we discussed why we each had made the assessment we did. After that discussion, we reassessed to see how well our opinions had consolidated. From that discussion, we created a "cheat sheet" for each class that clarified what we expected to see in a "Clearly Competent," a "Barely Competent," and an "Incompetent" folder. (See Appendix B.)

We spent May 20, from 9:00 a.m. to 4:00 p.m., reading and assessing 106 E101 folders for the following competencies:

Students can produce writing that has a purpose, as well as a clear focus and

point.

Students can produce prose without mechanical errors that distract readers

from attending to the meaning and purpose of the writing.

May 21 was devoted to reading and assessing 100 E102 folders on these competencies:

Students can do more than simply report information they gather from outside

sources, but use that information purposefully

Students have an understanding of the purposes and uses of documentation

and competency in using MLA or APA citations.

Each folder had to be assessed by at least two readers. If the first two readers could not agree on their assessment, the folder was sent to a third reader. The reading process went as follows:

All folders were placed in a "First Reader" pile. Readers took folders from that pile and read them, using the "Cheat Sheet" to help them with their assessment and marking their judgment on the "Assessment Sheet."
After making judgment, the reader placed the "Assessment Sheet" face down in the back of the folder and put the folder in the "Second Reader" pile.
When readers moved on to second readings, they read and assessed the folder in the same manner. They only looked at the previous reader’s assessment after making their own on a separate "Assessment Sheet" or if they were stuck and needed some help in making their own assessment.
If the two readers agreed, the folder was placed in a "Finished" pile.
If the readers disagreed, the folder was set aside. After all folders had been read, second readers met with first readers try to reach agreement on a disputed assessment. If they could not, the folder was placed in the "Third Reader" pile.

After the reading and assessing was complete, we took some time to reflect on the process and discuss what worked and what could be improved. No one complained of any major problems. The time frame of two and a half days seemed to work fine. Two competencies seemed the right number to assess. Any more would be too much work.

Some of us had trouble distinguishing between "Clearly Competent" and "Barely Competent" portfolios, and discussion revealed that while the cheat sheets did help a lot, the degree to which a portfolio was "Clearly" or "Barely Competent" was difficult to assess.

We debated the question of whether we needed assignment sheets to accurately assess a portfolio. It would be helpful to know what the student was assigned to write. But, some wondered, would that change the final assessment? Does it matter if a portfolio is "Incompetent" because the assignments were bad or because the response to them was bad? This is an issue we will have to look at and experiment with in coming years.

All readers agreed that the most valuable aspect of the three days was how the process made us reflect on our own teaching and evaluation skills. Readers especially commented about the helpfulness of the norming that occurred not only during the training session but throughout the two reading days as first and second readers of a portfolio met to try to reach agreement on their assessment. During the second day, readers were much more secure with the work and with compromise, and fewer portfolios went to third readers. This helped us gain confidence in our evaluation skills for our own classes. The chance to talk about evaluation and assessment of student writing in such an intense but collegial atmosphere was an enormous learning process and may be one of the most important steps toward improving the writing instruction in our program.

The Data

E101

106 of 115 classes were evaluated.

Image5.gif (37085 bytes) Image6.gif (37084 bytes)

Focus, Purpose, Point: Mechanical Errors:

Clearly Competent 52 portfolios Clearly Competent 47 portfolios

Barely Competent 38 " Barely Competent 37 "

Incompetent 16 " Incompetent 22 "

Of this total, 7 portfolios were judged Incompetent in both categories.

E102

100 of 111 classes were evaluated.

Image7.gif (36455 bytes)

Image8.gif (32742 bytes)

Purposeful Use of Research Materials: Skill in Documentation:

Clearly Competent 49 portfolios Clearly Competent 32 portfolios

Barely Competent 38 " Barely Competent 38 "

Incompetent 13 " Incompetent 30 "

Of these, 12 portfolios were judged Incompetent in both categories, 6 because they contained no researched essays.

Reflect on the Data

One of the most encouraging facts is the high level of compliance from instructors, above 91% for both courses. This indicates that the program was simple enough not to be so burdensome or suspect that a large number of instructors would forget or refuse to participate. Low compliance would have compromised the results.

Overall, the results are pleasing. The evidence is clear that in both courses the students have learned to take control of their own texts, writing essays and research papers that show a sense of purpose. That is something the program stresses. However, the levels of Incompetence for Mechanics and Documentation suggest cause for concern in those areas.

There were significant differences in the competency levels of portfolios from the

Spring and Fall semesters. In E101, the Fall semester portfolios were noticeably more competent, while in E102, the Spring semester portfolios were of a higher competence.

Over twice the percentage of Spring E101 portfolios were judged Incompetent in Focus, Purpose, & Point as Fall, and a third more were Incompetent in Mechanics. In contrast, over 50% of Fall E101 portfolios were judged Clearly Competent in both categories. Only 41% of Spring E101 portfolios were judged Clearly Competent in Focus, Purpose, & Point, and even less, 32% were Clearly Competent in Mechanics.

The figures are not quite as dramatic for E102, although they do show a clear superiority of the Spring semester portfolios. Fifty-eight percent of these were judged Clearly Competent in Purposeful use of Research Materials, while only 37% of the Fall semester portfolios were. Twenty-four percent of the Spring portfolios were judged Incompetent in Documentation Skills, compared to 37% from the Fall.

Fall is the semester when most students take E101; 74 classes were offered in Fall 1998 compared to 42 in Spring 1999. So it follows that most of those students go on to take E102 in the Spring; 65 classes were offered in Spring 1999 while only 46 were offered the preceding Fall. The assessment evidence suggests that students do better when they take E101 or 102 in the "traditional" semester.

There are a variety of reasons that may contribute to this discrepancy between semesters. They range from the assumption that students who fail E101 in the Fall or E102 in the Spring repeat the class the following semester and may fail or do poorly again, to the evidence that students, especially E101 students, who take the course during the "off" semester are often part-time or non-degree seeking students who are "trying out" college. We don’t really know. But these results are consistent with the findings of the Writing Program Descriptive Report compiled in 1997, so we do know that this is a pattern.

The data was presented during the Writing Program’s annual meeting on August 20, and the faculty had the opportunity to question and comment on it and then participate in a mock assessment of some of the portfolios. This helped an even larger number of faculty norm to the standards and also produced an opportunity to discuss pedagogy as well as the assessment process itself.

The question of the necessity of assignment sheets was raised again. The E101 competency of Mechanics was cause for concern too. People wanted to know how to tell specifically how many errors are too many. This was a concern during the May reading as well. An instructor who does not use portfolios was concerned that the assessment reading favors portfolios over folders. She suggested that they be labeled as either portfolios or folders so that readers could look for progression in competency in folders where students have not had the chance to revise work.

Overall, the faculty didn’t really know what to make of the data. The question, "What level of incompetence is acceptable?" was raised, but no one could answer it. This is a vital question, and one we must address as more assessment evidence becomes available over the years.

Curricular Response

It is important to remember that 1998-99 was only the second year we had the new course descriptions (& therefore these competencies) in place, so some people may not be used to teaching to them yet, or may not have mastered teaching them well. With that in mind, the Writing Program will focus its efforts on helping teachers develop better ways of teaching both mechanical skills and the intricacies of MLA or APA documentation.

Faculty Development Programs will focus on better ways to teach mechanics and documentation.
COMPTALK will host a moderated discussion on these topics.
The Writing Program will help develop Classroom Assessment Techniques (CATs) so teachers can better understand how well students are mastering material.
The Writing Program will collect teachers’ handouts, assignments, & exercises into a booklet shared among the faculty.
The Writing Program will buy books and copy articles on the teaching of grammar to be available in their library.

WordWorks, the Writing Center’s publication that is distributed to teachers across the disciplines, will feature articles on these topics.

Changes to the Assessment Program

Major changes to the Assessment Program will not be conducted until 2000, when we have enough information for a more thorough and thoughtful evaluation. But our experience in this first year has led us to make a few small changes. If these improve the process, they will become a permanent part of the program.

Repeat assessment of the competency we are most concerned about for each course every year in order to build a more thorough picture of the results.
Change the language of the assessment categories from Clearly Competent, Barely Competent, Incompetent to Competent, Minimally Competent, Incompetent.
Include a category on the Assessment Sheet that says "______ # of papers were not assessable."
Include a place on the Assessment Sheet to note how many pages and papers the folder/portfolio contained.
When each portfolio is submitted, have the instructor designate on the cover sheet whether it is a portfolio or a folder.

Conclusion

The Writing Program is generally pleased with the results of this first year of assessment. The structure of the program was successful, the faculty was involved, and the date has given us ideas for some specific improvements to focus on in our classes.

This has also raised the possibility of revising the course descriptions or the competencies. As we continue with assessment and get a more complete picture of the elephant, the logical step would be to review these and see if we still agree on the importance of certain points, or if certain competencies are assessable. The assessment program gives a formal opportunity to reflect on our teaching practices and philosophies.

Probably the best thing about the whole assessment program this year is the way it has refocused and energized the entire writing program. It has forced us to work towards agreement about the things we want to do, and got us talking with each other about the best ways to do them. The key has been to make the process matter, to really spend time designing a means for the faculty to be heard. The unqualified support of the Dean's office – including financial support – also helped us all believe that what we were doing was taken seriously.

Appendix A

English 101

Statement of Mission and Minimum Requirements

Fall 1998

Introduction

One of the strengths and challenges of Boise State is the rich diversity of its students. Boise State students often do not look like conventional college students, a profile that is more common at other college campuses in the state. BSU students may be adults, returning to college after many years. They may be learning English as their second language; they may work full or part-time as they attend school. Only about 10% live on campus. E101, which the great majority of incoming students take their first or second semester, will likely be an important introduction to the culture of the academy—its habits of mind, responsibilities, and conventions. Though many BSU students will not permanently enter our discourse communities, E101 should certainly help students acquire key academic literacies. However, a central purpose of E101 is to encourage students to see writing as a lifetime practice by helping them to see the connection between writing and their lives outside of the academy.

Transforming Attitudes

Evidence shows that how students view themselves as learners and what motivates them to acquire a particular body of knowledge strongly influences their learning. Often E101 students enter the course with little confidence in their writing abilities. As teachers of an entry-level writing course, we believe that students’ experience with language and language use in the course should be a positive one, and this will provide the basis for the development of writing skills. As a consequence, E101 focuses, in part, on the affective dimension of writing and thinking processes; that is, the course hopes to encourage students to believe that reading and writing are meaning-making activities that are relevant to their lives, within school and without. One way to measure the success of E101 is whether the course makes students more likely to want to make writing a part of their lives, not just during their years at college, but after college as well.

Writing as a Mode of Thinking and Learning

Part of helping students to embrace writing as a lifelong practice is to emphasize that it is a mode of thinking and learning, not simply a means of writing down what one already knows. We want to create the conditions that allow students to discover what they think through writing, and help them to see that this process can be used in any subject, any discipline, and almost any situation that demands thought.

Writing With Authority

We want our students to take responsibility for the ideas they discover as writers in E101. If the composition program represents a transition for students between being receivers of knowledge and makers of knowledge, then E101 is a course that first demonstrates how writing

can allow our students to demonstrate their authority over subject matter, often for the first time. We hope that the course frequently puts students at the center of their own discourse, challenging

Page 2 of E101 Mission Statement

them to discover and express their own ideas, and to make their ideas convincing or compelling to others.

Rhetorical Awareness

Another way in which students exercise this authority is by encouraging them to see that effective writing involves making a multitude of choices, and giving students control over those choices. Many of these choices are determined by the rhetorical situation—the writer’s purpose, the writer’s audience, the nature of the writer’s subject matter, the writer’s relationship to the subject. E101 is intended to increase students’ awareness of rhetorical situations and how they might affect the decision they make about how to approach a topic. Students learn that language has consequences and writers must take responsibility for what they write.

Writing in a Social Context

In E101 we try to create a community of writers in which students understand that membership implies engagement with each others’ struggles to make meaning. We want them to experience writing as a social interaction. Knowledge is not created in isolation but through dialogue and writing which is shared with a real audience. We also want them to see the writing classroom as an intellectual community in which they are encouraged to think freely and deeply, where difference is not only tolerated but seen as an opportunity for learning.

Minimum requirements

Specifics

Students in writing classes should continuously produce written work. This includes evaluated work, such as formal assignments and subsequent revisions, as well as informal and non-evaluated work, such as journal entries, in-class writing exercises, rough drafts, and peer responses. E101 students will produce, on average, the equivalent of five double-spaced and typed pages—1500 words—a week. The equivalent of at least 20 pages of double-spaced and typed writing will be the basis for assessing students' final grades in the course.

Students will write at least one essay that is about a reading or uses outside sources (reading, interview, observation) to help inform their thinking or point-of-view.

Students will receive an introduction to library references and methods of citing sources.

Students will meet all the attendance and class participation requirements and submit

required assignments on deadline.

Competencies

Students will demonstrate that:

they can produce writing that has a purpose, as well as a clear focus and point.

Page 3 of E101 Mission Statement

they can use revision to extend their thinking about a topic, not just to rearrange material or "fix" mechanical errors.

they can make assertions which are supported by evidence gathered from their experience, their reading, their observations, or their conversations with others, and the nature of these assertions and evidence is appropriate to the particular genre in which they are writing (e.g. narrative, argument, etc.)

they can articulate the rhetorical choices they have made, illustrating their awareness of a writer’s relationship to the subject, a text’s purpose, and audience.

they can use a variety of strategies for generating ideas for writing, for planning and organizing material, for identifying purpose and audience, and for providing useful feedback to peers during the writing process.

by the end of the course, they can produce prose without mechanical errors that distract readers from attending to the meaning and purpose of the writing.

English 102

Statement of Mission and Minimum Requirements

Fall 1999

Introduction

Students will emerge from E101 with more confidence in their writing skills, more awareness of the ways they can control the features of their writing for particular situations, and more understanding of writing as a mode of learning and thinking. In E102, students will build on these understandings and extend them. In addition, the course will further students' transition from receivers of knowledge to makers of knowledge. In E102, students will come to understand the spirit and methods of inquiry, especially the idea that knowledge is made through conversation with others. In this model for inquiry, ideas about what might be true are not simply reported, but negotiated. Writers consider many voices on their subjects, speak to and sometimes against them as they explore what they think. This conversation is often enacted through writing as students interact with outside texts. But such conversations also take place through collaborative work—working with each other and possibly non-students in off-campus settings—as well as discussions with authorities and others.

Research as a "Writerly" Activity

E102 is a course that emphasizes research, but research broadly conceived; that is, students will come to see that information can come not just from the library, but through field observations, personal experience, interviews, and other sources. They will also recognize that research is not a separate activity confined to the "research paper," but can inform any piece of writing.

Students in E102 will be encouraged to take charge of their own intellectual projects. Whenever possible, they will define for themselves how they approach their investigations (i.e. topic choice, focus, research strategy, and treatment) and take responsibility for the ideas that emerge from them. Students will learn that research—like many of their previous writing projects—should be initially driven by questions, not answers, and that research is a process that often involves reformulating the question as the writer learns more about the topic.

Methods of Reporting Research

While the process of inquiry is often driven by questions, the product of an investigation is often built around an answer writers discover as they explore their topics. Argument is the dominant (though not the only) mode of communicating the results of academic inquiry. In E102, students may be introduced to the many ways that research is conducted and reported in writing, but they should also have the experience of using the evidence they have gathered to make a persuasive point.

Page 2 of E102 Mission Statement

Minimum Requirements

Specifics

Students in writing classes should continuously produce written work. This includes evaluated work, such as formal assignments and subsequent revisions, as well as informal and non-evaluated work, such as journal entries, in-class writing exercises, rough drafts, and peer responses. E102 students will produce, on average, the equivalent of five double-spaced and typed pages—1500 words—a week. Assessing the equivalent of at least 25 pages of double-spaced and typed writing will be the basis for determining students' final grades in the course.

Students will write at least two research-based essays that use multiple sources of information. One of these may be an extended essay.

Students will write at least one essay that builds an argument in support of their point of view.

Students will meet all course requirements for attendance, submission deadlines, and class participation.

Competencies.

E102 students will demonstrate that:

they can do more than simply report information they gather from outside sources, but use that information purposefully.

they can take charge of their own investigations, formulating the question(s) that will drive the research, finding the information they need to explore those questions, and coming to their own conclusions about what they discover.

they can read critically, including an ability to discern what the author is asking the reader to believe, and to evaluate the evidence and means of persuasion.

they have an understanding of the purposes and uses of documentation and competency in using MLA or APA citations.

they are competent in using library reference sources to find information.

Page 3 of E102 Mission Statement

Appendix B

Cheat Sheets & Score Sheets

Cheat Sheet E101

Purpose, Focus, Point

Competent:

Purpose becomes apparent within the opening paragraphs, or the closing may pull the essay together and make the purpose apparent

All paragraphs generally support and develop a well-defined focus

Answers the "So What?" question

Barely Competent:

Purpose does not become apparent until late in the essay, and the closing does not pull it together

Focus may be very broad; paragraphs may not always stay within the focus

Sort of answers the "So What?" question, may have more than one possible answer

Incompetent:

Purpose never becomes apparent

Focus may never be apparent, or may be so broad that supporting paragraphs can not adequately develop it

Does not answer the "So What?" question

Mechanics:

Competent:

Generally error-free

May be an occasional typo or error in punctuation, sentence structure, or usage, but this does not indicate a pattern

Barely Competent:

May be several typos or errors in punctuation, sentence structure, or usage, but they generally indicate poor editing and not patterns of misuse.

ESL tendencies such as misuse of prepositions may be noticeable but not overwhelming

Incompetent:

There are many errors, indicating sloppiness in editing or patterns of misuse.

Common ESL errors indicate a lack of mastery of the language

Cheat Sheet 102

Using information purposefully:

Competent:

Purpose of essay becomes evident

Writer guides reader through info

Introduces & comments on info

Barely Competent:

Less guided, info seems to wander

Source driven rather than writer driven,

The work is structured around the sources

Incompetent:

Lists info from sources

No purpose to info

No writer presence

Documentation:

Competent:

Shows control in introducing sources

Proper placement & form

Barely Competent:

Improper attribution of sources

Missing or incorrect punctuation

Errors in citing electronic sources

Incompetent:

No documentation

Mixed format

No understanding of how or why documentation is used

Missing information

Improper form

Plagiarism

E101

Reader’s Initials:_____

Class & Section:___________

Competency: The portfolio demonstrates that the student can produce writing that has a purpose, as well as a clear focus and point.

Clearly Competent Barely Competent Incompetent

Comments:

Competency: The portfolio demonstrates that the student can produce prose without mechanical errors that distract readers from attending to the meaning and purpose of the writing.

Clearly Competent Barely Competent Incompetent

Comments:

E102

Reader’s Initials:_____

Class & Section #:____________

Competency: The portfolio demonstrates that the student can do more than simply report information they gather from outside sources, but use that information purposefully.

Clearly Competent Barely Competent Incompetent

Comments:

Competency: The portfolio demonstrates that the student has an understanding of the purposes and uses of documentation and competency in using MLA or APA citations.

Clearly Competent Barely Competent Incompetent

Comments:

10-99

256\Assessment\Assessmentt Reports\Portfolio Assess Reports\F98-SP99 Assessmt Report-Elephant’s Tail