Deanna Mascle’s Field Methods blog

I almost put a question mark on the end of that title as I feel like that final deliverable is still a question mark…I’m working on something but it isn’t what I said it would be and I don’t think it will go beyond my small circle of co-researchers. I do think it will be an useful document for me but is that good enough? Guess I better check with Becky.

When we had to propose our research topic at the beginning of the semester I had some difficulty, in my mind, presenting my project as field research. But now that I have come to more fully understand what field research is — I am definite that this is field research. The work I’m doing is situated very firmly within a specific field — first-year writing or freshman composition — and studying the work of that field in the classrooms and offices (not to mention copy rooms) of the writing teacher.

I think field methods is a much more effective way to study a process such collaborative grading than a quasi-experiment and I think education and teaching in general. Anyone who has ever taught knows that the classroom experience cannot be controlled — and I think the lessons learned from a “controlled” experiment cannot hold very true for real teachers and real classrooms.

So what am I doing now? First I gathered all my material together and it is a pretty impressive list of artifacts:

Artifacts:

• Databases created for data mining
• Data mining project
• Field methods report (setting and timeline)

• Summary assignment
• Summary rubric
• Student summary papers
• Completed summary rubrics
• Synthesis assignment
• Synthesis rubric
• Student synthesis papers
• Completed synthesis rubrics
• Engle, Mascle, Reding interviews
• Emails

Then I realized that there was something very important lacking from my original list — a good narrative timeline of the project.  I should have done this as I went but this is my first project of this scope and there are a lot of things I should have done differently (I won’t mention my 36-hour panic when I thought I lost a set of scored rubrics…if you could see my office you will marvel that I ever found them). So that is what I’m currently working on. I think this document will be a key element for me — especially in the future. Right now all the details are fresh but…

So where do I go from here?

• I will have my collaborators “member check” my timeline
• We will have a debrief to go over our assignments and rubrics to tweak them
• We will have a collaborative session to discuss some of the papers with more disparate grade spreads

When those things are done I will feel that the pilot study is complete but I still have questions…

1. Will Becky accept my timeline as a deliverable or does she want me to “deliver” what I promised?
2. What do I do with all this info now? I can think of several avenues for publishing (and am in fact collaborating with Ronda Wery and Fred Kemp on a 4Cs proposal right now) — obviously different folks would be interested in different aspects of the project — researchers, teachers, administrators
3. Where do I go with the study? Do I expand the study in the fall semester to include more instructors– do we do it the same or simply collect the scores (and maybe papers?) of the larger pool for future study or do we harness technology (ala TTU) to ease the process?
4. Do I expand the study in the spring semester to include more assignments?
5. Do I expand the study to include more institutions?
6. And my final question…not to be sneezed at in terms of importance…what do I report to my dept. Just the bare facts, numbers, and statistics? Do I go for a more narrative approach? Some middle ground? Should I put it to a vote of the class?

Thank you everyone for your support and help this semester! I couldn’t have done it without you — or certainly with a lot more angst!

As I struggle with my data and end-of-the-semester reports/papers, I finally identified the main cause for my struggle with my paper for this class…I don’t know my audience.

As I see it, my collaborative grading project could be of interest to three groups of people:

1. researchers (specifically those interesting in writing research) — which could mean targeting such journals as Journal of Writing Research or Assessing Writing. The paper I’m writing for data mining would seem to fit more in this vein as it focuses more the numbers and patterns in those numbers and findings
2. writing program administrators — what can we learn from this study to inform program administration and planning in terms of assignments, rubrics, and faculty collaboration (for starters anyway). This might be the approach for 4Cs conference presentation and/or an article for WPA Journal. Both qualitative and quantitative data would play a part with perhaps more emphasis on qualitative to help interpret what quantitative tells us? Or rather to tell the story of the data?
3. teachers — what can this study teach writing instructors about assignments, grading, and collaboration, I’m guessing for this audience the data would play a minimal role with more focus on the story behind the data and what this experience meant for the instructors involved and what they learned from the experience and how it changed their pedagogy and methods. Perhaps a better audience for this would be the Journal of Teaching Writing or Teaching English in the Two-year College

Actually, thinking this through has helped me — I think for my final deliverable for this class I will focus on #2 and hold off on #3 until I’ve gone through more data collection (as mentioned in other posts — more teachers, more assignments, more institutions)

How much “milking” of one research project can you honorably do? I could see doing something for all 3 of these audiences and they would be very different approaches plus I would pull in some different data.

Is my novice showing?

Re: The call for NEXT year’s CCCC (Conference on College Compositition and Communication) has gone out already!  It will be in Louisville, KY, and the theme is “The Remix:  Revisit, Rethink, Revise, Renew”.  Deadline is May 1 for paper proposals, May 8th for electronic submissions.  You might consider proposing something from this class or another you’ve taken.

Becky says: Well, you could definitely pitch the collaborative grading as a “remix” of “traditional” grading, particularly if you have data! You could also pitch it for a reason: Moving towards portfolio, creating rubrics, getting faculty to talk, and so forth. I can see a panel, though they prefer panels made up of folks from different institutions.

Ronda has expressed an interest in working on a panel but we are floundering a bit as there are a lot of different directions we could take this. How many people do we need for a panel? How do we pitch a panel?

Do we work up new assignments and rubrics that cross institutional lines? See how well our MSU assignments and rubircs cross institutional lines? Oh dear my head hurts just thinking about  all the ways we can remix…

Susan Lang suggests:

one thing I’m thinking that might add to your study’s impact is if you could use the rubrics in more traditional settings—that is, if you could use these in classes where the instructor grades his or her students’ work and get a baseline of average scores given under those circumstances—then compare to the experimental sections. Obviously, that’s not something you can do in the scope of this class, but I think it’s something that would help verify your results—that, and having someone examine the students’ writing independently and determine whether or not the writing seemed to deserve the scores it was given.

I am taking two classes this semester — Field Methods and Data Mining. I am using my collaborative grading project for both classes although obviously my approaches to what I do with this information are different. It is actually interesting to look at how things differ and compare when using different lenses. That is not the reason for this post. I have an initial analysis due for data mining this week and I decided to focus on the data from the first set of collaborative grades (we used two assignments but synthesis is not yet in — plus summary data alone offers more than enough information to write a short report). That is in part why I’m posting here. In hopes that mulling this over will help me focus on what aspects of the data are of interest to begin with.

The data set under study includes six sections of a first-year writing class taught by three instructors. Each of the three instructors has two classes participating in the study. Although each section started the semester with 18 students, attrition has lowered those numbers. In addition, not all students turned in the required assignments.

Each student assignment received two sets of scores and was not scored by the student’s instructor. All scores were assigned using a rubric specifically created for this assignment by the three instructors for the purpose of this study. The total score is calculated by looking at five key areas for each summary and synthesis. For the summary these were: introduction, content, use of source, writing and mechanics, and MLA format and documentation.

There are obviously lots of different ways to look at this data — focusing on instructor grading or on student performance. I’m thinking about starting with instructor grading — specifically looking at variation in grades.

I’m thinking about possible publication opportunities for my collaborative grading project.

One is:

The call for NEXT year’s CCCC (Conference on College Compositition and Communication) has gone out already!  It will be in Louisville, KY, and the theme is “The Remix:  Revisit, Rethink, Revise, Renew”.  Deadline is May 1 for paper proposals, May 8th for electronic submissions.  You might consider proposing something from this class or another you’ve taken.

But want to think about other possibilities as well.

Still thinking and open to suggestions. These are not in order of preference or anything as I haven’t progressed beyond thinking stage yet:

Assessing Writing

WPA Journal

Composition Forum

NCTE’s:

Research in the Teaching of English

Teaching English in the Two-Year College

Obviously I need to think about the right fit for each journal but also its standing (thinking about job search in the future)

As part of my collaborative grading project, I would like to either interview — or perhaps have each partner write reflectively — about various topics, questions that connect to this project. Clearly I will have a lot of hard data to work with but I think for other instructors more will be needed to understand the full implications of the data and the conclusions that can be reached from it.

What questions/topics should I address?

Describe your usual assignment development/creation process. How do you usually give student assignments?

Have you used a rubric before? If so, what did it look like?

What made you agree to take part in this project?

What qualms did you have about this project?

How do you feel about this project now?

Do you think collaborating on assignment and/or rubric design makes for a stronger/weaker assignment/rubric?

What was it like to collaboratively grade (as a grader and as an instructor)?

Has this process changed your teaching? How?

What have you learned from this project?

New questions:

What do you see as the benefits and drawbacks of this process for instructors and students?