An analysis of my course evaluations for Spring 2015

They are in: recently I received the email containing my course evaluations for the Spring 2015 semester, and I am ready to analyze them in their full glory. As I indicate on my Course Evaluations page, I have a somewhat-tortured relationship with student evaluations. Let me just start out by saying that I would be totally fine with all of my students saying they loved my course to death and giving me all perfect marks. But that never happens, so what I am always left to do is interpret the meaning of my imperfect marks. So here goes…

Let me start with the pleasant surprise: my lone section of Evolution. This is a course that just started teaching again after nearly four years of putting the course on hiatus. Last semester I taught two sections of the course, so I had the chance to really refine it. And this semester I made a big decision: I greatly reduced student workload by eliminating the requirement of a Term Paper. This version of the course did feature a Midterm Exam for the first time, but in all other ways it was a light version of previous iterations.

While the course was less rigorous, students overall did not do very well. I had a lot of students who missed classes, missed assignments, and generally struggled with coursework. And on top of that, I had one student who was disgruntled throughout the course: after skipping the first day of class and failing to read about course requirements in the syllabus, he accused me of bias against him because I had not properly informed him of how he would be graded. Ugh. A class struggling overall, and one really delusional student: I thought for sure that this would be my worst-rated course of the semester.

So I was kind of shocked when I got the ratings back for this course. Overall, my rating in the course was 3.72 out of 4.00 (93.0%). Well over 90%, which is really great as far as I am concerned. Digging down a little deeper, things look even better. One student — almost certainly my disgruntled fellow — gave me the lowest possible rating in all but two categories. With that rating removed, my overall average is 3.91 out of 4.00 (97.7%), one of my highest ever. There is an important lesson to be learned from this semester: the fact that your students are not achieving in your class does not mean that they do not appreciate what your providing for them.

Overall, the ratings for Evolution were comparable across categories, with the following categories being rated a bit lower:

• 3.54 out of 4: The quantity of assigned work was appropriate to goals of the course.
• 3.23 out of 4: The course helped me improve my problem solving skills.
• 3.31 out of 4:  I would recommend this course to another student.

Students always resent workload, so I am okay with this rating… especially given that they no longer have a Term Project to complete. The problem-solving skills category is an interesting one. I certainly would like to have my students learn to better solve problems — in particular being able to explain how particular species or species traits evolved — after taking this course, but I am not sure how students are interpreting the term problem-solving. The slightly lower I would recommend this course rating is very telling to me, and consistently one that I get rated lower on. What does it mean when students think that a course is well-designed and well-executed but are less enthusiastic to recommend it to friends? Just a hypothesis, but perhaps students recommend easy General Education courses rather than well-designed General Education courses to their peers.

One thing that does not get captured in the numerical ratings of course evaluations is the qualitative experience of each student; luckily, our course evaluations ask students to leave detailed feedback on: 1) the best features of the course, 2) changes that would most improve the course, and 3) anything else students want to add. I obviously cannot provide you with an unbiased and comprehensive overview of these comments, so here are some biased and incomplete impressions:

• Students like the overall design of the course, commenting on the predictability/consistency of workload, the structure of the class and the use of the learning management system to run the course, the diversity of in-class activities, and the many opportunities for students to improve their work.
• Students think I am enthusiastic about the topic.
• The course is challenging, but students appreciated the challenge overall even when they could not quite meet the challenge.
• Students have a variety of suggestions for improving the course, including giving students more opportunity for making up missed work, giving students more time for in-class assignments, providing more readings outside of the textbook, and getting rid of quizzes at the start of class. None of these suggestions was made by more than a couple of students, so there was no consensus on how to improve the course.

I am very happy with these evaluations. I really appreciate that this group of students differentiated between how they were being taught and what grade they earned. I think that the design of this course has reached a nice equilibrium, and I do not see very much in these evaluations that would push me to change this course in any major way. I have been considering the elimination of quizzes at the beginning of class, mostly because I would love to have those fifteen minutes of class time back, but I would have to replace the quiz with some other assignment that assures that students do the readings. As you will see below, an alternative I favor might not be loved by students.

My other two classes sections for this semester were for Ecology for Architects, which I seem to be on track to teach every Spring semester. As the only full-time ecologist in my department, it makes sense that I would be the person to ‘curate’ the content for this course. Nonetheless, the course has a weird history. My colleague Damon Chaky taught it for many years, creating a very realized course design. Aman Gill, who is also an ecologist and has had a couple of appointments at Pratt as a full-time instructor over the past five years, has also taught the course and developed his own version of it. Two years ago I was asked to teach the course, and I inherited materials generously provided by Damon and Aman. I also had been teaching the Ecology course for art and design majors during my previous six years at Pratt, so I came into my new assignment to Ecology for Architects well-equipped to teach the course.

Sometimes it is really great to be given a new course assignment. I am always refining my existing courses, but once you get into a particular mode of teaching it is really hard to make wholesale changes to an existing course. I guess what I would say is that courses one has taught for a long time have a lot of momentum, and it is difficult to force oneself to redirect all the inertia embodied in an well-worn course trajectory. But when given a new course to teach, one has the option to perform a complete re-design. So that is what I decided to do in preparation for teaching Ecology for Architects for the first time in the Spring of 2014. Rather than re-purpose my existing Ecology course or simply use the materials that Aman and Damon provided to me, I decided to build the course from the foundation up. I would use the materials I had available if they fit my larger design plan, but I would also be open to throwing things out that did not seem right, creating new curriculum where needed. I was in an ideal place to re-design this course.

I will go into the full rationale for the course at another time, but what I produced was an environmental science course with a strong ecological focus. The course is designed to teach all of the basic ideas of ecology and a lot of environmental science concepts, mostly using examples with relevance to architects. For example, in my regular Ecology course we spend a lot of time using the phenomenon of overfishing and fisheries collapse to learn about population growth and decline, trophic cascades, and sustainability. I have a lot of great material on this topic, including a teaching tool that I created with a former Pratt graduate student. But I decided not to spend any time on overfishing in Ecology for Architects, because overfishing is only tangentially a problem of the built environment. Instead, we tackle conservation issues more directly related to how and where humans build structures.

A big feature of Ecology for Architects as I designed it are the weekly group activities that students complete in class. A typical class starts with some review and discussion of the previous week’s activity, then moves on to a slideshow and discussion related to the current week’s readings, and then ends with students working on some task in class. Although time management is always an issue because I try to squeeze so much into this course, I try to make close to half of the class time spent on these group activities. I am proud of how diverse these activities are: each week, students are asked to tackle a different challenge, with minimal overlap or similarity in the challenge presented. And students struggle with these challenges… but in a good way. I give them tons of feedback on their work, both individually and as a class, so any student who spends some time thinking about their work and my impressions of that work will learn a lot. No other course that I teach even comes close to having such a robust and well-honed collection of group activities.

You can see that I have a lot of pride about how I have designed this course, pride largely connected to how much time and effort I have expended designing the course. So you can imagine how frustrating it is to me when I get poor ratings for that course. As you can see by checking my course evaluations page, by far my worst ratings have been for Ecology for Architects. It started last Spring 2014, my first semester teaching the course. One section of the course gave me ratings that were relatively low for me, and the other section gave me my lowest ratings ever. I took a step back and asked myself “what should I do to make this course more palatable to students?”. A major complaint was that the course required too much work, and I think that this might have been a reasonable complaint. I had asked students to complete a Term Paper over the course of the semester, and they had struggled to complete the various stages of this assignment on time and at sufficient quality.

So this semester, I decided to make two fundamental changes to the course. First, I decided to eliminate the Term Paper. This was a dramatic change, because I added no additional work to replace the Term Paper; Spring 2015 Ecology for Architects students simply had to do less work in order to complete the course than Spring 2014 students. My second change was to replace beginning-of-class quizzes with a new assignment related to assuring that students do the readings before coming to class. The quizzes — which I still use in most of my courses — are a great way to force students to come to class prepared, but they are expensive in terms of time, basically causing the three-hour class to start twenty minutes late. In an attempt to save this time and maintain the benefits of some kind of assignment related to the readings, I decided to create what I knew would be a challenging assignment: for each week’s readings, I made a list of concepts for each reading and asked students to identify which of these concepts were covered by that reading. These Reading Questions proved to be difficult for students, although many of them dramatically improved over the course of the semester.

So you can imagine that having made only two changes to the course — one that dramatically decreased student workloads — and taking a second crack at teaching a new course, I was hoping for some better evaluations this semester. And then you can imagine how tough it was for me to swallow the fact that I got lower ratings this semester. Let me put the hard cold facts out there before I digest them. One section gave me a 3.36 out of 4.00 (84.0%), while the other gave me a 3.26 out of 4.00 (81.5%). Ugh. These are some of my lowest scores ever, and they average out to even lower than my previous Ecology for Architects ratings. Man, what is going on here?

I am not sure that I can tell you why I have gotten slammed pretty hard by three out of the four Ecology for Architects course sections I have taught, but I will try to get some clues on the table. Let me start by looking at the categorical ratings from each section to see where they really do not approve of my work:

• 2.24 & 2.50 out of 4.00: The quantity of assigned work was appropriate to goals of the course.
• 2.76 & 3.19 out of 4.00: The instructor stimulated my interest.
• 2.93 & 3.00 out of 4.00:  The instructor’s evaluation/grading of my work was fair.
• 2.36 & 2.60 out of 4.00: The course helped me improve my problem solving skills.
• 2.71 & 2.69 out of 4.00: I would recommend this course to another student.
• 2.47 & 2.69 out of 4.00: I would recommend this instructor to another student.

Although my ratings in other categories are not phenomenal, they are all ‘solid’: well above 3.00.

Digging into student comments is one way to try to get context for the numerical ratings that students are asked to give. As indicated above for my Evolution evaluations, there is no way for me to be either comprehensive or objective in how I summarize these comments. So here is my incomplete and biased account of overall student comments:

• Overall, students think that I am knowledgeable and enthusiastic.
• Most students appreciate that the class is structured and well-organized.
• A lot of students echoed the idea that the course requires too much work. Students particularly are frustrated by the readings. Some comments were revealing: students expect that the readings should be short enough so they can finish them the night before, right after their design classes… I guess that making time to do the readings at some other point in the week is impossible? Student expectations for reading load and my own understanding of what is reasonable for a three-credit class seem very misaligned.
• Students do not like the Reading Questions. They find them to be too difficult, not valuable, and tricky.
• Students overall complain about the grading system in the class, both as too difficult and too unforgiving. Several students are frustrated by the fact that they do not get full credit for simply being present in class, or that they do not get a free pass for missing one class.
• Students diverge on the issue of whether Group Activities in class are valuable; some students like them, others do not.
• Some students like the course’s use of the learning management system, others strongly dislike it.
• One student — who was positive on the course overall — recommended the following: “Never ever requiring another Scientific American reading again. They were extraneous and biased. I think a lot of criticism that this course is just one big guilt trip comes from frivolous readings like these.” I found this comment really interesting, in part because the Scientific American articles are characterized as having a strong political bent, but also because — at least in this student’s mind — the course is regarded as a big “guilt trip”. I think that most of us in the fields of ecology and environmental science face an interesting challenge when engaging students with what is mostly factual material. Another student commented “I… think the class was affected by his personal opinions and beliefs“. While these comments were pretty isolated, I find them fascinating because throughout the course I am pretty neutral about what I see as the solutions to the ecological problems that I present.
• Sometimes an isolated comment can get to you. One student said “Please do not make fun of people’s answers in class. This discourages students to participate in class even more“. Man, I wish I had a video of whatever event in class inspired this comment! Although no other students suggested that I was nasty to them, I hate to get even one comment like this.

Comments that students make probably represent the most valuable source of information on these evaluations. They explain that a lot of student frustration is linked to work-load and rigor. They also suggest that different students have very different reactions to different facets of the course.

How do I interpret my low ratings, particularly in the categories outlined above? That question encapsulates the major problem with course evaluations: if I could ask follow-up questions of each student, I might be able to ascertain their rationale for marking me low. But in the absence of such follow-up questions, I need to make some guesses.

Let me start off with the worst rating, which was for the quantity of work assigned in the class. Clearly students feel that they should be asked to do less in the course. This is ironic given that I cut a time-consuming major assignment for this version of the course. But looking at the students’ comments, they very consistently ask for less reading and they particularly hate the Reading Questions that I require them to do on a weekly basis. To be clear, reading is pretty much the only major thing out of class that I ask students to do. For a three-credit class the typical reading load is 3-5 articles, usually about 6-8 pages each. So figure about forty pages of reading per week. This does not seem like a lot of reading to me, but apparently it does to my students. Perhaps what particularly bothers them is that the Reading Questions I assign make it impossible to ‘skim’ the readings, generating low scores for students who do not closely or carefully read the assigned texts; ironically, it was this very characteristic of the Reading Questions students complained about. To me, that sounds a lot like we just want you to be less rigorous. What is interesting is that several students suggest that I pretty much eliminate readings and rely on the class to deliver content, a suggestion that would require that I just lecture rather than providing students with weekly challenges. A problem with course evaluations is that students can suggest a change without having any sense of the ‘cascading effects’ such a change would have on the design of the course.

The complaint that I don’t grade fairly is also interesting. Again, it is hard to know exactly what facet of the grading students are basing this low rating on, but given that most of my assignments are graded by rubric (if not simply by right-and-wrong answer), the fact that most students felt that my grading was unfair to me comes down to my grading was too rigorous. Many student comments complained about the fact that I give students a lower grade for participation (80%) on days that they do not say a single thing in class. Again, I have the option to yield to student desires by giving every student 100% credit just for being in class in bodily form. But I know from past experience that at least some students respond to lower participation grades by improving their participation, the exact result that the lower grade of 80% is designed to produce. Squeezing sufficient participation out of students in order to create a productive classroom environment is difficult enough with the grade incentive to participate… I can only imagine in horror what my classroom would look like if students had no incentive to participate.

As I indicated above for Evolution, it is really hard for me to know what to do with the low rating for teaching problem-solving skills. Ironically, each week’s group activity is essentially a problem to solve. So either students think that these activities do not actually teach them to solve these problems, or they do not see this learning as a form of problem-solving. Without further commentary from students, this rating is very hard to interpret.

If there is one of these low ratings that I would most aspire to improve, it would be the one related to stimulating student interest. Okay, I get that you do not like the way that you are being graded — or that I ask you to do this much work — but is the course really not that interesting? If so, I have some work to do. Again, I have to turn to student comments to try to figure out what is not stimulating: the course content, the class activities, or me standing in front of the room.

I am really bummed out that overall students would neither recommend my version of this course nor me as an instructor. I figure that you have to look at the other ratings that are low to figure out why I do not merit recommendation. Clearly it is not because students think that I am disorganized or too easy or clueless about my subject area. But is it because I am very rigorous, and the goal of every architecture student is to avoid all distractions from their studio work? Or is it because I am not that interesting? I wish I knew!

One interesting thing about the ratings for Ecology for Architects is that they reflect a fairly-consistent student attitude about their own responsibilities to the course. I would label this attitude the I have the right to remain passive approach to being a student. Students feel that they should be allowed to treat the readings as they deem appropriate (including ‘skimming’ or skipping them), that they should be able to sit through an entire class without contributing to the dialogue at all, that they should not be asked to collaborate with their fellow students during class time, and that they should not be held accountable for what they have learned on a weekly basis. All these desires contradict what we know produces the best student learning outcomes: being asked to perform close reading of literature, actively testing out one’s ideas through in-class discussion, working with others to solve problems related to the material learned, and being given constant feedback on their understanding. I get it, learning less would take a lot less time from students’ studio work, but should I honor students’ desires to learn less?

This brings me to the crux of why I have gotten dramatically lower ratings from architecture students. The way I see it, I face a perfect storm with this course. To start, it is a required course. Students may not even have the chance to choose which instructor they get for Ecology for Architects, and they have to take the course. That is fundamentally different from every other course I teach. Some of my art and design students may regret their choice in the end, but all of them decided to take courses in ecology or evolution from me. My architecture students were stuck taking my course, and they were probably stuck with me as their instructor. Required courses are going to frustrate more students, because no one can avoid them.

Because this is a required course, Ecology for Architects is taught by a number of instructors. This semester was interesting because of the three other instructors for the course, two were using the same basic curriculum and assignments as me. But historically — and even this past semester — different instructors employ radically different grading systems, require radically different work from students, and conduct the classroom in radically different manners. Given that most of the Ecology for Architects
students are second-year students who interact intensively through their studios, you have to imagine that they talk about the different versions of this course they each receive. They must be aware of the inconsistency. This inconsistency must rub students the wrong way, especially when one version of a required course is more rigorous than another. It is possible that many of the complaints about my class’ level of rigor — and the tendency of students to not recommend me — stem from the fact that my version of the course is deemed more demanding. In a competitive and demanding major, all students should face the same demands from a required course, but that consistency has never been established at Pratt.

I was really surprised by the fact that of my two sections of this course it was my high-achieving section that gave me the lower ratings. This is generally exactly the opposite of what happens (at some point I want to actually graph course average grade versus average course rating), as classes who struggle tend to take out their frustration with the course via ratings. But this past semester, I had two very different sections. One section was pretty diligent about their work, but struggled on some assessments… especially exams. The other section was pretty neglectful of their work, but managed to do better overall on assessments. It was almost like the high-achieving section was taking me more seriously, while the other section was more dedicated to the idea of I have to survive this course, but not necessarily get a good grade in this course. The high-achieving section rated me more poorly, which I can only interpret as some resentment for requiring them to do work that distracted them from other pursuits (especially their studio work); the other section was less upset with me simply because they took my whole class less seriously. I cannot come up with any other explanation for this very counter-intuitive trend.

I recognize that a lot of what I have written above comes off as — well, really is — pretty defensive. And that is because… I feel pretty defensive about these ratings. I feel as though I have worked hard to provide students with a high-quality course, and many of them do not appreciate that. There is also the ‘banging your head against the wall’ feeling emerging for me about this course: other than making it a lot less rigorous and therefore less educationally valuable, I do not know what I can do to achieve higher ratings. And that makes me prone to not worrying about lower ratings in this course.

That all said, I am still looking for ways to improve student experiences of this course. One thing that I am going to try in future versions of Ecology for Architects is to be a lot nicer. I think that I am pretty nice and welcoming, but apparently a lot of my students do not share that feeling. I can be intellectually tough in class, which some students may interpret as mean (see the “do not make fun of people’s answers” comment above). Maybe I am too rough — after all, I am a native New Yorker (via Long Island!) — but after visiting some of the second-year studio final reviews, I know they receive harsher criticism from their studio professors. I may just have to recognize that architecture students have a different sensitivity than students in other majors, and I may have to adapt my in-class approach to that sensitivity.

I had a really great conversation with a colleague (whom I trust completely) about how it might be a good idea to capitulate a little to what seems to be a growing expectation among our students that they will be treated like consumers. My instinct is to say “come on, are you kidding me, give in to the idea that because a student paid tuition they are entitled to be coddled by me?”, but what my colleague nicely pointed out is that it really does not require a huge amount of work on our part to send a standard email to students who are missing assignments, struggling, or otherwise falling short of course expectations. An email that says I noticed you are struggling and I would like to help you improve your work might go a long way in changing how students interpret the rigor of the course. Or maybe not… but this is an experiment worth trying!

I am also going to take a close look at the reading materials that I provide to students to see if there is a way to make them more interesting while still teaching the same overall conceptual material. Some comments about the readings — paired with the low rating for stimulating student interest — make me wonder if tweaking readings would make students more interested in the material. In this course this is a difficult balancing act, because although there are some readings that are directly about architecture it is impossible to have all the readings be clearly and directly related to architecture. I think that a lot of my students would like to have everything tied to their major, but you cannot become fluent in ecological science by only reading articles about buildings.

Students do not like the Reading Questions, but at this point I am not ready to drop them based on their criticism. But, I think that they can be improved. One student commented that there is no feedback on these questions, and I think that this is a very valid criticism. If each right and wrong concept came with an explanation of what aspects of the article relate to why that concept was or was not expressed by the reading, I think that a lot of students would get more out of the Reading Questions… and probably eventually do better on them. Of course if students want to just skim, or work on the readings at the last minute, there is nothing that I can do about that.

The other thing I plan to do in the future for Ecology for Architects is to more clearly lay out the rationale for each of the ritualized assignments I require (Reading Questions, Group Activities, and Follow-Up Questions) on the first day of class. Maybe what students really care about is knowing why they are being asked to do all these different things for my course. I will try in the future to better explain why some of the more challenging facets of the course exist. Perhaps I should also risk infantilizing my students by suggesting to them that they need to find a time during each week to complete class readings, and that that time might not be best selected as the night before class.

Four sections of a course over two years is a tiny snapshot, so I will wait to see what happens next Spring. But to be honest, I am not overly hopeful for a major change. If I had to bring up my course evaluation scores (for example, if I did not already have tenure), there is only one plan of action that I would feel confident in following: just make the course easier. That is not something I plan to do. At this point I believe too strongly in the importance of my curricular content — and what it teaches students — to simply water it down. Barring some larger discussion of what students need to learn in Ecology for Architects, which would have to be paired with harmonizing the approach of each section of this course, I do not see my evaluations coming up by any other method. But let us see…