Frequently Asked Questions (for instructors) - Online SIRS

Which courses are using the online survey?

You can view a full list of surveys in the online system, for the current semester. All Winter and Summer session courses use the online survey system. For Fall and Spring semesters, most departments have chosen to use the online system but a small number of departments continue to use the paper system.

What questions are on the survey?

The questions are identical to the ones on the paper survey. We have an example of the online survey , and we can also provide interested people with a fully functional demonstration. Please contact us at info@brokenemail.rutgers.edu please replace "brokenemail" with ctaar.rutgers.edu to arrange a demonstration.

Can I add my own questions?

By request we can add additional questions if they are to be used department-wide, but because this is a manual process we cannot individualize the survey for every instructor. Eventually we hope to add the ability for instructors to add their own questions but this is still at least a year or more off. In the meantime, we suggest instructors use alternate methods to run their own informal surveys, such as a poll or an anonymous quiz in Sakai or a Google Docs web form.

Can I see how many students have replied to my surveys?

We will send an email update every Tuesday and Thursday during the run of your survey to let you know how many students have replied up to that point. You can also check on our current surveys web page (data updated daily).

How do I see the results of my surveys?

We will email the results of the surveys to each instructor on the day after the registrar closes the grading period. Exact calendar dates vary from year to year, but roughly:

• Fall Semester: results distributed first week of January
• Spring Semester: results distributed prior to Memorial day
• Summer session: results for all sessions distributed in the last week of August
• Winter session: results distributed prior to the start of the Spring semester

The survey report sent by email is misaligned and difficult to read, can I get a PDF copy?

We send the report as an ".html" attachment to an email message. Some email software may reformat the report only when viewed within the email message, causing the font and text size to change. Simply save the attachment, then open the saved file to view the results with the original formatting. There is no advantage to using the "PDF" format, but if you would like a PDF version, you may "print to PDF" after saving and opening the attached ".html" file (you may need Adobe Acrobat). You can also open the file in Microsoft Word however this will also alter the formatting. To restore the formatting in Word you will need to select all the text, change the font to "Courier New", set the page layout to landscape, and perhaps decrease the font size.

How are surveys for lectures with multiple recitations or labs handled?

Partly this depends on how your department lists the course in the Schedule of Classes. If the lecture and labs or recitations are listed separately (e.g. the lecture is listed as "102:01" and the labs are listed as "103:01", "103:02" etc.), we will create a survey for each section independently as with any other course. If the lecture is made up of multiple sections but does not have it's own course number (e.g., "101:01", "101:02" and "101:03" all meet together in a lecture hall one day a week), we will create a survey for each of the individual sections (one for each TA), plus a survey that combines all of the sections together (for the lecturer). If we do not know the names of the TAs we will contact you or the department administrator.

If you teach both the lecture and a recitation for the same course you will typically have two surveys so you can gather feedback about the lecture and the recitation separately. However we will need to alter the survey titles so the students can tell the surveys apart. Please contact us to request the change.

How are courses with more than one lecturer handled?

Will will create a separate survey for each lecturer, provided we know the lecturers names. We collect instructor's names from the Schedule of Classes and from department administrators at the start of the semester. You can specify dates to run the surveys, so if you would like to run each survey at different points in the term for the different lecturers, please contact us or ask your department to enter the dates when updating the survey information, at least two weeks before the survey should run.

Will this change the faculty rating? Will students who do not attend class but respond to the survey bias the outcome?

In our studies to date, changes in the average rating are not significant but we are continuing to collect data to determine if the online system affects the ratings. On average, individual ratings varied by ±0.48 points between Fall 2007 paper surveys and Fall 2008 online surveys. For comparison, individual ratings varied by an average of ±0.40 points between Fall 2006 and Fall 2007, both semesters using paper surveys. Comparisons were limited to instructors teaching the same course for more than one semester (i.e., we did not compare ratings for the same instructor teaching different courses, nor did we compare ratings for the same course taught by different instructors).

Will the number of students who respond drop?

The response rate (the number of students who filled out the survey divided by the enrollment) for the online surveys for each semester ranges on average between 50% to 65%. This represents a drop for some departments, and an increase for others. We are closely tracking response rates as we implement our online ratings, and slowly expanding the use of online surveys so we can make adjustments. Based on evidence from other universities, we expect to see an initial drop in the overall number of students replying to the surveys followed by a gradual increase as students and instructors become more familiar with the system. More importantly, the evidence suggests that the change in response rate does not significantly affect the average ratings for individual instructors or departments as a whole.
Response rates to individual surveys may either decrease or rise depending on how the instructor communicates the details of the survey with the students, and is affected by factors such as class size, attendance policies, and mode of instruction (lecture versus lab, etc.).

What can I do to get more students to take the survey?

Faculty and instructors are essential to ensuring that the students respond to the survey. Above all else, communicate with your students regarding the importance of the survey to improve your own teaching, as well as the importance to the university as a whole. Consider taking the following actions:

• While the survey is running, direct your students to http://sakai.rutgers.edu and tell them to click "All Surveys" after logging in.
• Do not rely on our email reminders - students may not read the email, and we cannot send email to students who do not provide an accurate email address to the university. Talk about the survey in class.
• Include a statement on your syllabus that you expect all students to complete the SIRS survey.
• Use informal, midcourse surveys throughout the term.
• When the survey begins, take some class time to discuss the importance of the survey.
• Give the students personal examples of how you have used prior surveys to improve your teaching.
• Inform the students that the surveys are used by the University in promotion, tenure, and reappointment decisions.
• Assure students that their comments and responses will always remain anonymous.
• Invite students to view survey data from previous semesters at http://sirs.rutgers.edu/
• Read more about student participation

How do you enforce student responses?

The survey is voluntary. The system notifies students by email of the availability of the survey, students who do not reply to the survey will receive repeated reminders until they respond or until the survey ends. Additional methods of enforcement cannot be implemented until the university community has an opportunity to discuss the implications and practicality.

If the response rate is very low, can one student unfairly impact my ratings?

Relative to the enrollment in the course, the impact of an outlier may be exaggerated by a low response rate. Although possibly disproportionate, these responses do reflect some student opinions and care must be taken when interpreting the survey. Multiple surveys across courses and semesters should be considered together, and outliers should be recognized as such. Please refer to the guidelines for interpreting the Student Instructional Rating Survey. SIRS is not intended to be the sole determiner in the assessment of teaching; other evidence of teaching ability can and should be included to offset the impact of an outlier.

If the outlier is due to student error (e.g., filling out the wrong survey, reversing the scale), please refer to our policy for requesting corrections.

A student made a mistake on the survey form, can it be corrected?

We do not interpret the students' responses on the survey, and we cannot examine student responses while the survey is still running. Any instructors who feel that a student has incorrectly submitted comments for another instructor or reversed the answer scale should communicate this to his or her department chair. The department chair should request in writing that the survey responses be reviewed and reprocessed.

In many cases students report that they they have made a mistake, but they merely misremember the details of filling out the survey. Requests for corrections should only be made after reviewing the survey results and ascertaining that a mistake does in fact exist and impacts the outcome. All requests for the department must be submitted together because each correction may affect the entire department mean.

Why do students need to log in? Does this violate their anonymity?

Student log-in information is used only to determine which surveys a student can take, and to prevent the students from responding more than once to the same survey. The survey software never reveals the students' identities, and all reports generated by CTAAR only include anonymous, aggregate data. See the privacy policy for more information. It is important that you communicate to your students that you will only see anonymous data, and only after final grades have been submitted.

Can I get a list of students who have not responded to the survey?

No, the system is designed to protect student identities and does not report who did or did not respond to the survey.

Can students who withdraw from the course take the survey?

No. The survey system only allows students who are currently enrolled in the course to take the survey. Roster information is updated daily.  

Why is my course enrollment in your email wrong?

While the survey is running, we send periodic updates to instructors to let them know the current response rate. We only update the enrollment numbers in this email at certain points in the semester, so at times the enrollment number in the email will be slightly out of date. This does not affect the survey since the survey runs from the student perspective. When a student logs into the system, it checks to see if that individual student is registered for the course - the total enrollment is inconsequential for making that determination. We update the enrollment numbers before issuing the final report, but if you find a discrepancy in the final report please let us know and we will correct it.

How are the results used?

University-wide, the survey results are used as part of the faculty promotion and tenure review process. While the use of the survey data varies within individual academic units, it is often used as part of a review process for improving the curriculum, implementing changes to teaching strategies, reappointment review for part-time lecturers and teaching assistants. Many faculty and instructors use the survey data, in particular the comments, to assess and improve their own teaching methods.

Who gets to to see the results?

The anonymous summary statistic sheets for faculty and part-time lecturers are available to the entire university community at http://sirs.rutgers.edu/ beginning with data from 2004, Fall and Spring semesters only. Older data is available on CD-ROM at the University Libraries. Data for Teaching Assistants is no longer published because of the requirements of the Family Educational Right to Privacy Act (FERPA). Student comments are not published.

CTAAR distributes all the reports, including the data for teaching assistants and the student comments, directly to the academic departments and to the individual instructors shortly after the grading period ends. By request, CTAAR also provides the raw, numerical student response data to departments that want to run their own statistical analysis.

Who gets to see the comments?

CTAAR sends the comments directly to the instructors and to the academic departments. The comments are completely anonymous and grouped by question. Comments are not published.