EFFECTIVE FEEDBACK TO THE INSTRUCTOR FROM ONLINE HOMEWORK

Gerd Kortemeyer, Matthew Hall, Joyce Parker, Behrouz Minaei-Bidgoli, Guy Albertelli II, Wolfgang Bauer, and Edwin Kashy
Michigan State University

ABSTRACT

The paper describes different feedback mechanisms available to instructors during the deployment of online formative assessment exercises.

KEYWORDS

Online Homework, Online Formative Assessment

I. Introduction

Technology has enabled instructors to create and distribute a wide variety of educational materials efficiently. These include numerous types of formative conceptual and algorithmic exercises for which prompt feedback and assistance can be provided to students. While several meta-analyses of the effects of assessment with immediate feedback to the student on learning are positive [1, 2], the range of effect size is considerable [3], and can even be negative [2–6]. Even within our own model systems CAPA, LectureOnline, and LON-CAPA, when used just for homework, a range of partly contradictory observations were made [7, 8]. There will not be a general answer to the question of whether or not systems like LON-CAPA are beneficial—after all, they are just tools, not a curriculum. Instead, effectiveness will depend on how they are used, and with which material. There is no doubt however that timely feedback to the instructor, as discussed in this paper, is crucial for ensuring effective use—both during selection and deployment of online educational materials.

II. THE TOOL

LON-CAPA, (The LearningOnlineNetwork with a Computer-Assisted Personalized Approach) [10], while similar to many others in most aspects, differs in three important ways:

• The first is its capability to randomize problems, both algorithmic numerical exercises as well as problems that are qualitative and conceptual, so that numbers, options, images, graphs, formulas, labels, etc., differ from student to student [11]. The students can thus (and are encouraged to) discuss the assignments, but cannot simply exchange answers.
• The second is in the tools provided that allow instructors to collaborate in the creation and sharing of content in a fast and efficient manner, both within and across institutions, thus implementing the initial goals of the WWW [12]. The majority of course management systems are built around the course as the main entity, and learning content is then uploaded to the courses. At the end of the semester, most systems allow export of the content to an instructor’s personal computer, and then require re-uploading in another semester. Within LON-CAPA, content is stored independently of a specific course in a shared cross-institutional content pool.
• The third is its one-source multiple target capabilities, that is, its ability to automatically transform one educational resource, for example a numerical or conceptual homework question, into a format suitable for multiple uses: the same source code, which is used to present problems for online homework, can also generate them for an online examination, or for a printed version suitable for a proctored bubble sheet examination which is later machine scored [13].

III. AUTOMATED FEEDBACK MECHANISMS

The amount of data gathered from large enrollment courses (200–400 students) with over 200 randomizing homework problems, each of them allowing multiple attempts, can be overwhelming. Figure 1 shows just a small excerpt of the homework performance in an introductory physics course, students in the rows, problems in the columns, each character representing one online homework problem for one student. A number shown is the number of attempts it took that particular student to get that particular problem correct— “*” means more than nine attempts; “.” denotes an unsolved problem; blank denotes an un-attempted problem. This view is particularly useful before the problem deadline, where columns with a large number of dots or blank spaces indicate problems that the students have difficulties with.

Figure 1: A Small Excerpt of the Performance Overview for a Small Introductory Physics Class

An important task of the feedback tools for the instructor is to help identify the source of difficulties and the misconceptions students have about a topic. There are basically three ways to look at such homework data: by student, by problem, or cross-cutting. For a per-student view, each of the items in the table in Figure 1 is clickable and shows both the students’ version of the problem (since each is different), and their previous attempts. Figure 2 is an example of this view, and indicates that in the presence of a medium between the charges, the student was convinced that the force would increase, but also that this statement was the one he was most unsure about: His first answer was that the force would double; no additional feedback except “incorrect” was provided by the system. In his next attempt, he changed his answer on only this one statement (suggesting that he was convinced of his other answers) to “four times the force”—however, only ten seconds passed between the attempts, showing that he was merely guessing by which factor the force increased.

Figure 2: Student-centered View of a Problem

The per-problem view in Figure 3 shows which statements were answered correctly course-wide on the first and on the second attempt, respectively, the graphs on the right show which other options the students chose if the statement was answered incorrectly. Clearly, students have the most difficulty with the concept of how a medium acts between charges, with the absolute majority believing the force would increase, and about 20% of the students believing that the medium has no influence. The concept should be dealt with again in class.

In the analysis illustrated in Figure 3, a simple item-analysis on statements was performed, with only the added difficulty of keeping track of the randomized order in which these statements appeared to individual students. A more sophisticated analysis involves keeping track of the concepts each statement addresses, especially if there is more than one statement addressing the same concept, and different students see different versions of it. To this end, internally, the statements can be grouped into six so-called “concept groups,” each focusing on a particular physics aspect of the problem. Every student gets one statement (with the correct labels filled in) from each one of these concept groups. The item analysis on the result in this mode is done by concept group, not by statement, and can thus be carried out independently of the randomization.

The simplest function of cross-cutting statistics tools in the system is to quickly identify areas of student difficulties. This is done by looking at the number of submissions students require in reaching a correct answer, and is especially useful early after an assignment is given. A high degree of failure indicates the need for more discussion of the topic before the due date, especially since early responders are often the more dedicated and capable students in a course. Figure 4 shows a plot of the ratio of number of submissions to number of correct responses for 17 problems, from a weekly assignment five days before it was due. About 15% of the 400 students in an introductory physics course had submitted part or most of their assignment.

Figure 3: Compiled Student Responses to a Problem

The data of Figure 4 are also available as a table, which in addition lists the number of students who have submissions on each problem. Figure 4 shows that five of the questions are rather challenging, each requiring more than 4 submissions per success on average (for example, problem 1 requires a double integral in polar coordinates to calculate a center of mass). Note that an error in the unit of the answer or in the formatting of an answer is not counted as a submission—in those instances, students re-enter their data with proper format and units, a skill that students soon acquire without penalty.

Figure 4: One Early Measure of a Degree of Difficulty

Figure 5: Online Student Discussions on Two Problems, One Numeric and One Conceptual in Nature

IV. FREE-FORM FEEDBACK

Within LON-CAPA, every online resource in a course is automatically associated with a threaded online discussion, which is attached to the bottom of the page. Students and instructors can post anonymously, or with names or screen names. Students are extremely vocal online, even though they are made aware that instructors are always able to see their full names, independent of posting mode. Figure 5 shows the online student discussion associated with two problems which address the same physics, but are different in nature: the left problem is numerical in nature, the right one conceptual. Due to the randomization of the questions, student cannot simply exchange answers with each other, and are forced to discuss the questions on a level which is more insightful to the instructor. Reading at least some of the online discussions before class gives instructors some insight into the general climate in the course, as well as student areas of difficulty.

V. EVALUATIVE FEEDBACK FOR MATERIALS SELECTION

Feedback to the instructor should begin during the resource selection process, particularly when considering material from other authors. The LON-CAPA shared resource pool currently spans over 20 universities and colleges, as well as over 20 high and middle schools and several publishing companies.

As the resource pool grows, selecting an appropriate content resource becomes an increasingly challenging task. In addition to the “Browse” view of the resource pool, instructors can search the cataloguing information. LON-CAPA has two categories of cataloguing or metadata (“data about data”) mechanisms: static metadata provided by the authors, such as title, subject, keywords, etc, and dynamics metadata, gathered by the system based on the use of the resource.

The latter provides information similar to amazon.com’s dynamic metadata; that is, it provides information in which context a resource has been used by other instructors, see Figure 6.

Figure 6: Dynamic Metadata, Context

Besides providing contextual information, the dynamic metadata provides what amounts to a peer-review mechanism: resources that have been selected by a number of instructors presumably are the ones having passed careful consideration by a number of peers.

Figure 7: Left Side: User Interface Used by LON-CAPA to Collect User Evaluation Data; Right Side: Excerpt from the Summary Information Metadata Presented to the Resource Creator

Any instance in which a given resource is used in an exam setting thus collects information on degree of difficulty and discrimination. This information can be archived and used to create random tests that are generated from a large bank of testing resources. The computer can then create tests that do not rely on the selection of the instructor, but instead allow for a comparison relative to an objective standard. This is important particularly when one allows for creation of individual tests for students, in which the questions are allowed to vary from student to student.

VI. CONCLUSIONS

Technology does indeed provide means to get considerable feedback on many aspects of teaching and learning. To make good use of that feedback is a far greater challenge. We have been using LON-CAPA for both formative and summative assessment.

The LON-CAPA course management software has reached a state of maturity; its resource pool has reached a size that now allows novel approaches to old problems. With over 60,000 individual resources, with many tens of thousands of students enrolled each semester at approximately 50 institutions, with automated metadata collection, and with resource sharing across the LON-CAPA member network we have entered a new phase in the use of educational technology. It has now become possible to think about multiple content representations to provide a more customized accommodation of individual learning styles. In addition, we are now in the position to establish more objective measurement tools for learning outcomes that utilize large test banks of individual test items for which standardized statistical information has been collected across many educational settings.

VII. REFERENCES

1. Azevedo, R. and R. M. Bernard. A meta-analysis of the effects of feedback in computer-based instruction. Journal of Educational Computing Research 13 (2): 111–127, 1995.
2. Mason, B. and R. Bruning. Providing feedback in computer-based instruction: What the research tells us, 2003. http://dwb.unl.edu/Edit/MB/MasonBruning.html.
3. Bonham, S., R. Beichner and D. L. Deardorff. Online homework: Does it make a difference? Phys. Teach. 39 (5): 293, 2001.
4. Bransford, J. D., A. L. Brown and R. R. Cocking. (Eds). How People Learn: Brain, Mind, Experience, and School. Washington, D.C.: National Academy Press, 2000.
5. Kluger, A. N. and A. DeNisi. The effects of feedback interventions on performance: Historical review, a meta-analysis and a preliminary feedback intervention theory. Psyc. Bull. 119: 254–284, 1996.
6. Kluger, A. N. and DeNisi, A. Feedback interventions: Toward the understanding of a double-edged sword. Cur. Dir. Psyc. Sci. 7: 67, 1998.
7. Pascarella, A. M. The influence of web-based homework on quantitative problem-solving in a university physics class. Proc. NARST Annual Meeting, 2004.
8. Kotas, P. Homework behavior in an introductory physics course. Masters Thesis (Physics), Central Michigan University, 2000.
9. Albertelli, G., II., B. Minaei-Bidgoli, W. F. Punch, G. Kortemeyer and E. Kashy. Concept feedback in computer-graded assignments. Proc. IEEE Frontiers in Education 32: 2002.
10. Kortemeyer, G., W. Bauer, D. A. Kashy, E. Kashy and C. Speier. The LearningOnline Network with CAPA Initiative. Proc. IEEE Frontiers in Education 31: 2001. http://www.lon-capa.org/.
11. Kashy, E., S. J. Gaff, N. Pawley, W. L. Stretch, S. Wolfe, D. J. Morrissey and Y. Tsai. Conceptual questions in computer-assisted assignments. Am. J. Phys. 63 (11): 1000, 1995.
12. Proposal for a HyperText Project: http://www.w3.org/History/19921103-hypertext/hypertext/WWW/Proposal.html.
13. Multiuser considerations: http://www.w3.org/History/19921103-hypertext/hypertext/WWW/DesignIssues/Multiuser.html.
14. Albertelli, G., II., G. Kortemeyer, A. Sakharuk and E. Kashy. Personalized examinations in large on-campus classes. Proc. IEEE Frontiers in Education 33: 2003.
15. Kortemeyer, G. and W. Bauer. Multimedia collaborative content creation (mc 3)—The MSU LectureOnline System. J. Eng. Educ. 88 (4): 405, 1999.
16. Kashy, D. A., G. Albertelli, II., E. Kashy and M. Thoennessen. Teaching with ALN technology: Benefits and costs. J. Eng. Educ. 90 (4): 499, 2001.
17. Minaei-Bidgoli, B., D. A. Kashy, G. Kortemeyer and W. F. Punch. Predicting student performance: An application of data mining methods with an educational web-based system, Proc. Frontiers In Education 33: 2003.
18. Thoennessen, M., E. Kashy, Y. Tsai and N. Davis. Impact of asynchronous learning networks in large lecture classes. Group Discussion and Negotiation 8: 371, 1999. And private communication.

VIII. ACKNOWLEDGMENT

We thank the National Science Foundation (ITR-0085921; CCLI-ASA-0243126) for the support of this project. Any opinions, findings, and conclusions or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. Support in earlier years was also received from the Alfred P Sloan and from the Andrew W Mellon Foundations. We are grateful to our own institution, Michigan State University and to its administrators for over a decade of encouragement and support. Discussions with our colleagues D.A. Kashy, M. Thoennessen, F. Berryman, and S. Raeburn were particularly useful.

IX. ABOUT THE AUTHORS

Gerd Kortemeyer is a professor of Physics Education in the Lyman Briggs School of Science and the Division of Science and Mathematics Education at Michigan State University, and the principal investigator of the LON-CAPA project.
Contact: Lyman-Briggs School of Science, Michigan State University, East Lansing, MI 48825, korte@lon-capa.org

Matthew Hall is a specialist in the Division of Science and Mathematics Education at Michigan State University, working among other projects on the statistical functionality of the LON-CAPA software project.
Contact: Division of Science and Mathematics Education, Michigan State University, East Lansing, MI 48824, hallmat3@msu.edu

Joyce Parker is a professor of Teacher Education at Michigan State University.
Contact: Division of Science and Mathematics Education. Michigan State University, East Lansing, MI 48824, parker13@msu.edu

Behrouz Minaei-Bidgoli recently completed his doctoral studies in Computer Science, working on data-mining mechanisms for online educational systems.
Contact: Division of Science and Mathematics Education, Michigan State University, East Lansing, MI 48824, minaeibi@msu.edu

Guy Albertelli II is a specialist in the Division of Science and Mathematics Education at Michigan State University, and the technical director of the LON-CAPA software project.
Contact: Division of Science and Mathematics Education, Michigan State University, East Lansing, MI 48824, albertel@msu.edu

Wolfgang Bauer is a professor of Physics at Michigan State University.
Contact: Department of Physics and Astronomy, Michigan State University, East Lansing, MI 48823, bauer@pa.msu.edu

Edwin Kashy is a professor of Physics at Michigan State University.
Contact: National Superconducting Cyclotron Laboratory, Michigan State University, East Lansing, MI 48824, kashy@nscl.msu.edu