The SCALE Efficiency Projects

Download PDF version:

* Sloan Center For Asynchronous Learning Environments, University of Illinois at Urbana-Champaign and
**Office of Instructional Resources, University of Illinois at Urbana-Champaign

Correspondence on this paper should be sent to:
Lanny Arvan,
Director, SCALE
1406 West Green Street
Urbana, Illinois 61801-2291
Phone: 217-333-7054
Email: l-arvan@uiuc.edu

ABSTRACT
This paper presents evidence from nine "Efficiency Projects" that were SCALE’s focus in the 1997-98 academic year. The Efficiency Projects were specifically aimed at using ALN to achieve higher student/faculty ratios, without sacrificing instructional quality. The study concentrates on data amassed for the fall 1997 semester. Evidence was collected on the cost side, for ALN development and delivery, and the performance/attitude side, from both student and faculty perspectives. The study supports the view that when a sensible pedagogic approach is embraced that affords the students with avenues to communicate about their learning, ALN can produce real efficiency gains in courses without sacrificing the quality of instruction.

KEY WORDS
Efficiency Projects
Quality of learning

I. INTRODUCTION

This paper presents evidence from nine "Efficiency Projects" that were SCALE’s focus in the 1997-98 academic year. The study concentrates on data amassed for the fall 1997 semester. Evidence was collected on the cost side, for ALN development and delivery, and the performance/attitude side, from both student and faculty perspectives.

The Efficiency Projects were specifically aimed at using ALN to achieve higher student/faculty ratios, without sacrificing instructional quality. The higher student/faculty ratios occurred in some cases by increasing the number of students taught, in other cases by reducing the size of the instructional staff. One common feature shared by these projects is class size. All were in large undergraduate classes. Another common feature was the reliance to some degree on automatic, Web-based grading software. Yet there were substantial differences across these projects.

The courses were in Chemistry (General Chemistry and Advanced Organic Chemistry for Biology majors), Circuit Analysis (Introductory), Differential Equations, Economics (Microeconomics Principles and Intermediate Microeconomics), Microbiology (Introductory), Spanish (Intermediate Grammar), and Statistics (Introductory for non-technically oriented students). The Principal Investigators (PIs) on these projects differed in their experience in teaching with ALN. Some were among the original SCALE grantees (and among these some had significant relevant prior experience). Others had less experience. Indeed, the Spanish PI was a relative computer novice and was using ALN for the first time. Some of the courses relied heavily on graduate assistants. Others used undergraduate peer tutors. Some of the courses used asynchronous conferencing primarily as a means for providing help to students. Others used synchronous text-based chat for this purpose and used asynchronous conferencing as a means for students to do written work online. In some cases the developers of the online materials were also providing the face-to-face delivery of instruction. In other cases the authoring, presentation, and coaching functions were separated across individuals. Some courses retained the traditional lecture intact. Others substantially reduced face-to-face contact hours.

With all this variation, it is probably better for the reader to interpret the results as a collection of case studies rather than as a cross section of evidence on ALN, viewed as a precisely pinpointed approach to online instruction. We do try to draw some general conclusions where we think it appropriate, both about ALN instruction in a large class setting and about using ALN to attain efficiency ends.

A critical issue is the extent to which the findings presented in this paper are replicable. A big part of the replicability question is attributing the results to ALN or to the PIs themselves. In the vast majority of the Efficiency Projects, the PIs were early adopters with a great deal of enthusiasm for online teaching. Whether the results translate to mainstream faculty remains an open question. Another factor is the general computing environment. Computer technology permeates daily life at the University of Illinois at Urbana-Champaign (UIUC) and thus it might be equally important to ask whether the results would hold at a campus where computing is less firmly imprinted into the culture. Yet another significant issue is the extent of up-front learning needed, for the support organization as well as for the instructors, as a precursor to any program aimed at utilizing ALN for efficiency ends. Whether others would require the two-year lead for general ALN development, as did SCALE, is an open question. To give the reader more of context for these and related issues, we briefly review how SCALE has grown, from its origins to the present.

A. Brief SCALE History
SCALE was formed in spring 1995 with a $2.1 million grant from the Alfred P. Sloan foundation and a generous match from the University of Illinois at Urbana-Champaign. The grant covered a three-year period that ended after the spring 1998 semester. The goal was to bring 15 ALN courses online a year. In fact, in the 1997-98 academic year there were approximately 80 courses per semester supported by SCALE. These courses enrolled about 8000 students per semester. The Efficiency Projects represent only a small number of the courses supported by SCALE, but account for about half the enrollments.

SCALE’s primary mission was to support ALN course development in an on-campus setting. Initially, Sloan had set four targets for this on-campus ALN to achieve. These were to improve retention, to decrease time to degree, to demonstrate verifiable increases in student learning, and to lower the cost of instruction. Over time, these targets have been modified, based on the experience with ALN and its implementation on the UIUC campus.

While there were some Web-based activities in SCALE courses at the outset, the bulk of the initial ALN work entailed use of asynchronous conferencing. At the start there were two asynchronous products supported, the now defunct PacerForum and the still popular FirstClass. Both of these are client-server software. The user must have the client installed on the desktop computer where the user is working. The client allows the user to access the server over the network. It is the server where all the information is stored. In 1996-97 SCALE dropped support of PacerForum but began to support Web-based conferencing, where the Web browser (initially Netscape Navigator, later also Microsoft Internet Explorer) serves as the client. Web-based conferencing allows for a more seamless movement between course-related Web materials and the conferencing environment, a distinct advantage. In 1996-97 SCALE supported the product WebNotes. In 1997-98, SCALE switched to WebBoard and will continue to support use of this product in the upcoming year. In spite of the increasing popularity of the Web, many SCALE faculty continue to use FirstClass. The reasons for this loyalty to FirstClass are many and varied: 1) they have had good success with it in the past, 2) it is what they know and don’t want to have learn something else, and 3) they view the current Web-based alternative as inferior.

Increasingly, SCALE faculty have come to put their ALN materials on the Web. In addition to the standard syllabus and lecture notes, simulations (primarily in science and engineering courses) that had previously been delivered via dedicated client software were moved to the Web. Moreover, after the pioneering projects in 1995-96, much effort was put into authoring questions for CyberProf [1] and Mallard [2], products developed at UIUC for allowing students to self-teach via intelligent assessment of short-answer questions delivered through the Web. Without a doubt, Web delivery became an increasingly important component of the teaching strategy in SCALE-supported courses.

Apart from an evolution in the technology, there was also a transformation in the pedagogy. Over time, the original grantees came to increasingly trust their ALN teaching approach. ALN became less of an experiment and more an established style, with a heavy emphasis on the assignments that students were to complete. This regularizing of ALN allowed SCALE to provide its support in a consistent, well-prescribed manner. It also allowed grantees who got started in year two of the grant and even more so in year three to get current with ALN teaching in an accelerated manner. They had to learn the ALN software, to be sure, but there was less need to tinker with the pedagogy and wonder if it would work.

There has been an independent evaluation team from the start of the SCALE project. That team is headed by John C. Ory and includes Cheryl Bullock and Kristine Burnaska. They produced semester-by-semester evaluations starting in fall 1995 and culminating in spring 1997 [3],[4],[5],[6]. Matthew Hanson joined the evaluation team in summer 1997, to work exclusively on the Efficiency Projects. While the evaluation team has been in frequent contact with SCALE administration and, in particular, the evaluation strategy of the Efficiency Projects was discussed extensively, the actual data collection effort has been the sole province of the evaluation team. This independence helped to minimize the chance of misrepresentation of the findings and to reduce the awkwardness involved in the data collection, particularly in those cases when students or faculty reported that things weren’t going so well.

B. Sources of Productivity Improvement
Studies of computer technology use aimed at increasing instructional productivity are quite rare. The Rensselaer Studio Courses offer one example [7]. Some work done at Michigan State University by Ed Kashy, Michael Thoennessen, et. al., [8], [9], is closer in spirit to the present study. That is essentially the entire list. That such work is indeed rare is confirmed by Polley Ann McClure [10]. "While there are some cases in which we can document improved educational output as the result of technology intervention, in a brief survey of the literature, I could find no studies documenting improved educational output per unit cost. The educational gains have been at huge cost, in terms of investment in both equipment and software, but more significantly, in faculty and support staff time." Similarly, David Noble [11], a notable opponent of online education, cites the work of Kenneth Green [12] when arguing, "Recent surveys of the instructional use of information technology in higher education clearly indicate that there have been no significant gains in either productivity improvement or pedagogical enhancement."

That such documentation is so rare suggests two potential explanations: (1) it is not possible to generate productivity increases with computer technology, and (2) it is possible, but the incentives are not right for us to witness them. Robert Koob [13] makes a convincing case for the second hypothesis. Yet affirming that incentives are weak does not in itself prove that computer technology can generate instructional productivity gains. More direct evidence is needed and that provides the raison d’être for our study.

Strategic thinking about how instructional technology should be used for advancing productivity ends has clearly outstripped the empirical work in this area. Much of this strategic thinking has come out of Educom’s National Learning Infrastructure Initiative (NLII). Examples include the papers by Carolyn Twigg [14], [15], D. Bruce Johnstone [16], and William F. Massy and Robert Zemsky [17]. The ideas behind the SCALE Efficiency projects have been influenced by this work. Yet it should be understood that making these ideas operational requires compromise, in both the implementation and in the measurement. It is our hope that this paper gives the reader some insight into the type of compromises that are needed to get actual productivity projects underway and the variety of measurement problems that arise as a consequence.

Furthermore, there is a fundamental conceptual point that should be considered where the NLII philosophy departs from the ALN philosophy. The basis of the NLII thinking is that educational technology is capital and that any productivity gains must come as capital input substitutes for labor input. While this capital for labor approach is not entirely absent in the ALN approach, it is not the whole story. With ALN, much of the productivity increase comes from labor-for-labor substitution – inexpensive student labor for expensive faculty labor. (The TLT Affiliate of AAHE headed by Steven Gilbert and Stephen Ehrmann, [18], vigorously argues for more of this type of labor-for-labor substitution, but to date they have concentrated their focus on the instructional technology support arena rather than in the online classroom itself.) Viewing the students’ time as a productive input, as suggested by Lanny Arvan [19], some of this productivity gain arises from peer-to-peer communication. (Note that we don’t cost-out this student time in the measurement component of this paper, however, some demographic evidence suggests how such a costing out should be done[20].) Additional productivity gains emerge from student interaction with peer tutors who receive remuneration for the help they provide. In the ALN approach, it is critical to view networked computers as, in part, communication tools. This allows the ALN approach to make the instruction more personal while simultaneously increasing productivity. At least, that is the ideal.

There has been a change of thinking within SCALE administration about how to deliver on the Sloan objectives. During the first year of the SCALE project, there was an expectation that the desired efficiency outcomes would come as a byproduct of ALN implementation. This due to the enhanced peer-to-peer interaction and the avoidance of wasteful duplication of effort through the instructor answering common student questions once, via posts to a public class conference. It was also expected that efficiency gains could be had in all ALN courses. In fact, most SCALE-affiliated faculty reported increased time involved in instruction as a long-term proposition, because of the increased contact with the students online. Subsequently, some of these instructors have modified their views about the need to be online so frequently for their students to have good access. But the view remains that ALN teaching is arduous. Thus, it became apparent that the byproduct approach would not achieve the desired results. Moreover, it also became clear that efficiency outcomes would be difficult or impossible to attain in small ALN classes. There were two reasons for this that perhaps should have been obvious at the outset of the project but were not. First, if there was substantial up-front development in a small class, such development could not be amortized over a large number of students. Second, in a small class there is very limited opportunity to exploit labor-for-labor substitution. When SCALE administration ultimately contracted for the efficiency projects [21], SCALE targeted large classes only.

Another consequence of abandoning the byproduct approach was the need to put in specific incentives to produce efficiency outcomes. After the first year of SCALE, grants to PIs were reduced so that more projects could receive funding. This trend was reversed for some of the Efficiency Projects, which received grants that were as large as those grants given in the first year. Moreover, SCALE was able to obtain assurances from the UIUC administration that any savings produced could be retained within the department where those savings were generated.

C. Further Caveats
In the main, the SCALE Efficiency projects represent mature ALN development in large classes where the ALN has now been focused on efficiency ends. There are many other ALN courses that SCALE currently supports where no attempt is being made to produce efficiency outcomes. Among these are some large classes. Thus, we are not arguing that large size per se makes a class a good candidate for an efficiency project. For example, SCALE supports an introductory comparative literature course that enrolls about 250 students a semester. The course is taught with a lecture once a week. There are also small sections run by graduate assistants under the supervision of the faculty member who delivers the lecture. The course is writing-intensive and satisfies the campus Composition II requirement. In spite of the course size, the possibility for capital substitution is limited here. Competent evaluators must assess the students’ written work. Computer assessment of the writing is not possible, because the assessment is so contextually based. It can’t be done via a search for key words. This requirement of competent assessment also limits the possibility of labor-for-labor substitution in this course. We think that ALN is improving learning, but we have no way to quantify the learning, so this course is not one of our Efficiency Projects. There are also SCALE-supported courses currently taught in such an inexpensive manner – large lecture with few if any graduate assistants to support the course – that it seems foolhardy to try to further reduce the cost of instruction.

We are also not arguing that the SCALE approach can work everywhere, technological considerations aside. The reliance on peer tutors, in particular, requires highly able students who can serve in this capacity and feel they are doing something socially beneficial in the process. The SCALE approach likely can work well at other institutions in the Big Ten and at other similarly regarded public campuses. To what other institutions the approach can be profitably extended is an open question.

One further point bears mention here. There has been a negative reaction to using educational technology for efficiency ends, emerging from various pockets of concerned faculty [22], [23]. Much of this reaction relates to the effect on faculty employment. The capital substitution argument would seem to suggest a need for fewer faculty. Certainly there is a fear that this will be the case. Reducing faculty employment is viewed as ‘bad’ in many quarters. It is our view that on the UIUC campus the SCALE Efficiency Projects will have little or no impact on faculty employment, though we do anticipate a big impact from these projects overall. It is graduate student employment that will be affected the most dramatically, if the SCALE Efficiency Projects become more widespread on campus. The reason for this is simple. In the vast majority of the courses that SCALE has been targeting, graduate students do the bulk of the teaching. The course coordination function remains in the hands of a faculty member, even with ALN. The upside of this is to reduce the pressure on graduate student enrollment to staff large introductory undergraduate courses. This should allow graduate student enrollment to better track the new Ph.D. job market in the individual discipline and to better match the quality of the particular degree program. Furthermore, to the extent that the changes in graduate student enrollment can be made without disenfranchising students who are currently enrolled, simply by adjusting the size of entering cohorts, it is not obvious that there is a downside to this approach.

II. A SIMPLE PRODUCTIVITY MODEL

All the Efficiency Projects entailed at least some up-front development. This development can be thought of as part learning -- both attaining a comfort level with the software and formulating a successful pedagogic strategy -- and part authoring/online publishing. In collecting the data, development costs were grouped into three categories. First, there are faculty costs (e.g., course buyouts and summer support). Second, there are programming costs (e.g., hourly wages or assistantship support of student programmers). Last, there are equipment costs (e.g., the cost of desktop computers, the pro rata share of server and license costs allocated to the particular project, and the cost of software).

Subsequent to the up-front development, each Efficiency Project produced some recurrent benefit. In courses where the overall enrollment remained unchanged, this benefit can be envisioned as a reduction in operating costs, measured on a semester-by-semester basis. Operating costs also include three components. First, there is the pro rata share of the faculty member’s salary (plus benefits) allocated to teaching the ALN course. Second, there is the cost of other course personnel, either graduate students on assistantship or student hourlies. Last, there is the pro rata share of common costs, particularly SCALE support staff.

It is helpful to think of the total benefit per semester as the product of two factors: 1) the reduction in operating cost per student and 2) the number of students in the class. This is the entire benefit when overall course enrollment remains unchanged. The benefit calculation is a bit more complicated when ALN allows for an expansion of overall enrollment. Enrollment expansion can occur only if there had been unsatisfied demand for the course, in which case the benefit on the cost side itself has two components and there is a benefit on the demand side as well. These three components are the operating cost reduction on the original class size, the imputed operating cost reduction on the increase in class size, and the benefit that accrues to those students who would have been rationed out had course capacity not expanded. We did not try to measure this third component. We simply note that measurement of the first two components understates the recurrent benefit.

A goal from the outset was to measure all costs in dollar terms, to best make comparisons between the various cost components. Thus, with the exception of student hourlies, there was no attempt made to characterize the time entailed in doing the work, say for the instructor authoring the on-line materials. The approach we took differs markedly from time and motion studies. Instead of measuring the time input directly, we measured the dollar amount needed to elicit the requisite time input. In most cases we did this by directly measuring outlays, either in SCALE grants to the PIs, or in actual salary numbers. In some cases we had to make imputations for compensation and where we did that we go to some length to describe the calculations. The most contentious of these imputations was determining the share of faculty salary allocated to teaching a course. In most cases we simply assigned 25% of the nine-month salary to the course, because the typical campus-teaching load is two courses per semester in both the fall and the spring.

That we used actual outlays bears mention for several reasons. First, the Efficiency Projects are among a group of successful SCALE projects. A few of the original projects did not succeed. We made no attempt to adjust our development cost measure for the risk of project failure, because there are offsetting biases in measurement. Second, many of the Efficiency Projects were among SCALE’s original projects and have received several rounds of funding. The funding was given out annually and with each grant there was no guarantee of further funding in a subsequent grant. Thus, the measured outlays might differ substantially from the case where a multiyear development cycle was planned from the outset. Third, returning to the question raised in the introduction about whether the results apply to mainstream faculty, there is the issue of how the size of the SCALE grants compare to right level of compensation for the effort; too much, too little, or just right. Last, because others who might embark on such an ALN program now will have the benefit of learning from those who have preceded them, their development costs will likely not be as great as in the case of the SCALE projects.

Where possible, a computation was made to determine the number of semesters it would take to recover the up front development cost. This calculation was done twice, once in the case of no discounting and then again when future benefits were discounted. The effect of discounting is to lengthen the period of cost recovery. We used a rather conservative interest rate, 9%, so that we would have reliable bounds on the period of cost recovery [24].

This cost model clearly oversimplifies matters. Authoring does not occur entirely up front. Teaching with ALN is an iterative process. Modifications are made to the on-line materials based on the actual experience of teaching a class. The philosophy behind the way these authoring costs were allocated in this study is as follows. Authoring that takes place during the semester is treated as operating cost. Authoring that occurred in the summer and for which the PI was compensated via a grant from SCALE is treated as up front development.

III. CATEGORIZING THE EFFICIENCY PROJECTS

In an attempt to make some relevant cross-project comparisons, we categorize the productivity increase by whether the scheduled contact hour is with a teaching assistant or a faculty member. This distinction is relevant for at least two reasons. First, in terms of converting instructor time to dollars, teaching assistants are paid on a more or less uniform basis. There is much more variation in faculty members' salary, due to variation in rank and variation in compensation across disciplines. Moreover, since faculty obligations are a bundle of teaching, research, and service, with the fraction of the obligation somewhat idiosyncratic, it is hard to parse out the teaching component. The precision of the cost estimates should be understood in that light. Second, this distinction, at least on the UIUC campus, represents the extent to which the instructor must adopt the ALN innovation as terms of employment. Departmental standards determine the graduate assistant load associated with a 50% time appointment, a 33% time appointment, etc. Departments can and do change these standards upon occasion, for reasons quite unrelated to adoption of ALN. Moreover, if some courses are taught with ALN and other not, departments can make the standard flexible to accommodate that distinction. Teaching Assistants with the same fractional appointment may teach three ALN sections but only two non-ALN sections. The point is that departments can implement this, as long as in their judgment the burden on the TAs is roughly equal under either approach and matches the fraction of the appointment. In contrast, such differential loads cannot be imposed upon faculty members without prior consult and approval.

When the department that houses the project captures the productivity increase, it occurs either by the instructor teaching more sections or by the instructor teaching more students per section. With the first, students should perceive no difference in quality, as from their perspective the class has not changed. With the second, student perceptions of quality (as well as objective measures of student performance) provide evidence about the consequences on course quality. The SCALE project in introductory Chemistry had the TAs teach more sections. Seven of the other projects achieved the increase in productivity via larger sections. When the instructor captures the productivity increase, the instructor's workload should drop. The SCALE project in Microbiology introduced virtual labs to replace wet labs in some cases. During the weeks when the virtual labs were given, the graduate lab assistants were able to focus their attention on their research rather than their teaching. Though how the productivity increase is captured is an important variable, we chose not to use this dimension to differentiate our projects because there was not enough variation.

All but the project in Mathematics utilized on-line quizzing with automated grading to some degree. Students receive the benefits of immediate feedback and repeated tries at the material. A concomitant benefit is that instructors are relieved from the burden of grading. Yet there is more to a large ALN course than automated grading of assignments. These ALN courses have a substantial component of people-to-people interaction online. Some of this is peer-to-peer interaction. The rest is between student and course staff, much of it through scheduled on-line office hours, which are more extensive than their traditional counterpart. The on-line office hours are frequently in the evening, when they are convenient for the students. Office hour staffing is made affordable by having undergraduate peer-tutors, by reducing the number of class contact hours, or some mixture of the two. Though there were some qualitative differences across projects in how these office hours were conducted, all of the projects relied on the on-line office hours extensively. Again, though automated quizzes and on-line office hours are very important components of our Efficiency Projects, there is not enough variation across projects to focus on these components as a means to categorize the projects.

For this reason, we searched for another dimension along which the SCALE Efficiency Projects are differentiated. We ended up focusing on whether all the assignments are machine graded (short answer), or if there was still some long answer work graded by TAs. In so doing there are several issues we were trying to address. First, automated grading may seem reasonable to some instructors in some disciplines, but may appear inappropriate elsewhere. As it turned out, in several of the projects the two forms of grading were used in mixed mode with the intent of achieving the best that both have to offer. (Here we are focusing on assignments completed out of class. One could also look at this distinction with exams. Interestingly, some SCALE supported courses that have all short answer assignments have long answer exams while other courses with some long answer assignments have short answer exams.) Second, in some of the courses undergraduate peer tutors have also been utilized as graders. This is controversial. On the one hand, by using undergraduate graders there is an inexpensive supply of grading assistance that the instructor can tap from those students who have taken the course previously. On the other hand, some faculty are suspicious that undergraduates do not have the requisite depth of knowledge of the subject to provide good written feedback to the students. Third, it may be that written work done online and the assessment that goes along with it are simply different from the paper analog. In SPAN 210 much of the written feedback was actually given by other students as they made responses to original posts. The graders' job was much more to acknowledge these efforts by the students and much less to participate in the dialog with additional feedback. In ECON 300, the rapid transmission of the work coupled with the large number of graders, allowed for rapid response to the submission. This, in turn, allowed the teams to resubmit homework in response to the comments of the grader, a practice that is exceedingly rare with paper-based homework.

Table 1. Efficiency Matrix

IV. MEASURING QUALITY OF INSTRUCTION

In the contract made with Frank Mayadas, the efficiency project courses were to be redesigned so as to improve the quality of learning or to hold the quality of learning constant. Apparent efficiency gains that resulted in a deterioration of quality were deemed out of bounds.

As we subsequently document, these projects have been successful in lowering expenditure per student. Elsewhere, expenditure per student is itself regarded as a quality indicator, with greater expenditure indicating higher quality. (For example, see Money Magazine’s college ratings [25].) Expenditure per student can be viewed as a measure of input quality. We report on our efforts to measure the effect of ALN on the output quality in several Efficiency Projects.

To assess changes in course quality due to the use of ALN, in the ideal, we would have the same instructor teach both an ALN section and a non-ALN section of the course and administer a common performance standard for both sections. (Jerald Schutte has done a study along these lines [26]. He finds that the on-line approach significantly outperforms the traditional approach. But he is unable to control for his own teaching effort across the two sections.) In addition, students would be randomly assigned to the sections. We could then look at indicators of student performance and of student satisfaction as measures of output quality. Moreover, we would have an appropriate benchmark to which we could compare the ALN approach.

As we described in the introduction, we had to make several compromises in implementing our study. In retrospect, we think an ideal study may be impossible to implement, because of limited resources and the ethical issues such a study raises. The ideal study can be implemented most easily when beliefs about the teaching approach are neutral. The more it is believed that ALN is superior to the traditional approach (or vice versa) the harder it is to implement the study. Instructors do not want to be shown that their teaching is inferior. Students do not want to take the version of the course that will make them less prepared to do well on the exams. And administrators do not want to continue with the traditional approach if ALN appears to afford a productivity advantage. We tried to come as close as possible to the ideal in our investigations of instructional quality. Whenever possible we attempted to collect quality measures that included assessments of student performance. We have more plentiful information about student and instructor attitudes.

V. THE FINDINGS

In this section we proceed through the efficiency matrix presenting brief descriptions of the projects, a summary of the cost information, and the available output/quality information. Rather than go through the efficiency matrix in alphabetical order of the cells, we proceed on the basis of the quality of our evidence.

A.Cell B of the Efficiency Matrix – SPAN 210
In 1996-97, an Italian professor successfully developed an ALN approach for her ITAL 101 and 102 courses. She designed vocabulary/grammar exercises for the students to complete using Mallard as well as writing assignments done using FirstClass. This professor serves as course coordinator while graduate TAs teach independent discussion sections.

SPAN 210, a basic course in Spanish grammar, has a similar structure to ITAL 101 and 102. The idea behind this project was to build on the course development experience of the Italian professor and to thereby use ALN to begin to address the "Spanish Problem" on campus. At UIUC, and at most universities nationwide, the demand for Spanish language courses far exceeds actual enrollment, primarily because the ability to staff these courses is limited. This demand is fueled by the increasing internationalization of our economy. Students who wish to have a minor in international studies need competency in a second language. Spanish is the language of choice. On the UIUC campus, the Spanish problem will be exacerbated by a recently imposed increase in the foreign language requirement. Though much of the demand for Spanish is in the introductory courses, SPAN 210 has also had a chronic excess demand problem. There are students who have wanted to take the course but who have been unable to do so because all the slots were filled.

To initiate this Efficiency Project, the Italian Professor searched the Spanish faculty for a willing participant, ultimately enlisting the SPAN 210 course-coordinator, who was drawn into this out of dissatisfaction with the exercises in the textbook she was using. This search occurred in spring 1997 in response to a call from SCALE administration. At the outset of the ITAL 101-102 project, in summer 1996, it was not envisioned that it would lead to a subsequent SPAN 210 project. For this reason, we are not including the development cost of the Italian project in the cost calculation for SPAN 210. In summer 1997 the Spanish professor and a graduate assistant began developing on-line materials for SPAN 210. In fall 1997 two out of nine regular sections of Span 210 were taught with ALN (utilizing both FirstClass and Mallard). Each ALN section was twice as large as a traditional section. The ALN section met only once a week while the traditional section met 3 times a week. This helped to keep the workload for the instructors uniform across sections. The professor also used ALN in a Discovery section. The campus Discovery program includes a set of courses that are for freshmen only, that are taught by tenured or tenure-track faculty, and that have class size capped at 20 students. All ALN and non-ALN sections used similar exams.

In fall 1997 the use of ALN allowed the department to increase class size from 19 students to approximately 38 students in each of two sections. The department believed that by using ALN to teach all sections of SPAN 210 in the future they would be able to teach approximately twice as many students, without adding personnel. In fact, in spring 1998, all sections of SPAN 210 were taught with ALN and all have experienced a doubling of enrollment relative to historical norms.

It is important to observe that the development period for ALN in Span 210 was only summer and fall 1997, where the Spanish professor authored the on-line material for Mallard. This shorter development cycle can be attributed, in large part, to the Italian professor’s prior development. As it turns out, however, the Spanish professor did not merely translate the questions that had been written in Italian, but rather wrote her own questions that suited her view of how to teach the subject. This wasn’t planned for at the beginning of summer 1997. It turned out that way because the Spanish professor developed an increasing enthusiasm for the enterprise. Nonetheless, we expect to see shorter development cycles in projects that are derivatives of earlier development. If these derivative projects are aimed at efficiency ends, they are likely to produce a dividend quickly.

That is the case for the SPAN 210 project. Total development cost (measured by the size of the SCALE grant) was $15,336, divided roughly evenly between faculty summer support and student programming support. Even with discounting, it is clear the first full-scale ALN offering of this course in spring 1998 produced a cost saving that more than covered this development cost. This project has already produce a dividend. It has also paved the way for a further, more ambitious Efficiency Project in the introductory Spanish sequence.

The SPAN 210 course is the closest we came to conducting the ideal study of output quality. The two ALN sections were compared to two non-ALN sections used as a control group. These four sections had common exams. We have the results from the two midterms. The ALN sections had approximately twice the number of students as the traditional sections, so in comparing the distributions one should focus on the cumulative distribution functions, not on the absolute number within each category. There was also a common attitudinal survey administered and a focus group for each section.

Table 2 shows the results for Midterm 1 and Table 3, the results for Midterm 2 for the ALN and non-ALN sections of Spanish 210. Table 2 shows that the non-ALN section had more students at the extremes of the distribution. This implies the two distributions cannot be ranked via first order stochastic dominance. The non-ALN section had a slightly higher median, in the 91 – 93 range. The ALN section had a median in the 87 – 90 range. Table 3 shows the reverse. The ALN section had more students at the extremes. There were some drops in both sections, more percentage-wise in the non-ALN section. This explains, perhaps, the result at the lower extreme of the distribution. The medians were the same for Midterm 2, in the 87 – 90 range.

Table 2. Comparison of ALN and non-ALN Midterm 1 Results in Spanish 210

Table 3. Comparison of ALN and non-ALN Midterm 2 Results in Spanish 210

We also perform a comparison of means, under the assumption that all observations occur at the midpoint of the cell. For example, all observations in the 97 – 100 range are treated as occurring at 98.5. Letting x denote the ALN outcome and y the non ALN outcome, we report values of the statistic

z_m = (x_m – y_m)/(s_x²/N_x + s_y²/N_y)^1/2

Under the null hypothesis that there is no difference in the means for the two classes, z_m should have a standard normal distribution. For Midterm 1, we see that the non-ALN sections did score significantly higher than the ALN sections at the 90% confidence level, but not at the 95% confidence level. For midterm 2, there is no significant difference in the scores of the two sections, even at the 99% confidence level.

Table 4. Comparison of means between ALN and non-ALN Midterm Scores in SPAN 210

Based on the Midterm 2 performance, it appears that doubling the class size and meeting with the TA only 1 day a week (instead of 3 days a week in the non-ALN sections) has not had an adverse affect on student performance.

The attitude surveys indicate that students in the ALN sections had significantly less contact with their peers and with their instructor than students in the traditional section. Interestingly, the students in the ALN sections indicated they had significantly greater access to their instructors. Thus, the reduced interaction with the instructor in the ALN sections must be, in part, a matter of choice by the students.

These findings are explained by the following. The traditional sections met for 3 hours a week. The ALN sections met one hour a week. During the other two scheduled hours, the instructor held office hours. (There were also office hours at other times.) This meant the students had no other obligations during at least some of the scheduled office hours and therefore should have been expected to report they had good access to the instructor. Office hours were voluntary. Many students did not avail themselves of this contact opportunity. The focus group discussions indicate that for a grammar course with the exercises in Mallard, many students did not perceive the need to discuss the material with the instructor. This is why there was less contact. The students perceived the course to be self-paced (though it did have the one weekly class session). Apparently, they did the work on their own, rather than in groups. In general, we would like to see a lot of student-to-student contact in an ALN course. But for this type of material, that contact may be unnecessary.

There were three summative questions posed in the survey:

1. How difficult was the material?

2. Would you recommend the course to a friend?

3. How much did you learn?

The responses to questions (1.) and (2.) did not significantly vary from the ALN to the non-ALN sections. Students found the material moderately easy and more would recommend the course than would not. On question (3.), the ALN students reported learning less than did the non-ALN students. This is somewhat surprising in light of the exam results. From the responses to this question and to the question about Mallard in the focus group, it appears that about 75% of the class liked the Mallard approach and thought they got something worthwhile out of it. But some of the students thought the Mallard exercises dull and would have preferred more human interaction. We suspect it is those students who reported not learning much in the course. It would be very interesting to know 1) whether these students had problems due to computer literacy and 2) how these students actually did on the exams. The surveys were anonymous so we do not have this information. It does raise the question, however, whether the students might be learning, at least as measured by the exams, without their perceiving it.

B. Cell C of the Efficiency Matrix – STAT 100, ECE 100, and CHEM 331
STAT 100
STAT 100 is a course that fills the UIUC’s relatively new General Education quantitative requirement. For most of these students it is the only quantitative course to be taken in college. Several years ago the experienced ALN instructor developed a software package for the course, WinPop, designed to be easy to use by the students (because many of these students were not very computer savvy) and to demonstrate fundamental statistical concepts in a highly visual manner. When SCALE came along, this instructor introduced FirstClass conferencing into the course. (He has since switched to WebBoard.) He also converted his software to Java, so it is visible in Netscape. He has recently started to use Mallard for administering on-line quizzes.

By using ALN, this experienced instructor has increased the number of students he teaches in his section of STAT 100. As shown in Table 4, enrollments in STAT 100 continue to increase. The experienced ALN instructor is working with other professors who teach the course to get them to use ALN. The hope is that all professors who teach STAT 100 can raise their enrollment levels by using ALN and subsequently accommodate the expanding enrollments without increasing instructional delivery costs.

Table 5. Overall Enrollment in STAT 100 by Semester

Table 6 presents information on section-by-section enrollments in STAT 100. (Note that Discovery sections of STAT 100 are not included in the calculations in Table 6, but they do count in the overall enrollments in Table 4.) The ALN section that was taught by the experienced ALN instructor has been the largest section of STAT 100 since fall 1995. For the three semesters, fall 1995 – fall 1996, the ALN section had about 25 more students than the traditional sections. Overall demand ratcheted upwards in fall 1996. This was initially met by adding a traditional section. By spring 1997 it was apparent that this demand increase was permanent. Moreover, the department owed the campus a Discovery section of the course that it had not taught the previous spring. Thus, it had to accommodate over 420 students with only four sections. As an experiment, it was decided to have the entire increased capacity be borne by the ALN section. This experiment proved successful and enrollments in this ALN section remained high in fall 1997. Moreover, some of the other instructors began to experiment with ALN. To keep this experimentation from being too onerous, section size was reduced. This explains the increase in section size in the traditional sections that semester. The long-term plan is to have all sections be ALN and to have only four non-Discovery sections.

Table 6. Average Enrollment Per Section in STAT 100 by Semester

The large ALN section uses an undergraduate peer tutor at an estimated cost of $1,000 per semester, paid out of funds supplied by SCALE. This constitutes the only difference in operating costs between the big ALN section all other non-Discovery sections. The department provides approximately the same amount of money to be spent on faculty, TA, and grader support in all sections. Thus, it is clear that the big ALN section is more cost-effective than the other sections.

We have a modest amount of comparative exam results from the fall 1997 semester. These come from common questions on one midterm. Four of the sections were involved in making these cross section comparisons. These were ALN1, the largest section and taught by the instructor who developed the materials; ALN2, the smallest of the four and taught by an instructor was using ALN for the first time, adopting the approach used in the largest section; and two other sections, non-ALN1 and non-ALN2, that used the traditional approach.

ALN1 and non-ALN1 had questions 1 – 4 in common. ALN1, ALN2, and non-ALN2 had questions 5 – 7 in common. We did a comparison of means, on a question by question basis. Table 7 shows that ALN1’s students out performed the others. They had the highest mean scores on each question, significantly higher than non-ALN1’s students on questions 1 and 3, significantly higher than non-ALN2’s students on questions 5 – 7, and significantly higher than ALN2’s students on questions 5 and 7 at the 90% confidence level. The latter suggests that we cannot be sure whether the results are attributable to ALN or instead to instructor-specific effects. Nevertheless, the results should make one optimistic about ALN. It seems that in this class ALN is boosting student exam performance. This is all the more impressive considering that ALN1 is larger and has a lower cost per student.

Table 7. Comparison of Student Performance on Common Exam Items Administered in ALN and non-ALN Sections of STAT 100

Regarding student attitudes, we have survey data only from the large ALN section. Thus we have no comparative information on student attitudes.

The professor used Mallard for on-line quizzes. He used WebBoard for computer conferencing and has developed quite a lot of Web-based, highly graphical statistical material, to illustrate basic principles. In this use of the virtual environment, STAT 100 was quite similar to ECE 110. Moreover, the summative questions in the survey for STAT 100 were identical to those for ECE 110. Interestingly, the student responses were quite similar to those in the engineering course.

Over 90% said they found using the Web easy or somewhat easy. Eighty-five percent rated their overall experience as good or better. And not quite 90% said they would probably or definitely take another course that used the Web.

It must be emphasized that the STAT 100 students are non-technical (in contrast to the ECE 110 students). Consequently, it might be a reasonable inference that it is the teaching approach coming through in the responses to the summative questions.

ECE 110
Because of rapid changes in the field, the Department of Electrical and Computer Engineering has deemed it necessary that students get more hands-on experience in the laboratory beginning their freshman year. To achieve this end, the department has decided to move circuit analysis from the second year to the first year of instruction. The emphasis on hands-on instruction requires some de-emphasis on theory. The new course, ECE 110, takes a more basic approach to the theory than the old course, ECE 270. The PI is the main instructor in the lecture component of ECE 110. He has developed extensive materials for delivery in Mallard. All the homework is on-line and is automatically graded. The PI also makes extensive use of newsgroups/conferencing. In this way he can easily keep up with problems that students may be having with the material.

ECE 110 has been taught with ALN from the outset. Thus there is no basis for comparison with a traditional version of the course. The lecture part of ECE 110 is less labor intensive than the analogous part of ECE 270. Indeed, with ECE 270 as the base, all development costs, about $65,000, had been recovered by the end of fall 1997. This should be interpreted cautiously, however. While some of the cost savings are undeniably due to ALN, some of the savings must be attributed to course restructuring.

We have absolute attitudinal information from the course survey. Because a newsgroup was used in addition to Mallard, the survey refers to Web use rather than Mallard use. There were three summative questions in the survey.

1. How easy did you find using the Web for purposes of this course?
2. How would you rate your overall experience using the Web in this course?
3. Would you take another course using the Web?

Over 90% of the class reported that using the Web was easy or somewhat easy. Eighty percent of the class rated it as good, very good, or excellent. And Eighty-eight percent of the class reported they would probably or definitely take another class with the Web.

The survey results indicate that the bulk of the students are happy with the way the course is delivered. We cannot tell, however, if that is due to student characteristics, the vast majority being electrical engineering students, or if instead it is due to characteristics of course design. For that reason, the comparison with STAT 100 is helpful.

CHEM 331
CHEM 331 is the organic chemistry field course intended for students in the Life Sciences. The project is actually an outgrowth of the PI’s (and the students') displeasure with the use of lectures in the course. The instructor believed she could greatly improve the course by teaching it entirely on-line. The absence of face-to-face contact constitutes a more radical experiment than SCALE’s other projects. There are several potential efficiencies if the project is successful, including larger enrollments, lack of classroom-facility requirements, and additional revenue from extramural student tuition rates.

The movement to a totally on-line format was more radical than was taken in the other SCALE courses. The fall 1997 semester was the first offering of the course in this mode. The professor advertised that the course would be taught this way. Nonetheless, she reported that a full Twenty-five percent of the students who signed up for the course did not have the requisite computer literacy. Either the advertising failed or the students ignored it, perhaps because there is only one section of CHEM 331 per semester. It is fair to say that some of the negative responses to the summative questions on the CHEM 331 survey are due to these problems with computer background, rather than with the teaching approach itself.

The three summative questions in this survey were:

1. Compared to traditional (i.e., non-online) courses, how much did you learn in this course?
2. How would you rate the overall quality of this course?
3. Would you recommend this course to your friends?

The results show that 47% thought there was either no difference or more learning in this ALN course than in the traditional course. 38% thought the quality was good or better. And 38% would probably or definitely recommend the course to a friend.

Section Analysis
A comparison of the STAT 100 and the ECE 110 courses suggest that we are seeing the success of the ALN approach itself. It is unlikely that the high marks on the summative questions in the survey could be explained by a matching of student characteristics to the particular teaching style, given how disparate the audiences for the two classes are.

The more mixed responses to the summative questions for CHEM 331 suggest either that some students perceive a benefit to lecture that others, including the instructor, may not acknowledge, or that there needs to be much more help for the students at the outset when an on-campus course is taught totally online.

C. Cell D of the Efficiency Matrix – ECON 300 and MATH 285
ECON 300
Among the original SCALE projects, ECON 300 is the only one that from the outset was designed with the goal of achieving cost savings in instruction. After some trial experimentation with ALN, the instructor has gone from teaching a traditional course of 60 to teaching 180 students with ALN. The traditional class size is explained as follows. In the Economics department as elsewhere on campus, it is believed that making eye contact with all students in lecture is critical to being able to offer a high quality course. Countering this, overall enrollments for ECON 300 are around 700 and the department is pressed to find enough instructors to staff the course. The number 60 represents the high end of a balance between these competing needs. The department has had access to amphitheater classroom seating for teaching ECON 300 and that creates an ability to teach relatively large numbers in an intimate setting. Absent this capacity, section size in ECON 300 might have been smaller. Indeed, not all sections use the amphitheater classrooms and these other sections do tend to be smaller.

The large ALN section represents an abandoning of the eye-contact model and a general de-emphasis of the lecture in favor of on-line activities, with the aim that the overall quality of the course would improve. The pedagogic strategy behind the ALN approach has several components. There is a self-teaching component that utilizes Mallard for on-line quizzes. There are also written problem sets done online in FirstClass. The problem sets are assessed on a team basis. An individual team member submits a proposed solution to a particular homework problem on behalf of the team. The submission is graded rapidly, within 48 hours, and returned online to a team conference. As long as this occurs before the deadline for the problem set, another team member can resubmit the problem, taking account of the grader’s comments. The rapid turnaround time is facilitated by use of undergraduate graders. These same peer tutors also provide office hour help, both face to face and online during the evening. The large class size justifies having many of these peer tutors. An absolute grading scale has been imposed to encourage the students within a team to collaborate. More than half the credit for the course is based on the homework. This is approximately equally divided between the self-teaching work done in Mallard and the group work done in FirstClass.

In fall 1997 (and spring 1998) there was another instructor who had his own ALN section of 60 students. This instructor was ‘apprenticing’ with the PI and used the previously developed course materials and same on-line pedagogy. The undergraduate TAs for both sections provided common office hours, both face to face and online. The TAs grading time, however, was devoted only to the section to which the TA was assigned. In the cost part of the study on ECON 300, the entire focus is on the original PI. The apprentice instructor is ignored. In the output/quality part of the study, extensive comparisons are made between the two sections, both in student performance and in student attitude. Here we are not measuring how ALN compares to the traditional approach, but rather whether ALN pedagogy can translate well from the creator to another instructor.

In Table 8, we present some summary results on the cost estimates for the ECON 300 project. In spring 1996 and again in spring 1997, the course was taught with 150 students. (The ALN version was not offered in fall 1996.) Then in fall 1997, the enrollments in the ALN section were allowed to increase to 180. Since our most precise operating cost information is for fall 1997, we have calculated the numbers in Table 8 using fall 1997 data. The numbers in the first column of Table 8 are based on a pro rata share of the fall 1997 costs rather than on the historical data. If anything, this understates the cost savings, as the peer tutor/student ratio increased from spring 1996 to fall 1997. That there are four rows to the table reflect the issue of how to allocate faculty salary to teaching. Rows one and two reflect a low fraction, one ninth of the annual nine-month salary. With a four course annual load, this leaves more than half the faculty time to be allocated for research and service and in that allocation these alternative uses of faculty time are viewed as orthogonal rather than as complementary. Rows three and four are based on a higher fraction of faculty time, one quarter of the annual salary. This makes sense if there is a strong complementarity between all uses of faculty time and hence it is appropriate to allocate it all to teaching. Another issue reflected in the table is workload. Going from teaching a traditional course of 60 students to teaching an ALN course of 180 increased the workload of the faculty member. In general, to encourage professors to teach larger ALN sections, these sections may have to be given more weight in the calculation of faculty workload. (For ECON 300, the PI believed a multiple of 1.5X would be a fair estimate.) The increased compensation for the large ALN section is reflected in rows two and four of the table. It should be noted that this represents a hypothetical case. No actual compensation of this type was paid.

The ECON 300 project was funded in each year of the original grant. Funding was greatest during the first year, when there was both a course buyout for the PI as well as summer support. Total development costs are estimated at $56,224, with around 75% of that figure counting as compensation for the PI. In Table 8 we provide an estimate of the period of cost recovery under each scenario of faculty compensation. To get some reasonable bounds, we do this twice. First, we do this without discounting the future cost savings. These estimates are given in column three of the table. Then we repeat this with discounting, using the 9% as the annual interest rate. As should be obvious from the table, the period of cost recovery is sensitive to how faculty salary is imputed (and to the discount factor). Under the most optimistic scenario, the large ALN section of ECON 300 was already producing a hefty surplus by the end of fall 1997. Under the most pessimistic scenario, we still have to wait a year to recoup all the development costs.

Table 8 – Per Student Cost Savings and Length of Recovery Period

We do not have any ALN versus non-ALN information for ECON 300. We do have substantial information, however, that compares the two ALN sections. We have exam results for common question on two midterms and a final. (Note that on the final there were quite a few students who took a conflict exam. The scores of those students are not included in the sample.) In this case, we are providing evidence on whether the ALN approach transfers readily from the developer of the materials to another instructor. Again we perform a comparison of means. 0n each of the midterms there were 5 common questions on the exams (4 points a piece). On the final, there were 14 common questions.