Using Grades to Evaluate a Career
Course:
Technical Report No. 43[1]
By
Robert C. Reardon, Ph.D.
Professor and Program Director
Stephen J. Leierer, Ph.D.
Associate Professor,
Donghyuck (Ryan) Lee, M.Ed.
February 22, 2006
Center for the Study of Technology in Counseling and
Career Development
Department of Educational Psychology and Learning
Systems
UCA 4100
Ph: 850-644-6431
Using Grades to Evaluate a Career
Course: Technical Report No. 43
Abstract
Grades in a standardized career
course offered at a large southeastern university over a 26-year period were
analyzed to measure the course impact on student learning. The analysis
examined the extent to which course structures and procedures affected grades,
how grades differed across terms and whether there were changes in grade trends
over time. The relationship between expected and earned grades was also
examined along with instructor effectiveness ratings.
Results indicated that did not
differ across quarter and semester systems or following the introduction of a
plus/minus grading scheme. However, grades were lower toward the end of the
26-year period following the introduction of a career theory base to the course
and coinciding with the more widespread use of the Internet in career research
report writing. The results revealed that 74 percent of the 6,176 students
completing the course met the learning objectives of the course with a grade of
B+ or higher. Grades in the summer were significantly higher than grades in the
spring or fall, and grades in the spring were significantly higher than the
fall. When grades were examined over six time periods, the grades in the most
recent time period were lower than those in any other time period.
To examine the relationship between
expected and earned grades, data were obtained from instructors during the
period of fall 1999 through summer 2004. A difference was found between
aggregated Expected GPA and Earned GPA across semesters, with the same order of
differences as noted above with overall grades: summer was higher than spring
or fall, and spring higher than fall. Each term, students expected higher
grades than they earned. In addition, students meeting 4 times per week had
larger differences in expected and earned GPAs than students meeting once,
twice, or three times per week. With respect to ratings of instructor
effectiveness, students in both the fall and spring rated instructors higher
than in the summer term, and students in the spring gave higher ratings than in
the fall. Finally, we found that the Expected GPA and the instructor rating
were significant predictors of the Earned GPA.
The findings of this study of career
course grades were discussed in terms of the issue of grade inflation in
postsecondary education, the use of grades as a method of career course
evaluation, how the structure of the course grading rubric is based on the
instructional goals and philosophy of the instructor, and the deliberate use of
career courses as a career intervention in colleges and universities.
Using Grades to Evaluate a Career
Course: Technical Report No. 43
Courses have become a popular method for helping college students learn more about the career development area and improve their skills in career problem solving and decision making. A decade ago, Mead and Korschgen (1994) conducted a national survey of postsecondary schools and found that 62% of the respondents offered some kind of career course, and later Halasz and Kempton (2000) reported the number at 70%. A more conservative estimate of career course implementation was reported by Collins (1998), who surveyed 1,688 college members of the National Association of Colleges and Employers and found that 30% of respondents were offering credit-bearing courses and 24% were offering non-credit courses, a figure that had held steady since 1981.
Career courses are quite varied in design, scope, and function (Folsom & Reardon, 2003; Folsom, Reardon, & Lee, 2005). For example, they reported that courses were offered for credit or not; credits ranged from 1 to 3 hours, and some were variable credit. Some courses were designed for entering first year students, and others were designed for upper division students already in their majors or about to graduate. Some courses were elective and others were required in a specific major. Some courses were highly structured and others were more open-ended in format. Some courses focused on self-assessment and career planning, and others included knowledge about labor markets, employing organizations, employment, and substantive behavioral sciences content. Some courses were offered in a more stand-alone format, and others were fully integrated into ongoing career services programs available on the campus. Some courses were taught by career counseling staff, others were taught by regular faculty in varied academic programs, and still others were team taught by combinations of personnel.
Evaluating the impact of career courses has been a recurring focus of research in this area. Folsom and Reardon (2003) and Folsom et al. (2005) reported that career courses have a positive impact on the cognitive functioning of students in several areas, and these courses also appeared to have a positive impact on student outcomes, including satisfaction with career courses and increased college retention. They found that 34 of the 38 studies reported positive career course results; 90% of the studies showed positive gains in vocational identity, career decision making, or other output variables. Similarly, of the 15 studies involving outcomes such as retention in college, 87% showed positive results.
However, one area of course evaluation that has not received much attention is related to the use of student grades in designing a career course and evaluating the impact of a course on student learning. While there is considerable controversy surrounding the use of grades in evaluating student academic performance (Kohn, 2002), grades are ubiquitous in higher education. Grades can be used to reflect relative standing or absolute accomplishment. A faculty colleague once quipped, “Grades are the coin of the realm.” Course grades, while not a standardized, objective measure, could be used to assess the impact of student learning and the degree to which the objectives of a career course intervention are met.
The issue of how to grade student progress in a career course was discussed by Filer (1986). He noted that the unique nature of a career course could make it difficult to determine what an “A” grade actually means, e.g., career decision-making knowledge, employability skills, self-knowledge. Course grades need to be understood in terms of the course objectives, which typically focus upon increased knowledge about self, educational, and occupational options, and job search skills. Given this goal, does a low grade mean that the student did not increase self-knowledge about interests, values, or abilities? Recalling Bloom’s (1956) taxonomy of educational objectives, student learning could be assessed in terms of the recall of facts about career theories or occupational classifications, or the evaluations and judgments of the meaning of personal life events. Another grading option reported by Michelozzi (2000) and Reed, Reardon, Lenz, and Leierer (2001) involved the use of a performance contract and the awarding of grading points as course activities were completed, e.g., attendance, completion of interest inventories, writing a resume. In this instance, students can choose the grade they want to receive and then work towards that outcome through completion of contracted learning activities. Ideally, this would increase their motivation to engage the instructional activities in the course.
Ultimately, a course grading system reflects the instructor’s values for assigning importance to various learning events and activities in a course. Filer (1986) noted that a grading system stimulates self-management on the part of students seeking a good grade, and it also simulates a work setting in that rewards (grades) are awarded to those who execute tasks (course assignments) well.
The present study involved a career course offered at a large southeastern research university since 1973 (Lee & Anthony, 1974). Several studies have been conducted in an effort to learn more about the impact of this course. Reardon and Regan (1981) used student course evaluation ratings and found no significant differences between the career planning course and a comparison group of other university courses with regard to levels of instructor involvement, student interest, and course demands; however, the career course received higher ratings in amount of student-instructor interaction and level of course organization. More recently, Vernick, Reardon, and Sampson (2004) used student course ratings of the same course and found that student perceptions of the quality of the career course have been quite consistent over time. However, in this later study students reported the career course to be characterized by greater course demands, greater student-instructor involvement, greater course organization, and lower student interest than other credit courses at the university. They also found that student perceptions of career courses appeared to be more positive when the class met more than one time per week, allowing students more opportunities to integrate and apply what they were learning.
Two other studies examined the impact of this course using objective measures and outcomes other than student course ratings. Folsom, Reardon, Peterson, and Mann (2005) assessed the impact of the course in terms of time taken to graduate, graduation rate, credit hours taken, number of course withdrawals, and cumulative GPAs. Student course participants (N = 544) were compared to a matched sample of non-course participants (N = 544) on these variables after five years. Results showed that the two groups differed with respect to hours taken to graduate and the number of course withdrawals. Women participants graduated in less time than non-participants but had more course withdrawals. Men took longer to graduate but had fewer course withdrawals and higher GPAs. Reed et al. (2001) examined the impact of this course on cognitive information processing. Students showed a significant decrease in their negative career thoughts when the Career Thoughts Inventory (CTI; Sampson, Peterson, Lenz, Reardon, & Saunders, 1996) was used as a pretest and posttest measure. The greatest decrease in negative thinking was found in students with the highest level of negative thinking at the beginning of the course. Specific components of negative career thinking, decision-making confusion, and commitment anxiety, contributed significantly to the main effect. There were no significant interactions with race or gender.
We sought to extend the earlier research on the impact of a career course by examining course grades over a 26-year period, from the fall of 1978 through the summer of 2004. We believed that course grades could be used as a dependent variable and a practical measure of course impact. However, the literature on grades and grading practices in higher education reveals controversy. Considerable discussion has appeared regarding the matter of “grade inflation.” Kuh and Hu (1999) and Levine and Creton (1998) have reported that grade inflation appeared in the early 1960s and continued through the 1990s, but at a slower rate in later years. On the other hand, Johnson (2003) and Kohn (2002) suggested that grade inflation may not exist and is an artifact of such things as student selection of courses more likely to produce a high grade. This study provided an opportunity to examine the presence of possible grade inflation in the same course using the same grading rubric over 26 years.
In addition, the present study provided an opportunity to examine the relationship between students’ earned and expected grades and their ratings of overall instructor effectiveness. Typically, students of lower academic ability tend to over estimate their grades while those of higher academic ability tend to be more accurate in predicting their grades or even underestimate their performance (Balch, 1992; Prohaska, 1994). On the last day of class, students in each class were asked to indicate the grade they expected to receive. This overall expected “class grade” was compared to the overall earned “class grade.” In addition, we were able to retrieve overall class ratings of instructors for an ad hoc sample of career classes from 1999-2004. These data enabled us to examine the relationships between expected and earned grades and instructor ratings.
Several research questions guided this study.
Method
The majority of students completed this course titled “Introduction to Career Development,” to fulfill elective requirements for the baccalaureate degree. Since it was developed in 1973, the course has typically enrolled about 60% females and 40% males, including freshmen (15%), sophomores (45%), juniors (20%), and seniors (20%). Ethnic diversity is generally proportional to the general student population of the university: American Indian, .4%; African American, 12%; Asian, 3%; Hispanic American, 7%; Caucasian, 74%; and Other, 4%. Depending on the semester, anywhere from 15% to 25% of the class is composed of students with officially undeclared majors. The large percentage of sophomores enrolled is the result of academic advisors referring these undeclared students to the class. When a typical class section is asked to describe their satisfaction with their career, the responses show the following pattern.
1. well satisfied with my choice, 14%.
2. satisfied but have a few doubts, 23%;
3. not sure, 16%;
4. dissatisfied but intend to remain, 6%;
5. very dissatisfied and intend to change, 1%;
6. undecided about my future, 40%;
While almost 40% of the members in a typical class are satisfied with their present career situation, about 60% are unsure, dissatisfied, or undecided.
The career course examined in this study has been in existence since 1973 (Lee & Anthony, 1974; Peterson, Sampson, & Reardon, 1991; Peterson, Sampson, Lenz, & Reardon, 2004), but the focal point of this investigation is from 1978-2004. This course was also described as an intervention in a case study report by Reardon and Wright (1999). The original course was a series of career seminars, which were eventually developed into a formal three credit hour course taught by staff in the counseling center and the career placement center. Instructional systems specialists further developed and improved the course design and integrated multimedia career development resources available through the university’s career resource center. In 1984, the conceptual base of the course changed to include a systems approach, and in 1993, a foundation in cognitive information processing (CIP) theory was added. The present course is based on CIP theory (Peterson, Sampson, & Reardon, 1991; Peterson, Sampson, Reardon, & Lenz, 2004; Sampson, Lenz, Reardon, & Peterson, 1999), which is incorporated into the text, Career Planning and Development: A Comprehensive Approach (Reardon, Lenz, Sampson, & Peterson, 2005).
Since 1975, the course has included eight instructional objectives. As a result of completing the course learning activities students will be able to:
The course
is comprised of three units. Unit I, “Career Concepts and Applications,”
focuses on self-knowledge, knowledge about options, and decision making.
Assignments include writing an autobiography, completing the Self-Directed
Search (
Unit II, “Social Conditions Affecting Career Development,” focuses on current social, economic, family, and organizational changes affecting the career planning process and the need for students to develop more complex cognitive schema to solve career problems. Unit III of the course focuses on employability skills and strategies for implementing academic/career plans. Assignments include two information interview reports, the completion of a resume and cover letter, and a strategic/academic career plan paper. This final paper utilizes the CASVE cycle from CIP theory as an over-arching cognitive strategy to help students integrate their learning into the career problem-solving and decision-making process.
Since 1980, the course has been team-taught annually by a lead instructor and 3 to 4 co-instructors, typically graduate students, with an instructor/student ratio of about 1:8. The class involves a mixture of lecture, panel presentations, and small and large group activities. Each instructor is assigned a small group of students who meet throughout the semester during class time. The instructors also meet individually with the students at least once during the semester to discuss their assessments and progress in the class.
Several events occurred during the period of this investigation which had the potential to impact course grades:
Fall 1981 University switched from the quarter to semester system
Fall 1984 Plus/minus (+/-) grading system implemented
1984-1988 Catalyst materials on work/family life introduced (Gerken, Reardon, & Bash, 1988)
Fall 1994 CIP theory introduced into the course (Peterson, Sampson, & Reardon, 1991)
Spring 1996-Spring 2000 Once per week sections offered
Summer 1998 Last time Friday section was offered
Fall 1999 CIP text and student manual introduced (Reardon et al., 2000)
Fall 1999 Internet increasingly used in students’ research
The impact of these structures and procedures on student learning is examined in the results section.
Grades are based on the successful execution of a performance contract (PC) by the student. This procedure has been in place since 1975. The PC includes 16 different graded activities spread across the three units of the course; each activity has a point value ranging from 5 to 100 and some activities are counted separately in each unit, e.g., a unit test worth 25 points is given in each unit. Table 1 shows the graded activities, the points assigned to the activity, the course units that include the activity, and the course objectives related to the activity. A student enrolled for three credits is evaluated on 28 different activities and a letter grade is assigned on the basis of successful completion of a percentage of the 653 available points, e.g., 90-99% of available points for an A grade. Figure 1 shows the percentage of course grade points related to the learning activities if the student were enrolled for three credits. More information about this career course is available at http://www.career.fsu.edu/techcenter/instructor/undergraduate/index.html.
------------------------------------------------------------------
Insert Table 1 course contract x objectives about here
------------------------------------------------------------------
------------------------------------------------------------------
Insert Figure 1 course grade by activity percentage about here
------------------------------------------------------------------
As reported
earlier by Reardon and Reagan (1981) and Vernick, Reardon, and Sampson (2004),
the present study drew in part upon the results of a standardized instrument
that has been used at the university since 1971, the Student Instructional
Rating System (SIRS; Arreola, 1973), as well as academic records provided by
the university registrar’s office. The SIRS is a standardized student course
evaluation form developed at
SIRS
provided an opportunity for instructors to obtain reactions to their
instructional effectiveness and course organization and to compare these
results to similar courses offered within the university. At our university,
SIRS consists of 32 items, and 25 items enable students to express their degree
of satisfaction with the quality of instruction provided in the course by using
a five-point Likert scale. For example, “The course was well organized,” could
be marked Strongly Agree, Agree, Neutral, Disagree, or Strongly Disagree. The
SIRS includes five composite factors related to student evaluation of
courses: (1) instructor involvement, (2)
student interest, (3) student/instructor interaction, (4) course demands, and
(5) course organization. Originally established at
A second instructional rating instrument was also employed in this study, the State University System Student Assessment of Instruction (SUSSAI). This instrument consists of eight items focused on class and instructor evaluation. SUSSAI has been used in instructor evaluations since 1995 and there is no psychometric information available on the instrument. One item was of particular interest in this study: “Overall assessment of instructor: Excellent, Very Good, Good, Fair, Poor.” Results of this item were examined in relation to actual and expected grades obtained in sections of the course from 1999 through 2004.
Archived course grade data were obtained from the university registrar after permission was obtained from the university institutional research board. The registrar had electronic grade records from 1978 through 2004, and these data were retrieved for each quarter or semester during the period. The unit of measurement, then, was aggregate course data for each academic term, quarter or semester, fall 1978 through summer 2004. In addition, course data for each course section was provided by registrar records from the 1999 fall semester through the 2004 summer term. During this latter five-year period, 62 sections of the course were led by 12 different instructors, who taught the course from 1 to 10 times.
Because data related to the SIRS form are considered confidential personnel records, we obtained permission from lead instructors to use their SIRS data from the affected sections from fall 1999 through summer 2004 and agreed to report results of our investigation without using instructor names. Unfortunately, some instructors had misplaced their SIRS reports or they had been discarded and we could not obtain the information provided by students about their expected grade in that section of the course. Of the 62 course sections offered during the period, we obtained SIRS data from 46 (74%).
The results of SUSSAI course ratings by students are not part of the faculty confidential personnel records and we were able to obtain responses regarding 57 of the 63 sections (92%) of the course during this time period. Some records were missing because the instructor did not administer the form during the spring or summer terms when university policy does not require it, or because reports were simply lost or unrecorded over the years. No instructors declined to provide any of the data we requested.
Results
The results below are presented in reference to the questions that guided the study.
Several course changes occurred over the 26-year history of this study. In order to understand each of these changes, we examined the aggregated quarter or semester grade point averages of the students in this course before and after a particular change in the course (Table 2). We examined these changes in their historical order.
We found the aggregated GPA of students enrolled in this course under the quarter system was not significantly lower than the GPAs of students enrolled under the semester system [F (1, 6174) = .28, p = ns, Cohen’s d = .035]. In 1984, FSU changed from a five-point grading scale (A, B, C, D, F) to an 11-point plus/minus grading scale, and there was no significant difference between these two methods of grading [F (1,6174) = 1.09, p = ns, Cohen’s d = .045]. From 1985 fall semester through the 1991 fall semester, the Catalyst materials related to gender, family, and career were integrated into the course (Gerken et al., 1988). During these 21 semesters, the aggregated student GPAs were significantly higher than other semesters, 3.55 versus 3.44, [F (1, 6174) = 25.30, p < .0005, Cohen’s d = .148]. Two findings emerged in relation to the introduction of CIP theory into the course. First, students had lower aggregated GPAs after CIP theory was integrated into the course [F (1, 6174) = 115.06, p < .0005, Cohen’s d = .274]. Second, students made lower grades after the textbook (Career Development and Planning: A Comprehensive Approach) was introduced in the fall of 1999 [F (1, 6174) = 127.75, p < .0005, Cohen’s d = .317] and when Internet-based sources were used increasingly for occupational research (career field analysis research paper).
----------------------------------------------------
Insert Table 2 GPA variations over time by
grading systems, etc.
---------------------------------------------------
We were interested in learning if grades provided evidence that students were meeting the learning objectives for the course. From the fall of 1978 through summer of 2004, of the 6,303 students who enrolled in the course, 6,176 completed it. Of those students completing the course, approximately 74 percent earned a B+ or better. Table 3 further shows that 2,743 (44.4%) earned an A, 1,183 (19.2%) received an A-, and 619 (10.0%) earned a B+; by contrast, four percent of the students completing the course earned less than a C-. The mean grade point for the 6,176 completers was 3.44, SD = .84. Because 74 percent of the students received a B+ or higher, we concluded that these students met the learning objectives of the course. Interestingly, for these 75 semesters, there was a significant negative correlation between the mean semester GPA for the class and the semester identification number, r = -.38, p = .002. This finding suggested that class GPAs decreased over time, and the next section provides more detail about changes in course GPA over time.
----------------------------------------------------
Insert Table 3 Distribution of GPA from Fall 1978- Summer 2004
----------------------------------------------------
To answer questions about possible
changes in student grades over time, we conducted a semesters (fall, spring,
summer) ´
time periods (fall 1978 to spring 1981, fall 1981 to summer 1985, fall 1985 to
summer 1990, fall 1990 to summer 1995, fall 1996 to summer 2000, fall 2000 to
summer 2004) Analysis of Variance (ANOVA). The overall model was significant F (17, 6159) = 6280.86, p < .0005, h2 = .94). The
mean GPA for the spring, summer, and fall semesters, was 3.48, 3.58, and 3.39,
respectively (see Figure 2). The ANOVA revealed a significant effect for the
semester variable [F (2, 6159) = 13.79, p < .0005, h2=
.004]. A post-hoc analysis indicated the GPA of students in the summer semester
was significantly higher than the GPA students in the fall or spring semesters.
Furthermore, students in the spring semester earned a significantly higher GPA
than students in the fall semester.
-------------------------------------------------------
Insert Figure 2 mean scores of aggregated
GPAs for 3 semesters about here
-------------------------------------------------------
Grade Distribution over Time
We also examined the mean career course grades over six time periods (see Figure 3). There was a significant main effect for time period, [F (5, 6159) = 23.69, p < .0005, h2= .02]. Post-hoc tests indicated that students in the fall 2000 to summer 2004 period had a significantly lower GPA than students in any other time period. Conversely, students who took the course from the fall 1990 to summer 1995 had a significantly higher mean GPA than students in four of the five other time periods. In examining the trend of the GPAs for the six time periods, we did not find a significant GPA increase from the first (M = 3.47, SD = .89) to the second period (M = 3.54, SD = .87) or the second to the third time period (M = 3.50, SD = .77); however, there was a statistically significant increase (p = .001) of 0.13 points in the mean GPA from the third to the fourth period (M = 3.63, SD = .64). After the third time period there was a statistically significant reduction (p < .0005) of 0.32 points from the fourth to the fifth (M = 3.31, SD = .93). Likewise, there was a statistically significant (p < .0005) decrease of 0.12 points in mean GPA from the fifth to the sixth time period (M = 3.23, SD = .95). There was not a significant interaction of semester by time period [F (9, 6159) = 1.76, p = ns].
------------------------------------------------------
Insert Figure 3 trend of Aggregated GPA for Six Time Periods about here
---------------------------------------------------------
In order to examine
matters related to students’ expected and earned grades in the career course,
along with their perceptions of instructor effectiveness, we used data from two
instruments discussed earlier, the State University System Student
Assessment of Instruction (SUSSAI) and
the Student Instructional Rating System (SIRS).
To explore
the relationship between expected grades and earned grades, we obtained the
aggregated data for instructors of course sections from the fall of 1999 to the
summer of 2004. The class met four times per week during the summer semester;
therefore, grade frequencies used to generate the aggregate grade point average
(GPA) for the four days per week variable were exactly the same grade
frequencies used to compute the aggregate GPA for the summer semester. Because
of this redundancy, we were not able to examine the interaction of classes per
week across semesters. Therefore, we examined the aggregated GPA with Two-Way
Repeated Measure ANOVA, instead of a single Repeated Measure MANOVA. In the first
Two-Way Repeated Measures ANOVA, we contrasted the aggregated earned GPA (EGPA)
and expected GPA (XGPA).
As shown in Figure 4, we found a significant difference between the students’ aggregated EGPA (M = 3.28, SD = .14) and XGPA (M = 3.52, SD = .20), [F (1, 1560) = 1517.16, p < .0005, h2 = .48]. There was a significant main effect of semester for the dependent variable of EGPA [F (2, 1,560) = 1054.06, p < .0005, h2 = .36], with the summer semester (M = 3.50, SD = .10) significantly higher than the spring semester (M = 3.28, SD = .10). The fall semester (M = 3.19, SD = .07) was significantly lower than the spring semester.
---------------------------------------------------------
---------------------------------------------------------
Likewise, there was a significant
difference in the main effect semester variable for the means of XGPA [F
(2, 1560) = 104.93, p < .0005, h2
= .12]. Students in the summer semester had significantly higher expected GPA
(3.60, SD = .07) than students in the spring (M = 3.56, SD =
.17); conversely, the aggregate XGPA of students in the fall (M = 3.44, SD
= .22) was significantly lower than students in the spring semester. We found a
semester ´
GPA interaction; that is, the difference between the students’ EGPA versus XGPA
varied by semester [F (2, 1560) = 84.97, p < .0005, h2 = .10. Students in each
semester expected higher grades than they earned, spring (3.56, 3.28), summer
(3.60, 3.50), and fall (3.44, 3.19) (see Figure 4). The difference between the
aggregated earned GPA and the expected GPA of summer students was significantly
smaller than students in the fall and spring semesters. In addition, the
students in the fall semester had a significantly larger difference between the
earned and expected GPA.
With the second Two-Way Repeated Measures ANOVA we examined the differences between the earned and the expected GPA by the number of classes per week. The aggregated earned GPA for students enrolled in one class per week, two classes per week, three classes per week, and four classes per week were 3.25, 3.24, 3.22, and 3.50, respectively (see Figure 5). The aggregated expected GPAs were 3.59, 3.55, 3.25, and 3.60, for students who were enrolled in one class per week, two classes per week, three classes per week, and four classes per week respectively (see Figure 5). When considering differences by the number of classes per week, students expected a higher GPA than they earned (3.50 versus 3.30). These differences were significant [F (1, 1,559) = 942.87, p < .0005, h2 = .38. In addition, there were significant differences in the earned GPA for the number of classes per week [F (3, 1559) = 330.44, p < .0005, h2 = .39]. There was a significant EGPA ´ Classes per week interaction [F (3, 1559) Expected = 211.00, p < .0005, h2 = .29]. Post-hoc tests indicated that students in the four day a week classes had significantly larger differences between expected and earned GPA than classes meeting once a week, twice a week, and three times a week. Conversely, those students meeting three days per week had significantly smaller differences between earned and expected GPA than the classes meeting once, two, and four times a week.
-------------------------------------------------------------------
Insert Figure 5 Average Mean Score of Earned GPA by Number of Classes Per Week about here
-------------------------------------------------------------------
The following section examines
research questions related to the student ratings of instructor effectiveness
(RATE), earned grade point average (EGPA), and expected grade point average
(XGPA). The means, standard deviations, and correlation are provided in Table
4. Because of the redundancy of the summer semester variable with the four
class days per week variable, we conducted two linear regression analyses to
address the unique effect of the students’ EGPA and XGPA with the semester or
number of classes per week on the RATE average (See Table 5). In the first linear regression, to examine
the influence of each semester on predicting the dependent variables, we dummy
coded the semester variable. Although we created three binary variables, fall
(1 = Enrolled, 0 = Not Enrolled), spring (1 = Enrolled, 0 = Not Enrolled), and
summer (1 = Enrolled, 0 = Not Enrolled), only the fall and spring semesters
were entered into the regression equation. The summer semester variable did not
need to be loaded into the equation because summer was coded as being neither
fall nor spring. The dummy coding matrix was as follows:
F S
Fall 1 0
Spring 0 1
Summer 0 0
We also entered the
continuous variables of EGPA and XGPA grade point average into the regression
equation. The adjusted R2 was .04; that is, these predictor
variables accounted for 4 percent of the variance in predicting instructor
rating [F = (4, 1558) = 19.47, p < .00005].
-------------------------------------------------------------
Insert Table 4 correlation of classes, GPAs, and instructor evaluation about here
-------------------------------------------------------------
-------------------------------------------------------------
Insert Table 5 multiple regression of semesters, classes, GPAs,
and instructor evaluations about here
-------------------------------------------------------------
The regression revealed that students in both the fall and spring
semesters were evaluating their instructors significantly higher than the
students in the summer semester. Students in fall semester evaluated their
instructors significantly higher than students in the summer semester (b = .43,
F = 61.70, p < .00005), and
students in the spring semester gave their instructors significantly higher
scores than summer semester students (b =
.33, F = 48.71, p < .0005). Interestingly, students in the spring semester rated
their instructors more favorably than students in the fall semester (b = .11,
F = 14.30, p < .00005). The
continuous variables of EGPA (b =
.15, F = 15.78, p < .0005) and XGPA (b =
.11, F = 15.78, p < .0005) were significant predictors of instructor
effectiveness ratings (RATE). The interpretation of these b weights is that when controlling for the
other variables in the regression equation, for a .15 point increase in RATE,
there was a one point increase in EGPA. Likewise, there was .11 point increase
in RATE, for each one point increase in XGPA when controlling for the other
variables in the equation.
Although the number of classes per
week could be considered a continuous independent variable, there was not a
linear relationship between the number of classes per week and the mean RATE,
EGPA and XGPA. That is, increasing the number of classes days per week did not
result in systematic increase or decrease in RATE, EGPA, XGPA. We created four
binary variables, one class per week (Enrolled=1, Not Enrolled = 0), two
classes per week (Enrolled=1, Not Enrolled = 0), three classes per week
(Enrolled=1, Not Enrolled = 0), and four per week (Enrolled=1, Not Enrolled =
0). However, only the two-classes per week, three-classes per week, and
four-classes per week and spring semesters were entered into the regression
equation. (The one-semester class variable did not need to be loaded into the
equation because summer was coded as being neither fall nor spring.) The dummy
coding matrix was as follows:
2 3 4
One class per week 0 0 0
Two classes
per week 1 0 0
Three
classes per week 0 1 0
Four classes
per week 0 0 1
The adjusted R2 was .20; that is, predictor variables
accounted for 20% of the variance in RATE [F
= (5, 1562) = 78.87, p <
.00005]. Students in the one-class per week section rated instructors
significantly less favorably than students in the two-classes per week (b = -.74,
F = 226.83, p < .0005),
three-classes per week (b = .75, F = 314.18, p < .0005), and four-classes per week (b = -.32,
F = 45.17, p < .0005) sections
of the course. Other comparisons of the contribution of classes per week to
RATE showed students in the three classes per week sections rated their
instructors significantly more favorably than students in two classes per week
(b = .23,
F = 74.30, p < .0005) or four
classes per week (b = -.51, F = 151.78, p < .0005). Finally, students in the four classes per week
sections rated their instructors less favorably than the students in the two
classes per week sections (b = -.16, F = 62.41, p < .0005). The continuous variables of EGPA (b = .08,
F = 6.43, p = .011) and XGPA (b = .24, F = 76.35, p < .0005) were significant
predictors of RATE. That is, when all other predictor variables are controlled,
for a one point in EGPA there was .14 point increase in RATE. Likewise, for a
one point increase in XGPA there was a .24 increase in RATE.
Earned Grade Point Average
We also
used linear regression equations to predict average earned GPA using the
semester enrolled, number of classes per week, students’ rating of instructor
effectiveness (RATE) and expected grade point average (see Table 6). As in the
first linear regression to predict RATE, we examined the contribution of the
semester, fall, spring, and summer, the continuous variables of XGPA and RATE
to predict EGPA. The adjusted R2 was .59; that is, these
predictor variables accounted for 59 percent of the variance in predicting EGPA
[F = (4, 1558) = 560.86, p < .00005]. The regression revealed
that mean EGPA for the both the fall and spring semesters were significantly
lower than the mean EGPA for summer semester. On average, those students in
fall semester earned a significantly lower EGPA than students the summer
semester (b = -1.07, F =1833.98, p < .00005). Likewise, students in the spring semester earned a
significantly lower average EGPA than students in the summer semester (b = -.81, F = 1166.02, p < .00005. Interestingly,
students in the spring earned a higher mean GPA than students in the fall
semester (b = .08,
F =235.01, p < .00005). Instructor rating (b = .11, F = 36.47, p < .0005) was
a significant predictor of EGPA; for a one point increase in RATE, there was a
.11 increase in EGPA when controlling for the other independent variables.
Likewise, XGPA (b
= .07, F = 15.78, p < .0005) was a significant predictor of EGPA; for a one
point increase in XGPA there was a .07 increase in EGPA when controlling for
the other independent variables.
------------------------------------------------------------------
Insert Table 6 regression of semesters,
classes per week, expected GPA
and instructor evaluation on earned GPA
-------------------------------------------------------------------
In the final linear regression to predict
EGPA, we used the variables class days per week as dummy coded above, and the
continuous variables of XGPA and RATE. The adjusted R2 was
.53; these predictor variables accounted for 53 percent of the variance in
predicting EGPA [F (3, 1559) =
350.33, p < .0005]. The EGPA of
students in one-class per week (b =
-.39, F = 386.52, p < .0005), two-classes per week (b = -.88,
F = 1377.89, p < .0005), and
three-classes per week (b = -.58, F = 492.40, p < .0005) earned significant lower GPA than students in the
four-classes per week. Other comparisons of the contribution of classes per
week to predicting EGPA showed that students in the three classes per week sections
did not obtain significantly higher earned grade point averages than students
in the two classes weekly sections (b
=.005, F = 0.05, p = ns) or the three classes per week sections (b = .05,
F = 2.66, p = ns). Likewise,
there was not a significant difference in the EGPA of those students in
sections meeting one or two days per week (b= -.04, F = 0.86 , p = ns).
The continuous variables of XGPA and RATE were significant predictors of
EGPA. XGPA (b = .20,
F = 90.44, p < .0005) was a
significant predictor of EGPA; for a one point increase in XGPA, there was an
increase of .20 in EGPA when controlling for the other independent
variables. Likewise, instructor rating
was a significant predictor of EGPA (b =
-.07, F = 16.66, p < .0005); for each unit increase in instructor evaluation
there was .05 increase in earned GPA when controlling for the other
independent variables.
Discussion
Grade Variations over Twenty-Six
Years across Academic Terms
This study examined
student grades over a 26-year period in a career course in an effort to learn
more about how various course variables affected student learning. One of the
variables examined was course structure and procedure, specifically quarter
versus semester course offerings and 5 versus 11 point grading scales. The
university switched from a quarter to a semester system in 1981 and introduced
the +/- grading system in 1984. No differences were found in course grades for
these two sets of variables. However, the intensive infusion of the work-family
life balance materials from Catalyst from 1984-1988 led to higher grades, while
the infusion of cognitive information processing theory in 1994 and the related
textbook in 1999 may have made the course more challenging and led to lower
grades. Closer inspection reveals that since 1999 students have increasingly
used Internet-based sources in researching occupations, and their selection and
use of these resources has often resulted in lower grades on the career field
analysis research paper. This paper accounts for about 30 percent of the course
grade. These shifts in earned grades during these periods may have been
influenced by other unknown events or historical factors. Nevertheless, it is
important to note that the grading procedures and basic structure of the course
did not change over the 26-year period of the study, e.g., performance
contract, variable credit. In effect, the dependent variable or grading system
did not change over time and we believe it represents a stable outcome measure
of the course during the period of this study.
There is evidence that
course grades measure the learning objectives of the course. An analysis of
Table 1 shows how the 8 course objectives were connected to the 16 graded
course learning activities. Of the 6,176 students completing the course from
1978 through 2004, 74% received a B+ or better grade, meaning that they
received at least 87% of the available points earned in completing course
assignments. Sixty-four percent of the students received a grade of A or A-. It
may be noted that 127 of 6,303 students enrolled in the course over 26 years
did not complete the course and were excluded from the analysis. These students
may have withdrawn from the university or been dismissed for various reasons.
Of the 12 grading categories for the course shown in Table 3, the sixth most
frequent course grade was F, or almost 3%, which provides some evidence of the
course demands.
Course grades, as an
indicator of student academic performance and learning, varied by semester.
Grades in the summer were significantly higher than the fall or spring terms,
and grades in the spring were higher than in the fall. There are several
possible explanations for this. Students in the summer term have a more
intensive course experience and are enrolled in fewer other courses. Summer
class sections also meet four times per week. Altogether, these conditions may
provide for a more intensive learning experience in the summer. Students
enrolled in the fall often schedule the class in the spring, four months before
the class begins, which leads to considerable schedule changing in the fall.
Class enrollments get set later in the fall than in the spring, which may
reflect more compromises that students have to make in creating their course
schedules. There is more continuity between registration and the beginning of
class in the spring and summer terms.
When grades over the
26-year history were analyzed in terms of six time periods, students in the
latest time period (fall 2000-summer 2004) had a significantly lower grades
than those in any other time period. While grades were higher in the fall 1990
to summer 1995 period, they declined steadily in the two time periods after
that. While some institutions have reported grade inflation, this does not
appear to be the case with this career course.
Using aggregated data
for instructors from 46 of 62 course sections from fall 1999 through summer
2004, we found differences between the grades that students expected and
earned. As noted earlier when examining grades across academic terms, the
earned GPA was higher in the summer than the fall and spring, and the spring
was higher than the fall. This finding was also present when grades across
course sections were examined. Moreover, the same pattern was present in this
analysis of the students’ expected GPA across course sections. In addition, the
difference between expected and earned GPA was smallest in the summer and
largest in the fall. We also found differences in earned and expected GPA with
respect to the number of class meetings per week. Students meeting four times
per week had the largest difference in earned and expected GPA, while students
meeting two times per week had the smallest differences.
We examined the
instructor ratings of overall effectiveness provided by students in 57 of 63
course sections. Students in the summer gave lower instructor ratings than
those in the fall or spring, and students in the fall gave lower ratings than
those in the spring. In a further analysis, we found that instructor
effectiveness ratings were related to both earned and expected GPA. When
instructor rating increased by .15 points, earned GPA increased by 1.0, and
when instructor rating increased by .11, expected GPA increased by 1.0. When
analyzed in terms of class meetings per week, students meeting once per week
had the lowest instructor ratings and students meeting three times per week had
the highest ratings. In addition, a 1.0 increase in earned GPA was related to a
1.4 increase in instructor ratings, and a 1.0 increase in expected GPA yielded
a .24 increase in instructor ratings.
The mean earned GPA
across course sections was lower in the fall and spring than the summer, and
students in the fall had a lower earned GPA than those in the spring.
Instructor ratings were a significant predictor of earned GPA, with a 1.0
increase in instructor rating yielding a 1.1 increase in earned GPA. Expected
GPA was also predicted by earned GPA with a 1.0 increase in expected GPA yielding
a .07 increase in earned GPA. Finally
when class days per week, expected GPA, and instructor ratings were used to
predict earned GPA, 53% of the variance was accounted for. Students meeting
four times per week had a higher earned GPA.
Conclusions
and Implications
One limitation of this
study pertained to the use of aggregate grades across semesters or course
sections. This study did not examine individual student grades across the 26
years, or connect individual student grades to expected grades. The use of
aggregate data also precluded an examination of ethnicity, gender, or other
learner characteristics in this research. Another limitation pertained to the
incomplete data provided by instructors regarding student ratings of teaching
effectiveness. In general, there are difficulties in analyzing instructor
effectiveness because some of this information is confidential and not a public
record.
This study provides
evidence that grades might be used as a valid dependent measure of the impact
of career course interventions, especially if the treatment variables are
carefully described and the grading procedures are fully explained and
replicable by other researchers. Indeed, using grades as a measure of career
course student learner outcomes is a logical and practical method for studying
a career course. Students (74 percent) achieved course objectives indicating
that the course was an effective career intervention for most students.
Moreover, grades for specific course learning activities provided a dependent
measure of various predictor variables. For example, is extroversion related to
the frequency of information interviewing as a course assignment, or is
conscientiousness related to the technical grade for a research paper? Is
academic self-efficacy related to scores on a unit test in a career course?
Does use of a performance contract (students can determine the amount of time
and effort to invest in course learning activities) differentially affect
students in terms of personality characteristics, readiness for career decision
making, current academic standing?
It is apparent that
this career course, with its array of learning activities and procedures is a
unique, complex career intervention. Grades did not change when the course
moved from quarter to semester system, when a plus/minus grade system was
introduced, or when family-gender topics were added. However, in recent years,
perhaps in conjunction with the introduction of CIP theory into the course, the
use of a textbook, and the use of the Internet, grades dropped. The
introduction of the theory and the text brought a slight change to the course
in that external knowledge came more into focus in the course. The
figure-ground relationship between the student-learner and the course contents
changed because the student’s interests, values, skills and goals were no
longer the only subject matter to be learned. In a more practical sense, the
CFA paper accounting for over 30 percent of the grade and reflecting competency
in research and writing, together with poor documentation and the use of
questionable sources on the Internet, combined to lower student grades. This
has occurred in spite of extra attention to instruction regarding high
standards for research writing.
In the process of
reflecting on the way grades are used to assess the student learning in a
course, we became aware of how the design of the course and grading reflect the
philosophy and values of the course designer. How much is a particular learning
event worth? For example, 46% of a student’s grade in this course is based on
two formal papers, a career field research paper (31%) and a strategic career
plan paper (15%). Only 13% of the course grade is based on test results from
reading the textbook. A course grading system might be designed to measure
knowledge recall from the text and lectures, or the completion of learning
activities as in the present course. From a student perspective, it is a matter
of “how much do I want to know” versus “how much do I want to participate or get
involved.” Therefore, in using grades to evaluate a career course intervention,
it is important to fully and carefully describe the nature and content of the
course so replication of research findings can be generalized to other courses.
This understanding would also apply to the use of outcome measures other than
grades. In other words, not all career courses are the same and this should be
recognized in generalizing the results of such research.
Similarly, in
analyzing the results of this study we became aware of the potential benefits
of qualitative research designs in evaluating the impact of a career course.
Such designs might enable us to learn more about students’ perceptions of
course learning activities, their goals in taking a career course, their
readiness to engage the career decision-making process, the impact of selecting
a course grade outcome on motivation and performance. Reardon and Wright (1999)
reported a case study that illustrates the kind of research that would be
helpful. Focus groups aimed at undecided lower division students, graduating
seniors, and students with high or low grade point averages might be helpful in
this regard. What factors contributed to students’ overestimation of expected
grades in the career planning class, e.g., current GPA, academic standing,
negative career thoughts, ethnic/cultural background? Follow-up studies of
students completing a career course would also be helpful.
Brown and Krane (2000), in reviewing a series of meta-analyses, concluded that demonstrably effective career interventions have five components: (1) allow clients to clarify career and life goals in writing; (2) provide clients with individualized interpretations and feedback, e.g., test results; (3) provide current information on the risks and rewards of selected occupations and career fields; (4) include study of models and mentors who demonstrate effective career behavior; and (5) assistance in developing support networks for pursuing career aspirations. Brown and Krane suggest that persons designing and evaluating the impact of career courses should assess the extent to which at least three of the five components are included in the course. We believe that all five of these components are present in this career course, and offer the 16 learning activities in Table 1 as evidence of this.
There is considerable
money involved in career courses. Our quick survey of Amazon.com revealed about
22 published texts for career courses. Many of these texts included
instructor’s manuals, student workbooks, Web-based resources, and the like.
When the student fees for these courses are added to the costs of the
instructional materials, and the large number of institutions reporting the use
of career courses is considered, then the magnitude of the economic impact of
career courses begins to emerge. Information about the effectiveness of these
career materials is not typically provided by the publishers of textbooks, and
few reports have appeared in refereed journals on this matter. Indeed, the publication
of such research raises a variety of ethical questions about journals reporting
studies of career interventions that are proprietary in nature.
Instructors responsible for the design and teaching of career courses have considerable freedom. They can develop a course to reflect their particular content preferences, their desired teaching styles, the special needs of learners, and the results of successful prior work. We are struck by the fact that successful prior work based on research does not seem to be a priority criterion in the design of career course instruction. For example, in selecting a career text and related instructional materials, how often is prior research on the effectiveness of the material used in course development? One option might be for textbook publishers to support research on their instructional materials in the same way that test publishers support studies of psychological tests. Another option might involve the support of universities, government agencies, and foundations for long-term research on the impact of career courses related to student retention, academic performance, and lifelong career development.
References
Arreola, R. A. (1973). A cross-institutional factor structure replication of the Michigan State University SIRS faculty evaluation model. College Student Journal, 7, 38-42.
Balch, W. R. (1992). Effect of class standing on students; prediction of the final exam scores. Teaching of Psychology, 19, 136-141.
Bloom, B. S., Engelhart, M. D., Furst, F. J., Hill, W. H., &
Krathwohl, D. R. (1956). Taxonomy of
educational objectives: Cognitive domain.
Brown, S. D.,
& Krane, N. E. R. (2000). Four (or five) sessions and a cloud of dust: Old assumptions and new observations about
career counseling. In S. B. Brown & R. W. Lent (Eds.), Handbook of counseling psychology (3rd ed., pp.
740-766).
Collins, M. (1998). Snapshot of the profession. Journal of Career Planning & Employment, 41, 32-36, 51-55.
Filer, R. D. (1986). Assigning grades in career planning courses: A neglected issue. Career Development Quarterly, 35, 141-147.
Folsom, B., Peterson,
G., Reardon, R., & Mann, B. (2004-2005). Impact of a career-planning course
on academic performance and graduation. Journal of College Retention, 6,
461-473. Also available at Folsom, B., Peterson, G., Reardon, R., &
Mann, B. (2001). Career course effects on
academic behavior & retention. (Technical Report. No. 34).
Folsom, B., & Reardon, R. (2003). College career
courses: Design and accountability. Journal
of Career Assessment, 11, 421-450.
Folsom, B.,
Reardon, R., & Lee, D. (June 28,
2005). The
effects of college career courses on learner outputs and outcomes (Technical Report No. 44).
Gerken, D., Reardon, R., & Bash, R. (1988). Revitalizing a career course: The gender roles infusion. Journal of Career Development, 14, 269-278.
Halasz, T. J., &
Kempton, C. B. (2000). Career planning courses and workshops (pp. 157-170). In
D. A. Luzzo (Ed.), Career counseling of
college students: An empirical guide to strategies that work.