The Effect of a
Workbook Intervention on
College Students’ Reframes of Dysfunctional Career Thoughts:
Technical Report 37
Darrin L. Carr
February 27, 2004
Center
for the Study of Technology in Counseling and Career
Development
Dunlap Success Center, 100 S Woodward Avenue,
The Florida State University
Tallahassee, FL 32306-4162
(850) 644-6431 (voice) (850) 644-3273 (FAX)
http://www.career.fsu.edu/techcenter
Copyright
© 2004 by Florida
State University
All Rights Reserved
Darrin
L. Carr is a Research Assistant, Instructor, and Career Advisor at The Career
Center and a doctoral student in the Combined Program in Counseling Psychology
and School Psychology at The Florida State University. James P. Sampson, Jr. is
a Professor in the Department of Educational Psychology and Learning Systems
and co-director of the Center for the Study of Technology in Counseling and
Career Development at The Florida State University. The author wishes to thank
James P. Sampson, Jr., Gary W. Peterson, Robert C. Reardon, Janet G. Lenz, Page
Purgar, James Murray, Scott Arkin, and Selah Rhoden for their support of and
participation in this evaluation. Furthermore, this report would not have been
possible without the data rating efforts of Beth Kegler, Katie Meyer, and Jon
Shy. Inquiries and comments should be addressed to Darrin Carr, at dcarr@admin.fsu.edu.
Table of Contents
Abstract
This technical report documents the results from a
quasi-experimental evaluation of the efficacy of Improving Your Career Thoughts: A Workbook for the Career Thoughts
Inventory on college students’ acquisition of the skill of reframing
dysfunctional career thoughts. Participants in this evaluation study were 86
undergraduate students enrolled in five sections of a career development course
at a large southeastern university. A repeated measures ANOVA of effectiveness
of reframed dysfunctional statements by initial level of dysfunctional career
thinking and treatment status was conducted. A significant main effect and
interaction between time and treatment status was found on an initial posttest
measure. However, no effect or interaction was found for level of dysfunctional
career thoughts on reframe effectiveness. At a two week, delayed posttest, a
significant decay effect occurred for the treatment group. Furthermore,
possible demand characteristics and treatment diffusion led to a significant
improvement in the effectiveness of reframed thoughts of the control group.
The
Effect of a Workbook Intervention on
College Students’ Reframes of Dysfunctional Career Thoughts:
Technical Report 37
Most clients are unaware of the biases, misinformation,
and distorted beliefs that they bring to career choice and that these
presuppositions can lead to self-defeating and disabling experiences (Krumboltz, 1983). It is thought that these
beliefs exist due to limited learning opportunities and act to limit additional
learning (Krumboltz &
Jackson, 1993).
For example, Elliott (1995) noted that negative
self-statements can impair a client’s ability to utilize occupational
information, lead to career indecision, and inappropriate choices.
Anecdotal and empirical evidence suggest the existence of
such dysfunctional career beliefs and thoughts in both adolescence and
adulthood. This phenomenon has been labeled by various professionals as myths (Dorn & Welch, 1985), self-defeating assumptions (Dryden, 1979),
dysfunctional career beliefs (Krumboltz, 1990), and dysfunctional cognitions (Corbishley & Yost, 1989). Furthermore, Dorn and Welch (1985) found that high school
students subscribed to various (nine of 13 postulated) career myths as measured
by the Survey of Career Attitudes.
Similarly, Krumboltz has standardized the Career
Beliefs Inventory on a wide variety of groups, providing further evidence
of dysfunctional beliefs and thoughts occurring independent of group membership
(Krumboltz, 1994).
In response to the evidence of both the negative impact
and commonness of dysfunctional thinking on career choice, the Career Thoughts Inventory (CTI) and a
companion intervention Improving Your
Career Thoughts: A Workbook for the Career Thoughts Inventory (Workbook)
were developed (Sampson, et al., 1996a, 1996b). While the CTI has been used in
research since its release (Vernick, 1999; Sampson,
Reardon, Peterson, & Lenz, 2004), the associated Workbook has
received little attention in the literature. Specifically, Gilbert (1996) and Fontaine (2001) noted the lack of evidence in
the CTI professional manual of the workbook’s ability to correct dysfunctional
career thinking or improve career problem-solving and decision-making skills. A
search of the literature also revealed no formal, empirical evidence of its
efficacy.
This gap in the literature may be a specific instance of a
more general lack of evidence on the efficacy of cognitive-behaviorally based
interventions in teaching the skill of cognitive reframing. Instruction in
reframing, by which clients learn to alter their dysfunctional thoughts to more
functional ones, is often an important part of cognitive restructuring
treatments for depression and anxiety. However, much of the focus in the
literature has been on outcome measures (e.g., degree of change in depression,
anxiety, career decidedness, vocational identity, etc.) of treatment and not on
the intervening learning outputs that occur.
Therefore, this evaluation study attempts to fill the gap
in the literature on the efficacy of the CTI Workbook, specifically
investigating the effect of a cognitive behaviorally-based workbook
intervention on college students’ skills to effectively reframe dysfunctional
career thoughts. In the process, it is hoped that more general information on
effective instruction related to cognitive reframing skills will also be
gained. To achieve these goals, this paper will first briefly review the
relevant theoretical and empirical literature and state the hypothesis of
interest. Next, the method of investigation will be outlined, including a
description of participants, instrumentation, and study design and procedure.
Then, results will be presented and the paper will close with a discussion of
study findings, their limitations, and resulting implications.
Review of the Literature
This section of the paper will review two theories
underlying the CTI and Workbook; cognitive theory and cognitive information
processing theory of career decision-making. Also the literature on the
efficacy of interventions associated with each theory will be summarized.
Furthermore, a model for evaluating the effectiveness of reframed dysfunctional
thoughts based in both clinical experience and the theoretical literature will
be introduced.
Cognitive Theory and Related Interventions
Assumptions. A basic assumption of cognitive theory is that
individuals use information processing to represent themselves and the world in
cognitive organizational structures known as schemata (Bartlett, 1958). These schemata, which
represent individuals’ beliefs or assumptions about themselves and the world,
yield the automatic thoughts of consciousness, which interact with affect and
behavior. However, these thoughts can be vulnerable to systematic distortions
which bias the processing of information from the environment (Beck, 1967; Beck, 1976;
Beck, Emery, & Greenberg, 1979). Cognitive theory conceptualizes
psychopathology (e.g., anxiety or depression) as a dysfunctional bias in the
content, amount, or function of an individual’s thoughts as well as the
monitoring and control (metacognition) of those thoughts (Beck & Weishaar,
2000; Wells, 2000).
Furthermore, cognitive theory generally accepts the important role of
physiological factors (e.g., neurotransmitter levels) as well as dysfunctional
cognition in contributing to vulnerability to psychopathology (Strunk, 2001).
Cognitive theory has led to the development of two related
kinds of interventions: self-instruction training and cognitive restructuring.
Self-instruction training focuses on the proactive learning of metacognitive
skills necessary to perform a novel task (Meichenbaum, 1974). Cognitive restructuring
focuses on actively changing the preexisting schema underlying automatic
thoughts that may be inhibiting performance of a specific task (Kinnier &
Krumboltz, 1986).
It should be noted that the distinction between the two interventions may be
somewhat artificial, given that the two may be used interchangeably by both
counselor and client.
Self-Instruction
Training. In response to the limitations of behavioral techniques and to
foster learning that was more durable and generalizable, Meichenbaum (1977) developed a skills-oriented
therapy in which learners were instructed in how to “talk to themselves” in
preparing for a stressor, confront and handle the stressor, cope with the
feeling of being overwhelmed, and reinforce progress by use of these self-statements.
This cognitive-behavioral training followed a sequence similar to Luria’s functional interiorization of language process
for children (Luria, 1962). The training consisted of
cognitive modeling by an expert, then overt, external guidance by an expert
while the learner performed the task, then overt, self-guidance by the learner,
then faded, overt self-guidance by the learner, and finally covert
self-guidance by the learner. “The focus was not to teach the child [learner]
what to think but how to think” (Meichenbaum &
Cameron, 1974, p. 410).
In short, Meichenbaum developed a process by which metacognition could be
taught.
Self-instruction
training has been found to be effective with a wide variety of ages and
problem domains. Through self-instruction training impulsive hyperactive
children have developed greater self control and greater cognitive reflectivity
(Meichenbaum &
Goodman, 1971);
in-patients with schizophrenia improved on measures of interview behavior,
proverb and inkblot responses, and measures of attention (Meichenbaum &
Cameron, 1974);
and college undergraduates significantly increased flexibility and originality
on measures of divergent thinking relative to a control group (Meichenbaum, 1975). Also, standard behavior
therapy procedures augmented with self-instruction have produced greater
treatment efficacy, more generalization, and greater persistence of treatment
effects than behavior therapy alone (Meichenbaum &
Cameron, 1974).
Furthermore, Dush, Hirt, and
Schroeder (1983, 1989) conducted two meta-analyses of the efficacy literature
on self-statement modification. Their first meta-analysis of 69 controlled studies
demonstrated that self-instruction combined with other cognitive methodologies
is an effective technique. A second meta-analysis of 48 controlled studies of
the effects of self-statement modification in children was less conclusive. A
more recent review of the literature concluded that studies on self-instruction
were confounded by the presence of more than one treatment technique (e.g.,
progressive relaxation for treatment of anxiety) (Lange, Richard, Gest,
de Vries, & Lodder, 1998).
Cognitive restructuring.
Bandura (1977) has stated that one of the
most effective ways to change cognitions is to change performance (behavior).
Based on this premise, cognitive restructuring uses experience to create
feedback loops, which challenge dysfunctional automatic thoughts and
assumptions and increase positive feelings. The desired outcome of cognitive
restructuring is increased self-control by enabling more accurate processing of
information and interpretation of events. “The ultimate goal in cognitive
restructuring interventions should be to teach clients the art of auto cognitive restructuring. Clients
should be trained to recognize, examine, and modify or refute maladaptive
cognitions whenever they emerge in the future” (Kinnier &
Krumboltz, 1986, p. 314).
Cognitive restructuring has become popular in the
self-help literature. Burns (1999) offers The Feeling Good Handbook which emphasizes ten forms of twisted thinking and ten methods of untwisting thinking. Also, Bourne (1990) has authored a workbook for
mistaken beliefs associated with anxiety and phobia. This workbook employs
affirmations (short, easily repeated positive statements that counter mistaken
beliefs) that are used to habituate positive thinking. Bourne suggests that the
power of affirmation can be increased through repetition and injection of
feeling.
Given the popularity of such interventions, it is
fortunate that the empirical literature suggests that cognitive restructuring
is effective in a wide variety of clinical applications. Such applications
include the reduction of panic attacks (Salkovskis, Clark,
& Hamel, 1991),
mitigation of disruptions in public speaking associated with perfectionism (DiBartolo, Frost,
Dixon, & Almodovar, 2001), and improvement in inmate institutional
behaviors (Baro, 1999). As with self-instruction
training, one of the difficulties in determining the efficacy of cognitive
restructuring is that several techniques are often used concurrently during
cognitive-behavioral therapy.
Jacobson and Dobson (2000) attempted to isolate the
effect of the three major components of cognitive-behavioral therapy
(activation of behavior, modification of automatic thoughts, and modification
of core schemata) on therapeutic outcomes, as measured by several screening
instruments including the Beck Depression Inventory at treatment, termination,
and six month follow-up, for 150 outpatients with major depression. Despite
adherence to treatment regimens, superior outcomes could not be attributed to
either the modification of automatic thoughts or core schemata conditions over
the behavioral activation condition. Thus, while cognitive restructuring may
have been found to be effective for specific problem domains, the exact
mechanisms that contribute to success may not always be clear.
Regardless of the active mechanism of cognitive therapy,
an important factor in the effectiveness of self-instruction training and
cognitive restructuring interventions may be the client’s belief that they will
work. Meichenbaum (1977) demonstrated that the
difference between subjects who successfully tolerated a cold pressor test and
those who did not was in their belief that they could successfully use coping
strategies rather than their actual use of self-statements and images. Thus, it
may not be enough for a person to learn coping strategies. The individual must
also possess the metacognitive awareness of their coping strategies, know when
and how to use the strategies, and understand the potential, beneficial
outcomes of their application.
In review, self-instruction
or self-statement training is the proactive integration of metacognitive
knowledge and process into instruction for a given performance task through a
variety of techniques. With careful planning, counselors can help clients learn
novel tasks and accompanying facilitative self-talk and then generalize this
new knowledge to other performance situations. Cognitive restructuring is the effort to change a client’s
preexisting negative thoughts and dysfunctional schemata through experience.
Various forms of cognitive restructuring have been widely circulated in the
self-help literature. While successful in specific applications, the exact
mechanism of action of cognitive restructuring is unclear because efficacy
studies often combine behavioral with cognitive treatments. In general,
cognitive theory can inform therapy that is preventative in nature (e.g.,
self-instruction training) or palliative (e.g., cognitive restructuring). Next,
this paper will turn to the specific content domain of career choice and how
cognitive theory and therapy can be combined with the CIP approach to assist
clients with career choice.
The CIP Approach and the CTI Workbook
Assumptions. The
Cognitive Information Processing Approach to Career Problem Solving and
Decision Making (CIP) assumes that career choice involves complex problem
solving, which can be vulnerable to dysfunctional cognitions (Sampson et al., 2004). These dysfunctional
cognitions can interfere in the career choice process and impact career
decidedness. Indecisive individuals can be characterized by their high degree
of anxiety, while undecided individuals may be developmentally immature. In
both cases, engaging in learning experiences may help to modify preexisting
schemata, thereby reducing dysfunctional cognitions and anxiety and enabling
growth and movement.
Krumboltz (Krumboltz, 1976;
Krumboltz, Mitchell, & Jones, 1976; Krumboltz & Jackson, 1993) noted the importance of using
assessment not only for matching and evaluation purposes in career assessment,
but also to promote learning in career decision making. Thus, assessment
becomes the mediator between identifying values, interests, skills, and beliefs
based upon past experiences and identifying needs that can be fulfilled by
future learning. It is through learning that clients can begin to “unblock”
their problem-solving attempts, develop new self-knowledge, and move toward
fulfilling career goals.
In keeping with Krumboltz’s emphasis on learning, CIP
identifies the content and process of career choice while emphasizing the
important role of metacognitions (Peterson, Sampson,
& Reardon, 1991; Peterson, Sampson, Lenz, & Reardon, 2002; Sampson et
al., 2004).
This approach is illustrated by two main constructs the Pyramid of Information Processing Domains and the CASVE Cycle.
The Pyramid of
Information Processing is comprised of three content domains; knowledge,
decision making, and executive processing. The Knowledge Domain is further divided into areas of self-knowledge
(e.g., values, interests, skills, and employment preferences) and options
knowledge (e.g., knowledge of specific options and a schema for organizing
options). The Decision-Making Domain
of the pyramid contains the CASVE cycle,
the process component of CIP Theory. The Executive
Processing Domain of the pyramid emphasizes the importance of the
metacognitive skills of self-awareness, monitoring and control, and self-talk
in career problem solving and decision making.
The five step decision-making or CASVE cycle is defined
as: 1) Communication (understanding
internal and external cues), 2) Analysis
(clarifying self, option, decision making, and metacognitive knowledge), 3) Synthesis (elaborating and crystallizing
options), and 4) Valuing (arriving at
a tentative primary and secondary choice after weighing the costs and benefits
of and prioritizing each option relative to self, significant others, cultural
group, and community/society at large), and 5) Execution (creating a plan for the pursuing the tentative choice).
The cycle completes with a return to the Communication
stage, where internal and external cues are checked to see if the problem (or
gap) has been resolved. (Peterson et al., 1991;
Reardon, Lenz, Sampson, & Peterson, 2000; Peterson et al., 2002; Sampson et
al., 2004).
The intervention of focus in this study, the Career
Thoughts Inventory and Workbook, is grounded in the CIP approach as each
inventory item corresponds to one of the eight elements of the pyramid of
decision-making domains and CASVE cycle. The inventory and workbook are also
rooted in the tradition of cognitive theory and interventions, applying both
self-statement instruction and cognitive restructuring. The CTI, workbook, and
related literature will now be examined.
Career Thoughts
Inventory and Workbook. The CTI, is a self-administered and objectively
scored measure of content and degree of dysfunctional career thinking. The
total score of the CTI has been found to be both reliable and valid for a
college student sample. Sampson, et al. (Sampson, Peterson,
Lenz, Reardon, & Saunders, 1996) reported internal consistency
coefficients for the total score scale ranging from .93 to .97. Stability for
total scores was also adequate as measured by four-week test-retest stability (r = .86). The CTI items and scales also
possess content, construct, and criterion-related validity. The CTI has
demonstrated convergent validity with several other instruments including the
identity scale and occupational information and barriers items of the My Vocational Situation (MVS)
questionnaire (Holland, Daiger, &
Power, 1980)
and the Neuroticism domain of the NEO
PI-R (including Anxiety, Angry Hostility, Depression, Self-Consciousness,
Impulsiveness, and Vulnerability) (NEO PI-R) (Costa & McCrae,
1992).
Additional data on CTI factor scores and corresponding reliability and validity
can be found in Sampson, et al. (1996).
The Improving your
Career Thoughts Workbook (Workbook) is a therapeutic companion to the
Career Thoughts inventory, which “…is intended to assist individuals in
interpreting their CTI scores and in completing the cognitive restructuring,
action planning, and learning necessary to effectively engage in exploratory,
problem solving, and decision making behaviors” (Sampson, et al., 1996b, p.
15). Using the metaphor of a wall to represent dysfunctional thinking, the
workbook offers four sections intended to help the individual to understand the
relationship between dysfunctional thoughts and the need for additional
supports. These sections help the client to: 1) identify the degree and content of their dysfunctional career
thinking, 2) challenge such thinking,
3) alter their thoughts to be more
adaptive, and 4) create a plan to act
on new, more adaptive thoughts. This study attempted to isolate the effect of
the second and third sections of the workbook, which employ stimulus statements
(i.e., short paragraphs of counseling feedback) and a written response exercise
to promote cognitive reframing. In order to accomplish this task, a model for
evaluating the effectiveness or “quality” of individual participant’s cognitive
reframes was necessary.
A Model for Evaluating Reframe Effectiveness.
In an effort to operationally define reframe
effectiveness, Carr (2003) conducted think aloud
protocols with four expert, doctoral level career counselors. The task of these
experts was to evaluate the efficacy of presented reframes of negative career
thoughts and to relate both the reasoning behind and evidence for their
judgments. A qualitative analysis of the resulting data yielded the model
available in Table 1. This model holds that more effective
cognitive reframes are characterized by an internal locus of control, explicit
detail, acknowledgement of affect, sensitivity to time, selection of positive
words, accurate problem definition, and a structure parallel to that of the original
negative thought. Space limitations prohibit a thorough discussion of the
procedure used to construct this model. However, this information as well a
review of the model’s grounding in the theoretical and empirical literature is
available (Carr, 2003)
Summary of Literature and Research Hypothesis
In summary, the CTI (a measure of dysfunctional career
thinking), and the companion workbook (an intervention that applies both self
statement modification and cognitive restructuring techniques) attempt to identify
and alleviate dysfunctional career thoughts through the application of
cognitive theory and the CIP approach to career decision making. While the CTI
has been found to be a valid and reliable instrument, the efficacy of the
Workbook has not been determined. However, interventions similar to the CTI
Workbook have been found to be effective in a wide variety of clinical domains.
Furthermore, meta-analytic studies of the literature have
noted that efforts to determine the efficacy of cognitive techniques have been
stymied by the inclusion of both cognitive and behavioral elements in
treatment. The literature also appears to have focused on the outcome of
interventions (e.g., lowered anxiety and depression), not the actual learning
outputs that may lead to these outcomes. Finally, given past findings that
dysfunctional thoughts may interfere with knowledge acquisition and schema
modification, it follows that the initial level of dysfunctional career
thinking may also impact the learning of the cognitive reframing skill. Thus,
the primary purpose of this study was to determine the effectiveness of the CTI
Workbook in imparting the skill of cognitive reframing to college students. Of
additional interest was obtaining information about the process of instructing
the skill of cognitive reframing and the efficacy of purely cognitively based
techniques. In support of these research goals, it was hypothesized that there
would be no significant interaction between experimental group status (i.e.,
treatment vs. control) and level of dysfunctional career thinking (e.g., high
vs. low) over time and the college students’ acquisition of the skill of
reframing dysfunctional career thoughts as measured by reframe effectiveness.
Method
The study method will now be described including
participant recruitment, instrumentation, study design and procedure, and rater
training and scoring.
Participant Recruitment
Participants were recruited from students enrolled in five
sections of a career development course at a major southeastern university.
Each participant provided his or her informed consent (Appendix
A) to participate in the evaluation project and was not required to
participate in the evaluation as part of the class grade. Participants were informed that they were being asked to evaluate
a workbook they might use later in the course. Students who did not wish to
participate were asked to remain in class and use the time to read for class or
complete class assignments. This request was made to maintain the
confidentiality of their decision to not participate in the evaluation.
Descriptive data on participants’ college, ethnicity, year
in school, and sex can be viewed in Table 2. As
would be expected in a career development course, the sample was more undecided
in their major than the undergraduate university population as a whole.
Furthermore, a proportionally greater number of Social Science majors
participated in the evaluation study than in the university at large (22% vs.
10%). The sample was also less diverse than the greater university population
with over 81% of participants self-reporting as Caucasian versus 74% of the
undergraduate university population. The majority (61%) of the sample was of
either sophomore (30%) or senior (31%) class standing. Furthermore, the sample
also included proportionally more women than the university population (66% vs.
56%). Finally, participants’ ages ranged from 18 to 31 years with an average
age of 20 years (σ = 2.41) as compared to an average age of
22.2 years for the university undergraduate population.
Instrumentation
A self-report Student Data Sheet (Appendix
B) regularly collected by class instructors at the beginning of each
semester was used to gather archival information on participant sex, ethnicity,
intended major, number of credit hours enrolled, and satisfaction with first
occupational choice. Participants’ career decidedness level was also determined
from the data sheet through the Occupational Alternatives Question (OAQ) (Slaney, 1980). This question asked
participants to list all of the occupations they were considering and their
first choice occupation. The OAQ was
scored as 1 if a first occupational choice was listed with no alternatives; 2
if a first choice was listed with alternatives; 3 if no first was choice
listed, just alternatives; and 4 if neither an occupational choice or
alternatives were listed. Additional data on first time in college vs.
transfer status and year in school was collected using a separate smaller,
demographic form attached to the Student Data Sheet.
Both the 48-item Career
Thoughts Inventory (CTI) and Improving
your Career Thoughts: A Workbook for the Career Thoughts Inventory (Workbook)
(Sampson, et al., 1996a, 1996b) described previously served as sources for the
instruments created for this evaluation. For this study, the 48 items of the
CTI were divided into a 32-item screening instrument, an eight-item pretest/treatment/posttest
instrument packet, and an eight-item delayed posttest instrument (Appendix C). The first 32-item instrument was used to
establish the level of dysfunctional career thoughts present among
participants. The second instrument packet (based on items 33 through 41) was
used to establish a baseline of the participants’ skill at reframing
dysfunctional career thoughts, and to provide training to the treatment group
in challenging and altering dysfunctional thoughts. Two versions of this packet
were created: 1) a control version without the instructions and reframe
statements of the workbook; 2) a treatment version containing instructions and
reframe stimulus statements from the actual CTI Workbook (Appendix
C). For the control group, the simple instruction “Make these sentences
more positive” was provided along with stimulus statements of general and
neutral “advice” that participants might receive from non-practitioners. A
third instrument comprised of CTI items 42 through 48 was used with both
treatment and control to collect data on participant retention of the reframing
skill two weeks after training (Appendix C).
Design and Procedure
A double blind, quasi-experimental
design was applied during this study. Selection occurred by students enrolling
in a career development class and then volunteering to participate in the
evaluation study. As random assignment of individual participants was not
practical, two of five course sections were randomly assigned to treatment
status. Data were collected during the final 20 minutes of the class at three
points in time over a 22-day period in the Spring of 2002. At prearranged times
(Appendix D), course instructors left the room and one
of four confederates (2 Caucasian males and 2 Caucasian females), reading from
scripts (Appendix E), employed the previously described
instrumentation to collect data from participants. Both confederates and
participants were blind to the hypotheses of the study and to their treatment
or control status. Due to staff shortages, one confederate collected data from
two different course sections of the control group. Each confederate collected
data at all three times, with the exception of one who had to be replaced due
to a scheduling conflict.
The first data collection was conducted during the second
week of class. After establishing informed consent, the paper and pencil
demographic form was completed and the 32-item screening instrument was
administered using an optical scan form. The later forms were then scanned and
imported in a database to obtain a median split and identify high scoring
participants on the CTI (i.e., those individuals with higher levels of
dysfunctional career thinking) for later data analysis.
One week later, confederates conducted the second data
collection. Both treatment and control groups were first presented with brief
reframing (i.e., “rewrite”) instructions. Next, items 33–40 of the CTI labeled
as “Old Thought” were presented and participants were invited to cognitively
restructure thoughts from participants labeled as “New Thought.” At this point
the treatment group received actual reframing stimulus statements corresponding
to items 33 to 40, while the control group received paragraphs of “plausible
neutral advice” such as one might receive from a friend. Often this advice
focused on an obscure feature of the original item. Participants were again
provided an opportunity to reframe items 33-40 of the CTI.
Two weeks later, a
third and final data collection was conducted. A brief prompt to reframe
the eight provided dysfunctional thoughts, identical to those in the pretest
during the second data collection, was presented to both treatment and control
groups. At the end of data collection, confederates announced to participants
that the evaluation was complete and provided debriefing materials that
explained the project’s purpose and their membership in either the treatment or
control group (Appendix F). In addition, each
participant was provided with a complete 48-item Career Thoughts Inventory to complete as a regularly occurring
course assignment. All participants later individually discussed their
inventories with their instructors. Those students endorsing a high degree of
dysfunctional career thinking on the 48-item inventory were assigned the
complete Career Thoughts Inventory
Workbook by their instructor.
Rater Training and Data
Rating
Raw data resulting from this procedure were 2,064
reframes. These reframes were typed, verbatim (i.e., grammar and spelling
mistakes included) into a Microsoft Access database for later evaluation and
rating. Two primary raters (a female, masters student in career counseling and
a male, doctoral student in counseling psychology) and a third “back-up” rater
(a female, doctoral student in counseling psychology) volunteered to complete
this task. Volunteers were Caucasian and ranged in age from 24 to 26 years.
Each volunteer was trained to apply the model created by Carr (2003) to globally rate reframes as
to their effectiveness in supporting career decision making (Table
1).
This training familiarized raters with the model and how
to reliably apply it to reframed statements using an unbalanced, four point
rating scale. On this scale, “less effective” reframes received a rating of –1
while “more effective” reframes received ratings of +2. Those reframes which
were thought to have “no change,” for which the “task was not understood,” or
the “item did not apply” to the participant received a rating of 0. Reframes in
which partial, positive change was perceived to have occurred by the rater
received a rating of +1. It was emphasized throughout the training that “global
judgments” of reframe effectiveness were desired and raters were discouraged
from “adding” up criteria to arrive at their ratings.
Rater training was delivered via a web-based resource that
allowed for identical delivery of content but flexibility in location and
timing of instruction (Appendix G). Duration of
training for each rater ranged from approximately 1.5 to 2 hours. First, raters
were shown a CTI item paired with four possible reframes and asked to place
each reframe according to the previously described four point scale. Next, for
each of the four reframes, raters checked the model criteria they believed were
applicable. Raters were then given feedback comparing their perceptions of
applicable model criteria to those of the investigator. Each CTI item was then
summarized by positioning the four reframes and their matching criteria from
the model along the rating scale. After a brief stretch break, this procedure
was repeated a second time for a different CTI item and reframes.
At the end of training, raters were shown a “learning
review” which summarized the major points of the training. Raters were then
asked to apply their learning by rating 24 practice reframes derived from
incomplete data records previously culled from the larger dataset. As each
rater finished training, a notification email was sent to the investigator who
debriefed the rater about their experience and reviewed the summary data.
Analysis of this practice data using a weighted kappa (a
measure of agreement with provision for scaled disagreement or partial credit)
yielded modest reliability coefficients (Table 3) (Cohen, 1968). However, the level of
agreement between rater 2 and rater 3 was notably less than that of rater 1
with either rater 2 or 3. Given the “tie breaker” role to be played by rater 3,
this difference was considered acceptable. While interrater reliability was
found to be modestly acceptable, a significant oversight by the investigator
during training and subsequent data rating process was the establishment of
intrarater reliability. This could have been achieved by repeating the
administration of practice items after an intervening period of time to account
for memory.
To evaluate the reframed statements, raters used a similar
Web-based application to rate each individual reframe according to the provided
model (Appendix H). Each of the 2,064 reframes and
its corresponding CTI item was presented in a random sequence, thus raters were
blind to participants’ identities, preexisting levels of dysfunctional career
thoughts, and treatment status. After evaluating the reframe using the model
and rating guidelines, which could be easily referenced on screen, raters used
the four point Likert scale to rate the quality of the reframe. For any given
reframe, if judgments by the two primary raters differed by more than one scale
position, the third rater was asked to review the statement.
Given the schedules of the student raters, the rating
process took approximately three months to complete. When the process was
finished, ratings for each reframe were averaged by summing the raters’
judgments and dividing by the number of raters required to arrive at a
decision. Ratings for the eight items on each instrument administered at
pretest, posttest, and posttest + two weeks were then averaged, creating three
measures of the degree to which participants successfully applied the reframing
skill over time.
Results
A repeated measures ANOVA with between-subjects factors
was conducted to perform an omnibus test of the hypothesis. For this analysis,
the within-subjects factor was the average effectiveness of career thoughts as
determined by the rating panel. The between-subjects factors were assigned
treatment status (e.g., treatment or control group) and initial level of
dysfunctional career thinking (e.g., high or low) as measured by the screening
version of the CTI. The preliminary analyses required for executing this test
will now be detailed, followed by a review of the findings from the omnibus,
individual factor, and pair wise tests.
Preliminary Analyses
Rater Agreement.
Out of the raters’ 2,064 judgments of effectiveness of reframed dysfunctional
thoughts, 1,367 (66%) were exact agreement and 548 (27%) were partial
agreement. Raters disagreed on only 150 ratings (7%) or extremely disagreed
(i.e., opposite ends of the scale) for only 5 ratings (.2%) thus requiring
input from the third rater. The weighted Kappa coefficient for the entire
sample for the two primary raters was .43 (p
< .001) indicated a modest amount of agreement between the two primary
raters.
As the dependent
measure for this study was decided a priori to be the mean ratings of the judges, the average intraclass correlation was derived. This
coefficient for a two-way mixed effect model (absolute agreement
definition) was calculated to be .56 (p
< .0001) (Shrout & Fleiss, 1979).
The two-way mixed model was selected
as each of the two primary judges saw all of the reframes, thus they were
considered a fixed effect while the ratings of reframes were a random effect.
Equivalence of
Groups. As nonequivalent cohort groups were employed in this study, tests
were conducted to insure that treatment and control groups were equivalent in
terms of known variables. Pairwise comparisons between treatment and control
using Student Neuman-Keuls’ (Kirk, 1982) revealed there were no significant
differences (p < .05) with respect
to age, sex, self reported credit hours, year in school, lower vs. upper
division status, OAQ, satisfaction with choice, and transfer status (Table 4). Thus, any difference between the treatment and
control groups occurred due to chance factors alone.
Normality of
Distribution. An important assumption of repeated measures ANOVA is
normality of the distribution of the dependent measure. Thus,
Kolmogorov-Smirnov (K-S) tests (Roscoe, 1975) were conducted on the three
measures of reframe effectiveness for the entire sample (n = 86) (Table 5). Distributions were not
normally distributed at pretreatment and two weeks post treatment (K-S = .108, 86, p = .015 and K-S = .118,
86, p = .005 respectively).
Inspection of descriptives for the pretest measure indicated a positively
skewed and somewhat leptokurtic distribution, while the delayed posttest
measure distribution was less positively skewed, but more leptokurtic. Scrutiny
of data indicated that outliers existed three standard deviations above the
mean at pretest and delayed posttest; thus, these two participants were removed
from the analysis. Unfortunately, an additional K-S test indicated continued
non-normality (K-S = .098, 84, p = .045 and K-S = .114, 84, p = .009)
(Table 6). However, inspection of descriptives of this
smaller sample’s pretest and delayed measures measure indicated a reduction in
skewness and kurtosis.
In an effort to continue to improve normality, square root
and logarithmic transformations were applied to both dependent measures. These
additional K-S tests indicated poorer fit to the normal curve after
transformation (p = .000 to p = .039). When sample data were
separated by treatment and control status, pretest and delayed posttest measures
were found to be normally distributed (Table 7 and Table 8). Given the failure of data transformation to
improve normality, the improvement in skewness and kurtosis of the total sample
with the elimination of the outliers, and a return to normality when the sample
was divided into treatment and control groups, it was decided that the analysis
of untransformed dependent measures from the smaller data set could continue.
However, the outcomes of the ensuing ANOVAs would need to be interpreted with
caution due to the possibility of a violation of normality.
Sphericity and
Homogeneity of Variance. As measures are repeated they are considered by
default to not be independent; thus, Mauchly’s test of sphericity (Edwards,
1993) was performed to insure equal correlations among treatment pairs.
Furthermore, Levene’s test (Draper, 1998) for homogeneity of variance was
consulted at appropriate times. No violations of these assumptions were
indicated at anytime throughout the analysis and specific results are included
in the following findings.
Findings
The repeated measures ANOVA multivariate test indicated
that while time and the interaction of time and treatment status captured
significant amounts of the variance in reframe effectiveness, initial level of
dysfunctional career thinking did not (Table 9). Time of
data collection accounted for 28.7% of the variance in effectiveness of
reframed dysfunctional career thoughts (F
= 15.726, 2, p < .001). The
interaction between time and treatment status accounted for 19.2% of the
variance in reframed dysfunctional career thoughts (F = 9.273, 2, p <
.001). Initial level of dysfunctional career thinking, as measured by the CTI
screener, was not found to capture significant variance when combined with time
and/or treatment status (1.4% of variance, F
= .568, 2, p = .569 and .7% of
variance, F = .280, 2, p = .757 respectively). Furthermore, the
assumption of equal correlations among treatment pairs of the dependent measure
was met (W = .989, C 2 = .835, 2, p = .659).
As the variable of initial level of dysfunctional career
thoughts did not demonstrate a statistically significant effect, it was removed
from the analysis. This decision dictated the inclusion of an additional
subject for which CTI screening data was missing. Thus, an additional repeated
measures ANOVA with only treatment status as the between subjects variable and
time as the within subjects variable was conducted on the entire sample (Table 10). This yielded differences in the amount of
variance accounted for by the independent variables. In this new analysis, time
accounted for 29% of the variance in effectiveness of reframed dysfunctional
career thoughts (F = 16.546, 2, p < .001) and the interaction between
time and treatment status accounted for 18.2% of the variance in reframed
dysfunctional career thoughts (F =
9.016, 2, p < .001). Again, the
assumption of equal correlations among treatment pairs of the dependent measure
was met (W = .987, C 2 = 1.031, 2, p = .59).
Additional testing using ANOVA’s with Student-Neuman-Keuls
control of family wise error confirmed the within subjects effect of time (F =
16.726, p < .001) and interaction
of Time and Treatment status (F =
9.655, p < .001). Furthermore,
post hoc contrasts between pretest and posttest and posttest and delayed
posttest were also found to be significant (p
< .01 or better) (Table 11).
Given the interaction found between Time and Group,
individual post hoc tests of reframe effectiveness between groups at each time
were conducted using Student-Neuman-Keuls (Table 12).
Levene's test for equality of error variances across dependent measures was met
for these tests (smallest p >
.145). No significant differences were found between treatment and control
groups at pretest or delayed posttest (F
= 0.94, 1, p = .760 and F = .382, 1, p = .538 respectively). However, a significant difference in
reframe effectiveness was found between treatment and control conditions on
immediate post-test (F = 12.371, 1, p = .001). This difference was also
found to practically significant with a moderate effect size of .77.
Post hoc contrasts of reframe effectiveness within control
and treatment groups across time are summarized in Table 13
and Table 14 respectively. Here, practically significant
effect sizes of .51 and .73 (p <
.01 or better) were found between posttest and delayed posttest and pretest and
delayed posttest for the control group. However, the contrast between pretest
and posttest for the control group was not found to be statistically
significant. For the treatment group, statistically and practically significant
differences occurred among all times. A large initial effect size of 1.0 was
found between pretest and posttest (p
< .001). However, a negative effect size of -.40 was found between posttest
and delayed posttest (p = .021)
indicating a loss in the effectiveness of the treatment group’s reframing
skill. Thus, an overall effect size between pretest and delayed posttest of .60
was found (p = .001).
The findings of this study are summarized by Figure 1.
Control and treatment groups started at pretest with statistically equivalent
positions, with control group generating reframes of slightly greater
effectiveness than treatment group participants. An immediate, after treatment
posttest indicated a statistically and practically significant difference in
the quality of reframes generated by the treatment group versus the control
group. Two weeks after treatment, the difference in groups diminished as
quality of reframes generated by the control group again slightly exceeded
those of the treatment group by a statistically insignificant amount. This was
due to a significant decrease in effectiveness of reframes generated by the
treatment group and a significant increase in the effectiveness of the control
group’s reframes.
Discussion
This discussion of the findings of this study will begin
with an acknowledgement of possible limitations, continue with an
interpretation of results, and conclude with potential implications for
research and practice.
Limitations
This study has several limitations in the form of threats
to both internal and external validity. One threat to internal validity is the
possibility of nonequivalence of groups, due to the assignment of cohorts, instead
of individual participants to treatment or control groups (Smith & Glass, 1987). While a difference in these groups may have
contributed to the treatment effect, tests of equivalency on variables thought
to be related to the dependent variable (e.g., level of career decidedness)
yielded no significant differences between groups. Thus, nonequivalence of
groups is a possible, yet unlikely threat to this study’s findings.
A second threat
to internal validity is the possibility of treatment diffusion occurring
after the posttest. While assigning treatment and control groups by cohort
should have limited this possibility, results indicate significant movement of
the control group between posttest and delayed posttest measures. This may be
due in part to all participants (treatment and control) receiving approximately
five clock hours of instruction in the CIP approach (the theoretical
basis of the CTI items used in study) (Peterson, Sampson, & Reardon, 1991;
Peterson, Sampson, Reardon, & Lenz, 1996; Sampson, Lenz, Reardon, &
Peterson, 1999) during the two weeks
intervening the measures. This instruction was delivered using the text,
Career Planning and Development: A
Comprehensive Approach and a related workbook, (Student Manual; Reardon, Lenz, Sampson, & Peterson, 2000).
All sections of the course follow the same syllabus, use
similar materials, and participate in like activities. Between the first and
second data collection participants received instruction and participated in
activities focusing on improvement of self-knowledge, including writing a
career focused autobiography, clarifying their values, and completing the Self-Directed Search (Holland, 1994) and a skills assessment
activity. After the second but before the third data collection, students were
provided opportunities to broaden their occupational knowledge through an
introduction to computer-assisted career guidance systems (e.g., Choices and
SIGI PLUS or Discover). Students also participated in a career center tour and
career library scavenger hunt designed to familiarize them with career
information resources that could be used in later course assignments. Prior to
the third data collection, participants were also instructed in the CASVE cycle
of career problem solving and career decision making. A chapter on “Exploring
Metacognitions”, which covered the influence of metacognitions on career
decision making, was taught after the study was concluded. The data collection
schedule can be viewed in Appendix D.
Reed, et. al. (2001) found significant decreases in scores
on the CTI when it was used as a pre-test and posttest measure with this career
development course. The greatest decrease in negative thinking was found in
students with the highest initial levels of negative thinking. Two factors of
the CTI, decision-making confusion and commitment anxiety, contributed
significantly to the main effect and there were no significant interactions
with race or gender. In short, Reed found that the course (which based upon
individual student needs may or may not have included the CTI Workbook) can
impact levels of dysfunctional career thinking as measured by the CTI. Reed
noted several threats to the validity of this quasiexperimental study,
specifically the absence of a control group. Furthermore, Reed pointed out that
no single element of the full course, which contains some 60 separate elements,
could be attributed with contributing to the observed effect. Thus this course,
which lowered endorsement of dysfunctional career thoughts on the CTI, may also
have transmitted knowledge or skills necessary for reframing of these thoughts,
thereby contaminating the control group in this study.
Possible threats to external validity, which may limit the
generalizability of this study’s findings, are demand characteristics and testing
effect. Demand characteristics are cues in the environment to which
subjects react (Smith & Glass,
1987).
As these cues may not be present, in other environments, the desired effects
may not be created outside of the experimental setting. While care was taken to
avoid the influence of such cues by scripting confederate interactions with
participants, some cuing may simply be unavoidable. For example, the Hawthorne Effect may have impacted this
study. That is, participants may have improved the effectiveness of their reframes
due to just being observed by the investigator’s confederates. Results may also
have been generated by the Testing Effect,
which occurs when participants learn by merely engaging in a task several times
over a brief period. In short, this study may be confounded by several threats
to internal and external validity. However, the results of this study, while potentially
suspect, may still offer some interpretive value.
Interpretation of Results
The overall hypothesis of this study was that there would
be no significant interaction between experimental group status (i.e.,
treatment vs. control) and level of dysfunctional career thinking (e.g., high
vs. low) over time and college students’ acquisition of the skill of reframing
dysfunctional career thoughts as measured by reframe effectiveness. Based upon
the findings of this study, this hypothesis can be rejected. While there was no
effect or interaction of initial level of dysfunctional career thinking on
effectiveness of reframed thoughts, there was a significant effect and
interaction of treatment group and time on reframe effectiveness.
Both treatment and control groups began the study with
near identical skill at reframing dysfunctional career thoughts into more
effective ones. Judges rated their reframing efforts as being only slightly
more effective than no “change at all” (i.e., a score of 0 on a –1 to +2 scale).
Posttest measures showed that the CTI Workbook improved the ability of the
treatment group to create more effective reframes of dysfunctional career
thoughts. This improvement was found to be both a statistically and practically
significant change both in terms of the standard deviation of the distribution
of the measure and in terms of movement on the rating scale. Treatment
participants moved approximately one-half of a rating point on the four-point
rating scale after exposure to CTI Workbook instructions and eight
stimulus-reframing statements. There was no significant movement of the control
group between pretest and posttest.
However, there was an apparent decay in this treatment
effect between posttest and delayed posttest. This may be due to the absence of
the CTI Workbook reframing stimulus statements in the delayed posttest. In
essence, the first posttest for the treatment group demonstrated immediate
skill learning, while the delayed posttest demonstrated reduced performance
when participants were engaged in a generalized, delayed application of the
reframing skill to an alternate problem set. Thus, treatment participants may
have learned the skill given their immediate previous exposure to instruction;
however, some of this learning was lost when applied to a novel situation.
Despite the decay effect, a significant overall improvement in reframe
effectiveness occurred in the treatment group between pretest and delayed
posttest.
During the interval between posttest and delayed posttest,
the control group also improved significantly in their reframing skill. In
fact, the control group finished the study with reframes slightly more
effective than those of the treatment group. As previously discussed, this
finding may be due to the Hawthorne
effect, treatment diffusion, or some other confound. Furthermore, it is
possible that the different item set used in the delayed posttest was somehow
more salient to the control group than the treatment group. Another possible
interpretation is that practice at the skill of reframing in general may be as
important as the CTI Workbook treatment. Finally, another interpretation of these findings may be that
the treatment group was more confident in their ability to complete the
task and became bored with a repetitive assignment. The control group, however,
having less instruction in cognitive reframing, may have been more challenged
by and motivated to engage the final reframing task.
In
conclusion, at immediate posttest the CTI Workbook appears to be effective in instructing
participants in the skill of creating more effective reframes of dysfunctional
career thoughts. However, this effect may decay with time or as the learner is
asked to generalize their learning to other dysfunctional thoughts without the
benefit of supporting reframe stimulus statements. Furthermore, repeated
practice at making a thought “more positive” may lead to an improvement in
effectiveness of reframes of dysfunctional career thoughts regardless of
treatment. The possible implications of these interpretations for research and
practice will now be discussed.
Implications for Research and Practice
The primary purpose of this study was to determine the
effectiveness of the CTI Workbook in conveying the skill of cognitive reframing
to college students, thereby filling a gap in the literature noted in two
reviews of the Career Thoughts Inventory. This gap has now been
partially bridged as results demonstrate that the CTI Workbook can improve
college students’ reframing skill as measured by reframe effectiveness.
However, the question of the CTI Workbook’s effectiveness as a cognitive
restructuring intervention has not been completely resolved as this study only
focused on the cognitive skill of reframing (i.e., self-statement modification)
and excluded the behavioral feedback necessary to achieve cognitive
restructuring.
Thus, in an effort to isolate the skill of cognitive
reframing this study disregarded an assumption of cognitive theory that
modeling is a necessary, but not sufficient, condition for learning.
Self-instructional rehearsal by the learner (acting) is also necessary for
improved task performance. Thus, training tasks should actively involve the
learner and require mental transformation, not simply rote learning and
repetition. Furthermore, training must help learners identify and alter
existing maladaptive thoughts into self-statements incompatible with the
previous ones, rather than just adding more positive self-statements. “What is
required is modeling and practice in synthesizing and internalizing the meaning
of one’s self-statements” (Meichenbaum &
Cameron, 1974, p. 106).
Thus, the Workbook’s behavioral components (i.e., the creation and execution of
Individual Action Plans) should be included in the treatment condition of
further research on its effectiveness. This may reinforce the cognitive
component of the reframing skill, thereby extending and generalizing the
workbook’s effect beyond the immediate and somewhat abstract learning context
of this study.
As this study was an analog of how the CTI and Workbook
are actually applied in counseling, an investigation of the effect of the CTI
on the learning outputs of clients receiving individual case managed services
from counselors may also be helpful. Through the counseling relationship,
clients might be better oriented to and understand the need for completing the
CTI Workbook (e.g., seeing their results compared to group norms), which in
turn may impact client motivation for, and belief in, the intervention. Such a
study could utilize the complete workbook, asking clients to both cognitively
reframe their dysfunctional career thoughts and act on these new thoughts;
thus, possibly resulting in the restructuring of clients’ schema through
behavioral feedback.
Another goal of this study was to obtain information about
factors that may influence the instruction of cognitive reframing. The
improvement in control group reframes over time, may indicate that the eight
items used in the delayed posttest may have somehow been more salient to the
control group than the treatment group. Similarly, the decline in the
effectiveness of the treatment group’s reframes may be due to the limited salience
of the new items. Such a scenario would be important to explore given that
clients apply the CTI Workbook to Career Thoughts Inventory items that
they have “strongly agreed” or “agreed” with (i.e., indicated item saliency).
For example, Lange et al. (1998) designed a treatment that
employed only positive self-instruction statements created by participants. The
goal of the treatment was not to bolster self-esteem by offering the
individuals new information about themselves, but rather to manipulate “…the
salience of a subset of information already present within the person’s own
thought system” (Lange et al., 1998, p.
226).
Participants in the treatment condition showed significantly greater
improvement on measures of self-esteem compared to control. Furthermore,
intrinsically motivated participants showed greater improvement in self-esteem
with respect to extrinsically motivated participants after engaging in
self-instruction. Unfortunately, a separate treatment group of
investigator-created, positive self-instructions to compare against the
efficacy of participant created positive self-instructions was not included in
the study.
To further investigate the effect of item salience it may
be possible to create a computer-based form of the Workbook, which would allow
for treatment based on CTI items endorsed as more negative by individual
participants. Furthermore, treatment could be varied by using the externally
sourced stimulus reframe statements in the CTI Workbook or through internally
sourced stimuli created via instruction of participants in important reframing
concepts consistent with the themes of reframing model (e.g., concreteness,
time sensitivity). Such a study may more accurately simulate the actual manner
in which the CTI Workbook is used and yield data that could inform future
workbook improvements.
Conclusion
This study has shown that the cognitive component of the
CTI Workbook alone can immediately improve the effectiveness of reframes of
dysfunctional career thoughts. However, several questions remain about the
longevity and generalization of learning, and whether or not additional
variables such as salience of dysfunctional thoughts and externality of
reframing stimuli may influence workbook effectiveness. Thus, additional
research should be conducted on the effects of both the cognitive and
behavioral elements of the CTI Workbook. Regardless of the need for additional
research, the cognitive component of the CTI Workbook alone is successful in
instructing the skill of reframing dysfunctional career thoughts. Furthermore,
this part of the workbook is likely to be effective regardless of a client’s
initial level of dysfunctional career thinking. Follow-up studies on the CTI
Workbook might include a more functional study of clients receiving individual
case managed services or the creation of a computer-based version, which allows
manipulation of the variable of item salience.
References
Bandura, A. (1977). Social
learning theory. Englewood
Cliffs, NJ: Prentice-Hall.
Baro, A. (1999). Effects of a cognitive restructuring
program on inmate institutional behavior. Criminal
Justice and Behavior, 26, 466-484.
Bartlett, F. (1958). Thinking:
An experimental and social study. Oxford,
England: Basic
Books.
Beck, A. T. (1967). Depression:
Clinical, experimental, and theoretical aspects. New York: Harper and Row.
Beck, A. T., Emery, G., & Greenberg, R. L. (1979). Cognitive therapy of depression. New York: Guilford.
Beck, A. T., & Weishaar, M. E. (2000). Cognitive therapy.
In R.J.Corsini & D. Wedding (Eds.), Current
psychotherapies (6th ed., pp. 241-272). Itasca, IL:
F. E. Peacock Publishers.
Beck, A. T. (1976). Cognitive
therapy and the emotional disorders. New York:
International University Press.
Bourne, E. J. (1990). The
anxiety and phobia workbook. Oakland,
CA: New Harbinger Publications.
Burns, D.
D. (1999). The feeling good handbook. (revised ed.)
New York.
Carr, D. (2003). A
Model for evaluating the effectiveness of cognitive reframes of dysfunctional
career thoughts (Rep. No. 36). Florida
State University,
Tallahassee, FL: Center for the Study of Technology in
Counseling and Career Development.
Cohen, J. (1968). Weighted kappa: Nominal scale agreement
with provision for scaled disagreement or partial credit. Psychological Bulletin, 70, 213-220.
Corbishley, M. A., & Yost, E. B. (1989). Assessment and
treatment of dysfunctional cognitions in career counseling. Career Planning and Adult Development
Journal, 5, 20-26.
Costa, P. T., Jr., & McCrae, R. R. (1992). Revised NEO Personality Inventory (NEO PI-R)
and NEO Five-Factor Inventory (NEO-FFI): Professional manual. Odessa, FL:
Psychological Assessment Resources.
DiBartolo, P. M., Frost, R. O., Dixon, A., & Almodovar, S. (2001). Can
cognitive restructuring reduce the disruption associated with perfectionistic
concerns? Behavior Therapy, 32,
167-184.
Dorn, F. J., & Welch, N. (1985). Assessing career
mythology: A profile of high school students. The School Counselor, 33, 136-142.
Draper, N., & Smith, H. (1998). Applied regression analysis. New
York: Wiley.
Dryden, W. (1979). Rational-emotive therapy and its
contribution to careers counseling. British
Journal of Guidance and Counselling, 7, 181-187.
Dush, D. M., Hirt, M. L., & Schroeder, H. E. (1983).
Self-statement modification with adults: A meta-analysis. Psychological Bulletin, 94, 408-422.
Dush, D. M., Hirt, M. L., & Schroeder, H. E. (1989).
Self-statement modification in the treatment of child behavior disorders: A
meta-analysis. Psychological Bulletin,
106, 97-106.
Edwards, L. K. (1993). Applied
analysis of variance in behavioral science. New York: Marcel Dekker.
Elliott, K. J. (1995). Anthetic dialogue: A new method for
working with dysfunctional beliefs in career counseling. Journal of Career Development, 22, 141-148.
Fontaine, J. H. (2001). Career Thoughts Inventory. In B. S.
Plake & J. C. Impara (Eds.), The
fourteenth mental measurements yearbook (pp. 228-230). Lincoln, NE:
Buros Institute of Mental Measurement.
Gilbert, H. B. (1996). Career
Thoughts Inventory: A review and critique (Rep. No. ED408526).
Holland,
J. L., Daiger, D. C., & Power, G. (1980). My Vocational Situation. Palo
Alto, CA: Consulting
Psychologist Press.
Holland,
J. L. (2004). Self Directed Search: Form
R (4th ed.). Odessa,
FL: PAR.
Jacobson, N., & Dobson, K. S. (2000). A component
analysis of cognitive-behavioral treatment for depression. Journal of Consulting and Clinical Psychology, 64, 295-304.
Kinnier, R. T., & Krumboltz, J. D. (1986). Procedures of
successful career counseling. In N.C.Gysbers (Ed.), Designing careers (pp. 307-335). San Francisco: Jossey-Bass.
Kirk, R. E. (1982). Experimental design : Procedures for the behavioral sciences (2nd ed.). Monterey, CA:
Brooks/Cole.
Krumboltz, J. D. (1983). Private
rules in career decision making (Special Publications Series No. 38) (Rep.
No. ED 229 608). Columbus, OH;
Ohio State
University, National
Center for Research in Vocational
Education, Advanced Study
Center: ERIC Document
Reproduction Service.
Krumboltz, J. D. (1990). Helping
clients change dysfunctional career beliefs. Paper presented at the annual
meeting of the American Association for Counseling and Development, Cincinnati, OH.
Krumboltz, J. D. (1976). This Chevrolet can't float or fly. The Counseling Psychologist, 76, 17-19.
Krumboltz, J. D. (1994). The Career Beliefs Inventory. Journal of Counseling & Development,
72, 424-428.
Krumboltz, J. D., & Jackson, M. A. (1993). Career
assessment as a learning tool. Journal of
Career Assessment, 1, 393-409.
Krumboltz, J. D., Mitchell, A., & Jones, G. (1976). A
social learning theory of career selection. The
Counseling Psychologist, 76, 71-80.
Lange, A., Richard, R., Gest, A., de Vries, M., &
Lodder, L. (1998). The effects of positive self-instruction: A controlled
trial. Cognitive Therapy and Research,
22, 225-236.
Luria, A. R. (1962). On the variability of mental functions
in the process of the development of a child: On material from a comparative
investigation of twins. Voprosy
Psychologii, 3, 15-22.
Meichenbaum, D., & Cameron, R. (1974). The clinical
potential of modifying what clients say to themselves. Psychotherapy: Theory, Research, & Practice, 11, 103-117.
Meichenbaum, D. (1974). Self-Instructional Strategy
Training: A cognitive prosthesis for the aged. Human Development, 17, 273-280.
Meichenbaum, D. (1975). Enhancing creativity by modifying
what subjects say to themselves. American
Educational Research Journal, 12, 129-45.
Meichenbaum, D. (1977). Cognitive-behavior
modification: An integrative approach. New York: Plenum.
Meichenbaum, D. H., & Goodman, J. (1971). Training
impulsive children to talk to themselves: A means of developing self-control. Journal
of Abnormal Psychology, 77, 115-126.
Peterson, G. W., Sampson, J. P. Jr., Lenz, J. G., &
Reardon, R. C. (2002). A cognitive information processing approach to career
problem solving and decision making. In D.Brown (Ed.), Career choice and
development (4th ed., pp. 312-369). San
Francisco, CA:
Jossey-Bass.
Peterson, G. W., Sampson, J. P. Jr., & Reardon, R. C.
(1991). Career development and services:
A cognitive approach. Belmont,
CA: Brooks/Cole.
Peterson, G. W., Sampson, J. P., Jr., Reardon, R. C., &
Lenz, J. G. (1996). Becoming career problem solvers and decision makers: A
cognitive information processing approach. In D. Brown & L. Brooks (Eds.), Career choice and development (3rd. Ed.)
(pp. 423-475). San Francisco,
CA: Jossey-Bass.
Reardon, R. C., Lenz, J. G., Sampson, J. P., & Peterson,
G. W. (2000). Career planning and
development: A comprehensive approach. Belmont,
CA: Wadsworth.
Reed, C. R., Lenz, J. L., Reardon, R. C., & Leierer, S.
(2000). Using the Career Thoughts Inventory to study the impact of a career course
(technical report No. 25). Tallahassee, FL: Florida
State University. Also available at http://www.career.fsu.edu/documents/technical
reports/technical reports.html
Reed, C., Reardon, R., Lenz, J., & Leierer, S. (2001).
Reducing negative career thoughts with a career course. Career Development
Quarterly, 50, 158-167
Roscoe, J. T. (1975). Fundamental
research statistics for the behavioral sciences (2nd ed.). New
York: Holt.
Salkovskis, P. M., Clark,
D. M., & Hamel, D. A. (1991). Treatment of panic attacks using cognitive
therapy without exposure or breathing retraining. Behavior Research and Therapy, 29, 161-166.
Sampson, J. P., Jr., Lenz, J. G., Reardon, R. C., &
Peterson, G. W. (1999). A cognitive information processing approach to employment
problem solving and decision making. The
Career Development Quarterly, 48, 3-18.
Sampson, J. P. Jr., Peterson, G. W., Lenz, J. G., Reardon,
R. C., & Saunders, D. E. (1996a). Career
Thoughts Inventory. Odessa,
FL: Psychological Assessment
Resources.
Sampson, J. P. Jr., Peterson, G. W., Lenz, J. G., Reardon,
R. C., & Saunders, D. E. (1996b). Improving
your career thoughts: A workbook for the Career Thoughts Inventory. Odessa, FL:
Psychological Assessment Resources.
Sampson, J. P. Jr., Peterson, G. W., Lenz, J. G., Reardon,
R. C., & Saunders, D. E. (1996c). Career
Thoughts Inventory professional manual. Odessa, FL:
Psychological Assessment Resources.
Sampson, J. P. Jr., Reardon, R. C., Peterson, G. W., &
Lenz, J. G. (2004). Career counseling and
services: A cognitive information processing approach. Belmont, CA:
Brooks/Cole.
Shrout, P. E., & Fleiss, J. L. (1979). Intraclass
correlations: Uses in assessing rater reliability. Psychological Bulletin, 2, 420-428.
Slaney, R. B. (1980). Expressed vocational choice and
vocational indecision. Journal of
Counseling Psychology, 27, 122-129.
Smith, M. L., & Glass, G. V. (1987). Research and evaluation in education and the
social sciences. Boston,
MA: Allyn and Bacon.
Strunk, D. R. (2001). Cognitive therapy for depression: A
review of its efficacy. Journal of
Cognitive Psychotherapy: Special issue: Review of cognitive behavioral therapy,
15, 289-297.
Vernick, S. H. (1999). The
Career Thoughts Inventory (CTI) in research and practice (Rep. No.
ED445304).
Wells, A. (2000). Emotional
disorders and metacognition: Innovative cognitive therapy. New York: Wiley.
(
(