A Differential Feature-Cost
Analysis of Internet-Based Career Information Delivery Systems (CIDS):
Technical Report Number 24
James P.
Sampson, Jr.
Jill A. Lumsden
Darrin L. Carr
Elisa A. Rudd
December 1999
Copyright 1999
by Florida State University
All Rights Reserved
Center for the
Study of Technology in Counseling and Career Development
University Center, Suite A4100, The Florida State University
Tallahassee, FL 32306-2490
(850) 644-6431 (voice) (850) 644-3273 (FAX)
http://www.career.fsu.edu/techcenter
James P.
Sampson, Jr. is Professor in the Department of Human Services and Studies and
co-director of the Center for the Study of Technology in Counseling and Career
Development at The Florida State University. Jill A. Lumsden is Coordinator,
Career Development Services and Darrin L. Carr is a Career Advisor at The
Career Center, and both are doctoral students in the Department of Human
Services and Studies at The Florida State University. Elisa A. Rudd is
Assistant Director of Career Services at the University of North Carolina at
Asheville.
Current
support for this analysis has been provided by the U. S. and Florida
Departments of Labor and The Florida State University (FSU) Career Center.
Previous support for this analysis was provided by the National Occupational
Information Coordinating Committee, The FSU College of Education, The FSU
Division of Student Affairs, and The FSU Department of Human Services and
Studies, the American Counseling Association Foundation, Barnett Banks of
Florida, Inc., DANTES (Defense Activities for Non-Traditional Education
Support), the Florida Department of Education Bureau of Career Development, the
Office of Special Education and Rehabilitative Services of the United States
Department of Education, and the W. K. Kellogg Foundation.
Table of Contents |
|
|
|
Page |
|
3 |
|
|
4 |
|
|
4 |
|
|
Selection Criteria |
5 |
|
6 |
|
|
6 |
|
|
8 |
|
|
9 |
|
|
10 |
|
|
List of Tables |
||
|
|
|
Page |
|
Feature
Comparison of Internet-Based Career Information Delivery Systems |
10 |
|
|
Cost
Comparison of Internet-Based Career Information Delivery Systems |
13 |
|
|
Additional
Career Information Sites Judged As Partially Meeting Internet-Based CIDS
Criteria |
14 |
|
|
Contact
Information of Internet-Based CIDS Developers |
15 |
|
|
Contact
Information for Internet-Based State CIDS Delivered by Career Information
System |
16 |
|
A Differential Feature-Cost Analysis of Internet-Based Career Information Delivery Systems (CIDS): Technical Report Number 24
The primary
purpose of this study was to highlight similarities and differences among
Internet-based career information delivery systems (CIDS) so that
practitioners, system developers, policy makers, and researchers may make
informed decisions concerning such sites. The specific Internet-based CIDS
included in this analysis were: 1) e-Choices, 2) Career Information System –
National site, 3) Embark.com - Education and Career Opportunities System
(ECOS), 4) Georgia Career Information System, 5) Illinois Career Information
System (Horizons), 6) Indiana Career and Postsecondary Advancement Center
(ICPAC), and 7) Washington Occupational Information System (WOIS). For the
purposes of this analysis, features included: 1) site content, 2) user
friendly features, 3) support resources, and 4) access policy, while costs
included license fees. The data presented in this analysis were gathered from
on-line CIDS use, support materials provided on-line, and telephone interviews
with the developers.
For the
purposes of this paper, an Internet-based career information delivery system
(CIDS) is defined as including the following three components: 1) assessment,
2) search capability, 3) information delivery (educational and occupational
data), and these components must be integrated to assist the individual
in using the information from one element of the web site to complete another
element of the site. Examples of integration include using assessment results
to search for occupations, or looking up educational programs for an occupation
being researched. This definition is congruent with the definition of CIDS
adopted in the standards of the Association of Computer-Based Systems for
Career Information, “A computer-based career information system delivers
integrated assessment and search functions linked to career and educational
information in an environment that provides user support.”
An essential
element in evaluating the appropriateness of potential sites for various users
and populations involves an analysis of data on the effectiveness of CIDS with
different populations. The process of completing research and evaluation
studies is, however, a time-consuming process. Furthermore, CIDS are extremely
dynamic, in that they are continuously being revised and updated as the nature
of the Internet allows for immediate modifications. The rapidity of site change
complicates the evaluation of Internet-based CIDS. These two problems, the time
lag in publishing research and evaluation studies and the rapid evolution of
CIDS on the Internet, necessitate a multidimensional approach to the evaluation
process.
The use of a
differential feature-cost analysis offers a potential solution to the above
problems. A differential feature analysis allows the comparison of two or more
CIDS in terms of the features available. In regards to previous feature-costs
analyses of computer-assisted career guidance systems (CACGS) (e.g., Sampson,
et al. , 1998), Gati (1990) stated, "a feature analysis of the systems may
be used to eliminate a particular system because of the presence (or absence)
of a critical undesirable (or necessary) feature" (p. 122). This also
applies to the analysis of Internet-based CIDS. An assumption made here is that
literature relevant to CACG systems is also relevant for this analysis of
Internet-based CIDS. Examining the features of an Internet-based CIDS allows
practitioners to understand the most optimal ways to integrate the site into
their existing service delivery.
For the
purposes of this analysis, features include 1) site content, 2)
user-friendly features, 3) support resources, and 4) access policy, while costs
include license fees. Because this type of analysis is limited to features and
costs, which are known at the time the site is reviewed, the findings can be
made available in a timely fashion. This report continues with the purposes of
the study, methodology, results, discussion, and the conclusion.
The primary
purposes of this study are to identify what Internet-based CIDS exist and to
highlight similarities and differences among Internet-based CIDS, so that practitioners
may make more informed decisions concerning the integration of sites into
service delivery, Internet-based CIDS developers may more systematically
present information about their site, policy makers may monitor the
developing scope and direction of site features and costs, and researchers
may more fully describe Internet-based CIDS in their studies. The specific
Internet-based CIDS included in this analysis are: 1) e-Choices, 2) Career
Information System - National site, 3) Embark.com - Education and Career
Opportunities System (ECOS), 4) Georgia Career Information System, 5) Illinois
Career Information System (Horizons), 6) Indiana Career and Postsecondary
Advancement Center (ICPAC), and 7) Washington Occupational Information System
(WOIS). National Career Information System (CIS) delivers sites for the states
of Alaska, Idaho, Massachusetts, Nebraska, Nevada, Ohio, and Oregon that
contain state-specific information. While these sites were reviewed, they were
not listed individually due to substantial similarities to the Career
Information System (National site). Contact information for the organizations
that publish these CIS-based sites is provided later in this document.
The
following methodology will describe the selection criteria for including
Internet-based CIDS in this analysis, how features were established, procedures
for collecting and analyzing the data, and limitations of the study.
Internet-Based CIDS Selection Criteria
The following
criteria were used in selecting Internet-based CIDS for inclusion in this
analysis: 1) the state provider is included in the Association of
Computer-Based Systems for Career Information (ACSCI) directory and the
state has an Internet version of their system, 2) the site includes components
that address self-assessment, the generation of alternatives through search,
and the delivery of occupational and educational information, as well as
integration among these components. Many web sites offer valuable information
but do not have the full features or integration typically associated with
CIDS. Examples of Internet sites that do not meet this full criteria, yet may
contain valuable information, are listed later in this report. This table
illustrates which criteria used in this study are met and not met by each site.
Establishment of Features
Features
were established by beginning with the features utilized by Sampson, et al.
(1998) in the cost-feature analysis of computer-assisted career guidance
systems (CACGS). As the CIDS were reviewed, features were added or removed to
best reflect the existing features of sites. Some features were included based
on the emerging national standards for Internet delivery of CIDS, such as the
NCDA Guidelines for the Use of the Internet for Provision of Career Information
and Planning Services (National Career Development Association, 1997).
Procedures
A four member
research team was assembled to conduct this analysis. The research team met to
review the purposes of the present research and then met as a group several
times to discuss common criteria for features and to suggest the addition of
new features or the deletion of previous features. Developers were contacted to
ask if they were willing to participate in the study. All developers agreed to
participate and supplied information (e.g., passwords) required to access their
Internet-based CIDS.
Each developer's Web site was then
reviewed using the checklist in Table 1 by two members of the research team.
Disagreements between team members over the presence or absence of specific
features were resolved through group discussion and a second review of the site
for the feature in question.
When
different terminology was used by developers to represent similar features, a
"/" mark was used to combine terms, e.g., earnings/wages. In
reviewing the sites, information was not considered to be part of the Web site
if it originated from another site. Instead, this information was considered a
link.
A draft of the
report was then made available to the developers of each site to identify
factual errors and discuss the criteria for receiving a "yes" or
"no" for specific features in question or specific cost data. Factual
errors were then corrected and developer comments were taken into consideration
by the authors in completing the analysis and the discussion. The authors
assume full responsibility for the quality of the analysis and related
interpretations that are included in this study.
Limitations
While
every attempt has been made to be accurate, there are inherent limitations in
any methodology. First, the following analysis does not examine the effectiveness
or desirability of the features identified for the sites. Although a
site may be granted a “yes” for a particular feature, this feature may not be
particularly effective or desirable for the goals of one’s clientele. It is
important not to confuse effectiveness with quantity of features. Jepsen (1990)
stated that, “… the mere presence of a feature as part of any complex system
does not assure its effectiveness” (p. 130). Furthermore, features that are
present may not be desirable. Gati (1990; 1994; 1996) cautions that features
initially perceived as desirable may actually be detrimental to good practice.
A second
limitation involves the use of a "checklist" approach in presenting
the data. In an effort to present feature data in a succinct manner, a dichotomous
yes - no "checklist" comparison of sites was used, i.e. "Feature
X: Does Site A have it? Does Site
B?" This approach effectively
simplified a massive amount of data. However, potential problems occur when
this methodology oversimplifies and obscures reality. Certain features cannot
be adequately explained by this "yes" - "no" analysis. This
type of analysis does not capture some of the true differences. For example,
Internet-based CIDS may contain financial aid information for education
institutions. However, there may be wide differences in the quantity and
quality of information provided about financial aid. There are no quality
standards required to achieve a “yes” for any particular feature. For example,
there were no checks on the source validity and bias that may be inherent in
outlook information.
Also related
to the limitation of using a "checklist" approach, the awarding of a
"no" for any given site feature is not necessarily "bad"
and the awarding of a "yes" for any given site feature is not
necessarily "good. " In some
cases a "no" may not indicate the lack of a relevant
site feature. For example, if the site administers an assessment on-line that a
professional views as inappropriate for her client, then a "yes" for
this site feature is irrelevant. Oliver (1990) noted that the checklist
approach is a tool to be used in evaluating a system for a specific population.
The checklist serves as an organizer for further investigation, not as a final
evaluation of the site.
A
third limitation of this study involves the dynamic nature of the Internet. A
strength of Internet-based CIDS is the ability to constantly update and add
features and content. This strength poses a challenge to any study attempting
to review these features and content. Any attempt to do so is simply a snapshot
in time.
The results of
the analysis are provided in a series of Tables. Table 1 provides data on seven Internet-based CIDS
available on-line. It includes site content, user-friendly features, support
resources, and access policy. Table 2 provides data on
costs to access each of the Internet-based CIDS. Table 3 identifies other state career information
sites which were judged as not meeting the CIDS criteria defined earlier. Table 4 provides the contact information for Internet CIDS
site developers. Table 5 provides contact information for
state sites delivered by Career Information System.
In
light of the results obtained in this study, the following section will discuss
trends in funding and costs, user support and identification of user needs,
information provided by sites on identity and qualifications of the developers,
data provided on validity of assessments and information, use of various media
and audio by sites, integration features and strategies for access, and
reference to ACSCI standards.
The
growth and development of these sites appears to have been supported by four
funding mechanisms: 1) separate fees charged to organizations (e.g., schools)
that in turn provide access to their members (e.g., students), 2) access is
bundled with fees paid for PC-based systems, 3) access is free and granted to
members of a certain group (e.g., an entire state) due to underwriting by a
government agency, and 4) access is universal due to underwriting by a
government agency. There is currently no evidence of an Internet CIDS that
derives direct support from an individual, fee per service basis.
While
these Internet-based CIDS are not directly supported by individual users, it is
the individual who ultimately benefits from these sites. However, not all of
these individuals have the capability to initiate or sustain the career
exploration and decision-making process (Sampson, Peterson, Reardon, &
Lenz, in press). Varying degrees of professional support for CIDS use may be
necessary depending upon user needs (Sampson, 1997). None of the sites include
a description of circumstances where users may need help applying the
information to their specific needs. It would benefit these users if sites
included contact information for local referral sources based upon user needs.
Basing
the design of Web sites for career information delivery on a needs-based model,
helps individuals to more quickly find information that meets their needs while
understanding how to use the information they receive (Sampson, 1999).
Traditionally, the user has been required to learn the navigation metaphors and
"rules" of a particular Web site in order to find the information
which they seek. In a needs-based design, CIDS features are categorized by
commonly occurring user needs. For example, college graduates often need to
conduct a job search and high school students often desire to explore their
occupational interests. This approach requires user reflection and provides a
framework with which users can match their needs to features available on the
site.
A great deal of the information about
site developers and site content is contained in separate Web sites about the
developer or sponsoring organization. This information is not well integrated
into the Internet CIDS themselves. Users may begin their search for career
information at the Web site address for the Internet CIDS, thus missing the
opportunity to find this valuable information. For example, while developers
often identify themselves on their CIDS site, this identification may not be
linked to the qualifications of the developer, which is posted on a different
Web site. Therefore, consumers have less information with which to make
informed choices about the quality of available sites. Although a qualified
developer does not ensure the existence of a valid CIDS, it is one useful piece
of information individuals may use in selecting quality sites.
Similarly,
none of the sites provide information on validity of assessments. This
information is necessary to allow practitioners to ethically discharge their
responsibilities and users to determine if the assessment will meet their
needs. Likewise, while original sources were cited in much of the information
content, none of the sites provided a general statement about information
gathering and authoring procedures or quality control methods. Practitioners
and users must have confidence that the information delivered is accurate and
relevant to their needs.
Users
may potentially benefit from exposure to "original source content"
via links to other sites. While there is a natural desire to keep users on
one's own Web site, experience with real world content may be a valuable aspect
of the exploration process. For example, links to union or employer web pages
offering apprenticeships would provide the learner with 1) evidence to validate
information in the CIDS, 2) a chance to build a schema by comparing and
contrasting options; and 3) information on which to take further action (e.g.,
information interview, job shadow, or application). The downside to providing
users direct access to such content is that they can easily become lost and
they may not be able to judge the variable quality of information presented.
The
Internet offers a wide variety of media creation and integration tools to
developers, however sites currently rely on text-based strategies for
information presentation. This is most likely due to delivery issues that are
associated with multimedia and the Internet (e.g., bandwidth, platform
standards, and production costs). However, the user experience could be
enhanced by presenting complex information (e.g., wage and Labor Market
Information) in the form of simple charts delivered as graphics.
One
form of information delivery offering rich possibility to developers is audio.
None of the sites appear to use this presentation method, even though audio is
a relatively bandwidth friendly form of multimedia. It can also be a relatively
easy to produce and cost effective method of integrating "real world"
content (e.g., vocational biographies). Audio offers additional benefits to those
with visual disabilities or limited reading skills. It is also important for
developers to consider other accessibility issues when designing and delivering
their products. For example, all developers may wish to provide alternative
means of navigation or text only versions of their Web sites for individuals
with visual disabilities, who may depend on text-to-speech tools for navigating
the World Wide Web.
Overall,
Internet-based CIDS appear to be in an early stage of development in comparison
with their older, PC-based siblings. While the core information contents of
Internet-based and PC-based CIDS are almost identical, tools that users can use
to integrate this information in the decision-making process appear less
developed. For example, none of the sites offer the capability to compare two
education providers or two occupations, a common feature of standalone CIDS.
This situation will probably improve over time, as developers are more able to
apply their past experience with CACG systems to an Internet environment that
is more responsive and fully featured.
Similarly,
only two CIDS currently offer an online portfolio and keep a user record
through an individualized access strategy (i.e., users receive their own
username and password). While group access strategies are easier to deliver and
manage, they may possibly provide a less powerful experience for the user. By
users having their own unique records, tools for portfolio development (e.g.,
resumes, assessment records, and education) and information management (e.g.,
individual learning plans and histories of past site utilization) become
possible. One possible disadvantage the individual access strategy presents is
a potential learning barrier or concern about confidentiality that users may
choose to avoid. Such barriers and concerns may decrease site utilization.
Currently,
there is no reference to ACSCI standards on any of the Internet-based CIDS.
These standards represent a powerful tool for educating users and practitioners
about what they should expect from providers in terms of quality and use of
career information. Rather than failing to meet standards, it may be that in
the rush to make sites available, time was not allocated for showing adherence
to standards.
While
tremendous progress has been made over the past two years in the development
and delivery of Internet-based CIDS, this career resource is still young.
Similar to the original transition from mainframe minicomputer-based career
information delivery systems to PC platforms, CIDS have begun the transition
from PCs to those that leverage the full potential of the Internet. Based upon
a feature-cost analysis of seven existing Internet-based CIDS, it appears that
practitioners, site developers, policy makers, and researchers may wish to
focus their thinking about the future of Internet-based CIDS around five key
activities. These include: 1) providing information to users about developer
qualifications, information validity, and assessment validity, 2) including
needs-based information access strategies for users, 3) integrating multiple
media formats and original source material into each site, 4) developing tools
which assist users in integrating site information into their individual career
decision-making processes, and 5) educating users about standards of practice
in career information delivery. The rapid pace of Internet technology
development will most likely bring an accelerated rate of change to career
information delivery systems and their features and costs. Furthermore, it is
possible that we will see the development of CIDS that base their cost recovery
on a fee per service basis. Both of these eventualities will require ongoing
analysis and discussion among practitioners, site developers, policy makers,
and researchers to ensure that users find and use the information they need to
make effective career decisions.
Association of
Computer-Based Systems for Career Information. (1999). Handbook of standards
for the operation of computer-based career information systems. Alexandria,
VA: Author, ACSCI Administrative Office. [On-line]. Available: http://www.acsci.org/standards2.htm
Gati, I.
(1990). The contribution of differential feature-cost analysis to the
evaluation of computer-assisted career guidance systems: A comment on Sampson,
Reardon, Humphreys, Peterson, Evans, and Domkowski (1990). Journal of Career
Development, 17, 119-128.
Gati, I.
(1994). Computer-assisted career counseling: Dilemmas, problems, and possible
solutions. Journal of Counseling and Development, 73, 51-56.
Gati, I.
(1996). Computer-assisted career counseling: Challenges and prospects. In M. L.
Savickas & B. W. Walsh (Eds.). Handbook of career counseling theory and
practice (pp. 169-190). Palo Alto, CA: Consulting Psychologist Press.
Jepsen, D. A.
(1990). A useful but limited consumer's guide. Journal of Career
Development, 17, 129-132.
National
Career Development Association. (1997). NCDA guidelines for the use of the
Internet for provision of career information and planning services
[On-line]. Alexandria, VA: Author. Available: http://ncda.org/polweb.html
Oliver, L. W.
(1990). Evaluating computer-assisted career guidance systems: A critique of the
differential feature-cost approach. Journal of Career Development, 17,
137-141.
Sampson, J.
P., Jr. (1997, January). Ethical delivery of computer-assisted career guidance
services: Supported vs. stand-alone system use. In R. C. Reardon (chair), Ethical
issues in using computers to deliver career services: Counseling intervention,
equality of access, and the Internet. Paper presented at the National
Career Development Association Conference, Daytona Beach, FL.
Sampson, J. P., Jr. (1999). Integrating
Internet-based distance guidance with services provided in career centers. The
Career Development Quarterly, 47, 243-254.
Sampson, J.
P., Jr., Peterson, G. W., Reardon, R. C. & Lenz, J. G. (in press). Using
readiness assessment to improve career services: A cognitive information processing
approach. The Career Development Quarterly.
Sampson, J.
P., Jr. , Reardon, R. C., Reed, C., Rudd, E., Lumsden, J., Epstein, S., Folsom,
B., Herbert, S. M., Johnson, S., Simmons, A., Odell, J., Rush, D., Wright, L.,
Lenz, J. G., Peterson, G. W., & Greeno, B. P. (1998). A differential
feature-cost analysis of seventeen computer-assisted career guidance systems:
(technical report No. 10) (8th Ed. ). Tallahassee, FL: Florida State
University, Center for the Study of Technology in Counseling and Career
Development [On-line]. Available: http://www.career.fsu.edu/techcenter/Tr10.html
|
|
EC1 |
CIS |
ES |
GA |
IL |
IN |
WA |
|
SITE
CONTENT |
|
|
|
|
|
|
|
|
Web Site Introduction |
|
|
|
|
|
|
|
|
Identification of developers |
Y |
Y |
Y |
Y |
Y |
Y |
Y |
|
Information on the qualifications of
developer |
N |
Y |
N |
Y |
Y |
Y |
Y |
|
Information on validity of assessments |
N |
N |
N |
N |
N |
N |
N |
|
Information on validity of the information |
N |
N |
N |
N |
N |
N |
N |
|
Description of circumstances where user
may need professional assistance |
N |
N |
N |
N |
N |
N |
N |
|
Description of limits of confidentiality
of user data input on the Web site |
Y |
N |
Y |
N |
N |
N |
N |
|
Recommendation on use of site based on
identified user needs |
N |
N |
Y |
N |
N2 |
Y |
N |
|
Notice of how to obtain local assistance
with career problem solving |
N |
N |
N |
N |
N |
Y |
N |
|
Notice of voluntary adherence to ACSCI
standards |
N |
N |
N |
N |
N |
N |
N |
|
Indication of how ACSCI standards are met |
N |
N |
N |
N |
N |
N |
N |
|
|
|
|
|