VALENCIA COMMUNITY COLLEGE

Alternative Self-Study: Strategic Topics Report 2003

Strategic Topic #1 - Core Competency Integration and Assessment

Executive Summary

Development of the current core competencies at Valencia grew from simultaneous initiatives begun nearly a decade ago: grass-roots faculty and staff work on improving student outcomes across diverse populations (supported with Title III grant funds) and an administratively-initiated, collaboratively-led exploration of becoming a more learning-centered institution (enhanced through college participation in the Pew Higher Education Roundtables and in an ACE/Kellogg project, “Leadership and Institutional Transformation”).

Through a massive, college-wide collaborative process over a three-year period, we moved from seven discipline-specific competencies to four curriculum-integrating global competencies:  Think, Value, Communicate and Act (TVCA).  Now listed and explicated in the College Catalog, course syllabi, the Strategic Learning Plan, and other college publications, discussion continues on how best to understand, teach toward, and measure growth in these key life abilities.

This segment of our Self Study Report documents the progression of TVCA development from concept to reality; the substantial on-going faculty/staff engagement in learning about learning and about authentic measurement thereof; the various and continuing pilot projects in assessment of learning in the core competencies; other developments resulting from competency-based thinking, such as the new Teaching/Learning Academy for tenure-track faculty; and, finally, a projection of “what next?”  How do we move from peripheral experimentation to core process?

Accomplishments include:

• Creation of new core competencies that serve to integrate the Valencia learning experience for students, faculty and staff.
• Dissemination and explication of core competencies in college publications, including course syllabi
• Multiple, on-going opportunities for faculty and staff to research, develop, and experiment with learning activities for and assessment of core competencies
• Documented results of pilot projects in integration and assessment of core competencies
• College-wide familiarity with the core competencies; a shared vocabulary
• Community College Survey of Student Engagement (CCSSE) evidence that core competencies are being addressed within the curriculum.
• A developed, fully operational e-portfolio, currently in pilot testing (an eventual repository for rich evidence of student growth in core competencies that can inform our analysis of degree program effectiveness in the future).

Work remaining includes:

• Moving assessment of core competencies from periphery to center, from “special assignment” to “regular assignment” people and practices
• Further developing understandings of the relationship between course outcomes achievement and measurable growth in mastery of the core competencies
• Continuing to provide learning opportunities across the college on the core competencies and their assessment 
• Structuring on-going opportunities for collegial conversation to ensure shared understandings of intention and purpose, insights and challenges
• Further developing MyPortfolio and other tools that provide scaled and
• authentic assessment possibilities
• Broadly practicing assessment of student growth in TVCA, building toward habit and expertise as well as a body of useful information
• Widely learning and practicing the application of evidence to a continuous-improvement cycle.

We acknowledge the TVCA Project as a work still in early-stage progress and as complexly inter-related with other learning-centered initiatives at the College.  We welcome the perspectives and guidance that this accreditation process will contribute toward accomplishment of our learning-centered commitment to know what our students are learning and to apply those understandings to improve and enhance subsequent learning at the College.

Strategic Topic #1 - Core Competency Integration and Assessment

Overview
Think, Value, Communicate, Act (TVCA): Core Competency Integration and Assessment

The Valencia core competencies (Think, Value, Communicate, and Act {TVCA}) provide a holistic description of what “educated” persons (life-long learners) can do increasingly well.  The conceptual model of TVCA is outlined in the Valencia Community College Alternative Self-Study Proposal (page 8).  This report on the TVCA Core Competency Integration and Assessment Project (2001-2003) fulfills one of the three specific objectives of the proposal for an Alternative Self-Study for Reaffirmation of Accreditation for Valencia Community College.  Specifically, the objective of the core competency project is to “design and implement a system that integrates the core competencies into the curriculum through faculty development, curriculum design, and student assessment” (Proposal, page 8). 

The student Core Competencies are now well established as goals of teaching and learning in all programs at Valencia (College Catalog, 2000-01, page 13, http://valenciacc.edu/catalog/00-01/pdf/toc.pdf).  The challenge addressed by the TVCA Integration and Assessment project is that “The core competencies now need to be fully integrated into all courses by faculty through learning and assessment strategies” (Proposal, page 8).  Core competencies unify the curriculum for both faculty and students, causing us to explicitly think about and focus the purposes of our work.  We assume that each course contributes to growth in the core competencies as well as to increased learning in the specific disciplines.  When working with faculty, our approach has been to “excavate,” that is, to uncover how growth is fostered and learning outcomes met, and to document evidence of this.  We are now challenging ourselves to name and demonstrate progress and mastery levels in those key abilities:  to measure deep, long-term, transferable learning and the ability to think critically and creatively; to exercise good judgment; to understand and effectively articulate ideas and perspectives; to act responsively, responsibly, and effectively across courses and disciplines—in life, in fact.

Our current system of institutional research does not collect and analyze evidence of student learning in terms of TVCA as outlined above.  The Office of Institutional Research (IR) does expertly gather data on many vital indicators, such as rates and patterns of enrollment, retention, passing, graduation, etc. (reports available in the SACS Resource Rooms).  Our curriculum does have some assessment checkpoints, such as the Computerized Placement Test (CPT) upon admission (to place students into remedial courses as needed), the college preparatory level exit exams in Reading, English, and Mathematics, and the Freshman Composition I exit writing exam.  But on the whole, the curriculum does not thoughtfully produce evidence of student learning, nor is such evidence systematically collected, analyzed, and shared back to the departments and professors so that it can be used to plan improvements.  There is a gap between our goals for the curriculum (student learning in terms of TVCA) and our systems for data collection, analysis, and application toward improvement.  

Valencia’s chosen method for bridging this gap is via professional development programs for faculty and academic staff.  This is clearly indicated in the self-study proposal: “Professional development will be provided for department chairs and lead faculty, within and across disciplines, to work with other faculty to integrate the core competencies into their course syllabi and curricula” (Proposal, page 8).  There are many dimensions to the professional development that will create real changes in the basic teaching, learning and assessment functions of the college: some will require support of academic deans as they develop departmental assessment plans; many others will require intensive work with faculty to support the changes needed to shift to learning-centered practices.  It is our goal to support our educators as they devise ways to know and to show what their students have learned.  Both deans and teachers need time, resources and support to learn how to develop assessments that reflect more than information-recall; to collect the evidence of what students know how to do, and how well they can do it.

Bridging this gap actually means creating substantive changes in the culture of the college.  Creating a cultural change in an organization as large and diverse as Valencia is a long-term goal, one of great complexity.  The College’s stated intention is to move toward a “culture of evidence” to support all key decisions, with evidence of student learning at the center.  The method of effecting this change is appropriately dependent upon comprehensive professional development programming.  John Tagg, co-author of the oft-cited “Learning Paradigm” article from Change magazine (Barr and Tagg, 1995) has commented on Valencia’s apparent success by conventional community college standards.  He noted that this very success would make cultural change that much harder-why change when what you do seems to work?  As he told us during a consulting visit in Fall 2000:

“…I suspect that the core culture of the college has not yet faced up to the paradox that in order to start doing new things you will have to stop doing some old things.  This is a possible source of conflict in the future.  It will require an extended and open conversation about the trade-offs involved in change in order to break through some of the institutional defensive routines that are likely to appear to deflect changes.” 

(Tagg Report to Valencia, 9/24/00)

Valencia Community College President Sanford Shugart understands the changes that need to be made in terms of systems and organizational culture.  He made this clear to the college in his presentation to the entire college assembly, November 5, 2002.  The following is a paraphrase:

Education reform of the past 50 years has failed because it is phrased as a people problem, i.e., that the teachers and/or the students are the “problem,” that one or both are deficient and therefore in need of fixing.  Instead, we need to focus on a systematic approach to what is essentially a systems problem.  What we need is organizational change that empowers people to innovate and improve the conditions that support learning.  We may have to break and remake our systems at the college to reach these new outcomes.

We will need to collect results of our efforts.  We don’t measure results.  We need to work together collegially to identify who’s learning and who’s not, and discover what we need to do to move the students to the next level of learning.  We need to agree upon what needs to be learned and how we will know when it has been learned.  And we need to approach this collegial work in the way that it increases liberty, our freedom to create the best conditions for learning.

(Shugart, 11/5/2002)

The TVCA Integration and Assessment Project needs to be seen within Dr. Shugart’s frame as an attempt to foster positive changes in the organizational culture that will result in a “culture of evidence” of student learning.  Our vision is that assessment of TVCA will inform the teaching and learning process and feed back into the curricular and professional development planning.  What we have to report here is a description of the ongoing work of core competency integration and assessment, its results to date, and the timeline for moving this learning-centered work outward into ever-expanding circles of awareness and engagement.

Incremental Progress in Mainstreaming the Learning-Centered Initiative (LCI), 1994-2000

Valencia has made steps toward the goal of bringing the LCI into the mainstream of the college through engaging individual faculty in curricular and professional development projects.  The impact of the 1994-99 Title III grants on the approximately 300 faculty who participated is documented in research articles published in peer-edited academic journals (Nellis, DiMartino, Hosman, Clarke, 2000; Pedone and Bonsangue, 2000; Nelson, 1998), numerous professional conference presentations (Reagan, Puyana, Hosman, Nellis, 2000), and annual reports to funding agencies (copies of academic journals available in SACS Resource Rooms).  Additional information on performance results can be found at http://faculty.valenciacc.edu/development/archives/results.shtm, and faculty participant lists, http://faculty.valenciacc.edu/development/archives/pastrosters.shtm.

Since 2000, the Title III West and Title V Osceola grants, and the Destinations summer development program have adopted an Action Research model as a method to bring more members of the college community into a “culture of evidence.”  Examples of faculty designs are available on the Faculty Development website, http://faculty.valenciacc.edu/development.  The summer program Destinations has stressed assessment for years (http://faculty.valenciacc.edu/destination.), especially Classroom Assessment Techniques (CATs).  Since 2002, the Destinations program has also emphasized standards of scholarship in faculty projects, which has deepened the commitment to collection of data on student learning results and the sharing of those results in ways that help the college better understand student learning.   In all three of these programs, which engage over 140 faculty per year in sustained and intensive (40 to 60 hours of time) curricular and professional development projects, the explicit integration of TVCA into teaching and learning is an integral component (http://www.valenciacc.edu/tla/documents/LC_Curriculum.pdf).

There are additional professional development programs, such as the Teaching/Learning Academy (TLA) -- a redesigned approach to orientation and learning for tenure-track faculty (http://www.valenciacc.edu/tla/documents/Mission_Vision%20TLA.pdf). The TLA is currently serving 74 new faculty.  TVCA and its assessment are central to the program (http://www.valenciacc.edu/tla/about_tla.cfm).  Furthermore, Valencia has helped develop and currently employs the Scenarios online course in learning-centered teaching. The primary audience for Scenarios is adjunct faculty who often cannot participate in other professional development offerings because of scheduling and compensation complications.  Approximately 100 faculty have participated to date in this 20-hour, web-mediated course.  Research on the initial success of the Scenarios project has recently been published in the New Directions for Community Colleges series (Nellis, Hosman, King, Armstead, 2002; article available in SACS Resource Rooms, see bibliography, page 43). 

The programs described here are part of the delivery of a comprehensive faculty development program.  Apart from the major efforts of the West Campus Title III and Osceola Campus Title V grants, Valencia has supported these development opportunities through the budget initiative process, which allows faculty and staff to request budgetary support on an annual basis.  For two years, 2001-2002 and 2002-2003, Valencia budget initiatives have funded the Destinations summer program, the Scenarios online project, the TLA program for tenure-track faculty, the TVCA Integration and Assessment Project, and the Start Right projects in college preparatory Reading, Math, English and English for Academic Purposes (EAP, the non-native English speakers’ preparatory program).

Valencia has evidence of student learning in aggregate data that is presented in the report on Strategic Learning Plan Goal 4: Learning by Design, and Strategic Learning Plan Goal 2: Start Right (http://valenciacc.edu/lci/news.asp).  Additional data on student engagement with the college from the students’ point of view was obtained via Valencia’s participation in the national Community College Survey of Student Engagement (CCSSE), the results of which indicate that Valencia does a consistently above average job of engaging its students when compared to the national norm for Community Colleges.

Valencia is making steady progress on the implementation and assessment of core competencies.  We are indeed “mainstreaming” the Learning-Centered Initiative (LCI) through the TVCA project and the three other strategic topics, all of which are driven by the Strategic Learning Plan.  Evidence that Valencia is meeting the goals of this strategic topic include the following:

• Creation of new core competencies that serve to integrate the Valencia learning experience for students, faculty and staff.
• Dissemination and explication of core competencies in college publications, including course syllabi.
• Multiple, on-going opportunities for faculty and staff to research, develop, and experiment with learning activities for and assessment of core competencies.
• Documented results of pilot projects in integration and assessment of core competencies.
• College-wide familiarity with the core competencies; a shared vocabulary.
• Community College Survey of Student Engagement (CCSSE) evidence that core competencies are being addressed within the curriculum.
• A developed, fully operational e-portfolio, currently in pilot testing (an eventual repository for rich evidence of student growth in core competencies that will inform our analysis of degree program effectiveness in the future).

There is still plenty of work yet to do.  Definition of learning outcomes for courses in terms compatible with TVCA is an ongoing task (see Learning Outcomes).  Assessment of student learning in courses is being accomplished on an individual faculty basis through the Title III and Title V grant-funded programs; shared assessments by groups of faculty teaching the same course were pilot tested in fall 2002 as a result of the TVCA project (see Model Lessons and Case Studies).  Ongoing course review at the departmental level, using evidence of student learning to inform a systematic improvement process is still beyond our reach at this time, but the TVCA project offers a vehicle for discovering how to best develop such a system.  For an outline of the vision for course and program assessment, see http://faculty.valenciacc.edu/pbishop/tvca/tvca-home.htm.  The work of creating cultural change in a large organization takes time.  Valencia is engaged in this difficult work and discovering those systems it must change and create to support our progress toward demonstrating our status as an authentic learning college.

2001-2003 Core Competency Initiative

As stated in the Self-Study Proposal, the problem addressed by the TVCA Integration Project is that “The core competencies now need to be fully integrated into all courses by faculty through learning and assessment strategies” (Proposal, page 8).   During 2000-2001, the College was involved in construction of the new Strategic Learning Plan (SLP), and the need to integrate and assess the core competencies emerged as a major goal to be accomplished.  The charge for the TVCA project is drawn from SLP Goal 4, Learning By Design:  “Create a culture where clearly specified learning outcomes and assessments engage (learners) as responsible partners ... and where the College’s learning leaders can effectively create the best conditions for learning”  (http://valenciacc.edu/lci/essays/Goal4Essay.htm).

Action Items are sub-elements of the SLP that are designed to move the college incrementally toward achievement of the desired outcomes for each goal. Goal 4, Learning By Design, Action Item 1, 2, 3, and 5 are directly impacted by the efforts of the TVCA Integration and Assessment project:

• Create a multi-year strategy to incorporate core competencies
• Identify learning outcomes and assessment procedures, with a focus on “front door” courses
• Promote the use of classroom assessment models
• Create a multi-year course review and approval process.

The work of the TVCA Integration and Assessment project is clearly at the heart of this segment of Valencia’s SLP (see Goal 4 Report: http://valenciacc.edu/gtreports/).

To help accomplish this goal, President Shugart named humanities professor Dr. Philip Bishop as a “Faculty Fellow,” in Fall 2001 to lead the design and implementation of the TVCA project.  Bishop was supported throughout the project by Ann Puyana, Interim Chief Learning Officer, and Patrick Nellis, Faculty Development Coordinator.  Additional support came from the management team of the Title III and Title V grant projects, Coordinator of the Teaching/Learning Academy (TLA), Coordinator of the Destinations summer program, and the College Prep Coordinators for Mathematics and Communications (i.e., the staff of the Curriculum Development, Teaching and Learning [CDTL] department). 

Supporting internal expertise, three very important consultant visits regarding the TVCA core competency framework took place during1999 and 2000.  The suggestions by these consultants were influential in the design of the self-study proposal and they continue to guide our thinking about core competencies and assessment.  A synopsis of each follows:

Kings College

This was the third visit by consultants from Kings. This time, a team of three faculty, led by Academic Vice President Don Farmer, presented a seminar on Evaluation of Core Competencies.  Twenty-six people attended, among them two Vice Presidents, six Deans, one Provost and 17 faculty leaders and CDTL staff.

Four discussion groups worked on course assignments, teaching strategies and assessment criteria related to TVCA.  A variety of possible course-embedded assessment activities to holistically assess student performance were demonstrated, and possible strategies to implement such assessments were outlined. 

In his report to Valencia, Dr. Farmer suggested:

• Wide discussion among faculty needs to be facilitated in order to create the expectation that TVCA will move beyond rhetoric to reality for both faculty and students.
• Adjunct faculty need to become aware of the TVCA competencies.
• Pilot tests should be developed on each campus; these need to be supported with funds for release time and stipends.
• College-wide project teams should oversee each of the four core competencies as they are further developed, implemented, and integrated into the curriculum.
• A team of faculty should be sent to the Assessment Forum sponsored by the American Association of Higher Education.

(Farmer Report to Valencia, 11/12/99)

Robert Diamond

In one of many visits to Valencia, Dr. Diamond used participant teams in this workshop to analyze the indicators of TVCA in specific detail.  During the summary stage of the workshop, Dr. Diamond outlined important next steps for the development of the TVCA curriculum:

• More work is needed to clarify the meaning of the TVCA indicators in terms of student learning outcomes.
• Editing of the TVCA indicators should proceed as problems are uncovered during the step above.
• Discussion should be facilitated across the college on the sequence and flow of the curriculum so that it has the best chance of fostering student growth in TVCA.
• A review of course outcomes by each discipline should be conducted to infuse the TVCA indicators into the courses.

(Nellis, CDTL Report on Diamond seminar, September 2000)

John Tagg

In his report to Valencia following his visit (see bibliography, page 43), Professor Tagg offered the following suggestions on assessment of student learning in terms of TVCA:

• Assessment of TVCA must be an institutional function.
• This approach takes on a student’s perspective, whose learning experience is not bounded by departments.
• It is important to include department chairs in the change process.
• We must create trans-departmental structures that can plan and develop TVCA assessment from a more holistic perspective.

(Tagg Report to Valencia, September 2000)

In January 2001, Interim CLO Ann Puyana designed the budget initiative proposal to make TVCA integration and assessment an ongoing and authentic effort, rather than merely for external review.  The influence of the consultants above will be clear in the following aspects of the TVCA integration project:

• Open dialog on TVCA was to be the overall method of communication to promote awareness, interest and engagement in the project.
• Entry-level activity focused on identifying current course outcomes that are aligned with the global TVCA outcomes.
• Pilot testing was the approach to all implementation work.
• Course-embedded assessments were to be designed that could show development of TVCA.
• A holistic perspective of student development in TVCA was to be obtained through the use of an electronic portfolio.

The first steps of the curriculum development had already been accomplished when the proposal for the Self Study was written in September 2000 - redefining the core competencies of Valencia graduates (College Catalog 2000-01, http://valenciacc.edu/catalog/00-01/pdf/toc.pdf), followed by the declaration of learning as the primary goal of the college (SLP Goal 1: Learning First, http://valenciacc.edu/lci/essays/Goal1Essay.htm)

The next steps were to define learning outcomes for courses that linked what is learned in the class to the overarching core competencies (TVCA), and to plan to assess student achievement across courses and programs in terms of these core competencies.  Valencia’s strategy was to target the courses with very high student enrollment for the outcomes work, seeking maximum impact from the project in terms of the influence it could have over the learning experiences of as many students as possible.

The course review efforts followed a three-stage process:

• Definition of course outcomes
• Design of “model lessons”
• Broad dialog on outcomes and assessment

A section providing details of each of these phases of the course review for TVCA integration follows.

At the program level, the general education program that defines the AA Degree must be evaluated by a different means.  TVCA is a holistic framework of student competence, which is what makes this assessment work both fascinating and very difficult.  The electronic portfolio holds much promise as the method to gather evidence of student achievement of the TVCA competencies while proceeding through the general education program.  The MyPortfolio product designed at Valencia for this purpose will become the database from which an assessment team can sample complex student evidence (creative projects that demonstrate student ability to synthesize knowledge from across the general education experience) and evaluate it in terms of TVCA.  An interdisciplinary team of faculty will commence work on a TVCA rubric for assessment of portfolio projects in the Spring term, 2003.  The Nursing program is testing a version of MyPortfolio that has been customized for their curriculum during the 2002-03 academic year.  The Honors Program will be participating in the further development of My Portfolio as a teaching and learning tool.  We plan to have the assessment team begin to test their rubric for TVCA by using samples of student work drawn from pilot portfolios during Summer 2003.  From this pilot test the assessment team will make recommendations on the uses of electronic portfolio for TVCA assessment of the general education program.

MyPortfolio is embedded in the Atlas learning support system, under the MyLifemap tab. There are significant connections between the portfolio as a tool for assessment of a student’s growth in core competencies and their progress in planning their career and academic goals.  The e-portfolio holds more than evidence of academic achievement; it allows the student to bring the parts of their college experience into a coherent whole, and to share that with professors as well as prospective employers.  As this system matures we expect students and faculty to discover the reinforcing nature of the advising or student development curriculum and the academic curriculum.

It is our belief that our over-arching core competencies unify the curriculum on both conceptual and practical levels for both faculty and students, in ways heretofore not broadly understood or experienced at Valencia. The ultimate goal of the TVCA Integration and Assessment Project is for significantly more students to complete their courses and programs and to be better able to use their knowledge and skills to function competently and successfully in the “real world.”  Valencia is committed to gathering evidence of authentic learning across courses and programs that will validate those programs and help us to improve them.  The following sections provide detail about the Learning Outcomes, Model Lessons, and Case Studies of the changes in particular courses.  

TVCA Integration and Course Outcomes

Although learning outcomes are identified and course outlines are available for all courses (Valencia AA Program Review, 2002, details available in SACS Resource Rooms), they are for the most part not aligned with the current student core competencies.  As a point of departure, this phase of the TVCA Integration and Assessment project selected courses to review on the basis of their high enrollment and importance to general education.  Our general education curriculum does not have a prescribed core of courses arranged in a developmental sequence, apart from a pre-requisite system driven by the Computerized Placement Test (CPT) to identify those students whose skills in English and mathematics fall below the college level.  There are the usual prerequisite sequences in mathematics and science, but apart from Freshman Composition One (which is a prerequisite for other writing-intensive courses), students can and do take courses based upon course availability and student scheduling preference.  Following a suggestion by Valencia consultant and critical thinking expert Joanne Gainen, we have identified the de-facto core of the Valencia student experience by examining enrollment patterns.  Thus the very high enrollment courses, a combination of college preparatory (remedial) and college-level introductory courses, have been identified as the “Front Door” courses.  Because students stumble over the threshold of the “front door,” working with the faculty who teach these courses is seen as the most learning-centered approach to integration of TVCA into the curriculum.

A general call to all faculty for participation was sent out via email.  At the same time, deans were asked to nominate full-time faculty for participation in the TVCA work.  Stipends were available to faculty participants. The approach adopted was to guide lead faculty in the development of course outcomes and assessment standards that define what students will be able to do in the discipline and measure how well they'll be able to do it, as a result of what they learn in our courses. Work with 40 full-time faculty members began on 15 high-enrollment courses and two new Nursing courses in Spring 2002, and development continued through Summer 2002 (http://faculty.valenciacc.edu/pbishop/tvca/tvca-crsoutcms.htm).

A background file was distributed to participants that consisted of the existing course outline (usually 10 or more years old) and any other course development work that had been done through the grant-funded efforts.  These documents are primarily of two types: a content topic outline (usually drawn from textbook materials), with an analysis of the key skills addressed that would contribute to the students’ ability to pass the College Level Academic Skills Test (CLAST, the Florida “rising junior” exam); a second type of outline followed a competency-based approach guided by the WIDS (Wisconsin Instructional Design System) software, which is a precise analysis of discipline-based skills and knowledge needed to do well in the particular course.  These outlines are neither analytically nor philosophically aligned with the recently defined Valencia Core Competencies.

Our approach was to engage faculty in the consideration of the outcomes of the courses that they teach-- relatively broad statements of the integrated knowledge, skills, and values that a student is able to demonstrate upon successful completion of a course or program. They are frequently found by asking the faculty member to speculate on the question, “What is it that you want the student to be able to know and do five years after completing your course?”  From a broad, perhaps philosophical and synthetic goal statement, we have the raw material of a course outcome statement.  The following six points by consultant Mark Battersby clarify the outcomes approach to course and program design:

• It differs from more traditional academic approaches that emphasize coverage by its emphasis on basing curriculum on what students need to know and be able to do as determined by student and societal needs, not disciplinary tradition.
• It differs from competency-based approaches in its emphasis on integration and the development of more general abilities [such as TVCA] that are often overlooked in a competency approach.
• [The focus is on] what students should be able to do rather than merely what knowledge they should possess as a result of a course or program, making explicit the development and assessment of generic abilities [such as TVCA].
• A key element is the role of assessment.  Assessment choices give clear meaning to the more abstract formulations of the learning outcomes.
• Assessment tasks (assignments) should also be seen as a primary means of learning.
• Assessment methods should provide the opportunity for students to demonstrate the learning outcomes in as integrated and realistic use as possible.

(Battersby, http://faculty.valenciacc.edu/pbishop/tvca/whats-lrng-outcm.htm)

The goal of this work is to state learning outcomes clearly (in terms that students as well as faculty will understand) and provide evaluation based on explicit standards, both of which will facilitate student learning (Stiehl, 2002; Diamond, 1998 see bibliography, page 43).  Participants were also given a copy of Learner-Centered Assessment on College Campuses, Huba and Freed (2000), as a reference and guide to the work ahead.  This book is a comprehensive overview of learning-centered practices that provides dozens of appropriate and concrete examples from colleges around the nation.   Faculty also had access to conceptual supports on the Valencia Faculty Development web page (TVCA Glossary http://faculty.valenciacc.edu/pbishop/tvca/gloss/tvca-glossary_copy(1).htm; also the Learning-Centered Reference Guide http://faculty.valenciacc.edu/development/resources/flipbook/default.htm).

The design of course outcomes began with a two–day workshop lead by Dr. Mark Battersby on February 22-23, 2002.  Battersby began with an explication of his concept of the “competent lay person” as the desired outcome of a general college education.  This is a powerful image of college graduates as persons who can reason their way through life’s problems, most of which do not come to us as tidy, academic discipline-based homework tasks.   It is an appealing rationale for revisiting course design on the basis of well-articulated learning outcomes.  Following this introduction faculty formed work teams by course and plunged into the work of identifying their outcomes, frequently drawing upon the course outlines of record as a good starting point.   Battersby led the two days of intensive workshop with facilitation by Faculty Fellow Philip Bishop, Coordinator of Faculty Development Patrick Nellis, and a team of Faculty Facilitators consisting of Linda Anthon, Melody Boeringer-Hartnup, and Penny Villegas (faculty participants are listed on the website that follows).

Following the workshop, faculty were encouraged to discuss their draft outcomes with colleagues, circulate the documents via email, and conduct on-line conversations on a web-based discussion board.  This process of faculty-to-faculty discussion met with limited success.  The outcomes were submitted to Dr. Bishop and a feedback process was begun.  See the TVCA Integration and Assessment web page for faculty products from this work – there are 17 courses, with 15 high enrollment, front door courses; two are Nursing courses, a vital AS Degree program. Forty faculty served on course specific teams (http://faculty.valenciacc.edu/pbishop/tvca/tvca-crsoutcms.htm).

The overall goal of the Outcomes work was two-fold: course outcomes and assessment criteria for high-enrollment courses.  The quality of the results of this phase of the work varied widely, perhaps due to the background and preparation of the faculty authors, but surely also due to the varying levels of time that they devoted to the task.  During the feedback cycle it was clear that defining assessment criteria was a struggle for most participants.  A solution to both of these issues was to provide a second phase of support to faculty volunteers through the intensive summer program of faculty development that we call Destinations.  Through the Destinations teamwork in summer 2002, faculty engaged in designing concrete assessment tasks based upon the learning outcomes products of the Spring Term 2002 (see Model Lessons).

A second and deeper challenge was that there exists no strong connection between the TVCA Integration and Assessment Project and the academic departments and their deans.  This is not a failure of good will, but a system flaw.  We lack the infrastructure to support ongoing course review and analysis of student evidence through well-designed and authentic assessment tasks that can be evaluated according to criteria shared by all faculty teaching the course.  We have begun the process at a logical point by initially reviewing and creating draft redesigns of course learning outcomes and assessment criteria.  We are now conducting discipline dialogs on selected courses to engage a much broader spectrum of key full-time faculty in this discussion (see Case Studies).  We pilot tested assessment techniques in selected courses during Fall Term 2002, and are helping faculty analyze the results in Spring Term 2003. 

TVCA Integration and “Model Lessons”

“Destinations” is our intensive summer faculty development workshop, coordinated by Math Professor Melissa Pedone, that has been running continuously since 1998.  It is a 40-hour commitment (20 contact hours and 20 hours of “homework”) by approximately 100 faculty (usually a 60/40 split of adjunct and full-time). Typically, faculty will hear a keynote workshop by a nationally prominent specialist in higher education, followed by workshops to guide participants in the production of learning activities and assessment for their classroom (see the Destinations database of learning activities http://net4.valenciacc.edu/cp/destination//).  It has been an excellent vehicle for raising faculty awareness of the TVCA core competency framework and for learning about classroom assessment techniques.

The theme of Destinations 2002 was “The Scholarship of Teaching and Learning.”  Donna Duffy of Middlesex Community College, a Carnegie Scholar, was the keynote speaker.   Dr. Duffy explained the scholarly framework of teaching that she promotes at her community college and demonstrated many approaches to the assessment of course outcomes that are practiced by her colleagues.  Faculty were assigned to one of two tracks: an Action Research track for individual work, or a Model Lessons track for group work.  All work had clear required elements, including a rubric for evaluating their team products, based on the Standards of Scholarship.

The Model Lessons track was coordinated by Faculty Fellow Philip Bishop, Valencia’s TVCA Integration and Assessment project leader. The goal was for faculty to take the product of the Outcomes phase of the TVCA project and develop exemplary course units that target an essential course outcome, employ best practices of teaching and learning, and embed assessment tasks that enable shared assessment of student mastery while also providing students with useful feedback about their achievement. This practical application was accomplished as part of the Destinations program in Summer 2002. Implementation of model units and shared assessment took place in Fall 2002 by teams in 10 high-enrollment courses, with 40 faculty involved (many different from those in the Outcomes phase; part-time and full-time faculty included). 

Liaison work to encourage implementation of the projects is currently being done by faculty leaders Penny Villegas and Linda Anthon, under the supervision of Faculty Fellow Philip Bishop and Faculty Development Coordinator Patrick Nellis. 

There are some distinct advantages to using an intensive summer program such as Destinations to help faculty learn to assess course outcomes, as opposed to the disjointed experience of ad hoc work conducted over an entire term.  The products of the Model Lesson projects are coherent and scholarly.  One weakness is that participants in Destinations are not typically accountable to implement their projects, so the new expectation to do so has met with limited success.  Another weakness is that the Model Lessons did not involve enough full-time faculty who are central to the development of curriculum.  The main weakness of trying to implement assessment of learning through this method is the same one identified in the review of the Outcomes phase of TVCA Integration and Assessment:  We lack the infrastructure to support ongoing course review and analysis of student evidence through well-designed and authentic assessment tasks that can be evaluated according to criteria shared by all faculty teaching the course.

Each of the 10 Destinations Model Lesson projects resides on the TVCA website, http://faculty.valenciacc.edu/pbishop/tvca/tvca-crsoutcms.htm.

TVCA Integration: Case Studies

To view each case study in greater detail, see the following web page: http://faculty.valenciacc.edu/pbishop/tvca/tvca-casestudies.htm.

All of the courses under review in these case studies are highly enrolled, front door courses that are key to a student’s success in our college.  The dialogs that are briefly described here have grown out of the work encouraged by the TVCA Integration Project.  These discipline-based discussions begin with a consideration of the course outcomes and proceed to the questions of evidence of student learning.  The assessment of learning is a difficult topic and it is likely that dialogs will need to continue for some time to come.  Below are mini-status reports on several courses that are making progress toward the assessment of student learning.

The role of at least one dean is prominent in each of these case studies.  To establish an improvement process based upon cyclical analysis of evidence of student learning is one of the goals of the TVCA Integration Project.  Dean leadership of this effort will be crucial, as will faculty control of the design and analysis of the assessments.  These case studies represent incremental steps toward a new structure that will sustain an ongoing improvement process.  A brief statement on the progress to date with the College Prep Mathematics and Reading, Speech, Humanities, and Spanish courses follows. 

College Preparatory Mathematics

Valencia has been continuously exerting effort and resources on the seemingly intractable problem of student preparedness and performance in mathematics courses over the past several years.  Coherent re-design efforts have met with many roadblocks, including changes in the course numbering and descriptions at the state level, imposition of state- mandated exit exams, and re-shuffling of college personnel and resources in the creation of course labs and math support areas. 

Prep Math Institute
Our most promising approach to college prep mathematics is being led by College Prep Math Coordinator William Johnson (see http://faculty.valenciacc.edu/pbishop/tvca/tvca-casestudies.htm).  Johnson has established a coherent approach to curriculum design and faculty development that is based in both learning theory and research (local student performance and national data as well).  Johnson’s model for learning is holistic, and in this manner engages the student core competencies (TVCA) throughout the curriculum (mathematical thinking; communicating through a student math journal/portfolio; valuing mathematics and learning to learn skills; acting on good decisions about study habits and academic planning).  Johnson has worked with a core of six math professors who designed the curriculum modules and taught the pilot sections of the program.  The college has supported this effort, begun in Fall 2001 with “hard” money from the college budget.  In summer 2002, a team of faculty, including Johnson and two math professors, attended the Kellogg Institute for the Training and Certification of Developmental Educators, led by Hunter Boylan.  The West Campus Dean of Mathematics, Dr. Cliff Morris, and West Campus Provost Dr. Paul Kinser have been supporters of Johnson’s work and have helped make the logistics of the program possible.

SPC 1600, Fundamentals of Speech

SPC 1600 is the basic public speaking course at Valencia; it is required for the AA degree and for most AS degrees.  This is a highly enrolled “front door” course that reaches most of our student population. 

Work by faculty to update the design of the course has been intermittent since the creation of the official course outline in 1992.  In 1996, several faculty used grant-funded resources to attempt a redesign with the goal of moving toward “real world” communications. Ultimately, however, the changes tested in the pilot phase were not accepted college wide, and the design of the course was not modified.  Participating faculty enjoyed many benefits, nevertheless, including the opportunity to enhance their expertise in learning theory and pedagogy and to experience new leadership roles.

In the meantime, the new student core competencies (TVCA) were developed.  It was now necessary to review the course again to align its outcomes with the TVCA framework.  In January 2002, SPC 1600 was placed on the list of highly enrolled courses that should be addressed by the TVCA Integration project.

Before the new design work was undertaken, it happened that West Campus faculty member Dr. Mayra Holzer was concluding her dissertation research, including an inter-rater reliability study among four teachers of Speech on that campus.  Holzer captured a sample of 60 student speeches on videotape and had each of the faculty rate the student performance, using shared criteria.  One of the findings was that the faculty have very close agreement on what constitutes a good quality college speech.  This was the first systematic use of a sample of student evidence to assess an aspect of teaching effectiveness since the 1996 grant-supported projects.  The West Campus Communications department has shown that shared assessment of samples of student work using common criteria can be accomplished.  It was good news to have evidence that we do have strong inter-rater reliability.  A college-wide team of faculty in Speech are now discussing how similar samplings of student artifacts can be used to inform the effectiveness of teaching and the integration of TVCA (Holzer, 2002).

In 2002, the TVCA Integration Project initiated work with faculty to identify the outcomes of the Speech course.  Mayra Holzer and Beth Perell, both of West Campus, volunteered to lead the outcomes review. They shared the draft with colleagues college-wide (primarily via email), and by the end of Summer 2002, a strong draft of outcomes showing a relationship to the core competencies, and a full course outline including course competencies and learning objectives, were ready for review by speech teachers college-wide.

Some of this progress was aided by the summer professional development program, Destinations 2002 (see Model Lessons).  Holzer participated on a team that included an adjunct professor from West Campus and three full-time professors from East Campus.  The five speech professors worked out some sharable teaching activities that fit the course outcomes and designed assessments, including rubrics for student performance.  This project was pilot-tested during fall 2002, and analysis of evidence of student learning collected by faculty participants is planned for early Spring 2003.

A Discipline Dialog for speech professors college-wide was hosted by Communications Dean Karen Borglum of West Campus and Faculty Development Coordinator Patrick Nellis on October 24, 2002.  Seven full-time speech professors attended, with East, West, and Winter Park campuses represented.  Consensus was reached on the course outcomes and on the revised outline.  A decision was made to constitute an SPC 1600 work team to further discuss the process of departmental assessment of student learning in the course.  Another major issue still to be addressed is whether to shift the course toward a more general communications orientation; the group agreed that more study of this issue would be necessary to reach college-wide consensus.

Dialog on the teaching and learning of the Fundamentals of Public Speaking (SPC 1600) and the wider issue of communication across the curriculum will continue into the coming terms.  Plans for faculty to meet and discuss course design and assessment are already in the making; they will be informed by as much evidence of student learning as there is agreement by faculty to collect.

REA 0002, College Preparatory Reading

The pattern identified in the SPC 1600 (Fundamentals of Speech) course dialog is also evident in many of the case studies:  an old course outline, revitalized by grant-funded faculty experimentation, but not widely addressed or agreed upon beyond the pilot. Two East Campus Communications faculty, Dr. Janice Hunter and Ms. Nora Woodard worked on the TVCA Integration project in 2002, but this work had little impact across the college. 

A Discipline Dialog for reading professors college-wide was hosted on November 1, 2002 by Communications Dean Karen Borglum of West Campus, Dean Michele McArdle of Winter Park Campus, College-wide Coordinator of College Prep Communications, Dr. Nick Bekas, TVCA Faculty Fellow Dr. Philip Bishop and Faculty Development Coordinator Patrick Nellis.  Six full-time reading professors attended; all four campuses were represented.  Definitive consensus was not reached on the course outcomes.  Instead, faculty identified the twelve reading sub-skills that are tested on the state-mandated exit exam; however, they all agreed that the real outcomes of the course are much broader and deeper than those measured by this exam.  A robust discussion of what those major outcomes are and how they should be phrased ensued.  It was agreed that more work was needed on an outcomes statement for the course.

Two faculty volunteered to collect some evidence of student learning of a specific course skill, and this assessment material will be analyzed by the reading faculty in January 2003.  There is a sense that the outcomes should be clearly defined before systematic assessment procedures are put into place, but this effort to collect some student evidence is seen as a worthwhile “reality-check” for the group.  It is the first attempt to collect and analyze student performance across reading course sections.  The work on defining outcomes and finding informative assessment methods will continue in the Spring 2003 term.

HUM 1020, Introduction to the Humanities

The Introduction to the Humanities course follows the pattern identified above, although the effort to integrate TVCA into this course has been underway for about 3 years.  Humanities faculty Jean-Marie Fuhrman, Karen Styles and Wendy Schwam took part in the 2002 TVCA Integration Project and continued to refine the course outcomes and the criteria for student performance.  Jean-Marie Furhman also took part in the Destinations work and led the design of the Model Lesson for that group.

A Discipline Dialog for humanities professors college-wide was hosted by East Campus Humanities Dean Rick Rietveld, TVCA Faculty Fellow Dr. Philip Bishop and Faculty Development Coordinator Patrick Nellis on October 31, 2002.  Six full-time humanities professors attended, with East, Osceola and Winter Park campuses represented.   Significant progress toward consensus on the core outcomes of the course was made.

Several faculty volunteered to collect some evidence of student learning of a specific course outcome, and this assessment material will be analyzed by the humanities faculty in spring 2003.  It is the first attempt to collect and analyze student performance across humanities course sections.  The work on defining outcomes and finding informative assessment methods will continue in the spring term.

Meanwhile, East Campus humanities faculty have been participating in an assessment project led by Dr. Bishop.  For the past two terms, faculty on his team have used a survey of student critical thinking, based upon the work of Dr. Judy Ruland of the University of Florida.  This assessment offers faculty a view of their course from the students’ perspective and can help them find areas in their pedagogy that could be improved upon. Results of the survey are found at http://faculty.valenciacc.edu/pbishop/tvca/tvca-casestudies.htm.  The combination of student surveys with student artifacts should make for well-informed discussions of the effectiveness of the HUM 1020 (Introduction to Humanities) curriculum.  These dialogs will continue in the spring term.

SPN 1000, Basic Spanish

The curricular design work in the Spanish language area is the least typical of the courses in the case studies.  East Campus Foreign Language Coordinator Aida Diaz and East Dean of Humanities Rick Rietveld have been leading faculty and curricular development in Spanish for over three years.  This pioneering work, informed by the national standards set by the American Council of Teachers of Foreign Languages (ACTFL) and the Valencia core competencies, is already stated as clear outcomes.  Diaz and Ana Caldero of West Campus worked together on the TVCA Integration Project in 2002 to share their work to date and verify that it was an appropriate design for student learning.

In Fall 2002, several faculty on East Campus recorded student oral interviews as a means of assessing communicative ability.  This evidence of student learning will be analyzed in an assessment project during spring term to discover if students are in fact able to communicate at an appropriate level and if professors in fact apply the agreed-upon criteria in a comparable fashion.  Information from this project will be the basis for further discipline dialogs in Spanish and perhaps provide a basis for adjustments in the design of the curriculum.

Summary and Future Plans

Our current work in the TVCA Integration project is built upon several years of individual faculty development projects such as the Title III and Title V grants, the summer Destinations program, and the new Teaching/Learning Academy.  These programs are where the evidence of integration of TVCA exists to date, in the results of individual faculty members’ action research projects. 

Valencia has accepted the challenge of becoming a more learning-centered college.  We now need to generate informative assessments of student learning that will offer deans and faculty in their campus departments the opportunity to conduct evidence-based reviews of course effectiveness.  This is new work for us at Valencia.  Among other changes, this work requires a collective rather than an individual effort, which in itself is a major cultural shift.  Another cultural shift is involved in using a variety of quantitative and qualitative evidence of student learning to inform collective decisions about curriculum and pedagogy.  A realigned infrastructure must support both course and program review.  The following excerpts are taken from the TVCA Integration project website (http://faculty.valenciacc.edu/pbishop/tvca/overvw.htm):

The Big Picture
Valencia seeks to develop an institutional assessment process for gathering qualitative evidence of student learning and using this evidence to improve learning. The primary purpose of assessment is to inform and facilitate learning. Assessment must be part of a feedback loop that informs students about their own learning and guides educators in efforts to improve student learning.

An Emerging Design
As curriculum leaders envision the process now, Valencia can assess student learning at two levels: course review and program review. The TVCA Initiative has supported preliminary work by faculty, deans, and other leaders in shaping the key elements, but the real nuts-and-bolts of both levels remain to be designed in.

Course Review
Course review focuses on assessment of student learning at the course level, especially important to us in the high-enrollment, front-door courses that are decisive in student progress toward graduation. Course review rests on four elements that are led and designed by faculty and deans:

• Course outcomes: what students will be able to do in real life with what they learn in the course; outcomes involve students' core competencies (think, value, communicate, and act) because these competencies are woven into the academic disciplines.
• Course assessment tasks ("thin-slice"): small-scale, course-embedded assignments that demonstrate students' mastery of course outcomes and can be sampled to assess learning in the course.
• Course or discipline assessment teams: faculty teams who assess samples of student work on course assessment tasks; likely to be more effective if their make-up is inter-disciplinary (i.e., they include faculty from outside the discipline).
• Feedback that improves learning: assessment results that inform and facilitate conversations about improving student learning in our courses.

Program Review
Program review assesses student learning across a program, enabling the college and students themselves to assess their growing mastery of the core competencies (TVCA) and other program outcomes, as demonstrated by work they do in courses. Program review rests on five elements:

• The core competencies: the fundamental competencies of an educated person essential to success in the world beyond college.
• Portfolio assessment tasks ("thick-slice"): large-scale, course-embedded assignments that demonstrate students' mastery of the core competencies in a disciplinary context.
• The student portfolio: a mechanism for publishing large-scale student work for the purpose of the student's own self-assessment and the college's assessment of program effectiveness.
• Inter-disciplinary assessment team: a faculty team that assesses student portfolio work for its mastery of TVCA.
• Feedback that improves learning: assessment results that inform and facilitate conversations about improving student learning in our courses.

Some Strategic Principles for Discussion

• TVCA assessment should be designed and implemented by the faculty, deans, and other learning leaders-- those who will do the assessing and can use it to improve learning.
• TVCA assessment should involve students in their own self-assessment, especially at the level of program or portfolio assessment.
• Faculty participation in the TVCA assessment process needs to be voluntary.  It is best carried forward by people who believe in its value to teaching and learning.
• TVCA assessment should never be used to evaluate individual professors.
• In course review, student samples should be anonymous. Course review seeks to assess the quality of student learning in the course, not to grade students or evaluate professors.
• TVCA assessment is only worthwhile if it feeds back into improvements of teaching and learning. This feedback loop needs to be designed and implemented locally, by discipline faculty, deans, and other learning leaders. See http://faculty.valenciacc.edu/pbishop/tvca/overvw.htm for more information.

An examination of four dimensions of the implementation of assessment of learning:  the institutional culture, the level of shared responsibility, institutional support, and efficacy of assessment (NCA, The Higher Learning Commission, 2000) shows that Valencia is making progress but has by no means reached a mature stage of continuous improvement. 

In terms of the official institutional culture, there is evidence of shared values regarding learning-centeredness.  The recent collaborative re-write of the college vision and mission statements reflects these values.  Shared learning-centered values are also embodied in the student core competencies themselves, and in the new “Essential Competencies of a Valencia Faculty Educator,” developed for the Teaching/Learning Academy and endorsed by the Faculty Association and the College Learning Council.  A full understanding of general education assessment is still not widely held, however, and much work remains in this area to create a shared purpose, vocabulary of assessment, and a habit of practice.  Assessment of student competencies is progressing (see the Case Studies) but has not yet been broadly implemented. 

In terms of shared responsibility for assessment, those faculty who have taken part in the individual faculty development offerings over the past several years have the greatest understanding and commitment to this process.  The executive administrators and the Board of Trustees are very supportive of this movement toward continuous quality improvement as well.  Growing the practice will require deans to take a prominent leadership role in the future direction of TVCA Integration and Assessment, engaging their faculty in the process.  Students show some awareness of TVCA through surveys conducted under the Title III and Title V faculty development projects, and the Community College Survey of Student Engagement (CCSSE) shows that Valencia students experience significant curricular attention to these core abilities in their study at the college.

Institutional support for TVCA Integration and Assessment has been sufficient in terms of material resources; the real shortcoming is the lack of a clearly structured framework for making this work central to the functioning of the college.  TVCA work to date shows a pattern that is consistent with grant-funded efforts: the innovations and insights discovered are too easily marginalized.  TVCA Integration and Assessment needs to evolve into a systematic function carried on in campus departments to improve the results of our teaching, i.e., student learning.  This will require a new leadership role for both deans and veteran faculty.  It is a core element of the cultural shift that is alluded to above.

The efficacy of our TVCA Integration and Assessment project is not yet clear.  Implementation of assessment of learning is clearly in its early stages at Valencia.  We have made good pilot beginnings in research and practice. Through a collaborative process, the new Chief Learning Officer, Dr. Tracy Edwards, along with the President, is currently engaging deans and their faculty in a new wave of learning conversations that should serve to enhance the TVCA initiative and move it from the status of innovation at the margins to a core function of each academic department.  The timeline on the next page represents a possible approach to this effort.

	’02-‘03	’03-‘04	’04-05 and thereafter …
Fall	Discipline dialogue on course outcomes and embedded course assessment tasks in selected courses.	Discipline dialogue on course outcomes, with leadership from deans. Continuation of collection of assessment artifacts from courses addressed in previous year.	Implementation of enhancement designs and embedding of assessment tasks in front door courses; continued discipline dialogue on implementation and assessment process.
Spring	Dean program led by CLO to enhance dean ability to lead the evolving assessment program. More discipline dialogue on course outcomes and embedded assessment tasks in a few additional front-door courses. Assessment and analysis of samples of student work collected from fall term by discipline teams in courses selected above.	Discipline dialogue on course outcomes with leadership from deans. Assessment and analysis of samples of student work collected in fall term by discipline teams in courses selected above.	Assessment and analysis workshops by discipline, with leadership from deans: Based on the evidence from sample student performance, what can we learn about how students are learning in our course?
Summer	Deans engaged in design of Destinations ’04 as means of supporting faculty design of shared course assessments. Destinations ’03 continues individual approach to faculty development with emphasis on Action Research within individual classrooms.	Assessment development seminars under umbrella of Destinations ‘04, with Deans in leadership role, faculty in course-based teams. Assessment for improvement of student learning in front-door courses; design of course enhancements and course assessment tasks for fall implementation	Assessment development seminars by discipline: What do we need to learn and what do we need to do differently to improve student learning? Design of course enhancements and new or revised course assessment tasks for fall implementation.