Assessment Implementation Team Progress Report

Assessment Implementation Team Report
University of Oregon Division of Student Affairs

Initial Draft - June 1, 2009

Introduction

In December 2008, the Division of Student Affairs completed its first strategic plan outlining the Division’s core ideology and establishing seven goals to be met in the next three to five years. “Research and Assessment” was set as one of the goals and in February 2009 the Vice President for Student Affairs, Dr. Robin Holmes, charged the Assessment Implementation Team to outline a plan for meeting the following objectives meant to advance research and assessment within the division:

(1) Establish the infrastructure to lead the division’s assessment and research efforts
(2) Implement a division-wide assessment program and research agenda
(3) Increase research and assessment support to division units

In responding to this charge, the Assessment Implementation Team met extensively through March 2009 – June 2009. In an effort to understand the current state of assessment in student affairs, a thorough review of the literature was conducted (Aloi, Green and Jones, 2007, 2008; Anderson, Bresciani, and Zelna, 2004; Bresciani, 2002a, 2002b; Dungy, Keeling, Underhile, and Wall, 2008; Harms, 2001; Henni, Mitchell, and Maki, 2008; Onge, 1999; Upcraft & Schuh, 1996, 1998; Schuh and Upcraft, 2001, 2008), multiple peer institution’s division of student affairs assessment programs were discussed, and consultation with Oregon State University’s Office of Student Affairs Research and Evaluation was held.

Overall, this undertaking proved to be quite challenging given the developing nature of assessment within student affairs. From the onset of the committee’s initial meeting it was clear that assessment meant various things to different people. Individual explanations ranged in scale from the discrete and detailed to the vague and all-encompassing; each dependent upon one’s context (e.g. academic background, professional experience, current position). At times this posed significant communication challenges and often resulted in confusion, misunderstandings, and frustration. However, this process underscored the importance of developing a common language and striving to describe assessment in the most accessible and straightforward manner.

To that end, this report begins with a description of the assumptions that underpin the committee’s understanding of assessment and is followed by several key definitions. Shared philosophical foundation and language will lead to common practices within the division. In addition to a shared understanding of assessment, the proposed plan addresses the following areas which are meant to bring assessment into the espoused practices of the division:

• Working definition of assessment and the assessment process
• Division-wide assessment model
• Infrastructure to support division assessment program
• Divisional and departmental roles and responsibilities
• Implementation timeline
  

Philosophy of Assessment

Establishing a shared conceptual understanding of why we are engaging in assessment (Anderson, Bresciani, and Zelna, 2004). Nationally, the impetus for student affairs practitioners to engage in assessment has come in response to two interconnected factors: 1) growing pressure for increased fiscal accountability and education reform from accreditation bodies, the legislator, scholars and the public (Upcraft and Schuh, 2008); and 2) a call within the profession to align the work of student affairs professionals with the academic mission of their institutions and become active partners in intentionally advancing student learning (ACPA, 1996; ACPA and NASPA, 2004).

For the Division of Student Affairs at the University of Oregon, it is clear that these same influences had a part in setting the tenor and trajectory for the Division’s Strategic Plan and the subsequent need to engage in division-wide assessment. Hence the reason assessment and research was named as a goal.

The committee feels strongly that assessment is something that must become engrained in practices within the division. Consistent with the Division of Student Affairs Strategic Plan, we will engage in assessment for the collective purpose of:

(1) Improving our role as educators and aspiring to our core purpose, “to transform and advance student learning”
(2) Improving the effectiveness of our programs and services
(3) Generating evidence to guide decision making and strategic planning
(4) Substantiating our impact on students, campus life, and learning
(5) Unifying the diverse aims of divisional units

Definitions

The following definitions support the philosophy of assessment and begin to clarify which assessment practices align with and support the division’s needs.

Assessment – The ongoing, systematic process of collecting, analyzing, and using information about divisional, departmental, and programmatic effectiveness, in order to improve student learning (Upcraft & Schuh, 1996; Anderson, Bresciani, & Zelna, 2004).

Evaluation – “[A]ny effort to use assessment evidence to improve institutional, departmental, divisional, or agency effectiveness” (Upcraft & Schuh, 1996, p.19)

Research – In contrast to assessment, which “guides good practice,” research “guides theory development and tests concepts” and has “broader implication for student affairs and higher education” (Upcraft & Schuh, 2001, p. 5).

Student Learning – “Learning is a complex, holistic, multi-centric activity that occurs throughout and across the college experience. Student development, and the adaptation of learning to students’ lives and needs, are fundamental parts of engaged learning and liberal education. True liberal education requires the engagement of the whole student – and the deployment of every resource in higher education” (Learning Reconsidered, 2004, p. 6).

Working Definition of Assessment

For the purpose of this report and subsequent assessment plan, the Assessment Implementation Team has defined assessment as the ongoing, systematic process of collecting, analyzing, and using information about divisional, departmental, and programmatic effectiveness, in order to improve student learning. While this formal definition adequately encapsulates the parameters and purpose of assessment for the division, it may not be the ideal vehicle for clearly and consistently communicating what assessment means to all members of the division, representing a diversity of backgrounds, job functions, and perspectives of assessment.

To that end, we have also developed the following working definition:

Assessment is the systematic process that allows us to answer the question, “Are we accomplishing what we have set out to do?”

Assessment Process

Though simple, responding to the question requires attention to each step of the assessment process (See Fig. 1):

• Articulate Mission, Goals, and Outcomes – In order to answer the question we must know what we intend to do. This begins with the recognition that we, the Division of Student Affairs, do our work as a collective whole with a core purpose, “to transform and advance student learning.” It also requires that each unit, from the top-down, spell out what they do in light of this purpose and their unique function.
• Implement Methods to Deliver Outcomes – A response also demands that we have determined how we intend to do it. The intended mission, goals and outcomes of the division and departmental units are implemented through intentionally designed programs and services.
• Identify Methods to Gather Evidence – Determining accomplishment necessitates that we know what it will look like when we have achieved our outcomes. Markers of success are identified for each outcome, and appropriate assessment methods and tools are selected to measure them.
• Interpret Evidence – Using the methods identified above evidence is collected and analyzed. Resulting information is reviewed and level of accomplishment is interpreted.
• Use Evidence to Evaluate Effectiveness – Once results are reviewed and interpreted, this information is used to take note of accomplishments and improve what we do. We evaluate our efforts by taking the information collected to determine how we can make our programs and services more effective in achieving our desired outcomes.
• Repeat – In answering the question the process naturally begins again. Programs and services are tweaked, updated, and re-implemented. New programs are researched, designed, and initiated for the first time. And again, the question is posed, “Are we accomplishing what we have set out to do?”
  

Figure 1: The Assessment Process

(Adapted from Bresciani, 2003)

Division of Student Affairs Assessment Model

Consistent with the Division of Student Affairs Strategic Plan and relevant assessment literature (Anderson, Bresciani, and Zelson, 2004; Schuh and UpCraft, 1996, 2001, 2008), the Assessment Implementation Team proposes that the above assessment process be adopted across the Division, at the divisional, unit, and program/services level, and implemented in a comprehensive manner.

Components of Assessment Model

In their extensive work on student affairs assessment, Schuh and Upcraft (1996, 2001) identified the essential components of a comprehensive model of assessment: tracking, needs assessment, satisfaction assessment, campus environment and student culture, outcomes assessment, comparable institution assessment, national standards assessment, and cost effectiveness assessment.

Although the measurement of these elements is critical, it is not expected that information about each component be gathered on an annual basis. Some components will be assessed annually, such as tracking and outcomes, and others on a more infrequent basis (i.e. comparable institution assessment, national standards assessment). Each component is further described below:

Tracking – monitoring who uses our programs, services and facilities (e.g. raw numbers, frequency, age, class standing, gender, race, residence, etc).

Needs Assessment – identifying needs of our students and clientele (e.g. student perceived, research supported, and institutionally expected).

Satisfaction Assessment – measuring the level of student and clientele satisfaction with our programs, services, and facilities.

Student Cultures and Campus Environments Assessment – assessing the collective perception of campus and student experience (e.g. campus climate, academic environment, residential quality of life)

Outcomes Assessment – measuring the impact our services, programs and facilities have on students’ learning, development, and student success.

Comparable Institution Assessment – identifying how the quality of our programs, services and facilities compare with peer institutions’ best practices.

National Standards Assessment – using nationally accepted standards to assess our programs and services (e.g. national assessment inventory– EBI, CAS standard self-assessment, departmental review by consulting group).

Cost Effectiveness Assessment – determining whether the programs, services and facilities we offer to students are worth the cost.

Setting Division Assessment Priorities

The adoption of a comprehensive assessment model across the Division of Student Affairs is not a neutral undertaking. The Division’s strategic plan sets a distinct trajectory for our work and in doing so, suggests an order of priority for our assessment efforts. Our core purpose, “to transform and advance student learning,” and impeding curricular framework, emphasizes our role as educators and distinct impact on the learning process.

As such, the priorities of the Division of Student Affairs in implementing an inaugural assessment model are as follows:

(1) Delineate, implement, and measure student learning outcomes consistent with Division of Student Affairs Strategic Plan and the Curriculum Implementation Team’s divisional curriculum framework, to improve our effectiveness as educators and approach student learning with greater intentionality.

(2) Delineate, implement, and measure outcomes related to our facilities and services.

(3) Maintain existing assessment efforts.

(4) Work to incorporate new components of the comprehensive assessment model, based on unit need, to improve overall divisional effectiveness

Figure 2: Division of Student Affairs Comprehensive Assessment Model – Assessment Priorities

Implementation of the Model

The implementation of the proposed model is a significant undertaking that will require the establishment of new divisional roles and responsibilities, infrastructure and staff development plan.

Divisional Roles and Responsibilities

In addition to new infrastructure the implementation of a comprehensive assessment process will require divisional units, and respective staff, to acquire new roles and responsibilities.

1) Divisional-Level Assessment – assessment at the divisional level will be centralized and administered by the Student Affairs Office of Research and Assessment in conjunction with Division of Student Affairs Assessment Committee.

2) Departmental and Programs/Service Assessment – assessment at the departmental and programs/services level will be decentralized and the principal responsibility of each respective unit of the Division of Student Affairs. Each unit will be required to fulfill following functions:

a. Align respective mission and goals to the Division of Student Affairs Strategic Plan and University Mission

b. Delineate outcomes for programs and service in light of the Curriculum Implementation Team’s divisional curricular framework and the division’s strategic plan

c. Develop an annual assessment plan, delineating assessment priorities, identifying assessment measures and outlining an implementation timeline

d. Coordinate and administer assessment efforts throughout the year

e. Write a year-end assessment report

f. Use assessment information to update and redesign unit programs and services

Figure 2: Division of Student Affairs Comprehensive Assessment Model – Assessment Responsibilities

Infrastructure

To oversee and manage the assessment needs and requirements across the division it is the recommendation of the Assessment Implementation Team that two new roles be created within the division. Since our recommended model includes both centralized and decentralized assessment practices, the establishment of both a unit within the division and a divisional committee will ensure the successful implementation of the proposed model.

Personnel

It is the recommendation of the team that a new Division of Student Affairs unit be established:

1) Proposed Unit Name – Office of Student Affairs Research and Assessment

2) Unit Role

a. Provide leadership in the areas of research and assessment to the division

b. Coordinate assessment efforts across the division at all levels

c. Set division-level research and assessment agenda and implement research/assessment projects

d. Provide consultation and support to units of the division in the areas of outcome development, instrument construction and implementation, data collection and statistical analysis, and report writing.

e. Provide ongoing staff development to increase the assessment capacity of the division

f. Establish and maintain vehicle for cataloging and communicating the division’s research and assessment results

g. Initiate campus-wide assessment efforts and develop cross-divisional relationships

3) Staffing

a. Director of the Student Affairs Office of Research and Assessment

i. Provide leadership, supervision, and management of the research and assessment efforts for the Division of Student Affairs research

ii. Set division-level research and assessment agenda and initiate research/assessment projects

iii. Initiate collaborative interactions between divisional and campus-wide groups to support and manage assessment efforts

iv. Provide consultation and support to divisional units

v. Chair the Student Affairs Assessment Committee

b. Assistant Director/Coordinator of Student Affairs Assessment

i. Establish and maintain communication and cataloging vehicle for the division’s research and assessment results

ii. Administer and facilitate division-wide staff development plan

iii. Review unit assessment plans and offer ongoing consultation

iv. Manage the Student Affairs Office of Research and Assessment website

v. Manage StudentVoice

c. Assessment Support Staff (2) – graduate /professional staff

i. Facilitate staff development efforts

ii. Assist in implementation the development and implementation of division-level research and assessment project

iii. Administer data analysis for division and unit assessment efforts

iv. Assist in research and assessment report writing

In addition to the creation of a new unit the Assessment Implementation Team recommends the establishment of a division assessment committee.

1) Proposed Committee Name- Division of Student Affairs Assessment Committee

2) Committee Charge

a. Establish unit-level assessment expectations

b. Establish formalized structure for unit-level assessment plans and reports

c. Provide leadership and guidance for division-wide assessment model by outlining divisional assessment and research agenda

d. Assist in the coordination of assessment efforts within the Division of Student Affairs

e. Serve as unit liaison to Office of Student Affairs Research and Assessment, communicating departmental interests and training needs

f. Assist in the facilitation of staff development and consultation initiatives

g. Assist in the implementation of division-level research and assessment projects

3) Representation – the assessment committee would be comprised at least one representative from each divisional unit.

Assessment Resources

StudentVoice

The division has contracted with StudentVoice to contribute to the infrastructure that will support multilevel assessment efforts. StudentVoice can assist the division in the data collection step of the assessment process through the creation and administration of web-based surveys.

The committee recommends the following to make optimal use of StudentVoice within the division:

(1) The Office of Student Affairs Research and Assessment will manage divisional relationship and communication with StudentVoice

(2) StudentVoice may be used on a divisional, unit and program/service level and supports the assessment model. Therefore, access to StudentVoice should be granted to individuals within the division who wish to make use of the services provided to fulfill unit or program assessment agendas.

Additional Resources

In addition to StudentVoice, the committee recommends that supplementary resources be identified that will expand the divisions ability to implement the proposed assessment model. These include:

• Identifying campus experts in assessment and research design
  
• Identifying existing programs at the UO or in the Eugene community for long term or limited use
• Identify national assessment and research instruments and coordinate participation with other campus entities
  

Staff Development Plan

As the division prepares to implement the assessment process, it will be imperative for professional development opportunities to be offered that will increase our ability to create a culture where assessment is practiced. Since we are interested in creating a culture of assessment, we will follow the assessment process previously defined in this report to develop our professional development program for the division.

• Articulate Mission, Goals, and Outcomes – What do we intend to do?
  We intend to increase the assessment capacity of the division.
• Implement Methods to Deliver Outcomes – How we intend to do it?
  We intend to develop a comprehensive professional development program for individuals within the division of student affairs. The ACPA Assessment Skills and Knowledge Content Standards (ASK Standards), provides defined learning outcomes that would demonstrate a professional’s knowledge about assessment. Programs and services would be developed in each of these areas and offered through a variety of mediums to increase the knowledge and skills associated with assessment.
  
  Content Standard 1: Assessment Design
  Content Standard 2: Articulating Learning and Development Outcomes
  Content Standard 3: Selection of Data Collection and Management Methods
  Content Standard 4: Assessment Instruments
  Content Standard 5: Surveys Used for Assessment Purposes
  Content Standard 6: Interviews and Focus Groups Used for Assessment Purposes
  Content Standard 7: Assessment Methods: Analysis
  Content Standard 8: Benchmarking
  Content Standard 9: Program Review and Evaluation
  Content Standard 10: Assessment Ethics
  Content Standard 11: Effective Reporting and Use of Results
  Content Standard 12: Politics of Assessment
  Content Standard 13: Assessment Education
• Identify Methods to Gather Evidence – How will we measure learning outcomes?
  
  Division Level
  We will use a survey to measure professionals’ aptitudes, experiences and desired experiences with assessment using a survey administered through Student Voice prior to offering the professional development opportunities. We will administer this survey in the summer of 2009 to establish a baseline. Then, professional development programs will be available for individuals over the course of the academic year. We will then administer the same survey through Student Voice in the summer of 2010.
  
  Program and Service Level
  After each individual program or service is offered, participants will be asked to respond to a short survey to measure the learning that occurred in the particular area covered during that session.
  
  Individual Level
  Each individual will have the opportunity to track their own knowledge and skills in each content area and identify areas where they wished to gain. This will assist the individual in selecting and attending programs that will best fit their learning needs.
• Interpret Evidence – What do these results mean?
  What did professionals in the division report learning during their engagement with the various programs designed to increase their knowledge of assessment? What can we report as the impact of the designed program and services? What were our shortfalls?
• Use Evidence to Evaluate Effectiveness – Now that we have results and have interpreted their meaning, we must use this information to improve what we do. We evaluate our efforts by taking the information collected and determining how we can make our programs and services more effective in achieving our desired outcomes.
• Assessment is an Iterative Process - Now we return to the beginning step of this process, creating a continuous loop. The results will allow us to answer the question at the beginning, “Are we accomplishing what we have set out to do?” If we were not successful, then we can use the results to inform what we change about the vehicles we use to impact learning. If we were successful, then we know that we can continue what we are doing because we know how it impacts learning.

Implementation Timeline (incomplete)

Short Term (3 – 6 months)

• Reconcile the proposed assessment model with the recommendations of Curriculum Implementation Team
• Identify an Interim Director of Assessment
• Establish Student Affair’s Assessment Council. Identify at least one representative from each unit.

Medium Term (6 - 12 months)

• Provide intensive training for committee members of the Student Affair’s Assessment Committee. (American College Personnel Association, American Educational Research Association, Association for Institutional Research, Association for the Study of Higher Education, Association for the Study of Higher Education) ACPA Student Affairs Assessment Institute, NASPA International Assessment and Research Conference
• Introduce the curricular goals and assessment plan to the Division of Student Affairs
• Provide division-wide professional development opportunities around outcomes…
• Require each unit to develop and/or refine their respective mission, goals, and outcomes…
• StudentVoice…

Long Term (12-24 months)

• End of Winter Term 2010 each unit submit mission, goal, outcomes

 

Conclusion

The work of assessment is a comprehensive and dynamic process aimed at improving the overall effectiveness of all levels of the division. The implementation of a division-wide assessment program is crucial to achieving the goals of the strategic plan and operationalizing our core purpose, “to transform and advance student learning”.

This report is by no means complete. There are many more important questions that have yet to be answered and in some cases this report has only added to that list. However, this plan represents an important first step in reaching an assessment plan for the division and will hopefully serve as a catalyst for future planning and conversations.

 

References

Aloi, S., Green, A. S., & Jones, E. (2008). An exploration of high-quality student affairs learning outcomes assessment practice. NASPA Journal, 45(1), pp. 133-157

Aloi, S. L., Green, A. S., & Jones, E. A. (2007). Creating a culture of assessment within western virginia university’s student affairs division. Assessment Update, 19 (2), pp. 7-9.

American College Personnel Association. (1996). The student learning imperative: Implications for student affairs. Journal of College Student Development 37, 118-122.

American College Personnel Association, & National Association of Student Personnel Administrators (2004). Learning reconsidered: A campus-wide focus on the student experience. Washington, DC: Authors.

Anderson, J. A., Bresciani, M. J.,& Zelna, C. L. (2004). Assessing student learning and development: A handbook for practitioners. Washington, DC: National Association Student Personnel Administrators.

Bresciani, M. J. (2002a). An updated outline for assessment plans. NetResults [Online]. Retrieved from http://www.naspa.org/

Bresciani, M. J. (2002b). Criteria checklist for an assessment program. NetResults [Online]. Retrieved from http://www.naspa.org/

Dungy, G. J., Keeling, R. P., Underhile, R., & Wall, A. F. (2008). Assessment reconsidered: Institutional effectiveness for student success. International Center for Student Success and Institutional Accountability.

Harms, J. Y. (2001). Identifying the assessment needs of student affairs professionals using a web-based survey. Paper presented at the Annual Conference of the National Association of Student Personnel Administrators. Seattle, WA.

Henning, G. W., Mitchell, A. A., & Maki, P. L. (2008). The Assessment Skills and Knowledge Standards: Professionalizing the Work of Assessing Student Learning and Development. About Campus. 13(4), 11-17.

Onge, S. S. (1999). Developing a strategic plan for assessment in student affairs. Assessment Update, 11 (4).

Oregon State University Student Affairs Assessment Council. (March 2009). Why do assessment? Oregon State University. Downloaded on May 30, 2009 from http://oregonstate.edu/studentaffairs/docs/white.paper.03.09.pdf

Schuh, J. H., & Upcraft, M. L. (1996). Assessment in student affairs: A guide for practitioners. San Francisco: Jossey-Bass

Schuh, J. H., & Upcraft, M. L. (1998). Facts and myths about assessment in student affairs. About Campus, 2 (5), pp. 2-8.

Schuh, J. H., & Upcraft, M. L. (2001). Assessment practice in student affairs: An application manual. San Francisco: Jossey-Bass.

Schuh, J. H., & Upcraft, M. L. (2008). Assessment methods for student affairs. San Francisco: John Wiley & Sons, Incorporated