3/30/16
AUN Criterias, SELF ASSESSMENT REPORT (SAR) V.3 Leni S. Heliani Kantor Jaminan Mutu UGM 2016
Status Akreditasi/sertifikasi P rodi (S1) Status P enjaminan Mutu UGM Internal : Audit Mutu Internal siklus 1 tahun Ekternal Nasional : 72 % terakreditasi A (dari 258 prodi) Regional ( sertifikasi AUN): -‐ 26 Prodi (S1), 2016 ( 6 prodi) • Internasional: -‐ AACSB : 8 prodi fakultas Ekonomi B isnis ( 2014) -‐ T. Kimia : I-‐Chem-‐E (1 prodi) -‐ Mipa Kimia : R oyal Chemistry -‐ Sistem informasi Medis (1 prodi) -‐ S1 Kedokteran : F ascu • • • •
1
3/30/16
References • AUN-‐QA , Guide to AUN-‐QA Assessment at Programme Level, 2015 • AUN-‐QA , Guide to AUN-‐QA Assessment at Programme Level, 2012 • Materi Pengembangan SAR: Jonson Ong, Titi Savitri
1. Quality Assurance in Higher Education • Internal Quality Assurance • External quality Assurance -‐ Accrediation -‐ Certification -‐ Audit
4
2
3/30/16
Certification AUN • Type of certification: -‐ Instiutional -‐ Program : undergraduate • Fokus on quality improvement process • Outcome based assessment • Discipline: All
5
AUN-‐QA Models for Higher Educatio
3
3/30/16
AUN-‐QA Model for Institutional Level
AUN-‐QA Model for IQA System
4
3/30/16
AUN-‐QA Model for Programme Level (1st)
AUN-‐QA Model for Programme Level (2nd)
5
3/30/16
AUN-‐QA Model for Programme Level (3rd)
Comparison of 3 versions criterias
6
3/30/16
Reference 2011
Ong, 2012
14
7
Reference 2015
3/30/16
Structure of each criteria (in the manual) • AUN –QA standard: requirements for each criteria to meet the AUN-‐standard • Checklist: assesment/evaluation points • Explanation of criteria • Diagnostic questions: list of questions that help the team to develop the AUN-‐SAR and the assessor to find the evidence in meeting w. the criteria • Sources of Evidence 16
8
3/30/16
AUN-‐ Criteria 1. Expected Learning Outcome Criteria/standard:
17
b.Expected Learning Outcome (ELO)
18
9
3/30/16
c. Checklist
19
d. Explanation • Outcomes-‐based education (OBE) : a w ay in w hich curriculum is defined, organised and directed based on all the things that learners would learn and demonstrate successfully w hen they complete the study programme. • The focus of OBE is on the results of learning, w here the knowledge, skills and attitudes including habits of m ind, the learners are expected to learn are clearly identified and expressed as ELOS • The ELOs are formulated from the needs of the stakeholders, form the starting point of the programme design. • ELOs are concerned with the achievements of the learner rather than the intentions of the teacher, w hich are often w ritten as aims, goals or objectives of the programme. 20
10
3/30/16
International accreditation/certification • Outcome based education è Outcame based assessment
21
2. Learning design
22
(Felder&Bert dalam Samadi, 2015)
11
3/30/16
Outcome Base Education OBE addresses following key questions: -‐ What do the student to have to be able to do? -‐ How can we best help student achieve it? -‐ How we will know whether they have achieved it? -‐ How do we close the loop for futher improvement (CQI)? 23
Steps Of Learning Design: • Develop learning objectives/outcomes • Check for alignment between the curriculum and the objectives/outcomes • Develop an assessment plan • Collect assessment data • Use results to improve the program • Routinely examine the assessment process and correct, as needed (Allen 2004) 24
12
3/30/16
Outcomes Pyramid: Goals, Objectives and Outcomes
http://assessment.uconn.edu/
25
Goals, Objectives, and Outcome
http://assessment.uconn.edu/
26
13
3/30/16
Assessment plan
Student learning outcomes, s hould derive f rom t he objectives which f low f rom the goals and mission of t he program. An Assessment Plan includes all t hese components and f ocuses on t he assessment of each learning outcome and how the evaluation of t he results of t he assessment are used f or program improvement (http://assessment.uconn.edu/, 2015) 27
d. Diagnostic Questions
28
14
3/30/16
e. Sources of Evidence Programme and course specifications Course brochure and prospectus or bulletin Skills matrix Stakeholders’ input University and faculty websites Curriculum review minutes and documents Accreditation and benchmarking reports
29
2. Programme Specification
15
3/30/16
b. AUN Programme Specificaton format
31
c. Checklist
32
16
3/30/16
3. Programme Structure and Content Criteria/standard:
33
b. Checklist
34
17
3/30/16
c. Structured-‐ Curricullum Teaching methods
Student assessmnet methods
Course LO
ELO at Program level
Staff quality & infrasructures
35
c. Structured-‐Curriculum • Constructive alignment between the courses, teaching and learning also assessment methods • Constructive alignment includes: ü defining ELOs that are measurable; ü selecting teaching and learning methods that are likely to ensure that the ELOs are achieved; and ü assessing how well the students have achieved the ELO as intended.
18
4. Teaching and Learning Approach
3/30/16
The Learning Pyramid
19
3/30/16
b. Checklist
5. Student Assessment
20
3/30/16
b. Checklist
c. Evident • Samples of in-‐course assessment, project work, thesis, final examination, etc. • Rubrics • Marking scheme • Moderation process • Appeal procedure • Programme and course specifications • Examination regulations
21
3/30/16
SAR Development Approach
or P DCA
Ong, 2012
22
3/30/16
23
3/30/16
24
3/30/16
PDCA APPROACH Plan Do Check Act
P A
D C
What Make a Good SAR? No. 1
Criteria Expected Learning Outcomes
What a Good SAR s hould have? • • • • • • •
How is feedback from s takeholders gathered and u sed for the formulation and revision of ELOs? ( related to criterion 1 3) Who are involved in the formulation and revision of ELOs and h ow often are ELOs u pdated or revised? How ELOs are formulated for the p rogramme and all courses, and the relationship b etween them? (related to criterion 3 ) What taxonomy of educational objectives (e.g. Bloom) is u sed for the formulation of ELOs? How are ELOs aligned to teaching & learning activities and assessment? ( related to criterion 4 and 5 ) What life-‐long learning skills are p romoted and taught? What p athways and p rofessional p rogression or d evelopment are available to s tudents and graduates for life-‐long learning? 50
25
3/30/16
What Make a Good SAR? No. 2
Criteria
What a Good SAR s hould have?
Programme • Specification • •
Is t he programme specification c onformed t o t he AUN’s programme specification (page 17) of t he “Guide t o AUN Actual Assessment at Programme Level? How is programme specification (print and non-print) communicated and made available t o s takeholders? How many languages is t he programme specification translated?
51
What Make a Good SAR? No. 3
Criteria
What a Good SAR s hould have?
Program • Structure and Content •
•
•
What are t he university and f aculty v ision and mission and how these are reflected in t he programme objectives and ELOs? How is t he c urriculum and s yllabi developed and how are they s tructured, organised and related? (related t o c riterion 11) What benchmark is used f or t he design and development of the c urriculum and syllabi? How often are t he c urriculum and s yllabi revised and updated and who are involved? Related t o c riterion 11).
52
26
3/30/16
What Make a Good SAR? No.
4
Criteria
Teaching and Learning Strategy
What a Good SAR s hould have?
• What is the teaching a nd learning strategy a nd what philosophy is b ehind it? • How is it demonstrated in teaching a nd learning activities a nd a ssessment? (related to criterion 5 ) • What a nd h ow is academic staff trained in the teaching and learning strategy a nd methods? (related to criterion 6 a nd 1 2) • What a nd h ow is the e ffectiveness o f teaching a nd learning e valuated? What is the trend a nd what is d one to improve the trend (related to criterion 11)
53
What Make a Good SAR?
No. 5
Criteria
Student Assessment
What a Good SAR s hould have?
• • •
• • • •
How are ELOs, t eaching and learning s trategy and assessment aligned? (related t o c riterion 1 and 4) What are t he t ypes and methods of assessment used throughout t he c ourse of s tudy? What is t he assessment rubric and how it is f ormulated t o t est the achievement of ELOs f or each c ourse? (related t o criterion 1) When and how is t he assessment requirements, methods and criteria c ommunicated t o s tudents? How is in-course assessment and f inal results c ommunicated to s tudents? What is t he s tudent appeal process f or examination results? What is t he quality assurance process f or s tudent assessment t o ensure f air, accurate and c onsistent marking by academic s taff? (related t o c riterion 11) 54
27
3/30/16
What Make a Good SAR?
No. 6
Criteria
What a Good SAR s hould have?
Academic Staff • • Quality •
• • • • • • •
What are the roles, responsibilities, ethics and accountability of academic staff? What is the current number, qualifications, experiences, and profile (age and gender) of full-time and part-time academic staff? How has the number of academic staff grown in the past? What is the forecasted number of academic staff in the future? How they have grown or shrink in respond to student cohort sizes? What is the ratio of staff/student? How is this compared to other programmes, faculties and universities? How are academic staff selected, appointed, promoted? How are academic staff appraised and rewarded for their performance (teaching, research and service)? (related to criterion 14) How are training needs of academic staff identified? (related to criterion 12) What is the current and future training and development plan for academic staff (related to criterion 12) How are academic staff redeployed, transferred, terminated and retired? What are the research activities (projects, papers, presentation, publication, etc.) and funds carried out by academic staff over the past 5 years? (relate to criterion 14) 55
No.
8
Criteria
Staff Quality
What a Good SAR s hould have?
• What is the current a nd future training p lan & b udget for academic a nd support staff? (related to criterion 6 , 7 ) • What is the training & d evelopment p rocess a nd h ow are training n eeds identified for a cademic a nd support staff? (related to criterion 6 , 7 ) • What a re the key p erformance indicators for measuring and monitoring training a nd d evelopment a ctivities? How did they p erform o ver the last 5 years? What is being d one to improve the trend (related to criterion 6 , 7)
28
3/30/16
What Make a Good SAR? No. 7
Criteria
What a Good SAR s hould have?
Support s taff • Quality • • • •
What is t he number, qualifications, experiences and profile of support s taff in t he libraries, laboratories, c omputer f acilities, student s ervices at t he university, f aculty and department? How are t he s upport s taff appointed, s elected, promoted, appraised, rewarded? How are t raining needs of s upport s taff identified? (related t o criterion 12) What is t he c urrent and f uture t raining and development plan for s upport s taff? (related t o c riterion 12) What are t he k ey performance indicators of s ervice provided by libraries, laboratories, c omputer f acilities and s tudent services? How are these KPIs monitored and reported? What is t rend of KPIs performance in t he last 5 y ears? What is done to improve t he t rend (related t o c riterion 10, 13, 15) 57
What Make a Good SAR?
No. 8
Criteria
Student Quality and Support
What a Good SAR s hould have?
• •
• • • •
•
How many s tudent intakes in a y ear? What is t he number and profile of s tudents applied and enrolled for t he programme over t he last 5 y ears? What is t he acceptance ratio over t he s ame period? What are t he basic requirements f or s tudents t o apply f or t he programme? Where are the s ources of s tudents and how are t hey s elected and admitted? How the quality of admitted s tudents f rom different s ources co-relate t o t heir academic performance (GPA)? What is t he c redit s ystem and how many c redits (minimum and maximum) can a student t ake in a s emester? I s t he s tudy load related t o GPA? How it works? What are t he t ypes of s cholarship available t o s tudents? 58
29
3/30/16
No.
What Make a Good SAR? Criteria
8
What a Good SAR s hould have?
• • • • • • • • •
What student advice and services are available at the university and faculty level? (related to criterion 7) What are the roles of academic staff and support staff in providing student advice and support (related to criteria 6 and 7) How students get their feedback on in-course assessments, project works, assignments, examinations, etc. (related to criterion 5, 15) How are academic staff appointed as academic advisers and how are they assigned to the students? (related to criterion 6). What is the system of communication and monitoring of student academic performance? (related to criterion 15) What are the types of activity (cultural, social, sports, recreation, etc.) organised for students? (related to criterion 15) What are the campus amenities available to students? (related to criterion 15) What mental well-being services (i.e. counseling, psychiatry, stress management, etc.) available to students? (related to criterion 15) What career services and advice are provided to students? (related to 59 criterion 15)
What Make a Good SAR?
No. 9
Criteria
What a Good SAR s hould have?
Facilities and • Infrastructure • •
• •
• •
What are the types and number of facilities and infrastructure available (lecture facilities, libraries, laboratories, computer facilities)? What amount of budget per annual is allocated for facilities and infrastructure maintenance, replacement and upgrading? What are the key performance indicators for monitoring user satisfaction, condition and usage of the facilities and infrastructure? What is the trend of user satisfaction in the last 5 years? What is being done to improve the trend? (related to criterion 13, 15) What activities on safety, health and environment are organised? What are the personal protective equipment (gowns, gloves, helmets, safety shoes, etc.) available to students and staff to protect their personal well-being? How are fire-fighting equipment and medical aids placed and maintained? What is the fire evacuation plan? 60
30
3/30/16
What Make a Good SAR? No. 10
Criterion Quality Enhancement
What Make a Good SAR? •
• •
•
What is t he c urriculum design & development process and who are t he s takeholders involved? (related t o criterion 1, 3, 6, 13) What improvements have been made to t he c urriculum over t he y ears? (related t o c riterion 3) How is quality assurance organised at t he university, faculty and department level and what quality assurance activities are being c arried out? What activities are c arried t o ensure quality in t eaching & learning and student assessment? What improvements have been made in t hese areas? (related t o c riterion 4, 5)
Copyright: Johnson Ong, AUN-‐QA Expert
No.
10
Criteria
What a Good SAR s hould have?
Quality • What is the p rocess o f g athering feedback from Enhancement students, a lumni members, a cademic a nd support staff, e mployers, p rofessional b odies, e tc? (related to criterion 1 , 7 , 1 0, 11, 1 5) • What is the frequency, response rate a nd sample size of e ach feedback from students, a lumni members, a cademic staff, support staff, e mployers, etc. o ver time? What a re b eing d one to improve the response rate? • How is the feedback u sed for improvement? (related to criterion 11)
31
3/30/16
What Make a Good SAR?
No. 11
Criteria
Output
What a Good SAR s hould have?
•
•
•
What is t he t rend of s tudent pass rates, t ime t o graduate and employability over t he last 5 y ears? How do they perform against benchmarked universities? What are being done to improve t he t rend? What is t he allocated f und f or research over t he last 5 years? How is t his c ompared t o other programmes, faculties and universities? What are t he t ypes and v olume of research activities carried out by s tudents and academic s taff over t he last 5 years? How do they perform against benchmarked universities? What are being done to improve t he t rend? (related t o c riterion 5, 6) 63
What Make a Good SAR?
No.
Criteria
What a Good SAR s hould have?
• What indicators are used to measure stakeholders’ satisfaction (students, staff, alumni, employers, etc? What is the trend of the indicators in the last 5 years? (related to criterion 13) • How satisfied are employers with graduate quality as compared to graduates from other universities? • What is the plan to improve stakeholders’ satisfaction? 64
32
3/30/16
Collect Data and EVIDENCES Evidences AUN-QA Criteria 1 Expected Learning Outcomes 2 Programme Specification 3 Programme Structure & Content
Possible Evidences Programme & module specifications, c ourse brochure & prospectus, s kills matrix, s takeholders’ inputs, university & f aculty website, c ommunication media & plans t o s takeholders, c urriculum review minutes, accreditation & benchmarking reports, Programme Specification
4 Teaching & Learning Strategy
Teaching & learning strategy, evidence of action learning, s tudent f eedback, online learning portal, module specification
5 Student Assessment
Assessment c riteria, s amples of in-course assessment, project work, f inal examination, marking scheme, moderation process, appeal procedure
6 Academic Staff Quality
Recruitment c riteria, s taff qualifications, peer review & appraisal s ystem, c areer plan, s tudent f eedback, award & recognition s ystems, s taff workload, allocation of roles and duties, t ermination & retirement s chemes 65
Do
P53-66
Collect Data and Evidences AUN-QA Criteria
Possible Evidences Education, t raining and development policy and plan, training places & hours, s cholarships, job rotation & deployment s cheme
7
Support Staff Quality
Number, type and qualification of s upport s taff, c areer plan, t raining plan, appraisal s ystem, award & recognition s chemes, s tudent/faculty f eedback, Education, t raining and development policy and plan, training places & hours, s cholarships, job rotation & deployment s cheme
8
Student Quality and s upport Student s election process, t rend of s tudent intakes, credit s ystem, s tudent workload, s tudent performance reports, Mechanisms t o report and f eedback on student progress, c oaching, mentoring and counselling s chemes, s tudent f eedback
9
Facilities and I nfrastructure Number and t ype of f acilities, utilisation rates, downtime/uptime, maintenance plan, new facilities and upgrading plans, s afety & health policy, f acilities booking s ystem 66 Do P 53-66
33
3/30/16
Collect Data and Evidences
AUN-QA Criteria
Possible Evidences
10 Quality Anhancement
Curriculum design, review & approval process and minutes, QA of assessments/examinations, stakeholders’ inputs, external examiners, s tudents feedback
11 Output
Pass/drop-out rates, employment s tatistics, entry-level salary, employers f eedback, average t ime t o graduate, research, Process f or measuring s takeholders satisfaction, s takeholders s atisfaction t rends, perceptions of graduates & university
67
Do
P53-66
SAR Contents
68
34
3/30/16
SAR Contents
69
To prepare a creditable and objective report, the assessment team has to verify the evidences gathered and agree on the strengths and weaknesses of the QA practices adopted by the university. Next is to establish the gaps against the AUN-QA criteria and suggest areas for improvement. Based on the findings, the assessment team has to establish and agree on the level of performance or rating. Any differences should be resolved through factual and objective evidences against the best known practices. Reconciliation of ratings of common criteria across programmes should be carried out to ensure consistency of results.
Scoring Scale
A 7-point rating scale is used for AUN-QA assessment. It provides universities and assessors an instrument for scaling their verdicts and to see how far they have progressed in their AUN-QA journey. The 7-point rating scale is described below. Rating
Description
1
Absolutely Inadequate The QA practice to fulfil the criterion is not implemented. There are no plans, documents, evidences or results available. Immediate improvement must be made.
2
Inadequate and Improvement is Necessary The QA practice to fulfil the criterion is still at its planning stage or is inadequate where improvement is necessary. There is little document or evidence available. Performance of the QA practice shows little or poor results.
3
Inadequate but Minor Improvement Will Make It Adequate The QA practice to fulfil the criterion is defined and implemented but minor improvement is needed to fully meet them. Documents are available but no clear evidence to support that they have been fully used. Performance of the QA practice shows inconsistent or some results.
4
Adequate as Expected The QA practice to fulfil the criterion is adequate and evidences support that it has been fully implemented. Performance of the QA practice shows consistent results as expected.
5
Better Than Adequate 70 The QA practice to fulfil the criterion is better than adequate. Evidences support that it has been efficiently implemented. Performance of the QA practice shows good results and positive improvement trend.
6
Example of Best Practices The QA practice to fulfil the criterion is considered to be example of best practices in the field. Evidences support that it has been effectively implemented. Performance of QA practice shows very good results and positive improvement trend.
7
Excellent (Example of World-class or Leading Practices) The QA practice to fulfil the criterion is considered to be excellent or example of world-class practices in the field. Evidences support that it has been innovatively implemented. Performance of the QA practice shows excellent results and outstanding improvement trends.
In assigning rating to criterion and sub-criterion, only whole number should be used.
35
carried out to ensure consistency of results. A 7-point rating scale is used for AUN-QA assessment. It provides universities and assessors an instrument for scaling their verdicts and to see how far they have progressed in their AUN-QA journey. The 7-point rating scale is described below. Rating
3/30/16
Description
1
Absolutely Inadequate The QA practice to fulfil the criterion is not implemented. There are no plans, documents, evidences or results available. Immediate improvement must be made.
2
Inadequate and Improvement is Necessary The QA practice to fulfil the criterion is still at its planning stage or is inadequate where improvement is necessary. There is little document or evidence available. Performance of the QA practice shows little or poor results.
3
Inadequate but Minor Improvement Will Make It Adequate The QA practice to fulfil the criterion is defined and implemented but minor improvement is needed to fully meet them. Documents are available but no clear evidence to support that they have been fully used. Performance of the QA practice shows inconsistent or some results.
4
Adequate as Expected The QA practice to fulfil the criterion is adequate and evidences support that it has been fully implemented. Performance of the QA practice shows consistent results as expected.
5
Better Than Adequate The QA practice to fulfil the criterion is better than adequate. Evidences support that it has been efficiently implemented. Performance of the QA practice shows good results and positive improvement trend.
6
Example of Best Practices The QA practice to fulfil the criterion is considered to be example of best practices in the field. Evidences support that it has been effectively implemented. Performance of QA practice shows very good results and positive improvement trend.
7
Excellent (Example of World-class or Leading Practices) The QA practice to fulfil the criterion is considered to be excellent or example of world-class practices in the field. Evidences support that it has been innovatively implemented. Performance of the QA practice shows excellent results and outstanding improvement trends.
Scoring Scale
In assigning rating to criterion and sub-criterion, only whole number should be used. The overall verdict of the assessment should be computed based on the arithmetic average of the 11 criteria with only one decimal place. 64 | P a g e
Appendix A Checklist for AUN-QA Assessment at Programme Level 1 1.1 1.2 1.3
2 2.1 2.2 2.3
3 3.1 3.2 3.3
4 4.1 4.2 4.3
Expected Learning Outcomes The expected learning outcomes have been clearly formulated and aligned with the vision and mission of the university [1,2] The expected learning outcomes cover both subject specific and generic (i.e. transferable) learning outcomes [3] The expected learning outcomes clearly reflect the requirements of the stakeholders [4] Overall opinion Programme Specification The information in the programme specification is comprehensive and up-to-date [1, 2] The information in the course specification is comprehensive and up-to-date [1, 2] The programme and course specifications are communicated and made available to the stakeholders [1, 2] Overall opinion Programme Structure and Content The curriculum is designed based on constructive alignment with the expected learning outcomes [1] The contribution made by each course to achieve the expected learning outcomes is clear [2] The curriculum is logically structured, sequenced, integrated and up-to-date [3, 4, 5, 6] Overall opinion Teaching and Learning Approach The educational philosophy is well articulated and communicated to all stakeholders [1] Teaching and learning activities are constructively aligned to the achievement of the expected learning outcomes [2, 3, 4, 5] Teaching and learning activities enhance life-long learning [6] Overall opinion
1
2
3
4
5
6
7
72
36
3/30/16
WRITING UP of SAR
• Keys: –Clarity; –Relevant; –Concise.
(Priyanto, 2012)
Clarity:
Organize information in a simple pattern that readers can easily understand
Relevant:
Use various types of material to maintain readers interested in explaining the facts (graphs, tables, and quotes)
Concise:
Use introduction to set the reader ’s expectation, no redundancy, brief but meaningful; Have a special summary/conclusion ready; Anticipate questions (you’ll be asked and have answer ready). è SAR report ± 50 pages (excluded appendix) (Priyanto, 2012)
37
3/30/16
Status Akreditasi/sertifikasi P rodi (S1) Status P enjaminan Mutu UGM Internal : Audit Mutu Internal siklus 1 tahun Ekternal Nasional : 65 % terakreditasi A ( dari 258 prodi) Regional ( sertifikasi AUN): -‐ 22 Prodi (S1) • Internasional: -‐ AACSB : 8 prodi fakultas Ekonomi B isnis ( 2014) -‐ T. Kimia : I-‐Chem-‐E (1 prodi) -‐ Mipa Kimia : R oyal Chemistry -‐ Sistem informasi Medis (1 prodi) • • • •
Develop Plan Activity/Week P L A N
1
2
3
4
5
6
7
8
9
1 0
1 1
1 2
Deadline
Assigned to
Status
Communicate Intent Organise Team Develop Plan Understand AUN QA criteria and process
D O
Self-assessment Collect data & evidences
Close gaps Write SAR Review SAR C H E C K
A C T
Verify SAR Gather Feedback Improve QA Finalise SAR Communicate SAR Get Ready
PlanChange Management
76
38
3/30/16
GinÖng Prati Dino
Terima Kasih
39
