АНОТАЦІЯ

| UKRAINE RULE OF LAW PROJECT

QUALITY OF COURT PERFORMANCE:
EXTERNAL EVALUATION

Pilot Program for Court Performance Quality Evaluation:
Citizen Report Card Survey among Court Visitors (Round III)

Analytical Report based on Pilot Survey Results
JUNE 2011
This publication is made possible by the generous support of the American people through the United States Agency for International Development (USAID). The contents are the responsibility of the authors and do not necessarily reflect the views of USAID or the United States Government.
TABLE OF CONTENTS
I. INTRODUCTION
As a part of its strategy promoting judicial standards and enhancing accountability of the judicial
system through court performance evaluation, the USAID Ukraine: Rule of Law Project gets
partner civil society organizations involved in introducing Citizen Report Card (CRC) surveys
among court visitors.
CRC surveys have proved to be an efficient tool to measure the level of citizens’ satisfaction
with the quality of municipal services in many countries worldwide. The Project was the first in
the world who adopted this methodology not only to assess citizens’ satisfaction with court
services but to serve the overall objective – to get feedback from the civil society to improve
court performance, citizen access to justice and to create well-grounded basis for development of
court performance standards in Ukraine. The CRC survey is an extremely valuable tool as it
gives an actual input from court users and involves courts, court staff and civil society
organizations in all stages of the process.
By using CRCs, partner civil society organizations assess perception of court performance by
court users through several measures such as: 1) territorial accessibility of courts; 2) level of
comfort in a courthouse; 3) accessibility of court information; 4) payments affordability; 5)
timeliness in considering cases; 6) court staff performance; and 7) the quality of judge
performance. Based on these measures, civil society organizations have developed a
questionnaire with a set of questions giving a total score ranging from 0 (the lowest score) to 1
(the highest score) for each of 7 measures.
Instead of focusing on the quality of court decisions or correct application of law, CRCs
demonstrate court performance in a wider sense including duration of proceedings, competence
and professionalism of judges and court staff, and, inter alia, attitude towards the parties. With
CRCs, Ukrainian courts could identify weaknesses in servicing and methods of external court
performance evaluation to enhance court accountability before the public and find a tool to
follow up improvements of court service quality over the time. CRC project partners make a
platform for development of court performance standards in Ukraine. These standards can assess
to what extent courts succeed (or fail) in abiding by the fundamental principles such as fairness,
timeliness, and integrity.

With the Project’s support, CSO volunteers interviewed more than 2,300 court visitors at every of
three rounds of the CRC Survey launched in 2008. Up to date, over 7,500 citizens provided their
feedback on court performance to develop recommendations to improve court performance.
Following the recommendations based on the first round of the survey completed in 2008 – 2009,
two courts made court premises more comfortable for visitors, while four courts made court
information more accessible. As a result, visitors of those courts valued court staff performance
more highly during the second round of the CRC survey. For example, 61% of court visitors
founded court staff diligent, knowledgeable, and competent during the first round against 70%
during the second round. Furthermore, three courts have improved timeliness of servicing between
the first and the second rounds. Six of seven courts participating in both rounds improved their
overall performance score: about 57% during the first round vs. 71% during the second one.
Upon the second round of the CRC Survey, 15 courts received more than 200 recommendations
based on citizens input.
This report summarizes the results of the third round of the CRC survey of court visitors’
satisfaction with court performance.
The survey has been carried out by five Ukrainian civil society organizations in October 2010 -
May 2011. The report is intended for CSOs, court presidents, judges, court administrations, and
other court staff members and stakeholders interested in promoting a dialogue between courts and
the civil society to improve access to justice, enhance efficiency, transparency, and accountability
of the judicial system, and to enhance public trust in courts. The report mainly focuses on the
analysis of Index of Court Performance Perception by the Public and its quality measures.
II. ORGANIZATION OF SURVEY
Pilot courts and non-governmental organizations (NGOs) able to interview respondents using the
proposed methodology were selected at the initial stage. Five Ukrainian civil society
organizations were selected on a competitive basis. The courts selected for the second round of
the CRC survey (2009-2010) were those which participated in the first round to assess
performance over the time. Therefore, the third round of the CRC survey involved visitors of 15
courts (Table 1).
Table 1: Pilot courts covered by the survey
Petrovsky District Court, city of Donetsk Chornobay District Court, Cherkasy oblast Yarmolyntsi District Court, Khmelnytsky oblast Krasyliv District Court, Khmelnytsky oblast Novovolynsk District Court, Volyn oblast Stara Vyzhivka District Court, Volyn oblast An additional point to emphasize is that the Project arranged and conducted a training session in CRC surveys of court services for representatives of the selected NGOs, including trainings for interviewers (volunteers) and data input operators. At the initial stage of the CRC survey, the Project established partnerships with representatives of pilot courts, including chief officials of territorial departments of the State Judicial Administration of Ukraine in pilot regions and presidents of courts. Numerous focus group meetings were held to discuss and improve survey instruments. Such meetings brought together both representatives of pilot courts and citizens who directly applied to courts. Such approach allowed us to agree upon a system of court performance measures and their respective indicators, on the one hand, and to probe a standardized questionnaire, on the
other.
Court visitors were interviewed directly at courthouses in face-to-face interviews using the
standardized questionnaire. Approximately 210 visitors were interviewed at each court, 3,088
visitors in total.
Several focus group discussions were held based on the survey results to discuss the main
findings of the survey and to develop practical suggestions on how to improve the quality of
court performance.
The Project has developed CSPro-based data input software application for processing of
questionnaires. A statistical analysis was carried out with SPSS software.
At the final stage of the CRC survey, the survey findings were notified to stakeholders and
measures to implement CRC-based recommendations were initiated including court performance
monitoring.
III. SURVEY METHODOLOGY

Although CRC methodology is based on survey tools which are traditional for social researches,
its objectives and arrangements differ. In fact, it is a form of civil society control over the quality
of different services offered to the public.
The key objectives of such surveys of court services include providing judicial information to
courts and the society in a timely manner; identifying efficient ways to overcome existing
challenges, and monitoring changes in problematic areas of the judicial system. At the same
time, such surveys serve as a source of reliable information and are effective provided the
following conditions are met:
a) the survey methodology takes current international court performance standards and domestic
judicial practice into consideration;
b) there is an efficient and effective communication based on mutual trust between courts and the
public, in particular, NGOs and mass media;
c) all stakeholders have unbiased attitude towards the survey results; and
d) joint efforts are being taken by experts, lawyers, sociologists and the public during the process
of survey organization.
Although identification of principles, methods, and technical tools to assess court performance
quality is a high priority for Ukraine, these tasks have not been reduced to practice posing the
main challenge at the stage of methodology development. As a burden of Soviet legacy, the
judicial system has still been using approaches which assess court performance against the
number of revoked/changed judgments, claims, disciplinary proceedings, and other quantitative
measures. Therefore, the underlying approach is innovative and combines Ukrainian context and
the best world practice.
Quality measures – the key elements of the survey – are used to comprehensively assess the
quality of court performance. The quality measures are court performance quality standards
which embody expectations of the public from court performance in the democratic society.
Each quality measure is explained through the system of court performance indicators which make the standard more specific and allow to measure it in quantitative way. Although there are several approaches to assess court performance in international practice, mechanical transfer of the available tools would hardly be efficient since such approaches fail to take into account the context of the Ukrainian judicial system. Therefore, the Project offered the system of 7 quality measures which, in turn, are assessed through 25 indicators which have been used to make questions for the questionnaire (Table 2). Table 2: Main Measures for Assessment of Court Performance Quality
• Easiness to find a courthouse for the first time • Easiness to get a courthouse using public transportation • Comfortableness while being in court • Enough of space for waiting, preparation of documents, review of case files, and preparation for court hearings • Cleanness and neatness of court premises • Accessibility of court premises and usability of court services • Convenience of secretariat business hours • Convenience of information stands location • Availability of necessary information in court premises about location of rooms, courtrooms, and other premises; rules of entry to and stay in courthouse; pending cases; general information about the court; template documents; procedure for payment of court fees and stamp duties; bank details and amounts of payments • Availability of necessary information on court web-site • Access to lawyer services where necessary • Consideration of wishes when assigning time and date for • Timeliness of receiving summons and notifications of hearings • Justification of delays/rescheduling of court hearings • Timeliness of getting acquainted with case files • Court staff demonstration of respect, politeness, willingness to help, equal treatment to any visitor irrespective of his/her social status or any other factor, diligence, discipline, respect for promises, patience, tolerance, professionalism and expertise. • Perception of judge attitude to court case participants as correct • Perception of judge readiness to review case • Perception of judge’ compliance with case review procedures • Opportunity to comment the allegation of opposite side • Opportunity to justify own position during case hearing • Perception of judicial independence and impartiality • Timeliness of receiving court decision • Intelligibility and understandability of the text of court decision • Perception of reasoning of court decision
An efficient way to analyze the system of quality measures in a comprehensive manner is to use
approaches allowing to synthesize a total score. For the purposes of this survey, Index of Court
Performance Perception by the Public (the Index) was used as such total score.
Index of Court Performance Perception by the Public is a total score derived as a mean of values
(scores) for each of seven quality measures (individual indices). The Index is calculated based on
the system of statistical weights, i.e. weight coefficients which characterize the input of each
measure to the total score. The system of statistical weights is prioritized by importance for court
visitors. The total score is defined as the sum of weight coefficients by all quality measures.
Standardized values of weight coefficients for each quality measure based on court visitors’
assessments are presented on Figure 1 below.
Completeness, accessibility and clarity of Convenience and comfortableness of court Figure 1: Statistical weights of quality measures used in
Index of Court Performance Perception by the Public
This system of statistical weights shows that the input of elements to the total score of the Index varies. According to court visitors, timeliness in considering cases (index score: 0.18) is the most important court performance indicator, while territorial accessibility of court (index score: 0.06) appears to be the least important priority for improvement of court performance. Comparison of index elements with those of an ideal court is an important methodological issue for analysis of the Index of Court Performance Perception by the Public. In other words, if every quality measure got its highest score, the total index score would be 1.00 and such court would be considered an ideal one in terms of performance. However, it is almost impossible to achieve the ideal quality in practice. Therefore, the analysis of court performance should be measured against the actual maximum scores of quality measures for all participating courts and the court getting the highest (maximum) score should be considered as an ideal court. The comprehensive study of court performance should be hierarchical and cover analyses such as: − Index of Court Performance Perception by the Public Assessment of quality measures as Index elements Court performance indicators within certain quality measures Such approach offers a number of analytical opportunities: ƒ to get an overview of performance of every participating court and define its rating to define measures to be taken to improve court performance to achieve the maximum quality score within the possible index score range; ƒ to identify the most crucial elements of quality measures which affect the total score of to quantitatively assess input of each quality measure to the total deviation of the actual index score vs. the possible maximum index score; ƒ to identify priority areas for improvement of court performance based on the actual scores and their weights within the total index score; ƒ to develop steps to improve the scores of quality measures based on their respective indicators.

IV. OVERALL ASSESSMENTS OF COURT PERFORMANCE QUALITY
For the purposes of comprehensive assessment of court performance quality, two quality
assessment criteria have been used to measure court performance:
1) Integral Court Performance Perception Index score;
2) Answers to the direct question of the questionnaire: How do you assess the overall court
performance?
Based on the methodology described above, each court got its Court Performance Perception
Index score. The analysis of current scores against those of past years demonstrated an overall
positive trend (Table 3).
Table 3: Pilot courts by Court Performance Perception Index score
As shown above, 9 of 15 pilot courts improved their quality index score, while 4 courts got a somewhat lower score over the last year. Notably, such deviations are minor for all courts over two rounds, and such results should be considered as a trend and any quantitative characteristics should therefore be avoided. The analysis of respondents’ answers to the direct question supports the conclusion on the overall positive trend (Table 4). Table 4: Pilot courts by answers to the question:
How do you assess the overall court performance?
Furthermore, there are positive changes in the levels of two scores which became almost equal in 2011 (vs. 2010; Figure 1) as the court performance quality score (defined as an answer to the direct question: How do you assess the overall court quality?) became somewhat higher over the two rounds of the CRC survey. How do you assess the overall court performance? Figure 1: Pilot courts by court performance quality, 2010
Such positive trends in the quality assessment may demonstrate that court visitors are getting on to the fact that court performance quality is a comprehensive issue which cannot be reduced to judge/claimant (or other party) relations only. The itemized analysis of the General Index shows that courts are differentiated by individual quality measures to the certain extent as shown by the range of scores of all courts (Figure 2). At the same time, court visitors’ requirements to the level of quality or any other quality measure vary depending on the type of court and group of respondents. Figure 2: Quality measures by range of variation (survey round III, 2010-2011)
Similar to the last year’s results, court visitors consider payments affordability and convenience and comfortableness of stay at courthouse as the most problematic issues. Figure 2 however shows that service provision timeliness and judge performance are the most crucial court performance quality criteria which should be improved first. Table 5 summarizes the levels of Index of Court Performance Perception elements by pilot courts. Table 5: Index of Court Performance Perception by the Public by quality measures and weight coefficients
(Survey Round III, 2010-2011)
V. SUMMARY

In summary, performance of the judicial system in Ukraine requires comprehensive assessment
based not only on statistical information about internal court performance but on external
assessments such as public surveys among those who directly deals with courts. Such surveys are
very important since even when a court operates efficiently (based on certain measures) and
provides excellent services (in terms of compliance with standards) the quality of servicing may
remain poor as long as a court visitor, as an external evaluator, is unsatisfied with such service and
such service does not meet his/her specific needs (expectations).
The methodology offered by the Project is adapted in general to assessment of court performance
quality at the local level based on the needs of specific pilot courts. The Index of Court Performance
Perception by the Public is a powerful tool to assess court performance against different quality
measures.
This methodology serves a very useful purpose – it allows using the survey findings for the
judicial system administration and building a platform for courts to take public opinion into
consideration in a more complete and adequate way as this will make a court more open to people
even when financing is insufficient, the judiciary branch lacks clear administration, and laws have
many discrepancies. This methodology may be applied by both civil society organizations and
courts themselves when court administrations seek to improve court performance and have
necessary resources, both human and financial.

Source: http://www.fair.org.ua/content/library_doc/CRC_summary_report_Round_3_ENG.pdf