Benchmarking Website Performance in the Public Sector: A Non Parametric Approach

Please download to get full document.

View again

of 8
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Document Description
In this paper the outcome of a benchmarking study that compares organization websites in the public sector is presented. In particular, 31 websites of Italian public universities are compared considering the website cognitive efficiency as a measure
Document Share
Document Tags
Document Transcript
  Benchmarking Website Performance in the Public Sector: A Non Parametric Approach Corrado lo Storto Dept. of Industrial Engineering, University of Naples Federico II, Naples, Italy Email: Abstract   — In this paper the outcome of a benchmarking study that compares organization websites in the public sector is presented. In particular, 31 websites of Italian public universities are compared considering the website cognitive efficiency as a measure of its overall performance, i.e. usability and accessibility. Data Envelopment Analysis is performed to generate a measurement for the cognitive efficiency, while cross-efficiency is used to alleviate the weak discriminating capability of the basic DEA model, and have a ranking of sample websites. Seven university websites are 100% cognitive efficient and average efficiency is at 61.63%. Results also show that website cognitive efficiency is positively influenced by the user perceived attractiveness and negatively by the time over-consumption during navigation.   Index Terms   —Benchmarking, Data Envelopment Analysis, cross-efficiency, Public Administration, website, efficiency, cognition, human-computer interaction I.   B ACKGROUND  Since late 1970s, both in developed and underdeveloped countries, the public sector is facing a continuous pressure to increase the amount of value for money delivered to citizens and businesses, by lowering its operating costs to become more efficient, assuring greater accountability and improving the quality of services provided to users, being consequently more customer focused and responsive to the needs of stakeholders [1]. Following this general trend, a significant reform of the public sector organizations was started since the early 1990s in Italy, too. A wide range of initiatives that affected different areas (i.e., health services, education, etc.) and levels (local and central government) of the public sector were launched, all of them aimed at improving value for money, ensuring an easy access to more common administrative services, modernizing them, simplifying and making the administrative action more efficient. The emergence of the pervasive ICT paradigm is having an important weight in the reorganization and innovation process occurring in the public sector. Indeed, recent trends in the digital technologies are revolutionizing the way the  public sector organizations, citizens and businesses are interacting together. The WWW based-technologies, supporting an easy and less costly access to integrated databases, huge data storage, and a rapid information exchange between the citizens and the public sector organizations, allowed the establishment of a new reference framework in which the public sector services classified as front office services are supplied to citizens via internet, while services classified as back office services are supplied to other offices through conventional networks. The digital technologies have a significant potential impact on public services and value for money, and help providing citizens with services anytime and anywhere by using portable devices. The wide diffusion of the World Wide Web (WWW) infrastructure in the recent years has made available online a large variety of cost efficient e-government services, and a growing number of people may access to a huge amount of information published on the public administration websites [2]. The WWW offers several  benefits and opportunities to citizens and public sector offices, i.e. to transfer and exchange information in a timely fashion, to publish up to date news, to publish a huge amount of information without incurring high  printing and distribution costs, to increase transparency and accountability of the Public Administration, and finally enhancing trust of people. However, such benefits and opportunities cannot be achieved if the website interfaces that users can access are scarcely usable and accessible [3]. Indeed, according to the Technology Acceptance Model [4], the perceived usefulness and ease of use of a website positively influences the users’ intention to use it. While until the last decade most of the effort in the  public sector organizations was focused on the measurement and improvement of internal efficiency, now the effort has shifted to measure the level of users (citizens and businesses) satisfaction and their appreciation for services provided. With the emergence of the e-governance paradigm, the benchmarking practice in the public sector has received more attention than in the past [1]. As the public sector organizations are facing an increasing pressure to improve service quality on the one side, and, in the same time, to reduce costs on the other side, benchmarking has become one important  practice that may successfully support the public sector organizations in their effort to increase the value for money delivered to citizens, identifying performance gaps, developing and implementing action plans in order to improve performance in terms of cost efficiency and customer satisfaction [5, 6, 7]. Indeed, benchmarking may help the public sector organizations to set goals that 636JOURNAL OF COMPUTERS, VOL. 9, NO. 3, MARCH 2014© 2014 ACADEMY PUBLISHER  doi:10.4304/jcp.9.3.636-643  reflect more their strategic mission, which are more realistic and achievable, thus creating worthwhile challenges, and allowing greater efficiency and accountability, and to better understand and meet the citizens needs in the provision of services through the WWW infrastructure [8, 9, 10]. For this reason, evaluating the performance of an e-government website has become an important concern in the implementation of benchmarking studies, and website usability and accessibility are considered important factors when the  public sector performance has to be assessed relatively to issues such as administrative efficiency, social inclusion, transparency, etc., and measuring the website-user interaction performance in this modified citizen centric framework cannot be disregarded any longer. 1  This paper adopts the integrative framework developed in ref. [14, 15] to implement a benchmarking study in the field of higher education in the public sector. In particular, the cognitive efficiency of the websites of a sample of 31 Italian public universities is measured. This framework is  based on concepts driven from theories of information  processing and cognition, and considers the website cognitive efficiency as a measure of its overall  performance, i.e. usability and accessibility. Cognitive efficiency is then calculated performing Data Envelopment Analysis (DEA). II.   T HE C ONCEPT OF W EBSITE C OGNITIVE E FFICIENCY  The performance of websites has been evaluated in the literature using different approaches. Many methods and approaches to website performance measurement have  been suggested focusing on website functionality, i.e. audits and expert technical reviews, testing involving real users, automatic testing tools, contextual interviews, focus groups, heuristic evaluation, individual interviews, use cases, observations and log activity [16, 17, 18]. A number of approaches privilege users’ needs satisfaction associated to their interaction with the website when they use it, the level of website usability and ease of use, adopting the Technology Acceptance Model as reference framework [4], i.e. the data quality measurement models [19, 20], the WebQual TM  model [21], the service quality models, such as the e-TailQ [22, 23], the SITEQUAL model [24], multivariate models [25] the SERVQUAL model [26, 27], the E-S-QUAL model [28]. Generally, in all these approaches, the website  performance is evaluated adopting subjective assessments that are based on the website user personal judgments. 1  As the International Organization for Standards (ISO) claims, the usability a website is the extent to which it can be used by users to achieve specified goals with effectiveness, efficiency and satisfaction in a given context of use (ISO 9241 standards), or the capability of the website to be understood, learned, used and liked by the user, when used under specified conditions (ISO/IEC CD 9126-1 standards). Website accessibility relates to the extent a website can  be used by all potential users, i.e. by people with all abilities and disabilities (WCAG 1.0 and WCAG 2.0 guidelines) [11]. Both website usability and accessibility influence the users satisfaction when they interact with the interface and their perceived efficiency of this interaction. Behind these two concepts there are two complementary design approaches that, to a certain extent, share common principles [12, 13]. They implicitly assume that website performance fundamentally depends on the cognitive capabilities of the website users when they interact with the website and give meanings to the website [29, 30]. Cognition and information processing capabilities are important issues that determine how users perceive website quality [31]. The perceived ease of navigation, the website information structure comprehension, and the easy search for important information are indeed critical concepts in any website evaluation. However, the traditional website evaluation methods privilege the use of qualitative data, and in general they are unable to take into account the users’ emotions and experiences [32].   Further, these approaches are unable to model the cognitive process of website users that determines their feeling about the website when they experience it. The integrative framework that is used here to assess the website  performance is based on the idea that the user-website interaction forms a unique coupled cognitive system in which the websites, providing the users with patterns of action and an additive short-memory, support them in their work keeping their internal mental information  processing activity less complex [14, 15]. Thus, the external information resources provided by the websites  become a part of the users cognitive system [33]. The websites address how users’ cognition evolves during interaction as the websites dynamically assume new configurations that better fit the users information  processing capability, while information and knowledge are continuously transformed through dynamically adapting mental, external and technological representational states as the physical structures in the external representations (the website) constrain the range of possible cognitive behaviors, allowing some of them  but prohibiting others. When the users interact with the websites, they have to bear a cognitive cost because of the efforts made to search, interpret and process information. If the website users are either unable to locate critical information or to manage the information overload, being more or less disoriented because information already collected is either redundant or ambiguous, they will find difficult to proceed with their search or retrieve supplementary information necessary to accomplish the task. That will make their search activity  psychologically and cognitively costly. In the course of their interaction with the websites, the cognitive state of the website users may thus be characterized in terms of  being more or less ambiguous and uncertain. Interacting with the website may be also a source of gratification and satisfaction to users, both of them associated to the quality of interaction itself and the way this latter develops, and, finally, to the successful goal attainment. Moreover, a reduced amount of time necessary to achieve a task is usually associated to more efficient information  processing and better website design. These perceived  benefits are important to the website users as they reflect a positive outcome from the outlay of scarce resources and the fulfillment of previously unmet needs [14, 15]. Thus, a website should be capable to eliminate or at least reduce the users working memory overload associated to JOURNAL OF COMPUTERS, VOL. 9, NO. 3, MARCH 2014637© 2014 ACADEMY PUBLISHER   the need to mentally retain and integrate different information and knowledge amounts, and, in the same time, to provide the users with a pleasant and satisfying experience. Cognitive efficiency (CEff) may thus be considered an important measure of the amount of cognitive work the users have to perform outside their working memory to carry on a task, given the constraining nature of the website interface: cognitive benefitsCEff cognitive costs =  ∑∑   Table 1 displays the cognitive dimensions – benefits (O) and costs (I) – of the framework used to measure the website cognitive efficiency. The framework considers 3 types of cognitive costs generated by the perceived uncertainty, ambiguity and over-consumption of time related both to the website structure characteristics and navigation of users, and 3 types of cognitive benefits associated to the experience of the users in their interaction with the website as perceived usefulness, satisfaction and attraction for website. TABLE   1 DEA   M ODEL V ARIABLES   Variable Type Associated measure UNC_S I cognitive cost due to perceived uncertainty determined by website structure AMB_S I cognitive cost due to perceived ambiguity determined by website structure TIME_S I cognitive cost due to perceived time over-consumption influenced by website structure UNC_N I cognitive cost due to perceived uncertainty determined by website navigation AMB_N I cognitive cost due to perceived ambiguity determined by website structure TIME_N I cognitive cost due to perceived time over-consumption affected by website structure USE_E O cognitive benefit determined by perceived website usefulness SAT_E O cognitive benefit determined by perceived satisfaction for website usage ATT_E O cognitive benefit determined by perceived attractiveness for website III.   T HE M EASUREMENT OF THE W EBSITE C OGNITIVE E FFICIENCY BY I MPLEMENTING DEA The benchmarking analysis and measurement of websites cognitive efficiency can be implemented by  performing Data Envelopment Analysis (DEA). DEA is a non parametric technique that identifies the efficient units from a given set and combine them to generate an efficient frontier that is used as a benchmark to measure the relative efficiency of the inefficient units. The efficiency measurements depends on a restricted set ( the reference set  ) of units with which the unit under evaluation is compared to [34]. Variables associated to measures of the user perceived cognitive costs and  benefits are adopted as inputs and outputs, without any explicit assumption about the underlying relation between them. The website cognitive efficiency (CEff) is thus calculated as the classical engineering ratio of outputs to inputs. Figure 1 graphically shows how DEA works and measures efficiency. Five websites, WS A , WS B , WS C , WS D , WS E  should be compared. It is assumed that the usage of each website produces only 2 kind of cognitive  benefits and 1 kind of cost. Let us assume that each website generates the same amount of cognitive cost. Under the assumption of convexity, the line segments that respectively connect websites WS A  to WS C  and WS C  to WS E  are called the efficient frontier. The efficient frontier defines the maximum combinations of benefits that can  be produced for the given amount of cost. Indeed, the segment WS A WS C lies beyond both the segment WS A WS B  that can be drawn between websites WS A  and WS B  and the segment WS B WS C drawn between WS B  and WS C . As a consequence, a convex combination of WS A  and WS C  will generate the most benefit for the given cost. Websites WS A  and WS C  are efficient as they are on the efficient frontier, but as WS B  lies under the efficient frontier, it should be considered inefficient and its cognitive inefficiency CEff can be measured by the relative length of segment WS AC WS B  (where WS AC  is a virtual website formed through a combination of website WS A  and website WS C which form the reference set of WS B , and   the ratio OWS B /OWS AC  measures the efficiency). In a similar way, segment WS C WS E is beyond the segments WS C WS D  and WS E WS D , and segment   WS CE WS D  measures the inefficiency of website WS D . The efficient frontier is generated solving a sequence of linear programming problems, one for each unit included in the sample. Given the set of units, for each unit DEA determines the optimal set of input weights and output weights that maximize its efficiency score. Assuming that there are n units, r   inputs and  s  outputs, the relative efficiency measurement of a unit k   is obtained  by solving the following fractional model [35]: 1111 max .. =1,..., ,0 =1,..., =1,...,  siik ir  jjk  j siipir  jjp jij vyuxvy stpnuxvuisjr  ==== ∑∑∑∑ ≥  where  y ip  is the amount of output i  produced by unit  p ,  x  jp  is the amount of input  j  used by unit  p , v i  is the weight given to output i , and u  j  is the weight given to input  j . This model can be converted into n  linear programs which can be more easily solved to calculate the relative efficiency of each unit, as follows: 1111 max .. 1 0 =1,..., =1,..., =1,..., , 0 === = ∑ = ∑ − ≤ ∑ ∑ ≥  siik ir  jjk  j sr iipjjpijij vy stuxvyux pnisjrvu   638JOURNAL OF COMPUTERS, VOL. 9, NO. 3, MARCH 2014© 2014 ACADEMY PUBLISHER   Several scholars have suggested different methods  based on DEA to have units rankings [36, 37]. The cross-efficiency approach is particularly useful to alleviate the weak discrimination capability of the basic DEA model [38]. The calculation of each unit cross-efficiency occurs in two steps [39]. In the first step, the basic DEA evaluation is carried out and optimal weights assigned to inputs and outputs for every unit are determined. In the second step, the cross-efficiencies matrix X is generated. The element ce ij  in i -th row and  j -th column of cross-matrix X   is the ratio ce ij   = eff   j ( u i  , v i ) of outputs and inputs of unit  j , weighted by the optimal weights of unit i . As a consequence, column  j  contains the efficiency scores of unit  j  calculated using optimal weights of unit i , for i =1,…, n . The average score of elements in column  j   provides the cross-efficiency score of unit  j . The cross-efficiency scores calculated for all units are used to rank these latter. All measurements of elements in matrix X are between 0 and 1, while the elements in the diagonal are the basic DEA efficiency scores. 1 1 2 21 1 += w benefit w benefit CEff v cost    Figure 1. An example about how DEA works IV.   T HE B ENCHMARKING S TUDY    A.    Method The sample contains 31 websites of Italian public research and teaching universities, and is largely representative as the most important academic institutions in terms of students enrolled and amount of researchers and professors are included. In general, universities  provide the online users with several services, i.e. online enrollment to single courses and undergraduate or graduate diploma degrees, download of teaching material, the availability of a great amount of information relative to course programs, examinations, records of their academic career, etc. that can be more or less easily searched and retrieved. As the website usability and accessibility evaluation depends on the assessment context, goals and task performed during the interaction of the users with the website, major attention was necessary in the analysis to identify and circumscribe a specific task and a homogenous group of website users-evaluators. The search for information relative to the graduation and professional qualification examinations was considered as a reference task for the benchmarking analysis, while 5 young engineers graduated from the University of Naples Federico II were involved as website user-evaluators of all sample websites. An experimental laboratory setting was implemented to carry on the empirical study. The 5 young engineering graduates were invited to browse the 31 websites and  provide their judgments. A questionnaire purposefully created was used to collect data from users during their interaction with websites and measure their cognitive costs and benefits. Nine groups of statements allowed to have the same number of measurement scales of the framework cognitive constructs. Users were requested to express their agreement or disagreement for the statement content using a 1-9 level measurement grid. As Table 2 shows, the α  Cronbach index higher than 0.7 indicates a good internal fit and consistency of single scales [40]. TABLE   2 A LFA -C RONBACH I  NDEX OF S CALES  variable α  # scale items UNC_S 0.92 3 AMB_S 0.90 9 TIME_S0.87 3 UNC_N 0.85 4 AMB_N0.94 13 TIME_N0.79 5 USE_E 0.91 7 SAT_E 0.89 4 ATT_E 0.83 5 Judgments provided by users for every scale were averaged generating a unique measurement for the underlying construct. These measurements were used to run DEA.  B.    Results Table 3 shows the Pearson correlations between cognitive cost and benefit variables. Cognitive costs associated to website structure and navigation are negatively correlated to the cognitive benefits perceived  by the users when they interact with the websites. The correlation signs support the rationale behind the website cognitive efficiency evaluation framework adopted. The strong measurements and positive signs of correlations  between cognitive costs are also indicating that ambiguity and uncertainty influence each other in the interaction of the users with the websites. Table 4 reports the outcome of DEA efficiency calculation. The CCR DEA efficiency (CCReff) was calculated, adopting a radial measure for efficiency and an input orientation approach [35]. In the input orientation approach it is assumed that cognitive costs should be reduced to make a not efficient website 100% efficient keeping the same amount of perceived cognitive  benefits. Cognitive efficiency measures show that 7 out 31 websites result 100% cognitive efficient. The average efficiency of the sample is at 61.63% with a standard deviation of 27.25%. JOURNAL OF COMPUTERS, VOL. 9, NO. 3, MARCH 2014639© 2014 ACADEMY PUBLISHER   TABLE   3 P EARSON C ORRELATIONS B ETWEEN C OGNITIVE V ARIABLES  1 2 3 45 6 7 891 UNC_S 1.00 2 AMB_S .95 1.00 3 TIME_S .94 .91 1.00 4 UNC_N .63 .65 .71 1.00 5 AMB_N .60 .63 .68 .941.00 6 TIME_N .67 .65 .71 .80.82 1.00 7 USE_E -.27 -.36 -.35 -.43-.47 -.44 1.00 8 SAT_E -.31 -.38 -.34 -.50-.53 -.51 .95 1.009 ATT_E -.37 -.39 -.41 -.59-.61 -.63 .86 .901.00  The university websites considered as inefficient have an efficiency score lower than 100%. According to the framework, a website is inefficient if a virtual website can be generated as a linear combination of some websites in the sample, and this virtual website offers at least the same amount of cognitive benefits to the user charging a lower amount of cognitive costs compared to the real website under examination. The average efficiency of the 24 inefficient websites is 54.43% with a standard deviation of 19.74%, while the scores of inefficient websites range between 12.44%  and 86.42%. Table 4 also shows the reference sets of the inefficient websites. The number of times each 100% efficient website compares in an inefficient website reference set is useful to understand to what extent an efficient website has unique features in terms of cognitive benefits delivered to users and costs borne by them. TABLE   4 DEA   O UTCOME  WS Description CCReffCROSSeff  peers WS10WS18WS19WS22 WS24 WS27WS30WS1 Università Politecnica Marche 25.1819.62+ WS2 Università di Bari 71.8562.99+ WS3 Politecnico di Bari 66.8159.05+ + + + WS4 Università di Bergamo 65.5148.91+ + WS5 Università di Bologna 50.2542.93+ + + + WS6 Università di Brescia 50.7244.12+ + WS7 Università di Cagliari 24.5421.60+ + WS8 Università della Calabria 63.6453.88+ + WS9 Università di Camerino 39.7232.05+ + WS10 LIUC 10092.43+ WS11 Università di Catania 12.448.29+ + WS12 Università di Ferrara 42.7439.23+ + + + WS13 Università di Firenze 40.3535.48+ + WS14 Università di Genova 60.9152.31+ + WS15 Università di Lecce 26.7922.96+ + WS16 Politecnico di Milano 86.4264.76+ + WS17 Università di Napoli Federico II 54.9542.90+ + WS18 Seconda Università di Napoli (SUN) 10094.60+ WS19 Università di Napoli Parthenope 10089.22+ WS20 Università di Padova 69.6459.82+ + + + WS21 Università di Palermo 52.6547.35+ + + WS22 Università di Parma 10094.81+ WS23 Università di Pisa 36.9728.00+ + + WS24 Università di Roma La Sapienza 10092.89+ WS25 Università di Roma Tor Vergata 58.4849.29+ + + WS26 Università di Salerno 14.9710.76+ + WS27 Università del Sannio 10087.61+ WS28 Università di Sassari 75.557.24+ + WS29 Politecnico di Torino 68.1859.56+ + + WS30 Università di Trento 10091.87+ WS31 Università di Udine 51.1836.19+ In the sample, the websites of the Università di Napoli  Parthenope  and the Università del Sannio  have only themselves as a reference website. On the contrary, WS10, WS18, WS22, WS24 and WS30 are frequently included in the reference sets of several inefficient websites. In particular, WS18 and WS22 are respectively in the reference sets of 16 and 15 inefficient websites. This information may be used to make more focused comparisons of websites, privileging only a restricted number of websites and/or cognitive dimensions. The cognitive efficiency rate calculated by the CCR DEA model for an inefficient website does not provide any ranking among all websites, but only its relative inefficiency compared to that of the websites which form its reference set. Thus, the website of the Università di  Napoli Federico II   is only 54.95% efficient in comparison to the Università LIUC   and Seconda Università di Napoli , while the website of the  Politecnico di Milano  is inefficient at 13.58% (or 86.42% efficient) compared to a virtual website that combines together the websites of the Università La Sapienza  of Roma and the Università di Trento  that have generated its reference set. 640JOURNAL OF COMPUTERS, VOL. 9, NO. 3, MARCH 2014© 2014 ACADEMY PUBLISHER 
Similar documents
View more...
Search Related
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks