Retrieval evaluation measures, Management Information Sys

Assignment Help:

Retrieval Evaluation Measures 

Objective retrieval evaluation measures have been used from the beginning to assess the retrieval system performance. Many different parameters can in principle be used to measure retrieval performance. Of these measures, the best known, and the most widely used, are: recall (R) and precision (P), reflecting the proportion of relevant items retrieved in answer to a search request, and the proportion of retrieved items that are relevant, respectively. The main assumption behind the use of measures such as recall and precision is that the average user is interested in retrieving large amounts of relevant materials (producing a high recall performance) while at the same time rejecting a large proportion of the extraneous items (producing high precision). These assumptions may not always be satisfied. Nevertheless, recall-precision measurements have formed the basis for the evaluation of the better known studies of both operational and laboratory-type retrieval systems. 

It may be mentioned that recall-precision measurements have not proved universally acceptable. Objections of a theoretical and practical nature have been raised. The most serious questions relate to the fact that recall, in particular, is apparently incompatible .with the utility theoretic approach to retrieval, which forms the basis of a good deal of existing information retrieval theory. Under the utility theoretical perspective, retrieval effectiveness is measured by determining the utility to the users of the documents retrieved in answer to a user query. The problem is that a document may be relevant to a query while nevertheless proving useless for a variety of reasons. The system utility might be optimised in some cases by bringing a single relevant document to the user's attention, at which point the recall might be very low. Hence, recall and relevance are quite different notions. 

The most important factor in determining how well a system is working is the relevance of the items retrieved from the system in response to a query from the user. Relevancy of an item, however, is not a binary evaluation, but a continuous function between the items being exactly what is being looked for and is being totally unrelated. To discuss relevance, it is necessary to define the context under which the concept is used. From a human judgement stand point, relevancy can be considered: 

  1. Subjective, i.e., depends upon a specific user's judgement. 
  2. Situational, i.e., relates to a user's requirements. 
  3. Cognitive, i,e., depends on human perception and behaviour. 
  4. Temporal, i.e., changes over time. 
  5. Measurable, i.e., observable at points in time, 

The subjective nature of relevance judgements has been documented by Saracevic and was shown in TREC experiments. (Saracevic, 1995). In a dynamic environment each user has his own understanding.of the requirement and the threshold on what is acceptable. Based upon his cognitive model of the information and the problem, the user judges a particular item to be relevant or not. Some users consider the information they already know to he non-relevant to their information need. Also, judgement of relevance can vary over time. Thus, relevance judgement is measurable at a point in time constrained by the particular users and their thresholds on acceptability of information. 

Another method of specifying relevance is from information, system and situational views. Here again, the information view is subjective in nature and pertains to human judgement of the conceptual relatedness between an item and the search. It involves the user's personal 'judgement of the relevancy of the item to the user's information need. When information professionals assist the user, it is assumed that they can reasonably predict whether certain information will satisfy the user's needs. Ingwersen (1992) categorises the informations view into four categories of 'aboutness': 

  1. 'Author Aboutness'- determined by the author's language as matched by the system in natural language retrieval. 
  2. 'Index Aboutness'- determined by the indexer's transformation of the author's natural language into a controlled vocabulary. 'Request Aboutness'- determined by the user's or intermediary's processing of a search statement into a query. 
  3. 'User Aboutness'- determined by the indexer's attempt to represent the document according to presupposition about what the user will want to know. 

In this context, the system view relates to a match between query terms and terms within an item. It can be objectively observed, tested without relying on human judgement. On the other hand, the situation view pertains to relationship between information and the user's information problem situation. It assumes that only users can make valid judgements regarding the suitability of information to solve their information need. Lancaster and Warner refer to information and situation views as relevance and pertinence respectively. Pertinence can be defined as those items that satisfy the user's information need at the time of retrieval. Pertinence depends on each situation. 

It may be mentioned here that evaluation of IR Systems is, essential to understand the source of weakness in existing systems and to improve their effectiveness. The standard measures of Precision, Recall and Relevance have been used for the last 25 years as the major measures of algorithmic effectiveness. 

The major problem associated with evaluation of IR Systems is the subjective nature of the information. There is no deterministic methodology for understanding what is relevant to a user's search. Users have trouble in translating their mental perception of information being sought into the written language of a search statement. When fe. is are needed users are able to provide a specific relevance judgement on an item. But when general information is needed relevancy goes from a classification process to a continuous function. They are not able to easily distinguish relevant items from non-relevant ones. 

The Text Retrieval Evaluation Conferences (TRCSs) provide yearly forums where developers of algorithms can share their techniques with their peers and contribute theories of evaluation, which may lead to the design and development of effective and efficient ER systems in future.  


Related Discussions:- Retrieval evaluation measures

Searching on the internet, SEARCHING ON THE INTERNET   A vast array of ...

SEARCHING ON THE INTERNET   A vast array of databases and services are available through the Internet. It is necessary to design interfaces that help users who search for infor

Explain how the swift fin y-copy works, Question: a) A wide spectrum o...

Question: a) A wide spectrum of mobile/branchless banking models is evolving. These models differ primarily on the question that who will establish the relationship with the e

Research theory, Ask question #MinimumWrite a theory application in which y...

Ask question #MinimumWrite a theory application in which you: Discuss the concepts behind your chosen theory including how the theory has been developed or changed over time.

Compare and contrast the implementation strategies, Problem: (i) ‘Many ...

Problem: (i) ‘Many Information Systems projects in the public sector have failed due to poor change management procedures'. Discuss this statement with examples. (ii) An I

Word Memo report, How to do a Word Memo report for MIS course?

How to do a Word Memo report for MIS course?

Information system, Discuss three (3) major challenges that typically user...

Discuss three (3) major challenges that typically users face in building and/or using information systems AND elaborate the ways to overcome those challenges

Write a long note on international communication, Question 1 What is the C...

Question 1 What is the Communist perspective of Media (or the Communist Media Theory) and what is the Free Press Theory (or Libertarian Perspective)? Question 2 Write a long

Current lists - selection tools of information, Current Lists: Current...

Current Lists: Current books - those that are published during the year - represent the majority of materials usually acquired by most libraries, although it may not always be

Subject directories, Subject Directories   Subject directories, also kn...

Subject Directories   Subject directories, also known as subject guides allow people to browse information by subject. They are hierarchically organised indexes of different su

The ChoicePoint Attack Case Study, It is a case study The ChoicePoint Att...

It is a case study The ChoicePoint Attack ChoicePoint, a Georgia-based corporation, provides risk-management and fraud-prevention data. Traditionally, ChoicePoint provided motor

Write Your Message!

Captcha
Free Assignment Quote

Assured A++ Grade

Get guaranteed satisfaction & time on delivery in every assignment order you paid with us! We ensure premium quality solution document along with free turntin report!

All rights reserved! Copyrights ©2019-2020 ExpertsMind IT Educational Pvt Ltd