1 Introduction
Modern mobile devices have evolved in their screen size, processing power and connectivity possibilities, transforming them into a powerful Internet access device, thus bringing the Web search capacities to the palm of our hands.
Our work focuses on mobile Web search. Web search is a rather different search paradigm when realized from mobile devices. In this situation, the user needs a time- and cost-efficient system for Web search that is effective in finding relevant results and that requires the least user effort, all in a limited-interface device. We believe that the effectiveness of mobile search can be enhanced by taking into account the user's context.
Previous research (Kamvar & Baluja, Reference Kamvar and Baluja2008) showed that, related to the mobile search task, it is desirable to have an advanced interface that helps the user to write a query with fewer keystrokes by means of a word recommender or autocompletion system. Our first experience shows that if these recommendations were also to consider context information (Arias et al., Reference Arias, Cantera, Vegas, de la Fuente, Alonso, Bernardo, Llamas and Zubizarreta2008), the recommended words will be more precise and the search effective. However, it is necessary to accurately model the search context to provide precise word recommendation and help the user effectively. As an example, a recommender system will not propose in the first place the term ‘snowboard’ when the user is at the beach or in summer time.
The ontologies could let us model concepts related to the context of the mobile search query. A complete context definition could provide a solid base for the word recommendation system. This initial context definition is evaluated in this paper by means of an online survey with real users, with the intention of providing better support in the context definition.
The rest of the document is organized as follows: in Section 2, we will explain how context ontology could enhance the mobile Web search experience; Section 3 describes how the mobile context ontology is structured. The evaluation of the concepts in the ontology is described in Section 4, and finally Section 5 has the conclusions and future work.
2 Improving the mobile Web search experience
The improvement of Web search from mobile devices is a widely studied effort with different points of view. Some other proposals that use ontologies for the Web search enhancement can be found, but they were mostly associated with a desktop environment (Pretschner & Gauch, Reference Pretschner and Gauch1999; Gauch et al., Reference Gauch, Chaffee and Pretschner2003; Sieg et al., Reference Sieg, Mobasher and Burke2007). Some other related studies use the context as an improvement in mobile devices, but strongly focus on the semantic discovery and communication (Chen et al., Reference Chen, Finin and Joshi2003; Weißenberg et al., Reference Weißenberg, Voisard and Gartmann2004) or even creating users’ profiles focused on Web services (Khushraj & Lassila, Reference Khushraj and Lassila2005). There are also some ontologies that provide a context definition in the mobile device environment, but these are focused on the specification of servicesFootnote 1 or device characteristicsFootnote 2.
Also we find some improvements in using ontologies for the mobile Web search, but these are related to the reranking of results (Leung et al., Reference Leung, Lee and Lee2010) or even including filtering with a sharing system that allows distributing the results between desktop and mobile devices (Jeon et al., Reference Jeon, Kim and Choi2008).
As a part of the My Mobile Search Project (MYMOSE)Footnote 3, we are creating the Context Ontology for Mobile Devices (COMoDE), which conceptualizes the mobile context elements such as the location, the device and network characteristics, the environmental information and the personal preferences, among others. This context definition will allow to identify whether a word is suitable for recommendation according to the actual search situation.
The word recommendation system obtains the words from a thesaurus; these words will be showed to the user when typing only if they are suitable for the actual context. The ontologies and rules are used to determine whether they are proper to be recommended or not. If the user selects the recommendation, then the search query is enhanced using the user's preferences. As an example, if the user starts typing ‘re’, the recommender will show ‘Restaurant’ because it is an appropriate word in the actual context. If this word is selected, the system will enhance the search query using the context and user's preferences, as the location or restaurant preferences giving ‘Restaurant AND Madrid AND Japanese’ as a final search string (Figure 1).
Word recommendation system
Because an ontology is a conceptualization of knowledge, a very subjective task, there is no ‘one and only’ way to create it. We intend to develop the ontology following the (Noy & Mcguinness, Reference Noy and Mcguinness2000) seven steps, which can be summarized as follows:
1. determining the domain,
2. identifying related existing ontologies,
3. identifying the knowledge domain terms,
4. creation of the hierarchy of classes,
5. recognizing the properties for the previously defined classes,
6. defining the properties features,
7. creating instances for the ontology.
The first steps are a fundamental base to obtaining a better semantic framework, especially the terms’ definition because these will be the classes and properties in the final ontology. Even when these steps should be followed in order, the ontology creation is an iterative and cyclic process in the last creation steps.
In the next section, we will explain the definition and creation of the ontology, which will add the user context support for the word recommendation system in the mobile Web search.
3 Creating the ontology structure
We have developed our ontology following the previously mentioned seven steps; the first three steps’ results will be briefly explained in this section. It is on these initial steps where the ontology domain and fundamental terms are defined and these terms are evaluated in Section 4.
First step: determining the domain
We determinate the domain to conceptualize, understanding the context as the interrelated conditions in which something exists or occurs Footnote 4. Transferring this definition to the domain of the word recommendation for the mobile Web search, now we have that the mobile context is all the conceptualized elements that support a word recommendation system and could enhance the search query in mobile devices.
Second step: identifying existing ontologies
In this step, we search and analyze any standardized and tested ontology structure that could be functional for our purpose. We found ontologies related to the definition of the context associated with mobile devices, or used as a support for the search enhancement; however, these ontologies have a lot of differences with our needs.
Some previous context ontologies were used to enhance the Web search (Pretschner & Gauch, Reference Pretschner and Gauch1999; Gauch et al., Reference Gauch, Chaffee and Pretschner2003; Sieg et al., Reference Sieg, Mobasher and Burke2007); these efforts were focused on the desktop environments and do not include the conceptualizations required for the mobile search characteristics. Other proposals use context ontologies as an improvement for mobile applications (Weißenberg et al., Reference Weißenberg, Voisard and Gartmann2004), or strongly focused on the semantic service discovery and communication (Rios et al., Reference Rios, Costa, Pires, Gonçalves, Filho and van Sinderen2003), user's profiles for Web services (Khushraj & Lassila, Reference Khushraj and Lassila2005) or even the interaction between mobile devices (Chen et al., Reference Chen, Finin and Joshi2003) in a limited space, all of them with a significant difficulty to adapt to the word recommendation and the Web search.
Moreover, we have also considered two ontologies that have a strong definition of the context on mobile devices: the SPICE Project Ontology and the W3C's Delivery Context Ontology.
The SPICE project Mobile Ontology (Villalonga et al., Reference Villalonga, Strohbach, Snoeck, Sutterer, Belaunde, Kovacs, Zhdanova, Goix and Droegehorn2009) is created as a framework for mobile services; the ontology's use was required to obtain a shared and agreed view. Their premise was to develop a standardized way to establish a stretch collaboration between the user and the device and its services.
The fundamental approach on this ontology is related to home networking, mainly in the devices and service interaction. This approach is a complete and well-structured effort, but is not compatible with our needs because this does not take into account other contextual elements such as location, season or weather concepts, or even some time factors (i.e. early morning)—concepts strongly attached to the word recommendation that we want to provide as a support for mobile Web search.
The Delivery Context Ontology is a big effort made by the World Wide Web Consortium, contains classes and properties that include the characteristics of the device, the software used to access the service and the network providing the connection among others.
Even when this ontology perfectly describes the device-related concepts, it does not take into account the concepts related to the environment of the search query and the user preferences, focusing only on the description of the device's characteristics in a very specialized way.
As a resume, these ontologies do not include elements required for our needs and cannot be used as context definition base for the word recommendation to enhance Web search.
Third step: identifying the knowledge domain terms
In this third step, we discovered the terms related to the domain of study that should be considered. We identified six categories of concepts, which could be relevant when thinking in the word recommendation as a support for the mobile search task. These categories are.
A. The device and network characteristics. Terms related to the device processing and displaying capabilities, and the network characteristics where the user is connected from.
B. The user preferences. Concepts that represent the user's preferences for the search task.
C. The physical situation. Related to the location of the user and the device and its characteristics.
D. The temporal conditions and circumstances. In this group, the characteristics of the date, time and its relativeness are conceptualized.
E. The Environmental Conditions and Circumstances. Information such as climatic or daylight elements is conceptualized.
F. The user movement characteristics. This allows to conceptualize the terms relative to the movement of the user/device and its characteristics.
These categories are top level for all the terms we considered to use in our ontology. All the classes and properties are classified within these categories. The second level of concepts and a more detailed description can be viewed onlineFootnote 5.
This structure of concepts will be completed with a series of rules to determinate whether a word is appropriate to be recommended by the system. We tried to identify all the possible factors, which can be significant to consider. Moreover, the ontology must be designed to identify the user's preferences, which also must be relevant in the word recommendation and query enhancement. As an example, the word ‘beach’ can be recommended to the user if his location is near to the shore, is not winter and the weather forecast does not show a rainy day.
To identify whether these proposed categories are adequate as the ontology structure, we considered making an evaluation; this evaluation is explained in the next section of this document.
4 Evaluation of the ontology concept structure
To endorse the foundation structure of our ontology, we have evaluated the six categories of concepts we have proposed.
Ontologies are a conceptualization in a determinate knowledge area or domain, and as a user-defined element could exist in many different ontologies related to the same concepts. Ontology evaluation is a main issue to ensure that the best ontologies provide a stable foundation required by any project; this evaluation can determinate its correctness or validness to certain required characteristics in a specific situation.
The ontology evaluation is a complex and tough task, which needs a strong base and support in a proven methodology. We analyzed some different evaluation proposals such as Staab (Staab et al., Reference Staab, Sure, Gomez-Pérez, Daelemans, Reinberger, Guarino and Noy2004), Guarino (Guarino & Welty, Reference Guarino and Welty2002; Maedche & Staab, Reference Maedche and Staab2002; Guarino & Welty, Reference Guarino and Welty2004) as well the characteristics on the (Gómez-Pérez, Reference Gómez-Pérez1995; Gómez-Pérez, Reference Gómez-Pérez2001) approaches. Brank et al. (Reference Brank, Grobelnik and Mladenic2005) also summarizes the classification of ontology evaluation according to the approach used to evaluate the ontology and the level of the ontology, giving a broader point of view. Both categories are briefly described as follows.
According to the Approach of the evaluation:
• Comparing the ontology with a ‘golden standard’. Is related to the standards or base definitions on which the ontology will be created, that is, the language used to develop an ontology.
• Using the ontology in an application and evaluating the results. Evaluates the performance issues about the ontology by testing on a real-world application and comparing results.
• Comparisons with a source of data about the domain to be covered by the ontology. Can be realized with a comparison with another well-known ontology or a structured set of data, as proposed by Brewster et al. (Reference Brewster, Alani, Dasmahapatra and Wilks2004).
• Evaluation done by humans. Where the experts on some knowledge area evaluate the structure, consistency and relations between the concepts in an ontology.
According to the Level of the evaluation:
• Lexical, vocabulary or data layer. Here the focus is on which concepts, instances, facts, etc. have been included in the ontology, and the vocabulary used to represent or identify these concepts.
• Hierarchy or taxonomy. Although various other relations between concepts may also be defined, the ‘is-a’ relationship is often particularly important and may be the focus of specific evaluation efforts.
• Other semantic relations. The ontology may contain other relations besides ‘is-a’, and these relations may be evaluated separately.
• Context or application level. Evaluation looks at how the results of the application are affected by the use of the ontology.
• Syntactic level. The ontology is usually described in a particular formal language and must match the syntactic requirements of that language.
• Structure, architecture and design. We want the ontology to meet certain pre-defined design principles or criteria; structural concerns involve the organization of the ontology and its suitability for further development.
Not all these evaluation approaches are suitable for all the evaluation levels; there are some better-fit evaluations according to the evaluation requirements. The Brank design (Table 1) summarizes the possible combination between approaches and levels in the evaluation process.
Ontologies evaluation different approaches and levels
There are different approaches to face an ontology evaluation, and even when there are some recommended methods for some situations, there is not a ‘one size fits all’ method of evaluation.
We need to evaluate the base concepts of our ontology, and thus we need a lexical, vocabulary or data layer revision. At this point, as any kind of evaluation can be useful to evaluate the proposed concepts, we decided to conduct a survey that supported human-made evaluation. This decision was made because we wanted to take into account the final user's opinions.
To accomplish this task, we have created a survey that has the intention to evaluate according to the interviewed users, if these concepts could enhance the Web search.
4.1 Survey for the base concept evaluation
The survey is a formal set of questions developed to obtain information from the interviewed; in this case, the survey intends to evaluate whether our base concepts could represent the context for mobile Web search, and simultaneously we obtained a basic user profile related to the mobile Internet usage.
We decided to maintain the survey as small as possible to prevent it from becoming a tedious task for the interviewed. It was divided into eight sections. The first two sections will be devoted to profiling the users, obtaining the different groups of user characteristics and their mobile Internet usage representative data. The remaining six sections evaluate the concepts related to the context definition for mobile users.
For the survey design, we have made 46 questions grouped as follows:
• six items to define the survey participant profile,
• seven items plus three conditional questions about the Internet usage,
• eight items related to the Device and Network Characteristics concepts,
• four items related to concepts of the User Profile,
• five items for the Physical Situation concept evaluation,
• five items in the evaluation of concepts related to the Temporal Conditions and Circumstances,
• four items to evaluate the Environmental Conditions and Circumstance concepts,
• four items associated with the User Movement Characteristic concepts.
At the start of the survey, the users viewed an introduction as small and clear as possible that explains about the Web search and how the context can influence in the search results when in mobile environments. The quiz was designed in Spanish and English languages, and could be selected by the user before starting the quiz.
The quiz was applied on an online basis; basically we invited people from all the world. We spread the invitation over e-mail, blogs, microblogging and social networks. We suppose the users that take the quiz are technology's early adopters because of the way of invitation distribution.
We used LimeSurveyFootnote 6 as our survey application, a free open-source server to manage online surveys. This application was installed at a Computer Science Department server in the University of Valladolid and was available online 24/7 in 3 weeks from 24 June to 15 July on 2009.
User profiling
The first two sections from the survey were intended to identify the basic user characteristics, to create a user's basic profile:
• Age: this item was created to identify age ranges. Less than 25 years, between 25 and 39 years, and 40 years or more.
• Sex: should be selected between male and female.
• Level of studies: allow to identify the level of the user, this should be selected between ‘Middle school’, ‘High school’, ‘College or University’ and ‘Postgraduate’.
• Computer-related student/worker: intended to identify whether the user is a computer science student or worker to obtain the user's professional profile.
The second section of the user modeling has questions related to the user habits associated with the mobile environment and the Internet usage:
• Internet services on the desktop: this item let us identify the commonly used services when the user is in a desktop environment (Web search, e-mail, blogs, etc.).
• Mobile connection usage profile: a set of items that allows to create the user's mobile Internet profile. The user needs to answer a simple question: Does he use the mobile Internet connection? If yes, he is selected, and then he should answer three extra questions related to the device type, mobile Internet connection type and frequency of use.
• Internet services on the mobile: this item serves as a point of comparison between the desktop and mobile environments analyzing the related previous question.
• Type of content: to identify the type of content that the user will search if connected from the mobile devices.
• Two questions about reasons to not be connected from mobile devices and intention of future use.
• Does the user think that a personalized Web search for mobile devices that considers the context would be useful?: an explicit question created to identify the willingness of the user to use personalized Web search services.
Context terms evaluation
Shows to the user the context-related terms for their evaluation. These concepts will be showed in six groups of related terms:
• The Device and Network Characteristics.
• The User Preferences.
• The Physical Situation.
• The Temporal Conditions and Circumstances.
• The Environmental Conditions and Circumstances.
• The User Movement Characteristics.
In each group, the most representative-related concepts are featured; these must be punctuated between 1 and 5 to determinate the users’ point of view about their significance using the Likert Item scale (1 = totally disagree, 2 = mostly disagree, 3 = indifferent, 4 = mostly agree, 5 = totally agree). At the end of each group of concept evaluation, a final optional open question was introduced to receive any other concept that the users consider appropriate.
To evade extending the survey, only some representative concepts were taken; the rest of the non-evaluated concepts have a hierarchical relation with the evaluated ones. The complete survey can be found onlineFootnote 7.
4.2 Survey results
We received a total of 60 responses from all over the world. There are some interesting discovered issues based on the survey results. Processed data from each survey item obtained from the LimeSurvey software can be found in the author's WebsiteFootnote 8.
Participants profiling
The profile obtained from the participants give us as a result that most of them are young adults with university or postgraduate studies, and much more important, 55 of the total have studies or work related to computer science. This last observation, complemented with the fact that they know about this survey through services such as Facebook or Twitter, allowed us to assume that they are early adapters in technology issues.
Another element that defines the profile of the participants was the use of mobile Internet connection; 29 of all 48 participants use the Internet connection from mobile devices (Figure 2).
Mobile Internet usage percentage
Of the participants who use the mobile Internet connection, 21 users, who represent 72%, connect from cellular phones, and 18 users connect from a portable computers (netbook, laptop, etc.), who denote a 62%. This represents that almost a half of the participants use mobile internet connection from portable computers and a third from cellular phones.
We also identify that 40 of the 60 contributors (66,66) consider the Web search a usual task to perform from mobile devices.
After the participant profiling, we analyzed by item and group from Sections 3 to 8 of the survey, where the proposed concepts were evaluated to determinate how much they could be considered appropriate to take it into account to enhance a mobile search.
Every item and group of items were analyzed by their Arithmetic Mean, Standard Deviation and Coefficient of Variation; the latter represents a normalized measure of dispersion and allows to identify a more (or less) agreed point of view in the survey results.
Device and Network Characteristics items
In the third group, the survey items contain the concepts related to the Device and Network Characteristics, which includes eight items, with one optional question. We obtained the results showed in Table 2.
Device and Network Characteristics group of concepts analysis
In the optional question, we received 14 answers; the emphasized concepts appear in more than one of the responses: Processor speed, Input methods, mobile specially designed pages, environment, time and date, historic use, device security issues, cost of use, reliability, signal strength, hyperlink dial, third-party apps, communication technology and locally offered services. Moreover, some of these concepts have already taken into account in subsequent concept groups.
We realize that there was a very standard behavior in the responses, obtaining a similar arithmetic mean and standard deviation for all the items and by the group (Figure 3). In this set of questions, the items 3.3 (Mobile device screen characteristics) and 3.6 (Localization method used by the device) obtained the better results and the worst qualification was for the item 3.5 (Characteristics of device's sensors). In this way, we can identify that the survey's participants consider all the devices and network characteristics as a valuable element for the mobile search enhancement, giving the identification of device's sensors the less important value.
Device and network characteristic group of concepts graph
User Preferences concept items
In the fourth group, the items are related to the User Preference concepts; this group contains four items, with one optional question included. The results are presented in Table 3.
User preferences concepts group of concepts analysis
The optional question received 10 answers; the emphasized concepts appear in more than one of the responses: age, economic activity, location, visited page categorization, time and date, gender, ease of use, previous search statistics and speed of access. Some of these concepts have already been taken into account in later concept groups.
This group shows a similar behavior in the previous group (Figure 4(a)), with a minimal difference within the items, where question 4.3 (Preferred results, obtained from previous search queries and results) obtained better qualification with a minor difference among the other items. It is important to note that this group of questions has the most uniform results and better ratings; the users consider the preferences as an important search enhancement.
(a) User preferences and (b) Physical situation groups of concepts graph
Physical Situation concepts items
In Section 5 of the survey, the group was presented with concepts related to the Physical Situation with five items including the optional open question, thereby obtaining the following results shown in Table 4.
Physical situation concepts group of concepts analysis
The optional question has obtained four answers: speed, medical emergency situation, civil emergency situation and geographical independence.
This set of items has similar conduct than the previous group (Figure 4(b)), the arithmetic mean has very similar values; however, here we can find that question 5.3 (Type of place (airport, beach, downtown, etc.)) has the most differentiated qualification with a total standard deviation of 1.21 and a coefficient of variation of 0.31. The item 5.4 (Type of situation (business, pleasure, etc.)) had the better results. This let us identify that the survey's participants gave a significant relevance to the location when searching, but essentially to the type of situation against the type of place.
Temporal Conditions and Circumstance concept items
Dedicated to the Temporal Conditions and Circumstance concepts, the sixth group was composed of five items including the optional question, and the results are presented in Table 5.
Temporal Conditions and Circumstances concepts group of concepts analysis
The optional question has two answers that include gender, user interests and weather information. Some of these concepts have already been taken into account in other concept groups.
This group shows similar results as the preceding groups (Figure 5(a)), with the question 6.1 (Actual date and time) as the better qualified item, and the items 6.2 (Actual season of the year) and 6.4 (Relative date) as the worst qualified one. This led us to believe the users consider that actual date and time are the most relevant enhancement for the mobile Web search within the temporal conditions variables.
(a) Temporal Conditions, (b) Environmental Conditions and (c) User Movement Characteristics groups of concepts graph
Environmental Conditions and Circumstance concept items
Dedicated to the Environmental Conditions and Circumstance concept, the seventh group was composed of four items that include the optional open question, and the results are described in Table 6.
Environmental Conditions and Circumstances concepts group of concepts analysis
In the open optional question, we received two answers that include: kind of transport and UV index. One of these concepts has already been taken into account in other concept group.
The more relevant data obtained in this group are related to the question 7.3 (Actual sunrise and sunset time), which obtained the worst qualification from the group; the rest had an average behavior (Figure 5(b)). This group gives us the chance to determinate that the weather and its forecast have significant value in the improvement that we are developing for the mobile Web search.
User Movement Characteristic concept items
The eighth group with the User Movement Characteristic concepts was composed of four items, including the optional question, with the following results shown in Table 7.
User Movement Characteristics concepts group of concepts analysis
Optional question received only one answer related to the run distance.
Finally, this last group of items follows the regular behavior showed in the survey (Figure 5(c)), the most relevant issue in this group is that which has the lowest qualifications of all the surveys in the arithmetic mean, but the coefficient of variation shows a regular behavior. In addition, the item 8.1 (Speed of user movement) has the worst value; this means that the participants believe that the speed movement is not as important as the type of movement.
As we can see, the behavior of the data obtained by the survey is very uniform, essentially because there are no large variations between items. Even when the results are very similar for all the items and groups, there are some points to highlight.
As an individual analysis, the items 3.3 (Mobile device screen characteristics), 3.6 (Localization method used by the device) and 4.3 (Preferred results) obtained the overall results with a coefficient of variation of 0.25, whereas the item 3.5 (Characteristics of device's sensors) obtained the overall worst result with 0.36 of coefficient of variation, followed by the item 7.3 (Actual sunrise and sunset time) with a value of 0.34.
In the general analysis, the fourth group of concepts related to the User Preferences obtained the better qualifications (0.26 coefficient of variation), upon the seventh group associated with the Environmental Conditions and Circumstances that obtained the worst (0.31 coefficient of variation); this reflects that the users weighed the Users’ Preferences as the most valuable item in the Web search enhancement and the Environmental Conditions and Circumstances as the less important (Figure 6).
Survey's results graph by groups of concepts
The survey's results allowed us to identify concepts that can be considered most useful as a context when searching over the Web in mobile environments. In general terms, the group of items related to the User Preferences obtained better results, showing that the users have a tendency to give more value to characteristics that consider their preferences over other features.
Some of the better qualified individual items are related to the device's characteristics; this could be interpreted as the users have also identified that mobile devices have special elements, which also must be taken into account.
The items from the environmental conditions obtained the worst results as a group, but essentially because of the item 7.3 (Actual sunrise and sunset time) that obtained the worst of all results; the rest of items in the group has an average behavior. A similar behavior occurs with the item 3.5 (Characteristics of device's sensors (accelerometer, biometric, thermometer, etc.)), which obtained the second worst result, but the rest of the group has an average performance. Thus, these individual elements must have a lower value or even be removed to be considered from the conceptualization of the context.
All the items in the survey opened to us a more clear path to follow in the word recommendation system we are developing. This system will show the user's more suitable words according to their context, and this context will be more precise considering the user's point of view, which has been obtained analyzing the survey's results.
5 Conclusions
Although there are some proposals related to the mobile Web search enhancement, we are presenting a development based on the word recommendation as an assistant for the mobile Web search situations. These recommendations are supported by means of a context ontology, which determinates whether a recommendation is suitable or not in the actual search context condition. To accomplish this, we need an adequate context definition, taking into account what the users believe are the concepts that represent the most valuable aid.
We proposed a user's evaluation for the general concepts from the context ontology that we have created; these concepts were evaluated by users to determine their validity. This evaluation was made through the application of a survey. The evaluation brings us helpful information about the user's considerations and clear ideas about what the users are expecting from the word recommendation as a search enhancement when made from mobile devices.
The main result of the survey is the identification of items that are most valuable according to the users; the acquired knowledge will improve the ontology that supports our word recommendation system.
The group of User Preference items obtained the first place between the participants’ choices. We will improve our ontology within these concepts to take into account more clearly and efficiently the stuff related to the preferences.
In the same way, we have identified the items with the worst user's consideration from the users; these elements will be analyzed to determine whether they are modified or deleted.
Acknowledgments
This work is partially funded by MICINN (grant TIN2009-14009-C02-02).
