1 Introduction
Privacy should not be seen as a problem associated only to new technologies (Yao et al., Reference Yao, Rice and Wallis2007). Indeed, privacy has been a concern long before the emergence of information technologies and the explosive growth of the Internet. There are studies that suggest that privacy is probably as old as the human race itself (Schermer, Reference Schermer2007). Even in primitive societies individuals have always had a desire for privacy (Westin, Reference Westin1984). This desire for privacy is usually related to the tendency toward territoriality that most animals have. Moreover, the claim of a right for privacy is often related to the instinct of defending against intrusion.
The modern conception of privacy started more than a hundred years ago, with the seminal work of Warren and Brandeis (Reference Warren and Brandeis1890): The right of privacy. These two lawyers defined privacy as ‘the right to be let alone’. They were pioneers in considering the implications of technology in privacy. Specifically, they were very concerned about the implications of instantaneous photographs and portraits in injuring the feelings of the people in those photographs and portraits. Privacy was later recognized as a fundamental human right by the United Nations Declaration of Human Rights, the International Covenant on Civil and Political Rights, the Charter of Fundamental Rights of the European Union, and many other international treaties (Acquisti et al., Reference Acquisti, Gritzalis, Lambrinoudakis and di Vimercati2008).
In the second part of the 20th century, Alan Westin defined privacy as ‘the claim of individuals, groups, or institutions to determine for themselves when, how, and to what extent information about them is communicated’ (Westin, Reference Westin1967). This is what is currently known as the informational self-determination right (Rannenberg et al., Reference Rannenberg, Royer and Deuker2009). The concept of informational self-determination changed the right to privacy from the right to be let alone to its current incarnation as a means to limit the abuse of personal data (Schermer, Reference Schermer2007). Informational self-determination represents today's European understanding and regulation of privacy in the context of information and communication technology (EU Directives 95/46/EC, 45/2001/EC, and 2002/58/EC).
Despite all these regulations, as the Internet has no governing or regulating body, privacy breaches are still possible. Nowadays, in the era of global connectivity (everything is inter-connected anytime and everywhere) with almost 2 billion world-wide users with connection to the Internet as of 2010Footnote 1, privacy is of great concern. In the real world, everyone decides (at least implicitly) what to tell other people about themselves. In the digital world, users have more or less lost effective control over their personal data. Users are therefore exposed to constant personal data collection and processing without even being aware of it (Fischer-Hübner & Hedbom, Reference Fischer-Hübner and Hedbom2008). Garfinkel (Reference Garfinkel2001) suggests that nowadays users have only one option to preserve their privacy: becoming hermits and not using online social networks, e-commerce sites, etc. Considering the increasing power and sophistication of computer applications that offer many advantages to individuals, becoming a hermit may not really be an option. However, all of these advantages come at a significant loss of privacy (Borking et al., Reference Borking, Van Eck, Siepel and Bedrijf1999). Recent studies show that 90% of users are concerned or very concerned about privacy (Taylor, Reference Taylor2003). Moreover, almost 95% of web users admitted they have declined to provide personal information to web sites at one time or another when asked (Hoffman et al., Reference Hoffman, Novak and Peralta1999).
Autonomous agents are likely used for personal purposes, so that they usually encapsulate personal information of real people. Consider the case in which agents act on behalf of their principalsFootnote 2. In this case, an autonomous agent usually encapsulates personal information describing its principal (Fasli, Reference Fasli2007b), such as preferences, names, and other information. Moreover, agents carry out interactions on behalf of their principals so that they exchange personal information. Agents act on behalf of their principals in agent-mediated e-commerce (Sierra, Reference Sierra2004), as personal assistants (Mitchell et al., Reference Mitchell, Caruana, Freitag, McDermott and Zabowski1994), in virtual worlds like Second LifeFootnote 3 (Weitnauer et al., Reference Weitnauer, Thomas, Rabe and Kopp2008), as recommenders (Montaner et al., Reference Montaner, López and De La Rosa2003), in agent-mediated knowledge management (Elst et al., Reference van Elst, Abecker and Dignum2004), in agent-based semantic Web services (Gibbins et al., Reference Gibbins, Harris and Shadbolt2004), in distributed problem solving (Wallace & Freuder, Reference Wallace and Freuder2005), and many other current and future applications. Therefore, they play a crucial role in safeguarding and preserving their principals’ privacy. Piolle et al. (Reference Piolle, Demazeau and Caelen2007) claim that a great number of researchers in the agent community acknowledge the importance of privacy and believe that more efforts should be made to improve privacy in multi-agent systems.
Surveys such as the one presented in this article are crucial to promote and encourage research and advances in the field of privacy and multi-agent systems. This article introduces the problem of preserving privacy in computer applications and its relation to autonomous agents and multi-agent systems. Therefore, it can be used as a reference for researchers interested in both privacy and multi-agent systems. This article surveys studies that enhance privacy in multi-agent systems as well as studies that enhance privacy through the use of multi-agent systems. It is intended to provide balanced but critical presentations of these studies. These studies are classified according to the type of privacy threats they avoid. Finally, we identify some open challenges in the area of privacy and multi-agent systems.
The remainder of the paper is organized as follows. Section 2 overviews information privacy and its relation to multi-agent systems. Section 3 presents a survey of privacy-preserving mechanisms developed against information collection. Section 4 surveys studies that deal with the avoidance of information processing. Section 5 surveys studies that focus on limiting information dissemination. Section 6 discusses open challenges regarding privacy and multi-agent systems. Finally, section 7 presents some concluding remarks.
2 Privacy and agents
There are many studies that treat privacy as a unitary concept with a uniform value, which is unvarying across different situations. However, privacy violations usually involve several types of harmful or problematic activities (Solove, Reference Solove2002). There are some taxonomies and categorizations that aim to classify these activities (Bostwick, Reference Bostwick1976; Kang, Reference Kang1998; Solove, Reference Solove2006; Spiekermann & Cranor, Reference Spiekermann and Cranor2009). From these taxonomies, Solove (Reference Solove2006) and Spiekermann and Cranor (Reference Spiekermann and Cranor2009) are commonly accepted as covering most of the privacy aspects. Although we find the taxonomy proposed by Solove (Reference Solove2006) and the one proposed by Spiekermann and Cranor (Reference Spiekermann and Cranor2009) equally correct and suitable, we use the taxonomy proposed by Solove (Reference Solove2006) because we think it helps the reader to figure out which entity holds the sensitive information at a given moment and which actions the subject of this information can do (or could have done) to protect its privacy.
According to Solove (Reference Solove2006), privacy can be threatened by three main information-related activities: information collection, processing, and dissemination. Information collection refers to the process of gathering and storing data about an individual. Information processing refers to the use or transformation of data that has been already collected. Information dissemination refers to the transfer of collected (and possibly processed) data to other third parties (or making it public knowledge).
Figure 1 depicts a visual scheme that details when information-related activities can be performed in the process of information exchanges among agents. Information collection occurs when agent A communicates personal information about its principal to agent B. In this case, agent B is the one that collects the information. Moreover, although not depicted in the figure for the sake of clarity, a malicious agent could overhear the communications between agent A and agent B and collect information about A. Once agent B has collected information about agent A, it can then process this information. Finally, agent B can disseminate the information it has about agent A (processed or not) to agent C.
Information-related activities that can threaten privacy
The information-related activities described above can represent a chance to breach the privacy of an agent's principal. Examples of possible privacy breaches that can emerge due to these activities are, but not limited to:
• Secondary use refers to the use of collected information for purposes different from the purposes for which the information was initially collected and without the data subject's consent for doing so (Solove, Reference Solove2006). There are potentially infinite types of secondary uses. In the following, we describe some of these possible secondary uses:
– Profiling. Hildebrandt and Gutwirth (Reference Hildebrandt and Gutwirth2008) define profiling as ‘the process ‘of “discovering” patterns in data that can be used to identify or represent a human or non-human subject (individual or group) and/or the application of profiles (sets of correlated data) to individuate and represent an individual subject or to identify a subject as a member of a group (which can be an existing community or a discovered category) and/or the application of profiles to individuate and represent principals or groups’. According to this definition, profiling can be achieved through information collection and processing. One of the most common types of profiling is called buyer profiling in e-commerce environments, in which vendors obtain detailed profiles of their customers and tailor their offers regarding customer's tastes.
– Price discrimination. Vendors could charge customers different prices for the same good according to the customers’ profiles (Odlyzko, Reference Odlyzko2003), that is, if a vendor knows that some good is of great interest to one customer, the vendor could charge this customer more money for this good than other customers for the same good. For instance, in 2000, Amazon started to charge customers different prices for the same DVD titles (Spiekermann, Reference Spiekermann2006). When the story became public, Amazon claimed that this was part of a simple price test and discontinued this practice.
– Poor judgment. This is when principals are judged and subsequently treated according to decisions made automatically based on incorrect or partial personal data (Smith & Milberg, Reference Smith and Milberg1996). For instance, companies usually divide their potential customers into similar groups based on customers’ characteristics (known as customer segmentation). This practice can lead to exclusion of people from services based on potentially distorted judgments (Spiekermann & Cranor, Reference Spiekermann and Cranor2009).
• Identity theft is ‘fraud or another unlawful activity where the identity of an existing person is used as a target or tool without that person's consent’ (Koops & Leenes, Reference Koops and Leenes2006). For instance, Bilge et al. (Reference Bilge, Strufe, Balzarotti and Kirda2009) present how to clone an existing account in an online social network and to establish a friendship connection with the victim in order to obtain information about her/him.
• Spy agents. An agent could transfer information about its principal to other third parties without its principal's consent and without its principal being aware of the transfer. For instance, an agent provider that hires or sells agents to principals may design and develop these agents so that they collect information on the principals and their activities (Bygrave, Reference Bygrave2001).
• Unauthorized access. Sensitive information about principals is transferred on-line even across the Internet and is stored in local and remote machines. Without appropriate protection mechanisms, a potential attacker could easily obtain information about principals without their consent. For instance, an attacker can be listening to transferred information over the network (files, messages, e-mails, etc.) and simply gather the information flowing in the network (Stallings, Reference Stallings2010). This is usually solved by encrypting the information exchanged over a network.
• Traffic analysis. Although information exchanged over a network is encrypted, a potential attacker could also gather information about who is communicating with whom. This is because there is information such as the IP address and other where about information of both sender and receiver that is available even if the content of the transferred network packet is encrypted. Thus, this potential attacker could also know how often two individuals communicate to each other and even infer that two individuals are closely related to each other (Korba et al., Reference Korba, Song and Yee2002).
• Unauthorized dissemination or exposure refers to the transfer of previously collected and possibly processed information to other third parties, which are different from the one that collected (processed) the information, without the consent of the subject of this information. For instance, an agent A collects (and possibly processes) the information that it receives about another agent B. Agent A can transfer information about agent B to another agent C for whatever reason, for example, to receive a monetary compensation. Thus, agent C can perform any of the aforementioned privacy breaches. Moreover, agent C could even make the information about agent A public knowledge, for example, if agent C publishes the information about agent A, for example, in an (online) journal/blog.
All of these privacy breaches may injure the feelings of the principal involved, as pointed out more than a 100 years ago by Warren and Brandeis (Reference Warren and Brandeis1890). Moreover, these privacy breaches can cause other consequences to the principal involved, such as money loss. These privacy breaches could even cause a principal to be summoned by a court, for example, an attacker can steal the identity of a principal and impersonate her/him to carry out unlawful behaviors.
Agents play a crucial role in safeguarding and preserving their principals’ privacy (Such, Reference Such2011). They usually have a detailed profile of their principals’ names, preferences, tastes, location (permanent address, geo-location at a given time), social characteristics (affiliation to groups, friends), roles in organizations and institutions, transactions performed, credit card numbers, and other personal information. To our knowledge, privacy is seldom considered in the multi-agent systems research field. This leads to agent-based applications that invade principals’ privacy, causing concerns about their use and the privacy breaches explained above.
It is crucial for multi-agent systems to consider privacy in order to be of wide use. This can potentially promote principals’ trust in agent-based technologies. This trust is needed for principals to be willing to engage with and delegate tasks to agents (Fasli, Reference Fasli2007a). To this aim, studies that enhance privacy in multi-agent systems technologies are needed. Moreover, agent designers and developers also need to be mindful of possible privacy implications when developing agent-based applications (Chopra & White, Reference Chopra and White2007). This means that agent designers and developers should choose to apply multi-agent systems technologies that preserve privacy, instead of multi-agent systems technologies that are unconcerned about privacy.
Despite having the potential to compromise their principals’ privacy, multi-agent systems can also be used to preserve it (Solanas & Martìınez-ballesté, Reference Solanas and Martìınez-ballesté2009). Multi-agent systems can offer themselves opportunities to enhance privacy beyond what other disciplines in information sciences can do due to their intrinsic features such as intelligence, pro-activeness, autonomy, and the like. According to (Westin, Reference Westin1967), privacy can also be seen as a ‘personal adjustment process’ in which individuals balance ‘the desire for privacy with the desire for disclosure and communication’.
Humans have different general attitudes toward privacy that influence this adjustment process (Westin, Reference Westin1967; Ackerman et al., Reference Ackerman, Cranor and Reagle1999; Olson et al., Reference Olson, Grudin and Horvitz2005): privacy fundamentalists are extremely concerned about privacy and reluctant to disclose personal information; privacy pragmatists are concerned about privacy but less than fundamentalists and they are willing to disclose personal information when some benefit is expected; and finally, privacy unconcerned do not consider privacy loss when disclosing personal information. This view of privacy requires a dynamic management of privacy instead of a static one (Palen & Dourish, Reference Palen and Dourish2003). Multi-agent systems can help to support this dynamism, as we will see during this survey.
3 Protection against information collection
As described above, information collection can play a key role in breaching privacy, that is, collected data about a principal can be used to breach her/his privacy. In this section, we describe works in the agent research field that prevent undesired collection of sensitive information. According to Spiekermann and Cranor (Reference Spiekermann and Cranor2009), information collection involves data transfer and data storage. For the case of agents, this means that information collection involves one agent sending sensitive information to another agent, and that both agents are able to store this sensitive information. Therefore, it is crucial for agents to first decide which information to transfer to which other agent by means of a decision-making mechanism (as described in Section 3.1), and then transfer and store it securely using traditional security mechanisms, such as those that provide confidentiality (as described in Section 3.2).
Another approach for avoiding undesired information collection is the use of third parties. In this case, agents does not send the information directly to the intended destination agents, instead agents provide sensitive information to third parties. These third parties process the information and return the obtained outcomes to the intended destination agents. We describe studies that follow this approach in Section 3.3.
Figure 2 depicts a conceptual map for all of the studied approaches that provide support for protecting against information collection.
Information Collection Conceptual Map
3.1 Disclosure decision making
The first important approach to prevent information collection is to decide exactly which information to disclose to other agents. Agents should be able to decide which information to disclose according to their principals’ preferences about privacy. This is crucial to prevent undesired information collection. Thus, agents need to incorporate disclosure decision-making mechanisms allowing them to decide whether disclosing personal information to other agents is acceptable or not.
3.1.1 Based on policies
One approach for disclosure decision making is based on policies. In this approach, agents usually specify their policies for both disclosing information and the information they want to collect from others. Then, if an agent's policy for disclosing information matches another agent's policy for collecting information from others, the former agent sends the information to the latter.
Tentori et al. (Reference Tentori, Favela and Rodriguez2006) presents a privacy-aware agent-based framework that allows agent developers to indicate two privacy-related policies (following an XML schema) per agent: one specifying the privacy policy for information that the agent communicates to others and the other specifying the privacy policy for the information that it requires. There is an agent broker that checks that both policies are compatible. Then, the agent broker monitors and ensures that the information that the two agents exchange complies with the policies. Although it allows the real compliance of privacy policies to be checked, the agent broker becomes a clear performance bottleneck and a single point of failure (SPOF). Moreover, the agent broker itself can be a source of privacy concerns because it knows all the information that two agents communicate to each other.
Lee and Stamp (Reference Lee and Stamp2008) present an approach based on P3PFootnote 4. The privacy-enhancing agent (PEA) is in charge of automatically retrieving P3P policies of service providers and evaluates whether or not these policies are compliant with its principal's policy. When a principal attempts to access a website, PEA automatically retrieves the website P3P policy and compares it to its principal's preferences. If PEA detects potential privacy violations (i.e. the principal's preferences and the website's P3P policy do not match) or is unable to read the policy of the website, it notifies its principal so that the principal can decide to desist in accessing the website. This approach does not consider that a website may not comply with its announced policy, and, thus, principals’ privacy breaches are still possible.
Crépin et al. (Reference Crépin, Demazeau, Boissier and Jacquenet2009) present an ontology described using Web Ontology LanguageFootnote 5. Agents can define their policies using this ontology in terms of the hippocratic MAS (Crépin et al., Reference Crépin, Vercouter, Boissier, Demazeau and Jacquenet2008) concepts. They differentiate between data provider and data consumer agents. Both of them define their privacy policies according to this ontology. They also propose a protocol by which data consumers request sensitive data from data providers. Data consumers include their policies in the request. If the policy matches the data provider's preferences, the data provider sends the consumer the requested sensitive data. If not, the data provider proposes some modifications to the policy in order to reach an agreement. If the data consumer accepts these adaptations, the data provider sends the requested sensitive data to the data consumer, otherwise the consumer cancels the transaction. Again, they do not consider that data consumers may not comply with the policy they committed to.
Udupi and Singh (Reference Udupi and Singh2010) present InterPol, a policy language and a framework for interaction in multi-agent referral networks. Policies are logic rules that can be implemented in Prolog. Policies can dynamically adapt to changes in the relationships with other agents. Policies are dynamic in the sense that new predicates can be added to the agent's Knowledge Base. InterPol provides two privacy mechanisms: (i) marking a rule of fact with its visibility (public or private); and (ii) using two privacy-related predicates, servicePrivacyNeed and agentPrivacyTrust, that dynamically manage the privacy decision making. This decision making is based on the privacy that an agent needs for a service and the trust it has in other agents when dealing with privacy issues. However, they do not provide any mechanism for deciding how and when both servicePrivacyNeed and agentPrivacyTrust should be updated.
All of the works presented in this section check that the data exchanged complies with the policies. However, none of them checks that once the data is collected it is treated as stated in the policies. Thus, an agent can disclose personal information to another agent expecting that this second agent will comply with its policy. However, this second agent may not comply with the policy, incurring in possible breaches of privacy.
3.1.2 Based on privacy-utility tradeoffs
There are a great number of people that are willing to trade part of their privacy in exchange for some benefit (64% of US citizens according to Taylor (Reference Taylor2003)). They are known as privacy pragmatists, as mentioned above. There are many studies that have focused on providing models of the so-called privacy-utility tradeoff (Lebanon et al., Reference Lebanon, Scannapieco, Fouad and Bertino2006; Krause & Horvitz, Reference Krause and Horvitz2008). The decision in this case is whether or not a particular privacy-utility tradeoff is acceptable for disclosing information, and then allowing the destination party to collect this information.
The privacy-utility tradeoff is usually modeled as follows. Given a set of personal data attributes A, the utility function of disclosing these attributes U(A), and the privacy cost function of disclosing these attributes C(A), the privacy-utility tradeoff is modeled as 
$$\[--><$>A{\rm^{\ast}}\, = \,\arg \,{{\max }_A}\,U(A){\rm{ - }}C(A)<$><!--$$
. An example of utility function is the one used by Krause and Horvitz (Reference Krause and Horvitz2008) that measures the reduction of time for performing an online search if some personal data attributes such as the geographical location are given. The privacy cost is usually defined taking into account the sensitivity of the information to be disclosed.
Yassine and Shirmohammadi (Reference Yassine and Shirmohammadi2009) present an agent-based architecture that negotiates a reward to be paid to agents’ principals by the service providers in return for their disclosures. The data categorization agent is in charge of classifying principals’ information into different categories (Yassine et al., Reference Yassine, Shirehjini, Shirmohammadi and Tran2010). The data categorization agent is able to calculate the privacy cost of the information about its principal considering the categories that this information falls into and the sensitivity for these categories. Principals define the sensitivity for each category and each service provider. The privacy cost is used to calculate an expected reward for disclosing the information. The payoff negotiator agent negotiates a reward for the information disclosed with the service provider, discarding any deal that provides its principal less than her/his expected reward. In this approach, principals must define the expected privacy cost for each category and for each service provider. This can be a burden for principals when considering a large number of service providers.
Another different approach is the one presented by van Otterloo (Reference van Otterloo2005). The author does not focus on information directly disclosed to another party but on the information that can be collected by observing the strategies an agent follows in a game. The author defines minimal information games as games in which the agent tries to maximize its utility while minimizing the privacy loss. Privacy loss is calculated as the uncertainty (Shannon (Reference Shannon1948) entropy) of the strategy that the agent will use. Thus, if an agent uses strategies with high uncertainty, other agents cannot predict their behavior. The author also defines most normal games as games in which the agent tries to maximize its utility while deviating the minimum from the normal strategy that other agents will play. In this sense, the agent tries to hide the preferences that differ from the normal behavior of the rest of the agents. The deviation from the normal strategy is calculated as the relative entropy between the agent strategy and the normal strategy. The author of this work does not consider that different actions may have different privacy sensitivity.
The research based on the privacy-utility tradeoff does not consider that there are also other reasons that make people decide whether or not to disclose information about them rather than an increase in utility or a decrease in privacy loss. There are many cases where the direct benefit of disclosing information is not known in advance. The decisions on whether or not to disclose information are based on other reasons in these situations. For instance, the well-known psychological phenomenon called disclosure reciprocity (Green et al., Reference Green, Derlega and Mathews2006) states that one person's disclosure encourages the disclosure of the other person in the interaction, which in turn, encourages more disclosures from the first person.
3.1.3 Based on social relationships
In this section, we describe studies that consider concepts of social relationships when disclosing personal information. In these studies, agents disclose personal information according to the trust and intimacy they have in other agents. In this way, agents decide whether or not to disclose personal information considering trust and intimacy on the one hand and privacy loss on the other hand.
The privacy-trust tradeoff (Seigneur & Jensen, Reference Seigneur and Jensen2004; Lilien & Bhargava, Reference Lilien and Bhargava2008) states that an agent is willing to disclose personal information in order to increase the trust others have in it. Agents are meant to maximize privacy-trust tradeoffs to gain a certain level of trust with the least loss of their privacy. However, the authors of these works associate the different levels of payoff for an agent with the different trust levels that others have in an agent. Then, a trust level is matched with a direct benefit so that an increase in trust results in an increase in utility. As a result, this can be finally modeled as a privacy-utility tradeoff.
There are many cases where the direct benefit of disclosing personal data attributes is not known in advance. This is the case in human relationships, where the disclosure of personal data attributes in fact plays a crucial role in the building of these relationships (Green et al., Reference Green, Derlega and Mathews2006). These relationships may or may not eventually report a direct benefit for an individual. For instance, a close friend tells you what party he voted for. He may disclose this information without knowing (or expecting) the future gains this may cause in utility. Indeed, this disclosure may not report him any benefit. Moreover, current disclosure decision-making models based on the privacy-utility tradeoff do not consider repeated disclosures and their implications. These implications have been broadly studied in psychology, which has lead to findings regarding how humans disclose personal information in the building of their relationships, such as the well-known disclosure reciprocityphenomenon (Green et al., Reference Green, Derlega and Mathews2006). This phenomenon is based on the observation that one person's disclosure encourages the disclosure of the other person in the interaction, which in turn, encourages more disclosures from the first person.
Such et al. (Reference Such, Espinosa, Garcia-Fornes and Sierra2011d, Reference Such, Espinosa, Garcia-Fornes and Sierra2012a) propose a disclosure decision-making mechanism that considers the disclosure reciprocity phenomenon and that disclosures may not report any benefit (or this benefit may not be known in advance). It is based on intimacy and privacy measures to deal with these situations. Then, agent A may choose to disclose a piece of personal information to agent B so that it maximizes the estimation of the increase in intimacy while at the same time minimizing the privacy loss. Agent A does so expecting that this disclosure would encourage agent B to disclose personal information to agent A. To deal with agents that do not reciprocate the disclosures they receive, the authors propose a trust model that considers all the disclosures an agent has made to other agent and the reciprocations it received from this agent in order to predict future reciprocations. Moreover, this proposal also considers the reliability of agents when they disclose information about them, that is, how sincere agents are when they disclose information about themselves. The authors of this work assume that personal attributes can be verified and propose a reliability model based on this. However, although most personal attributes can be verifiable directly or indirectly (e.g. tastes can be inferred from what an agent buys), a few others may not be verifiable, for example, the party an agent votes for using an e-voting system.
3.2 Secure data transfer and storage
Once an agent has decided what information to disclose to which agent, this information must be protected from access by any other third party that is different from the agent that the information is intended for. This includes parties from their local computer and network but also different locations, even across the Internet. As stated by Petkovic and Jonker (Reference Petkovic and Jonker2007), privacy protection is only possible using secure systems. Security and privacy are often related to each other, but they are not the same (Head & Yuan, Reference Head and Yuan2001). On the one hand, security is the control of information in general (Camp, Reference Camp1996). Thus, information is secure if the owner of the information can control it. On the other hand, information is private if the subjectof information can control it. Thus, privacy requires security to control access and distribution of information (Garfinkel, Reference Garfinkel2009).
Agent platforms (APs) provide all the basic infrastructure (for message handling, tracing and monitoring, run-time management, and so on) required to create a MAS (Wooldridge, Reference Wooldridge2002). There are many APs developed by the agent community—for an overview of current APs and the features they provide refer to Alberola et al. (Reference Alberola, Such, Garcia-Fornes, Espinosa and Botti2010). As APs are in charge of executing MAS, they need to be concerned about basic security concepts. However, only a few of them currently take security concerns into account. For instance, Jade (JADE Board, 2005), Magentix (Such et al., Reference Such, Alberola, Espinosa and Garcia-Fornes2011a), AgentScape (Quillinan et al., Reference Quillinan, Warnier, Oey, Timmer and Brazier2008), SECMAP (Ugurlu & Erdogan, Reference Ugurlu and Erdogan2005), Tryllian ADK (Xu & Shatz, Reference Xu and Shatz2003), Cougaar (Newman, Reference Newman2004), SeMoA (Roth & Jalali-Sohi, Reference Roth and Jalali-Sohi2001), the one presented by Ismail (Ismail, Reference Ismail2008) and Voyager (Recursion Software Inc., 2008) are security-concerned APs.
There are security concepts that are necessary for preserving privacy, such as confidentiality (Gangopadhyay, Reference Gangopadhyay2001). All of the above APs use different mechanisms for providing confidentiality. Confidentiality is a security property of a system that ensures the prevention of unauthorized reading of information (Stamp, Reference Stamp2006). This involves both the control of access to information and its distribution. Confidentiality requires authorization mechanisms being in place as well as mechanisms for protecting the transmission of data over a network. They are key mechanisms for avoiding the leak of sensitive information, and, thus, protecting the subject of this information from a privacy breach.
Authorization is the act of determining whether a particular user (or computer system) has the right to carry out a certain activity (Longstaff et al., Reference Longstaff, Ellis, Shawn, Lipson, Mcmillan, Pesante and Simmel1997). The term access control is often used as a synonym for authorization. As an example of a scenario where authorization is necessary for preserving privacy, imagine that two agents A and B are running on the same host and represent two different principals. Agent A may contain a detailed profile about its principal and save this profile in a local file. In this situation, agent B must only be able to access this local file if it is authorized to do so. If agent B succeeds in accessing the file despite not being authorized to do so, this could represent a privacy breach for the principal of agent A.
Security-concerned APs use different authorization mechanisms. These mechanisms allow the specification of rights for agents to carry out activities ranging from traditional access control lists (Jade, Voyager) to other approaches, such as capability-based access control (Ismail, Reference Ismail2008), role-based access control (RBAC; SeMoA, AgentScape, and Tryllian ADK), policy-based access control (Cougaar and SECMAP), and mandatory access control (Magentix). Most APs enforce the access rights that are defined using some of these approaches by means of sandboxing agents.
Confidentiality also implies the protection of transmitted data across a network (Stallings, Reference Stallings2010). In these situations, confidentiality usually means that sensitive information is encrypted into a piece of data so that only parties that are able to decrypt that piece of encrypted data can access the sensitive information. Current security-concerned APs provide confidentiality for the messages exchanged by the agents running on top of them. To this aim, APs use existing secure data transfer technologies such as Kerberos (Neuman et al., Reference Neuman, Yu, Hartman and Raeburn2005), SSL (Frier et al., Reference Frier, Karlton and Kocher1996), and TLS (Dierks & Allen, Reference Dierks and Allen1999). These technologies allow the encryption of messages before transferring them and the decryption of messages once they are received. As a result, if an agent A sends a message to an agent B, A is sure that B will be the only one able to read this message.
Without appropriate confidentiality mechanisms, privacy cannot be preserved. All of the APs above provide the needed secure features to secure data transfer and storage. Therefore, they are suitable to prevent undesired collection of information. However, there are also security concepts that can represent by themselves an actual threat for privacy (Petkovic & Jonker, Reference Petkovic and Jonker2007), even though they are mandatory for the system to be secure. For instance, to achieve access control, each entity trying to gain access must first be authenticated so that access rights can be tailored to it (Stallings, Reference Stallings2010). As we will see in Section 4.3, authentication can itself be a threat for privacy. Only some of the security-concerned APs provide mechanisms for preserving-privacy authentication. This should be considered when choosing an AP if there are privacy concerns to be considered.
3.3 Trusted third party computation
Another approach to prevent information collection is based on third parties. Agents provide sensitive information to third parties that process this information and return the outcomes obtained to the intended destination agents. The agent-based information filtering community has developed some proposals that are based on trusted third parties (Cissèe, Reference Cissèe2003; Aïmeur et al., Reference Aïmeur, Brassard, Fernandez and Onana2006; Cissèe & Albayrak, Reference Cissèe and Albayrak2007). Information filtering architectures take user profiles and generate personalized information based on them. User profiles usually contain information about preferences, rated items, etc. The resulting systems can be recommender systems, matchmaker systems, or can be a combination of both. The proposals we describe in this section enhance privacy by decoupling the three main parts in an information filtering architecture: users, service providers, and filters.
Aïmeur et al. (Reference Aïmeur, Brassard, Fernandez and Onana2006) present a software architecture that they call ALAMBIC. ALAMBIC considers three main parties: users, service providers, and the Still Maker. The Still Maker is a secure platform that generates mobile agents (with a unique public/private key pair) that migrate to service providers. These agents are in charge of filtering the information about users. The code of ALAMBIC agents is encrypted and obfuscated. Moreover, users’ profiles are encrypted with the public key of the mobile agents before being transferred to the service provider. As a result, it is very difficult for service providers to obtain more information than the outputs of the filtering process that is carried out inside the mobile agents. However, according to Cissèe and Albayrak (Reference Cissèe and Albayrak2007), this architecture addresses two aspects inadequately: the protection of the filter against manipulation attempts, and the prevention of collusion between the filter and the provider.
To overcome these aspects, Cissée and Albayrak (2007) propose separating the filter from the service provider. This proposal is based on the use of a trusted AP. Users, service providers, and filter entities (the party that provides filtering functionalities) can deploy agents in the trusted AP. In a nutshell, the information filtering process involves the following steps: (i) the filter entity deploys a temporary filter agent in the trusted AP; (ii) the user entity also deploys an agent, which is called relay agent, in the trusted AP; (iii) the relay agent establishes control of the temporary filter agent (by using mechanisms provided by the trusted AP) and sends the user profile to the temporary filter agent; (iv) the provider profile is propagated from the service provider to the temporary filter agent via the relay agent; (v) the temporary returns the recommendations to the service provider via the relay agent. The authors of this work assume that all providers of APs are trusted. This assumption may be not valid in truly open multi-agent systems in which there could be untrusted APs.
3.4 Summary of proposals against information collection
To sum up, there are three different approaches that deal with avoiding undesired information collection. First, disclosure decision-making mechanisms (Section 3.1) provide agents with reasoning capabilities for deciding which information to disclose to other agents. Disclosure decision-making mechanisms are currently based on: policies (if the policy of the sender agent matches the policy of the receiver agent, then the sender agent decides to send the corresponding information); privacy-utility tradeoffs (information is only revealed if it is worth it in terms of both the utility and the privacy loss of disclosing this information); and social relationships (information is revealed if the receiver agent is intimate enough or to further increase the intimacy with the receiver agent). Second, secure data transfer and storage mechanisms (Section 3.2) are needed to protect the confidentiality of personal information, so that only authorized parties can access this information. In this way, many APs include mechanisms based on traditional security techniques to provide confidentiality. Third, trusted third party computation mechanisms (Section 3.3) avoid unnecessary information collection by receiver agents. Specifically, agents provide sensitive information to third parties that process this information and return the outcomes obtained to the intended destination agents. Finally, to consult the specific open challenges regarding information collection, we refer the reader to Section 6.
4 Protection against information processing
Information processing refers to the use or transformation of data that has been already collected (Spiekermann & Cranor, Reference Spiekermann and Cranor2009). Information processing usually involves various ways of connecting data together and linking it to the individuals to whom it pertains (Solove, Reference Solove2006). For instance, a vendor could have a complete profile of a customer containing relevant data collected from the purchases made by the customer's agent. The vendor can then use information filtering techniques to obtain detailed information on the customer's tastes. Then, the vendor can infer which goods the customer is more willing to acquire and offer them in advance through personalized advertising. Moreover, the vendor could even incur in price discrimination practices, that is, the vendor could charge different prices to different customers depending on the desire that the customer has to acquire a product according to their tastes.
Most of the work for protecting against the processing of information already collected is based on the principle of data minimization. Data minimization states that disclosed personal data should preserve as much unlinkability as possible (Pfitzmann & Hansen, Reference Pfitzmann and Hansen2010). This is a way to reduce the probability of different pieces of data being connected to each other and linked to an individual. Therefore, privacy threats are reduced while still allowing information to be collected.
Spiekermann and Cranor (Reference Spiekermann and Cranor2009) state that ‘Identifiability can be defined as the degree to which (personal) data can be directly linked to an individual’. The degree of privacy of a system is inversely related to the degree of user data identifiability (Pfitzmann & Hansen, Reference Pfitzmann and Hansen2010). The more identifiable data that exists about a person, the less that person is able to control access to information about herself/himself, and the greater the privacy risks. Identifiability ranges from completely identified to anonymous. Throughout this section, we survey different studies in MAS that prevent information processing through minimizing the collection of identifiable data.
Figure 3 depicts a conceptual map for all of the studied approaches that provide support for protecting against information processing.
Information Processing Conceptual Map
4.1 Anonymity
Anonymity is the maximum degree of privacy, so it plays a crucial role in preserving privacy in agent technologies (Brazier et al., Reference Brazier, Oskamp, Prins, Schellekens and Wijngaards2004). The main property of anonymity is that collected data cannot later be attributed to a specific individual. Anonymity is commonly defined in terms of a possible attacker. Pfitzmann and Hansen (Reference Pfitzmann and Hansen2010) define anonymity as ‘Anonymity of a subject from an attacker's perspective means that the attacker cannot sufficiently identify the subject within a set of subjects, the anonymity set.’
Many anonymity systems can be modeled in terms of unlinkability (Diaz, Reference Diaz2006). Pfitzmann and Hansen (Reference Pfitzmann and Hansen2010) define unlinkability as follows: ‘Unlinkability of two or more items of interest (IOIs, e.g., subjects, messages, actions, etc.) from an attacker's perspective means that within the system (comprising these and possibly other items), the attacker cannot sufficiently distinguish whether these IOIs are related or not’. Anonymity can be achieved when a given IOI cannot be linked to a given subject. For instance, the sender of a message is anonymous if the message cannot be linked to a particular subject from a set of subjects that may have potentially sent the message (the anonymity set).
4.1.1 Anonymity in multi-agent problem solving
The agent community has developed algorithms that aim at preserving anonymity in multi-agent problem solving, including both distributed constraint satisfaction (DisCSP) and distributed constraint optimization (DCOP). In these problems, agents need to share information in order to solve a problem of mutual interest. The major concern in DisCSP and DCOP algorithms is that they usually leak information that can be exploited by some agents to infer private information of other agents (Greenstadt et al., Reference Greenstadt, Pearce and Tambe2006). The anonymity set here is the set of agents that share information. The main aim of these protocols is that shared information cannot be linked to the corresponding agent. A typical application is that of meeting scheduling in which agents arrange meetings according to their principals’ schedules. Private information in these problems usually refer to information about: (i) agent preferences (domain privacy), that is, whether an agent can attend a meeting in each time slot in DisCSP or the utility valuations for each agent for each time slot in DCOP; and (ii) the assignment for each agent once a final solution is reached as well as partial solutions during the solving process (assignment privacy).
Yokoo et al. (Reference Yokoo, Suzuki and Hirayama2005) and Silaghi and Mitra (Reference Silaghi and Mitra2004) present secure DisCSP algorithms based on multiparty computation. Multi-party techniques compute general functions with secret inputs. Therefore, these techniques allow the collection of information in a way that cannot be linked back to the agents. Theoretical proofs show that these secure DisCSP algorithms do not leak private information, that is, there is no chance for either an agent or an external entity to link variable assignment to the agents taking part in the problem-solving process. However, these approaches have a high computational cost. DisCSP algorithms require many comparisons and these protocols require exponentiation operations for each comparison. Therefore, these protocols should be used when privacy concerns are very high.
There are other computationally cheaper approaches, such as (Greenstadt et al., Reference Greenstadt, Grosz and Smith2007) and (Brito & Meseguer, Reference Brito and Meseguer2003). However, these approaches still leak information. These works try to reduce privacy loss of existing DisCSP/DCOP algorithms. Metrics based on the Valuations of Possible States (Maheswaran et al., Reference Maheswaran, Pearce, Bowring, Varakantham and Tambe2006) framework are usually considered to quantify the reduction in privacy loss. Greenstadt et al. (Reference Greenstadt, Grosz and Smith2007) present the DPOP with Secret Sharing (SSDPOP) algorithm, which is an extension of DPOP based on the efficient cryptographic technique of secret sharing. Agents use secret sharing to send aggregate values, and, thus, they do not reveal their individual valuations. Brito and Meseguer (Reference Brito and Meseguer2003) present the Distributed Forward Checking (DisFC) algorithm, which is an approach without using cryptography. In DisFC, agents exchange enough information to reach a global consistent solution without making their own assignment public. To this aim, DisFC sends filtered domains (agent preferences) to other agents and replaces their own value by a sequence number.
4.1.2 Anonymizers
There are technologies developed outside the agent community called anonymizers. These anonymizers can be used to obtain communication anonymity. They hide the IP address and other where about information from the messages they receive and forward these messages (Menczer et al., Reference Menczer, Street, Vishwakarma, Monge and Jakobsson2002). If an agent sends a message to another agent using these anonymizers, the receiving agent is not able to identify the sender from the potential senders (the anonymity set in this case). Chaum (Reference Chaum1981) first introduced MIX-networks as a means to counteract traffic analysis. A MIX-network is composed of a set of MIX nodes. Each MIX node receives a number of messages, modifies them (using some cryptographic transformation), and sends them randomly. Moreover, each MIX node in the network knows only the previous and next node in a received message's route. Therefore, an external observer is not able to correlate incoming and outcoming messages.
Onion routing (Goldschlag et al., Reference Goldschlag, Reed and Syverson1999) is based on Chaum's MIX-networks. While MIX nodes could store messages for an indefinite amount of time while waiting to receive an adequate number of messages to mix together, an onion router is designed to pass information with low latency. However, large message traffic is vital to strengthen anonymity in onion router networks. An example of implementation of onion routing is Tor (Dingledine et al., Reference Dingledine, Mathewson and Syverson2004).
Anonymizers also prevent an external observer from inferring a possible relationship between the sender and the receiver of a message (known as traffic analysis). Although authorization and confidentiality are ensured by secure APs (explained in Section 3.2), a potential attacker could also gather information about who is communicating with whom. This is because the IP of both sender and receiver can be known even though the content of the message exchanged is encrypted. Moreover, this potential attacker could also know how often two agents communicate to each other, and then infer relationship patterns between the two agents.
There are some agent architectures and implementations that use anonymizers (Menczer et al., Reference Menczer, Street, Vishwakarma, Monge and Jakobsson2002; Jaiswal et al., Reference Jaiswal, Kim and Gini2004). IntelliShopper (Menczer et al., Reference Menczer, Street, Vishwakarma, Monge and Jakobsson2002) is an intelligent shopping agent that aids customers who are shopping for a product on an e-commerce site. This agent is in charge of monitoring e-commerce sites to notify the customer about updates related to products she/he is interested in. To this aim, the agent is able to collect information about the customer's activities at an e-commerce site to determine interesting products and decisions about whether or not she/he buys the products. Customers connect to IntelliShopper through an anonymizer so that IntelliShopper cannot know about the IP and other where about information of the requests it receives. The MAGNET architecture (Jaiswal et al., Reference Jaiswal, Kim and Gini2004) provides support for auction-based business-to-business marketplaces. MAGNET agents participate in auctions that are reverse, that is, contracting agents present call for bids to supplier agents. MAGNET uses an anonymizer between the market and the bidders. This is intended to reduce market–supplier collusion by making the supplier's bids unlinkable until the end of auction.
There are also anonymizers specially developed for APs. Korba et al. (Reference Korba, Song and Yee2002) present an alternate Onion Routing approach for Jade. Each Jade AP has several onion agents that provide an anonymous data forwarding service, and at least one onion monitor agent that keeps track of the location of all the other onion agents in the system. Onion monitor agents exchange information in order to maintain a valid topology of the complete onion agent network. The main drawback of this approach is that agents in Jade do not communicate directly with each other; instead, it is the container where agents live that finally sends the message over the network to the container where the recipient agent lives. Therefore, an external observer could track the path of a message through the containers and infer possible relations among agents living in these containers. The lower the number of agents in a container, the higher the probability for an external observer to link the message to one particular agent (as sender or receiver). Moreover, the Jade AP itself could monitor the path of a message from one agent to another. A possible solution for this may be that containers connect to each other through the general purpose anonymizers presented above.
4.2 Pseudonymity
Pseudonymity (Chaum, Reference Chaum1985) is the use of pseudonyms as identifiers. A pseudonym is an identifier of a subject other than one of the subject's real names (Pfitzmann & Hansen, Reference Pfitzmann and Hansen2010). Pseudonyms have been broadly used by human beings in the real world. For instance, in the 19th century when writing was a male-dominated profession, some female writers used male names for their writings. Nowadays, in the digital world, there are a great number of pseudonyms such as usernames, nicknames, e-mail addresses, sequence numbers, public keys, etc.
The most important trait of pseudonymity is that it comprises all degrees of identifiability of a subject (from identified to anonymous) depending on the nature of the pseudonyms being used. Complete identification is when the linking between a pseudonym and its holder is publicly known. Anonymity can be achieved by using a different pseudonym for each different transaction known as transaction pseudonyms (Chaum, Reference Chaum1985), unless the information contained in these transactions establishes linkability (Bhargav-Spantzel et al., Reference Bhargav-Spantzel, Camenisch, Gross and Sommer2007).
4.2.1 Ad hoc mechanisms
There are some agent-based approaches that implement ad hoc mechanisms for implementing pseudonymity. Some of these approaches have been proposed in the agent-based information filtering domain (Aïmeur et al., Reference Aïmeur, Brassard, Fernandez and Onana2006; Cissée & Albayrak, 2007). Cissée and Albayrak (Reference Cissèe and Albayrak2007) provide an approach based on transaction pseudonyms. They aim at providing anonymity, that is, the recommendations must not be linkable to the identity of the principal if the principal wants these recommendations to be unlinkable to himself. They propose that principals use a different agent each time they ask for a recommendation. Aïmeur et al. (Reference Aïmeur, Brassard, Fernandez and Onana2006) provide a completely different approach; the principal ‘identifies’ herself/himself when she/he communicates with the service provider by using a pseudonym. Thus, the service provider can build a profile to better aid recommendations but without establishing linkability to the identity of the agent's principal. However, none of these approaches consider that the principal may want some recommendations to be unlinkable while others be linkable; instead they provide the use of either only one pseudonym or a different pseudonym for each recommendation.
Other approaches to pseudonymity come from the agent-based e-commerce domain (Menczer et al., Reference Menczer, Street, Vishwakarma, Monge and Jakobsson2002). Users connect to the IntelliShopper agent using a pseudonym to avoid the link between the profiles that IntelliShopper has about customers and their real identity. Moreover, users can use different pseudonyms for IntelliShopper to have separate profiles for separate activities. Therefore, users can decide whether or not to use a new pseudonym in each transaction, instead of forcing the same pseudonym for all transactions or a different pseudonym for each transaction (as in the approaches described in the above paragraph). However, the authors of this work leave the user with the responsibility of creating their pseudonyms and they do not provide any pseudonym management facility.
4.2.2 Support from APs
Another approach for providing general support for pseudonymity for agent technologies instead of ad hoc solutions is to provide this support from APs. Thus, this support aids agent developers to use pseudonymity without having to implement their own solutions. However, only a few of the APs explained in Section 3 implement some kind of support for pseudonymity. Magentix (Such et al., Reference Such, Alberola, Espinosa and Garcia-Fornes2011a), Secmap (Ugurlu & Erdogan, Reference Ugurlu and Erdogan2005), AgentScape (Quillinan et al., Reference Quillinan, Warnier, Oey, Timmer and Brazier2008) and Cougaar (Newman, Reference Newman2004) assign a unique identity for each agent that it can use to authenticate itself to other agents. Using this identity, agents can act pseudonymously, that is, agents can act on behalf of their principal without using the identity of their principal. However, agents cannot hold more than one pseudonym, that is, principals should use a different agent each time they want to use a different pseudonym (similarly to what is proposed by Cissée and Albayrak (2007) explained above).
Warnier and Brazier (Reference Warnier and Brazier2010) also present a mechanism for the AgentScape AP that offers pseudonymity by means of what they call handlers. Handlers are pseudonyms that agents can use to send/receive messages to/from other agents. At will, agents can request the AP for new handlers. Moreover, the AP is the only one that knows the association between handlers and GUIDs (global unique identities of the agents). An agent can also obtain anonymity by simply using a different handler for each transaction (transaction pseudonyms). AgentScape also offers an automatic anonymity service. Agents can send messages anonymously without having to manage pseudonyms. This service is provided by agents called anonymizers Footnote 6. When an agent wants to send a message anonymously, this message is redirected to an anonymizer. Then, this anonymizer is in charge of removing the original handler of the sender from the message, replacing it with another (possibly new) handler, and sending the message to the intended recipient. If the intended recipient replies, this reply is forwarded to the sender of the original message.
The original sender of the message must notify when a transaction ends. For each new transaction, the anonymizer generates a new handler.
APs that provide support for pseudonymity (e.g. by providing APIs to create and manage pseudonyms) do not consider that pseudonyms can be issued by external third parties. That is, APs themselves are in charge of issuing the pseudonyms. Thus, the AP itself (and the anonymizer agents for the case of AgentScape) must be trusted. This is because the AP knows the relation of pseudonyms to each other and to the principal involved. This usually implies that the organization or company that hosts the specific system (e.g. eBay in the case of an e-marketplace) knows the association of pseudonyms to each other and to principals. Therefore, this organization or company can collect and process information about the principals that run their agents on the system.
Other more general approaches have been proposed to provide pseudonymity to agent technologies (van Blarkom et al., Reference van Blarkom, Borking and Olk2003; Such et al., Reference Such, Espinosa, Garcia-Fornes and Botti2011b). Both approaches propose the integration of Privacy-Enhancing Technologies (PETs; Senicar et al., Reference Senicar, Jerman-Blazic and Klobucar2003) into agent technologies. PETs can be defined as ‘a system of ICT measures protecting informational privacy by eliminating or minimising personal data thereby preventing unnecessary or unwanted processing of personal data, without the loss of the functionality of the information system’ (van Blarkom et al., Reference van Blarkom, Borking and Olk2003).
Van Blarkom et al. (Reference van Blarkom, Borking and Olk2003) propose the use of Identity Protectors. Identity Protectors are PETs that are in charge of converting the identity of the principal involved (the person whose data are being processed) into one or more pseudonyms. They propose that the Identity Protector is placed either between the principal and the agent or between the agent and the environment. The Identity Protector in an information system can take several forms: (i) a separate function implemented in the data system; (ii) a separate data system supervized by the individual; (iii) a data system supervized by a trusted third party. They present in which places of a specific agent architecture an Identity Protector can be placed. However, they do not provide any specific design or implementation of an Identity Protector and the integration of it into an agent architecture.
Such et al. (Reference Such, Espinosa, Garcia-Fornes and Botti2011b) present a proposal for an Identity Protector based on trusted third parties. Moreover, this proposal has been integrated into the Magentix2 AP (Such et al., Reference Such, Garcia-Fornes, Espinosa and Bellver2011c). Specifically, this proposal builds on modern Privacy-Enhancing Identity Management Systems (PE-IMS) (Hansen et al., Reference Hansen, Berlich, Camenisch, Clau, Pfitzmann and Waidner2004). PEIMS are PETs that support the management of pseudonyms to control the nature and amount of personal information disclosed. These systems are aimed at first providing the controlled pseudonymity of the principals; and second, the reliability of the principals. Controlled pseudonymity implies unlinkability between the pseudonym and the real identity of the principal behind the partial identity. Controlled pseudonymity also implies that the pseudonyms of the same principal are unlinkable if they are used in different contexts. The reliability of the users implies that, at first, there is unlinkability between pseudonyms and the real identity of the entities behind them, but under special circumstances the issuer of the pseudonym can make a pseudonym and the real identity of an entity linkable. The authors of this approach assume an identity infrastructure to be used as a trusted third party. This assumption can be made in networks such as the Internet, as contemplated by standards such as the Identity meta system interoperability standardFootnote 7. However, this assumption may not hold in environments with very scarce resources such as sensor networks.
4.3 Implications in security, trust, and reputation
4.3.1 Implications in security
As stated in Section 3, security plays a crucial role in preventing undesired information collection. However, there are also security concepts themselves that can represent an actual threat to privacy (Petkovic & Jonker, Reference Petkovic and Jonker2007) even though they are mandatory for the system to be secure. These security concepts include authentication and accountability. Minimizing data identifiability may affect authentication and accountability if specific countermeasures are not considered.
Authentication binds a principal to a digital representation of her/his identity (Bishop, Reference Bishop2002). To authenticate something on the Internet is to verify that its identity is as claimed (Jøsang et al., Reference Jøsang, Patton and Ho2001). In the case of a message, the function of authentication is to assure the recipient that the message is from the source that it claims to be from (Stallings, Reference Stallings2010). For instance, if an agent A sends a message to an agent B, B should be able to authenticate A as the sender of the message. Authentication of the entities existing in an AP is the basis for confidentiality (Such et al., Reference Such, Alberola, Espinosa and Garcia-Fornes2011a), explained in Section 3. All of the APs that provide some support for pseudonymity (Magentix (Such et al., Reference Such, Alberola, Espinosa and Garcia-Fornes2011a), Secmap (Ugurlu & Erdogan, Reference Ugurlu and Erdogan2005), AgentScape (Quillinan et al., Reference Quillinan, Warnier, Oey, Timmer and Brazier2008) and Cougaar (Newman, Reference Newman2004)) support authentication based on pseudonyms.
The other security concept that has a direct impact on privacy (and vice versa) is accountability. Accountability refers to the ability of holding entities responsible for their actions (Bhargav-Spantzel et al., Reference Bhargav-Spantzel, Camenisch, Gross and Sommer2007). Accountability helps to promote trust in the system. This is because if an agent misbehaves and there are no accountability consequences, there is a sense of impunity that could even encourage abuse. This trust is crucial for systems in which users can be seriously damaged by losing money, such as agent-based e-commerce (Fasli, Reference Fasli2007b).
Accountability usually requires an unambiguous identification of the principal involved (Bishop, Reference Bishop2002). Then, this principal can be held liable for their acts. For instance, a customer agent pays a vendor agent for a good. The agent vendor commits to shipping the good to the customer agent's principal. In the event that the customer agent's principal does not receive the good, the vendor's principalFootnote 8 may be held liable for this. Although determining exactly who should be held liable for this depends on the applicable laws in the specific country, it usually requires the identification of the vendor's principal. Then, the vendor agent's principal can be sued for fraud.
Pseudonyms can be utilized to implement accountability (Hansen et al., Reference Hansen, Berlich, Camenisch, Clau, Pfitzmann and Waidner2004). AgentScape and Magentix keep track of the association between principals and pseudonyms. Therefore, these two APs can disclose the principal behind the pseudonym, removing pseudonymity and producing identity and accountability as a result. The main drawback of this approach is that the AP itself (including the anonymizer agents for the case of AgentScape) must be trusted. This is because the AP knows the relation of pseudonyms to each other and to the principal involved. Although this is needed to ensure accountability (agent principals can still be held liable for their agents behavior even when pseudonyms are used), this usually implies that the organization or company that hosts the specific marketplace (e.g. eBay) knows the association of pseudonyms to each other and to principals. Therefore, this organization or company can collect and process information about the principals that run their agents in the marketplace.
Other approaches such as Such et al. (Reference Such, Espinosa, Garcia-Fornes and Botti2011b) allow APs and agents to use pseudonyms generated by trusted third parties that do not participate in the specific marketplace. Therefore, the AP itself does not know the association between pseudonyms and their principals. These third parties may disclose this information only if required by a court. However, this does not prevent these third parties and the AP (or an agent) from colluding to finally obtain this information.
4.3.2 Implications in trust and reputation
In a multi-agent system, agents usually need to assess trust toward other agents as well as their reputation. To this aim, the agent community has developed a vast number of trust and reputation models (Ramchurn et al., Reference Ramchurn, Huynh and Jennings2004; Sabater & Sierra, Reference Sabater and Sierra2005). An agent can build a reputation by using the same pseudonym more than once. In the same way, an agent can be trusted by a transaction partner by using the same pseudonym for different transactions. Current trust and reputation models are usually based on the assumption that pseudonyms are long lived, so that ratings about a particular entity from the past are related to the same entity in the future. However, when these models are actually used in real domains, this assumption is no longer valid. For instance, an agent that has a low reputation due to its cheating behavior may be really interested in changing its identity and restarting its reputation from scratch. This is what Jøsang et al. (Reference Jøsang, Ismail and Boyd2007) called the change of identities problem. This problem has also been identified by other researchers under different names, for example, whitewashing (Carrara & Hogben, Reference Carrara and Hogben2007).
Kerr and Cohen (Reference Kerr and Cohen2009) also point out the fact that entities could create new accounts (identity in the system) at will, not only after abandoning their previous identity but also holding multiple identities at once. This is known as the sybil attack (Jøsang & Golbeck, Reference Jøsang and Golbeck2009). An example of this attack could be an agent that holds multiple identities in a marketplace and attempts to sell the same product through each of them, increasing the probability of being chosen by a potential buyer.
These vulnerabilities can be more or less harmful depending on the final domain of the application. However, these vulnerabilities should, at least, be considered in domains in which trust and reputation play a crucial role. For instance, in e-marketplaces, these vulnerabilities can cause users to be seriously damaged by losing money. Another example can be a social network like Last.fmFootnote 9 in which users can recommend music to each other. A user who always fails to recommend good music to other users may gain a very bad reputation. If this user creates a new account in Last.fm (a new identity in Last.fm) her/his reputation starts from scratch, and is able to keep on recommending bad music. Users may be really bothered by these recommendations and move to other social networks. In this case, the one seriously damaged is the social network itself by losing users.
A possible solution for these vulnerabilities is the use of once-in-a-lifetime pseudonyms (Friedman & Resnick, Reference Friedman and Resnick1998). Agents can only hold one once-in-a-lifetime pseudonym in each marketplace. Therefore, they cannot get rid of the trust other agents have in them as well as the reputation they earned in the multi-agent system. A model for agent identity management based once-in-a-lifetime pseudonyms has been proposed in Such et al. (Reference Such, Espinosa, Garcia-Fornes and Botti2011b). This model considers two kinds of pseudonyms: permanent and regular. Agents can only hold one permanent pseudonym in a given system. Regular pseudonyms do not pose any limitation. Although both kinds of pseudonyms enable trust and reputation relationships, only permanent pseudonyms guarantee that identity-related vulnerabilities are avoided. Then, agents will choose to establish trust and reputation through permanent pseudonyms if they want to avoid identity-related vulnerabilities. If they want to avoid information processing, they will use as many regular pseudonyms as needed to achieve their desired privacy level (they reach the maximum level by using a different pseudonym for each different transaction). However, this model needs the existence of trusted third parties called Identity Providers to issue and verify pseudonyms. While this may not be a difficulty in networks such as the Internet, this may not be appropriate in environments with very scarce resources such as sensor networks.
There are also other solutions for identity-related vulnerabilities of trust and reputation models that can be used when trusted third parties cannot be assumed (Hoffman et al., Reference Hoffman, Zage and Nita-Rotaru2009). Yu et al. (Reference Yu, Kaminsky, Gibbons and Flaxman2006) present an approach based on social networks represented as a graph in which nodes represent pseudonyms and edges represent human-established trust relationships among them in the real world. They claim that malicious users can create many pseudonyms but few trust relationships. They exploit this property to bind the number of pseudonyms to be considered for trust and reputation. However, this approach is not appropriate for open MAS in which agents act on behalf of principals that may not be known in the real world. (Cheng & Friedman, Reference Cheng and Friedman2005) have demonstrated several conditions using graph theory that must be satisfied when calculating reputation in order for reputation models to be resilient to sybil attacks. This approach needs a particular and specific way to calculate ratings about an individual. Thus, this approach cannot be applied to trust and reputation models that follow other approaches for managing trust and reputation ratings.
4.4 Summary of proposals against information processing
Over the course of this section, we have detailed two different (but at the same time related) approaches to protect against information processing. The first one is based on anonymity (Section 4.1). Specifically, we have presented approaches that provide anonymity in multi-agent problem solving, in which agents need to share information in order to solve a problem of mutual interest but they want to avoid leaking information that can be exploited by some agents to infer private information. We have also shown technologies that support the anonymization of communications over the Internet by means of what is known as anonymizers. The second approach is based on pseudonymity (Section 4.2). We have shown several ad hoc approaches that provide limited pseudonym support. Moreover, we have also described complete pseudonym management systems that some APs provide and that facilitate the development of agents that make extensive use of pseudonyms (which can also lead to achieve anonymity if, for instance, an agent uses a different pseudonym for each new communication). We have also shown in Section 4.3 the interplay between anonymity and pseudonymity on the one hand, and security, trust and reputation on the other hand. We have concluded that anonymity and pseudonymity can have a direct impact on security, trust and reputation, which also play a crucial role to preserve privacy. In this way, we have described works that enhance privacy (by means of pseudonymity) but without impacting on security, trust, and reputation. Finally, we refer the reader to Section 6 to consult the open challenges on this.
5 Protection against information dissemination
Information dissemination refers to the transfer of previously collected and possibly processed data to other third parties. It should be pointed out that protection against dissemination in an open environment such as open multi-agent systems is a very hard problem. This is mainly because when a sender agent passes information to a receiver agent, the former usually loses control over that information. Moreover, it is very difficult for the sender agent to verify whether or not the receiver agent passes this information to other third parties. In the following, we outline some approaches to protect against information processing based on concepts usually used in agent-based technologies: trust and reputation, and norms.
Figure 4 depicts a conceptual map for all of the studied approaches that provide support for protecting against information dissemination.
Information Dissemination Conceptual Map
5.1 Based on trust and reputation models
One approach to prevent information dissemination is based on trust and reputation models. There are works that assume that the reputation of another agent with regard to how they use the information they collect/process is available (Yassine & Shirmohammadi, Reference Yassine and Shirmohammadi2009). Thus, agents can choose not to send information to agents that have a bad reputation. In this case, having a bad reputation means that the agent usually disseminates personal information about other agents. To measure the trustworthiness of agents regarding whether they disseminate personal information or not, one of the many trust and reputation models developed by the agent community could be used (refer to Ramchurn et al. (Reference Ramchurn, Huynh and Jennings2004), and Sabater and Sierra (Reference Sabater and Sierra2005) for reviews on trust and reputation models).
These models usually need to verify the behavior of an agent in the past to predict their future behavior. However, how could an agent verify that another agent has disseminated information about it? The verification of whether or not an agent disseminates personal information about other agents or not is not straightforward. One approach could be that an external entity controls all the communications among agents. Thus, this external entity is able to know if an agent is disseminating information about another. This approach, however, cannot be applied due to its privacy implications (this external entity would act as a big brother). Instead, we envision multi-agent systems in which communications between each agent pair can be encrypted (by using mechanisms such as the ones presented in Section 3.2) to avoid undesired information collection by any other external entity.
Sierra and Debenham (Reference Sierra and Debenham2008) present a model for detecting undesired information dissemination based on information-theoretic measures. They consider that agents are uncertain about their world model. An agent estimates the amount of information that another agent possibly disseminated about the former agent from the information in the messages that the agent receives from other agents. To this aim, the agent sets update functions of its uncertain world model based on the messages received. For instance, if an agent A sends to agent B that A likes the color pink, agent A can set an update function of the messages received that scan for information related to the color pink. Thus, if an agent C sends agent A a message offering pink dresses, A could infer that B probably disseminated its color preferences to C. According to this, agent A can revisit the trustworthiness of agent B regarding information dissemination. This model only considers what an agent can observe by itself. However, other agents could also warn this agent about the fact that another agent is disseminating information about it. For instance, in the previous example, if agent A and agent C are known to each other, agent C may not take advantage of knowing A's color preferences. Instead, agent C can warn agent A that agent B disseminates information about it.
5.2 Based on normative multi-agent systems
This approach is based on using norms for governing the dissemination of information in a so-called normative multi-agent system (Criado et al., Reference Criado, Argente and Botti2011). According to Boella et al. (Reference Boella, van der Torre and Verhagen2007), a norm is ‘a principle of right action binding upon the members of a group and serving to guide, control, or regulate proper and acceptable behavior’. In this case, proper and acceptable behavior means that agents should not be able to disseminate sensitive information about other agents without their consent.
Barth et al. (Reference Barth, Datta, Mitchell and Nissenbaum2006) present a logical framework for expressing and reasoning about norms of transmission of personal information. This framework formalizes the main ideas behind contextual integrity. Contextual integrity (Nissenbaum, Reference Nissenbaum2004) is a legal framework for defining the appropriateness of the dissemination of information based on the context, the role of the entities taking part in the context, and the subject of the personal information being transferred. In the framework of Barth et al. (Reference Barth, Datta, Mitchell and Nissenbaum2006), privacy norms are expressed as linear temporal logic (LTL) formulas. These formulas are used to define the permissions and prohibitions when disseminating private information about other agents. For instance, it can be expressed that, in a medical context, an agent playing the role of doctor is allowed to pass medical personal information to an agent only if this agent is the subject of the information and this agent is playing the role of patient. Note the difference to an RBAC approach, which allows the definition of permission based on the roles of the entities taking part in the system. However, it cannot use the information about who is the subject of the information being transferred. Barth et al. (Reference Barth, Datta, Mitchell and Nissenbaum2006) assume a closed system in which all agents abide by the norms, so they do not provide any enforcement mechanism.
Krupa and Vercouter (Reference Krupa and Vercouter2010) present a position paper that includes an initial proposal for controlling personal information dissemination in open multi-agent systems based on contextual integrity (explained above). They consider that agents may not abide by the norms. They propose five privacy-enforcing norms to promote privacy-preserving behaviors when disseminating information: (i) respect the appropriateness of the information to be transferred according to contextual integrity, that is, agents should not transfer information that is not appropriate regarding the context, the roles of the agents involved, and the subject of the information; (ii) sign the transmission chain before sending the information, so agents that transmit information remain liable for this transmission; (iii) do not send information to violating agents, that is, agents that do not abide by the norms; (iv) delete information received from violating agents so that this information is no longer transferred; and finally (v) punish agents violating these norms (including this one) by sending punishment messages (messages that inform that a given agent has performed a violation) to the subject of the information and also other agents in the system.
Krupa and Vercouter (Reference Krupa and Vercouter2010) suggest the use of trust models based on the punishment messages to isolate violating agents, that is, if an agent is said to violate norms, other agents will not send personal information to it. Thus, trust and reputation models can be used based on these punishment messages. This is because, in this case, it is assumed that all of the agents will follow the norms, and in the event of not doing so, these punishment messages can act as the verification mechanism needed for trust and reputation models, as explained in Section 5.1. However, this work is at an initial stage. Specifically some major issues remain open (according to the authors): (i) the real connection of their proposal to trust models needs to be specified; (ii) two or more agents can easily collude by passing information to each other without other benevolent agents being aware of it; (iii) one agent can consider that another agent is not trustworthy according to its trust model, while another can consider it to be trustworthy (i.e. some transmissions can be viewed as appropriate by an agent and the same transmissions can be viewed as inappropriate by another agent); and (iv) the system can be subject to strategic manipulation, such as agents sending fake punishment messages that do not really correspond to real violations.
6 Open challenges
Over the course of this article, we have identified many possible lines for future research. In this section, we outline some of the most challenging possible future directions in the research field of privacy and MAS. These possible directions are open challenges identified during the realization of this survey. To our knowledge, all these open challenges will play a crucial role in broadening the use of agent technologies. On the one hand, principals will be more willing to engage with and delegate tasks to agents. On the other hand, agent technologies can be mixed/integrated with/into other information technologies to enhance the privacy that these information technologies provide.
6.1 Interoperability and openness
As stated by Luck et al. (Reference Luck, McBurney, Shehory and Willmott2005), interoperability is crucial for the medium-term development of MAS. Interoperability is a basic requirement for building open MAS, in which heterogeneous agents can enter and leave the MAS at any moment, can interact with each other, and can span organizational boundaries. For instance, agent-based e-marketplaces are open MAS (Fasli, Reference Fasli2007a), in which buyer, seller, and broker agents can be developed by different developers using different languages and frameworks, so heterogeneity is inherent. Thus, agents and their interaction protocols need to allow interoperation. Thus, standards that help to allow interopertion are of crucial importance.
Although there are some standards proposed for agents and their interaction protocols, as yet there is no standard focusing on privacy issues. As described in Section 3.2, a first requirement for privacy is security. There have been some standards for security in MAS. FIPA defined a standard for security in MAS, but this standard soon became obsolete (FIPA, 1998). There have been some studies that consider this obsolete standard as a basis to analyze and propose guidelines for FIPA-based security standards (Poslad & Calisti, Reference Poslad and Calisti2000; Poslad et al., Reference Poslad, Charlton and Calisti2003). However, there has not been another proposal for a security standard for FIPA platforms since the obsolete standard from 1998.
The OMG Mobile Agent System Interoperability Facility (MASIF; Milojicic et al., Reference Milojicic, Breugst, Busse, Campbell, Covaci, Friedman, Kosaka, Lange, Ono, Oshima, Tham, Virdhagriswaran and White1998) is a standard for mobile agents. Mobile agents are a species of autonomous agents that are capable of transmitting (migrating) themselves—their program and their state—across a computer network, and recommencing execution at a remote site (Wooldridge, Reference Wooldridge2002). MASIF specifies security mechanisms for mobile agents to migrate among hosts and also secure communication mechanisms. However, the security of MASIF is dependent on Corba-IDL mechanisms. No other transport mechanisms are considered, such as HTTP, AMQP, and others.
Security standards play a crucial role in preserving confidentiality in agent interactions. However, there are many other mechanisms that are needed for preserving privacy that also need to be standardized. For instance: how can agents selectively disclose parts of their identities in a standard way (as required by the disclosure decision-making mechanisms in Section 3.1)? How can agents change the pseudonyms they use in a standard way (as required for pseudonym management technologies in Section 4.2)? These standards do not need to be built from scratch. Instead, existing standards can be used as the basis for them. For instance, the OASIS Identity Metasystem Interoperability standardFootnote 10 is a standard for mechanisms that support pseudonymity and the selective disclosure of identity attributes.
6.2 Pseudonym changer agent
According to Hansen et al. (Reference Hansen, Schwartz and Cooper2008), one of the main questions that is relevant for pseudonyms to be privacy preserving is the amount of information that can be gathered by linking data disclosed under the same pseudonym. Social security numbers in the United States are a clear example that if a pseudonym is used for a long time, even spanning different contexts, different pieces of personal information disclosed in different contexts can be linked to each other, and also allow the inference of other personal information emerging from the combination of data in different contexts and applying learning and inference techniques. Moreover, linking a pseudonym to the real identity only once is sufficient to be able to associate all of this personal information to a real identity. This link can remain over time. Thus, other personal information disclosed under this pseudonym can be linked to the real identity of the principal subject of this personal information.
Clauβ et al. (Reference Clauβ , Kesdogan and Kölsch2005) point out that different pseudonyms should be used in different contexts if the principal wants to maintain the personal information disclosed under each of these pseudonyms unlinkable. The most privacy-preserving option is to use transaction pseudonyms, which treat each transaction as a different context. However, there are many cases in which the principal can be interested in reusing the same pseudonym, for example, a social network that focus on a specific topic in which the principal is willing to establish friendships and other kinds of relationships that need the reuse of pseudonyms for recognizing entities from one time to another. Another example is that the principal itself could be willing to provide his/her profile to the seller agent in an e-commerce scenario in exchange for a discount or a reward, as pointed out in Section 3.1.
We mentioned several approaches for pseudonym support in Section 4.2. However, we could not find any approach that suggests pseudonym changes. In other words, the needed mechanisms for agents to be able to change their pseudonym exist, but there is no study or proposal for agents to decide when to change their pseudonym. This responsibility is given to the agent's designer or agent's principal. We envision pseudonym changer agents. These agents would be in charge of suggesting pseudonym changes by evaluating the privacy risks of reusing a pseudonym. Moreover, the models detailed in Section 3.1 could also be applied to make the decision of whether or not a pseudonym change is appropriate. This decision would take into account: the privacy risks due to not changing the pseudonym, and the utility or intimacy that would be lost by changing the pseudonym. Such et al. (Reference Such, Serrano, Botti and Garca-Fornes2012b) propose the first step toward this ambitious endeavor.
6.3 Disclosure decision making based on multiple criteria
As stated in Section 3.1, current disclosure decision-making mechanisms are based on policies, the privacy-utility tradeoff, or social relationships. However, there is no proposal that brings these mechanisms together. This could be very appropriate for situations in which each one of these mechanisms is not enough by itself to cope with the requirements of agents’ principals.
There are many examples of environments and situations in which these disclosure decision-making mechanisms can be combined. For instance, in a controlled environment in which policies are known to be enforced, the policies themselves can be used for agents that disclose personal information based on the privacy-utility tradeoff. In this way, agents are able to valuate the privacy loss they will suffer in the event of disclosing a specific piece of data, according to the policy of the intended destination agent. Then, based on this estimated privacy loss they can determine if the expected benefit for disclosing the information is worth it.
Another example could be the combination of the privacy-utility tradeoff with other more social approaches. For instance, suppose that an agent knows the benefit that disclosing personal information to another agent may cause to itself. Also suppose that this benefit is not worth the disclosure according to the privacy-utility tradeoff. An agent can still decide to disclose this information if it has a relationship that is intimate enough with the intended destination agent. Moreover, the agent could also decide to disclose this information if it does not have a relationship that is intimate enough with the intended destination agent, but it wants to reciprocate a previous disclosure of the intended destination agent.
Now, suppose that an agent has very low intimacy with another agent. Moreover, suppose that this low level of intimacy is due to the fact that the second agent deceived the first agent by not reciprocating its disclosures. The question arises as to whether or not the first agent should disclose personal information to the second agent if the first agent knows the utilitarian benefit of doing so and this benefit is high enough. In other words, how could the agent decide which of the two mechanisms to follow in a given situation?
6.4 Collective disclosure decision making
All of the disclosure decision-making mechanisms presented in Section 3.1 assume that the decision of whether or not to disclose is to be taken by only one agent. However, what to do when disclosing is a collective decision? There are many application domains in which the items to be disclosed may involve more than one agent, so that the privacy preferences of all of the agents involved should be satisfied when disclosing those items. For instance, in social networks there are items such as photos than may depict different people so that the privacy preferences of all of the people depicted should be respected (Gross & Acquisti, Reference Gross and Acquisti2005). This can also occur in domains such as Virtual Organizations (Criado et al., Reference Criado, Argente, Julian and Botti2009) of agents that can produce items (that can contain sensitive information) and that must decide whether or not to disclose that items to other agents from outside the Virtual Organization. Finally, this can also occur generally in any of the existing collaborative software applications (Leuf & Cunningham, Reference Leuf and Cunningham2001) in which different parties must collectively decide whether or not to disclose and to whom to disclose the documents that are collaboratively authored.
A simple solution for this is to let agents to define their preferred privacy preferences with respect to an item and then merge these preferences into a disclosure policy for a given item. This merge would be valid as long as agent preferences do not conflict. However, there can be situations in which privacy preferences may conflict and this merge may not be obvious or may even be impossible. For these situations, we encourage research on methods and mechanisms to reach a common decision about whether or not to disclose and to whom to disclose items that involve more than one agent. These methods and mechanisms can be based on existing technologies to reach agreements among agents. These technologies are commonly referred to as Agreement Technologies (Sierra et al., Reference Sierra, Botti and Ossowski2011), and involve technologies such as automated negotiation (Jennings et al., Reference Jennings, Faratin, Lomuscio, Parsons, Wooldridge and Sierra2001). Thus, agents could negotiate a common disclosure decision on a particular item.
6.5 Learning the privacy sensitivity of personal information
Most of the approaches presented in this survey assume that agents know the real privacy sensitivity of each personal attribute of their principals. However, this assumption is not always realistic. For instance, the number of personal attributes can be very large, so principals may not feel comfortable specifying the sensitivity for each of their personal attributes. Some of the approaches try to minimize this burden by clustering attributes into categories so that principals specify the sensitivity for each category (Yassine et al., Reference Yassine, Shirehjini, Shirmohammadi and Tran2010). However, this can be also a burden if there is a huge number of categories or if the categories must be defined per target agent and there is a huge number of possible target agents.
A possible future line of research could be to automatically learn the privacy sensitivity of personal information based on studies such as the one presented by Huberman et al. (Reference Huberman, Adar and Fine2005). They carried out an experiment that validates that people are usually more willing to disclose certain private attributes that are typical or positively atypical compared with the target group. The experiment assesses the value (in terms of monetary compensation) that people give to disclose personal attributes like weight and age. They gathered interesting results regarding weight (for age the significance was less): people who weigh less than the average required little compensation to disclose their weight, while people who weigh more than the average required a large compensation to disclose their weight. This is due to the fact that people who weigh more are afraid to feel embarrassment or stigmatization. The authors found a linear relationship about a trait and the value one places on it. The less desirable the trait, the more reluctant a person is to disclose the information. However, small deviations in a socially positive direction are associated with a lower monetary compensation request.
6.6 Personal data attribute inference
The decision-making models presented in Section 3.1 consider the privacy loss of disclosing a personal attribute before deciding whether they finally disclose it or not. This privacy loss usually considers the sensitivity of the personal attribute to be disclosed and the probability of linking this personal attribute to the real identity of the principal behind the agent. Although the agent can decide whether or not to disclose each attribute, it cannot control that other agents can infer other attributes that it does not want to disclose. This is known as the inference problem (Farkas & Jajodia, Reference Farkas and Jajodia2002). For instance, in the United States, if a principal discloses its driver license number, she/he is also disclosing that she/he is, at least, 16 years old.
Only a few of the decision-making models consider what could be inferred due to the disclosure of a personal attribute. Moreover, the decision-making models that consider these inferences provide very simple inference models. Several approaches tackle this problem in different computer science disciplines. These approaches are intended to infer the probabilities of linking personal data attributes to each other and to the principal they describe. For instance, there are approaches that deal with the inference problem when querying databases (Cuenca Grau & Horrocks, Reference Cuenca Grau and Horrocks2008), when applying data mining techniques (Zhu et al., Reference Zhu, Wang and Du2009), in social networks (Zheleva & Getoor, Reference Zheleva and Getoor2009), and in general, in all activities that require the publication of data (Chen et al., Reference Chen, Kifer, LeFevre and Machanavajjhala2009). All of these approaches consider complex models of personal information inference. The disclosure decision-making mechanisms for agents can either be based on these models or they can be adapted for the case of agents.
6.7 Information dissemination detection
As shown in Section 5, there are few studies that focus on information dissemination. Although these studies solve some of the problems that must be dealt with for protecting information dissemination, there are still many challenges that remain open. One such open challenge involves mechanisms for agents to detect when other agents disseminate information about them.
Sierra and Debenham (Reference Sierra and Debenham2008) propose an approach for an agent to detect that another agent is disseminating information about it based on scanning all the information the first agent receives in the search for clues of possible information disseminations. However, an agent may not be able to detect by itself that other agents are disseminating information about it. Another approach for information dissemination detection is based on notifications sent by other agents warning of possible dissemination of information. These notifications can play a crucial role when an agent itself is unable to detect that other agents are disseminating information about it. Krupa and Vercouter (Reference Krupa and Vercouter2010) use notifications of disseminations in the form of what they call punishment messages. These messages are sent by the agent that detects an inappropriate dissemination to the rest of the agents, so that agents can know which agents perform inappropriate information disseminations. However, this mechanism can be subject to strategic manipulation, such as agents sending messages containing fake norm violations that do not really correspond to real violations.
6.8 Integration of trust, reputation, and norms for protecting against information dissemination
The real connection of norm-based approaches to trust-based approaches for avoiding information dissemination needs to be specified. This open challenge is also closely related to the information dissemination detection problem. If an agent is able to detect that another agent has performed information dissemination, it could revise the trust the first agent has in the second agent. Moreover, an agent could earn a very bad reputation with other agents by performing undesired information dissemination. In this way, both trust and reputation would act as privacy-enforcer mechanisms, isolating agents that disseminate information in an inappropriate way.
Krupa and Vercouter (Reference Krupa and Vercouter2010) suggest that messages informing agents that violate the corresponding information dissemination norms can be used as inputs for trust and reputation models. Therefore, agents that do not abide by the norms would be considered as untrustworthy and would earn a bad reputation. This would finally result in the isolation of agents that do not abide by the norms. However, the authors of this work do not discuss how this integration of trust, reputation, and norms can be effectively achieved.
6.9 Avoiding collusion for protecting information dissemination
As shown in Section 5, current norm-based approaches to information dissemination are vulnerable to collusion. Thus, two or more agents could easily collude by passing information to each other without other benevolent and norm-abiding agents being aware of it. This could be addressed by using a central authority that would control and monitor the information that agents exchange. However, this may not be possible for various reasons. The main one, in line with this article, is privacy preservation. This is because this authority would become a big brother. Moreover, there may be other reasons, such as to prevent this authority from becoming a performance bottleneck and a SPOF.
Moreover, the problem of collusion could even be worse if we consider collusion in which one agent decides to disseminate information but without revealing the source of the information. Krupa and Vercouter (Reference Krupa and Vercouter2010) identified this problem and called the agent that disseminates the information ‘journalist’. As they state, a journalist agent would be an agent that decides to sacrifice himself to become a relay for information that violates the information dissemination norms. Therefore, the agent that is the source of the violation will never be known, and only the journalist will be seen as a violator of these norms. A journalist agent could even be rewarded with a monetary benefit in exchange for its practices.
6.10 Protection against information collection and dissemination
The combination/mixing of disclosure decision-making models with information dissemination models can play a crucial role preventing both information collection and information dissemination. All of the disclosure decision-making models presented in Section 3.1 assume separate interactions among agents for evaluating the privacy cost that a disclosure may cause. That is, these models do not consider that the agent that received the disclosure can share the received information with other agents.
An illustrative example can be: agent A decides by means of a disclosure decision-making models not to disclose its attribute location to agent B, for example, the expected utilitarian benefit for agent A to disclose its location to agent B is not high enough compared with the privacy loss this disclose may cause to agent A. However, in a different interaction, agent A decides, by means of a disclosure decision-making model, to disclose its attribute location to agent C, for example, the expected utilitarian benefit for agent A to disclose its location to agent C is high enough compared with the privacy loss. After this, agent A effectively discloses its location to agent C. Then, suppose that agent B and agent C are known to each other, so agent C can finally disclose the location of agent A to agent B. Therefore, B finally knows the location of agent A, even though agent A decided not to disclose it directly to agent B. Thus, we consider that if the dissemination risk is known, it should be considered when deciding whether to disclose information because if information collection is prevented, information dissemination cannot occur.
Another example can be: agent B has a bad reputation in a society because it usually disseminates the information it receives about other agents. Therefore, other agents in the society can decide to avoid disclosing information to B. However, suppose that an agent A and agent B have a very close relationship, that is, they have a medium/high degree of intimacy. Then, suppose that agent A has to decide whether to disclose its location to agent B. In this particular case, agent A could consider that the intimacy it has with agent B is high enough to assume that if it discloses new information to agent B, agent B will not disseminate it to other agents.
7 Conclusions
In this paper, we have introduced the issue of privacy preservation and its relation to multi-agent systems. We have also surveyed the state of the art on studies that fall into the intersection between privacy and multi-agent systems. We have classified them regarding the information-related activities that these studies aim to prevent: information collection, information processing, and information dissemination. First, there are works that prevent undesired information collection by means of disclosure decision- making mechanisms, by means of secure data transfer and storage infrastructures, and by means of trusted third parties that mediate communications. Second, there are works that prevent undesired information processing by using anonymity and pseudonymity techniques. Finally, there are works that prevent undesired information dissemination based on trust and reputation on the one hand and normative multi-agent systems on the other hand.
Although we have presented many studies that provide satisfactory solutions for some specific problems, we consider that research on privacy and multi-agent systems is still in its infancy. As pointed out in Section 6, there are still a great number of possible research lines that remain unexplored. To our knowledge, privacy will be a matter of major concern and will be the subject of many research efforts during this century. Multi-agent systems can play a crucial role for preserving privacy. To this aim, multi-agent systems need to preserve the privacy of the personal information that the agents that take part in these systems hold about real people (e.g. the personal information an agent could have about its principal in applications in which agents act on behalf of their principals). Agent-based solutions should also be integrated into other information technologies to enhance privacy. Thus, approaches based on concepts that are usually used in agent-based technologies (such as trust and reputation, and norms) can further enhance privacy in other existing information technologies.
Acknowledgments
This work has been partially supported by CONSOLIDER-INGENIO 2010 under grant CSD2007-00022, and project TIN2011-27652-C03-00 of the Spanish Government.
