An Incremental Algorithm for Mining Privacy-Preserving Frequent Itemsets Jinlong Wang, Congfu Xu∗, Yunhe Pan Institute of Artificial Intelligence, Zhejiang University Hangzhou, 310027, China [email protected] [email protected] [email protected]

Abstract Privacy preserving data mining is a novel research direction in data mining and statistical databases, where data mining algorithms are analyzed for the side-effects they incur in data privacy. There have been many studies on efficient discovery of frequent itemsets in privacy preserving data mining. However, it is nontrivial to maintain such discovered frequent itemsets because a database may allow frequent itemsets updates and such frequent itemsets may be turned into infrequent itemsets. In this paper, an incremental updating algorithm IPPFIM is proposed for efficient maintenance of discovered frequent itemsets when new transaction data are added to a transaction database in privacy preserving. The algorithm makes use of previous mining results to cut down the cost of finding new frequent itemsets in an updated database, the performance evaluation shows the efficiency of this method. Keyword: Data mining, privacy-preserving, incremental

1

Introduction

With the development of computer hardware and software and the rapid computerization of business, the amount of data available for analysis has grown exponentially in many areas such as retail industry, financial forecast, decision support and intrusion detection. When the scale of data manipulation, exploration and inferencing went beyond human capacities, a new technique named data mining emerged. The term data mining refers to the nontrivial extraction of valid, implicit, potentially useful and ultimately understandable information in large databases ∗ Correspondence

author: Congfu Xu

with the help of the ubiquitous modern computing devices [1, 2]. During the past decade, many successful applications in data mining have been reported from varied sectors such as marketing, finance, banking, manufacturing and telecommunication. As a valuable technique, data mining is developing flourishly, meanwhile, there arise serious concerns over individual privacy in data collection, processing and mining [3], as a result preserving privacy appears as a prime concern in the field of data mining. The conventional wisdom held that data mining, with its promise to efficiently discover valuable, non-obvious information from large databases, is particularly vulnerable to misuse [4]. [5] predicted the making of a conflict between data mining and privacy. The reality is that data mining results rally violate privacy. The objective of data mining is to generalize across populations, rather than reveal information about individuals. The hitch is that data mining works by evaluating individual data that is subject to privacy concerns. So, the true problem is not data mining, but the way data mining is done. [6] and [7] proposed the concept of privacy preserving data mining (PPDM) aimed at alleviating the conflict between data mining and privacy. As a novel research direction in data mining and statistical databases [8], PPDM has begun to receive attentions and be investigated by many researchers [6, 7, 9, 10, 11, 12, 13, 14, 15]. PPDM is defined as “getting valid data mining results without learning the underlying data values” [9]. PPDM encompasses the dual goal of meeting privacy requirements and providing valid data mining results [10]. As a first introduction of PPDM, the idea of [6] is to perturb individual data values. By the perturbation, the original data is hidden and only the randomized values are revealed, but the statistical characteristics have been kept in order that accurate models without access to the precise information in individual data records can be developed. An alternative approach [7] in PPDM is to build a data mining model from local data sets of various participating sites without revealing individual records of one site to other participating sites. The

method is usually based on SMC (Secure Multiparty Computation) [16]. Although the security definition of the randomization model is much weaker than the one in the SMC model, the randomization model aims to protect the (exact) actual data value, and it can gain the higher efficiency than the SMC model (When the number of participants becomes large, the performance will not be desirable.), because of this, randomization method is currently greatly applied to privacy preserving data mining [12, 13, 14, 15]. Randomization methods address the issue of privacy preserving by perturbing the data and reconstructing the distributions at an aggregate level in order to perform mining. All these papers [12, 13, 15] consider randomization techniques in privacy preserving frequent itemset mining. These algorithms try to extract the data itemsets without directly accessing the original data and attempt to guarantee that the mining process does not get sufficient information to reconstruct the original data. However, in reality, data changes from time to time. The itemset mined can present some development trends, when the database dynamically incrementing, some new frequent itemsets can appear, and some old frequent itemsets can disappear, so the incremental mining is paramount important. When database changes, mining the updating database again will not meet the requirements of the real-time response, so, the efficient updating algorithm must be devised to update, maintenance and management the mined knowledge. In this paper, an efficient updating technique will be applied to privacy preserving frequent itemset mining, and an incremental algorithm, called IPPFIM (Incremental Privacy-Preserving Frequent Itemset Mining) will be proposed to improve the efficiency in database incremental updating. The remainder of this paper is organized as follows. The related concepts are described in Section 2. In Section 3, an efficient incremental privacy preserving frequent itemset mining algorithm IPPFIM is presented. The performance study of IPPFIM is reported in Section 4, which shows the efficiency of this method. Finally, Section 5 concludes this paper.

2 2.1

Preliminaries Frequent Itemset Mining

As a key stage in many data mining applications, including the discovery of association rules, strong rules, correlations, sequential rules, episodes, multidimensional itemsets, and many other important discovery tasks, frequent itemset (itemset) mining problem has received a great deal of attentions since its introduction in 1993 by Agrawal et al. [17]. Let I = {i1 , i2 , . . . , im } be a set of distinct literals, usually called items. Let D, the task-relevant data, be a set of

database transactions where each transaction has a unique identifier, called T ID, and contains a set of items. For a set of items A ⊆ I, a transaction T is said to contain A if A ⊆ T . A set of items is referred to as an itemset. An itemset that contains k items is a k-itemset. Definition 1 Suppose A ⊆ I is an itemset, then the frequent itemset in a given database D with respect to a frequency threshold min sup is F (D, min sup) = {A ⊆ I|sup(A, D) ≥ min sup}, where, sup(A, D), the support of an itemset A, is the relative frequency of an item set A in transaction databases D, and min sup is the least support defined by users, min sup ∈ (0, 1). The problem of mining frequent itemset is to mine all itemsets whose support is no less than s, a user-specified minimum support threshold. It usually makes use of the download closure of frequent itemsets: all subsets of a frequent itemset are frequent and that all supersets of an infrequent itemset are infrequent. These two properties are usually to be applied to prune elements of the itemset lattice.

2.2

Privacy Preserving Frequent Itemset Mining

In [13], Rizvi and Srikant presented a scheme called MASK (Mining Associations with Secrecy Konstraints), based on a simple probabilistic distortion of user data, employing random numbers generated from a pre-defined distribution function. In order to address the runtime efficiency issue in MASK, [15] achieved the improvement through changes in both the distortion process and the mining process of MASK, presenting a new algorithm EMASK (Efficient MASK). By generalizing the distortion process to perform symbol-specific distortion and appropriately choosing the distortion parameters and applying a variety of optimizations of set theory in the reconstruction process, runtime efficiencies are well achieved. By the virtue of randomization to distort the original database, frequent itemset can be mined in privacy preserving, the definition is as the following. Definition 2 Let D be a transactional database, D∗ be a distorted database from D in order to preserve individual privacy in D. It is this distorted database D∗ that is eventually supplied to the data miner, along with a description of the distortion procedure. The data miner mines the distorted database D∗ to estimate the frequent itemsets with support count satisfying the minimal support in the original database D, by virtue of the distribution procedure. Figure.1 illustrates the process.

Table 1. Incremental Mining Relationship Table

Figure 1. Privacy preserving frequent itemset mining process.

X ∈ F pd Yes Yes No No

X ∈ Fp Yes No Yes No

X ∈ F p′ Yes Visiting D to computing Visiting d to computing No

Property 3 For c ∈ F p, when c ∈ F pd , c ∈ F p′ .

3

Efficient Incremental Privacy Preserving Frequent Itemset Mining

Property 4 For c ∈ F p, when c 6∈ F pd , c ∈ F p′ ⇔ sup(c, D ∪ d) ≥ s.

In reality, data changes from time to time in many areas, including the retail industry and the finance sector. The itemset mined in these wide applications can present some development trends. When the transaction database changes with time, dynamically increments, some new frequent itemsets can appear, and some old frequent itemsets can disappear, which induces the incremental mining, a paramount important mining method. When increasing, mining the updating database, composed with the original and update database again, will not meet the requirements of the real-time response. In these dynamic databases, the knowledge acquired can facilitate successive discovery processes, and the efficient updating algorithm must be devised to updating, maintenance and management such knowledge,

2. For infrequent itemsets in D, computing the support of the frequent itemsets in F pd visiting D∗ .

Definition 3 Let D be the original database and s be the minimum support threshold, and d denote the incremental database where new transactions or new customers are added to D. D∗ and d∗ denote the database after distorted. F p expresses the frequent itemsets in the original database D, F pk expresses k-frequent itemsets, F p′ is the set of frequent itemsets in D ∪ d, F p′k expresses k-frequent itemsets in D ∪ d. Property 1 Suppose c ∈ F pk . c 6∈ F p′k ⇔ sup(c, D∪d) < s. Property 2 Suppose c 6∈ F pk . c ∈ F p′k ⇔ sup(c, D ∪d) ≥ s. In Table.1, we summarize the relationship of the frequent itemset in an incremental update environment (Let F pd denote the frequent itemset in incremental database d.). From Table.1, we can find the two key problems in incremental mining: 1. For frequent itemsets F p in D, find the not or still available frequent itemsets.

Property 5 For c 6∈ F p, when c 6∈ F pd , c 6∈ F p′ . Property 6 For c 6∈ F p, when c ∈ F pd , c ∈ F p′ ⇔ sup(c, D ∪ d) ≥ s. In this paper, an efficient incremental privacy preserving frequent itemset mining algorithm is proposed to address the updating problem in PPDM, when new transactions appended. In the algorithm, we make use of the distortion technique mentioned in [15], and some optimizations in [15]. Because the reconstruction procedure is cost, a k-itemset may be distorted to produce any of 2k combinations, in order to accurately reconstruct the support of the k-itemset, we need the counts of all these 2k combinations in the distorted database. Through the basic formula from set theory, the support of all combinations of the itemset can be computed efficiently [15]. In mining, for the frequent itemset, the support can be obtained in the file F p, but for the infrequent itemset not in the file F p, a new scan must be done to recomputed the support, decreasing the efficiency. In order to overcome this, a minor modification is proposed, when mining, the support of the frequent itemsets in the distorted database is also registered along with registering the support in the original database, conveniencing the incremental mining, improving the efficiency. For example, itemsets {A} and {B} are frequent, but {AB} is infrequent, in incremental mining, if we need compute the support of {AB} in the original database D, when the support counts of {A} and {B} in the distorted database D∗ are registered, the support of {AB} can be computed quickly through the basic formula from set theory [15] without scanning the database D∗ . In the following, an efficient incremental privacy preserving frequent itemset mining IPPFIM is shown

2. When sup(X, d) < s, itemset X is infrequent in incremental database d.

in Algorithm.1, the input F pD∗ denotes the corresponding frequent itemsets F p and their support in the distorted database D∗ . Algorithm 1: An efficient incremental privacy preserving frequent itemset mining algorithm IPPFIM. The objective of this algorithm is to process privacy preserving frequent itemset mining in an incremental updating environment. Input: D∗ , d∗ , F p, F pD∗ , minimum support s, distorted parameter p, q. Output: F p′ (Frequent itemset and the support in D ∪ d) Method: As Fig.2

(a) Making use of the Property 5, X is a infrequent itemset in the whole database file if X 6∈ F p. (b) Step 3 - 5 make use of Property 4, when sup(X, D ∪ d) ≥ s, put the itemset X into F p′ .

4

Experimental Results

In order to assess the performance of IPPFIM, experiments are conducted to compare its performance with that of EMASK [15], an efficient privacy preserving frequent itemset mining algorithm. Both algorithms are implemented using Cygwin with gcc 2.9.5. Our target platform is a Pentium4 1.6GHz processor, with 384MB memory, using a Maxtor IDE disk (7200rpms, 80GB). The operating system is Windows Xp with service pack 2. Benchmark data sets. The performance tests are performed on synthetic database benchmark, publicly available from IBM synthetic market-basket data generator [18]. The data sets, using the IBM data generator, mimic the transactions in a retailing environment. In the data sets, the meanings of the parameters are shown in Table.2. In the following experiments, we use the data set T25.I4.D1M.N1K as the original database.

D T I N

Figure 2. An efficient incremental privacy preserving frequent itemset mining.

When computing 1-itemset, itemset X is brought through scanning d∗ . When computing k-itemset (k > 1), X is brought through Ck+1 , generated by F p′k . In incremental mining, we use the aforesaid Property 3-6. 1. When sup(X, d) ≥ s, itemset X is frequent in the incremental database d. (a) Step 1 and 2 make use of the Property 3, put the frequent itemset X into F p′ . (b) Step 6 - 10 make use of the Property 6, computing the support of X in the original database, if the condition is satisfied, put it into F p′

Table 2. Parameters meanings Number of transactions Average size of the transactions Average size of the maximal potentially large itemsets Number of items

Performance Testing. We compare the performance of IPPFIM against EMASK for different data sets as Fig.3 and Fig.4. In the experiments, every databases are distorted with parameter p=0.5, q=0.97 as EMASK [15]. The first experiment is done on the original database T25.I4.D1M.N1K with an update database T25.I4.D25K.N1K. The execution time and performance ratio (execution time ratio) with various minimum support thresholds are evluated as Fig.3. Fig.3(a) shows that the execution time of IPPFIM is much less than the time of EMASK. When the setting on minimum support is decreased, the execution time of both algorithms increases, moreover, the increasing rates of both algorithms are dissimilar. As Fig.3(b), for the same data sets, the less minimum support is, the more the execution time ratio (EMASK/IPPFIM) is (The fluctuate in the Fig.3(b) is relate to the synthetic database, Fig.4(b) is the same.). For small support, IPPFIM is 10 to 20 times faster than EMASK. For larger support, it is less costly to re-run the mining

algorithm on the updated database since the number of large itemsets is relatively smaller. For example, when he support increases from 0.75% to 1.0%, the execution time ration decreases from 10.24 to 3. In general, the larger the increment is, the longer it would take to do the update operations. Also, the gain in speedup would slow down. Fig.4 shows the execution time and ratio of the two methods on the data set T25.I4.D1M.N1K with updates of 5K, 10K ,25K, 50K, 100k and 250K for the same minimal support. Just as Fig.4, for the same support, the efficiency is improved. When update size increases, the speed-up ratio decreases. For example, when the incremental databases increase from 50K to 250K, the execution time ratio decreases from 15.98 to 4.74. As the size increases in the update, the performance ratio will decrease, because IPPFIM adopts the incremental updating technique, it only need to compute a fraction of the candidate itemsets generated newly, which saves the time of scanning the original database compared to re-run the EMASK on the update database, eventually the performance of IPPFIM will not be under that of EMASK.

5

(a) Execution Time

Conclusion

In this paper, we have presented an efficient, incremental update algorithm IPPFIM for the maintenance of the frequent itemsets discovered by data mining in privacy preserving when a set of new transactions are added to the transaction database. Our algorithm strives to reduce the I/O requirements for updating the frequent itemsets. IPPFIM uses the information available from a previous mining to reduce the amount of work that has to be done to remove itemsets that no longer exist in the updated database, and to add new itemsets which were not in the set of old transactions but now exist in the updated database. The experiments on data sets show that IPPFIM achieves a better performance than re-running EMASK (an efficient privacy preserving frequent itemset mining algorithm) over the whole set of transactions. To extend IPPFIM algorithm by applying the strategies for mining evolving data streams with privacy preservation is under our current study.

Acknowledgements This paper was supported by the Natural Science Foundation of China (No. 60402010), Zhejiang Provincial Natural Science Foundation of China (Y105250) and the Science-Technology Progrom of Zhejiang Province of China (No. 2004C31098).

(b) Execution Time Ratio

Figure 3. Performance Comparison with Different Minimum Support

References [1] U. M. Fayyad, G. Piatesky-Shapiro, P. Smyth, and R. Uthurusamy. Advances in Konwledge Discovery and Data Mining. AAAI/MIT Press, 1996. [2] J. Han, M. Kamber. Data Mining: Concepts and Techniques. Morgan Kaufmann Publishers, August 2000. [3] The Economist. The end of Privacy. May 1st, 1999. pp. 15. [4] C. Clifton and D. Marks. Security and privacy implications of data mining. ACM SIGMOD WOrkshop on Research Issues on Data Mining and Knowledge Discovery, 1996. pp. 15-19. [5] K. Thearling. Data mining and privacy: a conflict in making. DS, November 1998.

[11] D. Agrawal and C. Aggarwal. On the Design and Quantification of Privacy Preserving Data Mining Algorithms. PODS 2001. pp. 247-255. [12] A. Evfimievski, R. Srikant, R. Agrawal and J. Gehrke. Privacy preserving mining of association rules. SIGKDD 2002. pp. 217-228. [13] S. Rizvi and J. Haritsa. Maintaining data privacy in association rule mining. VLDB 2002. pp. 682-693. [14] W. Du and Z. Zhan. Using Randomized Response Techniques for Privacy-Preserving Data Mining. SIGKDD 2003. pp. 505-510. (a) Execution Time

[15] S. Agrawal and J. Haritsa. On addressing efficiency concerns in privacy-preserving mining. DASFAA 2004. pp. 113-124. [16] A. C. Yao. Protocols for secure computations (Extended Abstract). FOCS 1982. pp. 160-164. [17] R. Agrawal, T. Imielinski and A. N. Swami. Mining association rules between sets of items in large databases. SIGMOD 1993. pp. 207-216. [18] R.Agrawal and R.Srikant. Fast Algorithms for Mining Association Rules. VLDB 1994. pp. 487-499.

(b) Execution Time Ratio

Figure 4. Performance Comparison with Different Incremental Database

[6] R. Agrawal and R. Srikant. Privacy-preserving data mining. SIGMOD 2000. pp. 439-450. [7] Y. Lindell and B. Pinkas. Privacy preserving data mining. Crypto 2000. pp. 36-54. [8] N. R. Adam and J. C. Wortmann. Security Control Methods for Statistical Databases: A Comparison Study. ACM Comput. Surv. 21(4), 1989. pp. 515-556. [9] C. Clifton, M. Kantarcioglu, and J. Vaidya. Defining Privacy For Data Mining. Proc. of the National Science Foundation Workshop on Next Generation Data Mining, 2002. pp. 126-133. [10] S. R. M. Oliveira and Osmar R. Zaiane. Toward Standardization in Privacy-Preserving Data Mining. DMSSP 2004 (In conjunction with SIGKDD 2004).