Expert Systems with Applications 37 (2010) 6182–6191

Contents lists available at ScienceDirect

Expert Systems with Applications journal homepage: www.elsevier.com/locate/eswa

DASA: Dissatisfaction-oriented Advertising based on Sentiment Analysis q Guang Qiu a,*, Xiaofei He b, Feng Zhang a, Yuan Shi a, Jiajun Bu a, Chun Chen a a b

Zhejiang Key Laboratory of Service Robot, College of Computer Science, Zhejiang University, Hangzhou 310027, China College of Computer Science, Zhejiang University, Hangzhou 310027, China

a r t i c l e

i n f o

Keywords: Contextual advertising Advertising keyword extraction Ad selection Sentiment analysis Syntactic parsing

a b s t r a c t Online advertising has become one of the major revenue sources of today’s Internet ecosystem. The main advertising channels used to distribute textual ads are sponsored search and contextual advertising. Here we consider the problem of contextual advertising, i.e. associating ads with a Web page. Most of previous work only focuses on topical relevance of ads whereas the consumer attitudes are ignored. In this paper, we propose a novel advertising strategy, called Dissatisfaction-oriented Advertising based on Sentiment Analysis (DASA), to simultaneously improve ad relevance and user experience. Specifically, by using syntactic parsing and sentiment dictionary, we propose a rule based approach to extract topic words of opinion sentences associated with negative sentiment, which are regarded as the advertising keywords. We also design a prototype system for product information submission for the sake of ad selection. We take into account the consumer attitudes and promote the competitors of those products with which the consumers are not satisfied. The experimental results on advertising keyword extraction and ad selection have demonstrated the effectiveness of the proposed approach. Ó 2010 Elsevier Ltd. All rights reserved.

1. Introduction The growth of the Internet signified a dawn of a new age for advertising. Compared with that on traditional media like newspaper and television, advertising on the Internet (online advertising) boasts advantages including wide coverage, low cost and so on. Driven by these unique advantages of online advertising and the rapid development of World Wide Web, advertisers have paid increasing attention to this novel and impressive advertising media. As mentioned in (Broder, Fontoura, Josifovski, & Riedel, 2007), the total Internet advertiser cost in US alone in 2006 is estimated at over 17 billion dollars with a growth rate of almost 20% year over year which means more than 29 billion dollars in 2009. The large investment in online advertising market provides main funding for the majority of Internet companies. For some Web sites, especially those for-profit non-transactional ones, revenue from advertising maintains not only current performance but also future development. Therefore, it seems that these advertising-relied Web sites should get ads published as many as possible to gain high revenue from advertisers. However, Bhargava et al. have show in their work (Bhargava & Feng, 2002) that advertising would create a disutility to Web users, consequently reducing sites’ market share and consumer-side revenue, further the profits from advertisers. Also, Mccoy et al. have drew a similar conclusion in q This work is supported by China National Key Technology R&D Program (2008BAH26B00). * Corresponding author. Tel.: +86 571 87953955. E-mail address: [email protected] (G. Qiu).

0957-4174/$ - see front matter Ó 2010 Elsevier Ltd. All rights reserved. doi:10.1016/j.eswa.2010.02.109

their investigation (Mccoy, Everard, Polak, & Galletta, 2007). In other words, from the perspective of consumers, advertising should be constrained or even boycotted. The tradeoff between financial revenue and market share triggers the emergence of relevant advertising to emphasize the relevance between ads and Web pages for the sake of consumers. Broder et al. categorize relevant advertising (textual ads) according to the distribution channels into sponsored search (SS) and contextual match (CM) (Broder et al., 2007). SS aims to place originating query driven ads on the result pages from a Web search engine. This kind of advertising is used by major Web search engines like Google, Yahoo! and Microsoft Live Search. CM aims to place the commercial ads in the content of generic Web pages. Some major Web search engines also provide such service and there are also many smaller players. Fig. 1 gives illustrative examples for these two types of advertising. The primary focus of this paper is on CM which is also referred to as contextual advertising. The dominant strategy of most of current sponsored search and contextual match is the keyword targeted marketing (RibeiroNeto, Cristo, Golgher, & Moura, 2005). Previous approaches to targeted/relevant advertising consider short-term and long-term interests, however suffer from privacy issues (Wang et al., 2002). In keyword targeted marketing, keywords are extracted from search queries (in the case of SS) or Web pages (in the case of CM) to match against those keywords associated with ads provided by advertisers. It would be important to note that the privacy issues can be avoided in this kind of strategy. The problem of placing relevant ads using the keyword targeted marketing strategy (mostly for contextual advertising) has

G. Qiu et al. / Expert Systems with Applications 37 (2010) 6182–6191

6183

Fig. 1. Examples of sponsored search and contextual match. (a) Yahoo! search results for query ‘‘Honda automobiles”, in which we can see an ad of ‘‘Honda Autos” on the right. (b) A snapshot of the home page of automotiveforums.com. There are ads provided by Google adsense on the left.

received a lot of attentions recently (Broder et al., 2007; Chakrabarti, Agarwal, & Josifovski, 2008; Lacerda et al., 2006; Ribeiro-Neto et al., 2005; Wu & Bolivar, 2008; Yih, Goodman, & Carvalho, 2006). These approaches only focus on topical relevance between ads and Web pages, which may not be optimal in some situations. For example, for a blog written by someone who is complaining about the safety of Honda Accord, the ads related to Honda Accord are good matches if we simply consider the topical relevance. However, the consumer reading this blog may have a negative feeling about Honda Accord and therefore may not click the ads. Thus, it might be desirable to place the ads of other automobiles with high safety standard. In this paper, we propose a novel ad targeting mechanism, called Dissatisfaction-oriented Advertising based on Sentiment Analysis (DASA), to address the above mentioned problem. Our basic idea is to combine traditional keyword relevance matching with sentiment analysis (also known as opinion mining). In fact, sentiment analysis of texts is a kind of non-topical text analysis technique which is orthogonal to the topical based technique (Esuli & Sebastiani, 2005). By sentiment analysis, we aim to discover the author’s underlying sentiment when he/she is writing the text, and eventually determine his/her attitude towards the mentioned

topics. This way, we are able to promote ads which not only relate to the topics of the Web pages but also meet the consumers’ potential information needs. Naturally, in the example given above, given the author’s negative attitude towards the safety of Honda Accord, we may consider promoting automobiles which have higher safety standard like Volvo. The rest of the paper is organized as follows. Section 2 discusses previous work on contextual advertising and sentiment analysis. In Section 3, we provide an overview of our DASA advertising strategy, along with some discussions on application issues. In Sections 4 and 5, we describe our approaches for advertising keyword extraction and ad placement in details, respectively. In Section 6, we describe our experimental methodology and data sets, and demonstrate experimental results with discussions. Finally, we provide some concluding remarks in Section 7.

2. Related work In this section, we provide a brief description of related work on contextual online advertising and sentiment analysis which is the key technique employed in our approach.

6184

G. Qiu et al. / Expert Systems with Applications 37 (2010) 6182–6191

2.1. Contextual online advertising Contextual advertising is a major type of online advertising, in which ads are placed on Web pages according to their content. Published literature on contextual advertising is still limited. One reason is that it is an emerging research area; in addition, commercial secrets also limit publications of research achievements. In existing researches, some reduce the problem to keyword matching (Wu & Bolivar, 2008; Yih et al., 2006), while others inspect relevance of the entire text on the Web page (Broder et al., 2007; Chakrabarti et al., 2008; Lacerda et al., 2006; Ribeiro-Neto et al., 2005). In the first line, Yih et al. (2006) emphasize the extraction of appropriate advertising keywords in contextual advertising and propose a classification based approach. That is, by using documents manually annotated with advertising keywords, they apply logistic regression to train a classifier to classify a keyword as an advertising keyword or not. The features used to represent keywords include linguistic features, html metadata, TFIDF, query log, etc. A similar approach is recently proposed by Wu and Bolivar (2008) which adopts linear regression to determine if a keyword is an advertising one. The features used in their work are similar to those in (Yih et al., 2006). It would be important to note that, in our work, we develop a novel strategy to extract advertising keywords in which no training process is required. In the second line, previous work on relevance measures between Web pages and ads can be roughly categorized into syntactic based and semantic based approaches. Ribeiro-Neto et al. provide a syntactic solution to ad placement based on the cosine similarity measuring between Web page and ad vectors (RibeiroNeto et al., 2005). They also propose a solution to solve the impedance problem between the vocabulary of ads and Web pages. In their follow-up work (Lacerda et al., 2006), they design a ranking strategy for displaying ads according to their relevance by effectively leveraging all individual features using genetic programming. They aim to optimize overall precision and minimize the number of misplacements. The semantic approach proposed in (Broder et al., 2007) stems from the linguistic phenomena of polysemy that a word has different meanings in different semantic contexts, which is contrary to the impedance problem in (Ribeiro-Neto et al., 2005). They use the proximity of the ad and page classes as the semantic score. Their experimental results show that semantic score is significant as to improving the ad quality. However, all previous approaches above focus only on the topical relevance and ignore the sentiment information between the lines. In this work, we propose a novel strategy which considers both the topic relevance and sentiment coherence between ads and Web pages. 2.2. Sentiment analysis Another work related to our approach is sentiment analysis. Current researches on sentiment analysis mainly focus on topic related issues (Hu & Liu, 2004; Lu & Zhai, 2008; Mei, Cai, Zhang, & Zhai, 2008; Mei & Ling, 2007) and sentiment classification (Ding & Liu, 2007; Hatzivassiloglou & McKeown, 1997; Pang & Lee, 2004; Turney, 2002; Turney & Littman, 2003). Here, we only summarize some of previous approaches on topic modeling and word sentiment classification, which are the most closely related to our work. In the work of (Mei & Ling, 2007), Mei et al. propose a novel probabilistic model called Topic-Sentiment Mixture to capture the mixture of topics and sentiment simultaneously. A topic model and two sentiment models are defined in their work based on language models to model the probabilistic distribution of words in different topics and sentiment polarities. The work by Lu and Zhai

(2008) studies the problem of how to integrate a well-written expert review about an arbitrary topic with many ordinary opinions expressed in a text collection. They propose a semi-supervised topic modeling approach to cast the expert review as a prior in the probabilistic topic model PLSA and fit the model to the text collection. Hatzivassiloglou et al. are the earliest to tackle the problem of determining the semantic orientation of sentiment words (Hatzivassiloglou & McKeown, 1997). Their proposed method tries to predict the orientation of subjective adjectives by analyzing pairs of adjectives extracted from a large unlabeled document set. The underlying intuition is that the act of conjoining adjectives subjects to linguistic constraints on the orientation of the adjective involved. Turney et al. adopt a different methodology which requires little linguistic knowledge (Turney & Littman, 2003). They first define two minimal sets of seed terms as descriptive of the categories Positive Sp and Negative Sn. Then they compute the point-wise mutual information (PMI) of the target term t with each seed term ti as a measure of their semantic association. Sentiment analysis of sentences is a major building block of our proposed approach. The specific task here is to extract the topic words of sentences that the contained sentiment words are about. In our work, we propose a rule based approach to accomplish this extraction task. 3. Overview of DASA advertising strategy Our contextual advertising strategy DASA works as follows: (1) Extract topic words as the candidate topics that ads may be about from the Web pages (the traditional way to promote ads); (2) Determine consumers’ sentiment on the extracted topic words and select those towards which consumers have negative sentiment as the final advertising keywords; (3) Match the advertising keywords with appropriate ads. In other words, an ad is considered to be relevant to a Web page only if it is associated with the topic words towards which consumers have negative attitudes. The most crucial steps of our approach are topic word extraction and its sentiment classification, which eventually lead to advertising keyword selection. Fig. 2 shows the architecture of DASA. 3.1. Application issues There are several application issues we have to address before the detailed elaboration of our approach. Since our advertising approach depends on the analysis of consumers’ attitudes, the first issue is that whether there exists large scale Web environment containing plenty of personal opinions where ads can be placed. The answer to this problem is User Generated Content (UGC), which is also known as Consumer Generated Media (CGM), referring to various kinds of media content produced by consumers. It has been characterized as ‘‘conversational media” which is a two-way process encouraging the publishing of users’ own content and commenting on others’ (http://www.en.wikipedia.org/wiki/User-generated content). Typical applications include forums and blogs, which have already been prevailing Web applications on the Internet (http://www.businessweek.com/globalbiz/ content/jan2007/gb20070109559223.htm). Therefore, UGC is ideal environment for our advertising strategy. The second issue is the targeted audience of our advertising. In UGC, there are two types of users participating in Web activities: writers and readers of the content. When a writer is complaining

G. Qiu et al. / Expert Systems with Applications 37 (2010) 6182–6191

6185

Fig. 2. Illustration of the workflow of DASA. TWn means the topic word extracted from Web pages by the topic word extraction technique. AKk is the final advertising keyword selected from the topic words by sentiment classification technique. These two steps form the advertising keyword extraction phase as described by the dashed rectangle.

about some product in his/her article, our advertising scheme tends to place ads which are about rival products. This can potentially help the writer find a new product that satisfies his/her needs. Besides, this scheme also provides alternatives to the readers who have similar opinions with the writer. When there are no personal comments or only positive opinions contained in the article, traditional advertising strategy can be applied. 4. Advertising keyword extraction As stated above, we reduce the task of advertising keyword extraction to topic extraction and sentiment classification subtasks, i.e. to identify topics towards which consumers have negative attitudes. Often the consumers are more concerned about the topics on which they have personal comments. Therefore, we only consider the sentences which contain sentiment words. And these sentences are referred to as opinion sentences throughout this paper. Our extraction approach can be summarized in three steps:

mar, the relation between two words A and B can be described as A(or B) depends on B(or A). There are also other complicated relations between A and B. For example, A depends on another word C while C depends on B. We define two categories to summarize all the relations, which are also illustrated in Fig. 3. Definition 1. Direct relation (DR): A direct relation means one word depends on the other one directly or they both depend on a third word directly. In the context of sentiment word S and topic word T, this relation means that S(or T) depends on T(or S), or S and T both depend on another word H. Definition 2. Indirect relation (IDR): An indirect relation means one word depends on the other one through another word or they both depend on a third word indirectly. In the context of sentiment word S and topic word T, this relation means that S(or T) depends on T(or S) through another word H, or S(or T) depends on a word H1 which depends on the same word H which T(or S) depends on.

(1) Identify the opinion sentences in the texts; (2) Extract the topics in the opinion sentences given the labeled sentiment words; (3) Select advertising keywords from the extracted topic words.

4.1. Identify opinion sentences In this step, as we define opinion sentences as sentences that contain sentiment words, the identification task can be tackled by discriminating if any of the contained words are sentiment ones. In our work, we employ a dictionary based approach to judge if a word is a sentiment word. The dictionary used in our work is the General Inquirer (http://www.wjh.harvard.edu/inquirer/) which consists of 1892 negative words such as ‘‘abandon”, ‘‘abuse”, ‘‘poor”, and 1563 positive ones such as ‘‘able”, ‘‘appreciate”, ‘‘good”. 4.2. Extract topic words of opinion sentences using rules Given opinion sentences identified from the texts and the contained sentiment words as well, topic words associated with the sentiment words can be extracted. Here we propose our solution using pre-set rules based on syntactic parsing of sentences. We take advantage of Minipar (http://www.cs.ualberta.ca/lindek/minipar.htm) as the syntactic parsing tool, which uses the dependency grammar (Tesniere, 1959). After parsing, words in a sentence are assigned with corresponding syntactic categories and linked to each other by certain relations. In dependency gram-

Fig. 3. Different relations between sentiment word S and topic word T. (a–c) are the three kinds of direct relations. (d–g) illustrate the four kinds of indirect relations.

6186

G. Qiu et al. / Expert Systems with Applications 37 (2010) 6182–6191

Note that in IDR, there may be more than one H or H1 in the dependency path. In our work, we only consider the one word situation as compact dependency relations would guarantee the association between S and T with high certainty. Given the seven relations between S and T, we design our rules which are able to capture the relations as well as the word syntactic categories. The specific rules are listed in Table 1. In our rule representations, S(T, H1, H)-Category and S(T, H1, H)-Relation denote the syntactic category and relation of the word S(T, H1, H), respectively. We also assign priority Pi (i = 1,...,7) to each rule in case that more than one rule can be applied to the extraction task. In the seven rules, R1 is assigned with the highest priority P1 and R2 to R7 get lower priorities in descending order. We set the priority for each rule by considering the certainty of the association between S and T that the corresponding relation can guarantee. For the sake of simplicity, we only consider the topological structures implied by these rules, ignoring most of the detailed information of relation and syntactic category. One constraint is that the category of topic word is limited to N (assigned by Minipar) which means the part-of-speech of the topic word should be noun. Note that Minipar parses all pronouns and numbers also as N. We do not consider the noun phrases in our current work, since the state-of-the-art phrase identification is still not accurate enough. There are also some exceptions we need to exclude, such as the conj relation in R1 and R2 which means the two related words are of equal syntactic role in the sentence. Using these rules and their priorities, we are now able to extract the topic word in the opinion sentence. Take R1 for example, given the sentiment word S parsed as S-Category, word with category TCategory and relation S-Relation with S is taken as the topic word T of S with priority P1. The algorithm shown in Fig. 4 elaborates how to extract topic word T using these rules. 4.3. Select advertising keywords Clearly not all the above extracted topic words can be taken as advertising keywords directly as explained in the introduction section. We should identify users’ attitudes towards the topics and then select appropriate topic words as advertising keywords. From the perspective of sentiment analysis, the implied sentiment of the topic word is classified to be positive or negative based on the polarity of the related sentiment word. As stated in the very beginning, advertising on topics towards which consumers have negative attitudes is more helpful and targeted to users. Therefore, we take topic words which correspond to negative sentiment words as the advertising keywords. 5. Appropriate ad placement After the advertising keywords are obtained through topic extraction and sentiment classification, we can select appropriate

Table 1 Rules defined in our work. S(T, H1, H)-Category and S(T, H1, H)-Relation mean the syntactic category and relation of the word S(T, H1, H), respectively. We use these rules to extract topic words in opinion sentences. No. Rules R1 R2 R3 R4 R5 R6 R7

S-Category < S-Relation > T-Category T-Category < T-Relation > S-Category S-Category < S-Relation > H-Category < T-Relation > T-Category S-Category < S-Relation-T-Crelation > T-Category T-Category < T-Relation-S-Crelation > S-Category S-Category < S-Relation-H1-Relation > H-Category < T-Relation > T-Category T-Category < T-Relation-H1-Relation > H-Category < S-Relation > S-Category

Fig. 4. Algorithm for extracting topic word.

ads that are relevant to these keywords and simultaneously meet consumers’ needs. As stated above, our DASA ad placement strategy only concerns topics towards which consumers have negative attitudes. In real world applications, advertising keywords can be brand names (such as BMW in the sentence ‘‘I do not like BMW”), as well as composed features (such as safety in the sentence ‘‘I am not satisfied with the safety of Honda Accord”). Therefore, different strategies should be adopted to select corresponding ads for these two situations. In our work, we propose to promote rivals’ products when the advertising keywords are brand names and select products which have excellent performance in certain features when the advertising keywords are feature names. In case of multi-candidates, WTP (willingness to pay) (Feng, Bhargava, & Pennock, 2003) can be employed to determine which one to place. In order to select appropriate brands efficiently for the two situations, we need to know which two brands are rivals and which features a brand is good at. In traditional advertising systems, an ad is always represented by its bid phrases, self-explanatory title, short description and URL of the target Web page. These pieces of information can only provide a general description about the product. However, we argue that ad description should be more targeted and expressive in terms of its advantages over its rivals so that the ad-network can be aware of products’ advantages and consequently select appropriate ads to meet consumers’ needs. In our work, we design a browser based system to construct a knowledge base for different products. Fig. 5 is a snapshot of the system. Any manufactories or dealers can provide their product information through the system. Currently, we take automobile domain as an example. Note that our approach can also be applied other domains. The product information is stored in XML files as shown in Fig. 6. The element brand in the node AUTO is the brand name of the auto, such as BMW, BENZ, etc. The element name in the node MODEL is the model name, such as Z, X serials of BMW. The node RIVAL stores brand names of rivals and node ADVANTAGE_FEATURE contains the outstanding features of the model. An example is that 750li of BMW may be labeled as rival of S600 of BENZ. Another example is that Volvo has excellent performance in safety. Thus, we can promote ads of S600 when the consumer is complaining about 750li and place ads of Volvo when he is unsatisfied with the safety feature of Honda Accord.

6187

G. Qiu et al. / Expert Systems with Applications 37 (2010) 6182–6191

Fig. 5. Snapshot of our system for product information submission. Manufactories or dealers can provide advantageous feature information and rivals of their products through this system.

D#2 for advertising keyword evaluation. Table 2 shows the detailed information of each data set. In D#1, the average length of the 3783 opinion sentences is 21 words. All these sentences are labeled manually with the topic words. We develop a tool for volunteers to do the labeling with the instruction as follows: Fig. 6. XML format for product information storage.

6. Experiments and results Three experiments are conducted to evaluate the performance of our approaches to topic word and advertising keyword extraction and ad placement. We begin with a description of the evaluation corpus. 6.1. Evaluation corpus Since our advertising approach considers not only topical relevance but also consumers’ opinions, we use Web forums as the evaluation corpus. Particularly, we choose the popular auto forum automotiveforums.com as the source data in our experiments. In the forums, the users’ posts are organized as one thread if they all respond to the same post. We crawl and store these posts in unit of threads, i.e. store all the posts of one thread in a single file. We call this file an article hereafter. Let D#0 denote the set of articles. Another two data sets are constructed from D#0 for the first two experiments respectively, D#1 for topic word evaluation and

Suppose you are the writer of the sentence, given the sentiment word, please figure out in the sentence what is the topic word that the sentiment word related to. For example, in the sentence ”The z3 was awful and z4 is not much better”, the corresponding topic word for ”awful” should be ”z3” and that for ”better” should be ”z4”.

We collect the articles which contain the 3783 opinion sentences and the final collection D#2 consists of 647 articles. For advertising keyword labeling, we divide the 647 articles into 10 groups, and then request 10 annotators to label the keywords in articles whose corresponding ads they consider appropriate (”appropriate” means not only relevant but also not annoying). These keywords are then taken as the correct advertising keywords for the articles.

Table 2 Statistical information for the three data sets for D#0, D#1 and D#2. D#0

D#1

D#2

128,425 articles

3783 opinion sentences

647 articles

6188

G. Qiu et al. / Expert Systems with Applications 37 (2010) 6182–6191

6.2. Experiments on topic word extraction Topic word extraction is the key to the extraction of advertising keywords, therefore we conduct separate experiments on it. We begin with the evaluation strategy and criteria.

6.2.1. Evaluation strategy and criteria To the best of our knowledge, there is no previous work on sentiment based topic word extraction in the research area of sentiment analysis. Therefore, in our experiments, we compare our approach with a naive distance based approach that extracts the nearest noun of the sentiment word as the topic word. We use the accuracy as the evaluation criteria. The accuracy is defined to be the percentage of the sentences whose automatically extracted topic words (either by our approach or the distance based approach) are the same as the user pre-labeled ones.

6.2.2. Results and discussions Table 3 shows the results of the 3783 opinion sentences in D#1 by using our rule based approach, the distance based approach and two random approaches: Random-All and Random-Noun. The Random-All approach selects any one word from the sentence randomly as the topic word. Thus its corresponding accuracy value is calculated as the inverse of the average sentence length (21 words as shown before). Similarly, the Random-Noun approach is to select any noun in the sentence as the topic word and thus its accuracy value is the inverse of the average number of words parsed as N contained in the sentence (The number is 10.25. This number is a little bit high because Minipar parses all pronouns and numbers in addition to actual nouns as N and meanwhile some parsing errors also exist.). We take the -Noun case into account as both rule and distance based approaches impose the noun constraint. As can be seen, both our proposed approach and the distance based approach significantly outperform random approaches. Particularly, our approach obtains 55% accuracy, while the distance based approach obtains 42% accuracy. There is almost 31% performance improvement. This shows that our approach is effective in extracting the topic word. Moreover, this result reveals the fact that the relationship between sentiment word and topic word

Table 3 Performance results of our rule based approach compared with the distance based approach and random approaches. The accuracies of random approaches are calculated as the inverse of average sentence length (for Random-All) or the inverse of average number of contained words parsed as N (for Random-Noun). The CorrectSent column is the number of sentences whose topic words are extracted correctly. Approaches

CorrectSent

Accuracy

Rule Distance Random-Noun Random-All

2063 1581 – –

0.55 0.42 0.10 0.05

can be better characterized by syntactic structure rather than distance. Table 4 gives some examples of our rule based approach compared with other approaches.

6.3. Experiments on advertising keyword extraction In this section, we first describe our evaluation criteria and strategy, and then show the results and discussions.

6.3.1. Evaluation criteria and strategy In order to evaluate the performance of different advertising keyword extraction methods, we consider the quality of advertising keywords from two aspects: the commercial value (CV) and traditional measurements (precision, recall and F1-measure) which are calculated based on the manually labeled results. The commercial value of extracted keywords in an article is defined as the percentage of extracted keywords which are of real value in some particular online advertising environment, like Google’s AdWords and Yahoo’s Sponsored Search. It measures the quality of keywords from the perspective of advertisers while traditional measurements emphasize the consumers’ interests. For comparison, we implement another keyword extraction approach based on the famous keyword weighting schema TFIDF, which is regarded as the most useful feature for advertising keyword extraction (Yih et al., 2006). It would be important to note that, however, our proposed approach can be easily incorporated into previous approaches, since most of them only consider topical relevance while our approach considers consumers’ attitudes. Our aim in the experiments is to show the effectiveness of our approach instead of showing superior performance over sophisticated previous approaches. Therefore, a comparison to a typical feature in advertising keyword extraction is enough. For our rule based approach, the advertising keywords of an article are composed of those of the opinion sentences in the article. All these keywords are not weighted. For the TFIDF based approach, we remove the stop words and select the top 10 keywords with the highest TFIDF values as the advertising keywords. The average number of the labeled keywords in an article is 2.76 and that of those extracted by our rule based approach is 2.26. One important issue in our evaluation strategy is about the recognition of the commercial value of a word. We argue that a word is commercially valuable if and only if there is at least one advertiser that bids on it. Particularly, in order to quantitatively evaluate it, we submit a keyword to some search engine and check if there is any ad shown in the result page. If there is ad shown, this keyword is considered commercially valuable. In current experiments, we make use of Google’s search engine and its AdWords service. Fig. 7 gives two search examples (‘‘shoes” and ‘‘black”) on Google. From the results, we can see that there are ads in the search result page of ‘‘shoes” but no ads for ‘‘black”. So the keyword ‘‘shoes” is considered commercially valuable while ‘‘black” is not. This conclusion is also consistent to common sense.

Table 4 Some examples for different approaches. The Sentiment column shows the sentiment word contained in the sentence. The Nouns column shows the words which are parsed as N by Minipar in the sentence. The Labeled column shows manually labeled topic words. The Rule and Distance columns show the extracted topic words by rule and distance based approaches, respectively. Sentences

Sentiment

Nouns

Labeled

Rule

Distance

So the next thing is, does anyone know where I could get a cheap dealers manual for the car? The coil tower on these vehicles are notorious for corrosion. But in the end, a nice turbo setup would cost approx.

Cheap

Thing, anyone, where, manual, dealers, car Tower, coil, vehicles, corrosion End, setup, turbo, approx

Manual

Manual

Dealers

Tower Setup

Tower Setup

Vehicles Approx

Notorious Cost

G. Qiu et al. / Expert Systems with Applications 37 (2010) 6182–6191

6189

Fig. 7. Google search results for words ‘‘shoes” and ‘‘black”. As can be seen, there are ads in the result page of ‘‘shoes” but no ads for ‘‘black”. This means word ‘‘shoes” has higher commercial value than ‘‘black”.

6.3.2. Results and discussions In our experiments, we report CV, precision, recall and F1-measure of the top 10 selected keywords as the overall results for the TFIDF based approach. Considering that the average number of advertising keywords manually labeled and extracted by rule based approach is close to three, we also evaluate the performance of TFIDF based approach for the top three selected keywords. Table 5 lists all the results of different approaches. From the table we can see that our approach outperforms the TFIDF based approach (both TFIDF-Average and TFIDF-Top3) in all the four measurements, except the precision in TFIDF-Top3. The improvement of CV (17.3% over TFIDF-Average and 8.6% over TFIDF-Top3) means that the advertising keywords extracted by

our approach are more valuable from the perspective of advertisers and thus ensures these keywords would be related with ads instead of being abandoned because of being meaningless for advertisers. The superior performance of our approach in traditional measurements precision, recall and F1-measure shows that our approach can extract more advertising keywords with high accuracy than other approaches. Note that, the F1-measure can be essentially taken as the targeted-ness of the keywords from the view of consumers as it is a combination of precision and recall. Usually the more targeted the advertising keywords are, the higher revenue the advertisers will obtain. The reason is that with the improvement in the targeted-ness of keywords, ad relevance

6190

G. Qiu et al. / Expert Systems with Applications 37 (2010) 6182–6191

Table 5 Results of CV, precision, recall and F1-measure for our rule based approach and the TFIDF based approach. For TFIDF, we consider two different variations TFIDF-Average and TFIDF-Top3 which take the average value of top 10 and top 3 keywords, respectively. Approaches

CV

Precision

Recall

F1

Rule TFIDF-Average TFIDF-Top3

0.265 0.226 0.244

0.101 0.092 0.104

0.375 0.211 0.145

0.159 0.128 0.121

increases, therefore consumers will get less disturbed by ads and thus will remain well or even better impression on advertising Web sites. Finally, the market share would not be ill affected a lot by ads and advertisers and Web sites will be likely to get more profits. Therefore, F1-measure is more important than the other two measurements. In the exception of the results, although our approach is a little inferior in precision compared with TFIDFTop3, we gain 31.4% improvement in the F1-measure, which means our approach still outperforms TFIDF-Top3. Note that all the values in our measurements are relatively low. However, in online advertising-like applications, an improvement of X% in ad matching can lead to an improvement of X% in the end result unlike many other tasks where the effect of performance enhancement on the end result is not linear (Mishne & Rijke, 2006). Therefore, regarding the magnitude of online advertising market, a tiny improvement would lead to a considerable increase in the final revenue for advertisers. 6.4. Experiments on ad selection The above experiments have examined the performance of our approach for advertising keyword extraction, which is a key step towards ad placement. We now conduct experiments on selecting appropriate ads. Since we do not have the corpus of real world ads, we represent our recommendations as descriptions such as ”autos those perform excellent in safety”. As described before, there are two kinds of ads that would be promoted to consumers as they may complain on a certain brand or feature. For simplicity, we focus only on features in our current experiments. We compare DASA with the naive strategy that extracts advertising keywords in the same way but does not consider the atti-

tudes of consumers, i.e. take all the topic words as the advertising keywords (TE-ALL). Volunteers are requested to make the judgment on the recommended ads (the ad descriptions indeed). In our experiments, we construct a prototype system in which volunteers provide texts commenting on some products (only on features of autos in current work) and then get two lists of ads returned by DASA and TE-ALL. Volunteers are unaware of which list is the result of which approach. They are then asked to determine which one they think is better to meet their needs with less annoying. Fig. 8 shows the comparison between DASA and TE-ALL. The Yaxis is the number of times that volunteers regard the results of an approach are better than those of the other one. An obvious observation from the figure is that our approach (DASA) outperforms TEALL. For 35 times, volunteers judge that the results of our approach are better than those of TE-ALL, while the contrast is 22 times. The results show that it is reasonable to take consumers’ attitudes into account.

7. Conclusions Targeted advertising is of great importance for Internet companies to gain revenue from both advertisers and consumers. Previous approaches focus only on the topical relevance while the consumers’ attitudes are ignored. These approaches fail to meet the actual needs of consumers especially when they may have negative attitudes towards the mentioned topics. In this paper, we take the attitudes of consumers into account and propose a novel advertising strategy DASA to promote ads according to what they are unsatisfied with. We use pre-set rule based sentiment analysis techniques to tackle topic word extraction and consumers’ attitude identification in advertising keyword extraction. We also design a prototype system for users to provide product information for the sake of ad selection. For evaluation, we conduct separate experiments to measure the quality of topic words, advertising keywords and placed ads. Specifically, we propose to use commercial value to measure the quality of advertising keywords from the perspective of advertisers. Experiments show that our strategy outperforms the TFIDF based approach in advertising keyword extraction and the naive approach which does not consider the attitudes of consumers in ad selection. The results indicate that it is reasonable to take the attitudes of consumers into consideration. References

Fig. 8. Comparison results of our approach DASA and TE-ALL which does not consider the attitudes of consumers.

Bhargava, H. K., & Feng, J. (2002). Paid placement strategies for internet search engine. In Proc. 2002 Int. WWW Conf. (WWW’02) (pp. 117–123). Broder, A., Fontoura, M., Josifovski, V., & Riedel, L. (2007). A semantic approach to contextual advertising. In Proc. 2007 Int. Conf. on research and development in information retrieval (SIGIR’07), 2007 (pp. 559–566). Chakrabarti, D., Agarwal, D., & Josifovski, V. (2008). Contextual advertising by combining relevance with click feedback. In Proc. 2008 Int. WWW Conf. (WWW’08) (pp. 417–426). Ding, X., & Liu, B. (2007). The utility of linguistic rules in opinion mining. In Proc. 2007 Int. Conf. on research and development in information retrieval (SIGIR’07) (pp. 811–812). Esuli, A., & Sebastiani, F. (2005). Determining the semantic orientation of terms through gloss classification. In Proc. 14th ACM Int. Conf. on information and knowledge management (CIKM’05) (pp. 617–624). Feng, J., Bhargava, H. K., & Pennock, D. (2003). Comparison of allocation rules for paid placement advertising in search engines. In Proc. 5th Int. Conf. on electronic commerce (ICEC’03) (pp. 294–299). Hatzivassiloglou, V., & McKeown, K. R. (1997). Predicting the semantic orientation of adjectives. In Proc. 1997 ACL (ACL’04). Hu, M., & Liu, B. (2004). Mining and summarizing customer reviews. In Proc. 2004 Int. Conf. knowledge discovery and data mining (KDD’04) (pp. 168–177). Lacerda, A., Cristo, M., Goncalves, M., Fan, W., Ziviani, N., & Ribeiro-Neto, B. (2006). Learning to advertise. In Proc. 2006 Int. Conf. on research and development in information retrieval (SIGIR’06) (pp. 549–556). Lu, Y., & Zhai, C. (2008). Opinion integration through semi-supervised topic modeling. In Proc. 2008 int. WWW Conf. (WWW’08) (pp. 121–130).

G. Qiu et al. / Expert Systems with Applications 37 (2010) 6182–6191 Mccoy, S., Everard, A., Polak, P., & Galletta, D. (2007). The effects of online advertising. Communications of the ACM, 50(3), 84–88. Mei, Q., Cai, D., Zhang, D., & Zhai, C. (2008). Topic modeling with network regularization. In Proc. 2008 Int. WWW Conf. (WWW’08) (pp. 101–110). Mei, Q., & Ling, X. (2007). Topic sentiment mixture: Modeling facets and opinions in weblogs. In Proc. 2007 Int. WWW Conf. (WWW’07) (pp. 161–175). Mishne, G., & Rijke, M. (2006). Language model mixtures for contextual ad placement in personal blogs. In Proc. 5th Int. Conf. NLP (FinTAL). Pang, B., & Lee, L. (2004). A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proc. 42nd Ann. meeting of the association for computational linguistics (ACL’04) (pp. 271–278). Ribeiro-Neto, B., Cristo, M., Golgher, P., & Moura, E. (2005). Impedance coupling in content-targeted advertising. In Proc. 2005 Int. Conf. on research and development in information retrieval (SIGIR’05) (pp. 496–503).

6191

Tesniere, L. (1959). Elements de syntaxe structurale. Paris: Librairie C. Klincksieck. Turney, P. (2002). Thumbs up or thumbs down? Semantic orientation applied to unsupervised classification of reviews. In Proc. 40th Ann. meeting of the association for computational linguistics (ACL’02) (pp. 417–424). Turney, P. D., & Littman, M. L. (2003). Measuring praise and criticism: Inference of semantic orientation from association. ACM Transactions on Information Systems, 21(4), 315–346. Wang, C., Zhang, P., Choi, R., & D’Eredita, M. (2002). Understanding consumers attitude toward advertising. In Proc. 8th Americas Conf. on information system (pp. 1143–1148). Wu X., & Bolivar, A. (2008). Keyword extraction for contextual advertisement. In Proc. 2008 Int. WWW Conf. (WWW’08) (pp. 1195–1196). Yih, W., Goodman, J., & Carvalho, V. R. (2006). Finding advertising keywords on web pages. In Proc. 2006 Int. WWW Conf. (WWW’06) (pp. 213–222).