Machine Learning Techniques for Detecting Identifying Linguistic Patterns in News Media

,


Introduction
An author's language patterns influence how readers will perceive them and their message.For example, psychological and linguistic studies demonstrate how subconscious 'asymmetries' in word selection can reveal attributes of a speaker's identity and viewpoints (Hippel, Sekaquaptewa, and Vargas, 1997;Porter, Rheinschmidt-Same, and Richeson, 2015;Vaux et al., 2003).In addition to these unintentional phenomena, speakers may also consciously introduce bias while making an argument, influencing opinion through deliberately selected modifiers or framing (de Bruijn and de Vries, 2018).For instance, a defender of a welfare program may call its supporting taxes 'a small necessary burden' while their opponent might attack the same as 'wasteful and stifling.'This study's model examines these characteristic linguistic markers to detect patterns separating each agency from the rest, highlighting what those markings say both about news organizations and the machine learning-rich platforms disseminating their work.

Prior work
Related work in media studies, machine learning, and psychology inform this paper's modeling.To begin, prior research scores news publishers' audience ideological leaning and examine how trust in agencies differs demographically and politically (Mitchell et al., 2014;Knight Foundation, 2018).Bridging audience to content, earlier research also finds consumption bias towards material that agrees with a reader's pre-existing opinions (Knobloch-Westerwick and Meng, 2009).In addition to investigation of audience behavior, previously published text mining techniques uncover coverage bias towards political entities within the context of specific events and infer political orientation from social media posts (Fortuny et al., 2012;Chang, Chiu, and Hsu, 2017).Meanwhile, prior Bayesian analysis and more recent neural network based approaches can determine authorship of political or scientific writing (Mosteller and Wallace, 1964;Hitschler, Berg, and Rehbein, 2017).Finally, as this study earlier references, psychological human studies also show that perception of subtle patterns (like linguistic inter-group biases) allow audiences to infer attributes of the speaker's identity and beliefs (Porter, Rheinschmidt-Same, and Richeson, 2015;Hippel, Sekaquaptewa, and Vargas, 1997).This study extends this prior work by learning linguistic patterns that identify a news agency across a large body of work despite variations in authorship and subject matter.

Hypotheses
This study uses the following three hypotheses to investigate news media linguistic patterns: • Given article titles, this paper's method can predict publishing agency better than chance.
• In comparison to using titles which are often very short, this study's model can better predict the originating agency using article descriptions.
• Using an example set of topics, this method's performance increases when filtering for articles discussing a common subject by isolating different treatment of similar issues.

Methods and Materials
A probabilistic predictor for publishing agency investigates journalistic patterns by learning from approximately 49,000 articles to generate scores from 0 to 1 for each of the ten agencies examined.In this neural network-based classifier, 0 represents low likelihood that an article came from a particular agency and 1 represents high likelihood (Lan, 2017).The agency of highest score for an article becomes the model's overall prediction.

Data
A program run daily uses BeautifulSoup and Requests to capture public Really Simple Syndication ('RSS') feeds from the BBC News ('BBC'), Breitbart, CNN, Daily Mail, Drudge Report ('Drudge'), Fox News ('Fox'), New York Times ('NYT'), NPR, Vox, and Wall Street Journal ('WSJ') (Richardson, 2019;Reitz, 2018).RSS feeds are built for machine parsing and, among many uses, inform search engines of new content availability (Slegg, 2015).This ideologically broad sample of news sources are persisted into a SQLite database with Pandas and Numpy providing data manipulation (SQLite, 2019;Pandas, 2019;Numpy, 2019).The code for gathering these data are released under a permissive open source license (Pottinger, 2019a).The resulting 48,997 unique articles from January 29, 2019 to May 27, 2019 are divided into training, validation, and test sets of 80%, 10%, and 10% respectively.This study publishes its dataset under a creative commons license (Pottinger, 2019d;Pottinger, 2019h).Note that article title and description come from the news agency itself with the later often containing article snippets or matching preview text on the source's website.Additional information is available at the dataset's webpage (Pottinger, 2019h).

Abuse and standards
These data are collected in compliance with the Robots Exclusion Protocol by observing robots.txtfiles, industry-standard instructions which convey to systems like this crawler what data is or is not permissible to automatically process (Koster, 1996).This helps prevent this study from abusing services while building an index for its demonstrational search engine.

Neural network architecture
This study compares feed-forward (FF) and Long-Short Term Memory (LSTM) neural networks (Hochreiter and Schmidhuber, 1997).FF uses an input boolean vector encoding if a word is found within a document or not as described in Formula 1.These one dimensional vectors are processed through two fully connected 'dense' layers of variable length (using ReLUs) and a final softmax layer for output (Agarap, 2018;Lan, 2017).Meanwhile, the LSTM architecture features an integer encoding of the input words as a sequence where each unique word from the corpus has a unique integer index to which it is converted.These one dimensional vectors are fed into an encoding layer converting those word indices into a one-hot form before forwarding them to a variable number of LSTM units whose output are processed through a final softmax layer.
Formula 1: Occurrence Vector Formulation Though both offer advantages, LSTMs often better fit text-based tasks as the recurrent structure can take advantage of the 'ordered' nature of language whereas FF networks treat text as unordered tokens (Brownlee, 2019).However, prior work does see FF outperform LSTM in some circumstances (Miller, 2018).To contrast these structures, Figure 1 summarizes both architectures.

Regularization
To prevent overfitting where performance in training exceeds that in a hidden validation set, regularization techniques in both FF and LSTM improve predictive generalization.First, this paper varies L2 activation regularization within interior layers of both network types (dense layers and LSTM units) before also varying dropout rates within the same layers (Ng, 2004;Srivastava et al., 2014).
Additionally, as uncommon words may uniquely identify articles, the number of words included in the encoding provide a form of regularization.Therefore, the frequency of a word is calculated across the entire dataset and only the 'top n' words are included in the encoding (with multiple values of n explored).

Other preprocessing
Some articles 'leak' information about the publisher, preventing actual extraction of linguistic patterns.This includes, for example, hyperlinks to the agency's website or a publisher referring to itself (an article mentioning it is part of a Vox series reveals its publisher to be Vox).With this in mind, HTML elements are converted to the text visible to the user and the name of the publisher is removed from the title and description text before training.Some agencyspecific features that are not part of the 'body' of the title or description such as agency-specific advertising text are also hidden from the neural network.

Voice similarity
Incorrectly classified articles may indicate that two news agencies exhibit similar ways of speaking, helping explain model behavior and the nature of media linguistic patterns.For example, a large number of articles from the BBC classified as published by NPR may indicate that the two share similar characteristics.With that motivation in mind, this study offers bidirectional similarity calculations through a Jaccard Index-like measure (Jaccard, 1912).In Formula 2, s1 refers to the set of articles from source 1, s2 refers to the set of articles from source 2, classif ied(s1, s2) refers to the set of articles incorrectly classified as being from source 2 while actually being from source 1, and similarity(s1, s2) refers to the bidirectional similarity between s1 and s2.A larger similarity(s1, s2) could indicate a greater overlap in patterns.

Comparison to prior work
In order to discuss findings within the context of prior work, this paper presents additional formulations for comparing results to other pre-existing measures.

Formulation for comparison with Pew
Pew audience ideology scores provide a conservative to liberal scale for news agencies (Mitchell et al., 2014;Knobloch-Westerwick and Meng, 2009).However, Formula 2's method presents some issues when comparing Pew's work to this model.First, Formula 2 does not provide a one dimensional scale, requiring this study to measure distances from a reference agency in this paper's model to all other outlets to create a single axis of comparison.Breitbart in particular provides a convenient reference as it is the most conservative source this paper examines according to Pew (Mitchell et al., 2014).Second, values generated using Formula 2 will be close to 0 if s1! = s2 and close to 1 if s1 = s2.This causes a 'crowding' of values on the extremes of a distance measure when agencies' distance to themselves are included.That in mind, Formula 3's alternative distance from source 1 (s1) to source 2 (s2) calculates the percent of incorrectly classified articles from s1 that were predicted as s2.In contrast to Formula 2, this affords comparison to Pew scores but is only suited to 'directional' tasks (no guarantee that distance(s1, s2 otherwise Formula 3: Directional Distance Function

Formulation for comparison with sentiment polarity
To understand model behavior, this study also contrasts results to a pre-trained sentence polarity model, examining model relationship to not only agencies' average title polarity score but also the percent of articles from an agency whose titles exhibit 'substantial overall' positive or negative polarity (found by filtering for titles with polarity averages < −0.3 or > 0.3) (Loria, 2019).

Understanding feature importance
To understand the types of patterns that this model identifies, this study presents findings through Local Interpretable Model-agnostic Explanations (LIME) (Ribeiro, Singh, and Guestrin, 2016).This method can quantify the importance of features (words) to model predictions.

Finding exemplars and examining topics
Qualitatively exploring the nature of news media patterns, the final softmax layers' 0 to 1 scores indicate likelihood that an article came from a particular news agency such that filtering to the top 'n' documents sorted by an agency's score in descending order identifies news articles that are most like their publishing source.This reveals, for example, which NPR articles are the most prototypical of NPR coverage within the model.Similarly, one can find cross-agency exemplars and sorting Fox articles by their Breitbart score allows one to find the most 'Breitbart-like' articles published by Fox, revealing possible similarities in voice.Finally, filtering for exemplars within a topic affords comparison of news sources within a single issue or event, subject matter exemplars this paper explores through an example topic set that the author believes a priori may provoke interesting comparisons: school, climate, democrat, Trump, and immigration.Though this paper uses this set below, an open source application enables investigation of other topics (Pottinger, 2019g).

Supplemental resources and reproducibility
Multiple configurations for neural networks are considered through a newly released open source system and, in addition to code and data in Code Ocean for reproducibility, this study supplements this methods section via Protocols.io(Pottinger, 2019c;Pottinger, 2019d;Pottinger, 2019e).

Results
The study finds evidence to support all three originally posed hypotheses (restated and referred to as Hypotheses 1 through 3): 1. Given article titles, this paper's method can predict publishing agency better than chance.
2. This study's model can better predict the originating agency using article description compared to using title.
3. This method's performance will increase when filtering for articles discussing a common topic from within an example subjects set.

Model performance
Both FF and LSTM outperform the naive ('by chance') method and use of description outperforms title (Figure 2), supporting both hypotheses 1 and 2. Note that the 'naive' approach is to guess the source with the largest number of articles published (Daily Mail), resulting in a 32% accuracy.This paper later provides per agency scores and class imbalance is further discussed below.

Feed-forward or LSTM
To decide between FF and LSTM, both are trained on the same corpus using a set of similar parameters (descriptions, 10,000 words, 0.01 L2 regularization, no dropout).LSTM achieves 70% validation set accuracy while FF achieves 73%, justifying a preference for FF.As cataloged, training within other parameter sets similarly sees FF outperform LSTM in this study (Pottinger, 2019f).

Architectural variation
In FF, all dense layer sizes yield comparable validation accuracy as shown in Table 1.Similarly, the number of LSTM units does not drastically impact validation accuracy (16 units yield 68%, 32 yield 70% Table 1: Feed forward dense layer sizes and validation set performance

Regularization
Regularization dramatically impacts performance and moderate L2 regularization yields preferred accuracies as shown in Figure 3.The study hypothesizes that the number of words included in the encoding could provide a form of regularization as highly uncommon words may uniquely identify articles or a small set of articles.The effect of encoder size is tested as reported in Figure 5.In general, the model saw performance degradation in smaller dictionary sizes (fewer unique words encoded) and, while showing minor over-fitting, larger dictionary sizes lead to higher accuracy on the test set.However, the marginal utility of additional words appears to decrease after around 10,000 words.Specifically, as shown in Figure 5, 10,000 words provide slightly improved validation set accuracy compared to 30,000 words.

Selected configuration
Given empirical results, the study recommends the following: • 32 node first dense layer, 16 node second dense layer FF.
The model achieves achieves a 74% test set accuracy after selecting this configuration, a result in line with its 73% validation set accuracy.This paper uses these settings in presenting results below.

Topical performance
Next, supporting Hypothesis 3, accuracy increases when comparing different organizations coverage of the same subjects (Figure 6).However, uneven performance distribution across topics may indicate that some subjects showcase differences in voice better than others.Note that this test uses accuracy across all sets (test, validation, and training) as topic filtering causes a sharp reduction in n size, especially when considering either just validation or test sets.Also, due to filtering on topic yielding low sample sizes, observe that the model is not retrained when evaluating each topic.As explored later, this may suggest that the model compares signals within the context of a topic (different agencies' 'treatments' of an issue) but the same model can perform across different subject matter.
Figure 6: Performance after filtering for keywords within the example topics set

Voice similarity
Helping to understand these linguistic relationships, consider that two news agencies may have similar ways of speaking if articles from one source are often confused as having come from another source.Figure 7 shows the results of applying Formula 2 and is consistent with prior work on political ideology, demonstrating clear links between NPR, CNN, and the New York Times as well as between Breitbart and Fox (Mitchell et al., 2014).That said, some edges may indicate pattern similarities less emergent from politics.Though prior work would suggest that their political leaning differs, Fox's strong connection to CNN indicates that the 'voice' in which they deliver their different message is still itself similar (Mitchell et al., 2014).This is explored further below.
Figure 7: News agency graph with line width proportional to similarity score

Quantitative understanding of patterns
To understand the types of patterns identified, this study compares this model's behavior to related work and calculates feature (word) importance.

Ideology and polarity
Using Formula 3, Figure 8 describes a modest correlation between the Pew score and the distance from Breitbart in this model (R 2 = 0.3).This relationship suggests that, while this classifier may interpret political signals, model behavior is poorly explained by political lean alone.

Word importance
As polarity and ideology fail to fully explain model behavior, this study next evaluates the importance of different words using LIME (Ribeiro, Singh, and Guestrin, 2016).However, using the prototypical articles' described in Section 2.8, even the ten most 'important' words identified per article only weakly influence predictions (Figure 10).
Figure 10: LIME analysis of prototypical articles Indeed, 98% of words examined shift an agency's score by less than 0.2 (57% under 0.05) despite this prototypical set's agency scores nearing 1.Furthermore, as published in supplemental material, LIME observes the model using logical candidates for discrimination (Donald, growth, wounded, police, federal, 'England' rugby team, etc.) even if no token achieves overwhelming power (Pottinger, 2019i).This highly dispersed token importance distribution implies that this model behavior is complex beyond simple word or topic probabilities, possibly indicating that the model is interpreting something like contextual sentiment towards topics or framing devices.This study further explores this in the qualitative results below.

Qualitative understanding of patterns
To give context to quantitative exploration, prototypical articles provide qualitative understanding of these patterns as they manifest in individual stories.

Agency exemplars
Starting with full corpus exemplars, consider

Topical exemplars
Prototypical articles can also isolate differences in coverage in the context of a particular topic, sharpening a view into the model's learned trends.For example, Table 3 displays prototypical stories discussing climate.While again no single attribute of 'voice' seems to define all of the sources, some observations from that set which may reveal framing devices: • Breitbart frames climate change within context of political power.
• Fox's exemplar places the topic within a political frame whereas NPR places it within a scientific one.
• Wall Street Journal situates the issue within an economic context, consistent with its economic focus.
• Both BBC News and the Daily Mail discuss climate protesters but one puts an emphasis on violence.

Cross-agency exemplars
Finally, to qualitatively understand why some agencies exhibit connections in Figure 7, Table 4 examines articles from one agency showing patterns of another using the method from Section 2.8.The high polarity of CNN / Fox articles contrasts low polarity economic focus of the Wall Street Journal / NPR articles.Though prior work would not expect a connection between CNN and Fox or NPR and the Wall Street Journal, these cross-exemplars highlight possible nonpolitical connections shared between agencies (Mitchell et al., 2014).

Other observations
This study wishes to highlight final observations within results before discussion.

Per-agency performance
Figure 11 shows some like Vox have 'high separability' whereas some like Fox saw more confusion.That said, a weak correlation of F1 score to sample size (R 2 = 0.2) suggests added data aids learning but class imbalance does not bias towards more strongly represented agencies (Figure 12).Therefore, Figure 11 likely sees feature discernibility lead some agencies to be more identifiable than others.

Fox News
In early training, Fox News' recall in the validation set was below 1% and, to remedy this, Fox is re-sampled (articles duplicated) so that additional instances appear in training (but not in test or validation sets).The amount of resampling (from 0% to 100%) of Fox does not impact overall validation accuracy by more than 1% but, using 100% duplication (each Fox article in the training set duplicated exactly once), Fox's recall reaches 43%.This study does not resample other agencies as performance is not as disparate for any other outlet.Depending on use case, derived work may consider this resampling optional.

Search engine
To enable researchers to further explore the exemplars generated by this model, a public demo search engine is made available at whowrotethis.comusing the Flask framework and an 'inverted index' (Ronacher, 2019;Benedetti, 2018).The application (Figure 13) is available as open source (Pottinger, 2019b).
Figure 13: Who Wrote This running in Firefox on Mac OS X

Discussion
This study's model demonstrates how linguistic patterns can not only identify individual agencies but also reveal characteristic tone and framing.Furthermore, while this neural network-based classifier may include signals of sentiment polarity and political ideology, those measures fail to fully describe these predictions which are more complex than an individual word's probabilistic relationship with a publisher.After demonstrating predictive power and information retrieval capability, this study now turns to discussion of findings.

Implications for modeling
The length of documents, number of training instances, and choices made in regularization / model architecture all impact accuracy.However, this study finds complex relationships between outcome and those inputs and structures.

Sensitivities to amount of input data
As described, amount of input data impacts accuracy but more data may not always yield better outcome.Most notably, description out-performs title likely because the later is longer (Figure 14).That said, increasing dictionary size does improve performance but only up to a certain point of rapidly diminishing returns and, while increased sample sizes may aid learning for some agencies, more training instances do not guarantee increased performance.This model may require regularization because articles have highly identifying word co-occurrences which enable them to be identified without extracting broader patterns.That said, weight penalties like L2 regularization outperform dropout in this model and results suggest L2 weight penalties may also improve regression in related tasks such as topic modeling or authorship assignment.

Feed-forward vs LSTM
FF prevails in this task possibly because, given recent research in useable context, the number of tokens per instance may simply be too short to take full advantage of LSTM's properties (Khandelwal et al., 2018).Note that there have been other findings of FF outperforming recurrent structures in some circumstances (Miller, 2018).Beyond accuracy, FF sees notably lower wall clock training time than LSTM as cataloged (Pottinger, 2019f).

Emergent signals of ideology
While model behavior seems broader than conservative versus liberal or sentiment polarity, some learned patterns are likely ideological in nature.For example, a learned belief around agency co-occurrence of 'economy' and 'climate' or 'Trump' and 'honorable' could be a form of 'commentary' on a publisher, an emergent 'form of speech' without explicitly ideological feature engineering.While this model seeks that linguistic bias, this could have implications for other machine learning systems desiring or requiring neutrality.Even without express instruction to extract ideology, consider that news recommenders could learn ideological-correlated features to suggest articles matching users' subjectivities, creating or reinforcing 'filter bubbles' (Delaney, 2017).Alternatively, in 'cold start' or a non-personalized system where all users are presented the same recommendations, an unintended non-neutrality in the platform could emerge to reflect majority preferences (Gaspar, 2018).Similar to prior work discussing racial or sexual disparity in algorithm performance, this unintended bias may impact discussion of technology companies' ethical responsibilities when they act as mediators of the news media ecosystem or host public discourse (Levin, 2018;Santamicone, 2019).Furthermore, like earlier work in biased word embeddings, recognition of possible content-emergent bias could open avenues of remediation in both recommenders and other systems processing this kind of text (Bolukbasi et al., 2016).

Implications for news media
Americans' trust in different news agencies is neither high nor uniform across demographic group or political affiliation but, while many studies, news stories, and even commercial products examine differences in coverage on the axis of politics, this study suggests that journalistic voice is more complex than ideological lean (Knight Foundation, 2018;Mitchell et al., 2014;Knobloch-Westerwick and Meng, 2009;Hess, 2017;AllSides, 2019).In addition to recommending future news media studies consider non-political angles, this study in particular calls for work in determining if non-political patterns influence news consumption behavior because, if these other attributes of 'voice' like framing or polarity influence consumption in ways similar to politics, an article's addressable audience may not only be limited by perceptions of its political bias but by audience affinity for the other linguistic markers with which it is written, further dividing the modern media landscape beyond politics alone (Knight Foundation, 2018;Knobloch-Westerwick and Meng, 2009).In short, if voice 'mediates' the way an article is received, these linguistic patterns not only present a challenge to a journalist attempting to reach an audience but non-political signals may further shrink the number of avenues through which 'facts' are established for a population.This is discussed further in future work.

Class imbalance
Though this study does take some steps to address class imbalance when it impacts performance, future studies could explore instance weighting and resampling to confront the fact that some agencies publish more work.That said, this paper suggests that an increased number of instances may or may not improve performance and some infrequent classes do see high performance in the current model iteration.

Full article length
This study does not attempt classification using full article text which could potentially improve performance.Still, given that the number of words included in an encoding offers only a diminishing return after a certain point, this study suggests that regressing on the full article may not necessarily improve performance substantially.

Testing voice preferences
As discussed, this paper cannot empirically establish that the identified patterns increase appeal to certain audiences and, in the case of non-political patterns, it is possible but not necessarily true that these markers impact media consumption (Knobloch-Westerwick and Meng, 2009).Future work could close this gap by exploring the interaction of favorability and voice in different populations similar to prior work but outside an explicitly political lens, comparing framing devices (economic vs political) or polarity for example (Knobloch-Westerwick and Meng, 2009).

Topic modeling and sentiment analysis
This paper simply filters for keywords within a title when exploring a topic and brief experiments with Latent Dirichlet Allocation and Tf-Idf did not result in sensible topic modeling yet ready for publication (Blei et al., 2003;Bafna, Pramod, and Vaidya, 2016).Furthermore, this study only uses pre-existing sentiment models.Therefore, it is possible other studies could create more performant sentiment or topic modeling system with tailoring to this dataset, possibly providing alternatives to this paper's example subject set.

Conclusions
Learning from publicly available RSS feeds for ten news organizations, this machine learning method uncovers different agencies' linguistic characteristics through a neural network-based model capable of identifying the publisher of an article with 74% test set accuracy.This probabilistic classifier allows for quantitative contrasting of different organizations' patterns and empirical findings provide recommendations for related tasks both on this study's newly released open database and in similar news media datasets.Furthermore, an 'exemplar' search engine using the learned model enables researchers to qualitatively uncover new insight into organization's coverage.Finally, ending with commentary rooted in previous linguistic, psychological, and media studies research, discussion highlights possible implications of these patterns for the media landscape and, in demonstrating a mechanism for inadvertent 'filter bubble' emergence, describe practical ethical implications for other machine learning efforts.That said, in addition to recommending investigation outside an explicitly liberal versus conservative lens, this paper invites future work to further investigate observed phenomena both through machine learning methods like topic modeling and through psychological / linguistic studies.

Reproducibility and data
This study releases code under a permissive open source license and provides runnable containers via Code Ocean (Pottinger, 2019c;Pottinger, 2019a;Pottinger, 2019b;Pottinger, 2019d).Empirical results for attempted model configurations are also archived via Weights and Biases and the methods section is supplemented via Protocols.io(Pottinger, 2019f;Pottinger, 2019e)

Figure
Figure 4: Dropout rates

Figure 5 :
Figure 5: Performance across different number of words encoded

Figure 8 :
Figure 8: Pew audience ideology score and Breitbart distance

Figure 9 :
Figure 9: Percent articles with negative polarity and Breitbart distance

Figure 11 :
Figure 11: Validation set performance per agency

Figure 12 :
Figure 12: Validation set F1 score versus number of articles for an agency

Figure 14 :
Figure 14: Lengths of inputs across all agencies ).Based on this, FF with the (32, 16) layer configuration is used to classify news agency source in subsequent evaluations as it achieves the highest validation set accuracy.

Table 2
Of course, mirroring token importance above, Table2may also reveal some possible topical bias like in BBC's reference to England's rugby team.That said, in addition subject matter discrimination, this collection also highlights characteristic vantage points for some agencies (Fox discussing violent crime and the Wall Street Journal's economic focus) that emerge again below.Still, consistent with the quantitative results, none of these features (sentiment, topical bias, and framing) seem to alone describe this exemplar set.

Table 2 :
Overall exemplar articles per agency

Table 3 :
Prototypical articles discussing climate

Table 4 :
Selection of cross-agency articles Koster, 1996)the spirit of a publisher's wishes, data are made available under the Attribution-NonCommercial-ShareAlike 4.0 International Creative Commons license (Pottinger, 2019h; CNN RSS 2019).It is advised that derived work ensure compliance with governing restrictions around use of article information including possible applicability of 'fair use' and, especially if crawling, updated state of publishers' robots.txtfiles(USCopyright Office, 2019;Koster, 1996).