Why qualitative tools matter in finance
Qualitative tools process qualitative data and produce qualitative information.
Quantitative data is easy to recognise: share prices, historical time series of shares and bonds, inflation, interest rates and every other kind of relevant numerical data. It can be fed straight into mathematical or statistical equations. Qualitative data is harder to define — and generally cannot.
Qualitative data is difficult, if not impossible, to express as numbers: rumours, fears, brokers’ recommendations, takeovers. A sentence such as “there are rumours of a possible takeover of Apple, the troubled computer manufacturer” is highly relevant to financial operators, because it is likely to move Apple’s share price immediately — and those of potential buyers. Yet it would be extremely difficult, if not impossible, to fit into a mathematical or statistical equation. News like this matters because it shapes operators’ expectations about a share, and how each operator reacts depends on how they perceive it. Even if a complete econometric model of every variable and every player’s expected behaviour were possible in theory, the complexity of the financial world makes it impossible in practice — and it would demand enormous computing power.
Macroeconomics shows the same pattern. The sophisticated econometric models used by central banks often fail to predict economic cycles, crises and expansions. Only a handful of macroeconomic relationships (such as interest rates and investment) are widely used, where the effect of a change in one variable is easy to predict.
Because qualitative data is so much harder to process, the market is full of quantitative financial tools while qualitative information is still largely left to the operators themselves.
Operators are influenced far more by news than by analysts’ forecasts or historical price analysis.
In the financial community, news, rumours and facts are among the most important drivers of behaviour. When news such as “the inflation rate is expected to increase next month” arrives, the consequences are immediate: operators act on their own experience and on how others behave, not on expensive forecasts from complex neural-network systems. Quantitative methods are useful — they suggest a “normal” path for prices — but in the end what counts is the news and how people react to it.
Operators and information providers recognised long ago that qualitative data is the key to trading decisions, so the emphasis has been on delivering as much relevant qualitative information as possible. Operators receive real-time news about companies (announcements, rumours, profit forecasts), the economy (inflation, unemployment) and politics (shifts in government economic or tax policy), together with vast archives of past information.
The ideal would be a system that processes qualitative information, weighs every relevant factor and returns a decision such as “buy” or “sell”. Traditional mathematical and statistical techniques are unlikely to manage this, and current artificial intelligence is not yet sophisticated enough — so decisions are still taken mainly by the operators.
Qualitative tools today therefore focus on reducing, summarising or classifying news according to specific criteria rather than inferring decisions from it — the equivalent of simple quantitative analysis (a moving average, say) rather than a clear trading decision. Explanatory and forecasting qualitative tools have yet to be built, and Natural Language Processing is a strong candidate for building them.
Natural Language Processing as a support tool
As a general support tool, Natural Language Processing summarises, reduces or classifies qualitative input rather than analysing the action an operator should take. The aim is to give the operator a summary of the most important data, not to recommend a trade — much like a moving average, which captures and identifies a trend but leaves its interpretation, and the decision, to the operator. In the same way, an NLP tool might report that the main theme across a group of news stories is a probable rise in inflation, leaving the operator to interpret it.
The main task of NLP support tools is therefore to help operators cope with qualitative information overload, simplifying and reducing the information they need to make decisions.
Information extraction is the NLP technique that identifies relevant information in a source text and extracts it into predefined structures (templates). For this reason, most NLP-based financial tools perform information extraction.
So far, very few NLP-based tools have been successfully adopted by the financial community, and most were designed for very specific tasks in very narrow domains. They usually rely on techniques closer to information retrieval and word matching than to true NLP. No information extraction system yet processes large volumes of qualitative data and produces sensible results across a reasonably broad financial domain.
NLP and information extraction systems are mainly developed for, and used by, information providers, who are keen to classify, reduce and summarise the huge volume of financial information they deliver to customers. Dow Jones’ Dow-vision Internet news service, for example, automatically adds information to articles in real time — but that information is fairly basic, such as the market sector and category of the company, and is obtained mainly through pattern matching rather than NLP, or even entered by hand by the provider’s staff.
ATRANS (Automatic Processing of Money Transfer Messages) has been used successfully on real tasks, but it dealt with a specific type of inter-bank message in a very narrow domain, not directly related to trading securities on the stock exchange.
Natural Language Processing as an explanatory tool
We believe Natural Language Processing can also be used as an explanatory and predictive tool — something no natural-language-based tool has yet achieved. In our view, an NLP explanatory financial tool could be built on the following points:
- Inference beyond the text. An NLP information extraction system can infer knowledge that is not stated directly in an article. A text can describe a takeover without ever using the word “takeover” or “acquisition”: the system infers the underlying concept and extracts the relevant explanatory information. Operators can then use this to explain price behaviour in terms of qualitative information — work that normally has to be done by hand.
- Meta-analysis. Prices are driven by people’s perception of events rather than by logical or mathematical equations, so operators care about how a piece of information was reported: the way an event is described reflects the writer’s perception and usually influences the reader’s. Meta-analysis adds this information to the summary or template extracted from the source text, in two ways:
- How often a topic is mentioned in the article. When a piece of news is repeated many times, the number of mentions and its importance are likely to be directly related, so the system should identify the number and relevance of each mention and report them — using a semantic rather than a superficial comparison.
- How the information is expressed. The system can identify the way the news is told and present it to the user.
Natural Language Processing as a forecasting tool
Finally, Natural Language Processing could serve as a forecasting tool, suggesting final buy or sell decisions. The idea is that operators often act on real-time news rather than on quantitative predictions, following these steps:
- the operator reads the news;
- the operator interprets the new information;
- the operator compares and analyses it against what he or she already knows;
- a final decision — buy or sell — is taken.
An NLP forecasting tool would automate this process: process the new data, identify what is relevant, interpret it with domain knowledge specific to the situation, and present the operator with a suggested decision alongside the summary or template of the original article. Alternatively, it could work with existing prediction tools such as expert systems and neural networks, with the NLP system extracting the relevant information and passing it to them in the appropriate form.
