5.1 Introduction to Discourse Processing

Natural language is rarely expressed in isolated, disconnected sentences. Instead, we communicate using discourse—coherent groups of sentences that build upon one another to convey a larger message. Discourse processing is the area of NLP that focuses on building theories and models of how individual utterances connect to form coherent and meaningful text. Understanding these connections is essential for deep language comprehension and is considered one of the most difficult challenges in Artificial Intelligence.

5.2 The Concept of Coherence

Coherence is the essential property that makes a sequence of sentences feel like a unified whole rather than a random collection of statements. A text that lacks coherence is just a jumble of grammatically correct but unrelated sentences. For a discourse to be coherent, it must possess two key properties:

1. Coherence Relations: There must be meaningful connections between the utterances. The relationship between one sentence and the next should be clear; for example, one might provide an explanation for, a result of, or an elaboration on the other.
2. Entity-Based Coherence: The entities (people, places, objects) mentioned throughout the text must be related to each other in a sensible way. This often involves tracking entities as they are introduced and referred to again later in the discourse.

5.3 Discourse Segmentation

Discourse segmentation is the task of identifying the structure of a large discourse by breaking it down into smaller, meaningful segments (such as paragraphs or passages about a specific topic). This process is vital for applications like information retrieval (finding specific passages within a long document) and automatic text summarization (identifying the most important segments to include in a summary).

5.3.1 Unsupervised Segmentation

This approach, often called linear segmentation, operates without hand-labeled training data. It relies on the principle of cohesion, which refers to the linguistic devices that tie text units together. In particular, it often leverages lexical cohesion, identifying segment boundaries at points where the vocabulary shifts significantly. For example, a boundary is likely to occur where the words in one block of text (e.g., synonyms, related terms) are very different from the words in the next block.

5.3.2 Supervised Segmentation

This approach requires a training corpus where the segment boundaries have been manually labeled. It then trains a model to recognize these boundaries. Supervised methods often rely on discourse markers or cue words (e.g., “however,” “in conclusion,” “firstly”), which are words or phrases that explicitly signal the structure of the discourse. These markers can be highly domain-specific.

5.4 Analyzing Text Coherence Relations

To achieve coherence, utterances within a discourse must be connected by specific logical relations. The following are some of the key coherence relations that can exist between two sentences or clauses (represented as S0 and S1).

• Result: The state asserted in S0 is inferred to be the cause of the state asserted in S1.
  • Example: (S0) Ram was caught in the fire. (S1) His skin burned.
• Explanation: The state asserted in S1 is inferred to be the cause of the state asserted in S0.
  • Example: (S0) Ram fought with Shyam’s friend. (S1) He was drunk.
• Parallel: S0 and S1 assert similar propositions about different entities.
  • Example: (S0) Ram wanted a car. (S1) Shyam wanted money.
• Elaboration: S0 and S1 both infer the same proposition, with one providing more detail.
  • Example: (S0) Ram was from Chandigarh. (S1) Shyam was from Kerala. (Note: This example from the source appears to be mislabeled and would better fit a “Parallel” relation. A better example for Elaboration would be: Ram bought a new car. It is a sleek, red convertible.)
• Occasion: A change of state can be inferred from S0, the final state of which is the initial state for S1.
  • Example: (S0) Ram picked up the book. (S1) He gave it to Shyam.

5.5 Hierarchical Discourse Structure

The coherence of a longer text is often not just a linear chain of relations but forms a hierarchical structure. Relations can be nested, with one relation connecting two individual sentences, and another relation connecting that two-sentence block to a third sentence.

Consider the following passage:

• (S1) Ram went to the bank to deposit money.
• (S2) He then took a train to Shyam’s cloth shop.
• (S3) He wanted to buy some clothes.
• (S4) He did not have new clothes for the party.
• (S5) He also wanted to talk to Shyam regarding his health.

In this example, S4 provides an Explanation for S3. The combined unit (S3-S4) then provides an Explanation for S2. Similarly, S5 stands in a Parallel relation to the (S2-S3-S4) block, as it describes another purpose of the visit. This nesting of relations creates a tree-like structure for the entire discourse.

5.6 Reference Resolution

To interpret a discourse, we must know who or what is being talked about. Reference is the use of a linguistic expression (like a noun or pronoun) to denote an entity in the world. Reference resolution is the crucial task of determining which entities are referred to by which expressions.

5.6.1 Key Terminology

• Referring expression: The linguistic expression used to refer to something (e.g., “Ram,” “he,” “his friend”).
• Referent: The actual entity in the world that is being referred to (e.g., the person named Ram).
• Corefer: When two or more expressions refer to the same entity, they are said to corefer.
• Antecedent: An expression that introduces an entity into the discourse, which is later referred to by another expression. For example, in “Ram saw his friend,” “Ram” is the antecedent of “his.”
• Anaphora: The act of referring back to an entity that has been previously introduced. The referring expression is called an anaphoric expression.
• Discourse model: A computational representation of the entities that have been mentioned in the discourse and the relationships between them.

5.6.2 Types of Referring Expressions

There are several types of linguistic expressions used to refer to entities:

1. Indefinite Noun Phrases: These typically introduce new entities into the discourse. (e.g., “some food,” “one day”).
2. Definite Noun Phrases: These refer to entities that are already known or identifiable to the listener from the context. (e.g., “The Times of India“).
3. Pronouns: A common form of definite reference used to refer to a recently mentioned entity. (e.g., “Ram laughed as loud as he could.”).
4. Demonstratives: Pronouns like “this” and “that” which point to entities.
5. Names: Proper nouns that refer to specific people, organizations, or locations. (e.g., “Ram”).

5.6.3 Core Reference Resolution Tasks

There are two primary sub-tasks within reference resolution:

• Coreference Resolution: This is the task of finding all expressions in a text that refer to the same entity. The result is a set of coreference chains. For example, in a text about the CEO of a company, the expressions “The Chief Manager,” “he,” and “his” might all belong to the same coreference chain. The pronoun “it” is particularly challenging in English because it can refer to specific entities as well as abstract situations (e.g., “It’s raining.”).
• Pronominal Anaphora Resolution: This is the more specific task of finding the antecedent for a single pronoun. For a pronoun like “his,” the task is to identify which previously mentioned noun phrase (the antecedent) it refers to.

This module has demonstrated how discourse processing techniques allow NLP systems to understand the flow and connectivity of text beyond the sentence level. In our next module, we will examine some of the core, practical NLP tasks that build upon these foundational concepts.