Figure 1. An Illustration of the transition from storytelling to advanced conceptualization. Image Generated by DALL-E.
In my previous blog,—“Do Large Language Models Agree on Entity Extraction?”— I explored how different Large Language Models (LLMs) approach Named Entity Recognition (NER) tasks, highlighting the inconsistencies in entity identification and categorization. By leveraging techniques such as Sentence-BERTembeddings, and cosine similarity, I demonstrated how to unify outputs from multiple LLMs into a coherent and reliable list of entities, which is essential for tasks requiring consistent information extraction.
This blog transitions from entity extraction to a broader exploration of how LLMs move from narrative analysis to conceptual modeling and simulation. While extracting entities is a critical step in understanding narratives, the next challenge lies in structuring those narratives into meaningful models that can inform decision-making and support real-world applications. Here, I delve into how narratives can be conceptualized, represented, and transformed into actionable models using LLMs, bridging the gap between descriptive prose and computational simulations. If you want to replicate these experiments, you can access the GitHub repository.
Introduction
LLMs have showcased remarkable potential across various domains, from security to scientific research, mathematics, and education. In cybersecurity, LLMs can bolster code security by pinpointing vulnerabilities, and they excel in data privacy management, often surpassing traditional security frameworks (Yao et al., 2024). In scientific research, LLMs expedite the discovery process by condensing vast volumes of literature and enhancing the clarity and structure of scientific writing (Boyko et al., 2023). In mathematics, LLMs are on the brink of revolutionizing problem-solving for algebraic equations and calculus. However, they face challenges regarding abstract proofs or problems that necessitate higher reasoning (Ahn et al., 2024). In the educational sphere, LLMs are becoming increasingly indispensable for tutoring subjects like mathematics, programming, and reasoning while transforming knowledge-based question-answering systems in online education platforms (Li et al., 2023).
However, despite these advances, challenges persist. LLMs heavily rely on vast datasets for training, which can introduce biases and raise ethical concerns around privacy, fairness, and misinformation (Boyko et al., 2023; Yao et al., 2024). Additionally, LLMs often need help producing concise and executable models, especially when tasked with programming languages they are less familiar with (Frydenlund et al., 2024). An example of this limitation is their inability to generate accurate simulation models from a given narrative. Frydenlund et al. (2024) evaluated ChatGPT’s ability to generate simulation models from a narrative about disrupted student transportation during the COVID-19 pandemic, finding that it struggled to produce concise, executable models, mainly when working with unfamiliar programming languages. This highlights the significant limitations in converting descriptive prose into complex simulations. Giabbanelli (2023) further emphasizes that while GPT-based models like ChatGPT show potential in simulation tasks, they encounter significant limitations when performing complex modeling, such as explaining simulation structures or interpreting dynamic outcomes.
Despite the strides made with LLMs in tasks such as text generation and code development, there is a glaring gap in integrating LLMs with real-world simulation processes. Current research, including studies by Frydenlund et al. (2024) and Giabbanelli (2023), underscores this. While LLMs show promise in simulating narrative-based phenomena, their inability to generate concise, executable models and accurately interpret dynamic outcomes reveals the limitations of existing models. This gap urgently calls for more advanced frameworks that can effectively bridge narrative understanding and simulation tasks.
How do we represent a narrative structure?
Narratives convey valuable information and can be coherently conceptualized, capturing key elements such as events, experiences, or real-world phenomena. They are essential for understanding how humans interpret and communicate information, shaping how stories are told and understood. Narratives simplify complex ideas by organizing them into sequences that provide context, purpose, and direction. Ideally, narratives should be modeled to describe real-world problems. However, while LLMs hold promise, they are limited in their ability to model narratives.
To conceptualize a narrative, we begin by identifying its fundamental components: the entities and their relationships. Some studies demonstrate how Natural Language Processing (NLP) can aid this task. For instance, Holur et al. (2023) extracted aligned actor groups from news articles using question-answering models, identifying key actors and their relationships to reveal dynamics within the narrative and better understand how these actors are positioned within the broader story. Similarly, Soni et al. (2023) extracted spatial relationships between characters and locations within narratives, using techniques like NER and semantic role labeling (SRL) to identify critical actors and map their interactions, providing insights into how these elements contribute to the overall structure of the narrative. Likewise, Hsu et al. (2022) used sequence tagging and priming techniques to extract relational structures like events, arguments, and relationships from the text, improving the extraction of entity relationships by integrating priming techniques with pre-trained models.
By organizing these parts into a coherent structure, we gain a deeper understanding of the narrative’s essence. After conceptualizing, narratives can be presented through visual tools, textual formats, simulations, and graphical representations, each offering unique ways to simplify and communicate complex ideas. For instance, Giabbanelli and Witkowicz (2024) developed a GPT-based question-answering system to extract causal relationships from text and visualize them in causal maps, identifying concept nodes and their interactions to represent the narrative structure.
Narrative Characterization: Actor, Factor, Mechanism
When characterizing a narrative in terms of actor, factor, and mechanism (AFM) involves identifying the main participants (actor), the contributing elements or circumstances (factors), and the processes or actions taken (mechanisms). In the context of a knowledge graph, the nodes represent actors, factors, and mechanisms. At the same time, the edges show the relationships on how actors use mechanisms to interact with factors (actions taken by actors), such as “upheld the redistricting,” “criticized the decision,” and “benefited from redistricting.” Actors can range from governmental bodies to grassroots movements, each playing a distinct role in the narrative construction. Factors encompass the various elements influencing the narrative, such as values, interests, and divergences among stakeholders. Conversely, mechanisms delve into the actors’ actions, strategies, and processes to advance their narratives and agendas. By mapping out these components in a knowledge graph, one can visualize the intricate web of relationships and connections that underpin the storytelling in politics and policymaking. In a simulation process, actors represent entities that perform a function or are impacted by another entity, factors are variables influencing the narrative, and mechanisms refer to actions or effects within the narrative framework (Shuttleworth & Padilla, 2021).
Shuttleworth and Padilla (2021) used natural language understanding techniques to extract actors, factors, and mechanisms from narratives by applying text processing methods to identify key entities, attributes, and interactions. In contrast, Shuttleworth and Padilla (2022) utilized NLP techniques, such as dependency parsing and pattern-based grammatical rules. Both studies aimed to transform these components into conceptual models and simulation specifications by identifying the relationships between the entities and then visualizing these relationships through network graphs to represent the narrative structure. No evidence exists that LLMs have yet to be applied to this specific NER task.
Experimental Setting by Using LLMs
For the experiment, I used a hypothetical text illustrating a real-world scenario involving political figures and judicial decisions: “President Joe Biden criticized the Supreme Court’s decision on the redistricting of the South Carolina district by issuing a public statement highlighting concerns about racial discrimination.”
In reality, the focus of the news was on the Supreme Court's ruling, which allowed South Carolina’s racially discriminatory congressional map to stand. Civil rights organizations, such as the ACLU, responded to the ruling, stating that the map unfairly targeted Black voters and diluted their electoral influence. The ACLU emphasized that this decision undermines principles of equality and justice protected under the Equal Protection Clause.
It is important to clarify that President Biden did not make such a statement. The hypothetical text was crafted to represent a scenario, while the actual reactions centered on the judicial ruling and the concerns raised by civil rights advocates.
From this text, we can identify the AFMs as follows: “President Joe Biden (actor) criticized the Supreme Court's decision (factor) on the redistricting of the South Carolina district (factor) by issuing a public statement (mechanism), highlighting concerns about racial discrimination (factor).”
To perform the narrative characterization, I used GPT-4o (OpenAI) and Ollama, an open-source platform for running LLMs locally. Specifically, I utilized Llama 3 8B (Meta) and Mistral 7B (Mistral AI).
LLMs have significantly advanced the automation of such tasks, enabling the extraction of AFMs directly from narratives. For example, by prompting an LLM such as GPT-4o—using the ChatGPT interface—we can 1) decompose the narrative into its AFM components and 2) extract the necessary nodes and edges to create a visualization, facilitating a deeper understanding of the relationships within the narrative.
Narrative Characterization
System: You are a helpful assistant who characterizes narratives into actors, factors, and mechanisms. Actors can range from governmental bodies to grassroots movements, each playing a distinct role in the narrative construction. Factors encompass socioeconomic, cultural, and environmental variables, while mechanisms refer to processes or actions taken to generate events, analyze a text, and extract the actors, factors, and mechanisms.
Prompt: What are the actors, factors, and mechanisms in the following text?
Text: President Joe Biden criticized the Supreme Court’s decision on the redistricting of the South Carolina district by issuing a public statement highlighting concerns about racial discrimination.
Table 1. Narrative characterization from gpt-4o, llama3 and mistral
Extract nodes and edges for the visualization by using gpt-4o.
Prompt: Make a visual representation of the narrative characterization. Consider that the nodes represent actors, factors, and mechanisms in the context of a knowledge graph. At the same time, the edges show the relationships on how actors use mechanisms to interact with factors (actions taken by actors), such as “upheld the redistricting,” “criticized the decision,” and “benefited from redistricting.”
Output: Here is the visual representation of the narrative characterization as a knowledge graph. The nodes represent actors, factors, and mechanisms, while the edges show the relationships and how actors use mechanisms to interact with factors. The colors differentiate between actors (skyblue), factors (lightgreen), and mechanisms (lightcoral).
· Actors:
· President Joe Biden
· Supreme Court
· South Carolina district
· Factors:
· Racial Discrimination
· Values and Interests
· Divergences among Stakeholders
· Mechanisms:
· Public Statement
· Highlighting Concerns
· Criticism
· Relationships:
· President Joe Biden uses a Public Statement to highlight Racial Discrimination.
· President Joe Biden uses Highlighting Concerns to bring attention to Racial Discrimination.
· President Joe Biden uses Criticism to challenge the Supreme Court.
· The Supreme Court influences Values and Interests.
· The South Carolina district is affected by Racial Discrimination.
· The Supreme Court creates Divergences among Stakeholders.
Figure 2. AFM Visual representation of the narrative by using ChatGPT interface—model gpt-4o.
Conclusions
In conclusion, while LLMs have demonstrated significant progress in tasks such as entity extraction and narrative characterization, their full potential in bridging narratives to conceptual models and simulations remains a work in progress by leveraging advanced techniques like NER and frameworks like AFM. However, challenges persist in ensuring consistency, accuracy, and the ability to generate executable models from descriptive text.
This exploration highlights the transformative possibilities of LLMs in automating narrative analysis and visualization, underscoring their role in addressing real-world problems. Refining these methods and integrating contextual understanding into LLM workflows will be pivotal in enhancing their utility for modeling, simulation, and other complex computational tasks. These advancements will pave the way for more effective tools to interpret and act upon narratives, bridging the gap between textual data and actionable insights.
To further contribute to the fields of Modeling & Simulation (M&S) and Computer Science (CS), we propose key steps to advance state of the art: (1) Evaluating the capabilities of NER models like Gliner (Zaratiana et al., 2023), NuNER (Bogdanov et al., 2024), and UniversalNER (Zhou et al., 2023) for identifying and classifying entities as actors, factors, or mechanisms; (2) Developing a benchmark dataset specifically for AFM tasks to standardize evaluation; (3) Creating a dedicated NER model optimized for AFM extraction; (4) Extracting and mapping relationships between entities to provide richer contextual insights; and (5) Visualizing these characterizations to make the narrative structure more accessible and actionable.
These contributions are essential to closing the gap between narrative analysis and practical application in computational models. They also highlight the importance of advancing LLM methodologies to handle complex narrative tasks effectively, ultimately enabling more robust tools for tackling multifaceted real-world challenges.
References
Ahn, J., Verma, R., Lou, R., Liu, D., Zhang, R., & Yin, W. (2024). Large language models for mathematical reasoning: Progresses and challenges. arXiv preprint arXiv:2402.00157.
Bogdanov, S., Constantin, A., Bernard, T., Crabbé, B., & Bernard, E. (2024). NuNER: Entity Recognition Encoder Pre-training via LLM-Annotated Data. arXiv preprint arXiv:2402.15343.
Boyko, J., Cohen, J., Fox, N., Veiga, M. H., Li, J. I., Liu, J., Modenesi, B., Rauch, A. H., Reid, K. N., & Tribedi, S. (2023). An interdisciplinary outlook on large language models for scientific research. arXiv preprint arXiv:2311.04929.
Frydenlund, E., Martínez, J., Padilla, J. J., Palacio, K., & Shuttleworth, D. (2024). Modeler in a box: how can large language models aid in the simulation modeling process? SIMULATION, 00375497241239360. https://doi.org/10.1177/00375497241239360.
Giabbanelli, P., & Witkowicz, N. (2024). Generative AI for Systems Thinking: Can a GPT Question-Answering System Turn Text into the Causal Maps Produced by Human Readers? In Proceedings of the 57th Hawaii International Conference on System Sciences. https://hdl.handle.net/10125/107291.
Giabbanelli, P. J. (2023, December). GPT-based models meet simulation: how to efficiently use large-scale pre-trained language models across simulation tasks. In 2023 Winter Simulation Conference (WSC) (pp. 2920-2931). IEEE. https://doi.org/10.1109/WSC60868.2023.10408017.
Holur, P., Chong, D., Tangherlini, T., & Roychowdhury, V. (2023, July). My side, your side and the evidence: Discovering aligned actor groups and the narratives they weave. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 8938-8952). https://doi.org/10.18653/v1/2023.acl-long.497.
Hsu, I., Huang, K.-H., Zhang, S., Cheng, W., Natarajan, P., Chang, K.-W., & Peng, N. (2022). TAGPRIME: A unified framework for relational structure extraction. arXiv preprint arXiv:2205.12585.
Li, Q., Fu, L., Zhang, W., Chen, X., Yu, J., Xia, W., Zhang, W., Tang, R., & Yu, Y. (2023). Adapting large language models for education: Foundational capabilities, potentials, and challenges. arXiv preprint arXiv:2401.08664.
Shuttleworth, D., & Padilla, J. (2022, December). From Narratives to Conceptual Models via Natural Language Processing. In 2022 Winter Simulation Conference (WSC) (pp. 2222-2233). IEEE. https://doi.org/10.1109/WSC57314.2022.10015274.
Shuttleworth, D., & Padilla, J. J. (2021, December). Towards semi-automatic model specification. In 2021 Winter Simulation Conference (WSC) (pp. 1-12). IEEE. https://doi.org/10.1109/WSC52266.2021.9715393.
Soni, S., Sihra, A., Evans, E. F., Wilkens, M., & Bamman, D. (2023). Grounding characters and places in narrative texts. arXiv preprint arXiv:2305.17561.
Yao, Y., Duan, J., Xu, K., Cai, Y., Sun, Z., & Zhang, Y. (2024). A survey on large language model (LLM) security and privacy: The Good, The Bad, and The Ugly. High-Confidence Computing, 4(2), 100211. https://doi.org/10.1016/j.hcc.2024.100211
Zaratiana, U., Tomeh, N., Holat, P., & Charnois, T. (2023). Gliner: Generalist model for named entity recognition using bidirectional transformer. arXiv preprint arXiv:2311.08526.
Zhou, W., Zhang, S., Gu, Y., Chen, M., & Poon, H. (2023). Universalner: Targeted distillation from large language models for open named entity recognition. arXiv preprint arXiv:2308.03279.
- Brian Llinás (bllin001)