 |
| Figure 1: Chen et al. NetLogo Chat asking for details about human’s needs. |
Introduction
Data
The study gathered insights from 30 participants with diverse backgrounds, ranging from academia to industry, who were familiar with ABM or NetLogo. Participants were divided into novices and experts based on their self-reported expertise and observed behaviors. The study data comprised video recordings and transcriptions of 40 hours of qualitative interviews, complemented by behavioral observations during task completion. Tasks involved open-ended modeling scenarios using both ChatGPT and NetLogo Chat, allowing researchers to capture user perceptions, interactions, and feedback.
Methodology
NetLogo Chat System - Design
One of the primary design principles of NetLogo Chat is to empower users to program interactively rather than passively relying on AI-generated outputs. Over-reliance on LLM-based systems, where users follow instructions without engaging critically, undermines the learning process. This phenomenon aligns with Seymour Papert’s warning about "computers programming children," as learners bypass opportunities to construct their knowledge and skills.
User-Led Programming with Clarification Questions
 |
| Figure 2: Chen et al. ChatGPT assuming details of human’s needs. These questions serve two purposes: |
- Inspiration: Helping users explore potential directions for their task when they feel stuck.
- Efficiency: Offering actionable suggestions that users can directly implement if aligned with their goals.
This interaction style forces users to actively think about their task, building a deeper understanding of the modeling process. Using few-shot learning, NetLogo Chat is trained to generate high-quality clarification prompts that are relevant and contextual. Future iterations could expand this by incorporating larger template sets to dynamically retrieve the most suitable clarification prompts for user interactions.
Reducing Hallucinations with Authoritative Sources
Another challenge in AI-driven programming is hallucination—when LLMs generate inaccurate or nonsensical outputs. This issue is particularly prevalent in low-resource programming languages (LRPLs) like NetLogo, where even advanced LLMs like GPT-4, PaLM2, and Falcon-180B fail to produce syntactically correct code consistently. For instance, code samples generated by these models often contained syntax errors or conceptual misinterpretations that required expert intervention to resolve.
NetLogo Chat addresses this by integrating authoritative resources, such as:
- Official NetLogo documentation: Providing reliable syntax and usage guidelines.
- Curated code examples: Offering tested, context-specific models for reference.
 |
| Figure 3: Chen et al. NetLogo Chat’s embedded editor for generated code |
Instead of simply incorporating these resources into LLM prompts, NetLogo Chat surfaces them directly to users alongside AI-generated outputs (Figure 3). This dual-layered approach improves transparency, fosters trust in the system, and equips users with tools to cross-check and validate the AI’s suggestions.
Code Snippet Interaction Within the Conversational Interface
In Figure 3, the conversational interface displays generated code snippets alongside the chat. Instead of requiring users to copy and paste the code into a separate editor, NetLogo Chat allows learners to:
- Inspect for syntax issues: Users can immediately verify if the generated code has errors.
- Run the code directly: The system enables real-time testing of snippets within the conversation, allowing users to see how the code performs without switching between tools.
- Iteratively refine code: Learners can ask follow-up questions, request clarifications, or seek modifications to the code directly within the chat, ensuring they return a working snippet to their project.
This seamless integration streamlines the process of testing and refining code, creating a user-friendly environment that aligns with constructionist learning principles.
Technical Implementation
 |
| Figure 4: Chen et al. A brief outline for NetLogo Chat’s LLM workfow |
ReAct Framework: A Step-by-Step Workflow
The ReAct framework introduces an iterative process where the LLM not only generates responses but also plans actions and delegates tasks to external agents when needed. This design allows for integrating external inputs, such as user clarifications and authoritative documentation, into the LLM's workflow, ensuring more accurate and trustworthy outputs.
Here’s how the workflow operates, using the example request to "create a predation model":
Planning the Response: The LLM first interprets the user's request and plans a response. For example, it might note: "The user intends to create an agent-based biology model related to predation. However, it is unclear what exactly the user wants. We need to ask follow-up questions."
Choosing an Action: Based on the planning phase, the LLM selects an action from predefined options, such as:
- Ask clarification questions
- Search for documentation
- Write a response
- Apologize for lack of information
In this case, the LLM chooses to ask clarification questions.
Generating Questions: The LLM generates context-specific questions to refine the user’s request, such as:
- "What species do you want to include in the model?"
- Example suggestions: "Wolf, Sheep."
Iterating with User Input: Once the user provides additional details, such as "Wolf and Sheep," the workflow restarts. The LLM uses this information to further refine the plan, deciding to search for relevant documentation or examples.
Semantic Search Integration: The system conducts a semantic search within a pre-assembled database of NetLogo’s official documentation and example models. For instance, it might search for: "Wolf-sheep predation model in NetLogo." The retrieved results are then fed back into the system as additional inputs.
Final Response Generation: With sufficient inputs from both the user (clarified request) and the database (relevant examples), the LLM plans its final response. It composes a detailed output that integrates the user’s needs and authoritative resources, producing a complete and accurate response.
Conclusion
References
John Chen, Xi Lu, Yuzhou Du, Michael Rejtig, Ruth Bagely, Mike Horn, Uri Wilensky "Learning agent-based modeling with LLM companions: Experiences of novices and experts using ChatGPT & NetLogo chat" in CHI'24: Proceedings of the CHI Conference on Human Factors in Computing Systems. DOI: https://doi.org/10.1145/3613904.3642377
-Nithiya