CEAA: A Cognitive Embodied Agents Architecture for Interactive Computing Systems

CEAA: A Cognitive Embodied Agents Architecture for Interactive Computing Systems

This paper proposes CEAA, a modular, implementation-oriented architecture for deploying embodied Intelligent Virtual Agents (IVAs) with cognitive capabilities in real-time interactive virtual environments. The architecture addresses the persistent gap between high-level cognitive agent models and their practical deployment in real-time 3D systems. Existing architectures are often focused on either the implementation of low-level reactive control systems constrained by commercial game engines, or high-level representations of reasoning models that are difficult to implement in virtual worlds. CEAA builds on established frameworks such as the Sense-Think-Act paradigm and the Belief-Desire-Intention (BDI) cognitive model, among others, and aims to provide a reusable implementation-oriented framework as a template for deploying IVA brains in interactive 3D computing systems.

The architecture is separated into three layers and twelve interconnected components. The first layer is the User and Environment Layer where users, agents, and virtual materials are situated. The second layer is the Knowledge Layer, responsible for capturing and maintaining an up-to-date log and state of the environment, keeping track of all events and storing them in a meaningful structure. The last layer is the Agent Layer, responsible for enabling the virtual agent to perform human-like reasoning and decision-making to act in the virtual environment.

The components of the architecture include: an Environment component representing the external virtual world; a Knowledge Base implemented as a shared blackboard between agents; a Sense component for selective perception; a Memory component and associated Memory Processor for experience management; a Think component for cognitive orchestration; a Cognitive Construct reasoner operating over Belief, Desire, and Intention structures; a Reasoner to perform informed decisions based on the Cognitive Constructs; a Planner for generating action sequences; an Agent Capabilities model defining executable affordances; a Behavior Mapper for grounding abstract actions; and an Act component responsible for executing behaviors within the virtual environment.

The Environment component provides stimuli to agents but performs no reasoning. Events are captured through event-driven publish–subscribe mechanisms, if–else statements, or symbolic AI methods, then converted into structured information and passed to the Shared Knowledge Base. The Knowledge Base maintains a structured record of environmental events and current world state, functioning as a centralized shared knowledge structure based on the blackboard architecture paradigm. The Sense component enables the agent to sense information through sensors acting on the Knowledge Base, evaluating events based on criteria of interest such as roles, beliefs, desires, and intentions. The Memory component stores and organizes the agent's past experiences, encompassing episodic information, semantic knowledge, and current understanding of the world. The Think component serves as the central cognitive coordinator, integrating information from the Memory component and communicating with the Reasoner and Planner components to decide appropriate actions. The Cognitive Construct component comprises the agent's Beliefs, Desires, and Intentions, representing core internal mental states that shape behavior. The Reasoner operates over the Cognitive Construct component, retrieving beliefs, desires, and intentions to generate dynamic structures determining whether an agent is interested in a problem. The Planner transforms beliefs, desires, and intentions into structured sequences of actions. The Act component retrieves decisions from the Think component and communicates with the Behavior Mapper to plan how to perform actions based on agent capabilities. The Behavior Mapper transforms abstract planned actions into concrete executable behaviors, translating symbolic action descriptions into specific behavior specifications. The Agent Capabilities component defines the set of actions an agent can perform, including animations, behaviors, body movements, locomotion, facial expressions, voice recordings, gaze, and gestures.

CEAA informed the implementation of pedagogical-agent-driven learning environments (two identical environments: one for computer desktops and one for VR) developed in Unity3D, representing a virtual museum dedicated to ENIAC and early computing history. The museum contained learning materials, an interactive three-dimensional ENIAC representation, and four embodied pedagogical agents with different responsibilities. The agents served as navigators and instructors, delivering presentations and supporting learning as evaluators. The implementation was evaluated through a mixed-methods comparative study involving 92 undergraduate Computing and Engineering students. Results provided evidence that VR and desktop implementations supported statistically significant knowledge acquisition, with positive pre-test-to-post-test gains in both conditions. Technology acceptance was also positive in both conditions. The findings demonstrate that a multi-agent, knowledge-aware and embodied learning system could be implemented and used successfully in a realistic educational activity.

The primary contribution of CEAA is an implementation-oriented architectural framework for deploying IVAs that exhibit cognitive, goal-directed, adaptive, and embodied behaviour in real-time virtual environments. It addresses the persistent gap between cognitive agent models and their practical deployment by translating established theoretical concepts into modular components that can be implemented within interactive 3D systems. CEAA integrates multiple approaches into a development-oriented framework that explicitly maps cognitive processes to deployable IVA components. Its contribution lies not in proposing a new reasoning model, but in fusing established frameworks into a coherent implementation structure. CEAA decomposes cognition and decision-making into concrete, interconnected modules that can be mapped to complex real-time systems, including VR applications, serious games, Metaverse applications, and CPSS. CEAA also integrates cognition and embodiment within a single decision-making process in which both have equal importance, embedding behavior expression within the decision-making pipeline. It provides a reusable and extensible reference model rather than a domain-specific solution, functioning both as a theoretical framework and as a generalizable architectural backbone.

The architecture's limitations include that it remains at a conceptual and theoretical level, requiring further studies to benchmark its capabilities against other architectures across dimensions including latency, ease of development, and user experience. Future work focuses on the implementation of extension tools for Unity and Unreal that implement this architecture and effectively ease the development of agents.

Improvements to AI Systems Based on CEAA:

1. Modular Cognitive-Embodied Pipeline for Real-Time Agents

• Implement a three-layer architecture (User/Environment, Knowledge, Agent) with twelve decoupled components (Sense, Think, Act, Memory, Planner, etc.) as a reusable template.

• The improved AI system can run in real-time 3D environments (VR, games, digital twins) without sacrificing high-level reasoning, by separating perception, memory, and action execution into independently scalable modules.

2. Shared Blackboard Knowledge Base for Multi-Agent Coordination

• Use a centralized, event-driven blackboard that logs all environmental changes and agent actions in a structured format.

• The improved AI system enables multiple agents to share situational awareness, coordinate tasks (e.g., one agent navigates, another evaluates), and avoid redundant sensing, leading to efficient team behavior in collaborative or competitive scenarios.

3. Selective Perception via Interest-Based Filtering

• Implement a Sense component that filters environmental events based on agent roles, beliefs, desires, and intentions.

• The improved AI system reduces computational load and noise by only processing stimuli relevant to the agent’s current goals, improving response latency and decision accuracy in dynamic environments.

4. Experience-Driven Memory with Episodic and Semantic Stores

• Integrate a Memory component that separates episodic (past events), semantic (world knowledge), and working memory, managed by a Memory Processor.

• The improved AI system can recall past interactions, generalize from experiences, and adapt its behavior over time, enabling personalized responses and long-term learning without retraining.

5. BDI-Based Cognitive Construct for Goal-Directed Reasoning

• Embed Belief-Desire-Intention structures as a core reasoner that dynamically updates based on new information.

• The improved AI system can deliberate over conflicting goals, revise intentions when beliefs change, and exhibit human-like persistence or abandonment of tasks, making it robust in uncertain or changing environments.

6. Planner-to-Behavior Mapping with Capability Constraints

• Use a Planner to generate abstract action sequences, then a Behavior Mapper to ground them into executable animations, gestures, or locomotion based on an Agent Capabilities model.

• The improved AI system can execute complex multi-step tasks (e.g., giving a guided tour, assembling an object) while respecting physical embodiment constraints (e.g., reach, speed, available animations), ensuring believable and feasible actions.

7. Unified Decision-Making with Embodiment Integration

• Fuse cognitive reasoning and physical expression into a single pipeline, where behavior expression is part of the decision process, not an afterthought.

• The improved AI system can simultaneously reason about what to do and how to do it (e.g., choosing a gesture to emphasize a point), producing more natural, context-aware interactions in embodied settings like virtual assistants or training simulators.

8. Implementation-Oriented Framework for Rapid Deployment

• Provide a reference architecture that maps directly to game engines (e.g., Unity, Unreal) with plug-and-play components.

• The improved AI system can be deployed by developers without deep AI expertise, reducing time-to-market for interactive applications (e.g., educational VR, serious games, Metaverse avatars) while maintaining cognitive depth.

9. Event-Driven Stimulus Conversion for Heterogeneous Inputs

• Standardize event capture (publish-subscribe, if-else, symbolic AI) into a uniform structured format for the Knowledge Base.

• The improved AI system can seamlessly integrate diverse sensor data (e.g., user input, physics collisions, voice commands) into a single reasoning loop, enabling cross-modal understanding and response.

10. Benchmarkable and Extensible Architecture for Future Research

• Use CEAA as a baseline to test latency, development ease, and user experience against other architectures.

• The improved AI system can be systematically evaluated and extended with new components (e.g., emotion models, learning modules) without redesigning the core, fostering iterative advancement in cognitive embodied AI.

The development of embodied Intelligent Virtual Agents (IVAs) that have cognitive capabilities in real-time interactive virtual environments remains a challenge, even with today's advancements in technology. Existing architectures are often focused on either the implementation of low-level reactive control systems that are constrained by commercial game engines, or high-level representations of reasoning models that can be difficult to implement in virtual worlds. This paper builds on that notion and proposes a modular cognitive architecture for deploying embodied IVAs. This architecture builds on existing, pre-established frameworks such as the Sense-Think-Act paradigm and the Belief-Desire-Intention cognitive model, among others, and aims to provide a reusable implementation-oriented framework as a template for deploying IVA "brains" in interactive 3D computing systems. The proposed architecture contributes by providing a modular, implementation-oriented framework for the deployment of embodied, cognitive-capable IVAs and bridges the gap between high-level agent reasoning models with real-time embodied execution, for scalable, adaptive, and explainable agents in complex interactive virtual environments.