Keynotes | K&Ts | GACs | Talks | Posters | Search

Poster A in Poster Session A: Tuesday, August 4, 9:30 – 11:15 am, Kimmel Center, Shorin & Rosenthal Rooms

Learning Flexible Task Representations from Temporal and Sequential Regularities in Continuous Experience

Aidan Higgs¹, Angela J Langdon¹; ¹National Institutes of Health

Presenter: Aidan Higgs

Cognitive agents must represent their environment in order to learn and act appropriately to obtain their goals. In reinforcement learning, state representations provide the critical scaffolding for update-based learning rules but are themselves oftentimes assumed. In cases where they are learned, that learning is typically over the rigid structure of classic Markov decision processes, forcing environmental variability and sequential dependencies to be captured through increased size and complexity of the state transition model as a whole. However, from an agent’s perspective, experience during learning often contains characteristic and reliable structure in time: outcomes follow cues at reliable intervals and task epochs persist for predicable lengths of time. We present a perspective on state representation learning in which agents exploit temporal and sequential regularities in their environment to learn compositional task representations that are both efficient and flexible. Using a semi-Markov framework, agents infer hidden task-states using a world model that explicitly includes state dwell-time distributions along with state transition probabilities. We derive learning rules for both the duration of task-states and their transition probabilities, allowing for tandem online adaptation of these properties to the intervals between and sequence of observations during a task. We demonstrate the utility of this approach in simulated task settings, showing that adaptation to temporal structure enables cognitive agents to develop succinct and task-aligned state representations even in conditions of high observation uncertainty. Further, we show how bidirectional interactions between temporal and sequential learning mechanisms explain divergent adaptations in state representation in a non-stationary environment, despite algorithmic invariance in the model. Our findings reveal the centrality of temporal structure for the formation and evolution of cognitively-interpretable task-state representations direct from continuous experience, and its role in the emergence of distinct representational approaches to the same sequence of events during cognitive tasks.

Topic Area: Memory, Learning & Knowledge Structures