A hierarchy of associations in hippocampo-cortical systems: cognitive maps and navigation strategies

In this letter we describe a hippocampo-cortical model of spatial processing and navigation based on a cascade of increasingly complex associative processes that are also relevant for other hippocampal functions such as episodic memory. Associative learning of different types and the related pattern encoding-recognition take place at three successive levels: (1) an object location level, which computes the landmarks from merged multimodal sensory inputs in the parahippocampal cortices; (2) a subject location level, which computes place fields by combination of local views and movement-related information in the entorhinal cortex; and (3) a spatiotemporal level, which computes place transitions from contiguous place fields in the CA3-CA1 region, which form building blocks for learning temporospatial sequences.At the cell population level, superficial entorhinal place cells encode spatial, context-independent maps as landscapes of activity; populations of transition cells in the CA3-CA1 region encode context-dependent maps as sequences of transitions, which form graphs in prefrontal-parietal cortices. The model was tested on a robot moving in a real environment; these tests produced results that could help to interpret biological data. Two different goal-oriented navigation strategies were displayed depending on the type of map used by the system.Thanks to its multilevel, multimodal integration and behavioral implementation, the model suggests functional interpretations for largely unaccounted structural differences between hippocampo-cortical systems. Further, spatiotemporal information, a common denominator shared by several brain structures, could serve as a cognitive processing frame and a functional link, for example, during spatial navigation and episodic memory, as suggested by the applications of the model to other domains, temporal sequence learning and imitation in particular.     

A Natural Feature Representation for Unstructured Environments

This paper addresses the long-standing problem of feature representation in the natural world for autonomous navigation systems. The proposed representation combines Isomap, which is a nonlinear manifold learning algorithm, with expectation maximization, which is a statistical learning scheme. The representation is computed off-line and results in a compact, nonlinear, non-Gaussian sensor likelihood model. This model can be easily integrated into estimation algorithms for navigation and tracking. The compactness of the model makes it especially attractive for deployment in decentralized sensor networks. Real sensory data from unstructured terrestrial and underwater environments are used to demonstrate the versatility of the computed likelihood model. The experimental results show that this approach can provide consistent models of natural environments to facilitate complex visual tracking and data-association problems.      

Simultaneous learning of perception and action in mobile robots

One of the main problems of robots is the lack of adaptability and the need for adjustment every time the robot changes its working place. To solve this, we propose a learning approach for mobile robots using a reinforcement-based strategy and a dynamic sensor-state mapping. This strategy, practically parameterless, minimises the adjustments needed when the robot operates in a different environment or performs a different task.Our system will simultaneously learn the state space and the action to execute on each state. The learning algorithm will attempt to maximise the time before a robot failure in order to obtain a control policy suited to the desired behaviour, thus providing a more interpretable learning process. The state representation will be created dynamically, starting with an empty state space and adding new states as the robot finds new situations that has not seen before. A dynamic creation of the state representation will avoid the classic, error-prone and cyclic process of designing and testing an ad hoc representation. We performed an exhaustive study of our approach, comparing it with other classic strategies. Unexpectedly, learning both perception and action does not increase the learning time.     

融合对比预测编码的深度双Q网络

在模型未知的部分可观测马尔可夫决策过程（partially observable Markov decision process,POMDP）下,智能体无法直接获取环境的真实状态,感知的不确定性为学习最优策略带来挑战。为此,提出一种融合对比预测编码表示的深度双Q网络强化学习算法,通过显式地对信念状态建模以获取紧凑、高效的历史编码供策略优化使用。为改善数据利用效率,提出信念回放缓存池的概念,直接存储信念转移对而非观测与动作序列以减少内存占用。此外,设计分段训练策略将表示学习与策略学习解耦来提高训练稳定性。基于Gym-MiniGrid环境设计了POMDP导航任务,实验结果表明,所提出算法能够捕获到与状态相关的语义信息,进而实现POMDP下稳定、高效的策略学习。     

Temporal BYY encoding, Markovian state spaces, and space dimension determination.

As a complementary to those temporal coding approaches of the current major stream, this paper aims at the Markovian state space temporal models from the perspective of the temporal Bayesian Ying-Yang (BYY) learning with both new insights and new results on not only the discrete state featured Hidden Markov model and extensions but also the continuous state featured linear state spaces and extensions, especially with a new learning mechanism that makes selection of the state number or the dimension of state space either automatically during adaptive learning or subsequently after learning via model selection criteria obtained from this mechanism. Experiments are demonstrated to show how the proposed approach works.     

Continuous state/action reinforcement learning: A growing self-organizing map approach

This paper proposes an algorithm to deal with continuous state/action space in the reinforcement learning (RL) problem. Extensive studies have been done to solve the continuous state RL problems, but more research should be carried out for RL problems with continuous action spaces. Due to non-stationary, very large size, and continuous nature of RL problems, the proposed algorithm uses two growing self-organizing maps (GSOM) to elegantly approximate the state/action space through addition and deletion of neurons. It has been demonstrated that GSOM has a better performance in topology preservation, quantization error reduction, and non-stationary distribution approximation than the standard SOM. The novel algorithm proposed in this paper attempts to simultaneously find the best representation for the state space, accurate estimation of Q-values, and appropriate representation for highly rewarded regions in the action space. Experimental results on delayed reward, non-stationary, and large-scale problems demonstrate very satisfactory performance of the proposed algorithm.     

广义的以物体为中心的行程编码：一种新的三维物体模型

三维物体的表达和识别是图象理解和场景分析的核心问题,三维模型在三维物体的识别和场景分析中具有十分重要的作用.三维模型应该是以物体为中心的,能够提供该场景的所有有用信息.物体的大小,形状及朝向应均可从该模型中提取得到.本文提出了一种新的三维物体模型——广义的以物体为中心的行程编码.它包括物体的GORC物理数据结构,详细的形状描述和抽象描述.物体的高层次的表达可以通过以GORC编码的物理数据直接提取得到.三维的GORC是二维的以物体为中心的行程编码在三维上的推广,它兼有物体的体积表达和表面表达的优点.三维物体的GORC模型可以很容易地由其深度信息构造得出,基于GORC的投影运算,图象代数运算以及特征提取均可非常有效地实现.     

A neural network model for the acquisition of a spatial body scheme through sensorimotor interaction

This letter presents a novel unsupervised sensory matching learning technique for the development of an internal representation of three-dimensional information. The representation is invariant with respect to the sensory modalities involved. Acquisition of the internal representation is demonstrated with a neural network model of a sensorimotor system of a simple model creature, consisting of a tactile-sensitive body and a multiple-degrees-of-freedom arm with proprioceptive sensitivity. Acquisition of the 3D representation as well as a distributed representation of the body scheme, occurs through sensorimotor interactions (i.e., the sensory-motor experience of the creature). Convergence of the learning is demonstrated through computer simulations for the model creature with a 7-DoF arm and a spherical body covered by 20 tactile fields.     

基于目标导向行为和空间拓扑记忆的视觉导航方法

针对在具有动态因素且视觉丰富环境中的导航问题,受路标机制空间记忆方式启发,提出一种可同步学习目标导向行为和记忆空间结构的视觉导航方法.首先,为直接从原始输入中学习控制策略,以深度强化学习为基本导航框架,同时添加碰撞预测作为模型辅助任务;然后,在智能体学习导航过程中,利用时间相关性网络祛除冗余观测及寻找导航节点,实现通过情景记忆递增描述环境结构;最后,将空间拓扑地图作为路径规划模块集成到模型中,并结合动作网络用于获取更加通用的导航方法.实验在3D仿真环境DMlab中进行,实验结果表明,本文方法可从视觉输入中学习目标导向行为,在所有测试环境中均展现出更高效的学习方法和导航策略,同时减少构建地图所需数据量;而在包含动态堵塞的环境中,该模型可使用拓扑地图动态规划路径,从而引导绕路行为完成导航任务,展现出良好的环境适应性.     

Building, visualizing, and computing on surfaces of evolution

An approach is presented to giving a robot the ability to move safely through a scene using its own vision that depends, as in humans, on the ability to operate explicitly in both space and time and to exploit the massive redundancy present in the hundreds of views that can be obtained when moving through a scene. The mechanism for integrating these space-time factors is a 3-D surface-building process called the Weaving Wall. In robotic navigation work the 3-D surfaces built by its process represent the space-time evolution of scene images, and this representation, in conjunction with geometric constraints, enables the 3-D structure of the scene to be determined. In other domains where there is a gradual evolution of data over a third dimension (e.g. medical tomography), the surfaces constructed by the Weaving Wall are immediately of value for their topographic structure. The designs of both the surface-building and scene-reconstruction processes make them well suited for real-time operation, given appropriate hardware     

Self-organizing continuous attractor networks and path integration: one-dimensional models of head direction cells

Some neurons encode information about the orientation or position of an animal, and can maintain their response properties in the absence of visual input. Examples include head direction cells in rats and primates, place cells in rats and spatial view cells in primates. 'Continuous attractor' neural networks model these continuous physical spaces by using recurrent collateral connections between the neurons which reflect the distance between the neurons in the state space (e.g. head direction space) of the animal. These networks maintain a localized packet of neuronal activity representing the current state of the animal. We show how the synaptic connections in a one-dimensional continuous attractor network (of for example head direction cells) could be self-organized by associative learning. We also show how the activity packet could be moved from one location to another by idiothetic (self-motion) inputs, for example vestibular or proprioceptive, and how the synaptic connections could self-organize to implement this. The models described use 'trace' associative synaptic learning rules that utilize a form of temporal average of recent cell activity to associate the firing of rotation cells with the recent change in the representation of the head direction in the continuous attractor. We also show how a nonlinear neuronal activation function that could be implemented by NMDA receptors could contribute to the stability of the activity packet that represents the current state of the animal.     

Parametric POMDPs for planning in continuous state spaces

This work addresses the problem of decision-making under uncertainty for robot navigation. Since robot navigation is most naturally represented in a continuous domain, the problem is cast as a continuous-state POMDP. Probability distributions over state space, or beliefs, are represented in parametric form using low-dimensional vectors of sufficient statistics. The belief space, over which the value function must be estimated, has dimensionality equal to the number of sufficient statistics. Compared to methods based on discretising the state space, this work trades the loss of the belief space’s convexity for a reduction in its dimensionality and an efficient closed-form solution for belief updates. Fitted value iteration is used to solve the POMDP. The approach is empirically compared to a discrete POMDP solution method on a simulated continuous navigation problem. We show that, for a suitable environment and parametric form, the proposed method is capable of scaling to large state-spaces.     

A consideration of human immunity-based reinforcement learning with continuous states

Many reinforcement learning methods have been studied on the assumption that a state is discretized and the environment size is predetermined. However, an operating environment may have a continuous state and its size may not be known in advance, e.g., in robot navigation and control. When applying these methods to the environment described above, we may need a large amount of time for learning or failing to learn. In this study, we improve our previous human immunity-based reinforcement learning method so that it will work in continuous state space environments. Since our method selects an action based on the distance between the present state and the memorized action, information about the environment (e.g., environment size) is not required in advance. The validity of our method is demonstrated through simulations for the swingup control of an inverted pendulum.     

Length Estimation in 3-D Using Cube Quantization

Estimators for the original length of a continuous 3-D curve given its digital representation are developed. The 2-D case has been extensively studied. The few estimators that have been suggested for 3-D curves suffer from serious drawbacks, partly due to incomplete understanding of the characteristics of digital representation schemes for 3-D curves.The selection and thorough understanding of the digital curve representation scheme is crucial to the design of 3-D length estimators. A comprehensive study on the digitization of 3-D curves was recently carried out. It was shown that grid intersect quantization and other 3-D curve discretization schemes that lead to 26-directional chain codes do not satisfy several fundamental requirements, and that cube quantization, that leads to 6-directional chain codes, should be preferred.The few 3-D length estimators that have been suggested are based on 26-directional chain coding that naturally provides a classification of the chain links, which is necessary for accurate length estimation. Cube quantization is mathematically well-behaved but the symmetry and uniformity of the 6-directional digital chain elements create a challenge in their classification for length estimation.In this paper length estimators for 3-D curves digitized using cube quantization are developed. Simple but powerful link classification criteria for 6-directional digital curves are presented. They are used to obtain unbiased length estimators, with RMS errors as low as 0.57% for randomly oriented straight lines.     

1D representation of Isomap for united video coding

This paper proposes a 1D representation of isometric feature mapping (Isomap) based united video coding algorithms. First, 1D Isomap representations that maintain distances are generated which can achieve a very high compression ratio. Next, embedding and reconstruction algorithms for the 1D Isomap representation are presented that can transform samples from a high-dimensional space to a low-dimensional space and vice versa. Then, dictionary learning algorithms for training samples are proposed to compress the input samples. Finally, a unified coding framework for diverse videos based on a 1D Isomap representation is built. The proposed methods make full use of correlations between internal and external videos, which are not considered by classical methods. Simulation experiments have shown that the proposed methods can obtain higher peak signal-to-noise ratios than standard highly efficient video coding for similar bit per pixel levels in the low bit rate situation.     

连续状态自适应离散化基于K-均值聚类的强化学习方法

使用聚类算法对连续状态空间进行自适应离散化．得到了基于K-均值聚类的强化学习方法．该方法的学习过程分为两部分：对连续状态空间进行自适应离散化的状态空间学习，使用K-均值聚类算法；寻找最优策略的策略学习．使用替代合适迹Sarsa学习算法．对连续状态的强化学习基准问题进行仿真实验，结果表明该方法能实现对连续状态空间的自适应离散化，并最终学习到最优策略．与基于CMAC网络的强化学习方法进行比较．结果表明该方法具有节省存储空间和缩短计算时间的优点．     

A description of dynamic behavior of sensory/motor systems with fuzzy symbolic dynamic systems

This paper presents a model of continuous sensory/motor systems for autonomous agents in naviga-tion problems. The Markov environmental model and sequential plan are extended with fuzzy sets, which present the mathematical transformation from discrete state space to continuous state space. The extended fuzzy environmental model and fuzzy sequential knowledge allow the identification of continuous sensory/motor systems with a gradient descent-based parameter estimation algorithm. A simulation demonstrates the feasibility of the proposed method. This work was presented, in part, at the Fourth International Symposium on Artificial Life and Robotics, Oita, Japan, January 19–22, 1999     

一种基于海马认知机理的仿生机器人认知地图构建方法

海马结构空间细胞的放电活动被认为能够形成对环境内在地图的表达,即所谓的认知地图.先前的仿生环境认知地图构建方法（例如RatSLAM）以及传统的SLAM方法均缺乏足够的生理学依据,不能准确地体现出生物在导航中的生理学现象和认知功能实现过程.本文模仿海马结构空间细胞的认知机理提出了一种构建精确的环境认知地图的方法,其特点在于通过构建统一的空间细胞吸引子计算模型对自运动线索进行路径积分;网格细胞和位置细胞对环境的表达来源于条纹细胞的前向驱动作用;通过环境的颜色深度图像进行闭环检测,对空间细胞路径积分进行误差修正,最终生成精确的环境认知地图.该认知地图是一种拓扑度量地图,包含了环境特征点坐标、视觉线索以及特定位点的拓扑关系.本文通过仿真实验和机器人平台物理实验验证了方法的有效性,研究成果为仿海马认知机理的机器人导航方法研究奠定了基础.     

State representations form-D systems with generalized causality structures

This paper considers the definitions of recursive and statelike representations form-D systems modeled as operators on a partially ordered Hilbert resolution space.Using only the causality structure we develop a second-order transition representation which encompasses previously studied models. The same representation is shown to be valid for both quarter plane and arbitrary conic causality structures.Transformation of the transition representation into a first-orderm-D local state model leads to the concept of structural minimality. We develop explicit conditions which apply to both stationary and nonstationary cases.The transition representation also enables us to establish the existence of general 1-D wave advance model representations. Minimality of the wave advance model is also discussed.Supported in part by SDIO/IST and managed by ARO under Contract D24962-MA SDI.     

Is the early visual system optimised to be energy efficient?

This paper demonstrates that a representation which balances natural image encoding with metabolic energy efficiency shows many similarities to the neural organisation observed in the early visual system. A simple linear model was constructed that learned receptive fields by optimally balancing information coding with metabolic expense for an entire visual field in a 2-stage visual system. The input to the model consists of a space variant retinal array of photoreceptors. Natural images were then encoded through a bottleneck such as the retinal ganglion cells that form the optic nerve. The natural images represented by the activity of retinal ganglion cells were then encoded by many more 'cortical' cells in a divergent representation. Qualitatively, the system learnt by optimising information coding and energy expenditure and matched (1) the centre surround organisation of retinal ganglion cells; (2) the Gabor-like organisation of cortical simple cells; (3) higher densities of receptive fields in the fovea decreasing in the periphery; (4) smaller receptive fields in the fovea increasing in size in the periphery; (5) spacing ratios of retinal cells; and (6) aspect ratios of cortical receptive fields. Quantitatively, however, there are small but significant discrepancies between density slopes which may be accounted for by taking optic blur and fixation induced image statistics into account. In addition, the model cortical receptive fields are more broadly tuned than biological cortical neurons; this may be accounted for by the computational limitation of modelling a relatively low number of neurons. This paper shows that retinal receptive field properties can be understood in terms of balancing coding with synaptic energy expenditure and cortical receptive fields with firing rate energy expenditure, and provides a sound biological explanation of why 'sparse' distributions are beneficial.