Saturday mornings always bring some deep thoughts. This week I'm thinking about memory, and how to store and retrieve patterns with spiking network nature.
Every data scientist must be familiar with the MNIST dataset. So let's take a network trained with MNIST. It can recognize digits quite well. In this limited model, numbers are the whole world for the model. It sees them everywhere. It's very familiar with them, recognition is a mundane task.
But what if we present to this model a new symbol? For example, a star. This symbol is a supernova for the model. Most likely it will falsely classify it as 4. But what should it do? It should quickly understand that it cannot match the star to anything known and start a process of remembering.
To make the new symbol into memory, special neurotransmitters must be released, which confirms that the stimuli look for a new way. Will it put the star in a random space in memory? Either way, when this is done, dopamine will be released, and everyone will be happy.
For every new pattern, it will search for an empty location. It will be a random place for every instance of this model. And you probably think that this is what makes all individuals different because the learning process is unpredictable.
Actually, no, it is predictable. Because patterns do not take a new space randomly out of all empty ones. The star will find its way to a place where similar patterns are stored. This will ignite some old memories which we call associations, and it will activate similar neurotransmitters.
Memories heavily vary from person to person based on their previous experience. If that area in memory is sad, then the star will be considered as sad experience. Also, the star will share some cells with other patterns, some spiky ones, like number 4 or 7. And this is how the learning process that takes random spots can create a stable system with predictable inference results.
Hopefully, not everything is sad in our lives. When we witness something unexpected, it will be seen with extreme delight. That’s why some art could be boring because it is associated with mundane stuff, but we very much appreciate even some strange abstract splashes because they represent new textures not related to our previous experience, and that excites us.