Spiking Convolutional Neural Network in Speck#
In this section you will learn about Speck, a neuromorphic system that integrates a Dynamic Vision Sensor and a Spiking Convolutional Neural Network on device. In the following sections, you will learn how to interact with the Speck data, visualize them and how to train a network that can be loaded to Speck.
You will train an SNN to solve a simple task and load it in Speck. You will then be able to run the network on inference and visualize its response in real time.
Speck device#
Speck is a neuromorphic device with characteristics in hardware and Samna has developed the software that we’ll see
Hardware#
Speck’s architecture consist of a Dynamic Vision Sensor and a Spiking Convolutional Neural Network with the following characteristics
DVS input of 128x128 resolution
9 convolutional layers
Implementation of Integrate and Fire neurons
Leak through bias and external clock
Software#
Synsense provides software libraries to develop neural networks and interact with Speck.
Communication with Speck relies on the Samna library.
Simulator sinabs-dynapcnn used to train models offline and load them to Speck.
Interaction with Speck#
In this section, we will see how to compute using speck input data, we will visualize events and develop a network for edge detection that we’ll load to speck.
Task 0: Import libraries and data#
We will start our preparation by importing the libraries necessary and the data for our tasks
Task 0.1: Import libraries#
We begin by importing the necessary libraries for our implementation, such as samna, sinabs, torch and libraries to help us fetch and vsualize our data.
!pip install -q samna
!pip install -q sinabs
import samna
import torch
import pickle
import tqdm
from matplotlib.animation import FuncAnimation
import matplotlib.pyplot as plt
from IPython.display import HTML
from torch.utils.data import DataLoader, Dataset, random_split
from urllib.request import urlretrieve
import sinabs.layers as sl
from torch import nn
from sinabs.activation.surrogate_gradient_fn import PeriodicExponential
from operator import iconcat
from functools import reduce
import pickle
Looking in indexes: https://gitlab.com/api/v4/projects/27423070/packages/pypi/simple
Collecting samna==0.38.5.0
Downloading https://gitlab.com/api/v4/projects/27423070/packages/pypi/files/00c6ac6fc03fecfb67fc33a1700b69e47eaf16ddaa700152354260b94680fb56/samna-0.38.5.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (141.6 MB)
?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/141.6 MB ? eta -:--:--
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.1/141.6 MB 4.7 MB/s eta 0:00:30
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.3/141.6 MB 5.0 MB/s eta 0:00:29
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.7/141.6 MB 6.7 MB/s eta 0:00:22
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/141.6 MB 9.9 MB/s eta 0:00:15
╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.2/141.6 MB 12.7 MB/s eta 0:00:12
━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.0/141.6 MB 19.5 MB/s eta 0:00:08
━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.3/141.6 MB 30.3 MB/s eta 0:00:05
━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.5/141.6 MB 46.5 MB/s eta 0:00:03
━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.6/141.6 MB 105.1 MB/s eta 0:00:02
━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.1/141.6 MB 118.0 MB/s eta 0:00:02
━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 24.9/141.6 MB 151.2 MB/s eta 0:00:01
━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 29.4/141.6 MB 169.3 MB/s eta 0:00:01
━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 32.8/141.6 MB 142.3 MB/s eta 0:00:01
━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━ 37.8/141.6 MB 118.8 MB/s eta 0:00:01
━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━ 43.2/141.6 MB 153.7 MB/s eta 0:00:01
━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━ 48.8/141.6 MB 161.8 MB/s eta 0:00:01
━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━ 54.0/141.6 MB 158.3 MB/s eta 0:00:01
━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━ 59.3/141.6 MB 155.1 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━ 64.8/141.6 MB 159.1 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━ 70.7/141.6 MB 167.2 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━ 76.1/141.6 MB 161.9 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━ 81.8/141.6 MB 165.0 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━ 87.6/141.6 MB 167.8 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━ 93.0/141.6 MB 164.8 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━ 94.0/141.6 MB 114.2 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━ 98.7/141.6 MB 105.3 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━ 104.3/141.6 MB 151.1 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━ 109.8/141.6 MB 163.9 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 115.3/141.6 MB 160.0 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━ 120.7/141.6 MB 160.1 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━ 126.6/141.6 MB 165.3 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 132.3/141.6 MB 170.6 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺ 138.3/141.6 MB 173.1 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 141.6/141.6 MB 180.5 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 141.6/141.6 MB 180.5 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 141.6/141.6 MB 180.5 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 141.6/141.6 MB 180.5 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 141.6/141.6 MB 180.5 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 141.6/141.6 MB 180.5 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 141.6/141.6 MB 180.5 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 141.6/141.6 MB 180.5 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 141.6/141.6 MB 180.5 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 141.6/141.6 MB 180.5 MB/s eta 0:00:01
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 141.6/141.6 MB 24.6 MB/s eta 0:00:00
?25h
Installing collected packages: samna
Attempting uninstall: samna
Found existing installation: samna 0.38.5.0
Uninstalling samna-0.38.5.0:
Successfully uninstalled samna-0.38.5.0
Successfully installed samna-0.38.5.0
---------------------------------------------------------------------------
ModuleNotFoundError Traceback (most recent call last)
Cell In[1], line 7
5 import torch
6 import pickle
----> 7 import tqdm
8 from matplotlib.animation import FuncAnimation
9 import matplotlib.pyplot as plt
ModuleNotFoundError: No module named 'tqdm'
Task 0.2: Fetch data#
Now we will fetch the data files that we will use for our next experiment.
You will notice that the format of the spikes is not the tensor format that has been previously discussed. What is this new format?
data_file, _ = urlretrieve("https://github.com/ncskth/phd-course/raw/main/book/module4/edge_detect")
with open(data_file, 'rb') as f:
data = pickle.load(f)
print(data[0])
Task 1: Visualize DVS output#
In this task you are asked to visualize your input data. Before working with the data, we should convert them in a friendly format (a tensor).
Note that we bin events every 10000μs (10ms). You will need this information when constructing the video from events.
def events_to_channels(events, time_interval = 1000):
frames = []
index = 0
current_frame = torch.zeros(2, 128, 128)
for (t, y, x, p) in tqdm.tqdm(events.int()):
if t // time_interval > index:
frames.append(current_frame.clone())
current_frame.fill_(0)
index += 1
current_frame[p, x, y] = +1
return torch.stack(frames)
def Speck_spikes_to_events(sp_events):
sp_events = reduce(iconcat, sp_events, [])
events = torch.zeros((len(sp_events), 4))
start = sp_events[0].timestamp
for i, e in tqdm.tqdm(enumerate(sp_events), total=len(sp_events)):
events[i][0] = e.timestamp - start
events[i][1] = e.x
events[i][2] = e.y
events[i][3] = e.feature
return events
# Convert Speck spikes to tensor
events = Speck_spikes_to_events(data)
events = events_to_channels(events, 10000)
Use the functions that are provided in the cell bellow to create a video of your input data.
def animate_frames(frames, figure=None, interval: int = 20, **kwargs):
if figure is None:
figure, _ = plt.subplots(**kwargs)
ax = figure.gca()
image = ax.imshow(frames[0]) # .T)
ax.set_axis_off()
def animate(index):
image.set_data(frames[index]) # .T)
return image
anim = FuncAnimation(figure, animate, frames=len(frames), interval=interval)
video = anim.to_html5_video()
html = HTML(video)
display(html)
plt.tight_layout()
plt.close()
def events_to_frames(frames, polarity: bool = True):
if len(frames.shape) == 3:
frames = frames.unsqueeze(-1).repeat(1, 1, 1, 3)
else:
if not polarity:
frames = frames.abs().sum(-1)
elif polarity:
frames = torch.concat([frames, torch.zeros(frames.shape[0], 1, *frames.shape[2:], device=frames.device)], dim=1).movedim(1, -1)
frames = ((frames / frames.max()) * 255).int().clip(0, 255)
return frames
# Visualize your input data
# ...
Task 2: Create edge detector in sinabs#
For this task you will have to create an edge detector. You will need to create a Spiking Convolutional Neural Network that detects vertical and horizontal edges.
Note that your neuros should be Integrate and Fire as Speck does not implement leak directly!
Task 2.1: Convolutional layer#
In the following cell, you have an example of an edge detector. Use this as your kernel to develop your network. What should the shape of the convolutional layer’s weights be?
kernel_size = 9
gaussian = torch.sigmoid(torch.linspace(-10, 10, kernel_size + 1))
kernel = (gaussian.diff()-0.14).repeat(kernel_size, 1)
plt.imshow(kernel)
plt.colorbar()
Task 2.2: Network implementation#
For this task, you will have to implement and run your network using the sinabs library. This library will allow you to transfer your network to the Speck chip. You can use sl.IAFSqueeze() to implement the integration of spikes.
Note: The biases should be set to None!
snn_bptt = nn.Sequential(
# [2, 128, 128] -> [2, 64, 64]
# Fill in your convolutional layer here. Note: Bias should be None!
sl.IAFSqueeze(batch_size=1, min_v_mem=-1.0),
)
out = snn_bptt(events)
# Every positive output of the network is a spike produced by the network. We threshold with out > 0 to normalize the output to 0 or 1 (spike or no spike) at each frame
print(torch.sum(out - out.int()))
out = (out > 0).float()
tensor(0.)
animate_frames(events_to_frames(out[:400], polarity=True))
Task 2.3: Visualize the output#
Using the provided function above, visualize the output. What do you observe? How can you suppress the noise?
Speck neurons can implement leak through their bias.
# Visualize the output
# ...