Train a spiking neural network#

In this project you will develop a Spiking Convolutional Neural Network to solve a simple classification problem. You are encouraged to use Sinabs, to be able to load your network to the Speck chip.

The data consist of Speck recordings where objects of each class have been recorded using the Dynamic Vision Sensor of speck2f.

In this task, the data consist of 5 classes (apple, car, flower, dog, ship). To reduce time consumption, the data are already given in tensor format. Each calss’s data are stored in the file {class_name}_tensor.pt (for example car_tensor.pt). Please note that the data are not in sparse format. However, they do follow the address event representation framework.

Due to computational power and memory limitations you are advised to start your exploration with 2 or 3 classes and only a subset of each recording

#### Task 3.0: Load the data

apple_file, _ = urlretrieve("https://github.com/ncskth/phd-course/raw/main/book/module4/apple_tensor.pt")
apple_events = torch.load(apple_file)

car_file, _ = urlretrieve("https://github.com/ncskth/phd-course/raw/main/book/module4/car_tensor.pt")
car_events = torch.load(car_file)

apple = events_to_channels(apple_events)
car = events_to_channels(car_events)

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[2], line 3
      1 #### Task 3.0: Load the data
----> 3 apple_file, _ = urlretrieve("https://github.com/ncskth/phd-course/raw/main/book/module4/apple_tensor.pt")
      4 apple_events = torch.load(apple_file)
      6 car_file, _ = urlretrieve("https://github.com/ncskth/phd-course/raw/main/book/module4/car_tensor.pt")

NameError: name 'urlretrieve' is not defined

Visualize the data#

Using the functions provided in previous tasks, visualize your input data. You can start by visualizing the first 10 seconds of the data.

# Visualize your data

# ...

Create the dataloader#

To be able to load the data to train the network, you should create a dataloader from the given data. In the cell bellow, you have to fill in the missing lines to create your dataloader

You might find useful the random_split function and the DataLoader module.

To create the dataset, we split the recording into pieces of sample_duration duration, so called sample. Each of these samples are one input of the network, for which the network will have to identify what is shown in the input signal.

We create the labels by creating tensors of zeros and ones (for the two classes) of size equal to the number of samples of each class.

class SpeckDataset(Dataset):
    def __init__(self, frames, targets, transform=None, target_transform=None):
        self.targets = targets
        self.frames = frames
        self.transform = transform
        self.target_transform = target_transform

    def __len__(self):
        return len(self.targets)

    def __getitem__(self, idx):
        frames = self.frames[idx]
        label = self.targets[idx]
        if self.transform:
            image = self.transform(image)
        if self.target_transform:
            label = self.target_transform(label)
        return frames, label


sample_duration = 1   # change this to suit your experiment. Higher value will result in longer sequence (video) for each input data point (sample) but less samples overall
car_samples = car.shape[0]//sample_duration
apple_samples = apple.shape[0]//sample_duration

train_perc = 0.8
batch_size = 10

c = car[:sample_duration*car_samples].reshape((car_samples, sample_duration, *car.shape[1:]))
a = apple[:sample_duration*apple_samples].reshape((apple_samples, sample_duration, *apple.shape[1:]))

c_t = torch.zeros(c.shape[0])
a_t = torch.zeros(a.shape[0])+1

data = torch.cat((c, a), dim=0)
targets = torch.cat((c_t, a_t), dim=0)


# Create the dataset using the SpeckDataset module
# dataset = ...

# Split the data to trainset and testset
# trainset, testset =

# Define the trainloader and testloader DataLoaders
# ...

Develop your network#

Use the torch sequential and the sl.IAFSqueeze() modules to develop your network. Be aware os the speck’s architecture regarding the size of the network.

snn_bptt = nn.Sequential(
    # ...
)

Train your network#

You should implement your training and testing loop. Fill in the train and test function and evaluate the result.

You are advised to use Google Colab’s GPU resources to accelerate the training.

Note that the sl.IAFSqueeze() module has the peculiarity that it cannot process data in the form of [Batch, Time, Channel, Height, Width], but you first have to reshape your input data to the form [Batch x Time, Channel, Height, Width]. This is equivalent to stacking one sample after the other. Be aware that the output will have the same form as the input tensor, so you will have to reshape the output back to [Batch, Time, ...].

Your output labels are not in the form of your target labels! Your decision on how you handle your output sequences to retrieve your output labels could affect the performance of your network.

# define the functions' signatures (parameters and return)
def train():
  for data, label in tqdm.tqdm(trainloader):
    # reshape the input from [Batch, Time, Channel, Height, Width] into [Batch*Time, Channel, Height, Width]
    data = data.reshape(-1, 2, 128, 128)

    # fill in the rest of the function




def test():
  with torch.no_grad():
    for data, label in tqdm.tqdm(testloader):
      # reshape the input from [Batch, Time, Channel, Height, Width] into [Batch*Time, Channel, Height, Width]
      data = data.reshape(-1, 2, 128, 128)

      # fill in the rest of the function

# Define the optimizer and the loss function
# optimizer = ...
# criterion = ...

# Train your network