Exploiting heterogeneous delays for efficient computation in low-bit neural networks

Authors: Pengfei Sun, Dan Goodman, Danyal Akarca

Presentation type: Flash talk at SNUFA 2025 online workshop (5-6 Nov 2025)

Abstract

Neural networks rely on learning neural weights. However, this overlooks other neural parameters that can also be learned and may be utilised by the brain. One such parameter is the delay: the brain exhibits complex temporal dynamics with heterogeneous delays, where signals are transmitted asynchronously between neurons. It has been theorised that this delay heterogeneity, rather than a cost to be minimised, can be exploited in embodied contexts where task-relevant information naturally sits contextually in the time domain. We test this hypothesis by training spiking neural networks to modify not only their weights but also their delays. We find that delay heterogeneity enables state-of-the-art performance on temporally complex neuromorphic problems and can be achieved even when weights are extremely imprecise (1.58-bit ternary precision: just positive, negative, or absent). This parameter efficiency of requiring fewer parameters at lower precision, enables performant solutions highly compressible relative to comparable weight-only networks. We show how delays and time-constants trade-off, with performance depending on a small number of longer delays that provide favorable scaling behavior. Our results suggest temporal heterogeneity is an important principle for efficient computation, particularly when task-relevant information is temporal, as in the physical world.