We determined the temporal matched filter kernel by averaging calcium transients within a moderate SNR range; these transients likely represent the temporal response to single action potentials².a, Example data show all background-subtracted fluorescence calcium transients of all GT neurons in all videos in the ABO 275 μm dataset that showed peak SNR (pSNR) in the regime 6 < pSNR < 8 (gray). We minimized crosstalk from neighboring neurons by excluding transients during time periods when neighboring neurons also had transients. We normalized all transients such that their peak values were unity, and then averaged these normalized transients into an averaged spike trace (red). We used the portion of the average spike trace above e^–1 (blue dashed line) as the final template kernel.b, When analyzing performance on the ABO 275 μm dataset through ten-fold leave-one-out cross-validation, using the temporal kernel determined in (a) within our temporal filter scheme achieved significantly higherF₁ score than not using a temporal filter or using an unmatched filter (*P < 0.05, **P < 0.005; two-sided Wilcoxon signed-rank test,n = 10 videos) and achieved a slightly higherF₁ score than using a single exponentially decaying kernel (P = 0.77; two-sided Wilcoxon signed-rank test,n = 10 videos). Error bars are s.d. The gray dots represent scores for the test data for each round of cross-validation. The unmatched filter was a moving-average filter over 60 frames.c-d, are analogous to (a-b), but for the Neurofinder dataset. We determined the filter kernel using videos 04.01 and 04.01.test.

We explored multiple potential CNN architectures to optimize performance.a-d, Various CNN architectures having depths of (a) two, (b) three, (c) four, or (d) five. For the three-depth architecture, we also tested different numbers of skip connections, ReLU (Rectified Linear Unit) instead of ELU (Exponential Linear Unit) as the activation function, and separable Conv2D instead of Conv2D in the encoding path. The dense five-depth model mimicked the model used in UNet2Ds⁹. The legend ‘0/n_i + n_i’ represents whether the skip connection was used (n_i + n_i) or not used (0 + n_i).e, TheF₁ score and processing speed of SUNS using various CNN architectures when analyzing the ABO 275 μm dataset through ten-fold leave-one-out cross-validation. The right panel zooms in on the rectangular region in the left panel. Error bars are s.d. The legend (n₁,n₂, …,n_k) describes architectures withk-depth andn_i channels at thei^th depth. We determined that the three-depth model, (4,8,16), using one skip connection at the shallowest layer, ELU, and full Conv2D (Fig.1c), had a good trade-off between speed and accuracy; we used this architecture as the SUNS architecture throughout the paper. One important drawback of the ReLU activation function was its occasional (20% of the time) failure during training, compared to negligible failure levels for the ELU activation function.

We tested if the accuracy of SUNS when analyzing the ABO 275 μm dataset within the ten-fold leave-one-out cross-validation relied on intricate tuning of the algorithm’s hyperparameters. The evaluated training parameters included (a) the threshold of the SNR video (th_SNR) and (b) the training batch size. The evaluated post-processing parameters included (c) the threshold of probability map (th_prob), (d) the minimum neuron area (th_area), (e) the threshold of COM distance (th_COM), and (f) the minimum number of consecutive frames (th_frame). The solid blue lines are the averageF₁ scores, and the shaded regions are mean ± one s.d. When evaluating the post-processing parameters in (c-f), we fixed each parameter under investigation at the given values and simultaneously optimized theF₁ score over the other parameters. Variations in these hyperparameters produced only small variations in theF₁ performance. The orange lines show theF₁ score (solid) ± one s.d. (dashed) when we optimized all four post-processing parameters simultaneously. The similarity between theF₁ scores on the blue lines and the scores on the orange lines suggest that optimizing for three or four parameters simultaneously achieved similar optimized performance. Moreover, the relatively consistentF₁ scores on the blue lines suggest that our algorithm did not rely on intricate hyperparameter tuning.

The (a,d) recall, (b,e) precision, and (c,f)F₁ score of all the (a-c) batch and (d-f) online segmentation algorithms in the presence of increasing intensity noise. The test dataset was the ABO 275 μm data with added random noise. The relative noise strength was represented by the ratio of the standard deviation of the random noise amplitude to the mean fluorescence intensity. As expected, theF₁ scores of all methods decreased as the noise amplitude grew. TheF₁ of SUNS was greater than theF₁’s of all other methods at all noise intensities.g-l, are in the same format of (a-f), but show the performance with the presence of increasing motion artifacts. The motion artifacts strength was represented by the standard deviation of the random movement amplitude (unit: pixels). As expected, theF₁ scores of all methods decreased as the motion artifacts became stronger. TheF₁ of SUNS was greater than theF₁’s of all other methods at all motion amplitudes. STNeuroNet and CaImAn batch were the most sensitive to strong motion artifacts, likely because they rely on accurate 3D spatiotemporal structures of the video. On the contrary, SUNS relied more on the 2D spatial structure, so it retained the accuracy better when spatial structures changed position over different frames.

SUNS segmented active neurons within each individual frame, and then accurately collected and merged the instances belonging to the same neurons. We selected two example pairs of overlapping neurons from the ABO video 539670003 identified by SUNS, and showed their traces and instances when they were activated independently.a, The SNR images of the region surrounding the selected neurons. The left image is the maximum projection of the SNR video over the entire recording time, which shows the two neurons were active and overlapping. The right images are single-frame SNR images at two different time points, each at the peak of a fluorescence transient where only one of the two neurons was active. The segmentation of each neuron generated by SUNS is shown as a contour with a different color. The scale bar is 3 μm.b, The temporal SNR traces of the selected neurons, matched to the colors of their contours in (a). Because the pairs of neurons overlapped, their fluorescence traces displayed substantial crosstalk. The dash markers above each trace show the active periods of each neuron determined by SUNS. The colored triangles below each trace indicate the manually-selected time of the single-frame images shown in(a).c-d, are parallel to (a-b), but for a different overlapping neuron pair.e, We quantified the ability to find overlapped neurons for each segmentation algorithm using the recall score. We divided the ground truth neurons in all the ABO videos into two groups: neurons without and with overlap with other neurons. We then computed the recall scores for both groups. The recall of SUNS on spatially overlapping neurons was not significantly lower (and was numerically higher) than the recall of SUNS on non-spatially overlapping neurons (P > 0.8, one-sided Wilcoxon rank-sum test,n = 10 videos; n.s.l. – not significantly lower). Therefore, the performance of SUNS on overlapped neurons was at least equally good as the performance of SUNS on non-overlapped neurons. Moreover, the recall scores of SUNS in both groups were comparable to or significantly higher than that of other methods in those groups (**P < 0.005, n.s. – not significant; two-sided Wilcoxon signed-rank test,n = 10 videos; error bars are s.d.). The gray dots represent the scores on the test data for each round of cross-validation.

We evaluated the contribution of each pre-processing option (spatial filtering, temporal filtering, and SNR normalization) and the CNN option to SUNS. The reference algorithm (SUNS) used all options except spatial filtering. We compared the performance of this reference algorithm to the performance with additional spatial filtering (optional SF), without temporal filtering (no TF), without SNR normalization (no SNR), and without the CNN (no CNN) when analyzing the ABO 275 μm dataset through ten-fold leave-one-out cross-validation.a, The recall, precision, andF₁ score of these variants. The temporal filtering, SNR normalization, and CNN each significantly contributed to the overall accuracy, but the impact of spatial filtering was not significant (*P < 0.05, **P < 0.005, n.s. - not significant; two-sided Wilcoxon signed-rank test,n = 10 videos; error bars are s.d.). The gray dots represent the scores on the test data for each round of cross-validation.b, The speed andF₁ score of these variants. Eliminating temporal filtering or the CNN significantly increased the speed, while adding spatial filtering or eliminating SNR normalization significantly lowered the speed (**P < 0.005; two-sided Wilcoxon signed-rank test,n = 10 videos; error bars are s.d.). The light color dots representF₁ scores and speeds for the test data for each round of cross-validation. The execution of SNR normalization was fast (~0.07 ms/frame). However, eliminating SNR normalization led to a much lower optimalth_prob, and thus increased the number of active pixels and decreased precision. In addition, ‘no SNR’ had lower speed than the complete SUNS algorithm due to the increased post-processing computation workload for managing the additional active pixels and regions.

a, Training on one ABO 275 μm video and testing on nine ABO 275 μm videos (each data point is the average over each set of nine test videos,n = 10);b, Training on ten ABO 275 μm videos and testing on ten ABO 175 μm videos (n = 10);c, Training on one Neurofinder video and testing on one paired Neurofinder video (n = 12);d, Training on three-quarters of one CaImAn video and testing on the remaining quarter of the same CaImAn video (n = 16). TheF₁ scores of SUNS were mostly significantly higher than theF₁ scores of other methods (*P < 0.05, **P < 0.005, ***P < 0.001, n.s. - not significant; two-sided Wilcoxon signed-rank test; error bars are s.d.). The gray dots represent the individual scores for each round of cross-validation.

a,e, Training on one ABO 275 μm video and testing on nine ABO 275 μm videos (each data point is the average over each set of nine test videos,n = 10);b,f, Training on ten ABO 275 μm videos and testing on ten ABO 175 μm videos (n = 10);c,g, Training on one Neurofinder video and testing on one paired Neurofinder video (n = 12);d,h, Training on three-quarters of one CaImAn video and testing on the remaining quarter of the same CaImAn video (n = 16). TheF₁ score and processing speed of SUNS online were significantly higher than theF₁ score and speed of CaImAn Online (**P < 0.005, ***P < 0.001; two-sided Wilcoxon signed-rank test; error bars are s.d.). The gray dots in (a-d) represent individual scores for each round of cross-validation. The light color dots in (e-g) representF₁ scores and speeds for the test data for each round of cross-validation. The light color markers in (h) representF₁ scores and speeds for the test data for each round of cross-validation performed on different CaImAn videos. We updated the baseline and noise regularly after initialization for the Neurofinder dataset, but did not do so for other datasets.

The (a-c)F₁ score and (d-f) speed of SUNS online increased as the number of frames per update (n_merge) increased for the (a, d) ABO 275 μm, (b, e) Neurofinder, and (c, f) CaImAn datasets. The solid line is the average, and the shading is one s.d. from the average (n = 10, 12, and 16 cross-validation iterations for the three datasets). In (a-c), the green lines show theF₁ score (solid) ± one s.d. (dashed) of SUNS batch. TheF₁ score and speed generally increased asn_merge increased. For example, theF₁ score and speed when usingn_merge = 500 were respectively higher than theF₁ score and speed when usingn_merge = 20, and some of the differences were significant (*P < 0.05, **P < 0.005, ***P < 0.001, n.s. - not significant; two-sided Wilcoxon signed-rank test;n = 10, 12, and 16, respectively). We updated the baseline and noise regularly after initialization for the Neurofinder dataset, but did not do so for other datasets. Then_merge was inversely proportional to the update frequency or the responsiveness of SUNS online to the appearance of new neurons. A trade-off exists between this responsiveness and the accuracy and speed of SUNS online. At the cost of less responsiveness, a highern_merge allowed the accumulation of temporal information and improved the accuracy of neuron segmentations. Likewise, a highern_merge improved the speed because it reduced the occurrence of computations for aggregating neurons.

We compared theF₁ score and speed of SUNS online with or without baseline and noise update for the (a) ABO 275 μm, (b) Neurofinder, and (c) CaImAn datasets. TheF₁ scores with baseline and noise update were generally higher, but the speeds were slower (*P < 0.05, **P < 0.005, ***P < 0.001, n.s. - not significant; two-sided Wilcoxon signed-rank test; error bars are s.d.). The light color dots representF₁ scores and speeds for the test data for each round of cross-validation. The improvement in theF₁ score was larger as the baseline fluctuation becomes more significant.d, Example processing time per frame of SUNS online with baseline and noise update on Neurofinder video 02.00. The lower inset zooms in on the data from the red box. The upper inset is the distribution of processing time per frame. The processing time per frame was consistently faster than the microscope recording rate (125 ms/frame). The first few frames after initialization were faster than the following frames, because the baseline and noise update was not performed in these frames.

Supplementary Figs. 1–14, Tables 1–10 and Methods.

Demonstration of how SUNS online gradually found new neurons on an example raw video. The example video is selected frames in the fourth quadrant of the video YST from the CaImAn dataset. We showed the results of the SUNS online without the ‘tracking’ option enabled (left) and with the ‘tracking’ option enabled (right). The red contours were the segmented neurons from all frames before the current frame. We updated the identified neuron contours every second (ten frames), so the red neuron contours appeared with some delay after the neurons’ initial activation. The green contours in the right panels were the neurons found in previous frames and appeared as active in the current frames. Such tracked activity enables follow-up analysis of animal behaviours or brain network structures in real-time feedback neuroscience experiments.

Demonstration of how SUNS online gradually found new neurons on an example SNR video. The example video is selected frames in the fourth quadrant of the video YST from the CaImAn dataset after pre-processing and conversion to an SNR video. We showed the results of the SUNS online without the ‘tracking’ option enabled (left) and with the ‘tracking’ option enabled (right). The red contours were the segmented neurons from all frames before the current frame. We updated the identified neuron contours every second (ten frames), so the red neuron contours appeared with some delay after the neurons’ initial activation. The green contours in the right panels were the neurons found in previous frames and appeared as active in the current frames. Such tracked activity enables follow-up analysis of animal behaviours or brain network structures in real-time feedback neuroscience experiments.