TactiGraph: An Asynchronous Graph Neural Network for Contact Angle Prediction Using Neuromorphic Vision-Based Tactile Sensing

1. Introduction

1.1. Sense of touch and vision-based tactile sensing

The sense of touch is an important feedback modality that allows humans to perform many tasks. Consider, for example, the task of inserting a key into a lock. After obtaining an estimate of the keyhole’s position, we rely almost exclusively on the sense of touch to get the key from being in the general vicinity of the keyhole to inserting it in the keyhole. During these fleeting seconds, we rely on tactile feedback to adjust both the position as well as the orientation of the key until insertion is obtained. More subtly, we also use tactile feedback to adjust how much force is needed to keep the grasped key from slipping. For robots to perform tasks such as grasping [1,2], peg-in-hole insertion [3,4], and many tasks that require dexterity, it becomes paramount that robotic systems have a sense of touch [5,6,7]. Much work has been done on augmenting robots with an artificial sense of touch [8]. Several tactile sensor conceptions exist within the literature. These include sensors based on transduction (capacitive, resistive, ferromagnetic, optical, etc.) as well as those based on piezoelectric material [7]. However, these sensors have high instrumentation costs and are thus hard to maintain over long periods of time [9]. A particularly promising tactile sensing technology is vision-based tactile sensors (VBTS) [9].

VBTS systems consist of an imaging device capturing the deformation of a soft material due to contact with the external environment. The soft material is equipped with a pattern that allows the image sensor to capture deformations clearly. Such patterns include small colored markers [10,11], randomly dispersed fluorescent markers [12], and colored LED patterns [13]. Compared to other tactile sensors, VBTSs do not require much instrumentation; only the imaging device and a source of illumination are required to be instrumented and maintained. This is important for the longevity and usability of the sensor. Although VBTSs have low instrumentation overhead, they also provide high-resolution tactile feedback. VBTS output images, and thus can be processed using classical and learning methods for image processing. Some algorithms build on intermediate features such as marker position, optical flow, or depth [3,10,11,14,15,16,17,18], while others build and train end-to-end models [3,18,19,20]. The high resolution of VBTS allows robots to perform tasks such as detecting slip [2,21,22], estimating force distribution [11,23], classifying surface textures [24,25], and manipulating of small and soft objects [26,27] as well as many tasks that require dexterity and fine detailed pose estimation.

In previous work of ours [10,28], we have introduced a multifunctional sensor for navigation and guidance, precision machining, and vision-based tactile sensing. Concretely, the sensor is introduced in the context of the aerospace manufacturing industry where maintaining high-precision while machining is imperative. The proposed sensor configuration is depicted in Figure 1. When the hatch is open, the camera used for VBTS is used to navigate to the desired position. Once the target position is achieved, the hatch is closed and the deburring or drilling process starts. The tactile feedback from the VBTS is used to ensure that the robot is machining while maintaining perfect perpendicularity to the workpiece. The video stream of the imaging device is processed by a convolutional neural network (CNN). Ensuring perpendicularity is crucial when working in the aerospace manufacturing industry [29,30,31]. Failure to abide by perpendicularity requirements leads to an increase in bending stress and a decrease in fatigue life, thus lowering the reliability of aircraft parts [32,33]. Several other works in the literature use VBTS for contact angle prediction [19,34,35]. Lepora et al. [19] use a CNN on binarized images from the camera to estimate the contact pose. Psomopoulou et al. [34] also use a CNN to estimate the contact angle while grasping an object. Finally, [35] extract the markers’ positions using blob detection, then construct a 2D graph with markers as nodes using Delaunay triangulation. The graphs are then processed using a graph neural network to predict contact angle. All of the afromentioned work uses conventional VBTS thus requiring internal illumination as well as having a limited temporal update-rate. Due to advancements in vision sensor technologies, the neuromorphic event-based camera has emerged as a critical tool for achieving accurate visual perception and navigation. This bio-inspired device offers unprecedented capabilities compared to standard cameras, including its asynchronous nature, high temporal resolution, high dynamic range, and low power consumption. Therefore, by utilizing an event camera instead of a standard camera, we can enhance the potential of our previous sensor and improve navigation or machining performance in challenging illumination scenarios. In this work, we use an event-based camera for VBTS. This allows us to obtain less expensive computations, fast update-rate, and relinquishing the need for internal illumination that adds instrumentation complexity.

1.2. Neuromorphic vision-based tactile sensing

Most VBTSs, like the ones used in our prior work, use traditional frame-based cameras which record all pixels and return their intensity at a fixed rate. This leads to the redundant processing of pixels that do not change between consecutive frames. The fixed rate of frame-based cameras also poses a problem when recording fast-paced scenes. Furthermore, the downtime between consecutive frames can contain information that is of crucial importance to precision machining. To alleviate these issues, neuromorphic event-based cameras are being employed for vision-based tactile sensing [1,16,20,21,36,37,38,39].

Neuromorphic cameras (also known as event-based cameras) are a relatively new technology, first introduced in [40], that aims to mimic how the human eye works. Neuromorphic cameras report intensity changes, at the pixel level, in the scene in an asynchronous manner rather than report the whole frame at a fixed rate. This mode of operation makes event-based cameras exhibit no motion blur. The pixel-wise intensity changes, called events or spikes, are recorded at a temporal resolution on the order of microseconds. Event-based cameras have been applied in autonomous drone racing [41], space imaging [42], space exploration [43], automated drilling [44], and visual servoing [45,46]. Neuromorphic cameras’ fast update rate, along with their high dynamic range (140 dB compared to conventional cameras with 60 dB [47]) and low power consumption, make them apt for robotics tasks [48]. In particular, Event-based cameras are capable of providing adequate visual information in challenging lighting conditions without requiring an additional light source, owing to their high dynamic range. Due to not needing a source of illumination, a VBTS system that utilizes an event-based camera will have a lower instrumentation cost and thus require less maintenance in the long run. While some VBTSs use a transparent tactile surface to overcome the need for a source of illumination [9], this will make training end-to-end machine learning models difficult as the camera will capture extraneous information from the environment making it dependent on the object the sensor is contacting and the environment, thus limiting generalization. Event-based cameras allow us to overcome the instrumentation and maintenance costs of having a source of illumination while still maintaining the potential for training end-to-end models. These features of event-based camera make them an attractive choice for VBTS. However, dealing with event-based data still poses a challenge as will be discussed in the following subsection.

1.3. Challenges with event-based vision and existing solutions

The temporally dense, spatially sparse, and asynchronous nature of event-based streams poses a challenge to traditional methods of processing frame-based streams. Early work on neuromorphic vision-based tactile sensing (N-VBTS) constructs images out of event streams by accumulating events over a period of time and applying image-processing techniques. Such approaches are called event-frame methods. These approaches usually use synchronous algorithms and apply them over constructed frames sequentially thus, event-frame approaches do not exploit the temporal density and spatial sparsity of event streams. For instance, Amin et al. [36] detect the incipient slip of a grasped object by applying morphological operations over event-frames and monitoring blobs in the resulting frame. This approach is not asynchronous and does not generalize well to tasks beyond slip detection. Ward-Cherrier et al. [16] construct encodings relevant to the markers’ position of the tactile sensors and then use a classifier to detect the object’s texture in contact. This approach assumes knowledge of the markers’ positioning at all times. To our knowledge, the authors do not provide an algorithm for obtaining the markers’ position beyond the case where the sensor is resting. Fariborz et al. [37,38] use Conv-LSTMs on event-frames constructed from event streams to estimate contact forces. Faris et al. [22] uses CNN over accumulated event heatmaps to detect slip. This approach is not asynchronous hence has downtime between constructed event-frames. To our knowledge, the only asynchronous deep learning method that makes use of spatial sparsity and temporal density applied in the N-VBTS setting is the work of MacDonald et al. [39]. Building on [16]’s NeuroTac, they propose using a spiking neural network (SNN) to determine the orientation of contact with an edge. While this is a promising step towards neuromorphic tactile sensing, the training of the SNN is done in an unsupervised manner. Another classifier is run on top of the SNN to make the prediction. This approach does not generalize well beyond simple tasks. Furthermore, training SNNs is still challenging due to their non-differentiable nature [49,50].

Outside the N-VBTS literature, event-frame and voxel methods also persist [47,51,52,53,54]. An emerging line of research investigates the use of graph neural networks (GNNs) to process event streams [55,56,57,58,59]. GNNs operate on graphs by learning a representation that takes into account the graph’s connectivity. This representation can be used for further processing via classical machine and deep learning methods. GNNs generalize convolutional networks for irregular grids and networks [60]. This is done using a mechanism called message passing. Given a graph G with nodes V, node features X, and edges E, a single layer of message passing will have nodes obtaining messages from their neighboring nodes and using those messages to update their node features. These messages are learnable functions of the neighboring nodes’ features. A particular type of message passing is the convolutional type, where nodes update their representation in a way that resembles convolutions. A node $u\in V$ updates its own representation from $x_u^{\ell }$ at layer ℓ, to $x_u^{\ell +1}$ at layer $\ell +1$ by

$$\begin{array}{cc}\hfill m_{uv}& =a_{uv}\phi (x_i^{\ell },x_j^{\ell })\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill m_u& =\sum _{v\in \mathcal{N}\left(u\right)}{m}_{uv}\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill x_u^{\ell +1}& =\psi (x_i^{\ell },m_i)\hfill \end{array}$$

(3)

Where $v\in V$ is a neighboring node of u, $a_{uv}$ are edge attributes, and $\phi$ and $\psi$ are learnable functions [60]. By constructing a graph over events from an event-based camera, GNNs can perform spatially sparse and temporally dense convolutions. GNNs can also operate in an asynchronous mode by applying the methods proposed in [61,62] to match the nature of event-based streams. This mode of operation ensures that calculations only occur when there are events, as opposed to event-frame methods. The earliest work utilizing GNNs for event streams, [55], investigates object classification on neuromorphic versions of popular datasets such as Caltech101 and MNIST. Other works also tackle object detection and localization [55,56,61]. Alkendi et al. [59] use a GNN fed into a transformer for event stream denoising. Furthermore, [63] shows that GNNs do well in object detection tasks while performing considerably fewer floating point operations per event compared to CNNs operating on event-frames.

Graphs inherently do not encode geometric information pertaining to their nodes. They only encode information concerning the topological relationships between the nodes as well as the node features. Nonetheless, constructing useful and meaningful representations of event data requires more than just the topological structure of a graph. Thus, it becomes imperative to choose $\psi$ and $\phi$ in a way that encapsulates the geometry of events. Several graph geometric deep learning methods have been applied to event-based data in the literature. This includes Mixture model network (MoNet), graph convolutional networks (GCN), SplineConv, voxel graph CNNs, as well as EventConv [55,56,57,58,59]. Although all of these methods take geometry into account, SplineConv has been shown to perform better and faster than MoNet [57]. Furthermore, unlike voxel graph CNNs and EventConv [58,59], SplineConv can be run asynchronously using the method proposed in [63]. SplineConvs are also more expressive than GCNs which can only use one-dimensional features [64,65]. In the case of geometric graphs, this feature is usually taken as the distance between nodes. This is problematic for two reasons 1) messages shared from two equidistant nodes will be indistinguishable and 2) the messages will be rotation invariant and hence lose all information about orientation.

1.4. Contributions

In this work, we use a SplineConv-based graph neural network to predict the contact angle of a neuromorphic vision-based tactile sensor. This proposed framework is depicted in Figure 1. Our contributions can be summarized as follows.

• We introduce, TactiGraph, a graph neural network based on SplineConv layers to handle data from a neuromorphic vision-based tactile sensor. TactiGraph is able to fully account for the spatially sparse and temporally dense nature of event streams.
• We deploy TactiGraph to solve the problem of contact angle prediction using the neuromorphic tactile sensor. We obtain an error of $0.{62}^{\circ }$.
• TactiGraph maintains a high level of accuracy, with only an error of $0.{71}^{\circ }$, even when the tactile sensor is used without an illumination source. This demonstrates that it is possible to reduce the cost of operating a vision-based tactile sensor by eliminating the need for LEDs, while still achieving reliable results.

1.5. Outline

The rest of this work is organized as follows. In Section 2, we discuss the data collection apartus as well as the sensor design. In Section 3, we benchmark TactiGraph against other VBTS and N-VBTS methods. Finally, we conclude in Section 4.

2. Materials and Methods

In this section, we describe the experimental setup used to generate the data in this paper. Furthermore, we also describe the tactile sensor design, the sensor’s output format, as well as how to handle this output using TactiGraph.

2.1. Data collection

The VBTS system consists of a camera, an enclosure containing the camera, and a hemispherical soft surface attached to the hatch as seen in Figure 1. Small beads are placed on the insides of the soft tactile surface to generate contrast, allowing the camera to capture the deformation of the sensor with clarity. The event-camera used is an IniVation DAVIS 346 with resolution $346\times 260$ and a latency of $20\mu s$[66]. The sensor enclosure is made of 3D-printed ABS. Two LED strips are placed above and below the camera. More details on the manufacturing process of the elastomer surface and the enclosure can be found in [10,28]. The whole apparatus is attached to the Universal Robotics UR10 [67]. The UR10 pushes the tactile sensor against a flat surface at various angles of contact. This is shown in Figure 1. The contact angle is controlled by two parameters, $\theta \in \Theta$ and $\varphi \in \Phi$ where $\Theta =\{0,1,\cdots ,9\}$ and $\Phi$ is a collection of 20 angles around the circle. This variation in $\theta$ and $\varphi$ can be seen in Figure 1. We collect 12 samples of each contact angle, thus ending up with a total of $n=1836$ samples in total. The depth of each contact case is chosen randomly between 5mm and 15mm from the tip of the sensor when relaxed. The randomness in contact depth ensures that our model can generalize to different contact profiles from light to heavy contact cases. To evaluate the performance of N-VBTS without internal illumination, this process is done twice, once with the LED strips on and another time with the LED strips turned off.

2.2. Preprocessing the event stream

Let $C=\{1,\cdots ,240\}\times \{1,\cdots ,346\}$ the set of all pixels in the DAVIS 346. The output of the event camera is then a stream $\mathcal{S}={\left\{e_k\right\}}_{k=1}^N$ of asynchronously reported intensity changes in the frame. The $k^{\mathrm{th}}$ intensity change, an event, is a 4-tuple

$$e_k=(x_k,y_k,t_k,p_k)$$

(4)

where $(x_k,y_k)\in C$, $t_k\in {\mathbb{R}}^{+}$, and $p_k\in \{-1,1\}.$ The $(x,y)$ components represent where the event has happened, the t component is a timestamp indicating when the event has happened, and p, the polarity, represents whether the intensity change is positive or negative. Figure 2 shows histograms of the number of events generated by contact cases in both LED-on and LED-off scenarios. The variance in the depth of the contact translates itself into the variance of the number of events generated; a light contact case will cause a smaller displacement to the markers hence generating less events and vice versa. Overall, fewer events are generated in LED-off contact cases. However, dark scenes cause a lot of background noise [68,69]. While many denoising methods exist in the literature [59,68,69], we use the background activity filter [70] with a time window of 1.25 milliseconds when the LED strips are off. For further robustness against noise, the dataset is augmented via jittering events. Given an event $e=(x,y,t,p)$, we spatially jitter the event by at most one pixel $\tilde{e}=(x+\delta x,y+\delta y,t,p)$ where $\delta x$ and $\delta y$ are drawn uniformly from {-1, 0, 1}. The effect of this jittering will be investigated later in this work.

Out of the stream $\mathcal{S}$, for the $j^{\mathrm{th}}$ contact case we capture a spatiotemporal volume ${\mathcal{E}}_j\subset \mathcal{S}$ such that

$${\mathcal{E}}_j=\{(x,y,t,p)\in \mathcal{S}|t_j\le t\le t_j+\Delta T_j\}$$

(5)

where $t_j$ the timestamp at which the $j^{\mathrm{th}}$ contact case starts and $90\mathrm{ms}\le \Delta T_j\le 200\mathrm{ms}$ is the window size chosen to be at most $200\mathrm{ms}$. The window size is adaptive to adjust for various depths of contact cases; a heavy contact case takes more time than a light contact. For each event $e_k\in {\mathcal{E}}_j$ we assign a normalized spatiotemporal postion $r_k=({\widehat{x}}_k,{\widehat{y}}_k,{\widehat{t}}_k)=\left(\frac{x}{W},\frac{y}{H},\frac{t_k-t_j}{\Delta T_j}\right)$.

2.3. Graph construction

For each spatiotemporal volume of events ${\mathcal{E}}_j$ a graph $G_j=(V_j,E_j,X_j)$ is construced. The nodes of the graph are events themselves $V_j={\mathcal{E}}_j.$ The edges of the graph are determined using the $k^{\mathrm{th}}$ nearest neighbors algorithm. The distance between two events $e_i,e_k\in {\mathcal{E}}_j$ is calculated using the Euclidean distance between their normalized spatiotemporal positions $r_i$ and $r_k.$ Letting $k\mathrm{NN}\left(e_k\right)$ denote the set of the k nearest events to $e_k$, the set of edges of the graph is defined by

$$E_j=\{(e_i,e_k)\in {\mathcal{E}}_j\times {\mathcal{E}}_j|e_k\in k\mathrm{NN}\left(e_i\right)\}$$

(6)

While this method of constructing the graph does not always result in a graph with one connected component, we found it always results in one large connected component with a few smaller components consisting mostly of less than 10 events. Thus these smaller components are dropped out from the dataset. Finally, each node of the graph $e_k$ has the polarity $p_k$ as its node features, $X_j=\left\{p_k\right|e_k\in {\mathcal{E}}_j\}$. The dataset is now the collection of the n graphs, with labels correponding to the contact angle of the contact case, collected from each of the n contact case, ${\mathcal{D}}_{\mathrm{LED}-\mathrm{on}}={\left\{(G_j,{\mathrm{Roll}}_{\mathrm{j}},{\mathrm{Pitch}}_j)\right\}}_{j=1}^n$. We note that we use a roll-pitch representation of the contact angle. This is to avoid singularities caused around 0. Another dataset ${\mathcal{D}}_{\mathrm{LED}}\mathrm{off}$ with $n=1836$ samples is generated identically while the LED is off. The two datasets are used separately. Each dataset is randomly split into train, validation, and test subsets with $70\%$ of $\mathcal{D}$ being used for training while $15\%$ is used for validation, and the last $15\%$ is used for testing.

2.4. TactiGraph

A graph neural network, namely TactiGraph, with a stack of SplineConv layers and node pooling layers is used for contact angle prediction. TactiGraph consists of SplineBlocks, node pooling layers, a final pooling layer, and a final multilayer perceptron (MLP). Graphical depiction of TactiGraph can be seen in Figure 3 (a). SplineBlocks consist of a SplineConv layer, an exponential linear unit (ELU), and a batch normalization layer. The node pooling layer reduces the number of nodes in the graph $G_j$ from $N_i$ to $N_{i+1}$ by first constructing a voxel grid over the volume ${\mathcal{E}}_j$ then pooling all nodes within a voxel unit into one node; inheriting edges, if any, from the unpooled nodes. For the layers before pooling, messages passed between events. This ensures the initial low-level geometry is preserved. After pooling, higher-level nodes are created and message passing occurs between high-level features. The final pooling layer will convert the variable size graph into a fixed-size vector by max pooling over a $4\times 4$ voxel grid. A voxel grid is generated over the spatiotemporal volume $\mathcal{E}$. Skip connections are also added via adding node features. We ablate over the number of SplineConv layers, the number of node pooling layers, the number of skip layers, and the node embedding dimension. The scope of this ablation study is shown in Table 1. A final pooling layer is applied, followed by fully connected layers predicting the contact angles.

2.5. Training setup

We use version 2.0.4 of the Pytorch Geometric library [71] to implement our models. Training is done over 1000 epochs using the Adam optimizer [72] with an adaptive learning rate and default values as per Pytorch version 1.11.0 [73]. The learning rate starts at 0.001 but is reduced to 0.00001 when the validation loss plateaus. The training is carried out on a PC running Ubuntu 20.04, with an Intel i7-12700H CPU and an NVIDIA RTX 3080 Laptop GPU.

3. Results and Discussion

In this section, we present our findings in benchmarking TactiGraph against other methods of contact angle prediction as well as other methods of processing event streams. We demonstrate the abilities of TactiGraph on N-VBTS with and without internal illumination. We conduct a computational analysis comparing TactiGraph on N-VBTS with conventional VBTS approaches.

3.1. Evaluation of TactiGraph and the effect of jittering

The best model from the ablation study is shown in Figure 3. The training results on both LED-on and LED-off datasets are shown in Table 2. We display TactiGraph’s mean absolute error (MAE) on the test dataset. The model is trained with and without applying the 1px jitter augmentation on the training datasets. We note that applying the jittering augmentation strategy when training improves accuracy in the test dataset. The effect of jittering is amplified on the LED-off dataset. We argue that this is due to the noisy and sparse nature of event-based cameras in the LED-off environment [68,69]. Thus exposing the model to jittered events makes the model more robust to the noisy test dataset. It is worth noting that jittering events by more than one pixel proved to be an ineffective strategy that gave worse results than not jittering. This might be due to the fact that the event-based camera used, the DAVIS346, is of relatively low resolution. Thus jittering by more than one pixel can cause a change in the true dynamics of the scene.

3.2. Visualizing TactiGraph’s embedding space

To get a better understanding of what TactiGraph has learned, we visualize the node embedding generated for each sample in the dataset. This is done by saving the values obtained from the last node pooling layer, right before the fully connected layer, during a forward pass on the trained model. These values live in a high-dimensional space. To this end, we use the t-distributed stochastic neighbor embedding (t-SNE) algorithm [74]. t-SNE, a manifold learning technique, nonlinearly projects high-dimensional data onto a low-dimensional space more tangible for visualization. The results of the t-SNE visualization are shown in Figure 4. Each point in the scatter plot represents a contact case j associated with $(G_j,{\mathrm{Roll}}_j,{\mathrm{Pitch}}_j)\in {\mathcal{D}}_{\mathrm{LED}}\mathrm{on}$. The points are colored according to the angles of contact $\theta$ and $\varphi$. Even though TactiGraph was trained on roll-pitch representation, what we see in these plots is that TactiGraph has learned to embed similar contact angles $\theta$ and $\varphi$ next to each other. Looking at how different values of $\varphi$ and $\theta$ vary in the embedding space, we see that the model has learned an embedding that emphasizes variance in $\varphi .$ This is due to the fact that $\varphi$ varies more than $\theta$. The clearly visible gradients in these plots confirm that TactiGraph has actually done a good job of learning the dynamics of the problem.

3.3. Benchmark results

The only contact angle estimation approach in the literature that uses neuromorphic vision-based tactile sensing is the work of MacDonald et al. [39]. However, unlike our work, MacDonald et al. estimate the contact angle with an edge rather than a flat surface; thus these results are not directly comparable. They build an embedding using a spiking neural network in an unsupervised fashion, which is coupled with a supervised KNN classifier. We also compare our results against other works using traditional vision-based tactile sensing approaches. The results are tabulated in Table 3. We split the results between N-VBTS methods and conventional VBTS methods. Given the relatively low dynamic range of conventional cameras, conventional VBTSs do not work without an internal source of illumination. While Halwani et al. [28] achieve better results, this is done using a CNN operating on a conventional camera, hence requiring a source of illumination, being susceptible to motion blur, as well as being computationally more expensive as we show in Section 3.4. The same applies to Tac-VGNN [35], which uses a GNN. However, their GNNs operate synchronously on graphs constructed using a conventional camera, where graph nodes are made of internal markers.

Finally, we compare the performance of TactiGraph against using CNNs on event-frames. We use each ${\mathcal{E}}_j$ from both LED-on and LED-off datasets to construct an event-frame. This is done to make a fair comparison, as both methods will have access to the same amount of information. A grayscale event-frame $F_j\in {\mathbb{R}}^{346\times 260}$ corresponding to volume ${\mathcal{E}}_j$ is constructed as follows.

$${\left(F_j\right)}_{ab}=\sum _{e_i\in {\mathcal{E}}_j}{p}_i\delta (x_i=a)\delta (y_i=b),$$

(7)

where $\delta$ is the Dirac delta function. Sample event-frames are shown in Figure 5. We see that the temporal information is lost. Starting from the best CNN model obtained for VBTS contact angle prediction in [28], we perform an ablation study over the number of channels in the 2nd, 3rd, and 4th convolutional layers. Furthermore, we also look at how many convolutional layers are needed. Lastly, the ablation also studies the number and sizes of the fully connected layers. This ablation study is presented in Table 5. The benchmark results of the best CNN model are displayed in Table 4. TactiGraph achieves better contact angle estimation than the optimal CNN architecture on event-frames constructed from the same volumes of events that the graphs are made of. Better results hold in both LED-on and LED-off scenarios.

3.4. Runtime analysis

Given a live event stream $\mathcal{S}$ that started running at time $t_0$, in other words, for every $(x,y,t,p)\in \mathcal{S}$ we have $t_0\le t\le t_c$ where $t_c$ is the current time, TactiGraph operates on a graph constructed from a sliding window.

$$\mathcal{W}=\{(x,y,t,p)\in \mathcal{S}|t_c-\Delta T\le t\le t_c\}$$

(8)

As events asynchronously enter and exit $\mathcal{W}$, the graph is updated accordingly and TactiGraph acts on it. Instead of having a GNN rerun the whole forward pass as events slide in and out of $\mathcal{W}$, Schaefer et al. [61] propose AEGNN, a method by which redundant computations are not repeated. By looking at the neighborhoods of incoming and outcoming events, AEGNN is able to asynchronously determine which computations need to be recomputed. We modify TactiGraph to utilize the same mechanism as AEGNN. With these optimizations in mind, a prediction using TactiGraph consists of two steps: graph construction and the forward pass. In the worst-case scenario where the whole scene changes, the graph construction step takes an average of 34.5 milliseconds. Also, in the worst-case scenario, the forward pass takes an average of 58.1 milliseconds. These results were obtained on the same hardware mentioned above in Section 2.5.

The combination of N-VBTS and TactiGraph is computationally much cheaper than the CNN and VBTS of Halwani et al. [28]. We validate this by looking at the total compute-time taken by both methods in processing the same contact cases. We record a dataset 20 seconds long containing five contact cases. We run the CNN model from [28] on the active pixel sensor stream of the same DAVIS 346 used in this work. The total compute time the CNN takes to process this stream is 3.93 seconds. TactiGraph operating on the event stream, on the other hand, took only 0.22 seconds, $5.5\%$ of the CNN compute time. This is attributed to the redundant output the VBTS gives, which leads to redundant computations done by the CNN. Therefore, TactiGraph operating on N-VBTS streams is much faster than the CNN model operating on VBTS streams from [28].

3.5. Future work

Our neuromorphic vision-based tactile sensor has shown remarkable performance in contact angle prediction. Therefore, the TactiGraph capabilities can be extended further to perform other tactile sensing applications such as force sensing, texture recognition, and object identification in parallel. We plan on also including a recurrent or attentional mechanism in TactiGraph. This will give TactiGraph the generalization ability to operate on multiple tasks. It is also worth noting that the forward pass in TactiGraph can be further improved by replacing SplineConvs with a corresponding look-up table as proposed in [63], which claims a 3.7-fold reduction in inference time.

4. Conclusions

We introduced a neuromorphic vision-based tactile sensor (N-VBTS) that is able to run at a faster rate than the traditional vision-based tactile sensor (VBTS). We developed TactiGraph, a graph neural network, to operate on the raw asynchronous event stream exploiting the spatiotemporal correlations between events. Notably, TactiGraph is utilized to predict the contact angle of the sensor and achieves an error of $0.{62}^{\circ }$ degrees. We demonstrated the effectiveness of the proposed N-VBTS in terms of efficacy and accuracy compared to VBTS. Particularly, N-VBTS were capable of functioning without internal illumination hence leading to a reduction in long-term instrumentation and maintenance requirements. When tested on the same scenario, N-VBTS requires only 5.5% of the compute-time needed by VBTS.