Publications - Saras Project

Action detection Artificial Intelligence Cognitive control Collision avoidance Computer Science Computer vision Laparoscopy Laparoscopy Learning machine learning Manipulators Medical robotics Minimally invasive surgery Object recognition Pattern Recognition POMDP R-MIS Robot Robot Robot kinematics Robotic surgery Robotics Robots Surgery Surgical robotics Surgical robots Surgical robots Teleoperation Three-dimensional displays Tools 
Show all

                    
                        2021
                    
                
 Khan, Salman;  Cuzzolin, Fabio
Spatiotemporal Deformable Scene Graphs for Complex Activity Detection Conference 
2021 The British Machine Vision Conference (BMVC), 2021.
Abstract | Links | BibTeX | Tags: Action detection, activity detection, autonomous driving, complex activity detection, deformable pooling, graph convolutional network, parts deformation, scene graph, Surgical robotics
@conference{Khan2021,

title = {Spatiotemporal Deformable Scene Graphs for Complex Activity Detection},

author = {Salman Khan and Fabio Cuzzolin},

url = {https://www.bmvc2021-virtualconference.com/assets/papers/0706.pdf},

year  = {2021},

date = {2021-11-24},

booktitle = {2021 The British Machine Vision Conference (BMVC)},

abstract = {Long-term complex activity recognition and localisation can be crucial for decision making in autonomous systems such as smart cars and surgical robots. Here we address the problem via a novel deformable, spatiotemporal scene graph approach, consisting of three main building blocks: (i) action tube detection, (ii) the modelling of the deformable geometry of parts, and (iii) a graph convolutional network. Firstly, action tubes are detected in a series of snippets. Next, a new 3D deformable RoI pooling layer is designed for learning the flexible, deformable geometry of the constituent action tubes. Finally, a scene graph is constructed by considering all parts as nodes and connecting them based on different semantics such as order of appearance, sharing the same action label and feature similarity. We also contribute fresh temporal complex activity annotation for the recently released ROAD autonomous driving and SARAS-ESAD surgical action datasets and show the adaptability of our framework to different domains. Our method is shown to significantly outperform graph-based competitors on both augmented datasets.},

keywords = {Action detection, activity detection, autonomous driving, complex activity detection, deformable pooling, graph convolutional network, parts deformation, scene graph, Surgical robotics},

pubstate = {published},

tppubtype = {conference}

}

Close
Long-term complex activity recognition and localisation can be crucial for decision making in autonomous systems such as smart cars and surgical robots. Here we address the problem via a novel deformable, spatiotemporal scene graph approach, consisting of three main building blocks: (i) action tube detection, (ii) the modelling of the deformable geometry of parts, and (iii) a graph convolutional network. Firstly, action tubes are detected in a series of snippets. Next, a new 3D deformable RoI pooling layer is designed for learning the flexible, deformable geometry of the constituent action tubes. Finally, a scene graph is constructed by considering all parts as nodes and connecting them based on different semantics such as order of appearance, sharing the same action label and feature similarity. We also contribute fresh temporal complex activity annotation for the recently released ROAD autonomous driving and SARAS-ESAD surgical action datasets and show the adaptability of our framework to different domains. Our method is shown to significantly outperform graph-based competitors on both augmented datasets.
Close
https://www.bmvc2021-virtualconference.com/assets/papers/0706.pdf
Close

                    
                        2020
                    
                
 V. Singh Bawa  G. Singh, Kaping’A Skarga-Bandurova Leporini Landolfo Stabile Setti Muradore Oleari Cuzzolin F  I  A  C  A  F  R  E  F
ESAD: Endoscopic Surgeon Action Detection Dataset Online 
 arXiv,  (Ed.): 2020, visited: 25.06.2020.
Abstract | Links | BibTeX | Tags: Action detection, endoscopic video, surgeon action detection, Surgery
@online{Bawa2020,

title = {ESAD: Endoscopic Surgeon Action Detection Dataset},

author = {V. Singh Bawa, G. Singh, F. Kaping’A, I. Skarga-Bandurova, A. Leporini, C. Landolfo, A. Stabile, F. Setti, R. Muradore, E. Oleari, F. Cuzzolin},

editor = {arXiv},

url = {https://zenodo.org/record/4471476#.YBFMT-hKiXI},

year  = {2020},

date = {2020-06-12},

urldate = {2020-06-25},

abstract = {In this work, we take aim towards increasing the effectiveness of surgical assistant robots. We intended to make assistant robots safer by making them aware about the actions of surgeon, so it can take appropriate assisting actions. In other words, we aim to solve the problem of surgeon action detection in endoscopic videos. To this, we introduce a challenging dataset for surgeon action detection in real-world endoscopic videos. Action classes are picked based on the feedback of surgeons and annotated by medical professional. Given a video frame, we draw bounding box around surgical tool which is performing action and label it with action label. Finally, we presenta frame-level action detection baseline model based on recent advances in ob-ject detection. Results on our new dataset show that our presented dataset provides enough interesting challenges for future method and it can serveas strong benchmark corresponding research in surgeon action detection in endoscopic videos.},

keywords = {Action detection, endoscopic video, surgeon action detection, Surgery},

pubstate = {published},

tppubtype = {online}

}

Close
In this work, we take aim towards increasing the effectiveness of surgical assistant robots. We intended to make assistant robots safer by making them aware about the actions of surgeon, so it can take appropriate assisting actions. In other words, we aim to solve the problem of surgeon action detection in endoscopic videos. To this, we introduce a challenging dataset for surgeon action detection in real-world endoscopic videos. Action classes are picked based on the feedback of surgeons and annotated by medical professional. Given a video frame, we draw bounding box around surgical tool which is performing action and label it with action label. Finally, we presenta frame-level action detection baseline model based on recent advances in ob-ject detection. Results on our new dataset show that our presented dataset provides enough interesting challenges for future method and it can serveas strong benchmark corresponding research in surgeon action detection in endoscopic videos.
Close
https://zenodo.org/record/4471476#.YBFMT-hKiXI
Close

                    
                        2018
                    
                
 Behl, Harkirat Singh;  Sapienza, Michael;  Singh, Gurkirt;  Saha, Suman;  Cuzzolin, Fabio;  Torr, Philip H S
Incremental Tube Construction for Human Action Detection Proceeding 
2018.
Abstract | Links | BibTeX | Tags: Action detection, Artificial Intelligence, Computer Science, Computer vision, Detection, Pattern Recognition, Robot
@proceedings{Behl2018,

title = {Incremental Tube Construction for Human Action Detection},

author = {Harkirat Singh Behl and Michael Sapienza and Gurkirt Singh and Suman Saha and Fabio Cuzzolin and Philip H. S. Torr},

editor = {British Machine Vision Conference (BMVC). Newcastle-Upon-Tyne, UK},

url = {https://arxiv.org/abs/1704.01358},

year  = {2018},

date = {2018-07-23},

abstract = {Current state-of-the-art action detection systems are tailored for offline batch-processing applications. However, for online applications like human-robot interaction, current systems fall short, either because they only detect one action per video, or because they assume that the entire video is available ahead of time. In this work, we introduce a real-time and online joint-labelling and association algorithm for action detection that can incrementally construct space-time action tubes on the most challenging action videos in which different action categories occur concurrently. In contrast to previous methods, we solve the detection-window association and action labelling problems jointly in a single pass. We demonstrate superior online association accuracy and speed (2.2ms per frame) as compared to the current state-of-the-art offline systems. We further demonstrate that the entire action detection pipeline can easily be made to work effectively in real-time using our action tube construction algorithm.},

keywords = {Action detection, Artificial Intelligence, Computer Science, Computer vision, Detection, Pattern Recognition, Robot},

pubstate = {published},

tppubtype = {proceedings}

}

Close
Current state-of-the-art action detection systems are tailored for offline batch-processing applications. However, for online applications like human-robot interaction, current systems fall short, either because they only detect one action per video, or because they assume that the entire video is available ahead of time. In this work, we introduce a real-time and online joint-labelling and association algorithm for action detection that can incrementally construct space-time action tubes on the most challenging action videos in which different action categories occur concurrently. In contrast to previous methods, we solve the detection-window association and action labelling problems jointly in a single pass. We demonstrate superior online association accuracy and speed (2.2ms per frame) as compared to the current state-of-the-art offline systems. We further demonstrate that the entire action detection pipeline can easily be made to work effectively in real-time using our action tube construction algorithm.
Close
https://arxiv.org/abs/1704.01358
Close