Publications - Saras Project

Action detection Artificial Intelligence Cognitive control Collision avoidance Computer Science Computer vision hierarchical finite state machine Laparoscopy Laparoscopy machine learning Manipulators Medical robotics Minimally invasive surgery mobile robots Object recognition Pattern Recognition POMDP R-MIS Robot Robotic surgery Robotics Robots Surgery Surgical robotics Surgical robots Surgical robots Teleoperation Three-dimensional displays Tools Training 
Show all

                    
                        2021
                    
                
 Piccinelli, Nicola;  Muradore, Riccardo
A bilateral teleoperation with interaction force constraint in unknown environment using non linear model predictive control Journal Article 
European Journal of Control, 62 (November 2021), pp. 185-191, 2021, ISSN: 0947-3580.
Abstract | Links | BibTeX | Tags: Bilateral teleoperation, Model Predictive Control, Optimal control, Robotics
@article{Piccinelli2021c,

title = {A bilateral teleoperation with interaction force constraint in unknown environment using non linear model predictive control},

author = {Nicola Piccinelli and Riccardo Muradore},

doi = {https://doi.org/10.1016/j.ejcon.2021.06.030},

issn = {0947-3580},

year  = {2021},

date = {2021-07-10},

journal = {European Journal of Control},

volume = {62},

number = {November 2021},

pages = {185-191},

abstract = {In critical scenarios, the interaction forces between a robot and the environment could lead to damages and dangerous situations. Complex tasks like grasping fragile objects or physical human-robot interaction in collaborative robotics require the capability of controlling forces. In bilateral teleoperation, force feedback is used to provide telepresence to the operator. In such situation, the force is commonly measured by a force/torque sensor at the end effector of the remote robot. Even if force feedback allows the operator to feel the interaction with the environment this does not prevent unsafe motion. In this paper, we propose a model predictive control (MPC) based bilateral teleoperation able to guarantee safe interaction with the environment by constraining the forces. The method does not assume any prior knowledge of the environment.},

keywords = {Bilateral teleoperation, Model Predictive Control, Optimal control, Robotics},

pubstate = {published},

tppubtype = {article}

}

Close
In critical scenarios, the interaction forces between a robot and the environment could lead to damages and dangerous situations. Complex tasks like grasping fragile objects or physical human-robot interaction in collaborative robotics require the capability of controlling forces. In bilateral teleoperation, force feedback is used to provide telepresence to the operator. In such situation, the force is commonly measured by a force/torque sensor at the end effector of the remote robot. Even if force feedback allows the operator to feel the interaction with the environment this does not prevent unsafe motion. In this paper, we propose a model predictive control (MPC) based bilateral teleoperation able to guarantee safe interaction with the environment by constraining the forces. The method does not assume any prior knowledge of the environment.
Close
doi:https://doi.org/10.1016/j.ejcon.2021.06.030
Close

                    
                        2018
                    
                
 Roberti, Andrea;  Muradore, Riccardo;  Fiorini, Paolo;  Cristani, Marco;  Setti, Francesco
An energy saving approach to active object recognition and localization Conference 
Annual Conference of the IEEE Industrial Electronics Society (IECON).Washington, DC, USA.  2018.
Abstract | Links | BibTeX | Tags: Active object recognition, Artificial Intelligence, Computer Science, Learning, Object recognition, Pattern Recognition, POMDP, Robotics
@conference{Roberti2018b,

title = {An energy saving approach to active object recognition and localization},

author = {Andrea Roberti and Riccardo Muradore and Paolo Fiorini and Marco Cristani and Francesco Setti},

editor = {IECON 2018},

doi = {10.1109/IECON.2018.8591411},

year  = {2018},

date = {2018-10-21},

organization = {Annual Conference of the IEEE Industrial Electronics Society (IECON).Washington, DC, USA. },

abstract = {We propose an active object recognition (AOR) strategy explicitly suited to work with a real robotic arm. So far, AOR policies on robotic arms have focused on heterogeneous constraints, most of them related to classification accuracy, classification confidence, number of moves etc., discarding physical and energetic constraints a real robot has to fulfill. Our strategy adjusts this discrepancy, with a POMDP-based AOR algorithm that explicitly considers manipulability and energetic terms in the planning optimization. The manipulability term avoids the robotic arm to encounter singularities, which require expensive and straining backtracking steps; the energetic term deals with the arm gravity compensation when in static conditions, which is crucial in AOR policies where time is spent in the classifier belief update, before to do the next move. Several experiments have been carried out on a redundant, 7-DoF Panda arm manipulator, on a multi-object recognition task. This allows to appreciate the improvement of our solution with respect to other competitors evaluated on simulations only.},

keywords = {Active object recognition, Artificial Intelligence, Computer Science, Learning, Object recognition, Pattern Recognition, POMDP, Robotics},

pubstate = {published},

tppubtype = {conference}

}

Close
We propose an active object recognition (AOR) strategy explicitly suited to work with a real robotic arm. So far, AOR policies on robotic arms have focused on heterogeneous constraints, most of them related to classification accuracy, classification confidence, number of moves etc., discarding physical and energetic constraints a real robot has to fulfill. Our strategy adjusts this discrepancy, with a POMDP-based AOR algorithm that explicitly considers manipulability and energetic terms in the planning optimization. The manipulability term avoids the robotic arm to encounter singularities, which require expensive and straining backtracking steps; the energetic term deals with the arm gravity compensation when in static conditions, which is crucial in AOR policies where time is spent in the classifier belief update, before to do the next move. Several experiments have been carried out on a redundant, 7-DoF Panda arm manipulator, on a multi-object recognition task. This allows to appreciate the improvement of our solution with respect to other competitors evaluated on simulations only.
Close
doi:10.1109/IECON.2018.8591411
Close
 Singh, Gurkirt;  Saha, Suman;  Cuzzolin, Fabio
Predicting action tubes Journal Article 
2018, (Proceedings of the ECCV 2018 Workshop on Anticipating Human Behaviour (AHB 2018), Munich, Germany, Sep 2018).
Abstract | Links | BibTeX | Tags: Artificial Intelligence, Computer Science, Computer vision, Object recognition, Pattern Recognition, Robot, Robotics
@article{Singh2018cb,

title = {Predicting action tubes},

author = {Gurkirt Singh and Suman Saha and Fabio Cuzzolin},

editor = {ECCV 2018 Workshop on Anticipating Human Behaviour (AHB 2018), Munich, Germany, Sep 2018},

url = {http://openaccess.thecvf.com/content_ECCVW_2018/papers/11131/Singh_Predicting_Action_Tubes_ECCVW_2018_paper.pdf},

doi = {10.5281/zenodo.3362942},

year  = {2018},

date = {2018-08-23},

abstract = {In this work, we present a method to predict an entire `action tube' (a set of temporally linked bounding boxes) in a trimmed video just by observing a smaller subset of it. Predicting where an action is going to take place in the near future is essential to many computer vision based applications such as autonomous driving or surgical robotics. Importantly, it has to be done in real-time and in an online fashion. We propose a Tube Prediction network (TPnet) which jointly predicts the past, present and future bounding boxes along with their action classification scores. At test time TPnet is used in a (temporal) sliding window setting, and its predictions are put into a tube estimation framework to construct/predict the video long action tubes not only for the observed part of the video but also for the unobserved part. Additionally, the proposed action tube predictor helps in completing action tubes for unobserved segments of the video. We quantitatively demonstrate the latter ability, and the fact that TPnet improves state-of-the-art detection performance, on one of the standard action detection benchmarks - J-HMDB-21 dataset.},

note = {Proceedings of the ECCV 2018 Workshop on Anticipating Human Behaviour (AHB 2018), Munich, Germany, Sep 2018},

keywords = {Artificial Intelligence, Computer Science, Computer vision, Object recognition, Pattern Recognition, Robot, Robotics},

pubstate = {published},

tppubtype = {article}

}

Close
In this work, we present a method to predict an entire `action tube' (a set of temporally linked bounding boxes) in a trimmed video just by observing a smaller subset of it. Predicting where an action is going to take place in the near future is essential to many computer vision based applications such as autonomous driving or surgical robotics. Importantly, it has to be done in real-time and in an online fashion. We propose a Tube Prediction network (TPnet) which jointly predicts the past, present and future bounding boxes along with their action classification scores. At test time TPnet is used in a (temporal) sliding window setting, and its predictions are put into a tube estimation framework to construct/predict the video long action tubes not only for the observed part of the video but also for the unobserved part. Additionally, the proposed action tube predictor helps in completing action tubes for unobserved segments of the video. We quantitatively demonstrate the latter ability, and the fact that TPnet improves state-of-the-art detection performance, on one of the standard action detection benchmarks - J-HMDB-21 dataset.
Close
http://openaccess.thecvf.com/content_ECCVW_2018/papers/11131/Singh_Predicting_Ac[...]
doi:10.5281/zenodo.3362942
Close
 Arturo, Marbán;  Srinivasan, Vignesh;  Samek, Wojciech;  Fernández, Josep;  Casals, Alicia
Estimation of interaction forces in robotic surgery using a semi-supervised deep neural network model Conference 
2018.
Abstract | Links | BibTeX | Tags: Learning, Robot, Robotic surgery, Robotics, Surgery, Training
@conference{González2018,

title = {Estimation of interaction forces in robotic surgery using a semi-supervised deep neural network model},

author = {Marbán Arturo and Vignesh Srinivasan and Wojciech Samek and Josep Fernández and Alicia Casals},

editor = {IEEE},

url = {https://upcommons.upc.edu/bitstream/handle/2117/132610/iros2018_paper_26_07_2018.pdf?sequence=3&isAllowed=y},

doi = {10.1109/IROS.2018.8593701},

year  = {2018},

date = {2018-08-09},

abstract = {Providing force feedback as a feature in current Robot-Assisted Minimally Invasive Surgery systems still remains a challenge. In recent years, Vision-Based Force Sensing (VBFS) has emerged as a promising approach to address this problem. Existing methods have been developed in a Supervised Learning (SL) setting. Nonetheless, most of the video sequences related to robotic surgery are not provided with ground-truth force data, which can be easily acquired in a controlled environment. A powerful approach to process unlabeled video sequences and find a compact representation for each video frame relies on using an Unsupervised Learning (UL) method. Afterward, a model trained in an SL setting can take advantage of the available ground-truth force data. In the present work, UL and SL techniques are used to investigate a model in a Semi-Supervised Learning (SSL) framework, consisting of an encoder network and a Long-Short Term Memory (LSTM) network. First, a Convolutional Auto-Encoder (CAE) is trained to learn a compact representation for each RGB frame in a video sequence. To facilitate the reconstruction of high and low frequencies found in images, this CAE is optimized using an adversarial framework and a L1-loss, respectively. Thereafter, the encoder network of the CAE is serially connected with an LSTM network and trained jointly to minimize the difference between ground-truth and estimated force data. Datasets addressing the force estimation task are scarce. Therefore, the experiments have been validated in a custom dataset. The results suggest that the proposed approach is promising.},

keywords = {Learning, Robot, Robotic surgery, Robotics, Surgery, Training},

pubstate = {published},

tppubtype = {conference}

}

Close
Providing force feedback as a feature in current Robot-Assisted Minimally Invasive Surgery systems still remains a challenge. In recent years, Vision-Based Force Sensing (VBFS) has emerged as a promising approach to address this problem. Existing methods have been developed in a Supervised Learning (SL) setting. Nonetheless, most of the video sequences related to robotic surgery are not provided with ground-truth force data, which can be easily acquired in a controlled environment. A powerful approach to process unlabeled video sequences and find a compact representation for each video frame relies on using an Unsupervised Learning (UL) method. Afterward, a model trained in an SL setting can take advantage of the available ground-truth force data. In the present work, UL and SL techniques are used to investigate a model in a Semi-Supervised Learning (SSL) framework, consisting of an encoder network and a Long-Short Term Memory (LSTM) network. First, a Convolutional Auto-Encoder (CAE) is trained to learn a compact representation for each RGB frame in a video sequence. To facilitate the reconstruction of high and low frequencies found in images, this CAE is optimized using an adversarial framework and a L1-loss, respectively. Thereafter, the encoder network of the CAE is serially connected with an LSTM network and trained jointly to minimize the difference between ground-truth and estimated force data. Datasets addressing the force estimation task are scarce. Therefore, the experiments have been validated in a custom dataset. The results suggest that the proposed approach is promising.
Close
https://upcommons.upc.edu/bitstream/handle/2117/132610/iros2018_paper_26_07_2018[...]
doi:10.1109/IROS.2018.8593701
Close
 Singh, Gurkirt;  Saha, Suman;  Cuzzolin, Fabio
TraMNet - Transition Matrix Network for Efficient Action Tube Proposals Proceeding 
2018.
Abstract | Links | BibTeX | Tags: Computer Science, Computer vision, Electrical Engineering, Image processing, Pattern Recognition, Robot, Robotics, Systems Science, Visual processing
@proceedings{Singh2018,

title = {TraMNet - Transition Matrix Network for Efficient Action Tube Proposals},

author = {Gurkirt Singh and Suman Saha and Fabio Cuzzolin},

url = {https://arxiv.org/abs/1808.00297},

year  = {2018},

date = {2018-08-01},

abstract = {Current state-of-the-art methods solve spatio-temporal ac-tion localisation by extending 2D anchors to 3D-cuboid proposals onstacks of frames, to generate sets of temporally connected bounding boxescalled action micro-tubes. However, they fail to consider that the underly-ing anchor proposal hypotheses should also move (transition) from frameto frame, as the actor or the camera do. Assuming we evaluate n2D an-chors in each frame, then the number of possible transitions from each2D anchor to he next, for a sequence of fconsecutive frames, is in theorder of O(nf), expensive even for small values of f.To avoid this problem we introduce a Transition-Matrix-based Network(TraMNet) which relies on computing transition probabilities betweenanchor proposals while maximising their overlap with ground truth bound-ing boxes across frames, and enforcing sparsity via a transition threshold.As the resulting transition matrix is sparse and stochastic, this reducesthe proposal hypothesis search space from O(nf) to the cardinality ofthe thresholded matrix. At training time, transitions are speciﬁc to celllocations of the feature maps, so that a sparse (eﬃcient) transition ma-trix is used to train the network. At test time, a denser transition matrixcan be obtained either by decreasing the threshold or by adding to itall the relative transitions originating from any cell location, allowingthe network to handle transitions in the test data that might not havebeen present in the training data, and making detection translation-invariant. Finally, we show that our network is able to handle sparseannotations such as those available in the DALY dataset, while allowingfor both dense (accurate) or sparse (eﬃcient) evaluation within a singlemodel. We report extensive experiments on the DALY, UCF101-24 andTransformed-UCF101-24 datasets to support our claims.},

keywords = {Computer Science, Computer vision, Electrical Engineering, Image processing, Pattern Recognition, Robot, Robotics, Systems Science, Visual processing},

pubstate = {published},

tppubtype = {proceedings}

}

Close
Current state-of-the-art methods solve spatio-temporal ac-tion localisation by extending 2D anchors to 3D-cuboid proposals onstacks of frames, to generate sets of temporally connected bounding boxescalled action micro-tubes. However, they fail to consider that the underly-ing anchor proposal hypotheses should also move (transition) from frameto frame, as the actor or the camera do. Assuming we evaluate n2D an-chors in each frame, then the number of possible transitions from each2D anchor to he next, for a sequence of fconsecutive frames, is in theorder of O(nf), expensive even for small values of f.To avoid this problem we introduce a Transition-Matrix-based Network(TraMNet) which relies on computing transition probabilities betweenanchor proposals while maximising their overlap with ground truth bound-ing boxes across frames, and enforcing sparsity via a transition threshold.As the resulting transition matrix is sparse and stochastic, this reducesthe proposal hypothesis search space from O(nf) to the cardinality ofthe thresholded matrix. At training time, transitions are speciﬁc to celllocations of the feature maps, so that a sparse (eﬃcient) transition ma-trix is used to train the network. At test time, a denser transition matrixcan be obtained either by decreasing the threshold or by adding to itall the relative transitions originating from any cell location, allowingthe network to handle transitions in the test data that might not havebeen present in the training data, and making detection translation-invariant. Finally, we show that our network is able to handle sparseannotations such as those available in the DALY dataset, while allowingfor both dense (accurate) or sparse (eﬃcient) evaluation within a singlemodel. We report extensive experiments on the DALY, UCF101-24 andTransformed-UCF101-24 datasets to support our claims.
Close
https://arxiv.org/abs/1808.00297
Close