Nowadays, autonomous underwater vehicles (AUVs) are expected be one of the essential tools in applications such as inspection of underwater structures (e.g., dams and bridges) and underwater cable tracking [1]. Even though many studies have been conducted and published worldwide, researchers face rapidly increasing demands to expand the roles of AUVs. In spite of developing technologies related to power storage devices, underwater vehicles are especially limited in operations that take longer than the duration supported by the vehicle’s power capacity. A recharging unit with an underwater docking function can enable AUVs to operate for extended periods in the sea independently of a surface vessel, making the docking operation important not only for battery recharging applications but also other applications such as sleeping under the mother ship or new mission downloading. Moreover, the docking capacity can be extended to provide navigation for other underwater vehicles on the way to their own missions. However, a number of challenging issues hinder these applications, which require high accuracy and robustness against disturbances that occur in the underwater environment. To achieve these tasks in underwater vehicles, we have developed a vision-based docking system using stereo vision.

 Recently, due to the progress in computer vision, a vision-based system has been highlighted as a promising navigation system. As in land and space systems, numerous studies on underwater vehicles using visual servoing have recently been conducted worldwide. Each study has different merits and limitations depending on the intended application. Most research is based on monocular vision. In contract, we have developed a 3D pose tracking system for docking operation using stereo vision providing high homing accuracy. To the best of our knowledge, our proposed system is the world first initiated research using two cameras as stereo vision in underwater vehicle environment. We have developed an efficient and robust real-time autonomous docking system by means of visual servoing using stereo vision, named as Three-Dimensional Move on Sensing (3D-MoS). The system recognizes a relative pose between a ROV and a target object through a newly proposed 3D model-based recognition method using Real-time Multi-step GA in real-time.

 We conducted many docking experiments in the pool and in real sea using an ROV and also an AUV. The robustness of visual servoing has been evaluated against different disturbances while the underwater vehicle is controlled by our stereo vision based visual servoing. We conducted docking experiment in the sea of Wakayama prefecture to evaluate how much our 3D-MoS system would be robust against natural sea environment. According to the experimental result, it was confirmed that docking performance in sea using proposed system was achieved successfully with centimeter level accuracy in recognition and visual servoing.

On November 9th 2018, we conducted 2 times continuously repeated Z-shaped docking using a ROV (DELTA-150) as shown in the following videos.

●Video: Z-shaped docking in the pool.


On December 18th 2017, we conducted 39 times continuously repeated docking using a newly developed ROV (DELTA-150) as shown in the following videos.

●Video 1 shows docking number 9 to 10 when ROV’s lighting source was not used.




●Video 2 shows docking number 12 to 13 when lighting from ground was used to see ROV.




●Video 3 shows docking number 30 to 31 when ROV’s lighting source was used during docking experiment.








Fig. A newly developed ROV (DELTA-150) and docking station in the sea, Ushimado, Japan



Confirmed:


According to this experiment, we confirmed that sea docking using a new AUV (DELTA-150) can be conducted with the conditions of turbidity 4.0 FTU and night environment with and without using ROV’s lighting source. We conducted 39 times continuously repeated docking successfully.


Approach:


ROV: In this experiment, we developed a ROV (DELTA-150) manufactured by QI. Since the configuration of thrusters in the new ROV is not simple same as the one we used in previous experiments, the hardware structure of the DELTA-150 has been developed to be stable in rolling, pitching and yawing. In addition to hardware modification, we improved the control system considering the dynamic coupling between each other.


Active Marker and Fitness Function: We used an active marker and newly modified RM-GA using H(hue), S(saturation), V(value) and Brightness.


Homing method: After docking completion, the vehicle is attached to the square shaped docking hole that has a close surface at the back by pushing the vehicle forward with constant thrust. During homing at the docking hole, all experimental data including left and right camera images are stored into the hard disk of PC for further analysis.


We installed three underwater cameras in different directions in docking station to check docking performance during experiment and to do offline analysis after experiment. These three cameras are connected to another ground PC and the images are recorded by using some recording software.



Condition: The condition of this 39 times continuously repeated docking experiment is shown in the right side of video.




*This research has been supported by Mitsui Engineering & Shipbuilding.



 At present almost all the metals and industrial minerals are extracted from onshore resources. To become independent of those onshore resources, it is important to utilize the resources from the ocean. Among them, searching expensive rare metals and methane hydrate in the seabed are economically important. Japan is seriously considering deep-sea mining of methane hydrate that is expected to be a future energy resource. Therefore, advanced technologies are needed in the field of deep sea research. To fulfill social demands from deep sea as discussed above, underwater robots have been developed worldwide. AUVs have become essential in deep ocean work such as ocean bottom exploration and underwater surveying. However, AUVs are limited in operation time because of its energy independent and their limited power capacity. Going back surface for battery recharging reduces the ability of AUVs in terms of time-consuming and manpower on the surface ship. To overcome this problem, a underwater battery recharging unit with a docking function is one of the solutions for AUVs to continue their tasks in the seabed without need to go back surface station for recharging.

 Even though AUVs do not need to descend to the sea bottom for some tasks such as bottom topology survey, almost ocean exploration works like oil pipe inspection, searching precious metals and picking up them need AUVs to work near sea bottom. At this time, the most challenging and unavoidable problem is turbidity that deteriorates the ability of the AUVs in terms of visibility. Turbidity is defined as the cloudiness in a liquid caused by the presence of suspended particles that scatter and absorb light. Therefore, how to verify the turbidity tolerance and how to overcome the disturbance of turbidity are important research questions not only for AUV environment but also for the field of vision-based underwater systems. Since the intended application in this study is underwater battery recharging in sea bottom to extend the working period of AUVs, we cannot avoid the turbidity by just doing research in clean water.

 According to above discussion, turbidity is a practical problem for AUVs that are really used at the sea bottom. There is no study, as far as authors know, on turbidity tolerance of real-time visual servoing based docking at the sea bottom by conducting in the sea practically. We also have yet to discuss the robustness of our stereo-vision based pose estimation method against turbidity and conducted in turbid sea in previous studies. Therefore, as the main contribution of this study, practicality of sea docking against turbidity that has not discussed in other studies is presented in this report. Visual servoing in a turbid environment using dual eyes-based pose estimation is verified experimentally with the analysis on the performance of the proposed method against different turbidity levels. Although visual servoing is not the only solution for docking technologies of AUVs, it is practically important to evaluate the tolerance ability before combining the visual servoing technology with integrated control system. Not only pool tasks but also sea docking tasks in turbid environment were conducted to verify the effectiveness of the turbidity on pose estimation for real-time visual servoing and turbidity tolerance of the proposed method. This study is expected to extend the sphere of field of robotics related to the underwater environment.

 Continuous repeated docking was conducted in the shallow sea near Ushimado town, Japan. Since the study focus on turbidity tolerance, the challenging turbid coastal environment was selected to conduct sea docking experiment rather than clear oceanic water. We conducted experiments up to 5 times in the sea with improvement ( including using a new active 3D marker) for each time to robust against high turbidity ( The maximum turbidity when the final experiment was conducted is 15 FTU measured by turbidity sensor TD-500) . Each docking experiments in the sea except the first experiment ( we surveyed the condition of sea and check pre tasks ) are shown by the following movies.

 This work was supported by JSPS KAKENHI Grant Number JP16K06183. This research has been cooperated by MITSUI Engineering and Shipbuilding Co.,LTD and Kowa cooperation.

●Video 1 shows fourth trial continuously repeated docking experiment in the Ushimado sea. Docking number 1,2 in day time and 20, 21 in night time are selected and illustrated in this video.


Confirmed:

According to this experiment, we confirmed that docking in the sea can be conducted when the condition of turbidity 15 FTU and evening to night environment. We conducted 37 times continuously repeated docking successfully.

Approach:

In this experiment, we used an active marker and newly modified RM-GA using H(hue), S(saturation), V(value). The previous trials (Video 2- 4) are based on only Hue information.

Homing method: After docking completion, the vehicle is attached to the square shaped docking hole that has a close surface at the back by pushing the vehicle forward with constant thrust. During homing at the docking hole, all experimental data including left and right camera images are stored into the hard disk of PC for further analysis.

We installed three underwater cameras in different directions in docking station to check docking performance during experiment and to do offline analysis after experiment. These three cameras are connected to another ground PC and the images are recorded by using some recording software.

Condition:

The condition of this 37 times continuously repeated docking experiment is shown in the right side of video. The experiments were conducted after a rainy day so that the turbidity is the highest one comparing to previous experiments (Video 2- 4).

Note:

Docking experiment could not be conducted during 3:00 PM to 4:00 PM with the conditions of turbidity (13.0 to 14.1 FTU), illuminance at the sea surface (17100 to 22000 Lx), illuminance at 1m depth (1600 to 3200 Lx), and depth of sea bottom (2.7 m).


 

 

 

●Video 2 shows third trial continuously repeated docking experiment in the Ushimado sea. Docking number 1,2 in day time and 82, 83 in night time are selected and illustrated in this video.


Confirmed:

According to this experiment, we confirmed that docking in the sea can be conducted when the condition is turbidity 8 FTU and day to night environment. We conducted 83 times continuously repeated docking successfully.

Approach:

In this experiment, we used an active marker after confirming the performance of our proposed system in the condition of high turbidity, and day and night environment.

We applied our homing method to attached the vehicle to the docking hole (two square shaped ones) by pushing the vehicle forward with constant thrust.

Condition:

The condition of this 83 times continuously repeated docking experiment is shown in the right side of video. The experiments were conducted day to night environment.

Note:

The LEDs are switched on from the docking in day time, but the three spheres of 3D marker can be seen as a marker that does not light in the day time condition.

At night, light of green sphere has been found unfortunately to be not lighting, but the docking continued 83 times from 6:24 PM to 7:45 PM.


 

 

 

●Video (ushimado-3rd): Video 3 shows second trial continuously repeated docking experiment in the Ushimado sea.


Confirmed:

According to this experiment, we confirmed that docking in the sea can be conducted when the condition is turbidity 7 FTU and evening environment. We conducted 20 times continuously repeated docking successfully.

Approach:

In this experiment, we used our newly designed active marker after confirming a better performance of active maker over passive marker in the condition of high turbidity and day and night environment.

We applied our homing method to attached the vehicle to the docking hole (two square shaped ones) by pushing the vehicle forward with constant thrust.

Condition:

The condition of this 20 times continuously repeated docking experiment is shown in the right side of video. The experiments were conducted in the evening when there was less lighting.

Note:

The currents for each LED of 3D marker’s ball and the brightness of 3D marker is lower that the experiment in video1.


 

 


●Video 4 shows the first sea trial of continuously repeated docking in the Ushimado sea. In this video, the first 5 times docking experiments are shown.


Confirmed:

In this experiment, we confirmed that docking in the sea can be conducted when the condition of turbidity is about 7.7 FTU in day time. We conducted 20 times continuously repeated docking succesfully.

Approach:

In this experiment, we used a passive 3D marker and ROV’s LED. Our proposed RM-GA is based on hue value. We used a circular shaped docking hole with a diameter of 130 mm, and homing method (as in Video1) was not included. Therefore, the vehicle stops visual servoing for a few seconds after docking completion to store experimental data.

Condition:

The condition of this 20 times continuously repeated docking experiment is shown in the right side of video. There was a strong wave during this experiment comparing to other experiments (Video 1, 3-4).

Note:

There were some problems in storing experiment data. We could store 19 times docking data and the first 7 times left and right camera images.

We installed three underwater cameras in different directions in docking station and ground one camera to check docking performance during experiment and to do offline analysis after experiment. However, there was a problem in recording one of three underwater camera. Therefore, this experiment video includes images from left and right camera, two underwater cameras and ground camera.


 

 


●Video 5 shows the first sea trial of continuously repeated docking in the Uhimado sea. In this video, the first 4 times docking experiments are shown.


Confirmed:

In this experiment, we confirmed that docking in the sea can be conducted when the condition of turbidity is about 6 FTU in day time. We conducted 4 times continuously repeated docking successfully.

Approach:

In this experiment, we used a passive 3D marker and ROV’s LED. Our proposed RM-GA is based on hue value. We used a circular shape docking hole with a diameter of 130 mm, and homing method (as in video 1) was not included. Therefore, the vehicle stops visual servoing for a few seconds after docking completion to store experimental data.

Condition:

The condition of this 4 times continuously repeated docking experiment is shown in the right side of the video. There were some waves during this experiment.

Note:

We installed three underwater cameras in different directions in docking station to check the docking performance during the experiment and to do offline analysis after experiment. However, there was a problem in recoding one of three underwater camera. Therefore, this experiment video includes images from left and right camera, two underwater cameras.


 

 


●Video1: Stereo Vision-based Docking Experiment of AUV (Tuna Sand2) for Sea Bottom Battery Recharging (Conducted in Tokyo University on October 4th, 2016)



The AUV docking experiments using our proposed stereo vision based visual servoing for sea bottom battery recharging application was conducted on October 4th, 2016. Our real-time 3D pose tracking system was installed in an AUV “Tuna-Sand 2.” An underwater battery recharging unit with a unidirectional docking function was designed and fixed in a pool. Tuna-Sand 2 approached to the docking station using other sensors and final docking operation was performed by our newly proposed stereo vision based visual servoing system. The main task in this experiment is to insert the docking pole (that is attached in AUV) into the docking hole (that is fixed with a 3D marker in the docking station) automatically by visual servoing. There are two steps in this experiment: (1) Approaching to the station following preset waypoints using other sensors, and (2) Docking step by visual servoing. In the first step, AUV followed the preset way points to approach the station using other sensors until 3D marker was detected by two cameras. When AUV approached to the station and 3D marker was in the field of view of two cameras, AUV switched to docking step in which the vehicle is controlled by visual servoing to insert docking pole into the docking hole precisely. The experimental results showed the performance of the proposed system with accurate docking accuracy. This project was conducted in Tokyo University in cooperation with Project Assistant Prof. Yuya NISHIDA from Kyushu Institute of Technology and Assoc. Professor,Toshihiro MAKI from Tokyo University.

(The study of this research was reported in the following papers.)

[1] Xiang Li , Yuya Nishida , Myo Myint , Kenta Yonemori , Naoki Mukada , Khin Nwe Lwin , Matsuno Takayuki, and Mamoru Minami, Dual-eyes Vision-based Docking Experiment of AUV for Sea Bottom Battery Recharging, the International Conference OCEANS17 MTS/IEEE, Aberdeen, Scotland, June 19-22, 2017. PDF (will be published soon)


●Video2: Stereo Vision-based Docking Experiment in a real sea for Sea Bottom Battery Recharging Application (Conducted in the sea near Wakayama city in Japan on December 16th, 2015)



The docking experiments using stereo vision-based docking approach for sea bottom battery recharging application was conducted on December 16th, 2015. A docking station was designed as a unidirectional one to which the AUV has to dock in a specific entry. Real-time pose tracking using stereo vision was developed using two cameras mounted on an underwater vehicle and a known 3D marker fixed at the docking station. Real-time relative pose (position and orientation) estimation was implemented utilizing 3D model-based matching method and Real-time Multi-step Genetic Algorithm. A remotely Operated Vehicle (ROV) was used as a test bed. To verify the proposed approach for underwater battery recharging, a docking pole attached to the vehicle and a docking hole with a diameter of 70 mm fixed with a 3D marker at the docking station were designed. The main task in this experiment is to insert the docking pole into the docking hole automatically by visual servoing. Firstly, the vehicle approaches the docking station manually until the 3D marker is in the field of view with 1 m distance. Then, final docking operation was performed by visual servoing. Totally four times success docking experiments using the proposed approach that simulate for underwater battery recharging were conducted in the real sea. The experimental results have confirmed the functionality and possibility of the proposed approach for the sea bottom docking application of AUVs, having proved the proposed homing approach to be practical under real-world sea conditions. This project was conducted in the sea near Wakayama City, Japan in cooperation with KOWA cooperation for development of the ROV.

(The study of this research was reported in the following papers.)

[1] Myo Myint, Kenta YONEMORI, Akira YANOU, Khin Nwe Lwin, Maoki Mukada and Mamoru MINAMI Dual eyes visual based sea docking for sea bottom battery recharging, Proceedings of the International Conference OCEANS16 MTS/IEEE, Monterey, USA, 2016. [PDF]

[2] Kenta YONEMORI , Akira YANOU , Myo MYINT , Khin Nwe LWIN and Mamoru MINAMI, Docking experiment of underwater vehicle by dual-eye visual servoing in sea, Transactions of the JSME (in Japanese), Vol.83, No.848, 2017. [PDF]


●Video3: Docking experiment from different arbitrary stating positions according to docking strategy that includes approaching step, visual servoing step, and docking step



This work is concentrated on the dual-eye visual servoing as a possible new docking strategy rather than conventional docking methods. The proposed docking strategy consists of three steps. First, the ROV has to approach the 3D target until the target is in its field of view. Second, detecting the object and regulating the vehicle to the defined relative pose of the target is performed in the visual servoing step. Third, the docking operation is completed. In this experiment, after approaching docking station with constant speed and a constant proceeding direction while trying to detect the 3D marker, the vehicle is stabilized in the visual servoing step and controlled to keep the ROV with a defined pose relative to the target. In the docking step, when the vehicle is stable within the tolerance of the position error for the defined time period, the forward thrust that enables the docking pole attached to the ROV to fit into the dock is generated by gradually decreasing the distance between the vehicle and the target object. A pool (2 m (L) × 3 m (W) × 0.75 m (H)) filled with tap water was used as an experimental tank for the underwater vehicle experiments. The ROV was tethered by a cable 200 m in length to receive image information and output control signals. Experiments were carried out with different start positions: (1) on the left side of the pool relative to the 3D marker, (2) in front of the 3D marker, and (3) on the right side of the pool relative to the 3D marker. The experimental results showed the proposed system could successfully carry out docking operations.

(The study of this research was reported in the following paper.)

[1] Akira YANOU , Shota OHNISHI , Shintaro ISHIYAMA and Mamoru MINAMI, Autonomous docking control of visual-servo type underwater vehicle system aiming at underwater automatic charging, Transactions of the JSME (in Japanese), Vol.81, No.832, 2015 [PDF]


●Video4: Docking performance against different disturbances (background, air bubbles, physical disturbance)



As in the space environment, the underwater world gives complexity to underwater vehicle operation due to disturbances. Because the proposed system is a vision-based system, not only the physical disturbances of ocean currents but also the noise in recognized images should be considered in the experiments. By completing the experimental tasks while including these considerations, the proposed docking system demonstrates its effectiveness against different disturbances. In this experiment, docking performance against different disturbances was verified. Three kinds of disturbances namely as background, air bubbles, and physical disturbance were given to the docking environment. The background sheet that has sea patterns was set behind the 3D marker to verify recognition of 3D marker against the real sea patterns. Air bubbles were generated in front of the 3D marker to provide random noise to the captured images. Additionally, physical disturbances simulating water currents by pushing the ROV using a rod in different directions during docking operation were given to the experimental conditions. The experimental result shows the successful performance of docking against different disturbances.


●Video5: Docking performance against different disturbances (background, air bubbles)



The robustness of 3D post estimation against air bubbles and background disturbances was verified. Air bubbles address not only noise to the captured images but also physical distance to the movement of the ROV. The background sheet that has sea patterns was set behind the 3D marker to verify recognition of 3D marker against the real sea patterns. In this experiment, the air bubbles were generated when the ROV approaches the station by visual servoing.


●Video6: Tracking a moving target by visual servoing against air bubbles disturbance



In this experiment, the tracking ability of the ROV using proposed stereo-vision based visual servoing was verified experimentally. The ROV can recognize the 3D pose of the 3D marker that is moving in a camera depth direction even though there are some air bubbles in front of the cameras, and track the moving 3D marker by visual servoing. Real time pose estimation through Real-time Multi-step GA was input directly in the feedback of the controller. 3D motion of the ROV is controlled by P controller. The experimental results show that the ROV can track the moving 3D marker by visual servoing.


●Video7:Tracking a moving target by visual servoing against air bubbles disturbance



In this experiment, the tracking ability of the ROV using proposed stereo-vision based visual servoing was verified experimentally. The ROV can recognize the 3D pose of the 3D marker for duty cycles of 20 s and an amplitude of 280 mm from the ROV that is moving in a camera depth direction even though there are some air bubbles in front of the cameras, and track the moving 3D marker by visual servoing. Real time pose estimation through Real-time Multi-step GA was input directly in the feedback of the controller. 3D motion of the ROV is controlled by P controller. The experimental results show that the ROV can track the moving 3D marker by visual servoing.

(The study of this research was reported in the following paper.)

[1] M. Mint,K. Yonemori,S. Ishiyama,A. Yanou,M. Minami,Real-time 3D Pose Estimation and Tracking 3D Marker using Dual-eye Cameras for Under Water Vehicle, Proceedings of the 24th Meeting of the Institute of Measurement and Automatic Control,pp.62-63,2015.11.28 [PDF]


●Video8: Evening news program broadcasted our research on August 27,2014. (By Sanyo Broadcasting)



This video is an evening news program broadcasted our research on August 27th, 2014 by Sanyo Broadcasting in Japan.


●Video9: Tracking a moving 3D marker by stereo-vision based visual servoing



In this experiment, The ROV can recognize the 3D pose of the 3D marker that is moving in the pool, and track the moving 3D marker by visual servoing in real-time. Real time pose estimation through Real-time Multi-step GA was input directly in the feedback of the controller. The experimental results show that the ROV can track the moving 3D marker by visual servoing.


2016年11月23日 日刊工業新聞で、水中ロボットが紹介されました。

 

2016年2月8日 日本経済新聞で、水中ロボットが紹介されました。

 

2014年11月11日 週刊ロビ「ロビクルをつくる」で水中ロボットが紹介されました。

週刊ロビ「ロビクルをつくる」記事

2014年10月21日 電気新聞で、水中ロボットが紹介されました。

水中ロボット科学新聞掲載記事

2014年8月19日 日経新聞で、水中ロボットが紹介されました。

        

2014年8月28日 科学新聞と山陽新聞で、水中ロボットが紹介されました。

水中ロボット科学新聞掲載記事 水中ロボット山陽新聞掲載記事
pagetop