Autonomous Fire Fighting Robot Arm
Sensing and Localization of Fire & Manipulation of Fire Extinguisher
HDT Adroit 6DOF A24 Pincer Manipulation of a fire extinguisher using a thermal camera
This is a Winter Project of MSR Program at Northwestern University.
Code: [GitHub]
Background
The idea of a firefighting robot is not a new concept, and there have been many proposals for using firefighting robots. In fact, a robot called Colossus helped save much of the historic structure by braving conditions deemed too dangerous for human firefighters after Notre Dame Cathedral in Paris caught fire in 2019 [1], and the LA Fire Department tested autonomous firefighting robots in a major downtown blaze in 2020 [2].
The problems with currently existing firefighting robots are that they are not suitable for domestic or commercial use because they are large and built specifically to operate in firefighting missions. I believe that manipulating a robot arm to detect a fire and use a fire extinguisher can be beneficial in industrial settings such as warehouses that have already adopted robot arms. In particular, maintenance costs associated with fire sprinklers and frozen pipes can be reduced in warehouses. Additionally, implementing a fire safety feature in domestic service robot arms can appeal to a broader audience.
Therefore, I implemented a firefighting mechanism using HDT Adroit 6 DOF A24 Pincer robot arm (I will refer it as an Adroit throughout this post) and FLIR Lepton 2.5 thermal camera for this project. The goal of this project is to detect a heat source (fire), grab the fire extinguisher, and operate it.
Hardware
SYSTEM OVERVIEW
The Adroit operates on ROS. I used ROS-noetic version for this project.
ROS Packages Overview
Package: thermal_image_processing
This package takes care of detecting the hottest spot in the camera view and reading its temperature from the thermal camera. I implemented the package from the PureThermal 1 / PureThermal 2 FLIR Lepton Dev Kit (https://github.com/groupgets/purethermal1-uvc-capture).
Thermal Image Processing
FLIR Lepton 2.5 thermal camera can detect the temperature using its radiometry in a gray scale. The pixel value in the gray scale, for example, shows some value such as 40,000 which represents 400.00 K. The node thermal_detection detects the pixel with the highest temperature and converts it into Celsius scale. In addition, it publishes the temperature value and 2D x-y coordinates of the image as rostopic so that the Adroit can execute its movements based on those values.
Contour Detection From Thermal Image
The previous method of detecting the heat source was to get the highest-value pixel from the radiometric image. This method was rather unstable because the target pixel often fluctuated. Therefore, I calculated the contour from the radiometric image and its centroid to get a target pixel of the heat source.
Combined Vision Processing
Getting the pixel coordinate (u,v) from the centroid of thermal image’s contour, it’s fed into the Realsense depth camera to get depth data at (u,v).
To evaluate depth sensing, the result was compared to measurements with a ruler. The error range was about ±100 mm at maximum when measuring distances from 1.2 m to 0.3 m. The error became larger as the depth camera moved farther away from the target object.
When it comes to combining two cameras, the thermal image and depth image are offset from each other. Therefore, it is important to align the two cameras properly. rostopic combined_image2 provides combined images of the color image and thermal image to show how well the two cameras are aligned while combined_image provides combined images of color aligned to the depth image and the thermal image. The thermal camera and Realsense camera should be calibrated to avoid errors due to camera misalignment.
Testing with Electric Heater
Testing with Fire
Package: control
This package controls the Adroit’s arm joints through MoveIt and the pincer through the pincer joint controller. I have implemented the node arm_control for Adroit to detect the fire, grab the fire extinguisher, aim it to the fire, and press the lever.
Fire Detection
arm_control node subscribes to the temperature value and x-y coordinates that thermal_detection is publishing.
The Adroit makes joint movements on its joint 1 and 6 to scan the environment until it detects something with a high temperature value. The threshold for the temperature was set to 70°C (~158°F) for this project since I used a heater as my heat source, but this value would have been set higher if it were to work with a real fire.
When the Adroit detects something above 70°C, it will try to align the center of the camera view to the heat source by adjusting joint 1 and 3. The figure above represents its mechanism. When the robot aligns with the heat source, it remembers joint positions so that Adroit can return to this position after grabbing the fire extinguisher. In this case, the Adroit’s joint 1 is a little bit smaller than the remembered joint position because the fire extinguisher will be located slightly left to the thermal camera.
Grab & Aim
Since I am working with a single arm robot, it is necessary for the pincers to be able to perform two tasks: picking up the fire extinguisher & pressing the lever. Therefore, I implemented custom safety pins to prevent the Adroit from accidentally pressing the lever when it is trying to grab the fire extinguisher. I made two types of pins: the first one is an electric wire covered by rubber, and the other one is a 16-gauge copper wire. The copper wire one is sturdier, so the Adroit needs to exert more force to press the lever compared to the rubber one. I attached anti-slip stickers on the levers of the fire extinguisher to get a better grip.
I used a fixed coordinate for the location of the fire extinguisher, and I found this reasonable because fire extinguishers are usually located at specific and fixed spots. The Adroit successfully grabs the fire extinguisher 8 out of 10 times, and this error was due to incorrect orientation of the fire extinguisher.
I installed a laser pointer to accurately evaluate the Adroit’s aim to the heat source. When the Adroit goes back to its remembered joint positions, the fire extinguisher hose’s aim is aligned to the heat source.
Press
arm_control publishes std_msgs/Float64 to pincer_position_controller to control the pincers. When the Adroit presses the lever, the fire extinguisher’s aim goes down a little bit. Overall, this project achieved its goal.
Future Improvements
- Testing with a real fire
- Publishing transformation of the fire
- Separating the thermal camera from the robot’s pincer
- Implementing depth sensing of the fire
- Mounting the Adroit on a wheeled mobile robot
- Applying SLAM to the mobile robot
- Using a fire extinguisher with a flexible hose
- Picking up a fire extinguisher using object detection