None Notebook

This notebook contains material from CBE30338; content is available on Github.

< B.1 Diabetes: Controlling Blood Glucose Concentrations | Contents | Tag Index |

Open in Colab

Download

B.2 Visual Tracking of an Object with a Drone

CBE 30338 Chemical Process Control, Spring 2019

Elizabeth Innis, Kari Minnich, Omosefe Obanor, Alyssa Schuettpelz

B.2.1 Problem Statement

The object recognition and tracking of a specific object using a DJI Tello requires a method to communicate changes in the object's position as well as manipulation of the RBG values in a CV2 code. For object recognition, the DJI Tello will need to recognize the specific color of the object. The RBG values chosen for the object are unlikely to be exact, but it will need a specific range to be close enough in order for the Tello to recognize the specific object. To track the object, the DJI Tello will need to process that there is an offset in the original distance of the object from the center of frame and react accordingly to refocus.

B.2.1.1 Technology Challenge

This project uses software and coding to operate a DJI Tello Drone. Working with hardware presents a set of challenges that modeling with only software or running simulations does not provide. The issues of dying batteries and overheating had to be overcome before the code could be tested to run commands. It should be noted that if the battery life of the Tello is too low, the drone will not take flight. Before running code, charge all of the battery packs and have replacements ready if the drone will not lift off. In addition to this, there was a point during exploration that the batteries were known to be fully charged, yet the drone would not lift from the ground. The DJI Tello will not take flight if it senses that it is overheating. Carpeted floors have been observed to prevent the DJI Tello from cooling down. This can be circumvented by taking off from solid surfaces or placing a notebook, folder or other convenient smooth, flat object under the drone as a launch surface. Finally, several libraries must be downloaded to run the video feed of the Tello with CV2. Those libraries can be found in the appendix.

B.2.1.2 Project Goal

The goal of this project is to manipulate the operation of a working drone to enable recognition and tracking. The target detection will be optimized to incorporate internal camera streaming, an analysis of the stream, and subsequent reaction. Existing code repositories are utilized as a reference at a foundational level to guide the target acquisition method. Several applications exist in which such process control for target detection is beneficial. Key areas of use are in search and rescue, as well as other humanitarian efforts.

B.2.2 Results and Theory

Operating the DJI Tello with keyboard commands is an important first step to completing the goal for the tracking of an object. The keyboard commands help the user to place the drone where he or she prefers, (proper height, angle, distance from object, etc.) in a ready position for tracking.

Once the DJI Tello can be controlled remotely, the next step to color tracking is to display the camera feed that the Tello sees on the computer using CV2. The pixels in the image can be searched for pixels of a certain color by analyzing RGB values. In this project we used a range of blue values. The code will blur together pixels within the largest area blue color range and draw a circle around the blue object we hope to track.

Tracking the object is done using a PID control loop incorporating the offset coordinates of the center of the object and the center of the frame. This is done by using the coordinates from the previous frame and comparing it to the current frame. These coordinate distances are computed in pixels. The goal of tracking is to track where the blue object is "now" based off of where it was in the last frame.

The control loop uses the offset coordinates and the distance from the center to find velocities. These velocities are used to command the drone to move left, right, up, or down. It runs the loop for each frame and moves the drone accordingly. The drone should move either up, down, left, or right by observing the sign of the velocity in either the x or y direction. The velocity at which the drone moves is the magnitude of the value computed by the control loop. All of this should result in a drone tracking a blue object.

More details about each step of or project are found below. For the full commented code see the appendix.

B.2.2.1 Controller

This code utilizes key presses to control the movements of the drone when tracking is not enabled. These are described by the following code:

B.2.2.2 Image Recognition

This code conducts image recogniton using CV2 and the video feed from the Tello camera. A range of color values is determined for the desired object and the video stream is converted, frame by frame, into an array of RGB values. The location of the object is determined by analyzing the RBG values and looking for values within the defined range. A circle is drawn surrounding object within the RBG range, and the drone attempts to maintain that circle around the object as it adjusts in flight.

B.2.2.3 PID Control for Object Tracking

We wanted to use a feedback control loop to direct the drone towards the recognized object from tracker.py to do this we will used a PID controller.
This code finds an optimal x and y velocity (pixles/second) then tells the drone to go right, left, up, or down, at the designated velocity. The governing equations for the PID loop were found and adjusted from the course notes linked below. https://nbviewer.jupyter.org/github/jckantor/CBE30338/blob/master/notebooks/04.01-Implementing_PID_Control_with_Python_Yield_Statement.ipynb

B.2.3 Executable Element

https://github.com/einnis01/CBE30338_Drone_Project/blob/master/DroneSmaller.mp4

The link above contains a video of the DJI Tello drone tracking a blue ball.

B.2.4 Conclusion and Summary

Our goal for this project was to enable object recognition and tracking. We were able to accomplish our goal with the implementation of a PID controller and adjusting the RGB values in the CV2 code for our specific object's color.

The RGB values we chose were lower bounds: [90,50,50] and upper bounds: [110,255,255]. These values gave us the best range for the specific blue of our object. The DJI Tello was able to recognize our object very well, but it is important to note that the object could not be in front of a black background or in extremely bright yellow lights. Through various trials, it was evident that a black background and/or bright yellow lights weakened the Tello's ability to recognize the object due to the color feedback the Tello was receiving, which was no longer in the specified range.

We successfully tracked our object using the object tracker in the CV2 code and the PID controller. Moving the object slowly away from the drone allowed the object tracker and PID controller time to process the changes and provide feeback to the drone. There were slight issues with oscillations when the object was moving too fast away from the center of frame. Therefore, slight adjustments were made to the constants in the PID controller to improve the stability of the drone.

B.2.5 Appendices

All code can be found at https://github.com/einnis01/CBE30338_Drone_Project

B.2.5.1 A.) TelloCV.py

This code is the main controller for drone.

B.2.5.2 B.) Tracker.py

This code is responsible for the image recognition and object tracking.

< B.1 Diabetes: Controlling Blood Glucose Concentrations | Contents | Tag Index |

Open in Colab

Download