Computer Vision Project

Abstract

So far, you have successfully programmed robot (simulators) using reactive control strategies and taken a deep dive into the robot localization problem. Next, you will be learning about computer vision and the role it plays in robotics. In contrast to previous project, this time you will have a great deal of freedom to shape your project towards your interests and learning goals.

Learning Objectives

  • Self-directed learning of a new robotics algorithm.

  • Implementing a robotics system with minimal scaffolding.

Teaming

For this project, you should work with one other student. Since we have an odd number of students in the class, one team of three will be allowed.

Please fill out this Google Sheet when you have a project team and Github repo.

Project Topic

Your project should be about computer vision and its intersection with robotics. In our simulated robot reality of the pandemic, this means that you should choose a computer vision topic that has some plausible application in robotics. You will carry out that project either using a robot simulator, a computer vision dataset, or some combination of both.

In your project proposal you will be coming up with an implementation plan. That is, if you are using a particular algorithm to solve a problem, which parts of the algorithm will you implement, and which will you use pre-built implementations for? Be strategic in these decisions to balance learning about algorithms with system building (e.g., programming the simualted Neatos (or another robot) to do something interesting). You have substantial lattitude in shaping your project to focus on the parts you really want to learn (e.g., system design versus basic understanding of algorithms). That said, we expect that you will do some exploration of algorithms as part of this project (which could could include implementation or perhaps subsantial learning about an algorithm or class of algorithms).

Potential Algorithm Topics

  • Object tracking

  • Image segmentation

  • Object detection

  • Text recognition

  • Fiducial tracking

  • Visual odometry

  • Structure from motion

(Note: there are some resources for these topics later in the document)

Robot Platform

  • One option is to continue to work with the Neato simulator in Gazebo, and use the simulated world for your computer vision project. You will probably want to build a more complex world (see instructions for doing that in the how-tos for the Neato Simulator . While we’ve mostly been using simple shapes in previous projects, there are a lot of built-in models that you can use to build a world.

  • We’ve been working with the Neato simulator, but there are many other simulators out there, which you’re welcome to use in this project. You can find some of them here: Other Robot Simulators.

  • You may want to use an external dataset for your project. Here are some possible starting points.

    • Visual Data

      has a nice collection of computer vision datasets and projects.

    • If you want a huge (but very cool) dataset for self-driving vehicles, consider usign Waymo’s Open Dataset

    • Here's another list of datasets specifically for robotics (not all are about computer vision)

    • If you’re interested in machine learning for robot control, you might consider building off some of the datasets (and code) from the 2018 Robot Learning project You also might want to check out the Google Slides presentation summarizing their results.

    • A lof these datasets are big and fancy, but don’t be afraid to start with smaller, more classic datasets. That’s a great way to learn without having to deal with the greater complexity that comes with some of this data.

Deliverables

There are four deliverables for this project.

Project Proposal (due 10/26)

At a minimum, please include the answers to the following questions. You should include enough detail for us to be able to give you useful feedback.

  • Who is on your team?

  • What is the main idea of your project?

  • What are your learning goals for this project?

  • What algorithms or computer vision areas will you be exploring?

  • What components of the algorithm will you implement yourself, which will you use built-in code for? Why?

  • What is your MVP?

  • What is a stretch goal?

  • What do you view as the biggest risks to you being successful (where success means achieving your learning goals) on this project?

  • What might you need from the teaching team for you to be successful on this project?

In-class Presentation / Demo (11/9)

We’d like each team to spend about 10 minutes presenting what they did for this project. You can structure the presentation in whatever manner you’d like, however, you should try to meet these goals:

  • Explain the goal of your project

  • At a high-level explain how your system works

  • Demonstrate your system in action (either in a video [recommended] or live). If your system doesn’t work completely yet, that is fine, try to show at least one component of your system in action.

  • This presentation / demo should be very informal. This presentation will be assessed in a purely binary fashion (basically did you do the things above).

Code (Due 11-9)

  • You should turn in your code and writeup via Github. Please fork your repo from this one.

Writeup (Due 11-9)

In your ROS package create a README.md file to hold your project writeup. Your writeup should touch on the following topics. We expect this writeup to be done in such a way that you are proud to include it as part of your professional portfolio. As such, please make sure to write the report so that it is understandable to an external audience. Make sure to add pictures to your report, links to Youtube videos, embedded animated Gifs (these can be recorded with the tool peek).

  • What was the goal of your project? Since everyone is doing a different project, you will have to spend some time setting this context.

  • How did you solve the problem (i.e., what methods / algorithms did you use and how do they work)? As above, since not everyone will be familiar with the algorithms you have chosen, you will need to spend some time explaining what you did and how everything works.

  • Describe a design decision you had to make when working on your project and what you ultimately did (and why)? These design decisions could be particular choices for how you implemented some part of an algorithm or perhaps a decision regarding which of two external packages to use in your project.

  • What if any challenges did you face along the way?

  • What would you do to improve your project if you had more time?

  • Did you learn any interesting lessons for future robotic programming projects? These could relate to working on robotics projects in teams, working on more open-ended (and longer term) problems, or any other relevant topic.

Resources and Potential Project Directions

Resources

  • In-Class Activities from Past CompRobo Offerings
    • Object Tracking
    • 3D Structure from Motion
    • Image Filtering
    • Object Recognition

      (the instructions on running the code are out of date, but the ideas might help).

  • Convert ROS image messages to OpenCV using CvBridge
  • Connecting a webcam to ROS and OpenCV
  • An overview of methods for object tracking
  • Slides from the CompRobo Learning Project 2018 Class Report Out
  • Mastering OpenCV with Practical Computer Vision Projects

  • Visual odometry resources (one example)

  • Canny edge detection
  • Contours
  • Template matching
  • Hough line transform
  • Basics of histograms

    and histogram equalization

  • Basic Numpy tutorials
  • GUI Features in OpenCV
  • Basic Operations on Images
  • Arithmetic Operations on Images
  • Corner Detection
  • Numpy Examples List

Previous Student Projects to Draw From

  • Visual Navigation for Flying Robots

    is a course on said topic. The linked page includes lectures and even some bag files.

  • Robot Learning Report out 2018

  • Computer Vision Project Writeups from 2017

    • Self Driving Neato
    • Computer Vision Emotion Detection
    • Visual Localization
    • Predicting Paths of Tracked Objects
    • Neato Keeper
    • Lane Follower
    • Meal Recognition
    • Pac Neato
    • Neato Augmented Reality Parking