OpenCV

One of the biggest library for real-time computer vision. Written in C/C++.

OpenCV.js

OpenCV compiled to asm.js or Webassembly with the help of Emscripten.

Emscripten

• Comparable to what we did with wasm-bindgen
• Glue Code on JS and wasm side
• Allows high-level interaction between JS and C/C++

Let's do some image processing!

Checkout git repo

https://github.com/r3dDoX/wasm-exercise.git

Setup

                
                    npm i
                    npm start
                
                

Visit: http://localhost:8080

You should see "runtime ready" in the browser console after a couple of seconds

Project Structure

• bin
  • opencv.js
• index.html
• main.js
• ...

opencv.js

• JS bindings
• base64 encoded wasm binary
• 8MB 😬
• creates window.cv object

Get video stream

WebRTC

• Capture audio/video streams
• Exchange data between browsers
• Basic features in all major browsers

Get stream

                
                    navigator.mediaDevices
                        .getUserMedia({ audio: false, video: true })
                        .then(stream => {
                            ...
                        })
                        .catch(console.error);
                
                

Find width/height

                
                    const videoTrack = stream.getVideoTracks()[0];
                    const settings = videoTrack.getSettings();

                    // settings.width / settings.height
                
                

Allow video element to play stream

                
                    inputVideo.srcObject = stream;
                
                

Exercise Time!

• Get the stream as mentioned above
• Set it on the video element

Let's add OpenCV to the mix

VideoCapture

• OpenCV.js can read from video element
• Creating a matrix from the picture
• We can then apply OpenCV functions to this matrix

Setup VideoCapture

                
                    let cap;

                    function startStreaming() {
                        ...
                        cap = new cv.VideoCapture(inputVideo);
                        ...
                    }
                
                

This is an OpenCV.js specific functionality

Setup output canvas

                
                    function startStreaming() {
                        ...
                        outputCanvas.setAttribute('width', videoWidthPx);
                        outputCanvas.setAttribute('height', videoHeightPx);
                        ...
                    }
                
                

Setup OpenCV Matrices

                
                    let src;
                    let dst;

                    function startStreaming() {
                        ...
                        src = new cv.Mat(settings.height, settings.width, cv.CV_8UC4);
                        dst = new cv.Mat(settings.height, settings.width, cv.CV_8UC1);
                        ...
                    }
                
                

OpenCV Constants

• cv.CV_8UC4 / cv.CV_8UC1
• 8U is an 8-bit unsigned integer
• C4/C1 stands for the channels
• Our source has 4 channels per pixel (RGBA)
• Our output will only have 1 channel per pixel

Create function to process frames

1. Read frame from VideoCapture
2. Apply OpenCV color transformation
3. Draw to canvas
4. Loop

1. Read frame from VideoCapture

                
                    cap.read(src);
                
                

Read frame and write it to src-matrix

2. Apply OpenCV color transformation

                
                    cv.cvtColor(src, dst, cv.COLOR_RGBA2GRAY);
                
                

Transform src- to dst-matrix reducing colors to grayscale

3. Draw to canvas

                
                    cv.imshow(outputCanvas, dst);
                
                

Draw dst-matrix to canvas. This is an OpenCV.js specific functionality.

4. Loop

                
                    window.requestAnimationFrame(processVideo);
                
                

Exercise Time!

• Initialize matrices and VideoCapture
• Create function with draw loop
• Call this function after setting up the stream

Memory Leaks?

.delete()

• All objects created with cv.* from JS are shared
• Not deleted by OpenCV
• Have a *.delete() method to free memory

Let's do some basic face detection

How do we detect faces?

1. We need a classifier
2. Run it over each frame
3. Get all detected objects
4. Draw them on the dst-matrix

Haar Cascade classifier

• Classifier trained to detect an object
• We're not going to train it ourselves
• OpenCV provides a pretrained one

Loading the classifier

• Load an XML file from the webserver
• Create that file in OpenCV
• Instantiate Classifier with this file
• This is already done in the exercise

Running the classifier

• The classifier has to be run on a grayscale matrix
• Dst-matrix should not be grayscale
• We need another copy of the src-matrix

Create new grayscale matrix

                
                    gray = new cv.Mat(); // instantiate once
                    ...
                    src.copyTo(dst);
                    cv.cvtColor(dst, gray, cv.COLOR_RGBA2GRAY);
                
                

Call classifier

                
                    faces = new cv.RectVector(); // instantiate once
                    ...
                    classifier.detectMultiScale(gray, faces, 1.1, 3, 0);
                
                

Loop through all faces

• The vector is now filled with found faces
• With JS we can loop over this Vector
• Vector has size() and get(i) methods

Loop through all faces

                
                    for (let i = 0; i < faces.size(); i++) {
                        let face = faces.get(i);
                        ...
                    }
                
                

Draw them on dst-matrix

• Every face has: x, y, width and height properties
• To draw a rectangle we need two cv.Point objects
• Top left and bottom right
• cv.rectangle(topLeft, bottomRight, color)

Draw them on dst-matrix

                
                    let topLeft = new cv.Point(face.x, face.y);
                    ...
                    cv.rectangle(dst, topLeft, bottomRight, [255, 0, 0, 255]);
                
                

Scaling

• We will scale down the input to 320px
• Also render it scaled down for the sake of simplicity
• You would only scale it down for the algorithm and render it in full scale

Exercise Time!

• Checkout branch exercise/face-detection
• Enhance the steps mentioned above
• Have fun!
• Check out deployed version

To round it up

Pick the right tool for the task!