Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Crazy computer vision stuff in the browser
Search
Zeno Rocha
April 28, 2015
Programming
5
940
Crazy computer vision stuff in the browser
Zeno Rocha
April 28, 2015
Tweet
Share
More Decks by Zeno Rocha
See All by Zeno Rocha
The Next Generation of Developer-First Products
zenorocha
1
550
7 Habits of Highly Productive Developers
zenorocha
1
370
7 Hábitos de Desenvolvedores Altamente Produtivos
zenorocha
1
440
What's new in the Liferay Community
zenorocha
0
650
Launching Liferay Projects Faster with WeDeploy
zenorocha
1
530
How Liferay fits into the real of latest technologies
zenorocha
0
550
Estoicismo e JavaScript
zenorocha
3
1.1k
Por que ninguém se importa com seu novo projeto open source?
zenorocha
2
890
Como investir em... você!
zenorocha
1
530
Other Decks in Programming
See All in Programming
自分ひとりから始められる生産性向上の取り組み #でぃーぷらすオオサカ
irof
8
2.6k
なぜイベント駆動が必要なのか - CQRS/ESで解く複雑系システムの課題 -
j5ik2o
7
2.5k
Amazon S3 TablesとAmazon S3 Metadataを触ってみた / 20250201-jawsug-tochigi-s3tables-s3metadata
kasacchiful
0
100
Honoとフロントエンドの 型安全性について
yodaka
4
250
AWSマネコンに複数のアカウントで入れるようになりました
yuhta28
2
160
SwiftUI Viewの責務分離
elmetal
PRO
0
140
Amazon ECS とマイクロサービスから考えるシステム構成
hiyanger
2
490
2024年のkintone API振り返りと2025年 / kintone API look back in 2024
tasshi
0
210
Conform を推す - Advocating for Conform
mizoguchicoji
3
680
DevinとCursorから学ぶAIエージェントメモリーの設計とMoatの考え方
itarutomy
1
640
Lottieアニメーションをカスタマイズしてみた
tahia910
0
120
Grafana Loki によるサーバログのコスト削減
mot_techtalk
1
110
Featured
See All Featured
The Cost Of JavaScript in 2023
addyosmani
47
7.3k
Learning to Love Humans: Emotional Interface Design
aarron
273
40k
Raft: Consensus for Rubyists
vanstee
137
6.8k
YesSQL, Process and Tooling at Scale
rocio
171
14k
For a Future-Friendly Web
brad_frost
176
9.5k
The MySQL Ecosystem @ GitHub 2015
samlambert
250
12k
Side Projects
sachag
452
42k
[RailsConf 2023] Rails as a piece of cake
palkan
53
5.2k
Mobile First: as difficult as doing things right
swwweet
223
9.3k
Designing for humans not robots
tammielis
250
25k
A Philosophy of Restraint
colly
203
16k
Measuring & Analyzing Core Web Vitals
bluesmoon
6
240
Transcript
CRAZY COMPUTER vision stuff in the browser
.html .css .js
zenorocha
None
None
None
None
None
computer vision ? what is
Persistence of vision
Persistence of vision
Motion perception
Motion perception
vimeo.com/108331968
Face detection
Face recognition
Autonomous robots
Autonomous robots
Virtual reality
Augmented reality
Augmented reality
how to do that?
None
None
how to do that in the browser?
1. Access the camera
None
Request user’s webcam navigator.getUserMedia({ video: true }, onSuccess, onFail);
Request user’s webcam navigator.getUserMedia({ video: true }, onSuccess, onFail);
Request user’s webcam navigator.getUserMedia({ video: true }, onSuccess, onFail);
2. Reproduce in a video
<video>
Get User Media callback function onSuccess(stream) { var video =
document.querySelector('video'); video.src = URL.createObjectURL(stream); video.onloadedmetadata = function(e) { // Done }; }
Get User Media callback function onSuccess(stream) { var video =
document.querySelector('video'); video.src = URL.createObjectURL(stream); video.onloadedmetadata = function(e) { // Done }; }
Get User Media callback function onSuccess(stream) { var video =
document.querySelector('video'); video.src = URL.createObjectURL(stream); video.onloadedmetadata = function(e) { // Done }; }
Get User Media callback function onSuccess(stream) { var video =
document.querySelector('video'); video.src = URL.createObjectURL(stream); video.onloadedmetadata = function(e) { // Done }; }
3. Obtain pixel matrix
<canvas>
4. Tracking algorithms
Request accepted Request access to the camera Reproduce camera <video>
Obtain pixel matrix <canvas> Tracking algorithms Result Summary
tracking.js
None
Maira Bello Team eduardo lundgren java pablo carvalho zeno rocha
100% JavaScript! Open source No dependencies Simple and intuitive API
Automatic setup Built-in tracking algorithms Easily extensible High performance Unit & performance regression tests ~ 7 Kb
tracker color
Demo
var tracker = new tracking.ColorTracker('magenta'); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
var tracker = new tracking.ColorTracker('magenta'); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
var tracker = new tracking.ColorTracker('magenta'); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
tracker object
Demo
var tracker = new tracking.ObjectTracker('face'); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
var tracker = new tracking.ObjectTracker('face'); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
var tracker = new tracking.ObjectTracker('face'); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
tracker custom
var MyTracker = function() { MyTracker.base(this, 'constructor'); }; tracking.inherits(MyTracker, tracking.Tracker);
MyTracker.prototype.track = function(pxs, width, height) { // Your code here this.emit('track', { // Your code here }); };
var MyTracker = function() { MyTracker.base(this, 'constructor'); }; tracking.inherits(MyTracker, tracking.Tracker);
MyTracker.prototype.track = function(pxs, width, height) { // Your code here this.emit('track', { // Your code here }); };
var MyTracker = function() { MyTracker.base(this, 'constructor'); }; tracking.inherits(MyTracker, tracking.Tracker);
MyTracker.prototype.track = function(pxs, width, height) { // Your code here this.emit('track', { // Your code here }); };
var MyTracker = function() { MyTracker.base(this, 'constructor'); }; tracking.inherits(MyTracker, tracking.Tracker);
MyTracker.prototype.track = function(pxs, width, height) { // Your code here this.emit('track', { // Your code here }); };
var tracker = new tracking.MyTracker(); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
var tracker = new tracking.MyTracker(); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
var tracker = new tracking.MyTracker(); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); });
who is using it?
kdzwinel.github.io/JS-face-tracking-demo
thedevilini.com/#masks
lessons learned
24 fps 60 fps 1000ms / 24fps = 41ms per
frame
24 fps 60 fps 1000ms / 24fps = 41ms per
frame
None
None
None
Int32Array Uint16Array Float64Array Uint8ClampedArray [ ] Typed arrays
jsperf.com/tracking-js-arrays
jsperf.com/tracking-js-arithmetic
Demo
None
None
web components > > -
how we create a component nowadays?
1. Never create! Just use a plugin
2. Copy & paste someone’s code
3. And hope it works
3. And hope it works
web components
web components Templates
web components Templates Custom Elements
web components Templates Custom Elements Shadow DOM
web components HTML Imports Templates Custom Elements Shadow DOM
None
var tracker = new tracking.ColorTracker('magenta'); tracking.track('#video', tracker, { camera: true
}); tracker.on('track', function(event) { event.data.forEach(function(rect) { // Your code goes here }); }); Remember?
Using Web Components… <video is="video-color-tracking" target="magenta" camera="true" ontrack="onResult" </video>
more ?
trackingjs.com
Where the magic happens Your comfort zone
Thanks! zenorocha.com @zenorocha
Thanks! zenorocha.com @zenorocha