Putting Point Cloud into AR with Kinect and Processing

Transcript

1. Putting Point Cloud into AR with Kinect and Processing
2. Setting up • Kinect for Windows SDK v1.8 https://www.microsoft.com/en-us/download/details.aspx?id=40278 • Processing 2.2.1 https://processing.org/download/ Unzip and put following folders into libraries of Processing (C:¥Users¥YOU¥Documents¥Processing¥ libraries) • SimpleOpenNI 1.96 https://code.google.com/archive/p/simple-openni/downloads • NyARToolKit (nyar4psg.zip) https://github.com/nyatla/NyARToolkit-for-Processing/releases Download and unzip following file into any folder. https://drive.google.com/file/d/1ahzAzp_s5gqg5WbfJkF5JSv5v87qm KZv/view?usp=sharing
3. Goal https://youtu.be/WeorvV9vZh8
4. Initialization of Kinect import SimpleOpenNI.*; //Load SimpleOpenNI SimpleOpenNI kinect; //Variable to access Kinect void setup(){ //Insatiate and initialize Kinect. kinect = new SimpleOpenNI(this); if(kinect.isInit() == false){ exit(); return; } }  Run processing.  Write following code.  Ctrl + s to save this code.
5. Run Application IR projection will started
6. Preparation of using NyARToolKit Move camera_para.dat & patt.sample1 from Sample to the same directory of your code. 【NyARToolKit】 【Sample3】
7. Preparation of using NyARToolKit import SimpleOpenNI.*; import jp.nyatla.nyar4psg.*; //Import NyARToolKit SimpleOpenNI kinect; MultiMarker ar; //Variable to access ARToolKit void setup(){ size(640,480,P3D); /*Initialization of Kinect (Abb)*/ ar = new MultiMarker( this, width, height, "camera_para.dat" ,NyAR4PsgConfig.CONFIG_PSG_PV221); } Window size Camera parameter
8. Show Color Image with NyARToolKit void draw(){ //Update Kinect kinect.update(); //Grab current color image. PImage rgbImage = kinect.rgbImage(); //Draw color image as background. ar.drawBackground(rgbImage); } void setup(){ /*Initialization of Kinect(Abb)*/ ar = new MultiMarker(this, width, /*Abb*/); kinect.enableRGB(); //Enable acquiring color image }
9. Marker Recognition with ARToolKit (1/2) void setup(){ /*Initialization of Kinect(Abb)*/ ar = new MultiMarker(this, width, /*Abb*/); //Load marker file (width is 80mm) ar.addARMarker("patt.sample1", 80); kinect.enableRGB(); } 【ARToolKit】  Using square frame and inner figure.  Do not use rotation symmetric figure.  You can use pre-defined markers if original ARToolkit is downloaded.
10. Marker Recognition with ARToolKit (2/2) void draw(){ kinect.update(); PImage rgbImage = kinect.rgbImage(); ar.drawBackground(rgbImage); } ar.detect(rgbImage); //Marker detection. if(!ar.isExistMarker(0)){ return; //Return if no marker is detected } ar.beginTransform(0); //Put on following object on marker fill(0,0,255); box(40); //Draw box as a test. ar.endTransform();
11. Run application It’s not on marker
12. Modification of CG’s Position ar.beginTransform(0); translate(0,0,20); fill(0,0,255); box(40); ar.endTransform(); Translate box to upper. x Y Z (0,0,0) (0,0,20) 40mm
13. Acquireing Depth Data void setup(){ /*Initialization of Kinect (Abb)*/ kinect.enableRGB(); kinect.enableDepth(); //Enable Depth Image } void draw(){ kinect.update(); PImage rgbImage = kinect.rgbImage(); ar.drawBackground(rgbImage); /* following code is abbreviated in this slide */ }
14. Acquiring Point Cloud Data void draw(){ kinect.update(); PImage rgbImage = kinect.rgbImage(); //Acqureing point cloud. //Point clouds is in 1D array. PVector[] realWorld = kinect.depthMapRealWorld(); /*以下省略*/ pixelと濃淡 3D 実空間
15. Drawing Point Cloud (1/2) void draw(){ /*Acquiring image from kinect (省略)*/ /*Marker detection (Abb.)*/ ar.beginTransform(0); // translate(0,0,20); fill(0,0,255); box(40); ar.endTransform(); } beginShape(POINTS); //Draw points of each position (Shown in the next page) endShape(); translate(0,0,20); fill(0,0,255); box(40); Remove Box Add
16. Drawing Point Cloud (2/2) beginShape(POINTS); endShape(); //Ａｄｄ ｆｏｌｌｏｗｎｉｎｇ ｃｏｄｅ for(int v=0; v<kinect.depthHeight(); v+=4){ for(int u=0; u<kinect.depthWidth(); u+=4){ int index = u + v * kinect.depthWidth(); PVector point = realWorld[index]; stroke(255,255,0); //Draw as yellow point if(point.z>0) { vertex(point.x, point.y, point.z); } } } X Y Z
17. Run Application Point are shown but not easy to see. Yellow points
18. Modification of Drawing Setting of Point Cloud scale(0.3); //shrink whole size of point cloud translate(0,0,-600); //Translate point cloud beginShape(POINTS); for(int v=0; v<kinect.depthHeight(); v+=4){ for(int u=0; u<kinect.depthWidth(); u+=4){ int index = u + v * kinect.depthWidth(); PVector point = realWorld[index]; // stroke(255,255,0); //描画色 if(point.z>0) { vertex(point.x, point.y, point.z); } } } X Y Z
19. Modification of Each Point Size strokeWeight(2); //Draw as 2 pix point. scale(0.3); translate(0,0,-600); beginShape(POINTS); for(int v=0; v<kinect.depthHeight(); v+=4){ /*Drawing code (Abb.)*/
20. Rotate Point Clouds strokeWeight(2); scale(0.3); //Rotate 90 degree(=π/2) to X axis. rotate(PI/2, 1, 0, 0); translate(0,0,-600); beginShape(POINTS); 回転
21. Remove Far Point Cloud beginShape(POINTS); for(int v=0; v<kinect.depthHeight(); v+=4){ for(int u=0; u<kinect.depthWidth(); u+=4){ int index = u + v * kinect.depthWidth(); PVector point = realWorld[index]; // stroke(255,255,0); if(point.z>0) { vertex(point.x, point.y, point.z); } } } endShape(); if(point.z>0 && point.z<1500) { vertex(point.x, point.y, point.z); } 除去 If distance<1500mm
22. Modify Position of Point Cloud Again 移動 strokeWeight(2); scale(0.3); rotate(PI/2, 1, 0, 0); translate(0,0,-600); //600mm後ろに戻す (場合による) beginShape(POINTS); for(int v=0; v<kinect.depthHeight(); v+=4){ /*Drawing point cloud (Abb.)*/ translate(0,300,-600); //Move upper direction (prox. 300mm)
23. Assign Image Color to Point Cloud beginShape(POINTS); for(int v=0; v<kinect.depthHeight(); v+=4){ for(int u=0; u<kinect.depthWidth(); u+=4){ int index = u + v * kinect.depthWidth(); PVector point = realWorld[index]; // color pixelColor = rgbImage.pixels[index]; stroke(255,255,0); if(point.z>0) { vertex(point.x, point.y, point.z); } } } endShape(); stroke(pixelColor);
24. Run Application Not much the position between point and color
25.  Position of color camera and depth camera are different → Pixel coordinate is not match each other. Reason of Mismatch Color Depth
26. Match Color and Depth Images void setup(){ size(640, 480, P3D); kinect = new SimpleOpenNI(this); if(kinect.isInit() == false){ exit(); return; } ar = new MultiMarker( this, width, height, "camera_para.dat"); ar.addARMarker("patt.sample1", 80); kinect.enableRGB(); kinect.enableDepth(); kinect.alternativeViewPointDepthToImage(); } Add this code
27. Finished