fast I can walk/run • Whether the car turns/does not turn • Whether the car will accelerate quickly Whether or not I'll have an accident based on past experience and common sense
subject will do next and take action accordingly Ø learn how the world works and background knowledge from a few interactions and observations • change point, common sense • Understanding the real world through language Ø have linguistic information such as common sense and knowledge Ø gain a deeper understanding of the real world by connecting language to the real world Real World Cognition of Humans BUT… • Machine learning for real-world recognition prediction Ø input (observation) is an image → equivalent to human vision Ø predictions of image features are considered real-world predictions • ML doesn't make predictions based on physical properties of objects or physical laws, as humans do • real world and understanding the real world through language have not yet been linked • proposes a predictive inference model that can detect and predict physical change points based on the physical laws of real-world objects. • To connect the real world and language, the inference is expressed as a language. Purpose
of the next step can be displayed correctly Proposed Model Graph structure Representation of a set of physical properties PredNet VTA, graph VTA Image when looking at the real world from the visual Generate inference content as a language Experiment 1 Experiment 2 • Object detection • speed • acceleration • image features, etc
characteristics of the environment 20 object recognition object position velocity acceleration Position direction flags between objects graph structure embedding vector
characteristics of the environment 21 object recognition object position velocity acceleration Position direction flags between objects graph structure embedding vector
Ø{shape, color, material} of an object Calculate location information • Calculate the coordinates of the object center from the acquired bounding box coordinates Dataset physical training dataset 23 (𝑥& , 𝑦&) (𝑥' , 𝑦') 𝑐 = 𝑥, 𝑦 = ( 𝑥& + 𝑥' 2 , 𝑦& + 𝑦' 2 ) c Before detecting After detecting
characteristics of the environment 24 object recognition velocity acceleration Position direction flags between objects graph structure embedding vector object position
created from physical characteristics of the environment 28 object recognition combination velocity acceleration Position direction flags between objects embedding vector
predicted change point of an event can be extracted correctly Setting • Data Set ØCLEVRER ØPhysical training data Scope of coverage: 6 patterns x 10 frames Situations in which physical changes of objects occur, such as collision, disappearance, appearance. Experiment Summary 30
annotation collision information and flag timing Example • collision→19 frame, by eye → 21 frame • The correct answer range was set to 19-21 frame • flag︓18, 19, 20, 22 → accuracy︓2/4×100=50 (%) 31 19 frame 20 frame 21 frame
33.3 50 50 33.3 66.7 50 annotation 66.7 50 66.7 40 50 50 Accuracy Original image Predicted image t=1 t=12 m=1 m=0 m=0 m=1 m=1 m=0 m=1 m=1 collision accuracy︓2/6*100=33.3% Result of range i m=0 m=1 accuracy with physical training data predictions with accuracy equivalent to annotation data
predicted change point of an event can be extracted correctly Setting • Data Set ØCLEVRER ØPhysical training data Scope of coverage: 6 patterns x 10 frames Situations in which physical changes of objects occur, such as collision, disappearance, appearance. Ex 2: Text Generation Purpose • Express reasoning as language to connect the real world and language Setting • Dataset ØPaired data of graph embedding vectors and language data • Collision situations only Experiment Summary 34
Object type Ø”color” “shape” Øex) blue sphere, gray cylinder, etc. 35 「⻘⾊の球と灰⾊の球が近づく」 “Blue sphere and gray sphere approach.” 「⻘⾊の球が灰⾊の球に近づく」 “Blue sphere approaches gray sphere.” 「灰⾊の球が⻘⾊の球に近づく」 “Gray sphere approaches blue sphere.” 「⻘⾊の球と灰⾊の球がぶつかる」 “Blue sphere and gray sphere collide.” 「⻘⾊の球が灰⾊の球にはじかれる」 “Blue sphere is repulsed by gray sphere.” 「灰⾊の球が⻘⾊の球にはじかれる」 “Gray sphere is repulsed by blue sphere.” collision before collision after collision 「⻘⾊の球と灰⾊の球が離れる」 “Blue sphere and gray sphere leave.” 「⻘⾊の球から灰⾊の球が離れる」 “Gray sphere away from blue sphere.” 「灰⾊の球から⻘⾊の球が離れる」 “Blue sphere away from gray sphere.” Example of text templates:Colliding Objects “blue sphere”, “gray sphere” 5 frames 5 frames • A and B approach • A approaches B • B approaches A • A and B collide • A is repulsed by B • B is repulsed by A • A and B leave • A away from B • B away from A before collision after template ※ A・B︓objects
original image Predicted image 「緑⾊の球と⾚⾊の円柱がぶつかる」 “Green sphere and red cylinder collide." 「緑⾊の球が⾚⾊の円柱にはじかれる」 “Green sphere is repulsed by red cylinder.” 「⾚⾊の円柱が緑⾊の球にはじかれる」 “Red cylinder is repulsed by green sphere.” correct text 緑⾊の円柱が⾚⾊の円柱にはじかれる Green cylinder is repulsed by red cylinder. generated text 「灰⾊の球と⻘⾊の円柱がぶつかる」 “Gray sphere and blue cylinder collide." 「灰⾊の球が⻘⾊の円柱にはじかれる」 “Gray sphere is repulsed by blue cylinder.” 「⻘⾊の円柱が灰⾊の球にはじかれる」 “Blue cylinder is repulsed by gray sphere.” 灰⾊の球が⻘⾊の⽴⽅体にはじかれる Gray sphere is repulsed by blue cube. 「⽔⾊の⽴⽅体と⽔⾊の円柱がぶつかる」 “Cyan cube and cyan cylinder collide." 「⽔⾊の⽴⽅体が⽔⾊の円柱にはじかれる」 “Cyan cube is repulsed by cyan cylinder.” 「⽔⾊の円柱が⽔⾊の⽴⽅体にはじかれる」 “Cyan cylinder is repulsed by cyan cube.” ⽔⾊の⽴⽅体が⻘⾊の球にぶつかる Cyan cube is repulsed by blue sphere. 「緑⾊の円柱と茶⾊の⽴⽅体がぶつかる」 “Green cylinder and brown cube collide." 「緑⾊の円柱が茶⾊の⽴⽅体にはじかれる」 “Green cylinder is repulsed by brown cube.” 「茶⾊の⽴⽅体が緑⾊の円柱にはじかれる」 “Brown cube is repulsed by green cylinder.” 緑⾊の円柱が茶⾊の⽴⽅体にぶつかる Green cylinder is repulsed by brown cube. object’s color ✔,shape ✘ object’s color ✔,shape ✔ object’s color ✔,shape ✘ object’s color ✘,shape ✘ correct text generated text correct text generated text correct text generated text original image original image original image Predicted image Predicted image Predicted image
20 frames before 25 frames before collision 15 frames before 5 frames before 10 frames before collision Incorrect reason for both color and shape of object Possibility that "cyan cube" and "blue sphere" were judged to have collided Range vi る er. る 「⽔⾊の⽴⽅体と⽔⾊の円柱がぶつかる」 “Cyan cube and cyan cylinder collide." 「⽔⾊の⽴⽅体が⽔⾊の円柱にはじかれる」 “Cyan cube is repulsed by cyan cylinder.” 「⽔⾊の円柱が⽔⾊の⽴⽅体にはじかれる」 “Cyan cylinder is repulsed by cyan cube.” ⽔⾊の⽴⽅体が⻘⾊の球にぶつかる Cyan cube is repulsed by blue sphere. 「緑⾊の円柱が茶⾊の⽴⽅体にはじかれる」 “Green cylinder is repulsed by brown cube.” 「茶⾊の⽴⽅体が緑⾊の円柱にはじかれる」 “Brown cube is repulsed by green cylinder.” 緑⾊の円柱が茶⾊の⽴⽅体にぶつかる Green cylinder is repulsed by brown cube. object’s color ✔,shape ✔ object’s color ✘,shape ✘ correct text generated text generated text original image Predicted image Predicted image
Range i Range ii Range iv Range vi original image Predicted image 「緑⾊の球と⾚⾊の円柱がぶつかる」 “Green sphere and red cylinder collide." 「緑⾊の球が⾚⾊の円柱にはじかれる」 “Green sphere is repulsed by red cylinder.” 「⾚⾊の円柱が緑⾊の球にはじかれる」 “Red cylinder is repulsed by green sphere.” correct text 緑⾊の円柱が⾚⾊の円柱にはじかれる Green cylinder is repulsed by red cylinder. generated text 「灰⾊の球と⻘⾊の円柱がぶつかる」 “Gray sphere and blue cylinder collide." 「灰⾊の球が⻘⾊の円柱にはじかれる」 “Gray sphere is repulsed by blue cylinder.” 「⻘⾊の円柱が灰⾊の球にはじかれる」 “Blue cylinder is repulsed by gray sphere.” 「⽔⾊の⽴⽅体と⽔⾊の円柱がぶつかる」 “Cyan cube and cyan cylinder collide." 「⽔⾊の⽴⽅体が⽔⾊の円柱にはじかれる」 “Cyan cube is repulsed by cyan cylinder.” 「⽔⾊の円柱が⽔⾊の⽴⽅体にはじかれる」 “Cyan cylinder is repulsed by cyan cube.” 「緑⾊の円柱と茶⾊の⽴⽅体がぶつかる」 “Green cylinder and brown cube collide." 「緑⾊の円柱が茶⾊の⽴⽅体にはじかれる」 “Green cylinder is repulsed by brown cube.” 「茶⾊の⽴⽅体が緑⾊の円柱にはじかれる」 “Red cylinder is repulsed by green sphere.” 緑⾊の円柱が茶⾊の⽴⽅体にぶつかる Green cylinder is repulsed by brown cube. object’s color ✔,shape ✘ object’s color ✔,shape ✔ correct text correct text correct text generated text original image original image original image Predicted image Predicted image Predicted image since the average is taken, it is possible that the score is a little low
hierarchical structure of the human brain Øadd flag “m” representing change points to the hierarchical structure of PredNet Øbased on experimental results, timing of change points can also be obtained for predictive content • generated a language of inference to connect real-world events and objects as a language Øon the basis of the experimental results, it was possible to generate a language for the content of the inferences 40
cooking Øgo to kitchen → prepare cutting board → cut ingredients → fry • in the real-life environment, extraction of easy-to-understand change points and prediction of what actions will be necessary 41