2023DFKI, UdS talk

30.10.2023

Transcript

1. Predictive Inference Model of the Physical Environment that mimics Predictive Coding Eri Kuroda Ochanomizu University Material My HP
2. Eri Kuroda introduction myself • Prof. Ichiro Kobayashi’s student, Ochanomizu University (in Tokyo) • Ph.D 2nd year student • stay at Saarbücken from Oct.4 to Mar.26 2 • my research interest: Ø how humans understand the world (the environment). Ø Prediction Ø Physical Reasoning Ø World Model • my Ph.D main theme: To create a mechanism for thinking about real-world events using language as human do.
6. 1 6
7. 2 7
8. 8 1 2 Crossing in a hurry before the car arrives Crossing the street after the car has passed
9. 9 Where is our judgment Estimate distance Speed • How fast I can walk/run • Whether the car turns/does not turn • Whether the car will accelerate quickly Whether or not I'll have an accident based on past experience and common sense
10. 10 Background・Purpose • Recognition and Prediction Ø predict what the subject will do next and take action accordingly Ø learn how the world works and background knowledge from a few interactions and observations • change point, common sense • Understanding the real world through language Ø have linguistic information such as common sense and knowledge Ø gain a deeper understanding of the real world by connecting language to the real world Real World Cognition of Humans BUT… • Machine learning for real-world recognition prediction Ø input (observation) is an image → equivalent to human vision Ø predictions of image features are considered real-world predictions • ML doesn't make predictions based on physical properties of objects or physical laws, as humans do • real world and understanding the real world through language have not yet been linked • proposes a predictive inference model that can detect and predict physical change points based on the physical laws of real-world objects. • To connect the real world and language, the inference is expressed as a language. Purpose
11. 11 Overview CLEVRER whether the timing of the change point of the next step can be displayed correctly Proposed Model Graph structure Representation of a set of physical properties PredNet VTA, graph VTA Image when looking at the real world from the visual Generate inference content as a language Experiment 1 Experiment 2 • Object detection • speed • acceleration • image features, etc
12. 12 PredNet [Lotter+, 2016]
13. 13 PredNet [Lotter+, 2016] Hierarchical Model
14. 14 PredNet [Lotter+, 2016] the process of predictive coding Hierarchical Model
15. 15 PredNet [Lotter+, 2016] Propagation of predictions Updating the prediction model Error generation Error propagation input (observation)
16. 16 Variational Temporal Abstraction [Kim+, 19] when walking on the blue road when walking on the red road all events chang points all events chang points
17. 17 Variational Temporal Abstraction [Kim+, 19] difficult to decide when to transition 𝑍 problem Human: easy ↔ Model: difficult Observation (Input) Observation abstraction temporal abstraction
18. 18 Variational Temporal Abstraction [Kim+, 19] Determines the flag (0 or 1) of 𝑚 by the magnitude of the change in latent state compared to the previous observation Introduced flags
19. 19 Proposed Model 𝐸!"_ℓ%& 𝐸!"_ℓ ⊝ ⊝ 𝑅!"_ℓ%& 𝑥" Input # 𝐴!"_ℓ%& 𝐴!"_ℓ%& # 𝐴!"_ℓ 𝐴!"_ℓ 𝐸'"_ℓ%& 𝐸'"_ℓ ⊝ ⊝ 𝑅'"_ℓ%& 𝑅'"_ℓ # 𝐴'"_ℓ%& 𝐴'"_ℓ%& # 𝐴'"_ℓ 𝐴!"_ℓ img Output 𝑑𝑖𝑓𝑓 !" 𝑅!"_ℓ 𝑑𝑖𝑓𝑓'" 𝑚( Output 𝑑𝑖𝑓𝑓 > 𝛼 physical training data Input Error Representation Prediction time t 𝛼︓ threshold Difference Graph structure prediction based on physical properties Image Prediction 𝑑𝑖𝑓𝑓 = 𝑑𝑖𝑓𝑓!" + 𝑑𝑖𝑓𝑓%"
20. Dataset︓CLEVRER [Yi+,2020] • CLEVRER [Yi+, 2020] ØCoLlision Events for Video REpresentation and Reasoning 20 Number of videos 20,000 (train:val:test=2:1:1) Video Length 5 sec Number of frames 128 frame Shape cube, sphere, cylinder Material metal, rubber Color gray, red, blue, green, brown, cyan, purple, yellow Event appear, disappear, collide Annotation object id, position, speed, acceleration
21. combination Dataset physical training dataset • Dataset created from physical characteristics of the environment 21 object recognition object position velocity acceleration Position direction flags between objects graph structure embedding vector
22. combination Dataset physical training dataset • Dataset created from physical characteristics of the environment 22 object recognition object position velocity acceleration Position direction flags between objects graph structure embedding vector
23. object recognition • YOLACT Ø[Bolya+,2019] ØA type of instance segmentation Ø{shape, color, material} of an object Dataset physical training dataset 23 Before detecting After detecting
24. object recognition • YOLACT Ø[Bolya+,2019] ØA type of instance segmentation Ø{shape, color, material} of an object Calculate location information • Calculate the coordinates of the object center from the acquired bounding box coordinates Dataset physical training dataset 24 (𝑥& , 𝑦&) (𝑥' , 𝑦') 𝑐 = 𝑥, 𝑦 = ( 𝑥& + 𝑥' 2 , 𝑦& + 𝑦' 2 ) c Before detecting After detecting
25. combination Dataset physical training dataset • Dataset created from physical characteristics of the environment 25 object recognition velocity acceleration Position direction flags between objects graph structure embedding vector object position
26. Velocity・Acceleration Dataset physical training dataset 26 velocity acceleration 𝑎!" = (𝑣!" − 𝑣!# )/(𝑒𝑡"#$%&×𝑡) 𝑎'" = (𝑣'" − 𝑣'# )/(𝑒𝑡"#$%&×𝑡) ※ 𝑒𝑡()*+, = 5/128 time elapsed between frames 𝑣!" = (𝑥( − 𝑥()*)/𝑒𝑡"#$%& 𝑣'" = (𝑦( − 𝑦()* )/𝑒𝑡"#$%&
27. Velocity・Acceleration Position direction flags between objects Dataset physical training dataset 27 velocity acceleration 𝑎!" = (𝑣!" − 𝑣!# )/(𝑒𝑡"#$%&×𝑡) 𝑎'" = (𝑣'" − 𝑣'# )/(𝑒𝑡"#$%&×𝑡) ※ 𝑒𝑡()*+, = 5/128 time elapsed between frames 𝑣!" = (𝑥( − 𝑥()*)/𝑒𝑡"#$%& 𝑣'" = (𝑦( − 𝑦()* )/𝑒𝑡"#$%& x main object others main object = (𝑥&'%( , 𝑦&'%( ) others = (𝑥)"*+, , 𝑦)"*+, ) 𝑥-%.. = 𝑥)"*+, − 𝑥&'%( 𝑦-%.. = 𝑦)"*+, − 𝑦&'%( 𝑥-%.. 𝑦-%.. + + − − y 1st Quadrant 2nd Quadrant 3rd Quadrant 4th Quadrant 1st Quadrant 2nd Quadrant 4th Quadrant 3rd Quadrant
28. graph structure • Node information Øshape, color, material embedding vector • node2vec [Grover+, 2016] Dataset physical training dataset 28 [[0.54, 0.29, 0.61…], [[0.82, 0.91, 0.15…], … [[0.14, 0.35, 0.69…]] Example of embedding vector
29. graph structure object position Dataset physical training dataset • Dataset created from physical characteristics of the environment 29 object recognition combination velocity acceleration Position direction flags between objects embedding vector
30. Ex 1: Extracting Predicted Change Points Ex 2: Text Generation Experiment Summary 30
31. Ex 1: Extracting Predicted Change Points Purpose • whether the predicted change point of an event can be extracted correctly Setting • Data Set ØCLEVRER ØPhysical training data Scope of coverage: 6 patterns x 10 frames Situations in which physical changes of objects occur, such as collision, disappearance, appearance. Experiment Summary 31
32. Ex1︓ Accuracy Calculation Method • Examine the F-measure (%) of annotation collision information and flag timing correct answer range • collision→19 frame， by eye → 21 frame • The correct answer range was set to 19-21 frame 32 19 frame 20 frame 21 frame
33. Ex1︓ Setting • Training data : 600,000 • Test data : 80,000 • epoch︓500,000 • batch-size︓100 • Optimization ︓Adam • Error function︓KL divergence 33
34. 34 Ex1︓Result Physical training data i ii iii iv v vi Created based on 2D coordinates obtained from object recognition 40.0 50.0 50.0 40.0 57.1 50.0 Created from accurate 3D information (annotation) 57.1 50.0 57.1 44.4 50.0 50.0 F-measure (%) Original image Predicted image t=1 t=12 m=1 m=0 m=0 m=1 m=1 m=0 m=1 m=1 collision Result of range i m=0 m=1
35. 35 Ex1︓Result Physical training data i ii iii iv v vi Created based on 2D coordinates obtained from object recognition 40.0 50.0 50.0 40.0 57.1 50.0 Created from accurate 3D information (annotation) 57.1 50.0 57.1 44.4 50.0 50.0 F-measure (%) Original image Predicted image t=1 t=12 m=1 m=0 m=0 m=1 m=1 m=0 m=1 m=1 collision Result of range i m=0 m=1 accuracy with 2D based-data predictions with accuracy equivalent to 3D based-data (annotation)
36. Ex 1: Extracting Predicted Change Points Purpose • whether the predicted change point of an event can be extracted correctly Setting • Data Set ØCLEVRER ØPhysical training data Scope of coverage: 6 patterns x 10 frames Situations in which physical changes of objects occur, such as collision, disappearance, appearance. Ex 2: Text Generation Purpose • Express reasoning as language to connect the real world and language Setting • Dataset ØPaired data of graph embedding vectors and language data • Collision situations only Experiment Summary 36
37. Ex2︓ Creation of Templates • nine templates Ø3(before・collision・after)×3(sentence type) • Object type Ø”color” “shape” Øex) blue sphere, gray cylinder, etc. 37 「⻘⾊の球と灰⾊の球が近づく」 “Blue sphere and gray sphere approach.” 「⻘⾊の球が灰⾊の球に近づく」 “Blue sphere approaches gray sphere.” 「灰⾊の球が⻘⾊の球に近づく」 “Gray sphere approaches blue sphere.” 「⻘⾊の球と灰⾊の球がぶつかる」 “Blue sphere and gray sphere collide.” 「⻘⾊の球が灰⾊の球にはじかれる」 “Blue sphere is repulsed by gray sphere.” 「灰⾊の球が⻘⾊の球にはじかれる」 “Gray sphere is repulsed by blue sphere.” collision before collision after collision 「⻘⾊の球と灰⾊の球が離れる」 “Blue sphere and gray sphere leave.” 「⻘⾊の球から灰⾊の球が離れる」 “Gray sphere away from blue sphere.” 「灰⾊の球から⻘⾊の球が離れる」 “Blue sphere away from gray sphere.” Example of text templates：Colliding Objects “blue sphere”, “gray sphere” 5 frames 5 frames • A and B approach • A approaches B • B approaches A • A and B collide • A is repulsed by B • B is repulsed by A • A and B leave • A away from B • B away from A before collision after template ※ A・B︓objects
38. 38 Ex2︓ text generating model test Trained Decoder Model generated text indicating predicted content pred graph embedding input # 𝐴!"_ℓ Decoder Softmax <bos> w1 w2 wt <eos> … w1 w2 wt … Decoder train model text pair data train Linear graph embedding 219,303 pieces 10,965 pieces
39. 39 Ex2︓result Range i Range ii Range iv Range vi original image Predicted image 「緑⾊の球と⾚⾊の円柱がぶつかる」 “Green sphere and red cylinder collide." 「緑⾊の球が⾚⾊の円柱にはじかれる」 “Green sphere is repulsed by red cylinder.” 「⾚⾊の円柱が緑⾊の球にはじかれる」 “Red cylinder is repulsed by green sphere.” correct text 緑⾊の円柱が⾚⾊の円柱にはじかれる Green cylinder is repulsed by red cylinder. generated text 「灰⾊の球と⻘⾊の円柱がぶつかる」 “Gray sphere and blue cylinder collide." 「灰⾊の球が⻘⾊の円柱にはじかれる」 “Gray sphere is repulsed by blue cylinder.” 「⻘⾊の円柱が灰⾊の球にはじかれる」 “Blue cylinder is repulsed by gray sphere.” 灰⾊の球が⻘⾊の⽴⽅体にはじかれる Gray sphere is repulsed by blue cube. 「⽔⾊の⽴⽅体と⽔⾊の円柱がぶつかる」 “Cyan cube and cyan cylinder collide." 「⽔⾊の⽴⽅体が⽔⾊の円柱にはじかれる」 “Cyan cube is repulsed by cyan cylinder.” 「⽔⾊の円柱が⽔⾊の⽴⽅体にはじかれる」 “Cyan cylinder is repulsed by cyan cube.” ⽔⾊の⽴⽅体が⻘⾊の球にぶつかる Cyan cube is repulsed by blue sphere. 「緑⾊の円柱と茶⾊の⽴⽅体がぶつかる」 “Green cylinder and brown cube collide." 「緑⾊の円柱が茶⾊の⽴⽅体にはじかれる」 “Green cylinder is repulsed by brown cube.” 「茶⾊の⽴⽅体が緑⾊の円柱にはじかれる」 “Brown cube is repulsed by green cylinder.” 緑⾊の円柱が茶⾊の⽴⽅体にぶつかる Green cylinder is repulsed by brown cube. object’s color ✔，shape ✘ object’s color ✔，shape ✔ object’s color ✔，shape ✘ object’s color ✘，shape ✘ correct text generated text correct text generated text correct text generated text original image original image original image Predicted image Predicted image Predicted image
40. Ex2︓ discussion of the results of the range vi 40 20 frames before 25 frames before collision 15 frames before 5 frames before 10 frames before collision Incorrect reason for both color and shape of object Possibility that "cyan cube" and "blue sphere" were judged to have collided Range vi る er. る 「⽔⾊の⽴⽅体と⽔⾊の円柱がぶつかる」 “Cyan cube and cyan cylinder collide." 「⽔⾊の⽴⽅体が⽔⾊の円柱にはじかれる」 “Cyan cube is repulsed by cyan cylinder.” 「⽔⾊の円柱が⽔⾊の⽴⽅体にはじかれる」 “Cyan cylinder is repulsed by cyan cube.” ⽔⾊の⽴⽅体が⻘⾊の球にぶつかる Cyan cube is repulsed by blue sphere. 「緑⾊の円柱が茶⾊の⽴⽅体にはじかれる」 “Green cylinder is repulsed by brown cube.” 「茶⾊の⽴⽅体が緑⾊の円柱にはじかれる」 “Brown cube is repulsed by green cylinder.” 緑⾊の円柱が茶⾊の⽴⽅体にぶつかる Green cylinder is repulsed by brown cube. object’s color ✔，shape ✔ object’s color ✘，shape ✘ correct text generated text generated text original image Predicted image Predicted image
41. Ex2︓ BLEU 41 Range i Range ii Range iv Range vi original image Predicted image 「緑⾊の球と⾚⾊の円柱がぶつかる」 “Green sphere and red cylinder collide." 「緑⾊の球が⾚⾊の円柱にはじかれる」 “Green sphere is repulsed by red cylinder.” 「⾚⾊の円柱が緑⾊の球にはじかれる」 “Red cylinder is repulsed by green sphere.” correct text 緑⾊の円柱が⾚⾊の円柱にはじかれる Green cylinder is repulsed by red cylinder. generated text 「灰⾊の球と⻘⾊の円柱がぶつかる」 “Gray sphere and blue cylinder collide." 「灰⾊の球が⻘⾊の円柱にはじかれる」 “Gray sphere is repulsed by blue cylinder.” 「⻘⾊の円柱が灰⾊の球にはじかれる」 “Blue cylinder is repulsed by gray sphere.” 「⽔⾊の⽴⽅体と⽔⾊の円柱がぶつかる」 “Cyan cube and cyan cylinder collide." 「⽔⾊の⽴⽅体が⽔⾊の円柱にはじかれる」 “Cyan cube is repulsed by cyan cylinder.” 「⽔⾊の円柱が⽔⾊の⽴⽅体にはじかれる」 “Cyan cylinder is repulsed by cyan cube.” 「緑⾊の円柱と茶⾊の⽴⽅体がぶつかる」 “Green cylinder and brown cube collide." 「緑⾊の円柱が茶⾊の⽴⽅体にはじかれる」 “Green cylinder is repulsed by brown cube.” 「茶⾊の⽴⽅体が緑⾊の円柱にはじかれる」 “Red cylinder is repulsed by green sphere.” 緑⾊の円柱が茶⾊の⽴⽅体にぶつかる Green cylinder is repulsed by brown cube. object’s color ✔，shape ✘ object’s color ✔，shape ✔ correct text correct text correct text generated text original image original image original image Predicted image Predicted image Predicted image since the average is taken, it is possible that the score is a little low BLEU@2 BLEU@3 BLEU@4 score 79.7 74.5 68.8
42. Conclusion • Construct a predictive inference model that mimics the hierarchical structure of the human brain Øadd flag “m” representing change points to the hierarchical structure of PredNet Øbased on experimental results, timing of change points can also be obtained for predictive content • generated a language of inference to connect real-world events and objects as a language Øon the basis of the experimental results, it was possible to generate a language for the content of the inferences 42
43. Future Tasks • Use of real world-like data • When cooking Øgo to kitchen → prepare cutting board → cut ingredients → fry • in the real-life environment, extraction of easy-to-understand change points and prediction of what actions will be necessary 43