$P)0(<8BUBOBCF *14+> าߦऀͷߏతͳྨࣅੑΛଊ͑ΔͨΊʹෳͷಛྔͷڞىੑΛදݱ ମݕग़λεΫʹ͓͚ΔϋϯυΫϥϑτಛʢϛυϧʣ +PJOU)BBSMJLFGFBUVSF ϙδςΟϒΫϥε ωΨςΟϒΫϥε j =ʢ̍̍̍ʣ= ̓ ͖͍͠ॲཧ )0(ಛྔͷڞىදݱ த ߨ ࢁ ػ 4 Ѫ Te F ya h த෦େֶ ֶ෦ϩϘο τཧֶՊ ڭत ౻٢߂ ػց֮ˍϩϘςΟ Ϋεάϧʔϓ 487-8501 Ѫݝय़Ҫࢢদຊொ1200 Tel 0568-51-9096 Fax 0568-51-9409 [email protected] http://vision.cs.chubu.ac.jp ത࢜ ʢֶʣ M த෦େֶ ֶ෦ϩϘο τཧֶՊ ڭत ౻٢߂ ػց֮ˍϩϘςΟ Ϋεάϧʔϓ 487-8501 Ѫݝय़Ҫࢢদຊொ1200 Tel 0568-51-9096 Fax 0568-51-9409 [email protected] http://vision.cs.chubu.ac.jp ത࢜ ʢֶʣ MACHINE PERCEPTION AND ROBOTICS GROUP Chubu University Department of Robotics Science and Technology College of Engineering Professor Dr.Eng. Hironobu Fujiyoshi Machine Perception and Robotics Group 1200 Matsumoto-cho, Kasugai, Aichi 487-8501 Japan Tel +81-568-51-9096 Fax +81-568-51-9409 [email protected] http://vision.cs.chubu.ac.jp w )0(ಛྔͷޯͷؔੑΛଊ͑Δ r $P)0(<8BUBOBCF> w ہॴྖҬͷޯϖΞΛྦྷੵͨ͠ಉ࣌ਖ਼ىߦྻ r +PJOU)0(<ࡾҪ> w #PPTJOHʹΑΓࣝผʹ༗ޮͳہॴྖҬͷؔੑΛ֫ಘ $P)0( )0(ಛྔͷڞىදݱ த෦େֶ ֶ෦ใֶՊ ߨࢣ ࢁԼོٛ ػց֮ˍϩϘςΟ Ϋ 487-8501 Ѫݝय़Ҫࢢদຊ Tel 0568-51-9670 Fax 0568-51-1540 [email protected] http://vision.cs.chubu MACHINE PERCEPTIO Chubu University Department of Compu த෦େֶ ֶ෦ϩϘο τཧֶՊ ڭत ౻٢߂ ػց֮ˍϩϘςΟ Ϋεάϧʔϓ 487-8501 Ѫݝय़Ҫࢢদຊொ1200 Tel 0568-51-9096 Fax 0568-51-9409 [email protected] http://vision.cs.chubu.ac.jp ത࢜ ʢֶʣ MACHINE PERCEPTION AND ROBOTICS GROUP Chubu University Department of Robotics Science and Technology ࢁԼོٛ ػց֮ˍϩϘςΟ Ϋεάϧʔ 487-8501 Ѫݝय़Ҫࢢদຊொ1200 Tel 0568-51-9670 Fax 0568-51-1540 [email protected] http://vision.cs.chubu.ac.jp MACHINE PERCEPTION AND R Chubu University Department of Computer Scie College of Engineering Lecturer Dr.Eng. Takayoshi Yam Machine Perception and Robo 1200 Matsumoto-cho, Kasuga 487-8501 Japan Tel +81-568-51-9670 Fax +81-568-51-1540 [email protected] http://vision.cs.chubu.ac.jp MACHINE PERCEPTION AND R ౻٢߂ ػց֮ˍϩϘςΟ Ϋεάϧʔϓ 487-8501 Ѫݝय़Ҫࢢদຊொ1200 Tel 0568-51-9096 Fax 0568-51-9409 [email protected] http://vision.cs.chubu.ac.jp ത࢜ ʢֶʣ MACHINE PERCEPTION AND ROBOTICS GROUP Chubu University Department of Robotics Science and Technology College of Engineering Professor Dr.Eng. Hironobu Fujiyoshi Machine Perception and Robotics Group 1200 Matsumoto-cho, Kasugai, Aichi 487-8501 Japan Tel +81-568-51-9096 Fax +81-568-51-9409 [email protected] http://vision.cs.chubu.ac.jp MACHINE PERCEPTION AND ROBOTICS GROUP w )0(ಛྔͷޯͷؔੑΛଊ͑Δ r $P)0(<8BUBOBCF> w ہॴྖҬͷޯϖΞΛྦྷੵͨ͠ಉ࣌ਖ਼ىߦྻ r +PJOU)0(<ࡾҪ> w #PPTJOHʹΑΓࣝผʹ༗ޮͳہॴྖҬͷؔੑΛ֫ಘ $P)0( +PJOU)0(
<%PTPWJUTLJZ *$-3> Figure 1: The Transformer - model architecture. 3.1 Encoder and Decoder Stacks Scaled Dot-Product Attention Multi-Head Attention Figure 2: (left) Scaled Dot-Product Attention. (right) Multi-Head Attention consists of several attention layers running in parallel. 3.2.1 Scaled Dot-Product Attention We call our particular attention "Scaled Dot-Product Attention" (Figure 2). The input consists of queries and keys of dimension dk , and values of dimension dv . We compute the dot products of the query with all keys, divide each by p dk , and apply a softmax function to obtain the weights on the values. In practice, we compute the attention function on a set of queries simultaneously, packed together into a matrix Q. The keys and values are also packed together into matrices K and V . We compute the matrix of outputs as: Attention(Q, K, V ) = softmax( QKT p dk )V (1) The two most commonly used attention functions are additive attention [2], and dot-product (multi- plicative) attention. Dot-product attention is identical to our algorithm, except for the scaling factor of 1 p dk . Additive attention computes the compatibility function using a feed-forward network with a single hidden layer. While the two are similar in theoretical complexity, dot-product attention is much faster and more space-efficient in practice, since it can be implemented using highly optimized matrix multiplication code. Scaled Dot-Product Attention Multi-Head Attention Figure 2: (left) Scaled Dot-Product Attention. (right) Multi-Head Attention consists of several attention layers running in parallel. 3.2.1 Scaled Dot-Product Attention We call our particular attention "Scaled Dot-Product Attention" (Figure 2). The input consists of queries and keys of dimension dk , and values of dimension dv . We compute the dot products of the query with all keys, divide each by p dk , and apply a softmax function to obtain the weights on the values. In practice, we compute the attention function on a set of queries simultaneously, packed together 5SBOTGPSNFS 7J5 7J5ͷΈ
w ࣝৠཹʹΑΔޮతͳ7J5ͷֶश %FJ5<5PVWSPO *$.-> w 7J5ͷࣗݾڭࢣ͋Γֶश ."&<)F $713> 7J5ʹΑΔଟ༷ͳλεΫͷԠ༻ Overlap Patch Embeddings Transformer Block 1 MLP Layer ! " × # " ×"$ ! % × # % ×"& ! '& × # '& ×"" ! $( × # $( ×"' ! " × # " ×4" MLP ! " × # " ×$)*+ Transformer Block 2 Transformer Block 3 Transformer Block 4 Overlap Patch Merging Efficient Self-Attn Mix-FFN ×" UpSample MLP ! "!"# × # "!"# ×"$ ! "!"# × # "!"# ×" ! % × # % ×" Encoder Decoder Figure 2: The proposed SegFormer framework consists of two main modules: A hierarchical Transformer encoder to extract coarse and fine features; and a lightweight All-MLP decoder to directly fuse these multi-level features and predict the semantic segmentation mask. “FFN” indicates feed-forward network. in an end-to-end manner. After that, researchers focused on improving FCN from different aspects such as: enlarging the receptive field [17–19, 5, 2, 4, 20]; refining the contextual information [21– 29]; introducing boundary information [30–37]; designing various attention modules [38–46]; or using AutoML technologies [47–51]. These methods significantly improve semantic segmentation performance at the expense of introducing many empirical modules, making the resulting framework computationally demanding and complicated. More recent methods have proved the effectiveness of Transformer-based architectures for semantic segmentation [7, 46]. However, these methods are still computationally demanding. Transformer backbones. ViT [6] is the first work to prove that a pure Transformer can achieve state-of-the-art performance in image classification. ViT treats each image as a sequence of tokens and then feeds them to multiple Transformer layers to make the classification. Subsequently, DeiT [52] further explores a data-efficient training strategy and a distillation approach for ViT. More recent methods such as T2T ViT [53], CPVT [54], TNT [55], CrossViT [56] and LocalViT [57] introduce tailored changes to ViT to further improve image classification performance. Beyond classification, PVT [8] is the first work to introduce a pyramid structure in Transformer, demonstrating the potential of a pure Transformer backbone compared to CNN counterparts in dense prediction tasks. After that, methods such as Swin [9], CvT [58], CoaT [59], LeViT [60] and Twins [10] enhance the local continuity of features and remove fixed size position embedding to improve the performance of Transformers in dense prediction tasks. Transformers for specific tasks. DETR [52] is the first work using Transformers to build an end-to- end object detection framework without non-maximum suppression (NMS). Other works have also used Transformers in a variety of tasks such as tracking [61, 62], super-resolution [63], ReID [64], Colorization [65], Retrieval [66] and multi-modal learning [67, 68]. For semantic segmentation, Training data-efficient image transformers & distillation throug VHOIDWWHQWLRQ ))1 UHVLGXDO0/3 FODVV WRNHQ GLVWLOODWLRQ WRNHQ SDWFK WRNHQV Figure 2. Our distillation procedure: we simply include a new dis- tillation token. It interacts with the class and patch tokens through the self-attention layers. This distillation token is employed in a similar fashion as the class token, except that on output of the network its objective is to reproduce the (hard) label predicted by the teacher, instead of true label. Both the class and distillation tokens input to the transformers are learned by back-propagation. true label. Let yt = argmaxc Zt (c) be the hard decision of the teacher, the objective associated with this hard-label distillation is: LhardDistill global = 1 2 LCE (ψ(Zs ), y) + 1 2 LCE (ψ(Zs ), yt ). (3) For a given image, the hard label associated with the teacher may change depending on the specific data augmentation. We will see that this choice is better than the traditional one, while being parameter-free and conceptually simpler: The teacher prediction yt plays the same role as the true label y. Label smoothing. Hard labels can also be converted into soft labels with label smoothing (Szegedy et al., 2016), where the true label is considered to have a probability the output of the teache remaining complement Fine-tuning with disti and teacher prediction d resolution. We use a tea typically obtained from method of Touvron et a true labels only but thi and leads to a lower pe Classification with ou test time, both the class duced by the transform fiers and able to infer th is the late fusion of thes add the softmax outpu prediction. We evaluate 5. Experiments This section presents a sults. We first discuss comparatively analyze vnets and vision transfo 5.1. Transformer mod As mentioned earlier, o the one proposed by Do volutions. Our only dif and the distillation toke for the pre-training bu any confusion, we refe work by ViT, and prefix refers to our referent m architecture as ViT-B. W resolution, we append t the end, e.g, DeiT-B↑38 procedure, we identify The parameters of ViT-B %&53<$BSJPO &$$7> 4FH'PSNFS<9JF /FVS*14> ."&<,)F $713> %FJ5<5PVWSPO *$.->
ςΩετΛҙຯۭؒʹຒΊࠐΈɺΦʔϓϯϘΩϟϒϥϦମݕग़Λ࣮ݱ :P-P8PSME<$IFOH $713> YOLO Backbone Text Encoder A man and a woman are skiing with a dog caption, noun phrases, category… User Text Embeddings Multi-scale Image Features Text Contrastive Head Box Head man woman dog Vocabulary Embeddings man woman dog User’s Vocabulary Dog Image-aware Embeddings Multi-scale Image Features Training: Online Vocabulary Deployment: Offline Vocabulary Vision-Language PAN Object Embeddings Region-Text Matching Input Image Extract Nouns
$713> bridge sky A white cute cat lying on the ground. (a) CLIP is pre-trained with natural images CLIP Mask proposal generator … CLIP classification A photo of a {bridge} (b) Skeleton of two-stage approaches (c) Bo … 20 40 60 mIoU on ADE20K-150 Orac Orac masked images … 0.99 0.00 0.01 0.30 0.35 0.35 … … … 0.35 0.28 0.47 0.01 0.02 0.97 apple orange teapot CLIP image Mask Former There are apple and orange and teapot. Noun Parser CLIP text … ! "! ! "" ! "# ! "#$! #! #" #% teapot apple orange Query: golden gate, yacht Ukulele Query: saturn V, blossom Query: golden gate, yacht Query: Oculus, Ukulele ఆٛ͞Ε͍ͯͳ͍ΦϒδΣΫτ֓೦ʹରׂ͕ͯ͠Մೳ
*NBHF&ODPEFS 7J5 Preprint. Under review. Transformer Encoder MLP Head Vision Transformer (ViT) * Linear Projection of Flattened Patches * Extra learnable [ cl ass] embedding 1 2 3 4 5 6 7 8 9 0 Patch + Position Embedding Class Bird Ball Car ... Embedded Patches Multi-Head Attention Norm MLP Norm + L x + Transformer Encoder eprint. Under review. Transformer Encoder MLP Head Vision Transformer (ViT) * Linear Projection of Flattened Patches * Extra learnable [ cl ass] embedding 1 2 3 4 5 6 7 8 9 0 Patch + Position Embedding Class Bird Ball Car ... Embedded Patches Multi-Head Attention Norm MLP Norm + L x + Transformer Encoder gure 1: Model overview. We split an image into fixed-size patches, linearly embed each of them, d position embeddings, and feed the resulting sequence of vectors to a standard Transformer coder. In order to perform classification, we use the standard approach of adding an extra learnable 5FYU%FDPEFS --. ςΩετೖྗɿ࣭จʢϓϩϯϓτʣ ςΩετग़ྗɿճจ $SPTT"UUFOUJPO 4FMG"UUFOUJPO ೖྗτʔΫϯɿ ग़ྗτʔΫϯɿ ը૾ೖྗɿ
ω0: A Vision-Language-Action Flow Model for General Robot Control Physical Intelligence Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, Ury Zhilinsky https://physicalintelligence.company/blog/pi0 Fig. 1: Our generalist robot policy uses a pre-trained vision-language model (VLM) backbone, as well as a diverse cross- embodiment dataset with a variety of dexterous manipulation tasks. The model is adapted to robot control by adding a separate action expert that produces continuous actions via flow matching, enabling precise and fluent manipulation skills. The model IUUQTXXXZPVUVCFDPNXBUDI W +65ZCM0&X
An Open Foundation Model for Generalist Humanoid Robots Robot State “Pick up the industry object and place in yellow bin.” Joint Positions Joint Velocities Base Position EEF Poses … Tokenize Encode System 2 Vision-Language Model Image Tokens Text Tokens Encode Action Tokens Denoising System 1 Diffusion Transformer Motor Action Image Observation Language Instruction Figure 2: GR00T N1 Model Overview. Our model is a Vision-Language-Action (VLA) model that adopts a dual-system design. We convert the image observation and language instruction into a sequence of tokens to be processed by the Vision-Language Model (VLM) backbone. The VLM outputs, together with robot state and action encodings, are passed to the Di!usion Transformer module to generate motor actions. 2. GR00T N1 Foundation Model ࣮ੈք ݴޠɾ ࣝܥ ֮ܥ ӡಈܥ γεςϜ ख़ߟతೳ γεςϜ ଈԠతೳ ᶃঢ়گΛཧղ͠ ᶅߦಈ͢Δ ᶄஅɾਪ͠ ैདྷͷਂֶश ʢγεςϜ૬ʣ ݱࡏͷج൫Ϟσϧ ʢγεςϜͷҰ෦·ͰΧόʔʣ ࣍ੈ"*Ϟσϧ ʢγεςϜ ΛΧόʔʣ IUUQTXXXNFYUHPKQDPOUFOUNYU@DIPVTFJ@QEG
Robotics: Bringing AI into the Physical World Figure 1 | Overview of the Gemini Robotics family of embodied AI models. Gemini 2.0 already exhibits capabilities relevant to robotics such as semantic safety understanding and long contexts. The robotics-specific training and the optional specialization processes enable the Gemini Robotics models to exhibit a variety of 7-"ϞσϧʹΑΔϩϘοτಈ࡞ੜɿ(FNJOJ3PCPUJDT (PPHMF IUUQTXXXZPVUVCFDPNXBUDI W .W(ONN1D
hf149
.png){kind=avatar}
satai
skoyamalab
yuukit
ikuyamada
matsumoto_r
datascientistsociety
laysakura
hponka
yumulab
trafficbrain
tsurubee
tomoyanonymous
danielanewman
bluesmoon
rasmusluckow
reverentgeek
cassininazir
shlominoach
scottboms
tammyeverts
sstephenson
lauravandoore
jcasabona
stephaniewalter
