Understanding Filesystem using go-fuse, from scratch

https://github.com/hanwen/go-fuse
https://github.com/juntaki/bucketsync

Transcript

1. Understanding Filesystem using go-fuse, from scratch Jumpei Takiyasu M3, Inc.
2. = &Me{ Name: "Jumpei Takiyasu", Company: "M3, Inc.", // ex-Filesystem engineer Github: "juntaki", Twitter: "juntaki", Web: "https://juntaki.com", }
3. Contents 1. Filesystem basics 2. How to make Filesystem 3. Mounting S3 using go-fuse
4. Filesystem basics
5. Filesystem OS abstraction for data I/O.
6. Filesystem of Linux Implementation of the VFS(Virtual File System) interfaces. Source of data are various: • Memory: procfs, ramfs • Block device: XFS, ext4, btrfs • Network: NFS, CIFS
7. VFS operation example read(2) System call: vfs_read() User Kernel file->f_op->read() struct file_operations { struct module *owner; loff_t (*llseek) (struct file *, loff_t, int); ssize_t (*read) (struct file *, char __user *, size_t, loff_t *); ssize_t (*write) (struct file *, const char __user *, size_t, loff_t *); …. Application glibc VFS xfs, ext4, etc. Block layer
8. Data structure of VFS on-memory structure • inode (index node) • dentry (directory entry) http://www.coins.tsukuba.ac.jp/~yas/classes/os2-2012/2013-02-19/index.html
9. Rough example of on-disk data structure Regular file On-disk inode Directory “Filename1” : <inode number> “Filename2” : <inode number> On-disk inode data[0] data[1] Root inode number Super block On-disk inode Directory [...]
10. How to make Filesystem
11. FUSE Easy to implement filesystem Slow performance due to context switch and memory copy User Kernel Application glibc VFS FUSE Daemon
12. hanwen/go-fuse https://github.com/hanwen/go-fuse native bindings for the FUSE kernel module.
13. pathfs.FileSystem interface (Omitted) type FileSystem interface { // These should update the file's ctime too. Chmod(name string, mode uint32, context *fuse.Context) (code fuse.Status) Chown(name string, uid uint32, gid uint32, context *fuse.Context) (code fuse.Status) Utimens(name string, Atime *time.Time, Mtime *time.Time, context *fuse.Context) (code fuse.Status) Truncate(name string, size uint64, context *fuse.Context) (code fuse.Status) Access(name string, mode uint32, context *fuse.Context) (code fuse.Status) // Tree structure Mkdir(name string, mode uint32, context *fuse.Context) fuse.Status Rename(oldName string, newName string, context *fuse.Context) (code fuse.Status) Rmdir(name string, context *fuse.Context) (code fuse.Status) Unlink(name string, context *fuse.Context) (code fuse.Status) // File handling. If opening for writing, the file's mtime // should be updated too. Open(name string, flags uint32, context *fuse.Context) (file nodefs.File, code fuse.Status) Create(name string, flags uint32, mode uint32, context *fuse.Context) (file nodefs.File, code fuse.Status) // Directory handling OpenDir(name string, context *fuse.Context) (stream []fuse.DirEntry, code fuse.Status) // Symlinks. Symlink(value string, linkName string, context *fuse.Context) (code fuse.Status) Readlink(name string, context *fuse.Context) (string, fuse.Status) }
14. Mounting S3 using go-fuse
15. juntaki/bucketsync ★ S3 backed FUSE Filesystem written in Go with dedup, compression and client-side encryption. https://github.com/juntaki/bucketsync TODO :p
16. S3 backed Filesystem, most simple way S3 on AWS console looks like Filesystem! • Full path = Object key • Data = Object contents Problem: • Directory rename performance • Random read/write with compression/encryption
17. Bucketsync data structure overview Directory Meta obj. Directory Meta obj. File Meta obj. DataChunk object root File Meta obj. Data chunk’s key is Hash of content. No upload needed for existing key Random generated key Meta object is cached in memory (LRU)
18. Demo
19. Conclusion Go is productive! Let’s try your idea with go-fuse!