2017 - latus - a Personal Cloud Storage App written in Python

Description

Many people use personal cloud storage applications to keep all of their files accessible, backed up and secure. However, they generally are closed source, so they can't easily be viewed or modified. latus is a personal cloud storage application that is open source and written in Python. It provides flexible file sync across computers (nodes) and the cloud, as well as 'zero knowledge' encryption.

Abstract

Personal cloud storage solutions are common and popular. Many companies provide this capability such as DropBox, Google Drive, Microsoft One Drive, Amazon Drive, etc. However, these applications are closed source, making it difficult for the community to innovate on top of them. latus is a client application written in Python for personal cloud storage. It is open source, and utilizes open source libraries/packages. latus uses 'zero knowledge' encryption so that files can be encrypted prior to being stored in the cloud, providing an additional level of security. It also provides the ability to control where files are stored long term - in the cloud, cold storage, or only on personal systems. latus performs file sync, so that all computers (or "nodes") contain the most current file versions, as well as a file version history. For the cloud storage itself, latus can use existing cloud storage vendors or AWS directly. In this talk I will discuss the major subsystems of latus, the Python packages used, testing, freezing/installing and lessons learned.

Bio

James Abel is a HW and SW engineering consultant in the Bay Area. In 2016 he retired from Intel Corporation, most recently as a Principal Engineer working on microprocessor performance architecture, simulation and design automation. He holds MSCS and BSEE degrees. His other interests are music recording, production and performance (bass/guitar).

https://www.youtube.com/watch?v=Mrl7YjkOFgU

Transcript

1. James Abel – PyBay Aug 2017 Latus® A Personal Cloud Storage App written in Python www.github.com/jamesabel/latus James Abel Aug 12, 2017 j@abel.co @jamesabel www.abel.co Latus is a registered trademark of James Abel
2. James Abel – PyBay Aug 2017 Intro • James Abel • HW/SW consultant • Former Intel Principal Engineer • Some Python for several years (otherwise C/C++, ASM, etc.) • Python ‘go to’ programming language for the last few years
3. James Abel – PyBay Aug 2017 Agenda • What is latus? • Why should I care? • Cloud based File Synchronization • File system events • Event database • What’s in the cloud • Synchronization • Encryption • Task Bar Application • Using latus • Next Steps • Summary latus is on github at www.github.com/jamesabel/latus
4. James Abel – PyBay Aug 2017 What is latus? • File storage and synchronization across your computers • AKA cloud storage • Automatically mirrors files across your computer and to the cloud • Make files accessible • Inherent backup • App that runs in the background (‘client sync’) • Open Source (GPLv3) • “Zero Knowledge” Encryption • Inherent versioning • Python 3.6 • Utilizes lots of awesome packages! Latus is a cloud file sync app written in Python
5. James Abel – PyBay Aug 2017 Why should I care? • You like cloud storage file sync, but you want .. • Open Source • Python • Zero Knowledge Encryption to the cloud • Control over exactly where files are stored in the cloud (e.g. for regulatory, compliance and/or business requirements) • Control over versioning history • Control over what takes up space in the cloud in the long term • Custom integrated applications • ‘Independent Web’ • Provide a customized branded (‘white label’) cloud storage/sync offering • Or just example uses of several awesome packages and capabilities Open Source File Sync written in Python!
6. James Abel – PyBay Aug 2017 What is file sync? Computer A Computer B The Internet a.txt “latus” folder “latus” folder
7. James Abel – PyBay Aug 2017 What is file sync? Computer A Computer B The Internet a.txt a.txt “latus” folder “latus” folder
8. James Abel – PyBay Aug 2017 Cloud Based File Sync File System Events  Database  Cloud  Synchronization Local File System (PC/Mac) Watchdog Event Database (AWS DynamoDB + local SQLite cache) Events Sync Engine File Cloud Storage (AWS S3) Files Computer A Cloud Computer B Local File System (PC/Mac)
9. James Abel – PyBay Aug 2017 Cloud Based File Sync Local File System (PC/Mac) Watchdog Event Database (AWS DynamoDB + local SQLite cache) Events Sync Engine File Cloud Storage (AWS S3) Files (encrypted) Computer A Cloud Computer B Local File System (PC/Mac) Step 1: File “a.txt” created
10. James Abel – PyBay Aug 2017 Cloud Based File Sync Local File System (PC/Mac) Watchdog Event Database (AWS DynamoDB + local SQLite cache) Events Sync Engine File Cloud Storage (AWS S3) Computer A Cloud Computer B Local File System (PC/Mac) Step 2.a.: “a.txt Created” Event Step 2.b.: “a.txt” to S3 Files (encrypted)
11. James Abel – PyBay Aug 2017 Step 3: “a.txt” mirrored Cloud Based File Sync Local File System (PC/Mac) Watchdog Event Database (AWS DynamoDB + local SQLite cache) Events Sync Engine File Cloud Storage (AWS S3) Computer A Cloud Computer B Local File System (PC/Mac) Files (encrypted) Events Files (decrypted)
12. James Abel – PyBay Aug 2017 mivui – Monotonically Increasing Value • Events are strictly sequenced • Monotonically Increasing Value in Micro-Seconds (uS) since Epoch as an Integer • Equivalent to: int(round(time.time()*1E6)) • Integer allows reliable comparison and DB indexing/sort key • Server provides mivui • Optionally can be locally created (but is then not absolutely guaranteed to be monotonically increasing) • http://api.abel.co/miv { "mivui": 1502435085769804, "toc": 0, "mivf": 1502435085.769804, "dur": 0.0018029212951660156, "success": true, "mivs": "1502435085.769804“ }
13. James Abel – PyBay Aug 2017 File System Event Database Items • originator – node (computer) where this event was detected (normally a UUID) • mivui – monotonically increasing integer in micro-seconds from epoch (from server) • detection – how was event detected – initial scan, watchdog, periodic poll • event_type – create, delete, move, modify • file_hash – SHA512 of file • file_path – file path in the latus folder • mtime – mod time of the file • size – file size (in bytes) • src_path – source in the case of moves (otherwise unused)
14. James Abel – PyBay Aug 2017 Cloud and Local event databases Cloud database is ‘the’ database – local is a cache Cloud (AWS DynamoDB) Local Computer Nodes (SQLite cache) a b AWS SQLite AWS DynamoDB
15. James Abel – PyBay Aug 2017 Synchronization • Sync is hard • Compute (algorithm) on the client • All nodes see the events on all other nodes and use the same algorithm • ‘winner’ is based on ordered events • Default algorithm is that the most recent event (across all nodes) wins • Other algorithms are possible • Files are available in the cloud to provide them locally • Encrypted in the cloud
16. James Abel – PyBay Aug 2017 Sync example $vi a.txt $vi b.txt $mv a.txt c.txt $rm b.txt DynamoDB S3 class DetectionSource(IntEnum): unknown = 0 initial_scan = 1 watchdog = 2 periodic_poll = 3 class LatusFileSystemEvent(IntEnum): created = 1 modified = 2 deleted = 3 moved = 4 Winner!
17. James Abel – PyBay Aug 2017 Encryption • Keep unencrypted files off the internet/cloud • Uses the most excellent Python cryptography package • Key exists on client nodes (computers) • Local computer is assumed secure • In the cloud file contents are stored as AWS S3 objects • S3 object names are SHA512 hashes of the files contents ‘salted’ with encryption key • Thwarts dictionary lookup attacks • For security, the encryption key should not be transferred over the internet • Keep the key ‘out of band’ • Once generated, a key can be exported to a file (simple JSON format) • USB stick works well (put it in a safe place!) • On first run: • Makes a new encryption key if no one exists or • Asks the user for an existing key to be imported
18. James Abel – PyBay Aug 2017 boto3 • Amazon’s Python library to access Amazon Web Services (AWS) • AWS credentials stored on the local file system or passed in explicitly to boto3 APIs • ~/.aws/credentials [default] aws_access_key_id = YOUR_ACCESS_KEY aws_secret_access_key = YOUR_SECRET_KEY region=us-west-1 # means Northern California • Access AWS import boto3 s3 = boto3.resource('s3’) # Upload a new file data = open('test.jpg', 'rb') s3.Bucket('my-bucket').put_object(Key='test.jpg', Body=data)
19. James Abel – PyBay Aug 2017 AWS S3 (with encryption) ‘a’ This object’s contents contains the bytes corresponding to a file with contents ‘b’ latusstorage S3 bucket
20. James Abel – Pyninsula Mar 2017 latus is a Task Bar App …
21. James Abel – Pyninsula Mar 2017 What is a Task Bar Application? • A taskbar is an element of a graphical user interface which has various purposes. It typically shows which programs or applications are running on the device, as well as provide links or shortcuts to other programs or places, such as a start menu, notification area, and clock. https://en.wikipedia.org/wiki/Taskbar • AKA System Tray
22. James Abel – Pyninsula Mar 2017 PyQt • Qt (/kjuːt/ "cute"[7][8][9]) is a cross-platform application framework that is used for developing application software that can be run on various software and hardware platforms with little or no change in the underlying codebase, while still being a native application with native capabilities and speed. • https://en.wikipedia.org/wiki/Qt_(software) • What is PyQt? • PyQt is a set of Python v2 and v3 bindings for The Qt Company's Qt application framework and runs on all platforms supported by Qt including Windows, OS X, Linux, iOS and Android. • https://riverbankcomputing.com/software/pyqt/intro • Currently using PyQt5
23. James Abel – Pyninsula Mar 2017 Hello World Task Bar App class HelloWorldSystemTray(QSystemTrayIcon): def __init__(self): import icons icon = QIcon(QPixmap(':icon.png')) super().__init__(icon) menu = QMenu() menu.addAction("About").triggered.connect(self.about) menu.addAction("Exit").triggered.connect(self.exit) self.setContextMenu(menu) def about(self): about_box = QDialog() layout = QGridLayout(about_box) layout.addWidget(QLabel('hello world')) about_box.setLayout(layout) about_box.show() about_box.exec_() def exit(self): QApplication.exit() app = QApplication(sys.argv) app.setQuitOnLastWindowClosed(False) # so popup dialogs don't close the system tray icon system_tray = HelloWorldSystemTray() system_tray.show() app.exec_()
24. James Abel – PyBay Aug 2017 •Latus Task Bar App class LatusSystemTrayIcon(QSystemTrayIcon): def __init__(self, app, latus_appdata_folder): self.app = app self.latus_appdata_folder = latus_appdata_folder menu = QMenu(parent) menu.addAction("Open Latus Folder").triggered.connect(self.open_latus_folder) menu.addAction("Preferences").triggered.connect(self.preferences) menu.addAction("Export Latus Key").triggered.connect(self.export_latus_key) menu.addAction("About").triggered.connect(self.about) menu.addAction("Nodes").triggered.connect(self.nodes) menu.addAction("Exit").triggered.connect(self.exit) self.setContextMenu(menu) def start_latus(self): self.sync = latus.aws.sync_aws.Sync(self.latus_appdata_folder) def show(self): QSystemTrayIcon.show(self) def exit(self): self.sync.request_exit() QApplication.exit() app = QApplication(sys.argv) system_tray = LatusSystemTrayIcon(app, latus_appdata_folder) system_tray.start_latus() system_tray.show() app.exec_()
25. James Abel – PyBay Aug 2017 Preferences • Local preferences kept in preferences.db • SQLite DB • Latus crypto key • Latus folder path • AWS configuration (credentials, region) • Accessible via icon pulldown
26. James Abel – PyBay Aug 2017 Using latus • Can be cloned and executed from source • www.github.com/jamesabel/latus • latus as a native app • latus uses many packages – it can be a challenge to freeze/install • cryptography and PyQt can be problematic to freeze • Embedded Python interpreter tends to work best • Freezers/Installers • Windows: pynsist, osnap using the ‘embedded’ Python interpreter • MacOS: osnap currently latus can be run for source or as a native app for Windows and MacOS
27. James Abel – PyBay Aug 2017 Hacking latus • Fork from • www.github.com/jamesabel/latus • Run regression tests • Uses py.test, localstack • Has create, delete, move, etc. • But can use more tests. ☺ • Pull requests • File issues
28. James Abel – PyBay Aug 2017 Some Next Steps … • Add AWS pubsub to get file system events instantly (AWS SNS/SQS) •Cloud storage efficiency enhancements • Select folders for ‘offline only’ •Node management • Filter out special files (e.g. “.*”) • Code review ☺ Try out latus! Its on github at www.github.com/jamesabel/latus
29. James Abel – PyBay Aug 2017 Summary and Thank You • latus is an Open Source Cloud Storage app in Python 3.6 • Hack on latus at SF Python Project Night, chat at Pyninsula or Bay Piggies • Thank you • Glyph • Thomas Kluyver Try out latus! Its on github at www.github.com/jamesabel/latus