Implementing Data Visualization Apps on iOS Devices

Slides from an invited presentation on building data visualization apps on iOS Devices presented to the software development engineers at The Broad Institute of MIT and Harvard.

1. Elastic Image Software LLC Douglass Turner Elastic Image Software tweets: @dugla email: douglass.turner@gmail.com Implementing Data Visualization Apps on iOS Devices
2. Elastic Image Software LLC iOS software development is done using Objective-C an object-oriented superset of C. It was developed in the spirit of Smalltalk. 2
3. Elastic Image Software LLC • Objective-C is simple, approachable, lightweight, and pragmatic. No frills. • Objective-C and C can be intermingled freely. • Think OOP for C hackers and Unix heads. 3
4. Elastic Image Software LLC • Class: Grouping of data + code. The type of an object. • Instance: A specific copy of a class. • Method: A message that an object can respond to. • Instance variable (ivar): A piece of data belonging to an object 4 Objective-C Supports
5. Elastic Image Software LLC 5 Interface - .h file @interface className : superclassName @property(nonatomic, retain) Type *propertyForType; +(return_type)classMethod; +(return_type)classMethodWithParam:(paramType) paramName; -(return_type)instanceMethodWithParam1:(param1Type)param1Name andParam2:(param2Type) param2Name; @end
6. Elastic Image Software LLC 6 @implementation classname @synthesize propertyForType; +(return_type)classMethod { // do stuff } +(return_type)classMethodWithParam:(paramType) paramName { // do stuff } -(return_type)instanceMethodWithParam1:(param1Type)param1Name andParam2:(param2Type) param2Name { // do stuff } @end Implementation - .m file
7. Elastic Image Software LLC Apple style tends towards longSelfDocumentingMethodNames. 7
8. Elastic Image Software LLC 8 Instantiation self.window = [[UIWindow alloc] initWithFrame:[[UIScreen mainScreen] bounds]]; Property setting self.window.backgroundColor = [UIColor whiteColor]; Message passing [self.window makeKeyAndVisible];
9. Elastic Image Software LLC 9 Objective-C Types
10. Elastic Image Software LLC 10 Dynamically-typed: id whoCaresWhatThisIs; Statically-typed: Thang* aThang;
11. Elastic Image Software LLC 11 Selectors identify methods by name @interface Observer : NSObject - (id)initWithTarget:(id)object action:(SEL)action; @property(nonatomic, strong) id targetObject; @property(nonatomic) SEL targetAction; @end
12. Elastic Image Software LLC 12 Selectors identify methods by name observer = [[Observer alloc] initWithTarget:self action:@selector(updateDisplay:)];
13. Elastic Image Software LLC 13 Objective Lifecycle • Create an instance. • Futz with it (Send messages. Pass it around.) • Discard it.
14. Elastic Image Software LLC iOS is designed around one foundational pattern: Model View Controller. Much of iOS development - excluding Model development - involves customization and extension of this pattern. 14
15. Elastic Image Software LLC 15 Controller Model View reference reference reference
16. Elastic Image Software LLC 16 Controller Model View Target - Action Key-value Observing or Notification Key-value Observing or Notification
17. Elastic Image Software LLC 17 iOS has rich support for a loose, flat, decoupled style of programming • Notification • Target - Action • Key-value Observing (KVO) • Block & Dispatch Queue • Delegation (Protocol)
18. Elastic Image Software LLC 18 Notifications
19. Elastic Image Software LLC 19 [[NSNotificationCenter defaultCenter] postNotificationName:MyNotification object:self]; Notification Notifier
20. Elastic Image Software LLC 20 [[NSNotificationCenter defaultCenter] addObserver:self selector:@selector(respondToMyNotification:) name:MyNotification object:nil]; Notification - (void) respondToMyNotification:(NSNotification *)notification { // do stuff } Notification respondent
21. Elastic Image Software LLC 21 Target - Action
22. Elastic Image Software LLC 22 Target - Action
23. Elastic Image Software LLC 23 Target - Action
24. Elastic Image Software LLC 24 Target - Action
25. Elastic Image Software LLC 25 Target - Action @interface Counter : NSObject @property(nonatomic, strong) NSNumber *count; - (IBAction)trackSlider:(UISlider *)slider; @end
26. Elastic Image Software LLC 26 Target - Action @implementation Counter -(IBAction) trackSlider:(UISlider *)slider; {! self.count = [NSNumber numberWithFloat:slider.value]; } @end
27. Elastic Image Software LLC 27 Target - Action @class Observer; @class Counter; @interface EIViewController : UIViewController @property(nonatomic, strong) IBOutlet UILabel *label; @property(nonatomic, strong) Observer *observer; @property(nonatomic, strong) IBOutlet Counter *counter; - (void)updateLabel:(NSNumber *)newValue; @end
28. Elastic Image Software LLC 28 Target - Action @implementation EIViewController - (void)viewDidLoad { self.observer = [[Observer alloc] initWithTarget:self action:@selector(updateLabel:)]; [self.counter addObserver:self.observer forKeyPath:@"count" options:NSKeyValueObservingOptionNew context:NULL]; } -(void) updateLabel:(NSNumber *)newValue { self.label.text = [NSString stringWithFormat:@"%.2f", [newValue floatValue]]; } @end
29. Elastic Image Software LLC 29 Key-value Observing (KVO)
30. Elastic Image Software LLC 30 Any property is by default “Key-value Compliant” allowing it to be observed.
31. Elastic Image Software LLC 31 Example: HelloSlider GitHub: http://bit.ly/XDxIvp
32. Elastic Image Software LLC 32 Example: HelloSlider GitHub: http://bit.ly/XDxIvp
33. Elastic Image Software LLC 33 Example: HelloSlider GitHub: http://bit.ly/XDxIvp
34. Elastic Image Software LLC 34 Example: HelloSlider GitHub: http://bit.ly/XDxIvp @interface Counter : NSObject @property(nonatomic, strong) NSNumber *count; - (IBAction)trackSlider:(UISlider *)slider; @end
35. Elastic Image Software LLC 35 Example: HelloSlider GitHub: http://bit.ly/XDxIvp @implementation Counter -(IBAction) trackSlider:(UISlider *)slider; {! self.count = [NSNumber numberWithFloat:slider.value]; } @end
36. Elastic Image Software LLC 36 @interface Observer : NSObject - (id)initWithTarget:(id)object action:(SEL)action; @property(nonatomic, strong) id targetObject; @property(nonatomic) SEL targetAction; @end Example: HelloSlider GitHub: http://bit.ly/XDxIvp
37. Elastic Image Software LLC 37 @implementation Observer - (void)observeValueForKeyPath:(NSString *)keyPath ofObject:(id)object change:(NSDictionary *)change context:(void *)context { [self.targetObject performSelectorOnMainThread:self.targetAction withObject:[object valueForKeyPath:keyPath] waitUntilDone:NO]; } @end Example: HelloSlider GitHub: http://bit.ly/XDxIvp
38. Elastic Image Software LLC 38 Example: HelloSlider GitHub: http://bit.ly/XDxIvp @implementation EIViewController - (void)viewDidLoad { self.observer = [[Observer alloc] initWithTarget:self action:@selector(updateLabel:)]; [self.counter addObserver:self.observer forKeyPath:@"count" options:NSKeyValueObservingOptionNew context:NULL]; } -(void) updateLabel:(NSNumber *)newValue { self.label.text = [NSString stringWithFormat:@"%.2f", [newValue floatValue]]; } @end
39. Elastic Image Software LLC 39 Blocks & Dispatch Queues
40. Elastic Image Software LLC 40 ^{ NSLog(@"Doing something"); } Blocks & Dispatch Queues dispatch_queue_t queue = dispatch_get_global_queue(0,0); dispatch_async(queue, ^{ NSLog(@"Doing something"); }); Block Dispatch Queue
41. Elastic Image Software LLC 41 Blocks & Dispatch Queues - (void)updateFeaturesWithNotification:(NSNotification *)notification { dispatch_async([UIApplication sharedIGVAppDelegate].trackControllerQueue, ^{ [self updateFeatures]; dispatch_async(dispatch_get_main_queue(), ^{ [self.coverageTrack setNeedsDisplay]; [self.track setNeedsDisplay]; }); }); }
42. Elastic Image Software LLC 42 Delegation (Protocol)
43. Elastic Image Software LLC 43 A protocol is identical to an interface in Java. A collection of method signatures implemented by the object that “conforms” to the protocol. The delegate/protocol pattern is ubiquitous throughout iOS. Delegation (Protocol)
44. Elastic Image Software LLC 44 @interface UITableViewController:UIViewController <UITableViewDelegate, UITableViewDataSource> UITableViewController inherits from UIViewController and conforms to the UITableViewDelegate and UITableViewDataSource protocols Delegation (Protocol)
45. Elastic Image Software LLC 45 UITableViewDelegate @protocol UITableViewDelegate<NSObject, UIScrollViewDelegate> @optional - (NSIndexPath *)tableView:willSelectRowAtIndexPath:; - (NSIndexPath *)tableView:willDeselectRowAtIndexPath:; @end
46. Elastic Image Software LLC 46 @protocol UITableViewDataSource<NSObject> @required - (NSInteger)tableView:numberOfRowsInSection:; @optional - (NSInteger)numberOfSectionsInTableView:; - (NSArray *)sectionIndexTitlesForTableView:; @end UITableViewDataSource
47. Elastic Image Software LLC 47 iOS devices combine mobility, gesture, and a powerful GPU. This makes iOS App development an entirely new form of software development.
48. Elastic Image Software LLC 48 Mobility introduces context as a driver for usage. New app types appear to meet user needs driven by context.
49. Elastic Image Software LLC 49 Gesture forces a “no interface”, approach to app design and U/X mindset. Data representation drives interaction and associated affordances.
50. Elastic Image Software LLC 50 GPU. That cute little iPhone in your hand is a graphics processing beast. It is a GPU device tamed for domestic use. The entire interface is GPU driven. That is why iOS apps feel the way they do. Light. Effortless. Friction free. Like butter.
51. Elastic Image Software LLC 51 Developers must discard many of their desktop assumptions when developing for iOS • No mouse • No interface • Minimal keyboard • Arms length interaction • One handed Interaction • Two handed Interaction • Untethered resources
52. Elastic Image Software LLC 52 Demos & Code
53. Elastic Image Software LLC 53 Multi-resolution Image •CATiledLayer •UIScrollView •Amazon Web Services (S3)
54. Elastic Image Software LLC 54 Multi-resolution Image • 8k x 8k image resolution. 101MB on disk. • Subsampled 4 successive times • Diced into 256 x 256 tiles • Stored on Amazon S3
55. Elastic Image Software LLC 55
56. Elastic Image Software LLC 56
57. Elastic Image Software LLC 57
58. Elastic Image Software LLC 58 IGV for iPad •CoreGraphics •Dispatch Queues •UIScrollView
59. Elastic Image Software LLC 59
60. Elastic Image Software LLC 60
61. Elastic Image Software LLC 61
62. Elastic Image Software LLC 62
63. Elastic Image Software LLC 63
64. Elastic Image Software LLC 64
65. Elastic Image Software LLC 65
66. Elastic Image Software LLC 66
67. Elastic Image Software LLC 67
68. Elastic Image Software LLC 68
69. Elastic Image Software LLC 69
70. Elastic Image Software LLC 70
71. Elastic Image Software LLC 71
72. Elastic Image Software LLC 72 The Elastic Image •OpenGL •GLSL - OpenGL Shading Language •Shader/Gesture plug-ability via plist •UISplitViewController
73. Elastic Image Software LLC 73
74. Elastic Image Software LLC 74
75. Elastic Image Software LLC 75
76. Elastic Image Software LLC 76
77. Elastic Image Software LLC 77
78. Elastic Image Software LLC 78
79. Elastic Image Software LLC 79
80. Elastic Image Software LLC 80
81. Elastic Image Software LLC 81
82. Elastic Image Software LLC 82
83. Elastic Image Software LLC 83
84. Elastic Image Software LLC 84
85. Elastic Image Software LLC 85
86. Elastic Image Software LLC 86
87. Elastic Image Software LLC 87
88. Elastic Image Software LLC 88 self.panGesture = [[UIPanGestureRecognizer alloc] initWithTarget:self action:@selector(handlePanGesture:)]; [self addGestureRecognizer:self.panGesture]; The Elastic Image Gestures are fundamental to iOS apps. A gesture is attached to a UIView. Gestures come in different flavors. self.scaleGesture = [[UIPinchGestureRecognizer alloc] initWithTarget:self action:@selector(handleScaleGesture:)]; [self addGestureRecognizer:self. scaleGesture]; self.toggleGesture = [[UITapGestureRecognizer alloc] initWithTarget:self action:@selector(handleToggleGesture:)]; self.toggleGesture.numberOfTapsRequired!! = 1; self.toggleGesture.numberOfTouchesRequired! = 1; [self addGestureRecognizer:self.toggleGesture]; Pan Pinch Tap
89. Elastic Image Software LLC 89 The Elastic Image Photo f/x are implemented in GLSL. The OpenGL Shading Language. Shaders are written in a C-like language and evaluated in a SIMD manner on the entire image in realtime.
90. Elastic Image Software LLC 90 The Elastic Image varying! vec2 v_st; uniform sampler2D hero; void main() { ! ! gl_FragColor = (heroChannels == 1) ? vec4(texture2D(hero, v_st).a) : texture2D(hero, v_st); ! } varying! vec2 v_st; uniform sampler2D hero; void main() { ! ! gl_FragColor = (heroChannels == 1) ? vec4(texture2D(hero, v_st).a) : texture2D(hero, v_st); ! } TextureMapShader - apply a texture (the photo) to a quad (the rendering surface).
91. Elastic Image Software LLC 91 The Elastic Image Property lists enable simple specification of a shader gesture and its handler. The Objective-C runtimes enables easy conversion from string to class instance
92. Elastic Image Software LLC 92 - (UIGestureRecognizer *)createGestureWithDictionary:(NSDictionary *)gestureDictionary { NSString *gestureClassName = [gestureDictionary objectForKey:@"class"]; Class _gesture = NSClassFromString(gestureClassName); NSString *selectorName = [gestureDictionary objectForKey:@"selector"]; SEL _selector = NSSelectorFromString(selectorName); UIGestureRecognizer *shaderGesture = [[[_gesture alloc] initWithTarget:self action:_selector] autorelease]; shaderGesture.delegate = self.detailController; return shaderGesture; } The Elastic Image Property lists enable simple specification of a shader gesture and its handler. The Objective-C runtimes enables easy conversion from string to class instance
93. Elastic Image Software LLC 93 •OpenGL •GLSL - OpenGL Shading Language •Proprietary Panorama Engine •UIPopoverController Beautiful Panoramas
94. Elastic Image Software LLC 94
95. Elastic Image Software LLC 95
96. Elastic Image Software LLC 96
97. Elastic Image Software LLC 97
98. Elastic Image Software LLC 98
99. Elastic Image Software LLC 99
100. Elastic Image Software LLC 100
101. Elastic Image Software LLC 101 •OpenGL •GLSL - OpenGL Shading Language •Proprietary Panorama Engine •3D Spatial Picking BMW Interior Panorama
102. Elastic Image Software LLC 102
103. Elastic Image Software LLC 103
104. Elastic Image Software LLC 104
105. Elastic Image Software LLC 105
106. Elastic Image Software LLC 106
107. Elastic Image Software LLC 107
108. Elastic Image Software LLC 108 RadPad •OpenGL •GLSL - OpenGL Shading Language •Wavelet Image Decompression •UISplitViewController •UIScrollViewController
109. Elastic Image Software LLC 109
110. Elastic Image Software LLC Thank You Douglass Turner Elastic Image Software tweets: @dugla email: douglass.turner@gmail.com