A Functional Programming Approach To Data Processing In Python

LambdaConf Workshop on Functional Programming with Python

Transcript

1. A FUNCTIONAL PROGRAMMING APPROACH TO LambdaConf — Boulder, Colorado — May 26, 2017 DATA PROCESSING IN PYTHON By Reuben Cummings
2. Reuben Cummings λ @reubano λ #LambdaConf Who am I? Managing Director, Nerevu Development Founder, Arusha Coders Author of several popular Python packages
3. WHAT IS DATA? I dare you, I double dare you! Image Credit: www.emaze.com SAY BIG DATA ONE MORE TIME
4. Reuben Cummings λ @reubano λ #LambdaConf Organization language presenter mercury alex scala gleb haskell michael "This session seeks to entertain and teach the developer who is already..." structured unstructured
5. Reuben Cummings λ @reubano λ #LambdaConf Storage type,duration leap,360 hop,120 de novo,60 inspire,10 00103e0 b0e6 04... 00105f0 e4e7 03... 0010600 0be8 04... 00105b0 c4e4 02... 00106e0 b0e9 04... flat/text binary
6. Reuben Cummings λ @reubano λ #LambdaConf Organization vs Storage flat/text binary structured unstructured
7. What is it data processing good for?
8. Spotify's Discovery Weekly playlist of new songs you like adapts to user's shifting musical tastes handles outliers and seasonality Image Credit: www.spotify.com/int/discoverweekly/
9. What is functional programming good for?
10. Om rapid UI re-renders serializable application state time travel/undo Image Credit: circleci.com
11. A BRIEF INTRO TO PYTHON Ooouuu, stickers! Image Credit: www.pythongear.com
12. Reuben Cummings λ @reubano λ #LambdaConf Presentation: GitHub repo github.com/reubano/lambdaconf- tutorial
13. Reuben Cummings λ @reubano λ #LambdaConf Presentation: Jupyter Notebook beta.mybinder.org/v2/gh/reubano/ lambdaconf-tutorial/master (presentation.ipybn)
14. Naive Image Credit: (Alika Seu) www.flickr.com
15. Reading data
16. Reuben Cummings λ @reubano λ #LambdaConf Reading data (naive) from urllib.request import urlopen from json import loads BASE = 'https://api.github.com/search' _url1 = '{}/repositories?q={}' q = 'data&per_page=100' url1 = _url1.format(BASE, q) f = urlopen(url1)
17. Reuben Cummings λ @reubano λ #LambdaConf Reading data (naive) from urllib.request import urlopen from json import loads BASE = 'https://api.github.com/search' _url1 = '{}/repositories?q={}' q = 'data&per_page=100' url1 = _url1.format(BASE, q) f = urlopen(url1)
18. Reuben Cummings λ @reubano λ #LambdaConf Reading data (naive) from urllib.request import urlopen from json import loads BASE = 'https://api.github.com/search' _url1 = '{}/repositories?q={}' q = 'data&per_page=100' url1 = _url1.format(BASE, q) f = urlopen(url1)
19. GitHub API Image Credit: https://api.github.com/search/repositories?q=data
20. Reuben Cummings λ @reubano λ #LambdaConf >>> data = loads(f.read().decode('utf-8')) Reading data (naive)
21. Reuben Cummings λ @reubano λ #LambdaConf >>> repos = data['items'] >>> repos[0]['description'] 'Jargon from the functional programming world in simple terms!' >>> repos[0]['full_name'] 'hemanth/functional-programming-jargon' >>> data = loads(f.read().decode('utf-8')) Reading data (naive)
22. Processing data
23. Reuben Cummings λ @reubano λ #LambdaConf Processing data (naive) def rate(repos): rated = [] for repo in repos: rated.append(repo['watchers'] * 2) return rated
24. Reuben Cummings λ @reubano λ #LambdaConf Processing data (naive) def rate(repos): rated = [] for repo in repos: rated.append(repo['watchers'] * 2) return rated
25. Reuben Cummings λ @reubano λ #LambdaConf Processing data (naive) def rate(repos): rated = [] for repo in repos: rated.append(repo['watchers'] * 2) return rated
26. Reuben Cummings λ @reubano λ #LambdaConf Processing data (naive) def rate(repos): rated = [] for repo in repos: rated.append(repo['watchers'] * 2) return rated
27. Reuben Cummings λ @reubano λ #LambdaConf Processing data (naive) >>> rate(repos)[:5] [36520, 30174, 28576, 26842, 24092]
28. Reuben Cummings λ @reubano λ #LambdaConf >>> from itertools import count >>> >>> inf_repos = ( ... {'watchers': c} for c in count()) Processing infinite data (naive)
29. Reuben Cummings λ @reubano λ #LambdaConf >>> from itertools import count >>> >>> inf_repos = ( ... {'watchers': c} for c in count()) >>> >>> rate(inf_repos) Processing infinite data (naive)
30. Reuben Cummings λ @reubano λ #LambdaConf KeyboardInterrupt Traceback (most recent call last) <ipython-input-212-e2ea27b0be2f> in <module>() >>> from itertools import count >>> >>> inf_repos = ( ... {'watchers': c} for c in count()) >>> >>> rate(inf_repos) Processing infinite data (naive)
31. Reuben Cummings λ @reubano λ #LambdaConf rated = [] for repo in repos: rated.append(repo['watchers'] * 2) return rated def rate(repos): Processing expensive data (naive)
32. Reuben Cummings λ @reubano λ #LambdaConf def exp_rate(repos): rated = [] for repo in repos: rated.append(repo['watchers'] * 2) return rated Processing expensive data (naive)
33. Reuben Cummings λ @reubano λ #LambdaConf from time import sleep def exp_rate(repos): rated = [] for repo in repos: rated.append(repo['watchers'] * 2) return rated Processing expensive data (naive)
34. Reuben Cummings λ @reubano λ #LambdaConf from time import sleep def exp_rate(repos): rated = [] for repo in repos: rated.append(repo['watchers'] * 2) return rated sleep(5) Processing expensive data (naive)
35. Reuben Cummings λ @reubano λ #LambdaConf >>> exp_rate(repos)[:5] Processing expensive data (naive)
36. Reuben Cummings λ @reubano λ #LambdaConf [36520, 30174, 28576, 26842, 24092] >>> exp_rate(repos)[:5] Processing expensive data (naive)
37. Lazy evaluation Image Credit: (Mark Turnauckas) www.flickr.com
38. Lazy intro
39. Reuben Cummings λ @reubano λ #LambdaConf >>> next(lazy_list) 0 >>> eager_list = list(range(5)) >>> eager_list [0, 1, 2, 3, 4] >>> lazy_list = iter(eager_list) >>> lazy_list <list_iterator at 0x10c2af978> Iterators
40. Reuben Cummings λ @reubano λ #LambdaConf >>> next(lazy_list) StopIteration Traceback (most recent call last) <ipython-input-68-898b6387b693> in <module>() ----> 1 next(lazy_list) Iterators >>> list(lazy_list) [1, 2, 3, 4]
41. Reading data
42. Reuben Cummings λ @reubano λ #LambdaConf $ pip install ijson Reading data (lazy evaluation)
43. Reuben Cummings λ @reubano λ #LambdaConf >>> from ijson import items >>> >>> f = urlopen(url1) >>> repos = items(f, 'items.item') >>> repos <generator object items at 0x110c70db0> >>> repo = next(repos) >>> repo['full_name'] 'hemanth/functional-programming-jargon' Reading data (lazy evaluation)
44. Processing data
45. Reuben Cummings λ @reubano λ #LambdaConf rated = [] for repo in repos: rated.append(repo['watchers'] * 2) return rated def rate(repos): Processing data (lazy evaluation)
46. Reuben Cummings λ @reubano λ #LambdaConf rated = [] for repo in repos: rated.append(repo['watchers'] * 2) return rated def gen_rates(repos): Processing data (lazy evaluation)
47. Reuben Cummings λ @reubano λ #LambdaConf def gen_rates(repos): for repo in repos: yield repo['watchers'] * 2 Processing data (lazy evaluation)
48. Reuben Cummings λ @reubano λ #LambdaConf >>> rates = gen_rates(repos) >>> next(rates) 36520 >>> next(rates) 30174 >>> gen_rates(repos) <generator object gen_rate at 0x160c70db0> Processing data (lazy evaluation)
49. Reuben Cummings λ @reubano λ #LambdaConf Processing infinite data (lazy evaluation) >>> rates = gen_rates(inf_repos) >>> next(rates) 42220156
50. Reuben Cummings λ @reubano λ #LambdaConf Processing expensive data (lazy evaluation) def gen_exp_rates(repos): for repo in repos: sleep(5) yield repo['watchers'] * 2
51. Reuben Cummings λ @reubano λ #LambdaConf Processing expensive data (lazy evaluation) def gen_exp_rates(repos): for repo in repos: sleep(5) yield repo['watchers'] * 2
52. Reuben Cummings λ @reubano λ #LambdaConf >>> list(result) >>> from itertools import islice >>> >>> rates = gen_exp_rates(repos) >>> result = islice(rates, 5) Processing expensive data (lazy evaluation)
53. Reuben Cummings λ @reubano λ #LambdaConf [36520, 30174, 28576, 26842, 24092] >>> list(result) >>> from itertools import islice >>> >>> rates = gen_exp_rates(repos) >>> result = islice(rates, 5) Processing expensive data (lazy evaluation)
54. Reuben Cummings λ @reubano λ #LambdaConf >>> from itertools import islice >>> >>> rates = gen_exp_rates(repos) >>> result = islice(rates, 5) >>> list(result) [36520, 30174, 28576, 26842, 24092] >>> next(rates) 648 Processing expensive data (lazy evaluation)
55. Grouping data
56. Reuben Cummings λ @reubano λ #LambdaConf Grouping data >>> f = urlopen(url1) >>> repos = items(f, 'items.item') >>> repo = next(repos) >>> repo.keys() dict_keys(['id', 'name', 'full_name', 'owner', 'private', 'html_url', 'description', 'fork', 'url', 'forks_url', 'keys_url', ...])
57. Reuben Cummings λ @reubano λ #LambdaConf Grouping data >>> repo['has_issues'] True
58. Reuben Cummings λ @reubano λ #LambdaConf Grouping data >>> import itertools as it >>> from operator import itemgetter >>> >>> keyfunc = itemgetter('has_issues') >>> sorted_repos = sorted(repos, key=keyfunc) >>> grouped = it.groupby( ... sorted_repos, keyfunc) >>> data = ( ... (k, len(list(g))) for k, g in grouped)
59. Reuben Cummings λ @reubano λ #LambdaConf Grouping data >>> next(data) (False, 3) >>> next(data) (True, 96)
60. Memoization Image Credit: (olho wodzynski) www.flickr.com
61. Processing data
62. Reuben Cummings λ @reubano λ #LambdaConf def gen_exp_rates(repos): for repo in repos: sleep(5) yield repo['watchers'] * 2 Processing expensive data (memoization)
63. Reuben Cummings λ @reubano λ #LambdaConf def calc_rate(watchers): sleep(5) return watchers * 2 def gen_exp_rates(repos): for repo in repos: yield calc_rate(repo['watchers']) Processing expensive data (memoization)
64. Reuben Cummings λ @reubano λ #LambdaConf def _calc_rate(watchers): cacher = lru_cache() calc_rate = cacher(_calc_rate) from functools import lru_cache sleep(5) return watchers * 2 Processing expensive data (memoization)
65. Reuben Cummings λ @reubano λ #LambdaConf @lru_cache() from functools import lru_cache def calc_rate(watchers): sleep(5) return watchers * 2 def gen_exp_rates(repos): for repo in repos: yield calc_rate(repo['watchers']) Processing expensive data (memoization)
66. Reuben Cummings λ @reubano λ #LambdaConf [10, 10, 10, 10, 10] >>> list(result) >>> repos = it.repeat({'watchers': 5}) >>> rates = gen_exp_rates(repos) >>> result = islice(rates, 5) Processing expensive data (memoization)
67. EXERCISE #1 Mount Meru — Arusha, Tanzania Image Credit: Reuben Cummings
68. Reuben Cummings λ @reubano λ #LambdaConf Exercise #1: Problem display the total # of watchers per language (ignore repos w/o a language)
69. Reuben Cummings λ @reubano λ #LambdaConf Exercise #1: Result C# 32 C++ 63 HTML 349 JavaScript 3881 Jupyter Notebook 5481 PHP 201 Python 37007 R 18
70. Reuben Cummings λ @reubano λ #LambdaConf Exercise #1: Data source https://api.github.com/search/ repositories?q=data
71. Reuben Cummings λ @reubano λ #LambdaConf Exercise #1: Jupyter Notebook beta.mybinder.org/v2/gh/reubano/ lambdaconf-tutorial/master (exercises.ipybn)
72. Reuben Cummings λ @reubano λ #LambdaConf from urllib.request import urlopen from itertools import groupby from operator import itemgetter from ijson import items url2 = '{}/repositories?q=data'.format(BASE) f = urlopen(url2) repos = items(f, 'items.item') Exercise #1: Solution
73. Reuben Cummings λ @reubano λ #LambdaConf keyfunc = itemgetter('language') cleaned = filter(keyfunc, repos) records = sorted(cleaned, key=keyfunc) grouped = groupby(records, keyfunc) for key, group in grouped: cnt = sum(g['watchers'] for g in group) print(key, cnt) Exercise #1: Solution
74. Reuben Cummings λ @reubano λ #LambdaConf Exercise #1: Solution beta.mybinder.org/v2/gh/reubano/ lambdaconf-tutorial/master (solutions.ipybn)
75. INTRODUCING MEZA Because you might not need Pandas Image Credit: github.com/reubano/meza
76. Reuben Cummings λ @reubano λ #LambdaConf $ pip install meza Meza demo
77. Reuben Cummings λ @reubano λ #LambdaConf Meza demo: Jupyter Notebook beta.mybinder.org/v2/gh/reubano/ lambdaconf-tutorial/master (presentation.ipybn)
78. Reading data
79. Reuben Cummings λ @reubano λ #LambdaConf Reading data >>> from urllib.request import urlopen >>> from meza.io import read_json >>> >>> f = urlopen(url2) >>> records = read_json(f, path='items.item') >>> repo = next(records) >>> repo['full_name'] 'emberjs/data'
80. Reuben Cummings λ @reubano λ #LambdaConf Reading data >>> len(list(records)) 29
81. Reuben Cummings λ @reubano λ #LambdaConf Reading data >>> from io import StringIO >>> from meza.io import read_csv >>> >>> f = StringIO( ... 'greeting,location\nhello,world\n') >>> >>> next(read_csv(f)) {'greeting': 'hello', 'location': 'world'}
82. Reuben Cummings λ @reubano λ #LambdaConf Reading data >>> from os import path as p >>> from meza.io import join >>> >>> url3 = '{}&page=2'.format(url2) >>> files = map(urlopen, [url2, url3]) >>> records = join( ... *files, ext='json', path='items.item')
83. Reuben Cummings λ @reubano λ #LambdaConf Reading data >>> repo = next(records) >>> repo['full_name'] 'emberjs/data'
84. Reuben Cummings λ @reubano λ #LambdaConf Reading data >>> repo = next(records) >>> repo['full_name'] 'emberjs/data' >>> repo['language'] 'JavaScript'
85. Reuben Cummings λ @reubano λ #LambdaConf Reading data >>> repo = next(records) >>> repo['full_name'] 'emberjs/data' >>> repo['language'] 'JavaScript' >>> len(list(records)) 59
86. Transforming data
87. Reuben Cummings λ @reubano λ #LambdaConf Transforming data >>> from meza.process import merge >>> >>> records = [ ... {'a': 200}, {'b': 300}, {'c': 400}] >>> >>> merge(records) {'a': 200, 'b': 300, 'c': 400}
88. Reuben Cummings λ @reubano λ #LambdaConf Transforming data >>> from meza.process import group >>> >>> records = [ ... {'item': 'a', 'amount': 200}, ... {'item': 'a', 'amount': 200}, ... {'item': 'b', 'amount': 400}] >>> >>> grouped = group(records, 'item')
89. Reuben Cummings λ @reubano λ #LambdaConf Transforming data >>> key, _group = next(grouped) >>> key 'a' >>> _group [{'amount': 200, 'item': 'a'}, {'amount': 200, 'item': 'a'}]
90. Reuben Cummings λ @reubano λ #LambdaConf Transforming data >>> from meza import process as pr >>> >>> f = urlopen(url2) >>> raw = read_json(f, path='items.item') >>> fields = [ ... 'full_name', 'language', 'watchers', ... 'score', 'has_wiki'] >>> >>> cut = pr.cut(raw, fields)
91. Reuben Cummings λ @reubano λ #LambdaConf Transforming data >>> cut <generator object cut.<locals>.<genexpr> at 0x10b0410f8> >>> cut, preview = pr.peek(cut) >>> cut <itertools.chain at 0x10c2ad5f8> >>> len(preview) 5
92. Reuben Cummings λ @reubano λ #LambdaConf Transforming data >>> preview[0] {'full_name': 'substance/data', 'has_wiki': True, 'language': 'JavaScript', 'score': Decimal('72.90926'), 'watchers': 678}
93. Reuben Cummings λ @reubano λ #LambdaConf Transforming data >>> filled = pr.fillempty( ... raw, value='', fields=['language']) >>> >>> pivoted = pr.pivot( ... filled, 'score', 'language', ... rows=['has_wiki'], op=min)
94. Reuben Cummings λ @reubano λ #LambdaConf Transforming data >>> next(pivoted) {'HTML': Decimal('73.52254'), 'JavaScript': Decimal('53.48755'), 'PHP': Decimal('41.3122'), 'Python': Decimal('42.49319'), 'has_wiki': False}
95. Reuben Cummings λ @reubano λ #LambdaConf Transforming data >>> next(pivoted) {'': Decimal('44.83392'), 'C#': Decimal('47.793495'), 'HTML': Decimal('69.20008'), 'JavaScript': Decimal('70.15174'), 'PHP': Decimal('44.251198'), 'Python': Decimal('45.78215'), 'R': Decimal('46.23451'), 'has_wiki': True}
96. Reuben Cummings λ @reubano λ #LambdaConf | full_name | language | score | has_wiki | | --------- | ---------- | ------ | -------- | | 'aptnote…' | '' | 76.11… | True | | 'GSA/dat…' | 'HTML' | 73.52… | False | | 'substan…' | 'JavaScr…' | 72.83… | True | | 'GoogleT…' | 'JavaScr…' | 70.15… | True | | 'curran/…' | 'HTML' | 69.20… | True | Transforming data (before)
97. Reuben Cummings λ @reubano λ #LambdaConf | has_wiki | '' | HTML | JavaScript | | -------- | -------- | -------- | ---------- | | False | | 73.52254 | | | True | 76.11933 | 69.20008 | 70.15174 | Transforming data (after)
98. EXERCISE #2 Image Credit: Reuben Cummings Mount Kilimanjaro — Kilimanjaro Region, Tanzania
99. Reuben Cummings λ @reubano λ #LambdaConf Exercise #2: Problem display the language with the most # of watchers per owner_type per has_pages
100. Reuben Cummings λ @reubano λ #LambdaConf Exercise #2: Result (partial) {'has_pages': True, 'language': 'JavaScript', 'owner_type': 'Organization', 'watchers': 128605}
101. Reuben Cummings λ @reubano λ #LambdaConf Exercise #2: Data source https://api.github.com/search/ repositories? q=data&sort=stars&order=desc
102. Reuben Cummings λ @reubano λ #LambdaConf Exercise #2: Hint from meza.fntools import flatten # and one of the following from meza.process import normalize # this from meza.process import aggregate # or this
103. Reuben Cummings λ @reubano λ #LambdaConf Exercise #2: Jupyter Notebook beta.mybinder.org/v2/gh/reubano/ lambdaconf-tutorial/master (exercises.ipybn)
104. Reuben Cummings λ @reubano λ #LambdaConf from urllib.request import urlopen from operator import itemgetter from functools import partial from meza import process as pr, fntools as ft from meza.io import read_json q = 'data&sort=stars&order=desc' url4 = '{}/repositories?q={}'.format(BASE, q) f = urlopen(url4) Exercise #2: Solution
105. Reuben Cummings λ @reubano λ #LambdaConf records = read_json(f, path='items.item') filled = pr.fillempty( records, value='', fields=['language']) flat = (dict(ft.flatten(r)) for r in filled) args = ('watchers', 'language') rows = ['has_pages', 'owner_type'] Exercise #2: Solution
106. Reuben Cummings λ @reubano λ #LambdaConf spun = pr.pivot( flat, *args, rows=rows, op=sum) spun, preview = pr.peek(spun) Exercise #2: Solution
107. Reuben Cummings λ @reubano λ #LambdaConf >>> preview[0] {'C#': 7675, 'C++': 55602, 'Go': 13223, 'Objective-C': 10556, … 'has_pages': False, 'owner_type': 'Organization'} Exercise #2: Solution
108. Reuben Cummings λ @reubano λ #LambdaConf >>> kw = {'rows': rows, 'invert': True} >>> normal = pr.normalize(spun, *args, **kw) >>> normal, preview = pr.peek(normal) >>> preview[0] {'has_pages': False, 'language': 'Objective-C', 'owner_type': 'Organization', 'watchers': 10556} Exercise #2: Solution
109. Reuben Cummings λ @reubano λ #LambdaConf akeyfunc = itemgetter('watchers') gkeyfunc = lambda x: tuple(x[r] for r in rows) aggregator = partial(max, key=akeyfunc) kwargs = { 'tupled': False, 'aggregator': aggregator} grouped = pr.group(normal, gkeyfunc, **kwargs) Exercise #2: Solution
110. Reuben Cummings λ @reubano λ #LambdaConf >>> grouped, preview = pr.peek(grouped) >>> preview[0] {'has_pages': False, 'language': 'C++', 'owner_type': 'Organization', 'watchers': 55602} Exercise #2: Solution
111. Reuben Cummings λ @reubano λ #LambdaConf sgrouped = sorted( grouped, key=akeyfunc, reverse=True) for record in sgrouped: print(record) Exercise #2: Solution
112. Reuben Cummings λ @reubano λ #LambdaConf | language | watchers | owner_ty… | has_pages | | -------- | -------- | --------- | --------- | | 'JavaS…' | 128605 | 'Organi…' | True | | 'C++' | 55602 | 'Organi…' | False | | 'Python' | 54269 | 'User' | False | | 'Jupyte…'| 12046 | 'User' | True | Exercise #2: Result (full)
113. Reuben Cummings λ @reubano λ #LambdaConf Exercise #2: Solution beta.mybinder.org/v2/gh/reubano/ lambdaconf-tutorial/master (solutions.ipybn)
114. Thanks! Reuben Cummings @reubano
115. Extra Slides
116. Reuben Cummings λ @reubano λ #LambdaConf def gen_rates(repos): for repo in repos: yield repo['watchers'] * 2 Processing data (lazy evaluation)
117. Reuben Cummings λ @reubano λ #LambdaConf def gen_rates(repos): return ( r['watchers'] * 2 for r in repos) Processing data (lazy evaluation)
118. Reuben Cummings λ @reubano λ #LambdaConf from urllib.request import urlopen from operator import itemgetter from functools import partial from meza import process as pr, fntools as ft from meza.io import read_json q = 'data&sort=stars&order=desc' url4 = '{}/repositories?q={}'.format(BASE, q) f = urlopen(url4) Exercise #2: Alt. solution
119. Reuben Cummings λ @reubano λ #LambdaConf records = read_json(f, path='items.item') filled = pr.fillempty( records, value='', fields=['language']) flat = (dict(ft.flatten(r)) for r in filled) akeyfunc = itemgetter('watchers') Exercise #2: Alt. solution
120. Reuben Cummings λ @reubano λ #LambdaConf rows = [ 'has_pages', 'owner_type', 'language', 'watchers'] def grouper(records, rows, aggregator): kwargs = {'aggregator': aggregator} key = lambda x: tuple(x[r] for r in rows) _grouper = partial(pr.group, tupled=False) return _grouper(records, key, **kwargs) Exercise #2: Alt. solution
121. Reuben Cummings λ @reubano λ #LambdaConf def agg1(records): args = (records, 'watchers', sum) return pr.aggregate(*args) grouped = grouper(flat, rows[:3], agg1) agg2 = partial(max, key=akeyfunc) regrouped = grouper(grouped, rows[:2], agg2) cut = pr.cut(regrouped, rows) Exercise #2: Alt. solution
122. Reuben Cummings λ @reubano λ #LambdaConf >>> cut, preview = pr.peek(cut) >>> preview[0] {'has_pages': False, 'language': 'C++', 'owner_type': 'Organization', 'watchers': 55602} Exercise #2: Alt. solution
123. Reuben Cummings λ @reubano λ #LambdaConf sgrouped = sorted( cut, key=akeyfunc, reverse=True) for record in sgrouped: print(record) Exercise #2: Alt. solution