2017 - Preventing headaches with linters and automated checks

Description

This talk will teach you how to use and extend tools for automated checks on Python projects. Are your imports a mess? No reasonable order, stdlib modules mixed with third-party ones? There's a tool to fix that. Tired of checking for security patches of the libs in your requirements.txt? Let a tool do this for you. We'll learn about many other tools and we'll also discuss how to build new ones.

Abstract

While it's very common to enforce PEP8 code style with tools like pycodestyle or flake8, it's rare for Python projects to use other types of tools for automated checks. However, there are many common problems in readability, security, configuration, etc that could be avoided by using other linters and tools, for example:

Are your imports a complete mess, with third-party modules mixed with stdlib ones? You can use "isort" to organize and separate them.
Tired of checking if your project requirements received security patches? Let "safety" do that for you.
Hated when that fellow developer pushed a huge file that slowed your repository pulls forever? A "pre-commit" hook with a simple check could've prevented that.
Is your code cluttered with unused legacy functions and classes? Check and prevent that with "vulture".
As we can see from the list above, many issues can be prevented at commit or CI time with automated tools. In this talk, we'll discuss how to configure and use those tools. Also, we'll learn the role of static analysis in those tools, which will enable us to extend them and build new ones.

Here is a non-exhaustive list of tools that will be presented:

prospector: https://github.com/landscapeio/prospector
pylint: https://github.com/PyCQA/pylint
safety: https://github.com/pyupio/safety
bandit: https://github.com/openstack/bandit
pre-commit: http://pre-commit.com/
isort: https://github.com/timothycrosley/isort
vulture: https://github.com/jendrikseipp/vulture
pycycle: https://github.com/bndr/pycycle
pyt: https://github.com/python-security/pyt
Django System check framework: https://docs.djangoproject.com/en/1.10/ref/checks/

Bio

Web developer from Brazil. Loves beautiful high-quality products, from UX to code, and will defend them against unreasonable deadlines and crazy features. Partner at Vinta (https://www.vinta.com.br/), a web consultancy specialized in building products with React and Django.

https://www.youtube.com/watch?v=mXqKaM8Ddac

Transcript

1. Flávio Juvenal @flaviojuvenal vintasoftware.com Preventing headaches with linters and automated checks
2. Slides are here: bit.ly/djangocon-linters
4. Linters and automated checks: 1. What are they 2. Why using them 3. How to implement them 4. When to run them 5. Which ones exist
5. What?
6. lint
7. lint (uncountable) 1. clinging fuzzy fluff that accumulates... and we hate. en.wiktionary.org/wiki/lint (not exactly)
8. linter
9. Also valid for software...
10. linter (countable) 1. tool to analyze code to find flaws and errors, helping to remove that clinging fuzzy fluff we hate. en.wikipedia.org/wiki/Lint_(software) (not exactly)
11. $ cat my_sum.py def my_sum(x, x): return x + y $ pyflakes my_sum.py my_sum.py:1: duplicate argument 'x' in function definition my_sum.py:2: undefined name 'y'
12. Why?
13. from django.db import models class PersonQuerySet(models.QuerySet): def admin_authors(self): self.filter( role='A', is_admin=True) What's wrong?
14. Can a linter really detect this…?
15. Yes! We'll see how
16. Linters prevent: • bad style (e.g. pycodestyle) • bad patterns (e.g. flake8-bugbear) • bugs (e.g. pylint) • security vulnerabilities (e.g. bandit)
17. Expert devs know language/framework/library-related quirks and common mistakes. They should write this knowledge down as linter checks to perpetuate it.
18. For orgs, linters consolidate knowledge in executable form • enforce long-lasting good practices • automate code quality checks • help to train new developers
19. Orgs are doing this! • twisted/twistedchecker • openstack-dev/hacking • edx/edx-lint • saltstack/salt-pylint • all via custom checks plugins that tools support: pycodestyle, flake8, pylint, bandit, coala, etc.
20. Orgs are checking: • blacklisted modules, e.g. test-only libs • inconsistent string formatting • i18n calls on non-literal strings • missing try/except on optional third-party imports • prohibit locals() calls • missing super() call on unittest setUp/tearDown • too broad assertRaises on tests • etc…
21. Linters make better UX for libraries and frameworks • error prevention is good UX • Django does that...
22. python manage.py check docs.djangoproject.com/en/dev/topics/checks/
23. $ cat example_project/urls.py urlpatterns = [ url(r'^admin/', admin.site.urls), url(r'^app1/', include('app1.urls', namespace='app1')), url(r'^app2/', include('app2.urls', namespace='app1')), ] $ python manage.py check System check identified some issues: WARNINGS: ?: (urls.W005) URL namespace 'app1' isn't unique. You may not be able to reverse all URLs in this namespace System check identified 1 issue (0 silenced).
24. class ArrayField(base_field, size=None, **options) If you give the field a default, ensure it’s a callable. Incorrectly using default=[] creates a mutable default shared between all instances of ArrayField. docs.djangoproject.com/en/1.11/ref/ contrib/postgres/fields/#arrayfield
25. Suggestion for Django: Perhaps every "ensure", "remember to", "don't forget", or similar warnings on Django docs should become a new system check…
26. How?
27. dynamic analysis or static analysis
28. dynamic analysis • performed by executing code, not necessarily at runtime • in order to work, certain checks need to be dynamic, i.e., they need to execute Django. E.g., a check for unapplied migrations • Django system check framework is dynamic
29. Let's solve this with a dynamic check: from django.db import models from django.contrib.postgres.fields import ArrayField class Post(models.Model): tags = ArrayField( models.CharField(max_length=200), default=[])
30. Demo! system_checks/checks.py
31. static analysis • performed without actually executing code • safer and more general than dynamic analysis, it can analyse all code flows • similar to code review, but performed by machines
32. static analysis types 1. text/regex-based 2. token-based 3. AST-based 4. Inference-based
33. text/regex-based • Ideal for simple checks • Example: dodgy, looks at Python code to search for things which look "dodgy" such as passwords or diffs
34. whitespace_before_parameters @ pycodestyle.py#L699 $ cat tokenize_me.py print ("Hi") $ python -m tokenize tokenize_me.py ... 1,0-1,5: NAME 'print' 1,6-1,7: OP '(' ... token-based
35. token-based • Better to get structure than raw text • Does not lose info: x == untokenize(tokenize(x)) • Ideal for style checks • Example: pycodestyle (formerly pep8), part of flake8
36. import ast, astor tree = ast.parse( ''' def add(x, y): return x + y ''') print(astor.dump(tree)) Abstract Syntax Tree-based
37. AST-based Module( body=[ FunctionDef(name='add', args=arguments( args=[arg(arg='x'), arg(arg='y')]), body=[Return( value=BinOp( left=Name(id='x'), op=Add, right=Name(id='y')))])]) greentreesnakes.readthedocs.io/en/latest/nodes.html
38. AST-based • Abstract Syntax Tree: tree structure that represents code • Abstracts some info (e.g. "if-elif-else" becomes nested "if-else"s) • …
39. AST-based • Ideal for checks that need to analyze structure of the code as a whole, checking the relationships between parts, like logic errors (e.g. "undefined name") • Example: pyflakes, part of flake8
40. AST is made for walking ♪ ♬ ♩ class FuncLister(ast.NodeVisitor): def visit_FunctionDef(self, node): print(node.name) self.generic_visit(node) FuncLister().visit(tree)
41. from django.db import models class PersonQuerySet(models.QuerySet): def admin_authors(self): self.filter( role='A', is_admin=True) Let's solve this with AST walking:
42. Let's solve this with AST walking: greentreesnakes.readthedocs.io/en/latest/nodes.html#Call
43. Let's solve this with AST walking: greentreesnakes.readthedocs.io/en/latest/nodes.html#Expr
44. Demo! ast_qs.py
45. class PersonQuerySet(models.QuerySet): def authors(self): return self.filter(role='A') def admin_authors(self): self.authors().filter( is_admin=True) What if…
46. Would be great if we could infer the type of self.authors()...
47. ...actually, we can!
48. Inference-based • Can infer info from AST nodes, like imports/variables/attrs resolution, literal op results, classes MRO, types, etc. • "An interpreter that doesn't execute the code" • Example: pylint, thanks to astroid lib
49. Demo! ast_qs_inferred.py
50. What about mypy type inference?
51. github.com/python/mypy/ issues/2097 Not yet :(
52. However… disobeying_guido_on_mypy.py
53. google/pytype • Used in 500+ internal Google projects, all in Python 2 • Only initial Python 3 support pycharm • Can work with incomplete ASTs • Implemented in Java Both • Make use of type annotations • Have inference capabilities
54. When?
55. when to run linters 1. programming-time 2. commit-time 3. continuous integration-time 4. code review-time
56. programming-time
57. Some stuff should never be committed
58. commit-time • Git hook to run linters before commit • pre-commit.com - written in Python!
59. continuous integration-time github.com/vintasoftware/django-react-boilerplate Fail your build if any linter report any issue
60. code review-time github.com/lyft/linty_fresh
61. Which?
62. dozens of linters available! • quality: flake8, flake8-bugbear, pylint, pydiatra, vulture • imports: isort, pycycle • docs: pydocstyle • security: bandit, dodgy, pyt, hacking, safety, dependency-check • packaging: pyroma, check-manifest • cpython: cpychecker • spelling: scspell3k • typing: mypy • wrap them all with prospector (python-only) or coala (general)
63. github.com/vintasoftware/python-linters-and-code-analysis
64. some ideas for new Django checks: 1. string formatting in raw/extra queries 2. null=True in CharField/TextField 3. null=True in BooleanField instead of NullBooleanField 4. method that overrides save doesn't use commit argument 5. ModelForm save doesn't return saved instance 6. Celery task call with ORM model instance as argument 7. ATOMIC_REQUESTS=True + Celery delay inside view 8. etc… Let's sprint on those for django-bug-finder
65. If it's difficult for code analysis to understand your code, maybe it'll be difficult for your fellow developers too…
66. Feel free to reach me: twitter.com/flaviojuvenal flavio@vinta.com.br vintasoftware.com Let's contribute: django-bug-finder python-linters-and-code-analysis Slides for this and other Vinta talks at: bit.ly/vinta2017 Thanks! Questions?
67. References ‒ Pylint - an overview of the static analysis tool for Python, Claudiu Popa https://www.youtube.com/watch?v=p1wPOIYt8Ws ‒ 12 years of Pylint (or How I learned to stop worrying about bugs) https://www.youtube.com/watch?v=0jKbNpEjkhI Slides: http://pcmanticore.github.io/pylint-talks/#slide:1 ‒ Andrey Vlasovskikh - Static analysis of Python https://www.youtube.com/watch?v=lJtED-xN-HE ‒ Dave Halter - Identifying Bugs Before Runtime With Jedi https://www.youtube.com/watch?v=yPSmj2kmX8g ‒ Static Code Analysis with Python https://www.youtube.com/watch?v=mfXIJ-Fu5Fw ‒ Writing custom checkers for Pylint https://breadcrumbscollector.tech/writing-custom-checkers-for-pylint/ ‒ Writing pylint plugins https://nedbatchelder.com/blog/201505/writing_pylint_plugins.html ‒ Pylint and dynamically populated packages http://blog.devork.be/2014/12/pylint-and-dynamically-populated.html
68. ‒ To AST and Beyond by Curtis Maloney https://www.youtube.com/watch?v=N_Q3i3oaZ6w ‒ Andreas Dewes - Learning from other's mistakes: Data-driven analysis of Python code - PyCon 2015 https://www.youtube.com/watch?v=rN0kNQLDYCI ‒ Why Pylint is both useful and unusable, and how you can actually use it https://codewithoutrules.com/2016/10/19/pylint/ ‒ Interview: Claudiu Popa – Using Pylint for Python Static Analysis https://blog.sqreen.io/interview-pylint-for-python-static-analysis/ ‒ Andrey Vlasovskikh - Static analysis of Python https://www.youtube.com/watch?v=lJtED-xN-HE ‒ Static Code Analysis for All Languages - coala (by Lasse Schuirmann) https://www.youtube.com/watch?v=oFawYQ0EonY ‒ Hacking Python AST: checking methods declaration https://julien.danjou.info/blog/2015/python-ast-checking-method-declaration References
69. References ‒ How Python Linters Will Save Your Large Python Project https://jeffknupp.com/blog/2016/12/09/how-python-linters-will-save-your-lar ge-python-project/ ‒ https://github.com/jwilk/check-all-the-things/blob/master/data/python.ini
70. Extra slides
71. fixers • some linters have the ability to automatically fix the code, this is great UX! • e.g. isort, autoflake, autopep8, etc. • Coala integrates checkers with fixers seamlessly
72. more examples of dynamic analysis • executes in a real-like env or in the real env • real-like env: pyroma, which runs setup.py to check Python packages health • real env: a check for unapplied migrations in Django