Upgrade to Pro — share decks privately, control downloads, hide ads and more …

2017 - Preventing headaches with linters and au...

PyBay
August 21, 2017

2017 - Preventing headaches with linters and automated checks

Description

This talk will teach you how to use and extend tools for automated checks on Python projects. Are your imports a mess? No reasonable order, stdlib modules mixed with third-party ones? There's a tool to fix that. Tired of checking for security patches of the libs in your requirements.txt? Let a tool do this for you. We'll learn about many other tools and we'll also discuss how to build new ones.

Abstract

While it's very common to enforce PEP8 code style with tools like pycodestyle or flake8, it's rare for Python projects to use other types of tools for automated checks. However, there are many common problems in readability, security, configuration, etc that could be avoided by using other linters and tools, for example:

Are your imports a complete mess, with third-party modules mixed with stdlib ones? You can use "isort" to organize and separate them.
Tired of checking if your project requirements received security patches? Let "safety" do that for you.
Hated when that fellow developer pushed a huge file that slowed your repository pulls forever? A "pre-commit" hook with a simple check could've prevented that.
Is your code cluttered with unused legacy functions and classes? Check and prevent that with "vulture".
As we can see from the list above, many issues can be prevented at commit or CI time with automated tools. In this talk, we'll discuss how to configure and use those tools. Also, we'll learn the role of static analysis in those tools, which will enable us to extend them and build new ones.

Here is a non-exhaustive list of tools that will be presented:

prospector: https://github.com/landscapeio/prospector
pylint: https://github.com/PyCQA/pylint
safety: https://github.com/pyupio/safety
bandit: https://github.com/openstack/bandit
pre-commit: http://pre-commit.com/
isort: https://github.com/timothycrosley/isort
vulture: https://github.com/jendrikseipp/vulture
pycycle: https://github.com/bndr/pycycle
pyt: https://github.com/python-security/pyt
Django System check framework: https://docs.djangoproject.com/en/1.10/ref/checks/

Bio

Web developer from Brazil. Loves beautiful high-quality products, from UX to code, and will defend them against unreasonable deadlines and crazy features. Partner at Vinta (https://www.vinta.com.br/), a web consultancy specialized in building products with React and Django.

https://www.youtube.com/watch?v=mXqKaM8Ddac

PyBay

August 21, 2017
Tweet

More Decks by PyBay

Other Decks in Programming

Transcript

  1. Linters and automated checks: 1. What are they 2. Why

    using them 3. How to implement them 4. When to run them 5. Which ones exist
  2. lint (uncountable) 1. clinging fuzzy fluff that accumulates... and we

    hate. en.wiktionary.org/wiki/lint (not exactly)
  3. linter (countable) 1. tool to analyze code to find flaws

    and errors, helping to remove that clinging fuzzy fluff we hate. en.wikipedia.org/wiki/Lint_(software) (not exactly)
  4. $ cat my_sum.py def my_sum(x, x): return x + y

    $ pyflakes my_sum.py my_sum.py:1: duplicate argument 'x' in function definition my_sum.py:2: undefined name 'y'
  5. Linters prevent: • bad style (e.g. pycodestyle) • bad patterns

    (e.g. flake8-bugbear) • bugs (e.g. pylint) • security vulnerabilities (e.g. bandit)
  6. Expert devs know language/framework/library-related quirks and common mistakes. They should

    write this knowledge down as linter checks to perpetuate it.
  7. For orgs, linters consolidate knowledge in executable form • enforce

    long-lasting good practices • automate code quality checks • help to train new developers
  8. Orgs are doing this! • twisted/twistedchecker • openstack-dev/hacking • edx/edx-lint

    • saltstack/salt-pylint • all via custom checks plugins that tools support: pycodestyle, flake8, pylint, bandit, coala, etc.
  9. Orgs are checking: • blacklisted modules, e.g. test-only libs •

    inconsistent string formatting • i18n calls on non-literal strings • missing try/except on optional third-party imports • prohibit locals() calls • missing super() call on unittest setUp/tearDown • too broad assertRaises on tests • etc…
  10. Linters make better UX for libraries and frameworks • error

    prevention is good UX • Django does that...
  11. $ cat example_project/urls.py urlpatterns = [ url(r'^admin/', admin.site.urls), url(r'^app1/', include('app1.urls',

    namespace='app1')), url(r'^app2/', include('app2.urls', namespace='app1')), ] $ python manage.py check System check identified some issues: WARNINGS: ?: (urls.W005) URL namespace 'app1' isn't unique. You may not be able to reverse all URLs in this namespace System check identified 1 issue (0 silenced).
  12. class ArrayField(base_field, size=None, **options) If you give the field a

    default, ensure it’s a callable. Incorrectly using default=[] creates a mutable default shared between all instances of ArrayField. docs.djangoproject.com/en/1.11/ref/ contrib/postgres/fields/#arrayfield
  13. Suggestion for Django: Perhaps every "ensure", "remember to", "don't forget",

    or similar warnings on Django docs should become a new system check…
  14. dynamic analysis • performed by executing code, not necessarily at

    runtime • in order to work, certain checks need to be dynamic, i.e., they need to execute Django. E.g., a check for unapplied migrations • Django system check framework is dynamic
  15. Let's solve this with a dynamic check: from django.db import

    models from django.contrib.postgres.fields import ArrayField class Post(models.Model): tags = ArrayField( models.CharField(max_length=200), default=[])
  16. static analysis • performed without actually executing code • safer

    and more general than dynamic analysis, it can analyse all code flows • similar to code review, but performed by machines
  17. text/regex-based • Ideal for simple checks • Example: dodgy, looks

    at Python code to search for things which look "dodgy" such as passwords or diffs
  18. whitespace_before_parameters @ pycodestyle.py#L699 $ cat tokenize_me.py print ("Hi") $ python

    -m tokenize tokenize_me.py ... 1,0-1,5: NAME 'print' 1,6-1,7: OP '(' ... token-based
  19. token-based • Better to get structure than raw text •

    Does not lose info: x == untokenize(tokenize(x)) • Ideal for style checks • Example: pycodestyle (formerly pep8), part of flake8
  20. import ast, astor tree = ast.parse( ''' def add(x, y):

    return x + y ''') print(astor.dump(tree)) Abstract Syntax Tree-based
  21. AST-based • Abstract Syntax Tree: tree structure that represents code

    • Abstracts some info (e.g. "if-elif-else" becomes nested "if-else"s) • …
  22. AST-based • Ideal for checks that need to analyze structure

    of the code as a whole, checking the relationships between parts, like logic errors (e.g. "undefined name") • Example: pyflakes, part of flake8
  23. AST is made for walking ♪ ♬ ♩ class FuncLister(ast.NodeVisitor):

    def visit_FunctionDef(self, node): print(node.name) self.generic_visit(node) FuncLister().visit(tree)
  24. Inference-based • Can infer info from AST nodes, like imports/variables/attrs

    resolution, literal op results, classes MRO, types, etc. • "An interpreter that doesn't execute the code" • Example: pylint, thanks to astroid lib
  25. google/pytype • Used in 500+ internal Google projects, all in

    Python 2 • Only initial Python 3 support pycharm • Can work with incomplete ASTs • Implemented in Java Both • Make use of type annotations • Have inference capabilities
  26. commit-time • Git hook to run linters before commit •

    pre-commit.com - written in Python!
  27. dozens of linters available! • quality: flake8, flake8-bugbear, pylint, pydiatra,

    vulture • imports: isort, pycycle • docs: pydocstyle • security: bandit, dodgy, pyt, hacking, safety, dependency-check • packaging: pyroma, check-manifest • cpython: cpychecker • spelling: scspell3k • typing: mypy • wrap them all with prospector (python-only) or coala (general)
  28. some ideas for new Django checks: 1. string formatting in

    raw/extra queries 2. null=True in CharField/TextField 3. null=True in BooleanField instead of NullBooleanField 4. method that overrides save doesn't use commit argument 5. ModelForm save doesn't return saved instance 6. Celery task call with ORM model instance as argument 7. ATOMIC_REQUESTS=True + Celery delay inside view 8. etc… Let's sprint on those for django-bug-finder
  29. If it's difficult for code analysis to understand your code,

    maybe it'll be difficult for your fellow developers too…
  30. Feel free to reach me: twitter.com/flaviojuvenal [email protected] vintasoftware.com Let's contribute:

    django-bug-finder python-linters-and-code-analysis Slides for this and other Vinta talks at: bit.ly/vinta2017 Thanks! Questions?
  31. References ‒ Pylint - an overview of the static analysis

    tool for Python, Claudiu Popa https://www.youtube.com/watch?v=p1wPOIYt8Ws ‒ 12 years of Pylint (or How I learned to stop worrying about bugs) https://www.youtube.com/watch?v=0jKbNpEjkhI Slides: http://pcmanticore.github.io/pylint-talks/#slide:1 ‒ Andrey Vlasovskikh - Static analysis of Python https://www.youtube.com/watch?v=lJtED-xN-HE ‒ Dave Halter - Identifying Bugs Before Runtime With Jedi https://www.youtube.com/watch?v=yPSmj2kmX8g ‒ Static Code Analysis with Python https://www.youtube.com/watch?v=mfXIJ-Fu5Fw ‒ Writing custom checkers for Pylint https://breadcrumbscollector.tech/writing-custom-checkers-for-pylint/ ‒ Writing pylint plugins https://nedbatchelder.com/blog/201505/writing_pylint_plugins.html ‒ Pylint and dynamically populated packages http://blog.devork.be/2014/12/pylint-and-dynamically-populated.html
  32. ‒ To AST and Beyond by Curtis Maloney https://www.youtube.com/watch?v=N_Q3i3oaZ6w ‒

    Andreas Dewes - Learning from other's mistakes: Data-driven analysis of Python code - PyCon 2015 https://www.youtube.com/watch?v=rN0kNQLDYCI ‒ Why Pylint is both useful and unusable, and how you can actually use it https://codewithoutrules.com/2016/10/19/pylint/ ‒ Interview: Claudiu Popa – Using Pylint for Python Static Analysis https://blog.sqreen.io/interview-pylint-for-python-static-analysis/ ‒ Andrey Vlasovskikh - Static analysis of Python https://www.youtube.com/watch?v=lJtED-xN-HE ‒ Static Code Analysis for All Languages - coala (by Lasse Schuirmann) https://www.youtube.com/watch?v=oFawYQ0EonY ‒ Hacking Python AST: checking methods declaration https://julien.danjou.info/blog/2015/python-ast-checking-method-declaration References
  33. References ‒ How Python Linters Will Save Your Large Python

    Project https://jeffknupp.com/blog/2016/12/09/how-python-linters-will-save-your-lar ge-python-project/ ‒ https://github.com/jwilk/check-all-the-things/blob/master/data/python.ini
  34. fixers • some linters have the ability to automatically fix

    the code, this is great UX! • e.g. isort, autoflake, autopep8, etc. • Coala integrates checkers with fixers seamlessly
  35. more examples of dynamic analysis • executes in a real-like

    env or in the real env • real-like env: pyroma, which runs setup.py to check Python packages health • real env: a check for unapplied migrations in Django