Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Development and Infrastructure for Microservice...

Yosuke
September 21, 2016

Development and Infrastructure for Microservice Architecture

Yosuke

September 21, 2016
Tweet

More Decks by Yosuke

Other Decks in Technology

Transcript

  1. About me Engineer at Nikkei Inc. 2 Yosuke Suzuki @yosukep

    Python, Golang, JavaScript (Worked as an editor and staff writer also)
  2. About Nikkei.com 4 • Launched on Mar 2010 • More

    than 480,000 paid users • 300,000,000 pv / month
  3. Dev team in Nikkei.com 5 • API • Mobile App

    • Front end • Infra • Bigdata Hiring inhouse engineers, migrating to inhouse development
  4. Collaboration with FinancialTimes 6 • Acquired on Dec 2015 •

    Share development technique • English!! Some team use English for internal communication
  5. Definition of Microservices 8 • Small in size✔ • Autnomously

    developed✔ • Independently deployable✔ • Built and released with automated processes✔ • Decentralized O’Reilly: Microservice Architecture
  6. API Gateway Architecture 9 Cache/router (Fastly) UI service UI service

    Internet API gateway Search Content Auth DNS (Internal / Route53) API service API service Image API gateway is not decentralized?
  7. Gateway pattern 10 • Authentication and Authorization ◦ Front-end layer

    sometime does authorization • Some services like image delivery are directly served • Wrapper for old APIs
  8. Small in size✔ 11 • Django based python application ◦

    Each application has a few folders ◦ But API gateway has many folders • NodeJS application adopts express as framework ◦ 200 - 300 lines of JavaScript
  9. Autonomously developed✔ 12 • Do not share resources like DB

    ◦ Use S3 to share data • Use web APIs to exchange data between services • statelss • generate a service from template
  10. Template includes 14 • requirements.txt • tox.ini ◦ Test settings

    and format check by Flake8 • Django application • Dockerfile • ElasticBeanstalk config file • CircleCI config file
  11. Generate by cookiecutter 15 pip install cookiecutter cookiecutter gh:Nikkei/django-project-template Interactive

    command line • repo_name: GitHub repo name • project_name: Django project name • subdomin: domain name • short_description: description
  12. Follow template updates 16 pip install git+https://github.com/hirokiky/cookiepatcher cookiepatcher • Do

    not make service legacy • KY made cookiepatcher • cookiepatcher.json manages diffs
  13. Private PyPI 17 • Internal common codes ◦ Convert article

    id ◦ Replace CMS tags requirements.txt: --extra-index-url https://xxx.x/xxxxx/packages/simple/ nikkei-utils==0.6
  14. Private PyPI (2) 18 1. python setup.py sdist 2. Copy

    generated file like xx.0.1.tar.gz to specified directory 3. By pushing github repo for packages, CircleCI uploads packages automatically
  15. Private PyPI (3) 19 from nikkei_utils.kiji import extract_snippet from nikkei_utils.region

    import region_info def hoge(): snippet = extract_snippet(article.body)
  16. NodeJS app uses private npm 20 package.json: "dependencies": { "@nikkei/nikkei-ui":

    "^6.2.0", "@nikkei/rnikkei-express": "^2.32.2", "denodeify": "^1.2.1", "dom-delegate": "^2.0.3", "express-validator": "^2.20.8", "open-graph-scraper": "^2.2.2" } Get private npm packages from private npm server
  17. Local development environment 21 • It is easy for API

    development ◦ python manage.py runserver ◦ Not need to use Docker • Front-end development is not easy ◦ Routing some services to local computer
  18. Django Rest Framework 22 • Django Rest Framework • Django

    Rest Framework Swagger Productive framework for API development
  19. Rest Framework(2) 23 class FeedAnArticleView(APIView): http_method_names = ['get'] permission_classes =

    (IsAuthenticated,) def get(self, request, kiji_id_raw): """ # 検索エンジン用フィード --- parameters: - name: kiji_id paramType: path description: 対象の記事ID """ article = Article.objects.get(kiji_id_raw=kiji_id_raw, status=Article.POST) return Response(data={'article': feed_article_mapper(article)})
  20. Issues of ElasticBeanstalk command 26 • Cannot use different config

    file(Dockerrun.aws.json) • Blue/Green is not supported
  21. ebi command can do that! 27 pip install ebi ebi

    deploy <app_name> <env_name> • KY made it • Use different Dockerrun.aws.json for each env • Blue/Green deploy support
  22. CircleCI.yml 28 - sudo pip install --ignore-installed awsebcli - sudo

    pip install -U ebi - aws configure set aws_access_key_id $PROD_AWS_ACCESS_KEY_ID - aws configure set aws_secret_access_key $PROD_AWS_SECRET_ACCESS_KEY - aws configure set default.region ap-northeast-1 - aws configure set default.output json - ebi bgdeploy app-name env-name-blue env-name-green cname_prefix --profile default --noswap --region ap-northeast-1 --dockerrun Dockerrun.aws.prod.json - sleep 10 - sudo pip install -r e2etests/requirements.txt - py.test e2etests/tests.py - eb swap env-name-blue --profile default --destination_name env-name-green
  23. Batch 29 Docker Container Docker Container Batch process by Rundeck.

    Use same Docker Image both for Rundeck and ElasticBeanstalk Spot instance Docker Container
  24. Monitoring, logging 31 • Fluentd in another container aggregate logs

    ◦ github.com/bungoume/log-sender • Send error logs to Sentry • Send metrics data to New Relic • CloudWatch
  25. Incident alert 33 Send notification to Slack and call by

    PagerDuty when emegency situation happens
  26. Generic infrastructure 34 • Django/Python application • NodeJS(express) application •

    WordPress • Rundeck • Jenkins • Kibana • Nginx+plugin
  27. Easy to migrate 35 NodeJS application of FinancialTimes runs on

    Heroku Same application runs on ElasticBeanstalk without any modification
  28. Current issues 36 • Scaling out is not so ideal

    ◦ If spike more than expectation happens, scaling out is not fast enough ◦ It is better to use scheduled scaling out • Continuas update of libraries ◦ Django version 1.7 -> 1.10 ◦ Periodically check and update of requirements.txt is needed • AWS is not perfect, shipped with bugs sometime.