Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Developing microservice infrastructure for LINE...

Developing microservice infrastructure for LINE Securities

Kentaro Maeda
LINE Financial Development Center / SRE Team Server Side Engineer
https://linedevday.linecorp.com/2020/ja/sessions/7096
https://linedevday.linecorp.com/2020/en/sessions/7096

LINE DevDay 2020

November 27, 2020
Tweet

More Decks by LINE DevDay 2020

Other Decks in Technology

Transcript

  1. Agenda › About LINE Securities › LINE Securities system issues

    › Developing microservices platform › Future Prospects
  2. Familiar and Easy Investment › Small amounts › Launch on

    2019/8 › Easy processes › Simple UI
  3. “Ichi-Kabu” One stock trading › Trade until 9:00 p.m. ›

    Trade stock from one stock unit › Trading stocks increase to 1015
  4. eKYC – electronic Known your customer › Opening account without

    mailing › Start trading as soon as the next business day.
  5. LINE Securities services 2019/8 “Ichi-Kabu”(One stock trading) service launched. 2019/10

    eKYC Service launched. 2019/11 Investment trust service launched. 2020/3 LINE FX(Foreign Exchange) launched. 2020/5 Japanese stocks Market trading and Margin trading service launched. 2020/6 Periodic investment trust service launched. 2020/8 Opened accounts 300,000 Over! 2019/12 First “Time Sale Campaign”(Stock discount sale) started.
  6. Scale of LINE Securities system ※ excludes LINE FX, external

    services. 2019/8 2020/8 Services 11 70 Servers 65 180 The system was built on a dedicated network in LINE's data center.
  7. LINE Securities built on on-premise › We need a private

    network separate from other LINE’s services. › Complex network and dedicated connection is hampered development. › We cannot use Verda (Private Cloud) because it didn’t satisfy financial security criteria at the time.
  8. Complicated server setup process › Server setup took two weeks,

    even for a senior engineer. › Each service has cluster with Load Balancer, Domain, TLS Cert, Complex network configuration. › Development process was not common in each services.
  9. Ineffective use of server resources › Over 90% requests comes

    when market is opening (AM 9:00 – PM 3:00) › Scaling out on a per service requires a lot of servers.
  10. For rapid service development › We need common development rule.

    › We need easy server setup process. › We developed service platform with scalable and flexible resources.
  11. Service platform policy › Develop from well-known tools as building

    blocks. › Use OSS to be independent of specific infrastructure and tools. › Develop and Maintain with a minimal team.
  12. Architecture Overview › Sharing resources in all services › Load

    balancer in Edge server › Dynamic Deployment › All services connect directly
  13. Server Pool & Management Servers › Nomad: deploy container to

    server pool › Consul: catalog service address › Service: service and sidecar envoy container › Vault: manage secret secure › Control Plane: inject services address to envoy
  14. Docker: packaging and development rule › Setup server commonly with

    ansible › Keep clean server pool › Package binary and runtime as image › Separate developer and infra with minimum rule
  15. Nomad: Deploy service containers › Deploy container to available server

    following manifest › Container liveness check, auto recovery, rolling update. › Job: container count, image name, resource limit, other,,,
  16. Consul: Service Catalog, health check › Store service configuration in

    KVS › Store service dynamic address › Health check, alert changes quickly
  17. envoy & Control Plane: Service mesh › envoy is sidecar

    proxy for ingress/egress requests. › envoy configuration reloadable from API (xDS protocol) › Control Plane inject service address from consul to envoy.
  18. Result of service platform › Easy scale up. After market

    trading launched, traffic has more than doubled. 10 server setup and scaling up container finished one day. › Improved development speed Developers can focus on service and don’t concern to infrastructure. › Effective resource using. Sharing multiple service in one server. Batch processing run on server pool in nightly.
  19. Storage and Monitoring issues remain › Log storage: over 200

    GB log writes in one day › Complex monitoring tools: Prometheus, Zabbix, Inhouse monitoring tool,,, › Own running middleware: kafka, Elasticsearch, Airflow.
  20. Measures for improvement › Reorganization logging and monitoring tools. We

    are looking for new monitoring tool widely. SaaS, Cloud, OSS,,, › Challenge again to use Verda. More server scale up, management services, object storage, k8s, We are consulting with Verda team to satisfy security standard.
  21. Conclusion › We’ll continue to improve efficiency and automation. ›

    We’ve developed flexible and scalable platform on-premise. › We‘ll scaling LINE Securities over million account!