Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A Hitchhikers Guide to the Cloud Friendly Migra...

A Hitchhikers Guide to the Cloud Friendly Migration of Apache Solr

This talk will cover the process of migrating a large-scale, multi-language Apache Solr infrastructure from on-premise servers to AWS cloud infrastructure. The presentation will detail the challenges and best practices involved in the migration, including considerations for instance sizing, data transfer, configuring autoscaling and load balancing, while maintaining optimal search performance. The talk will also highlight the benefits of the migration, including increased scalability, cost savings, and improved disaster recovery capabilities. Real-world examples will be used to illustrate the process, building blocks and tools used during the migration and the talk will conclude with a discussion of future optimizations.

M.-Leander Reimer

June 21, 2023
Tweet

More Decks by M.-Leander Reimer

Other Decks in Technology

Transcript

  1. qaware.de A Hitchhikers Guide to the Cloud Friendly Migration of

    Apache Solr Mario-Leander Reimer [email protected] Managing Director | CTO QAware GmbH Wolfgang Klimt [email protected] IT Troubleshooter ConSol Software GmbH
  2. Old Architecture QAware | 5 28x 28x 28x 8 Apache

    Instances Search Load 28x Loader
  3. New Architecture, Step one QAware | 6 28 x 3(+)

    28x 8 Apache Instances Search 28x Loader Load
  4. New Architecture, Step one QAware | 7 28 x 3(+)

    28x 8 Apache Instances Search 28x Loader Load Staging 28 x
  5. New Architecture, Step two QAware | 9 28 x 3(+)

    Search 8 Apache Instances 28 x Staging 28 x Load Loader Loader Loader Loader ? x Loader
  6. Go Live of Step two QAware | 10 28 x

    3(+) Search 8 Apache Instances 28 x Staging Loader Loader Loader Loader 28x 28x Loader
  7. Go Live of Step two QAware | 11 28 x

    3(+) Search 8 Apache Instances 28 x Staging Loader Loader Loader Loader 28x 28x Loader
  8. Technologies QAware | 12 ▪ Mainly AWS EC2 machines and

    ALBs ▪ Terraform and Terragrunt used as Infrastructure as Code solution ▪ Hashicorp Packer used to create custom AMIs with Apache Solr distribution and AWS Corretto JDK pre-installed ▪ AWS Aurora but also Apache Ignite as staging databases ▪ Envoy Proxies to access on-premise datacenter from VPC, mainly to save IP addresses on direct connect ▪ EBS snapshots for faster startup of Solr instance on scale-up
  9. Learnings and Tips QAware | 13 ▪ Plan a gradually

    Go Live ▪ Have a fallback plan ▪ Many small instances behave different to few big instances ▪ Reserve time for improvements after go live ▪ Test thoroughly, but don’t expect to have tested everything
  10. Q&A

  11. qaware.de QAware GmbH Aschauer Straße 32 81549 München Tel. +49

    89 232315-0 [email protected] twitter.com/qaware linkedin.com/company/qaware-gmbh xing.com/companies/qawaregmbh slideshare.net/qaware github.com/qaware