Share
Embed
The Lost Chapter: Rediscovering Foundations in Software Design
Lemi Orhan Ergin PRO
October 07, 2023
Transcript
43 slides
-
THE LOST CHAPTER Rediscovering Foundations in So ft LEMİ ORHAN
ERGİN co-founder, craftgate -
THE LOST CHAPTER Rediscovering Foundations in So ft LEMİ ORHAN
ERGİN co-founder, craftgate -
We make the most significant decisions when we know least
Adding new features becomes harder and harder in time Complexity rises till the dead end You either redesign or rewrite to continue adding new features Microservices transformation projects usually aim to fix these kinds of legacy projects LIFE OF A SOFTWARE Feature Count Time Growth Limit Growth Limit let’s split it into microservices we need to rewrite it again Difficulty Level While Adding New Features Time refactor redesign zone Growth Limit Growth Limit the increase in endurance level is due to the new senior endurance level: do not touch if it works transfor- mation project zone go-live night -
No need for big redesigns or rewrites, refactoring is enough
Refactoring should be continuous The system should be open to extension and close to modification We learn from experiences and adapt the system accordingly No direct need for microservices, expansion in monolith should be fine Feature Count Time Difficulty Level While Adding New Features Time LIFE OF A SOFTWARE HOW IT SHOULD BE -
Payment API after a payment are completed, we save the
data to database every design starts with the simpliest solution IMPLEMENT PAYMENT OPERATIONS 1 Save payment Retrieve payment AN ORDINARY STORY OF LOVE & HATE OF AN API EVOLUTION -
Payment API We need to search existing payments FEATURE REQUEST
where do we need to add the search operation ? It is a payment operation, so add it to same service ? Save payment Retrieve payment -
Payment API it’s a simple as querying the database with
a search criteria and returning the results we add it to payment component as a new responsibility, a new flow 2 ADD SEARCH TO EXISTING FLOW Save payment Retrieve payment Search payment start with the simplest solution -
Payment API Search service requires additional models, introduces new dependencies
and differentiates from core payment flow payment service starts to be huge and unmaintainable (new dependencies for search operation are not used and relevant with core payment flow) search flow has different business needs and triggered by different roles a change in the repository for search improvements directly triggers change in core payment flow Save payment Retrieve payment Search payment -
Payment API SEPARATE SEARCH AS A NEW SERVICE 3 separate
the search logic from core payment logic, introduce a new service with its own dependencies Save payment Retrieve payment Search payment -
Payment API Single place for doing db operations becomes unmaintainable,
includes many responsibilities too many reponsibilities included, attracts more functionality as a single point for DB operations Save payment Retrieve payment Search payment -
Payment API separate the search db operation from other db
operations SPLIT INTO BOUNDED CONTEXTS 4 Save payment Retrieve payment Search payment H E X A O G N -
Search queries takes time and start to have impact on
save queries at database level both services depend on the same data and the performance of one side affects the other we moved read operations into a new repository Payment API Save payment Retrieve payment Search payment -
MASTER REPLICA search and payment flows do not depend on
the same database anymore Payment API Save payment Retrieve payment Search payment CHANGE THE DATA SOURCE 5 Read Write -
MASTER REPLICA Search results are expected to be returned as
files in csv and excel formats FEATURE REQUEST Payment API Save payment Retrieve payment Search payment Read Write -
Payment API MASTER REPLICA generate report simply searches the data
and fills it in a file with the given extension ADD NEW FLOW TO SEARCH MODULE 6 Save payment Retrieve payment Search payment Generate report Read Write -
Payment API MASTER REPLICA File generation is time and memory
consuming, and affects performance of other operations deploying the whole api is required whenever something added to report logic, that increases the risc on payment Save payment Retrieve payment Search payment Generate report Read Write -
Payment API MASTER REPLICA Report API it’s time to introduce
a new reporting api with generate report capabilities MOVE TO A NEW MICRO SERVICE 7 Save payment Retrieve payment Generate report Read Write Read Search payment -
Report generation takes time and users wait too long, timeout
errors start to occur Payment API MASTER REPLICA Report API things go even worse when multiple users trigger reports at the same time Save payment Retrieve payment Generate report Search payment Read Write Read -
Payment API MASTER REPLICA Report API async users just fire
and forget, then check if report is ready TRIGGER GENERATE REPORT ASYNC 8 Save payment Retrieve payment Generate report Search payment Read Write Read -
Users re-demand reports before previous ones complete, that creates unnecessary
load Payment API MASTER REPLICA Report API async users are usually impatient, they re-trigger report generation again and again Save payment Retrieve payment Generate report Search payment Read Write Read -
Payment API MASTER REPLICA Report API async lock adding a
lock per user lets us to prevent from being overloaded by the same duplicate report generations Save payment Retrieve payment Generate report Search payment THROTTLE DEMANDS FROM PER USER 9 Read Write Read -
Payment API MASTER REPLICA Report API async lock Throttling demands
per user does not prevent system from overloading by report demands by multiple users. Limit parallel report generations but not the demands. throttling demands per user is not enough, multiple demands overload the system Save payment Retrieve payment Generate report Search payment Read Write Read -
Payment API MASTER REPLICA Report API QUEUE each demand is
queued, the throttlling logic can be managed Save payment Retrieve payment Generate report Search payment Demand report QUEUE DEMANDS AND PROCESS 10 Read Write Read -
Payment API MASTER REPLICA Report API QUEUE Report generation affects
performance of search operations since the data source is shared, the load of reporting api has direct impact on payment operations Generate report Demand report Save payment Retrieve payment Search payment Read Write Read -
Payment API MASTER REPLICA Write Read Report API Read QUEUE
ELASTIC separate the read model of report generation from the model of payment operations Generate report Demand report Save payment Retrieve payment Search payment THROTTLE DEMANDS FROM PER USER 11 -
Payment API MASTER REPLICA Report API ELASTIC Inform the user
when the report is generated FEATURE REQUEST Save payment Retrieve payment Search payment QUEUE Generate report Demand report Report API Write Read Read -
Payment API MASTER REPLICA ELASTIC Notification API call email notification
api directly sends emails and slack messages that provided by the report api INTRODUCE NOTIFICATION SERVICE 12 Save payment Retrieve payment Search payment Report API QUEUE Generate report Demand report Report API Send Slack message Send email Write Read Read -
Payment API MASTER REPLICA ELASTIC Notification API call email Report
needs to know notification api and how to trigger its functionalities If notification api stops working, we lose messages Save payment Retrieve payment Search payment Report API QUEUE Generate report Demand report Report API Send Slack message Send email Write Read Read -
Payment API MASTER REPLICA ELASTIC email QUEUE keep messages in
queue and manage the error logic with retries QUEUE NOTIFICATION MESSAGES 13 Save payment Retrieve payment Search payment Report API QUEUE Generate report Demand report Report API Notification API Send Slack message Send email Write Read Read -
Payment API MASTER REPLICA ELASTIC email QUEUE Whenever queue stops
responding, the whole flow stops and messages are lost we lose messages when the queue is down :( Save payment Retrieve payment Search payment Report API QUEUE Generate report Demand report Report API Notification API Send Slack message Send email Write Read Read -
Payment API MASTER REPLICA ELASTIC email Outbox Inbox make messaging
transactional and be sure you create messages MAKE MESSAGING TRANSACTIONAL 14 Save payment Retrieve payment Search payment Report API Generate report Demand report Report API Notification API Send Slack message Send email Generate report Send notification Write Read Read -
Payment API MASTER REPLICA ELASTIC email Outbox Inbox We design
a kind-a maintainable, expandable, robust system at the end make messaging transactional and be sure you create messages Save payment Retrieve payment Search payment Report API Generate report Demand report Report API Notification API Send Slack message Send email Generate report Send notification Write Read Read -
https://twitter.com/KentBeck/status/1218307926818869248 %80 of software development is maintenance, pareto ? Adding
every new feature and every trouble you got trigger a change in the system, and the way you refactor the existing system determines how easy that can be. -
Extract to method, to classes, to functions Centralize decision making
(behavior) Make responsible of one behavior Apply design patters Apply design principles like SOLID Clearify boundaries of contexts Extract functionality as a new microservice Seperate by behaviors, not models or packages Avoid layered arc, build with hexagonal arch Avoid wrappers around CRUD operations Do not depend on same datasource Avoid distributed monoliths Avoid centralized common modules Do not depend on implementations Let the control of flow inverted (IoC) Prefer duplication to make independent Introduce abstrations to eliminate down times WE MANAGED COMPLEXITY BY DECREASING COUPLING AND INCREASING COHESION WHILE REFACTORING AND ADDING NEW CODE -
COUPLING & COHESION Structured Design book copyrighted on 1978, 1975
the most important fundemantal in software design Minimum Cost Systems Black Boxes Modular Design Modularity Software Engineering Complexity Coupling Cohesion Morphology of Simple Systems Design Heuristics Refactoring Transaction Analysis Communication in Modular Systems Packaging Optimization of Modular Systems Typology of System Components Recursive Structures Incremental Structures Program Quality Top-down Implementation Bottom-up Development Management Benefits THE LOST CHAPTER * the chapter has already been there for 40+ years FOR MANAGING SOFTWARE COMPLEXITY -
COUPLING In highly coupled systems, one change triggers a change
in somewhere else The measure of the strength of interconnection, measure of independence If one system waits, changes, works, fails, starts, stops due to another systems action, these systems are highly coupled with the butterfly effect, every change thiggers something bigger Controller Service Service Service Repository Repository Utility Rest Client -
COHESION The code that changes together (i.e. in the same
feature), stays together (only coupled to each other) Belonging together, relatedness, sharing same purpose/responsibility Cohesiveness exists inside boundaries, that makes it a module, a component, an element -
Controller Service Service Service Repository Repository Utility Rest Client Layered
communication increases coupling and destroys cohesion by every change in time -
WE ARE HIRING mail: [email protected] CRAFTGATE one-stop shop payment orchestration
payment hub for your virtual poses with dynamic pos routing, you always pay the lowest commission to banks craftgate.io @craftgateio https://tidyfirst.substack.com/p/how-i-came-to-write-tidy-first The goal of software design is to create chunks or slices that fit into a human mind. The software keeps growing but the human mind maxes out, so we have to keep chunking and slicing differently if we want to keep making changes. https://twitter.com/KentBeck/status/1354418068869398538 Kent Beck -
One-Stop-Shop Payment Orchestration [email protected] we are hiring! LEMİ ORHAN ERGİN
co-founder, craftgate