CSC510 Lecture 24

Reliability
(20261019)

Transcript

1. CSC 5100 Modern Software Engineering Lecture 24. Reliability Dr. Javier Gonzalez-Sanchez javiergs@calpoly.edu www.javiergs.phd ffi o ce: 14 -227
2. Homework :: Extended to Tuesday
3. Reliability From “it works” to “it works" when something fails
4. Key de initions Availability Fault tolerance Probability a system performs its intended function correctly over a period. Fraction of time a system can serve requests. Ability to continue providing an acceptable service despite component faults. f Reliability
5. Key de initions Observability Testability Ability to withstand disruption and recover to an acceptable state. Ability to infer internal behavior from logs, metrics, and traces. Ease of controlling inputs and observing outcomes during verification. f Resilience
6. Reliable systems are designed to fail predictably. 1. Observability ✅ 2. SOLID (Interfaces → Dependency Injec on) ti 3. Controlled Failures →
7. ✅ Observability
8. SOLID Separation of concerns Interfaces Dependency injection.
10. Blueprint
11. Step 1: define interfaces The interface expresses what the service needs—not how AWS works. public interface ResultStorage { void save(String id, String json); } public interface EventPublisher { void publish(String id, String json); } 12
12. Step 2: constructor injection Dependencies are supplied by the caller. No AWS SDK is required in domain code. public final class TemperatureService { private final ResultStorage storage; private final EventPublisher publisher; public TemperatureService(ResultStorage storage, EventPublisher publisher) { this.storage = storage; this.publisher = publisher; } } 13
13. Step 3: orchestrate the request First version: illustrates coupling removal, not yet fault tolerance. public String process(double fahrenheit) { String id = UUID.randomUUID().toString(); double celsius = (fahrenheit - 32) * 5.0 / 9.0; String json = /* serialize id, F and C safely */; storage.save(id, json); publisher.publish(id, json); return json; }
14. Testing with replaceable dependencies Real adapters In-memory fake Failing fake S3ResultStorage and SnsEventPublisher invoke AWS services. Stores results in a Map; no network or credentials. Throws predictable exceptions on selected attempts.
16. Controlled Failure
17. Failure scenario: SNS is unavailable 1 Convert °F → °C 2 Save to DynamoDB 3 Archive to S3 4 Publish to SNS 5 Attempt SQS / MQTT? Design question: does SNS failure prevent other independent attempts?
18. Why distributed failures are different • A process can be healthy while one downstream service is unavailable. • A timeout does not prove the remote operation failed. • One successful write does not imply that all writes succeeded. • Recovery may repeat operations: duplicate effects become possible. • Failure handling must be specified as part of the API contract.
19. Three design decisions Fail fast Best effort Durable handoff Stop processing when a required operation fails. Simple, but may leave partial state. Attempt independent effects and report which succeeded or failed. Persist an event before asynchronous delivery (e.g., outbox pattern). More robust, more complex.
20. What does “success” mean? • HTTP 200 does not guarantee all downstream effects occurred. • Lambda Errors counts invocation failures—not every business-level error. • Define required versus optional side effects before coding. • Use a request ID / UUID to correlate processing, storage, and events. • Return a truthful result: complete, partial, or failed.
21. To be continued...
22. Questions
23. CSC 5100 Modern Software Engineering Javier Gonzalez-Sanchez, Ph.D. javiergs@calpoly.edu Fall 2026 Copyright Notice. These slides are provided exclusively for educational use in CSC 5100 within the California State University system. Redistribution or use for any other purpose is prohibited without prior written permission.