DevOps Patterns & Antipatterns for Continuous Software Updates “What can possibly go wrong?!”
Slide 1
Slide 2
Why software updates? @jbaruch #TriangleDevOps #LiquidSoftware http://jfrog.com/shownotes
Slide 3
Slide 4
Slide 5
@jbaruch #TriangleDevOps #LiquidSoftware http://jfrog.com/shownotes
Slide 6
“As every company become a software company, Security vulnerabilities are the new oil spills” @jbaruch #TriangleDevOps #LiquidSoftware http://jfrog.com/shownotes
Slide 7
@jbaruch #TriangleDevOps #LiquidSoftware http://jfrog.com/shownotes
Slide 8
@jbaruch #TriangleDevOps #LiquidSoftware http://jfrog.com/shownotes
Slide 9
Slide 10
Slide 11
Slide 12
@jbaruch #TriangleDevOps #LiquidSoftware http://jfrog.com/shownotes
Slide 13
Slide 14
🎩 @jbaruch #dockercon jfrog.com/shownotes @ErinMeyerINSEAD’s “Culture Map”
Slide 15
shownotes http://jfrog.com/shownotes Slides Video Links Comments, Ratings Raffle @jbaruch #TriangleDevOps #LiquidSoftware http://jfrog.com/shownotes
Slide 16
So, you wanted to update faster… @jbaruch #TriangleDevOps #LiquidSoftware http://jfrog.com/shownotes
Slide 17
@jbaruch #TriangleDevOps #LiquidSoftware http://jfrog.com/shownotes
Slide 18
@jbaruch #TriangleDevOps #LiquidSoftware http://jfrog.com/shownotes
Slide 19
Slide 20
Slide 21
Update available Yes No Can we verify the update? No Yes Yes How about no Do we trust the update? Time consuming verification Let’s update! Yes Are there any high risks? No Do we want it? No
Slide 22
Slide 23
Features that we want @jbaruch #TriangleDevOps Acceptance tests costs #LiquidSoftware http://jfrog.com/shownotes
Slide 24
Slide 25
Your browser Twitter in your browser Twitter on your smartphone Your smartphone OS?! Update available Yes Are there any high risks? No Let’s update! Do we want it? No one asked you (auto update)
Slide 26
What can possibly go wrong?
Slide 27
@jbaruch #TriangleDevOps #LiquidSoftware http://jfrog.com/shownotes
Slide 28
Slide 29
Continuous updates pattern: Local rollback @jbaruch #TriangleDevOps Problem: update went catastrophically wrong and an over the-air patch can’t reach the device Solution: Have a previous version saved on the device prior to update. Rollback in case problem occurred #LiquidSoftware http://jfrog.com/shownotes
Slide 30
Slide 31
Slide 32
Slide 33
Continuous updates pattern: OTA software updates @jbaruch #TriangleDevOps Problem: physical recalls are costly. Extremely costly. Solution: Implement over the air software updates, preferably, continuous updates. #LiquidSoftware http://jfrog.com/shownotes
Slide 34
continuous OTA updates are like normal OTA updates, but better @jbaruch #TriangleDevOps #LiquidSoftware http://jfrog.com/shownotes
Slide 35
Slide 36
Slide 37
Continuous updates pattern: continuous updates @jbaruch #TriangleDevOps Problem: In batch updates important features wait for non-important features. Solution: Implement continuous updates. #LiquidSoftware http://jfrog.com/shownotes
Slide 38
@jbaruch #TriangleDevOps #LiquidSoftware http://jfrog.com/shownotes
Slide 39
Slide 40
Nub’s horror New feature update Uses templating with $ symbol Apple’s staging servers return prices without $ symbol Some Apple’s production servers return prices with $ symbol As a result, some users suffer crashes It took time to understand what went wrong It took time to get the fix through Apple review @jbaruch #LiquidSoftware #TriangleDevOps http://jfrog.com/shownotes
Slide 41
Continuous updates pattern: Canary releases @jbaruch #TriangleDevOps Problem: Releasing a bug affects ALL the users. Solution: Release to a small number of users first and observe. If a problem occurs, stop the release, revert or update the affected users. #LiquidSoftware http://jfrog.com/shownotes
Slide 42
Continuous updates pattern: observability @jbaruch #TriangleDevOps Problem: Some problems are hard to trace relying on user feedback only Solution: Implement tracing, monitoring and logging #LiquidSoftware http://jfrog.com/shownotes
Slide 43
Continuous updates pattern: Rollbacks @jbaruch #TriangleDevOps Problem: Fixes might take time, users suffer in a meanwhile Solution: Implement rollback, the ability to deploy a previous version without delay #LiquidSoftware http://jfrog.com/shownotes
Slide 44
Continuous updates pattern: feature flags @jbaruch #TriangleDevOps Problem: Rollbacks are not always supported by the deployment target platform Solution: Embed 2 versions of the features in the app itself and trigger them with API calls #LiquidSoftware http://jfrog.com/shownotes
Slide 45
Slide 46
You thought your problems are hard? Things under your control Server-side Updates IoT (Mobile, Automotive, Edge) Updates ✓ ✓ ✓ ✓ ✕ ✕ ✕ ✕ The availability of the target The state of the target The version on the target The access to the target @jbaruch #TriangleDevOps #LiquidSoftware http://jfrog.com/shownotes
Slide 47
Slide 48
KNIGHT-MARE @jbaruch #TriangleDevOps New system reused old APIs 1 out of 8 servers was not updated New clients sent requests to machine contained old code No monitoring, no alerting, no debugging Engineers undeployed working code from updated servers, increasing the load on the not-updated server #LiquidSoftware http://jfrog.com/shownotes
Slide 49
Continuous updates pattern: Automated deployment @jbaruch #TriangleDevOps Problem: People suck at repetitive tasks. Solution: Automate everything. #LiquidSoftware http://jfrog.com/shownotes
Slide 50
Continuous updates pattern: frequent updates @jbaruch #TriangleDevOps Problem: Seldom deployments generate anxiety and stress, leading to errors. Solution: Update frequently to develop skill and habit. #LiquidSoftware http://jfrog.com/shownotes
Slide 51
Continuous updates pattern: state awareness @jbaruch #TriangleDevOps Problem: Target state can affect the update process and the behavior of the system after the update. Solution: Know and consider target state when updating. Reverting might require revering the state. #LiquidSoftware http://jfrog.com/shownotes
Slide 52
@jbaruch #TriangleDevOps #LiquidSoftware http://jfrog.com/shownotes
Slide 53
Slide 54
Slide 55
Real life pattern: be kind @jbaruch #TriangleDevOps Problem: You shame someone publicly, week later shit happens to you. Solution: Don’t be a shmuck. #LiquidSoftware http://jfrog.com/shownotes
Slide 56
Cloud-dark @jbaruch #TriangleDevOps Deployment of a single misconfigured rule Included regex to spike CPU to 100% “Affected region: Earth” #LiquidSoftware http://jfrog.com/shownotes
Slide 57
Continuous updates pattern: Canary releases @jbaruch #TriangleDevOps Problem: Releasing a bug affects ALL the users. Solution: Release to a small number of users first and observe. If a problem occurs, stop the release, revert or update the affected users. #LiquidSoftware http://jfrog.com/shownotes
Slide 58
Slide 59
Continuous updates pattern: zero downtime updates @jbaruch #TriangleDevOps Problem: You will probably loose all your users if you shut down for 5 weeks (and counting) to perform an update. Solution: Perform zerodowntime OTA small and fequent continuous updates. #LiquidSoftware http://jfrog.com/shownotes
Slide 60
Continuous updates @jbaruch #TriangleDevOps Frequent Automatic Tested Canary State-aware Observability Local Rollbacks #LiquidSoftware http://jfrog.com/shownotes
Slide 61
Update available Yes Do we trust the update? Yes Do we want it? Are there any high risks? Sure, why not? (auto update) Yes Let’s update! No
Slide 62
” Our goal is to transition from bulk and rare software updates to extremely tiny and extremely frequent software updates; so tiny and so frequent that they provide an illusion of software flowing from development to the update target. We call it the Liquid Software vision. @jbaruch #TriangleDevOps #LiquidSoftware http://jfrog.com/shownotes
Slide 63
@jbaruch #TriangleDevOps #LiquidSoftware http://jfrog.com/shownotes
Slide 64
Corner cases? @jbaruch #TriangleDevOps #LiquidSoftware http://jfrog.com/shownotes
Slide 65
Q&A and twitter ads @jbaruch #TriangleDevOps #LiquidSoftware http://jfrog.com/shownotes
