DevOps Patterns & Antipatterns for Continuous Software Updates “What can possibly go wrong?!”
Slide 1
Slide 2
Why software updates?
Slide 3
@jbaruch #LiquidSoftware @CloudNativeFdn http://jfrog.com/shownotes
Slide 4
Slide 5
Slide 6
@jbaruch #LiquidSoftware @CloudNativeFdn http://jfrog.com/shownotes
Slide 7
“As every company become a software company, Security vulnerabilities are the new oil spills” @jbaruch #LiquidSoftware @CloudNativeFdn http://jfrog.com/shownotes
Slide 8
Slide 9
Identify @jbaruch #LiquidSoftware Fix @CloudNativeFdn Deploy http://jfrog.com/shownotes
Slide 10
Identify Fix Deploy Immediately OS upgrade years
Slide 11
Identify Fix Deploy 2 months Struts upgrade 2 months
Slide 12
@jbaruch #LiquidSoftware @CloudNativeFdn http://jfrog.com/shownotes
Slide 13
@jbaruch #LiquidSoftware @CloudNativeFdn http://jfrog.com/shownotes
Slide 14
@jbaruch #LiquidSoftware Identify As fast as possible Fix As fast as possible Deploy As fast as possible @CloudNativeFdn http://jfrog.com/shownotes
Slide 15
Slide 16
Slide 17
Slide 18
@jbaruch #LiquidSoftware @CloudNativeFdn http://jfrog.com/shownotes
Slide 19
This is not a new idea! @jbaruch #LiquidSoftware XP: short feedback Scrum: reducing cycle time to absolute minimum TPS: Decide as late as possible and Deliver as fast as possible Kanban: Incremental change @CloudNativeFdn http://jfrog.com/shownotes
Slide 20
Slide 21
shownotes http://jfrog.com/shownotes Slides Video Links Comments, Ratings Raffle @jbaruch #LiquidSoftware @CloudNativeFdn http://jfrog.com/shownotes
Slide 22
Slide 23
@jbaruch #LiquidSoftware @CloudNativeFdn http://jfrog.com/shownotes
Slide 24
@jbaruch #LiquidSoftware @CloudNativeFdn http://jfrog.com/shownotes
Slide 25
Slide 26
Slide 27
@jbaruch #LiquidSoftware @CloudNativeFdn http://jfrog.com/shownotes
Slide 28
Update available Yes No Do we trust the update? Yes How about no Let’s update! Yes Are there any high risks? No Do we want it? No
Slide 29
@jbaruch #LiquidSoftware @CloudNativeFdn http://jfrog.com/shownotes
Slide 30
number of artifacts as a symptom of complexity Today IoT Serverless Docker Microservices Infrastructure as Code Continuous Delivery Continuous Integration Agile 2000 @jbaruch @jfrog #LiquidSoftware www.liquidsoftware.com
Slide 31
The problem is not the code, it’s the data. Big data. @jbaruch #LiquidSoftware @CloudNativeFdn http://jfrog.com/shownotes
Slide 32
#emptyenvelopefromchina @jbaruch #LiquidSoftware @CloudNativeFdn http://jfrog.com/shownotes
Slide 33
Update available Yes No Can we verify the update? No Yes Yes How about no Do we trust the update? Time consuming verification Let’s update! Yes Are there any high risks? No Do we want it? No
Slide 34
Slide 35
Features that we want @jbaruch #LiquidSoftware Acceptance tests costs @CloudNativeFdn http://jfrog.com/shownotes
Slide 36
Slide 37
Your browser Twitter in your browser Twitter on your smartphone Your smartphone OS?! Update available Yes Are there any high risks? No Let’s update! Do we want it? No one asked you (auto update)
Slide 38
What can possibly go wrong?
Slide 39
@jbaruch #LiquidSoftware @CloudNativeFdn http://jfrog.com/shownotes
Slide 40
Slide 41
Continuous updates pattern: Local rollback @jbaruch #LiquidSoftware Problem: update went catastrophically wrong and an over the-air patch can’t reach the device Solution: Have a previous version saved on the device prior to update. Rollback in case problem occurred @CloudNativeFdn http://jfrog.com/shownotes
Slide 42
@jbaruch #LiquidSoftware @CloudNativeFdn http://jfrog.com/shownotes
Slide 43
Slide 44
Slide 45
Slide 46
Continuous updates pattern: OTA software updates @jbaruch #LiquidSoftware Problem: physical recalls are costly. Extremely costly. Also, you can’t force an upgrade. Solution: Implement over the air software updates, preferably, continuous updates. @CloudNativeFdn http://jfrog.com/shownotes
Slide 47
continuous OTA updates are like normal OTA updates, but better @jbaruch #LiquidSoftware @CloudNativeFdn http://jfrog.com/shownotes
Slide 48
Slide 49
Slide 50
Slide 51
Continuous updates pattern: continuous updates @jbaruch #LiquidSoftware Problem: In batch updates important features wait for non-important features. Solution: Implement continuous updates. @CloudNativeFdn http://jfrog.com/shownotes
Slide 52
Slide 53
You thought your problems are hard? Things under your control Server-side Updates IoT (Mobile, Automotive, Edge) Updates ✓ ✓ ✓ ✓ ✕ ✕ ✕ ✕ The availability of the target The state of the target The version on the target The access to the target @jbaruch #LiquidSoftware @CloudNativeFdn http://jfrog.com/shownotes
Slide 54
Slide 55
KNIGHT-MARE @jbaruch #LiquidSoftware New system reused old APIs 1 out of 8 servers was not updated New clients sent requests to machine contained old code Engineers undeployed working code from updated servers, increasing the load on the not-updated server No monitoring, no alerting, no debugging @CloudNativeFdn http://jfrog.com/shownotes
Slide 56
Continuous updates pattern: Automated deployment @jbaruch #LiquidSoftware Problem: People suck at repetitive tasks. Solution: Automate everything. @CloudNativeFdn http://jfrog.com/shownotes
Slide 57
Continuous updates pattern: frequent updates @jbaruch #LiquidSoftware Problem: Seldom deployments generate anxiety and stress, leading to errors. Solution: Update frequently to develop skill and habit. @CloudNativeFdn http://jfrog.com/shownotes
Slide 58
Continuous updates pattern: state awareness @jbaruch #LiquidSoftware Problem: Target state can affect the update process and the behavior of the system after the update. Solution: Know and consider target state when updating. Reverting might require revering the state. @CloudNativeFdn http://jfrog.com/shownotes
Slide 59
Slide 60
Cloud-dark @jbaruch #LiquidSoftware New rules are deployed frequently to battle attacks Deployment of a single misconfigured rule Included regex to spike CPU to 100% “Affected region: Earth” @CloudNativeFdn http://jfrog.com/shownotes
Slide 61
Continuous updates pattern: Progressive Delivery @jbaruch #LiquidSoftware Problem: Releasing a bug affects ALL the users. Solution: Release to a small number of users first effectively reducing the blast radius and observe. If a problem occurs, stop the release, revert or update the affected users. @CloudNativeFdn http://jfrog.com/shownotes
Slide 62
@jbaruch #LiquidSoftware @CloudNativeFdn http://jfrog.com/shownotes
Slide 63
Continuous updates pattern: observability @jbaruch #LiquidSoftware Problem: Some problems are hard to trace relying on user feedback only Solution: Implement tracing, monitoring and logging @CloudNativeFdn http://jfrog.com/shownotes
Slide 64
Continuous updates pattern: Rollbacks @jbaruch #LiquidSoftware Problem: Fixes might take time, users suffer in a meanwhile Solution: Implement rollback, the ability to deploy a previous version without delay @CloudNativeFdn http://jfrog.com/shownotes
Slide 65
Continuous updates pattern: feature flags @jbaruch #LiquidSoftware Problem: Rollbacks are not always supported by the deployment target platform Solution: Embed 2 versions of the features in the app itself and trigger them with API calls @CloudNativeFdn http://jfrog.com/shownotes
Slide 66
Slide 67
Continuous updates pattern: zero downtime updates @jbaruch #LiquidSoftware Problem: You will probably loose all your users if you shut down for 5 weeks to perform an update. Solution: Perform zerodowntime OTA small and frequent continuous updates. @CloudNativeFdn http://jfrog.com/shownotes
Slide 68
Continuous updates @jbaruch #LiquidSoftware Frequent Automatic Tested Progressively delivered State-aware Observability *Local Rollbacks @CloudNativeFdn http://jfrog.com/shownotes
Slide 69
Update available Yes Do we trust the update? Yes Do we want it? Are there any high risks? Sure, why not? (auto update) Yes Let’s update! No
Slide 70
” Our goal is to transition from bulk and rare software updates to extremely tiny and extremely frequent software updates; so tiny and so frequent that they provide an illusion of software flowing from development to the update target. We call it the Liquid Software vision. @jbaruch #LiquidSoftware @CloudNativeFdn http://jfrog.com/shownotes
Slide 71
@jbaruch #LiquidSoftware @CloudNativeFdn http://jfrog.com/shownotes
Slide 72
Corner cases? @jbaruch #LiquidSoftware @CloudNativeFdn http://jfrog.com/shownotes
Slide 73
Slide 74
Slide 75
@jbaruch #LiquidSoftware @CloudNativeFdn http://jfrog.com/shownotes
Slide 76
Q&A and twitter ads @jbaruch #LiquidSoftware @CloudNativeFdn https://liquidsoftware.com https://jfrog.com/shownotes
