DevOps Patterns & Antipatterns for Continuous Software Updates
A presentation at DevOps Toronto Meetup July 2020 in in Toronto, ON, Canada by Baruch Sadogursky
DevOps Patterns & Antipatterns for Continuous Software Updates “What can possibly go wrong?!”
Why software updates?
@jbaruch #LiquidSoftware #DevOpsTO http://jfrog.com/shownotes
“As every company become a software company, Security vulnerabilities are the new oil spills” @jbaruch #LiquidSoftware #DevOpsTO http://jfrog.com/shownotes
@jbaruch #LiquidSoftware #DevOpsTO http://jfrog.com/shownotes
This is not a new idea! @jbaruch #LiquidSoftware XP: short feedback Scrum: reducing cycle time to absolute minimum TPS: Decide as late as possible and Deliver as fast as possible Kanban: Incremental change #DevOpsTO http://jfrog.com/shownotes
shownotes http://jfrog.com/shownotes Slides Video Links Comments, Ratings Raffle @jbaruch #LiquidSoftware #DevOpsTO http://jfrog.com/shownotes
@jbaruch #LiquidSoftware #DevOpsTO http://jfrog.com/shownotes
@jbaruch #LiquidSoftware #DevOpsTO http://jfrog.com/shownotes
@jbaruch #LiquidSoftware #DevOpsTO http://jfrog.com/shownotes
Update available Yes No Do we trust the update? Yes How about no Let’s update! Yes Are there any high risks? No Do we want it? No
@jbaruch #LiquidSoftware #DevOpsTO http://jfrog.com/shownotes
The problem is not the code, it’s the data. Big data. @jbaruch #LiquidSoftware #DevOpsTO http://jfrog.com/shownotes
Update available Yes No Can we verify the update? No Yes Yes How about no Do we trust the update? Time consuming verification Let’s update! Yes Are there any high risks? No Do we want it? No
Features that we want @jbaruch #LiquidSoftware Acceptance tests costs #DevOpsTO http://jfrog.com/shownotes
Bigger and rarer updates Less trust in updates Pack more features
What can possibly go wrong?
@jbaruch #LiquidSoftware #DevOpsTO http://jfrog.com/shownotes
Continuous updates pattern: Local rollback @jbaruch #LiquidSoftware Problem: update went catastrophically wrong and an over the-air patch can’t reach the device Solution: Have a previous version saved on the device prior to update. Rollback in case problem occurred #DevOpsTO http://jfrog.com/shownotes
@jbaruch #LiquidSoftware #DevOpsTO http://jfrog.com/shownotes
@jbaruch #LiquidSoftware #DevOpsTO http://jfrog.com/shownotes
@jbaruch #LiquidSoftware #DevOpsTO http://jfrog.com/shownotes
Continuous updates pattern: OTA software updates @jbaruch #LiquidSoftware Problem: physical recalls are costly. Extremely costly. Also, you can’t force an upgrade. Solution: Implement over the air software updates, preferably, continuous updates. #DevOpsTO http://jfrog.com/shownotes
continuous OTA updates are like normal OTA updates, but better @jbaruch #LiquidSoftware #DevOpsTO http://jfrog.com/shownotes
Continuous updates pattern: continuous updates @jbaruch #LiquidSoftware Problem: In batch updates important features wait for non-important features. Solution: Implement continuous updates. #DevOpsTO http://jfrog.com/shownotes
KNIGHT-MARE @jbaruch #LiquidSoftware New system reused old APIs 1 out of 8 servers was not updated New clients sent requests to machine contained old code Engineers undeployed working code from updated servers, increasing the load on the not-updated server No monitoring, no alerting, no debugging #DevOpsTO http://jfrog.com/shownotes
Continuous updates pattern: Automated deployment @jbaruch #LiquidSoftware Problem: People suck at repetitive tasks. Solution: Automate everything. #DevOpsTO http://jfrog.com/shownotes
Continuous updates pattern: frequent updates @jbaruch #LiquidSoftware Problem: Seldom deployments generate anxiety and stress, leading to errors. Solution: Update frequently to develop skill and habit. #DevOpsTO http://jfrog.com/shownotes
Continuous updates pattern: state awareness @jbaruch #LiquidSoftware Problem: Target state can affect the update process and the behavior of the system after the update. Solution: Know and consider target state when updating. Reverting might require revering the state. #DevOpsTO http://jfrog.com/shownotes
Cloud-dark @jbaruch #LiquidSoftware New rules are deployed frequently to battle attacks Deployment of a single misconfigured rule Included regex to spike CPU to 100% “Affected region: Earth” #DevOpsTO http://jfrog.com/shownotes
Continuous updates pattern: Progressive Delivery @jbaruch #LiquidSoftware Problem: Releasing a bug affects ALL the users. Solution: Release to a small number of users first effectively reducing the blast radius and observe. If a problem occurs, stop the release, revert or update the affected users. #DevOpsTO http://jfrog.com/shownotes
@jbaruch #LiquidSoftware #DevOpsTO http://jfrog.com/shownotes
Continuous updates pattern: observability @jbaruch #LiquidSoftware Problem: Some problems are hard to trace relying on user feedback only Solution: Implement tracing, monitoring and logging #DevOpsTO http://jfrog.com/shownotes
Continuous updates pattern: Rollbacks @jbaruch #LiquidSoftware Problem: Fixes might take time, users suffer in a meanwhile Solution: Implement rollback, the ability to deploy a previous version without delay #DevOpsTO http://jfrog.com/shownotes
Continuous updates pattern: feature flags @jbaruch #LiquidSoftware Problem: Rollbacks are not always supported by the deployment target platform Solution: Embed 2 versions of the features in the app itself and trigger them with API calls #DevOpsTO http://jfrog.com/shownotes
Update available Yes Do we trust the update? Yes Do we want it? Are there any high risks? Sure, why not? (auto update) Yes Let’s update! No
” Our goal is to transition from bulk and rare software updates to extremely tiny and extremely frequent software updates; so tiny and so frequent that they provide an illusion of software flowing from development to the update target. We call it the Liquid Software vision. @jbaruch #LiquidSoftware #DevOpsTO http://jfrog.com/shownotes
@jbaruch #LiquidSoftware #DevOpsTO http://jfrog.com/shownotes
Corner cases? @jbaruch #LiquidSoftware #DevOpsTO http://jfrog.com/shownotes
@jbaruch #LiquidSoftware #DevOpsTO http://jfrog.com/shownotes
@jbaruch #LiquidSoftware Q&A and twitter ads #DevOpsTO https://liquidsoftware.com https://jfrog.com/shownotes
So, you want to update the software for your user, be it the nodes in your K8s cluster, a browser on user’s desktop, an app in user’s smartphone or even a user’s car. What can possibly go wrong?
In this talk, we’ll analyze real-world software update fails and how multiple DevOps patterns, that fit a variety of scenarios, could have saved the developers. Manually making sure that everything works before sending an update and expecting the user to do acceptance tests before they update is most definitely not on the list of such patterns.
Join us for some awesome and scary continuous update horror stories and some obvious (and some not so obvious) proven ideas for improvement and best practices you can start following tomorrow.
