#DataDrivenDevOps
A presentation at DevOpsDays Baltimore 2019 in in Baltimore, MD, USA by Baruch Sadogursky
#DataDrivenDevOps ”Without data you are just another person with an opinion” W. Edwards Deming @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
shownotes http://jfrog.com/shownotes Slides Video Links Comments, Ratings Raffle @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
Cobbler’s children have no shoes
Poll time! How many of you are software engineers? How many of you are optimists? How many of you are self-confident in their work? @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
Dunning-Kruger Effect a.k.a. ”optimism” “ People suffer from illusory superiority, mistakenly assessing their cognitive ability as greater than it is. Wikipedia @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
Second-system effect a.k.a. “self-confidence” “ The tendency of small, elegant, and successful systems, to be succeeded by overengineered, bloated systems, due to inflated expectations and overconfidence. The Mythical Man-Month @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
The consequences @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
disillusion Software I like @jbaruch @LeonStigter #DevOpsDaysBmore Software I know really well http://jfrog.com/shownotes
overconfidence “ Every software is 80% ready 80% of the time Every developer, ever @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
Hard to give up @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
@jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
How departments measure Support Marketing Sales HR Finance @jbaruch @LeonStigter Engineering Anecdotes from daily standup #DevOpsDaysBmore http://jfrog.com/shownotes
@jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
@jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
The speakers @LeonStigter @jbaruch @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
Engineering metrics in pre-devops era Ops Dev • Velocity • Architecture • Compliance @jbaruch @LeonStigter • SLA • Cost env setup • Average customer cost • Tools stability #DevOpsDaysBmore • • • • • QA Incidents Defects MTTR Code coverage Test suite stability http://jfrog.com/shownotes
DEVOPS! ? @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
Velocity to agile is like ? To devops… ? @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
Velocity to agile is like profit To devops… Profit @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
Very much alike! Metric Profit Velocity @jbaruch Easily understandable Unity actionable ✓ ✓ ✓ ✓ ✗ ✗ @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
Very much alike! Metric Profit Velocity @jbaruch Easily understandable Unity actionable ✓ ✓ ✓ ✓ ✗ ✗ @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
Very much alike! Metric Profit Velocity @jbaruch Easily understandable Unity actionable ✓ ✓ ✓ ✓ ✗ ✗ @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
Poll time! Who knows what velocity is? Who knows what burndown chart is? Who has a burndown chart? Who looks at the burndown chart? Who trusts the burndown chart? Who knows what to do if it doesn’t look right?
What’s the problem? Velocity @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
True for both! Metric Profit Velocity @jbaruch Easily understandable Unity actionable ✓ ✓ ✓ ✓ ✗ ✗ @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
Too coarse! Profit @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
Too coarse! DevOps @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
This is devops* Ops Deep specialization Dev *Unless you’re netfliX **You’re not @jbaruch @LeonStigter Common goals, tools, culture QA #DevOpsDaysBmore http://jfrog.com/shownotes
This is devops Ops Dev @jbaruch @LeonStigter QA #DevOpsDaysBmore http://jfrog.com/shownotes
Same for metrics! Ops Specialized metrics Dev @jbaruch @LeonStigter profit QA #DevOpsDaysBmore http://jfrog.com/shownotes
Same for metrics! Ops Dev @jbaruch @LeonStigter QA #DevOpsDaysBmore http://jfrog.com/shownotes
How do metrics collaborate? influenced affected dev ops qa dev ops qa • Time to build • False reopen count • Release pipelines • QA coverage stability • Artifact replication topology • Avg cost of customer • Smoke tests quality Test suite stability • Time to create new Incoming defect QA lab velocity @jbaruch• @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes Test execution time • •
Cooking with metrics @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
Metrics categorization @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
Why? @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
ROI – is this activity is the right thing to do?
Continuous improvement
trust
sampling @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
Complete sample
@jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
Representative sample
How effective are we? Samples are good enough for that As long as they are representative Need to be collected over time @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
Predicting the future
Meet scrum team “avengers”
anecdotes from the daily scrum stand-ups “ We never get enough testing environments from Ops, QA is suffering. Scrum team “Avengers” @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
Let’s gather some data! Environment Utilization 10 8 6 4 2 0 week 1 week 2 week 3 week 4 week 5 week 6 week 7 Utilized @jbaruch @LeonStigter week 8 week 9 week 10 week 11 week 12 week 13 week 14 Available #DevOpsDaysBmore http://jfrog.com/shownotes
That doesn’t look good! But why? Days the environment stays checked out 25 25 20 18 15 13 13 John Scott 14 15 16 10 10 7 6 5 1 2 3 0 Raj Jane Christina Helen Lee Baruch Nataly Ruppert Josh Patrick Michael Days checked out @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
Environment Utilization 16 14 12 Continuous improvement 10 8 6 4 2 0 week 1 week 3 week 5 week 7 week 9 week 11 week 13 week 15 week 17 week 19 week 21 week 23 Utilized @jbaruch @LeonStigter #DevOpsDaysBmore Available http://jfrog.com/shownotes
anecdotes from the daily scrum stand-ups “ Tests are failing, because the test suite is not stable, QA makes our builds fail without reason. Scrum team “Avengers” @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
Let’s gather some data!
Return on investment @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
anecdotes from the daily scrum stand-ups “ Those developers want to deploy new GC, its tons of work, without any profit. Scrum team “Avengers” @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
Let’s gather some data! @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
0:00 0:40 1:20 2:00 2:40 3:20 4:00 4:40 5:20 6:00 6:40 7:20 8:00 8:40 9:20 10:00 10:40 11:20 12:00 12:40 13:20 14:00 14:40 15:20 16:00 16:40 17:20 18:00 18:40 19:20 20:00 20:40 21:20 22:00 22:40 23:20 GC time Trust and predictability GC time @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
Dos and Don’ts
Measuring the right thing
LoC LOL, anyone? @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
Avoid data overload
Use reliable data
Metrics generate incentives, be aware! @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
Tweak as you go
Elevate! Integrated Dashboards Product Specific Dashboards Raw Data @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
@jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
How departments measure Support Marketing Sales HR Finance @jbaruch @LeonStigter Engineering Anecdotes from daily standup #DevOpsDaysBmore http://jfrog.com/shownotes
Step into data-driven life Removes blame game Builds accountability and trust Creates common base for discussion Enables DevOps @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes
Q&A and twitter ads @jbaruch @LeonStigter #DevOpsDaysBmore #DataDrivenDevOps http://jfrog.com/shownotes
“Without data, you’re just another person with opinions”.
In this talk, we’ll talk about data-driven DevOps and how the cross-cutting metrics from dev, QA, and ops can be integrated to provide you and the teams you support with an insight into the status of your engineering organization. As the DevOps Evangelists of your organization, you can help your teams to adopt data-driven decision making whereas it becomes more important due to cross-pillar influence and collaborated need for success. The practical aspect will cover dos and don’ts and examples of metrics that you can implement in to help your teams today.
