#DataDrivenDevOps
A presentation at GOTO Chicago 2019 in in Chicago, IL, USA by Baruch Sadogursky
#DataDrivenDevOps ”Without data you are just another person with an opinion” W. Edwards Deming
shownotes http://jfrog.com/shownotes Slides Video Links Comments, Ratings Raffle @jbaruch #DataDrivenDevOps #GOTOchgo http://jfrog.com/shownotes
Cobbler’s children have no shoes
Poll time! How many of you are software engineers? How many of you are optimists? How many of you are self-confident in their work? @jbaruch #DataDrivenDevOps #GOTOchgo http://jfrog.com/shownotes
Dunning-Kruger Effect a.k.a. ”optimism” “ People suffer from illusory superiority, mistakenly assessing their cognitive ability as greater than it is. Wikipedia @jbaruch #DataDrivenDevOps #GOTOchgo http://jfrog.com/shownotes
Second-system effect a.k.a. “self-confidence” “ The tendency of small, elegant, and successful systems, to be succeeded by overengineered, bloated systems, due to inflated expectations and overconfidence. The Mythical Man-Month @jbaruch #DataDrivenDevOps #GOTOchgo http://jfrog.com/shownotes
The consequences @jbaruch #DataDrivenDevOps #GOTOchgo http://jfrog.com/shownotes
disillusion Software I know really well Software I like @jbaruch #DataDrivenDevOps #GOTOchgo http://jfrog.com/shownotes
overconfidence “ Every software is 80% ready 80% of the time Every developer, ever @jbaruch #DataDrivenDevOps #GOTOchgo http://jfrog.com/shownotes
Hard to give up @jbaruch #DataDrivenDevOps #GOTOchgo http://jfrog.com/shownotes
@jbaruch #DataDrivenDevOps #GOTOchgo http://jfrog.com/shownotes
How departments measure Support Marketing Sales HR Finance @jbaruch #DataDrivenDevOps Engineering Anecdotes from daily standup #GOTOchgo http://jfrog.com/shownotes
How departments measure Support Marketing Sales HR Finance @jbaruch #DataDrivenDevOps Engineering Anecdotes from daily standup #GOTOchgo http://jfrog.com/shownotes
@jbaruch #DataDrivenDevOps #GOTOchgo http://jfrog.com/shownotes
@jbaruch #DataDrivenDevOps #GOTOchgo http://jfrog.com/shownotes
Erin Meyer, The Culture Map
Engineering metrics in pre-devops era Ops Dev • Velocity • Architecture • Compliance @jbaruch • SLA • Cost env setup • Average customer cost • Tools stability #DataDrivenDevOps #GOTOchgo • • • • • QA Incidents Defects MTTR Code coverage Test suite stability http://jfrog.com/shownotes
DEVOPS! ? @jbaruch #DataDrivenDevOps #GOTOchgo http://jfrog.com/shownotes
Velocity to agile is like ? To devops… ? @jbaruch #DataDrivenDevOps #GOTOchgo http://jfrog.com/shownotes
Velocity to agile is like profit To devops… Profit @jbaruch #DataDrivenDevOps #GOTOchgo http://jfrog.com/shownotes
Very much alike! Metric Profit Velocity @jbaruch Easily understandable Unity actionable ✓ ✓ ✓ ✓ ✗ ✗ #DataDrivenDevOps #GOTOchgo http://jfrog.com/shownotes
Poll time! Who knows what velocity is? Who knows what burndown chart is? Who has a burndown chart? Who looks at the burndown chart? Who trusts the burndown chart? Who knows what to do if it doesn’t look right?
What’s the problem? Velocity @jbaruch #DataDrivenDevOps #GOTOchgo http://jfrog.com/shownotes
True for both! Metric Profit Velocity @jbaruch Easily understandable Unity actionable ✓ ✓ ✓ ✓ ✗ ✗ #DataDrivenDevOps #GOTOchgo http://jfrog.com/shownotes
Too coarse! Profit @jbaruch #DataDrivenDevOps #GOTOchgo http://jfrog.com/shownotes
Too coarse! DevOps @jbaruch #DataDrivenDevOps #GOTOchgo http://jfrog.com/shownotes
This is devops* Ops Deep specialization Dev *Unless you’re netfliX **You’re not @jbaruch #DataDrivenDevOps Common goals, tools, culture QA #GOTOchgo http://jfrog.com/shownotes
This is devops Ops Dev @jbaruch #DataDrivenDevOps QA #GOTOchgo http://jfrog.com/shownotes
Same for metrics! Ops Specialized metrics Dev @jbaruch #DataDrivenDevOps profit QA #GOTOchgo http://jfrog.com/shownotes
Same for metrics! Ops Dev @jbaruch #DataDrivenDevOps QA #GOTOchgo http://jfrog.com/shownotes
How do metrics collaborate? influenced affected dev dev ops qa • Time to build • False reopen count • Release pipelines • QA coverage stability • Artifact replication topology ops • Avg cost of customer qa • • Test suite stability • Time to create new Incoming defect QA lab velocity #DataDrivenDevOps Test execution time #GOTOchgo http://jfrog.com/shownotes @jbaruch• • Smoke tests quality
Cooking with metrics @jbaruch #DataDrivenDevOps #GOTOchgo http://jfrog.com/shownotes
Metrics categorization @jbaruch #DataDrivenDevOps #GOTOchgo http://jfrog.com/shownotes
Why? @jbaruch #DataDrivenDevOps #GOTOchgo http://jfrog.com/shownotes
ROI – is this activity is the right thing to do?
Continuous improvement
trust
sampling @jbaruch #DataDrivenDevOps #GOTOchgo http://jfrog.com/shownotes
Complete sample
@jbaruch #DataDrivenDevOps #GOTOchgo http://jfrog.com/shownotes
Representative sample
How effective are we? Samples are good enough for that As long as they are representative Need to be collected over time @jbaruch #DataDrivenDevOps #GOTOchgo http://jfrog.com/shownotes
Predicting the future
Meet scrum team “avengers”
anecdotes from the daily scrum stand-ups “ We never get enough testing environments from Ops, QA is suffering. Scrum team “Avengers” @jbaruch #DataDrivenDevOps #GOTOchgo http://jfrog.com/shownotes
Let’s gather some data! Environment Utilization 10 8 6 4 2 0 week 1 week 2 week 3 week 4 week 5 week 6 week 7 Utilized @jbaruch #DataDrivenDevOps week 8 week 9 week 10 week 11 week 12 week 13 week 14 Available #GOTOchgo http://jfrog.com/shownotes
That doesn’t look good! But why? Days the environment stays checked out 25 25 20 18 15 13 13 John Scott 14 15 16 10 10 6 5 1 2 7 3 0 Raj Jane Christina Helen Lee Baruch Nataly Ruppert Josh Patrick Michael Days checked out @jbaruch #DataDrivenDevOps #GOTOchgo http://jfrog.com/shownotes
Environment Utilization 16 14 12 Continuous improvement 10 8 6 4 2 0 week 1 week 3 week 5 week 7 week 9 week 11 week 13 week 15 week 17 week 19 week 21 week 23 Utilized @jbaruch #DataDrivenDevOps #GOTOchgo Available http://jfrog.com/shownotes
anecdotes from the daily scrum stand-ups “ Tests are failing, because the test suite is not stable, QA makes our builds fail without reason. Scrum team “Avengers” @jbaruch #DataDrivenDevOps #GOTOchgo http://jfrog.com/shownotes
Let’s gather some data!
Return on investment @jbaruch #DataDrivenDevOps #GOTOchgo http://jfrog.com/shownotes
anecdotes from the daily scrum stand-ups “ Those developers want to deploy new GC, its tons of work, without any profit. Scrum team “Avengers” @jbaruch #DataDrivenDevOps #GOTOchgo http://jfrog.com/shownotes
Let’s gather some data! @jbaruch #DataDrivenDevOps #GOTOchgo http://jfrog.com/shownotes
0:00 0:40 1:20 2:00 2:40 3:20 4:00 4:40 5:20 6:00 6:40 7:20 8:00 8:40 9:20 10:00 10:40 11:20 12:00 12:40 13:20 14:00 14:40 15:20 16:00 16:40 17:20 18:00 18:40 19:20 20:00 20:40 21:20 22:00 22:40 23:20 GC time Trust and predictability GC time @jbaruch #DataDrivenDevOps #GOTOchgo http://jfrog.com/shownotes
Dos and Don’ts
Measuring the right thing
LoC LOL, anyone? @jbaruch #DataDrivenDevOps #GOTOchgo http://jfrog.com/shownotes
Avoid data overload
Use reliable data
”medium” In brazil ”medium” In russia Common vocabulary is important
Metrics generate incentives, be aware! @jbaruch #DataDrivenDevOps #GOTOchgo http://jfrog.com/shownotes
Tweak as you go
Elevate! Integrated Dashboards Product Specific Dashboards Raw Data @jbaruch #DataDrivenDevOps #GOTOchgo http://jfrog.com/shownotes
@jbaruch #DataDrivenDevOps #GOTOchgo http://jfrog.com/shownotes
@jbaruch #DataDrivenDevOps #GOTOchgo http://jfrog.com/shownotes
How departments measure Support Marketing Sales HR Finance @jbaruch #DataDrivenDevOps Engineering Anecdotes from daily standup #GOTOchgo http://jfrog.com/shownotes
Step into data-driven life Removes blame game Builds accountability and trust Creates common base for discussion Enables DevOps @jbaruch #DataDrivenDevOps #GOTOchgo http://jfrog.com/shownotes
Q&A and twitter ads @jbaruch #GOTOchgo #DataDrivenDevOps http://jfrog.com/shownotes
https://jbaru.ch/GOTOchgo 20% off!
“Without data, you’re just another person with opinions”.
In this talk, we’ll talk about data-driven DevOps and how the cross-cutting metrics from dev, QA, and ops can be integrated to provide you and the teams you support with an insight into the status of your engineering organization. As the DevOps Evangelists of your organization, you can help your teams to adopt data-driven decision making whereas it becomes more important due to cross-pillar influence and collaborated need for success. The practical aspect will cover dos and don’ts and examples of metrics that you can implement in to help your teams today.
