#DataDrivenDevOps
A presentation at Cincy Deliver 2019 in in Cincinnati, OH, USA by Baruch Sadogursky
#DataDrivenDevOps ”Without data you are just another person with an opinion” W. Edwards Deming @jbaruch #CincyDeliver #DataDrivenDevOps http://jfrog.com/shownotes
shownotes http://jfrog.com/shownotes Slides Video Links Comments, Ratings Raffle @jbaruch #CincyDeliver #DataDrivenDevOps http://jfrog.com/shownotes
Cobbler’s children have no shoes
Poll time! How many of you are software engineers? How many of you are optimists? How many of you are self-confident in their work? @jbaruch #CincyDeliver #DataDrivenDevOps http://jfrog.com/shownotes
Dunning-Kruger Effect a.k.a. ”optimism” “ People suffer from illusory superiority, mistakenly assessing their cognitive ability as greater than it is. Wikipedia @jbaruch #CincyDeliver #DataDrivenDevOps http://jfrog.com/shownotes
Second-system effect a.k.a. “self-confidence” “ The tendency of small, elegant, and successful systems, to be succeeded by overengineered, bloated systems, due to inflated expectations and overconfidence. The Mythical Man-Month @jbaruch #CincyDeliver #DataDrivenDevOps http://jfrog.com/shownotes
The consequences @jbaruch #CincyDeliver #DataDrivenDevOps http://jfrog.com/shownotes
disillusion Software I like @jbaruch #CincyDeliver #DataDrivenDevOps Software I know really well http://jfrog.com/shownotes
overconfidence “ Every software is 80% ready 80% of the time Every developer, ever @jbaruch #CincyDeliver #DataDrivenDevOps http://jfrog.com/shownotes
Hard to give up @jbaruch #CincyDeliver #DataDrivenDevOps http://jfrog.com/shownotes
@jbaruch #CincyDeliver #DataDrivenDevOps http://jfrog.com/shownotes
How departments measure Support Marketing Sales HR Finance @jbaruch #CincyDeliver Engineering Anecdotes from daily standup #DataDrivenDevOps http://jfrog.com/shownotes
@jbaruch #CincyDeliver #DataDrivenDevOps http://jfrog.com/shownotes
🎩 @jbaruch #dockercon jfrog.com/shownotes @ErinMeyerINSEAD’s “Culture Map”
Engineering metrics in pre-devops era Ops Dev • Velocity • Architecture • Compliance @jbaruch #CincyDeliver • SLA • Cost env setup • Average customer cost • Tools stability #DataDrivenDevOps • • • • • QA Incidents Defects MTTR Code coverage Test suite stability http://jfrog.com/shownotes
DEVOPS! ? @jbaruch #CincyDeliver #DataDrivenDevOps http://jfrog.com/shownotes
Velocity to agile is like ? To devops… ? @jbaruch #CincyDeliver #DataDrivenDevOps http://jfrog.com/shownotes
Velocity to agile is like Flow Efficiency To devops… Flow Efficiency @jbaruch #CincyDeliver #DataDrivenDevOps http://jfrog.com/shownotes
Very much alike! Metric Flow Efficiency Velocity @jbaruch Easily understandable Unity actionable ✓ ✓ ✗ ✓ ✓ ✗ #CincyDeliver #DataDrivenDevOps http://jfrog.com/shownotes
Poll time! Who knows what velocity is? Who knows what burndown chart is? Who has a burndown chart? Who looks at the burndown chart? Who trusts the burndown chart? Who knows what to do if it doesn’t look right?
What’s the problem? Velocity @jbaruch #CincyDeliver #DataDrivenDevOps http://jfrog.com/shownotes
True for both! Metric Flow Efficiency Velocity @jbaruch Easily understandable Unity actionable ✓ ✓ ✗ ✓ ✓ ✗ #CincyDeliver #DataDrivenDevOps http://jfrog.com/shownotes
Too coarse! Flow Efficiency @jbaruch #CincyDeliver #DataDrivenDevOps http://jfrog.com/shownotes
Too coarse! DevOps @jbaruch #CincyDeliver #DataDrivenDevOps http://jfrog.com/shownotes
This is devops* Ops Deep specialization Dev *Unless you’re netfliX **You’re not @jbaruch #CincyDeliver Common goals, tools, culture QA #DataDrivenDevOps http://jfrog.com/shownotes
This is devops Ops Dev @jbaruch #CincyDeliver QA #DataDrivenDevOps http://jfrog.com/shownotes
Same for metrics! Ops Specialized metrics Dev @jbaruch #CincyDeliver Flow Efficiency QA #DataDrivenDevOps http://jfrog.com/shownotes
Same for metrics! Ops Dev @jbaruch #CincyDeliver QA #DataDrivenDevOps http://jfrog.com/shownotes
How do metrics collaborate? influenced affected dev dev qa • Time to build • False reopen count • Release pipelines • QA coverage stability • Artifact replication topology ops • qa • • @jbaruch ops Avg cost of customer • Smoke tests quality Test suite stability • Time to create new Incoming defect QA lab velocity http://jfrog.com/shownotes • #CincyDeliver Test execution#DataDrivenDevOps time
Cooking with metrics @jbaruch #CincyDeliver #DataDrivenDevOps http://jfrog.com/shownotes
Metrics categorization @jbaruch #CincyDeliver #DataDrivenDevOps http://jfrog.com/shownotes
Why? @jbaruch #CincyDeliver #DataDrivenDevOps http://jfrog.com/shownotes
ROI – is this activity is the right thing to do?
Continuous improvement
trust
sampling @jbaruch #CincyDeliver #DataDrivenDevOps http://jfrog.com/shownotes
Complete sample
@jbaruch #CincyDeliver #DataDrivenDevOps http://jfrog.com/shownotes
Representative sample
How effective are we? Samples are good enough for that As long as they are representative Need to be collected over time @jbaruch #CincyDeliver #DataDrivenDevOps http://jfrog.com/shownotes
Predicting the future
Meet scrum team “avengers”
anecdotes from the daily scrum stand-ups “ We never get enough testing environments from Ops, QA is suffering. Scrum team “Avengers” @jbaruch #CincyDeliver #DataDrivenDevOps http://jfrog.com/shownotes
Let’s gather some data! Environment Utilization 10 8 6 4 2 0 week 1 week 2 week 3 week 4 week 5 week 6 week 7 Utilized @jbaruch #CincyDeliver week 8 week 9 week 10 week 11 week 12 week 13 week 14 Available #DataDrivenDevOps http://jfrog.com/shownotes
That doesn’t look good! But why? Days the environment stays checked out 25 25 20 18 15 13 13 John Scott 14 15 16 10 10 7 6 5 1 2 3 0 Raj Jane Christina Helen Lee Baruch Nataly Ruppert Josh Patrick Michael Days checked out @jbaruch #CincyDeliver #DataDrivenDevOps http://jfrog.com/shownotes
Environment Utilization 16 14 12 Continuous improvement 10 8 6 4 2 0 week 1 week 3 week 5 week 7 week 9 week 11 week 13 week 15 week 17 week 19 week 21 week 23 Utilized @jbaruch #CincyDeliver #DataDrivenDevOps Available http://jfrog.com/shownotes
anecdotes from the daily scrum stand-ups “ Tests are failing, because the test suite is not stable, QA makes our builds fail without reason. Scrum team “Avengers” @jbaruch #CincyDeliver #DataDrivenDevOps http://jfrog.com/shownotes
Let’s gather some data!
Return on investment @jbaruch #CincyDeliver #DataDrivenDevOps http://jfrog.com/shownotes
anecdotes from the daily scrum stand-ups “ Those developers want to deploy new GC, its tons of work, without any profit. Scrum team “Avengers” @jbaruch #CincyDeliver #DataDrivenDevOps http://jfrog.com/shownotes
Let’s gather some data! @jbaruch #CincyDeliver #DataDrivenDevOps http://jfrog.com/shownotes
0:00 0:40 1:20 2:00 2:40 3:20 4:00 4:40 5:20 6:00 6:40 7:20 8:00 8:40 9:20 10:00 10:40 11:20 12:00 12:40 13:20 14:00 14:40 15:20 16:00 16:40 17:20 18:00 18:40 19:20 20:00 20:40 21:20 22:00 22:40 23:20 GC time Trust and predictability GC time @jbaruch #CincyDeliver #DataDrivenDevOps http://jfrog.com/shownotes
Dos and Don’ts
Measuring the right thing
LoC LOL, anyone? @jbaruch #CincyDeliver #DataDrivenDevOps http://jfrog.com/shownotes
Avoid data overload
Use reliable data
”medium” In brazil ”medium” In russia Common vocabulary is important
Metrics generate incentives, be aware! @jbaruch #CincyDeliver #DataDrivenDevOps http://jfrog.com/shownotes
Tweak as you go
Elevate! Integrated Dashboards Product Specific Dashboards Raw Data @jbaruch #CincyDeliver #DataDrivenDevOps http://jfrog.com/shownotes
@jbaruch #CincyDeliver #DataDrivenDevOps http://jfrog.com/shownotes
How departments measure Support Marketing Sales HR Finance @jbaruch #CincyDeliver Engineering Anecdotes from daily standup #DataDrivenDevOps http://jfrog.com/shownotes
Step into data-driven life Removes blame game Builds accountability and trust Creates common base for discussion Enables DevOps @jbaruch #CincyDeliver #DataDrivenDevOps http://jfrog.com/shownotes
Q&A and twitter ads @jbaruch #CincyDeliver #DataDrivenDevOps http://jfrog.com/shownotes
“Without data, you’re just another person with opinions”.
In this talk, we’ll talk about data-driven DevOps and how the cross-cutting metrics from dev, QA, and ops can be integrated to provide you and the teams you support with an insight into the status of your engineering organization. As the DevOps Evangelists of your organization, you can help your teams to adopt data-driven decision making whereas it becomes more important due to cross-pillar influence and collaborated need for success. The practical aspect will cover dos and don’ts and examples of metrics that you can implement in to help your teams today.
