#DataDrivenDevOps

#DataDrivenDevOps ”Without data you are just another person with an opinion” W. Edwards Deming @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes

shownotes http://jfrog.com/shownotes Slides Video Links Comments, Ratings Raffle @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes

Cobbler’s children have no shoes

Poll time! How many of you are software engineers? How many of you are optimists? How many of you are self-confident in their work? @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes

Dunning-Kruger Effect a.k.a. ”optimism” “ People suffer from illusory superiority, mistakenly assessing their cognitive ability as greater than it is. Wikipedia @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes

Second-system effect a.k.a. “self-confidence” “ The tendency of small, elegant, and successful systems, to be succeeded by overengineered, bloated systems, due to inflated expectations and overconfidence. The Mythical Man-Month @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes

The consequences @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes

disillusion Software I like @jbaruch @LeonStigter #DevOpsDaysBmore Software I know really well http://jfrog.com/shownotes

overconfidence “ Every software is 80% ready 80% of the time Every developer, ever @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes

Hard to give up @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes

@jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes

How departments measure Support Marketing Sales HR Finance @jbaruch @LeonStigter Engineering Anecdotes from daily standup #DevOpsDaysBmore http://jfrog.com/shownotes

@jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes

The speakers @LeonStigter @jbaruch @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes

Engineering metrics in pre-devops era Ops Dev • Velocity • Architecture • Compliance @jbaruch @LeonStigter • SLA • Cost env setup • Average customer cost • Tools stability #DevOpsDaysBmore • • • • • QA Incidents Defects MTTR Code coverage Test suite stability http://jfrog.com/shownotes

DEVOPS! ? @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes

Velocity to agile is like ? To devops… ? @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes

Velocity to agile is like profit To devops… Profit @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes

Very much alike! Metric Profit Velocity @jbaruch Easily understandable Unity actionable ✓ ✓ ✓ ✓ ✗ ✗ @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes

Poll time! Who knows what velocity is? Who knows what burndown chart is? Who has a burndown chart? Who looks at the burndown chart? Who trusts the burndown chart? Who knows what to do if it doesn’t look right?

What’s the problem? Velocity @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes

True for both! Metric Profit Velocity @jbaruch Easily understandable Unity actionable ✓ ✓ ✓ ✓ ✗ ✗ @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes

Too coarse! Profit @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes

Too coarse! DevOps @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes

This is devops* Ops Deep specialization Dev *Unless you’re netfliX **You’re not @jbaruch @LeonStigter Common goals, tools, culture QA #DevOpsDaysBmore http://jfrog.com/shownotes

This is devops Ops Dev @jbaruch @LeonStigter QA #DevOpsDaysBmore http://jfrog.com/shownotes

Same for metrics! Ops Specialized metrics Dev @jbaruch @LeonStigter profit QA #DevOpsDaysBmore http://jfrog.com/shownotes

Same for metrics! Ops Dev @jbaruch @LeonStigter QA #DevOpsDaysBmore http://jfrog.com/shownotes

How do metrics collaborate? influenced affected dev ops qa dev ops qa • Time to build • False reopen count • Release pipelines • QA coverage stability • Artifact replication topology • Avg cost of customer • Smoke tests quality Test suite stability • Time to create new Incoming defect QA lab velocity @jbaruch• @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes Test execution time • •

Cooking with metrics @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes

Metrics categorization @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes

Why? @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes

ROI – is this activity is the right thing to do?

Continuous improvement

trust

sampling @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes

Complete sample

@jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes

Representative sample

How effective are we? Samples are good enough for that As long as they are representative Need to be collected over time @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes

Predicting the future

Meet scrum team “avengers”

anecdotes from the daily scrum stand-ups “ We never get enough testing environments from Ops, QA is suffering. Scrum team “Avengers” @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes

Let’s gather some data! Environment Utilization 10 8 6 4 2 0 week 1 week 2 week 3 week 4 week 5 week 6 week 7 Utilized @jbaruch @LeonStigter week 8 week 9 week 10 week 11 week 12 week 13 week 14 Available #DevOpsDaysBmore http://jfrog.com/shownotes

That doesn’t look good! But why? Days the environment stays checked out 25 25 20 18 15 13 13 John Scott 14 15 16 10 10 7 6 5 1 2 3 0 Raj Jane Christina Helen Lee Baruch Nataly Ruppert Josh Patrick Michael Days checked out @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes

Environment Utilization 16 14 12 Continuous improvement 10 8 6 4 2 0 week 1 week 3 week 5 week 7 week 9 week 11 week 13 week 15 week 17 week 19 week 21 week 23 Utilized @jbaruch @LeonStigter #DevOpsDaysBmore Available http://jfrog.com/shownotes

anecdotes from the daily scrum stand-ups “ Tests are failing, because the test suite is not stable, QA makes our builds fail without reason. Scrum team “Avengers” @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes

Let’s gather some data!

Return on investment @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes

anecdotes from the daily scrum stand-ups “ Those developers want to deploy new GC, its tons of work, without any profit. Scrum team “Avengers” @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes

Let’s gather some data! @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes

0:00 0:40 1:20 2:00 2:40 3:20 4:00 4:40 5:20 6:00 6:40 7:20 8:00 8:40 9:20 10:00 10:40 11:20 12:00 12:40 13:20 14:00 14:40 15:20 16:00 16:40 17:20 18:00 18:40 19:20 20:00 20:40 21:20 22:00 22:40 23:20 GC time Trust and predictability GC time @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes

Dos and Don’ts

Measuring the right thing

LoC LOL, anyone? @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes

Avoid data overload

Use reliable data

Metrics generate incentives, be aware! @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes

Tweak as you go

Elevate! Integrated Dashboards Product Specific Dashboards Raw Data @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes

@jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes

How departments measure Support Marketing Sales HR Finance @jbaruch @LeonStigter Engineering Anecdotes from daily standup #DevOpsDaysBmore http://jfrog.com/shownotes

Step into data-driven life Removes blame game Builds accountability and trust Creates common base for discussion Enables DevOps @jbaruch @LeonStigter #DevOpsDaysBmore http://jfrog.com/shownotes

Q&A and twitter ads @jbaruch @LeonStigter #DevOpsDaysBmore #DataDrivenDevOps http://jfrog.com/shownotes