Improving the Developer Experience with Developer Productivity Engineering
A presentation at Developer Week 2024 in in Oakland, CA, USA by Baruch Sadogursky
Baruch Sadogursky - @jbaru ch × Developer Productivity Advocate × Gradle Inc × Development -> DevOps -> #DPE
shownotes × × × × speaking.jbaru.ch Slides Video All the links!
Don’t ruin the flow
“The build takes forever, I am distracted to do other things and the context switch is terrible”
Don’t frustrate the developers
“we have a flanky test, it is irrelevant 99.5% of the time , but it always runs and it is last in the suite”
Don’t boil the frog
“I have a feeling that everything is slower somehow…”
Developer Productivity == A/M Autonomy /P == motivation Tools and people aren’t in my way Mastery Tools and processes help me to excel Purpose I want to be productive, i.e. create the product
Developer productivity Enginee ring!
Developer Productivity Enginee Foster Faster Feedback Collaborate through Effective Tooling Eliminate Toil for Developers Dedicated Organizational Mindset ring Embrace Rigorous Observability for Proactive Improvement Prioritize Automation and Eliminate Bottlenecks Outcomes Over Output
Talk is cheap, show me the goods!
Small DPE improvements make a × × × × × huge difference Generate code faster: Better IDE Test better: Testcontainers Enforce better code: Sonar Test more reliably: Flaky test detection Foster Faster Feedback:
feedback efficiency × × × × IDE: Sub-seconds (I type, it marks it red) Build: Seconds CI: Minutes Production: Hours/Days
Reverse dependency on distance from developers IDE Build CI Feedback Time Faster Slower Distance from Developers Expected Production
Reverse dependency on distance from developers IDE Build CI Production Feedback Time Faster Slower Distance from Developers Expected Real
It is slow!
It is slow and the developers h ave no idea why!
What is build? × × × × × Project setup Downloading the Internet Artifact generation: Compilation, packaging, etc Tests Artifact deployment
What can go wrong? × × × × × Project setup Downloading the Internet Artifact generation: Compilation, packaging, etc Tests Artifact deployment
When can it go wrong? At any time.
The Build frustrates the developers
Let’s ask Chatgpt
What the actual f*ck?! × Skipping tests defeats the purpose of the build! × How about skipping compilation? × We want faster feedback, not less feedback
What feedback do we want?
Ci/cd pipeline quality gates It compiles Basic Integration tests Basic Unit tests Basic Quality Linting Quality Code coverage Dependency scanning SAST/DAST Quality Sec Static code analysis Sec Sec Secrets scanning Resource Utilization Nonfunc Nonfunc Load Testing Nonfunc Compliance
Two types of feedback x e.g., CI/CD x we never wait for it x results are distracting x e.g., build x we’ll wait for it in the flow x we’ll be pissed off when it’s slow
Reverse dependency on distance from developers IDE Build CI Feedback Time Faster Slower Distance from Developers Production
Reverse dependency on distance from developers Faster IDE Build CI Feedback Time Commit time synchronous asynchronous Slower Distance from Developers Production
Ideal build time feedback It compiles Basic Integration tests Basic Unit tests Basic Quality Linting Quality Code coverage Dependency scanning SAST/DAST Quality Sec Static code analysis Sec Sec Secrets scanning Resource Utilization Nonfunc Nonfunc Load Testing Nonfunc Compliance
Delightful build (pick two): ☑ provides max feedback ☑ fast
Skip what can be skipped (but no more!)
Avoidance: Incremental build × Don’t build what didn’t changed × Don’t build what isn’t affected
Avoidance: Incremental build s hortcomings × Relies on produced artifacts × Relies on architectural decisions
Avoidance: Caching × × × × Makes the build faster Makes the build faster for everybody Makes the build faster always Makes all parts of the build faster
Avoidance: Predictive test selec tion × Learns de-facto code change effects × Skips tests with high degree of confidence
Predictive Test prediction What changed Where it changed Correlate with observed test failures Predictions which changes will fail which tests
Black magic in action × The more tests a project has, the less they break × Refactorings in Java break tests less than in JavaScript
Speed up what can’t be skipped
Test parallelization × Use max power of local machine × (Yes, your boss should buy you the bleeding edge)
Test distribution × × × × CI uses fan-out to speed-up tests Shouldn’t you enjoy it for local tests? Use the cloud to distribute test load RUN ALL THE NEEDED TESTS!
Don’t let it slide
Observe and improve × Measure local build times across time and environments × Detect downfacing trends × Find root causes and improve
The gains are real!
The software development process is rampant with bottlenecks that negatively impact developer productivity and ultimately, happiness. On a daily basis, developers are stymied by slow build and test feedback cycles, inefficient troubleshooting, and unreliable toolchains that don’t send valid signals (e.g. flaky tests). Antipatterns such as these create toil and inhibit their ability to create quality code, leaving developers feeling burnt out and unfulfilled. In fact, 85% of surveyed IT organizations reported spending too much time waiting on build and test feedback. Given the high rates of developer burnout and generative AI’s growing potential to upend the development process, it is more critical than ever for IT leaders to prioritize the developer experience.
In this session, Gradle Inc. Principal Developer Advocate Baruch Sadogursky will introduce the emerging practice of Developer Productivity Engineering (DPE) and discuss how it can increase developer productivity by improving the developer experience. DPE views developer productivity as a technology challenge that can be solved by engineering solutions, rather than a people problem addressable by management techniques. As such, it relies on acceleration technologies and failure analytics to address the toil, frustrations, and process bottlenecks that most negatively impact developers. DPE aims to achieve faster feedback cycles and provide actionable data so that developers can spend more time doing what they love—innovating and writing great code.
After this discussion, audiences will walk away with a better understanding of the benefits of adopting a DPE approach—like increased cost savings and delivering higher quality software, faster—as well as how teams can leverage it to maximize developer productivity and joy.
Video
Resources
The following resources were mentioned during the presentation or are useful additional information.
-
Drive - The Surprising Truth About What Motivates Us
-
DevOps Tools For Java Developers
-
Liquid Software: How to Achieve Trusted Continuous Updates in the DevOps World
-
The Problem With Gradle
-
Gradle Incremental Build
-
Gradle Build Cache
-
Gradle Remote Cache - Develocity
-
Predictive Test Selection - Meta Research
-
Predictive Test Selection - Develocity
-
Gradle Test Parallelization
-
Cloud Test Distribution - Develocity
-
Analytics and Observability - Develocity
-
Try It Yourself and Win Prizes: The Build Speed Challenge
-
Learn More About DPE: The DPE Handbook
