Troubleshooting & Debugging Production Microservices in Kubernetes
A presentation at Devoxx Ukraine 2018 in in Kyiv, Ukraine, 02000 by Baruch Sadogursky
Debugging & Troubleshooting Microservices w/ Kubernetes It runs in production, or is it? @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
Ray Tsang Developer Advocate Google Cloud Platform Java Champion Spring Cloud GCP cloud.spring.io/spring-cloud-gcp/ JHipster, JDeferred @saturnism @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
Baruch Sadogursky Developer Advocate JFrog @jbaruch @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
Shownotes! ● jfrog.com/shownotes ○ ○ ○ ○ ○ Slides Video (tomorrow) Links Feedback Raffle @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
@saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
Guestbook Service - Create Hello World Service - Greet Guestbook Service Retrieve @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
Guestbook UI greeting CRU D Guestbook Service session replication Redis Hello World Service MySQL @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
Let's see it! @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
2015 - FAIL! 14:01 @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
2016 - FAIL! 1:49:50 @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
2017 - FAIL! 31:00 @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
2018 - FAIL! 2:28:47 @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
Visibility & Insight is Key @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
Repository metadata - key insight into artifacts and their relationships @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
Production vs Non-Production :( @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
Distributed Systems are hard! @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
We've been there, done that @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
Information - Logs! kubectl logs -f pod_id @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
Inspecting the process / container kubectl exec -ti pod_id /bin/bash @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
Port forward to container kubectl port-forward pod_id local_port:remote_port @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
Isolate problematic instance w/ Labels kubectl label pod pod_id --overewrite serving=false @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
Pod Pod version = 1.0 frontend serving = true Pod Pod version = 1.0 version = 1.0 serving = true serving = true Service Label selectors: version = 1.0 serving = true @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
Pod Pod version = 1.0 frontend serving = true Pod Pod version = 1.0 version = 1.0 serving = true serving = false Service Label selectors: version = 1.0 serving = true @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
Log aggregation, at scale @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
Stackdriver Logging https://cloud.google.com/logging/ @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
@saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
Metrics from Logs → Alerting Near-realtime insight @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
Search and Query Your Logs @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
Export to BigQuery, Storage, Pub/Sub Near-realtime insight @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
@saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
@saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
Tracing at Google - Dapper https://research.google.com/pubs/pub36356.html @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
Zipkin @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
Stackdriver Trace https://cloud.google.com/trace/ @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
Zipkin → Stackdriver Trace https://cloud.google.com/trace/docs/zipkin @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
@saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
@saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
@saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
@saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
Automatic reports Detect performance regression @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
Oops… I forgot a log message. Darn! @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
Don't you wish you can use a debugger? @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
Stackdriver Debugger - Production Debugger https://cloud.google.com/debugger/ @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
Add debugger agent java -agentpath:/opt/cdbg/cdbg_java_agent.so ... -jar PATH_TO_JAR_FILE @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
Logpoint - dynamically add log messages! No redeployment, Magic. gcloud debug logpoints create HelloworldService.java:35 \ "Received endpoint: {endpoint}/{name}" --target helloworld-ui-1.0-SNAPSHOT @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
Snapshots Inspect call stack and variables @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
@saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
@saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
Trace Log Debug @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
Trace → Stackdriver Trace Logs → Stackdriver Logging Debug → Stackdriver Debugger https://cloud.google.com/stackdriver/ @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
Thanks! Source: https://github.com/saturnism/spring-boot-docker Shownotes: https://jfrog.com/shownotes Stackdriver: https://cloud.google.com/stackdriver/ Ray Tsang @saturnism Google Cloud Platform Baruch Sadogursky @jbaruch JFrog @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
@saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes
@saturnism @jbaruch @jfrog @googlecloud #devoxxFR jfrog.com/shownotes
Debugging applications in production is like being the detective in a crime movie. Especially with microservices. Especially with containers. Especially in the cloud. Trying to see what’s going on in a production deployment at scale is impossible without proper tools! Google has spent over a decade deploying containerized Java applications at unprecedented scale and the infrastructure and tools developed by Google have made it uniquely possible to manage, troubleshoot, and debug, at scale.
Join this session to see how you can diagnose and troubleshoot production issues w/ out of the box Kubernetes tools, as well as getting insight from the ecosystem with Weave Scope, JFrog Artifactory & Stackdriver tools.
