Troubleshooting & Debugging Production Microservices in Kubernetes

Debugging & Troubleshooting Microservices w/ Kubernetes It runs in production, or is it? @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Ray Tsang Developer Advocate Google Cloud Platform Java Champion Spring Cloud GCP cloud.spring.io/spring-cloud-gcp/ JHipster, JDeferred @saturnism @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Baruch Sadogursky Developer Advocate JFrog @jbaruch @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Shownotes! ● jfrog.com/shownotes ○ ○ ○ ○ ○ Slides Video (tomorrow) Links Feedback Raffle @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

@saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Guestbook Service - Create Hello World Service - Greet Guestbook Service Retrieve @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Guestbook UI greeting CRU D Guestbook Service session replication Redis Hello World Service MySQL @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Let's see it! @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

2015 - FAIL! 14:01 @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

2016 - FAIL! 1:49:50 @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

2017 - FAIL! 31:00 @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

2018 - FAIL! 2:28:47 @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Visibility & Insight is Key @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Repository metadata - key insight into artifacts and their relationships @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Production vs Non-Production :( @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Distributed Systems are hard! @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

We've been there, done that @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Information - Logs! kubectl logs -f pod_id @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Inspecting the process / container kubectl exec -ti pod_id /bin/bash @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Port forward to container kubectl port-forward pod_id local_port:remote_port @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Isolate problematic instance w/ Labels kubectl label pod pod_id --overewrite serving=false @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Pod Pod version = 1.0 frontend serving = true Pod Pod version = 1.0 version = 1.0 serving = true serving = true Service Label selectors: version = 1.0 serving = true @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Pod Pod version = 1.0 frontend serving = true Pod Pod version = 1.0 version = 1.0 serving = true serving = false Service Label selectors: version = 1.0 serving = true @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Log aggregation, at scale @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Stackdriver Logging https://cloud.google.com/logging/ @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

@saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Metrics from Logs → Alerting Near-realtime insight @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Search and Query Your Logs @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Export to BigQuery, Storage, Pub/Sub Near-realtime insight @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

@saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Tracing at Google - Dapper https://research.google.com/pubs/pub36356.html @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Zipkin @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Stackdriver Trace https://cloud.google.com/trace/ @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Zipkin → Stackdriver Trace https://cloud.google.com/trace/docs/zipkin @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

@saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Automatic reports Detect performance regression @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Oops… I forgot a log message. Darn! @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Don't you wish you can use a debugger? @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Stackdriver Debugger - Production Debugger https://cloud.google.com/debugger/ @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Add debugger agent java -agentpath:/opt/cdbg/cdbg_java_agent.so ... -jar PATH_TO_JAR_FILE @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Logpoint - dynamically add log messages! No redeployment, Magic. gcloud debug logpoints create HelloworldService.java:35 \ "Received endpoint: {endpoint}/{name}" --target helloworld-ui-1.0-SNAPSHOT @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Snapshots Inspect call stack and variables @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

@saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Trace Log Debug @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Trace → Stackdriver Trace Logs → Stackdriver Logging Debug → Stackdriver Debugger https://cloud.google.com/stackdriver/ @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Thanks! Source: https://github.com/saturnism/spring-boot-docker Shownotes: https://jfrog.com/shownotes Stackdriver: https://cloud.google.com/stackdriver/ Ray Tsang @saturnism Google Cloud Platform Baruch Sadogursky @jbaruch JFrog @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

@saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

@saturnism @jbaruch @jfrog @googlecloud #devoxxFR jfrog.com/shownotes