Troubleshooting & Debugging Production Microservices in Kubernetes

A presentation at Devoxx Ukraine 2018 in November 2018 in Kyiv, Ukraine, 02000 by Baruch Sadogursky

Slide 1

Debugging & Troubleshooting Microservices w/ Kubernetes It runs in production, or is it? @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 2

Ray Tsang Developer Advocate Google Cloud Platform Java Champion Spring Cloud GCP cloud.spring.io/spring-cloud-gcp/ JHipster, JDeferred @saturnism @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 3

Baruch Sadogursky Developer Advocate JFrog @jbaruch @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 4

Shownotes! ● jfrog.com/shownotes ○ ○ ○ ○ ○ Slides Video (tomorrow) Links Feedback Raffle @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 5

@saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 6

Guestbook Service - Create Hello World Service - Greet Guestbook Service Retrieve @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 7

Guestbook UI greeting CRU D Guestbook Service session replication Redis Hello World Service MySQL @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 8

Let's see it! @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 9

2015 - FAIL! 14:01 @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 10

2016 - FAIL! 1:49:50 @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 11

2017 - FAIL! 31:00 @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 12

2018 - FAIL! 2:28:47 @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 13

Visibility & Insight is Key @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 14

Repository metadata - key insight into artifacts and their relationships @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 15

Production vs Non-Production :( @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 16

Distributed Systems are hard! @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 17

We've been there, done that @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 18

Information - Logs! kubectl logs -f pod_id @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 19

Inspecting the process / container kubectl exec -ti pod_id /bin/bash @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 20

Port forward to container kubectl port-forward pod_id local_port:remote_port @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 21

Isolate problematic instance w/ Labels kubectl label pod pod_id --overewrite serving=false @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 22

Pod Pod version = 1.0 frontend serving = true Pod Pod version = 1.0 version = 1.0 serving = true serving = true Service Label selectors: version = 1.0 serving = true @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 23

Pod Pod version = 1.0 frontend serving = true Pod Pod version = 1.0 version = 1.0 serving = true serving = false Service Label selectors: version = 1.0 serving = true @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 24

Log aggregation, at scale @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 25

Stackdriver Logging https://cloud.google.com/logging/ @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 26

@saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 27

Metrics from Logs → Alerting Near-realtime insight @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 28

Search and Query Your Logs @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 29

Export to BigQuery, Storage, Pub/Sub Near-realtime insight @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 30

@saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 31

@saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 32

Tracing at Google - Dapper https://research.google.com/pubs/pub36356.html @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 33

Zipkin @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 34

Stackdriver Trace https://cloud.google.com/trace/ @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 35

Zipkin → Stackdriver Trace https://cloud.google.com/trace/docs/zipkin @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 36

@saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 37

@saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 38

@saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 39

@saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 40

Automatic reports Detect performance regression @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 41

Oops… I forgot a log message. Darn! @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 42

Don't you wish you can use a debugger? @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 43

Stackdriver Debugger - Production Debugger https://cloud.google.com/debugger/ @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 44

Add debugger agent java -agentpath:/opt/cdbg/cdbg_java_agent.so ... -jar PATH_TO_JAR_FILE @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 45

Logpoint - dynamically add log messages! No redeployment, Magic. gcloud debug logpoints create HelloworldService.java:35 \ "Received endpoint: {endpoint}/{name}" --target helloworld-ui-1.0-SNAPSHOT @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 46

Snapshots Inspect call stack and variables @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 47

@saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 48

@saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 49

Trace Log Debug @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 50

Trace → Stackdriver Trace Logs → Stackdriver Logging Debug → Stackdriver Debugger https://cloud.google.com/stackdriver/ @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 51

Thanks! Source: https://github.com/saturnism/spring-boot-docker Shownotes: https://jfrog.com/shownotes Stackdriver: https://cloud.google.com/stackdriver/ Ray Tsang @saturnism Google Cloud Platform Baruch Sadogursky @jbaruch JFrog @saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 52

@saturnism @jbaruch @jfrog @gcpcloud #devoxxua jfrog.com/shownotes

Slide 53

@saturnism @jbaruch @jfrog @googlecloud #devoxxFR jfrog.com/shownotes