Kubernetes is hard! Lessons learned taking our apps to Kubernetes
A presentation at Dev Dialogue Meetup May 2019 in in New York, NY, USA by Baruch Sadogursky
Kubernetes is hard! Lessons learned taking our apps to Kubernetes
Why? Environment flexibility Developer, QA, Support, Product, Solution… anyone! Per branch CI/CD Save money and resources Dogfooding @jbaruch #DevDialogue #k8S Ribbit with us: #swampUP http://jfrog.com/shownotes
The end Take it easy Your app is not ready Limits are good (my mom said that) Probes Observability Community @jbaruch #DevDialogue #k8S Ribbit with us: #swampUP http://jfrog.com/shownotes
🎩 @ErinMeyerINSEAD’s “Culture Map”
shownotes Slides Video Links Comments Ratings Raffle! https://jfrog.com/shownotes @jbaruch #DevDialogue #k8S http://jfrog.com/shownotes
What I’ve been promised @jbaruch #DevDialogue #k8S Ribbit with us: #swampUP http://jfrog.com/shownotes
What is it in real life @jbaruch #DevDialogue #k8S Ribbit with us: #swampUP http://jfrog.com/shownotes
@jbaruch #DevDialogue #k8S http://jfrog.com/shownotes
Ribbit with us: #swampUP
Your app is not ready. @jbaruch #DevDialogue #k8S http://jfrog.com/shownotes
From any “logging best practices” talk or article: @jbaruch #DevDialogue #k8S Ribbit with us: #swampUP http://jfrog.com/shownotes
Your app is not ready Data Persistency You need it. Or do you? @jbaruch #DevDialogue #k8S Ribbit with us: #swampUP http://jfrog.com/shownotes
Your app is not ready High availibility If you did HA before K8S, you are going to rewrite it Scaling Up and down Updates and downgrades Some pods will be newer than others (or older) @jbaruch #DevDialogue #k8S Ribbit with us: #swampUP http://jfrog.com/shownotes
Now let’s look at kubernetes @jbaruch #DevDialogue #k8S http://jfrog.com/shownotes
Limit everything … resources: requests: memory: “1Gi” cpu: “100m” limits: memory: “2Gi” cpu: “250m” … Always limit everything Coordinate with applications limits Java -Xms=1g -Xmx=2g RabbitMQ [rabbitmq.conf] total_memory_available_override_value = 1GB MongoDB —wiredTigerCacheSizeGB=1 @jbaruch #DevDialogue #k8S Ribbit with us: #swampUP http://jfrog.com/shownotes
Take care of the health It’s always a good idea, mandatory with K8S readinessProbe When the app is ready to serve livenessProbe Is this thing alive? Probe types Exec - returns 0 Http - returns < 400 … readinessProbe: httpGet: path: /api/system/health port: 8080 … livenessProbe: exec: command: - mongo - —eval - “db.adminCommand(‘ping’)” … livenessProbe: tcpSocket: port: 5672 … Tcp - did we manage to open a port If needed, write an exec script @jbaruch #DevDialogue #k8S Ribbit with us: #swampUP http://jfrog.com/shownotes
More than one container in a pod Initialization (before your container is up) Preheat your cache Run config scripts Sidecar design pattern Log collector Monitoring Network proxy (e.g. Istio) @jbaruch #DevDialogue #k8S Ribbit with us: #swampUP http://jfrog.com/shownotes
How to deploy anything to k8s Copy YAML Paste YAML Fix indents Repeat @jbaruch #DevDialogue #k8S http://jfrog.com/shownotes
Kubernetes resource { “kind”: “Deployment”, “apiVersion”: “extensions/v1beta1”, “metadata”: { “name”: “my-release-docker-app-chart” }, “spec”: { “containers”: [ { “name”: “docker-app-chart”, “image”: “docker.artifactory/docker-app:1.0”, @jbaruch #DevDialogue #k8S http://jfrog.com/shownotes
Let’s build a new one! > docker build –t docker.artifactory/docker-app:1.1 @jbaruch #DevDialogue #k8S http://jfrog.com/shownotes
One last thing… > sed –i.bak s#docker.artifactory/docker-app:1.1#${imageTag}# deployment.yaml
@jbaruch
#DevDialogue
#k8S
http://jfrog.com/shownotes
Or just use :latest “image”: “docker.artifactory/docker-app:latest” @jbaruch #DevDialogue #k8S http://jfrog.com/shownotes
Enter helm @jbaruch #DevDialogue #k8S http://jfrog.com/shownotes
Encapsulated packages of Kubernetes deployments All this… Becomes this xrayxray-analysis xray-event xray-indexer xray-nfs-server xray-persist xray @jbaruch #DevDialogue #k8S http://jfrog.com/shownotes
Powerful templating for descriptor files { “kind”: “Deployment”, “apiVersion”: “extensions/v1beta1”, “metadata”: { “name”: “{{ template “docker-app.fullname” . }}” }, “spec”: { “containers”: [ { “name”: “{{ template “docker-app.name” . }}”, “image”: “{{ .Values.image.repository }}: {{ .Values.image.tag }}” @jbaruch #DevDialogue #k8S http://jfrog.com/shownotes
Values: # Default values for docker-app. # This is a YAML-formatted file. # Declare name/value pairs to be passed into your templates. image: repository: docker.artifactory/docker-app tag: 1.1 secretName: regsecret pullPolicy: Always @jbaruch #DevDialogue #k8S http://jfrog.com/shownotes
Simple! Templates Values Metadata @jbaruch #DevDialogue #k8S http://jfrog.com/shownotes
Chart <-> image relationship Using templates we can reuse charts for multiple image versions Chart versions != Image versions @jbaruch #DevDialogue #k8S http://jfrog.com/shownotes
Helm repositories Official repository - kubeapps.com Get a local one! Option 1: Create your own: Run an http server with index.yaml Run helm repo index to generate one the index Option 2: Use JFrog Artifactory Universal Artifact Repository which supports Docker, Helm and everything else @jbaruch #DevDialogue #k8S http://jfrog.com/shownotes
observability No more ssh and grep in logs No need for production access for everybody (unless you pretend you’re Netflix) Observability tools: Log aggregation (Sumo Logic, Splunk, Log Entries) APM (New Relic, AppDynamics) Monitoring (DataDog, SignalFX) Cloud-Native tools @jbaruch #DevDialogue #k8S Ribbit with us: #swampUP http://jfrog.com/shownotes
Cloud-native observability - monitoring Prometheus Grafana @jbaruch #DevDialogue #k8S Ribbit with us: #swampUP http://jfrog.com/shownotes
Cloud-native observability - logs Fluentd ElasticSearch Kibana @jbaruch #DevDialogue #k8S Ribbit with us: #swampUP http://jfrog.com/shownotes
We’re big on those pipelines and promotions concepts If quality requirments are hit CI SERVER 1 If quality requirments are hit 2 Integration If quality requirments are hit 3 System Testing 4 Staging Production *
- Quality gates - @jbaruch #DevDialogue #k8S http://jfrog.com/shownotes
Integration with CI servers @jbaruch #DevDialogue #k8S Ribbit with us: #swampUP http://jfrog.com/shownotes
Jfrog and k8s GoCenter.io Distribution Development, testing @jbaruch #DevDialogue #k8S Ribbit with us: #swampUP http://jfrog.com/shownotes
Community Probably someone had this problem before CNCF Ambassadors Project maintainers and commiters Kudos RabbitMQ HA MongoDB (Bitnami) @jbaruch #DevDialogue #k8S Ribbit with us: #swampUP http://jfrog.com/shownotes
The end Take it easy Your app is not ready Limits are good (my mom said that) Probes Observability Community @jbaruch #DevDialogue #k8S Ribbit with us: #swampUP http://jfrog.com/shownotes
Twitter ads and Q&A http://jfrog.com/shownotes @jbaruch #DevDialogue #k8S Ribbit with us: #swampUP
Happy sailing! @jbaruch #DevDialogue #k8S Ribbit with us: #swampUP http://jfrog.com/shownotes
Today, Kubernetes is the defacto standard if you want to run container workloads in a production environment, though that wasn’t always the case. We had/have a fair amount of monolithic code for our products and to make sure we could build GoCenter in a way that would be easier to manage at the scale we needed to make changes. For us, that meant changes to the way we write code (more Go), how we build apps (using Docker and Kubernetes), and how we deploy (using Helm). We had to figure out a whole bunch of things as we looked at building proper microservices that we could deploy to Kubernetes. During the session, we will pay special attention to things like logging, persistence, limits, and probes because containerizing an app is not simply putting your existing app into a docker container. As we’re looking at containers, knowing exactly what is in the container in terms of licenses, packages, and vulnerabilities was, and is, incredibly important to us to keep our cluster and data secure.
