Istio is an open-source service mesh for microservices: it is the connective layer between your services that adds traffic control, load balancing, resiliency, observability, and security without embedding those concerns in application code. This post is based on Introducing Istio Service Mesh for Microservices (Posta & Sutter, O’Reilly) and walks through Istio’s architecture, installation, traffic control, resiliency, chaos testing, observability, and security.

Istio architecture (high level):

+------------------- CONTROL PLANE (istio-system) -------------------+
|  +-----------+  +-----------+  +-----------+  +------------------+ |
|  |  Pilot    |  |  Mixer    |  | Istio CA  |  | Ingress Gateway  | |
|  | discovery |  | policy,   |  | mTLS certs|  | (Envoy)          | |
|  | routing,  |  | telemetry |  |           |  | Gateway +        | |
|  | xDS config|  | ACLs      |  |           |  | VirtualService   | |
|  +-----+-----+  +-----+-----+  +-----------+  +--------+---------+ |
|        | xDS          | telemetry                    | external  | |
+--------+--------------+------------------------------+-------------+
         |              |                              |
         v              v                              v
+------------------- DATA PLANE (application pods) ---------------------+
|  +--------------------+         +--------------------+                |
|  | Pod A              |         | Pod B              |                |
|  | +---------------+  |  mesh   | +---------------+  |                |
|  | | app container |  |  ===>   | | app container |  |                |
|  | +-------+-------+  |  traffic| +-------+-------+  |                |
|  |         v          |         |         v          |                |
|  | +---------------+  |         | +---------------+  |                |
|  | | istio-proxy   |  |         | | istio-proxy   |  | (app+sidecar)  |
|  | | (Envoy)       |  |         | | (Envoy)       |  |                |
|  | +---------------+  |         | +---------------+  |                |
|  +--------------------+          +-------------------+                |
+-----------------------------------------------------------------------+

• Control plane: Pilot pushes routing and discovery config to sidecars (xDS). Mixer handles policy checks and telemetry. Istio CA issues certificates for mTLS. The Ingress Gateway is the Envoy at the edge for external traffic.
• Data plane: Each workload pod has an istio-proxy (Envoy) sidecar. All traffic to/from the app goes through the sidecar, which applies routing, retries, timeouts, and reports metrics/traces.

---

Why a service mesh?

Microservices communicate over the network, which is unreliable. With many deployments per week and polyglot runtimes, embedding resilience (retries, timeouts, circuit breakers) and observability in every service and language is costly and inconsistent. Kubernetes gives you deployment and basic round-robin load balancing but does not manage how services talk to each other.

A service mesh moves these cross-cutting concerns into the infrastructure: traffic is intercepted by a proxy next to each service, so you get uniform behavior regardless of language or framework. Istio was designed to work on multiple platforms but has first-class support for Kubernetes (and OpenShift).

---

Architecture: data plane and control plane

Istio has two main parts: the data plane (proxies that handle traffic) and the control plane (components that configure and manage those proxies).

Data plane

• Service proxy: Every outbound and inbound network call from your app goes through a proxy. The proxy can retry, timeout, circuit-break, collect metrics, and enforce policy. Istio’s default proxy is Envoy (Layer 7, C++, used in production at scale).
• Sidecar: The proxy runs as a sidecar container in the same pod as your app. Istio injects an istio-proxy (Envoy) container that intercepts all traffic to/from the business container. The app stays unaware; no SDK is required.

So: before Istio → app talks directly to the network. With Istio → app talks to localhost, sidecar intercepts and forwards; all policies and telemetry apply at the proxy.

Control plane

The control plane is the source of configuration and policy for the mesh.

Component	Role
Pilot	Service discovery and routing. Pushes routing tables and config to sidecars. Manages routing rules (e.g. RouteRule → today’s VirtualService) and destination policies (e.g. DestinationPolicy → today’s DestinationRule).
Mixer	Policy and telemetry. Receives metrics/traces from proxies; enforces ACLs, rate limits, and custom checks. Pluggable backends.
Auth (Istio CA)	Certificate authority. Issues x509 certs to workloads for mTLS (mutual TLS) between services, so mesh traffic can be encrypted and identity-based.

---

Installation (conceptual)

The book uses Istio 0.5.1 on OpenShift/minishift; current Istio is installed differently (e.g. istioctl install or Helm). The idea is:

1. Prerequisites: Kubernetes (or OpenShift), kubectl/oc, and optionally istioctl.
2. Install control plane: Deploy Istio (Pilot, Mixer, Auth, ingress gateway) into a dedicated namespace (e.g. istio-system).
3. Sidecar injection: Enable automatic or manual injection. Manual: istioctl kube-inject -f Deployment.yml and apply the result; automatic: label the namespace and Istio injects the proxy into new pods.
4. Add-ons: Prometheus, Grafana, Jaeger (or Zipkin), and optional service-graph for metrics and tracing.

After installation, deploy your apps as usual; with injection enabled, each pod gets the Envoy sidecar and joins the mesh.

---

Integration with Kubernetes

Istio is built to run on top of Kubernetes. It uses standard Kubernetes resources and adds its own Custom Resource Definitions (CRDs) and controllers so that mesh behavior is configured declaratively and stays in sync with the cluster.

How Istio uses Kubernetes

Kubernetes resource	How Istio uses it
Pods	Sidecar (Envoy) runs in the same pod as your app; all traffic to/from the app goes through the proxy.
Services	Istio discovers backends from Kubernetes Services and Endpoints. Routing rules (VirtualService) and destination config (DestinationRule) reference service names and pod labels.
Deployments	Replicas are unchanged; each pod gets a sidecar when injection is enabled. Version canaries use labels (e.g. version: v1, version: v2) on pods; Istio routes by these labels.
Namespaces	Mesh scope is per namespace. You enable injection per namespace; Istio CRDs are namespaced so policies apply per namespace (or cluster-wide where supported).

So: you keep using Deployments, Services, and Pods as usual. Istio extends them with the sidecar and with CRDs that describe how traffic flows between those services.

Sidecar injection

• Manual: Run your manifest through the injector and apply the result. The injector adds the istio-proxy container (and usually an init container that configures iptables so all TCP traffic is captured by the proxy).
  kubectl apply -f <(istioctl kube-inject -f deployment.yaml) -n my-ns
• Automatic: Add the label istio-injection=enabled to the namespace. The Istio admission webhook mutates new pods in that namespace so they get the sidecar and init container automatically. No change to your Deployment YAML.

Pods that have the sidecar are “in the mesh”; pods without it are not. You can run both in the same cluster (e.g. gradually roll out Istio by namespace).

Istio CRDs and the Kubernetes API

Istio installs CRDs such as VirtualService, DestinationRule, Gateway, PeerAuthentication, AuthorizationPolicy, ServiceEntry. You apply them with kubectl apply (or istioctl). The Istio control plane (e.g. Pilot) watches these resources and pushes config to the Envoy sidecars via the xDS API. So:

• VirtualService → “how to route traffic to service X” (hosts, routes, weights, timeouts, retries).
• DestinationRule → “how to talk to service X” (subsets by labels, load balancing, circuit breaker, TLS).
• Gateway → “how to expose the mesh at the edge” (hosts, TLS, ports).
• ServiceEntry → “how to reach external or legacy services” (egress).

Traffic flow stays Kubernetes-native: clients still call Kubernetes Service DNS names (e.g. http://recommendation:8080). The caller’s sidecar intercepts the call, consults the config from the control plane, and forwards to the right pod(s) according to VirtualService and DestinationRule. Kubernetes Services and Endpoints remain the source of which pods exist; Istio adds routing and resilience on top.

Network (traffic) flow: how Istio controls it

The diagram below shows how traffic flows through the mesh and where Istio (control plane and sidecars) takes control. All inter-pod and ingress traffic passes through Envoy proxies; the control plane configures those proxies from Kubernetes and Istio CRDs.

                    EXTERNAL CLIENT (e.g. browser, API caller)
                                      |
                                      | HTTPS
                                      v
    +------------------------------------------------------------------+
    |                    Istio Ingress Gateway (Envoy)                 |
    |                    Gateway + VirtualService (host, routes)       |
    +------------------------------------------------------------------+
                                      |
                                      | K8s Service: customer
                                      v
    +------------------+     outbound      +------------------+
    |  Pod: customer   |  ---------------> |  istio-proxy     |  (sidecar)
    |  [app container] |  localhost:8080   |  (Envoy)         |
    +------------------+  <--------------- +------------------+
         app calls                              |
         preference:8080                        | VirtualService /
         (captured by sidecar)                  | DestinationRule
                |                               | (from Pilot via xDS)
                v                               v
    +------------------+     outbound      +------------------+
    |  Pod: preference |  ---------------> |  istio-proxy     |
    |  [app container] |  <--------------- |  (Envoy)         |
    +------------------+                   +------------------+
         app calls                              |
         recommendation:8080                    | route to subset v1/v2,
                |                               | retry, timeout, etc.
                v                               v
    +------------------+     outbound      +------------------+
    |  Pod: recommend. |  ---------------> |  istio-proxy     |
    |  [app container] |  <--------------- |  (Envoy)         |
    +------------------+                   +------------------+
                                  ^
                                  | K8s Service: recommendation
                                  | (Endpoints = pod IPs by label)
    +------------------------------------------------------------------+
    |  CONTROL PLANE (istio-system)                                    |
    |  Pilot watches K8s Services/Endpoints + VirtualService/          |
    |  DestinationRule/Gateway → pushes config to every sidecar (xDS)  |
    +------------------------------------------------------------------+

(recommend. = recommendation pod; each pod has app container + istio-proxy sidecar)

Flow in words:

1. Ingress: External traffic hits the Istio Ingress Gateway (Envoy). The Gateway resource defines the listener; a VirtualService bound to that Gateway selects the Kubernetes Service (e.g. customer) and route. Traffic is then sent to a customer pod.
2. Pod egress: When the customer app calls http://preference:8080, the connection is captured by the pod’s init container (iptables rules) and sent to the istio-proxy sidecar on localhost instead of directly to the network. The app is unchanged; it still uses the K8s Service name.
3. Sidecar outbound: The customer sidecar (Envoy) receives the request. It has already received config from Pilot (xDS): VirtualServices, DestinationRules, etc. It resolves preference to a pod IP (using K8s Endpoints and any subset/weight from the rules), applies retries/timeouts/circuit breaker, and forwards to the preference pod’s sidecar.
4. Sidecar inbound: The preference pod’s sidecar receives the request, forwards it to the app container on localhost, and sends the response back. Same pattern for preference → recommendation.
5. Control plane: Pilot watches the Kubernetes API (Services, Endpoints, Pods) and Istio CRDs (VirtualService, DestinationRule, Gateway). It computes the routing and policy config and pushes it to every sidecar via the xDS protocol. So when you apply a VirtualService or DestinationRule, Pilot updates the sidecars and traffic behavior changes without pod restarts.

So: Kubernetes provides Services and Endpoints (who is who); Istio CRDs (VirtualService, DestinationRule, Gateway) describe how traffic is routed and secured; sidecars enforce that behavior on every request. The diagram above is the network flow that Istio controls for K8s.

Example: routing traffic (VirtualService + DestinationRule)

Suppose you have a Kubernetes Service named recommendation and two Deployments that back it: one with pods labeled version: v1, the other version: v2. Without Istio, the Service would round-robin across all pods. With Istio, you define subsets (which pods belong to which “version”) and routing (what percentage of traffic goes to each subset).

Step 1 — Define subsets (DestinationRule)
A DestinationRule attaches a name and a selector to a subset of pods that back a service. Istio uses the selector to match pod labels; the result is a named subset (e.g. v1, v2) that you can reference in a VirtualService.

# recommendation-destination.yaml
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: recommendation
  namespace: default
spec:
  host: recommendation          # Kubernetes Service name (DNS in the cluster)
  subsets:
    - name: v1
      labels:
        version: v1              # Pods with this label form the "v1" subset
    - name: v2
      labels:
        version: v2              # Pods with this label form the "v2" subset

• host: The service that this rule applies to. Must match the Kubernetes Service name (and optionally namespace) that your app calls (e.g. recommendation or recommendation.default.svc.cluster.local).
• subsets: Each subset has a name (used in the VirtualService) and labels. Only pods that match these labels and are in the Service’s Endpoints are part of that subset. So when the preference service calls http://recommendation:8080, the sidecar will know which pods are “v1” and which are “v2”.

Step 2 — Route traffic by weight (VirtualService)
A VirtualService says how to route traffic to that host. Here we send 80% of requests to subset v1 and 20% to subset v2 (canary).

# recommendation-virtualservice.yaml
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: recommendation
  namespace: default
spec:
  hosts:
    - recommendation             # Same as DestinationRule host: the K8s Service
  http:
    - route:
        - destination:
            host: recommendation
            subset: v1
          weight: 80
        - destination:
            host: recommendation
            subset: v2
          weight: 20

• hosts: List of hostnames this VirtualService applies to. Typically the same as the Kubernetes Service name so that any call to recommendation (from within the mesh) is handled by this rule.
• http.route: For HTTP traffic, you list one or more destination entries. Each has host + subset (from the DestinationRule) and weight (percentage, 0–100). Istio distributes requests according to these weights (e.g. 80% to pods in subset v1, 20% to subset v2).

What actually happens on a request

1. The preference app (in the same or another namespace) calls http://recommendation:8080. The call goes to the preference pod’s sidecar (Envoy), because all outbound traffic from the pod is captured.
2. The sidecar resolves recommendation via Kubernetes DNS and sees that there is a VirtualService for host recommendation. It evaluates the http route: with weights 80/20, it randomly (or deterministically) chooses one of the two destinations for this request—e.g. recommendation subset v2.
3. The sidecar looks up the DestinationRule for recommendation and finds subset v2 → selector version: v2. It gets the list of pod IPs for that subset from the Service’s Endpoints (filtered by label). It then forwards the request to one of those pods (e.g. using round-robin within the subset).
4. The request reaches a recommendation pod with label version: v2. That pod’s sidecar receives it, forwards to the app container, and the response goes back along the same path.

So: Kubernetes (Service + Endpoints + pod labels) defines who is in the mesh; DestinationRule defines groupings (subsets) by labels; VirtualService defines how traffic is split or matched to those subsets. No change is required in the preference or recommendation application code—they still use the same service URL.

Apply and verify

kubectl apply -f recommendation-destination.yaml
kubectl apply -f recommendation-virtualservice.yaml
# Send repeated requests; you should see ~80% v1 and ~20% v2 responses
for i in $(seq 1 20); do curl -s http://customer.default.svc.cluster.local/ | grep recommendation; done

You can change the weight in the VirtualService (e.g. 50/50 or 100% v2) and re-apply; the new split takes effect as soon as the control plane pushes updated config to the sidecars, with no pod restarts.

Ingress: Istio Gateway vs Kubernetes Ingress

• Kubernetes Ingress: Single resource for HTTP(S) routing into the cluster; controllers (e.g. NGINX, AWS ALB) implement it. Istio does not use the standard Ingress resource for mesh features.
• Istio Gateway: Defines listeners (host, port, TLS) at the edge of the mesh. It configures the Istio ingress gateway (Envoy) that runs as a Deployment/Service. VirtualService can bind to a Gateway to define routes for that host. So: Gateway = “what to expose,” VirtualService = “where to send traffic.” This gives you consistent routing, mTLS, and observability at the edge.

For external traffic: Client → Istio Ingress Gateway (Gateway + VirtualService) → mesh Services → pods with sidecars.

Summary: request path with Kubernetes

1. User hits https://api.example.com → DNS/load balancer → Istio Ingress Gateway (pod with Envoy).
2. Gateway matches Gateway + VirtualService → route to Kubernetes Service customer.
3. Request goes to a customer pod; sidecar receives it, applies policy, forwards to app container.
4. App calls http://preference:8080 → outbound goes to sidecar → Envoy uses VirtualService / DestinationRule for preference → forwards to a preference pod (e.g. by version label).
5. Same pattern for preference → recommendation. All inter-pod traffic is in the mesh (optional mTLS, retries, timeouts, metrics, tracing).

Kubernetes provides placement and discovery (Pods, Services, Endpoints); Istio adds traffic and security policy (VirtualService, DestinationRule, Gateway, auth) and observability without changing your application code.

---

Traffic control

Kubernetes Services give round-robin across pods. Istio adds routing rules so you can send traffic by version, weight, or header.

RouteRule (legacy) → VirtualService (current)

The book uses RouteRule (v1alpha2). In current Istio you use VirtualService and DestinationRule. The concepts map directly.

Canary and weighted routing

• Default: 100% to one version (e.g. version: v1).
• Canary: e.g. 90% v1, 10% v2. You specify destination (service name/namespace), route with labels (e.g. version: v2) and weight (percentage).
• Precedence: Higher precedence wins. So a “default” rule (precedence 1) sends all traffic to v1; a canary rule (precedence 5) overrides it with 90/10.
• Rollout: Gradually increase weight to v2 (e.g. 50/50, then 100% v2), then remove the canary rule and set the default to v2.

Example (book style, RouteRule):

# 90% v1, 10% v2 canary
spec:
  destination:
    name: recommendation
  precedence: 5
  route:
    - labels: { version: v1 }
      weight: 90
    - labels: { version: v2 }
      weight: 10

Header-based routing

Route only requests that match a predicate (e.g. User-Agent containing “Safari”) to a specific version. Use a match clause on request headers; that rule has higher precedence than the default so only matching traffic goes to the target version.

Dark launch (traffic mirroring)

Send live traffic to one version (e.g. v1) and mirror the same requests to another (e.g. v2) in a fire-and-forget way. The client only sees the response from v1; v2 receives a copy for testing. Configure mirror in the route to the v2 subset. Useful for pre-release validation with production-like traffic.

Egress

By default, Istio can block outbound traffic outside the cluster. You explicitly allow external hosts with EgressRule (legacy) or current egress configuration (e.g. ServiceEntry). Example: allow only httpbin.org:80 so a pod can call that service; other external calls are denied. This supports zero-trust and reduces the impact of a compromised pod.

---

Service resiliency

Resilience is enforced in the sidecar, so it works for any language or framework.

Load balancing

Beyond Kubernetes round-robin, Istio (via DestinationPolicy / DestinationRule) can set the algorithm:

• ROUND_ROBIN (default)
• RANDOM
• LEAST_CONN (weighted least request)

You target a destination (and optionally source) so that, for example, traffic from preference to recommendation uses RANDOM.

Timeout

Unbounded waits can cause cascading failures. In a RouteRule / VirtualService you set httpReqTimeout (e.g. 1s). If the upstream does not respond in time, the proxy returns 504 (e.g. “upstream request timeout”) and stops waiting.

Retry

For transient failures (e.g. 503), the proxy can retry to another pod. Configure httpReqRetries: number of attempts and perTryTimeout. So with 3 attempts and 2s per try, one failing call may still succeed on a retry. Combined with pool ejection, retries help avoid surfacing backend errors to the client.

Circuit breaker

Limit concurrent load on a failing or slow instance. Configure in DestinationPolicy / DestinationRule with circuitBreaker:

• maxConnections, httpMaxPendingRequests: cap connections and pending requests per host.
• When exceeded, the circuit “opens” and requests are rejected (or retried elsewhere if retry is configured) instead of piling up on that instance.

This prevents one bad pod from soaking all traffic and degrading the caller.

Pool ejection (outlier detection)

If a host returns errors (e.g. 5xx), Istio can eject it from the load-balancing pool for a sleep window (e.g. 15s). After that, the host is tried again. Settings include httpConsecutiveErrors, httpDetectionInterval, httpMaxEjectionPercent. Healthy pods keep serving; misbehaving ones are temporarily removed.

Combining resiliency

Use circuit breaker + pool ejection + retry together: circuit breaker limits concurrency per instance; pool ejection removes bad instances; retries send the request to another instance when the circuit is open or an instance is ejected. The book shows that with these in place, 503s from a misbehaving pod can be eliminated because traffic is retried to healthy pods.

---

Chaos testing

Chaos engineering is experimenting on the system to build confidence that it withstands failures. Istio injects faults at the proxy so you don’t need to change app code.

HTTP errors

Use a RouteRule / VirtualService with httpFault.abort: percent (e.g. 50) and httpStatus (e.g. 503). A fraction of requests to that destination get the synthetic error. You can then verify retries, circuit breakers, and user-facing behavior.

Delays

Use httpFault.delay: percent and fixedDelay (e.g. 7s). A fraction of requests are delayed before reaching the service. This simulates slow dependencies and helps verify timeouts and SLAs. The delay is applied in the proxy (Envoy), not in the application.

---

Observability

Tracing

Istio sends spans from the sidecars to a tracing backend (e.g. Jaeger, Zipkin). A trace is the path of a request through the mesh (a DAG of spans). To get complete traces, applications should forward tracing headers on outbound calls (e.g. x-request-id, x-b3-traceid, x-b3-spanid, x-b3-parentspanid, x-b3-sampled). The book’s sample uses a small interceptor to propagate these headers; no tracing SDK is required in the app for basic support.

Metrics

Mixer collects telemetry from proxies. With Prometheus and Grafana (or other backends) you get request rates, latencies, and error rates per service. The Istio dashboard in Grafana gives a view of the mesh. Metrics are language-agnostic because they are reported by the proxy.

---

Security

mTLS

Istio Auth (Citadel) issues certificates to workloads. Sidecars use mTLS so traffic between services in the mesh is encrypted and authenticated. This is transparent to the application.

Access control (blacklist and whitelist)

The book uses Mixer for ACLs:

• Blacklist: Explicitly deny certain paths. Example: deny requests from customer to preference (match on source.labels["app"] and destination.labels["app"]), return 403. Config uses denier handler, checknothing instance, and a rule that matches and invokes the handler.
• Whitelist: Deny by default; allow only certain sources. Example: only allow recommendation to call preference; customer → preference is denied. Config uses listchecker (with listentry for the source) and a rule.

In current Istio, AuthorizationPolicy is the main way to enforce allow/deny between workloads; the idea (identity-based and path-based access) is the same.

---

Summary

Area	What Istio provides
Architecture	Data plane (Envoy sidecar per pod) + control plane (Pilot, Mixer, Auth).
Kubernetes	Uses Pods (sidecar), Services/Endpoints (discovery); CRDs (VirtualService, DestinationRule, Gateway); sidecar injection per namespace.
Traffic	Weighted and header-based routing, canary, dark launch (mirroring), egress control.
Resiliency	Load balancing (round-robin, random, least_conn), timeout, retry, circuit breaker, pool ejection.
Chaos	Inject HTTP errors and delays at the proxy.
Observability	Tracing (Jaeger/Zipkin) and metrics (Prometheus/Grafana) from the proxy; propagate headers for full traces.
Security	mTLS between services; ACLs (blacklist/whitelist) → today’s AuthorizationPolicy.

Istio lets you offload traffic management, resiliency, observability, and security to the mesh so application code can stay focused on business logic. The APIs have evolved (RouteRule → VirtualService, DestinationPolicy → DestinationRule, EgressRule → ServiceEntry, Mixer ACLs → AuthorizationPolicy); the concepts in this post still apply. For up-to-date YAML and commands, see the Istio docs and the istio-tutorial (or the official Istio examples).

---

References

• Introducing Istio Service Mesh for Microservices, Christian Posta & Burr Sutter, O’Reilly — installation, traffic control, resiliency, chaos testing, observability, and security with Istio on Kubernetes/OpenShift.
• Istio documentation
• Principles of Chaos Engineering
• Jaeger — distributed tracing