Nginx In Deep — quyennv.com

NGINX is a high-performance web server designed to handle many simultaneous connections with very few system resources. It was created to address the C10K problem—designing a web server to handle 10,000 concurrent connections—through an event-based connection-handling model. This post is an in-depth guide drawn from Mastering NGINX (Dimitri Aivaliotis, Packt): installation, configuration structure, reverse proxy, upstream load balancing, HTTP server and locations, mail proxy, and troubleshooting.

Installing NGINX

Package manager

Debian/Ubuntu: sudo apt-get install nginx
RHEL/CentOS: Add the NGINX repository, then sudo yum install nginx
FreeBSD: sudo pkg_install -r nginx

Package installs use standard OS paths. For the latest stable (or development) build, use the official NGINX packages or compile from source.

Building from source

NGINX is modular. You can enable or disable features at compile time.

Requirements: compiler, PCRE (for rewrite/regular expressions), OpenSSL (for SSL). Optional: zlib (gzip), libatomic.

# Download from http://nginx.org/en/download.html
tar xzf nginx-<version>.tar.gz && cd nginx-<version>
./configure --prefix=/opt/nginx --user=www --group=www \
  --with-http_ssl_module --with-http_realip_module \
  --with-http_stub_status_module --with-pcre=... --with-openssl=...
make && sudo make install

Useful configure options:

Option	Purpose
`--prefix=<path>`	Installation root.
`--user`, `--group`	User/group for worker processes.
`--with-http_ssl_module`	HTTPS support (requires OpenSSL).
`--with-http_realip_module`	Use client IP from headers (e.g. behind L7 LB).
`--with-http_stub_status_module`	Expose status/metrics for monitoring.
`--with-mail`, `--with-mail_ssl_module`	Mail proxy (POP3/IMAP/SMTP).
`--without-http_*`	Disable unneeded modules to slim the binary.
`--add-module=<path>`	Third-party module (e.g. ngx_lua).

Branches: stable keeps the same internal API for third-party modules; mainline has newer features but may break module compatibility.

Configuration format and structure

The config is block-based. Each directive line ends with ;. Braces { } define a context.

<section> {
    <directive> <parameters>;
}

Global context

No surrounding braces. Directives here affect the whole server.

Directive	Meaning
`user <user> [group];`	User (and optional group) for worker processes.
`worker_processes <n>;`	Number of workers. Often set to CPU cores (CPU-bound) or 1.5–2× for I/O-bound.
`error_log <path> [level];`	Error log path; level: `debug`, `info`, `notice`, `warn`, `error`, `crit`, `alert`, `emerg`.
`pid <path>;`	PID file of the master process.
`worker_rlimit_nofile <n>;`	Max open files per process (tune if you hit “too many open files”).

events { } configures connection processing:

Directive	Meaning
`worker_connections <n>;`	Max simultaneous connections per worker. Total capacity ≈ `worker_processes × worker_connections`.
`use <method>;`	Connection method (e.g. `epoll`, `kqueue`, `/dev/poll` on Solaris). Usually auto-detected.

Example:

user www;
worker_processes 12;
error_log /var/log/nginx/error.log;
pid /var/run/nginx.pid;
events {
    worker_connections 2048;
}

Includes

You can split config with include:

include /opt/nginx/conf/mime.types;
include /opt/nginx/conf/vhost/*.conf;

Paths are relative to the main config file if not absolute. Test config: nginx -t -c <path-to-nginx.conf>.

HTTP context and virtual servers

The http { } block holds all HTTP-related configuration. Inside it, server blocks define virtual servers, selected by listen and server_name.

listen

listen [address][:port];
listen port;
listen unix:/path;

Optional parameters: default_server, backlog, ssl, deferred (Linux), etc.

server_name

Literal: server_name www.example.com;
Wildcard: *.example.com, www.example.*
Regex (prefix with ~): server_name ~^www(\d+)\.example\.com$; (captures like $1 usable later)
Empty "": matches requests with no or missing Host header

Virtual server selection order:

Exact IP/port match; 2) Exact server_name string; 3) Leading wildcard; 4) Trailing wildcard; 5) Regex (first match); 6) default_server; 7) First server with matching listen.

Always defining a default_server is recommended so unmatched hosts are handled predictably (e.g. return 444 to close connection):

server {
    listen 80;
    return 444;
}

Locations – where and how requests are handled

Locations live inside server and match the request URI.

location [modifier] uri { ... }
location @name { ... }   # named location: internal redirects only

Location modifiers

Modifier	Behavior
(none)	Prefix match.
`=`	Exact match; search stops.
`~`	Case-sensitive regex.
`~*`	Case-insensitive regex.
`^~`	Prefix match; if chosen, no regex locations are checked.

Selection:

Non-regex locations: “most specific” prefix wins (order does not matter).
Regex: first match in config order.
Comparison uses the decoded URI (e.g. %20 as space).

Location-only directives

Directive	Purpose
`alias <path>;`	Map this location to a filesystem path; alias replaces the matched URI segment.
`root <path>;`	Document root; full path = root + URI.
`internal;`	Only for internal requests (redirects, error_page, rewrite).
`limit_except <method> ...;`	Restrict to certain HTTP methods (GET implies HEAD).

try_files

try_files tries a list of path/URI in order; the first that exists (file or directory) is used. Often used to fall back to a backend or named location:

location / {
    try_files $uri $uri/ @backend;
}
location @backend {
    proxy_pass http://127.0.0.1:8080;
}

Reverse proxy

NGINX terminates client connections and opens new ones to upstream servers. The core directive is proxy_pass.

proxy_pass and URI handling

If proxy_pass includes a URI path (e.g. http://backend/newuri), the request URI is replaced by that path (except in the two cases below).
If the location is a regex, the URI is not replaced; pass the full URI by omitting the path: proxy_pass http://backend;.
If a rewrite in that location changes the URI and is evaluated, the URI is not replaced by the path in proxy_pass; again use proxy_pass http://backend; without path.

Example:

location /uri {
    proxy_pass http://localhost:8080/newuri;   # /uri → /newuri
}
location ~ ^/local {
    proxy_pass http://localhost:8080;           # /local stays /local
}

Essential proxy directives

Directive	Purpose
`proxy_pass <url>;`	Upstream URL.
`proxy_set_header <name> <value>;`	Set request headers to upstream (e.g. `Host`, `X-Real-IP`, `X-Forwarded-For`).
`proxy_redirect off \| default \| <replacement>;`	Rewrite `Location`/`Refresh` from upstream.
`proxy_connect_timeout`, `proxy_send_timeout`, `proxy_read_timeout`	Timeouts for connect, send, read.
`proxy_buffers`, `proxy_buffer_size`, `proxy_busy_buffers_size`	Buffering of upstream response.
`proxy_intercept_errors on;`	Handle upstream 4xx/5xx with `error_page`.

Typical proxy snippet (e.g. in an include):

proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_connect_timeout 30;
proxy_send_timeout 15;
proxy_read_timeout 15;
proxy_buffer_size 4k;
proxy_buffers 4 32k;
proxy_busy_buffers_size 64k;

Use client_max_body_size (e.g. 10m) when clients upload files so the body reaches the upstream.

Legacy apps and cookies

When putting legacy apps behind one host/path, cookie domain/path may need rewriting:

location /legacy1 {
    proxy_cookie_domain legacy1.example.com app.example.com;
    proxy_cookie_path $uri /legacy1$uri;
    proxy_redirect default;
    proxy_pass http://legacy1.example.com/;
}

Upstream and load balancing

Define a group of servers with upstream and reference it in proxy_pass (and similar).

upstream app {
    server 127.0.0.1:9000;
    server 127.0.0.1:9001 weight=2;
    server 10.0.0.2:80 max_fails=3 fail_timeout=30s;
    server 10.0.0.3:80 backup;
    server 10.0.0.4:80 down;
}
location / {
    proxy_pass http://app;
}

Server parameters: weight, max_fails, fail_timeout, backup, down.

Load-balancing algorithms

Algorithm	Directive	Behavior
Round-robin	(default)	Distribute in order; respect `weight`.
IP hash	`ip_hash;`	Same client IP (e.g. class-C) → same server; useful for session affinity.
Least connections	`least_conn;`	Send to server with fewest active connections; can use `weight`.

Keepalive to upstream

Reuse connections to upstream to avoid repeated TCP handshakes. For HTTP, use HTTP/1.1 and clear Connection:

upstream apache {
    server 127.0.0.1:8080;
    keepalive 32;
}
location / {
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_pass http://apache;
}

Reverse proxy: buffering, caching, compression

Buffering: proxy_buffers, proxy_buffer_size, proxy_busy_buffers_size, proxy_max_temp_file_size, proxy_temp_path control how much of the upstream response is buffered in memory (and when spooled to disk). Tuning reduces blocking and improves throughput.
Caching: proxy_cache_path defines a cache zone; proxy_cache, proxy_cache_key, proxy_cache_valid enable and control caching of upstream responses. Use for static or cacheable dynamic content.
Compression: gzip on, gzip_types, gzip_min_length (and optional gzip_static if built with the module) reduce payload size to clients.

HTTP server: files, logging, limits, access

File and I/O

Directive	Purpose
`root <path>;`	Document root for this context.
`sendfile on;`	Use sendfile(2) for efficient file delivery.
`tcp_nopush on;`	Send headers and full packets when using sendfile.
`open_file_cache max=N inactive=time;`	Cache open file descriptors and lookups.
`open_file_cache_valid <time>;`	Revalidate cache entries.
`client_max_body_size <size>;`	Max request body (e.g. uploads); 413 if exceeded.

Logging

Directive	Purpose
`access_log <path> [format];`	Access log path and optional format name.
`log_format <name> <string>;`	Define a format (variables like `$remote_addr`, `$request`, `$status`, `$body_bytes_sent`, `$http_referer`, `$http_user_agent`).
`error_log <path> [level];`	Override error log for this context.

Limits and access

limit_conn / limit_req: Limit connections or requests per key (e.g. per IP).
allow / deny: Restrict access by IP (use inside location or server).
auth_basic: HTTP Basic authentication.

PHP-FPM and FastCGI

Use the FastCGI module to proxy to PHP-FPM (or other FastCGI apps):

location ~ \.php$ {
    fastcgi_pass 127.0.0.1:9000;   # or unix:/run/php/php-fpm.sock
    fastcgi_index index.php;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    include fastcgi_params;
}

Similar patterns apply for uWSGI (uwsgi_pass) and SCGI (scgi_pass).

Rewrite and try_files (replacing “if”-fy configs)

Mastering NGINX and the NGINX rewrite guide recommend:

Prefer try_files over “if” for file/directory checks.
- Bad: using if (-f $request_filename) for existence.
- Good: try_files $uri $uri/ @fallback;
Prefer location blocks over “if” for URI matching.
- Match $request_uri or path with locations instead of if ($request_uri ~ ...).
Prefer server blocks for host-based routing.
- Use different server_name (or listen/default_server) instead of if ($host = ...).
Use “if” only for variable checks when no directive can express the same. Avoid if for hostname or URI switching when possible.

# Prefer:
location / {
    try_files $uri $uri/ @backend;
}
location @backend {
    proxy_pass http://app;
}

Mail proxy module

NGINX can act as a mail proxy for POP3, IMAP, and SMTP (built with --with-mail and optionally --with-mail_ssl_module). It does not store mail; it authenticates the client, then proxies the connection to the appropriate upstream mail server. The upstream is chosen by an authentication service that NGINX calls via HTTP.

Flow: Client → NGINX (listen on 110/143/25, etc.) → NGINX sends auth request (user/pass, protocol, etc.) to auth_http → Auth service returns Auth-Status: OK and Auth-Server / Auth-Port → NGINX proxies to that server.

Example (POP3/IMAP/SMTP with STARTTLS and SSL):

mail {
    server_name mail.example.com;
    auth_http localhost:9000/auth;
    proxy on;
    ssl_prefer_server_ciphers on;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_certificate /path/to/cert.pem;
    ssl_certificate_key /path/to/key.pem;
    # POP3
    server { listen 110; protocol pop3; starttls on; }
    server { listen 995; protocol pop3; ssl on; }
    # IMAP
    server { listen 143; protocol imap; starttls on; }
    server { listen 993; protocol imap; ssl on; }
    # SMTP
    server { listen 25; protocol smtp; }
    server { listen 587; protocol smtp; starttls on; }
    server { listen 465; protocol smtp; ssl on; }
}

The auth service must implement the NGINX mail auth protocol: read headers (e.g. Auth-User, Auth-Pass, Auth-Protocol), validate, and respond with Auth-Status, Auth-Server, Auth-Port (and optionally Auth-User/Auth-Pass for APOP). The book gives a full Ruby/Rack example; the same can be done in any language.

Determining client real IP

When NGINX is behind a load balancer or another proxy, $remote_addr is that proxy’s IP. Use realip (built with --with-http_realip_module) to set the client IP from a trusted header (e.g. X-Forwarded-For or X-Real-IP):

set_real_ip_from 10.0.0.0/8;   # trust proxy
set_real_ip_from 172.16.0.0/12;
real_ip_header X-Forwarded-For;
real_ip_recursive on;           # use last non-trusted as client IP

Error pages and upstream failures

Use error_page to serve custom pages or redirect:

error_page 404 /404.html;
error_page 502 503 504 /50x.html;

With proxy_intercept_errors on, upstream 4xx/5xx are handled by these error_page locations instead of being passed through.

Troubleshooting

Config test: nginx -t (optionally -c /path/to/nginx.conf).
Error log: Set a more verbose level (e.g. info or debug; debug needs --with-debug build) to see why a request was routed or rejected.
Common mistakes: Using if for file existence or host/URI switching instead of try_files and server/location; forgetting default_server; misordering locations or regex.
OS limits: Raise worker_rlimit_nofile and the system file descriptor limit (ulimit -n) when handling many connections; on a mail proxy, watch ephemeral port range for outbound connections.
Stub status: With --with-http_stub_status_module, expose a status location (restrict by IP or auth) to monitor active connections and request counts.

Summary

Area	Takeaways
Install	Package or source; choose modules (http_ssl, realip, stub_status, mail).
Config	Global → events → http → server → location; use include for clarity.
Virtual servers	`listen` + `server_name`; use `default_server`; match order: exact → wildcard → regex.
Locations	Prefix, `=`, `^~`, `~`/`~*`; prefer `try_files` and named locations over “if” for routing.
Reverse proxy	`proxy_pass`, `proxy_set_header`, timeouts, buffering; upstream keepalive for performance.
Upstream	Round-robin, ip_hash, least_conn; server params (weight, max_fails, backup); keepalive.
Mail	auth_http + backend; POP3/IMAP/SMTP with SSL/STARTTLS.
Troubleshooting	`nginx -t`, error_log level, avoid “if” for file/host/URI, tune OS limits and stub_status.

References

Mastering NGINX, Dimitri Aivaliotis, Packt Publishing — in-depth coverage of installation, configuration format, HTTP and mail modules, reverse proxy, upstream, buffering/caching, rewrite/try_files, and troubleshooting.
NGINX documentation
NGINX wiki – 3rd party modules