Performance Optimization

Optimization Principles

Performance optimization should follow the process of benchmarking, monitoring bottleneck location, step-by-step tuning, and regression verification. Do not tune parameters based only on experience, especially JVM settings, database connection pools, Redis memory policies, and MQTT connection limits. Always verify against the real device scale and API peak load.

Key Performance Points

Layer	Key Indicators	Optimization Direction
Nginx	Connections, status codes, response time, 502/504	Workers, timeouts, Gzip, upstream health
Java	CPU, heap memory, GC, thread pools, API latency	JVM parameters, thread-pool isolation, async processing
Redis	Hit rate, memory, slow commands, connections	TTL, warm-up, eviction policy, batch operations
Database	Slow SQL, lock waits, connections, TPS	Indexes, pagination, connection pool, transaction scope
MQTT	Online connections, message rate, authentication latency	Broker cluster, ACL, callback performance
Time-series DB	Write rate, query latency, disk space	Batch write, partitioning, retention period
Video service	Stream success rate, bandwidth, port usage	Bitrate, transcoding, recording directory, network policy

Nginx Optimization

• Adjust worker count according to CPU cores and tune connections according to concurrency.
• Enable reasonable caching for static assets and configure separate timeouts for APIs and WebSocket.
• In long-connection scenarios, watch proxy_read_timeout, upstream health, and connection release.
• For HTTPS, check certificates, TLS versions, session reuse, and cipher suites.

Java and Thread Pools

• In containers, explicitly set heap size, metaspace, direct memory, and GC strategy.
• Separate thread pools for Web requests, device access, rule execution, notifications, and scheduled tasks.
• Queues must be bounded and have rejection policies to avoid memory expansion during traffic spikes.
• Use asynchronous processing for time-consuming operations while keeping traceable logs and retry mechanisms.

Cache and Database

• Cache hot data first, with unified key naming and expiry policies.
• Prevent cache penetration, breakdown, and avalanche. Add null-value cache, distributed locks, and scattered TTL where needed.
• SQL optimization should prioritize indexes, condition filtering, and pagination. Avoid returning large result sets at once.
• When device history write volume is high, write to the time-series database first and set retention periods.

Benchmarking and Capacity Planning

1. Define targets: online device count, message frequency, API QPS, P95/P99 latency, and error rate.
2. Test by layer: Nginx/API first, then MQTT access, rule processing, database writes, and video flow.
3. Record baselines: save hardware specs, container configuration, JVM parameters, database settings, and test scripts.
4. Tune one variable at a time to identify the real optimization source.
5. Run regression verification to ensure features, error rate, resource usage, and data consistency do not regress.

Monitoring Indicators

Object	Suggested Metrics
System	CPU, memory, disk, network, file handles
Nginx	QPS, status codes, response time, upstream errors
Java	GC, heap memory, threads, API latency, exceptions
Redis	Memory, hit rate, slow commands, connections
MySQL	Slow SQL, lock waits, connection pool, TPS, replication delay
EMQX/MQTT	Online connections, message rate, authentication failures, subscriptions
Time-series DB	Write throughput, query latency, disk usage

Common Issues

• Slow pages: check static asset size, Nginx cache, API latency, and browser network waterfall.
• API timeout: check thread pools, database slow SQL, Redis slow commands, and external callbacks.
• Device message backlog: check Broker throughput, rule execution time, database writes, and consumer concurrency.
• Frequent JVM GC: check heap size, object allocation, cached objects, and large result-set queries.
• High database pressure: check indexes, pagination, transaction scope, hot tables, and archive strategy.