TCP Connection Budget
TCP Connection Budget
FastBee device-side web server is built on ESPAsyncWebServer + AsyncTCP, whose concurrency is limited by the lwIP TCP PCB table. Different chips have different resource capabilities, so connection budgets must be configured differently, otherwise pages may fail to load or SSE push may be lost.
Connection allocation follows these principles:
TCP_TOTAL = SSE (persistent push) + HTTP (page/API requests)- SSE: Reserved slots for
AsyncEventSource(real-time status push) - HTTP: Remaining slots for concurrent page loading and API calls
- Budget constants: Defined in
include/core/ResourceProfile.h - SSE limit:
MAX_SSE_CLIENTSfixed array inSSERouteHandler.h - TCP exhaustion threshold:
TCP_CONN_EXHAUSTION_THRESHOLDinWebConfigManager.hcontrols soft-restart trigger
Note: AsyncTCP v3.4.10 no longer reads
CONFIG_ASYNC_TCP_MAX_CONNECTIONSmacro; it is a legacy configuration. Actual connection count is determined by lwIPMEMP_NUM_TCP_PCB(default 16) and application-layer thresholds.
Chip Hardware Specifications
| Chip | Core | Internal SRAM | PSRAM | WiFi |
|---|---|---|---|---|
| ESP32 | Dual-core Xtensa 240MHz | 520 KB | None | 802.11 b/g/n |
| ESP32-S3 | Dual-core Xtensa 240MHz | 512 KB | 8 MB | 802.11 b/g/n |
| ESP32-C6 | Single-core RISC-V 160MHz | 512 KB | None | 802.11 ax (Wi-Fi 6) |
| ESP32-C3 | Single-core RISC-V 160MHz | 400 KB | None | 802.11 b/g/n |
Memory Budget Calculation
lwIP default configuration (all ESP32 variants, from ESP-IDF sdkconfig):
| Parameter | Default | Description |
|---|---|---|
MEMP_NUM_TCP_PCB | 16 | lwIP TCP PCB hard limit (all variants, cannot exceed) |
TCP_SND_BUF | 5744 B | Per-connection send buffer |
TCP_WND | 5760 B | Per-connection receive window |
| TCP PCB struct | ~172 B | lwIP internal control block |
| Per-connection total | ~11.7 KB | PCB + SND_BUF + WND |
Single-user access scenario analysis (browser single host max 6 concurrent):
| State | MQTT | SSE | HTTP | TIME_WAIT | Total PCB |
|---|---|---|---|---|---|
| Page load peak | 1 | 1 | 5-6 | 2-4 | 9-12 |
| Steady state | 1 | 1 | 0-2 | 1-2 | 3-6 |
| Multi-tab (S3) | 1 | 2 | 4-6 | 3-5 | 10-14 |
Connection Budget by Chip
| Chip | TCP Budget | SSE | HTTP | Exhaustion Threshold | Status |
|---|---|---|---|---|---|
| ESP32 classic | 6 | 1 | 5 | 12 | ✅ Implemented |
| ESP32-S3 | 8 | 2 | 6 | 14 | ✅ Implemented |
| ESP32-C6 | 6 | 1 | 5 | 12 | ✅ Implemented |
| ESP32-C3 | 4 | 1 | 3 | 10 | ✅ Implemented |
Exhaustion threshold must be < 16 (lwIP hard limit), triggering recovery before limit, leaving 2-6 slot buffer. Memory-constrained chips (C3) trigger earlier, memory-rich (S3) tolerate more connections.
PSRAM Impact on TCP Connection Capacity
ESP32-S3 with PSRAM significantly reduces internal DRAM contention, indirectly supporting more concurrent TCP connections:
| Scenario | No PSRAM | With PSRAM (threshold=512B) |
|---|---|---|
| Internal DRAM available | ~18 KB | ~22-35 KB (HTTP buffers offloaded to PSRAM) |
| lwIP TCP PCB space | Tight, easily exhausted | Sufficient (each PCB ~172B, must be in internal DRAM) |
| Concurrent HTTP requests | 2-3 triggers memory pressure | 6+ still handled normally |
CONFIG_ASYNC_TCP_MAX_CONNECTIONS=14(S3) is a build flag documentation; AsyncTCP v3.4.10 no longer reads it. Actual connection count constrained by lwIPMEMP_NUM_TCP_PCB=16and application-layerTCP_TOTAL_BUDGET=8. Raising this flag mainly serves as configuration intent documentation and backward compatibility with older AsyncTCP.
Key Files
| File | Purpose |
|---|---|
include/core/ResourceProfile.h | TCP_TOTAL_BUDGET / TCP_SSE_BUDGET / TCP_HTTP_BUDGET budget constants (chip-conditional compilation) |
include/network/handlers/SSERouteHandler.h | MAX_SSE_CLIENTS fixed array slot count |
include/network/WebConfigManager.h | TCP_CONN_EXHAUSTION_THRESHOLD soft-restart threshold; CHECK_INTERVAL_MS / TIME_WAIT_PRUNE_* pruning parameters; NO_REQUEST_WATCHDOG_MS no-request watchdog |
src/network/WebConfigManager.cpp | Proactive abort TIME_WAIT connections every 30s + threshold-based TIME_WAIT pruning + no-request watchdog + exhaustion detection and auto-recovery |
src/network/WebHandlerContext.cpp | Static file concurrency limiting: STATIC_MAX_INFLIGHT / STATIC_INFLIGHT_WATCHDOG_MS / small-resource exemption / 503 auto-retry page |
web-src/index.html | Frontend boot chunk serial loader: interval / start delay / backoff retry parameters |
platformio.ini | CONFIG_ASYNC_TCP_MAX_CONNECTIONS in each [xxx_runtime_flags] (legacy flag, AsyncTCP 3.4.10 no longer reads, S3 raised to 14) |
Management Measures
TIME_WAIT Connection Periodic Cleanup
After ESPAsyncWebServer closes a connection, TCP PCB enters TIME_WAIT state (default 2×MSL ≈ 60s), still occupying slots during this period. WebConfigManager traverses lwIP PCB linked list every 30 seconds, proactively aborting timed-out TIME_WAIT connections:
// WebConfigManager.cpp - Clean up TIME_WAIT every 30s
if (millis() - lastTimeWaitCleanup > 30000) {
lastTimeWaitCleanup = millis();
// abort TIME_WAIT connections to free TCP PCB slots
}TIME_WAIT Proactive Pruning
Web responses force Connection: close short-lived connections. Under dense access each request produces a TIME_WAIT PCB (lwIP keeps it for 2×MSL ≈ 120s), quickly exhausting MEMP_NUM_TCP_PCB=16 and stalling new requests. In addition to the 30s periodic cleanup above, WebConfigManager runs a TCP health check every 2s and prunes immediately when TIME_WAIT exceeds the threshold:
| Parameter | Value | Description |
|---|---|---|
CHECK_INTERVAL_MS | 2000ms | TCP health check interval |
TIME_WAIT_PRUNE_THRESHOLD | 4 | Start pruning when TIME_WAIT exceeds this |
TIME_WAIT_PRUNE_KEEP | 1 | TIME_WAIT count kept after pruning |
Pruning kills only the oldest TIME_WAIT (same strategy as lwIP tcp_kill_timewait). In this state tcp_abort removes the PCB without sending RST, so there is no side effect on already-closed connections.
No-Request Watchdog and SSE Protection
When isRunning=true but no HTTP request has arrived for 120s (NO_REQUEST_WATCHDOG_MS) while residual TCP connections exist, the watchdog reaps stale PCBs on port 80 in place (no riskier soft-restart). Two protection rules prevent false kills:
- Skip reaping while SSE clients are connected: SSE is a legitimate long-lived connection and does not count as HTTP requests. Long periods without HTTP requests are normal in this state, and aborting would cut the browser's in-flight request chain.
- Authenticated requests record activity uniformly: many handlers send responses via the chunked/stream paths of
HandlerUtils, which do not updatelastRequestSeenMs. ThereforerequireAuth()callsnoteWebRequestActivity()after successful authentication, preventing the watchdog from misjudging "no access".
Rule-query handlers read the rule table while holding a lock on the async_tcp task. They must use bounded waits (MutexGuard with an explicit timeout, e.g. 2000ms) and return 503 on timeout. The default portMAX_DELAY infinite wait is forbidden — an infinite wait while rule execution holds the lock would freeze all HTTP requests.
SSE Connection Limit
SSERouteHandler uses fixed array _slots[MAX_SSE_CLIENTS] to track clients. ESP32-S3 sets MAX_SSE_CLIENTS=2 (multi-tab support), other chips set MAX_SSE_CLIENTS=1 (single SSE connection). New connections are rejected when slots are full, preventing SSE from filling TCP PCB and blocking HTTP requests.
Static File Concurrency Limiting
A browser's initial navigation bursts ~12 resource requests (HTML document + JS chunks + CSS + images + API), and every response forces Connection: close, so resources can only queue with serial / low concurrency. Historically there were failures in both directions:
- Too much concurrency: 4-way concurrent streaming of a 128KB CSS exhausted DRAM (largest free block dropped to 220B),
operator newfailed and abort() sent the device into a crash loop - Limit too strict: with the cap set to 1, the first-screen burst only admitted 1 request; the rest were RST/timed-out or rejected with 503 "Static file busy", and rejecting the main document left the browser on a blank JSON screen
The current scheme (WebHandlerContext.cpp) balances the two:
| Parameter | Value | Description |
|---|---|---|
STATIC_MAX_INFLIGHT | 2 | In-flight cap for compressible resources (html/js/css); current resources are all ≤22KB gz, so 2-way concurrency stays safely within the 36KB DRAM gate |
STATIC_INFLIGHT_WATCHDOG_MS | 12000 | Stuck-counter watchdog: when onDisconnect never fires and the counter leaks, force-reset after 12s to avoid permanent 503 |
| Small-resource exemption | Uncompressed resources such as favicon/logo | bypassInFlightCap = !isCompressible; a few-KB single files do not recreate the historical crash scenario (large-file concurrent streaming), and first-screen essentials are no longer rejected |
| Slot release | onDisconnect | Connection: close guarantees release on completion/timeout/abort |
Rejection responses differ by resource importance to avoid the 503 blank screen:
| Request type | Rejection response | Recovery |
|---|---|---|
Core entry documents (/www/index.html, /www/setup.html incl. gz) | 503 + text/html auto-retry page (<meta http-equiv="refresh" content="1">) | Browser automatically re-navigates every second until recovery |
| Other static resources | 503 + JSON (retryAfter: 1) | Frontend loader onerror retries with incremental backoff |
Frontend loader (web-src/index.html) timing parameters paired with the server-side limiter:
| Parameter | Current | Old | Description |
|---|---|---|---|
| Boot chunk interval | 30ms | 80ms | 5 chunks load serially; the interval adds directly to first-screen time |
| Start delay | 20ms | 100ms | Loading begins after DOMContentLoaded |
| Failure retry backoff | 300/1200/2700ms | 500/2000/4500ms | 300×attempt², up to 4 attempts (CHUNK_MAX_RETRY) |
Measured after the fix (ESP32-F4R0, login page with ~12 resources): the login form is fully visible in ~2.8s (DOM load ≈ 2763ms), meeting the 3-second goal; when the main document is throttled there is no blank screen — the auto-retry page probes recovery every second.
Modification notes:
- Before raising
STATIC_MAX_INFLIGHT, confirm that the largest gz resource size × concurrency still fits the 36KB DRAM gate; measured 4-way 128KB concurrency caused a crash loop — do not raise it blindly - The watchdog value must exceed the legitimate transfer duration of a single file (seconds on weak networks), otherwise valid transfers get killed
- All core entry rejection paths (2 DRAM gates + 1 concurrency limit) must go through
sendStaticBusyRetryPage(); falling back to JSON brings back the 503 blank screen. The constants and behaviors above are locked bytest/test_static_concurrency.cpp; runpio test -e nativeregression after changes - The frontend loader timing parameters are also locked by
test_static_concurrency.cpp; before slowing the backoff, re-measure the device's actual rejection rate with a concurrency probe script
Modification Notes
- Any threshold modification must ensure
TCP_CONN_EXHAUSTION_THRESHOLD < 16(lwIP PCB hard limit) - After modifying static-limit constants (
STATIC_MAX_INFLIGHT/STATIC_INFLIGHT_WATCHDOG_MS) or frontend loader timing parameters, update and regress the locked assertions intest/test_static_concurrency.cpp - When tuning
TIME_WAIT_PRUNE_THRESHOLD/TIME_WAIT_PRUNE_KEEP/CHECK_INTERVAL_MS, also check the watchdog constant regression assertions intest_mqtt_protocol.cpp - After modifying the no-request watchdog reap logic, keep the SSE-active skip branch (locked by
test_mqtt_protocol.cpp) - After modifying
TCP_TOTAL_BUDGETinResourceProfile.h, must also update:MAX_SSE_CLIENTSinSSERouteHandler.h(≤TCP_SSE_BUDGET)TCP_CONN_EXHAUSTION_THRESHOLDinWebConfigManager.hCONFIG_ASYNC_TCP_MAX_CONNECTIONSin each[xxx_runtime_flags]ofplatformio.ini- Expected values in
scripts/validate-build-matrix.js
- ESP32-C3 has smallest RAM (400KB), TCP should not exceed 4
CONFIG_ASYNC_TCP_MAX_CONNECTIONSis deprecated by AsyncTCP 3.4.10, but retained as configuration documentation and backward compatibility