Concurrency & Lifecycle

AgentCore is designed to serve many concurrent conversations from one instance per event loop. This page is the contract: what is safe to share, what belongs to a single call, and what the application must supply.

The Short Version

Loop-safe, not thread-safe. Share one AgentCore across tasks of the same event loop. Do not share it across OS threads or event loops. This is the permanent contract, not a limitation planned for removal.
One agent per event loop — in practice, one per thread. Each thread that serves work owns its loop, its agent, its bindings, and its MCP client. Providers, MCP sessions, and media jobs are loop-bound. A binding remembers the loop of its first use and raises LoopAffinityError on a second one instead of failing inside the HTTP client. aclose() releases the claim when the binding can rebuild its provider; a caller-supplied loop-bound provider keeps it, because it stays open on its original loop. In-process local providers are the exception — see Threads and Multiple Event Loops.
Conversation state is per call. Pass conversation= on every call. Nothing about a conversation is stored on the agent.
Deadlines are per run. run_timeout bounds one whole run — preamble, model calls, tools, failover, event handlers. Expiry raises RunTimeoutError; the caller’s cancellation is never converted into it.
Credentials rotate explicitly. await agent.arefresh_credentials() rebuilds providers with a fresh credential; calls already in flight keep the provider they captured.
Configure before serving. Bindings, tools, skills, and MCP servers are shared, read-mostly state. Do not mutate them while calls are in flight. Each call reads the registry once when it starts: a tool registered mid-call appears on the next call, not the current one.
No built-in backpressure. Admission control, per-tenant quotas, and shutdown draining are the application’s job.

Binding State at a Glance

Two diagnostics make the loop contract observable without a debugger. binding.owner_loop is the event loop the binding first used (or None), and repr(binding) ends with loop=<id> — the loop’s id() in hex — and built=True/False. Neither ever shows the API key.

Provider source	`built` after first use	`owner_loop`	Who closes it
Definition-built (`api_key` / `base_url` / `provider_name`)	`True`	Set on first use; `None` before; released by `aclose()`	The binding — `aclose()` closes it; `arefresh()` closes a retired provider
Caller-supplied `provider=`	`False`	Set on the first loop-bound use and kept; stays `None` for a `loop_agnostic` provider	The caller — core never closes it

Moment	What happens	`built`	`owner_loop`
Construction	Definition stored; nothing built, no loop touched	`False`	`None`
First use — `chat()` / `stream()` / a capability call / `preflight()` / `build_provider()`	Provider built if definition-built; the running loop is claimed if the provider is loop-bound	`True` (built) / `False` (supplied)	The running loop, or `None` for a loop-agnostic provider
`aclose()`	Built providers are closed; a supplied provider stays open	`False` again	Released for a rebuildable binding or a loop-agnostic provider; kept for a supplied loop-bound provider

When a second loop is rejected, LoopAffinityError carries the same diagnostics: exc.model, exc.owner_loop, and exc.current_loop (the loop that raised). preflight() keeps them on the labeled error it raises.

Why Not Thread-Safe

Thread safety is deliberately out of scope. It is not something a few locks can add, because the state an agent caches is bound to the event loop that created it:

State	Why it cannot cross threads
Cached providers (`ModelBinding`)	`AsyncOpenAI` / `AsyncAnthropic` wrap `httpx.AsyncClient`; its connection pool and locks belong to one loop
`MediaJob`	A handle around an `asyncio.Task`; only its creating loop can drive or cancel it
Event handlers	Async handlers are awaited on the calling loop
`MCPClient`	Calls open an ephemeral session on the loop that runs them; safe per call, but live sessions are not shareable

A “thread-safe” wrapper would have to own a private loop and marshal every call into it. That is a deployment concern, not core behavior — the owner-loop pattern below shows how to build one in the application if a host cannot give each thread its own loop.

The rule is about the loop, not only the thread: an agent and its providers must be created and used within one loop’s lifetime. Do not build an agent on the main thread and hand it to worker threads, and do not cache a thread-local agent across asyncio.run() calls — each asyncio.run() creates and closes a new loop.

Model Calls Are Async

The built-in transports use the SDKs’ async clients (AsyncOpenAI and AsyncAnthropic), which run on httpx.AsyncClient. No model request is moved to a worker thread, so a slow upstream does not consume a tool-executor slot. The synchronous parts are message shaping (small), your own synchronous tools (run in the executor), and media loading (done by the caller — load a URL into MediaContent before calling chat()).

What Is Shared, What Is Per Call

Surface	Scope	Notes
`ModelBindings` and their providers	Shared	Providers are built lazily and cached on the binding; one provider serves every call using that binding
`ToolRegistry`	Shared	Register tools before serving; runtime registration is not synchronized
`SkillManager`	Shared	The catalogue is cached; `reload()` is the explicit refresh. `AgentCore` preloads it in a worker thread before the first skill access
`MCPClient`	Shared	One client per agent; sessions are opened per call, or pooled per server with `session_pool_size`
`messages` / `conversation`	Per call	The caller owns the history list
`extra_tools`, `tool_filter`, `mcp_servers`	Per call	The advertised set is also the execution set
`execution_policy`, permission callback	Per call (overrides)	Policy is ambient around tool execution only
`parallel_tool_calls`, `tool_timeout`, `run_timeout`, `context_compactor`	Per call (overrides)	Constructor values are defaults
`run_id`	Per call	Generated when absent; correlates every event of the run
`skill_state` (activation)	Per call	A skill activated in one run does not affect another
`on_event`	Per call (optional)	Instance and call-site handlers both fire for tool events
Builtin Todo list	Per run	`TodoWrite` / `TodoRead` state is isolated per `chat()` / `stream()` call
Media jobs (`submit_*`)	Per loop	The returned `MediaJob` is bound to the loop that created it

Run Deadline

run_timeout is one wall-clock budget for a whole run — skill preload, MCP start, tool/skill selection, every model call (failover retries included), every tool batch, and event-handler awaits:

agent = AgentCore(bindings, run_timeout=30.0)            # default for every call

response = await agent.chat(prompt, run_timeout=10.0)    # per-call override
response = await agent.chat(prompt, run_timeout=None)    # disabled for this call

Expiry raises RunTimeoutError — a TimeoutError carrying .timeout and .run_id. It is deliberately not a ProviderError, so no failover is attempted; the application decides whether to retry. The caller’s own cancellation still arrives as CancelledError.
run_skill() and chat_structured() own a single clock: the inner chat() calls never restart it.
Streaming starts the clock when the first chunk is pulled. No timer spans a yield: each pull is bounded by the remaining budget, and time the consumer spends parked between pulls still counts. On expiry or abandonment the provider stream is closed.
Synchronous tools run in a worker thread and cannot be killed: the run times out and discards the result while the thread finishes in the background. tool_timeout still bounds each tool on its own.

Event Correlation (`run_id`)

Every public entry point (chat, stream, run_skill, chat_structured, SkillPipeline.run) resolves one run_id — pass your own non-empty string, or let core generate one. It is carried on:

CLIverResponse.run_id;
every InferenceEvent and ToolEvent, beside the per-model-call request_id;
terminal stream chunks (the trace chunk and the final stop / iteration-limit chunk);
pipeline events, PipelineContext.run_id, and PipelineResult.run_id.

request_id stays per model call: one run can contain many requests. Log both — run_id groups a run’s events, request_id pinpoints one inference.

Deploying a Concurrent Server

For an async web framework (FastAPI, Starlette, aiohttp), build the agent once per worker process and share it across requests on that process’s single loop. Never share that instance with a worker thread of the same process. The FastAPI Server guide walks through the full lifespan, endpoint, streaming, and shutdown wiring:

import asyncio

from cliver_core import AgentCore, ModelBindings, presets
from cliver_core.tools import discover_builtin_tools

agent = AgentCore(
    ModelBindings(llm=[presets.deepseek(api_key="...")]),
    builtin_tools=discover_builtin_tools(),
)


async def handle(prompt: str, history: list) -> str:
    response = await agent.chat(prompt, conversation=history)
    return response.message.content or ""


# Application-level admission control — AgentCore does not provide one.
slots = asyncio.Semaphore(100)


async def bounded_handle(prompt: str, history: list) -> str:
    async with slots:
        return await handle(prompt, history)

Settings that keep one worker healthy under load:

Bound max_iterations per call (the default is 50).
Bound the conversation you pass in. For long runs, pass a context_compactor — core ships the seam and an opt-in LlmCompactor, never an automatic policy. See Context Window.
Cap tool output at the tool boundary — large results stay in the message history.
Give each tenant a separate agent if it needs a different tool set, system prompt, or permission policy. Otherwise share the agent and pass per-call options.
Never mutate a shared list or tool registry from a request handler.

Threads and Multiple Event Loops

A provider wraps an async HTTP client bound to one loop. Sharing it with another thread or loop is unsupported. Keep the agent inside the lifetime of one loop:

import asyncio

from cliver_core import AgentCore, ModelBindings, presets


def worker(prompts: list[str]) -> list[str]:
    async def main() -> list[str]:
        # open() is the explicit-lifecycle spelling of the constructor; the
        # context manager closes the agent (and the built providers) on exit.
        async with AgentCore.open(ModelBindings(llm=presets.deepseek(api_key="..."))) as agent:
            return [(await agent.chat(prompt)).message.content or "" for prompt in prompts]

    return asyncio.run(main())

That shape fits a thread that handles one unit of work. If a thread serves requests for its whole lifetime, give it one long-lived loop and create one agent inside that loop — do not call asyncio.run() per request and reuse an agent across the resulting loops.

Local in-process providers are the exception. LocalEmbeddingProvider and LocalRerankProvider own no loop-bound resources — blocking work runs in asyncio.to_thread and state is guarded by threading locks — so they declare loop_agnostic = True. A binding wrapping one does not claim a loop and can stay cached across asyncio.run() calls:

import asyncio

from cliver_core import ModelBinding
from cliver_core.capabilities import embed
from cliver_core.local import LocalEmbeddingProvider

# One process-wide provider; safe to share across threads and loops.
provider = LocalEmbeddingProvider(model="BAAI/bge-base-zh-v1.5", backend="sentence-transformers")
spec = ModelBinding.embed(model="BAAI/bge-base-zh-v1.5", provider=provider)


def embed_in_thread(text: str):
    return asyncio.run(embed(texts=text, binding=spec))


embed_in_thread("hello")
embed_in_thread("world")  # same cached binding — no loop claim, no error

Reuse is safe when the calls do not overlap. The provider object is thread-safe, but the binding’s lease bookkeeping is not synchronized — for concurrent calls from different threads, keep one binding per thread.

Keep one binding spec and need it on a different loop or thread? ModelBinding.for_current_loop() returns a sibling: it shares the model definition, params, and a supplied loop-agnostic provider, and starts with no loop claim. A binding that builds its provider builds a fresh one on the clone’s first loop (never share a loop-bound SDK client). It refuses a caller-supplied provider that owns loop-bound resources — rebuild that provider inside the target loop instead.

The alternative is one loop, many threads: run a single agent on a dedicated loop and submit work with asyncio.run_coroutine_threadsafe(...). For streaming, bridge chunks through a queue.Queue because the async generator cannot be consumed from another thread. The agent still belongs to exactly one loop; the other threads only hand it work. Use this only when a host cannot give each thread its own loop.

The CLI application is single-threaded and single-loop, so this does not affect it.

Startup Validation and Warm-Up

await bindings.preflight() is the cheap startup check: it builds every configured provider (primary and fallbacks) on the calling loop and resolves a credential_provider once, so a missing base_url or credential fails at startup instead of on the first request. It makes no model call, does not contact MCP servers, and is idempotent — a provider that already built is not rebuilt or closed. agent_lifespan() from cliver_core.integrations.lifespan runs it for you. Lazy construction remains the default; call preflight explicitly when startup validation pays off.

Choosing a startup strategy

Both helpers only move a configuration failure earlier — they add no capability. Pick the mode that matches who supplies the configuration and when a failure should surface:

Mode	Use when	Failure surfaces
`agent_lifespan()`, or explicit `preflight()` in your lifespan	A long-lived server owns a long-lived agent, and its configuration is complete at startup. A bad endpoint or missing key is a deployment error.	Startup: the process refuses to serve
Lazy construction only (the default)	A script, CLI, test, or short-lived job. The first call may pay the provider build.	The first call that needs the provider
Application-managed validation	Bindings are built per call or per tenant, or an absent credential is an expected state — a development or demo environment — that the app reports in its own UI.	The first call, routed through the app’s own error path

The third mode is not a workaround. An app that builds a binding per request — one key or endpoint per tenant, for example — has nothing to preflight at startup, and failing the whole process for a credential that may never be used would be wrong. Check what the application can check (a key present in settings, a workspace selected), then let the first call fail; the capability functions and AgentCore already raise the provider’s message. Do not call preflight() on a server whose configuration is incomplete by design — it builds every configured binding, so one missing credential refuses startup even for requests that never touch that purpose.

The first real call still does one-time work that is otherwise paid by a user request:

builds the provider(s) for the bound purpose (skipped if preflight already did);
starts the MCP client and discovers its tools;
loads the skill catalogue from disk (off the loop, in a worker thread).

Run a small warm-up call at startup, after configuration is final:

await bindings.preflight()  # endpoints and credentials, no network call
await agent.chat("Reply with OK.", max_iterations=1, skill_dispatch=False)

Warm-up also surfaces unreachable MCP servers before traffic arrives.

Tool Execution Under Load

Synchronous tools (all builtins) run via asyncio.to_thread. The default executor is per loop and limited to min(32, cpu_count + 4) threads for the whole process. If a run needs many tool calls, they queue behind other runs.
tool_timeout includes queued time. A call that waits for an executor slot and then runs is bounded by one wall-clock timeout. Python cannot kill a running thread: a timed-out synchronous tool keeps running in the background, and only its result is discarded.
Async @tool functions run on the agent’s loop (execute_async), so tool_timeout and caller cancellation reach them, and they may use loop-bound clients and objects. Tool bodies are still caller code: never block the loop inside an async tool.
MCP tool calls run on the agent’s loop. By default every call opens an ephemeral session: a stdio server starts a subprocess per session; streamable HTTP opens a new client and session per call. A streamable-http server configured with session_pool_size=N keeps up to N sessions open and leases each exclusively, so at most N calls run at once and a further call waits for an idle session instead of dialing again — see MCP session pooling. Size the pool per tenant and drain calls before mcp_client.close().

parallel_tool_calls=True is opt-in per agent or per call. It only makes sense for independent tools: permission checks still run sequentially in tool-call order, and results are appended in the original order. It does not raise the executor size.

Permission Callbacks and Event Handlers

The permission callback is consulted for ASK tools. It runs in the calling task:

Make it non-blocking. The CLI reads terminal input in a worker thread; a server should use a non-interactive policy instead of reading stdin.
It is serialized within one run’s batch, not globally. Concurrent runs can invoke the callback concurrently, so it must be concurrency-safe.
An exception fails closed — the tool is denied.

Tool events reach both the instance on_event= and the call-site on_event=. Inference events are emitted by the provider and are observed through the binding’s on_event= (they do not reach AgentCore’s handler). Handlers may be sync or async:

Do not block the loop inside a handler.
Handlers can run concurrently for different calls; treat shared state accordingly.
Handler exceptions propagate and abort the call on both surfaces — a raise from a tool handler is never converted into a tool-error result, exactly like a raise from an inference handler. A shared handler should be defensive.

Lifecycle and Shutdown

When the agent’s lifetime is lexical — a script, a test, a worker thread’s unit of work — use async with AgentCore.open(bindings) as agent:; it awaits aclose() on normal and exceptional exit. A server usually stops admitting work and drains in-flight runs first:

# 1. Stop accepting new work, then drain in-flight chat()/stream() tasks.
await asyncio.gather(*in_flight, return_exceptions=True)

# 2. Close the agent: cancels and drains the media jobs it started, then
#    closes the providers its bindings built.
await agent.aclose()

# 3. Close the MCP client you created — the agent never owns it.
await mcp_client.close()

agent.aclose() is idempotent and single-flight: concurrent callers await the same shutdown pass. It closes providers the agent’s bindings built — a caller-supplied provider= is never closed.
Media jobs started with submit_* are cancelled and awaited before the providers close. A job whose body ignores cancellation delays close; there is no way to force-kill a task.
Bindings can be shared by several agents. Closing one agent closes a provider its bindings built for all of them; give each agent its own bindings, or close only after every agent is done.
AgentCore does not close an injected MCPClient. Whoever built it closes it.
mcp_client.close() also stops every server’s session pool and closes its sessions — including leased ones, which is how the SDK cancels an in-flight call. Drain the calls first when their errors must not be observed.
A closed agent rejects every public entry point (chat, stream, run_skill, chat_structured, embedded model calls, submit_*) with RuntimeError. Build a new one for a new loop.

Standalone capabilities

A capability call used without an agent owns its binding’s lifetime. ModelBinding and ModelBindings are async context managers that await aclose() on normal and exceptional exit — the same lexical pattern as AgentCore.open:

from cliver_core import ModelBinding
from cliver_core.capabilities import generate_image

async with ModelBinding.image(
    model="gpt-image-1",
    api_key="...",
    base_url="https://api.openai.com/v1",
) as image:
    result = await generate_image(prompt="a red panda on a skateboard", binding=image)

Ownership is the same as aclose(): providers the binding built are closed, a caller-supplied provider= stays open. The context-manager form fits a binding whose lifetime is one operation or one script; a server that reuses providers across requests builds the bindings once and calls await bindings.aclose() at shutdown instead.

Streaming

Consume a stream from one task and in the same loop that created the agent:

async for chunk in agent.stream(prompt):
    if chunk.content:
        await send(chunk.content)

Do not share or concurrently iterate one async generator. Cancelling the consuming task propagates asyncio.CancelledError; run the normal shutdown drain rather than dropping the task silently.

Multi-Tenant Isolation

The core isolates per-call state: messages, skill activation, tool selection, policies, and the builtin Todo list. It does not isolate shared configuration. For tenants that must not see each other’s tools or prompts, build a separate AgentCore per tenant (often per loop) with its own bindings, registry, and policy; for tenants that differ only by request parameters, share the agent and pass options per call.

The advertised tool set is an execution boundary: a tool that is filtered out, disallowed by a skill’s allowed-tools, or belongs to an MCP server not selected for the call returns an error if the model tries to call it — and never triggers a permission prompt. It is not a sandbox for the tool body itself; use the permission callback and execution policy for that.

Credential Rotation

Provider clients bake their credential at construction. A binding built with an expiring key can be rotated explicitly:

from cliver_core import AgentCore, ModelBinding, ModelBindings

binding = ModelBinding.llm(
    model="deepseek-chat",
    protocol="openai",
    api_key=initial_key,
    credential_provider=load_current_key,   # sync or async, no arguments
)
agent = AgentCore(ModelBindings(llm=binding))

# Later — on a TTL timer, or after catching a ProviderError:
await agent.arefresh_credentials()          # refreshes every binding on the agent

credential_provider() is resolved first and must return a non-empty string. An exception or a bad value raises ValueError before anything changes; the old provider and key stay in use.
The replacement provider is built before the swap. Every model, capability, and System One call holds an in-flight lease on the provider it captured, so a call already running finishes undisturbed. A retired provider with no active lease is closed inside the refresh itself; one still serving a call is closed by the first refresh after that call finishes. Frequent rotation on a long-lived server therefore does not accumulate providers, and aclose() closes whatever remains.
If you use binding.build_provider() directly, wrap it — async with binding.lease_provider() as provider: — so a concurrent refresh cannot close it under an in-flight call.
A caller-supplied provider= is never rebuilt or closed — only api_key (used by media paths) is updated.
There is no automatic refresh on 401/403. The application owns the policy: catch a retryable ProviderError (or track the credential’s TTL) and call arefresh() / arefresh_credentials() itself.
Per tenant, keep one binding fed by that tenant’s secret source and refresh it on the tenant’s schedule:

async def refresh_all(tenants: dict[str, AgentCore]) -> None:
    for agent in tenants.values():
        await agent.arefresh_credentials()

Failure and Retries

SDK clients retry transient HTTP failures (the openai and anthropic SDKs default to two retries), then AgentCore may fail over to the next same-purpose binding on a retryable provider error. Every new call starts again at the primary binding.
There is no global circuit breaker or rate limiter in the core. If many concurrent runs hit one upstream limit, they retry and fail independently.
A tool error is a normal result; five consecutive tool errors or permission denials stop that run with a failure response. One run’s streak never affects another.

SDK Client Timeout and Retries

Both builtin SDK clients default to an SDK read timeout of 600 seconds and two retries — far larger than a typical request budget. A hung primary can spend the whole run_timeout before failover ever sees an error. Every binding factory (llm, image, video, tts, asr, embed, rerank, system_one) accepts two client-construction options:

binding = ModelBinding.llm(
    model="deepseek-chat",
    protocol="openai",
    api_key=key,
    client_timeout=30,        # seconds, every HTTP phase; None = SDK default (600 s read)
    client_max_retries=0,     # transport retries; None = SDK default (2)
)

These are SDK-client settings, not request parameters. A timeout inside params is the per-request value the SDK accepts in the request body — different knob, same word.
They apply to every call made through the binding — chat, streaming, media, embeddings, rerank, System One — and are re-applied when arefresh() rebuilds the provider.
Transport retries (this option) happen inside one model call; binding failover (D39) happens after a retryable ProviderError escapes it. Set client_max_retries=0 when you want the failover to react immediately.
They are rejected with ValueError on a supplied provider= (configure that provider’s client directly) and on provider classes without a builtin SDK transport.