The Model Server

Over the last two articles we went through everything needed to run a model: an SDK that runs inference, and a management layer that downloads models, sizes them for your hardware and loads them. You can use the SDK directly from your Go code, but one of the most popular ways of consuming LLMs is through an HTTP API, and that’s what’s left: the server that lets your existing OpenAI client talk to all of it.

It sounds boring, but it isn’t. A regular API computes the answer and then sends it; an inference server sends the headers first and generates the answer token by token, sometimes for minutes. That breaks a lot of the usual HTTP assumptions, and that’s what this article is about.

We’ll follow one request, POST /v1/chat/completions with "stream": true, and stop wherever streaming forces the server to do something special.

📌 A note on scope

Still Kronk v1.32.6, same as the previous article. I’m skipping the browser UI, startup, and the per-endpoint request/response formats.

The framework

Kronk’s server uses its own small web framework, and it has one unusual rule: handlers don’t write the response, they return it. Here’s the liveness check, the simplest handler in the server (checkapp.go:41):

func (a *app) liveness(ctx context.Context, r *http.Request) web.Encoder {
	info := Info{
		Status:     "up",
		Build:      a.build,
		Host:       a.host,
		GOMAXPROCS: a.maxProcs,
	}

	return info
}

There’s no http.ResponseWriter anywhere. The handler gets a context and a request, and returns a web.Encoder (web.go:23): any value with an Encode() method that turns it into bytes and a content type. Info is just a struct whose Encode() marshals itself to JSON. Then a single function, Respond, turns that value into the HTTP response and picks the status code from its type: an error is a 500, nil is a 204, anything else is a 200 (response.go:49).

That’s a nice design. Handlers are easy to test, and no handler can forget to set a status code. It works great for most regular API calls. But this post is about the one call that’s different, the streaming one, and there this approach doesn’t fit well: the first token and the last one can be minutes apart, and the client shouldn’t have to wait for the whole answer to see the start of it.

So the framework has two escape hatches:

• It stores the http.ResponseWriter in the request context (web.go:158). A handler that really needs to write directly can get it back with web.GetWriter(ctx).
• A handler can return NoResponse (response.go:16), which tells Respond: “I already wrote the response, don’t write anything”.

And that is what the chat handler does (chatapp.go:30): it decodes the JSON body, checks that there’s a model field, gets that model from the pool, and passes the response writer to the SDK’s ChatStreamingHTTP, which does the actual streaming. Then it returns NoResponse. The whole endpoint is about thirty lines.

So the interesting part is inside ChatStreamingHTTP. Let’s look at what it does.

Starting the stream

The red box is the key part to understand. Once the server has sent the status line and headers, there’s no way back, and the response is committed: you can’t change a 200 into a 500 anymore, because the client already received it.

Because of that commit point, Kronk does as much validation and preparation as it can before it, in the “BEFORE” box. That way, anything that can fail before inference starts is reported to the client as a regular HTTP error with the right status code. For example, each model only lets a limited number of requests in at once (by default, twice as many as it can run in parallel). When a new request arrives and the model is full, the request waits for a spot to free up, for up to three minutes. If a spot frees up in time, the request goes ahead; if not, it gives up and the client gets a 429, which tells it to retry later. Either way, that wait happens before the headers are sent (concurrency.go:33), so a busy server answers with a proper 429 instead of a 200 followed by nothing.

Once the request is validated and prepared, it’s time to commit: Kronk sends the status line and the headers, and flushes them right away (chat.go:104-109). Flushing means pushing the data to the network now instead of waiting for a buffer to fill up. The first token can take a while, because the model has to read the whole prompt first, and the flushed headers tell the client “your request was accepted, keep waiting”.

At this point the client has received the status and the headers, something like this:

HTTP/1.1 200 OK
Content-Type: text/event-stream

Now we can start sending the text generated by the model. For that, Kronk uses Server-Sent Events (SSE), a very simple format: each event is a data: line followed by an empty line, and lines starting with : are comments. Let’s see an example:

data: {"choices":[{"delta":{"content":"The"}}]}

data: {"choices":[{"delta":{"content":" capital"}}]}

: keep-alive

data: {"choices":[{"delta":{},"finish_reason":"stop"}]}

data: [DONE]

Each event carries the next piece of text. The last one has a finish_reason, and data: [DONE] tells the client the stream is over (chat.go:127). The : keep-alive line is a comment, which clients ignore, and Kronk sends one every fifteen seconds (chat.go:111-116).

Why send a comment nobody reads? Because the client isn’t the only one watching the connection. Reverse proxies and load balancers often close connections that have been quiet for too long; nginx, for example, closes them after sixty seconds by default. A long prompt on slow hardware can easily take longer than that before the first token comes out. A few bytes every fifteen seconds keep the connection looking alive.

That’s the happy path, but what happens when something goes wrong?

Failing after the 200

Failing before the commit was the easy part: we return an error with the right status code and we’re done. But how do we fail when we’ve already told the client that everything is OK? Let’s explore it through this diagram:

There are two kinds of failure after the commit.

On the left, the model fails while the client is still reading, for example because the conversation got too long for the model. The status code can’t change, but the stream is still open, so Kronk sends the error as one more event, data: {"error": {...}}, and closes the stream (chat.go:144-154). The client gets the error inside the body, as just another event in the stream, because that’s the only place left to put it.

On the right, the connection fails: the client disconnected, the request timed out, or writing to the socket failed. Now there’s nobody to tell. The SDK returns an error wrapped in ErrResponseCommitted (chat.go:16), which means “this failed, but the response was already sent”.

This is where “handlers return values” gets tricky. If the handler returns a plain NoResponse, the error is lost: no log, no metric, and it looks like a success. If it returns the error, Respond tries to write a 500 on top of a response that already started, so that’s not an option either.

The solution is that NoResponse can carry an error (response.go:27). The handler returns NewNoResponseError(err) when it sees ErrResponseCommitted (chatapp.go:57-62). The middleware (the functions that wrap every handler) sees the error inside, logs it and counts it in the metrics, and then passes the NoResponse along unchanged (errors.go:40-45). Respond sees NoResponse and writes nothing.

The result: the client’s stream just stops, and the server still has a full record of what went wrong.

Nuances depending on the protocol

Kronk speaks more than one protocol: it has OpenAI-compatible endpoints (/v1/chat/completions and /v1/responses) and an Anthropic-compatible one (/v1/messages). Each protocol has its own nuances and its own way of expressing things, like how events are named, how the stream ends, or how an error is reported after the commit. So far we’ve focused on OpenAI’s chat completions, and the details change slightly from one protocol to another, but the principles are the same: do as much as you can before the commit, and after it, the status can’t change.

For example, the Anthropic-compatible endpoint must send an anthropic-request-id header, but that id only arrives with the model’s first response. Since headers go out with the status, it can’t commit up front: it waits for that first response, sets the header, and then sends the 200 (msgsapp.go:112-120).

Now that we understand how both the regular API calls and the streaming ones work, let’s see how the rest of the request is handled.

The rest of the request path

To see how the rest of the request is processed, we need to talk about one last piece: the middlewares. Like in most web frameworks, they wrap our handler like the layers of an onion, and each request has to go through all of them. Let’s see what that looks like:

Every request goes through these middlewares from left to right before it reaches the handler, and the result comes back through them in reverse (mux.go:89). On the way in, they make the tracer available, log that the request started, and apply the per-route timeout and authentication. On the way out, they log that the request completed, count it in the metrics, and deal with any error or panic the handler produced.

If the handler returns an error, that error travels back out through the same middlewares: Errors logs it and Metrics counts it. Then Respond turns it into the response the client gets, using the same JSON error format as OpenAI, {"error": {"message", "type", "code"}}, so existing OpenAI clients know how to read it. Even a panic takes that path, because Panics is the innermost middleware: it turns the panic into a normal error right away (panics.go:22-27), and from there it travels out like any other error.

That’s all the pieces: the framework, the commit point, the failures after it, and the middlewares around the handler. Let’s put them together and follow a real request from start to finish.

Walking through an example

Let’s say your OpenAI client sends a POST /v1/chat/completions with "stream": true. What happens?

First, the request walks through the onion: the tracer is set up, the logger notes that the request started, and the per-route middlewares apply the 60-minute timeout and check authentication. Only then does it reach the chat handler, which decodes the body and asks the pool for the model, loading it if it isn’t in memory yet.

Now we’re in the “BEFORE” box. ChatStreamingHTTP checks that the writer can flush, waits for a spot if the model is busy, and prepares the prompt. If anything goes wrong here, the client still gets a normal HTTP error.

Everything went fine, so it’s time to commit: the server sends 200 OK with Content-Type: text/event-stream and flushes it. From here on, there’s no going back. The tokens start flowing as data: events, with a : keep-alive comment every fifteen seconds, until the final data: [DONE].

When the stream ends, the handler returns NoResponse. The middlewares log a successful request and count it, and Respond has nothing left to write, because the SDK already wrote everything.

Now, what if the user closes the tab halfway through the answer? Go’s HTTP server notices the connection is gone and cancels the request’s context. The engine sees that before generating the next token, stops, and frees its place for another request. The streaming loop notices too and returns ErrResponseCommitted, so the handler returns a NoResponse carrying that error. On the way out, the middlewares log and count the failure, and Respond writes nothing. There’s nobody listening anymore anyway.

And that’s the whole journey of a streaming request, from the first middleware to the last token. Let’s recap what we’ve learned.

Summary

The Kronk model server is a regular Go API in most ways. The interesting parts come from streaming:

• The commit point splits every request in two. Before the 200, errors are normal HTTP errors. After it, the status can’t change, so everything that can fail happens before.
• After the 200, errors go in the stream or in the logs. If the client is still listening, the error is sent as an event. If not, NoResponse carries the error so the server can still log it.
• Silence is dangerous. Keep-alive comments stop proxies from closing quiet connections.
• Each protocol has its own nuances, like the Anthropic-compatible endpoint delaying the commit until it knows its request id, but the principles are the same.

And that closes the series. We started with what inference actually is : a model file, a tokenizer, and a loop that produces one token at a time. Then yzma , calling a C++ engine from Go without a C compiler. Then the Kronk SDK , which serves many conversations at once. Then model management , getting models onto disk and into memory. And finally, the HTTP server on top.

With that, we’re ready to move on to a different topic. But until the next one, go ahead and run a model locally with Kronk.