Reimplementing Enough of Kubernetes to Fool kubectl

I have a repo that reimplements Kubernetes’ core APIs in Node (pods, deployments, replica sets, services, jobs, namespaces, configmaps, the rest) backed by MongoDB instead of etcd, running “pods” as sibling Docker containers. It’s the only project I’ve published that strangers actually looked at.

I don’t think that’s because the code is good. I think it’s because of the claim in the README: point your real kubectl at it. Not a diagram of Kubernetes’ architecture, not a from-scratch exercise, a thing you can falsify in one command. Every other repo I’ve written leads with how it was built, and none of them got read.

Making that claim true turned out to be almost entirely about protocol details that aren’t in the resource schemas at all. Here are the ones I didn’t see coming. Everything below is against kubectl v1.34.1, and you can reproduce the header captures yourself in about a minute.

kubectl’s first move is discovery, and it will not proceed without it

Before kubectl get pods fetches a single pod, it asks the server what exists. Point kubectl at a server that 404s and watch what it sends:

GET /api?timeout=32s
Accept: application/json;g=apidiscovery.k8s.io;v=v2;as=APIGroupDiscoveryList,
        application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,
        application/json

GET /apis?timeout=32s
Accept: (same)

It asks for aggregated discovery first (a single document describing every group and version) and falls back to plain application/json if the server doesn’t offer it. Then it retries. In my capture it hit /api and /apis five times each and gave up without ever requesting a pod.

This is the thing to implement first, and I didn’t. You can have a flawless PodList handler and kubectl will never reach it, because the client refuses to guess at what the server supports. Two endpoints returning almost-empty JSON are the difference between “nothing works” and “everything works.”

kubectl get asks the server to do the formatting

This is the one that genuinely surprised me. Once discovery succeeds, here’s what kubectl get pods actually requests:

GET /api/v1/namespaces/default/pods
Accept: application/json;as=Table;v=v1;g=meta.k8s.io,
        application/json;as=Table;v=v1beta1;g=meta.k8s.io,
        application/json

as=Table. kubectl is not asking for pods and then formatting them. It’s asking the server for a table (column definitions and rows of pre-rendered cells) and printing what comes back nearly verbatim. The NAME READY STATUS RESTARTS AGE header you’ve read ten thousand times is a string the API server chose.

Which means you can produce a convincing kubectl get pods without implementing Kubernetes at all. This is a complete server:

const http = require('http');
const J = (res, obj) => {
  res.writeHead(200, { 'Content-Type': 'application/json' });
  res.end(JSON.stringify(obj));
};

http.createServer((req, res) => {
  const path = req.url.split('?')[0];

  if (path === '/api')
    return J(res, { kind: 'APIVersions', versions: ['v1'] });

  if (path === '/apis')
    return J(res, { kind: 'APIGroupList', apiVersion: 'v1', groups: [] });

  if (path === '/api/v1')
    return J(res, { kind: 'APIResourceList', groupVersion: 'v1', resources: [
      { name: 'pods', singularName: 'pod', namespaced: true, kind: 'Pod',
        verbs: ['get', 'list', 'watch'] }] });

  if (path.includes('/pods'))
    return J(res, {
      kind: 'Table',
      apiVersion: 'meta.k8s.io/v1',
      columnDefinitions: [{ name: 'Name', type: 'string', format: 'name', priority: 0 }],
      rows: [{ cells: ['demo-pod'] }],
    });

  res.writeHead(404, { 'Content-Type': 'application/json' });
  res.end('{"kind":"Status","code":404}');
}).listen(8899);

Point a kubeconfig at http://127.0.0.1:8899 and run kubectl get pods:

NAME
demo-pod

That is real kubectl, fully satisfied, talking to sixty lines of Node that has no concept of a pod.

I find this genuinely good design once the surprise wears off. It’s why kubectl get works on resource types your kubectl binary has never heard of, including CRDs that shipped after it was built: the server knows how to display them and the client doesn’t have to. But it does mean “API-compatible” is a much bigger surface than the resource schemas suggest. Every one of the ~55 kinds I route needs its own table() returning its own columns. Pods return Name, Ready, Status, Restarts and Age; each column carries a type, a format, a priority and a description string.

There’s a subtlety in the streaming case too. When kubectl watches a table, the column definitions should only appear on the first event: repeat them on every update and the client re-prints headers. So the watch path sends them once and then nulls the field:

Model.table([asJson]).then((table) => {
  if (eventType !== 'ADDED') table.columnDefinitions = null;
  eventStream.push(`${JSON.stringify({ type: eventType, object: table })}\n`);
});

The protobuf wire format is not protobuf

kubectl itself asked for JSON in every capture above. But client-go (which is what operators and controllers are built on) negotiates application/vnd.kubernetes.protobuf, and if you want those clients to work you have to speak it.

I assumed that meant “serialize the object with the .proto definitions.” It’s two layers more than that. Every message is:

1. A four-byte magic prefix: 0x6b 0x38 0x73 0x00, the ASCII bytes k8s followed by a null. Decoding starts by skipping it.
2. An Unknown envelope, a protobuf message with a typeMeta field (kind, apiVersion) and a raw field.
3. The actual object, encoded separately, stuffed into raw as bytes.

So you encode twice (the object into bytes, then those bytes into a wrapper) and prepend the magic:

let dataInfo = dataType.encode(prepareForProto(data)).finish();
let encoded = unknownType.encode({
  typeMeta: { kind: data.kind ?? '', apiVersion: data.apiVersion ?? '' },
  raw: dataInfo,
  contentEncoding: '',
  contentType: '',
}).finish();
return Buffer.concat([Buffer.from([107, 56, 115, 0]), encoded]);

The typeMeta duplication is the interesting part: kind and apiVersion appear in the envelope and inside the payload, so a client can route a message to the right decoder without decoding the payload first. Reasonable, and impossible to guess from the schemas.

Watch events add a third layer; each event is a WatchEvent message wrapping the already-wrapped object.

The failure mode is a wrong number, not an error

Kubernetes has scalar types that are structs on the wire and strings in JSON, and converting between them is where I lost the most time. Four of them:

Quantity is the dangerous one. It looks like a string, so the obvious thing is to encode it as one. If you do, the Go client doesn’t reject it: it decodes as zero. A pod whose CPU limit you carefully set to 100m arrives with a limit of nothing, and every layer reports success.

The fix is to know which keys hold quantities and wrap their values, which can’t be inferred from the value’s own shape:

const QUANTITY_MAP_KEYS = new Set([
  'limits', 'requests', 'min', 'max', 'default', 'defaultRequest',
  'maxLimitRequestRatio', 'capacity', 'allocatable', 'hard', 'used',
]);

I’ve now written this same paragraph about three different projects. A crawler whose rate limiter reported a correct delay while sending a hundred simultaneous requests. A browser whose memory instrumentation was the largest consumer of the resource it measured. And a serializer that turns a resource limit into zero and returns 200. In all three the code was locally correct, nothing threw, and the only way to see the bug was to look at what the other side received.

What I couldn’t make work

One protobuf shape defeated me, and the comment in the code says so:

// Known-bad proto shape: our Event schema stores a populated
// series/deprecated* tree that protobufjs can't encode back into
// a wire-compatible EventSeries, which causes client-side decode
// failures. Force JSON for Event responses.

Events fall back to JSON. Clients that accept both are fine; a strict protobuf-only client would not be. I’d rather ship that with a comment than pretend the encoder is complete.

The larger honest list is in the README, and it’s long: no CNI, so pods get a synthetic ClusterIP that routes nowhere. No CoreDNS. No CRDs, which rules out most real operators. No RBAC enforcement: every request is effectively cluster-admin. No server-side apply; apply-patch+yaml is accepted and quietly treated as a strategic merge. Conformance tests that touch any of that will fail no matter how much API surface I add, and I’d rather say which ones up front than let someone discover it after an afternoon.

The part worth generalizing

The technical lesson is that wire compatibility lives almost entirely outside the data model. Discovery, content negotiation, server-side printing, envelope framing, scalar coercion: none of it appears in a resource schema, and all of it is load-bearing. If I’d written the schemas first and the protocol second I’d have had a thing that looked complete and worked with nothing.

The other lesson is about how I describe work. “Point your real kubectl at it” is a claim a stranger can refute in one command, and that turns out to be the whole difference. My other projects are described in terms of the effort that went into them (written from scratch, no dependencies, built by hand) which asks the reader to take my word for something and gives them nothing to do. This one handed them a test. It’s the only one anybody ran.