# My AI engineer roadmap example

I grow a small public talk-catalog question-answering application through model calls, evaluation, retrieval, bounded functions and an HTTP service. The article is https://tej.as/blog/ai-engineer-roadmap.

## Start

I use Node.js 24.3 or later with built-in TypeScript type stripping. It runs the annotations without type-checking them. No npm install is needed for this example.

```sh
npm test
cp .env.example .env
# Edit .env locally. Set your own API key and a random DEMO_TOKEN.
node --env-file=.env first-call.mjs
npm run ask -- "Which talk builds movie search?"
npm run eval
```

The `first-call.mjs` script starts with a single fact and question and prints a text answer. Offline checks use fixtures and make no provider calls. `first-call.mjs`, `ask`, `eval`, `compare` and embedding retrieval make billable API calls. The pinned GPT-4.1 mini snapshot is a reproducibility choice, not a recommendation that it is the newest or best model. Select another model that supports the Responses API and strict JSON-schema output with AI_MODEL.

## The stages

1. `node --env-file=.env roadmap.ts ask "Which talk teaches evals?"` sends all 4 small catalog entries to the model. Change `RETRIEVAL=none` to see why a model without this evidence should abstain.
2. `node --env-file=.env roadmap.ts eval` runs 6 fixed questions. Set `REPORT=all-report.json` to save outputs, verdicts, text-model usage and timings. Set `BASELINE=1` to run the original prompt and schema before the abstention changes. Outputs can differ when rerun. A failed check exits 1. Review the actual answers: these mechanical checks cannot establish factual support or catch every unsupported claim.
3. `RETRIEVAL=keyword npm run eval` and `RETRIEVAL=embedding npm run eval` compare retrieval using the same questions. `retrieve <query>` prints the chosen documents. Embedding mode uses text-embedding-3-small and recomputes all document vectors each time for transparency. Persist document vectors in a larger application. Reported response usage excludes embedding tokens; latency includes retrieval.
4. `node roadmap.ts tool "fuzzy unit tests"` executes a bounded read-only search function. Unit checks demonstrate denied permission, unknown functions, excess arguments and long input. This command does not run a model-driven tool loop. Add native function calling only after understanding the distinction between asking for an action and authorizing it.
5. `npm run serve` starts an authenticated local HTTP service with 500-character question bounds, a 20-second upstream timeout, 2 concurrent requests, a shared 10-request-per-minute limit and a 60-second in-memory answer cache. `GET /health` is public. `POST /ask` needs the bearer token. Prompts and answers are never logged by the service; it logs request IDs, model IDs, usage and timing. The in-memory cache temporarily retains questions as hashes and answers. A process restart clears it. Provider data policies apply separately even with store:false.
6. `node --env-file=.env roadmap.ts compare gpt-4.1-mini-2025-04-14 YOUR_OTHER_MODEL_ID` repeats the same eval on up to 3 models. Choose the other model deliberately, check its documentation and inspect failures before comparing averages.

```sh
curl http://127.0.0.1:8080/health
curl http://127.0.0.1:8080/ask \
  -H "Authorization: Bearer $DEMO_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"question":"Which talk builds movie search?"}'
```

The shell variable in this curl example must be exported in your shell; editing .env alone does not export it.

## Container

```sh
docker build -t roadmap-catalog .
docker run --rm -p 127.0.0.1:8080:8080 --env-file .env roadmap-catalog
```

Use TLS and managed authentication before exposing this demo remotely. The shared bearer and process-local rate limit are for a controlled demo, not a public multi-user deployment. A real multi-tenant system needs server-created identities, per-document permission filters before retrieval, distributed quotas, secret rotation and provider budget limits. The sample has public documents only and no write tools.

## Failure drills

Use an invalid API key, remove the key, try an oversized question, send a bad bearer token and request a price absent from the catalog. Those are different failures with different behavior. Test retrieved instruction attacks as well as hostile questions before introducing outside content: telling the model a document is untrusted does not prove resistance to prompt injection.

Catalog text is labeled editorial summary, checked against the linked talks and articles on November 12, 2026. It is deliberately incomplete and is not a synchronized copy of the site. Sample reports are single runs on this tiny test set, not a benchmark of general model ability.