Inference requests and APIs
Inspect bounded loopback requests, response envelopes, status codes, timeouts, retries, and streaming signals.
What you will practice
Check gateway health; List served models; Read a request envelope; Read a response envelope.
Use the worked examples when needed, then test your understanding without them.
- 01Check gateway healthA health endpoint can describe process liveness without proving model readiness. Read the scope of the endpoint before using its status to approve traffic.Practice
- 02List served modelsA served-model list describes identifiers the gateway accepts. Use the exact served identifier rather than a local filename or an unrelated registry name.Practice
- 03Read a request envelopeA request envelope combines model selection, messages and generation constraints. Invalid field types should be corrected at the contract boundary before retrying.Practice
- 04Read a response envelopeA response envelope contains output plus metadata such as completion status and usage. Inspect the finish reason before treating partial output as a complete answer.Practice
- 05Inspect token usageUsage accounts for input and generated tokens separately. Compare the sum with the recorded total and keep the tokenizer/model context with any estimate.Practice
- 06Distinguish client and server errorsClient and server status classes point to different boundaries. A malformed request calls for correction, while a transient server failure may warrant a bounded retry policy.Practice
- 07Reason about timeoutsTimeouts must identify which layer stopped waiting. If the client deadline is shorter than normal server work, the client can time out while the server keeps processing.Practice
- 08Review retry boundariesRetry policies need a maximum attempt count, delay and retryable failure set. Unbounded retries can amplify overload and duplicate work.Practice
- 09Inspect a request identifierA request identifier ties a client operation to gateway and runtime evidence. Record the identifier rather than copying the whole prompt into ordinary logs.Practice
- 10Follow a trace identifierA trace identifier connects spans across component boundaries. Use the shared trace ID and parent relationships to reconstruct one request path.Practice
- 11Compare batch and single requestsBatch execution has per-item outcomes. An accepted batch request does not mean every item completed successfully, so preserve item IDs and partial failure details.Practice
- 12Read a streaming event shapeStreaming chunks are increments, not necessarily complete messages. Assemble them in order and require the documented completion signal before marking the response done.Practice
- 13Inspect rate-limit metadataRate-limit metadata describes a specific quota scope and reset window. It does not authorize an unbounded retry loop or guarantee immediate capacity after a guessed delay.Practice
- 14Review idempotency guidanceIdempotency is an endpoint contract, not a property to assume. Reusing a key is useful only where the service documents how duplicate requests are handled.Practice
- 15Build an API evidence bundleAn API evidence bundle needs request IDs, status, timing, versions and a redacted contract example. It should explain the failure without exporting credentials or user prompts.Practice
- 16Inference API checkpointAPI incident review distinguishes validation, readiness, transport and overload failures. Classify the observed boundary before choosing correction, waiting or a bounded retry.Practice