Research notes

African AI systems as a research thread

A planned public thread: companies and architectures worth studying, not a finished survey.

In the 18-month announcement I said I would share African AI companies doing interesting work, and how I think certain systems probably work. That thread is not a finished write-up. I am recording it here so the archive does not pretend the only objects of study are American developer tools. Decide, as a spreadsheet agent with a Nigerian founding team, already sits in EXP-006. More entries will be added as studies are actually done.

A large part of public AI architecture discussion is centred on American model labs and developer tools. I want this archive to include African products that are solving equally interesting system problems under different market, infrastructure, and product constraints.

The goal is not to create a directory of companies. It is to study specific systems: what problem the product is solving, where AI appears in the workflow, what surrounding software likely makes it useful, what constraints are unique to the market, and which architectural decisions are transferable.

The Decide spreadsheet-agent study is one early entry in that direction. More studies will be added only when I have enough public evidence to separate observed behaviour from architectural inference.

This page is therefore intentionally an active research thread rather than a completed survey.

Context windows are not memory

A window is a budget. Memory is a policy for what should still be true later.

People collapse three ideas: the tokens present now, the logs of what was said, and the state that should persist. Only the third is memory in the sense I care about for agents.

A context window answers a narrow question: what information can the model attend to during this request? It does not answer what should still be remembered tomorrow, after a task changes, or after the current context has been replaced.

Conversation history is also not automatically memory. It is a record. A useful memory system needs a policy for selecting state that should persist, representing that state, retrieving it when relevant, updating it when reality changes, and discarding it when it stops being useful.

For agents, I care most about state that changes future decisions: objectives, constraints, accepted decisions, unresolved failures, permissions, and important discoveries.

A context window is a budget. Memory is a policy for what should remain true later.

Debugging layered AI versus debugging a prompt

If you cannot say which layer failed, you do not have an architecture. You have a blob.

QuoteSnap became easier to reason about when a wrong quote could be asked as four different questions: vision, validation, configuration, or rules. I now use that as a design test for other systems.

One test I now use for AI architecture is whether a wrong result can be decomposed into specific failure questions.

In QuoteSnap, a bad quote can be investigated as separate layers. Did the vision or language model interpret the job incorrectly? Did structured validation accept an invalid signal? Was the contractor configuration wrong? Did the deterministic pricing engine apply the wrong rule?

That separation is valuable because each failure has a different fix. Re-prompting the model will not repair a bad pricing rule. Changing the model will not repair stale contractor configuration.

If every failure can only be described as 'the AI got it wrong', the system is probably too tightly coupled.

Embeddings Are Infrastructure for Meaning

Embeddings matter far beyond training. They become the substrate for search, retrieval, and memory.

Embeddings allow systems to represent semantic relationships numerically. They become infrastructure for semantic search, retrieval, recommendation, similarity matching, and memory. Later research into coding assistants made the connection clearer. A model does not need to permanently hold an entire repository in context if another system can represent the repository, retrieve semantically relevant sections, and dynamically provide them to the model. Semantic similarity is still only one signal.

The mistake is to stop at “we added embeddings.” Retrieval quality, ranking, and disagreement between documents are the actual product problems.

Latency Changes Product Behaviour

Inference speed is not merely an infrastructure statistic. It changes what kind of product you can honestly ship.

A half-second interaction encourages a different interface from a twenty-second interaction. Systems involving several model calls accumulate latency even when each call appears acceptable. Parallel operations, smaller specialist models, caching, and deterministic replacements are therefore product decisions. The strongest possible model is not always the strongest possible system.

I now ask, when studying any AI product: what does this interaction cost at one hundred users, and what happens at one hundred thousand? If the architecture only works economically at prototype scale, it is not finished.

Maximise Useful Context Per Token

The goal is not to fill the window. The goal is to raise the density of information that can change the next decision.

Too much context can introduce irrelevant information, duplicate information, and conflicting information. Retrieval quality can directly affect answer quality because the model cannot reason over information it never receives. Simply retrieving more information does not guarantee better reasoning. The design target I keep returning to is useful context per token.

Context quantity and context quality are different variables.

Powerful Tools Without Boundaries Are a Product Failure

Once an agent can modify files, call APIs, or touch money, permissions are part of the research problem.

The transition from chatbots to agents changes the engineering problem. A chatbot primarily generates output. An agent can take action. Once a system can modify a file, execute code, update a spreadsheet, call an API, or send information, correctness becomes more important. You have to reason about state, permissions, execution, validation, traceability, retries, failure recovery, and human oversight. An agent with powerful tools but poor permission boundaries becomes dangerous. Increase autonomy only when verification and permission systems are strong enough to support it.

This is a current research question more than a finished design: how should agent permissions be represented when tools expose real business systems? I am not publishing a permission calculus here. I am recording the constraint, because it keeps getting ignored in demos.

Specialist small models plus tools

Current question: for constrained workflows, can a smaller specialist system beat a larger general model?

This is an open research question, not a result. The surrounding work — QuoteSnap’s split, Decide’s computational loop, quantization, cost — makes the question worth asking. I will not claim an answer until I have evidence.

Frontier general-purpose models are increasingly capable, but many production workflows are narrower than the benchmark tasks used to compare those models.

That raises a question I want to test directly: for a constrained workflow, can a smaller specialist model combined with good retrieval, deterministic tools, verification, and task-specific context outperform a much larger general model on the dimensions that actually matter to the product?

The relevant dimensions would include task success, latency, inference cost, failure recovery, and operational complexity rather than model quality in isolation.

I do not yet have enough evidence to state a result. This entry is a research hypothesis that follows naturally from the rest of the archive.

The hidden cost of autonomous agents

Loops multiply tokens, latency, and blast radius. Autonomy is an economic and safety decision.

Each extra step can retrieve, call a model, call a tool, retry, and write state. At prototype scale this is invisible. At product scale it is the architecture. I now treat unattended loops with suspicion until verification and permissions exist.

Autonomy changes the economics of an AI feature. A single user request can become multiple model calls, retrieval operations, tool executions, retries, validation steps, and state writes.

At prototype scale, this can feel invisible. At production scale, each extra loop contributes latency and variable inference cost. It also increases the number of actions that can fail or produce side effects.

This is why I do not treat 'more autonomous' as automatically 'more advanced'. The useful level of autonomy depends on whether the system has a strong oracle for success, bounded permissions, useful stop conditions, and enough economic value to justify the loop.

Autonomy is simultaneously a capability decision, a cost decision, and a safety decision.

The problem with giving an agent too many tools

Tool access is capability. Unbounded tool access is a permissions and selection problem.

Tool access increases agent capability, but also increases selection complexity, permission surface, and failure modes. My current design rule is to add tools that close a useful feedback loop and to treat tool catalogues as part of the agent architecture rather than as a feature checklist.

Giving an agent another tool increases capability, but it also increases the number of choices the agent can make and the number of actions the system must secure, describe, validate, and observe.

I have seen agents perform worse when tool catalogues become large, poorly differentiated, or weakly described. Similar tools create selection ambiguity. Broad permissions increase the blast radius of a wrong choice. Tools with unclear success conditions make verification difficult.

My current design rule is to add a tool when it closes a useful loop. A compiler closes the loop on compilation. A browser can close the loop on external information retrieval. A database tool can close the loop on application state. A tool that exists only because it looks powerful may increase complexity without increasing reliability.

This remains an open quantitative research area. I have not yet published an ablation measuring tool count against invalid selection or task completion.

Tokenization Is Part of the Intelligence Boundary

A language model does not receive language as humans experience it. It receives tokens.

Information is already being transformed before reasoning begins. The choice of tokenizer affects sequence length, vocabulary efficiency, and how information is represented. When working with models, the question is not only whether information exists in the prompt. The question is what representation of that information reaches the model. This became obvious while experimenting with a language model, and it later changed how I think about context packing in products.

This is a laboratory note, not a tokenizer write-up. The durable point is representation. If the system is stuffing prose, logs, and JSON into a window, it is making a tokenization bet whether it admits it or not.

What building a model changed about how I use APIs

Hosted models got less mystical and more like components with envelopes.

After working through tokenization, inference, and serving constraints, I stopped treating OpenAI-class APIs as oracles. I started treating them as components: what representation goes in, what contract comes out, what happens when they are slow, expensive, or wrong.

Before working directly with language-model components, hosted AI APIs were easy to experience as a single unit: send text in, receive intelligence out.

Working through tokenization, representation, inference, context, and serving constraints changed that mental model. I now think of a hosted model as a component with an input representation, a context budget, an output distribution, a latency profile, a price, and known failure modes.

That changes product architecture. I care more about the contract around the model: what information reaches it, what structure comes back, how results are validated, what happens during timeout or failure, and whether a deterministic component can own part of the task more reliably.

The API did not become less capable. I became more precise about what I was asking it to own.

Why agents sometimes appear smarter than their underlying model

Compilers, tests, retrieval, and memory can look like intelligence because they supply evidence the model would not have alone.

An agent that can run a compiler appears better at coding partly because the compiler tells it when it is wrong. High-quality retrieval appears like knowledge. Memory appears like consistency. Some apparent intelligence exists inside the model. Some useful intelligence emerges from the loop between the model and its environment.

It is easy to look at an impressive agent and attribute every useful behaviour to the model weights. That attribution is often incomplete.

A coding agent with a compiler can discover syntax and type failures the model did not know about when it generated the code. A test runner can reveal behavioural regressions. Retrieval can surface facts the model did not contain in its current context. A memory layer can preserve decisions that would otherwise disappear. A sandbox can let the system try an action and inspect the result.

None of those systems make the underlying model intrinsically smarter. They make the overall agent more capable because they add evidence and feedback to the reasoning loop.

Some intelligence is in the model. Some useful intelligence emerges from the loop between the model and its environment.

Why most AI evaluation is harder than unit testing

Deterministic tests still matter. They do not cover interpretation, retrieval, or judgement under uncertainty.

QuoteSnap’s pricing engine can be unit-tested. Visual interpretation cannot be reduced to the same tests. Coding agents can use compilers and still implement the wrong behaviour. Evaluation has to become layered or it becomes theatre.

Traditional unit tests remain extremely useful inside AI products. QuoteSnap's deterministic pricing rules can be tested with fixed inputs and expected outputs. Structured schemas can be validated. Tool functions can be tested like normal software.

The harder parts are the probabilistic boundaries. Visual interpretation can be reasonable without being exact. Retrieval can return relevant information without returning the best information. A coding agent can compile successfully and still implement the wrong behaviour.

That means evaluation should be layered. Test deterministic functions deterministically. Evaluate retrieval separately. Evaluate interpretation against labelled or reviewed examples. Measure end-to-end task success. Track latency, cost, retries, and failure recovery. Use human judgement where the task itself contains judgement.

Keep the unit tests. Add evaluation at every probabilistic boundary.

Why prompt engineering eventually becomes context engineering

Once the product has retrieval, tools, and state, the prompt is no longer the system.

Prompt phrasing still matters. It stops being the main lever as soon as the model is surrounded by selected files, tool traces, memories, and schemas. At that point you are designing a context pack, not a slogan.

Prompt wording matters, especially in small isolated interactions. It becomes a smaller part of the engineering problem once an AI feature gains retrieval, tools, memory, schemas, application state, user permissions, and intermediate results.

At that point the model is not responding to a single prompt. It is responding to an assembled context package. The quality of that package depends on what the system retrieved, what it omitted, which state it preserved, which tool results it exposed, and how conflicting information was resolved.

The design question therefore changes from 'what sentence should I write?' to 'what information does the model need for this decision, where does that information come from, how trustworthy is it, and how much of the context budget should it consume?'

Once the product has retrieval, tools, and state, the prompt is no longer the system.

Semantic similarity is not behavioural control

The login page matches “authentication.” The middleware may still own the behaviour.

A short note extracted from the Cursor study so it can be cited on its own. Ranking systems that only know cosine similarity will keep making this mistake.

Semantic search answers a useful question: which pieces of code look related to this request? It does not necessarily answer the more important question: which code controls the behaviour I need to change?

A login component is semantically close to authentication. Middleware, route guards, token utilities, or backend policy may be less semantically obvious while having more behavioural control.

That is why I think code retrieval should eventually combine semantic similarity with structural signals such as imports, references, symbol graphs, dependency relationships, and runtime evidence.

If you only retrieve what looks like the task, you can still miss what implements the task.

The model can do it is not an architecture

Capability is not permission, and it is not a product decision.

A one-line principle from QuoteSnap that now sits over most of the archive. Use AI where ambiguity, language, perception, or flexible reasoning creates an advantage. Keep deterministic behaviour deterministic where possible.

Modern language models can perform an unusually wide range of tasks. That makes it easy to turn model capability into an architecture decision without asking whether the model should own the responsibility.

QuoteSnap made the distinction concrete for me. A multimodal model can inspect job information and produce a price. That does not mean contractor pricing should be probabilistic. The model is more useful at interpreting messy input, while software is more useful at applying repeatable pricing rules.

I now ask two separate questions when designing an AI feature: can the model perform this task, and is the model the correct component to own this task in production?

Capability is not ownership.