Experiment · EXP-009

Is semantic similarity enough to find the controlling code?

Authentication-style queries retrieve related UI and still miss middleware and policy.

Thought experiment grounded in coding-agent behaviour: semantic search finds related language. Behaviour often lives in files that do not look like the query. I treat this as a completed qualitative finding and an incomplete quantitative ranking experiment. The graph-ranking architecture remains a hypothesis.

Hypothesis: The most semantically similar chunks are the chunks the model should see.

Research question

If a coding agent retrieves code only by semantic similarity to the user's request, will it consistently find the code that actually controls the behaviour?

Failure mode

Semantic search is good at finding code that talks about the same thing as the request. Behavioural control can live elsewhere. A search for authentication may strongly match login UI while the actual access policy is implemented in middleware, route guards, token utilities, or backend policy code.

Working hypothesis

Repository retrieval should combine semantic relevance with structural evidence such as symbols, imports, references, dependency relationships, folder boundaries, runtime traces, and potentially recent code history.

Semantic relevance
Symbol relationships
Imports and references
Dependency graph
Folder boundaries
Recent git activity
Runtime behaviour
Task history

Multi-signal codebase context

Task
  → Semantic retrieval
  → Symbol graph
  → Dependency graph
  → Runtime / test signals
  → Git history
  → Task memory
  → Ranked context pack
  → Model

Each signal answers a different question. Semantic search finds related language. The symbol graph finds definitions and references. The dependency graph finds what would break. Runtime and tests tell you what already failed. Git history tells you what recently moved. Task memory tells you what the agent already decided. The semantically closest file is not necessarily the file that controls a behaviour.

Current result

The qualitative failure mode is clear enough for me to reject semantic similarity as the only ranking signal. The stronger graph-aware ranking architecture remains a hypothesis until I run and publish a controlled retrieval evaluation.

What would make this conclusive

A proper experiment would require a repository benchmark with known controlling files, retrieval candidates, and measures such as recall at k across semantic-only, structural-only, and hybrid ranking strategies.

Abubakar, Imam. "Is semantic similarity enough to find the controlling code?." Imam Abubakar AI Research, 2026. https://imamabubakar.com/research/exp-009-semantic-only-code-search

This archive contains independent research, experiments, architecture studies, and technical observations. Some findings are exploratory and should not be interpreted as peer-reviewed scientific conclusions unless explicitly stated. Architecture studies of third-party products are based on publicly observable behaviour, published technical information, and engineering analysis. They are not claims of access to private source code or infrastructure.