Inside Model Reasoning

Fine-Tuning in Action

Barry S. Stahl

Solution Architect & Developer

@bsstahl@cognitiveinheritance.com

https://CognitiveInheritance.com

Favorite Physicists

1. Harold "Hal" Stahl
2. Carl Sagan
3. Richard Feynman
4. Marie Curie
5. Nikola Tesla
6. Albert Einstein
7. Neil Degrasse Tyson
8. Niels Bohr
9. Galileo Galilei
10. Michael Faraday

Other notables: Stephen Hawking, Edwin Hubble, Leonard Susskind, Christiaan Huygens

Favorite Mathematicians

1. Ada Lovelace
2. Alan Turing
3. Johannes Kepler
4. Rene Descartes
5. Isaac Newton
6. Emmy Noether
7. George Boole
8. Blaise Pascal
9. Johann Gauss
10. Grace Hopper

Other notables: Daphne Koller, Grady Booch, Leonardo Fibonacci, Evelyn Berezin, Benoit Mandelbrot

• See my presentation from CodeMash 2025: How the Fediverse Could Save Democracy & Why It Probably Won't

• Hear my thoughts on why a thriving Fediverse is so critical on David Giard's Technology and Friends show.

• Learn more about the Fediverse at fediverse.party.

• Get your own Mastodon account at Join Mastodon.

  • Don't stress on what server to use, just pick one and go. You can always move later.

• Follow me on Mastodon at @bsstahl@CognitiveInheritance.com.

• What does fine-tuning do?
• What problems does it solve?
• What problems doesn't it solve?
• What other methods are there?
• When should we use each?

• Same Knowledge, Different Behavior
  • We didn't add facts here
  • We added a pattern of behavior
• Other patterns we could tune for
  • Responses are in JSON format
  • Responses use our brand voice
  • Responses list options before answering
  • Responses are a label from a list

A neural network architecture that has enabled the recent advancements in NLP

• Based on the 2017 Google paper Attention is All You Need

• Key Breakthrough: The Self-Attention Mechanism

  • Focus on most relevant tokens
  • Add context even at a distance

• Enables Processing of Entire Sequences in Parallel

  • Unlike traditional RNNs that process data sequentially

• A point in multi-dimensional space
• Mathematical representation of a word or phrase
• Encode both semantic and contextual information

• Model: text-embedding-ada-002
• Vectors normalized to unit length
• Use 1536 dimensions

Creating Order from Chaos

• Unstructured string => Structured float[]
• Allows mathematical operations
  • Cosine Similarity & Distance
  • Nearest Neighbor Search
  • Clustering
  • Vector Addition & Subtraction
  • Dimensionality Reduction (e.g., PCA)
  • Anomaly Detection

• Base Models

  • Predict from the full training corpus
  • Provide a single response quickly
  • Only "consider" the most-likely response

• Fine-Tuned Models

  • Follow fine-tuned patterns
  • Can respond with multiple options initially
  • Then predict based on those options

• Reasoning Models

  • Pre-trained to follow reasoning patterns
  • May also have other tweaks

• The pattern can simulate multiple prompts
• Think first, then act
• Allows for multiple possibilities

Fine-tune data into the model when the "knowledge" is a pattern of static associations

• The information must be
  • Small - few facts
  • Stable & Unchanging
  • Closed-domain (won't expand)
  • Unlikely to conflict with future facts
• Examples
  • Error 123 indicates X
  • Status 234 indicates Y
  • This Acronym means Z

Fine-Tuning Changes Response Patterns

• Best for teaching the model how to respond
• Not for changing what the model should respond with

We'll look at better ways to adjust the model's knowledge later in this session

IntentBot on GitHub

• An OSS agentic framework
  • Modular
  • Microservices‑based
  • Designed before LLMs existed

• All components are pluggable
  • Intent determination
  • Intent-based Routing
  • Fulfillment Services

What if we give the model an utterance that it hasn't been fine-tuned to handle?

Will it:

• Throw an error?
• Fall-back to UnknownIntent?
• Make something up?

• Static Context Engineering
  • Add tokens to the designed prompt
• Dynamic Context Engineering
  • Add tokens at runtime
• Constrained Decoding
  • Restrict what tokens the model can produce
• Post‑Generation Validation
  • Reject or correct the output after it’s produced
• Fine-Tuning
  • Modify the model's behavioral priors

Add tokens to the designed prompt

• System prompts
• Instructions
• One-shot or few‑shot examples

Add tokens at runtime

• RAG
• GraphRAG
• Tool outputs
• Memory injection

• Combines the benefits of retrieval-based and generative models

• Identify and retrieve relevant information

  • Agument context of the generative models
  • Generative responses based on the augmented context

• Potential uses include

  • Explore large documentation sets
  • Summarize articles in light of known relevant information

Restrict what tokens the model can produce

• JSON schema enforcement
• Grammar constraints
• Token blacklists/whitelists

Reject or correct the output after it's produced

• Intent whitelists
• Schema validation
• Supervisory agents
• Client enforcement

Modify the model’s behavioral priors

• Classification patterns
• Formatting habits
• Stylistic tendencies

• Fine-Tuning : Intent Classification

  • Teaches the model the response pattern

• Static Context for Allowed Intents

  • System prompt provides the list of valid intents

• Post-Generation Validation

  • Checks that the returned intent is in the allowed set
  • If invalid, sends back with corrective feedback

• Constrained Decoding (Possible)

  • If single-token labels, restrict output to those tokens
  • Adds mechanical guardrails but also risk

• Tooling for Service Discovery (Future)

  • Query services to confirm an intent is supported
  • Resolve borderline user requests

• Fine-tuning → Behavior

  • Shapes how the model responds
  • Establishes patterns, structure, and consistency
  • Does not add facts or enforce hard constraints

• Context Engineering → Knowledge

  • Provides information the model didn't previously have
  • Static context = fixed instructions
  • Dynamic context (ie RAG) = runtime grounding

• Hybrid Approaches Are Common and Powerful

  • Combine behavior shaping + knowledge injection
  • Add constraints and validation for reliability
  • Architecture > any single technique

• Start with Prompt Engineering

  • Establish clear instructions and structure
  • Shape the model's behavior with minimal effort

• Move to RAG for More Complex Scenarios

  • Bring in external knowledge when needed
  • Keep information fresh, grounded, and scalable
  • Bring in Documents, Facts, Memory & Tools

• Fine‑Tune When Behavior Matters

  • Lock in consistent behaviors, formatting, and workflows
  • Improve reliability for repeated, structured tasks

• Use Combinations for Best Results

  • Real systems blend all of the above
  • Architecture beats any single technique

• From Me
  • AI Articles
  • Code Samples
  • IntentBot on GitHub
  • LLM Embeddings: Not Magic, Just Math
  • Future of Information Retrieval: A Deep Dive into RAG
• From Microsoft
  • Microsoft Foundry
  • Foundry Local
  • Using RAG with an AI Model
  • AI Architecture Center