Crafting AI

A Developer's Guide to Machine Learning

Barry S. Stahl

Principle Engineer

AZNerds.net

@bsstahl@cognitiveinheritance.com

https://CognitiveInheritance.com

Transparent Half Width Image 960x800.png

Favorite Physicists & Mathematicians

Favorite Physicists

Harold "Hal" Stahl
Carl Sagan
Richard Feynman
Marie Curie
Nikola Tesla
Albert Einstein
Neil deGrasse Tyson
Niels Bohr
Galileo Galilei
Michael Faraday

Other notables: Stephen Hawking, Edwin Hubble, Leonard Susskind, Christiaan Huygens

Favorite Mathematicians

Ada Lovelace
Alan Turing
Tim Berners-Lee
Isaac Newton
Emmy Noether
Johannes Kepler
René Descartes
George Boole
Carl Friedrich Gauss
Grace Hopper

Other notables: Blaise Pascal, Daphne Koller, Grady Booch, Evelyn Berezin, Pascal Van Hentenryck

Fediverse Supporter

See my presentation from CodeMash 2025: How the Fediverse Could Save Democracy & Why It Probably Won't
Hear my thoughts on why a thriving Fediverse is so critical on David Giard's Technology and Friends show.
Learn more about the Fediverse at fediverse.party.
Get your own Mastodon account at Join Mastodon.
- Don't stress on what server to use, just pick one and go. You can always move later.
Follow me on Mastodon at @bsstahl@CognitiveInheritance.com.

Some OSS Projects I Run

Liquid Victor : Media tracking and aggregation [used to assemble this presentation]
Prehensile Pony-Tail : A static site generator built in c#
TestHelperExtensions : A set of extension methods helpful when building unit tests
Conference Scheduler : A conference schedule optimizer
IntentBot : A microservices framework for creating conversational bots on top of Bot Framework
LiquidNun : Library of abstractions and implementations for loosely-coupled applications
Toastmasters Agenda : A c# library and website for generating agenda's for Toastmasters meetings
ProtoBuf Data Mapper : A c# library for mapping and transforming ProtoBuf messages

http://GiveCamp.org

Achievement Unlocked

Minimizing Error

Simple Linear Model

AI Models - Single Input Linear - 1280x720.png

2-Input Linear Model

AI Models - 2 Input Linear - 1280x720.png

Multiple-Input Linear Model

AI Models - Multiple Input Linear - 1280x720.png

Multi-Layer Perceptron

Crafting AI Models

Designing the Model Structure
- Define Inputs and Outputs
- Determine Layers and Neurons
- Select Activation Function
Training the Model
- Cost Function (Mean Squared Error)
- Gradient Descent Optimization
- Adjust Hyperparameters (e.g. learning rate)
These are the critical concepts for all ML tooling
- TensorFlow, Keras, ML.Net, etc.
- Master these concepts to harness any ML tool suite effectively

Linear Model

Component	Details
Input Var	\(X\)
Output Var	\(Y\)
Weight Param	\(m\)
Bias Param	\(b\)
Linear Eq	\(Y = mX + b\)

ML's Linear Linchpin

\(Y = mX + B\)

Every neuron gets its value from a linear transformation
Multiple inputs result in a sum of the linear transformations
- The sum of a linear transformation is linear
Only linearly-separable functions can be modeled without a non-linear activation

AI Models - Annotated Single Input Linear - 640x360.png

Mathematics

...we've invented a fantastic array of tricks and gimmicks for putting together the numbers, without actually doing it. We don't actually [apply \(Y = mX + b\) for every neuron] We do it by the tricks of mathematics, and that's all. So, we're not going to worry about that. You don't have to know about [Linear Algebra]. All you have to know is what it is, tricky ways of doing something which would be laborious otherwise.

With apologies to Professor Feynman, who was talking about the tricks of Calculus as applied to Physics, not the tricks of Linear Algebra as applied to Machine Learning.

Model Parameters

Parameters: Internal values learned during training
- Define the relationship between input and output
- Adjusted to minimize prediction error

Linear Regression Parameters

Linear Regression: \(Y = mX + b\)
- m: Weight (Slope) \(\to\) \(X\)'s influence on \(Y\)
In 3 Dimensions: \(Y = m_1X_1 + m_2X_2 + b\)
- m_1: Weight for \(X_1 \to X_1\)'s influence on \(Y\)
- m_2: Weight for \(X_2 \to X_2\)'s influence on \(Y\)
- Each weight is the slope of the plane in one dimension
Regardless of dimensionality
- b: Intercept (Bias) - Shifts the output vertically

Error Function

X-Axis: The value of the weight (m)
Y-Axis: The value of the bias (b)
Z-Axis: The size of the error

Weight (m) usually has the most impact on error

Training the Model

Objective: Minimize the error
- Error: The difference between the predicted value and the actual value
- MSE: Mean Squared Error - Avg of the squared differences between predicted and actual
Means: Gradient Descent Optimization
- Nearly any Optimization algorithm can be used
- Gradient Descent is the most common/suited

Gradient Descent

Optimization algorithm used to minimize the cost function of a model

Shifts the parameters in the opposite direction of the cost gradient
- The 1st derivative of the cost function
- Represents the slope of that function
Hyperparameters include:
- Number of iterations
- Learning rate
- Convergence criteria

AI Models - Gradient Descent - 400x800.png

Linear Regression Demos

AND Gate Model - Excel
Maintainability Model - Excel
Linear Regression - C# Code

Railroad Times Model

Train & Test Cycle

Voting Dataset

Voting Data

Voting Model

AI Models - SingleLayer Voter Model - 1280x720.png

Linear Separability

Not Linearly Separable

Activation

Functions applied to the output to introduce non-linearity

The sum of linear functions is linear
- i.e. multiple variables in a layer
- Y = M1*X1 + M2*X2 + ... + Mn*Xn + B
- Hyperplane in n-dimensional space
Composition of linear functions is linear
- i.e. multiple layers
- Y = M1*(M2*X1 + ... + Mn*Xn + B2) + B1
- Hyperplane in n-dimensional space
Composition of linear and non-linear is non-linear
- i.e. Activation functions
- Non-linear

Feature Analysis

Weights represent the influence of an input feature
The 13th weight is the smallest - least influential
- Least partisan: Superfund Right to Sue
The 4th weight is the largest - most influential
- Most partisan: Physician Fee Freeze

Trained Model Parameters - Voting Model - 800x800 - With Highlights.png

Feature Engineering

The process of identifying, selecting, and combining key attributes of a problem to improve a model

Choice and quality of features are critical
- Impact the model's ability to learn
Hierarchical layers enable complex feature extraction
Techniques like Dimensionality Reduction help reduce the number of features while preserving key information
- Example: Combine length and width into area
Requires experimentation and iteration

Features of the Voting Model

Current Model
- Inputs: 16 Features
- Output: 1 Neuron
- Architecture: Linear Perceptron (or Shallow Network)
Hypothesis: Existence of factions within parties
- Single-layer model may not capture this non-linear feature

Enhancing the Voting Model

Solution: Hidden Layer
Expected Outcome
- Better insight into party dynamics
- Improved accuracy
New Model
- Inputs: 16 Features
- Hidden Layer: 3 Neurons
- Output: 1 Neuron
- Architecture: Multi-layer Perceptron

AI Models - MultiLayer Voter Model - 1280x720.png

BackPropagation

A way to train multi-layer models by flowing deltas backwards

Create a feed-forward prediction
Calculate error at the output layer
Compute error gradient using the chain rule
Update Parameters using gradients
- Start from the end, work backwards
- Weights deltas are adjusted by the inputs
- Deltas to biases are not adjusted

Key Takeaways

Neural networks reduce to linear transformations
Non-linearity is required for non-separable problems
Training is optimization - minimize error
Gradient descent is the typical optimization
Engineer features to improve model performance
Master these fundamentals to build any ML system

Resources

Chain Rule of Calculus

To find the derivative of a composite function h(x) = f(g(x)), you take the derivative of the outer function f with respect to the inner function g, and multiply it by the derivative of the inner function g with respect to x

If h(x) = f(g(x)), then:
- dh/dx = df/dg * dg/dx
How changes in x affect the output h by accounting for how x influences g and how g influences f
Enables calculation of gradients for each layer by propagating errors backward through the network
Essential for training deep networks, as it helps adjust weights and biases to minimize prediction error

Overfitting

When a neural network learns the training data too well, capturing noise rather than the underlying pattern

Produces a model that performs well on training data but poorly on other data
Can happen when the model is too complex relative to the dataset
- Excessive parameters allow it to fit even noise within the data
Solutions include:
- Regularization: Add a penalty to the loss function
  - L1 (Lasso): Performs feature selection by setting some coefficients to zero
  - L2 (Ridge): Disperses the weights across all features
- Dropout: Randomly drop neurons during training to force the network to learn more robust features
- Early stopping: Stop training when performance on a validation set starts to degrade
- Data augmentation: Diversify the training set by applying transformations to the input data
- Weight initialization: Better initialization to avoid configurations that lead to overfitting
- Batch normalization: Normalize inputs on mini-batches to reduce fluctuations

Xavier/Glorot Initialization

Helps prevent gradients from becoming too small or large, aiding convergence

  // Xavier/Glorot initialization for better gradient flow
  int inputWeightCount = inputCount * hiddenLayerNodes;
  int totalWeightCount = inputWeightCount + hiddenLayerNodes;
  var weightScale = Math.Sqrt(2.0 / inputWeightCount);
  startingWeights = new double[totalWeightCount];
  for (int i = 0; i < startingWeights.Length; i++)
    startingWeights[i] = _random.GetRandomDouble(-weightScale, weightScale);

Balances the signal through layers
Prevents activation function saturation
Ideal for symmetric activations (e.g., sigmoid or hyperbolic tangent)