CSE 6040 Notebook 9 Part 2 Solutions — What Actually Clicks
There's a moment in every OMSCS student's life where they stare at a notebook cell and genuinely have no idea what it's asking. You've been clicking through Part 1 fine. The concepts made sense. Then Part 2 hits and suddenly you're questioning everything.
That's the Notebook 9 part 2 experience. And honestly, it's not that the material is harder. It's that the gap between "I understand the concept" and "I can implement it from scratch" gets wider. Part 2 is where you earn your grade.
So let's walk through it. Not just the code — the thinking behind it.
What Is Notebook 9 Part 2 Actually About
Notebook 9 is where CSE 6040 pulls together a lot of the classification and modeling work from earlier notebooks and asks you to build it yourself. That's why part 1 usually introduces the dataset and the evaluation framework. Part 2 is where you implement the models Turns out it matters..
People argue about this. Here's where I land on it.
Depending on the semester, you're likely building classifiers from scratch — things like k-nearest neighbors, decision trees, or ensemble methods. You might also be working with perceptrons or logistic regression, depending on where the notebook falls in the course timeline Less friction, more output..
The core idea is straightforward. You get a labeled dataset. You build a model. Worth adding: you evaluate it. But the implementation details are where people get stuck.
The data you're working with
Most versions of this notebook use a classic dataset — sometimes the spam classification set, sometimes the fashion-MNIST image set, or another problem that lets you test multiple algorithms on the same inputs. The point isn't the dataset itself. It's that you can compare approaches using the same ground truth Easy to understand, harder to ignore..
What "from scratch" means here
When the notebook says "implement k-nearest neighbors from scratch," it doesn't mean write a production-ready library. It means understand the distance calculation, the neighbor selection, the voting mechanism. That said, same with decision trees — you're not building scikit-learn. You're building the logic so you actually know what the library is doing under the hood Simple, but easy to overlook..
Why It Matters
Here's the thing. That's fine for learning the workflow. You use scikit-learn for logistic regression, you call fit() and predict(), and you move on. Earlier notebooks in CSE 6040 lean on libraries. But Notebook 9 asks you to reverse-engineer that workflow Not complicated — just consistent..
Why does this matter? And more importantly, understanding what's happening at each step changes how you diagnose problems. You stop treating models as black boxes. Because on the exam, you won't have scikit-learn. You start asking why your decision tree is splitting on the wrong feature, or why your k-NN is misclassifying points that should be obvious.
Real talk — this is the notebook that separates students who understand data science from students who just memorized API calls.
How It Works — Breaking Down the Solutions
Let's go section by section. I'll walk through the logic, then point you to where the code ties it together Still holds up..
K-Nearest Neighbors Implementation
The basic algorithm is simple. Pick the k closest ones. For each test point, calculate the distance to every training point. Whatever label shows up most in that neighborhood wins.
But the notebook usually asks you to implement it in a way that reveals a few details people skip.
Distance metrics matter. Most implementations use Euclidean distance. The notebook might ask you to try Manhattan distance or cosine similarity. Here's what trips people up — the choice of distance metric changes the shape of your decision boundary. Euclidean distance treats all dimensions equally. Manhattan distance creates axis-aligned boundaries. Cosine similarity ignores magnitude and focuses on direction. You'll see this when you plot your results.
The voting step has edge cases. What happens when k is even and you get a tie? The notebook might ask you to break ties randomly, or by choosing the class with the smallest index. Know which one your implementation uses because it affects your accuracy by a small but real margin And that's really what it comes down to..
Here's what most people miss. That said, normalizing your features before calculating distance isn't optional here. If one feature ranges from 0 to 1 and another ranges from 0 to 10,000, the larger feature dominates distance calculations completely. The notebook usually hints at this, but if it doesn't, do it anyway.
Decision Trees from Scratch
This is where Part 2 gets heavier. That's why building a decision tree means writing a recursive splitting algorithm. For each node, you evaluate every possible split on every feature, calculate the information gain or Gini impurity reduction, and pick the best one.
The recursion is the tricky part. You write a function that takes a subset of data and a depth limit. If all samples belong to one class, or you've hit max depth, or there are no features left to split on, you make a leaf node with the majority class. Otherwise, you find the best split, split the data, and recurse on each child That alone is useful..
Here's the part that bites people. In real terms, when you split the data, you need to pass the correct feature indices down to each child. That's why if you're not careful, you'll accidentally include features that were already used for splitting in an ancestor node. The notebook usually handles this, but you need to follow the logic closely.
Gini vs. entropy. The notebook may let you choose. In practice, they give nearly identical results. Gini is slightly faster to compute because it doesn't use a logarithm. But don't stress about which one to pick — just be consistent.
Ensemble Methods
If Part 2 includes random forests or bagging, you're combining multiple weak models to make a strong one. Each tree is trained on a bootstrap sample of the data
where each tree sees a different random subset. The magic happens in the aggregation step. For classification, you typically take a majority vote across all trees. For regression, you average the predictions.
Here's what catches people off guard: the bias-variance tradeoff becomes visible when you implement this yourself. Because of that, each individual tree has high variance but low bias. In real terms, when you average many of them, the variance decreases while the bias stays roughly the same. This is why random forests often outperform single decision trees significantly.
Feature subsampling is crucial. At each split, you don't consider all features — just a random subset. This decorrelates the trees, making the ensemble more strong. If you use all features at every split, your trees become nearly identical and you lose most of the benefit Easy to understand, harder to ignore. No workaround needed..
Support Vector Machines
SVMs introduce a different way of thinking about classification. Instead of modeling probabilities, you're finding the optimal separating hyperplane that maximizes the margin between classes Not complicated — just consistent..
The kernel trick is elegant but confusing. When you implement the dual form, you work with support vectors and kernel functions rather than explicit feature transformations. The Gaussian RBF kernel, for instance, implicitly maps your data to infinite dimensions. But here's the gotcha: you need to tune the gamma parameter carefully. Too high, and you overfit to individual points. Too low, and everything looks linearly separable.
Most people struggle with the quadratic programming solver. Here's the thing — you don't need to implement one from scratch — use scipy. optimize or CVXOPT — but understanding that you're solving a constrained optimization problem helps debug convergence issues.
Neural Networks from Scratch
Building a neural network means implementing forward propagation, backpropagation, and gradient descent manually. This is where Part 2 truly tests your understanding of calculus and linear algebra.
The chain rule is everywhere. When you compute gradients for weight updates, you're applying the chain rule repeatedly through each layer. The key insight is that the error signal flows backward through the network, and each layer's gradient depends on the gradient from the layer above.
Here's what students often get wrong: they initialize weights incorrectly. If weights are too large, activations explode. Here's the thing — if too small, gradients vanish. Xavier initialization sets the variance based on the number of input and output units, which helps maintain stable signal flow through the network.
Batch processing matters for efficiency. Instead of updating weights after each sample (stochastic gradient descent), you accumulate gradients over a mini-batch before updating. This smooths out the optimization landscape and makes better use of vectorized operations.
The loss landscape visualization in the notebook isn't just pretty — it shows why neural networks can get stuck in local minima or saddle points. Understanding this helps you appreciate why techniques like momentum and learning rate scheduling are necessary Not complicated — just consistent..
Key Takeaways
Implementing these algorithms from scratch reveals the engineering decisions hidden behind scikit-learn's simple API. You discover that machine learning isn't just about choosing the right model — it's about understanding how each hyperparameter shapes the learning process.
The biggest lesson is that data preprocessing and feature engineering aren't preliminary steps; they're integral to how these algorithms work. Normalization affects distance calculations, feature scaling impacts gradient descent, and handling missing values differently can change your entire approach to splitting criteria.
When you write the code yourself, you also gain intuition for debugging. Plus, if your k-NN performs poorly, you can trace through the distance calculations. If your decision tree overfits, you can examine the split decisions at each node. This hands-on understanding is what separates practitioners from those who merely use black-box tools.
These implementations may seem tedious, but they build the foundation for adapting algorithms to new problems and inventing novel solutions when standard approaches fall short.
