Parsa Mesgarha

SVD,Eigendecomposition and Mahalanobis (manobala) Distance

Mon, 06 Jul 2026 00:00:00 GMT

Need for Mahalanobis distance

Distance calculated using euclidean geometry provides the absolute value between an origin and the target, yet this quantity lacks on stating how unique/unremarkable a particular point is in a data cluster. Standard deviation of a probability distibution comments on how the variables are spread. Mahalanobis distance is that quantity that tells the measure of a point from the mean in terms of standard deviation. This is particularly required in point clouds because this helps in determining outliers/noise while performing Iterative Closing Point Algorithm.

Formulas to get familiar with

The Mahalanobis distance is calculated as:

$$ d_M (x) = \sqrt{(x - \mu)^\top \Sigma^{-1}(x - \mu)} $$

Setting $e = x - \mu$ (the deviation from the mean):

$$ d_M^2 = e^\top \Sigma^{-1} e = \Vert e \Vert_\Sigma^2 $$

$\Sigma$ is the $d \times d$ covariance matrix

$$ \Sigma_{d \times d} = \frac{1}{n - 1} \sum_{i=1}^{n} (\mathbf{x}_i - \bar{\mathbf{x}})(\mathbf{x}_i - \bar{\mathbf{x}})^\top $$

Let $d=2$, so each observation is a 2D vector $\mathbf{x}i = \begin{bmatrix}x{i1}\ x_{i2}\end{bmatrix}$ and the mean is $\bar{\mathbf{x}} = \begin{bmatrix}\bar{x}_1\ \bar{x}_2\end{bmatrix}$. The deviation vector for observation $i$

$$ \mathbf{x}i - \bar{\mathbf{x}} = \begin{bmatrix}x{i1} - \bar{x}1\ x{i2} - \bar{x}_2\end{bmatrix} $$

The outer product is $(\mathbf{x}_i - \bar{\mathbf{x}})(\mathbf{x}_i - \bar{\mathbf{x}})^\top$. This is a column vector times a row vector — giving a $2\times2$ matrix:

$$ (\mathbf{x}_i - \bar{\mathbf{x}})(\mathbf{x}i - \bar{\mathbf{x}})^\top = \begin{bmatrix}x{i1}-\bar{x}1\ x{i2}-\bar{x}2\end{bmatrix}\begin{bmatrix}x{i1}-\bar{x}1 & x{i2}-\bar{x}_2\end{bmatrix} $$

$$ = \begin{bmatrix}(x_{i1}-\bar{x}1)^2 & (x{i1}-\bar{x}1)(x{i2}-\bar{x}2)\ (x{i2}-\bar{x}2)(x{i1}-\bar{x}1) & (x{i2}-\bar{x}_2)^2\end{bmatrix} $$

note: d is number of features(columns) and n is number of observations (rows)

Sum over all $n$ observations

$$ \sum_{i=1}^{n}(\mathbf{x}_i - \bar{\mathbf{x}})(\mathbf{x}i - \bar{\mathbf{x}})^\top = \begin{bmatrix}\sum_i(x{i1}-\bar{x}1)^2 & \sum_i(x{i1}-\bar{x}1)(x{i2}-\bar{x}2)\ \sum_i(x{i2}-\bar{x}2)(x{i1}-\bar{x}1) & \sum_i(x{i2}-\bar{x}_2)^2\end{bmatrix} $$

Divide by $n-1$

$$ \Sigma_{2\times2} = \frac{1}{n-1}\begin{bmatrix}\sum_i(x_{i1}-\bar{x}1)^2 & \sum_i(x{i1}-\bar{x}1)(x{i2}-\bar{x}2)\ \sum_i(x{i2}-\bar{x}2)(x{i1}-\bar{x}1) & \sum_i(x{i2}-\bar{x}_2)^2\end{bmatrix} $$

Reading each entry

$$ \Sigma_{2\times2} = \begin{bmatrix}\text{Var}(x_1) & \text{Cov}(x_1,x_2)\ \text{Cov}(x_2,x_1) & \text{Var}(x_2)\end{bmatrix} $$

$(1,1)$: $\frac{1}{n-1}\sum_i(x_{i1}-\bar{x}_1)^2$ — variance of feature 1. Same feature squared with itself.
$(1,2)$: $\frac{1}{n-1}\sum_i(x_{i1}-\bar{x}1)(x{i2}-\bar{x}_2)$ — covariance of feature 1 and 2. Two different features multiplied.
$(2,1)$: same as $(1,2)$ — always symmetric since multiplication is commutative.
$(2,2)$: $\frac{1}{n-1}\sum_i(x_{i2}-\bar{x}_2)^2$ — variance of feature 2.

Connection to $\frac{1}{n-1}X_c^\top X_c$ (centered data)

The formula $\frac{1}{n-1}\sum_i(\mathbf{x}_i - \bar{\mathbf{x}})(\mathbf{x}_i - \bar{\mathbf{x}})^\top$ and $\frac{1}{n-1}X_c^\top X_c$ are the same thing written two different ways:

The summation form adds up $n$ outer products one observation at a time
The matrix form stacks all observations into $X_c$ and does one matrix multiplication

Both produce the identical $2\times2$ covariance matrix — the summation form makes the meaning explicit, the matrix form is compact and fast to compute.

⚠️ Centering requirement: The matrix form only equals the covariance matrix if $X_c$ is mean-centered — each row must be $(\mathbf{x}_i - \bar{\mathbf{x}})^\top$, so every column sums to zero. The summation form has $-\bar{\mathbf{x}}$ built into each term; the matrix form must bake it into $X_c$ itself.

On raw (uncentered) data, $\frac{1}{n-1}X^\top X$ gives the second moment matrix instead:

$$ \frac{1}{n-1}X^\top X = \Sigma + \tfrac{n}{n-1},\bar{\mathbf{x}}\bar{\mathbf{x}}^\top $$

equal to $\Sigma$ only when $\bar{\mathbf{x}} = \mathbf{0}$.

Centering ≠ normalising. The ladder:

Raw $X$ → second moment matrix

Centered $X$ → covariance matrix $\Sigma$ ← what Mahalanobis needs

Centered + scaled by std dev → correlation matrix $R$ Worked example (1 feature, $n=3$): let $x = [1, 2, 3]$, so $\bar x = 2$.

Centered: $x_c = [-1, 0, 1]$ $$ \frac{1}{n-1}x_c^\top x_c = \frac{1}{2}\left[(-1)^2+0^2+1^2\right] = 1 $$ This matches $\text{Var}(x) = \frac{1}{n-1}\sum_i(x_i-\bar x)^2 = 1$ exactly. ✓

Raw (uncentered): using $x = [1,2,3]$ directly

$$ \frac{1}{n-1}x^\top x = \frac{1}{2}\left[1^2+2^2+3^2\right] = \frac{14}{2} = 7 $$

7 is not the variance — it's inflated by the mean not being zero. This is exactly $\Sigma + \frac{n}{n-1}\bar x^2 = 1 + \frac{3}{2}(2)^2 = 1 + 6 = 7$. ✓

Skipping the centering step silently swaps "spread around the mean" for "spread around zero" — the two only agree when $\bar x = 0$.

Note: The query point $x$ in $d_M(x)$ never needs pre-centering — the $(x - \mu)$ inside the formula centers it explicitly. Centering only matters for how $\Sigma$ is computed from the data matrix.

What is covariance, variance and standard deviation? and Ohh my Gaussian Curve

$\sigma^2$ is variance — the average squared deviation of points from the mean; it measures how spread out the distribution is

$$ \sigma^2 = \frac{1}{n - 1}\sum_{i=1}^{n}(x_i - \bar{x})^2 $$

$\sigma$ is the standard deviation — the square root of the variance. Taking the root brings the units back to the first power (e.g. meters instead of meters²), same units as the data.
$\text{cov}(x_1, x_2)$ covariance tells how two features vary with respect to each other — positive when they move together, negative when they move oppositely, near zero when unrelated
variance ($\sigma^2$) can also be written as
$$ \sigma^2=E(X^2)-\mu^2 $$
Where $E(X^2)$ is: $$ \int_{-\infty}^{\infty} x^2 f(x), dx $$ And $f(x)$ is the Gaussian density formula: $$ \frac{1}{\sigma \sqrt{2\pi}} \exp\left(-\frac{(x - \mu)^2}{2\sigma^2}\right) $$
$\frac{1}{\sigma \sqrt{2\pi}}$ is the normalising constant to keep area under curve equal to 1.
A Gaussian random variable is written $X \sim N(\mu, \sigma^2)$ where $\mu$ is the mean and $\sigma^2$ is the variance
all Gaussian curves are bell shaped but they can be shifted (by tuning $\mu$) or stretched/squeezed (by tuning $\sigma$)

area under all theses curves is 1
Probability gives me a confidence/cerainity score of the belief we have on a method/result. A self driving car gets a LiDAR reading of 3.2 m to hit an obstacle, is it safe? It depends entirely on th threshold value of an acceptable uncertanity score. If $\sigma$ is less than 0.05m, then it is fine.

So what the f*ck is Singular Value Decomposition?????

Any matrix $A$ irrespective of its shape; number of independent columns it has (rank); and the numbers present in it, can be decomposed into 3 matrices. That is it!!! It is all that is need to be known to understand SVD apart from the formula.

Now, the formula is made up of 3 matrices which basically rotate, transform, and maybe rotate any input vector again. The 3 stooges of matrices are made of the square roots of the eigen values and eigen vector of $A^TA$ and eigen vector $AA^T$.

$$ A=US V^T $$

$V$ columns are eigen vectors of $A^TA$
$S$ is the singular values of $A$, which are the square roots of the eigen values of $A^TA$
$U$ matrix consist of columns which are eigenvectors of $AA^T$

NOTE: SVD works only if $U$ and $V$ are orthonormal vectors. We will see why.

$$ A^TA(v_{raw}) = \sigma^2(v_{raw}) \quad \textbf{--- (1)} $$

$$ (A^TA-\sigma^2 I)v_{raw}=0 $$

$$ v_i = \frac{v_{raw}}{||v_{raw}||} \quad \textbf{--- (2)} $$

Substituting the normalised input eigen vector from eqn$\textbf{(2)}$ in eqn $\textbf{(1)}$

$$ A^TA(v_{i}) = \sigma^2(v_{i}) \quad \textbf{--- (3)} $$

$$ ||Av_i||^2 = (Av_i)^T(Av_i) = v_i^TA^TAv_i \quad \textbf{--- (4)} $$

Substituting eqn$(3)$ in eqn$(4)$ we get

$$ ||Av_i||^2 = (Av_i)^T(Av_i) = v_i^TA^TAv_i = v_i^T\sigma^2(v_{i}) \quad \textbf{--- (5)} $$

Since $v_i$ is a unit vector, $v_i^Tv_i = 1$ (a scalar, not the identity matrix — that only applies when dotting a full orthonormal matrix with its transpose, e.g. $V^TV = I$). Therefore eqn$(5)$ is

$$ ||Av_i|| = \sigma \quad \textbf{--- (6)} $$

we know that the magnitude of $Av_i$ from eqn$(6)$

In eqn$(3)$ multiply both sides by $A$

$$ AA^TA(v_{i}) = \sigma^2(Av_{i}) \quad \textbf{--- (7)} $$

normalising $Av_i$ we get

$$ u_i=\frac{Av_i}{\sigma_i} \quad \textbf{--- (8)} $$

If we stack this relationship horizontally for all $n$ vector dimensions simultaneously, it maps perfectly to an equivalent matrix-matrix multiplication:

$$ A \underbrace{\begin{bmatrix} \mathbf{v}_1 & \mathbf{v}_2 & \cdots & \mathbf{v}n \end{bmatrix}}{V} = \underbrace{\begin{bmatrix} \mathbf{u}_1 & \mathbf{u}2 & \cdots & \mathbf{u}m \end{bmatrix}}{U} \underbrace{\begin{bmatrix} \sigma_1 & & \ & \sigma_2 & \ & & \ddots \end{bmatrix}}{\Sigma} $$

This simplifies neatly down to:

$$ AV = U\Sigma $$

Because $V$ is an orthogonal matrix built from an orthonormal basis of eigenvectors, its inverse is identical to its transpose ($V^{-1} = V^\top$). Right-multiplying both sides by $V^\top$ completely isolates $A$:

$$ AVV^\top = U\Sigma V^\top $$

$$ A = U\Sigma V^\top $$

So we have answered why we are using orthonormal eigenvectors and values of $A^TA$ for decomposing the matrix $A$ with the above derivation.

$$ Singular : Value : Decomposition $$

Why did you even bring up mahalanobis distance and singular value decomposition together?

Look at the formulas of Mahalanobis distance and SVD they look similar. But that isn't enough, we need to answer if Mahalanobis distance is doing SVD. This intuition is partly right. Mahalanobis distance is prefere because the formula converts an anisotropic distribution (uneven) to isotropic distribution.

From the previous formulas we can infer that covariance matrix is

$$ \Sigma_{cov} = \frac{1}{n-1}\mathbf{X_c^T}\mathbf{X_c} $$ $$ \mathbf{X_c} = US V^T $$

Substituting:

$$ \mathbf{X_c^T X_c} = (USV^\top)^\top(USV^\top) = VS^\top U^\top USV^\top = VS^2V^\top $$

(using $U^\top U = I$ and $S^\top S = S^2$ since $S$ is diagonal)

Therefore:

$$ \Sigma_{cov} = \frac{1}{n-1}\mathbf{V S^2 V^\top} $$

Note the order — it's $VS^2V^\top$, not $V^\top S^2 V$. This matters: it tells you $V$ (not $V^\top$) holds the eigenvectors as its columns, matching the standard eigendecomposition convention $\Sigma = V\Lambda V^\top$.

Hence eigenvalues of $\Sigma_{cov}$

$$ \lambda_i = \frac{\sigma^2_i}{n-1} $$

where $\sigma_i$ are the singular values of $\mathbf{X_c}$.

Is Mahalanobis distance doing SVD?

Short answer: partly right, partly wrong. Mahalanobis distance uses the same underlying machinery as SVD — rotation by eigenvectors, scaling by eigenvalues — but it is not performing SVD itself. What it's actually doing is eigendecomposition, applied specifically to the covariance matrix, followed by a distance computation that SVD alone doesn't give you.

What eigendecomposition actually is

For a symmetric matrix like $\Sigma$, eigendecomposition is:

$$ \Sigma = V \Lambda V^\top $$

$V$: columns are eigenvectors of $\Sigma$ (same $V$ on both sides — this only works because $\Sigma$ is symmetric)
$\Lambda$: diagonal matrix of eigenvalues $\lambda_i$

Compare this to general SVD:

$$ A = U \Sigma_A V^\top $$

Two different matrices $U$ and $V$ — because $A$ is not necessarily symmetric, so its "natural input direction" and "natural output direction" can differ
$\Sigma_A$ contains singular values $\sigma_i \geq 0$, not eigenvalues (which can be negative or complex for a non-symmetric $A$)

Why they coincide for $\Sigma$

Since the covariance matrix $\Sigma$ is symmetric and positive semi-definite:

Its eigenvalues are guaranteed real and $\geq 0$
Its eigenvectors are orthogonal
Therefore $U = V$ in its SVD, and its singular values equal its eigenvalues exactly

So for $\Sigma$ specifically, eigendecomposition and SVD collapse into the same operation. That's the source of the "Mahalanobis is doing SVD" intuition — it isn't wrong, it's just that SVD and eigendecomposition happen to be indistinguishable for this one special (symmetric PSD) case.

Building Mahalanobis distance from the eigendecomposition

Start from:

$$ d_M^2 = e^\top \Sigma^{-1} e $$

Substitute $\Sigma = V\Lambda V^\top \Rightarrow \Sigma^{-1} = V\Lambda^{-1}V^\top$ (valid since $V^{-1} = V^\top$ for orthogonal $V$):

$$ d_M^2 = e^\top V \Lambda^{-1} V^\top e = (V^\top e)^\top \Lambda^{-1} (V^\top e) $$

Let $\tilde{e} = V^\top e$ — this is $e$ rotated into the eigenvector basis. Then:

$$ d_M^2 = \tilde{e}^\top \Lambda^{-1} \tilde{e} = \sum_{i=1}^{d} \frac{\tilde{e}i^2}{\lambda_i} = \sum{i=1}^{d}\left(\frac{\tilde{e}_i}{\sqrt{\lambda_i}}\right)^2 = \left| \Lambda^{-1/2} V^\top e \right|^2 $$

The actual comparison: SVD pipeline vs Mahalanobis pipeline

Step	SVD applied to a vector ($A\mathbf{x}$)	Mahalanobis distance
1	rotate: $V^\top \mathbf{x}$	rotate: $V^\top e$
2	scale: multiply by $\sigma_i$	scale: divide by $\sqrt{\lambda_i}$
3	rotate again: $U(\cdot)$	no third rotation — take the norm instead
output	a transformed vector $\mathbf{y}$	a scalar distance $d_M$

Two concrete differences:

SVD stretches; Mahalanobis compresses. SVD's $\Sigma_A$ multiplies by $\sigma_i$ (stretch). Mahalanobis's $\Lambda^{-1/2}$ divides by $\sqrt{\lambda_i}$ (compress) — it's undoing the natural spread of the data, not applying it.
SVD ends in a rotation; Mahalanobis ends in a norm. The third SVD step rotates the scaled vector back into output space, giving another vector. Mahalanobis instead takes $|\cdot|^2$ at that point, collapsing everything to a single number.

What this operation is actually called

The two-step transform $\Lambda^{-1/2}V^\top$ (rotate, then scale-down) is called whitening — it takes an anisotropic (unevenly spread, correlated) distribution and turns it into an isotropic (evenly spread in all directions, uncorrelated) one:

$$ \hat{e} = \Lambda^{-1/2}V^\top e \quad \Rightarrow \quad \text{Cov}(\hat{e}) = I $$

Once whitened, ordinary Euclidean distance on $\hat{e}$ is the Mahalanobis distance on $e$ — because in the whitened space every direction is equally spread, so there's no shape left to correct for. That's the real punchline: Mahalanobis distance = Euclidean distance, computed after first eliminating the covariance.

Verdict

Mahalanobis distance is not doing SVD. It is doing:

Eigendecomposition of $\Sigma$ (which happens to coincide with SVD only because $\Sigma$ is symmetric PSD)
A whitening transform (rotate + inverse-scale) built from that eigendecomposition
A final Euclidean norm — a step SVD itself never includes

The formulas look similar because both are built from the same eigenvector/eigenvalue skeleton. But SVD's job is to decompose a general transformation into a vector output; Mahalanobis's job is to measure a distance that accounts for the shape of the data, ending in a scalar. Same skeleton, different destinations.

References

Get a detailed understanding here →

Building a $300 Research Robot — Part 0: Surviving the Setup

Thu, 28 May 2026 00:00:00 GMT

Intro

Robotics people use Linux. Not because they're trying to be difficult. Just because that's how it works — ROS2 (the standard robotics middleware everyone uses) runs natively on Ubuntu, drivers for robotics hardware are written for Linux first, and if you show up to a robotics lab running Windows you'll get a very polite but firm look.

So here I am. Ubuntu 24.04. Staring at a terminal. Building a robot.

This post is for anyone in the same situation — technically capable, completely new to Linux, and learning by breaking things.

What I'm building

I'm building a 2 DOF pan-tilt robot — a small assembly of two smart servo motors with a depth camera mounted on top. It can rotate left/right and tilt up/down. The camera watches the world, detects objects in 3D space, and the robot points toward them automatically — not because I hand-coded "if object is left, rotate left" but because a small AI policy learned how to do it through thousands of simulated practice runs.

Total hardware cost: about $239. The whole thing sits on a wooden block on my desk.

The software stack is:

ROS2 Jazzy — the robotics middleware that lets different parts of the system talk to each other
EMOS — a higher-level framework built on top of ROS2 that handles perception, planning, and control
Ubuntu 24.04 — the Linux distribution everything runs on

This post is just Part 0 — getting the environment working before touching any robot code.

Linux terms I had to learn before I could even read the error messages

Before the story, a quick glossary. Skip this if you already know Linux. If you don't — read it, because none of what follows will make sense without it.

The kernel is the core of the operating system. It is the first program that is uploaded to your device and the first program that activates. It acts as a bridge between the hardware and software input outputs. It controls which program gets access to how much RAM.

A driver is software that lets your OS talk to a specific piece of hardware. It allows the kernel to communicate with the hardware components.

DKMS (Dynamic Kernel Module Support) is a system that automatically recompiles drivers whenever the kernel updates. When a new kernel version installs, DKMS tries to rebuild all hardware drivers for it. If it fails — which it did in my case — you end up with a broken state where the new kernel exists but your GPU driver doesn't work on it.

apt is Ubuntu's package manager — the tool you use to install, update, and remove software. Most commands start with sudo apt .... The sudo part means "run this as administrator" — like right-clicking and choosing "Run as Administrator" on Windows.

dpkg is the lower-level tool that apt uses under the hood. You touch dpkg when apt can't fix itself and you have to go one level deeper.

When I say "broken packages," I mean packages that are partially installed or stuck in a state where the system doesn't know what to do with them.

HWE (Hardware Enablement Stack) is Ubuntu's system for keeping your kernel and drivers up to date on LTS (Long Term Support) releases. HWE kernels are newer than the default and pulled in automatically during upgrades. This is what pulled in kernel 6.17 without me asking for it.

My machine

Laptop: Dell G3 3500
GPU: NVIDIA GeForce GTX 1650 (4GB VRAM)
OS: Ubuntu 24.04.3 LTS
ROS2: Jazzy (pre-installed)
NVIDIA Driver: 580.65.06 (when I started)
CUDA: 13.0

The plan

Simple: update the system, then install EMOS.

sudo apt update && sudo apt upgrade -y

What this does: sudo apt update tells the system to check for newer versions of installed software. sudo apt upgrade -y actually downloads and installs those updates. The -y means "say yes to everything automatically."

This should have taken 5 minutes. Instead it took an hour.

What broke: kernel 6.17 vs NVIDIA 580

So the upgrade finished and I had a wall of red text staring back at me. Great start.

What happened, as far as I can tell: the upgrade quietly pulled in a new kernel — version 6.17 — as part of the routine update. I didn't ask for it, it just came along for the ride. The kernel is essentially the core of the operating system, the thing that talks to all your hardware. When it updates, all your hardware drivers need to be rebuilt for it — including the NVIDIA GPU driver. That rebuilding process is handled automatically by something called DKMS.

Now here's something that confused me at first — if 6.17 was broken, why was my laptop still running fine? The answer is that Linux doesn't replace the old kernel when a new one installs. It sits alongside it. Both live on your disk at the same time. When you boot up, a menu called GRUB appears (usually just for a second before it auto-selects) and picks which kernel to load. By default it picks the newest one.

Think of it like downloading an app update on your phone that gets stuck halfway. The old version still works. But now your storage has both the old and the broken new version sitting on it, and the app store is confused about what state everything is in.

So in my case:

Kernel 6.14 was already installed and working
The upgrade downloaded and partially installed kernel 6.17 alongside it
DKMS failed during that installation — 6.17 never fully completed
On reboot, GRUB tried to load 6.17 since it was newest — but fell back to 6.14 because the installation was broken
I ended up running 6.14 while 6.17 sat half-installed on the disk doing nothing except breaking apt DKMS failed with this:

Error! Bad return status for module build on kernel: 6.17.0-29-generic (x86_64)
dkms autoinstall on 6.17.0-29-generic/x86_64 failed for nvidia(10)

Honestly I'm still not 100% sure why. My best guess is that the kernel headers — the instruction manual DKMS needs to rebuild the driver — arrived in the wrong order during the upgrade and the rebuild started before everything was ready. It probably wasn't a genuine incompatibility, because later when I cleaned everything up the exact same driver compiled fine on kernel 6.17 without any changes. So something in the installation process just went sideways.

The result was four packages stuck in limbo — half installed, half not, blocking each other:

Errors were encountered while processing:
 linux-headers-6.17.0-29-generic
 linux-headers-generic-hwe-24.04
 linux-generic-hwe-24.04
 linux-image-6.17.0-29-generic
E: Sub-process /usr/bin/dpkg returned an error code (1)

The one good thing: my laptop was still running on the old kernel (6.14) so nothing was actually broken yet. The GPU still worked. I just had this mess sitting in the background that needed sorting out before I could do anything else.

Attempt 1: the wrong fix

My first instinct was sudo apt --fix-broken install followed by sudo apt autoremove.

apt --fix-broken install tries to resolve broken dependencies automatically. Reasonable first step.

apt autoremove removes packages that are "no longer needed." Sounds safe.

It wasn't.

autoremove looked at Ubuntu's dependency graph — which was unreliable because packages were in a broken state — and decided to remove:

The working NVIDIA 575 drivers
The older kernel 6.14.0-29 Quick detour: a dependency graph is just Ubuntu's way of tracking which software needs which other software to work. A video player might depend on an audio library, which depends on a codec package, and so on. Ubuntu keeps a map of all these relationships so when you install or remove something it knows what else to bring along or clean up. When packages are broken or half-installed, that map becomes unreliable — like Google Maps that doesn't know a road is closed. autoremove trusted the map, the map was wrong, and it removed things it shouldn't have.

Things I didn't ask it to remove. Things that were working fine.

The 6.17 kernel stayed because it had a dependency holding it in place:

linux-image-generic-hwe-24.04 depends on linux-image-6.17.0-29-generic

So now I had: broken 6.17 kernel, partially removed NVIDIA drivers, and a bigger mess than before.

Lesson: never run apt autoremove when packages are in a broken state.

What that means in plain terms: autoremove is a cleanup tool, not a repair tool. It's meant to be run when everything is healthy and you just want to tidy up unused packages. Running it when things are already broken is like asking someone to clean your house while it's on fire — they'll start throwing things out without knowing what's important and what isn't.

What I should have done instead:

Step 1 — Don't touch autoremove at all. When you see broken packages, fix them first.

Step 2 — Run these repair commands first:

sudo dpkg --configure -a
sudo apt --fix-broken install -y

dpkg --configure -a tells dpkg to finish anything it left half-done. apt --fix-broken install tells apt to look at what's broken and try to resolve it cleanly. These are the "try to repair yourself" commands.

Step 3 — Only run autoremove after those succeed. Once the system is healthy, autoremove can safely figure out what's genuinely unused.

The short version: see broken packages → run dpkg --configure -a and apt --fix-broken install → only then consider cleanup.

Attempt 2: trying to force-remove the 6.17 kernel

Since normal removal was blocked by dependencies, I tried force-removal:

sudo dpkg --remove --force-remove-reinstreq linux-image-6.17.0-29-generic linux-headers-6.17.0-29-generic linux-headers-generic-hwe-24.04 linux-generic-hwe-24.04

--force-remove-reinstreq removes packages stuck in a "needs reinstall" broken state, ignoring dependency checks. This got most of them — but not the image:

dpkg: dependency problems prevent removal of linux-image-6.17.0-29-generic:
 linux-image-generic-hwe-24.04 depends on linux-image-6.17.0-29-generic.

The meta-package linux-image-generic-hwe-24.04 was still holding the kernel image hostage. I had to remove that first.

The actual fix — in order

Step 1 — Remove the meta-package holding the kernel in place:

sudo dpkg --remove --force-remove-reinstreq linux-image-generic-hwe-24.04

A meta-package is a package that doesn't contain any software itself — it just declares dependencies on other packages. linux-image-generic-hwe-24.04 existed only to pull in linux-image-6.17.0-29-generic. Removing it broke the dependency chain.

Step 2 — Now remove the kernel image:

sudo dpkg --remove --force-remove-reinstreq linux-image-6.17.0-29-generic

Step 3 — Clean up dpkg's internal state:

sudo dpkg --configure -a
sudo apt --fix-broken install -y

dpkg --configure -a finishes configuring any packages that got interrupted mid-install. apt --fix-broken install resolves remaining dependency problems. Run both every time after force-removing packages.

Step 4 — Reinstall NVIDIA drivers:

sudo apt install nvidia-driver-580 -y

This time, apt pulled in version 580.159.03 — a newer release that actually supports kernel 6.17. The original upgrade had been trying to install this all along. The broken dpkg state prevented it from completing. Clearing the state let it finish.

Step 5 — Reboot:

sudo reboot

After reboot — everything working:

uname -r
# 6.17.0-29-generic ✓

nvidia-smi
# NVIDIA-SMI 580.159.03   Driver Version: 580.159.03   CUDA Version: 13.0 ✓

The irony: Think about what the upgrade was actually trying to do — bring in kernel 6.17 and a newer NVIDIA driver that supports it. That's the correct thing to do. The problem wasn't the plan, it was the execution. Packages installed in the wrong order, DKMS ran before the kernel headers were ready, and dpkg froze halfway through. When I force-removed the broken packages and reinstalled cleanly, all I did was give that original upgrade a clean slate to finish what it started. Same kernel, same driver, same end result — just without the half-done mess in the way. Mildly annoying in hindsight. Two hours of debugging to end up exactly where the upgrade was trying to take me anyway.. Once I cleared the mess, it completed exactly as intended — new kernel, new driver, everything working.

Installing EMOS

With the system clean, EMOS installed without drama.

Install the EMOS CLI:

curl -sSL https://raw.githubusercontent.com/automatika-robotics/emos/main/stack/emos-cli/scripts/install.sh | sudo bash

curl downloads a file from a URL. Piping it to sudo bash runs it immediately with admin privileges. This is standard practice for installing developer tools from trusted sources.

Install EMOS natively:

emos install --mode native

Native mode builds EMOS directly into your existing ROS2 installation. The other modes are Container (runs inside Docker, no ROS2 needed) and Pixi (isolated environment, no root required). Since I already have ROS2 Jazzy, native is cleanest.

Gotcha — ROS2 not sourced:

First attempt failed with:

E: Unable to locate package python3-ament-package

This happened because the terminal didn't know where ROS2 was installed. In Linux, software installed in non-standard locations needs to be "sourced" — you run a script that tells the current terminal session where everything lives.

Fix:

source /opt/ros/jazzy/setup.bash
emos install --mode native

source loads a configuration file into the current terminal session. You need to run this in every new terminal before using ROS2 commands. To make it automatic so you never have to think about it again:

echo "source /opt/ros/jazzy/setup.bash" >> ~/.bashrc

This adds the source command to ~/.bashrc — a file that runs automatically every time you open a new terminal. After running this once, every new terminal will have ROS2 available automatically.

After sourcing, EMOS installed cleanly.

TL;DR — What to do when a kernel upgrade breaks your NVIDIA driver

If you ran sudo apt upgrade, got a wall of red text, and nvidia-smi stopped working, this is probably what happened and how to fix it.

Step 1 — Try the gentle fix first:

sudo apt --fix-broken install -y
sudo dpkg --configure -a

Run these first. Sometimes this alone fixes it. If it doesn't, continue.

Step 2 — Find the broken kernel packages:

dpkg -l | grep linux-image
dpkg -l | grep linux-headers

Look for packages marked with iF or iU at the start of the line — those are broken. Note their names.

Step 3 — Force-remove them, meta-packages first:

sudo dpkg --remove --force-remove-reinstreq <package-name>

Meta-packages are the ones without a version number in the name — like linux-image-generic-hwe-24.04. Remove those before the versioned ones like linux-image-6.17.0-29-generic.

Step 4 — Clean up and reinstall your NVIDIA driver:

sudo dpkg --configure -a
sudo apt --fix-broken install -y
sudo apt install nvidia-driver-$(nvidia-smi --query-gpu=driver_version --format=csv,noheader | cut -d. -f1) -y

Step 5 — Reboot and verify:

sudo reboot
# After reboot:
uname -r
nvidia-smi

If nvidia-smi shows your GPU, you're done.