The elbow method helps determine the optimal number of clusters (K) by analyzing how the Sum of Squared Errors (SSE) changes with different K values.
Key Points:
Try it: Generate new data to see how the elbow curve changes. Look for where the curve starts to level off - this often suggests a reasonable choice for K.
Start with randomly chosen centroids \(\mu_1\), \(\mu_2\), ..., \(\mu_k\).
Let \(\,X = \{x_1, x_2, ..., x_n\}\) be our dataset, where each \(\,x_i\,\) is a \(\,d\)-dimensional vector.
The distance between two points \(\,x = (x_1, ..., x_m)\) and \(\,y = (y_1, ..., y_m)\) is calculated as:
$$d(x,y)^2 = \sum_{j=1}^m (x_j - y_j)^2 = \|x - y\|_2^2$$
This distance is used in the assignment step to determine the nearest centroid for each point.
The objective function to be minimized is:
$$SSE = \sum_{i=1}^n \sum_{j=1}^k w^{(i,j)} \|\mathbf{X}^{(i)} - \boldsymbol{\mu}^{(j)}\|_2^2$$
Where: