SVM:
Finally...
The default settings for the provided SVM performed quite reasonably on the baseline described so far.
It received a score on the order of .64.
When considering extra-credit opportunities, I decided that I don't know enough about basis expansions.
This led to a strong desire to try a couple.
The first basis expansion I tried is a pretty simple one. I decided to try concatenating the feature vector
with its point-wise square. I.e., something like newFeature = [oldFeature (oldFeature .* oldFeature)].
Adding the square terms allows the decision boundary to take the form of any quadratic surface, such
as a sphere, parabola, hyperbola, or other higher dimensional analogues.
Something really nice about this particular basis is, since x and x^2 have x DOT x^2 = 0 when the inner
product is taken on a symmetric interval, it is not necessary to employ a kernelized SVM for this expansion:
The kernel (in basis form) would turn out to be the identity matrix, so we can just create the feature vectors
procedurally and never inform the solver that this SVM is not linear.
This gave a point of boost or so, bringing the performance up to something on the order of .65~.66.
The second basis expansion I tried is the infamous Gaussian RBF kernel expansion. This required
a little more surgery
than the previous example.
First, at train time, it was necessary to generate the RBF kernel from the training images. This was done using the
compute_kernel() function provided on the primal SVM website.
Then the SVM solver had to be configured as non-linear, and the RBF model saved away. This new model was very high
dimensional, with as many dimensions as training images.
At test time, it was necessary to compute each test-histogram's representation in the RBF space. This was
accomplished by an elegant application of the compute_kernel() function mentioned above. Then the RBF representation
was tested against the RBF model in the usual way and the scores kept as before.
The RBF kernel gave another increase of a point or two, ending at .6707 once the regularization parameter was tuned.
An interesting thing to note is that there was actually a local maximum of performance when varying the regularization
parameter. Decreasing it from 1 to .1 gave a 10 point boost from .47 to .57, decreasing to .01 brought it up to .66 or so,
and .001 took it to .67. Continuing the logarithmic decay, performance decreased to .61 and then to .53.
RBF was fun, and I definitely understand kernelized SVMs better as a result of this exercise.