Arduino Low Source Machine Learning Model

So far, we have used SVM (Support Vector Machine) as our main classifier to move a Machine learning model to a microcontroller: but recently, we wanted to introduce you to an interesting alternative that can be much smaller, providing similar accuracy!

Relevance Vector Machine

We chose SVM as our focus for the MicroML framework because we knew that memory usage can be efficient after supporting vector encoding is moved to flat C.

We were able to apply many real-world models (motion identification, wake-up word detection) to small microcontrollers such as the old Arduino Nano (32 kb flash, 2 kb RAM).

The compromise of my application was to sacrifice the flash space (which is usually quite large) to save as much RAM as possible, which is usually the most limiting factor.

Because of this application, if your model grows in size (high-size data or data that cannot be well separated), the generated code still fits in RAM, but the available flash "overflows".

In a pair of our previous post, we warned that choosing a model before installing a model on an MCU can be a necessary step, since first you should check if it is suitable. If not, you should train another model in the hope of getting fewer support vectors, since each contributes to the increase in code size.

A new algorithm: Conformity Vector Machines

It was fortunate that we came across a new algorithm calledthe Relevance Vector Machine. It was patented by Microsoft until last year, but is now free to use.

Bayesan uses the same SVM formulation (an aggravated kernel total) by applying the model.

This serves to obtain the possibilities of classification results that are completely missing from the SVM in the first place.

In the second order, the algorithm tries to learn a much lesser representation of support vectors, as you can see in the picture below.

Relevance Vector Machine

A fairly lightweight model that can achieve high accuracy

Training the Classifier

Apparently, it didn't have an app because it was patented. Fortunately, there is an app that monitors the sklearn paradigm.You must install it:

pip install Cython
pip install https://github.com/AmazaspShumik/sklearn_bayes/archive/master.zip

It is very easy to train a classifier, since the interface is the usual compliance prediction.

from sklearn.datasets import load_iris
from skbayes.rvm_ard_models import RVC
import warnings

# I get tons of boring warnings during training, so turn it off
warnings.filterwarnings("ignore")

iris = load_iris()
X = iris.data
y = iris.target
clf = RVC(kernel='rbf', gamma=0.001)
clf.fit(X, y)
y_predict = clf.predict(X)

SVC The constructor's parameters are similar to those of the classifier in the sklearn:

  • kernel: linear, poly, one of the rbf
  • degree: if kernel=poly
  • gamma: if kernel=polyorkernel=rbf

You can read the documentation in sklearn to learn more.

Moving to C

Now that we have a trained classifier, we need to move it to flat C, which is compiled in the microcontroller of our choice.

 from micromlgen import port_rvm

 clf = get_rvm_classifier()
 c_code = port_rvm(clf)
 print(c_code)

You now have a plain C code that you can place in any microcontroller.

Performance Comparison

To test the effectiveness of this new algorithm, we applied it to the datasets we created in my previous submissions by comparing the size and accuracy of both SVM and RVM side by side.

DatasetDVMRVMDelta
Flash(bytes)Accuracy(%)Flash(bytes)Accuracy(%)Flashaccuracy
RGB colors45841003580100-%22-0%
Accelerometer movements(linear core)3688892705685-%80-7%
Accelerometer movements(gauss core)4534895776695-82%-0%
Wifi positioning46411003534100-24%-0%
Word wake-up(linear core)1809886360253-%80-%33
Wake-up with the hand(gauss core)2178890482662-78-%28

As you can see, the results are quite surprising:

  • You can achieve up to 82% space reduction in the high-dimensional data set without any loss of accuracy (gauss core and accelerometer movements).
  • Sometimes you may not get good accuracy (the wake-up word is no more than 62% in the data set).

As in any case, you should test which of the two algorithms gives the best result for your use case, but there are a few instructions you can follow:

  • If you need the highest accuracy, if you have enough space, SVM can probably get less better performance.
  • If you need small space or top speed, test whether RVM achieves satisfactory accuracy.
  • If both SVM and RVM achieve comparable performance, continue with RVM: in most cases it is much lighter than SVM and works faster.

Dimension Comparison

For reference, here are the codes created for an SVM classifier and an RVM classifier to classify the IRIS data set.

uint8_t predict_rvm(double *x) {
    double decision[3] = { 0 };
    decision[0] = -0.6190847299428206;
    decision[1] = (compute_kernel(x, 6.3, 3.3, 6.0, 2.5) - 72.33233 ) * 0.228214 + -2.3609625;
    decision[2] = (compute_kernel(x, 7.7, 2.8, 6.7, 2.0) - 81.0089166 ) * -0.29006 + -3.360963;
    uint8_t idx = 0;
    double val = decision[0];
    for (uint8_t i = 1; i < 3; i++) {
          (decision[i] > val) {
            idx = i;
            val = decision[i];
        }
    }
    return idx;
}

int predict_svm(double *x) {
    double kernels[10] = { 0 };
    double decisions[3] = { 0 };
    int votes[3] = { 0 };
        kernels[0] = compute_kernel(x, 6.7 , 3.0 , 5.0 , 1.7 );
        kernels[1] = compute_kernel(x, 6.0 , 2.7 , 5.1 , 1.6 );
        kernels[2] = compute_kernel(x, 5.1 , 2.5 , 3.0 , 1.1);
        kernels[3] = compute_kernel(x, 6.0 , 3.0 , 4.8 , 1.8 );
        kernels[4] = compute_kernel(x, 7.2 , 3.0 , 5.8 , 1.6 );
        kernels[5] = compute_kernel(x, 4.9 , 2.5 , 4.5 , 1.7 );
        kernels[6] = compute_kernel(x, 6.2 , 2.8 , 4.8 , 1.8 );
        kernels[7] = compute_kernel(x, 6.0 , 2.2 , 5.0 , 1.5 );
        kernels[8] = compute_kernel(x, 4.8 , 3.4 , 1.9 , 0.2 );
        kernels[9] = compute_kernel(x, 5.1 , 3.3 , 1.7 , 0.5 );
        decisions[0] = 20.276395502
                    + kernels[0] * 100.0
                    + kernels[1] * 100.0
                    + kernels[3] * -79.351629954
                    + kernels[4] * -49.298850195
                    + kernels[6] * -40.585178082
                    + kernels[7] * -30.764341769
        ;
        decisions[1] = -0.903345464
                    + kernels[2] * 0.743494115
                    + kernels[9] * -0.743494115
        ;
        decisions[2] = -1.507856504
                    + kernels[5] * 0.203695177
                    + kernels[8] * -0.160020702
                    + kernels[9] * -0.043674475
        ;
        votes[decisions[0] > 0 ? 0 : 1] += 1;
        votes[decisions[1] > 0 ? 0 : 2] += 1;
        votes[decisions[2] > 0 ? 1 : 2] += 1;
                int classVal = -1;
        int classIdx = -1;
        for (int i = 0; i < 3; i++) {
             if (votes[i] > classVal) {
                classVal = votes[i];
                classIdx = i;
            }
        }
        return classIdx;
}

As you can see, RVM actually calculates only 2 cores and performs 2 multiplications.SVM calculates 10 cores and performs 13 multiplications.

This is a recurring model, so RVM is much faster in the inference process.