LIBLINEAR FAQ

Some questions are listed in LIBSVM FAQ.

Table of Contents

Please check our explanation on the LIBLINEAR webpage. Also see appendix C of our SVM guide.

Please see the descriptions at LIBLINEAR page.

Please cite the following paper:

R.-E. Fan, K.-W. Chang, C.-J. Hsieh, X.-R. Wang, and C.-J. Lin. LIBLINEAR: A Library for Large Linear Classification, Journal of Machine Learning Research 9(2008), 1871-1874. Software available at http://www.csie.ntu.edu.tw/~cjlin/liblinear

The bibtex format is

@Article{REF08a,
  author = 	 {Rong-En Fan and Kai-Wei Chang and Cho-Jui Hsieh and Xiang-Rui Wang and Chih-Jen Lin},
  title = 	 {{LIBLINEAR}: A Library for Large Linear Classification},
  journal = 	 {Journal of Machine Learning Research},
  year = 	 {2008},
  volume =	 {9},
  pages =	 {1871--1874}
}

See the change log and directory for earlier/current versions.

Generally we recommend linear SVM as its training is faster and the accuracy is competitive. However, if you would like to have probability outputs, you may consider logistic regression.

Moreover, try L2 regularization first unless you need a sparse model. For most cases, L1 regularization does not give higher accuracy but may be slightly slower in training.

Among L2-regularized SVM solvers, try the default one (L2-loss SVC dual) first. If it is too slow, use the option -s 2 to solve the primal problem.

[rcv1_test_labels,rcv1_test_inst] = libsvmread('../rcv1_test.binary');
save rcv1_test.mat rcv1_test_labels rcv1_test_inst;

save -mat7-binary rcv1_test.mat rcv1_test_labels rcv1_test_inst;

load rcv1_test.mat

help save

-s 2

They should be very similar. However, sometimes the difference may not be small. Note that LIBLINEAR does not use the bias term b by default. If you observe very different results, try to set -B 1 for LIBLINEAR. This will add the bias term to the loss function as well as the regularization term (w^Tw + b^2). Then, results should be closer.

You can use grid.py of LIBSVM (after version 3.16) to check cross validation accuracy of different C. Two options must be specified.

1. `-svmtrain train': use the command `train' of LIBLINEAR
2. `-log2g null': do not grid with `g'

> python grid.py -log2c -3,0,1 -log2g null -svmtrain ./train heart_scale

We guess that you are comparing

> time ./train -s 0 -v 5 -e 0.001 data

1. The above timeing of LIBLINEAR includes time for reading data, but in the paper we exclude that part.
2. In the paper, to conduct 5-fold (or 2-fold) CV we group folds used for training as a separate matrix, but LIBLINEAR simply uses pointers of the corresponding instances. Therefore, in doing matrix-vector multiplications, the former sequentially uses rows in a continuous segment of the memory, but the latter does not. Thus, LIBLINEAR may be slower but it saves the memory.

We carefully studied such issues, and decided to use the current setting. For data classification, one doesn't need very accurate solution, so numerical issues are less important. Moreover, log1p is not available on all platforms. Please let us know if you observe any numerical problems.

Assume k is the total number of classes and n is the number of features. In the model file, after the parameters, there is an n*k matrix W, whose columns are obtained from solving two-class problems: 1 vs rest, 2 vs rest, 3 vs rest, ...., k vs rest. For example, if there are 4 classes, the file looks like:

+-------+-------+-------+-------+
| w_1vR | w_2vR | w_3vR | w_4vR |
+-------+-------+-------+-------+

Please see the answer in LIBSVM faq.

To correctly obtain decision values, you need to check the array

label

LIBSVM uses more advanced techniques for SVM probability outputs. The code is a bit complicated so we haven't decided if including it is suitable or not.

If you really would like to have probability outputs for SVM in LIBLINEAR, you can consider using the simple probability model of logistic regression. Simply modify the following subrutine in linear.cpp.

int check_probability_model(const struct model *model_)
{
	return (model_->param.solver_type==L2R_LR ||

int check_probability_model(const struct model *model_)
{
	return 1;

Some LIBLINEAR solvers consider the primal problem, so support vectors are not obtained during the training procedure. For dual solvers, we output only the primal weight vector w, so support vectors are not stored in the model. This is different from LIBSVM.

To know support vectors, you can modify the following loop in solve_l2r_l1l2_svc() of linear.cpp to print out indices:

	for(i=0; i<l; i++)
	{
		v += alpha[i]*(alpha[i]*diag[GETI(i)] - 2);
		if(alpha[i] > 0)
			++nSV;
	}

This FAQ is for solvers. For multiclass classification, please check How to speedup multiclass classification using OpenMP instead.

Because of the design of LIBLINEAR's solvers, it is not easy to achieve good speedup using OpenMP. However, by the following steps, we can still achieve some speedup for primal solvers (-s 0, 2, 11).

1. In Makefile, add -fopenmp to CFLAGS.
2. In classes l2r_lr_fun and l2r_l2_svc_fun of linear.cpp, modify the for loop Xv to:

   #pragma omp parallel for private (i)
   	for(i=0;i<l;i++)
   

   In l2r_l2_svc_fun, modify the for loop in subXv to:

   #pragma omp parallel for private (i)
   	for(i=0;i<sizeI;i++)
   

3. Using 8 cores on the sets rcv1_test.binary and mnist8m.scale.

   %export OMP_NUM_THREADS=8
   %time ./train -s 2 rcv1_test.binary
   0m45.250s
   %time ./train -s 2 mnist8m.scale
   59m41.300s
   

   Using standard LIBLINEAR

   %time ./train -s 2 rcv1_test.binary
   0m55.657s
   %time ./train -s 2 mnist8m.scale
   78m59.452s
   

Please take the following steps. Note that it works only for -s 0, 1, 2, 3, 5, 7.

In Makefile, add -fopenmp to CFLAGS.

In linear.cpp, replace the following segment of code

				model_->w=Malloc(double, w_size*nr_class);
				double *w=Malloc(double, w_size);
				for(i=0;i<nr_class;i++)
				{
					int si = start[i];
					int ei = si+count[i];

					k=0;
					for(; k<si; k++)
						sub_prob.y[k] = -1;
					for(; k<ei; k++)
						sub_prob.y[k] = +1;
					for(; k<sub_prob.l; k++)
						sub_prob.y[k] = -1;

					train_one(&sub_prob, param, w, weighted_C[i], param->C);

					for(int j=0;j<w_size;j++)
						model_->w[j*nr_class+i] = w[j];
				}
				free(w);

				model_->w=Malloc(double, w_size*nr_class);
#pragma omp parallel for private(i) 
				for(i=0;i<nr_class;i++)
				{
					problem sub_prob_omp;
					sub_prob_omp.l = l;
					sub_prob_omp.n = n;
					sub_prob_omp.x = x;
					sub_prob_omp.y = Malloc(double,l);

					int si = start[i];
					int ei = si+count[i];

					double *w=Malloc(double, w_size);

					int t=0;
					for(; t<si; t++)
						sub_prob_omp.y[t] = -1;
					for(; t<ei; t++)
						sub_prob_omp.y[t] = +1;
					for(; t<sub_prob_omp.l; t++)
						sub_prob_omp.y[t] = -1;

					train_one(&sub_prob_omp, param, w, weighted_C[i], param->C);

					for(int j=0;j<w_size;j++)
						model_->w[j*nr_class+i] = w[j];
					free(sub_prob_omp.y);
					free(w);
				}

%export OMP_NUM_THREADS=8
%time ./train -s 2 rcv1_test.multiclass
2m4.019s
%time ./train -s 1 rcv1_test.multiclass
0m45.349s

%time ./train -s 2 rcv1_test.multiclass
6m52.237s
%time ./train -s 1 rcv1_test.multiclass
1m51.739s

Please check this page