How to one hot encode variant length features?

Given a list of variant length features:

features = [
    ['f1', 'f2', 'f3'],
    ['f2', 'f4', 'f5', 'f6'],
    ['f1', 'f2']
]

where each sample has variant number of features and the feature dtype is str and already one hot.

In order to use feature selection utilities of sklearn, I have to convert the features to a 2D-array which looks like:

    f1  f2  f3  f4  f5  f6
s1   1   1   1   0   0   0
s2   0   1   0   1   1   1
s3   1   1   0   0   0   0

How could I achieve it via sklearn or numpy?

Answers:

Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.

Method 1

You can use MultiLabelBinarizer present in scikit which is specifically used for doing this.

Code for your example:

features = [
            ['f1', 'f2', 'f3'],
            ['f2', 'f4', 'f5', 'f6'],
            ['f1', 'f2']
           ]
from sklearn.preprocessing import MultiLabelBinarizer
mlb = MultiLabelBinarizer()
new_features = mlb.fit_transform(features)

Output:

array([[1, 1, 1, 0, 0, 0],
       [0, 1, 0, 1, 1, 1],
       [1, 1, 0, 0, 0, 0]])

This can also be used in a pipeline, along with other feature_selection utilities.

Method 2

Here’s one approach with NumPy methods and outputting as pandas dataframe –

import numpy as np
import pandas as pd

lens = list(map(len, features))
N = len(lens)
unq, col = np.unique(np.concatenate(features),return_inverse=1)
row = np.repeat(np.arange(N), lens)
out = np.zeros((N,len(unq)),dtype=int)
out[row,col] = 1

indx = ['s'+str(i+1) for i in range(N)]
df_out = pd.DataFrame(out, columns=unq, index=indx)

Sample input, output –

In [80]: features
Out[80]: [['f1', 'f2', 'f3'], ['f2', 'f4', 'f5', 'f6'], ['f1', 'f2']]

In [81]: df_out
Out[81]: 
    f1  f2  f3  f4  f5  f6
s1   1   1   1   0   0   0
s2   0   1   0   1   1   1
s3   1   1   0   0   0   0

All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0