Clustering

This endpoint is to process the data set, cluster the customers and visualize results.

• Data processing: features with a single value or with number of values equals to the number of samples ('customerId' excluded) will be removed, one of highly correlated features (threshold > 0.8) will be removed. Then categorical data will subject to one-hot-coding, and numeric data will be scaled.

• For the clustering, users are allowed to select features for training clustering model (if no features are selected, all the features will be included), linkage criteria (single, complete, average and ward, the default is ward method). Users are allowed to input a number of clusters (integer >= 0 and <=10). If 0 or 1 is given, the endpoint will run a silhouette analysis to get a recommended number of clusters, and shows the details of the clustering results.

• Finally, users are able to select features for visualization.

  1. If two features are selected, a scatter plot based on the clusters will be shown.

  2. If more than two features are selected, users are allowed to select methods to reduce dimension:

  • PCA: The selected features will subject to PCA procedure to obtain two prinical components, which will be used as x and y axis for visualization.

  • TSNE: There are a wide variety of tuning parameters for TSNE. The default values will be used for these parameters.

  • Note: For both methods, if the sample size is less than 3000, the whole dataset will be used for plot. If the sample size is more than 3000, 3000 samples will be selected by stratified sampling. For TSNE, since it is very expensive in running time, if the number of features is very high (>50), PCA will first be applied to select the 50 most important features.

  • PCA vs TSNE: (1) t-SNE is computationally expensive. (2) PCA it is a mathematical technique, but t-SNE is a probabilistic one. (3) Linear dimensionality reduction algorithms, like PCA, concentrate on placing dissimilar data points far apart in a lower dimension representation. But in order to represent high dimension data on low dimension, non-linear manifold, it is essential that similar data points must be represented close together. (4) Sometimes in t-SNE, different runs with the same hyperparameters may produce different results hence multiple plots must be observed before making any assessment. (5) Since PCA is a linear algorithm, it will not be able to interpret the complex polynomial relationship between features while t-SNE is made to capture exactly that.

Request Body

Response Body