Data Trigger

Gradient tree boosting in the cloud

Posted on November 13, 2020

A cloud computing experiment with two slightly different implementations of gradient boosted trees : LightGBM and XGBoost. Let us evaluate how these two algorithms do on a moderately large dataset, regarding both accuracy and speed. [Read More]

python xgboost lightgbm gradient boosted trees cloud computing machine learning superconductors paperspace

Visualizing the sum of two random variables

Posted on October 31, 2020

Take too independent random variables identically distributed. Question: if their sum is large, are they likely to be both large ? Let us examine this question with contour plots. [Read More]

r ggplot2 contour dataviz data visualization statistics

Adding totals and subtotals rows with pandas or the tidyverse

Posted on October 18, 2020

When dealing with a dataframe, generating aggregate data is a very common task. In my experience, presenting the summary statistics for the whole population or for subgroups directly in the dataframe can be useful, if not necessary. Today, I present my recipe to achieve this with the pandas and tidyverse packages. [Read More]

python r pandas tidyverse total row subtotals aggregate groupby

Back to basics : Scaling train and test samples.

Posted on October 12, 2020

Splitting and scaling a dataset seems easy. Well, it is admittedly not that hard, however it can be tricky. Today we will see how to properly split and scale a dataset, as this step if often necessary before any ML wizardry. Let us do this with a few R & Python packages/modules. [Read More]

scaling normalize standardize spark pyspark python r dplyr caret

Weighted Random Forest with Spark 3

Posted on September 6, 2020

The third version of the number one distributed computing framework Spark was released in June 2020. Sample weights support was implemented for tree-based algorithms : decision tree, gradient tree boosting and random forest. Today we experiment with this new feature on an imbalanced dataset about credit card fraud. [Read More]

spark pyspark python weight fraud random forest

Outlier detection

Posted on August 31, 2020

In this post, I try to define what an outlier is and I present several ways to approach the problem of anomaly detection. Then, I present the Local Outlier Factor algorithm and apply it on a specific dataset to show its power, using both Python and R. I also compare its performance with the Isolation Forest method. [Read More]

outlier r python anomaly isolation lof