Exploration of Linear Model fit using observation weights and parameter constraints to handle poor quality data

Author(s): Ullas M S Rao, Mattias Jönsson

Abstract

In the machine learning world, it is quite a common scenario to come across with the issues of variability and heteroscedasticity. One of the ways to address this is to assign weights to the observations. This helps us to ensure that the outliers are given lesser weights. Thus, reducing their effect on the model. Also, in certain cases, it is required to add some linear constraints on the estimates to restrict the predictor to be greater or lesser than a certain value or within a given range of values, this is mostly driven by the deep understanding of the real-world behavior and by certain business scenarios. Based on data analysis, it was found that market volume had an impact on the costs which required us to weigh the observations proportional to the volume of the container movement over twelve months. Added to it, algorithm the model had to accurately reflect the higher cost higher for refrigerated containers over the containers for dry cargos, given that all other features remained the same. This is a clear case of the combination of Linear regression model with weights and subjected to certain constraints.

Picture of Association of Data Scientists

Association of Data Scientists

The Chartered Data Scientist Designation

Achieve the highest distinction in the data science profession.

Elevate Your Team's AI Skills with our Proven Training Programs

Strengthen Critical AI Skills with Trusted Generative AI Training by Association of Data Scientists.

Our Accreditations

Get global recognition for AI skills

Chartered Data Scientist (CDS™)

The highest distinction in the data science profession. Not just earn a charter, but use it as a designation.

Certified Data Scientist - Associate Level

Global recognition of data science skills at the beginner level.

Certified Generative AI Engineer

An upskilling-linked certification initiative designed to recognize talent in generative AI and large language models

Join thousands of members and receive all benefits.

Become Our Member

We offer both Individual & Institutional Membership.

Subscribe to our Newsletter