Implementation and comparison of three basic Gradient Descent variants

Image by geralt from Pixabay modified by author

Gradient Descent algorithm is an iterative first-order optimisation method to find the function’s local minimum (ideally global). Its basic implementation and behaviour I’ve described in my other article here. This one focuses on three main variants in terms of the amount of data the algorithm uses to calculate the gradient and to make steps.

These 3 variants are:

  • Batch Gradient Descent (BGD)
  • Stochastic Gradient Descent (SGD)
  • Mini-Batch Gradient Descent (mBGD)

In this article, we will see their performance in a simple linear regression task.

A quick recap — a univariate linear function is defined as:

The Gradient Descent method lays the foundation for machine learning and deep learning techniques. Let’s explore how does it work, when to use it and how does it behave for various functions.

Image by JohnsonMartin from Pixabay

1. Introduction

Gradient descent (GD) is an iterative first-order optimisation algorithm used to find a local minimum/maximum of a given function. This method is commonly used in machine learning (ML) and deep learning(DL) to minimise a cost/loss function (e.g. in a linear regression). Due to its importance and ease of implementation, this algorithm is usually taught at the beginning of almost all machine learning courses.

However, its use is not limited to ML/DL only, it’s being widely used also in areas like:

  • control engineering (robotics, chemical, etc.)
  • computer games
  • mechanical engineering

That’s why today we will get a deep dive into the…

Smart homes and cities, Industry 4.0, drones and more

Photo by Johannes Plenio on Unsplash

Have you ever wondered how our life in the future may look like? What technology will bring us? How will it influence our lifestyle? Let us look at one possible scenario.

Smart Homes

It’s Saturday morning, automatic window curtains open when your alarm clock goes off. You open your eyes and your voice assistant Amazon Alexa welcomes you and briefly summarizes your today’s schedule. Meantime a smart cafe machine starts preparing your favorite café Americano and a toast roaster heats two slices of bread for your breakfast. When you go under the digital shower the intelligent system remembers your preferred…

Learn how to maximize the chances of getting a work offer by optimizing the number of interviews per month using a statistical approach and a bit of Python.

Photo by Cytonn Photography on Unsplash

On a job market, every job-seeker tries to maximize their chances of a successful outcome (getting a work offer). One of the popular discussion topics between themselves and job advisors is a recommended number of active applications and job interview per month. Having it too small means not giving yourself enough opportunities/chances for the success but also having too many appointment means not being able to properly prepare for them.

So how to strike a happy medium? So here’s where a particular statistical approach comes to play - thanks to a binomial distribution and some assumptions about ourselves we can…

It is not always necessary to run an optimization algorithm to perform linear regression. You can solve a specific algebraic equation — the normal equation — to get the results directly. Although for big datasets it is not even close to being computationally optimal, it‘s still one of the options good to be aware of.

Photo by Antoine Dautry on Unsplash

1. Introduction

Linear regression is one of the most important and popular predictive techniques in data analysis. It’s also one of the oldest - famous C.F. Gauss at the beginning of 19th-century was using it in the astronomy for calculation of orbits (more).

Its objective is to fit the best line (or a hyper-/plane) to the set of given points (observations) by calculating regression function parameters that minimize specific cost function (error), e.g. mean squared error (MSE).

As a reminder, below there is a linear regression equation in the expanded form.

Robert Kwiatkowski

An aerospace design engineer and a data science enthusiast.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store