Introduction to Bayesian statistics

2026
Oskar Soop

Thought experiment

How would you estimate the location of the ball?

Simplest method, don't overthink it!

The simplest approach is to use the sample average.

The sample average is also the maximum likelihood estimate (MLE).

Let be the location of the ball.

Let be the number of ants to the right (west) of the ball and be the number of ants to the up (north) of the ball.

Then

The likelihood function is

where

The MLE is

Can we do better?

We are now allowed to overthink!

Based on physical intuition, we can guess that the ball is located somewhere away from the closest edge of the table.

My prior belief about the location of the ball:

Samples were drawn according to the likelihood function:

where

We can use Bayes' theorem to update our belief about the location of the ball:

Course information

• Lectures/seminars:
  • Mondays 14:15-16:00, room 2039
  • Wednesdays 12:15-14:00, room 2039

Grading

Differentiated: A (90 - 100), B (80 - 89), C (70 - 79), D (60 - 69), E (50 - 59), F (0 - 49), not present

• ~45 min presentation (50 p) - required to pass the course
• final oral assessment (25 p)
• home work (25 p)

Course materials

The course materials will be available on https://courses.ms.ut.ee/2026.

• lecture notes by J. Lember
• handouts

What is probability?

What is probability?

Mathematical definition

Kolmogorov axioms:

1. For any event , .
2. , where is the sample space.
3. For any countable sequence of mutually exclusive events ,

What is probability?

Philosophical probability interpretations

Objective probability

aka frequentist or physical probability.

The probability of an event is defined as the limit of its relative frequency in an infinite number of trials:

where is the number of times event occurs in trials.

Subjective probability

aka Bayesian or epistemic probability.

The probability is interpreted as reasonable expectation representing a state of knowledge or as quantification of a personal belief.

1. A lady who drinks milk in her tea claims to be able to tell which was poured first, the tea or the milk. In ten trials, she determines correctly whether it was tea or milk that entered the cups first.
2. A music expert claims to be able to tell whether a page of music was written by Haydn or by Mozart. In ten trials conducted, he correctly determines the composer every time.
3. A drunken friend says that he can predict the outcome of a fair coin-flip. In ten trials, he is right every time.

There are other interpretations of probability

• Classical interpretation (symmetry based)
• Propensity interpretation
• etc.

Confusingly, the frequentist and Bayesian terms are used in different contexts to mean different things. In the context of statistics, they usually refer to different approaches to inference and are not necessarily disjoint.

Frequentist inference

Assumption that there is a "true distribution of the data".

Frequentist statistics resolves around the central question: "What can we say about the true distribution of the data based on the observed data?"

Three basic ingredients:

1. the data
2. a model
3. an estimation method

Frequentist inference

• p-values
• confidence intervals
• NHST (null hypothesis significance testing)
• MLE
• etc.

Frequentist inference

Let be an estimator of a true parameter .

Common frequentist properties:

• bias

• consistency

• coverage probability (confidence intervals)

• etc.

Bayesian inference

Four basic ingredients:

1. the data
2. a model
3. a prior distribution
4. a method for making inference based on the posterior distribution

Doesn't have to be, but can be frequentist. For example by setting priors according to certain "non-subjective" rules.

Let's move on to the Lecture Notes.