Probability Theory and Random variables

9 minute read


This Blog post briefly introduces some concepts in set theory and measure theory that are needed to define probability. It also talks about measurable transformations and random variables.


Set is a collection of some elements and space is the collection of all elements under consideration. For example, are all sets contained in the space of natural numbers, . A point set or atomic set is a set containing a single element such as above. The entire space itself is always a valid set, as is the empty set or null set, , which contains no elements at all. Sets are often defined implicitly via an inclusion criterion. These sets are denoted with the set builder notation. For example,

Algebra of sets

Let be a set, then is an algebra over ,i.e. subset of power set of , if it is closed under complements and under unions (hence intersections) of pairs of elements of .i.e.

  • (Includes null set)
  • if , then (closed under complement)
  • if , then (closed under finite unions)

- algebra

A sigma algebra is a set of subsets of , such that

  • (Includes null set)
  • if , then (closed under complement)
  • if , then (closed under countable unions)

-algebras are a subset of algebras in the sense that all -algebras are algebras, but not vice versa. Algebras only require that they be closed under pairwise unions while -algebras must be closed under countably infinite unions. -algebras can be defined over the real line as well as over abstract sets.

Topological space

There are many definitions of topological space based on open sets, closed sets, neighborhood etc. Here we give the definition open sets.

Let be a non-empty set, then a set of subsets of is said to be a topology if

  • (Includes null set and the set itself)
  • is closed under arbitrary unions (finite or infinite)
  • is closed under finite intersections.

The ordered pair is called topological space. The members of are called open-sets. For a given set , there can be many topologies.

Example: Let , then

  • is a trivial topology on .
  • is another topology.

Borel - algebra

Borel -algebra or Borel field on a topological space is a -algebra generated by a by a collection of subsets of whose elements are finite open intervals on Real numbers. This is a special case of -algebra. The Borel algebra on is the smallest -algebra containing all open sets (or, equivalently, all closed sets). The elements of Borel field are called Borel sets.

How to construct Borel field? Take all possible open intervals.Take their compliments. Take arbitrary unions. Include and . contains a wide range of intervals including open, closed, and half-open intervals. It also contains disjoint intervals such as . It contains (nearly) every possible collection of intervals that are imagined.

Measurable Space

A pair is a measurable space if is a set and is a -algebra of subsets of . Measurable space allows us to define a function that assigns real numbered values to the abstract elements of .

Measure ()

Let be a measurable space. A set function defined on is called a measure iff it has the following properties.

  • (measure is a non-negative real number)
  • (measure of empty set is zero)
  • For disjoint sets ,

A measure on a set, , is a systematic way to assign a positive number to each suitable subset of that set, intuitively interpreted as its size.

Examples of measures

  • Counting measure: no of elements in .i.e cardinality in case of discrete sets and in case of interval sets.
  • Lebesgue measure: Conventional length of , i.e. if , then . For discrete sets, lebesgue measure is zero.

A triplet is called a measure space if is a measurable space and is a measure.

Properties of Measure

  • Monotonicity: if , then
  • Subadditivity: if , then

Probability Space


  • - be the set of all possible outcomes of a random experiment. This is called as ‘Sample space’.
  • - be the -algebra over . Elements of are called events.
  • - is the probability measure defined on the space with the following properties
    • (non-negative quantity)
    • (probability assigned to the sample space is 1)
    • for disjoint sets in , i.e. for These are called Kolgromov’s axioms of probability.

Then the triplet is called a probability space. The construction of avoids some pathological subsets, called non-measurable sets. Non-measurable sets are those for which measure is not properly not defined, i.e. elements of the set can be rearranged in such a way that measure of the set changes. By restricting ourselves to the -algebra, we are making sure that events are assigned a specific, defined measure (probability in this case). Also by making the -algebra closed under unions and complements, we are making sure that the resulting events also have a definite measure. An important benefit of these closure properties is that they ensure that any non-constructible sets that may be lurking within the power set don’t persist into a -algebra. Consequently identifying any -algebra removes many of the pathological behaviors that arise in uncountably large spaces.

Measurable functions/transformations

Once we have defined a probability distribution on a space , and a well-behaved collection of subsets , we can then consider how the probability distribution transforms when transforms. In particular, let be a transformation from to another space . Can this transformation also transform our probability distribution on onto a probability distribution on , and if so under what conditions?

Let be the -algebra on . In order for to induce a probability distribution on we need the two -algebras to be compatible in some sense. In particular we need every subset to correspond to a unique subset . If this holds for all subsets in then we say that the transformation is measurable and we can define a measure on , which is induced by as

According to wikipidea, the definition of mesurable function is as follows: Let be measurable spaces meaning that are equipped with respective -algebras . A functions is said to be measurable, if for every , there is a pre-image of under in , i.e.

If is measurable from to then is a sub -field of . It is called -field generated by and denoted as .

Random Variables

Probability measure is a set function, i.e it defines the probability for the events the -algebra. It’d be easy for us to work with if we map everything onto the line,i.e. the sample space. A random variable is a convenient way to express the elements of as numbers rather than abstract elements of sets.
A random variable is a measurable function from the probability space to other probability space on real line which is where is the range of in , is the Borel field of and is the probability measure on induced by . The induced measure on which is is called the distribution of . Specifically, Cumulative distribution function is defined as follows

\[ \begin{aligned} F(x) &= P_{\mathcal{X}}(X \leq x) \newline &= P_{\mathcal{X}}(X \in (-\infty, x]) \newline &= P(\{\omega; X(\omega) \in (-\infty, x]\}) \newline &= P(\{\omega; -\infty \leq X(\omega) \leq x\}) \end{aligned} \]

and are really just two different manifestations, or parameterizations, of the same abstract probability system. The two parameterizations, for example, might correspond to different choices of coordinate system, different choices of units etc.

Not all the times we are interested in just calculating the probabilities, some times the random variables defined on the original probability space turns out to be quite useful. Consider the following statistical experiment. Go to the road outside the college building and consider the first car that goes left to right after your arrival. As we do not know/cannot predict which car in the city might be there it is a statistical experiment. The sample space is the set of all cars in your city (or in your country). Now consider the following questions

  1. How many people are in that car?
  2. What is the amount of petrol in the fuel tank at that time?
  3. How many kilometers the car has travelled that day before you noticed?

All of these are random variables on the same sample space. Answer to question 1 might be useful to a person who sells eatables on the roadside? (more passengers means more business). Answer to question 2 might help decide if it would be profitable to open a petrol-selling shop.

Various functions defined on random variables like expectation, variance etc shed further light on the understanding of these variables.


Even though the blogpost is short, it took time for me to understand certain concepts and put together these things. The following sources have been very helpful in understanding few concepts.

  1. Probability Theory: Introduction
  2. Probability Theory (For Scientists and Engineers)
  3. Why do we need sigma-algebras to define probability spaces?
  4. Why do we need random variables?
  5. What makes the elements of sigma algebra measurable (and measurable which measure)?
  6. Does the sigma algebra over the real line contain the singleton sets?