stat340s13

Introduction, Class 1 - Tuesday, May 7

Four Fundamental Problems

1. Classification: Given an input object X, we have a function which will take in this input X and identity which 'class (Y)' it belongs to (Discrete Case)
2. Regression: Same as classification but in the continuous case
3. Clustering: Use common features of objects in same class or group to form clusters.(in this case, y is unknown)
4. Dimensionality Reduction

Applications

Most useful when structure of the task is not well understood but can be characterized by a dataset with strong statistical regularity
Examples:

• Computer Vision, Computer Graphics, Finance (fraud detection), Machine Learning
  
• Search and recommendation (eg. Google)
  
• Automatic speech recognition, speaker verification
  
• Text parsing
  
• Face identification
  
• Tracking objects in video
  
• Financial prediction, fraud detection
  

Course Information

General Information

• No required textbook, recommended: "Simulation" by Sheldon M. Ross
• Computing parts of the course will be done in Matlab, but prior knowledge of Matlab is not essential (will have a tutorial on it)
• Learn for announcements, assignments, and emails.
• Other course material on: http://wikicoursenote.com/wiki/
• Log on to both Learn and wikicoursenote frequently.

Primary contributor: Put a summary of the lecture up within 48 hours.

General contributor: Elaborate on concepts, add example, add code, add pictures, reference, corrections etc… withing 2 weeks

Must do both All primary contributions are now considered general contributions you must contribute to 50% of lectures for full marks

• A general contribution can be correctional (fixing mistakes) or technical (expanding content, adding examples, etc) but at least half of your contributions should be technical for full marks

Do not submit copyrighted work without permission, cite original sources. Each time you make a contribution, check mark the table. Marks are calculated on honour system, although there will be random verifications. If you are caught claiming to contribute but didn't, you will lose marks.

Tentative Marking Scheme

Item	Value
Assignments (~6)	30%
WikiCourseNote	10%
Midterm	20%
Final	40%

Sampling (Generating random numbers), Class 2 - Thursday, May 9

Introduction

Some people believe that activities such as rolling a dice and flipping a coin are not truly random but are deterministic – that is, the result can be reliably calculated using things such as physics and math.

A computer cannot generate truly random numbers since computers can only run algorithms, which are deterministic in nature. They can, however, generate Pseudo Random Numbers; numbers that seem random but are actually deterministic.

Multiplicative Congruential Algorithm

This is an algorithm used to generate uniform, pseudo random numbers. It is also referred to as the Linear or Mixed Congruential Methods. The Multiplicative Congruential Method may also refer to the special case where c=0.

Take a number x and divide it by m
The remainder is a number between 0 and m-1
We use the operator “mod”
X mod m
1 = 10 mod 3
x = x mod m – we take the remainder and go remainder mod m, but we will get x every time we do this
We can modify this:
Z = (ax+b) mod m

Example: a=2, b=1, m=3; if x=10
Step 1: 0 = (2(10)+1) mod 3
Step 2: 1 = (2(0)+1) mod 3
Step 3: 0 = (2(1)+1) mod 3
You will get a sequence of numbers

MatLab for Multiplicative Congruential Algorithm:
Before your start:

>>Clear all
>>Close all

>>a=17
>>b=3
>>m=31
>>x=5
>>mod(a*x+b,m)
ans=26
>>x=mod(a*x+b,m)

(Note: Keep repeating this command over and over again and you will seem to get random numbers – this is how the command rand works in a computer.)

>>a=13
>>b=0
>>m=31
>>x(1)=1
>>for ii=2:1000
x(ii)=mod(a*x(ii-1)+b,m);
end
>>size(x)
ans=1    1000
>>hist(x)

(Note: The semicolon after the x(ii)=mod(a*x(ii-1)+b,m) ensures that Matlab will not show the entire vector that is x. It will instead calculate it internally and you will be able to work with it. Adding the semicolon to the end of this line reduces the run time significantly.)

This algorithm involves three integer parameters a, b, and m and an initial value, x_0 called seed. A sequence of numbers is defined by x(k+1) = a*x(k) + b mod m. Mod m means take the remainder after division by m.

Example: a=13, b=0, m=31
The first 30 numbers in the sequence are a permutation of integers for 1 to 30 and then the sequence repeats itself. Values are between 0 and m-1. If the values are normalized by dividing by m-1, then result is numbers uniformly distributed in the interval [0,1]. There are only a finite number of values (30 in this case).

Typically, it is good to choose m such that m is large, and m is prime. Careful selection of parameters helps generate relatively "random" output values, where it is harder to identify patterns.

For many years “rand” function in Matlab used this algorithm with these parameters A=7^5=16807, b=0, m=2^31 -1=2147483647 – recommended in a 1988 paper by Park and Miller (Important part is that m should be large)

Inverse Transform Method

This method is useful for generating other types of distributions other than uniform, such as exponential distribution and normal distribution. Exponential distribution have the property that numbers generated are more frequently close to 0. Normal distribution has the property that numbers generated are more frequently close to its mean.

Theorem:
Take U ~ U[0,1] and let x=F^(-1)(U)
Then x has distribution function F(.)
Where F(x) = P(X<=x) cdf; F^(-1)(U) denotes the function inverse of F(.) Or that F(x)=U -> x=F^(-1)(U)

Example: f(x) = λe^(-λx)
F(x)=∫0tox〖f(x)dx〗
= ∫0tox〖λe^(-λx) dx〗
=λ/(-λ) e^(-λx) [0 to x]
=(-e^(-λx)+e^0 )
=1-e^(-λx)

y=1-e^(-λx);
1-y=e^(-λx);
x=-ln(1-y)/λ;
y=-ln(1-x)/λ;
F^(-1) (x)=-ln(1-x)/λ;

Step 1: Draw U ~U[0,1];
Step 2: x=-ln(1-U)/ λ;

MatLab for Inverse Transform Method:

>>u=rand(1,1000)
>>hist(u)       #will generate a fairly uniform diagram

>>x=(-log(1-u))/2;
>>size(x)       #1000 in size 
>>figure
>>hist(x)       #exponential 

F(x)=P(X<=x)
=P(F^(-1)(U)<=x)
=P(F(F^(-1)(U))<=F(x))
=P(U<=F(x))

This method is flawed since not all functions are invertible. The method is limited also because not all functions are monotonic. Also, it may be impractical since some CDFs and/or integrals are not easy to compute.

Probability Distribution Function Tool in MATLAB

>>disttool