Problem Set 1

CS578/STAT590: Introduction to Machine Learning

Fall 2014

Problem Set 1
Joon Hee Choi

Handed In: 09/14/2014

Review Questions
1.
2.

3.

•

∂f
∂x

= 6x − y − 11,

∂f
∂y

•

∂f
∂x

= 6x − y − 11 = 0,

= 2y − x
∂f
∂y

= 2y − x = 0 ⇒ x = 2, y = 1 ⇒ (x, y) = (2, 1)

• (a) n = 2 w1 x1 and x =
. Then, w2 x2 wT x + b = w1 x1 + w2 x2 + b = 0 ⇒ −x1b +

Let w =

w1

x2
− wb

=1

2

(b) n = 3




 w1 x1
Let w =  w2  and x =  x2  . Then, w3 x3
T
w x + b = w1 x1 + w2 x2 + w3 x3 + b = 0 ⇒ −x1b + −x2b + −x3b = 1 w1 w2 w3 



x1 w1  x2 
 w2 




• Let w =  ..  and x =  .. . The distance between two hyperplanes is
 . 
 .  xn wn the same as the distance between a point (− wb11 , 0, ..., 0) on wT x + b1 = 0 and a hyperplane wT x + b2 = 0. Therefore,
D=

| − b1 + b 2 | w12 +

w22

Basic Concepts
1.
2.
3.
4.

1

|b1 − b2 |
= √
+ ... + wn2 wT w

Joon Hee Choi

2

Decision Trees
1.

7
7
• Entropy(S) = − 11 log2 ( 11
)−

4
4
log2 ( 11
)
11

= 0.946

2.
3.

•

Pages: 45 50 72 100 120 142 150
Class: N Y Y N
Y
Y
Y
Thresholds : 47.5, 86, 110, 175, 250, 675

200
N

300
Y

350
Y

(a) Threshold(T h) = 47.5
Entropy(ST h<47.5 ) = 0
7
3
3
7 log2 ( 10
) − 10 log2 ( 10
) = 0.881
Entropy(ST h≥47.5 ) = − 10
1
10
Gain(S, Pages) = 0.946 − 11 · 0 − 11 · 0.881 = 0.145
(b) Threshold(T h) = 86
Entropy(ST h<86 ) = − 32 log2 ( 32 ) − 13 log2 ( 31 ) = 0.918
Entropy(ST h≥86 ) = − 58 log2 ( 85 ) − 38 log2 ( 83 ) = 0.954
3
8
Gain(S, Pages) = 0.946 − 11
· 0.918 − 11
· 0.954 = 0.002
(c) Threshold(T h) = 110
Entropy(ST h<110 ) = 1
Entropy(ST h≥110 ) = − 75 log2 ( 57 ) − 27 log2 ( 72 ) = 0.863
7
4
· 1 − 11
· 0.863 = 0.033
Gain(S, Pages) = 0.946 − 11
(d) Threshold(T h) = 175
7
4
Gain(S, Pages) = 0.946 − 11
· 0.863 − 11
· 1 = 0.033
(e) Threshold(T h) = 250
8
3
Gain(S, Pages) = 0.946 − 11
· 0.954 − 11
· 0.918 = 0.002
(f) Threshold(T h) = 675
1
10
· 0.881 − 11
· 0 = 0.145
Gain(S, Pages) = 0.946 − 11

1000
N