An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
AnIntroductiontoLatentFeatureModel
JiguangLi
CenterforAppliedArtificialIntelligence
Oct28th,2021
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
Motivation
Assumewehave
N
observations
{
x
1
,
···
,
x
N
}
,suchthateach
x
i
∈
R
d
.Infinitemixturemodel,weassumeeachobservation
belongstoa
single
latentclass
c
i
:
Letthemixtureweightsbe
θ
andsupposethereare
K
classes,thedatalikelihoodis
P
(
X
|
θ
)=
Q
N
i
=1
P
K
k
=1
P
(
x
i
|
c
i
=
k
)
θ
k
Figure:
TwoClusterMixture
Problem
:whatifeachobservation
x
i
canbegeneratedby
morethanonelatentclass?
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
Motivation
Assumewehave
N
observations
{
x
1
,
···
,
x
N
}
,suchthateach
x
i
∈
R
d
.Infinitemixturemodel,weassumeeachobservation
belongstoa
single
latentclass
c
i
:
Letthemixtureweightsbe
θ
andsupposethereare
K
classes,thedatalikelihoodis
P
(
X
|
θ
)=
Q
N
i
=1
P
K
k
=1
P
(
x
i
|
c
i
=
k
)
θ
k
Figure:
TwoClusterMixture
Problem
:whatifeachobservation
x
i
canbegeneratedby
morethanonelatentclass?
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
FiniteFeatureModel:setup
Supposethereare
N
objectsand
K
features,andthefeatures
aregeneratedindependently.
Eachobject
i
hassomelatentfeaturevalues,let
F
=[
f
T
1
,
···
,
f
T
n
]
T
Dataisgeneratedfromtheselatentfeatures:
P
(
X
|
F
)
let
Z
bean
N
by
K
matrixsuchthat
Z
ik
=1ifobject
i
possessfeature
k
.
F
=
Z
⊗
V
,where
V
isthevalueofeachfeatureforeach
object.
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
FiniteFeatureModel:setup
Supposethereare
N
objectsand
K
features,andthefeatures
aregeneratedindependently.
Eachobject
i
hassomelatentfeaturevalues,let
F
=[
f
T
1
,
···
,
f
T
n
]
T
Dataisgeneratedfromtheselatentfeatures:
P
(
X
|
F
)
let
Z
bean
N
by
K
matrixsuchthat
Z
ik
=1ifobject
i
possessfeature
k
.
F
=
Z
⊗
V
,where
V
isthevalueofeachfeatureforeach
object.
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
FiniteFeatureModel:setup
Supposethereare
N
objectsand
K
features,andthefeatures
aregeneratedindependently.
Eachobject
i
hassomelatentfeaturevalues,let
F
=[
f
T
1
,
···
,
f
T
n
]
T
Dataisgeneratedfromtheselatentfeatures:
P
(
X
|
F
)
let
Z
bean
N
by
K
matrixsuchthat
Z
ik
=1ifobject
i
possessfeature
k
.
F
=
Z
⊗
V
,where
V
isthevalueofeachfeatureforeach
object.
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
FiniteFeatureModel:setup
Supposethereare
N
objectsand
K
features,andthefeatures
aregeneratedindependently.
Eachobject
i
hassomelatentfeaturevalues,let
F
=[
f
T
1
,
···
,
f
T
n
]
T
Dataisgeneratedfromtheselatentfeatures:
P
(
X
|
F
)
let
Z
bean
N
by
K
matrixsuchthat
Z
ik
=1ifobject
i
possessfeature
k
.
F
=
Z
⊗
V
,where
V
isthevalueofeachfeatureforeach
object.
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
FiniteFeatureModel:Priorfor
Z
InBayesianInference,weareinterestedintheposterior
P
(
F
|
X
)
∝
P
(
X
|
F
)
P
(
F
),where
F
=
Z
⊗
V
.
Howtospecifythepriorfor
Z
?
Assumeeachobjectpossessfeature
k
withprobability
π
k
.
P
(
Z
|
π
)=
Q
K
k
=1
Q
N
i
=1
P
(
Z
ik
|
π
k
)=
Q
K
k
=1
π
m
k
k
(1
−
π
k
)
N
−
m
k
Usually,weassume
π
k
∼
Beta
(
r
,
s
),suchthat
P
(
π
k
)=
π
r
−
1
k
(1
−
π
k
)
s
−
1
B
(
r
,
s
)
B
(
r
,
s
)=
R
1
0
π
r
−
1
k
(1
−
π
k
)
s
−
1
d
π
k
=
Γ(
r
)Γ(
s
)
Γ(
r
+
s
)
r
=
α
K
,
s
=1=
⇒
B
(
r
,
s
)=
Γ(
α
K
)
Γ(1+
α
K
)
=
K
α
(Γ(
X
)=(
X
−
1)Γ(
X
−
1))
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
FiniteFeatureModel:Priorfor
Z
InBayesianInference,weareinterestedintheposterior
P
(
F
|
X
)
∝
P
(
X
|
F
)
P
(
F
),where
F
=
Z
⊗
V
.
Howtospecifythepriorfor
Z
?
Assumeeachobjectpossessfeature
k
withprobability
π
k
.
P
(
Z
|
π
)=
Q
K
k
=1
Q
N
i
=1
P
(
Z
ik
|
π
k
)=
Q
K
k
=1
π
m
k
k
(1
−
π
k
)
N
−
m
k
Usually,weassume
π
k
∼
Beta
(
r
,
s
),suchthat
P
(
π
k
)=
π
r
−
1
k
(1
−
π
k
)
s
−
1
B
(
r
,
s
)
B
(
r
,
s
)=
R
1
0
π
r
−
1
k
(1
−
π
k
)
s
−
1
d
π
k
=
Γ(
r
)Γ(
s
)
Γ(
r
+
s
)
r
=
α
K
,
s
=1=
⇒
B
(
r
,
s
)=
Γ(
α
K
)
Γ(1+
α
K
)
=
K
α
(Γ(
X
)=(
X
−
1)Γ(
X
−
1))
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
FiniteFeatureModel:Priorfor
Z
InBayesianInference,weareinterestedintheposterior
P
(
F
|
X
)
∝
P
(
X
|
F
)
P
(
F
),where
F
=
Z
⊗
V
.
Howtospecifythepriorfor
Z
?
Assumeeachobjectpossessfeature
k
withprobability
π
k
.
P
(
Z
|
π
)=
Q
K
k
=1
Q
N
i
=1
P
(
Z
ik
|
π
k
)=
Q
K
k
=1
π
m
k
k
(1
−
π
k
)
N
−
m
k
Usually,weassume
π
k
∼
Beta
(
r
,
s
),suchthat
P
(
π
k
)=
π
r
−
1
k
(1
−
π
k
)
s
−
1
B
(
r
,
s
)
B
(
r
,
s
)=
R
1
0
π
r
−
1
k
(1
−
π
k
)
s
−
1
d
π
k
=
Γ(
r
)Γ(
s
)
Γ(
r
+
s
)
r
=
α
K
,
s
=1=
⇒
B
(
r
,
s
)=
Γ(
α
K
)
Γ(1+
α
K
)
=
K
α
(Γ(
X
)=(
X
−
1)Γ(
X
−
1))
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
FiniteFeatureModel:Priorfor
Z
InBayesianInference,weareinterestedintheposterior
P
(
F
|
X
)
∝
P
(
X
|
F
)
P
(
F
),where
F
=
Z
⊗
V
.
Howtospecifythepriorfor
Z
?
Assumeeachobjectpossessfeature
k
withprobability
π
k
.
P
(
Z
|
π
)=
Q
K
k
=1
Q
N
i
=1
P
(
Z
ik
|
π
k
)=
Q
K
k
=1
π
m
k
k
(1
−
π
k
)
N
−
m
k
Usually,weassume
π
k
∼
Beta
(
r
,
s
),suchthat
P
(
π
k
)=
π
r
−
1
k
(1
−
π
k
)
s
−
1
B
(
r
,
s
)
B
(
r
,
s
)=
R
1
0
π
r
−
1
k
(1
−
π
k
)
s
−
1
d
π
k
=
Γ(
r
)Γ(
s
)
Γ(
r
+
s
)
r
=
α
K
,
s
=1=
⇒
B
(
r
,
s
)=
Γ(
α
K
)
Γ(1+
α
K
)
=
K
α
(Γ(
X
)=(
X
−
1)Γ(
X
−
1))
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
FiniteFeatureModel:Priorfor
Z
InBayesianInference,weareinterestedintheposterior
P
(
F
|
X
)
∝
P
(
X
|
F
)
P
(
F
),where
F
=
Z
⊗
V
.
Howtospecifythepriorfor
Z
?
Assumeeachobjectpossessfeature
k
withprobability
π
k
.
P
(
Z
|
π
)=
Q
K
k
=1
Q
N
i
=1
P
(
Z
ik
|
π
k
)=
Q
K
k
=1
π
m
k
k
(1
−
π
k
)
N
−
m
k
Usually,weassume
π
k
∼
Beta
(
r
,
s
),suchthat
P
(
π
k
)=
π
r
−
1
k
(1
−
π
k
)
s
−
1
B
(
r
,
s
)
B
(
r
,
s
)=
R
1
0
π
r
−
1
k
(1
−
π
k
)
s
−
1
d
π
k
=
Γ(
r
)Γ(
s
)
Γ(
r
+
s
)
r
=
α
K
,
s
=1=
⇒
B
(
r
,
s
)=
Γ(
α
K
)
Γ(1+
α
K
)
=
K
α
(Γ(
X
)=(
X
−
1)Γ(
X
−
1))
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
FiniteFeatureModel:Priorfor
Z
InBayesianInference,weareinterestedintheposterior
P
(
F
|
X
)
∝
P
(
X
|
F
)
P
(
F
),where
F
=
Z
⊗
V
.
Howtospecifythepriorfor
Z
?
Assumeeachobjectpossessfeature
k
withprobability
π
k
.
P
(
Z
|
π
)=
Q
K
k
=1
Q
N
i
=1
P
(
Z
ik
|
π
k
)=
Q
K
k
=1
π
m
k
k
(1
−
π
k
)
N
−
m
k
Usually,weassume
π
k
∼
Beta
(
r
,
s
),suchthat
P
(
π
k
)=
π
r
−
1
k
(1
−
π
k
)
s
−
1
B
(
r
,
s
)
B
(
r
,
s
)=
R
1
0
π
r
−
1
k
(1
−
π
k
)
s
−
1
d
π
k
=
Γ(
r
)Γ(
s
)
Γ(
r
+
s
)
r
=
α
K
,
s
=1=
⇒
B
(
r
,
s
)=
Γ(
α
K
)
Γ(1+
α
K
)
=
K
α
(Γ(
X
)=(
X
−
1)Γ(
X
−
1))
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
FiniteFeatureModel:Priorfor
Z
InBayesianInference,weareinterestedintheposterior
P
(
F
|
X
)
∝
P
(
X
|
F
)
P
(
F
),where
F
=
Z
⊗
V
.
Howtospecifythepriorfor
Z
?
Assumeeachobjectpossessfeature
k
withprobability
π
k
.
P
(
Z
|
π
)=
Q
K
k
=1
Q
N
i
=1
P
(
Z
ik
|
π
k
)=
Q
K
k
=1
π
m
k
k
(1
−
π
k
)
N
−
m
k
Usually,weassume
π
k
∼
Beta
(
r
,
s
),suchthat
P
(
π
k
)=
π
r
−
1
k
(1
−
π
k
)
s
−
1
B
(
r
,
s
)
B
(
r
,
s
)=
R
1
0
π
r
−
1
k
(1
−
π
k
)
s
−
1
d
π
k
=
Γ(
r
)Γ(
s
)
Γ(
r
+
s
)
r
=
α
K
,
s
=1=
⇒
B
(
r
,
s
)=
Γ(
α
K
)
Γ(1+
α
K
)
=
K
α
(Γ(
X
)=(
X
−
1)Γ(
X
−
1))
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
FiniteFeatureModel:Distributionof
Z
Recallwehavedefinedpriorfor
π
k
∼
Beta
(
α
K
,
1)and
P
(
Z
|
π
):
P
(
Z
)=
K
Y
k
=1
Z
(
N
Y
i
=1
P
(
Z
ik
|
π
k
))
P
(
π
k
)
d
π
k
=
K
Y
k
=1
Z
N
Y
i
=1
π
m
k
k
(1
−
π
k
)
N
−
m
k
P
(
π
k
)
d
π
k
=
K
Y
k
=1
R
Q
N
i
=1
π
m
k
k
(1
−
π
N
−
mk
k
)
π
α/
K
−
1
k
B
(
α
K
,
1)
d
π
k
=
K
Y
k
=1
B
(
m
k
+
α/
K
,
N
−
m
k
+1)
B
(
α/
K
,
1)
=
K
Y
k
=1
α/
K
·
Γ(
m
k
+
α/
K
)Γ(
N
−
m
k
+1)
Γ(
N
+1+
α/
K
)
(1)
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
FiniteFeatureModel:
α
controlsSparsity
Theexpectationofthenumberofnon-zeroentriesinthe
matrixhasanupperboundthatisindependentof
K
!
E
[
X
i
,
k
Z
i
,
k
]=
E
[
T
Z
]=
KE
[
T
Z
1
]
=
K
N
X
i
=1
Z
1
0
π
k
P
(
π
k
)
d
π
k
=
KN
α/
k
1+
α/
K
=
N
α
1+
α/
K
≤
N
α
(2)
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
InfiniteFeatureModel
Motivation
:whatifwedon’tknowthenumberoflatent
features?Canbedefineadistributionofbinarymatrix
Z
which
has
N
rowsbutinfinitecolumns?
NaiveApproach
:recallinthefinitecase,wehaveshown
P
(
Z
)=
K
Y
k
=1
α/
K
·
Γ(
m
k
+
α/
K
)Γ(
N
−
m
k
+1)
Γ(
N
+1+
α/
K
)
Whatifwelet
K
→∞
?
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
InfiniteFeatureModel
Motivation
:whatifwedon’tknowthenumberoflatent
features?Canbedefineadistributionofbinarymatrix
Z
which
has
N
rowsbutinfinitecolumns?
NaiveApproach
:recallinthefinitecase,wehaveshown
P
(
Z
)=
K
Y
k
=1
α/
K
·
Γ(
m
k
+
α/
K
)Γ(
N
−
m
k
+1)
Γ(
N
+1+
α/
K
)
Whatifwelet
K
→∞
?
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
EquivalenceClasses
lof(
·
):mapbinarymatricestoleft-orderedbinarymatrices.
lof(
Z
)isobtainedbyreorderingthecolumnsofthebinary
matrix
Z
fromlefttorightbythemagnitudeofthebinary
numberexpressedbythecolumn:
[
Z
]:setofbinarymatricesthatarelof-equivalentto
Z
Figure:
Visualizationoflofoperation
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
EquivalenceClasses
lof(
·
):mapbinarymatricestoleft-orderedbinarymatrices.
lof(
Z
)isobtainedbyreorderingthecolumnsofthebinary
matrix
Z
fromlefttorightbythemagnitudeofthebinary
numberexpressedbythecolumn:
[
Z
]:setofbinarymatricesthatarelof-equivalentto
Z
Figure:
Visualizationoflofoperation
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
EquivalenceClasses
lof(
·
):mapbinarymatricestoleft-orderedbinarymatrices.
lof(
Z
)isobtainedbyreorderingthecolumnsofthebinary
matrix
Z
fromlefttorightbythemagnitudeofthebinary
numberexpressedbythecolumn:
[
Z
]:setofbinarymatricesthatarelof-equivalentto
Z
Figure:
Visualizationoflofoperation
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
MoreonEquivalenceClasses
The
history
offeature
k
atobject
i
is(
Z
1
,
k
,
···
,
Z
i
−
1
,
k
)
k
h
:numberoffeaturespossessingthehistory
h
K
0
:numberoffeaturesforwhich
m
k
=0
K
+
=
P
2
N
−
1
h
=1
k
h
:numberoffeaturesforwhich
m
k
>
0,
note
K
=
K
0
+
K
+
Cardinalityof
|
[
Z
]
|
:
|
[
Z
]
|
=
k
!
Q
2
N
−
1
h
=0
k
n
!
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
MoreonEquivalenceClasses
The
history
offeature
k
atobject
i
is(
Z
1
,
k
,
···
,
Z
i
−
1
,
k
)
k
h
:numberoffeaturespossessingthehistory
h
K
0
:numberoffeaturesforwhich
m
k
=0
K
+
=
P
2
N
−
1
h
=1
k
h
:numberoffeaturesforwhich
m
k
>
0,
note
K
=
K
0
+
K
+
Cardinalityof
|
[
Z
]
|
:
|
[
Z
]
|
=
k
!
Q
2
N
−
1
h
=0
k
n
!
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
MoreonEquivalenceClasses
The
history
offeature
k
atobject
i
is(
Z
1
,
k
,
···
,
Z
i
−
1
,
k
)
k
h
:numberoffeaturespossessingthehistory
h
K
0
:numberoffeaturesforwhich
m
k
=0
K
+
=
P
2
N
−
1
h
=1
k
h
:numberoffeaturesforwhich
m
k
>
0,
note
K
=
K
0
+
K
+
Cardinalityof
|
[
Z
]
|
:
|
[
Z
]
|
=
k
!
Q
2
N
−
1
h
=0
k
n
!
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
Let’smake
K
→∞
P
([
Z
])=
X
z
∈
[
Z
]
P
(
Z
)
=
K
!
Q
2
N
−
1
h
=0
k
n
!
K
Y
k
=1
α/
K
·
Γ(
m
k
+
α/
K
)Γ(
N
−
m
k
+1)
Γ(
N
+1+
α/
K
)
(3)
By2-pageofalgebraicmanipulations,mathematicianscan
show
lim
k
→∞
P
([
Z
])=
α
K
+
Q
2
N
−
1
h
=1
K
h
!
e
−
α
H
N
K
+
Y
k
=1
(
N
−
m
k
)!(
m
k
−
1)!
N
!
,where
H
N
=
P
N
j
=1
1
j
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
Let’smake
K
→∞
P
([
Z
])=
X
z
∈
[
Z
]
P
(
Z
)
=
K
!
Q
2
N
−
1
h
=0
k
n
!
K
Y
k
=1
α/
K
·
Γ(
m
k
+
α/
K
)Γ(
N
−
m
k
+1)
Γ(
N
+1+
α/
K
)
(3)
By2-pageofalgebraicmanipulations,mathematicianscan
show
lim
k
→∞
P
([
Z
])=
α
K
+
Q
2
N
−
1
h
=1
K
h
!
e
−
α
H
N
K
+
Y
k
=1
(
N
−
m
k
)!(
m
k
−
1)!
N
!
,where
H
N
=
P
N
j
=1
1
j
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
IndianBuffetProcess(IBP)
Figure:
IndianBuffetProcess
ThefirstcustomerentersanIndianBuffetwithinfinitely
manydishes.
Thefirstcustomersamplethefirst
Poisson
(
α
)dishes.
Thenthcustomerhelpshimselftoeachdishwith
probability
m
k
n
,where
m
k
isthenumberoftimesdish
k
hasbeensampled
Thenthcustomertries
poisson
(
α
n
)dishes.
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
IndianBuffetProcess(IBP)
Figure:
IndianBuffetProcess
ThefirstcustomerentersanIndianBuffetwithinfinitely
manydishes.
Thefirstcustomersamplethefirst
Poisson
(
α
)dishes.
Thenthcustomerhelpshimselftoeachdishwith
probability
m
k
n
,where
m
k
isthenumberoftimesdish
k
hasbeensampled
Thenthcustomertries
poisson
(
α
n
)dishes.
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
IndianBuffetProcess(IBP)
Figure:
IndianBuffetProcess
ThefirstcustomerentersanIndianBuffetwithinfinitely
manydishes.
Thefirstcustomersamplethefirst
Poisson
(
α
)dishes.
Thenthcustomerhelpshimselftoeachdishwith
probability
m
k
n
,where
m
k
isthenumberoftimesdish
k
hasbeensampled
Thenthcustomertries
poisson
(
α
n
)dishes.
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
IndianBuffetProcess(IBP)
Figure:
IndianBuffetProcess
ThefirstcustomerentersanIndianBuffetwithinfinitely
manydishes.
Thefirstcustomersamplethefirst
Poisson
(
α
)dishes.
Thenthcustomerhelpshimselftoeachdishwith
probability
m
k
n
,where
m
k
isthenumberoftimesdish
k
hasbeensampled
Thenthcustomertries
poisson
(
α
n
)dishes.
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
IndianBuffetProcess(IBP)
Figure:
IndianBuffetProcess
ThefirstcustomerentersanIndianBuffetwithinfinitely
manydishes.
Thefirstcustomersamplethefirst
Poisson
(
α
)dishes.
Thenthcustomerhelpshimselftoeachdishwith
probability
m
k
n
,where
m
k
isthenumberoftimesdish
k
hasbeensampled
Thenthcustomertries
poisson
(
α
n
)dishes.
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
ResearchIdeas
Supposehaveitemresponsematrix
X
withsize
N
by
K
,where
N
isthenumberofstudents,
K
isnumberofitems,and
X
ik
representwhetherstudent
i
answerquestion
k
correctly.
PartiallyCompensatoryMIRT
:
Supposethisexamtests
M
latentabilities,hence
students’latenttraits
θ
i
∈
M
.
α
k
∈
R
M
:discriminationterm,
d
k
∈
R
M
:difficulty
term.
P
(
X
ik
=1)=
Q
M
m
=1
1
1+exp(
−
α
k
(
θ
i
−
d
m
))
CurrentApproach
:let
c
km
∼
Beta
(2
,
2),
P
(
X
ik
=1)=
Q
M
m
=1
(
1
1+exp(
−
α
k
(
θ
i
−
d
m
))
)
c
km
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
ResearchIdeas
Supposehaveitemresponsematrix
X
withsize
N
by
K
,where
N
isthenumberofstudents,
K
isnumberofitems,and
X
ik
representwhetherstudent
i
answerquestion
k
correctly.
PartiallyCompensatoryMIRT
:
Supposethisexamtests
M
latentabilities,hence
students’latenttraits
θ
i
∈
M
.
α
k
∈
R
M
:discriminationterm,
d
k
∈
R
M
:difficulty
term.
P
(
X
ik
=1)=
Q
M
m
=1
1
1+exp(
−
α
k
(
θ
i
−
d
m
))
CurrentApproach
:let
c
km
∼
Beta
(2
,
2),
P
(
X
ik
=1)=
Q
M
m
=1
(
1
1+exp(
−
α
k
(
θ
i
−
d
m
))
)
c
km
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
ResearchIdeas
Supposehaveitemresponsematrix
X
withsize
N
by
K
,where
N
isthenumberofstudents,
K
isnumberofitems,and
X
ik
representwhetherstudent
i
answerquestion
k
correctly.
PartiallyCompensatoryMIRT
:
Supposethisexamtests
M
latentabilities,hence
students’latenttraits
θ
i
∈
M
.
α
k
∈
R
M
:discriminationterm,
d
k
∈
R
M
:difficulty
term.
P
(
X
ik
=1)=
Q
M
m
=1
1
1+exp(
−
α
k
(
θ
i
−
d
m
))
CurrentApproach
:let
c
km
∼
Beta
(2
,
2),
P
(
X
ik
=1)=
Q
M
m
=1
(
1
1+exp(
−
α
k
(
θ
i
−
d
m
))
)
c
km
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
ResearchIdeas
Supposehaveitemresponsematrix
X
withsize
N
by
K
,where
N
isthenumberofstudents,
K
isnumberofitems,and
X
ik
representwhetherstudent
i
answerquestion
k
correctly.
PartiallyCompensatoryMIRT
:
Supposethisexamtests
M
latentabilities,hence
students’latenttraits
θ
i
∈
M
.
α
k
∈
R
M
:discriminationterm,
d
k
∈
R
M
:difficulty
term.
P
(
X
ik
=1)=
Q
M
m
=1
1
1+exp(
−
α
k
(
θ
i
−
d
m
))
CurrentApproach
:let
c
km
∼
Beta
(2
,
2),
P
(
X
ik
=1)=
Q
M
m
=1
(
1
1+exp(
−
α
k
(
θ
i
−
d
m
))
)
c
km
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
ResearchIdeas
Supposehaveitemresponsematrix
X
withsize
N
by
K
,where
N
isthenumberofstudents,
K
isnumberofitems,and
X
ik
representwhetherstudent
i
answerquestion
k
correctly.
PartiallyCompensatoryMIRT
:
Supposethisexamtests
M
latentabilities,hence
students’latenttraits
θ
i
∈
M
.
α
k
∈
R
M
:discriminationterm,
d
k
∈
R
M
:difficulty
term.
P
(
X
ik
=1)=
Q
M
m
=1
1
1+exp(
−
α
k
(
θ
i
−
d
m
))
CurrentApproach
:let
c
km
∼
Beta
(2
,
2),
P
(
X
ik
=1)=
Q
M
m
=1
(
1
1+exp(
−
α
k
(
θ
i
−
d
m
))
)
c
km
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
ResearchIdeas(cont’d)
Wedon’tknowthenumberoflatentabilities,canweapply
IDBpriortoeachcolumnoftheitemresponsematrix
X
?
Advantages
:
Flexible,noneedforcross-validation.
Allowthenumberoflatentfactorstogrowoveryears.
Incorporateeducationexpertise?Fixsomeentriesof
latentmatrix
Z
?
Thelearned
Z
matrixcanbetreatedasanaive
representationofthelatentstructureofknowledges.
Drawbacks
:
High-dimensional?
Requirestoomanyitems?
WhataboutHierarchicalstructure?
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
ResearchIdeas(cont’d)
Wedon’tknowthenumberoflatentabilities,canweapply
IDBpriortoeachcolumnoftheitemresponsematrix
X
?
Advantages
:
Flexible,noneedforcross-validation.
Allowthenumberoflatentfactorstogrowoveryears.
Incorporateeducationexpertise?Fixsomeentriesof
latentmatrix
Z
?
Thelearned
Z
matrixcanbetreatedasanaive
representationofthelatentstructureofknowledges.
Drawbacks
:
High-dimensional?
Requirestoomanyitems?
WhataboutHierarchicalstructure?
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
ResearchIdeas(cont’d)
Wedon’tknowthenumberoflatentabilities,canweapply
IDBpriortoeachcolumnoftheitemresponsematrix
X
?
Advantages
:
Flexible,noneedforcross-validation.
Allowthenumberoflatentfactorstogrowoveryears.
Incorporateeducationexpertise?Fixsomeentriesof
latentmatrix
Z
?
Thelearned
Z
matrixcanbetreatedasanaive
representationofthelatentstructureofknowledges.
Drawbacks
:
High-dimensional?
Requirestoomanyitems?
WhataboutHierarchicalstructure?
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
ResearchIdeas(cont’d)
Wedon’tknowthenumberoflatentabilities,canweapply
IDBpriortoeachcolumnoftheitemresponsematrix
X
?
Advantages
:
Flexible,noneedforcross-validation.
Allowthenumberoflatentfactorstogrowoveryears.
Incorporateeducationexpertise?Fixsomeentriesof
latentmatrix
Z
?
Thelearned
Z
matrixcanbetreatedasanaive
representationofthelatentstructureofknowledges.
Drawbacks
:
High-dimensional?
Requirestoomanyitems?
WhataboutHierarchicalstructure?
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
ResearchIdeas(cont’d)
Wedon’tknowthenumberoflatentabilities,canweapply
IDBpriortoeachcolumnoftheitemresponsematrix
X
?
Advantages
:
Flexible,noneedforcross-validation.
Allowthenumberoflatentfactorstogrowoveryears.
Incorporateeducationexpertise?Fixsomeentriesof
latentmatrix
Z
?
Thelearned
Z
matrixcanbetreatedasanaive
representationofthelatentstructureofknowledges.
Drawbacks
:
High-dimensional?
Requirestoomanyitems?
WhataboutHierarchicalstructure?
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
ResearchIdeas(cont’d)
Wedon’tknowthenumberoflatentabilities,canweapply
IDBpriortoeachcolumnoftheitemresponsematrix
X
?
Advantages
:
Flexible,noneedforcross-validation.
Allowthenumberoflatentfactorstogrowoveryears.
Incorporateeducationexpertise?Fixsomeentriesof
latentmatrix
Z
?
Thelearned
Z
matrixcanbetreatedasanaive
representationofthelatentstructureofknowledges.
Drawbacks
:
High-dimensional?
Requirestoomanyitems?
WhataboutHierarchicalstructure?
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
ResearchIdeas(cont’d)
Wedon’tknowthenumberoflatentabilities,canweapply
IDBpriortoeachcolumnoftheitemresponsematrix
X
?
Advantages
:
Flexible,noneedforcross-validation.
Allowthenumberoflatentfactorstogrowoveryears.
Incorporateeducationexpertise?Fixsomeentriesof
latentmatrix
Z
?
Thelearned
Z
matrixcanbetreatedasanaive
representationofthelatentstructureofknowledges.
Drawbacks
:
High-dimensional?
Requirestoomanyitems?
WhataboutHierarchicalstructure?
An
Introduction
toLatent
FeatureModel
JiguangLi
Motivations
AFinite
FeatureModel
Infinite
FeatureModel
ResearchIdeas
Reference
ThomasL.Griffiths,ZoubinGhahramani(2011).The
IndianBuffetProcess:AnIntroductionandReview.
ZoubinGhahramani,ThomasL.Griffiths,PeterSollich
(2006).Bayesiannonparametriclatentfeaturemodels