Ethnicity
The are many different ethnic groups in the game, language being the primary factor determining who belongs to what group. It is more difficult to rule a fief containing a population belonging to a different ethnic group than the ruler.