Ancestral Rebellion occurs when the original ("ancestral") owner enters one of his fiefs that has been captured by another player and sends the fief into rebellion against its current owner. This can be done even if the fief is currently in regular rebellion. The probability of a successful ancestral rebellion would depend upon the current stature of the ancestral owner (1=10% chance...9=90% chance). Each attempt takes 30 days. It is important to note that an ancestral rebellion may be quelled by anyone, so it behooves the owner to quell the fief himself, or to have another friendly player on hand to help him.
If you quell an ancestral rebellion in a fief that you don't own that fief will be transferred to you. The examine fief screen from the travel menu shows rebellion status as well as who the ancestral owner is. You cannot start an ancestral rebellion if the owner or manager of the fief is there with an army, which may be inside the keep. That army must have at least one soldier for every 80 people in the fief and be defeated and driven out of the fief first before an ancestral rebellion can be attempted.
Basically, the Ancestral Rebellion feature is more useful to the smaller noble, who has only 2-4 fiefs to watch over. The intent, in large part, is to encourage these smaller players to stay in the game, by offering them a possibility of regaining lost lands in a less expensive manner. This feature also prevents them from being picked off early on in a game of "Medieval Magnate Monopoly." Ancestral Rebellion is not as helpful to the large magnates, proportionately, as to the minor lords.