كتاب العيدين
10  
The Two 'Ids

Yahya related to me that Malik said, "The sunna concerning the time of prayer on the ids of Fitr and Adha - and there is no disagreement amongst us about it - is that the imam leaves his house and as soon as he has reached the place of prayer the prayer falls due."

Yahya said that Malik was asked whether a man who prayed with the imam could leave before the khutba, and he said, "He should not leave until the imam leaves."

USC-MSA web (English) reference : Book 10, Hadith 14