Prince Andrew was one of the first royal family members to arrive atQueen Elizabeth’sChristmas lunch at Buckingham Palaceon Wednesday.

The royal’s appearance at the annual holiday party comes just weeks after his announcement that he is“stepping back from public duties”following his ties to disgraced financier Jeffrey Epstein.

Andrew’s oldest daughterPrincess Beatrice, who recentlyannounced her engagement, was also spotted arriving at the holiday lunch.

The fallout fromAndrew’s disastrous BBC interviewabout his friendship with convicted sex offender Jeffrey Epstein saw his official royal role collapse to almost nothing in a matter of days. He was removed from his hundreds of charitable patronages and his office was forced out of Buckingham Palace.

Prince Andrew drives to Queen’s Christmas lunch.Tim Rooke/Shutterstock

Prince Andrew

Princess Beatrice.Dominic Lipinski/PA Images/Getty

Princess Beatrice

The decision “shows that the royals can move very swiftly and very ruthlessly,” royal historian Robert Lacey tells PEOPLE.

Kate Middleton arrives at Queen’s Christmas lunch.Tim Rooke/Shutterstock

Catherine Duchess of Cambridge

Prince William and Prince George.Tim Rooke/Shutterstock

Prince William, Prince George

Queen’s Christmas lunch 2019

While Prince Andrew will not perform any more public duties as a member of the royal family, not much else will change at this time.

“He remains a member of the royal family,” a royal source says. “As a royal colonel and a war veteran, you can expect to see him at Remembrance Sunday. You can expect to see him on the balcony at Trooping the Colour too.”

Can’t get enough ofPEOPLE‘s Royals coverage?Sign up for our free Royals newsletterto get the latest updates onKate Middleton,Meghan Markleand more!

Prince Charles arrives at Queen’s Christmas lunch.Tim Rooke/Shutterstock

Prince Charles

Camilla, Duchess of Cornwall arrives at the Queen’s Christmas lunch.Aaron Chown/PA Images/Getty

The Duchess of Cornwal

source: people.com