Sevastova, the world number 386 from Latvia, snapped fourth-ranked Pegula's 11-match WTA Canada win streak, the longest since Serena Williams reeled off 14 consecutive wins in 2011, 2013, and 2014.
Sevastova, 35, won the most recent of her four career WTA titles at the 2019 Baltic Open in her homeland.
But she will try to extend her Montreal run in a round of 16 meeting with Osaka, a four-time Grand Slam champion from Japan who ousted another Latvian, 22nd seed Jelena Ostapenko, 6-2, 6-4.
"Somehow, I was down 2-0 in the second set and started to play better and better," Sevastova said. "Third set I played really good."
"Just trying to stay on the court as long as possible," added Sevastova, who has dealt with injury since returning from maternity leave in February 2024.
In the night session, second-seeded Iga Swiatek - playing her first tournament since winning Wimbledon - raced into the fourth round with a 6-2, 6-2 victory over Germany's Eva Lys.
Swiatek next faces Denmark's Clara Tauson, who beat Yuliia Starodubtseva 6-3, 6-0.
Pegula, last year's US Open runner-up in her best Slam showing, was the first woman to win back-to-back Canadian Open titles since Martina Hingis in 1999-2000.
Osaka, twice a winner at both the US and Australian Opens, is one match away from her first quarter-final run at either a Grand Slam or WTA 1000 event since she returned from maternity leave at the start of 2024.
Sevastova, who returned from her own maternity leave in February 2024, collected her first victory over a top-five opponent since beating Czech Karolina Pliskova in 2017.
Pegula broke to open the match and again at love to claim the first set when Sevastova sent a forehand long.
A forehand winner earned Sevastova a break for a 5-4 lead in the second set and she denied Pegula on three break chances before holding in the final game to force a third set.
Pegula sent a down-the-line forehand wide to surrender a break and a 2-1 lead to Sevastova, who broke again to grab a 4-1 advantage when Pegula swatted a forehand long.
Sevastova held to 5-1 and broke again on a forehand winner to end the match after one hour and 41 minutes.
"It was a weird match for me," Pegula said. "I felt like I had total control and then I just played a couple of terrible games for, like, three games."
"That totally flipped the momentum of the match, and I went from being up a set and 2-0 to being down very quickly."
"I don't really feel like I'm playing great tennis," Pegula admitted. "At times I am, but I feel very up and down, kind of sloppy, which I don't like. I've got to figure it out."
Osaka too good for Ostapenko
World number 49 Osaka broke on a double fault to capture the first set in 30 minutes.
In the second set, Ostapenko netted a forehand to hand Osaka a break for a 3-1 lead, but rallied and broke back in the next game on a forehand crosscourt winner.
Osaka broke again for a 4-2 lead when Ostapenko netted a forehand and Osaka served for the match, leading 5-3, but Ostapenko saved a match point on a forehand crosscourt winner and broke when Osaka sent a forehand beyond the baseline.
The Japanese star responded by breaking Ostapenko at love in the final game, the Latvian netting a backhand to conclude matters after 72 minutes.
"Granted, she broke me a couple times, but she's a really good returner, so I can't take that personally," Osaka said.
"I went in there knowing she's a great player and if I give her a chance she's going to hit a winner on me, so I just tried to keep my pace and stay as solid as I could."
Osaka improved to 3-0 lifetime against Ostapenko, having also beaten her at the 2016 French Open and last year's US Open in the first round without dropping a set.
Also advancing was Danish 16th seed Clara Tauson, who dispatched Ukraine's Yuliia Starodubtseva 6-3, 6-0.