Bagnaia, chasing his second world crown, had the first half of the sprint under control after starting from pole at the Sepang International Circuit, but the 26-year-old Italian was unable to hold off a late surge by Marquez.
Bagnaia had earlier set a lap record of 1:57.491.
The sprint result meant Bagnaia's championship lead over Spain's Martin was reduced to 11 points ahead of Sunday's race.
"We did the maximum and the maximum was not enough today to win," Bagnaia said.
"This race first we need to understand, something for tomorrow. My feeling wasn't the best, absolutely."
Martin will look to topple Bagnaia with just three races left this season and recreate his heroics from last month's Thailand Grand Prix.
"It was an amazing race. The pace was outstanding. I think all the riders were on the limit," Martin said.
"I wasn't able to win but I'm so happy about the second position in front of my fierce opponents. Happy with the performance."
Enea Bastianini took the fourth slot as Ducati bikes filled the first four places, with KTM's South African rider Brad Binder being the best among the non-Ducatis, finishing fifth.
Australian Jack Miller came in sixth, while another title contender, Marco Bezzecchi, occupied the seventh spot.
Frenchman Johann Zarco and Luca Marini were the other two riders in the points.
Marc Marquez's difficult season continued as the six-time MotoGP champion crashed for the 27th time in this campaign.
After Sunday's race, MotoGP travels to Qatar and Valencia later this month.