Raducanu took a “physical and mental” break after reaching the Miami Open quarter-finals last month and the opening round in Madrid on Wednesday was her first competitive match since.
It also marked a first win on the outdoor clay in almost three years.
But on Friday, competing in just her 18th competitive clay-court match, the former US Open champion had to claw her way back from a break down for 3-3 in the first set before immediately dropping her serve again after two double faults for Kostyuk to take the opener.
Raducanu, who continues to work with coach Mark Petchey on an informal basis, improved in the second set, winning four games in a row on her way to forcing a deciding set.
But a poor opening game from the Briton set Kostyuk up to race to victory and book her place in the third round, where she will meet Veronika Kudermetova.
Meanwhile, top seed Aryna Sabalenka comfortably beat Anna Blinkova 6-3 6-4 but eighth seed Zheng Qinwen struggled in a 6-4 6-4 defeat by Anastasia Potapova.
Spain’s Paula Badosa has withdrawn from the tournament with a back injury.
The world number nine had been due to take on Veronika Kudermetova in the second round.
