Those sacrifices included moving away from his family and friends in Devon to Dublin and a period of time away from international competition in order to become eligible for Ireland.
But Fannon always targeted the Olympics – despite what one coach once told him when he was young.
Having trained for just four hours a week in a small swimming pool outside Torquay United’s football ground, he became one of Great Britain’s top junior sprinters.
He was headhunted by Rudd to join him at Plymouth Leander where he trained alongside world 50m freestyle and butterfly champion Ben Proud and Olympic breaststroke champion Ruta Meilutyte.
That allowed him to continue his growth and he won a bronze medal at the 2016 European Junior Championships.
After the delayed Tokyo Olympics, he made the move to Dublin and has since become Irish record holder and one of five swimmers representing Ireland in individual events in Paris.
“Everybody wants a medal, that’s the mentality you have to have going into these games,” he tells BBC Radio Devon.
“But for me I think making it through the rounds is a big thing, especially at my first Olympics.
“Just having the ability to progress from heats to semis to finals and not buckle under the pressure I think is a massive thing going forward.
“It’s something that you can take going into 2028 and use that to your advantage.”
And he knows he must be at his absolute best if he is to be successful.
“I’d say break the Irish record and you’d definitely make a semi-final, and I’d say if you repeat that and hopefully go faster you’re pretty much going to make a final.
“The thing is with the 50m freestyle, and it’s the same thing with running, it’s so close and it’s so unpredictable on the day that it is very hard to say ‘this time will make a semi or this time will make a final’.
“But basing off how the world is swimming right now, and how the world has swam at previous Olympics, I think breaking the Irish record can put me in a good position.”