“Can you remind me why we’re here?” my husband asked, sounding slightly annoyed. I’d promised him a mini break by the sea, sans kids for the first time in five years. What the reality looked like, however, was pouring rain in a tiny Tasmanian town where low-rise houses were huddled by grey, choppy water and nary a shop in sight. It was winter and freezing cold. I could understand why he was less than impressed.

I feel like this is Australia’s birthright. And people just don’t know about it.

We’d just arrived in Southport, the most southern township in Australia, where we were staying at a deserted caravan park that declared itself “The Last Stop in Tassie” – whether that was proudly or despondently, I wasn’t quite sure. My husband also didn’t yet know that I had an ulterior motive to our child-free break. I’d been hearing more and more over the last few years about the Aurora Australis, the southern sibling to the Northern Lights. Although tourists flock to see the colourful, dancing lights that sweep across the skies in high-latitude regions from Canada to Iceland, few seem to know that the southern hemisphere has its own equally magical lightshow – and I was determined to see it.

“I feel like this is Australia’s birthright. And people just don’t know about it,” said Margaret Sonnemann, author of the Aurora Chaser’s Handbook and founder of the Aurora Australis Tasmania Facebook page.

The Southern (and Northern) Lights can be seen around the magnetic poles when the upper atmosphere is hit by energetic charged particles that travel along the Earth’s magnetic field lines. When those energetic electrons collide with gases in the atmosphere, they emit light.

“They occur when there’s a release of energy due to collisions between polar atmospheric gases of the Earth and precipitating high-energy charged particles guided by geomagnetic field from the magnetotail,” said Dr Rakesh Panwar at the Australian Bureau of Meteorology, Space Weather Services. He explained that they are seen mostly at high or polar latitudes, but occasionally may be seen at middle latitudes when a large solar event occurs.

Although it’s true that the Southern Lights have been seen from mainland Australia, they’re more commonly seen from Australia’s island-state of Tasmania (or from New Zealand), due to its clear skies and proximity to the pole, which is why I’d dragged my husband 600km south from Melbourne.

I wasn’t the only one fascinated by the magic of these dancing lights. Sonnemann, an American who moved to Tasmania 32 years ago, told me that she got hooked on aurora chasing back in 2005 when she was driving south to Hobart, the state capital. Although it wasn’t the first aurora she had seen, she said it was the most captivating. It was happening behind her as she drove, reaching up to the heavens, straight above her head.

“It was pretty much a black-and-white experience, but the movement of the beams was magical,” she said. “It changed dramatically from second to second, shooting up into the sky. It was beautiful and exhilarating.”

You may also be interested in:
• The food that defines Australia
• A ruggedly beautiful quarantine site
• Australia’s bizarre coastal desert

Although Sonnemann knew other locals who were interested in aurora chasing, there was no central source of information to draw on to find out when and where they were happening. She realised that a notification system was needed. “Since auroras are weather dependent, we needed to be able to not only say ‘it’s happening now’, but where it was happening,” she said.

Sonnemann started her Facebook page in 2010, mostly with a handful of interested locals (it now has more than 85,000 members and 10 admins). “But it quickly became photographers,” she said. “That’s because aurora photography can be the best way to see the lights.”

She said that although people associate auroras with streaks of colour lighting up the sky, in Tasmania, you’ll likely see a white light without a camera, perhaps with some coloured glow around it, unless there’s enough cosmic activity to make the aurora big enough. “You can’t see a lot of colour with the naked eye, as our eyes are not designed to see colour at night,” she explained. “To see strong colour, [the aurora] needs to be very bright.” However, use a camera and the results can be spectacular.

Because we’re looking more towards the horizon, we get just about every colour: red, greens, yellows, blues and purples

Ask Sonnemann what she loves most about the Aurora Australis, and her face lights up. She’s passionate about the topic and it makes sense when she tells me that people call her the “Aurora Lady”. “One of the nice things about our auroras is that most of them are not overhead,” she said, explaining that Tasmania is much further from the South Pole than many northern hemisphere aurora hotspots are to the North Pole. “What that means for photographers is that they get a better variety of colours if they’re looking at something that’s on the horizon. If you look at aurora photos from spots in the northern hemisphere, they’re often green, which is pretty much the colour you’re going to get if it’s overhead. Because we’re looking more towards the horizon, we get just about every colour: red, greens, yellows, blues and purples.”

“I like to say that it’s like seeing music,” she added, smiling.

What’s also unique about the Southern Lights is that they can be seen year-round. Although many people (myself included) assume that aurora chasing is a winter sport; that’s because northern hemisphere aurora spots have the midnight sun in summer. In the southern hemisphere, whether you’re in Tasmania, New Zealand or even parts of southern Victoria, you’ve got a real possibility of seeing an aurora in mild summer weather, although you’ll have to stay up later as it doesn’t get dark until 21:00 or 22:00. And the lights can be equally stunning, ranging from intense white pillars in the sky to ethereal streaks of greens and reds.

However, I’d clearly picked a bad weekend, with murky skies and fat drops of rain. One Southport local told us that he can often see the aurora from his back deck in summer. But not this time of year. Come back when it’s clear weather, he said. He was right. The sky was a blanket of grey cloud, and later, when it got dark, there was nothing to be seen in the thick, inky blackness.

However, when I’d met with Sonnemann earlier that day, she’d told me that I’d made another incorrect assumption – I didn’t need to be staking out Australia’s southernmost spot for my best chance of seeing the lights.

“Preferably you need to be away from city lights, but you can see them anywhere in this state,” she said. “You need a clear view to the south, so no hills, no trees. It’s just as good all over Tasmania, whether you’re in Devonport or Hobart. Anywhere where it’s darkish. You can see it in your own back yard.

So, the next morning we packed up and fled the tiny town for Bruny Island, two islands connected by an isthmus across the D’Entrecasteaux Channel from Southport. Bruny Island has a wealth of delights to interest the traveller, from its world-famous cheese to its unique white wallabies, so I figured there would be plenty to do even if we were thwarted by the fickle light show. The island is so remote that even without an aurora the night skies can be magical, with the Milky Way on full show and a blanket of stars cossetting you.

We drove from the ferry terminal, past wineries, breweries and cheese shops to our accommodation, down a single road with dense gums on either side that occasionally broke rank to give a startling view of the glistening ocean. The accommodation I’d booked had a 180-degree panorama over Cloudy Bay, where lulling waves broke into white-crested spumes as they flopped onto the shore, which was a strip of golden sandy beach surrounded by thickly forested hills of blue gum and stringybark. It was spectacularly beautiful.

“This is a photographer’s dream,” said Ben Kienhuis, operations manager at Bruny Island Coastal Retreats, as he unlocked the front door and showed us in.

He was right. From the living room window was a clear view across the bay, with nothing blocking our sightline to the horizon. This, I hoped, would be the place to see the ethereal aurora. Kienhuis explained that the night skies here can be so spectacular that his team have recently started offering aurora photography tours, which take amateur photographers to various locations on the island to take in the glory of these clear southern skies and, hopefully, the Aurora Australis. They believe their new aurora tour is the first of its kind in Tasmania.

“The auroras can be spectacular,” said photographer Luke Tscharke, who runs the tours with Kienhuis. “But there’s no aurora tourism [in Tasmania],” he added. “You don’t see people advertising to take you out to see the Southern Lights because it’s just so challenging to predict or guarantee that you’d see them.”

But the thrill of the chase is clearly part of the fun. To help me with my search, Sonnemann had introduced me to various apps and meteorological sites that are designed to help the amateur aurora chaser, which detail the wind speed, the Bz (the orientation of the interplanetary magnetic field) and the K-index (the level of disturbance in the Earth’s magnetic field), among other indicators. Combined, they give a guide as to the probability of an aurora happening. Facebook groups, such as Sonnemann’s, are also an excellent place to find out from locals on the ground where and when an aurora might occur. Understanding the readings was complex but addicting. I found myself feverishly checking the app on my phone, hoping for a change in conditions that might bring the aurora to me.

“There are never any guarantees,” Sonnemann reminded me. “People say, ‘I’ve heard this prediction…’, but you never know.”

Yet, despite the elusiveness of the lights, there’s been a surge in interest in the Aurora Australis over the last few years. Sonnemann explained that most of the members on her page are now not locals and photographers, but international tourists who want to see the aurora. Her concern is that there’s currently no infrastructure to handle them.

You don’t see people advertising to take you out to see the Southern Lights because it’s just so challenging to predict or guarantee that you’d see them

“It really disturbs me that we have people coming down here, from places like Singapore, who have never seen a dark sky before, and are petrified of driving in the dark,” she said. “Their whole lives have been fully lit. They’ve never seen ‘dark’. So they end up flying down and don’t even go out. They don’t want to drive in it. I just think that’s horrible.”

To try to meet the demand, she’s working on a project to create aurora viewing platforms, which she envisages as sheltered spots that people can drive or taxi to, where they’ll find explanatory signage and photography tips ­– and likely other aurora chasers.

“Even if there’s not an aurora – and there’s not one every night – they might be seeing the Milky Way for the first time,” Sonnemann said. “They can still see a beautiful night sky, even if there’s no aurora.”

That night I stayed up late, staring at the thick cloudy sky through the windows. The K-index was hovering at one and the wind speed was low, meaning an aurora was unlikely. It was freezing cold, and I fell asleep in front of the fire. Around 02:00, I woke up, startled by a noise perhaps, or maybe a sense that something was going to happen. I opened the sliding doors and stood in front of the bay, the frigid air pressing into me on all sides and making it hard to breathe. I looked up into the thick night sky. There was no aurora. But the clouds had finally cleared, leaving a never-ending blackness punctuated with stars. I stood there quietly, listening to the gentle roar of the ocean that stretched, unhindered, all the way to the South Pole.

I should have been disappointed, of course, but for some reason I wasn’t. Perhaps it was the magic of the experience or the magnitude of sky; whatever it was, I was sure that the aurora would reveal itself to me when the time was right. For now, though, I drank in the solitude, the space, the silence. Whether or not I was ever going to see the Southern Lights, I knew that I was glad to be here, on an island off an island off an island on the very edge of the world.

Join more than three million BBC Travel fans by liking us on Facebook, or follow us on Twitter and Instagram.

If you liked this story, sign up for the weekly bbc.com features newsletter called “The Essential List”. A handpicked selection of stories from BBC Future, Culture, Worklife and Travel, delivered to your inbox every Friday.