Phantom Falls are one of those 'lesser-visited-but-you-have-to-wonder-why' places. I'd say that I myself don't understand why, but when I think about it, it's because you don't find these falls in any of the major Great Ocean Road guides. Unless you happen to be visiting other nearby attractions you may never find this place.
So are they worth visiting? Absolutely! They are part of the proposed Trans Otway Waterfall Walk, which will one day (when the government throws some money at it!) link the Surf Coast Walk with the Great Ocean Walk, creating one continuous walk from Torquay to Port Campbell.
This section of the TOWW will use existing walking tracks, so you can enjoy the best of what it will offer today!
There are 3 ways to get to Phantom Falls, and choosing a route depends on how much you want to see of the area.
The direct Route
The most straight forward access is from the Old Allenvale Mill site carpark, just before the Sheoak Picnic Area. From here it's a 1.6km walk along St George River. From the Carpark you cross Allenvale Rd and follow the river until you cross a small bridge. From there it's through a private orchard briefly before heading uphill along a fire track. This section is steep, and some care is required, but it is also a scenic walk with good views down to St George River.
Just before a river crossing, you'll see the Phantom Falls viewing platform to your right. You won't see much from there, so be sure to explore the valley floor by continuing down the path, where you will be treated with a much better view.
Via Sheoak Picnic Area
Another option is to Take the walk from Sheoak Picnic Area, past Won Wondah Falls and Henderson Falls, through 'The Canyon', and then onto Phantom Falls. This is my preferred route, as you get 4 attractions for the price of 1, however you will have to walk considerably further!
You can do this walk as a 9km loop which has you returning via the track to the Allenvale Mill site, and then along Allenvale road and back to the Picnic area.
Via Cora Lynn Cascades
The third option is the most beautiful, but also the most difficult. From Blanket Leaf Picnic Area, which is on the way to Erskine Falls, you venture down to Cora Lynn Cascades, which is a moderate 2km walk. From here it becomes very beautiful, and difficult. It is only recommended for experienced walkers, as there are numerous creek crossings, steep and slippery terrain, and rocks and logs to scramble over. It does get easier as you approach Phantom, and if you're up to it, this walk (6.5km each way) is well worth the effort.
Phantom Falls --> Beautiful Otway Waterfalls Page
Phantom Falls --> Great Ocean Road Homepage