Figured it out.

public function getAllTags()
{
     $chapters = $this->chapters()->select('id')->get();
     $tags = Tag::where('taggable_type', Chapter::class)
             ->whereIn('taggable_id', $chapters->pluck('id'))
             ->orWhere(function($query) {
                $query->where('taggable_id', $this->id);
                $query->where('taggable_type', self::class);
             })
             ->join('taggables', 'taggables.tag_id', '=', 'tags.id')
             ->select('tags.*')
             ->groupBy('tags.id')
             ->get();

     return $tags;
}

I just had to start from Tag and build the query forcing the join by hand.

Did some benchmarking on the two different functions and am seeing a 65% speed improvement with 4 tags, which will only grow as the previous technique has a +n issue.

Leaving this here incase it helps anyone else.