Here are some ideas I had for my RPG, some I implemented some I didn't.
A boss battle in a room full of Zerg Creep. As you fight more and more Zerg Larva spawn into the room, crawling around everywhere. If you touch them, you die. So as the fight drags on it your Dodge the Rapist skills are increasingly taxed, and there's an emphasis to finish the fight quickly.
A coccoon that the boss flies into. You have to kill the coocoon before X seconds or else its health is regenerated and you have to basically start the fight from scratch.
Eggs scattered around the arena. If you touch them they hatch and lil minions come out to attack you. The boss has an ability to push or pull you around, to further increase the chance you may pop these eggs.
I have a boss that had two stages. First it's invulnerable, and doing all sorts of bound-style shit to you. (Big explosions in patterns that kill you) While this is happening you are trying to whittle down this building. Once you kill it it triggers the second stage and makes it vulnerable. Basically the theme to take away is this: have an unkillable boss, and instead give the player some other objective instead that they have to accomplish WHILE the boss is trying to kill them.
Second stage it can do things like create copies of itself, swap around with itself so you don't know its true location unless looking at unit name, it can detonate its copies or 'eat' them for boosts. It can also temporarily revert your ally status, so you default attack your allies instead of the boss, and you will show up as an enemy to your allies, so players can actually kill each other by mistake.
A Reaver (moves slowly) chasing you constantly throughout the fight. If it gets close enough to you it just squishes you dead. While this happens you can have other things going on, like a slowing spell that makes it harder for you to get away.
The opposite of the above. Make a boss that is immune to all damage UNLESS you are right next to it, or even under it (air unit), but then throw in the occasional spell where it does a super attack on the ground underneath it, so players are having to constantly dart in to do damage but then dart back out again when the pain comes.
Have an observer moving in a circular pattern around the arena. Have a burrowed unit constantly being moved to this observer. Then have a bunch of Lurkers targeting this burrowed unit. This creates a circular attack of Lurker spines that you have to avoid by running around the arena, staying out of reach of the spine attack which will kill/hurt you.
I like this ability, which I first saw in RUSH, and that is to use the Transmission action on a burrowed unit, which creates a flashing circle. This is a good way to telegraph some sort of heavy incoming attack. Players see this circle and have to react to it. (You can make it so it indicates a big blast is about to hit that area, and you have to run away, or you can make it so that circle indicates the only safe place in the arena, and you have to stand on it quick before you get fucked up!)
The getting-sucked-in mechanic. An oldie but a goodie. Use Recall animation for some sort of whirlpool effect and then issue order move to preserve. The player has to spam click to avoid getting sucked in. It's simplistic, but I think it's cool.
Don't forget the arena itself is a key component of a boss fight. Whether it's spacious or confined, whether it's full of obstacles or barren. The choices you make here can flavour the whole fight. Ideally you want synergy with your arena and your boss. Got a boss that deals heavy damage up close? Make the arena cramped.
Interactibles in the arena. Things like Health Fountains, or oil barrels (terran vespene) that you shoot to do damage, or Dark Swarm sprite (a poisonous cloud and you take damage while traversing it), or Beacons that allow you to teleport around the arena when you step on them. There are many things you can do.
Shrinking arena. Copy and paste your terrain multiple times, then make each terrain chunk smaller and smaller, give them all locations. As the fight progresses, move them from one location to another. This combined with a boss that excels when it's in melee range means your situation is going to get more and more desperate as the fight continues. And, of course, you can do the opposite as well.
Tuxedo's Boss in RUSH was cool. It was an Ultralisk in a cave that could one shot you. And you had to sneak around and get to the other side by outsmarting it. That's something to remember, not all bosses have to be combat encounters where you have to kill it. Have an unkillable boss (or optional killable) that you have to bypass instead.
Two mechanics I think are cool: one, a air unit picks you up and carries you somewhere. Two, a unit "eats you". While inside you have to hack your way out, or maybe just make it so you take X damage and then automatically get vomited back up.
A good thing to do with bosses as well is just to remove part of the player's arsenal, either temporarily or permanently. Things like:
* Disabling Consumables, or making it so Consumables only work X% of the time. (You try and eat a health or mana potion and it doesn't work but you still lose the consumable).
* Mana, either drain it via some effect or have it completely disabled.
* Spells - you can introduce X% failure chance for the spells, or make it so the spells may do something different to what they usually do (like heal instead of hurt the boss, or hit your allies instead of your intended target).
You can also make the player's arsenal work against the player. We all know how health potions keep us alive in lengthy fights, but what if the boss gets stronger each time you use one? Now, suddenly, the player is no longer that keen on drinking health potions. You've turned their own tools against them. So instead of disabling their arsenal entirely, it's like you're presenting the player with drawbacks. You can heal but X is gonna happen. You can cast a spell but Y is gonna happen, etc.
Other things that can ramp up the challenges:
* Time restraints: Like with my above Larva example, having mechanics where the fight gets harder as it goes on, or perhaps after X seconds you are screwed, or the arena starts shrinking. Time restraints are an easy way to up the challenge of ANY situation for the player. Look at most of the SC2 campaign missions and you'll see lots of times there were time restraints in place as part of the core objective.
* Multi-stage fights. If you're generous you can give them checkpoints, but if you're feeling brutal give them no checkpoints. They have to survive multiple stages of fights, with completely different mechanics in each one. The pressure is on to learn on the go or die and have to start it all again! Stage can mean either the arena itself changes, or the boss itself can evolve ("This isn't even my final form!")
* Bullet sponges - this one I am not a fan of, because I personally believe they don't really increase the challenge a lot of the time, but merely increase the time it takes to complete the fight. But yeah, one of the easiest things you can do is just give them a lot of HP, and really damaging attack values, and so the battle is a test of endurance. Can you whittle it down in time before it whittles you down??
* Bonus Objectives or Alluring Treasure - something the player has to do on top of just killing the boss can distract the player from the main goal and perhaps get them killed. You could have some treasure or a really powerful item or what not plainly visible in a dangerous part of the arena. Or you can award the player treasure for killing the boss quickly or via a certain way.
* Randomised boss spells - create, say, 10 spells for your boss, but it can only use 5 of them per fight. This will impede/delay a player's ability to learn a boss fight because each fight will be slightly different.
Ok haha enjoy that wall of text. That's all I can think of for now. TLDR: There's lots of shit you can do to make a boss battle interesting.
Yeah, you know that should be something someone requests as a feature/fix in SC Remaster - make the map pings focus more precisely depending on map size.