Of course! That's why it's good to be aware of your agent agreement. But every situation is different, so I can't really tell you "This is where X will happen."
If the agent put in significant work, if their direct negotiation resulted in rights being sold, if the deal was ginormous and they felt they had a case for their cut... then you might end up in a situation where litigation could happen. But the likelihood is soooo small.
Think of it this way, if you have a smashing agent who is negotiating great contracts for you and paving the way for film rights... you're probably not firing them anyway. Most unpublished clients fire their agents because the agent isn't actively pitching their work anymore, or isn't following up or is no longer responding. They're firing them because the agent isn't really working toward a sale for them anymore. In that case, it's highly unlikely anyone is going to sue anyone. The agent is probably just as happy to be rid of the client as the client is to be rid of them.
Once you're published, it gets more complicated. And again, every single situation is going to be totally different. But the agent is a professional and if you behave professionally, I'd say more often than not everything will work out okay. It's always awkward, but it is what it is.
So just be sure to check your agency agreement, ask questions about that event when you interview interested agents and just use common sense.