As an author, my chief financial concern is that I am paid fairly for my work.
In the realm of books, I want to be paid the equivalent of once per printed copy (we won't touch electronic distribution here. That's a much more squiggly can of worms) - if by some chance I get paid more then that, well, let's just call that gravy and pass the mashed potatoes!
I don't have a problem with book swapping. I think we've all gotten a book from a friend or relative to read. And I look at it this way. If someone that might never have read my work picks up one of my books at a swap and likes it, they may just buy the next one.
Dog-eared exposure and fan loyalty is just a different kind of royalty in my opinion.