My rescue story is similar.
I was on my way to the store when I saw a cocker trotting down the street, not far from my house. I pulled over to check for ID tags. None. The dog seemed very frightened and timid, but let me approach it, and it finally warmed up to me. My wife and I started visiting the surrounding neighbors looking for a lead.
Finally we found one that knew the dog. She seemed very troubled by it and told us the story. It belonged to her next door neighbor. Their previous dog had died of neglect (according to this neighbor). When the children cried, soft-hearted Grandma saved the day by buying them a new puppy. After the novelty wore off, this dog's life pretty much picked up where the last one left off. Most of its meals were provided by sympathetic neighbors. When the neighbor realized that we were "dog people", she pleaded with us to just take the dog and not say anything to the owner.
Now we were in a quandary. On the one hand, I didn't want to give the dog back to people that were going to give it a miserable life, but on the other, I know how I'd feel if my dog was "stolen". We sought the advice of a married couple who were good friends and dog professionals - Jeff a veterinarian, and Laurel a trainer. Their advice was to NOT bring the dog back to the owners and NOT advertise that we found a lost dog. We should simply see if they made any attempt to find the dog. Laurel said, "If I ever lost one of my dogs, within 24 hours, there would not be a single person in a 20 mile radius that didn't know about it."
We watched all local newspapers and grocery store bulletin board and saw no attempts to find her. And to make things even better, the family moved away, two years later. She's been with us for ten years.