One source says the Phoebe often nests in or around human habitations, building mud and grass nests under the eaves of buildings. The Pee-wee builds a nest on a horizontal limb well out from a trunk in a living tree. If it had sung, I could have recognized the song, but I didn't hear anything that day. Therefore, if the nest belongs to the bird I saw, I'd guess it's the Phoebe.
Second guessing on a bird's identity can drive you crazy. However, it is also dangerous to go with the first bird you see in the book that looks kind of like the one you saw. One time I had to ask a local expert about a picture, and he confidently said it was a mocking bird. That's what I thought the first time, in fact, but the longer I worked with it, the less confident I was. Will I ever have that confidence level? There are so many pieces to look at - color, lighting, stripes, legs, song. How can I get those birds to pose nicely and sing for me at the same time? Settle down there, girl, don't get all hyper. Remember, you are doing this for the fun of it, right? Don't get so wrapped up that you forget that part. This isn't a contest, and no one will give you a bad grade if you get something wrong. Every time you try to make a tough identification, you learn something else, and that alone is worthwhile. Take a deep breath and move on to the next bird.