Yeah spiffy, really, don't shoot it around other peoples' houses, that isn't smart at all. It would have been much better when you noticed him to either:
a) Gathered your stuff and gone away.
b) (This one requires some big brass dangly things...) You set the gun down on the ground and go apologize to him for disturbing him. Explain that what you are using is legal and since it uses airpower it is not a firearm. Tell him that you will not do it near there again.
<i>Note: option b is quite iffy, before you do this go and make damn sure that spud guns are legal in your state, county, and city.</i>
To summarize please find a better place to fire it, and if this happens again don't let that "oh sh*t" reflex take over where you start reciting "this is not a firearm" in a loud voice. Just walk away.