While searching I found Orrin's wife, Isabella and several of their adult children in Detroit from 1862 to 1880 in the city directory. Isabella is listed: Isabella Withey (wid. Orrin A.) which means he must have passed away after the 1870 Census for Wisconsin,Adams County, Dell Prairie. The thing I don't know is where and when? Was it in Wisconsin or Detroit?

Their eldest son William L. was employed in Detroit as first a cutter and later a foreman with H P Baldwin a shoemaker factory. William was also listed as a foreman with the Stephen F., and Co.
George A. was a clerk with the Michigan Central Railroad.
Stephen H. Withey was a carpenter.

George and Stephen would later serve long careers with Great Northern Railway. See entries