do you know where he died and his age at death?

Looking for this threw up that there was a James Martin Hillhouse of Bristol who built and completed the 'Severn' warship for the Royal Navy in 1786.

I would say that this is probably the Maternal grandfather of your James Martin Hillhouse Taylor.

Some indication of YOB would be helpful; this would help narrow the possibilities.

James died in Morpeth NSW on 23rd December 1875 and was about 61, so this would mean he was possibly born about 1814.

He died Christmas morning at his home, which was beautiful Marlborough House, 75 Swan Street. He built the house and it's still there. There should be a photo of it online somewhere.
Here is his obituary;
The Maitland Mercury,Thursday 30 December 1875.

DECEASE OF MR. JAMES TAYLOR, J.P.?
At about four o'clock on Christmas morning Mr. James Taylor, J.P. of Morpeth, paid the last debt of nature.
We understand that Mr. Taylor had been under medical attendance for about a fortnight previous to his death. The funeral took place at nine o'clock on Sunday morning the remains of the deceased being interred in the Church of England division of the Morpeth cemetery.
The deceased gentleman had been a resident of Morpeth for thirty-five or thirty-six years, and for several years after his arrival he very efficiently performed the duties of agent to the A.S.N. Company, besides carrying on his business as a merchant at the same time.
Mr. Taylor arrived in the colony about the same time as the three steamers of the A.S.N. Company, known as the Rose, the Thistle, and the Shamrock.
As a man of business Mr. Taylor was possessed of sterling qualities, and was highly respected for the honorable and straight forward character of all his transactions.
As a magistrate, too, the deceased gentleman was much respected by his fellow citizens. He was noted for what may be called a thoroughness in his work and his labours, whether devoted to building or to laying out his gardens, show him to have been a man of great refinement of taste.
In his private residence at Morpeth, and in the ornamental grounds surrounding it, he has left lasting monuments of his originality and cultivated taste.
Mr. Taylor, who was sixty-three years of age, leaves a wife, five sons, and three daughters

Taken in 1974. Marlborough House Morpeth

As you can see papers wrong about age.
Just to clarify, is it 'Hillhouse' or James Martin Millhouse Taylor.
James married Mary King at Morpeth in 1847.
James Taylor headstone
at Morpeth General Cemetery, Butchers Lane.

Thanks for your reply. I have been to Morpeth but did not get to see Marlborough House but did visit the Cemetery.
Have you got a picture of the Bond Store that James built? It has now been converted to units.
Are you related to this family and if so what branch to you come from?
From what we know it is 'Hillhouse' not Millhouse, it is possible that he is related to the Hillhouse family who built ships in Bristol.