Thereafter Facey and his mother saw each other sporadically until she died suddenly in September 1914, aged 51. However, he only stayed a short time before accepting work back in rural Western Australia. In 1908 Facey's mother remarried, and he travelled at her request to Perth to live with her and her second husband, a plumber named Arthur 'Bill' Downie at Subiaco. The scars on his back and neck from the injuries he had sustained remained evident for the rest of his life. Although the police were informed of the abuse, his employer was never prosecuted. After sustaining months of such abuse, Albert escaped by walking over 20 miles through the bush and luckily finding some new settlers that had camped. On his first job as a farm boy, his employer subjected him to virtual slavery and violent beatings with a horse whip. He was never able to attend school, but he managed to teach himself to read and write as a teenager. He began his working life around 1902, aged eight, and hardly ever lived with his family again. When his grandfather died in 1898, the grandmother, Mrs Jane Carr, (nee Barnett), moved with Albert and his siblings Roy (born 1890), Eric (born 1889) and Myra (born 1892) in 1899 from Barkers Creek near Castlemaine, Victoria, to Kalgoorlie, Western Australia. She left her younger children, including Albert, to the care of their grandmother. In 1898, Albert's mother departed for Western Australia to care for her older children, who had accompanied their father to the goldfields. His father died on the goldfields of Western Australia in 1896 of typhoid fever, when Albert was two years old. Facey was born in Maidstone, Victoria, the son of Joseph Facey and Mary Ann Facey, née Carr.

Bert facey a fortunate life