CALGARY, Alberta (Reuters) - Canada's oil sands producers are stuck in a rut.

The nation's oil firms are retrenching, with large producers planning little or no further expansion and some smaller projects struggling even to cover their operating costs.

As the era of large new projects comes to a close, many mid-sized producers - those with fewer assets and producing less than 100,000 barrels of oil a day in the oil sands - have shelved expansion plans, unable to earn back the high start-up costs with crude at around $50 per barrel. Larger Canadian producers, meanwhile, focus on projects that in the past were associated with smaller names.

The last three years have seen dozens of new projects mothballed and expansions put on hold, meaning millions of barrels of crude from the world's third-largest reserves may never be extracted.

Where industry groups in 2014 expected Canada's oil sands output to more than double to nearly 5 million barrels per day (bpd) by 2030, that forecast has been knocked down to 3.7 million bpd.

This follows a spell of consolidation that has seen foreign majors sell off more than $23 billion in Canadian assets in a year and turn to U.S. shale patches such as the Permian basin in Texas, which produce returns more quickly and where proximity to refiners means the barrels fetch a better price.

"We cannot compete with that huge sucking noise to the south that is called the Permian. Investment dollars are spiraling away down there," Derek Evans, chief executive of small oil sands producer Pengrowth Energy told Reuters in an interview.

Permian production rose 21 percent in 12 months through July compared to a 9 percent increase in Alberta's oil sands, according to Canadian and U.S. government data.

COSTLY STARTUP PHASE

Mid-sized producers are hurting the most, due to start-up costs that far exceed those in other major producing areas. Oil sands producers have slashed operating costs by a third since 2014, but building a new thermal project - in which steam is pumped as deep as one kilometer (1094 yards)underground to liquefy tar-like bitumen and bring it to the surface - requires U.S. crude benchmark at around $60 a barrel to break even, analysts estimate.

The North American benchmark West Texas Intermediate crude has traded between $42 and $55 a barrel so far this year. The U.S. Energy Information Administration forecasts it will average $49.69 a barrel in 2017 and $50.57 a barrel next year.

There are around half a dozen thermal projects in the costly start-up phase, when engineers steadily increase steam pressure to bring a reservoir's production up to full capacity.

One of those is Athabasca Oil Corp's Hangingstone project. It was originally conceived as a 80,000 bpd project, but instead will bring output to only 12,000 bpd from the current 9,000 bpd. The project can break even with U.S. crude prices of at least $53 a barrel, meaning right now Athabasca keeps losing money on Hangingstone production. Size is crucial in the oil sands; the more bitumen a company can squeeze out of a plant, the lower fixed costs per barrel will be.