The government is in "cuckoo world" when it talks of increasing produce, said a farmer who has lost £50,000 of asparagus due to a shortage of workers.
Norfolk farmer Andy Allen could not get enough pickers and so a 10-acre field has been left unpicked.
Farmers have been hit by the loss of EU workers after Brexit, the loss of Ukrainian seasonal workers and a limited number of foreign worker visas.
The government said it would be increasing the number of visas issued.
Mr Allen, of Portwood Farm, near Attleborough, said he had decided not to plant any more asparagus.
"We waited four years to get a harvest from this particular field and so to not have anything from it at the end of the day is pretty demoralising," he told BBC Politics East.
"We would normally need between 90 and 95 staff. At the maximum we had 75, and during a lot of the season we were dealing with 65.
"We would have expected to harvest 150-250 tonnes this year. We are 30% down."
When asked about the government's announcement this week that it wanted farmers to plant more produce, Mr Allen said: "They are in cuckoo world."
The May 2022 Food Labour Market Survey found that 49% of growers and food manufacturers had reduced their output due to a lack of workers, while 77% of businesses were experiencing shortages of low and unskilled workers.
Farmer Jack Pearce, who farms near King's Lynn, Norfolk, has also been struggling for workers, not so much for picking, but for processing.
He said he had just enough workers at present, but he has had to ask local farmers to grow fewer onions.
"In three generations of this business we've never had a myriad of successive challenges all happening concurrently," he said.
"We are not able to run extra shifts and it's become increasingly harder to sell more and develop as a business.
"Every time opportunities come along, the first thing we have to ask is 'do we have the labour to do it?'."
Charles Hesketh, regional policy manager for the National Farmer's Union in East Anglia, said: "I was talking to a guy who was growing a lot on the coast of Suffolk.
"His vegetable production has been scaled back 50% this year and this is solely because he can't take the risk of not having enough pickers and packers to bring in the crop.
"We are really worried and so are our members, and it's frustrating."
Government farming minister Victoria Prentis said: "Seasonal labour has long been a problem for farmers. These are seasonal jobs and a settled workforce wants a long-term job and doesn't want a job for a couple of weeks over summer.
"So we have always brought in seasonal workers and I am pleased to announce an extra 10,000 visas taking it to 40,000 for seasonal workers that we are able to bring in this year to harvest our crops.
"There is more that we can do. I'm very keen we should start to talk about food and farming jobs as the technology-savvy, exciting opportunities for young people.
"I'm also keen we work on automation because that will reduce the need for some of the workers."