The outreach team, which serves 32 users and is based at Park View Resource Centre, in Birch Tree Close, Lynn, will be disbanded at the end of August.
A statement issued by Norfolk Learning Difficulties Service, a partnership between Norfolk County Council and Norfolk Primary Care Trust (PCT), said: "The outreach team was providing a similar service to the community team by supporting people with learning difficulties in their own homes.
"In other parts of Norfolk this service is provided by the learning difficulties community teams, and this move to integrate them will help provide a consistent service throughout Norfolk."
The council said the equivalent of two-and-a-half full-time posts will be created in the community team as a result of the changes. The Lynn News understands seven people work in the outreach team.
A PCT spokesman said: "The PCT is doing everything it can to minimise job losses and find alternative opportunities within community services for the members of staff affected by the decision over the outreach service."
Josette Simpson (66) cares for her 48-year-old daughter, Tina, who has Down's Syndrome.
When Tina began the downward spiral into dementia in 2004 it was the Park View team which offered support to help the family cope, and who have been there, day-in, day-out, since.
The news of the team being cut coincides with a ruling by watchdog the National Audit Office calling for more specialist dementia services to be set up in England.
Mrs Simpson, of Castle Acre Road, Great Massingham, said: "We were getting support of the right calibre, but my family now feels we are
about to have a safety net removed.
"I am just concerned that it's going to be a dumbing down of services.
"It feels we've got a wonderful service here and it just seems we have been fortunate – but we want to stay fortunate."
The outreach team itself was formed after a four-bed inpatient service was pulled from Park View.
Chairman of West Norfolk Locality Parent Carer Group (for those caring for a child with a learning disability), Ian Stockwell, said: "The biggest concern of all is the total lack of any consultation.
"Each of these parents has got a care plan and none of them as far as I am aware have been individually approached, suggesting things are going to change and how it will affect them.
"I think it's diabolical, quite frankly. Many of these parents are totally reliant on these services."
A meeting to discuss the change with carers is being held on Tuesday.