As many item types are stored in the distributed cache (e.g. tags, document activities, authentication and security information etc.) it is not recommended that the distributed cache is disabled.
Many of the objects that end up in the distributed cache are computationally expensive, time-intensive to fetch, or a combination of the two. Storing these items in memory spread across a number of servers in the farm eases the pressure on the farm to return these items and allows operations to proceed more quickly.
Having said that, the default configuration option for the distributed cache on the farm is to have it running on all servers, a situation that Microsoft recommends against as this can also cause issues. This does however depend on the number of servers in the farm. If you have only a couple of servers in the farm (e.g. two servers that are acting as combined WFE and APP servers), then the cache service should run on both of those, if you have greater number of servers (e.g. two WFE servers, two APP servers), then consider running the cache on only two of those servers (generally the two WFE servers). Try to avoid running other services on the same servers as the distributed cache, specifically Search Services, Excel Services and Project Server Services.
The recommended approach is to use dedicated cache servers and allocate most of the RAM on the server to the cache service. For what would have been a four server 2010 farm (2x WFE, 2x APP), this ends up being a recommendation for a six server farm in 2013 (2x WFE, 2x APP and 2x distributed cache).
Also check the SharePoint logs for mentions of the distributed cache as anything that is causing issues should show up in the logs (e.g. permission errors, access issues etc.).