I don't know what specific NY dough formulation you are using, but your experience seems quite typical for a high-hydration NY style dough.
I often carefully observe my NY style doughs during their complete lifespan up to the point of use in order to detect patterns that might help me in future efforts, or simply to be able to learn enough to be able to answer questions like yours.
Depending on the formulation itself (mainly the amount of yeast) and the finished dough temperature, it is common to expect some rise in the dough after being placed in the refrigerator. Since it takes about a half hour for the yeast to start working, and since the dough often is placed in the refrigerator right after the dough has been made, you will usually not see a significant rise, if any, within the first hour or so. It will be like watching paint dry. Depending mainly on the amount of yeast used and the dough's temperature while in the refrigerator, the dough will often then noticeably increase in volume. After the passage of sufficient time, and certainly 3-4 days would qualify as "sufficient", the dough will often then flatten out, and especially if it is in an unbounded container such as a zip-type storage bag. It is also likely that the dough will become wet and a bit sticky. The reason these things happen is because the enzymes in the flour/dough, mainly protease enzymes, attack the gluten and relax and soften it. Water is also often released, creating some wetness in the dough. I might add that the higher the hydration the greater these effects, because high hydration doughs ferment faster. At this juncture, handling the dough, including the simple step of removing the dough from its container, can become a problem. And, as you noted, some of the imperfections you introduce at this point can carry through to the finished skin.
In terms of the amount of dough expansion you can reasonably expect, doubling or otherwise, will largely be dictated by the amount of yeast used and the temperature of the dough at its different stages. I have made what seemed to me to be identical doughs yet got different results in terms of dough expansion. Dough will behave differently at different times of year and water (tap) temperatures and refrigerator temperatures will vary seasonally. I have had winter doughs rise hardly at all while in the refrigerator yet produce very good results. But, on average, my doughs tend to increase by about 1 1/2 to to 2 times. Sometimes it is hard to tell because the dough will first rise and then flatten out, as discussed above. In almost all cases, the doughs behave similarly and end up at about the same point when they have been allowed to warm up in preparation for shaping, stretching, etc. It may take longer in winter than in summer, but that is what the laws of thermodynamics say should happen. I sometimes use a thermometer to check the dough temperature and use the dough when it gets to the desired temperature.
In your case, you may want to investigate better ways of holding your dough during fermentation. I usually use a fairly large metal container with a tight fitting lid (I use a lidded cookie tin). I am able usually to remove the dough without significantly maiming it. I often just turn the tin upside down and tap the bottom to let the dough fall out. I like zip-type bags also but I have discovered that it is hard to remove the dough after 3-4 days because of the flattening and spreading, and the wetness. In these case, I try to place my hand under the dough as gently as I can and then slowly retract the dough from the bag. The bigger the bag, the easier it is to do this.