We study a new variant of the well-studied Vehicle Routing Problem with Time Windows (VRPTW), called the fragility-constrained VRPTW, which assumes that 1) the capacity of a vehicle is organized in multiple identical stacks; 2) all items picked up at a customer are either ``fragile" or not; 3) no non-fragile items can be put on top of a fragile item (the \emph{fragility} constraint); and 4) no en-route load rearrangement is possible. We first characterize the feasibility of a route with respect to this fragility constraint. Then, to solve this new problem, we develop an exact branch-price-and-cut (BPC) algorithm that includes a labeling algorithm exploiting this feasibility characterization to efficiently generate feasible routes. This algorithm is benchmarked against another BPC algorithm that deals with the fragility constraint in the column generation master problem through infeasible path cuts. Our computational results show that the former BPC algorithm clearly outperforms the latter in terms of computational time and that the fragility constraint has a greater impact on the optimal solution cost (compared to that of the VRPTW) when vehicle capacity decreases, stack height increases and for a more balance mix of customers with fragile and non-fragile items.