Thats whats happening in Japan and Russia. Their birth rates are plummeting and in 20-30 years their workforce will be devestated by the lack of workers. When enough men decide they dont want to play the rat-race career and dating game anymore, it becomes socially acceptable. So more and more will do it. Inevitably the society collapses on itself when all the skilled workers die without replacements. Your only real choice then is a large influx of immigrants to breed with your population.