I love my boyfriend but I am afraid he has no room for me in his life.....?

I've been with my boyfriend for 9 months and I love him very much. Problem is he has two kids with his ex and was a stepdad to her kid from a previous relationship. When we first started seeing each other, everything was great! But he's received a lot of crap from his ex so now he is there with his kids everyday all day till they go to sleep. I only see him after 9pm on weekdays after kids in bed and almost never see him on weekends, unless again, it's after 9-10pm. I don't get to see him during holidays cuz he is with kids. He says it could be years before he is ready to be open with us about ex so until then his time with his kids was his time only. I tried to ask him for one weekend day for the two of us to spend the whole day together, to give him enough notice I said a month from now when you don't have kids. He can't even promise me that and seemed irritated I even asked. I tell him if he doesn't have room in his life for me it's okay I understand, commend him for being a great father, but I have needs too. He just gets mad and makes me feel like I am being unreasonable and making him choose between him and his kids. I'm not! I just need something...I don't want to see him for just a couple hours a night when he's done living his life and all tired. I have few friends and no kids of my own, and really feel like I should be out there starting my own family. It will be years before he's ready. I'm 30 he is 25. His kids are 2, 5 and his stepdaughter is 8. What should I do? Everytime I try to talk about it he gets mad or says he'll figure something out later and then doesn't. We really love each other, can anything be done? I don't want to interfere with his fathering but I deserve to spend an occasional afternoon with my boyfriend now and then? Do things other couples do, go to dinner, stuff like that. I want a real relationship.