An FDNY spokesman also said there may be a sixth victim, but would not elaborate on the person's age or condition.
The 3-alarm fire broke out in a two-story home on 48 Avenue in the Jamaica section of Queens.
Mayor de Blasio was rushing to the scene.
Firefighters rushed to the scene of the fire just after 2:30 p.m. Sunday.
"It's a very hard moment for people on this block, to see a family literally destroyed before their very eyes", de Blasio said, surrounded by elected officials and fire officials. He said it was "a fire that moved very, very quickly, and the loss was horrendous".
The fire department said it received a 911 call on the fire at 2:36 PM.
Fire Commissioner Daniel Nigro said the tragedy was taking a toll on firefighters who battled the blaze, many of whom are still reeling from the line-of-duty death last week of Firefighter William Tolley, who fell five stories from a roof while fighting another fire in Queens. The guy who was carrying the baby out, you could just see the stress on his face.
Two firefighters were injured in the blaze as well, but the extent of their injuries were not clear, an FDNY spokesman said.
The wood-frame home burned rapidly and was already engulfed in flames by the time firefighters arrived. Witnesses heard loud booms before the blaze broke out.
Denise Coleman lives about a block from the fire and said she saw a man she believes is the children's uncle trying to get into the house to rescue the victims.