the way Alien ended, you don't expect a sequel at all, a sequel could be made, but isn't necessary, and while some wouldn't say a seqeul is necessary to AVP(because it sucked) most would agree that the ending was done to sort of set up a sequel, most saw a sequel coming, with an ending like that, that is more or less a cliffhanger
unlike the ending to Alien, which as you said is quite resolved, the credits roll when expected, whereas in AVP, there is an extra surprise before the credits, letting you know that the sequel would generally head in that direction of the pred chestburster, it was a set up, whereas Alien didn't have a direct set up right before the end
if at the end of alien, you saw ripley go into the cryostasis tube, and then slowly panned over to an egg, accompanied with an ominous string of music, then its ending would have been comparable to AVP's ending.