Traditional encryption schemes work like this: Alice wants to send a secret message to Bob, so she encrypts message using a key (an alphanumeric series used to scramble data, so that the message can't be read by someone just picking it up). Alice sends the message to Bob, and Bob decrypts the message, using his own copy of the key.
There are several problems with the above situation, the main one being that Alice, somehow, has to securely get a copy of the key to Bob. If she is sending encrypted messages in the first place, it stands to reason that she has no secure delivery method, and thus, will always have to worry that someone else will be able to decrypt her messages.
During the 1970s, several mathematicians discovered a form of encryption that does not require both parties to have the same code. You would think that if Alice encrypts a message to Bob using Alice's key, and then Bob decrypts the message using his own, that the message would come out as gibberish. This isn't the case, because both Alice and Bob actually have two keys: one for encrypting, and one for decrypting. The key that people should use to encrypt messages to Alice is public. She can post it on her door, on her website, or in the local newspaper.
Alice can do this, because with PGP, her key is going to be the product of two very large prime numbers. Factoring large numbers is very difficult, and when the only two factors are primes, even more so. Alice's public key will be the product of these two prime factors, and her private key will be the two prime factors themselves.
Bob sends Alice a message encrypted using her public key. It doesn't matter if anyone else gets hold of the message, because only Alice will be able to interpret it. Bob doesn't need to private key, because he's not doing any decrypting.
Alice receives the message, and uses her private key to decrypt the message. If she wants to send a reply to Bob, she'll use the public key that he's posted online, and he'll use his private key to decrypt the reply.