What is an IPN?

(One of my favorite artists of all time!)


Week 2

Last week I learned that 0 values in the sales records are caused by a pending transaction that can be solved by retrieving Instant Payment Notifications from PayPal. This week I will be looking into how to receive IPN notifications from PayPal.

An IPN is an asynchronous message that notifies the host site about events occurring that were not initiated by the site itself. You can detect IPN messages with a method called a ‘listener’. When the listener is hit, the data from the message is similar to the feedback given from any API call. Since messages can become lost or delayed, PayPal periodically sends the message until our site responds that we have successfully received the message. According to the PayPal website, messages may be resent up to 4 days after the original message and they look something like this:

mc_gross=19.95&protection_eligibility=Eligible&address_status=confirmed&
payer_id=LPLWNMTBWMFAY&tax=0.00&address_street=1+Main+St&
payment_date=20%3A12%3A59+Jan+13%2C+2009+PST&payment_status=Completed&
charset=windows-1252&address_zip=95131&first_name=Test&mc_fee=0.88&
address_country_code=US&address_name=Test+User&notify_version=2.6&
custom=&payer_status=verified&address_country=United+States&
address_city=San+Jose&quantity=1&
verify_sign=AtkOfCXbDm2hu0ZELryHFjY-Vb7PAUvS6nMXgysbElEn9v-1XcmSoGtf&
payer_email=gpmac_1231902590_per%40paypal.com&txn_id=61E67681CH3238416&
payment_type=instant&last_name=User&address_state=CA&
receiver_email=gpmac_1231902686_biz%40paypal.com&payment_fee=0.88&
receiver_id=S8XGHLYDW9T3S&txn_type=express_checkout&item_name=&
mc_currency=USD&item_number=&residence_country=US&test_ipn=1&
handling_amount=0.00&transaction_subject=&payment_gross=19.95&
shipping=0.00

In order to receive these messages, there must be a url endpoint that PayPal can ping us with. The design of the url looks something like this https://bandcamp.com/ipn_endpoint, where ipn_endpoint is the same name given to the listener. Once we have the message, we need to use Ruby to read it somehow, that’s where the listener method comes in. One of the engineers had started looking into retrieving IPNs so I took a look at his old code. With a little bit of poking around, I found that there is checkout-ipn.rb which looks like it works as our listener and is what interprets the message sent to the HTTP endpoint. Interestingly, the listener was using a mutex for the case where two messages are sent at the same time with the same transaction ID.

The mutex is a clever way to avoid race conditions and similar problems by preventing processes or threads from occurring at the same time while they are sharing resources. From the time when the IPN is beginning to be processed to the instance where it is inserted into the table there should be a lock on all other messages that contain the same transaction ID, so we don’t find ourselves with a table insertion error since all transaction ids must be unique. It was interesting to see a high level implementation of a mutex when my experience with them mainly consisted of drawing circuits like these for my Digital Logic course:

mutex

As a side note, this was one of my favorite classes that I have taken at university so if you have the opportunity I highly recommend taking a digital logic course. Once you see the convenience of drawing karnaugh maps there’s no going back!

While investigating the implementation of a mutex, I noticed that the comments also referred to the mutex script as a semaphore. From what I’ve been able to gather, a semaphore can have a similar purpose as a mutex, but there is also much more that you can do with it. A mutex involves locking a shared resource for only one process to access at a time, and is released once the process is done using it. The semaphore can also do this, but uses a signal to let other processes know that a certain resource is being used. With this method you can control multiple resources that are being accessed by multiple processes without any collisions. I think technically in our case the design would be a mutex because we are locking and releasing the process as it accesses our one resource, the paypal_auth table.

theMoreYouKnow

The design for storing the IPN message before was that there would be a separate table with a few attributes, but I don’t think this is necessary to solve my problem because I only want to update the PayPal fee. So using mutex won’t be necessary since I’m simply going to be altering the paypal_auths table and won’t be running into insertion errors. By design, we will know if the table has already been updated for a pending transaction or not by adding two columns to the paypal_auths table: the payment pending reason and the date the transaction completed. Although, I’ll have to figure out if it’s a problem if a row in the table is altered more than once in the unlikely case where two threads will update the same row in the table if we happen to receive to IPNs simultaneously with the same transaction ID. It should be the case that the IPNs with the same transaction IDs that make it though all the filters are exactly the same message and this shouldn’t change the information in the table. Although if we allow this, we will be allowing redundancy by occasionally letting some IPN messages update the table with information that is already there.

Since this is more detail than is needed for this week, let’s get back to simply receiving the IPN. In order to really understand how we can see the message the URL and listener are receiving, I needed to read up about the web framework that Bandcamp uses. Instead of the ever popular Ruby on Rails, Bandcamp has decided to go for Ramaze, which has a reputation of being a leaner framework for Ruby.

The listener can use the instance method off Ramaze called request to store the message in a struct that can be easily searched. Once we have received the message we need to let PayPal know that we got it so they can stop sending the message (Also to make sure that the message was sent from PayPal). To do this, we make an API call back to PayPal with the exact message they sent us with this prefix: ‘cmd=_notify-validate’. PayPal should respond with either ‘VERIFIED’ or ‘INVALID’ and we should only process the payments that are verified.

Once the above is set up, we are ready to receive the IPN messages, but we still need to tell PayPal that we want them. There are two ways to do this: 1) specify our listener’s url in the IPN’s preferences on the PayPal website, or 2) Request the notification during the DoExpressCheckoutPayment API call. We have decided to use the later and dynamically request the IPNs so that our site can have more control over the requests.

Further checks we will need to do on the received IPNs:

  1. Make sure we have not already processed the transaction ID.
  2. Verify the transaction is ours by checking the email against our records.
  3. Only update our records with ‘completed’ payment types.
  4. Confirm currency and payment amount.

Week 3

All I need to do in order to complete this week’s goal is force my branch to only process messages through the Sandbox, take one of the webapps out of rotation and restart it with my branch of code, have this written as the listener:

class MainController
	def ppipn
		query_string = request.body.string
		BC::Log.debug("This is the IPN message: #{query_string}")
	end
end

and make sure this code is called from somewhere within our source code.

Since we’re making the requests for the IPN dynamic, we are actually asking for them when we send the DoExpressCheckoutPayment API by filling the field PAYMENTREQUEST_0_NOTIFYURL. To have more control we created a config feature that gives us the ability to turn on and off this request and also to allow only specific web servers to be able to send requests. So we can start testing by only requesting IPNs for DoExpressCheckoutPayment APIs that are requested on the webapp that is offline and is running with my code. But before we start allowing real IPNs through, I need to send some fake ones using the PayPal IPN simulator.

After sending the first one I immediately found an error in the log.

ESC[32m[07-11 10:55:20.072 23100 ramaze/log/hub.rb:34] ESC[0mESC[32mINFOESC[0m  Dynamic request from 127.0.0.1: /check?; host: bandcamp.com; referrer: none; client id: none
ESC[32m[07-11 10:55:20.511 85240 ramaze/log/hub.rb:34] ESC[0mESC[32mINFOESC[0m  Dynamic request from 127.0.0.1: /check?; host: bandcamp.com; referrer: none; client id: none
ESC[32m[07-11 10:55:22.650 29020 ramaze/log/hub.rb:34] ESC[0mESC[32mINFOESC[0m  Dynamic request from 127.0.0.1: /check?; host: bandcamp.com; referrer: none; client id: none
ESC[32m[07-11 10:55:22.655 99000 ramaze/log/hub.rb:34] ESC[0mESC[32mINFOESC[0m  Dynamic request from 127.0.0.1: /check?; host: bandcamp.com; referrer: none; client id: none
ESC[32m[07-11 10:55:23.269 54920 ramaze/log/hub.rb:34] ESC[0mESC[32mINFOESC[0m  Dynamic request from 127.0.0.1: /check?; host: bandcamp.com; referrer: none; client id: none
ESC[32m[07-11 10:55:23.523 96420 ramaze/log/hub.rb:34] ESC[0mESC[32mINFOESC[0m  Dynamic request from 127.0.0.1: /check?; host: bandcamp.com; referrer: none; client id: none
ESC[32m[07-11 10:55:25.460 13620 ramaze/log/hub.rb:34] ESC[0mESC[32mINFOESC[0m  Dynamic request from 127.0.0.1: /check?; host: bandcamp.com; referrer: none; client id: none
ESC[32m[07-11 10:55:25.598 57060 ramaze/log/hub.rb:34] ESC[0mESC[32mINFOESC[0m  Dynamic request from 127.0.0.1: /check?; host: bandcamp.com; referrer: none; client id: none
ESC[32m[07-11 10:55:26.309 13300 ramaze/log/hub.rb:34] ESC[0mESC[32mINFOESC[0m  Dynamic request from 127.0.0.1: /check?; host: bandcamp.com; referrer: none; client id: none
ESC[32m[07-11 10:55:26.433 77960 ramaze/log/hub.rb:34] ESC[0mESC[32mINFOESC[0m  Dynamic request from 127.0.0.1: /check?; host: bandcamp.com; referrer: none; client id: none
ESC[32m[07-11 10:55:27.995 24660 ramaze/log/hub.rb:34] ESC[0mESC[32mINFOESC[0m  Dynamic request from 173.0.82.126: /ppipn; host: bandcamp.com; referrer: none; client id: none
ESC[31m[07-11 10:55:27.997 24660 ramaze/log/hub.rb:34] ESC[0mESC[31mERRORESC[0m undefined method `Log' for BC:Module in: #
ESC[31m[07-11 10:55:27.997 24660 ramaze/log/hub.rb:34] ESC[0mESC[31mERRORESC[0m #
ESC[31m[07-11 10:55:27.998 24660 ramaze/log/hub.rb:34] ESC[0mESC[31mERRORESC[0m ./src/cart/checkout-ipn.rb:19:in `ppipn'

Yikes! But it’s actually not that bad because I was simply referencing the ‘Log’ class as a method like this BC.Log.debug, instead of as an embedded class, like this BC::Log.debug. Whoops.

I sent a couple more test IPNs and I eventually deemed my code ready to receive real IPNs. So first I turned them on using the config feature for one webabb, received about 7 per minute and decided it was safe to receive all the IPNs from each webapp and log them on the one I was using for testing. Now all I have to do is watch the log and check out what we get.

All right, so we’ll receive IPNs over the weekend and fish for them on Monday!

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s