(About time I posted some Coeur de pirate.)
At the end of the last post, I had found the information about IPNs and catching them. This post will be more about observing the execution of the plan developed from the last two weeks.
This week the final product is in sight! We set up receiving IPNs over the weekend and caught a good amount. That’s good news but I needed to figure out if any of these IPNs were completed pending payments. So I looked at all the transaction IDs of the pending transactions to see if I could find them with a ‘Completed’ status at a later date/time. It became obvious that grep would be the right tool for me so I started reading the man page learning some important features like
-c which counts all the instances of your match. I started thinking about how I could use regular expressions within the grep command to select a specific section of the line search. I started with something simple by printing just the pending transactions:
$ grep 'payment_status=Pending' log
Since the message is 1 long string, we print the whole transaction when we search like this. It would be better to only print out the transaction ID since that is all that we need to identify the transaction if it becomes completed. So with a bit more research of the man page, this should do that:
grep -Eo 'txn_id=[^[&]*]*\&'
The –Eo functionality is a combination of E, which interprets the pattern as an extended regular expression, and o, which shows only the part of a matching line that matches the pattern. Then we can combine the commands by piping one to the other from less, which can understand zip files:
$ less log | grep 'payment_status=Pending' | grep -Eo 'txn_id=[^[&]*]*\&'
Ok, so I have a list of transaction IDs, probably hundreds of them too, which will look something like this:
txn_id=39RKW48294IR823& txn_id=KF7EI393U2R849F& txn_id=E23J393093OW93H& txn_id=38DI2O4FJ8JF822& …
What would be ideal is if I could strip all of these down even further to just get the transaction ID without the ‘txn_id=’ and ‘&’. This took a fair bit of interneting to find the sed command, a text stream editor, that did the trick. And dun da da dah! The whole thing:
$ less log | grep 'payment_status=Pending' | grep -Eo 'txn_id=[^[&]*]*\&' | sed 's/txn_id=//;s/&//'
Now I can use the exact same command to find the completed payments by greping with ‘payment_status=Completed’ and then compare the two lists: pending transaction ids to completed transaction ids. Since I don’t have write privileges on the webapp where I was reading the logs, I copied the two types of lists to my machine and created a Python function to read two files and print out the ones that matched. Only about 10% of the pending transactions had completed. I think this is probably OK to see such a low number this far into the game because I’m sure we will see many more complete as time progresses. In fact, here is a sneak peak of what the trend of progression will look like over the next two weeks:
This is showing an interesting flat line at ~50% of transactions that begin as pending and never complete. We’ll see how this changes after another week of data. This is also giving us some foresight into what the average wait time is for the vast majority of pending transactions to complete.
Also notable is the kinds of pending transactions that were completing since we noticed before that the reasons for the most pending transactions are either unilateral (the seller’s email has not been verified) or echeck payments. Although over the weekend we mostly received unilateral pending payments, some other, some multi_currency, and some verify, none were echeck! Explicit definitions of all of these reasons can be found on the PayPal website. Although it is good that we are seeing payments completing, it is important to know why we aren’t seeing any echeck payments completing. It’s possible that the delay may be caused by the functioning hours of finantial institutions and mostly looking at data receiving over the weekend. And, according to the PayPal website, it usually takes between 3 and 5 business days for an eCheck to clear and the money to appear in the recipient’s PayPal account.
After a few days of viewing our very own IPNs, on Wednesday I finally found my first echeck payment completed. A monumental day indeed! The rest of the week I continued to look at how many payments had completed and approximately how many days it took to complete pending transactions.
After a bit more data I’ll post some interesting stats, in particular some graphs because pictures are way more fun.