My First Bug

(Here are some sweet tunes to listen to while you’re reading that will come up later.)

Something exciting about my work at Bandcamp thus far is that I’m rarely sure what I’m going to do next. The tentative plan coming in was to help one of the engineers, Michael, with his corner of Bandcamp: fan accounts. But that idea kind of dissolved when Michael became busy with orienting a new hire. So Joe has some project ideas that will be reasonable for someone with my experience and limited time here, and will also teach me a lot in the process. This is where I’d like to introduce my first bug, the shipping bug.

The fix that I had to make was to add the shipping address to the email receipt that a fan receives when she purchases merchandise, like an LP. Yep, that’s it. Although there is some more debugging that could be done. Several artists and labels had suggested this  because on occasion buyers weren’t receiving their purchases, and they’d find out later it was because items were being sent to the wrong address. By including the shipping address in the receipts, fans could double-check and contact the band to make corrections before it’s shipped out. A seemingly easy change, yes, but we also wanted to see if there was a reason the wrong address was being used in the first place.

The shipping bug was initially not even a bug at all, more of an edit to a print line. But nonetheless, it was an important enough glitch to be listed as a bug and logged in FogBugz. Fogbugz is a web tool based on the understanding that code is always evolving, and its vision is to provide a platform to support that reality. Buggy code is simply a fact of life and bugs are created even by the best programmers. Even if you write a beautiful program that fits the specs exactly and minimizes time and storage use, the technology and the program’s functionality will always be changing. Continually making improvements is what keeps a product competitive and desirable to consumers. So before I could mark this bug as ‘resolved’, I needed to know more about how Bandcamp processes sales.

Bandcamp uses PayPal to deal with payments, and all transactions are processed with PayPal’s Express Checkout. PPEC is a pretty well-written program that communicates with your website in order to exchange necessary information. The relationship is shown in the diagram below (courtesy PayPal’s website). First the request from our website is made to PayPal (PP) and a token is returned to indicate PP is ready for the transaction. The user is then redirected to the PP payment pages to login or enter their credit card info and the payment information is recorded. Then the user is returned to Bandcamp and Bandcamp sends a request to PP to make the payment. When that succeeds a final request is made to PP for the API operation GetExpressCheckoutDetails to retrieve all the details about the transaction. Finally, Bandcamp puts the information together in a receipt email. Phew.

(Sandbox is the testing interface for merchant sites)

(Sandbox is the testing interface for merchant sites)

Lucky for us PPEC is smart and knows that the user needs to enter a shipping address if they are ordering something physical. If you have a PP account you already have a billing address on file, but at payment time you can enter a different shipping address. Having two addresses, the suspicion was that maybe Bandcamp was grabbing the billing address instead of the shipping address and this would mean that packages would be sent to the wrong place, in the cases where the billing and shipping addresses were different. So, an investigation ensued! I spent a few hours exploring the information that PP sends back to Bandcamp in the GetExpressCheckoutDetails call. Turns out there’s a lot.

It was time to view some live transactions on one of the servers. Something I didn’t mention last post was the distributed system that Bandcamp has. They divide the work on the servers (located mostly at a colocation facility in Texas which no one at Bandcamp has ever been to) into eight major processes:

  • WEBAPP
  • API_SERVER
  • TRANSCODER
  • DOWNLOADER
  • SUMMARIZER
  • UPLOADER
  • DAEMON
  • IMAGER

The DAEMON is a process that works on its own to provide support for the custom distributed file system, but the rest of them work directly because of buttons you push on the website. In particular, when an artist uploads a song or a picture an UPLOADER works to collect it and place it in the file system. A TRANSCODER turns an uploaded audio file into different formats which a DOWNLOADER can then retrieve for download or streaming.

Bender1 (all of the servers are named after Futurama characters) is one of the many WEBAPPs. So we pulled it out of rotation, I ssh’d into it and we started looking through the log file (using less and grep) to find some merchandise purchases, noticing the kinds of shipping fields that are filled in when merchandise purchases are made. They look something like this:

all get_express_checkout_details response details: {"SHIPPINGAMT"=>"15.00", "L_PAYMENTREQUEST_0_QTY0"=>"1", "TAXAMT"=>"0.00", "PAYMENTREQUEST_0_ITEMAMT"=>"60.00", "L_PAYMENTREQUEST_0_AMT0"=>"10.00", "BUILD"=>"0123456", "L_PAYMENTREQUEST_0_ITEMWIDTHVALUE0"=>" 0.00000", "SHIPPINGOPTIONNAME"=>"Shipping to Canada (CA)", "L_ITEMWEIGHTVALUE1"=>" 0.00000", "SHIPTOZIP"=>"A1B 2C3", "SHIPPINGCALCULATIONMODE"=>"Callback", "INSURANCEOPTIONSELECTED"=>"false", "ACK"=>"Success", "SHIPTOCITY"=>"Edmonton", "SHIPTOCOUNTRYNAME"=>"Canada", "L_ITEMWIDTHVALUE0"=>" 0.00000", "CHECKOUTSTATUS"=>"PaymentActionNotInitiated", "INSURANCEAMT"=>"0.00", "VERSION"=>"91.0", "PAYMENTREQUEST_0_HANDLINGAMT"=>"0.00", "PAYMENTREQUEST_0_SHIPTOSTATE"=>"Alberta", "PAYMENTREQUEST_0_SHIPTOCOUNTRYNAME"=>"Canada", "EMAIL"=>"me@bandcamp.com", "L_PAYMENTREQUEST_0_ITEMLENGTHVALUE0"=>" 0.00000", "L_QTY0"=>"1", "PAYMENTREQUEST_0_SHIPTOCOUNTRYCODE"=>"CA", "PAYERSTATUS"=>"verified", "PAYMENTREQUEST_0_SHIPTOSTREET"=>"1234 1st Street", "SHIPPINGOPTIONAMOUNT"=>"15.00", "L_PAYMENTREQUEST_0_TAXAMT0"=>"0.00", "PAYMENTREQUEST_0_SHIPTOCITY"=>"Edmonton", "L_PAYMENTREQUEST_0_ITEMWEIGHTVALUE1"=>" 0.00000", "SHIPTOCOUNTRYCODE"=>"CA", "L_NAME0"=>"I'm the Best, by Yours Truly: Limited Edition - Vinyl", "AMT"=>"75.00", "PAYMENTREQUEST_0_SHIPPINGAMT"=>"15.00", "TOKEN"=>"EC-3VR045002S2330123", "PAYMENTREQUESTINFO_0_ERRORCODE"=>"0", "SHIPTOSTREET"=>"1234 1st Street", "PAYERID"=>"ZEXRCTVYBUNI", "INSURANCEOPTIONOFFERED"=>"false", "SHIPDISCAMT"=>"0.00", "ITEMAMT"=>"60.00", "PAYMENTREQUEST_0_ADDRESSSTATUS"=>"Confirmed", "SHIPPINGOPTIONISDEFAULT"=>"true", "L_ITEMWEIGHTVALUE0"=>" 0.00000", "PAYMENTREQUEST_0_INSURANCEAMT"=>"0.00", "ADDRESSSTATUS"=>"Confirmed", "PAYMENTREQUEST_0_AMT"=>"75.00", "SHIPTOSTATE"=>"Alberta", "L_AMT0"=>"10.00", "L_ITEMLENGTHVALUE0"=>" 0.00000", "CORRELATIONID"=>"84i4ff72649jt", "CURRENCYCODE"=>"CAD", "L_ITEMHEIGHTVALUE1"=>" 0.00000", "L_PAYMENTREQUEST_0_ITEMHEIGHTVALUE1"=>" 0.00000", "FIRSTNAME"=>"John", "L_PAYMENTREQUEST_0_QTY1"=>"1", "L_TAXAMT0"=>"0.00", "L_PAYMENTREQUEST_0_AMT1"=>"50.00", "LASTNAME"=>"Doe", "L_PAYMENTREQUEST_0_ITEMWEIGHTVALUE0"=>" 0.00000", "PAYMENTREQUEST_0_INSURANCEOPTIONOFFERED"=>"false", "PAYMENTREQUEST_0_SHIPDISCAMT"=>"0.00", "HANDLINGAMT"=>"0.00", "COUNTRYCODE"=>"CA", "L_PAYMENTREQUEST_0_NAME0"=>"Best Album Ever: Limited Edition - Vinyl", "PAYMENTREQUEST_0_SHIPTONAME"=>"John Doe", "PAYMENTREQUEST_0_CURRENCYCODE"=>"CAD", "PAYMENTREQUEST_0_TAXAMT"=>"0.00", "SHIPTONAME"=>"John Doe", "L_QTY1"=>"1", "L_PAYMENTREQUEST_0_ITEMLENGTHVALUE1"=>" 0.00000", "L_ITEMHEIGHTVALUE0"=>" 0.00000", "TIMESTAMP"=>"2013-01-01T01:01:01Z", "PAYMENTREQUEST_0_SHIPTOZIP"=>"A1B 2C3", "L_PAYMENTREQUEST_0_TAXAMT1"=>"0.00", "L_PAYMENTREQUEST_0_ITEMHEIGHTVALUE0"=>" 0.00000"}

Looks like quite a bit of nonsense and you’ll also notice a lot of the information is duplicated, but everything we need should be there. After looking at several of these and referring to the PP docs, we determined that they were consistently returning only one address. So, I tried to recreate the bug by making some transactions myself where my own shipping address was different from my billing address (using the out-of-rotation server so my actions were easy to follow in the log). As a test, I bought the LP of “The Cards” by Bevel Summers. (REALLY good album by the way, highly recommend it.) Then I checked out the logs again and we did in fact receive the correct address from PayPal.

From these tests we concluded that Bandcamp was always using the shipping address that the user specifies. No bug in the Bandcamp code, and showing in the receipt where the package will be sent seems like a reasonable solution to the problem. So, the next step is learning Bandcamp’s templating language liquid to change the email receipt.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s