Siegel.ie: September 2008

Sunday, September 28, 2008

GData Python API woes continue...

Unfortunately, using the GData API is starting to feel like being in a boat so hastily constructed, that every time you patch one leak, you need to run right to the next one. Today's issue either happened on Google's end or as a consequence of the recent 1.2.1 GData update.

My issue is similar enough to one on the mailing list, that I posted it there. The basic issue is that the gd_client.InsertRow() call works like a charm if your app happens to have a logged in Google-sign-in user. If not it fails in an absolutely unhelpful 404-Not Found (full text below). Moreover--Google's initial response seems to say that this change (requiring a logged in user) was intentional... but clearly breaks any app that relies on stored authsub tokens (up to this point a completely valid and encouraged method).

My hope is that this spate of issues is a short-term anomaly, as we can't afford this level of attention moving forward. I know Google has some good people like Jeff Scudder working on the project. I hope they give him enough team support and/or time to communicate to us early adopters. Specifically, if you are breaking the API expectations, tell us--rather than slipping the change in a 1.2.0=>1.2.1 update which was critical to fix other issues. Hrmph!

For the time being we're going to have to remove the GData code from our app, as we can't afford the downtime even on our relatively vanity app. Looking forward to Google's response.

Environment:Request Method: POSTRequest URL: http://localhost:8081/l/agpyaWdodHJlcGx5chELEgtMYW5kaW5nUGFnZRgCDA/widget.htmlDjango Version: 1.0-beta_2-SVN-unknownPython Version: 2.5.1Installed Applications:('myapp', 'appengine_django', 'django.contrib.auth', 'django.contrib.sessions')Installed Middleware:('django.middleware.common.CommonMiddleware', 'django.contrib.sessions.middleware.SessionMiddleware', 'django.contrib.auth.middleware.AuthenticationMiddleware', 'utils.auth.Auth')Traceback:File "/Users/.../google-app-engine/rightreply/django/core/handlers/base.py" in get_response  86.                 response = callback(request, *callback_args, **callback_kwargs)File "/Users/.../google-app-engine/rightreply/myapp/landing_pages/views.py" in widget_html  62.       lead.save()File "/Users/.../google-app-engine/rightreply/myapp/models.py" in save  217.     self.submit_to_google_docs()File "/Users/.../google-app-engine/rightreply/myapp/models.py" in submit_to_google_docs  253.     entry = gd_client.InsertRow(dict, self.landing_page.spreadsheet_id, self.landing_page.worksheet_id)File "/Users/.../google-app-engine/rightreply/gdata/spreadsheet/service.py" in InsertRow  325.         converter=gdata.spreadsheet.SpreadsheetsListFromString)File "/Users/.../google-app-engine/rightreply/gdata/service.py" in Post  831.         media_source=media_source, converter=converter)File "/Users/.../google-app-engine/rightreply/gdata/service.py" in PostOrPut  951.           'reason': server_response.reason, 'body': result_body}Exception Type: RequestError at /l/agpyaWdodHJlcGx5chELEgtMYW5kaW5nUGFnZRgCDA/widget.htmlException Value: {'status': 404, 'body': '<HTML>\n<HEAD>\n<TITLE>Not Found</TITLE>\n</HEAD>\n<BODY BGCOLOR="#FFFFFF" TEXT="#000000">\n<H1>Not Found</H1>\n<H2>Error 404</H2>\n</BODY>\n</HTML>\n', 'reason': ''}

Friday, September 26, 2008

GData and DJango's HttpResponse Clashing -- 'unicode' object has no attribute 'content'

GAE's GData API appears to be going through some rapid flux and our app is taking its punches. The changes are exposing some deeper issues which we've just tackled. Here's a doozy:

views.py
from django.http import HttpResponse
from myapp.models import *
...
  return HttpResponse("...")

models.py
from gdata.alt.appengine import *  # For new GData API calls
...

The above code listing illustrates our approach to patching the new GData APIs into our model calls. We didn't realize that we were also introducing the error below. Why? Because HttpResponse is defined in both django.http AND gdata.alt.appengine. This causes import-line-ordering-sensitivities.

The solution—even for a newbie python coder—is easy. You can reorder imports in the view to make sure the LAST imported namespace is the one you want. You can restrict the import to gdata.alt.* and then reference appengine.HttpResponse, etc as needed.

This error text is particular unhelpful and the error path unrelated to the root cause. Hopefully this will match others hitting the same problem.

AttributeError at /l/agpyaWdod...RgCDA/widget.js
'unicode' object has no attribute 'content'
Request Method: GET
Request URL: http://l.../l/agpyaW...CDA/widget.js
Exception Type: AttributeError
Exception Value: 
'unicode' object has no attribute 'content'
Exception Location: /.../gdata/alt/appengine.py in __init__, line 145
Python Executable: /System/Library/Frameworks/Python.framework/Versions/2.5/Resources/Python.app/Contents/MacOS/Python
Python Version: 2.5.1
Python Path: ['/.../rightreply', '/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine', '/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/django', '/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webob', '/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/yaml/lib', '/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine', '/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources', '/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python25.zip', '/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5', '/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/plat-darwin', '/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/plat-mac', '/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/plat-mac/lib-scriptpackages', '/System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python', '/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/lib-tk', '/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/lib-dynload', '/Library/Python/2.5/site-packages', '/System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python/PyObjC']
Server time: Thu, 25 Sep 2008 19:42:39 +0000

Note: This also had variants with the error of "'str' object has no attribute 'content'."

Friday, September 19, 2008

global name '_AppEngineHttpClient__ConvertDataPart'

Ugly! Google's spreadsheet API's had some recent hiccups and I was hoping to keep it as an integral part of a funnel system we use internally. If this bites anyone else, follow the directions and download the GData API Python Client 1.2.1. The previous rev 1.2.0 causes this issue.

Friday, September 12, 2008

[FIXED!] GData 1.2.0 and no module named alt.appengine

Just read Jeff Scudder's comment on my recent UpgradeSessionToken post. He pointed out that there is a newer gdata API which externally appears to be a minor change in the API usage. OK. Here goes!

He references the gdata docs which for my app required these changes (consolidated in a diff-like listing):

-import gdata.spreadsheet.service
+import gdata.service
+import gdata.alt.appengine
-gdata.service.http_request_handler = gdata.urlfetch
-gd_client = gdata.spreadsheet.service.SpreadsheetsService()
+gd_client = gdata.service.GDataService()
+gdata.alt.appengine.run_on_appengine(gd_client)

Great! Now we run the app and...

<class 'django.core.exceptions.ViewDoesNotExist'>: Could not import myapp.admin.views. Error was: No module named alt.appengine
Traceback (most recent call last):
  File "/base/data/home/apps/rightreply/1.73/main.py", line 45, in <module>
    main()
  File "/base/data/home/apps/rightreply/1.73/main.py", line 42, in main
    util.run_wsgi_app(application)
  File "/base/python_lib/versions/1/google/appengine/ext/webapp/util.py", line 76, in run_wsgi_app
    result = application(env, _start_response)
  File "/base/data/home/apps/rightreply/1.73/django/core/handlers/wsgi.py", line 222, in __call__
    response = self.get_response(request)
  File "/base/data/home/apps/rightreply/1.73/django/core/handlers/base.py", line 67, in get_response
    response = middleware_method(request)
  File "/base/data/home/apps/rightreply/1.73/django/middleware/common.py", line 72, in process_request
    urlresolvers.resolve("%s/" % request.path_info)
  File "/base/data/home/apps/rightreply/1.73/django/core/urlresolvers.py", line 299, in resolve
    return get_resolver(urlconf).resolve(path)
  File "/base/data/home/apps/rightreply/1.73/django/core/urlresolvers.py", line 240, in resolve
    sub_match = pattern.resolve(new_path)
  File "/base/data/home/apps/rightreply/1.73/django/core/urlresolvers.py", line 179, in resolve
    return self.callback, args, kwargs
  File "/base/data/home/apps/rightreply/1.73/django/core/urlresolvers.py", line 188, in _get_callback
    raise ViewDoesNotExist, "Could not import %s. Error was: %s" % (mod_name, str(e))
<class 'django.core.exceptions.ViewDoesNotExist'>: Could not import myapp.admin.views. Error was: No module named alt.appengine

Realized the article referenced an updated codebase released two days ago. The code affects the gdata and atom packages/directories. I'm not sure whether I downloaded the client code previously, or if it was created by google's app creation helper, but I had aging copies of the library in my project locally. So, I downloaded the updates and replaced the files:

jonathan$ wget http://gdata-python-client.googlecode.com/files/gdata.py-1.2.0.tar.gz
--2008-09-13 01:51:13--  http://gdata-python-client.googlecode.com/files/gdata.py-1.2.0.tar.gz
Resolving gdata-python-client.googlecode.com... 74.125.47.82
Connecting to gdata-python-client.googlecode.com|74.125.47.82|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 451038 (440K) [application/x-gzip]
Saving to: `gdata.py-1.2.0.tar.gz'

100%[=============================================================================================>] 451,038     92.5K/s   in 4.8s  

2008-09-13 01:51:20 (92.5 KB/s) - `gdata.py-1.2.0.tar.gz' saved [451038/451038]

jonathan$ tar xvzf gdata.py-1.2.0.tar.gz
gdata.py-1.2.0/
gdata.py-1.2.0/pydocs/
gdata.py-1.2.0/pydocs/atom.html
gdata.py-1.2.0/pydocs/atom.http_interface.html
gdata.py-1.2.0/pydocs/atom.mock_http.html
gdata.py-1.2.0/pydocs/atom.mock_service.html
gdata.py-1.2.0/pydocs/atom.service.html
gdata.py-1.2.0/pydocs/atom.token_store.html
gdata.py-1.2.0/pydocs/atom.url.html
gdata.py-1.2.0/pydocs/gdata.alt.appengine.html
gdata.py-1.2.0/pydocs/gdata.apps.html
gdata.py-1.2.0/pydocs/gdata.apps.service.html
gdata.py-1.2.0/pydocs/gdata.auth.html
gdata.py-1.2.0/pydocs/gdata.base.html
gdata.py-1.2.0/pydocs/gdata.base.service.html
gdata.py-1.2.0/pydocs/gdata.blogger.html
gdata.py-1.2.0/pydocs/gdata.blogger.service.html
gdata.py-1.2.0/pydocs/gdata.calendar.html
gdata.py-1.2.0/pydocs/gdata.calendar.service.html
gdata.py-1.2.0/pydocs/gdata.client.html
gdata.py-1.2.0/pydocs/gdata.codesearch.html
gdata.py-1.2.0/pydocs/gdata.codesearch.service.html
gdata.py-1.2.0/pydocs/gdata.contacts.html
gdata.py-1.2.0/pydocs/gdata.contacts.service.html
gdata.py-1.2.0/pydocs/gdata.docs.html
gdata.py-1.2.0/pydocs/gdata.docs.service.html
gdata.py-1.2.0/pydocs/gdata.exif.html
gdata.py-1.2.0/pydocs/gdata.geo.html
gdata.py-1.2.0/pydocs/gdata.html
gdata.py-1.2.0/pydocs/gdata.media.html
gdata.py-1.2.0/pydocs/gdata.photos.html
gdata.py-1.2.0/pydocs/gdata.photos.service.html
gdata.py-1.2.0/pydocs/gdata.service.html
gdata.py-1.2.0/pydocs/gdata.spreadsheet.html
gdata.py-1.2.0/pydocs/gdata.spreadsheet.service.html
gdata.py-1.2.0/pydocs/gdata.spreadsheet.text_db.html
gdata.py-1.2.0/pydocs/gdata.urlfetch.html
gdata.py-1.2.0/pydocs/gdata.youtube.html
gdata.py-1.2.0/pydocs/gdata.youtube.service.html
gdata.py-1.2.0/pydocs/generate_docs
gdata.py-1.2.0/samples/
gdata.py-1.2.0/samples/base/
gdata.py-1.2.0/samples/base/baseQueryExample.py
gdata.py-1.2.0/samples/base/dryRunInsert.py
gdata.py-1.2.0/samples/blogger/
gdata.py-1.2.0/samples/blogger/BloggerExample.py
gdata.py-1.2.0/samples/calendar/
gdata.py-1.2.0/samples/calendar/calendarExample.py
gdata.py-1.2.0/samples/contacts/
gdata.py-1.2.0/samples/contacts/contacts_example.py
gdata.py-1.2.0/samples/docs/
gdata.py-1.2.0/samples/docs/docs_example.py
gdata.py-1.2.0/samples/mashups/
gdata.py-1.2.0/samples/mashups/birthdaySample.py
gdata.py-1.2.0/samples/spreadsheets/
gdata.py-1.2.0/samples/spreadsheets/spreadsheetExample.py
gdata.py-1.2.0/src/
gdata.py-1.2.0/src/atom/
gdata.py-1.2.0/src/atom/__init__.py
gdata.py-1.2.0/src/atom/http.py
gdata.py-1.2.0/src/atom/http_interface.py
gdata.py-1.2.0/src/atom/mock_http.py
gdata.py-1.2.0/src/atom/mock_service.py
gdata.py-1.2.0/src/atom/service.py
gdata.py-1.2.0/src/atom/token_store.py
gdata.py-1.2.0/src/atom/url.py
gdata.py-1.2.0/src/gdata/
gdata.py-1.2.0/src/gdata/alt/
gdata.py-1.2.0/src/gdata/alt/__init__.py
gdata.py-1.2.0/src/gdata/alt/appengine.py
gdata.py-1.2.0/src/gdata/apps/
gdata.py-1.2.0/src/gdata/apps/__init__.py
gdata.py-1.2.0/src/gdata/apps/service.py
gdata.py-1.2.0/src/gdata/base/
gdata.py-1.2.0/src/gdata/base/__init__.py
gdata.py-1.2.0/src/gdata/base/service.py
gdata.py-1.2.0/src/gdata/blogger/
gdata.py-1.2.0/src/gdata/blogger/__init__.py
gdata.py-1.2.0/src/gdata/blogger/service.py
gdata.py-1.2.0/src/gdata/calendar/
gdata.py-1.2.0/src/gdata/calendar/__init__.py
gdata.py-1.2.0/src/gdata/calendar/service.py
gdata.py-1.2.0/src/gdata/codesearch/
gdata.py-1.2.0/src/gdata/codesearch/__init__.py
gdata.py-1.2.0/src/gdata/codesearch/service.py
gdata.py-1.2.0/src/gdata/contacts/
gdata.py-1.2.0/src/gdata/contacts/__init__.py
gdata.py-1.2.0/src/gdata/contacts/service.py
gdata.py-1.2.0/src/gdata/docs/
gdata.py-1.2.0/src/gdata/docs/__init__.py
gdata.py-1.2.0/src/gdata/docs/service.py
gdata.py-1.2.0/src/gdata/exif/
gdata.py-1.2.0/src/gdata/exif/__init__.py
gdata.py-1.2.0/src/gdata/geo/
gdata.py-1.2.0/src/gdata/geo/__init__.py
gdata.py-1.2.0/src/gdata/media/
gdata.py-1.2.0/src/gdata/media/__init__.py
gdata.py-1.2.0/src/gdata/photos/
gdata.py-1.2.0/src/gdata/photos/__init__.py
gdata.py-1.2.0/src/gdata/photos/service.py
gdata.py-1.2.0/src/gdata/spreadsheet/
gdata.py-1.2.0/src/gdata/spreadsheet/__init__.py
gdata.py-1.2.0/src/gdata/spreadsheet/service.py
gdata.py-1.2.0/src/gdata/spreadsheet/text_db.py
gdata.py-1.2.0/src/gdata/youtube/
gdata.py-1.2.0/src/gdata/youtube/__init__.py
gdata.py-1.2.0/src/gdata/youtube/service.py
gdata.py-1.2.0/src/gdata/__init__.py
gdata.py-1.2.0/src/gdata/auth.py
gdata.py-1.2.0/src/gdata/client.py
gdata.py-1.2.0/src/gdata/service.py
gdata.py-1.2.0/src/gdata/test_data.py
gdata.py-1.2.0/src/gdata/urlfetch.py
gdata.py-1.2.0/tests/
gdata.py-1.2.0/tests/atom_tests/
gdata.py-1.2.0/tests/atom_tests/__init__.py
gdata.py-1.2.0/tests/atom_tests/http_interface_test.py
gdata.py-1.2.0/tests/atom_tests/mock_http_test.py
gdata.py-1.2.0/tests/atom_tests/mock_server_test.py
gdata.py-1.2.0/tests/atom_tests/service_test.py
gdata.py-1.2.0/tests/atom_tests/token_store_test.py
gdata.py-1.2.0/tests/atom_tests/url_test.py
gdata.py-1.2.0/tests/gdata_tests/
gdata.py-1.2.0/tests/gdata_tests/apps/
gdata.py-1.2.0/tests/gdata_tests/apps/__init__.py
gdata.py-1.2.0/tests/gdata_tests/apps/service_test.py
gdata.py-1.2.0/tests/gdata_tests/base/
gdata.py-1.2.0/tests/gdata_tests/base/__init__.py
gdata.py-1.2.0/tests/gdata_tests/base/service_test.py
gdata.py-1.2.0/tests/gdata_tests/blogger/
gdata.py-1.2.0/tests/gdata_tests/blogger/__init__.py
gdata.py-1.2.0/tests/gdata_tests/blogger/service_test.py
gdata.py-1.2.0/tests/gdata_tests/calendar/
gdata.py-1.2.0/tests/gdata_tests/calendar/__init__.py
gdata.py-1.2.0/tests/gdata_tests/calendar/calendar_acl_test.py
gdata.py-1.2.0/tests/gdata_tests/calendar/service_test.py
gdata.py-1.2.0/tests/gdata_tests/contacts/
gdata.py-1.2.0/tests/gdata_tests/contacts/__init__.py
gdata.py-1.2.0/tests/gdata_tests/contacts/service_test.py
gdata.py-1.2.0/tests/gdata_tests/docs/
gdata.py-1.2.0/tests/gdata_tests/docs/__init__.py
gdata.py-1.2.0/tests/gdata_tests/docs/service_test.py
gdata.py-1.2.0/tests/gdata_tests/photos/
gdata.py-1.2.0/tests/gdata_tests/photos/__init__.py
gdata.py-1.2.0/tests/gdata_tests/photos/service_test.py
gdata.py-1.2.0/tests/gdata_tests/spreadsheet/
gdata.py-1.2.0/tests/gdata_tests/spreadsheet/__init__.py
gdata.py-1.2.0/tests/gdata_tests/spreadsheet/service_test.py
gdata.py-1.2.0/tests/gdata_tests/spreadsheet/text_db_test.py
gdata.py-1.2.0/tests/gdata_tests/youtube/
gdata.py-1.2.0/tests/gdata_tests/youtube/__init__.py
gdata.py-1.2.0/tests/gdata_tests/youtube/service_test.py
gdata.py-1.2.0/tests/gdata_tests/__init__.py
gdata.py-1.2.0/tests/gdata_tests/apps_test.py
gdata.py-1.2.0/tests/gdata_tests/auth_test.py
gdata.py-1.2.0/tests/gdata_tests/base_test.py
gdata.py-1.2.0/tests/gdata_tests/blogger_test.py
gdata.py-1.2.0/tests/gdata_tests/calendar_test.py
gdata.py-1.2.0/tests/gdata_tests/client_online_test.py
gdata.py-1.2.0/tests/gdata_tests/client_test.py
gdata.py-1.2.0/tests/gdata_tests/codesearch_test.py
gdata.py-1.2.0/tests/gdata_tests/contacts_test.py
gdata.py-1.2.0/tests/gdata_tests/docs_test.py
gdata.py-1.2.0/tests/gdata_tests/photos_test.py
gdata.py-1.2.0/tests/gdata_tests/service_test.py
gdata.py-1.2.0/tests/gdata_tests/spreadsheet_test.py
gdata.py-1.2.0/tests/gdata_tests/youtube_test.py
gdata.py-1.2.0/tests/__init__.py
gdata.py-1.2.0/tests/atom_test.py
gdata.py-1.2.0/tests/gdata_test.py
gdata.py-1.2.0/tests/module_test_runner.py
gdata.py-1.2.0/tests/run_all_tests.py
gdata.py-1.2.0/tests/run_data_tests.py
gdata.py-1.2.0/tests/run_service_tests.py
gdata.py-1.2.0/tests/testimage.jpg
gdata.py-1.2.0/README.txt
gdata.py-1.2.0/RELEASE_NOTES.txt
gdata.py-1.2.0/INSTALL.txt
gdata.py-1.2.0/MANIFEST
gdata.py-1.2.0/setup.py
gdata.py-1.2.0/PKG-INFO

# And finally... replace the local copies with the updated API libs.
jonathan$ cp -r gdata.py-1.2.0/src/gdata gdata.py-1.2.0/src/atom .

Try again and... Resolved! Thanks again for the pointer Jeff

Tuesday, September 09, 2008

Heroku Rocks!

In my recent perusal of hosted application hosters (HAH™) such as Google's AppEngine, I've come across Heroku but had not the chance to try it. Problem no more!

The opportunity arose as we transitioned a large server from our ranks. The machine hosted a free thumnail converter the ELC team wrote a while back. It is now in running well and in maintenance mode. I decided to give Heroku a try as an alternative hosting setup.

The site was in SVN, so the steps roughly were:

# Remove the .svn directories:
find . -name ".svn" -exec rm -rf \{\} \;
# Import into git follow from... http://heroku.com/docs/api/
git init
git add .
git commit -m "init"
heroku create elc-thumbnails-deploy
git remote add heroku git@heroku.com: elc-thumbnails-deploy.git
git push -f heroku

This alone was 95% of the work. The next steps polished things off--added two gems to the deploy that were not in the heroku list... agh! Just searching I realize I missed a beautiful part of the Heroku system--their gems and plugins manager. Anyway--finally, I created a config/heroku.yml file to remove the branding from the application. Along the way I had the pleasure of interacting with the Heroku team. I think they've got great prospects ahead and I look forward to seeing their growth as they come out of beta.

Monday, September 01, 2008

UpgradeToSessionToken Errors (no attribute 'socket') on AppEngine

Following the Google AppEngine AuthSub docs I get this error:

AttributeError at /admin/
'module' object has no attribute 'socket'
Request Method: GET
Request URL: http://www.host.com/admin/
Exception Type: AttributeError
Exception Value: 
'module' object has no attribute 'socket'
Exception Location: /base/python_dist/lib/python2.5/httplib.py in connect, line 1133
Python Executable: /base/
Python Version: 2.5.2
Python Path: ['/base/python_lib/versions/1', '/base/python_dist/lib/python25.zip', '/base/python_dist/lib/python2.5/', '/base/python_dist/lib/python2.5/plat-linux2', '/base/python_dist/lib/python2.5/lib-tk', '/base/python_dist/lib/python2.5/lib-dynload', '/base/data/home/apps/host/1.44/']
Server time: Mon, 1 Sep 2008 09:51:07 +0000

This happens both on the production deploy and on the local host. The answer is not obvious in a google search, but is found here. Looks like gdata relies on httplib which is replaced by urlfetch on AppEngine. The solution looks like:

import gdata.service
import gdata.urlfetch
# Use urlfetch instead of httplib
gdata.service.http_request_handler = gdata.urlfetch

Siegel.ie