JSON schema validation with Django Rest Framework

Django Rest Framework integrates well with models to provide out of the box validation, and ModelSerializers allow futher fine-grained custom validation. However, if you’re not using a model as the resource of your endpoint then the code required for custom validation of complex data structure can get hairy.

If there is a heavily nested data structure then a serializer can be used that has a nested serializer, which also has a nested serializer, and so on – or a JSON schema and custom JSON parser can be employed.

Using a JSON schema generation tool also provides a quick win: generate the canonical “pattern” of valid JSON structure using data known to be of the correct structure. This post will go through doing this JSON schema validation using Django Rest Framework.

Note my folder structure is like so:

apps/
    products/
        api/
            parsers.py
            negotiators.py
            schemas.py
            urls.py
            views.py

Usecase

If you find yourself in the following situations then this approach should come in useful:

  • Storing data from external service when you don’t have control over schema and don’t want to replicate it in a database.
  • Data not related to a specific resource.
  • Endpoint for saving data serialized from taffy.
  • Need to store data in flat file instead of database due to a technical constraint.

JSON Schema

We will validate the JSON posted to out endpoint against a JSON schema we define. The JSON schema standard is not yet finalized, but in a mature enough for our usecase. This example uses the jsonschema python package, and the following data:

Note for sake of briefity the example data structure below is simple, but just pretend its complex. If the data structure was as simple as bellow a serializer should be used.

# schemas.py
json = {
    "name": "Product",
    "properties": {
        "name": {
            "type": "string",
            "required": True
        },
        "price": {
            "type": "number",
            "minimum": 0,
            "required": True
        },
        "tags": {
            "type": "array",
            "items": {"type": "string"}
        },
        "stock": {
            "type": "object",
            "properties": {
                "warehouse": {"type": "number"},
                "retail": {"type": "number"}
            }
        }
    }
}

# The JSON Schema above can be used to test the validity of the JSON code below:
example_data = {
    "name": "Foo",
    "price": 123,
    "tags": ["Bar", "Eek"],
    "stock": {
        "warehouse": 300,
        "retail": 20
    }
}

For an easy look at validation in practice take a look here

Endpoint

Now to the impliment the DRF endpoint that uses JSON schema validation:

# views.py
from rest_framework.exceptions import ParseError
from rest_framework.response import Response
from rest_framework import views

from . import negotiators, parsers, utils


class ProductView(views.APIView):

    parser_classes = (parsers.JSONSchemaParser,)
    content_negotiation_class = negotiators.IgnoreClientContentNegotiation

    def post(self, request, *args, **kwargs):
        try:
            # implicitly calls parser_classes
            request.DATA
        except ParseError as error:
            return Response(
                'Invalid JSON - {0}'.format(error.message),
                status=status.HTTP_400_BAD_REQUEST
            )
        utils.store_the_json(request.DATA)
        return Response()
# parsers.py
import jsonschema
from rest_framework.exceptions import ParseError
from rest_framework.parsers import JSONParser

from . import schemas


class JSONSchemaParser(JSONParser):

    def parse(self, stream, media_type=None, parser_context=None):
        data = super(JSONSchemaParser, self).parse(stream, media_type,
                                                   parser_context)
        try:
            jsonschema.validate(data, schemas.json)
        except ValueError as error:
            raise ParseError(detail=error.message)
        else:
            return data
# urls.py
from django.conf.urls.defaults import url
from django.conf.urls import patterns

from . import views


urlpatterns = patterns(
    '',
    url(r'^/api/product/$', views.ProductView.as_view(), name='product_view'),
)

Content negotation

The `parse` method on each parser in `parser_classes` will get called only if the request’s “Content-Type” header has a value that matches the ‘media_type’ attribute on the parser, which means the JSON schema validation will not go ahead if no “Content-Type” header is set. If the schema validation must go ahead, I see a few options:

    • Assign `parser_classes = (PlainSchemaParser, JSONSchemaParser, XMLSchemaParser, YAMLSchemaParser, etc)` on ProductView and define the YAML, XML, etc schemas.
    • Force the view to use JSONSchemaParser parser regardless of if the client requests JSON, XML, etc.

To keep it simple this example will choose the second option by using custom content negotiation (which is pulled directly from DRF docs:

# negotiators.py

from rest_framework.negotiation import BaseContentNegotiation

class IgnoreClientContentNegotiation(BaseContentNegotiation):
    def select_parser(self, request, parsers):
        """
        Select the first parser in the `.parser_classes` list.
        """
        return parsers[0]

    def select_renderer(self, request, renderers, format_suffix):
        """
        Select the first renderer in the `.renderer_classes` list.
        """
        return (renderers[0], renderers[0].media_type)

Posting to the endpoint

lets define a simple webservice:

function createProduct(data){
    $.post('/api/product' data).
        done(function(resp){
            console.log('OK');
        }).
        fail(function(resp){
            var error = JSON.parse(xhr.responseText);
            console.log('error - ' + error.detail);
        });
}

createProduct({
    "name": "Foo",
    "price": 123,
    "tags": ["Bar", "Eek"],
    "stock": {
        "warehouse": 300,
        "retail": 20
    }
});
// OK

createProduct({
    "price": 123,
    "tags": ["Bar", "Eek"],
    "stock": {
        "warehouse": 300,
        "retail": 20
    }
});
// error - name property is required

createProduct({
    "price": 123,
    "tags": ["Bar", "Eek"],
    "stock": {
        "warehouse": 300,
        "retail": 20
    }
});
// error - price Number is less then the required minimum value

Maintenance

Word of caution: I find serializers much more maintainable than JSON schemas, so if you are 50/50 of whether to use JSON schema validation or a serializer then I suggest going for the serializer.

Advertisements

Authenticate using Django Rest Framework and Angular

In a previous post we went through how to authenticated using a DRF endpoint.

This post will expand on how to achieve persistent login with a one page app using Angular. Note as there are 2 frameworks in play there will be some need to workaround integrations problems. It doesn’t get messy, which is a testiment to the quality of the respective frameworks!

For a demo check out this a silly microblog tool I made where you can create an account and log into it: github

Note my folder strucure is like so:

    project/
        urls.py
        settings.py
        static/
            js/
                app.js
                ...[all third party js files]...
        apps/
            core/
                views.py
                urls.py       
            accounts/
                urls.py
                api/
                    views.py
                    serializers.py
                    authentication.py

Create the Angular module

In app.js we create a module that will call the DRF endpoints in order to authenticate:

angular.module('authApp', ['ngResource']).
    config(['$httpProvider', function($httpProvider){
        // django and angular both support csrf tokens. This tells
        // angular which cookie to add to what header.
        $httpProvider.defaults.xsrfCookieName = 'csrftoken';
        $httpProvider.defaults.xsrfHeaderName = 'X-CSRFToken';
    }]).
    factory('api', function($resource){
        function add_auth_header(data, headersGetter){
            // as per HTTP authentication spec [1], credentials must be
            // encoded in base64. Lets use window.btoa [2]
            var headers = headersGetter();
            headers['Authorization'] = ('Basic ' + btoa(data.username +
                                        ':' + data.password));
        }
        // defining the endpoints. Note we escape url trailing dashes: Angular
        // strips unescaped trailing slashes. Problem as Django redirects urls
        // not ending in slashes to url that ends in slash for SEO reasons, unless
        // we tell Django not to [3]. This is a problem as the POST data cannot
        // be sent with the redirect. So we want Angular to not strip the slashes!
        return {
            auth: $resource('/api/auth\\/', {}, {
                login: {method: 'POST', transformRequest: add_auth_header},
                logout: {method: 'DELETE'}
            }),
            users: $resource('/api/users\\/', {}, {
                create: {method: 'POST'}
            })
        };
    }).
    controller('authController', function($scope, api) {
        // Angular does not detect auto-fill or auto-complete. If the browser
        // autofills "username", Angular will be unaware of this and think
        // the $scope.username is blank. To workaround this we use the
        // autofill-event polyfill [4][5]
        $('#id_auth_form input').checkAndTriggerAutoFillEvent();

        $scope.getCredentials = function(){
            return {username: $scope.username, password: $scope.password};
        };

        $scope.login = function(){
            api.auth.login($scope.getCredentials()).
                $promise.
                    then(function(data){
                        // on good username and password
                        $scope.user = data.username;
                    }).
                    catch(function(data){
                        // on incorrect username and password
                        alert(data.data.detail);
                    });
        };

        $scope.logout = function(){
            api.auth.logout(function(){
                $scope.user = undefined;
            });
        };
        $scope.register = function($event){
            // prevent login form from firing
            $event.preventDefault();
            // create user and immediatly login on success
            api.users.create($scope.getCredentials()).
                $promise.
                    then($scope.login).
                    catch(function(data){
                        alert(data.data.username);
                    });
            };
    });

// [1] https://tools.ietf.org/html/rfc2617
// [2] https://developer.mozilla.org/en-US/docs/Web/API/Window.btoa
// [3] https://docs.djangoproject.com/en/dev/ref/settings/#append-slash
// [4] https://github.com/tbosch/autofill-event
// [5] http://remysharp.com/2010/10/08/what-is-a-polyfill/

Create the Angular view

Django and Angular both use curly braces to denote variables. Below we want Django to take care of defining the user (‘{{user.username}}’) based on the value passed in the Context to the template, while Angular should take care of everything else. We use verbatim template tag to prevent Django from stealing Angular’s thunder. In one_page_app.html we do:

<html ng-cloak ng-app>
<head>
	<link rel="stylesheet" href="/static/js/angular/angular-csp.css"/>
	<link rel="stylesheet" href="/static/js/bootstrap/dist/css/bootstrap.min.css"/>
</head>
<body>
<div class="navbar navbar-fixed-bottom" ng-controller="authController"
ng-init="user='{{user.username}}'">
{% verbatim %}
<div ng-show="user" class="navbar-form navbar-right">
<input type="submit" class="btn btn-default" ng-click="logout()"
value="logout {{user}}"/>
</div>
<div ng-hide="user">
<form id="id_auth_form" class="navbar-form navbar-right"
ng-submit="login()">
<div class="form-group">
<input ng-model="username" required name="username"
type="text" placeholder="username" class="form-control">
</div>
<div class="form-group">
<input ng-model="password" required name="password" type="password"
placeholder="password" class="form-control">
</div>
<div class="btn-group">
<input type="submit" class="btn btn-default" value="login">
<input type="submit" class="btn btn-default" value="register"
ng-click="register($event)">
</div>
</form>
</div>
</div>
</body>
<script src="/static/js/jquery/jquery.js"></script>
<script src="/static/js/bootstrap/dist/js/bootstrap.min.js"></script>
<script src="/static/js/angular/angular.min.js"></script>
<script src="/static/js/angular-resource/angular-resource.min.js"></script>
<script src="/static/js/autofill-event/src/autofill-event.js"></script>
<script src="/static/js/app.js"></script>
</html>
{% endverbatim %}

Requirements

Bower is fantastic tool to manage front-end requirements. Lets define bower.json like so:

{
  "dependencies": {
    "angular": "1.3.0-build.2419+sha.fe0e434",
    "angular-resource": "1.3.0-build.2419+sha.fe0e434",
    "autofill-event": "1.0.0",
    "jquery": "1.8.2",
    "bootstrap": "3.0.0",
  }
}

DRF Endpoint

We also need to update the apps.accounts.api.AuthView created in previous post: we need to call django.contrib.auth.login in order to attach the session id the request object, and cause the creation a Set-Cookie header on the respons, which causes the browser to create the defined cookie and so we will achieve persistent login upon reloading the page:

from django.contrib.auth import login, logout

class AuthView(APIView):
    authentication_classes = (QuietBasicAuthentication,)

    def post(self, request, *args, **kwargs):
        login(request, request.user)
        return Response(UserSerializer(request.user).data)

    def delete(self, request, *args, **kwargs):
        logout(request)
        return Response({})

django.contrib.auth.logout stops the session, and sends instructions to browser to make the cookie expire.

Serving the view

We also of course need to serve the one_page_app.html, so in core.views.py we do

from django.views.generic.base import TemplateView

class OnePageAppView(TemplateView):
    template_name = 'one_page_app.html'

And in core.urls we route a url to the Django view:

from django.conf.urls import patterns, include, url

from . import views

urlpatterns = patterns('',
    url(r'^$', views.OnePageAppView.as_view(), name='home'),
)

With this all plugged in we have a one page app that:

– Shows login form if user isn’t authenticated with their Django session cookie.
– Shows logout if user is authenticated by either session cookie or username and password.
– Allows logging in in a RESTful way.
– Allows creating new user and automatically logs after creation.
– Alerts login and registration errors.
– Took only about 150 lines of code to achieve.

Check credentials using Django Rest Framework

This post will cover how to authenticate a user’s username and password using a Django Rest Framework endpoint. This functionality allows checking credentials without the need for refreshing the browser.

This is a follow up to my previous post, which covered how to create a User model DRF endpoint (allowing creating, listing, deleting users).

Create the endpoint

note my folder strucure is like so:

apps/
    accounts/
        urls.py
        api/
            views.py
            serializers.py
            authentication.py

In views.py we need to create the AuthView which we will call to check credentials:

from . import authentication, serializers  # see previous post[1] for user serializer.

class AuthView(APIView):
    authentication_classes = (authentication.QuietBasicAuthentication,)
    serializer_class = serializers.UserSerializer

    def post(self, request, *args, **kwargs):
        return Response(self.serializer_class(request.user).data)

[1] https://richardtier.com/2014/02/25/django-rest-framework-user-endpoint/

Then in authenticaiton.py we define our authenticator: We inherit DRF’s BasticAuthentication to check the HTTP_AUTHORIZATION header for correct username and password.

from rest_framework.authentication import BasicAuthentication

class QuietBasicAuthentication(BasicAuthentication):
    # disclaimer: once the user is logged in, this should NOT be used as a
    # substitute for SessionAuthentication, which uses the django session cookie,
    # rather it can check credentials before a session cookie has been granted.
    def authenticate_header(self, request):
        return 'xBasic realm="%s"' % self.www_authenticate_realm

Notice we’re also overriding BasicAuthentication’s authenticate_header method to prevent undesirable behaviour: by default with Basic authentication if user provides wrong credentials the browser prompts the user for their credentials again using a native dialogue box. Rather ugly and bad user experience. To avoid this we ensure the schema returns a custom value other than ‘Basic‘.

We of course need to define the url in urls.py:

from django.conf.urls import patterns, url

from api import views as api_views

urlpatterns = patterns(
    '',
    url(r'^api/auth/$',
        api_views.AuthView.as_view(),
        name='authenticate')
)

Using the endpoint

Now we can do some fun stuff – attempt to authenticate using the endpoint. We need to set the header of the ajax request. We can do this by hooking in with $.ajax’s beforeSend:


function checkCredentials(username, password){
    function setHeader(xhr) {
        // as per HTTP authentication spec [2], credentials must be
        // encoded in base64. Lets use window.btoa [3]
        xhr.setRequestHeader ("Authorization", "Basic " +
                               btoa(username + ':' password));
    }

    $.ajax({type: "POST",  url: "/api/auth/",  beforeSend: setHeader}).
        fail(function(resp){
            console.log('bad credentials.')
        }).
        done(function(resp){
            console.log('welcome ' + resp.email)
        });

[2] https://tools.ietf.org/html/rfc2617
[3] https://developer.mozilla.org/en-US/docs/Web/API/Window.btoa

and to finally use it:


// pass in bad credentials...
checkCredentials('AzureDiamond', 'password');
// ...prints 'bad credentials.'

// pass in good credentials...
checkCredentials('AzureDiamond', 'hunter2');
// ...prints 'welcome AzureDiamond@bash.org'

Security considerations

Only serve this endpoint over HTTPS, in the same way you should serve a conventional login page using HTTPS. We dont wan’t a man in the middle attack getting our auth header!

In a future post we cover how to use this functionality to have persistent login.