Optimistic concurrency control in ASP.NET WebAPI

[warning: long post 🙂 ]

Setting the stage

The first question, and an expected one would be: what has to do ASP.NET Web API with concurrency control? Until now, in most applications, this problem was approached at persistence layer level or even on database server level: when implementing optimistic concurrency control, if during an update we detect that the data was modified in the meanwhile (since it was read), probably by another user or system, usually a specific exception was thrown (or a certain error code was returned) and the client application was supposed to treat this case one way or another – usually by displaying a dialog to the user that will allow to choose a way to solve the conflict, as appropriate: by cancelling his own changes, overwriting the changes done by the other user or by merging the changes, if it’s possible and depending on the application requirements.

I won’t go into details about the “theory” of optimistic concurrency control, when it’s used and when not, how it’s implemented etc. – it’s known stuff. What is less known is that HTTP (as an application-level protocol), among many other features, comes with a built-in feature that can be used (among other things) for making the optimistic concurrency control mechanism explicit: ETags. Usually used for caching in HTTP, ETags have another possible use, suggested even by the definition from the standard: “Entity tags are used for comparing two or more entities from the same requested resource. … An entity tag MUST be unique across all versions of all entities associated with a particular resource.”

Even if the terminology is a bit different, we realize that one resource (or document) could be associated to one (or more) records from a database, the ‘entities’ can be regarded as in-memory object instances (loaded from a database entity) – all this as an approximate equivalence.

Also as an analogy, an eTag could be considered (or implemented) as a ‘version’, ‘timestamp’ associated to a database record, in general an opaque value, without a meaning for the user. We realize that the ETag, apart from it’s usage for caching, it’s an ideal candidate as a standard way to transport the values used for concurrency control. The method is not something unheard of, being suggested even by those who worked at the standard long time ago (http://www.w3.org/1999/04/Editing/) and it’s used by various document/content management systems (where it appears under the name of “unreserved checkout”).

What have all of these to do with ASP.NET WebAPI? They have, because Web API tries to provide support for HTTP-based services, where HTTP is made explicit as an application-level protocol, instead of being hidden/abstracted away.

Back on earth..
Enough theory for now. How can be an ETag used in a practical way to offer support for optimistic concurrency control, at HTTP level?
Some pseudocode:

• a client application makes a request for a certain resource: GET /products/23
• if the resource does exists, the server will send back a response that will contain, apart from the resource itself, the ETag value for that resource, as a field in the HTTP header (ex.: ETag: “s2hk707Mvk+GqzxNe+lbOQ==”)
• the client application displays the resource, the user (1) modifies it (on the client)
• during this time, another user (2) (or another system) can edit and modify the resource without problems. On the server, the ETag is updated when the resource is persisted
• the first user (1) finishes it’s changes and press a ‘Save’ button
• the client application sends a PUT request to the server, with the modified resource and the original ETag, as a field in the HTTP header, like:
  If-Match: “<original ETag value>”
• on the server, if we can verify (one way or another) that the resource was modified in the meanwhile (usually by comparing the original and current etags), the update won’t be performed and a response will be returned with 412 (Precondition Failed) code, where ‘precondition’ is the “If-Match” condition from the change request (some services return 409 Conflict, so be prepared fro that)
• it’s up to the client application to decide what to do with this response, like: display to the user a message that let him to choose between different ways to solve the conflict (“owerwrite the changes done by other users”, “cancel my changes” etc.), optionally by showing a list of changes done by the other user

How does ASP.NET WebAPI helps us to implement something like this?
Easy: the actions from an ApiController offer an easy access to the HTTP headers – the rest is up to us. I let the code speak by itself 🙂

(if you read this a few years from now: the code is tested using .NET Framework 4.5 beta – it might be changed until the final release, as it happened in the past)

    public class ProductsController : ApiController
    {
        private static IList _products = new List(); // dummy product repository

        static ProductsController()
        {
            // dummy data
            _products.Add(new Product() { Id = 10, Code = "P1", Description = "Product 1", Price = 123.45m,
                                          Version = Guid.NewGuid().ToByteArray() });
            _products.Add(new Product() { Id = 11, Code = "P2", Description = "Product 2", Price = 567.47m,
                                          Version = Guid.NewGuid().ToByteArray() });
            _products.Add(new Product() { Id = 12, Code = "P3", Description = "Product 3", Price = 100.22m,
                                          Version = Guid.NewGuid().ToByteArray() });
        }

        // GET /api/products
        public IEnumerable Get()
        {
            return _products;
        }

        // GET /api/products/11
        public HttpResponseMessage Get(int id)
        {
            Product prod = (from p in _products
                         where p.Id == id
                         select p).FirstOrDefault();
            if (prod == null)
            {
                throw new HttpResponseException(HttpStatusCode.NotFound);
            }
            else
            {
                var response = new HttpResponseMessage(prod, HttpStatusCode.OK);
                response.Headers.ETag = new System.Net.Http.Headers.EntityTagHeaderValue("\""
                    + Convert.ToBase64String(prod.Version, Base64FormattingOptions.None) + "\"");
                return response;
            }
        }

        // PUT /api/values/5
        public void Put(int id, Product product)
        {
            // retrive the existing product from persitence
            Product existingProduct = (from p in _products
                            where p.Id == id
                            select p).FirstOrDefault();

            if (existingProduct != null)
            {
                // perform concurrency conflict check
                CheckIfProductWasModified(this.Request.Headers, existingProduct);

                //update the product
                existingProduct.Code = product.Code;
                existingProduct.Description = product.Description;
                existingProduct.Price = product.Price;
                // this should be done by the persistence layer (DB, etc..)
                existingProduct.Version = Guid.NewGuid().ToByteArray();

            }
            else
            {
                throw new HttpResponseException(HttpStatusCode.NotFound);
            }
        }

        private void CheckIfProductWasModified(HttpRequestHeaders requestHeaders, Product existingProduct)
        {
            if (requestHeaders.IfMatch != null)
            {
                // if the request contains an If-Match haeder with a non-empty ETag
                EntityTagHeaderValue firstHeaderVal = requestHeaders.IfMatch.FirstOrDefault();
                if ((firstHeaderVal != null) && (!string.IsNullOrEmpty(firstHeaderVal.Tag))
                    && (firstHeaderVal.Tag != "*")
                    )
                {
                    // compare the old and new ETag value (this can be done at DB-level using a WHERE clause)
                    string encodedNewTagValue = firstHeaderVal.Tag.Trim("\"".ToCharArray());
                    string encodeExistingTagValue = Convert.ToBase64String(existingProduct.Version, Base64FormattingOptions.None);
                    if (!encodedNewTagValue.Equals(encodeExistingTagValue, StringComparison.Ordinal))
                    {
                        // concurrency conflict if the resource was modified
                        throw new HttpResponseException(HttpStatusCode.PreconditionFailed);
                    }
                }
            }
        }

        // POST /api/products
        public HttpResponseMessage Post(Product product)
        {
            _products.Add(product);
            var response = new HttpResponseMessage(product, HttpStatusCode.Created);
            response.Headers.Location = new Uri(Request.RequestUri,
                "/api/products/" + product.Id.ToString(CultureInfo.InvariantCulture));
            return response;
        }

    }

The highlighted lines show the relevant parts:
– at GET, we set the ETag for the returned resource, by using HttpRespnseMessage.Headers.ETag (the value must be enclosed in double quotes (“)

– at PUT (modify), if this.Request.Headers.IfMatch has a value (that it’s not “*”), we compare it with the ETag of the persisted resource (in DB or cache)
– if they don’t match, we return a response with HttpStatusCode.PreconditionFailed

Finally, some tests that show what we expect to happen:

    [TestClass]
    public class ProductsControllerIntegrationTests
    {
        private HttpClient _client;
        private string _productUri = "http://ipv4.fiddler/TestETag/api/products/11";
        private readonly string jsonMediaType = "application/json";

        [TestInitialize]
        public void Init()
        {
            _client = new HttpClient();
        }

        [TestMethod]
        public async Task Get_ReturnsTheETagInTheHeaders()
        {
            HttpResponseMessage httpResp = await _client.GetAsync(_productUri);

            Assert.IsNotNull(httpResp.Headers.ETag, "No ETag was included in the response headers!");
            Assert.IsTrue(!string.IsNullOrEmpty(httpResp.Headers.ETag.Tag), "An empty ETag was received!");

        }

        [TestMethod]
        public async Task Put_IfTheProductWasModified_ReturnsPreconditionFailed()
        {
            // first user reads the product 11
            HttpResponseMessage getProdUser1 = await _client.GetAsync(_productUri);
            getProdUser1.EnsureSuccessStatusCode();
            Product prodUser1 = await getProdUser1.Content.ReadAsAsync();

            ///////////////
            // second user reads the product 11
            HttpResponseMessage getProdUser2 = await _client.GetAsync(_productUri);
            getProdUser2.EnsureSuccessStatusCode();
            Product prodUser2 = await getProdUser2.Content.ReadAsAsync();

            //second users modifies (PUT) and persist the product
            prodUser2.Description = prodUser2.Description + " modif by user 2";

            var putRequest2 = GetPutRequestMessage(prodUser2);
            // we have to use SendAsync in order to be able to set any header, including If-Match
            HttpResponseMessage putUser2Response = await _client.SendAsync(putRequest2);
            putUser2Response.EnsureSuccessStatusCode();

            ///////////////
            // first user modifies (PUT) and tries to persist the same product
            prodUser1.Description = prodUser2.Description + " modif by user 1";

            var putRequest1 = GetPutRequestMessage(prodUser1);
            HttpResponseMessage putUser1Response = await _client.SendAsync(putRequest1);

            /////////////////////
            Assert.AreEqual(HttpStatusCode.PreconditionFailed, putUser1Response.StatusCode,
                            "The response was not precondition failed!");

        }

        private HttpRequestMessage GetPutRequestMessage(Product product)
        {
            var requestMessage = new HttpRequestMessage(HttpMethod.Put, _productUri);
            // add an 'If-Match' header
            requestMessage.Headers.IfMatch.ParseAdd("\"" + Convert.ToBase64String(product.Version) + "\"");
            // add the modified object serialized as JSON
            ObjectContent prodObjContent =
                requestMessage.CreateContent(
                product,
                MediaTypeHeaderValue.Parse(jsonMediaType),
                new MediaTypeFormatter[] { new JsonMediaTypeFormatter() },
                new FormatterSelector());

            return requestMessage;
        }

        [TestCleanup]
        public void Cleanup()
        {
            if (_client != null)
            {
                _client.Dispose();
            }
        }

    }

Obviously, it doesn’t matter the language or framework used on the client – I used HttpClient from ASP.NET Web API because it was at hand, but I could use JavaScript, C++ or something else.

For a real application, of course, there are a few more steps to be done: a lot of refactoring for having some reusable code, extracting the code dealing with concurrency checks in a separate class – separation of concerns (probably using a DelagatingChanell, like it’s described at http://javiercrespoalvez.com/2011/06/etags-and-optimistic-concurrency.html or http://codebetter.com/howarddierking/2011/07/01/automatic-etag-management-with-web-api-message-handlers/), using a real persistence solution (like a database). Obviously, the above tests are only some ‘integration’ tests – some real unit tests would be useful.

What is not relevant in the above code:
– the format used for ETag – I used byte[] only because it’s easier to map to rowversion columns from MS SQL Server, when using EF, but equally good would be an int or GUID as long as make sure it’s unique
– the way in which the ETag is encodded in the HTTP header: I used Convert.ToBase64String only because it’s a convenient and safe way to encode an array of bytes (in both directions)
– the format used to serialize the resource (entity) – it can be JSON as above, but also XML or something else

What is not discussed in this post: the role played by ETag in HTTP caching, that is important, if used.

A bit of context:
In those cases when our REST-like service is simple enough to be exposed using the OData protocol proposed by Microsoft, the protocol will use ETags for concurrency support: http://www.odata.org/documentation/operations#ConcurrencycontrolandETags
Also the Microsoft framework that uses the OData protocol, like WCF Data Services (former ADO.NET Data Services, codename Astoria) is using HTTP Etags for concurrency, so when we can use that, it’s already baked for us (http://msdn.microsoft.com/en-us/data/hh127792 ; http://blogs.msdn.com/b/astoriateam/archive/2008/04/22/optimistic-concurrency-data-services.aspx).
Even if it might not be obvious, the OData protocol (based on HTTP) can be found in many places: Sharepoint 2010 services, Excel services, Azure Storage.

Outside Microsoft world, Raven DB API also is using ETags for concurrency: http://ravendb.net/docs/http-api/http-api-comcurrency, and some GData (Google Data) services do the same: https://developers.google.com/gdata/docs/2.0/reference#ResourceVersioning

Later edit: a short follow up: part 2