On closures and captured variables

A few days ago, on the project I’m working on, I’ve stumbled on an interesting bug – an example of why it pays off to learn the ‘deeper’ areas of C# language (or any other language).
Image copyright: Pavel Shlykov (Shutterstock)

Image copyright: Pavel Shlykov (Shutterstock)


Greatly simplified (and with the class names changed to protect the innocent :) ), we had:
– a structure of orders and order lines/items, something pretty straightforward:

// ...
    public class Order
    {
        public Order()
        {
            Items = new List<OrderItem>();
        }

        public string Number { get; set; }
        // ... other fields

        public IList<OrderItem> Items { get; set; }
    }
// ...
    public class OrderItem
    {
        public int ItemId { get; set; }
        public string ProductName { get; set; }
        public decimal Price { get; set; }
        // ... other fields
    }

An order contains several lines.

For one reason or another, let’s say that we want to deep copy this structure to another class, OrderLineDto, that flattens the structure:

    public class OrderLineDto
    {
        public string OrderNumber { get; set; } // the parent order number
        // OrderItem attributes:
        public int ItemId { get; set; }
        public string ProductName { get; set; }
        public decimal Price { get; set; }
        // ... other fields
    }

Because OrderItem has several hundred properties (don’t ask me why :) ), I’m using AutoMapper to simplify the mapping job.
We added a helper class that it’s supposed to keep the code nice and tidy:

public class OrderMapper
{
    private readonly Order _order;

    public OrderMapper(Order order)
    {
        _order = order;

        AutoMapper.Mapper.CreateMap<OrderItem, OrderLineDto>()
          .ForMember(orderLineDto => orderLineDto.OrderNumber, 
             config => config.MapFrom(sourceOrderItem => _order.Number));
     }

     public OrderLineDto GetLineDto(OrderItem orderItem)
     {
         var dtoLine 
           = AutoMapper.Mapper.Map<OrderItem, OrderLineDto>(orderItem);
         return dtoLine;
     }
}

The constructor gets the current Order instance, defined the mapping, and the GetLineDto method is doing the actual mapping from a OrderItem to a new OrderLineDto. Pretty simple..
Only for OrderLineDto.OrderNumber, we have to tell AutoMapper to take the value from the ‘parent’ _order.Number.

Let’s test it in a console application:

        static void Main()
        {
            var order1 = new Order {Number = "O1"};
            var orderItem1 = new OrderItem
            {
                ItemId = 1,
                ProductName = "Book 1",
                Price = 100
            };
            order1.Items.Add(orderItem1);

            var order2 = new Order {Number = "O2"};
            var orderItem2 = new OrderItem
            {                
                ItemId = 2,
                ProductName = "Book 2",
                Price = 200
            };
            order2.Items.Add(orderItem2);

            /////////////
            var orderMapper1 = new OrderMapper(order1);
            OrderLineDto dto1 = orderMapper1.GetLineDto(orderItem1);

            Console.WriteLine("\n\rItem 1 order number: {0}  == DTO 1 order number: {1}", 
                                order1.Number, dto1.OrderNumber);

            Console.WriteLine("Item 1 prod. name: {0}  == DTO 1 prod name: {1}", 
                                orderItem1.ProductName, dto1.ProductName);

            //////////////
            var orderMapper2 = new OrderMapper(order2);
            OrderLineDto dto2 = orderMapper2.GetLineDto(orderItem2);

            Console.WriteLine("\n\rItem 2 order number: {0}  == DTO 2 order number: {1}", 
                                order2.Number, dto2.OrderNumber);

            Console.WriteLine("Item 2 prod. name: {0}  == DTO 2 prod name: {1}", 
                                orderItem2.ProductName, dto2.ProductName);

            Console.ReadLine();
        }

I create 2 Order objects, each with one OrderItem, and for each OrderItem, I map it to a OrderLineDto object.
Finally, I compare the original and DTO properties to make sure they were copied properly.

However, the result is not the expected one:

Item 1 order number: O1  == DTO 1 order number: O1
Item 1 prod. name: Book 1  == DTO 1 prod name: Book 1

Item 2 order number: O2  == DTO 2 order number: O1
Item 2 prod. name: Book 2  == DTO 2 prod name: Book 2

Obviously, the 2’nd DTO object does not have the right order number (‘O2′), but the first one, ‘O1′.
Is AutoMapper broken? :)

No – the culprit is the was I’m defining the custom mapping for OrderNumber:

public class OrderMapper
{
    private readonly Order _order;

    public OrderMapper(Order order)
    {
        _order = order;

        AutoMapper.Mapper.CreateMap<OrderItem, OrderLineDto>()
          .ForMember(orderLineDto => orderLineDto.OrderNumber, 
             config => config.MapFrom(sourceOrderItem 
                                   => _order.Number));
    }
...
}

sourceOrderItem => _order.Number
is a lambda expression, but because _order field is referenced, a closure is created.
As for any closure, the _order variable instance is captured, and will be available each time the lambda expression is evaluated.

Nothing unexpected so far – the question is: which instance of _order?
The one from the moment OrderMapper constructor is called and the lambda expression is instantiated, right?
:)
That was the intention, at least.
However, AutoMapper has the good habit of caching the mappings in a static field, for good performance reasons.
So even if we try to redefine the mapping for a certain type, the first mapping is used.
In our case, the mapping will allways use the lambda expression created during the first call to Mapper.CreateMap, when the first OrderMapper is instantiated, so the first _order instance is captured by the closure and always used when OrderNumber is mapped.

How to fix this? Quite easy: copy the OrderNumber directly in code and don’t use AutoMapper for such a simple task:

    public class OrderMapper
    {
        private readonly Order _order;

        public OrderMapper(Order order)
        {
            _order = order;

            AutoMapper.Mapper.CreateMap<OrderItem, OrderLineDto>();
            
            //.ForMember(orderLineDto => orderLineDto.OrderNumber, 
            //        config => config.MapFrom(sourceOrderItem => _order.Number));
        }

        public OrderLineDto GetLineDto(OrderItem orderItem)
        {
            var dtoLine = AutoMapper.Mapper.Map<OrderItem, OrderLineDto>(orderItem);
            dtoLine.OrderNumber = _order.Number;
            return dtoLine;
        }
    }

To make sure that such a ‘bug’ is not introduced again by mistake, we can move the call to AutoMapper.Mapper.CreateMap in a static constructor that won’t be able to access instance fields.

More on closures and a comparation with Java: http://csharpindepth.com/articles/chapter5/closures.aspx
or http://martinfowler.com/bliki/Lambda.html
or in JavaScript: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Closures

Posted in .NET, C# | Tagged , , , | Lăsați un comentariu

On assumptions and formats

In .NET (and any other framework for that matter), it’s better to never assume anything, but to check twice.
Let’s take an example – what do you think, will the following unit test always pass?

[TestMethod]
public void ShortDateLength()
{
    DateTime d = new DateTime(2015, 01, 18);
    string dateString = d.ToString("yyyyMMdd");

    Assert.AreEqual(8, dateString.Length);
}

.
.
.
.
.
.
.
.
.
.
.
.
.
.
… well, usually, it will, but once every blue moon, it will fail :) .
All it takes it’s an user changing the regional settings of the computer, or adding the following 3 lines of code:

[TestMethod]
public void ShortDateLength()
{
    CultureInfo c = new CultureInfo("he-IL", false);
    c.DateTimeFormat.Calendar = new HebrewCalendar();
    Thread.CurrentThread.CurrentCulture = c;

    DateTime d = new DateTime(2015, 01, 18);
    string dateString = d.ToString("yyyyMMdd");

    Assert.AreEqual(8, dateString.Length);
}

Yes, on some cultures and calendars, the dates are not represented in arab numerals, and years might not fit in 4 chars.
In the above case, dateString will have the following ‘unexpected’ value:
תשע”הד’כ”ז

– a solid 10 chars in length.
When does this matter? When dateString is going to be displayed on the UI, probably not – in such cases I want to be formatted in the format chosen by the end-user.
However, if the DateTime value is going to be serialized to a text file or send to a web service, I want to make sure that I will be able to decode it later.
In such cases it’s better to replace the line 9 above with:
string dateString = d.ToString(“yyyyMMdd”, CultureInfo.InvariantCulture);

Somehow related – what do you think, which of the following tests will pass?

const string digits1 = "5678";
Assert.IsTrue(Regex.IsMatch(digits1, @"^\d+$"));

const string digits2 = "୮౪୩";
Assert.IsTrue(Regex.IsMatch(digits2, @"^\d+$"));

Unexpectedly for some, both will pass :)
\d in .NET matches any digit, and is Unicode-aware (https://msdn.microsoft.com/en-us/library/20bw873z(v=vs.110).aspx#DigitCharacter).
୮౪୩ are.. digits in some cultures (http://www.fileformat.info/info/unicode/category/Nd/list.htm).

Again, why is this relevant? Because many developers use \d as a quick way to validate input that it’s supposed to be only one of 0,1,2,…,9 – well, digit might mean more than that.
Some will say that in their application there is no risk that an end-user might enter something like ୮౪୩ – true, unless the input comes from a mis-behaving client application that calls a web service, and it just happen to send by accident the following sequence of bytes (hex values):
EF BB BF E0 AD AE
or
E0 B1 AA
or
E0 AD A9
– in UTF-8 these are.. digits.

Posted in .NET, C# | Tagged , , , | Lăsați un comentariu

How developers start to ignore code smells

Many people wonder how some developers blissfully ignore some best practices when writing code, or aren’t too bothered when they see a code smell in their project.
There are many explanations, but an old one is the code they see when working with Microsoft framework and samples (and not only Microsoft).

Even if Microsoft did great improvements in this direction in the recent years (clean code, best practices etc.), when some developers see code like the one below, in one of the most recent Microsoft frameworks, what conclusion will they draw? It’s from MS, so it must be right, no? :)

IdentityConfig.cs – part of the latest ASP.NET Identity 2.0 project template – 6 classes in one file:
IdentityConfig.cs

UserManager class – part of ASP.NET itself, new class added by Microsoft last year – the screenshot is truncated, it could be twice this size – I’m too tired to count how many public members are in there:
UserManager class

Posted in .NET, IT | Tagged , , , , | Lăsați un comentariu

Service Bus for Windows Server: How to define authorization rules at topic level

This is just a ‘reminder’ post for myself (and maybe others) when encountering the same issue.
For Service Bus 1.0 for Windows Server (not Azure), at least on a server not joined to a domain, when using local (workgroup) Windows users for authentication, in order to define the topic authorization rules, the AllowRule.ClaimValue must be the username (without machinename\ in front).
The IssuerName must be the namespace name, and ClaimType must be “nameidentifier”.

An example:

const string userClaim = "nameidentifier";
string userName = "TestUser"; // actual name of the local Windows user
string issuerNamespace = "TestNamespace"; // maybe dynamically obtained using namespaceManager.Address.PathAndQuery.Replace("/", "")
List<AccessRights> accessRights = new List<AccessRights> {AccessRights.Listen};
//...
var topicDescription = namespaceManager.GetTopic("MyTopic");
topicDescription.Authorization.Add(
         new AllowRule(issuerNamespace, userClaim, userName, accessRights));
//...
namespaceManager.UpdateTopic(topicDescription);

Obviously, the local Windows user must exists on both the Service Bus server machine and on the client computer, with the same name and password, and the client application must run using this user.
This type of authentication, Windows integrated using ‘workgroup users’, not joined to a domain, is not quite supported by Microsoft, that assumes that all computers will be joined to a Windows domain, but it works so far.

The MSDN documentation on this issue is not helpfull at all:
http://msdn.microsoft.com/en-us/library/microsoft.servicebus.messaging.authorizationrule.claimvalue.aspx
or http://msdn.microsoft.com/en-us/library/windowsazure/jj193003.aspx
– just auto-generated stuff, with examples taken from Azure Service Bus and usually not updated for Service Bus for Windows Server.

Posted in .NET | Tagged , , , , | Un comentariu

On confusing error messages

This morning I stumbled upon a strange error message, while running a .NET console application:

The application has failed to start because its side-by-side configuration is in
correct. Please see the application event log or use the command-line sxstrace.e
xe tool for more detail.

All went fine until yesterday.. I was ready to start digging using sxstrace and other esoteric tools, learning again about assembly side-by-side loading (http://msdn.microsoft.com/en-us/library/windows/desktop/ff951640(v=vs.85).aspx) or searching for a misterios .manifest file that I was sure does not exists there.
Instead I decided to actually.. read what the error message was telling me: ‘Please see the application event log’ :)

I was not expecting to find anything there, but, in EventLog/Windows/Application log, was a much more helpful message:
Activation context generation failed for “E:\[….]Service.exe”.Error in manifest or policy file “E:\[….]Service.exe.Config” on line 33. Invalid Xml syntax.

Looking in the .config file, the problem was obvious: a missing quote (“) at the end of a config setting.. :)
Fixed that and the error is gone.

The conclusion.. wtf ?

Posted in .NET, C# | Tagged , , , | Lăsați un comentariu

ADO.NET connection resiliency

In a previous post I mentioned a new feature that was going to be added in .NET Framework 4.5.1: ‘ADO.NET connection resiliency’.
Now that .NET 4.5.1 is RTM, I decided to check what exactly is this all about.I wrote a very simple console application, that uses ‘raw’ ADO.NET to connect to a database and repeatedly performs a simple query:

        static void Main(string[] args)
        {
            string connectionString = ConfigurationManager.ConnectionStrings["TestDB"].ConnectionString;

            string queryString = "SELECT Id, FirstName, LastName FROM dbo.Persons WHERE FirstName = @firstName";
            string firstName = string.Empty;

            while (firstName.ToLowerInvariant() != "q")
            {
                Console.Write("First name: ");
                firstName = Console.ReadLine();

                using (SqlConnection connection = new SqlConnection(connectionString))
                {
                    SqlCommand command = new SqlCommand(queryString, connection);
                    command.Parameters.AddWithValue("@firstName", firstName);

                    try
                    {
                        connection.Open();
                        SqlDataReader reader = command.ExecuteReader();
                        while (reader.Read())
                        {
                            Console.WriteLine("\t{0}\t{1}\t{2}", reader[0], reader[1], reader[2]);
                        }
                        reader.Close();
                    }
                    catch (Exception ex)
                    {
                        Console.WriteLine(ex.Message);
                    }
                }
            }
        }

All we do is to read a name, open a ADO.NET connection, execute a command, read the results, repeat.

Let’s build and run this code on a previous version of .NET Framework, let’s say 3.5:
Target NET 3.5
All works fine:
Run 3.5 Success

until somebody or something kills the connection from the ADO.NET connection pools that was still opened and connected to SQL Server:
CloseConnectionTcpView

If we try to execute a command on the database, ADO.NET uses the same connection from the pool, without knowing that it was closed by an external party:
Run 3.5 Error

This is not a permanent error and if we try again a new connection is created:

Run 3.5 Success

Let’s try to target .NET Framework 4.5.1 (under Visual Studio 2013):
Target .NET 4.5.1

Now, even if we kill the TCP/IP connection to the database server, no exception is triggered:
Run 4.5.1 Success

What happens under the covers? I didn’t found too much on MSDN, but most likely, ADO.NET catches the error and re-opens the connection.
All I found is a short info on a .NET 4.5.1 announcement post (http://blogs.msdn.com/b/dotnet/archive/2013/10/17/net-framework-4-5-1-rtm-gt-start-coding.aspx):
‘Under the covers, this new feature provides a robust connectivity system for recreating broken connections and re-trying transactions’

Unfortunately, it seems that this feature can be used only with SQL Azure or SQL Server 2014 (CTP2) (source).
This is somehow expected, because it’s a feature of the MS ADO.NET Provider for SQL Server, not ADO.NET in general.
Other third-party ADO.NET providers seem to have a similar feature for some time, like DataDirect ADO.NET providers:
http://www.datadirect.com/resources/resource-library/adonet-developer-center/adonet-tutorials/failover-support-in-datadirect-adodotnet-data-providers/connection-retry

It seems that by default, ADO.NET will retry only once, but this can be configured (http://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqlconnectionstringbuilder.connectretrycount(v=vs.110).aspx).

The same goes for the interval between retries (10s by default): http://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqlconnectionstringbuilder.connectretryinterval(v=vs.110).aspx

Posted in .NET, C# | Tagged , , , | Lăsați un comentariu

Connection resiliency in Entity Framework 6

The EF team has recently announced the new features added to EF 6 beta 1, and among them is one I found interesting: connection resiliency.
Despite the fancy name, it’s a simple concept: having built-in support for automatically retrying an EF operation, if the cause of the failure is a transient one, that has a high probablity to be “fixed” in a short interval of time.

While such “transient failures” are quite rare when using a single database server, located near the client (ex.: web application), on the same high-speed LAN, they are expected when the database server is located in a huge server farm, like Azure or Amazon ones. Expected not only once every blue moon, but quite often as servers are automatically relocated or connections are throttled when the load is high.
It’s true that such “retry” functionality it’s not so hard to build manually in our application, but it’s a repetitive process that I would rather have already available when needed.

As expected, currently this feature is available only on the EF provider for SQL Azure, and only in EF 6 beta 1 or newer. Since anyway I am on the bleeding edge on this case, I used the nightly EF6 build, taken with NuGet from MyGet feed (https://entityframework.codeplex.com/wikipage?title=Nightly%20Builds) instead of the official one.

Let’s write some code in a console application:

    class Program
    {
        static void Main(string[] args)
        {
            Database.SetInitializer<AdventureWorksContext>(null);

            Console.WriteLine("Persons starting with T: ...");
            using (var context = new AdventureWorksContext())
            {
                context.Database.Log = Console.Write;

                var personsStartingWithT = from p in context.Persons
                                           where p.FirstName.StartsWith("T")
                                           select p;

                foreach (Person person in personsStartingWithT)
                {
                    Console.WriteLine("{0} {1} {2}", person.Suffix, person.FirstName, person.LastName);
                }

                Console.WriteLine("\n\rPress any key to continue..");
                Console.ReadLine();

            }

        }
    }

So far nothing new.
I used the AdventureWorks2012 database, on a SQL Azure instance, so the connection string looks like this:

  <connectionStrings>
    <add name="AdventureWorksContext" providerName="System.Data.SqlClient" 
connectionString="Server=tcp:[.....].database.windows.net,1433;Database=AdventureWorks2012;User ID=tudor@[.......];Password=[.......];Trusted_Connection=False;Encrypt=True;Connection Timeout=30;" />
  </connectionStrings>

All we need to do in order to enable the connection resiliency for all actions is to add the SqlAzureExecutionStrategy to our custom DbConfiguration class, that is automatically discovered by EF at startup:

namespace TestConnectionResiliency.Models
{
    public class SqlAzureDbConfiguration : DbConfiguration
    {
        public SqlAzureDbConfiguration()
        {
            AddExecutionStrategy(() => new SqlAzureExecutionStrategy());
        }
    }
}

To test if it works, since it’s a bit hard to wait for a transient failure to happen on SQL Azure, I just disabled the network connection on the client just before the query was about to execute, and I enabled it back a few seconds later. The log on the debug console say it all:

A first chance exception of type 'System.Data.SqlClient.SqlException' occurred in System.Data.dll
A first chance exception of type 'System.Data.SqlClient.SqlException' occurred in System.Data.dll
A first chance exception of type 'System.Data.SqlClient.SqlException' occurred in System.Data.dll
A first chance exception of type 'System.Data.SqlClient.SqlException' occurred in System.Data.dll
A first chance exception of type 'System.Data.SqlClient.SqlException' occurred in System.Data.dll
A first chance exception of type 'System.Data.SqlClient.SqlException' occurred in System.Data.dll
A first chance exception of type 'System.Data.SqlClient.SqlException' occurred in System.Data.dll
A first chance exception of type 'System.Data.SqlClient.SqlException' occurred in System.Data.dll
A first chance exception of type 'System.Data.SqlClient.SqlException' occurred in System.Data.dll
A first chance exception of type 'System.Data.SqlClient.SqlException' occurred in System.Data.dll
'TestConnectionResiliency.vshost.exe' (Managed (v4.0.30319)): Loaded 'EntityFrameworkDynamicProxies-TestConnectionResiliency'
...
The thread '<No Name>' (0x2020) has exited with code 0 (0x0).
The program '[8832] TestConnectionResiliency.vshost.exe: Managed (v4.0.30319)' has exited with code 0 (0x0).

The timeout exception was thrown by ADO.NET, but EF caught it and retried the operation until it was successful.

What exactly is the EF provider for SQL Azure doing in this case?
Easy to find out – we have the source code for DbExecutionStrategy.cs and SqlAzureRetriableExceptionDetector.cs.

If a SqlException with a certain error code (full details in SqlAzureRetriableExceptionDetector.cs), or a TimeoutException, is thrown by the underlying provider, the operation is retried no more than 5 times until is successful, using exponentially increasing (random) delays between retries.
The formula used is: MIN(random(1, 1.1) * (2 ^ retryCount - 1), maxDelay)
where retryCount is by default 5, and maxDelay is by default 30s.
In other words, it retries slower and slower each time, but no slower than 30 seconds.
This is an exponential backoff algorithm, that is clasically used in such cases.
The list of SQl error codes that trigger a retry is similar with the ones from http://windowsazurecat.com/2010/10/best-practices-for-handling-transient-conditions-in-sql-azure-client-applications/ .

Some context: even since Azure was launched, various teams within Microsoft have tried to implement something similar:
– the ‘Transient Fault Handling Framework for SQL Azure, Windows Azure Storage, Service Bus & Cache’ (http://windowsazurecat.com/2011/02/transient-fault-handling-framework/) from the former Azure CAT (Customer Advisory Team)
– followed by Enterprise Library Transient Fault Handling Application Block (http://msdn.microsoft.com/en-us/library/hh680934(v=pandp.50).aspx) included in Enterprise Library 5.0 Integration Pack for Windows Azure by P&P team
– now EF 6 will includ this built-in: https://entityframework.codeplex.com/wikipage?title=Connection%20Resiliency%20Spec
– the Windows ODBC Driver for SQL Server/Azure offers something similar: ‘Connection Resiliency in the Windows ODBC Driver
– and in .NET 4.5.1 preview, it might seem that something similar will be included inside ADO.NET itself, as ‘ADO.NET idle connection resiliency’ (I found no details about this one yet)

Azure container

Source: Microsoft: http://news.cnet.com/2300-10805_3-10001679-1.html (Chicago Data Center)

Posted in .NET, Azure, Entity Framework, SQL Server | Tagged , , , | Un comentariu