OptionalFieldAttribute not needed with BinaryFormatter in .net v2.

[asanford]

The MSDN documentation, along with various articles online, such as this one

http://msdn.microsoft.com/msdnmag/issues/04/10/AdvancedSerialization/

claim that the BinaryFormatter will throw an exception if you try to
deserialize a stream that has missing members in it, and that the new
OptionalFieldAttribute allows you to avoid this problem, by marking the new
members as optional. Well, it seems that even if you don't specify this
attribute, no exceptions are thrown, no matter if you add members, remove
members, or change the assembly version. I also did tests with both signed
and unsigned assemblies, and it made no difference. Is there some other
global setting that could affect the behavior? I thought maybe the
BinaryFormatter.AssemblyFormat might be set to simple, but the doc says the
default is for Full, not Simple (I'm doing all my tests with the caching
application block's isolated storage backing, which I don't think changes the
AssemblyFormat from the default.) I also looked into the files containing
the serialized data, and I can see the assembly name, version number, public
key, etc., so I'm fairly sure AssemblyFormat is set to Full.

Any help?

BTW, I also found the following post indicating someone else already found
this problem:

http://groups.google.com/group/microsoft.public.dotnet.framework/browse_thread/thread/465a8713be2f153/173410a79ee7de1a?lnk=st&q=OptionalFieldAttribute+BinaryFormatter&rnum=1&hl=en#173410a79ee7de1a

Thanks!

[Jeffrey Tan[MSFT]]

Hi Asanford2000,

Thanks for your post!

I will try to write a sample project to reproduce this problem. For
efficiency's sake, is it convinient for you to provide a full code snippet
or a sample project for us to reproduce? This will help us identify the
problem more efficiently. Anyway, I will update you once I have reproduced
the problem. Thanks.

Best regards,
Jeffrey Tan
Microsoft Online Community Support
==================================================
When responding to posts, please "Reply to Group" via your newsreader so
that others may learn and benefit from your issue.
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.

[Jeffrey Tan[MSFT]]

Hi,

Sorry for letting you wait.

After performing some modification to the MSDN article project I can
reproduce the behavior: the OptionalFieldAttribute is not needed for
version tolerance.

Actually, this is caused by BinaryFormatter.AssemblyFormat property as you
suspected.

In .Net 1.1, we can set the "BinaryFormatter.AssemblyFormat" to
"FormatterAssemblyStyle.Simple" to allow more version tolerance. In this
way, no exception will be thrown if a particular field is not found from
the stored stream.
In .Net 2.0, the property is by default "Simple", not "Full"(You can
verify this by watching BinaryFormatter.AssemblyFormat in debugger), so
even we do not specify "OptionalFieldAttribute", no exception is thrown.
"OptionalFieldAttribute" is needed only when "FormatterAssemblyStyle.Full"
needs to work with missing fields.

Hope this helps.

Best regards,
Jeffrey Tan
Microsoft Online Community Support
==================================================
When responding to posts, please "Reply to Group" via your newsreader so
that others may learn and benefit from your issue.
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.

[asanford]

Thanks; I'm surprised the AssemblyFormat is set to simple, as the first
article I referenced from MSDN says "...The default value of AssemblyFormat
is FormatterAssemblyStyle.Full..." and the article discusses v1 and v2 of
the framework. I guess they're talking about v1 in that sentence?

Anyhow, I now have two problems:

1) If a serialize non-Dictionary classes, then setting AssemblyFormat to
simple seems to cause the OptionalFieldAttribute to not be needed, AND seems
to make the deserializer not care if the assembly version changes. However,
if I serialize dictionary classes, then the deserializer requires the
assembly version to match, even if I set AssemblyFormat to simple. Is there
a way to get the rules to apply consistently?

2) If I set AssemblyFormat=full, then I get exceptions if my assembly
versions don't match, regardless of the OptionalFieldAttribute. The only way
I can find to avoid this exception in that case is to set the formatter's
Binder or SurrogateSelectior properties. Is there another way to do this,
perhaps some attribute I can attach to my class, rather than having to
interact with the formatter? The reason I ask is that I'm using the
formatter indirectly, via the Caching Application Block v2.0 (with isolated
storage persistence.)

Thanks,
-Andy

Post by Jeffrey Tan[MSFT]
Hi,
Sorry for letting you wait.
After performing some modification to the MSDN article project I can
reproduce the behavior: the OptionalFieldAttribute is not needed for
version tolerance.
Actually, this is caused by BinaryFormatter.AssemblyFormat property as you
suspected.
In .Net 1.1, we can set the "BinaryFormatter.AssemblyFormat" to
"FormatterAssemblyStyle.Simple" to allow more version tolerance. In this
way, no exception will be thrown if a particular field is not found from
the stored stream.
In .Net 2.0, the property is by default "Simple", not "Full"(You can
verify this by watching BinaryFormatter.AssemblyFormat in debugger), so
even we do not specify "OptionalFieldAttribute", no exception is thrown.
"OptionalFieldAttribute" is needed only when "FormatterAssemblyStyle.Full"
needs to work with missing fields.
Hope this helps.
Best regards,
Jeffrey Tan
Microsoft Online Community Support
==================================================
When responding to posts, please "Reply to Group" via your newsreader so
that others may learn and benefit from your issue.
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.

[Jeffrey Tan[MSFT]]

Hi Andy,

Thanks for your feedback!

Post by asanford
I guess they're talking about v1 in that sentence?

Yes, I assume he is talking about .Net1.1. Also, since this article is
written in October 2004, which .Net2.0 is still in beta version, if the
author is talking about .Net2.0, I assume that this behavior has changed in
RTM.

Post by asanford
1) Is there a way to get the rules to apply consistently?

Sorry, I am not sure I understand you completely. Can you provided more
information regarding the "dictionary classes", which classes in FCL do you
refer to? Is it possible for you to provide a little sample project for us
to reproduce this problem? Thanks

Post by asanford
2) Is there another way to do this,
perhaps some attribute I can attach to my class, rather than having to
interact with the formatter?

I have performed some research in this issue and found the key point lies
in the System.Runtime.Serialization.Formatters.Binary.ObjectReader.Bind
method and
System.Runtime.Serialization.Formatters.Binary.ObjectReader.FastBindToType
method.(From Reflector), below is the code snippet:

internal Type Bind(string assemblyString, string typeString)
{
Type type1 = null;
if ((this.m_binder != null) && !this.IsInternalType(assemblyString,
typeString))
{
type1 = this.m_binder.BindToType(assemblyString, typeString);
}
if (type1 == null)
{
type1 = this.FastBindToType(assemblyString, typeString);
}
return type1;
}

internal Type FastBindToType(string assemblyName, string typeName)
{
....
if (this.bSimpleAssembly)
{
try
{
ObjectReader.sfileIOPermission.Assert();
try
{
StackCrawlMark mark1 =
StackCrawlMark.LookForMe;
assembly2 =
Assembly.LoadWithPartialNameInternal(assemblyName, null, ref mark1);
if ((assembly2 == null) && (assemblyName !=
null))
{
assembly2 =
Assembly.LoadWithPartialNameInternal(new AssemblyName(assemblyName).Name,
null, ref mark1);
}
}
finally
{
CodeAccessPermission.RevertAssert();
}
}
catch (Exception)
{
}
}
else
{
try
{
ObjectReader.sfileIOPermission.Assert();
try
{
assembly2 = Assembly.Load(assemblyName);
}
finally
{
CodeAccessPermission.RevertAssert();
}
}
catch (Exception)
{
}
}
....
}

As we can see, FastBindToType checks bSimpleAssembly field(which maps to
FormatterAssemblyStyle.Full/Simple) to use Assembly.Load or
Assembly.LoadWithPartialNameInternal. From Applied .net Framwork written by
"Jeffrey Richter", we know that Assembly.Load will check the version
explicitly, while Assembly.LoadWithPartialNameInternal will ignore the
assembly version.

Also, in Bind method, we can see that Binder.BindToType method gives us a
hook over the FastBindToType method, which may help us to jmp over
FastBindToType method.

OptionalFieldAttribute is used in
System.Runtime.Serialization.Formatters.Binary.ReadObjectInfo.GetMemberTypes
method:

internal Type[] GetMemberTypes(string[] inMemberNames, Type objectType)
{
......
if (!flag2)
{
object[] objArray1 =
this.cache.memberInfos[num2].GetCustomAttributes(typeof(OptionalFieldAttribu
te), false);
if (((objArray1 == null) || (objArray1.Length ==
0)) && !this.bSimpleAssembly)
{
throw new
SerializationException(string.Format(CultureInfo.CurrentCulture,
Environment.GetResourceString("Serialization_MissingMember"), new object[]
{ this.cache.memberNames[num2], objectType,
typeof(OptionalFieldAttribute).FullName }));
}
}
}
}
return typeArray1;
}

So FCL uses System.Reflection.MemberInfo.GetCustomAttributes method to
access the OptionalFieldAttribute. If you want to determine if there is any
additionally attribute can meet your need without changing the formatter
itself, you may set a breakpoint in
System.Reflection.MemberInfo.GetCustomAttributes method in debugger and
monitor if Deserialize method will trigger other breakpoint that is not
looking for OptionalFieldAttribute. Based on my research, I can not find a
such attribute. So you may still have to touch the formatter itself to hook
the process.

Hope this helps. If you have any further concern or need any other help,
please feel free to tell me, thanks!

Best regards,
Jeffrey Tan
Microsoft Online Community Support
==================================================
When responding to posts, please "Reply to Group" via your newsreader so
that others may learn and benefit from your issue.
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.

[asanford]

This post might be inappropriate. Click to display it.

[Jeffrey Tan[MSFT]]

Hi Andy,

Thanks for your feedback!

Yes, with your sample project, I can reproduce out this strange behavior.

I have performed some deeper debugging regarding this issue. I found that
the FileNotFoundException is generated by Assembly.GetType, which is
finally invoked by
System.Runtime.Serialization.FormatterServices.GetTypeFromAssembly.

By setting breakpoint at
System.Runtime.Serialization.FormatterServices.GetTypeFromAssembly and
performing some tracing, I found that this method is invoked several times.
Each time for a type, so it will loop through the deserialization object
and call FormatterServices.GetTypeFromAssembly for each child type in this
object.

Also, the exception is generated when using
FormatterServices.GetTypeFromAssembly for a strange type name:
"System.Collections.Generic.KeyValuePair`2[[System.String, mscorlib,
Version=2.0.0.0, Culture=neutral,
PublicKeyToken=b77a5c561934e089],[DemoDictionarySerializeIssue.Item,
DemoDictionarySerializeIssue, Version=1.0.0.4, Culture=neutral,
PublicKeyToken=3379bf4a1aba6400]]"

Since the currently debugging assembly version is 1.0.0.5, while
Assembly.GetType is trying to retrieve Version=1.0.0.4, it will attempt to
locate assembly with strong name below:
"DemoDictionarySerializeIssue, Version=1.0.0.4, Culture=neutral,
PublicKeyToken=3379bf4a1aba6400"

However, this assembly is changed to 1.0.0.5, so the loading fails.

So this issue seems to be with Generic of .Net2.0. I am not sure if this
issue is a bug or not. I will try to contact our CLR team regarding it.
Thanks.

Additionally, I found that without strong name to sign the assembly, it
will eliminate this issue, do you have any special request to strong name
the assembly?

Regarding the caching application block issue, I will also try to consult
internally and hope there is some workaround for it. I will get back to you
ASAP. Thanks.

Best regards,
Jeffrey Tan
Microsoft Online Community Support
==================================================
When responding to posts, please "Reply to Group" via your newsreader so
that others may learn and benefit from your issue.
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.

[Jeffrey Tan[MSFT]]

Hi Andy,

Have you managed to create a sample project for demonstration? Does my
reply make sense to you? If you have anything unclear or any concern,
please feel free to let me known, thanks!

Best regards,
Jeffrey Tan
Microsoft Online Community Support
==================================================
When responding to posts, please "Reply to Group" via your newsreader so
that others may learn and benefit from your issue.
==================================================
This posting is provided "AS IS" with no warranties, and confers no rights.