Business Rules vs. Sync workflows in CRM

With the introduction of the “Entity” scope in Business Rules, we can now configure them to execute on the server instead of a form (client-side) logic. However, this brings a dilemma on whether to use business rules or synchronous workflows for server level validations and business logic implementation.

Business rules that are defined on the scope of a form opens the door to defining form business logic that is configurable and requires no coding (something that previously would require JavaScript). Although there are many limitations to business rules, they enable simple business logic to be implemented by a configurator rather than a developer. However, in many scenarios the data in CRM is not created/updated via a CRM form. For example, bulk data import, workflows, plugins, API/SDK calls would bypass the CRM forms and therefore would also bypass any JavaScript or form Business Rule validations you might have configured. These type of server-side validation had typically required plugins or synchronous workflows.

Luckily, business rules now have a new scope “Entity” which applies at the server level instead of the form level. What this means is that these business rules, just like plugins and sync workflows, will execute even if you are not using a CRM form in your transaction because they happen on the platform (server side) instead of the form (browser/client side). There are certainly a large number of limitations on what business rules can do and there is still a large number of scenarios for which you must write code (plugins) to implement more complex rules. However, I sometimes get asked the question: If I can implement my server-side business logic as a business rule or a sync workflow, which one should I use?

Although I would love to hear from the broader community what is your position on the question above, I will provide my own personal view. If the question is asked, then I assume whatever you need to implement can be done with either business rules or sync workflows (what are the limitations of each is a separate topic not covered in this post). And in that case my answer is almost always to use business rules if you can. This is why:

1.       Business Rules are contained in the entity itself in the solution which I find somewhat cleaner and more organized, you can review which business rules apply to each entity and when exporting a solution (CRM 2016+) decide which business rules to export. You can argue processes have a primary entity and you can use that field for the same “organization” purpose (sure I give you that).
2.       Business Rules are actually implemented as sync workflows! The business rule is nothing more than an abstraction of a sync workflow but behind the scenes each business rule is converted to a sync workflow. However, I find the abstraction useful, mostly because I can interpret business rules a lot easier in my head than workflows. If you export a solution with a business rule you will notice the business rules are actually “processes” just like sync workflows so there is a conversion layer from entity business rules to sync workflows. What this means is that there are basically 2 separate designers (process designer and business rule designer) that produce the same workflow XAML definition that is executed by the same runtime. While both workflows and business rules are actually stored in the “Process” table/entity there seems to be a slight difference in the triggers: The workflow triggers are stored in the same Process table but the business rule triggers have their own “ProcessTrigger” (anyway that’s all implementation details).
3.       Workflow designer is quite outdated and probably reaching its end of life. However, it seems Microsoft is investing more and more in the business rules designer and supporting more and more scenarios with each CRM release. I personally find it more pleasant to work with the business rule designer than the old process designer.

On the other hand, sync workflows can be more useful if you want to at some point enable additional capabilities not supported by business rules. For example, user-level scope of a workflow (as opposed to Organization) and some of the other settings available in the process designer which are not available in the business rules designer.

In summary, sync workflows (and by extension process designer) have a lot more capabilities than business rules (and business rule designer). But if you have some business logic which can be implemented with either, then would suggest using business rules in that case instead of synchronous workflows.

OptionSet vs Lookup for implementing lists in CRM

In most CRM implementations we experience the dilemma of custom entity vs. OptionSet for implementing lists. This post seeks to point some of the considerations to keep in mind when making this decision.

Imagine you want to capture the "Industry" of your leads/accounts as this information might be valuable for reporting and BI. Every organization has different definitions of "industries" because there is no global standard list of industries that is useful for everyone. So you decide to create and maintain your own list of industries. Now the problem becomes: Should you implement this list as an "Industry" entity or should you simply create an OptionSet with the list of industries?
As everything in the technology consulting industry (pun intended), the answer is always: It depends! There is no exact formula of science to solve this dilemma but I've had numerous discussions with colleagues and customers in this respect and I thought of gathering some guidance that might help you decide:
 

1. How often does this list change and who maintains it?

If your list of industries is changing often or if you expect business users to contribute by maintaining this list, it is recommended you implement as a custom entity rather than option set. The reason is that the custom entity adds richer flexibility to create, update or remove values without having to involve IT. You can easily control who can edit the list by leveraging the security roles on the custom entity. However, if your list rarely changes (e.g. list of countries) and it needs to go through IT then you should probably use Option Set.

2. Do you need to support multiple languages?

For us in Canada we usually always implement CRM in both English and French. If your industry list is defined as an OptionSet you can support languages much easier because it is just a matter of translating labels so that each user will always see the list in their preferred language. However, if you use a custom entity, the list is data (instead of metadata) and data does not have the notion of language. Things get complicated here. The typical solution is to use a concatenated primary fields in which a workflow automatically appends English and French labels together with a separator. For example you have an industry record called "Travel / Voyage". But this gets too complex if you have more than 2 languages and creates unnecessary long labels. Not to mention language laws dictating which language should appear first can cause a headache. For this aspect, the winner is by far Option Sets.

3. Do you need to capture additional fields?

Imagine in our example, you also need to capture an industry code and size along with the industry name. It would be almost impossible to achieve this with Option Sets so you would definitely need to implement your industry list as a custom entity. Even if today you only need the name, consider whether it is likely that in the future you have to capture additional attributes and if so, you should favor using a custom entity over Option Sets.

4. Environment transport and synchronization

The nice thing about option sets is that when you transport a solution you are also transporting all the labels and values to your target environment and you are sure that the list is consistent in your dev/test/prod environments so you can rely on specific values existing in the list. However, if you implement as an entity, you will never know who will create, update or delete values from one environment and not the other. So you will end up with different value in each environment. Furthermore, you might end up with equal values but different Ids. For example, you might have industry "automobile" in dev with a specific GUID but the same industry in production with a different GUID. This takes me to the next point about:

5. Workflows and reports/views

You might want to do industry specific reports or workflows and include logic such as "if this customer is in the banking industry then do X". Using OptionSets this is safe because the value "Banking Industry" is consistent across all environments so you can add such condition to your workflow. However, if you use a custom entity, you cannot rely on values having the same GUID in all environments so you cannot define such logic safely. The workaround would be that in your workflow/view/report you say "if industry.name contains 'banking' then do X". This works in most cases but is not completely safe; for example, somebody might have renamed "banking" to "finance" and suddenly your workflow is broken. The safest is to use an OptionSet which you are sure will exist for a specific value. If you rely on reports and workflows such as the one described above you might be better off using Option Sets.

6. Dependent lists

Consider whether your list will have another dependent list. For example list of states/provinces depends on the value selected from the list of countries. You can achieve dependent lists as both OptionSets and custom entities. The only difference is that for custom entities it is easier because you don't need to write any code, simply configure the form to show only states/provinces related to the country selected (assuming there is a relationship between state/province and country). I think this is the least important factor but is a small advantage of custom entities as it is easier to filter the list based on another related value. If you define your list as a custom entity you can even do advanced filtering such as displaying only industries which exist in the country of the current lead (assuming N:N relationship between country and industry). That can be much harder to do via OptionSet (custom development) while it is just a matter of configuring the form in the case of custom entity.


I usually go in the order above to make a decision as the first items have more impact than the last items of the list above. I hope this can help you assess the best solution for your situation, let me know if you have other considerations I missed!

Agile CRM development with SCRUM

I recently worked in an agile CRM project as the scrum master and was able to learn some good tips of using scrum for CRM development.

I am of the belief that the waterfall days are coming to an end, mostly due to its unsustainable release cadence, inefficiency when reacting to the 'unknown' and the large amount of waste that goes towards tight controls and project management as opposed to focusing effort on product quality and continuous business value generation. In the world of CRM implementations this becomes even more evident because of various factors such as the ability of the platform to quickly produce business value and adapt in an iterative manner, even when the final outcome is yet to be discovered. CRM can be configured very quickly which allows you to achieve valuable work (increment) in iterations (sprints) that can be as short as 1-2 weeks. The fact that you can have stakeholders provide early feedback continuously (sprint reviews) allows you to adapt very quickly to changing demands and gives your users a chance to get familiar with CRM early on. Continuous feedback leads to more valuable user feedback because users have something tangible to review instead of having to imagine what everything will look like. If you are reading this you probably already understand the value of agile and I don't need to convince you further. Now let's look at the typical challenges we find with agile frameworks like scrum and some ideas from my experience that can make your life easier.

1. Your customer needs to fund the project and to do so they need the cost, scope and schedule upfront. This has been the hardest challenge, specially in the consulting world as organizations are used to think that to manage risk means to transfer to someone else (you) the risk of changes in scope, cost and schedule. This is an antiquated mentality when it comes to complex software development because you cannot plan for everything in advance and expect that the requirements you document today is exactly what you are going to need. The result is typically that consulting companies add a large amount of contingency and charge you more for any change in your original requirements (change requests). The end result is that your customer ends up paying more and not getting the optimal solution. What is worse, is that they will not know that the solution is not really doing what they need until it is too late, too much has been wasted and the budget cannot be increased. This is by far the main reason I see waterfall projects fail. But what can you do about it if your customer's culture is waterfall and they want a fixed-everything upfront before they can "fund the project"? I think the magic word here is 'trust'. The best you can do (other than finding another customer that understands that we are not in the 90s) is to earn their trust. Certainly this does not happen overnight but if you are able to build this trust you might eventually be able to propose 'giving it a try' and educating your customer on modern software development practices. The good thing about agile is that you see the results very quickly and you can start by funding a single sprint to see how things go (the bad part is that the first sprint is always the hardest!). One important tip is to make sure your customer trusts you enough to not think that you are just trying to make more money with them, this will not go well and is very far from your objective. In reality your objective is to save them money and increase their probability of success, they need to see that intention clearly. Regarding funding, you can help them switch from the 'fund a project' mentality to the mentality of 'fund a team that is going to give you optimal business value for your project(s)'.

2. You are not sure how to manage the CRM development and test environments for an agile project. This is one where you might have to try a few things and see what works best with your development team. My suggestion is to start with something as simple as possible and empower your development team to decide if they want to make changes to this during sprint retrospectives. The simplest is to have one (yes, one!) shared environment (CRM organization/tenant) for development and test. I have blogged in the past here about how to manage multiple dev and test CRM environments and some good practices about that, but I invalidate my own post when it comes to agile. You want to reduce waste and not have to spend time migrating code, merging solutions and fixing environment synchronization issues. This is why you should at least try to have everyone working on the same environment. You will certainly need some rules such as one developer cannot block the environment for debugging while everybody else needs it; but this is why the daily scrum exists. Make sure you have a daily scrum and that it is actually a 15 min maximum daily scrum and not some status report or defect logging meeting, do not invite anybody outside of your development team to the scrum meeting, this time should be wisely used so developers can synchronize and coordinate activities to be the most efficient. Developers can always have their own sandbox environments but these are not development environments as you should not promote code from sandbox to development.

Regarding test environment I initially had a separate test environment but the development team quickly realized that it was causing too much overhead with very little value. Instead, try to get everyone to test during the sprint (add testing to your definition of done). A very hard thing to do is to bring acceptance testing as part of the sprint and not an ad-hoc activity after the sprint. Get your product owner engaged to do the necessary acceptance testing prior to sprint review, and even consider incorporating UAT as part of your sprint. This is not easy but will give you increased agility and faster release cycles. If you succeed bringing acceptance testing as part of your sprint then it is more clear why you would not need a separate test environment.

3. You don't know how long your sprints should be and how much PBIs you should forecast for a given sprint. I think this is one of the places where CRM has an advantage over custom development, you can do a lot in very little time. Therefore, I would suggest to start with shorter sprints (maybe 2 weeks is a good start). Never do a sprint over 4 weeks or you might end up just doing waterfall! At first you will most likely underestimate and not complete all the PBIs by the end of the first sprint. This is not ideal but is acceptable as long as you don't "extend" the sprint (sprints are time boxed, never extend the end date of a sprint, no matter what). The reality is that at first you might struggle with sprint planning, this is normal, try to forecast how much you can complete in your first sprint by keeping in mind your definition of done (make sure you have one!). So think about how much time it will take you not just to code/configure but also to design, analyze, unit test, document, etc.) each PBI. Additionally you need to work with the product owner to select the most urgent PBIs while keeping a balance for the skills required to deliver the sprint. For example, if you have a dev team of 6 but only 1 person knows SSRS, then you cannot realistically include only SSRS reports in the sprint, even if those reports are the most urgent PBIs. You need to make sure your team skill set is proportionate to the skill set required to deliver the sprint otherwise you might end up with 1-2 very busy individuals while others have a hard time finding how to contribute. Additionally when you forecast your sprint make sure you reserve a 10% of total sprint time to do backlog refinement (most teams fail to do this). There are multiple estimation techniques such as planning poker, however, when forecasting your sprint always ask your team at the end to do a 'gut check' (are we confident to be able to deliver these PBIs within next sprint?). I have found gut checks to be very useful and accurate, everyone must keep in mind the time spent in meetings, backlog refinement, sprint reviews and retrospectives, etc. because all these activities are also part of your sprint, not just the development activities. So in reality you spend only about 80% of your sprint capacity developing PBIs (note that development includes design, code, unit test, etc). One more thing about sprints, try to keep the same duration of sprints and the same start and end day of the week. Eliminate any time between sprints (yes this can be hard). Sprint n+1 should start the day after sprint n finishes, this will give you greater agility, as opposed to having "ventilation" time between sprints to do activities that should actually have been done during the sprint.

4. You are uncertain about how you can incorporate UAT and deployment into the scrum process. This is one of those places where reality sometimes crashes with scrum. If you can get your customer to perform UAT during the sprint and you make that work then you are probably a scrum super ninja that is not learning anything new from this post. So that's the ideal scenario, but we know for various reasons this does not always work. In this case you will need to do as much acceptance testing with the product owner during the sprint as possible and then release to UAT as part of your definition of done. The problem is that your team will probably get interruptions for UAT support as the UAT testers are usually in a completely different cycle as your sprint cycles. You will have to deal with this and account for this during sprint planning, but again the more you can test earlier the better because you will get less UAT defects and distractions. In this case you might need a separate environment for UAT and you would need to forecast in your sprint planning the activities for UAT release. Regarding deployment to production, I have often heard the misunderstanding that in scrum you are supposed to deploy to production at the end of every sprint, that's not correct. You can release to production as often as you want, from multiple times per sprint to once very number of sprints. However, at the end of each sprint, your work should be "potentially releasable" according to the scrum guide, so you shouldn't have broken critical functionality at the end of the sprint, even if you don't plan to release to production. It is a good idea to release as often as you can, try to incorporate the DevOps approach and bring development and operations together as much as possible, this will help you achieve a faster release cycle and a smooth transition to production. Of course this is not something that you will get right from one day to the next, DevOps requires multiple changes and great amount of practice and experience to become efficient (beyond the scope of this post).

 

Scrum is an empirical process, so at its core it has transparency, inspection and adaptation. Make sure that you pay attention to these 3 values during your various events, the key is not how to successfully setup a scrum project but more about your agility to inspect and adapt your process to produce business value faster and reducing waste, that is your success criteria! There are certainly many other challenges and suggestions, I might blog more about this in the future, but I would conclude with one famous quote from scrum that couldn't be more accurate: scrum is a framework that is easy to understand but very hard to master.

Scalability considerations for CRM / SharePoint integration

SharePoint is probably the most natural and easiest to implement solution for document management for CRM records. However, what happens when you have millions of records in CRM which might have documents?

I recently worked on fairly large document migration solution to be able to associate documents to CRM records (e.g. cases, opportunities). There were a few hundreds of thousands of files amounting to about 50GB of data. So here are a few things to consider



1. FOLDER HIERARCHY



With the out of the box integration you have the choice to structure your folders based on accounts/contacts or simply to create a new library for each entity at the root of the site. It might be more convenient to navigate SharePoint if you can start from an account and then find which are the related cases/opportunities rather than starting with a global list of cases and try to find which cases are related to a specific account. In general, this structure is more convenient, until you consider security segregations.

If you need to segregate SharePoint security in such a way that some users can access documents of a given entity type (e.g. cases) but not be able to access documents of another entity type (e.g. opportunities) then the account/contact hierarchy becomes a problem because you cannot set the security at the library level (all documents would be on the same library) and you would have to do some acrobatics with folder security and inheritance breaking which would be a nightmare to maintain. Instead, if you have each entity type have its own SharePoint library then you can easily grant/deny access to documents of a given entity type.

Nonetheless, remember that with the out of the box CRM/SharePoint integration, there is no security synchronization between CRM and SharePoint. You need to keep this in mind throughout your design. Think for example that a user might not be able to access any opportunities in CRM but the user (if malicious) can always find a way to see the documents associated to CRM opportunities by going to SharePoint (unless you block the user completely from the given SharePoint library).

Also consider that changing the folder hierarchy after would be a major data migration effort so you should really think about what hierarchy makes more sense you’re your situation and consider current or future security requirements.

2. NUMBER OF ITEMS PER FOLDER / LIBRARY

Imagine that you have 100K cases in your system and you have enabled SharePoint integration for case entity. And perhaps every case has at least one document. In this case, your case library/folder will contain 100K items flat on the same list. This goes well beyond what SharePoint recommendations are for scalability and performance. It is not recommended to go beyond 5000 items (even that is already quite high). Of course you can always implement some sort of archiving or use multiple SharePoint sites depending on some criteria so you split this load. Another [reasonably] easy solution is to further structure your folders by year, quarter or month (or all of these). This way you will not end up with 100K folders under the “cases” folder. Instead, the maximum number of folders you will have under a single folder will be the maximum amount of cases that can be opened in a given month/quarter which might be a more reasonable number.

In the example above, we have 2 new layers: Year and Month. These correspond to the date on which the case was opened. By adding these additional layers, we can now guarantee that there will be no folder with more than a few hundred sub-folders, since we know that we only open a few hundred cases per month.

The downside is that this cannot be done by simple configuration or out of the box integration. This structure would require that you register a plugin on create of case, which will create the document locations and the SharePoint folder for the case being created. Some other disadvantages of this approach is that:

- Every case will now have a folder, even if the case has no documents. This should not be a problem if you know that anyway all cases have documents

- You are creating the SharePoint folder at the same time as you create the case, instead of the OOB behavior which is to create the folder on demand the first time document library is accessed in CRM. This is not necessarily a bad thing though.

- If your plugin is sync and it fails (e.g. SharePoint is down) then it will prevent the creation of the case in CRM. If your plugin is async and it fails, then you need to have a way to recover and create the correct folder when the user tries to access document library for this record.





3. SIZE AND GROWTH OF YOUR SHAREPOINT SITE

For sites with large volume of documents, large document sizes or rapidly growing volumes, you might also need to consider how long you have until you start having a performance or limitation problem with things like the max size of your content database, max number of files in the library or simply the maximum size of your site. O365 also has some limitations that you need to review. If you identify that this could be a potential problem you should consider implementing some sort of archival solution which will allow you to keep a link between CRM records and SharePoint files while at the same time optimizing for current and mostly used records. I have heard of people simply changing the CRM site every now and then and creating a new site once the old site is getting too large. I guess any of these strategies would work as long as you have defined a process to scale your site and monitor the volumes regularly. The most important thing is to consider this during your design phase and have identified your approach to handle scalability, even if you might not implement a long term maintenance strategy right away.

Should I register my plugins in Sandbox or no isolation?

As you are probably aware CRM plugins and custom workflow activities can either execute in isolation (sandbox) or without isolation. This post explores when you have that choice and what good practices you should consider for making that choice.

The CRM Sandbox is a feature released in CRM 2011 whose objective was to address one problem: How can we trust custom code to execute in a server that hosts multiple customers? Certainly, the core scenario is for CRM Online in which you have different tenants (customers) sharing the same infrastructure. You can imagine Microsoft would not allow any random .Net code to execute on their data centers without some controls. The same would apply if you are a partner who hosts tenants for a third party: You have to be careful about the code that executed in your infrastructure.

So in order to execute someone else’s code in a secure manner you must make sure the code executes in a sandbox environment which has certain limitations. For example:

1. Code is running in partial trust. This prevents operations like accessing the local file system, registry, event log and many more. This is managed by .Net Code Access Security feature, you can read more here: https://msdn.microsoft.com/en-us/library/930b76w0(v=vs.80).aspx . Note that by default sandbox plugins can sent HTTP or HTTPS messages to external endpoints, such as custom web services or Windows Azure. It is possible to overwrite this default to prevent sandbox plugins from calling external endpoints (other than localhost). You can overwrite this default in the registry (HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSCRM\SandboxWorkerOutboundUriPattern). More info on this here.

2. Throttling is applied to your plugin if it exceeds a given CPU/memory threshold. You can imagine this being useful to prevent someone’s code hijacking all the server resources. The process which executes your plugin can be killed if your plugin exceeds certain thresholds or becomes unresponsive.

3. Certain statistics are calculated for your sandbox plugins. These statistics typically include data such as how many times a given plugin type has crashed and what is the average execution time. These statistics are useful for providing throttling to someone else’s plugins.

The exact mechanisms used by Sandbox are not  all fully documented, this is perhaps because it reduces the risk of vulnerability of someone exploiting sandbox in CRM Online. However, what we do know is that sandbox plugins are executed in a special sandbox worker process that supports partial trust. Each CRM organization has its own sandbox worker process. Therefore, if someone’s malicious code results in the process getting killed, nobody else will be affected. Furthermore, we know there is a sandbox host process which you can see running as one of the windows services of CRM.

If you are working on CRM Online you have no choice and you must register all your plugins and custom workflow activities in isolation (sandbox mode) because of security reasons. However, if you are on premises, you have the choice, so which one should you pick?

You might hear that it is a good practice to always register on sandbox because if is more secure and you get free plugin statistics. Additionally, your solution will work for an Online customer or an on-premises customer without any modifications. While it is true that if you are an ISV and you sell solutions in the marketplace, it makes it easier to manage if your plugins are always in sandbox so you don’t need to release SKU-specific solutions (one solution for Online customers and another for on-premise customers). However, I disagree with anyone telling me that registering in sandbox is a good practice. It makes sense for the ISV scenario, but if you are simply writing a custom solution for your on-premise customer, you are not really getting any advantage by using sandbox. On the contrary, you are losing valuable system performance by doing so (more on this later).

So unless I have a real requirement to compute and regularly report my plugin statistics, I don’t see this feature adding much value to my on-premises customer. If your plugins crash or fail then you have a bigger problem to solve than looking at statistics. Perhaps measuring how long your plugins take to execute can be useful in some scenarios, but if you have never looked at these (or you don’t know of the existence of these statistics) then it is a good indication that you don’t need them. You might find them valuable in some cases, but at the end of the day, you are losing so many points for the slow performance of sandbox plugins that you really need to think twice what you value more: Some statistics you might never look at or a faster performance of your CRM?

The reason I talk about performance is because sandbox plugins are slower to execute that non-isolated plugins. I would have loved to have some data to show you but this is an easy test you can do, or I might do it in the future. Instead, I will explain why you can expect slower performance when using sandbox. Whenever CRM needs to execute a sandbox plugin, the process executing the operation (either IIS w3wp.exe or CrmAsyncService.exe) will need to do the following:

1. Temporarily suspend the current transaction (there is a time limit for this suspension)

2. Serialize the entire exeuction context of the current transaction and send a request to the sanbox host process to execute a given plugin on the current context

3. The sandbox host process will then deserialize this information and will then again serialize the information in order to delegate to the assigned “worker” process to execute the plugin.

4. The worker process then deserializes the current execution context and runs the plugin code in partial trust.

5. The result is then sent back to the sandbox host process.

6. The result is then sent back to the original process who requested a sandbox plugin (w3wp.exe or CrmAsyncService.exe).

Below are some diagrams to help you understand the flow in each scenario:

 

 

As you can see, this requires a lot of chatter between various processes and app domains, which is not exactly something you get for free. Additionally more room for errors in your infrastructure that can occur when you require multiple processes to be synchronized. Note that your plugin always executes in the sandbox worker process when registered in sandbox. If your plugin is not registered in isolation then it will be executed directly by the w3wp.exe process or the CrmAsynService.exe process (for async plugins and async workflow activities).

Therefore my conclusion is that you should not unnecessarily sacrifice system performance by registering your plugins in sandbox unless you have a good reason to. If you work with CRM Online you have no choice and if you are an ISV it might make your life easier to use sandbox; but I suggest you always make non-sandbox (no isolation) your default choice unless you have specific requirements or limitations that force you to register your plugin in sandbox mode. Additionally, you need to consider the limitations of sandbox plugins. For example, if you desire your plugin to use the event log or a local file system or shared drive, you cannot register your plugin in sandbox.

Different Entity Flavours: New entity, new form or same same?

Often times we have different “flavours” of the same entity. For example, we might have cases related to customer service enquiries and other type of cases related to product failures. So the question often arises: How to best model in CRM these different types of entities, should we use the same entity? the same form? This posts aim to provide some guidance for that scenario.

In the example above, you might be hesitating whether or not these 2 different type of cases should use the same entity and have a simple “qualifying” attribute (dropdown) to identify the case type or whether it makes more sense to have a different custom entity for each case type. If you select the same entity, you might also wonder whether the same form should be used for all case types or(maybe with some dynamic show/hide sections) or whether different forms should be used and route to the correct form depending on the value of the “case type” field.

For the sake of this example, I will stick to the “case type” example, although I have seen the same scenario come up with other entities such as contacts and opportunities. When you find yourself in the situation in which you are not sure whether or not to re-use the same entity or even whether or not to reuse the same forms, here are few questions that can help you get started with your assessment:

1. Do you have different security requirements for each case type? If you have a requirement such that a given role/team can only have access to a specific case type then you' should consider using different entities since it will be much easier to manage the security granularity for each of the case types without having to write and maintain tons of code for it.

2. Do you execute reports and BI on all cases aggregated? In this case if you split your case into multiple entities then your reporting can be more challenging and simple charts such as “case per type” would become a pain to do.

3. How much of the business logic is shared? If most of the business logic applies across the board (e.g. same escalation rules, same custom ribbon commands, etc) then it would be easier to re-use the same entity than having to duplicate all that business logic you implemented using JS or plugins on your entity. Also consider whether you will need some of the out-of-the-box business logic (e.g. escalations or allotments for cases) that you don’t want to re-invent if you use a new entity.

4. How much overlap do you have in the fields of each case type? If the only field that the different case types have in common is the “title” then this is a clear indication that your case types are in essence different entities. It would be annoying for end users when they use advanced find or they are creating views/dashboards that they see a long list of fields but they don’t know which field applies to what case type. If most of your fields are applicable across all case types then it would make more sense to share the same entity.

5. Are the optionset values the same for all your case types? Consider for example the “Source” field. Depending on your case types the applicable values might be different, for example “Twitter” might be a valid source for a customer service case but does not apply to an operational case of equipment failure. Think about the effort required to filter or validate option sets if they are too different for each of your case flavours.

 

By now you might have a better idea of whether or not to re-use the same entity or define a new one; there is no one-size-fits-all or blank/white answer, sometimes you need to consider multiple factors and make a difficult decision based on the information you know (e.g. the questions above). Now, if you decide to re-use the same entity you are left with the question: Should I use the same form or define a new form for each case type? Again, there are pros and cons of each approach and I’ll just attempt to provide you food for thought so you can make a better decision to the question above.

1. You can create a “base” form which has a dropdown for the case type. This would be the form that users would see when creating a new case. Depending on the case that they select then you have JavaScript onload that automatically navigates to the appropriate form. This works very well but there is a bad side effect from user experience: Each time you open a case, the last used form is opened by default even if it is not the correct form for the case type you opened. The user will see a delay in which the old form is loaded and after a few seconds it will forward to the correct form and reload it. If the same user has to deal with multiple forms all the time then this effect can be quite annoying and unfortunately there is no functionality in CRM such that the record opens on a specific form without the “jumping”. However, if typically users will only open a specific case type then it would work fine because the same form will always be used by default and rarely will the user see the form switching automatically.

2. Using additional forms allows you to configure role-base security. However, you should probably not leverage this because if you restrict who can see which forms then users might open a case record in the wrong form and the system is unable to navigate to the appropriate form if the user does not have the required role. If you leverage multiple forms per entity depending on case type then it is recommended you allow all users who have access to case to see all forms for case. You can leverage FLS if you want to hide specific fields.

3. Consider creating a common section/tab on the form which contains all the fields that apply to all cases. Then you can add one tab per each case type and then hide the tab dynamically on-load depending on the value of the case type. This works great from user experience because they don’t see the form “redirecting” and it is much faster than having multiple forms. The down-side is that it could get complex if you have many fields and subsections that overlap with some case types but not others. I usually prefer this approach when things are simple (only a few fields are different).

4. Remember that restricting access on a given form to  given role does not restrict the access on the record itself. If you don’t want your customer service team to see system failure case types then restricting the form will not be enough, they would still be able to open system failure cases but see them from the customer service form (which is odd and can cause confusion). If you really have strong security restrictions consider using separate entities or field level security (FLS).

Explaining the built-in SYSTEM and INTEGRATION users

If you have played with CRM long enough, you might have noticed the existence of 2 special user accounts: SYSTEM and INTEGRATION. In this post I’ll try to answer the typical questions I get around what these are and what you need to know about them.

Let me start with some of the facts and characteristics about these 2 user accounts which will help us draw some conclusions later on.

 

THE FACTS

1. SYSTEM and INTEGRATION users have a different SystemUserId (Guid) across all CRM organizations (and CRM Online organizations). To get their user ids you’d have to perform a query (or Advanced Find).

2. These users are very well hidden from the application. They are technically “disabled” and they are even filtered out from the “Disabled Users” view. If you’d like to see them you’d have to build your own Advanced Find without the default filters.

3. Nobody can log in CRM as either of these 2 users.

4. These users are read-only. You are not able to change their teams, security, FLS, business unit or any other field.

5. They do not consume a license.

6. These users are always on the root business unit.

7. They don’t have a mailbox or the ability to send or receive emails.

8. No security applies to these users (any action is allowed when executing as SYSTEM or INTEGRATION), all security validations are bypassed.

9. Unofficial fact: Seems like you cannot impersonate these users in CRM Online from outside of plugins. I haven’t been able to impersonate SYSTEM or INTEGRATION from an external application calling into CRM Online (however, works fine on my OnPrem orgs).

 

THE EFFECT

By now you might be wondering why do we even care about these obscure users. The answer is: we shouldn’t; there is a reason why they are so hidden and sometimes unheard of. However, you might also be thinking that if you have another application integrating with CRM then you could make use of [for example] the SYSTEM/INTEGRATION users to make all service calls into CRM. This way, whenever let’s say an audited record is updated via your integration then it will show as updated by INTEGRATION user which could be a neat indicator that the update came from an external system. This is technically possible (quite easy actually), all you have to do is set the CallerId in your proxy (OrganizationServiceProxy.CallerId or CrmConnection.CallerId) to their userId whenever you create your proxy from the external system to call into CRM, this is what it would look like:

Similarly, if you are working from a plugin, you can configure the plugin step to execute as SYSTEM or INTEGRATION user. You can configure that in the Plugin Registration Tool or the Development Toolkit:

So now that you know how to impersonate these users, let’s explore the details of why we would do so:

 

THE SCENARIOS

1. Elevation of Privileges.

You might have some business logic implemented in plugins which should bypass security checks (e.g. auto-calculated or rollup fields). In that case it might be useful to run the plugin as SYSTEM user. Additionally, you will see in the audit history that the record has been updated by SYSTEM user which gives us a hint that the update was made by an automated logic of a plugin. On the other hand, you have to be very careful when doing this. The reason is that a plugin can trigger another plugin or a workflow and you might end up with a chain of plugins/workflows triggering. Once you elevate the privileges, every action after that will also run in elevated privilege mode (SYSTEM). Therefore you need to make sure that whatever chain of plugins/workflows will be triggered is OK to execute as SYSTEM. For example, sending an email as SYSTEM will fail because that user does not have a mailbox/email address so if anywhere in your plugin chain you send an email then you might have a problem.

Here is an example: You have tasks with different priorities. The priority of the case is automatically taken from the highest priority of the tasks associated. Not all users have access to update the priority of the case but any user can update the priority of their task which will rollup to the case via a plugin. When a case priority is set to “1” a plugin will send a warning email to the case owner. If you impersonate SYSTEM to rollup the task priority to the case priority you need to make sure that in your plugin that sends the email you set the “from” field, otherwise CRM will try to send the email from SYSTEM which won’t work.

 

2. Generic “system” user for system operations

These special users can work whever you want to do actions on the system programmatically and tag the action to a generic system user. A typical example is when you have integrations with other systems and you want the automatic integration in CRM to execute as a generic system user. For example, if you have integration with ERP system, and whenever a new account is created in ERP it should also be created in CRM. In that case the accounts in CRM would be created by “SYSTEM” or “INTEGRATION” user if you use impersonation to perform the account create under one of those user accounts.

 

THE CAVEAT

As I mentioned earlier, impersonating these system accounts can have some bad side effects. The reason is that any action you perform under one of those user accounts can trigger plugins (which can trigger other plugins). You need to be very careful and make sure you understand that your impersonated action can trigger plugins then those plugins will also execute under the context of the SYSTEM /INTEGRATION accounts. Some plugins you might want to always execute under the context of the Calling User (depending on the scenario). The other things to consider is that these special accounts have no User Settings entity associated (they have no language, time zone, format, etc.) and in some cases your customizations might rely on the existence of user settings for every user account. For example, in some plugins you might check that the user’s language or time zone is in order to execute some business logic. This would break if the plugin executes as SYSTEM/INTEGRATION.

Another issue with these accounts is that they are always at the root business unit. If a record is created by one of these accounts, the record will belong to the root business unit which can mean that many users in child business units will have no visibility into the record (unless they have organization-level privileges). You would have to make sure that you assign the records when they get created if you need those records to sit on different business units other than the root.

 

THE CONCLUSION / ALTERNATIVE SOLUTION

Impersonating SYSTEM/INTEGRATION can be useful for some scenarios for privilege elevation purposes; however, it does have its side effects or considerations as explained above. You need to carefully consider whether it makes sense to impersonate these user accounts given the side effects. Another alternative would be to create a new user (e.g. “CRM System User”) in AD and CRM and chose to impersonate that user whenever you are performing “system” transactions that should execute with elevated privileges. The advantage of doing that is that now you can control exactly how you configure that “super user”. You can choose to give it System Administrator role or a more restrictive role. You can decide whether or not to give it field-level permissions and in what business unit the user should be. Furthermore, you can configure a “system” mailbox so that this user is able to send “system” communications to end users (which can be quite useful as well and cannot be done with built-in SYSTEM/INTEGRATION account).

Note that if your plugins are configured to run under the context of a specific user (e.g. CRM System User) then it’s a good practice that this same user exists in all your environments (DEV/QA/Prod) with the same full name. If that is the case then you can safely transport your solutions and the impersonation configuration will be preserved when you transport plugins across your environments because CRM will be able to find the impersonating user by full name (even though the systemuserid and the AD accounts might not match it will still be able to resolve by full name).

CRM 2011 to 2013 (Orion) Top 10 Upgrade Tips and Considerations (part 1)

This post aims at helping organizations understand the impact of upgrading to CRM 2013 and provides tips for assessing the upgrade considerations. It does not provide an exhaustive list of all the possible items you need to validate before upgrading but it provides a list of the most common ones.

So it’s time to upgrade to CRM 2013 and the questions we always get are:

What is the impact?

What do I need to do to make sure my organization will continue to work in CRM 2013?

What are the tips and tricks for upgrading to CRM 2013? How can I estimate the effort to upgrade?

 

So I will attempt to provide a checklist of the top 10 most common aspects of the upgrade that you should keep in consideration. In this post I will address the top 5 and you can expect a follow-up with the next 5.

 

1. Deprecated Features

Some features from CRM 2011 were already marked as obsolete and have been deprecated in the new version. I am providing a list to the best of my knowledge of those features. If you have a CRM 2011 environment that you upgraded from CRM 4.0 it is more likely that this will be a problem since many CRM 4.0 features/endpoints/APIs have been deprecated although they continued to work on CRM 2011.

A. CRM 4.0 plugins and custom workflow activities. If you have plugin assemblies that were compiled against the CRM 4.0 SDK then you will have to re-write and recompile against the CRM 2013 or CRM 2011 SDK.

B. CRM 4.0 client side scripting. If you have javascripts that make use of the CRM 4.0 client side scripting API (e.g. “crmForm”) you will need to re-write them to make use of the CRM 2013 or CRM 2011 API (e.g. Xrm.Page…).

C. 2007 web service endpoint. If you have any external applications or any code that is referencing the 2007 endpoint (asmx) then you need to update them and recompile then using the CRM 2013 or 2011 SDK and point to the endpoints supported in CRM 2013 such as the OrganizationService (WCF) or the REST endpoint (OData).

D. ISV folder. The ISV folder was made obsolete in CRM 2011 for hosting custom web applications inside the CRM ASP.net root. This folder is no longer supported in CRM 2013 so you would have to move those custom web pages to a separate IIS website.

E. Reduced ribbon support. With CRM 2013 new user experience you might have noticed that the ribbon has been replaced with the new command bar. The good news is that when you upgrade, most of your ribbon customizations will be automatically upgraded to the new command bar model, only a small subset of ribbon controls are no longer supported: Dropdown, MRU Split Button, Textbox, Insert Table, Gallery and Gallery Button, Combobox and Color Picker. It is quite rare to make use of those controls which are no longer supported so most organizations will be fine after the upgrade but it is always a good idea to review your custom ribbon rules and make sure you are not making use of those controls. In case you are, you’d need to replace them by some of the supported command bar controls (see SDK for details). The other important consideration with CRM 2013 is that the command bar will only display 5 commands and the rest need to be expanded by clicking on the “…” icon. Therefore, fore usability, you might need to refactor your command bar to place the true 5 most used commands in the bar and move all others to the “expand” section.

F. Read-optimized forms. This was a short-lived feature of CRM 2011 that came with CRM 2011 UR7 and has been removed in CRM 2013. More information on this feature here.

G. Duplicate detection during create and update in forms. In CRM 2013 duplicate detection will not kick-in during record creation or updating via the application forms. This is a well known limitation and although you might find tons of workarounds on the web (e.g. this article from MVP Adrii Butenko), it is possible that Microsoft will bring this functionality back. For now, you might need to do a custom extension for duplicate detection if you rely on it. Note that duplicate detection is still supported when making SDK calls and setting the SuppressDuplicateDetection optional parameter in your create or update request.

 

2. Unsupported customizations

Having unsupported customizations is never a good idea and I have blogged in the past why: The Risks of Unsupported Customizations in CRM. So it would be a great opportunity to review which unsupported customizations you have since they will likely break when you upgrade to CRM 2013. The tricky question is: How do I identify unsupported customizations? As a rule of thumb, any extensions to CRM that you make which are not explicitly documented in the CRM SDK are unsupported. Now, that might be too vague and sometimes not completely accurate but it is my favorite way of defining unsupported customizations. If you want to be a little more specific here is a list of unsupported customizations that I compiled from the SDK and a couple from experience:

• JavaScript developers are used to interacting with Document Object Model (DOM) elements in code. You might use the window.getElementById method or the jQuery library. You are free to use these techniques in your HTML web resources, but they are not supported to access elements in Microsoft Dynamics CRM application pages or entity forms. Instead, access to entity form elements are exposed through the Xrm.Page object model. The Microsoft Dynamics CRM development team reserves the right to change how pages are composed, including the ID values for elements, so using the Xrm.Page object model protects your code from changes in how pages are implemented.
• We do not recommend using jQuery in form scripts and ribbon commands (Note: previously this was called out as unsupported and in CRM 2011 has been updated to “not recommended” in CRM 2013). If jQuery accesses the page DOM then it would be clearly unsupported for CRM forms.
• Modifications to any .aspx, .css, .htm, .js, .xml, .jpg, or .gif files or the addition of files in the wwwroot directories of the Microsoft Dynamics CRM application, Microsoft Dynamics CRM tools, or Microsoft Dynamics CRM files located at Program Files\Microsoft Dynamics CRM. However, if you have made changes to these files, these files are checked for modifications and will not be overwritten.
• Modifications to the Microsoft Dynamics CRM website (file and website settings). Custom solutions should be installed in a different website. This includes modifications to the file system access control lists (ACLs) of any files on the Microsoft Dynamics CRM server.
• Use of client certificates is not supported by the Microsoft Dynamics CRM SDK. If you configure the Microsoft Dynamics CRM website to require IIS client certificates, you will get authentication failures for any applications that were built using the SDK.
• Modifications to the physical schema of the database, other than adding or updating indexes. This includes any actions performed against the database without using the System Customization capabilities in the web application or using the metadata APIs that are described in this SDK documentation. Modifying tables, stored procedures, or views in the database is not supported. Adding tables, stored procedures, or views to the database is also not supported because of referential integrity or upgrade issues. For Microsoft Dynamics CRM 2013 on-premises deployments, adding indexes is supported per the guidelines in the “Microsoft Dynamics CRM 2013 Implementation Guide.” This applies to all Microsoft Dynamics CRM databases and the Microsoft Dynamics CRM for Microsoft Office Outlook local database.
  When you change the database without using the support methods for system customization, you run the risk of problems occurring during updates and upgrades.
• Data (record) changes in the Microsoft Dynamics CRM database using SQL commands or any technology other than those described in the Microsoft Dynamics CRM SDK.
• Referencing any Microsoft Dynamics CRM dynamic-link libraries (DLLs) other than the following:
  • Microsoft.Xrm.Sdk.dll
  • Microsoft.Crm.Sdk.Proxy.dll
  • Microsoft.Xrm.Sdk.Workflow.dll
  • Microsoft.Xrm.Sdk.Deployment.dll
  • Microsoft.Crm.Outlook.Sdk.dll
  • Microsoft.Crm.Tools.EmailProviders.dll
• The use of application programming interfaces (APIs) other than the documented APIs in the web services DeploymentService, DiscoveryService, Organization Data Service, SOAP endpoint for web resources and OrganizationService.
• To achieve the appearance and behavior of Microsoft Dynamics CRM, the reuse of any Microsoft Dynamics CRM user interface controls, including the grid controls. These controls may change or be overwritten during an upgrade. We do not recommend that you use or change the Default.css file in the Microsoft Dynamics CRM root installation folder.
• The reuse of any Microsoft Dynamics CRM JavaScript code, including ribbon commands. This code may change or be overwritten during an upgrade.
• Modifications to any one of the Microsoft Dynamics CRM forms or adding new forms, such as custom .aspx pages, directly to Microsoft Office Outlook or making changes to .pst files. These changes will not be upgraded.
• Making customizations except when you use the Microsoft Dynamics CRM supported tools available offline in the CRM for Outlook.
• The use of custom HttpModules to inject HTML/DHTML into the Microsoft Dynamics CRM Forms.
• Creating a plug-in assembly for a standard Microsoft Dynamics CRM assembly (Microsoft.Crm.*.dll) or performing an update or delete of a platform created pluginassembly is not supported.
• Creating an Internet Information Services (IIS) application inside the Microsoft Dynamics CRM website for any VDir and specifically within the ISV folder is not supported. The <crmwebroot>\ISV folder is no longer supported.
• Editing a solutions file to edit any solution components other than ribbons, forms, SiteMap, or saved queries is not supported. For more information, see When to edit the customizations file. Defining new solution components by editing the solutions file is not supported. Editing web resource files exported with a solution is not supported. Except for the steps documented in Maintain managed solutions, editing the contents of a managed solution is not supported.
• Silverlight Application Library Caching is not supported.
• Adding custom indexes is not unsupported but I am not 100% clear about updating/deleting built-in indexes.

 

 

3. Code inspection or analysis tools

There are a number of tools that you can install to help you assess the code that you need to review before upgrading. These tools will typically find most of the issues with your existing code and tell you what action you need to fix them before upgrading. However, do not rely 100% on these tools as they are not fully deterministic, but it is a great deal of help. Here are some examples:

CRM 2013 Custom Code Validation Tool

CRM 2011 Custom Code Validation Tool

Microsoft Baseline Configuration Analyzer for CRM (determines if your CRM 2013 environment is configured according to best practices)

 

4. ISV solutions

If your CRM organization is using third-party solutions it is always required that you reach out to the ISV and validate the compatibility with CRM 2013 before making any assumptions. In some cases the solution will upgrade automatically while in other cases manual intervention will be required to make the ISV solution work with the new version.

 

5. Mobility

Because of the limited mobile functionality of CRM 2011 (mobile express) it is easy to assume that there is no effort required to migrate your mobile express application to CRM 2013 except perhaps some testing effort. However, if you are thinking of taking advantage of the new mobile application (e.g. CRM for tablets) in CRM 2013 then you need to consider that if you are in On-Premise you might need to include additional effort to enable IFD if it is not already configured. Mobile application in CRM 2013 is only supported with claims authentication and IFD mode. Additionally, if you have third party mobile solutions (e.g. Resco, CWR) then refer to the previous paragraph.

Setting a Blank Default URL for an IFrame in CRM 2011

It is common that we want an IFrame in an entity form and we need the URL of the IFrame to be set dynamically when the form loads using the setSrc() method in a script. But how can we set a blank default URL?

When you insert an IFrame in a form, the form designer forces you to enter a blank URL. Before Polaris (UR12) we could enter “about:blank” and we’re all set! However, you are now forced to enter a URL with a valid prefix (http, https, ftp, fttps):



If your IFrame will embed external web pages (such as www.bing.com ) then it is not a problem to enter the default URL and have the script modify the URL if required at run time. However, if your IFrame is used to display HTML pages that you have defined as CRM Web Resources then you will not want to enter the full URL (e.g. http://mydevcrmserver.com/myorg/WebResources/test.html ) because the URL would contain a link to your DEV organization and that means that every time you promote to test/prod or install the solution on a different server, you’d have to go and modify the URL manually. So what can you do?

One option is to set the defaul URL to some hard-coded external website (e.g. http://www.bing.com) and have your script update the URL at runtime so the IFrame loads your HTML web resource. Ugly!

The best option would probably be that Microsoft allows as to enter “about:blank” but it seems this is no longer allowed after UR12 (hopefully they will allow it in the future). But you don’t want to rely on that.

The second best option that should work for you is to define your own “blank” web resource (Thanks to Alex Ries for the suggestion):

1. Create a new HTML web resource, which is blank (<HTML/>)
2. In your form, instead of inserting an IFrame, insert a web resource and use your blank web resource you just created.
3. At run time, your script should still be able to modify the URL of your web resource to point to another web resource. The setSrc() method is supported on IFrame as well as web resource controls!

With this solution, you are no longer using an IFrame so you are not forced to set a default URL! Hope this helps you!

JavaScript, XML processing, DOM and CRM 2011 UR12

If you have scripts in CRM which somehow retrieve or process XML, chances are you will need to do some re-write or validations when applying Update Rollup 12 (UR12). This post explains how to fix you JavaScripts when you make use of the DOM parser for processing XML.

One of the highest-risk customizations to break with UR12 are JavaScripts and these are the 2 most common reasons why:

1. It is common for developers to perform unsupported operations with the page DOM instead of using the supported xRM APIs. Since UR12 introduces cross-browser support, accessing the DOM by unsupported methods such as ‘getElementById’ on CRM forms will no longer work because it has changed. Therefore, customizations like this one will break.

2. Because CRM has historically only been supported in IE, developers might have written JavaScripts that depend on IE extensions specific to Microsoft, these will also stop working because the assumption is no longer true (will be broken even in IE!). Even though you are not doing anything unsupported in your scripts, they can break and might need to be fixed.

There are plenty of tools and blog posts that talk about how to tackle #1, so in this article I will focus on problem #2 above and I will pick the specific example of processing XML.

Assume you are parsing and processing an XmlDocument in your JavaScript, perhaps you retrieved the xDoc from and XML HTTP request, using AJAX or even if you are simply parsing text into an XmlDocument object. In my case, I was retrieving an XML web resource using the following code:

1. function ShowHelpText() {
2.     var tooltipsTextFile = "ava_TooltipsText.xml";
3.  
4.     // Use AJAX to retrieve the tooltips texts
5.     $.ajax({
6.         type: "GET",
7.         async: true,
8.         url: tooltipsTextFile,
9.         dataType: "xml",
10.         success: parseTooltipsXml,
11.         error: function (xhr, textStatus, errorThrown) {
12.             alert("Error retrieving tooltips: " + xhr.message);
13.         }
14.     }); //end ajax
15. }

Before UR12, this code used to return a documentElement of type IXMLDOMElement which is a Microsoft extension to the W3C DOM. Therefore, I was happily calling methods such as selectNode and selectSingleNode. The problem is, after UR12, the same code will return a standard Document object which does not support selectSingleNode so I get the error “Object doesn’t support property or method ‘selectSingleNode’”:

 

In part, this is great news, because now your JavaScript will work in any browser (mostly), but you still need to do some clean-up to remove any dependencies on IE DOM extensions. In my case, I had to update my script to make use of getElementsByTagName and verify that I only use methods and properties that are W3C standard as documented here and process my XML according to that documentation (which works slightly different than if you have IE-specific JS).

Although my example is specific to parsing XML, the same can apply to any other scripts that rely on IE extensions not available in other browsers or not standard. I hope this post helps others be more aware and proactive before applying UR12 and in understanding how your scripts can break even if you are not using any unsupported customizations!

Problems with leading and trailing blank spaces in CRM data

I recently ran into a number of problems that ended all up being due to blankspaces in some CRM data. I thought of sharing where these problems come from, how blank spaces can affect CRM functionality and how you can resolve them.

You might wonder: how can you end up with leading or trailing blank spaces in text fields in CRM? While it is true that CRM forms are smart enough to remove these blank spaces upon form save, it might not be the case when the data comes in from data import or during integrations with external systems. It sounds like a small problem, but if you are not careful, blank spaces can become a data integrity nightmare for the following reasons:

1. You might end up with duplicate data that is hard to resolve later on. For example you might have account “123” and account “ 123 “ which reference to your same customer. Now all your activities, invoices and all related records are potentially spread across 2 different records. Fortunately, duplicate detection ignores leading/trailing blank spaces so these 2 records would be considered duplicate if you have a published rule on account number field. However, if these accounts are created from the SDK (or through an integration layer), chances are that the duplicate detection is not enforced because when making web service calls duplicate detection is off by default unless specified in the create/update message. Fixing duplicate parent records can be a really long and boring task.

2. Assume you have multiple contacts related to account “123” that you want to import using the data import wizard. However, your account appears in CRM as “ 123 “ (with blank spaces). It will be impossible for import wizard to match the contact’s parent account and import will fail because the parent account was not found. You might say: I can just change my import spreadsheet to reference account “ 123 “ instead of “123”. The answer is: it does not matter if in your import XML file you enter “123” or “ 123 “ as the parent account, CRM will not be able to resolve the parent account!! In short: You cannot use the out-of-the-box data import feature to import related records to that account until you fix the blank spaces, argh.

3. Once you fix all your integration points to trim all text fields before committing a create / update to CRM, how do you clean the thousands of records that already exist? My first guess was: Export to Excel for re-import, use some Excel functions to remove blank spaces and import back to CRM. Again to my surprise, removing leading/trailing blankspaces in text fields is not considered a change for CRM so during import, the records were not even processed! When you export records for re-import, CRM is smart enough to know that if you did not update a row, it will not re-import that row. However, it is not smart enough to figure out that you removed blank spaces so it also ignores your update.

4. Sometimes the CRM forms trim blank spaces for text fields. This can become a problem because if your data has trailing blank spaces, the CRM form will remove then on load, and automatically the form is marked as dirty! Therefore, just by opening the record your form is dirty and you get a warning about pending changes when you close the form even if you did not update the record. It also prevents some functionality since some of our ribbon buttons require the form to not be dirty. It is also not evident what was happening and took us a while to figure out why some forms where always getting marked as dirty.

 

We let our production CRM get really messy because we did not notice this problem until too late (we never noticed the external systems were sending data padded with blank spaces) so cleansing the data became quite challenging since it was spread to multiple fields in multiple entities, many duplicate parent and child records. As I mentioned earlier, fortunately duplicate detection does consider these 2 cases as duplicates (“123” and “ 123 “) so you can execute multiple duplicate detection jobs to find and resolve duplicates. However, it can be a really long task to do. In our case, because of the magnitude of the data cleanse that was required we had to invest in a data cleansing tool that would iterate through multiple records resolving the blank spaces and the duplicates which reduced considerably the amount of manual work required to solve the problem. The “tool” in a nutshell consisted simply in a custom workflow activity that takes as input a CRM query (Rollup Query) and then retrieves and cleans all the records specified as input. Then we can use on-demand workflows and provide different queries to resolve duplicates in multiple entities. In any case our conclusion is that we should have thought about this from the beginning and I hope this post helps other avoid the same problem before it is too late.

Custom Entity or Custom Activity in CRM 2011?

Not sure if you need a custom activity or a custom entity? This post mostly written by my co-worker Patrick Vantillard includes some of his conclusions after making some research about when to use a custom entity vs. a custom activity.

Difference	Custom Activity	Custom Entity
Logically	An activity represents an action such as making a call, or attending a meeting. This typically involves completing some “work” which is associated with the duration field (how long did it take you to complete the activity). Activities typically have a lifecycle: opened à assigned à worked on à closed. Activities in general cannot have sub-activities, at least not in the same native way that entities have associated activities.	Custom entities can represent any business entity. Custom entities store master data or transactional data and can have activities associated with the custom entity record.
Security	Given a security role, the same security applies to all activities (standard & custom) at once. You cannot give a different access level or privilege for a specific activity. For example, you cannot restrict “Delete” access to your custom activity but grant “Delete” access to other activities like Task.	You can control security privileges for each custom entity separately.
Visibility in the Application	When creating a custom activity you can check the “Display in Activity Menu” option which if selected, the custom activity will appear in all the Activity menus as well as the ribbon of all the activity-able entities (of course the ribbon can always be modified after). We can hide custom activities in the activity menus by unchecking the “Display in Activity Menu” box. NOTE: once the activity is created, that setting cannot be modified. By hiding the custom activity from the Activity Menu, your custom activities will also NOT show up in the “Activities” or “Closed Activities” views and associated views, so you would need a separate view for your custom activity (as if it were a custom entity).	Visibility of custom entities is controlled by the sitemap and form navigation menus.
Associating to the Case Entity	Because Case is an activity-able entity then your custom activity can be associated with a Case without the need of any special custom relationship and you can re-use the “Activities” and “Closed Activities” navigation menu to show all activities including your custom activity.	Custom entities will require a custom relationship to be able to associate them to a case. Therefore, you would require a new section in the form navigation to show specifically your associated custom entities.
Resolving a Case	By default, the case has been built in order to not allow a user to resolve a case as long as any of the related activities (standard & custom) are still open.	There is no validation on the status of the related custom entities when resolving a case.
Calculation of Time Spent on a Case	By default, on completion of the case, the Total time of the case will be calculated from the sum of duration values for associated activities. When a case is reactivated and then resolved again, Total Time field is calculated with two different options: 1. If no contract associated with case, time spent on additional activities added to original total, included in Total Time. 2. If contract associated with case, only time spent on additional activities included in Total Time.	Associated custom entities have no effect on the calculated Total Time Spent on the case.
Contracts Related to Case	While creating a contract, a user can select an Allotment Type. A user can therefore specify how the support will be “budgeted” (or, “allotted”) for a contract. You can specify number of incidents, or amount of time (linked to activities)	On the contrary, no such behavior has been created on the contract level linked to standard and custom entities.
Dashboards, Charts and Views	By default, will be able to create a single chart comparing different type of standard and custom activities. Since all activities roll up to the “Activity” (activitypointer) base entity, you can create a chart/dashboard/view which takes data from all activity types including your custom activities.	The system won’t allow a user to create a single chart comparing multiple standard or custom entities. One chart will be required by entity. Extensions in Silverlight or SSRS would be required to combine data from multiple entities.
Reports	Some out of the box reports are available will automatically include your custom activity. In addition, the report wizard will allow the user to create very easily some reports comparing all related activities linked to a case on a single Report	No reports are available out of the box for custom entities linked to a case. In addition, the report wizard could be used by the user but the user won’t be able to combine multiple custom entities linked to a specific case. Thus, there is no simple way to provide an overview of a specific case and its related entities. ( e.g. Case linked to two custom entities: Main entity and a Peer Review)