Author Archives: Bas Lijten

About Bas Lijten

Bas Lijten is a SharePoint architect at Achmea, with a special interest in search and integration with other platforms, such as Windows Phone and Windows 8

Fast Search for SharePoint: crawler process blocked due to memory capacity limit

This week I encountered the problem that the indexation of content didn’t seem to end. After 5 hours of crawling, my 1000 html pages of content still weren’t indexed by the Fast crawler. After putting on my Sherlock Holmes outfit, I started to investigate this problem. It turned out to be a real adventure to find a solution for this problem.

Continue reading

Indexing the new Geolocation field in SharePoint 2013 Technical preview is not possible

As I was playing around with the new geolocation column of SharePoint 2013, I wanted to try to migrate my geosearch solution for Fast Search for SharePoint 2010 to SharePoint 2013.  The addition of the geolocation column was opening up some interesting possibilities. Potentially, this solution would add some (near) out of the box geolocation solutions.

Unkown Datatype

After adding the geolocation column, adding some data to the last, I started to crawl the content. For some reason, the geolocation crawled property didn’t show up. And the reason why was logged to the Diagnostic log:

07/22/2012 21:14:27.12 mssdmn.exe (0x22BC) 0x1088 SharePoint Server Search Connectors:SharePoint dvs3 High Unknown datatype Geolocation [sts3util.cxx:6498] search\native\gather\protocols\sts3\sts3util.cxx

Apparently, the logic to index a geolocation column, hasn’t been implemented yet. Let’s hope it will be implemented when SharePoint 2013 will get released!

Expect some blogposts with a workaround for this issue soonish 😀

SharePoint 2013 geolocation column: a component is not installed

Today I was playing with the new geo location column. I added the Bing maps key, created a custom list and added the geo location column via code to that list. Upon submitting a new listitem, I received an error, mentioning that a component was not installed, to be able to use geolocation. This is a strange error, because I was able to add the column programmatically, while adding content to the list, is not possible. After checking the Diagnostic log, I noticed the following error:

A required component for using a geolocation field is not installed: Microsoft SQL Server System CLR Types

This remembered me that I had read somewhere (this means: It’s not in the geo location MSDN documentation.It’s written in the documenation on mobile apps and geolocation) that the Sql Server System CLR Types needed to be installed on the Web Front End servers. The package that needs to be installed is called “SqlSysClrTypes.msi”, and depending on the version of SQL that is used, one of the following feature pack needs to be installed:

Sql Server 2008 R2 SP1 Feature Pack

SQL Server 2012 Feature Pack

After the installation, I still couldn’t add any items to the list. An IIS Reset didnt work for me but a reboot did the trick.

Update: I found the place where I read about the installation package

WSP Deployment scopes and the development lifecycle in large environments

At our company, we use sharepoint to host all of our websites. As SharePoint is quite scalable, this is of course no problem and the production farm can handle it easily. These websites need to be developed and be tested, before they can be deployed to the production farm. This process is nothing special and doesn’t diverge from most popular software development methodologies.

Development Lifecycle

As stated before: Our development process is not something special: we develop, test it, the customer accepts the product and finally we take the product into production.

 

 

Most of the developers that I know, develop their products on their own systems and they test it locally before the product is pushed to the development farm. At those companies, there is just one project (2 at most) which makes use of the development farm. The developers are free to deploy whatever and whenever they want, because they don’t have to take other projects into account.

 

But our situation is quite different.We host a large amount of portals and websites (that number is still growing) and part of those existing portals are  continously upgraded. As opposed to the situation above (with a single project working at a farm), we do have 10+ projects working simulteaniously at a single farm.

 

Deployment cycle

When just developing websites in ASP.Net, this isn’t much of a problem, as all sites can easily be copied to it’s location. As we all know, the deployment scenario in SharePoint, is different. Developers create WSP’s, and those WSPs are deployed to the farm. In our case, code is checked into TFS, TFS builds the WSP’s and some tooling, ROSS, gets triggered and deploys the generated WSP’s to the development farm. That same tool deploys the solutions to Test, Acceptance and Production environments. This tool has been configured in such a way, that a solution only can be deployed to the production farmwhen it has passed the Development, Test and Acceptance stages.

 

 

This automated deployment is very useful, but causes some problems for us. Due to the fact that we have multiple projects at the same central development farm, which makes it easy for developers to deploy solution pacakges, this causes a lot of retractions and deployments of WSP’s. Every single WSP retraction and addition, causes the IIS application pools to recycle. This can (sometimes) take quite some time, can be frustating, and is, sadly enough, inherent to SharePoint Development. These deployments can cause other projects to recycle too. This is mainly due to the scope of the WSP’s being deployed: when a WSP is deployed with a global scope, or is retracted, this affects ALL webapplications on the farm. Every single application gets a “free” application pool recycle. With two projects, this doesn’t have to cause much problems: even when no deployment planning is made, those projects don’t have to suffer much from each other.

 

 

But when more projects come in, things get worse, as seen in the diagram below. The availability of all web applications gets lower, which results into less testing time per app, and each deployment of a WSP blocks all other projects from deploying/testing. It’s not only annoying, it also cost’s a lot of serious money. The next diagram is just an indication with 2 deployments per project a day and assumes that a deployment on a farm with 1 webapplication takes as long as a deployment to a farm with 10+ webapplications. In my experience this is not the case at all: the more applications that co-exist on a farm, the more time a (global) solution deployment takes.

 

 

A solution for this problem is easy: split de big central Develoment farm into several small ones, as shown in the picture below:

 

 

 

This is a situation that, in my opinion,  has a lot advantages (aside from the availability options), and fits into the cloud strategy of microsft (hosted development images), but doesn’t solve all problems. Where the problems will disappear for developers, the problems will stay the same for the Test, Acceptance and Production environments. Not as much as at the development stage, but the deployments are still blocking other projects. The Test environment is heavily used by our testers and needs a lot of uptime, too. When looking from a business perspective: it’s notacceptable that a commercial site, which sells products, has downtime because an other site isbeing deployed to production.

 

The Solution

The solution to get rid of the blocking deployments, but keep the flexibility of deploying “any time”, needs a small hack. The deployment scope of the solution needs to be changed from “Global” to “WebApplication”. This is not a setting in the package property window, but is determined “deploy time”. In short: whenever there is an artefact or configuration option that needs to be deployed at the webapplication scope, the solution scope is “webapplication”. In all other situations, the WSP will be globally deployed. The easiest way to do this, is to manually add a dummy safecontrol to the manifest.xml.  Note: this doesn’t work in all cases. There are a few situations where a solution only can globally be deployed to a farm. Try to bundle these functionalities into a single package, to keep downtime as low as possible.

 

 

As seen in the picture above, in the most ideal situation, projects don’t block each other from testing. The only influence that projects have on each other, is the fact that wsp’s can’t be deployed simultaneously.

 

Conclusion

SharePoint solution deployments can be quite time consuming, and when not properly planned and configured, it can even block other projects when working simultaneously on a farm. The more projects that are using the farm, the more projects that can suffer from each other. A simple solution for this, is to force the solutions to be deployed at webapplication scope. This can be achieved by manually adding a safecontrol to the project.

GeoSearch with Fast Search for SharePoint 2010 – Part 3: how to use the latitude and longitude in your queries

This is the third and last part of a series of blogposts about GeoSearch with Fast Search for SharePoint. I recently held some presentations about extending the power of Fast (on DIWUG and the Dutch TechDays 2012) and one of the subjects that I talked about was on how to search for locations, sort on distances etcetera. The recording of my sessions are available on Channel 9 in dutch.

Continue reading

Upgrading Masterpage and UIVersion from SharePoint 2007 to SharePoint 2010 without v4.master

When upgrading from SharePoint 2007 to SharePoint 2010, several scenarios are possible. When using the database attach method, it’s possible to stay in “V3” mode, or execute a visual upgrade too and make use of all SharePoint 2010 love. Whenever custom masterpages have been created for SharePoint 2007, the visual upgrade itself isn’t too much of an option: it upgrades the UIVersion of the current web to V4, but the custom masterpage is replaced by the default v4.master, which (of course) doesn’t provide any customizations. the next step is to create a new customized masterpage for SharePoint 2010, add it to your site and make sure to set this masterpage as the new default/custom masterpage.

The scenario I wanted to achieve, was the following:

  1. Deploy my custom solutions with all masterpages already upgraded to SP2010, so only the UIVersion should be upgraded
  2. Attach the content database
  3. Upgrade the site to v4 mode, without the need of a v4.master deployment

The problems of (visual) upgrading to SharePoint 2010

The problems of visual upgrading consist of several issues: it doesn’t work right out of the box (but that is expected), and some steps are needed to deploy and undeploy a masterpage, before the real SP2010 masterpage can be set, because the v4.master seems to be required

1 – Upgrade the site doesn’t work immediately

As stated above, a visual upgrade using the user interface or upgrade by code/powershell is needed to let the existing site run in v4 mode:

1: $web = Get-SPWeb http://site
2: $web.UIVersion = 4
3: $web.Update()

sadly, this will bug out, because SharePoint requires the v4.master to be able to upgrade to v4-mode. The v4.master has to be made available via a feature (or manually) in the masterpage gallery, before the upgrade can be executed.

2 – Upgrade to a masterpage that isn’t needed

Assuming that all customizations that did exist in SharePoint 2007 need to be retained, the above solution isn’t the final solution. Next steps are to create a SharePoint 2010 masterpage with the customizations, deploy that masterpage, and set that masterpage as the default/custom masterpage for that site. And because that v4.master wasn’t needed at all, that feature should be deactivated, or even better for hygienic reasons, be uninstalled and deleted.

why is this a problem?

As this isn’t to much of a hassle for a few sites, it does whenever some big portals/intranets are migrated, with a lot of different masterpages. When working with about 10.000 webs, this needs to be automated, but activating anddeactivating 10000 times a feature can be quite time consuming. A lot of time can be saved, whenever the customized masterpages are already “SP2010-ready”, so all is left to do, is to set the UIVersion to 4.

The solution of quickly applying a visual upgrade without a v4.master

As described above. I wanted to achieve the following scenario:

  1. Deploy my custom solutions with all masterpages already upgraded to SP2010, so only the UIVersion should be upgraded
  2. Attach the content database
  3. Upgrade the site.

what happens behind the scenes?

When firing up reflector, we see that the following happens in code:

When the UIVersion is set, an internal function called “SetUIVersion” is executed. This function takes the UIVersion as a parameter and a boolean which decides wether or not to set the masterpage. Unfortunately, this function is an internal function, so it can’t be used without using reflection.

The SetUIVersion function contains some logic to determine whether to rollback to UIVersion 3, or to upgrade to UIVersion 4. After the decision, the internal function SwitchMasterPages is called, with 5 parameters:

That function is called with the following parameters

And what happens in this function gets interesting. This function checks what default and custom masterpage are currently set (the V3 masterpages), and stores them in the propertybag of the current web, specified, by the parameters specified in the function. To determine what new masterpage needs to be set, that same propertybag is checked: in this case the properties “v4master” and “v4customMaster”. if they are empty, the default v4.master is used:

and in this part of the function, the error is thrown that the default masterpage couldn’t be found:

When the propertybag isn’t empty, the value of that propertybag is set as masterpage:

And that is our solution!

Simply set add the v4master and v4customMaster to the SPWeb propertybag, with the values of the current masterpage. Update the UIVersion of the web and update it. That should do the trick:

Summary

Upgrading a SP2007 site to SP2010 is easy to do, but it can consume quite some time and operations. To streamline the upgrade process of a SP2007 site to SP2010, it’s easier to upgrade the Sp2007 masterpages beforehand and include them, with the same name, in the solution. Deploy the solutions, attach the databases and upgrade every SPWeb using the above powershell. That should do the trick!

GeoSearch with Fast Search for SharePoint 2010 – Part 1

This is the first part of a series of blogposts about GeoSearch with Fast Search for SharePoint. I recently held some presentations about extending the power of Fast (on DIWUG and the Dutch TechDays 2012) and one of the subjects that I talked about was on how to search for locations, sort on distances etcetera. The recording of my sessions will be made available on Channel 9.

Why using Fast search for SharePoint 2010 for this purpose?

It’s possible to compute distances between locations in a lot different ways. For example, address can be queried against SQL Server (if the correct data is available), or those addresses can be used in conjunction with the Bing Geocode services. A custom webpart with logic to query one of those services with the users/objects current location and all list items with location information takes little time, but performance issues can pop up in no time. how is the performance when there are 200 items in a list? and 2000? 20000? maybe 200000? Surely, I can imagine that there are some smart solutions to send 200000 locations to the geocode service and receive them back, but remember: how did you extract that information from a sharepoint list? That takes quite some time. It is even getting harder when data comes from several lists, not even thought about data from several site collections, external data (BCS is an option here btw) or, location information that resides insidedocument.

Continue reading

GeoSearch with Fast Search for SharePoint 2010 – Part 2: Custom pipeline extensions

This is the second part of a series of blogposts about GeoSearch with Fast Search for SharePoint. I recently held some presentations about extending the power of Fast (on DIWUG and the Dutch TechDays 2012) and one of the subjects that I talked about was on how to search for locations, sort on distances etcetera. The recording of my sessions will be made available on Channel 9.

This blogpost describes how to create custom pipeline extensions

  • Part 1: why and how
  • Part 2: the custom pipeline extension to enrich the Fast index with spatial data (this post)
  • Part 3: how to use this metadata in your search queries
  • EDIT: Sourcode for the custom pipeline extension

Continue reading

Fast Search for SharePoint caveat: Apply a sortformula

Fast Search for SharePoint provides four great possibilities to influence the sort order of your resultset that was returned:

  • Sort by Managed property
  • Sort by Rank
  • Sort by Formula
  • Sort in Random order

When I was working on my presentaion and demo for the diwug (Dutch Information Worker User Group) last week, to show off on how you can improve your search experience by using custom ranking models, custom sort orders and extending the pipeline, I ran into an issue that I wasn’t aware off. When I used a custom sort formula, I did get the right result set, but not the right result order.

Continue reading