ArcGIS JSAPI Hack: Get around the 1000 limit on ArcGIS Server
Anyone who has developed against ArcGIS Server web services has probably encountered the limit on query results returned. Whether it’s 1000 features, or maybe 2000 with the new versions, it’s never enough for the kind of data you have to return. Contrary to what he or she believes, the typical map user can’t fathom having more than 1000 distinct points, lines, or polygons on a map at one time.
Why is there a limit, anyway?
TL;DR – It’s there for your protection.
Geographic data is pretty big data (not to be confused with Big Data, though). ArcGIS graphics are made up of both vector data (points, lines, or polygons) and attribute data. Requesting all data on 1,000 graphics in ArcGIS Server may take up to a minute to download to a mobile device with poor internet service. Add symbology to each graphic, and infoTemplates showing how popups should show them, and the amount of data in each graphic adds up. Rendering 1,000 polygons with custom symbology, a long list of attributes, and infoTemplates takes a lot of memory in the browser.
Now, imagine loading 75,000 graphics in the browser. Panning the map would slow to a crawl. Refresh rates on the browser would be pitiful, and many users would quit, thinking that the application was broken.
So, really, the 1,000 graphics limit is there to protect you, the developer, and the user experience of your application. Anything more than that could cause serious issues, noticed first in mobile browsers and older browsers, but affecting everybody.
What if I told you that there was a way to query for everything you want? There is, but you have to query twice. The first time, you have to use the executeForId() method provided by the QueryTask. If successful, it returns the ObjectIds for all the features that match the results.
Once you have all the ObjectIds, you can slice them up into groups of 1000, then query for all the results for each batch. You should consider staggering the requests so that the browser and/or server aren’t overwhelmed by all the requests. You can see an example BigQueryTask in the github gist below.
In the code above, the query takes a 1000 objectid slice, then queries for it. When results are returned, they are merged with any previous search results. Then, the code checks whether all the objectIds have been requested. If not, queries for the next slice. If all objectIds have been requested, they are resolved in the dojo/Deferred response.
I’ll have a working example here shortly.
How to show it
If you’re dealing with large numbers of features, and you have to show them on the map, that can be a pain. You may be tasked with showing residential properties within a county or a zip code. If you have 73,000 properties that match that query, you have a problem We already talked about how adding 73,000 graphics to your map will slow the average user’s computer to a crawl. On the other hand, if you only show the first 1,000 properties, the results may display only a corner of the county.
What if I told you there was a viable alternative? There is, but to find it, you have to let go of the preconceived notions you built up so far to solve the problem.
The solution is the ArcGISDynamicMapServiceLayer.
Yep, the old dynamic service layer comes in and saves the day. If you show the property layer in question using a dynamic map service layer, you can apply a layerDefinition, or a query string connected to a specific layer on the map service. This layerDefinition will be applied to the layer in ArcGIS Server before rendering, and the result will be shown on the map. Now, you can get your 73,000 properties and show them, too.
Please note, however, that this works only for text-based where clauses. At this time, you can’t pass in a geometry to the map service layer and collect the results. You could, however, construct a query where clause using all the objectIds from the search, like the following:
“OBJECTID in (1, 2, 3, 4, …, 9999, 10000)”
This is the method we at Bruce Harris and Associates chose when we developed the GIS-based property search website for the Osceola County Property Appraiser. By the way, congratulations goes out to them for winning a Special Achievement in GIS (SAG) Award for 2015.
If you have ArcGIS Server 10.3
If you’re working with ArcGIS Server 10.3 and you have a hosted feature service, you have more options for retrieving attribute data. You can quickly retrieve pages of data through the Query.num and Query.start attributes introduced in version 3.9. The num is short for the number of records you’d like to see, and start is the zero-based index of the feature you’d like to start with.
This may not help you see all the records, but it beets keeping up with what page you’re on, and the associated index that goes with it.