Paginating the data feed is an excellent way of making sure the server requests run smoothly and effectively. The server does not need to be tasked with an overload of requests to maintain your website's peak performance.
You can feed Doofinder data directly via API or allow Doofinder to read the data from a URL.
This way, URLs can point to already generated URLs (so they're basically downloads) or to server endpoints that dynamically return the most up-to-date data directly from the database. Such endpoints can be expensive in terms of resources. Therefore, Doofinder splits the download into multiple requests to maintain performance by enabling feed pagination.
Pagination, in this case, refers to the ordinal numbering of pages system.
Doofinder's feed pagination
To activate Doofinder's feed pagination, you need to contact the Support team through our contact form and make the request. If needed, simply log in to your Doofinder Admin Panel. Click on Help in the top blue bar, then select Contact our Support Team.
Before contacting us, ensure that your server has the following two parameters to retrieve a paginated feed:
– Limit: This is the maximum number of items returned in the response for each server call.
– Offset: This is the position of the first element that must be returned in response to the current request. The first value for this parameter is 0. This value increments limit with each request.
Also, note that depending on how the pagination is implemented, the feed can actually return "Limit" items or less.
For instance, if you implement it to get the items with offset <= ID < (offset + limit), you can get less than "Limit" because some items with an ID that matches the condition may be out of stock or deleted.
If you instead implement it just to request your database with the very same values for "Offset" and "Limit", your database will always return "Limit" items, until there're no more results or the last set of results is smaller than "Limit" in length.
Doofinder will start gathering pieces of the feed using the limit and offset parameters until it gets an empty response.
Upon setting the limit, Doofinder will make requests starting from offset=0, offset=1000, then offset=2000 and so on until an empty response is received. Of course, along with the offset parameter goes the limit parameter which specifies the size of the page (1000 in this case).
Example:
Say we have a feed at http://server.com/doofinder.cfm, which has 6300 products in its data feed. We then set its pagination limit to 2000 items.
The first server request to our parser would then be:
http://server.com/doofinder.cfm?limit=2000&offset=0
This would fetch the first 2000 items under the following conditions:
– When the feed is an XML, this would also fetch the declarations at the beginning of the XML file.
When the feed is TXT, this would also fetch the file's header.
Consequently, the next request would be:
http://server.com/doofinder.cfm?limit=2000&offset=2000
This would fetch the items between 2000 – 3999.
Special care should be taken when the feed is an XML or CSV feed such that the beginning of the XML is not resent with the data.
This would be under another request with offset 4000:
http://server.com/doofinder.cfm?limit=2000&offset=4000
A request with the offset set at 6000 would fetch the last 300 items. When the feed is in XML, this would also fetch the XML closing tags.
http://server.com/doofinder.cfm?limit=2000&offset=8000
The final request would then get an empty response, signalling Doofinder's downloader to end the download process.