Large Downloads in RETS Connector

Many RETS servers implement internal limits on the number of records that can be downloaded in one Search transaction. RETS specification provides 3 ways to override those limits. The following chapters describe how they are used, and also what are the pros and cons of each one.

Return ALL records

By specifying ALL in the “Maximum Records Returned” field, you are asking the server to suspend enforcement of its internal limits. This is the easiest way to ask for all data. However, many servers do not honor this request, and will limit the download anyhow. And even if the server does send the whole data, the download may be very big and in some servers it may cause the response to take a long time, resulting in timeout error I the Connector. Increasing the timeout for the task may resolve the last issue, but it’s not very flexible.

Use Offset to overcome server’s limits

RETS servers allow to specify an offset, effectively removing first few records from the download. That allows the client to ask for additional data if their first request was limited to a number of records lower than what the client needs. This is a good way to load large amounts of data, but represents a “dirty read”, which as not guaranteed to be accurate. Records added to or removed from the server between two reads may change the record positions so that some records are sent twice and some records may be skipped, even if they do match the requested criteria.

Due to performance reasons, some servers do not support offset. Unfortunately, there is no way in the RETS metadata to indicate whether the feature is supported Some servers will immediately fail if the Offset parameter is used in the request, while others will silently ignore the offset request and repeat the data from beginning.

RETS Connector allows for the Offset to be used. If the appropriate checkbox is checked, the Connector will try to download the requested data, repeating the request with increasing Offset as needed if the server does not send all matching records. It will also analyze the incoming data to see if the Offset feature is supported by the server, and fail the task if it is not.

Use Index to overcome server’s limits

Servers compliant with RETS/1.8 specification may implement an Index. The do not limit the number of records if only a small number of selected fields is requested. RETS Connector uses that feature to download a full list of record keys, and then to request full data for those records in batches. This strategy ensures that the connector is not missing or duplicating any records. Availability of the Index feature is indicated in the server’s Metadata, so the Connector does not offer it if the server does not support it.

Due to a problem in the RETS specification, the Connector cannot specify any limit when asking for the index keys. That may result in longer process time if a limited number of records is requested. However, the number of records stored in the Connector’s output will reflect the Maximum Records limit specified by the user.

If the server supports the Index, it is the safest and preferred way to overcome the server’s limits.

Batch Size

If Index or Offset is used, the data will be downloaded in several batches, each of which represents a separate request to the server. The Batch Size allows you to specify how many records should be requested in each batch. By default (automatic), the server is asked to provide as many data as possible in each batch, limiting the number of batches to a minimum. However, that may result in large responses, which may encounter timing problems on the Net. Specifying a lower batch size may resolve these problems by making more but tinier requests that have better chances to go thru. Another advantage of smaller batches is in resumable (see Resume options in the Download Schedule tab) tasks: the data from each completed batch are saved separately, so if the task fails for any reason, it does not have to reload so much when it is resumed.

Output format limitations

Because of the need to analyze the incoming data, the Offset and Index options are available only when data is requested in one of the Delimited formats (see the Data Format tab).