{"_id":"564d1afa4567342100ad96b7","user":"5539912a0074c80d00621b14","__v":0,"version":{"_id":"564d1af84567342100ad96aa","project":"551375e1d04af219007ddc52","__v":1,"createdAt":"2015-11-19T00:42:32.705Z","releaseDate":"2015-11-19T00:42:32.705Z","categories":["564d1af94567342100ad96ab","564d1af94567342100ad96ac","564d1af94567342100ad96ad","564d1af94567342100ad96ae","564d1af94567342100ad96af","564d1af94567342100ad96b0","564d1af94567342100ad96b1","564d1af94567342100ad96b2"],"is_deprecated":false,"is_hidden":false,"is_beta":false,"is_stable":true,"codename":"","version_clean":"1.1.0","version":"1.1"},"category":{"_id":"564d1af94567342100ad96b1","version":"564d1af84567342100ad96aa","__v":1,"pages":["564d1afa4567342100ad96b7","564d1afa4567342100ad96b8","564d1afa4567342100ad96b9","564d1afa4567342100ad96ba"],"project":"551375e1d04af219007ddc52","sync":{"url":"","isSync":false},"reference":false,"createdAt":"2015-04-28T19:56:44.186Z","from_sync":false,"order":6,"slug":"best-practices","title":"Best Practices"},"project":"551375e1d04af219007ddc52","updates":[],"next":{"pages":[],"description":""},"createdAt":"2015-04-28T19:57:20.563Z","link_external":false,"link_url":"","githubsync":"","sync_unique":"","hidden":false,"api":{"results":{"codes":[]},"settings":"","auth":"required","params":[],"url":""},"isReference":false,"order":0,"body":"In order to keep your user data as up-to-date as possible, we recommend a two-step approach using both [Notifications](doc:notifications) and [Batch Queries](doc:batch-queries) \n[block:api-header]\n{\n  \"type\": \"basic\",\n  \"title\": \"Primary Mechanism: Notifications\"\n}\n[/block]\nNotifications are sent whenever a user adds a new data source or has new data available. For this reason, notifications should be the primary mechanism for receiving the latest user data from Human API.  \n\nIn order to receive notifications, you must implement a listener on your server that will receive notifications and subsequently pull the updated data from the appropriate endpoint.\n[block:callout]\n{\n  \"type\": \"info\",\n  \"body\": \"To implement Notifications, see the guide [here](doc:notifications)\"\n}\n[/block]\n\n[block:api-header]\n{\n  \"type\": \"basic\",\n  \"title\": \"Secondary Mechanism: Batch Queries\"\n}\n[/block]\nA [Batch Query](doc:batch-queries) allows you to pull data from a specific endpoint for all of your users at once for a specified time period. We recommend that Batch Queries be implemented at regular intervals to ensure that your user data is up to date in the event that a notification fails to deliver. A good practice is to rely on notifications for the ongoing data synchronization and do a few catch-up synchronizations each 24 hour period.\n\nYou need to keep track of the last query time for your batch queries so that you only query for the data that was updated since the last time you ran the query. \n[block:callout]\n{\n  \"type\": \"warning\",\n  \"body\": \"Please do not perform regular batch queries without the `updated_since` query parameter. By supplying `updated_since` with your last query time, you can ensure that your queries are quick and efficient. You should schedule your queries frequently enough so that you regularly receive a small number of data pages per query.\"\n}\n[/block]\n\n[block:api-header]\n{\n  \"type\": \"basic\",\n  \"title\": \"Making Efficient Queries\"\n}\n[/block]\nFor both the individual and batch endpoints, you can utilize the `updated_since` query parameter to retrieve only data that has been updated since a certain date, often the 'last updated' timestamp for you local app. Doing so can make the synchronization efficient and reliable, and will ensure that you get data that is recently added or changed even it if is older than the most recent data retrieved. \n\nTo do this you should use the `updated_since` query parameter and a date in the format `YYYYMMDDTHHmmssZ`. This is an example query for an individual endpoint:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"curl https://api.humanapi.co/v1/human?...&updated_since=20140110T000000Z\",\n      \"language\": \"curl\"\n    }\n  ]\n}\n[/block]\n\n[block:callout]\n{\n  \"type\": \"info\",\n  \"body\": \"If the queries return more than the default 50 items per page, you should make sure you use the Link Header to retrieve all the pages of data. This can be done by walking through links until there are no more \\\"next\\\" links available.\"\n}\n[/block]\nSee [Patterns and Conventions](doc:patterns-and-conventions-reference) section for more details on using the `updated_since` query parameter and how to process paginated results.","excerpt":"","slug":"data-synchronization","type":"basic","title":"Data Synchronization"}

Data Synchronization


In order to keep your user data as up-to-date as possible, we recommend a two-step approach using both [Notifications](doc:notifications) and [Batch Queries](doc:batch-queries) [block:api-header] { "type": "basic", "title": "Primary Mechanism: Notifications" } [/block] Notifications are sent whenever a user adds a new data source or has new data available. For this reason, notifications should be the primary mechanism for receiving the latest user data from Human API. In order to receive notifications, you must implement a listener on your server that will receive notifications and subsequently pull the updated data from the appropriate endpoint. [block:callout] { "type": "info", "body": "To implement Notifications, see the guide [here](doc:notifications)" } [/block] [block:api-header] { "type": "basic", "title": "Secondary Mechanism: Batch Queries" } [/block] A [Batch Query](doc:batch-queries) allows you to pull data from a specific endpoint for all of your users at once for a specified time period. We recommend that Batch Queries be implemented at regular intervals to ensure that your user data is up to date in the event that a notification fails to deliver. A good practice is to rely on notifications for the ongoing data synchronization and do a few catch-up synchronizations each 24 hour period. You need to keep track of the last query time for your batch queries so that you only query for the data that was updated since the last time you ran the query. [block:callout] { "type": "warning", "body": "Please do not perform regular batch queries without the `updated_since` query parameter. By supplying `updated_since` with your last query time, you can ensure that your queries are quick and efficient. You should schedule your queries frequently enough so that you regularly receive a small number of data pages per query." } [/block] [block:api-header] { "type": "basic", "title": "Making Efficient Queries" } [/block] For both the individual and batch endpoints, you can utilize the `updated_since` query parameter to retrieve only data that has been updated since a certain date, often the 'last updated' timestamp for you local app. Doing so can make the synchronization efficient and reliable, and will ensure that you get data that is recently added or changed even it if is older than the most recent data retrieved. To do this you should use the `updated_since` query parameter and a date in the format `YYYYMMDDTHHmmssZ`. This is an example query for an individual endpoint: [block:code] { "codes": [ { "code": "curl https://api.humanapi.co/v1/human?...&updated_since=20140110T000000Z", "language": "curl" } ] } [/block] [block:callout] { "type": "info", "body": "If the queries return more than the default 50 items per page, you should make sure you use the Link Header to retrieve all the pages of data. This can be done by walking through links until there are no more \"next\" links available." } [/block] See [Patterns and Conventions](doc:patterns-and-conventions-reference) section for more details on using the `updated_since` query parameter and how to process paginated results.