Re: SYNC Module Question

classic Classic list List threaded Threaded
8 messages Options
Dave Thomas Dave Thomas
Reply | Threaded
Open this post in threaded view
|

Re: SYNC Module Question

Hi.  If you're not seeing any new records in sync_server_record on the parent, you're in trouble.  This is the table that is used to keep track of what records have propagated to what child servers.  Without rows being inserted into this table, no data will ever propagate to the children.  In your sync overview page on the parent, can you actually see the child servers, and are they configured correctly?

Did you change the server uuids at all, either on the children or on the parent?

On the parent, the state field in the sync_record table isn't important.  Its the state field in the sync_server_record that governs propagation.

d

On Mon, Apr 2, 2012 at 8:05 AM, Erick Mugoma <[hidden email]> wrote:
Hi Ben and Mark
Am just wondering what may or may not  be happening with the sync_server_record and sync_record tables on the central/parent server.
Plse help me understand something, I expect that every time a new record is created on the parent, the record details are logged to the above tables and you can even use these tables to track if the record was committed, sent, new or failed.  Why is it  that when I do a query to check if there any records in the sync_server_record table I can't see any record for example I should be able to see today's records (02-04-12) since the clean -up was last done on 31 march 2012. Any I deas on what is not Happenning?

On the parent sync_record table, the state field is not being updated. The state column value is 'NEW' for all the records. I expect that the state should change once the record is committed to all the children. Any reason why this is not so?

When I look at the same table on the child servers, the state column value is 'COMMITTED' 
Looking at the History Changes on the UI What I see are old changes like last updated is 28-03-2012. That is last week wed when we have new records being entered even today (02-04-12) and should be able to see them under history changes. Any Explanation why I can't see the newest/latest updates.

Thanks

Erick


[hidden email] from OpenMRS Developers' mailing list
Ben Wolfe (openmrs) Ben Wolfe (openmrs)
Reply | Threaded
Open this post in threaded view
|

Re: SYNC Module Question

Also see https://wiki.openmrs.org/display/docs/Sync+Module+Technical+Overview for a description of the tables.

To add to Dave's questions:
Did you set the "do not sync" objects to anything likw * or org.openmrs.* per chance?

Ben

On Mon, Apr 2, 2012 at 4:16 PM, Dave Thomas <[hidden email]> wrote:
Hi.  If you're not seeing any new records in sync_server_record on the parent, you're in trouble.  This is the table that is used to keep track of what records have propagated to what child servers.  Without rows being inserted into this table, no data will ever propagate to the children.  In your sync overview page on the parent, can you actually see the child servers, and are they configured correctly?

Did you change the server uuids at all, either on the children or on the parent?

On the parent, the state field in the sync_record table isn't important.  Its the state field in the sync_server_record that governs propagation.

d


On Mon, Apr 2, 2012 at 8:05 AM, Erick Mugoma <[hidden email]> wrote:
Hi Ben and Mark
Am just wondering what may or may not  be happening with the sync_server_record and sync_record tables on the central/parent server.
Plse help me understand something, I expect that every time a new record is created on the parent, the record details are logged to the above tables and you can even use these tables to track if the record was committed, sent, new or failed.  Why is it  that when I do a query to check if there any records in the sync_server_record table I can't see any record for example I should be able to see today's records (02-04-12) since the clean -up was last done on 31 march 2012. Any I deas on what is not Happenning?

On the parent sync_record table, the state field is not being updated. The state column value is 'NEW' for all the records. I expect that the state should change once the record is committed to all the children. Any reason why this is not so?

When I look at the same table on the child servers, the state column value is 'COMMITTED' 
Looking at the History Changes on the UI What I see are old changes like last updated is 28-03-2012. That is last week wed when we have new records being entered even today (02-04-12) and should be able to see them under history changes. Any Explanation why I can't see the newest/latest updates.

Thanks

Erick


[hidden email] from OpenMRS Developers' mailing list


[hidden email] from OpenMRS Developers' mailing list
Jim Grace Jim Grace
Reply | Threaded
Open this post in threaded view
|

Re: SYNC Module Question

Hi All,

Thank you Dave, Ben and Mark for the responses so far. If you can help us further to figure out what's going on, we would be grateful. It has us stumped. We're doing a production pilot of Sync from our central server to 2 nearby child sites in the same district, after successful internal testing. One site is synched through a cell modem, and the notebook running OpenMRS at the other site is brought back each night to sync through our LAN. (We expect to deploy both modalities, so we are piloting them both.) If it goes well, we want to roll out Sync (in stages) to our full network consisting of child nodes at our district hospitals and grandchild nodes in the clinics of those districts. We would be replacing Remote Form Entry, which we are currently using.

Everything went fine in both directions between our root parent server and the two production pilot children for over a week. Then the root server mysteriously stopped entering new sync rows in the database, even though it continued to be used heavily. We didn't change anything. Now our children databases are out of sync with the root, and it looks like we have to do a complete backup/restore to get them synchronized again. We would love to expand Sync to all our other sites, but not if we might have more such problems!

Rebooting the parent server made it start synching again perfectly (except that the databases are now out of sync from during the failure period -- so any new data synched from server to child blocks the queue if it is dependent on data that missed being synched.) I don't expect there are any configuration problems since we never changed the configuration between working - not working - working again. We didn't change UUIDs, configuration, not a single thing.

Here is the configuration we have from parent to child (taken from parent server) and child to parent (taken from child server):

[From Parent server] Objects NOT to Send/Export:
org.openmrs.Form
org.openmrs.module.reporting
org.openmrs.scheduler.TaskDefinition
org.openmrs.Cohort
org.openmrs.scheduler
org.openmrs.FormField
org.openmrs.hl7
org.openmrs.Field
org.openmrs.reporting
org.openmrs.logic.token.TokenRegistration
org.openmrs.GlobalProperty
org.openmrs.module.formentry
org.openmrs.module.patientflags
org.openmrs.module.dataintegrity
org.openmrs.module.logic
org.openmrs.api.db.SerializedObject
org.openmrs.notification.Alert
org.openmrs.logic

[From Parent server] Objects NOT to Receive/Import:
org.openmrs.Form
org.openmrs.module.reporting
org.openmrs.scheduler.TaskDefinition
org.openmrs.Cohort
org.openmrs.module.htmlformentry
org.openmrs.scheduler
org.openmrs.FormField
org.openmrs.hl7
org.openmrs.Field
org.openmrs.reporting
org.openmrs.logic.token.TokenRegistration
org.openmrs.GlobalProperty
org.openmrs.Concept
org.openmrs.module.formentry
org.openmrs.module.patientflags
org.openmrs.module.dataintegrity
org.openmrs.module.logic
org.openmrs.api.db.SerializedObject
org.openmrs.notification.Alert
org.openmrs.logic

[From Child server] Objects NOT to Send/Export:
org.openmrs.Field
org.openmrs.Concept
org.openmrs.api.db.SerializedObject
org.openmrs.scheduler
org.openmrs.module.logic
org.openmrs.FormField
org.openmrs.Form
org.openmrs.logic
org.openmrs.module.reporting
org.openmrs.GlobalProperty
org.openmrs.api.db.SerializedObject
org.openmrs.module.htmlformentry
org.openmrs.Cohort
org.openmrs.module.formentry
org.openmrs.logic.token.TokenRegistration
org.openmrs.scheduler.TaskDefinition
org.openmrs.reporting
org.openmrs.hl7
org.openmrs.notification.Alert

[From Child server] Objects NOT to Receive/Import:
org.openmrs.Field
org.openmrs.api.db.SerializedObject
org.openmrs.scheduler
org.openmrs.module.logic
org.openmrs.FormField
org.openmrs.Form
org.openmrs.logic
org.openmrs.module.dataintergrity
org.openmrs.module.patientflags
org.openmrs.module.reporting
org.openmrs.GlobalProperty
org.openmrs.api.db.SerializedObject
org.openmrs.Cohort
org.openmrs.module.formentry
org.openmrs.logic.token.TokenRegistration
org.openmrs.scheduler.TaskDefinition
org.openmrs.reporting
org.openmrs.hl7
org.openmrs.notification.Alert

We can see new sync_record rows created on our parent system until one with record_id 23722, timestamped 2012-03-28 09:28:20. Then there were no more sync_record rows created until after we rebooted. The following catalina.out entries from our parent server seem to show the last two parent sync_records 23721 and 23722 before they stopped, followed by a successful sync operation from the child server Rabuor:

INFO - LoggingAdvice.invoke(109) |2012-03-28 09:29:01,720| In method SyncService.updateSyncRecord. Arguments: SyncRecord=SyncRecord(23721) contains org.openmrs.Privilege,
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,720| Exiting method updateSyncRecord
INFO - LoggingAdvice.invoke(109) |2012-03-28 09:29:01,722| In method SyncService.updateSyncRecord. Arguments: SyncRecord=SyncRecord(23722) contains org.openmrs.Privilege,
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,722| Exiting method updateSyncRecord
INFO - LoggingAdvice.invoke(109) |2012-03-28 09:29:01,724| In method SyncService.saveRemoteServer. Arguments: RemoteServer=RemoteServer(1): Rabuor,
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,726| Exiting method saveRemoteServer
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,847| Exiting method saveEncounter
INFO - LoggingAdvice.invoke(109) |2012-03-28 09:29:01,848| In method HL7Service.encounterCreated. Arguments: Encounter=Encounter: [1171134 Tue Mar 27 00:00:00 EAT 2012 ADULTRETURN null 51216 Revised Adult Follow-Up Form num Obs: [Obs #51749135, Obs #51749164, Obs #51749129, Obs #51749108, Obs #51749111, Obs #51749131, Obs #51749130, Obs #51749125, Obs #51749124, Obs #51749115, Obs #51749126, Obs #51749121, Obs #51749123, Obs #51749119, Obs #51749153, Obs #51749122, Obs #51749148, Obs #51749149, Obs #51749150, Obs #51749151, Obs #51749147, Obs #51749140, Obs #51749141] num Orders: 0 ],
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,849| Exiting method encounterCreated
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,849| Exiting method processHL7Message

I've looked through catalina.out after this point, but can find no clue as to why the sync_record rows stopped being recorded. Is there something in particular I should look for?

Meanwhile on the child servers, new sync_records continued to be entered, and they were all synchronized successfully with the parent. It was only the flow from parent to both children that was interrupted because no new sync_records were being generated by the parent.

When we realized that nothing was moving from parent to child, we checked sync_record and sync_server_record tables and noticed that no new records were being generated by the parent. We verified that all child encounters were present on the parent, but no parent encounters had been transferred to the children after the last sync_record. We restarted the OpenMRS Ubuntu server, and the sync_records started being added again. There has been no problem since in the last few days, except new records that were dependent on unsynchronized objects have also caused problems. New sync_record are now being added again on the parent, starting from just after the system reboot.

We would greatly appreciate any further advice you can give us to understand this problem.

Thanks,
Jim

---------------------------------

At 11:32 PM 02-04-12, Ben Wolfe wrote:
Also see https://wiki.openmrs.org/display/docs/Sync+Module+Technical+Overview for a description of the tables.

To add to Dave's questions:
Did you set the "do not sync" objects to anything likw * or org.openmrs.* per chance?

Ben

On Mon, Apr 2, 2012 at 4:16 PM, Dave Thomas <[hidden email]> wrote:
Hi.  If you're not seeing any new records in sync_server_record on the parent, you're in trouble.  This is the table that is used to keep track of what records have propagated to what child servers.  Without rows being inserted into this table, no data will ever propagate to the children.  In your sync overview page on the parent, can you actually see the child servers, and are they configured correctly?
Did you change the server uuids at all, either on the children or on the parent?
On the parent, the state field in the sync_record table isn't important.  Its the state field in the sync_server_record that governs propagation.

d


On Mon, Apr 2, 2012 at 8:05 AM, Erick Mugoma <[hidden email] > wrote:
Hi Ben and Mark
Am just wondering what may or may not  be happening with the sync_server_record and sync_record tables on the central/parent server.
Plse help me understand something, I expect that every time a new record is created on the parent, the record details are logged to the above tables and you can even use these tables to track if the record was committed, sent, new or failed.  Why is it  that when I do a query to check if there any records in the sync_server_record table I can't see any record for example I should be able to see today's records (02-04-12) since the clean -up was last done on 31 march 2012. Any I deas on what is not Happenning?
On the parent sync_record table, the state field is not being updated. The state column value is 'NEW' for all the records. I expect that the state should change once the record is committed to all the children. Any reason why this is not so?
When I look at the same table on the child servers, the state column value is 'COMMITTED' 
Looking at the History Changes on the UI What I see are old changes like last updated is 28-03-2012. That is last week wed when we have new records being entered even today (02-04-12) and should be able to see them under history changes. Any Explanation why I can't see the newest/latest updates.

Thanks

Erick



[hidden email] from OpenMRS Developers' mailing list



[hidden email] from OpenMRS Developers' mailing list

[hidden email] from OpenMRS Developers' mailing list
Ben Wolfe (openmrs) Ben Wolfe (openmrs)
Reply | Threaded
Open this post in threaded view
|

Re: SYNC Module Question

If sync got turned off somehow and records weren't recorded on parent, then yes, there will be records out of sync and you might need to restore the children from the parent copy.  I'd recommend doing that asap.

I see your copied in your logs from the 28th.  Are there any rows with "ERROR" or "WARN" on them?

Just out of curiosity, what made you choose some of those Objects Not To Sync?  Specifically, why do you have org.openmrs.Form/Field/FormField set to not sync?  Are you moving those records manually?  Are you using formentry?  If so, you just need the org.openms.module.formentry package instead.

org.openmrs.module.htmlformentry should be ok to sync.

Is the cleanup scheduled task still running?  Or was it run manually about the time things stopped being recorded?

Ben

On Thu, Apr 5, 2012 at 4:43 AM, Jim Grace <[hidden email]> wrote:
Hi All,

Thank you Dave, Ben and Mark for the responses so far. If you can help us further to figure out what's going on, we would be grateful. It has us stumped. We're doing a production pilot of Sync from our central server to 2 nearby child sites in the same district, after successful internal testing. One site is synched through a cell modem, and the notebook running OpenMRS at the other site is brought back each night to sync through our LAN. (We expect to deploy both modalities, so we are piloting them both.) If it goes well, we want to roll out Sync (in stages) to our full network consisting of child nodes at our district hospitals and grandchild nodes in the clinics of those districts. We would be replacing Remote Form Entry, which we are currently using.

Everything went fine in both directions between our root parent server and the two production pilot children for over a week. Then the root server mysteriously stopped entering new sync rows in the database, even though it continued to be used heavily. We didn't change anything. Now our children databases are out of sync with the root, and it looks like we have to do a complete backup/restore to get them synchronized again. We would love to expand Sync to all our other sites, but not if we might have more such problems!

Rebooting the parent server made it start synching again perfectly (except that the databases are now out of sync from during the failure period -- so any new data synched from server to child blocks the queue if it is dependent on data that missed being synched.) I don't expect there are any configuration problems since we never changed the configuration between working - not working - working again. We didn't change UUIDs, configuration, not a single thing.

Here is the configuration we have from parent to child (taken from parent server) and child to parent (taken from child server):

[From Parent server] Objects NOT to Send/Export:
org.openmrs.Form
org.openmrs.module.reporting
org.openmrs.scheduler.TaskDefinition
org.openmrs.Cohort
org.openmrs.scheduler
org.openmrs.FormField
org.openmrs.hl7
org.openmrs.Field
org.openmrs.reporting
org.openmrs.logic.token.TokenRegistration
org.openmrs.GlobalProperty
org.openmrs.module.formentry
org.openmrs.module.patientflags
org.openmrs.module.dataintegrity
org.openmrs.module.logic
org.openmrs.api.db.SerializedObject
org.openmrs.notification.Alert
org.openmrs.logic

[From Parent server] Objects NOT to Receive/Import:
org.openmrs.Form
org.openmrs.module.reporting
org.openmrs.scheduler.TaskDefinition
org.openmrs.Cohort
org.openmrs.module.htmlformentry
org.openmrs.scheduler
org.openmrs.FormField
org.openmrs.hl7
org.openmrs.Field
org.openmrs.reporting
org.openmrs.logic.token.TokenRegistration
org.openmrs.GlobalProperty
org.openmrs.Concept
org.openmrs.module.formentry
org.openmrs.module.patientflags
org.openmrs.module.dataintegrity
org.openmrs.module.logic
org.openmrs.api.db.SerializedObject
org.openmrs.notification.Alert
org.openmrs.logic

[From Child server] Objects NOT to Send/Export:
org.openmrs.Field
org.openmrs.Concept
org.openmrs.api.db.SerializedObject
org.openmrs.scheduler
org.openmrs.module.logic
org.openmrs.FormField
org.openmrs.Form
org.openmrs.logic
org.openmrs.module.reporting
org.openmrs.GlobalProperty
org.openmrs.api.db.SerializedObject
org.openmrs.module.htmlformentry
org.openmrs.Cohort
org.openmrs.module.formentry
org.openmrs.logic.token.TokenRegistration
org.openmrs.scheduler.TaskDefinition
org.openmrs.reporting
org.openmrs.hl7
org.openmrs.notification.Alert

[From Child server] Objects NOT to Receive/Import:
org.openmrs.Field
org.openmrs.api.db.SerializedObject
org.openmrs.scheduler
org.openmrs.module.logic
org.openmrs.FormField
org.openmrs.Form
org.openmrs.logic
org.openmrs.module.dataintergrity
org.openmrs.module.patientflags
org.openmrs.module.reporting
org.openmrs.GlobalProperty
org.openmrs.api.db.SerializedObject
org.openmrs.Cohort
org.openmrs.module.formentry
org.openmrs.logic.token.TokenRegistration
org.openmrs.scheduler.TaskDefinition
org.openmrs.reporting
org.openmrs.hl7
org.openmrs.notification.Alert

We can see new sync_record rows created on our parent system until one with record_id 23722, timestamped 2012-03-28 09:28:20. Then there were no more sync_record rows created until after we rebooted. The following catalina.out entries from our parent server seem to show the last two parent sync_records 23721 and 23722 before they stopped, followed by a successful sync operation from the child server Rabuor:

INFO - LoggingAdvice.invoke(109) |2012-03-28 09:29:01,720| In method SyncService.updateSyncRecord. Arguments: SyncRecord=SyncRecord(23721) contains org.openmrs.Privilege,
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,720| Exiting method updateSyncRecord
INFO - LoggingAdvice.invoke(109) |2012-03-28 09:29:01,722| In method SyncService.updateSyncRecord. Arguments: SyncRecord=SyncRecord(23722) contains org.openmrs.Privilege,
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,722| Exiting method updateSyncRecord
INFO - LoggingAdvice.invoke(109) |2012-03-28 09:29:01,724| In method SyncService.saveRemoteServer. Arguments: RemoteServer=RemoteServer(1): Rabuor,
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,726| Exiting method saveRemoteServer
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,847| Exiting method saveEncounter
INFO - LoggingAdvice.invoke(109) |2012-03-28 09:29:01,848| In method HL7Service.encounterCreated. Arguments: Encounter=Encounter: [1171134 Tue Mar 27 00:00:00 EAT 2012 ADULTRETURN null 51216 Revised Adult Follow-Up Form num Obs: [Obs #51749135, Obs #51749164, Obs #51749129, Obs #51749108, Obs #51749111, Obs #51749131, Obs #51749130, Obs #51749125, Obs #51749124, Obs #51749115, Obs #51749126, Obs #51749121, Obs #51749123, Obs #51749119, Obs #51749153, Obs #51749122, Obs #51749148, Obs #51749149, Obs #51749150, Obs #51749151, Obs #51749147, Obs #51749140, Obs #51749141] num Orders: 0 ],
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,849| Exiting method encounterCreated
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,849| Exiting method processHL7Message

I've looked through catalina.out after this point, but can find no clue as to why the sync_record rows stopped being recorded. Is there something in particular I should look for?

Meanwhile on the child servers, new sync_records continued to be entered, and they were all synchronized successfully with the parent. It was only the flow from parent to both children that was interrupted because no new sync_records were being generated by the parent.

When we realized that nothing was moving from parent to child, we checked sync_record and sync_server_record tables and noticed that no new records were being generated by the parent. We verified that all child encounters were present on the parent, but no parent encounters had been transferred to the children after the last sync_record. We restarted the OpenMRS Ubuntu server, and the sync_records started being added again. There has been no problem since in the last few days, except new records that were dependent on unsynchronized objects have also caused problems. New sync_record are now being added again on the parent, starting from just after the system reboot.

We would greatly appreciate any further advice you can give us to understand this problem.

Thanks,
Jim

---------------------------------


At 11:32 PM 02-04-12, Ben Wolfe wrote:
Also see https://wiki.openmrs.org/display/docs/Sync+Module+Technical+Overview for a description of the tables.

To add to Dave's questions:
Did you set the "do not sync" objects to anything likw * or org.openmrs.* per chance?

Ben

On Mon, Apr 2, 2012 at 4:16 PM, Dave Thomas <[hidden email]> wrote:
Hi.  If you're not seeing any new records in sync_server_record on the parent, you're in trouble.  This is the table that is used to keep track of what records have propagated to what child servers.  Without rows being inserted into this table, no data will ever propagate to the children.  In your sync overview page on the parent, can you actually see the child servers, and are they configured correctly?
Did you change the server uuids at all, either on the children or on the parent?
On the parent, the state field in the sync_record table isn't important.  Its the state field in the sync_server_record that governs propagation.

d


On Mon, Apr 2, 2012 at 8:05 AM, Erick Mugoma <[hidden email] > wrote:
Hi Ben and Mark
Am just wondering what may or may not  be happening with the sync_server_record and sync_record tables on the central/parent server.
Plse help me understand something, I expect that every time a new record is created on the parent, the record details are logged to the above tables and you can even use these tables to track if the record was committed, sent, new or failed.  Why is it  that when I do a query to check if there any records in the sync_server_record table I can't see any record for example I should be able to see today's records (02-04-12) since the clean -up was last done on 31 march 2012. Any I deas on what is not Happenning?
On the parent sync_record table, the state field is not being updated. The state column value is 'NEW' for all the records. I expect that the state should change once the record is committed to all the children. Any reason why this is not so?
When I look at the same table on the child servers, the state column value is 'COMMITTED' 
Looking at the History Changes on the UI What I see are old changes like last updated is 28-03-2012. That is last week wed when we have new records being entered even today (02-04-12) and should be able to see them under history changes. Any Explanation why I can't see the newest/latest updates.

Thanks

Erick



[hidden email] from OpenMRS Developers' mailing list



[hidden email] from OpenMRS Developers' mailing list

[hidden email] from OpenMRS Developers' mailing list


[hidden email] from OpenMRS Developers' mailing list
Mark Goodrich-2 Mark Goodrich-2
Reply | Threaded
Open this post in threaded view
|

Re: SYNC Module Question

In reply to this post by Jim Grace

Could you check to see if sync_import records were still being created after 3-28-2012?

 

Could you create a ticket for this bug and upload the entire catalina.out log for 3-28-2012 and I will see if I can identify anything out of the ordinary?  Do you know if you installed any new modules on the parent server at this time?

 

Mark

 

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Jim Grace
Sent: Thursday, April 05, 2012 4:44 AM
To: [hidden email]
Subject: Re: [OPENMRS-DEV] SYNC Module Question

 

Hi All,

Thank you Dave, Ben and Mark for the responses so far. If you can help us further to figure out what's going on, we would be grateful. It has us stumped. We're doing a production pilot of Sync from our central server to 2 nearby child sites in the same district, after successful internal testing. One site is synched through a cell modem, and the notebook running OpenMRS at the other site is brought back each night to sync through our LAN. (We expect to deploy both modalities, so we are piloting them both.) If it goes well, we want to roll out Sync (in stages) to our full network consisting of child nodes at our district hospitals and grandchild nodes in the clinics of those districts. We would be replacing Remote Form Entry, which we are currently using.

Everything went fine in both directions between our root parent server and the two production pilot children for over a week. Then the root server mysteriously stopped entering new sync rows in the database, even though it continued to be used heavily. We didn't change anything. Now our children databases are out of sync with the root, and it looks like we have to do a complete backup/restore to get them synchronized again. We would love to expand Sync to all our other sites, but not if we might have more such problems!

Rebooting the parent server made it start synching again perfectly (except that the databases are now out of sync from during the failure period -- so any new data synched from server to child blocks the queue if it is dependent on data that missed being synched.) I don't expect there are any configuration problems since we never changed the configuration between working - not working - working again. We didn't change UUIDs, configuration, not a single thing.

Here is the configuration we have from parent to child (taken from parent server) and child to parent (taken from child server):

[From Parent server] Objects NOT to Send/Export:
org.openmrs.Form
org.openmrs.module.reporting
org.openmrs.scheduler.TaskDefinition
org.openmrs.Cohort
org.openmrs.scheduler
org.openmrs.FormField
org.openmrs.hl7
org.openmrs.Field
org.openmrs.reporting
org.openmrs.logic.token.TokenRegistration
org.openmrs.GlobalProperty
org.openmrs.module.formentry
org.openmrs.module.patientflags
org.openmrs.module.dataintegrity
org.openmrs.module.logic
org.openmrs.api.db.SerializedObject
org.openmrs.notification.Alert
org.openmrs.logic

[From Parent server] Objects NOT to Receive/Import:
org.openmrs.Form
org.openmrs.module.reporting
org.openmrs.scheduler.TaskDefinition
org.openmrs.Cohort
org.openmrs.module.htmlformentry
org.openmrs.scheduler
org.openmrs.FormField
org.openmrs.hl7
org.openmrs.Field
org.openmrs.reporting
org.openmrs.logic.token.TokenRegistration
org.openmrs.GlobalProperty
org.openmrs.Concept
org.openmrs.module.formentry
org.openmrs.module.patientflags
org.openmrs.module.dataintegrity
org.openmrs.module.logic
org.openmrs.api.db.SerializedObject
org.openmrs.notification.Alert
org.openmrs.logic

[From Child server] Objects NOT to Send/Export:
org.openmrs.Field
org.openmrs.Concept
org.openmrs.api.db.SerializedObject
org.openmrs.scheduler
org.openmrs.module.logic
org.openmrs.FormField
org.openmrs.Form
org.openmrs.logic
org.openmrs.module.reporting
org.openmrs.GlobalProperty
org.openmrs.api.db.SerializedObject
org.openmrs.module.htmlformentry
org.openmrs.Cohort
org.openmrs.module.formentry
org.openmrs.logic.token.TokenRegistration
org.openmrs.scheduler.TaskDefinition
org.openmrs.reporting
org.openmrs.hl7
org.openmrs.notification.Alert

[From Child server] Objects NOT to Receive/Import:
org.openmrs.Field
org.openmrs.api.db.SerializedObject
org.openmrs.scheduler
org.openmrs.module.logic
org.openmrs.FormField
org.openmrs.Form
org.openmrs.logic
org.openmrs.module.dataintergrity
org.openmrs.module.patientflags
org.openmrs.module.reporting
org.openmrs.GlobalProperty
org.openmrs.api.db.SerializedObject
org.openmrs.Cohort
org.openmrs.module.formentry
org.openmrs.logic.token.TokenRegistration
org.openmrs.scheduler.TaskDefinition
org.openmrs.reporting
org.openmrs.hl7
org.openmrs.notification.Alert

We can see new sync_record rows created on our parent system until one with record_id 23722, timestamped 2012-03-28 09:28:20. Then there were no more sync_record rows created until after we rebooted. The following catalina.out entries from our parent server seem to show the last two parent sync_records 23721 and 23722 before they stopped, followed by a successful sync operation from the child server Rabuor:

INFO - LoggingAdvice.invoke(109) |2012-03-28 09:29:01,720| In method SyncService.updateSyncRecord. Arguments: SyncRecord=SyncRecord(23721) contains org.openmrs.Privilege,
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,720| Exiting method updateSyncRecord
INFO - LoggingAdvice.invoke(109) |2012-03-28 09:29:01,722| In method SyncService.updateSyncRecord. Arguments: SyncRecord=SyncRecord(23722) contains org.openmrs.Privilege,
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,722| Exiting method updateSyncRecord
INFO - LoggingAdvice.invoke(109) |2012-03-28 09:29:01,724| In method SyncService.saveRemoteServer. Arguments: RemoteServer=RemoteServer(1): Rabuor,
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,726| Exiting method saveRemoteServer
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,847| Exiting method saveEncounter
INFO - LoggingAdvice.invoke(109) |2012-03-28 09:29:01,848| In method HL7Service.encounterCreated. Arguments: Encounter=Encounter: [1171134 Tue Mar 27 00:00:00 EAT 2012 ADULTRETURN null 51216 Revised Adult Follow-Up Form num Obs: [Obs #51749135, Obs #51749164, Obs #51749129, Obs #51749108, Obs #51749111, Obs #51749131, Obs #51749130, Obs #51749125, Obs #51749124, Obs #51749115, Obs #51749126, Obs #51749121, Obs #51749123, Obs #51749119, Obs #51749153, Obs #51749122, Obs #51749148, Obs #51749149, Obs #51749150, Obs #51749151, Obs #51749147, Obs #51749140, Obs #51749141] num Orders: 0 ],
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,849| Exiting method encounterCreated
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,849| Exiting method processHL7Message

I've looked through catalina.out after this point, but can find no clue as to why the sync_record rows stopped being recorded. Is there something in particular I should look for?

Meanwhile on the child servers, new sync_records continued to be entered, and they were all synchronized successfully with the parent. It was only the flow from parent to both children that was interrupted because no new sync_records were being generated by the parent.

When we realized that nothing was moving from parent to child, we checked sync_record and sync_server_record tables and noticed that no new records were being generated by the parent. We verified that all child encounters were present on the parent, but no parent encounters had been transferred to the children after the last sync_record. We restarted the OpenMRS Ubuntu server, and the sync_records started being added again. There has been no problem since in the last few days, except new records that were dependent on unsynchronized objects have also caused problems. New sync_record are now being added again on the parent, starting from just after the system reboot.

We would greatly appreciate any further advice you can give us to understand this problem.

Thanks,
Jim

---------------------------------

At 11:32 PM 02-04-12, Ben Wolfe wrote:
Also see https://wiki.openmrs.org/display/docs/Sync+Module+Technical+Overview for a description of the tables.

To add to Dave's questions:
Did you set the "do not sync" objects to anything likw * or org.openmrs.* per chance?

Ben

On Mon, Apr 2, 2012 at 4:16 PM, Dave Thomas <[hidden email]> wrote:

Hi.  If you're not seeing any new records in sync_server_record on the parent, you're in trouble.  This is the table that is used to keep track of what records have propagated to what child servers.  Without rows being inserted into this table, no data will ever propagate to the children.  In your sync overview page on the parent, can you actually see the child servers, and are they configured correctly?

Did you change the server uuids at all, either on the children or on the parent?

On the parent, the state field in the sync_record table isn't important.  Its the state field in the sync_server_record that governs propagation.

d

On Mon, Apr 2, 2012 at 8:05 AM, Erick Mugoma <[hidden email] > wrote:

Hi Ben and Mark

Am just wondering what may or may not  be happening with the sync_server_record and sync_record tables on the central/parent server.

Plse help me understand something, I expect that every time a new record is created on the parent, the record details are logged to the above tables and you can even use these tables to track if the record was committed, sent, new or failed.  Why is it  that when I do a query to check if there any records in the sync_server_record table I can't see any record for example I should be able to see today's records (02-04-12) since the clean -up was last done on 31 march 2012. Any I deas on what is not Happenning?

On the parent sync_record table, the state field is not being updated. The state column value is 'NEW' for all the records. I expect that the state should change once the record is committed to all the children. Any reason why this is not so?

When I look at the same table on the child servers, the state column value is 'COMMITTED' 

Looking at the History Changes on the UI What I see are old changes like last updated is 28-03-2012. That is last week wed when we have new records being entered even today (02-04-12) and should be able to see them under history changes. Any Explanation why I can't see the newest/latest updates.

Thanks

Erick


[hidden email] from OpenMRS Developers' mailing list


[hidden email] from OpenMRS Developers' mailing list


[hidden email] from OpenMRS Developers' mailing list


[hidden email] from OpenMRS Developers' mailing list
Dave Thomas Dave Thomas
Reply | Threaded
Open this post in threaded view
|

Re: SYNC Module Question

If the children were successfully propagating data to the parent (can you verify that you can look up data on the parent after sync stopped working that originated from a child server), but no rows were inserted into sync_server_record, i'd guess that this is an issue with core around interceptors failing to register (or be re-registered).  Were there any failed module loads, or module stopping and starting, on the day sync stopped working?

d

On Thu, Apr 5, 2012 at 7:12 AM, Mark Goodrich <[hidden email]> wrote:

Could you check to see if sync_import records were still being created after 3-28-2012?

 

Could you create a ticket for this bug and upload the entire catalina.out log for 3-28-2012 and I will see if I can identify anything out of the ordinary?  Do you know if you installed any new modules on the parent server at this time?

 

Mark

 

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Jim Grace
Sent: Thursday, April 05, 2012 4:44 AM
To: [hidden email]
Subject: Re: [OPENMRS-DEV] SYNC Module Question

 

Hi All,

Thank you Dave, Ben and Mark for the responses so far. If you can help us further to figure out what's going on, we would be grateful. It has us stumped. We're doing a production pilot of Sync from our central server to 2 nearby child sites in the same district, after successful internal testing. One site is synched through a cell modem, and the notebook running OpenMRS at the other site is brought back each night to sync through our LAN. (We expect to deploy both modalities, so we are piloting them both.) If it goes well, we want to roll out Sync (in stages) to our full network consisting of child nodes at our district hospitals and grandchild nodes in the clinics of those districts. We would be replacing Remote Form Entry, which we are currently using.

Everything went fine in both directions between our root parent server and the two production pilot children for over a week. Then the root server mysteriously stopped entering new sync rows in the database, even though it continued to be used heavily. We didn't change anything. Now our children databases are out of sync with the root, and it looks like we have to do a complete backup/restore to get them synchronized again. We would love to expand Sync to all our other sites, but not if we might have more such problems!

Rebooting the parent server made it start synching again perfectly (except that the databases are now out of sync from during the failure period -- so any new data synched from server to child blocks the queue if it is dependent on data that missed being synched.) I don't expect there are any configuration problems since we never changed the configuration between working - not working - working again. We didn't change UUIDs, configuration, not a single thing.

Here is the configuration we have from parent to child (taken from parent server) and child to parent (taken from child server):

[From Parent server] Objects NOT to Send/Export:
org.openmrs.Form
org.openmrs.module.reporting
org.openmrs.scheduler.TaskDefinition
org.openmrs.Cohort
org.openmrs.scheduler
org.openmrs.FormField
org.openmrs.hl7
org.openmrs.Field
org.openmrs.reporting
org.openmrs.logic.token.TokenRegistration
org.openmrs.GlobalProperty
org.openmrs.module.formentry
org.openmrs.module.patientflags
org.openmrs.module.dataintegrity
org.openmrs.module.logic
org.openmrs.api.db.SerializedObject
org.openmrs.notification.Alert
org.openmrs.logic

[From Parent server] Objects NOT to Receive/Import:
org.openmrs.Form
org.openmrs.module.reporting
org.openmrs.scheduler.TaskDefinition
org.openmrs.Cohort
org.openmrs.module.htmlformentry
org.openmrs.scheduler
org.openmrs.FormField
org.openmrs.hl7
org.openmrs.Field
org.openmrs.reporting
org.openmrs.logic.token.TokenRegistration
org.openmrs.GlobalProperty
org.openmrs.Concept
org.openmrs.module.formentry
org.openmrs.module.patientflags
org.openmrs.module.dataintegrity
org.openmrs.module.logic
org.openmrs.api.db.SerializedObject
org.openmrs.notification.Alert
org.openmrs.logic

[From Child server] Objects NOT to Send/Export:
org.openmrs.Field
org.openmrs.Concept
org.openmrs.api.db.SerializedObject
org.openmrs.scheduler
org.openmrs.module.logic
org.openmrs.FormField
org.openmrs.Form
org.openmrs.logic
org.openmrs.module.reporting
org.openmrs.GlobalProperty
org.openmrs.api.db.SerializedObject
org.openmrs.module.htmlformentry
org.openmrs.Cohort
org.openmrs.module.formentry
org.openmrs.logic.token.TokenRegistration
org.openmrs.scheduler.TaskDefinition
org.openmrs.reporting
org.openmrs.hl7
org.openmrs.notification.Alert

[From Child server] Objects NOT to Receive/Import:
org.openmrs.Field
org.openmrs.api.db.SerializedObject
org.openmrs.scheduler
org.openmrs.module.logic
org.openmrs.FormField
org.openmrs.Form
org.openmrs.logic
org.openmrs.module.dataintergrity
org.openmrs.module.patientflags
org.openmrs.module.reporting
org.openmrs.GlobalProperty
org.openmrs.api.db.SerializedObject
org.openmrs.Cohort
org.openmrs.module.formentry
org.openmrs.logic.token.TokenRegistration
org.openmrs.scheduler.TaskDefinition
org.openmrs.reporting
org.openmrs.hl7
org.openmrs.notification.Alert

We can see new sync_record rows created on our parent system until one with record_id 23722, timestamped 2012-03-28 09:28:20. Then there were no more sync_record rows created until after we rebooted. The following catalina.out entries from our parent server seem to show the last two parent sync_records 23721 and 23722 before they stopped, followed by a successful sync operation from the child server Rabuor:

INFO - LoggingAdvice.invoke(109) |2012-03-28 09:29:01,720| In method SyncService.updateSyncRecord. Arguments: SyncRecord=SyncRecord(23721) contains org.openmrs.Privilege,
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,720| Exiting method updateSyncRecord
INFO - LoggingAdvice.invoke(109) |2012-03-28 09:29:01,722| In method SyncService.updateSyncRecord. Arguments: SyncRecord=SyncRecord(23722) contains org.openmrs.Privilege,
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,722| Exiting method updateSyncRecord
INFO - LoggingAdvice.invoke(109) |2012-03-28 09:29:01,724| In method SyncService.saveRemoteServer. Arguments: RemoteServer=RemoteServer(1): Rabuor,
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,726| Exiting method saveRemoteServer
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,847| Exiting method saveEncounter
INFO - LoggingAdvice.invoke(109) |2012-03-28 09:29:01,848| In method HL7Service.encounterCreated. Arguments: Encounter=Encounter: [1171134 Tue Mar 27 00:00:00 EAT 2012 ADULTRETURN null 51216 Revised Adult Follow-Up Form num Obs: [Obs #51749135, Obs #51749164, Obs #51749129, Obs #51749108, Obs #51749111, Obs #51749131, Obs #51749130, Obs #51749125, Obs #51749124, Obs #51749115, Obs #51749126, Obs #51749121, Obs #51749123, Obs #51749119, Obs #51749153, Obs #51749122, Obs #51749148, Obs #51749149, Obs #51749150, Obs #51749151, Obs #51749147, Obs #51749140, Obs #51749141] num Orders: 0 ],
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,849| Exiting method encounterCreated
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,849| Exiting method processHL7Message

I've looked through catalina.out after this point, but can find no clue as to why the sync_record rows stopped being recorded. Is there something in particular I should look for?

Meanwhile on the child servers, new sync_records continued to be entered, and they were all synchronized successfully with the parent. It was only the flow from parent to both children that was interrupted because no new sync_records were being generated by the parent.

When we realized that nothing was moving from parent to child, we checked sync_record and sync_server_record tables and noticed that no new records were being generated by the parent. We verified that all child encounters were present on the parent, but no parent encounters had been transferred to the children after the last sync_record. We restarted the OpenMRS Ubuntu server, and the sync_records started being added again. There has been no problem since in the last few days, except new records that were dependent on unsynchronized objects have also caused problems. New sync_record are now being added again on the parent, starting from just after the system reboot.

We would greatly appreciate any further advice you can give us to understand this problem.

Thanks,
Jim

---------------------------------

At 11:32 PM 02-04-12, Ben Wolfe wrote:
Also see https://wiki.openmrs.org/display/docs/Sync+Module+Technical+Overview for a description of the tables.

To add to Dave's questions:
Did you set the "do not sync" objects to anything likw * or org.openmrs.* per chance?

Ben

On Mon, Apr 2, 2012 at 4:16 PM, Dave Thomas <[hidden email]> wrote:

Hi.  If you're not seeing any new records in sync_server_record on the parent, you're in trouble.  This is the table that is used to keep track of what records have propagated to what child servers.  Without rows being inserted into this table, no data will ever propagate to the children.  In your sync overview page on the parent, can you actually see the child servers, and are they configured correctly?

Did you change the server uuids at all, either on the children or on the parent?

On the parent, the state field in the sync_record table isn't important.  Its the state field in the sync_server_record that governs propagation.

d

On Mon, Apr 2, 2012 at 8:05 AM, Erick Mugoma <[hidden email] > wrote:

Hi Ben and Mark

Am just wondering what may or may not  be happening with the sync_server_record and sync_record tables on the central/parent server.

Plse help me understand something, I expect that every time a new record is created on the parent, the record details are logged to the above tables and you can even use these tables to track if the record was committed, sent, new or failed.  Why is it  that when I do a query to check if there any records in the sync_server_record table I can't see any record for example I should be able to see today's records (02-04-12) since the clean -up was last done on 31 march 2012. Any I deas on what is not Happenning?

On the parent sync_record table, the state field is not being updated. The state column value is 'NEW' for all the records. I expect that the state should change once the record is committed to all the children. Any reason why this is not so?

When I look at the same table on the child servers, the state column value is 'COMMITTED' 

Looking at the History Changes on the UI What I see are old changes like last updated is 28-03-2012. That is last week wed when we have new records being entered even today (02-04-12) and should be able to see them under history changes. Any Explanation why I can't see the newest/latest updates.

Thanks

Erick


[hidden email] from OpenMRS Developers' mailing list


[hidden email] from OpenMRS Developers' mailing list


[hidden email] from OpenMRS Developers' mailing list


[hidden email] from OpenMRS Developers' mailing list


[hidden email] from OpenMRS Developers' mailing list
Mark Goodrich-2 Mark Goodrich-2
Reply | Threaded
Open this post in threaded view
|

Re: SYNC Module Question

In reply to this post by Jim Grace

Jim & Erick—

 

Did you notice Mike’s email to the dev and implementer’s lists yesterday?  We discovered a serious bug where installing the Metadata Sharing module disables the sync interceptor, which would result in behavior like you were seeing.  Do you have the Metadata Sharing module installed?

 

This bug caused us to lose a few days of syncing for our Haiti deployment, though we were able to recover with some quick, hacky code that we wrote… were you able to recover?

 

Take care,

Mark

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Jim Grace
Sent: Thursday, April 05, 2012 4:44 AM
To: [hidden email]
Subject: Re: [OPENMRS-DEV] SYNC Module Question

 

Hi All,

Thank you Dave, Ben and Mark for the responses so far. If you can help us further to figure out what's going on, we would be grateful. It has us stumped. We're doing a production pilot of Sync from our central server to 2 nearby child sites in the same district, after successful internal testing. One site is synched through a cell modem, and the notebook running OpenMRS at the other site is brought back each night to sync through our LAN. (We expect to deploy both modalities, so we are piloting them both.) If it goes well, we want to roll out Sync (in stages) to our full network consisting of child nodes at our district hospitals and grandchild nodes in the clinics of those districts. We would be replacing Remote Form Entry, which we are currently using.

Everything went fine in both directions between our root parent server and the two production pilot children for over a week. Then the root server mysteriously stopped entering new sync rows in the database, even though it continued to be used heavily. We didn't change anything. Now our children databases are out of sync with the root, and it looks like we have to do a complete backup/restore to get them synchronized again. We would love to expand Sync to all our other sites, but not if we might have more such problems!

Rebooting the parent server made it start synching again perfectly (except that the databases are now out of sync from during the failure period -- so any new data synched from server to child blocks the queue if it is dependent on data that missed being synched.) I don't expect there are any configuration problems since we never changed the configuration between working - not working - working again. We didn't change UUIDs, configuration, not a single thing.

Here is the configuration we have from parent to child (taken from parent server) and child to parent (taken from child server):

[From Parent server] Objects NOT to Send/Export:
org.openmrs.Form
org.openmrs.module.reporting
org.openmrs.scheduler.TaskDefinition
org.openmrs.Cohort
org.openmrs.scheduler
org.openmrs.FormField
org.openmrs.hl7
org.openmrs.Field
org.openmrs.reporting
org.openmrs.logic.token.TokenRegistration
org.openmrs.GlobalProperty
org.openmrs.module.formentry
org.openmrs.module.patientflags
org.openmrs.module.dataintegrity
org.openmrs.module.logic
org.openmrs.api.db.SerializedObject
org.openmrs.notification.Alert
org.openmrs.logic

[From Parent server] Objects NOT to Receive/Import:
org.openmrs.Form
org.openmrs.module.reporting
org.openmrs.scheduler.TaskDefinition
org.openmrs.Cohort
org.openmrs.module.htmlformentry
org.openmrs.scheduler
org.openmrs.FormField
org.openmrs.hl7
org.openmrs.Field
org.openmrs.reporting
org.openmrs.logic.token.TokenRegistration
org.openmrs.GlobalProperty
org.openmrs.Concept
org.openmrs.module.formentry
org.openmrs.module.patientflags
org.openmrs.module.dataintegrity
org.openmrs.module.logic
org.openmrs.api.db.SerializedObject
org.openmrs.notification.Alert
org.openmrs.logic

[From Child server] Objects NOT to Send/Export:
org.openmrs.Field
org.openmrs.Concept
org.openmrs.api.db.SerializedObject
org.openmrs.scheduler
org.openmrs.module.logic
org.openmrs.FormField
org.openmrs.Form
org.openmrs.logic
org.openmrs.module.reporting
org.openmrs.GlobalProperty
org.openmrs.api.db.SerializedObject
org.openmrs.module.htmlformentry
org.openmrs.Cohort
org.openmrs.module.formentry
org.openmrs.logic.token.TokenRegistration
org.openmrs.scheduler.TaskDefinition
org.openmrs.reporting
org.openmrs.hl7
org.openmrs.notification.Alert

[From Child server] Objects NOT to Receive/Import:
org.openmrs.Field
org.openmrs.api.db.SerializedObject
org.openmrs.scheduler
org.openmrs.module.logic
org.openmrs.FormField
org.openmrs.Form
org.openmrs.logic
org.openmrs.module.dataintergrity
org.openmrs.module.patientflags
org.openmrs.module.reporting
org.openmrs.GlobalProperty
org.openmrs.api.db.SerializedObject
org.openmrs.Cohort
org.openmrs.module.formentry
org.openmrs.logic.token.TokenRegistration
org.openmrs.scheduler.TaskDefinition
org.openmrs.reporting
org.openmrs.hl7
org.openmrs.notification.Alert

We can see new sync_record rows created on our parent system until one with record_id 23722, timestamped 2012-03-28 09:28:20. Then there were no more sync_record rows created until after we rebooted. The following catalina.out entries from our parent server seem to show the last two parent sync_records 23721 and 23722 before they stopped, followed by a successful sync operation from the child server Rabuor:

INFO - LoggingAdvice.invoke(109) |2012-03-28 09:29:01,720| In method SyncService.updateSyncRecord. Arguments: SyncRecord=SyncRecord(23721) contains org.openmrs.Privilege,
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,720| Exiting method updateSyncRecord
INFO - LoggingAdvice.invoke(109) |2012-03-28 09:29:01,722| In method SyncService.updateSyncRecord. Arguments: SyncRecord=SyncRecord(23722) contains org.openmrs.Privilege,
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,722| Exiting method updateSyncRecord
INFO - LoggingAdvice.invoke(109) |2012-03-28 09:29:01,724| In method SyncService.saveRemoteServer. Arguments: RemoteServer=RemoteServer(1): Rabuor,
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,726| Exiting method saveRemoteServer
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,847| Exiting method saveEncounter
INFO - LoggingAdvice.invoke(109) |2012-03-28 09:29:01,848| In method HL7Service.encounterCreated. Arguments: Encounter=Encounter: [1171134 Tue Mar 27 00:00:00 EAT 2012 ADULTRETURN null 51216 Revised Adult Follow-Up Form num Obs: [Obs #51749135, Obs #51749164, Obs #51749129, Obs #51749108, Obs #51749111, Obs #51749131, Obs #51749130, Obs #51749125, Obs #51749124, Obs #51749115, Obs #51749126, Obs #51749121, Obs #51749123, Obs #51749119, Obs #51749153, Obs #51749122, Obs #51749148, Obs #51749149, Obs #51749150, Obs #51749151, Obs #51749147, Obs #51749140, Obs #51749141] num Orders: 0 ],
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,849| Exiting method encounterCreated
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,849| Exiting method processHL7Message

I've looked through catalina.out after this point, but can find no clue as to why the sync_record rows stopped being recorded. Is there something in particular I should look for?

Meanwhile on the child servers, new sync_records continued to be entered, and they were all synchronized successfully with the parent. It was only the flow from parent to both children that was interrupted because no new sync_records were being generated by the parent.

When we realized that nothing was moving from parent to child, we checked sync_record and sync_server_record tables and noticed that no new records were being generated by the parent. We verified that all child encounters were present on the parent, but no parent encounters had been transferred to the children after the last sync_record. We restarted the OpenMRS Ubuntu server, and the sync_records started being added again. There has been no problem since in the last few days, except new records that were dependent on unsynchronized objects have also caused problems. New sync_record are now being added again on the parent, starting from just after the system reboot.

We would greatly appreciate any further advice you can give us to understand this problem.

Thanks,
Jim

---------------------------------

At 11:32 PM 02-04-12, Ben Wolfe wrote:
Also see https://wiki.openmrs.org/display/docs/Sync+Module+Technical+Overview for a description of the tables.

To add to Dave's questions:
Did you set the "do not sync" objects to anything likw * or org.openmrs.* per chance?

Ben

On Mon, Apr 2, 2012 at 4:16 PM, Dave Thomas <[hidden email]> wrote:

Hi.  If you're not seeing any new records in sync_server_record on the parent, you're in trouble.  This is the table that is used to keep track of what records have propagated to what child servers.  Without rows being inserted into this table, no data will ever propagate to the children.  In your sync overview page on the parent, can you actually see the child servers, and are they configured correctly?

Did you change the server uuids at all, either on the children or on the parent?

On the parent, the state field in the sync_record table isn't important.  Its the state field in the sync_server_record that governs propagation.

d

On Mon, Apr 2, 2012 at 8:05 AM, Erick Mugoma <[hidden email] > wrote:

Hi Ben and Mark

Am just wondering what may or may not  be happening with the sync_server_record and sync_record tables on the central/parent server.

Plse help me understand something, I expect that every time a new record is created on the parent, the record details are logged to the above tables and you can even use these tables to track if the record was committed, sent, new or failed.  Why is it  that when I do a query to check if there any records in the sync_server_record table I can't see any record for example I should be able to see today's records (02-04-12) since the clean -up was last done on 31 march 2012. Any I deas on what is not Happenning?

On the parent sync_record table, the state field is not being updated. The state column value is 'NEW' for all the records. I expect that the state should change once the record is committed to all the children. Any reason why this is not so?

When I look at the same table on the child servers, the state column value is 'COMMITTED' 

Looking at the History Changes on the UI What I see are old changes like last updated is 28-03-2012. That is last week wed when we have new records being entered even today (02-04-12) and should be able to see them under history changes. Any Explanation why I can't see the newest/latest updates.

Thanks

Erick


[hidden email] from OpenMRS Developers' mailing list


[hidden email] from OpenMRS Developers' mailing list


[hidden email] from OpenMRS Developers' mailing list


[hidden email] from OpenMRS Developers' mailing list
Mark Goodrich-2 Mark Goodrich-2
Reply | Threaded
Open this post in threaded view
|

FW: [OPENMRS-DEV] SYNC Module Question

           

Erick,

 

I don’t have any personal experience with that kind of syncing, but it sounds find to me.

 

Mark

 

From: Erick Mugoma [hidden email]
Sent: Sunday, April 15, 2012 12:59 PM
To: Mark Goodrich
Subject: Re: [OPENMRS-DEV] SYNC Module Question

 

Hi Mark,
We don't run meta sharing module.
The parent server started syncing once again and some few records were not synced to the child servers.  Since we manage all the data from one central point, we are still safe. We've also decided to sync the data upstream only that is the from child servers to the parent.
We have blocked the syncing of  obs and encounters downstream that is from parent to child servers.

I there any harm in this approach?

Thanks

Erick.

On Sun, Apr 15, 2012 at 12:34 AM, Mark Goodrich <[hidden email]> wrote:

Jim & Erick—

 

Did you notice Mike’s email to the dev and implementer’s lists yesterday?  We discovered a serious bug where installing the Metadata Sharing module disables the sync interceptor, which would result in behavior like you were seeing.  Do you have the Metadata Sharing module installed?

 

This bug caused us to lose a few days of syncing for our Haiti deployment, though we were able to recover with some quick, hacky code that we wrote… were you able to recover?

 

Take care,

Mark

 

From: [hidden email] [mailto:[hidden email]] On Behalf Of Jim Grace
Sent: Thursday, April 05, 2012 4:44 AM
To: [hidden email]
Subject: Re: [OPENMRS-DEV] SYNC Module Question

 

Hi All,



Thank you Dave, Ben and Mark for the responses so far. If you can help us further to figure out what's going on, we would be grateful. It has us stumped. We're doing a production pilot of Sync from our central server to 2 nearby child sites in the same district, after successful internal testing. One site is synched through a cell modem, and the notebook running OpenMRS at the other site is brought back each night to sync through our LAN. (We expect to deploy both modalities, so we are piloting them both.) If it goes well, we want to roll out Sync (in stages) to our full network consisting of child nodes at our district hospitals and grandchild nodes in the clinics of those districts. We would be replacing Remote Form Entry, which we are currently using.

Everything went fine in both directions between our root parent server and the two production pilot children for over a week. Then the root server mysteriously stopped entering new sync rows in the database, even though it continued to be used heavily. We didn't change anything. Now our children databases are out of sync with the root, and it looks like we have to do a complete backup/restore to get them synchronized again. We would love to expand Sync to all our other sites, but not if we might have more such problems!

Rebooting the parent server made it start synching again perfectly (except that the databases are now out of sync from during the failure period -- so any new data synched from server to child blocks the queue if it is dependent on data that missed being synched.) I don't expect there are any configuration problems since we never changed the configuration between working - not working - working again. We didn't change UUIDs, configuration, not a single thing.

Here is the configuration we have from parent to child (taken from parent server) and child to parent (taken from child server):

[From Parent server] Objects NOT to Send/Export:
org.openmrs.Form
org.openmrs.module.reporting
org.openmrs.scheduler.TaskDefinition
org.openmrs.Cohort
org.openmrs.scheduler
org.openmrs.FormField
org.openmrs.hl7
org.openmrs.Field
org.openmrs.reporting
org.openmrs.logic.token.TokenRegistration
org.openmrs.GlobalProperty
org.openmrs.module.formentry
org.openmrs.module.patientflags
org.openmrs.module.dataintegrity
org.openmrs.module.logic
org.openmrs.api.db.SerializedObject
org.openmrs.notification.Alert
org.openmrs.logic

[From Parent server] Objects NOT to Receive/Import:
org.openmrs.Form
org.openmrs.module.reporting
org.openmrs.scheduler.TaskDefinition
org.openmrs.Cohort
org.openmrs.module.htmlformentry
org.openmrs.scheduler
org.openmrs.FormField
org.openmrs.hl7
org.openmrs.Field
org.openmrs.reporting
org.openmrs.logic.token.TokenRegistration
org.openmrs.GlobalProperty
org.openmrs.Concept
org.openmrs.module.formentry
org.openmrs.module.patientflags
org.openmrs.module.dataintegrity
org.openmrs.module.logic
org.openmrs.api.db.SerializedObject
org.openmrs.notification.Alert
org.openmrs.logic

[From Child server] Objects NOT to Send/Export:
org.openmrs.Field
org.openmrs.Concept
org.openmrs.api.db.SerializedObject
org.openmrs.scheduler
org.openmrs.module.logic
org.openmrs.FormField
org.openmrs.Form
org.openmrs.logic
org.openmrs.module.reporting
org.openmrs.GlobalProperty
org.openmrs.api.db.SerializedObject
org.openmrs.module.htmlformentry
org.openmrs.Cohort
org.openmrs.module.formentry
org.openmrs.logic.token.TokenRegistration
org.openmrs.scheduler.TaskDefinition
org.openmrs.reporting
org.openmrs.hl7
org.openmrs.notification.Alert

[From Child server] Objects NOT to Receive/Import:
org.openmrs.Field
org.openmrs.api.db.SerializedObject
org.openmrs.scheduler
org.openmrs.module.logic
org.openmrs.FormField
org.openmrs.Form
org.openmrs.logic
org.openmrs.module.dataintergrity
org.openmrs.module.patientflags
org.openmrs.module.reporting
org.openmrs.GlobalProperty
org.openmrs.api.db.SerializedObject
org.openmrs.Cohort
org.openmrs.module.formentry
org.openmrs.logic.token.TokenRegistration
org.openmrs.scheduler.TaskDefinition
org.openmrs.reporting
org.openmrs.hl7
org.openmrs.notification.Alert

We can see new sync_record rows created on our parent system until one with record_id 23722, timestamped 2012-03-28 09:28:20. Then there were no more sync_record rows created until after we rebooted. The following catalina.out entries from our parent server seem to show the last two parent sync_records 23721 and 23722 before they stopped, followed by a successful sync operation from the child server Rabuor:

INFO - LoggingAdvice.invoke(109) |2012-03-28 09:29:01,720| In method SyncService.updateSyncRecord. Arguments: SyncRecord=SyncRecord(23721) contains org.openmrs.Privilege,
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,720| Exiting method updateSyncRecord
INFO - LoggingAdvice.invoke(109) |2012-03-28 09:29:01,722| In method SyncService.updateSyncRecord. Arguments: SyncRecord=SyncRecord(23722) contains org.openmrs.Privilege,
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,722| Exiting method updateSyncRecord
INFO - LoggingAdvice.invoke(109) |2012-03-28 09:29:01,724| In method SyncService.saveRemoteServer. Arguments: RemoteServer=RemoteServer(1): Rabuor,
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,726| Exiting method saveRemoteServer
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,847| Exiting method saveEncounter
INFO - LoggingAdvice.invoke(109) |2012-03-28 09:29:01,848| In method HL7Service.encounterCreated. Arguments: Encounter=Encounter: [1171134 Tue Mar 27 00:00:00 EAT 2012 ADULTRETURN null 51216 Revised Adult Follow-Up Form num Obs: [Obs #51749135, Obs #51749164, Obs #51749129, Obs #51749108, Obs #51749111, Obs #51749131, Obs #51749130, Obs #51749125, Obs #51749124, Obs #51749115, Obs #51749126, Obs #51749121, Obs #51749123, Obs #51749119, Obs #51749153, Obs #51749122, Obs #51749148, Obs #51749149, Obs #51749150, Obs #51749151, Obs #51749147, Obs #51749140, Obs #51749141] num Orders: 0 ],
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,849| Exiting method encounterCreated
INFO - LoggingAdvice.invoke(134) |2012-03-28 09:29:01,849| Exiting method processHL7Message

I've looked through catalina.out after this point, but can find no clue as to why the sync_record rows stopped being recorded. Is there something in particular I should look for?

Meanwhile on the child servers, new sync_records continued to be entered, and they were all synchronized successfully with the parent. It was only the flow from parent to both children that was interrupted because no new sync_records were being generated by the parent.

When we realized that nothing was moving from parent to child, we checked sync_record and sync_server_record tables and noticed that no new records were being generated by the parent. We verified that all child encounters were present on the parent, but no parent encounters had been transferred to the children after the last sync_record. We restarted the OpenMRS Ubuntu server, and the sync_records started being added again. There has been no problem since in the last few days, except new records that were dependent on unsynchronized objects have also caused problems. New sync_record are now being added again on the parent, starting from just after the system reboot.

We would greatly appreciate any further advice you can give us to understand this problem.

Thanks,
Jim

---------------------------------

At 11:32 PM 02-04-12, Ben Wolfe wrote:
Also see https://wiki.openmrs.org/display/docs/Sync+Module+Technical+Overview for a description of the tables.

To add to Dave's questions:
Did you set the "do not sync" objects to anything likw * or org.openmrs.* per chance?

Ben

On Mon, Apr 2, 2012 at 4:16 PM, Dave Thomas <[hidden email]> wrote:

Hi.  If you're not seeing any new records in sync_server_record on the parent, you're in trouble.  This is the table that is used to keep track of what records have propagated to what child servers.  Without rows being inserted into this table, no data will ever propagate to the children.  In your sync overview page on the parent, can you actually see the child servers, and are they configured correctly?

Did you change the server uuids at all, either on the children or on the parent?

On the parent, the state field in the sync_record table isn't important.  Its the state field in the sync_server_record that governs propagation.

d

On Mon, Apr 2, 2012 at 8:05 AM, Erick Mugoma <[hidden email] > wrote:

Hi Ben and Mark

Am just wondering what may or may not  be happening with the sync_server_record and sync_record tables on the central/parent server.

Plse help me understand something, I expect that every time a new record is created on the parent, the record details are logged to the above tables and you can even use these tables to track if the record was committed, sent, new or failed.  Why is it  that when I do a query to check if there any records in the sync_server_record table I can't see any record for example I should be able to see today's records (02-04-12) since the clean -up was last done on 31 march 2012. Any I deas on what is not Happenning?

On the parent sync_record table, the state field is not being updated. The state column value is 'NEW' for all the records. I expect that the state should change once the record is committed to all the children. Any reason why this is not so?

When I look at the same table on the child servers, the state column value is 'COMMITTED' 

Looking at the History Changes on the UI What I see are old changes like last updated is 28-03-2012. That is last week wed when we have new records being entered even today (02-04-12) and should be able to see them under history changes. Any Explanation why I can't see the newest/latest updates.

Thanks

Erick


[hidden email] from OpenMRS Developers' mailing list


[hidden email] from OpenMRS Developers' mailing list


[hidden email] from OpenMRS Developers' mailing list

 


[hidden email] from OpenMRS Developers' mailing list