Migrating concept mappings to 1.9

classic Classic list List threaded Threaded
13 messages Options
Rafal Korytkowski-2 Rafal Korytkowski-2
Reply | Threaded
Open this post in threaded view
|

Migrating concept mappings to 1.9

Hi Andy,

We have noticed that MVP uses the comment field in the concept_map table. We are considering using that to determine the right map type in 1.9.

I ran the following query select comment, count(*) from concept_map group by comment; The results are below. I have also added corresponding map types from 1.9, but I am not sure if they match right now. We could correct them if needed.

+----------------------+----------+
| comment              | count(*) |
+----------------------+----------+
| NULL                 |    15516 |
| From Excel           |     2381 |
| From UMLS RxNORM Map |     3010 |
| Map Type: 1          |    46897 | (SAME-AS)
| Map Type: 10         |        1 | (Has specimen)
| Map Type: 17         |        5 | (Direct device)
| Map Type: 19         |        3 | (Direct substance)
| Map Type: 2          |     1880 | (NARROWER-THAN)
| Map Type: 24         |       18 | (Finding method)
| Map Type: 3          |    30841 | (BROADER-THAN)
| Map Type: 4          |      126 | (Associated finding)
| Map Type: 5          |       81 | (Associated morphology)
| Map Type: 6          |       19 | (Associated procedure)
| Map Type: 7          |        2 | (Associated with)
+----------------------+----------+
14 rows in set (2.12 sec)

Here's the proposed migration algorithm:

(1) if the comment matches "Map Type: (\d+)" then use that to determine the map type, and drop it

(2) otherwise move the comment to concept_reference_term.description (even though it doesn't really belong there)

(3) delete duplicate concept_reference_terms (having same source and source_code), though this means we may lose some concept_map.comment data

The reason for these changes is: TRUNK-3296: Found multiple reference terms
-Rafał

[hidden email] from OpenMRS Developers' mailing list
Andrew Kanter Andrew Kanter
Reply | Threaded
Open this post in threaded view
|

Re: Migrating concept mappings to 1.9

Yes, we did that from IMO and I included in that in some... however, it is not at all consistent. Where we have it, we should use it. There shouldn't be dupes with the same map type. I will look through this...

Thanks!
Andy

P.S. Great news about MDS... now just need to fix the concepts :)
 
--------------------
Andrew S. Kanter, MD MPH

- Director of Health Information Systems/Medical Informatics
Millennium Villages Project, Earth Institute, Columbia University
- Asst. Prof. of Clinical Biomedical Informatics and Clinical Epidemiology
Columbia University

Email: [hidden email]
Mobile: +1 (646) 469-2421
Office: +1 (212) 305-4842
Skype: akanter-ippnw
Yahoo: andy_kanter


From: Rafal Korytkowski <[hidden email]>
To: [hidden email]
Sent: Friday, May 4, 2012 11:08 AM
Subject: [OPENMRS-DEV] Migrating concept mappings to 1.9

Hi Andy,

We have noticed that MVP uses the comment field in the concept_map table. We are considering using that to determine the right map type in 1.9.

I ran the following query select comment, count(*) from concept_map group by comment; The results are below. I have also added corresponding map types from 1.9, but I am not sure if they match right now. We could correct them if needed.

+----------------------+----------+
| comment              | count(*) |
+----------------------+----------+
| NULL                 |    15516 |
| From Excel           |     2381 |
| From UMLS RxNORM Map |     3010 |
| Map Type: 1          |    46897 | (SAME-AS)
| Map Type: 10         |        1 | (Has specimen)
| Map Type: 17         |        5 | (Direct device)
| Map Type: 19         |        3 | (Direct substance)
| Map Type: 2          |     1880 | (NARROWER-THAN)
| Map Type: 24         |       18 | (Finding method)
| Map Type: 3          |    30841 | (BROADER-THAN)
| Map Type: 4          |      126 | (Associated finding)
| Map Type: 5          |       81 | (Associated morphology)
| Map Type: 6          |       19 | (Associated procedure)
| Map Type: 7          |        2 | (Associated with)
+----------------------+----------+
14 rows in set (2.12 sec)

Here's the proposed migration algorithm:

(1) if the comment matches "Map Type: (\d+)" then use that to determine the map type, and drop it

(2) otherwise move the comment to concept_reference_term.description (even though it doesn't really belong there)

(3) delete duplicate concept_reference_terms (having same source and source_code), though this means we may lose some concept_map.comment data

The reason for these changes is: TRUNK-3296: Found multiple reference terms
-Rafał

[hidden email] from OpenMRS Developers' mailing list



[hidden email] from OpenMRS Developers' mailing list
Rafal Korytkowski-2 Rafal Korytkowski-2
Reply | Threaded
Open this post in threaded view
|

Re: Migrating concept mappings to 1.9

Thanks Andy! When do you think you'll have time to look into that comments and see if they can be matched with proper mapping types from 1.9? Below is a full list of predefined mapping types in 1.9:

+---------------------+-------------------------------
| concept_map_type_id | name
+---------------------+-------------------------------
|                   1 | SAME-AS
|                   2 | NARROWER-THAN
|                   3 | BROADER-THAN
|                   4 | Associated finding
|                   5 | Associated morphology
|                   6 | Associated procedure
|                   7 | Associated with
|                   8 | Causative agent
|                   9 | Finding site
|                  10 | Has specimen
|                  11 | Laterality
|                  12 | Severity
|                  13 | Access
|                  14 | After
|                  15 | Clinical course
|                  16 | Component
|                  17 | Direct device
|                  18 | Direct morphology
|                  19 | Direct substance
|                  20 | Due to
|                  21 | Episodicity
|                  22 | Finding context
|                  23 | Finding informer
|                  24 | Finding method
|                  25 | Has active ingredient
|                  26 | Has definitional manifestation
|                  27 | Has dose form
|                  28 | Has focus
|                  29 | Has intent
|                  30 | Has interpretation
|                  31 | Indirect device
|                  32 | Indirect morphology
|                  33 | Interprets
|                  34 | Measurement method
|                  35 | Method
|                  36 | Occurrence
|                  37 | Part of
|                  38 | Pathological process
|                  39 | Priority
|                  40 | Procedure context
|                  41 | Procedure device
|                  42 | Procedure morphology
|                  43 | Procedure site
|                  44 | Procedure site - Direct
|                  45 | Procedure site - Indirect
|                  46 | Property
|                  47 | Recipient category
|                  48 | Revision status
|                  49 | Route of administration
|                  50 | Scale type
|                  51 | Specimen procedure
|                  52 | Specimen source identity
|                  53 | Specimen source morphology
|                  54 | Specimen source topography
|                  55 | Specimen substance
|                  56 | Subject of information
|                  57 | Subject relationship context
|                  58 | Surgical approach
|                  59 | Temporal context
|                  60 | Time aspect
|                  61 | Using access device
|                  62 | Using device
|                  63 | Using energy
|                  64 | Using substance
|                  65 | IS A
|                  66 | MAY BE A
|                  67 | MOVED FROM
|                  68 | MOVED TO
|                  69 | REPLACED BY
|                  70 | WAS A
+---------------------+-------------------------------

-Rafał


On 4 May 2012 23:35, Andrew Kanter <[hidden email]> wrote:
Yes, we did that from IMO and I included in that in some... however, it is not at all consistent. Where we have it, we should use it. There shouldn't be dupes with the same map type. I will look through this...

Thanks!
Andy

P.S. Great news about MDS... now just need to fix the concepts :)
 
--------------------
Andrew S. Kanter, MD MPH

- Director of Health Information Systems/Medical Informatics
Millennium Villages Project, Earth Institute, Columbia University
- Asst. Prof. of Clinical Biomedical Informatics and Clinical Epidemiology
Columbia University

Email: [hidden email]
Mobile: <a href="tel:%2B1%20%28646%29%20469-2421" value="+16464692421" target="_blank">+1 (646) 469-2421
Office: <a href="tel:%2B1%20%28212%29%20305-4842" value="+12123054842" target="_blank">+1 (212) 305-4842
Skype: akanter-ippnw
Yahoo: andy_kanter


From: Rafal Korytkowski <[hidden email]>
To: [hidden email]
Sent: Friday, May 4, 2012 11:08 AM
Subject: [OPENMRS-DEV] Migrating concept mappings to 1.9

Hi Andy,

We have noticed that MVP uses the comment field in the concept_map table. We are considering using that to determine the right map type in 1.9.

I ran the following query select comment, count(*) from concept_map group by comment; The results are below. I have also added corresponding map types from 1.9, but I am not sure if they match right now. We could correct them if needed.

+----------------------+----------+
| comment              | count(*) |
+----------------------+----------+
| NULL                 |    15516 |
| From Excel           |     2381 |
| From UMLS RxNORM Map |     3010 |
| Map Type: 1          |    46897 | (SAME-AS)
| Map Type: 10         |        1 | (Has specimen)
| Map Type: 17         |        5 | (Direct device)
| Map Type: 19         |        3 | (Direct substance)
| Map Type: 2          |     1880 | (NARROWER-THAN)
| Map Type: 24         |       18 | (Finding method)
| Map Type: 3          |    30841 | (BROADER-THAN)
| Map Type: 4          |      126 | (Associated finding)
| Map Type: 5          |       81 | (Associated morphology)
| Map Type: 6          |       19 | (Associated procedure)
| Map Type: 7          |        2 | (Associated with)
+----------------------+----------+
14 rows in set (2.12 sec)

Here's the proposed migration algorithm:

(1) if the comment matches "Map Type: (\d+)" then use that to determine the map type, and drop it

(2) otherwise move the comment to concept_reference_term.description (even though it doesn't really belong there)

(3) delete duplicate concept_reference_terms (having same source and source_code), though this means we may lose some concept_map.comment data

The reason for these changes is: TRUNK-3296: Found multiple reference terms
-Rafał

[hidden email] from OpenMRS Developers' mailing list



[hidden email] from OpenMRS Developers' mailing list


[hidden email] from OpenMRS Developers' mailing list
Andrew Kanter Andrew Kanter
Reply | Threaded
Open this post in threaded view
|

Re: Migrating concept mappings to 1.9

Rafal,

I am confused... the mapping below is correct. For the plan, I would do 1, not 2 (just ignore the comment) I would make these probably SAME AS for "from excel and from UMLS". For 3, I presume we will keep one of the duplicate maps...and if I had to choose a map type (if there is no map id:) then I would pick NARROWER-THAN, but perhaps I should review the list of dupes first.
 
--------------------
Andrew S. Kanter, MD MPH

- Director of Health Information Systems/Medical Informatics
Millennium Villages Project, Earth Institute, Columbia University
- Asst. Prof. of Clinical Biomedical Informatics and Clinical Epidemiology
Columbia University

Email: [hidden email]
Mobile: +1 (646) 469-2421
Office: +1 (212) 305-4842
Skype: akanter-ippnw
Yahoo: andy_kanter


From: Rafal Korytkowski <[hidden email]>
To: [hidden email]
Sent: Monday, May 7, 2012 11:04 AM
Subject: Re: [OPENMRS-DEV] Migrating concept mappings to 1.9

Thanks Andy! When do you think you'll have time to look into that comments and see if they can be matched with proper mapping types from 1.9? Below is a full list of predefined mapping types in 1.9:

+---------------------+-------------------------------
| concept_map_type_id | name
+---------------------+-------------------------------
|                   1 | SAME-AS = 1
|                   2 | NARROWER-THAN = 2
|                   3 | BROADER-THAN = 3
|                   4 | Associated finding = 4
|                   5 | Associated morphology = 5
|                   6 | Associated procedure = 6
|                   7 | Associated with = 7
|                   8 | Causative agent
|                   9 | Finding site
|                  10 | Has specimen = 10 
|                  11 | Laterality
|                  12 | Severity
|                  13 | Access
|                  14 | After
|                  15 | Clinical course
|                  16 | Component
|                  17 | Direct device
|                  18 | Direct morphology
|                  19 | Direct substance
|                  20 | Due to
|                  21 | Episodicity
|                  22 | Finding context
|                  23 | Finding informer
|                  24 | Finding method
|                  25 | Has active ingredient
|                  26 | Has definitional manifestation
|                  27 | Has dose form
|                  28 | Has focus
|                  29 | Has intent
|                  30 | Has interpretation
|                  31 | Indirect device
|                  32 | Indirect morphology
|                  33 | Interprets
|                  34 | Measurement method
|                  35 | Method
|                  36 | Occurrence
|                  37 | Part of
|                  38 | Pathological process
|                  39 | Priority
|                  40 | Procedure context
|                  41 | Procedure device
|                  42 | Procedure morphology
|                  43 | Procedure site
|                  44 | Procedure site - Direct
|                  45 | Procedure site - Indirect
|                  46 | Property
|                  47 | Recipient category
|                  48 | Revision status
|                  49 | Route of administration
|                  50 | Scale type
|                  51 | Specimen procedure
|                  52 | Specimen source identity
|                  53 | Specimen source morphology
|                  54 | Specimen source topography
|                  55 | Specimen substance
|                  56 | Subject of information
|                  57 | Subject relationship context
|                  58 | Surgical approach
|                  59 | Temporal context
|                  60 | Time aspect
|                  61 | Using access device
|                  62 | Using device
|                  63 | Using energy
|                  64 | Using substance
|                  65 | IS A
|                  66 | MAY BE A
|                  67 | MOVED FROM
|                  68 | MOVED TO
|                  69 | REPLACED BY
|                  70 | WAS A
+---------------------+-------------------------------

-Rafał


On 4 May 2012 23:35, Andrew Kanter <[hidden email]> wrote:
Yes, we did that from IMO and I included in that in some... however, it is not at all consistent. Where we have it, we should use it. There shouldn't be dupes with the same map type. I will look through this...

Thanks!
Andy

P.S. Great news about MDS... now just need to fix the concepts :)
 
--------------------
Andrew S. Kanter, MD MPH

- Director of Health Information Systems/Medical Informatics
Millennium Villages Project, Earth Institute, Columbia University
- Asst. Prof. of Clinical Biomedical Informatics and Clinical Epidemiology
Columbia University

Email: [hidden email]
Mobile: +1 (646) 469-2421
Office: +1 (212) 305-4842
Skype: akanter-ippnw
Yahoo: andy_kanter


From: Rafal Korytkowski <[hidden email]>
To: [hidden email]
Sent: Friday, May 4, 2012 11:08 AM
Subject: [OPENMRS-DEV] Migrating concept mappings to 1.9

Hi Andy,

We have noticed that MVP uses the comment field in the concept_map table. We are considering using that to determine the right map type in 1.9.

I ran the following query select comment, count(*) from concept_map group by comment; The results are below. I have also added corresponding map types from 1.9, but I am not sure if they match right now. We could correct them if needed.

+----------------------+----------+
| comment              | count(*) |
+----------------------+----------+
| NULL                 |    15516 |
| From Excel           |     2381 |
| From UMLS RxNORM Map |     3010 |
| Map Type: 1          |    46897 | (SAME-AS)
| Map Type: 10         |        1 | (Has specimen)
| Map Type: 17         |        5 | (Direct device)
| Map Type: 19         |        3 | (Direct substance)
| Map Type: 2          |     1880 | (NARROWER-THAN)
| Map Type: 24         |       18 | (Finding method)
| Map Type: 3          |    30841 | (BROADER-THAN)
| Map Type: 4          |      126 | (Associated finding)
| Map Type: 5          |       81 | (Associated morphology)
| Map Type: 6          |       19 | (Associated procedure)
| Map Type: 7          |        2 | (Associated with)
+----------------------+----------+
14 rows in set (2.12 sec)

Here's the proposed migration algorithm:

(1) if the comment matches "Map Type: (\d+)" then use that to determine the map type, and drop it

(2) otherwise move the comment to concept_reference_term.description (even though it doesn't really belong there)

(3) delete duplicate concept_reference_terms (having same source and source_code), though this means we may lose some concept_map.comment data

The reason for these changes is: TRUNK-3296: Found multiple reference terms
-Rafał

[hidden email] from OpenMRS Developers' mailing list



[hidden email] from OpenMRS Developers' mailing list


[hidden email] from OpenMRS Developers' mailing list



[hidden email] from OpenMRS Developers' mailing list
Rafal Korytkowski-2 Rafal Korytkowski-2
Reply | Threaded
Open this post in threaded view
|

Re: Migrating concept mappings to 1.9

To be clear "duplicate" mappings we ran into were cases where same source and source code were used to map more than one concept. It is not wrong, but we missed to notice that while writing the migration script. There is no need for you Andy to fix them.

Just for the record there are 44737 such mappings in MVP. I determined that with the following query:

select sum(c.sameCount) from (select count(*) as sameCount from concept_map group by source_code, source having count(*) > 1) as c;

We'll convert all MVP comments to map types as follows and then we'll drop the comments.

| Comment             | Map type
------------------------------------------------------------
| NULL                   | NARROWER-THAN
| From Excel          | SAME-AS
| From UMLS RxNORM Map | SAME-AS
| Map Type: 1          | SAME-AS
| Map Type: 10         | Has specimen
| Map Type: 17         | Direct device ??? 
| Map Type: 19         | Direct substance ???
| Map Type: 2          | NARROWER-THAN
| Map Type: 24         | Finding method ???
| Map Type: 3          | BROADER-THAN
| Map Type: 4          | Associated finding
| Map Type: 5          | Associated morphology
| Map Type: 6          | Associated procedure
| Map Type: 7          | Associated with

Andy, there are 3 map types left with "???". Map Type 17, 19 an 24. Can you confirm them?

I'm still wondering if we should move all other comments to the description field in concept_reference_term. I think it's better to leave the comment column as is in concept_reference_map and only deprecate it and mark to be removed in 1.10 since we can't really migrate it correctly.

-Rafał

On 7 May 2012 21:38, Andrew Kanter <[hidden email]> wrote:
Rafal,

I am confused... the mapping below is correct. For the plan, I would do 1, not 2 (just ignore the comment) I would make these probably SAME AS for "from excel and from UMLS". For 3, I presume we will keep one of the duplicate maps...and if I had to choose a map type (if there is no map id:) then I would pick NARROWER-THAN, but perhaps I should review the list of dupes first.
 
--------------------
Andrew S. Kanter, MD MPH

- Director of Health Information Systems/Medical Informatics
Millennium Villages Project, Earth Institute, Columbia University
- Asst. Prof. of Clinical Biomedical Informatics and Clinical Epidemiology
Columbia University

Email: [hidden email]
Mobile: <a href="tel:%2B1%20%28646%29%20469-2421" value="+16464692421" target="_blank">+1 (646) 469-2421
Office: <a href="tel:%2B1%20%28212%29%20305-4842" value="+12123054842" target="_blank">+1 (212) 305-4842
Skype: akanter-ippnw
Yahoo: andy_kanter


From: Rafal Korytkowski <[hidden email]>
To: [hidden email]
Sent: Monday, May 7, 2012 11:04 AM
Subject: Re: [OPENMRS-DEV] Migrating concept mappings to 1.9

Thanks Andy! When do you think you'll have time to look into that comments and see if they can be matched with proper mapping types from 1.9? Below is a full list of predefined mapping types in 1.9:

+---------------------+-------------------------------
| concept_map_type_id | name
+---------------------+-------------------------------
|                   1 | SAME-AS = 1
|                   2 | NARROWER-THAN = 2
|                   3 | BROADER-THAN = 3
|                   4 | Associated finding = 4
|                   5 | Associated morphology = 5
|                   6 | Associated procedure = 6
|                   7 | Associated with = 7
|                   8 | Causative agent
|                   9 | Finding site
|                  10 | Has specimen = 10 
|                  11 | Laterality
|                  12 | Severity
|                  13 | Access
|                  14 | After
|                  15 | Clinical course
|                  16 | Component
|                  17 | Direct device
|                  18 | Direct morphology
|                  19 | Direct substance
|                  20 | Due to
|                  21 | Episodicity
|                  22 | Finding context
|                  23 | Finding informer
|                  24 | Finding method
|                  25 | Has active ingredient
|                  26 | Has definitional manifestation
|                  27 | Has dose form
|                  28 | Has focus
|                  29 | Has intent
|                  30 | Has interpretation
|                  31 | Indirect device
|                  32 | Indirect morphology
|                  33 | Interprets
|                  34 | Measurement method
|                  35 | Method
|                  36 | Occurrence
|                  37 | Part of
|                  38 | Pathological process
|                  39 | Priority
|                  40 | Procedure context
|                  41 | Procedure device
|                  42 | Procedure morphology
|                  43 | Procedure site
|                  44 | Procedure site - Direct
|                  45 | Procedure site - Indirect
|                  46 | Property
|                  47 | Recipient category
|                  48 | Revision status
|                  49 | Route of administration
|                  50 | Scale type
|                  51 | Specimen procedure
|                  52 | Specimen source identity
|                  53 | Specimen source morphology
|                  54 | Specimen source topography
|                  55 | Specimen substance
|                  56 | Subject of information
|                  57 | Subject relationship context
|                  58 | Surgical approach
|                  59 | Temporal context
|                  60 | Time aspect
|                  61 | Using access device
|                  62 | Using device
|                  63 | Using energy
|                  64 | Using substance
|                  65 | IS A
|                  66 | MAY BE A
|                  67 | MOVED FROM
|                  68 | MOVED TO
|                  69 | REPLACED BY
|                  70 | WAS A
+---------------------+-------------------------------

-Rafał


On 4 May 2012 23:35, Andrew Kanter <[hidden email]> wrote:
Yes, we did that from IMO and I included in that in some... however, it is not at all consistent. Where we have it, we should use it. There shouldn't be dupes with the same map type. I will look through this...

Thanks!
Andy

P.S. Great news about MDS... now just need to fix the concepts :)
 
--------------------
Andrew S. Kanter, MD MPH

- Director of Health Information Systems/Medical Informatics
Millennium Villages Project, Earth Institute, Columbia University
- Asst. Prof. of Clinical Biomedical Informatics and Clinical Epidemiology
Columbia University

Email: [hidden email]
Mobile: +1 (646) 469-2421
Office: +1 (212) 305-4842
Skype: akanter-ippnw
Yahoo: andy_kanter


From: Rafal Korytkowski <[hidden email]>
To: [hidden email]
Sent: Friday, May 4, 2012 11:08 AM
Subject: [OPENMRS-DEV] Migrating concept mappings to 1.9

Hi Andy,

We have noticed that MVP uses the comment field in the concept_map table. We are considering using that to determine the right map type in 1.9.

I ran the following query select comment, count(*) from concept_map group by comment; The results are below. I have also added corresponding map types from 1.9, but I am not sure if they match right now. We could correct them if needed.

+----------------------+----------+
| comment              | count(*) |
+----------------------+----------+
| NULL                 |    15516 |
| From Excel           |     2381 |
| From UMLS RxNORM Map |     3010 |
| Map Type: 1          |    46897 | (SAME-AS)
| Map Type: 10         |        1 | (Has specimen)
| Map Type: 17         |        5 | (Direct device)
| Map Type: 19         |        3 | (Direct substance)
| Map Type: 2          |     1880 | (NARROWER-THAN)
| Map Type: 24         |       18 | (Finding method)
| Map Type: 3          |    30841 | (BROADER-THAN)
| Map Type: 4          |      126 | (Associated finding)
| Map Type: 5          |       81 | (Associated morphology)
| Map Type: 6          |       19 | (Associated procedure)
| Map Type: 7          |        2 | (Associated with)
+----------------------+----------+
14 rows in set (2.12 sec)

Here's the proposed migration algorithm:

(1) if the comment matches "Map Type: (\d+)" then use that to determine the map type, and drop it

(2) otherwise move the comment to concept_reference_term.description (even though it doesn't really belong there)

(3) delete duplicate concept_reference_terms (having same source and source_code), though this means we may lose some concept_map.comment data

The reason for these changes is: TRUNK-3296: Found multiple reference terms
-Rafał

[hidden email] from OpenMRS Developers' mailing list



[hidden email] from OpenMRS Developers' mailing list


[hidden email] from OpenMRS Developers' mailing list



[hidden email] from OpenMRS Developers' mailing list


[hidden email] from OpenMRS Developers' mailing list
Ben Wolfe (openmrs) Ben Wolfe (openmrs)
Reply | Threaded
Open this post in threaded view
|

Re: Migrating concept mappings to 1.9

Andy, in addition to those 3 map types, can you confirm that the comments are or are not important to you?  Some of the similar mappings have different comments, so we'd have to choose randomly which one to keep.

Ben

On Mon, May 7, 2012 at 5:31 PM, Rafal Korytkowski <[hidden email]> wrote:
To be clear "duplicate" mappings we ran into were cases where same source and source code were used to map more than one concept. It is not wrong, but we missed to notice that while writing the migration script. There is no need for you Andy to fix them.

Just for the record there are 44737 such mappings in MVP. I determined that with the following query:

select sum(c.sameCount) from (select count(*) as sameCount from concept_map group by source_code, source having count(*) > 1) as c;

We'll convert all MVP comments to map types as follows and then we'll drop the comments.

| Comment             | Map type
------------------------------------------------------------
| NULL                   | NARROWER-THAN
| From Excel          | SAME-AS
| From UMLS RxNORM Map | SAME-AS
| Map Type: 1          | SAME-AS
| Map Type: 10         | Has specimen
| Map Type: 17         | Direct device ??? 
| Map Type: 19         | Direct substance ???
| Map Type: 2          | NARROWER-THAN
| Map Type: 24         | Finding method ???
| Map Type: 3          | BROADER-THAN
| Map Type: 4          | Associated finding
| Map Type: 5          | Associated morphology
| Map Type: 6          | Associated procedure
| Map Type: 7          | Associated with

Andy, there are 3 map types left with "???". Map Type 17, 19 an 24. Can you confirm them?

I'm still wondering if we should move all other comments to the description field in concept_reference_term. I think it's better to leave the comment column as is in concept_reference_map and only deprecate it and mark to be removed in 1.10 since we can't really migrate it correctly.

-Rafał


On 7 May 2012 21:38, Andrew Kanter <[hidden email]> wrote:
Rafal,

I am confused... the mapping below is correct. For the plan, I would do 1, not 2 (just ignore the comment) I would make these probably SAME AS for "from excel and from UMLS". For 3, I presume we will keep one of the duplicate maps...and if I had to choose a map type (if there is no map id:) then I would pick NARROWER-THAN, but perhaps I should review the list of dupes first.
 
--------------------
Andrew S. Kanter, MD MPH

- Director of Health Information Systems/Medical Informatics
Millennium Villages Project, Earth Institute, Columbia University
- Asst. Prof. of Clinical Biomedical Informatics and Clinical Epidemiology
Columbia University

Email: [hidden email]
Mobile: <a href="tel:%2B1%20%28646%29%20469-2421" value="+16464692421" target="_blank">+1 (646) 469-2421
Office: <a href="tel:%2B1%20%28212%29%20305-4842" value="+12123054842" target="_blank">+1 (212) 305-4842
Skype: akanter-ippnw
Yahoo: andy_kanter


From: Rafal Korytkowski <[hidden email]>
To: [hidden email]
Sent: Monday, May 7, 2012 11:04 AM
Subject: Re: [OPENMRS-DEV] Migrating concept mappings to 1.9

Thanks Andy! When do you think you'll have time to look into that comments and see if they can be matched with proper mapping types from 1.9? Below is a full list of predefined mapping types in 1.9:

+---------------------+-------------------------------
| concept_map_type_id | name
+---------------------+-------------------------------
|                   1 | SAME-AS = 1
|                   2 | NARROWER-THAN = 2
|                   3 | BROADER-THAN = 3
|                   4 | Associated finding = 4
|                   5 | Associated morphology = 5
|                   6 | Associated procedure = 6
|                   7 | Associated with = 7
|                   8 | Causative agent
|                   9 | Finding site
|                  10 | Has specimen = 10 
|                  11 | Laterality
|                  12 | Severity
|                  13 | Access
|                  14 | After
|                  15 | Clinical course
|                  16 | Component
|                  17 | Direct device
|                  18 | Direct morphology
|                  19 | Direct substance
|                  20 | Due to
|                  21 | Episodicity
|                  22 | Finding context
|                  23 | Finding informer
|                  24 | Finding method
|                  25 | Has active ingredient
|                  26 | Has definitional manifestation
|                  27 | Has dose form
|                  28 | Has focus
|                  29 | Has intent
|                  30 | Has interpretation
|                  31 | Indirect device
|                  32 | Indirect morphology
|                  33 | Interprets
|                  34 | Measurement method
|                  35 | Method
|                  36 | Occurrence
|                  37 | Part of
|                  38 | Pathological process
|                  39 | Priority
|                  40 | Procedure context
|                  41 | Procedure device
|                  42 | Procedure morphology
|                  43 | Procedure site
|                  44 | Procedure site - Direct
|                  45 | Procedure site - Indirect
|                  46 | Property
|                  47 | Recipient category
|                  48 | Revision status
|                  49 | Route of administration
|                  50 | Scale type
|                  51 | Specimen procedure
|                  52 | Specimen source identity
|                  53 | Specimen source morphology
|                  54 | Specimen source topography
|                  55 | Specimen substance
|                  56 | Subject of information
|                  57 | Subject relationship context
|                  58 | Surgical approach
|                  59 | Temporal context
|                  60 | Time aspect
|                  61 | Using access device
|                  62 | Using device
|                  63 | Using energy
|                  64 | Using substance
|                  65 | IS A
|                  66 | MAY BE A
|                  67 | MOVED FROM
|                  68 | MOVED TO
|                  69 | REPLACED BY
|                  70 | WAS A
+---------------------+-------------------------------

-Rafał


On 4 May 2012 23:35, Andrew Kanter <[hidden email]> wrote:
Yes, we did that from IMO and I included in that in some... however, it is not at all consistent. Where we have it, we should use it. There shouldn't be dupes with the same map type. I will look through this...

Thanks!
Andy

P.S. Great news about MDS... now just need to fix the concepts :)
 
--------------------
Andrew S. Kanter, MD MPH

- Director of Health Information Systems/Medical Informatics
Millennium Villages Project, Earth Institute, Columbia University
- Asst. Prof. of Clinical Biomedical Informatics and Clinical Epidemiology
Columbia University

Email: [hidden email]
Mobile: +1 (646) 469-2421
Office: +1 (212) 305-4842
Skype: akanter-ippnw
Yahoo: andy_kanter


From: Rafal Korytkowski <[hidden email]>
To: [hidden email]
Sent: Friday, May 4, 2012 11:08 AM
Subject: [OPENMRS-DEV] Migrating concept mappings to 1.9

Hi Andy,

We have noticed that MVP uses the comment field in the concept_map table. We are considering using that to determine the right map type in 1.9.

I ran the following query select comment, count(*) from concept_map group by comment; The results are below. I have also added corresponding map types from 1.9, but I am not sure if they match right now. We could correct them if needed.

+----------------------+----------+
| comment              | count(*) |
+----------------------+----------+
| NULL                 |    15516 |
| From Excel           |     2381 |
| From UMLS RxNORM Map |     3010 |
| Map Type: 1          |    46897 | (SAME-AS)
| Map Type: 10         |        1 | (Has specimen)
| Map Type: 17         |        5 | (Direct device)
| Map Type: 19         |        3 | (Direct substance)
| Map Type: 2          |     1880 | (NARROWER-THAN)
| Map Type: 24         |       18 | (Finding method)
| Map Type: 3          |    30841 | (BROADER-THAN)
| Map Type: 4          |      126 | (Associated finding)
| Map Type: 5          |       81 | (Associated morphology)
| Map Type: 6          |       19 | (Associated procedure)
| Map Type: 7          |        2 | (Associated with)
+----------------------+----------+
14 rows in set (2.12 sec)

Here's the proposed migration algorithm:

(1) if the comment matches "Map Type: (\d+)" then use that to determine the map type, and drop it

(2) otherwise move the comment to concept_reference_term.description (even though it doesn't really belong there)

(3) delete duplicate concept_reference_terms (having same source and source_code), though this means we may lose some concept_map.comment data

The reason for these changes is: TRUNK-3296: Found multiple reference terms
-Rafał

[hidden email] from OpenMRS Developers' mailing list



[hidden email] from OpenMRS Developers' mailing list


[hidden email] from OpenMRS Developers' mailing list



[hidden email] from OpenMRS Developers' mailing list


[hidden email] from OpenMRS Developers' mailing list


[hidden email] from OpenMRS Developers' mailing list
Andrew Kanter Andrew Kanter
Reply | Threaded
Open this post in threaded view
|

Re: Migrating concept mappings to 1.9

In reply to this post by Rafal Korytkowski-2
Folks,
I don't think the original comments had anything to do with the list which appears in OpenMRS now for mapping sources... so the original map presented was definitely wrong. Please see the bottom of this email for corrected maps from the existing comments to the new map types.


Andy
 
--------------------
Andrew S. Kanter, MD MPH

Asst. Prof. of Clinical Biomedical Informatics and Clinical Epidemiology
Columbia University
Email: [hidden email]
Mobile: +1 (646) 469-2421
Office: +1 (212) 305-4842
Skype: akanter-ippnw
Yahoo: andy_kanter


From: Rafal Korytkowski <[hidden email]>
To: [hidden email]
Sent: Monday, May 7, 2012 11:04 AM
Subject: Re: [OPENMRS-DEV] Migrating concept mappings to 1.9

Thanks Andy! When do you think you'll have time to look into that comments and see if they can be matched with proper mapping types from 1.9? Below is a full list of predefined mapping types in 1.9:

+---------------------+-------------------------------
| concept_map_type_id | name
+---------------------+-------------------------------
|                   1 | SAME-AS  =
|                   2 | NARROWER-THAN
|                   3 | BROADER-THAN
|                   4 | Associated finding
|                   5 | Associated morphology
|                   6 | Associated procedure
|                   7 | Associated with
|                   8 | Causative agent
|                   9 | Finding site
|                  10 | Has specimen
|                  11 | Laterality
|                  12 | Severity
|                  13 | Access
|                  14 | After
|                  15 | Clinical course
|                  16 | Component
|                  17 | Direct device
|                  18 | Direct morphology
|                  19 | Direct substance
|                  20 | Due to
|                  21 | Episodicity
|                  22 | Finding context
|                  23 | Finding informer
|                  24 | Finding method
|                  25 | Has active ingredient
|                  26 | Has definitional manifestation
|                  27 | Has dose form
|                  28 | Has focus
|                  29 | Has intent
|                  30 | Has interpretation
|                  31 | Indirect device
|                  32 | Indirect morphology
|                  33 | Interprets
|                  34 | Measurement method
|                  35 | Method
|                  36 | Occurrence
|                  37 | Part of
|                  38 | Pathological process
|                  39 | Priority
|                  40 | Procedure context
|                  41 | Procedure device
|                  42 | Procedure morphology
|                  43 | Procedure site
|                  44 | Procedure site - Direct
|                  45 | Procedure site - Indirect
|                  46 | Property
|                  47 | Recipient category
|                  48 | Revision status
|                  49 | Route of administration
|                  50 | Scale type
|                  51 | Specimen procedure
|                  52 | Specimen source identity
|                  53 | Specimen source morphology
|                  54 | Specimen source topography
|                  55 | Specimen substance
|                  56 | Subject of information
|                  57 | Subject relationship context
|                  58 | Surgical approach
|                  59 | Temporal context
|                  60 | Time aspect
|                  61 | Using access device
|                  62 | Using device
|                  63 | Using energy
|                  64 | Using substance
|                  65 | IS A
|                  66 | MAY BE A
|                  67 | MOVED FROM
|                  68 | MOVED TO
|                  69 | REPLACED BY
|                  70 | WAS A
+---------------------+-------------------------------

-Rafał


On 4 May 2012 23:35, Andrew Kanter <[hidden email]> wrote:
Yes, we did that from IMO and I included in that in some... however, it is not at all consistent. Where we have it, we should use it. There shouldn't be dupes with the same map type. I will look through this...

Thanks!
Andy

P.S. Great news about MDS... now just need to fix the concepts :)
 
--------------------
Andrew S. Kanter, MD MPH

- Director of Health Information Systems/Medical Informatics
Millennium Villages Project, Earth Institute, Columbia University
- Asst. Prof. of Clinical Biomedical Informatics and Clinical Epidemiology
Columbia University

Email: [hidden email]
Mobile: +1 (646) 469-2421
Office: +1 (212) 305-4842
Skype: akanter-ippnw
Yahoo: andy_kanter


From: Rafal Korytkowski <[hidden email]>
To: [hidden email]
Sent: Friday, May 4, 2012 11:08 AM
Subject: [OPENMRS-DEV] Migrating concept mappings to 1.9

Hi Andy,

We have noticed that MVP uses the comment field in the concept_map table. We are considering using that to determine the right map type in 1.9.

I ran the following query select comment, count(*) from concept_map group by comment; The results are below. I have also added corresponding map types from 1.9, but I am not sure if they match right now. We could correct them if needed.

+----------------------+----------+
| comment              | count(*) |
+----------------------+----------+
| NULL                 |    15516 |
| From Excel           |     2381 |
| From UMLS RxNORM Map |     3010 |
| Map Type: 1          |    46897 | => SAME AS
| Map Type: 10         |        1 | Mistake... should be Map Type: 3
| Map Type: 17         |        5 | => Associated with
| Map Type: 19         |        3 | => Associated with
| Map Type: 2          |     1880 |=> BROADER-THAN
| Map Type: 24         |       18 | => Associated procedure
| Map Type: 3          |    30841 | => NARROWER-THAN
| Map Type: 4          |      126 | => Associated finding
| Map Type: 5          |       81 | => Associated Morphology
| Map Type: 6          |       19 | => Finding Site
| Map Type: 7          |        2 | => Associated with
+----------------------+----------+
14 rows in set (2.12 sec)

Here's the proposed migration algorithm:

(1) if the comment matches "Map Type: (\d+)" then use that to determine the map type, and drop it

(2) otherwise move the comment to concept_reference_term.description (even though it doesn't really belong there)

(3) delete duplicate concept_reference_terms (having same source and source_code), though this means we may lose some concept_map.comment data

The reason for these changes is: TRUNK-3296: Found multiple reference terms
-Rafał

[hidden email] from OpenMRS Developers' mailing list



[hidden email] from OpenMRS Developers' mailing list


[hidden email] from OpenMRS Developers' mailing list



[hidden email] from OpenMRS Developers' mailing list
Darius Jazayeri-3 Darius Jazayeri-3
Reply | Threaded
Open this post in threaded view
|

Re: Migrating concept mappings to 1.9

Hi Rafal et al,

Andy and I just discussed this on the phone, and as he says below, the "Map Type: #" comments in the MVP dictionary are totally unrelated to the actual map type ids we introduce in 1.9.

Andy is going to (today if he has time) change those in his database so they say something like "Map Type: SAME-AS" intstead, and re-export the dictionary for us.

Is it straightforward to change the upgrade scripts so that they look for "Map Type: NAME-OF-MAP-TYPE" instead?

-Darius

On Thu, May 10, 2012 at 7:30 AM, Andrew Kanter <[hidden email]> wrote:
Folks,
I don't think the original comments had anything to do with the list which appears in OpenMRS now for mapping sources... so the original map presented was definitely wrong. Please see the bottom of this email for corrected maps from the existing comments to the new map types.


Andy
 
--------------------
Andrew S. Kanter, MD MPH

Asst. Prof. of Clinical Biomedical Informatics and Clinical Epidemiology
Columbia University
Email: [hidden email]
Mobile: <a href="tel:%2B1%20%28646%29%20469-2421" value="+16464692421" target="_blank">+1 (646) 469-2421
Office: <a href="tel:%2B1%20%28212%29%20305-4842" value="+12123054842" target="_blank">+1 (212) 305-4842
Skype: akanter-ippnw
Yahoo: andy_kanter


From: Rafal Korytkowski <[hidden email]>
To: [hidden email]
Sent: Monday, May 7, 2012 11:04 AM
Subject: Re: [OPENMRS-DEV] Migrating concept mappings to 1.9

Thanks Andy! When do you think you'll have time to look into that comments and see if they can be matched with proper mapping types from 1.9? Below is a full list of predefined mapping types in 1.9:

+---------------------+-------------------------------
| concept_map_type_id | name
+---------------------+-------------------------------
|                   1 | SAME-AS  =
|                   2 | NARROWER-THAN
|                   3 | BROADER-THAN
|                   4 | Associated finding
|                   5 | Associated morphology
|                   6 | Associated procedure
|                   7 | Associated with
|                   8 | Causative agent
|                   9 | Finding site
|                  10 | Has specimen
|                  11 | Laterality
|                  12 | Severity
|                  13 | Access
|                  14 | After
|                  15 | Clinical course
|                  16 | Component
|                  17 | Direct device
|                  18 | Direct morphology
|                  19 | Direct substance
|                  20 | Due to
|                  21 | Episodicity
|                  22 | Finding context
|                  23 | Finding informer
|                  24 | Finding method
|                  25 | Has active ingredient
|                  26 | Has definitional manifestation
|                  27 | Has dose form
|                  28 | Has focus
|                  29 | Has intent
|                  30 | Has interpretation
|                  31 | Indirect device
|                  32 | Indirect morphology
|                  33 | Interprets
|                  34 | Measurement method
|                  35 | Method
|                  36 | Occurrence
|                  37 | Part of
|                  38 | Pathological process
|                  39 | Priority
|                  40 | Procedure context
|                  41 | Procedure device
|                  42 | Procedure morphology
|                  43 | Procedure site
|                  44 | Procedure site - Direct
|                  45 | Procedure site - Indirect
|                  46 | Property
|                  47 | Recipient category
|                  48 | Revision status
|                  49 | Route of administration
|                  50 | Scale type
|                  51 | Specimen procedure
|                  52 | Specimen source identity
|                  53 | Specimen source morphology
|                  54 | Specimen source topography
|                  55 | Specimen substance
|                  56 | Subject of information
|                  57 | Subject relationship context
|                  58 | Surgical approach
|                  59 | Temporal context
|                  60 | Time aspect
|                  61 | Using access device
|                  62 | Using device
|                  63 | Using energy
|                  64 | Using substance
|                  65 | IS A
|                  66 | MAY BE A
|                  67 | MOVED FROM
|                  68 | MOVED TO
|                  69 | REPLACED BY
|                  70 | WAS A
+---------------------+-------------------------------

-Rafał


On 4 May 2012 23:35, Andrew Kanter <[hidden email]> wrote:
Yes, we did that from IMO and I included in that in some... however, it is not at all consistent. Where we have it, we should use it. There shouldn't be dupes with the same map type. I will look through this...

Thanks!
Andy

P.S. Great news about MDS... now just need to fix the concepts :)
 
--------------------
Andrew S. Kanter, MD MPH

- Director of Health Information Systems/Medical Informatics
Millennium Villages Project, Earth Institute, Columbia University
- Asst. Prof. of Clinical Biomedical Informatics and Clinical Epidemiology
Columbia University

Email: [hidden email]
Mobile: +1 (646) 469-2421
Office: +1 (212) 305-4842
Skype: akanter-ippnw
Yahoo: andy_kanter


From: Rafal Korytkowski <[hidden email]>
To: [hidden email]
Sent: Friday, May 4, 2012 11:08 AM
Subject: [OPENMRS-DEV] Migrating concept mappings to 1.9

Hi Andy,

We have noticed that MVP uses the comment field in the concept_map table. We are considering using that to determine the right map type in 1.9.

I ran the following query select comment, count(*) from concept_map group by comment; The results are below. I have also added corresponding map types from 1.9, but I am not sure if they match right now. We could correct them if needed.

+----------------------+----------+
| comment              | count(*) |
+----------------------+----------+
| NULL                 |    15516 |
| From Excel           |     2381 |
| From UMLS RxNORM Map |     3010 |
| Map Type: 1          |    46897 | => SAME AS
| Map Type: 10         |        1 | Mistake... should be Map Type: 3
| Map Type: 17         |        5 | => Associated with
| Map Type: 19         |        3 | => Associated with
| Map Type: 2          |     1880 |=> BROADER-THAN
| Map Type: 24         |       18 | => Associated procedure
| Map Type: 3          |    30841 | => NARROWER-THAN
| Map Type: 4          |      126 | => Associated finding
| Map Type: 5          |       81 | => Associated Morphology
| Map Type: 6          |       19 | => Finding Site
| Map Type: 7          |        2 | => Associated with

+----------------------+----------+
14 rows in set (2.12 sec)

Here's the proposed migration algorithm:

(1) if the comment matches "Map Type: (\d+)" then use that to determine the map type, and drop it

(2) otherwise move the comment to concept_reference_term.description (even though it doesn't really belong there)

(3) delete duplicate concept_reference_terms (having same source and source_code), though this means we may lose some concept_map.comment data

The reason for these changes is: TRUNK-3296: Found multiple reference terms
-Rafał

[hidden email] from OpenMRS Developers' mailing list



[hidden email] from OpenMRS Developers' mailing list


[hidden email] from OpenMRS Developers' mailing list



[hidden email] from OpenMRS Developers' mailing list


[hidden email] from OpenMRS Developers' mailing list
Andrew Kanter Andrew Kanter
Reply | Threaded
Open this post in threaded view
|

Re: Migrating concept mappings to 1.9

New version openmrs_concepts_1.6.5_20120510.sql uploaded to CIEL dropbox. Rafal, please view this carefully as I generated things quickly...
 
--------------------
Andrew S. Kanter, MD MPH

Asst. Prof. of Clinical Biomedical Informatics and Clinical Epidemiology
Columbia University
Email: [hidden email]
Mobile: +1 (646) 469-2421
Office: +1 (212) 305-4842
Skype: akanter-ippnw
Yahoo: andy_kanter


From: Darius Jazayeri <[hidden email]>
To: [hidden email]
Sent: Thursday, May 10, 2012 11:47 AM
Subject: Re: [OPENMRS-DEV] Migrating concept mappings to 1.9

Hi Rafal et al,

Andy and I just discussed this on the phone, and as he says below, the "Map Type: #" comments in the MVP dictionary are totally unrelated to the actual map type ids we introduce in 1.9.

Andy is going to (today if he has time) change those in his database so they say something like "Map Type: SAME-AS" intstead, and re-export the dictionary for us.

Is it straightforward to change the upgrade scripts so that they look for "Map Type: NAME-OF-MAP-TYPE" instead?

-Darius

On Thu, May 10, 2012 at 7:30 AM, Andrew Kanter <[hidden email]> wrote:
Folks,
I don't think the original comments had anything to do with the list which appears in OpenMRS now for mapping sources... so the original map presented was definitely wrong. Please see the bottom of this email for corrected maps from the existing comments to the new map types.


Andy
 
--------------------
Andrew S. Kanter, MD MPH

Asst. Prof. of Clinical Biomedical Informatics and Clinical Epidemiology
Columbia University
Email: [hidden email]
Mobile: +1 (646) 469-2421
Office: +1 (212) 305-4842
Skype: akanter-ippnw
Yahoo: andy_kanter


From: Rafal Korytkowski <[hidden email]>
To: [hidden email]
Sent: Monday, May 7, 2012 11:04 AM
Subject: Re: [OPENMRS-DEV] Migrating concept mappings to 1.9

Thanks Andy! When do you think you'll have time to look into that comments and see if they can be matched with proper mapping types from 1.9? Below is a full list of predefined mapping types in 1.9:

+---------------------+-------------------------------
| concept_map_type_id | name
+---------------------+-------------------------------
|                   1 | SAME-AS  =
|                   2 | NARROWER-THAN
|                   3 | BROADER-THAN
|                   4 | Associated finding
|                   5 | Associated morphology
|                   6 | Associated procedure
|                   7 | Associated with
|                   8 | Causative agent
|                   9 | Finding site
|                  10 | Has specimen
|                  11 | Laterality
|                  12 | Severity
|                  13 | Access
|                  14 | After
|                  15 | Clinical course
|                  16 | Component
|                  17 | Direct device
|                  18 | Direct morphology
|                  19 | Direct substance
|                  20 | Due to
|                  21 | Episodicity
|                  22 | Finding context
|                  23 | Finding informer
|                  24 | Finding method
|                  25 | Has active ingredient
|                  26 | Has definitional manifestation
|                  27 | Has dose form
|                  28 | Has focus
|                  29 | Has intent
|                  30 | Has interpretation
|                  31 | Indirect device
|                  32 | Indirect morphology
|                  33 | Interprets
|                  34 | Measurement method
|                  35 | Method
|                  36 | Occurrence
|                  37 | Part of
|                  38 | Pathological process
|                  39 | Priority
|                  40 | Procedure context
|                  41 | Procedure device
|                  42 | Procedure morphology
|                  43 | Procedure site
|                  44 | Procedure site - Direct
|                  45 | Procedure site - Indirect
|                  46 | Property
|                  47 | Recipient category
|                  48 | Revision status
|                  49 | Route of administration
|                  50 | Scale type
|                  51 | Specimen procedure
|                  52 | Specimen source identity
|                  53 | Specimen source morphology
|                  54 | Specimen source topography
|                  55 | Specimen substance
|                  56 | Subject of information
|                  57 | Subject relationship context
|                  58 | Surgical approach
|                  59 | Temporal context
|                  60 | Time aspect
|                  61 | Using access device
|                  62 | Using device
|                  63 | Using energy
|                  64 | Using substance
|                  65 | IS A
|                  66 | MAY BE A
|                  67 | MOVED FROM
|                  68 | MOVED TO
|                  69 | REPLACED BY
|                  70 | WAS A
+---------------------+-------------------------------

-Rafał


On 4 May 2012 23:35, Andrew Kanter <[hidden email]> wrote:
Yes, we did that from IMO and I included in that in some... however, it is not at all consistent. Where we have it, we should use it. There shouldn't be dupes with the same map type. I will look through this...

Thanks!
Andy

P.S. Great news about MDS... now just need to fix the concepts :)
 
--------------------
Andrew S. Kanter, MD MPH

- Director of Health Information Systems/Medical Informatics
Millennium Villages Project, Earth Institute, Columbia University
- Asst. Prof. of Clinical Biomedical Informatics and Clinical Epidemiology
Columbia University

Email: [hidden email]
Mobile: +1 (646) 469-2421
Office: +1 (212) 305-4842
Skype: akanter-ippnw
Yahoo: andy_kanter


From: Rafal Korytkowski <[hidden email]>
To: [hidden email]
Sent: Friday, May 4, 2012 11:08 AM
Subject: [OPENMRS-DEV] Migrating concept mappings to 1.9

Hi Andy,

We have noticed that MVP uses the comment field in the concept_map table. We are considering using that to determine the right map type in 1.9.

I ran the following query select comment, count(*) from concept_map group by comment; The results are below. I have also added corresponding map types from 1.9, but I am not sure if they match right now. We could correct them if needed.

+----------------------+----------+
| comment              | count(*) |
+----------------------+----------+
| NULL                 |    15516 |
| From Excel           |     2381 |
| From UMLS RxNORM Map |     3010 |
| Map Type: 1          |    46897 | => SAME AS
| Map Type: 10         |        1 | Mistake... should be Map Type: 3
| Map Type: 17         |        5 | => Associated with
| Map Type: 19         |        3 | => Associated with
| Map Type: 2          |     1880 |=> BROADER-THAN
| Map Type: 24         |       18 | => Associated procedure
| Map Type: 3          |    30841 | => NARROWER-THAN
| Map Type: 4          |      126 | => Associated finding
| Map Type: 5          |       81 | => Associated Morphology
| Map Type: 6          |       19 | => Finding Site
| Map Type: 7          |        2 | => Associated with

+----------------------+----------+
14 rows in set (2.12 sec)

Here's the proposed migration algorithm:

(1) if the comment matches "Map Type: (\d+)" then use that to determine the map type, and drop it

(2) otherwise move the comment to concept_reference_term.description (even though it doesn't really belong there)

(3) delete duplicate concept_reference_terms (having same source and source_code), though this means we may lose some concept_map.comment data

The reason for these changes is: TRUNK-3296: Found multiple reference terms
-Rafał

[hidden email] from OpenMRS Developers' mailing list



[hidden email] from OpenMRS Developers' mailing list


[hidden email] from OpenMRS Developers' mailing list



[hidden email] from OpenMRS Developers' mailing list


[hidden email] from OpenMRS Developers' mailing list



[hidden email] from OpenMRS Developers' mailing list
Rafal Korytkowski-2 Rafal Korytkowski-2
Reply | Threaded
Open this post in threaded view
|

Re: Migrating concept mappings to 1.9

Andy, do we need to have two separate map types for "SAME-AS" and "SAME-AS from RxNORM" or the latter can be represented as "SAME-AS"?

-Rafał

On 10 May 2012 19:21, Andrew Kanter <[hidden email]> wrote:
New version openmrs_concepts_1.6.5_20120510.sql uploaded to CIEL dropbox. Rafal, please view this carefully as I generated things quickly...
 
--------------------
Andrew S. Kanter, MD MPH

Asst. Prof. of Clinical Biomedical Informatics and Clinical Epidemiology
Columbia University
Email: [hidden email]
Mobile: <a href="tel:%2B1%20%28646%29%20469-2421" value="+16464692421" target="_blank">+1 (646) 469-2421
Office: <a href="tel:%2B1%20%28212%29%20305-4842" value="+12123054842" target="_blank">+1 (212) 305-4842
Skype: akanter-ippnw
Yahoo: andy_kanter


From: Darius Jazayeri <[hidden email]>
To: [hidden email]
Sent: Thursday, May 10, 2012 11:47 AM

Subject: Re: [OPENMRS-DEV] Migrating concept mappings to 1.9

Hi Rafal et al,

Andy and I just discussed this on the phone, and as he says below, the "Map Type: #" comments in the MVP dictionary are totally unrelated to the actual map type ids we introduce in 1.9.

Andy is going to (today if he has time) change those in his database so they say something like "Map Type: SAME-AS" intstead, and re-export the dictionary for us.

Is it straightforward to change the upgrade scripts so that they look for "Map Type: NAME-OF-MAP-TYPE" instead?

-Darius

On Thu, May 10, 2012 at 7:30 AM, Andrew Kanter <[hidden email]> wrote:
Folks,
I don't think the original comments had anything to do with the list which appears in OpenMRS now for mapping sources... so the original map presented was definitely wrong. Please see the bottom of this email for corrected maps from the existing comments to the new map types.


Andy
 
--------------------
Andrew S. Kanter, MD MPH

Asst. Prof. of Clinical Biomedical Informatics and Clinical Epidemiology
Columbia University
Email: [hidden email]
Mobile: +1 (646) 469-2421
Office: +1 (212) 305-4842
Skype: akanter-ippnw
Yahoo: andy_kanter


From: Rafal Korytkowski <[hidden email]>
To: [hidden email]
Sent: Monday, May 7, 2012 11:04 AM
Subject: Re: [OPENMRS-DEV] Migrating concept mappings to 1.9

Thanks Andy! When do you think you'll have time to look into that comments and see if they can be matched with proper mapping types from 1.9? Below is a full list of predefined mapping types in 1.9:

+---------------------+-------------------------------
| concept_map_type_id | name
+---------------------+-------------------------------
|                   1 | SAME-AS  =
|                   2 | NARROWER-THAN
|                   3 | BROADER-THAN
|                   4 | Associated finding
|                   5 | Associated morphology
|                   6 | Associated procedure
|                   7 | Associated with
|                   8 | Causative agent
|                   9 | Finding site
|                  10 | Has specimen
|                  11 | Laterality
|                  12 | Severity
|                  13 | Access
|                  14 | After
|                  15 | Clinical course
|                  16 | Component
|                  17 | Direct device
|                  18 | Direct morphology
|                  19 | Direct substance
|                  20 | Due to
|                  21 | Episodicity
|                  22 | Finding context
|                  23 | Finding informer
|                  24 | Finding method
|                  25 | Has active ingredient
|                  26 | Has definitional manifestation
|                  27 | Has dose form
|                  28 | Has focus
|                  29 | Has intent
|                  30 | Has interpretation
|                  31 | Indirect device
|                  32 | Indirect morphology
|                  33 | Interprets
|                  34 | Measurement method
|                  35 | Method
|                  36 | Occurrence
|                  37 | Part of
|                  38 | Pathological process
|                  39 | Priority
|                  40 | Procedure context
|                  41 | Procedure device
|                  42 | Procedure morphology
|                  43 | Procedure site
|                  44 | Procedure site - Direct
|                  45 | Procedure site - Indirect
|                  46 | Property
|                  47 | Recipient category
|                  48 | Revision status
|                  49 | Route of administration
|                  50 | Scale type
|                  51 | Specimen procedure
|                  52 | Specimen source identity
|                  53 | Specimen source morphology
|                  54 | Specimen source topography
|                  55 | Specimen substance
|                  56 | Subject of information
|                  57 | Subject relationship context
|                  58 | Surgical approach
|                  59 | Temporal context
|                  60 | Time aspect
|                  61 | Using access device
|                  62 | Using device
|                  63 | Using energy
|                  64 | Using substance
|                  65 | IS A
|                  66 | MAY BE A
|                  67 | MOVED FROM
|                  68 | MOVED TO
|                  69 | REPLACED BY
|                  70 | WAS A
+---------------------+-------------------------------

-Rafał


On 4 May 2012 23:35, Andrew Kanter <[hidden email]> wrote:
Yes, we did that from IMO and I included in that in some... however, it is not at all consistent. Where we have it, we should use it. There shouldn't be dupes with the same map type. I will look through this...

Thanks!
Andy

P.S. Great news about MDS... now just need to fix the concepts :)
 
--------------------
Andrew S. Kanter, MD MPH

- Director of Health Information Systems/Medical Informatics
Millennium Villages Project, Earth Institute, Columbia University
- Asst. Prof. of Clinical Biomedical Informatics and Clinical Epidemiology
Columbia University

Email: [hidden email]
Mobile: +1 (646) 469-2421
Office: +1 (212) 305-4842
Skype: akanter-ippnw
Yahoo: andy_kanter


From: Rafal Korytkowski <[hidden email]>
To: [hidden email]
Sent: Friday, May 4, 2012 11:08 AM
Subject: [OPENMRS-DEV] Migrating concept mappings to 1.9

Hi Andy,

We have noticed that MVP uses the comment field in the concept_map table. We are considering using that to determine the right map type in 1.9.

I ran the following query select comment, count(*) from concept_map group by comment; The results are below. I have also added corresponding map types from 1.9, but I am not sure if they match right now. We could correct them if needed.

+----------------------+----------+
| comment              | count(*) |
+----------------------+----------+
| NULL                 |    15516 |
| From Excel           |     2381 |
| From UMLS RxNORM Map |     3010 |
| Map Type: 1          |    46897 | => SAME AS
| Map Type: 10         |        1 | Mistake... should be Map Type: 3
| Map Type: 17         |        5 | => Associated with
| Map Type: 19         |        3 | => Associated with
| Map Type: 2          |     1880 |=> BROADER-THAN
| Map Type: 24         |       18 | => Associated procedure
| Map Type: 3          |    30841 | => NARROWER-THAN
| Map Type: 4          |      126 | => Associated finding
| Map Type: 5          |       81 | => Associated Morphology
| Map Type: 6          |       19 | => Finding Site
| Map Type: 7          |        2 | => Associated with

+----------------------+----------+
14 rows in set (2.12 sec)

Here's the proposed migration algorithm:

(1) if the comment matches "Map Type: (\d+)" then use that to determine the map type, and drop it

(2) otherwise move the comment to concept_reference_term.description (even though it doesn't really belong there)

(3) delete duplicate concept_reference_terms (having same source and source_code), though this means we may lose some concept_map.comment data

The reason for these changes is: TRUNK-3296: Found multiple reference terms
-Rafał

[hidden email] from OpenMRS Developers' mailing list



[hidden email] from OpenMRS Developers' mailing list


[hidden email] from OpenMRS Developers' mailing list



[hidden email] from OpenMRS Developers' mailing list


[hidden email] from OpenMRS Developers' mailing list



[hidden email] from OpenMRS Developers' mailing list


[hidden email] from OpenMRS Developers' mailing list
Andrew Kanter Andrew Kanter
Reply | Threaded
Open this post in threaded view
|

Re: Migrating concept mappings to 1.9


 They are both Same-AS I kept that in the comment to keep the separate from my side...
--------------------
Andrew S. Kanter, MD MPH

Asst. Prof. of Clinical Biomedical Informatics and Clinical Epidemiology
Columbia University
Email: [hidden email]
Mobile: +1 (646) 469-2421
Office: +1 (212) 305-4842
Skype: akanter-ippnw
Yahoo: andy_kanter


From: Rafal Korytkowski <[hidden email]>
To: [hidden email]
Sent: Friday, May 11, 2012 5:41 AM
Subject: Re: [OPENMRS-DEV] Migrating concept mappings to 1.9

Andy, do we need to have two separate map types for "SAME-AS" and "SAME-AS from RxNORM" or the latter can be represented as "SAME-AS"?

-Rafał

On 10 May 2012 19:21, Andrew Kanter <[hidden email]> wrote:
New version openmrs_concepts_1.6.5_20120510.sql uploaded to CIEL dropbox. Rafal, please view this carefully as I generated things quickly...
 
--------------------
Andrew S. Kanter, MD MPH

Asst. Prof. of Clinical Biomedical Informatics and Clinical Epidemiology
Columbia University
Email: [hidden email]
Mobile: +1 (646) 469-2421
Office: +1 (212) 305-4842
Skype: akanter-ippnw
Yahoo: andy_kanter


From: Darius Jazayeri <[hidden email]>
To: [hidden email]
Sent: Thursday, May 10, 2012 11:47 AM

Subject: Re: [OPENMRS-DEV] Migrating concept mappings to 1.9

Hi Rafal et al,

Andy and I just discussed this on the phone, and as he says below, the "Map Type: #" comments in the MVP dictionary are totally unrelated to the actual map type ids we introduce in 1.9.

Andy is going to (today if he has time) change those in his database so they say something like "Map Type: SAME-AS" intstead, and re-export the dictionary for us.

Is it straightforward to change the upgrade scripts so that they look for "Map Type: NAME-OF-MAP-TYPE" instead?

-Darius

On Thu, May 10, 2012 at 7:30 AM, Andrew Kanter <[hidden email]> wrote:
Folks,
I don't think the original comments had anything to do with the list which appears in OpenMRS now for mapping sources... so the original map presented was definitely wrong. Please see the bottom of this email for corrected maps from the existing comments to the new map types.


Andy
 
--------------------
Andrew S. Kanter, MD MPH

Asst. Prof. of Clinical Biomedical Informatics and Clinical Epidemiology
Columbia University
Email: [hidden email]
Mobile: +1 (646) 469-2421
Office: +1 (212) 305-4842
Skype: akanter-ippnw
Yahoo: andy_kanter


From: Rafal Korytkowski <[hidden email]>
To: [hidden email]
Sent: Monday, May 7, 2012 11:04 AM
Subject: Re: [OPENMRS-DEV] Migrating concept mappings to 1.9

Thanks Andy! When do you think you'll have time to look into that comments and see if they can be matched with proper mapping types from 1.9? Below is a full list of predefined mapping types in 1.9:

+---------------------+-------------------------------
| concept_map_type_id | name
+---------------------+-------------------------------
|                   1 | SAME-AS  =
|                   2 | NARROWER-THAN
|                   3 | BROADER-THAN
|                   4 | Associated finding
|                   5 | Associated morphology
|                   6 | Associated procedure
|                   7 | Associated with
|                   8 | Causative agent
|                   9 | Finding site
|                  10 | Has specimen
|                  11 | Laterality
|                  12 | Severity
|                  13 | Access
|                  14 | After
|                  15 | Clinical course
|                  16 | Component
|                  17 | Direct device
|                  18 | Direct morphology
|                  19 | Direct substance
|                  20 | Due to
|                  21 | Episodicity
|                  22 | Finding context
|                  23 | Finding informer
|                  24 | Finding method
|                  25 | Has active ingredient
|                  26 | Has definitional manifestation
|                  27 | Has dose form
|                  28 | Has focus
|                  29 | Has intent
|                  30 | Has interpretation
|                  31 | Indirect device
|                  32 | Indirect morphology
|                  33 | Interprets
|                  34 | Measurement method
|                  35 | Method
|                  36 | Occurrence
|                  37 | Part of
|                  38 | Pathological process
|                  39 | Priority
|                  40 | Procedure context
|                  41 | Procedure device
|                  42 | Procedure morphology
|                  43 | Procedure site
|                  44 | Procedure site - Direct
|                  45 | Procedure site - Indirect
|                  46 | Property
|                  47 | Recipient category
|                  48 | Revision status
|                  49 | Route of administration
|                  50 | Scale type
|                  51 | Specimen procedure
|                  52 | Specimen source identity
|                  53 | Specimen source morphology
|                  54 | Specimen source topography
|                  55 | Specimen substance
|                  56 | Subject of information
|                  57 | Subject relationship context
|                  58 | Surgical approach
|                  59 | Temporal context
|                  60 | Time aspect
|                  61 | Using access device
|                  62 | Using device
|                  63 | Using energy
|                  64 | Using substance
|                  65 | IS A
|                  66 | MAY BE A
|                  67 | MOVED FROM
|                  68 | MOVED TO
|                  69 | REPLACED BY
|                  70 | WAS A
+---------------------+-------------------------------

-Rafał


On 4 May 2012 23:35, Andrew Kanter <[hidden email]> wrote:
Yes, we did that from IMO and I included in that in some... however, it is not at all consistent. Where we have it, we should use it. There shouldn't be dupes with the same map type. I will look through this...

Thanks!
Andy

P.S. Great news about MDS... now just need to fix the concepts :)
 
--------------------
Andrew S. Kanter, MD MPH

- Director of Health Information Systems/Medical Informatics
Millennium Villages Project, Earth Institute, Columbia University
- Asst. Prof. of Clinical Biomedical Informatics and Clinical Epidemiology
Columbia University

Email: [hidden email]
Mobile: +1 (646) 469-2421
Office: +1 (212) 305-4842
Skype: akanter-ippnw
Yahoo: andy_kanter


From: Rafal Korytkowski <[hidden email]>
To: [hidden email]
Sent: Friday, May 4, 2012 11:08 AM
Subject: [OPENMRS-DEV] Migrating concept mappings to 1.9

Hi Andy,

We have noticed that MVP uses the comment field in the concept_map table. We are considering using that to determine the right map type in 1.9.

I ran the following query select comment, count(*) from concept_map group by comment; The results are below. I have also added corresponding map types from 1.9, but I am not sure if they match right now. We could correct them if needed.

+----------------------+----------+
| comment              | count(*) |
+----------------------+----------+
| NULL                 |    15516 |
| From Excel           |     2381 |
| From UMLS RxNORM Map |     3010 |
| Map Type: 1          |    46897 | => SAME AS
| Map Type: 10         |        1 | Mistake... should be Map Type: 3
| Map Type: 17         |        5 | => Associated with
| Map Type: 19         |        3 | => Associated with
| Map Type: 2          |     1880 |=> BROADER-THAN
| Map Type: 24         |       18 | => Associated procedure
| Map Type: 3          |    30841 | => NARROWER-THAN
| Map Type: 4          |      126 | => Associated finding
| Map Type: 5          |       81 | => Associated Morphology
| Map Type: 6          |       19 | => Finding Site
| Map Type: 7          |        2 | => Associated with

+----------------------+----------+
14 rows in set (2.12 sec)

Here's the proposed migration algorithm:

(1) if the comment matches "Map Type: (\d+)" then use that to determine the map type, and drop it

(2) otherwise move the comment to concept_reference_term.description (even though it doesn't really belong there)

(3) delete duplicate concept_reference_terms (having same source and source_code), though this means we may lose some concept_map.comment data

The reason for these changes is: TRUNK-3296: Found multiple reference terms
-Rafał

[hidden email] from OpenMRS Developers' mailing list



[hidden email] from OpenMRS Developers' mailing list


[hidden email] from OpenMRS Developers' mailing list



[hidden email] from OpenMRS Developers' mailing list


[hidden email] from OpenMRS Developers' mailing list



[hidden email] from OpenMRS Developers' mailing list


[hidden email] from OpenMRS Developers' mailing list



[hidden email] from OpenMRS Developers' mailing list
Joaquín Blaya Joaquín Blaya
Reply | Threaded
Open this post in threaded view
|

Re: Migrating concept mappings to 1.9

In reply to this post by Andrew Kanter
Hi Andy,
I have access to the OpenMRS dropbox from you, but all I see there is the file openmrs_concepts_1.8.2_20110729.sql.zip. How do I get access to the concept dictionary you mentioned for 1.9?

Thanks,

Joaquín
___________________________________________________________________
Gerente de Desarrollo, eHealth Systems
Research Fellow, Escuela de Medicina de Harvard
Moderador, GHDOnline.org


On Thu, May 10, 2012 at 1:21 PM, Andrew Kanter <[hidden email]> wrote:
New version openmrs_concepts_1.6.5_<a href="tel:20120510" value="+5620120510" target="_blank">20120510.sql uploaded to CIEL dropbox. Rafal, please view this carefully as I generated things quickly...
 
--------------------
Andrew S. Kanter, MD MPH

Asst. Prof. of Clinical Biomedical Informatics and Clinical Epidemiology
Columbia University
Email: [hidden email]
Mobile: <a href="tel:%2B1%20%28646%29%20469-2421" value="+16464692421" target="_blank">+1 (646) 469-2421
Office: <a href="tel:%2B1%20%28212%29%20305-4842" value="+12123054842" target="_blank">+1 (212) 305-4842
Skype: akanter-ippnw
Yahoo: andy_kanter


From: Darius Jazayeri <[hidden email]>
To: [hidden email]
Sent: Thursday, May 10, 2012 11:47 AM

Subject: Re: [OPENMRS-DEV] Migrating concept mappings to 1.9

Hi Rafal et al,

Andy and I just discussed this on the phone, and as he says below, the "Map Type: #" comments in the MVP dictionary are totally unrelated to the actual map type ids we introduce in 1.9.

Andy is going to (today if he has time) change those in his database so they say something like "Map Type: SAME-AS" intstead, and re-export the dictionary for us.

Is it straightforward to change the upgrade scripts so that they look for "Map Type: NAME-OF-MAP-TYPE" instead?

-Darius

On Thu, May 10, 2012 at 7:30 AM, Andrew Kanter <[hidden email]> wrote:
Folks,
I don't think the original comments had anything to do with the list which appears in OpenMRS now for mapping sources... so the original map presented was definitely wrong. Please see the bottom of this email for corrected maps from the existing comments to the new map types.


Andy
 
--------------------
Andrew S. Kanter, MD MPH

Asst. Prof. of Clinical Biomedical Informatics and Clinical Epidemiology
Columbia University
Email: [hidden email]
Mobile: +1 (646) 469-2421
Office: +1 (212) 305-4842
Skype: akanter-ippnw
Yahoo: andy_kanter


From: Rafal Korytkowski <[hidden email]>
To: [hidden email]
Sent: Monday, May 7, 2012 11:04 AM
Subject: Re: [OPENMRS-DEV] Migrating concept mappings to 1.9

Thanks Andy! When do you think you'll have time to look into that comments and see if they can be matched with proper mapping types from 1.9? Below is a full list of predefined mapping types in 1.9:

+---------------------+-------------------------------
| concept_map_type_id | name
+---------------------+-------------------------------
|                   1 | SAME-AS  =
|                   2 | NARROWER-THAN
|                   3 | BROADER-THAN
|                   4 | Associated finding
|                   5 | Associated morphology
|                   6 | Associated procedure
|                   7 | Associated with
|                   8 | Causative agent
|                   9 | Finding site
|                  10 | Has specimen
|                  11 | Laterality
|                  12 | Severity
|                  13 | Access
|                  14 | After
|                  15 | Clinical course
|                  16 | Component
|                  17 | Direct device
|                  18 | Direct morphology
|                  19 | Direct substance
|                  20 | Due to
|                  21 | Episodicity
|                  22 | Finding context
|                  23 | Finding informer
|                  24 | Finding method
|                  25 | Has active ingredient
|                  26 | Has definitional manifestation
|                  27 | Has dose form
|                  28 | Has focus
|                  29 | Has intent
|                  30 | Has interpretation
|                  31 | Indirect device
|                  32 | Indirect morphology
|                  33 | Interprets
|                  34 | Measurement method
|                  35 | Method
|                  36 | Occurrence
|                  37 | Part of
|                  38 | Pathological process
|                  39 | Priority
|                  40 | Procedure context
|                  41 | Procedure device
|                  42 | Procedure morphology
|                  43 | Procedure site
|                  44 | Procedure site - Direct
|                  45 | Procedure site - Indirect
|                  46 | Property
|                  47 | Recipient category
|                  48 | Revision status
|                  49 | Route of administration
|                  50 | Scale type
|                  51 | Specimen procedure
|                  52 | Specimen source identity
|                  53 | Specimen source morphology
|                  54 | Specimen source topography
|                  55 | Specimen substance
|                  56 | Subject of information
|                  57 | Subject relationship context
|                  58 | Surgical approach
|                  59 | Temporal context
|                  60 | Time aspect
|                  61 | Using access device
|                  62 | Using device
|                  63 | Using energy
|                  64 | Using substance
|                  65 | IS A
|                  66 | MAY BE A
|                  67 | MOVED FROM
|                  68 | MOVED TO
|                  69 | REPLACED BY
|                  70 | WAS A
+---------------------+-------------------------------

-Rafał


On 4 May 2012 23:35, Andrew Kanter <[hidden email]> wrote:
Yes, we did that from IMO and I included in that in some... however, it is not at all consistent. Where we have it, we should use it. There shouldn't be dupes with the same map type. I will look through this...

Thanks!
Andy

P.S. Great news about MDS... now just need to fix the concepts :)
 
--------------------
Andrew S. Kanter, MD MPH

- Director of Health Information Systems/Medical Informatics
Millennium Villages Project, Earth Institute, Columbia University
- Asst. Prof. of Clinical Biomedical Informatics and Clinical Epidemiology
Columbia University

Email: [hidden email]
Mobile: +1 (646) 469-2421
Office: +1 (212) 305-4842
Skype: akanter-ippnw
Yahoo: andy_kanter


From: Rafal Korytkowski <[hidden email]>
To: [hidden email]
Sent: Friday, May 4, 2012 11:08 AM
Subject: [OPENMRS-DEV] Migrating concept mappings to 1.9

Hi Andy,

We have noticed that MVP uses the comment field in the concept_map table. We are considering using that to determine the right map type in 1.9.

I ran the following query select comment, count(*) from concept_map group by comment; The results are below. I have also added corresponding map types from 1.9, but I am not sure if they match right now. We could correct them if needed.

+----------------------+----------+
| comment              | count(*) |
+----------------------+----------+
| NULL                 |    15516 |
| From Excel           |     2381 |
| From UMLS RxNORM Map |     3010 |
| Map Type: 1          |    46897 | => SAME AS
| Map Type: 10         |        1 | Mistake... should be Map Type: 3
| Map Type: 17         |        5 | => Associated with
| Map Type: 19         |        3 | => Associated with
| Map Type: 2          |     1880 |=> BROADER-THAN
| Map Type: 24         |       18 | => Associated procedure
| Map Type: 3          |    30841 | => NARROWER-THAN
| Map Type: 4          |      126 | => Associated finding
| Map Type: 5          |       81 | => Associated Morphology
| Map Type: 6          |       19 | => Finding Site
| Map Type: 7          |        2 | => Associated with

+----------------------+----------+
14 rows in set (2.12 sec)

Here's the proposed migration algorithm:

(1) if the comment matches "Map Type: (\d+)" then use that to determine the map type, and drop it

(2) otherwise move the comment to concept_reference_term.description (even though it doesn't really belong there)

(3) delete duplicate concept_reference_terms (having same source and source_code), though this means we may lose some concept_map.comment data

The reason for these changes is: TRUNK-3296: Found multiple reference terms
-Rafał

[hidden email] from OpenMRS Developers' mailing list



[hidden email] from OpenMRS Developers' mailing list


[hidden email] from OpenMRS Developers' mailing list



[hidden email] from OpenMRS Developers' mailing list


[hidden email] from OpenMRS Developers' mailing list



[hidden email] from OpenMRS Developers' mailing list


[hidden email] from OpenMRS Developers' mailing list
Andrew Kanter Andrew Kanter
Reply | Threaded
Open this post in threaded view
|

Re: Migrating concept mappings to 1.9

Joaquin,

The only source currently is the 1.9 standalone version which has it built in as an option. I can easily extract the tables and add them to the dropbox, though. Rafal is working on a way for me to generate a 1.9 version regularly, although I would prefer to keep the 1.9 version as the primary maintenance version and back populate our 1.6 and other versions. I have not closely reviewed the enhanced concept content which is included with 1.9. There is going to be a new version shortly...

Andy
 
--------------------
Andrew S. Kanter, MD MPH

Asst. Prof. of Clinical Biomedical Informatics and Clinical Epidemiology
Columbia University
Email: [hidden email]
Mobile: +1 (646) 469-2421
Office: +1 (212) 305-4842
Skype: akanter-ippnw
Yahoo: andy_kanter


From: Joaquín Blaya <[hidden email]>
To: [hidden email]
Sent: Sunday, May 13, 2012 12:13 PM
Subject: Re: [OPENMRS-DEV] Migrating concept mappings to 1.9

Hi Andy,
I have access to the OpenMRS dropbox from you, but all I see there is the file openmrs_concepts_1.8.2_20110729.sql.zip. How do I get access to the concept dictionary you mentioned for 1.9?

Thanks,

Joaquín
___________________________________________________________________
Gerente de Desarrollo, eHealth Systems
Research Fellow, Escuela de Medicina de Harvard
Moderador, GHDOnline.org


On Thu, May 10, 2012 at 1:21 PM, Andrew Kanter <[hidden email]> wrote:
New version openmrs_concepts_1.6.5_20120510.sql uploaded to CIEL dropbox. Rafal, please view this carefully as I generated things quickly...
 
--------------------
Andrew S. Kanter, MD MPH

Asst. Prof. of Clinical Biomedical Informatics and Clinical Epidemiology
Columbia University
Email: [hidden email]
Mobile: +1 (646) 469-2421
Office: +1 (212) 305-4842
Skype: akanter-ippnw
Yahoo: andy_kanter


From: Darius Jazayeri <[hidden email]>
To: [hidden email]
Sent: Thursday, May 10, 2012 11:47 AM

Subject: Re: [OPENMRS-DEV] Migrating concept mappings to 1.9

Hi Rafal et al,

Andy and I just discussed this on the phone, and as he says below, the "Map Type: #" comments in the MVP dictionary are totally unrelated to the actual map type ids we introduce in 1.9.

Andy is going to (today if he has time) change those in his database so they say something like "Map Type: SAME-AS" intstead, and re-export the dictionary for us.

Is it straightforward to change the upgrade scripts so that they look for "Map Type: NAME-OF-MAP-TYPE" instead?

-Darius

On Thu, May 10, 2012 at 7:30 AM, Andrew Kanter <[hidden email]> wrote:
Folks,
I don't think the original comments had anything to do with the list which appears in OpenMRS now for mapping sources... so the original map presented was definitely wrong. Please see the bottom of this email for corrected maps from the existing comments to the new map types.


Andy
 
--------------------
Andrew S. Kanter, MD MPH

Asst. Prof. of Clinical Biomedical Informatics and Clinical Epidemiology
Columbia University
Email: [hidden email]
Mobile: +1 (646) 469-2421
Office: +1 (212) 305-4842
Skype: akanter-ippnw
Yahoo: andy_kanter


From: Rafal Korytkowski <[hidden email]>
To: [hidden email]
Sent: Monday, May 7, 2012 11:04 AM
Subject: Re: [OPENMRS-DEV] Migrating concept mappings to 1.9

Thanks Andy! When do you think you'll have time to look into that comments and see if they can be matched with proper mapping types from 1.9? Below is a full list of predefined mapping types in 1.9:

+---------------------+-------------------------------
| concept_map_type_id | name
+---------------------+-------------------------------
|                   1 | SAME-AS  =
|                   2 | NARROWER-THAN
|                   3 | BROADER-THAN
|                   4 | Associated finding
|                   5 | Associated morphology
|                   6 | Associated procedure
|                   7 | Associated with
|                   8 | Causative agent
|                   9 | Finding site
|                  10 | Has specimen
|                  11 | Laterality
|                  12 | Severity
|                  13 | Access
|                  14 | After
|                  15 | Clinical course
|                  16 | Component
|                  17 | Direct device
|                  18 | Direct morphology
|                  19 | Direct substance
|                  20 | Due to
|                  21 | Episodicity
|                  22 | Finding context
|                  23 | Finding informer
|                  24 | Finding method
|                  25 | Has active ingredient
|                  26 | Has definitional manifestation
|                  27 | Has dose form
|                  28 | Has focus
|                  29 | Has intent
|                  30 | Has interpretation
|                  31 | Indirect device
|                  32 | Indirect morphology
|                  33 | Interprets
|                  34 | Measurement method
|                  35 | Method
|                  36 | Occurrence
|                  37 | Part of
|                  38 | Pathological process
|                  39 | Priority
|                  40 | Procedure context
|                  41 | Procedure device
|                  42 | Procedure morphology
|                  43 | Procedure site
|                  44 | Procedure site - Direct
|                  45 | Procedure site - Indirect
|                  46 | Property
|                  47 | Recipient category
|                  48 | Revision status
|                  49 | Route of administration
|                  50 | Scale type
|                  51 | Specimen procedure
|                  52 | Specimen source identity
|                  53 | Specimen source morphology
|                  54 | Specimen source topography
|                  55 | Specimen substance
|                  56 | Subject of information
|                  57 | Subject relationship context
|                  58 | Surgical approach
|                  59 | Temporal context
|                  60 | Time aspect
|                  61 | Using access device
|                  62 | Using device
|                  63 | Using energy
|                  64 | Using substance
|                  65 | IS A
|                  66 | MAY BE A
|                  67 | MOVED FROM
|                  68 | MOVED TO
|                  69 | REPLACED BY
|                  70 | WAS A
+---------------------+-------------------------------

-Rafał


On 4 May 2012 23:35, Andrew Kanter <[hidden email]> wrote:
Yes, we did that from IMO and I included in that in some... however, it is not at all consistent. Where we have it, we should use it. There shouldn't be dupes with the same map type. I will look through this...

Thanks!
Andy

P.S. Great news about MDS... now just need to fix the concepts :)
 
--------------------
Andrew S. Kanter, MD MPH

- Director of Health Information Systems/Medical Informatics
Millennium Villages Project, Earth Institute, Columbia University
- Asst. Prof. of Clinical Biomedical Informatics and Clinical Epidemiology
Columbia University

Email: [hidden email]
Mobile: +1 (646) 469-2421
Office: +1 (212) 305-4842
Skype: akanter-ippnw
Yahoo: andy_kanter


From: Rafal Korytkowski <[hidden email]>
To: [hidden email]
Sent: Friday, May 4, 2012 11:08 AM
Subject: [OPENMRS-DEV] Migrating concept mappings to 1.9

Hi Andy,

We have noticed that MVP uses the comment field in the concept_map table. We are considering using that to determine the right map type in 1.9.

I ran the following query select comment, count(*) from concept_map group by comment; The results are below. I have also added corresponding map types from 1.9, but I am not sure if they match right now. We could correct them if needed.

+----------------------+----------+
| comment              | count(*) |
+----------------------+----------+
| NULL                 |    15516 |
| From Excel           |     2381 |
| From UMLS RxNORM Map |     3010 |
| Map Type: 1          |    46897 | => SAME AS
| Map Type: 10         |        1 | Mistake... should be Map Type: 3
| Map Type: 17         |        5 | => Associated with
| Map Type: 19         |        3 | => Associated with
| Map Type: 2          |     1880 |=> BROADER-THAN
| Map Type: 24         |       18 | => Associated procedure
| Map Type: 3          |    30841 | => NARROWER-THAN
| Map Type: 4          |      126 | => Associated finding
| Map Type: 5          |       81 | => Associated Morphology
| Map Type: 6          |       19 | => Finding Site
| Map Type: 7          |        2 | => Associated with

+----------------------+----------+
14 rows in set (2.12 sec)

Here's the proposed migration algorithm:

(1) if the comment matches "Map Type: (\d+)" then use that to determine the map type, and drop it

(2) otherwise move the comment to concept_reference_term.description (even though it doesn't really belong there)

(3) delete duplicate concept_reference_terms (having same source and source_code), though this means we may lose some concept_map.comment data

The reason for these changes is: TRUNK-3296: Found multiple reference terms
-Rafał

[hidden email] from OpenMRS Developers' mailing list



[hidden email] from OpenMRS Developers' mailing list


[hidden email] from OpenMRS Developers' mailing list



[hidden email] from OpenMRS Developers' mailing list


[hidden email] from OpenMRS Developers' mailing list



[hidden email] from OpenMRS Developers' mailing list


[hidden email] from OpenMRS Developers' mailing list



[hidden email] from OpenMRS Developers' mailing list