Python – Convert from XML comments to BRAT format

Convert from XML comments to BRAT format… here is a solution to the problem.

Convert from XML comments to BRAT format

I have a dataset of comments in XML format: see example below

Treatment of <annotation cui="C0267055">Erosive Esophagitis</annotation> in patients

The tagword is in the XML tag, as shown in the image. I need to put it in BRAT format, for example:

T1    annotation 14 33    Erosive Esophagitis

More examples can be found in http://brat.nlplab.org/standoff.html

I

can extract comments using regular expressions in Python, but I’m not sure how to convert it to the correct BRAT format. Is it possible to have such a tool?

Solution

If anyone still needs an answer to this question, here is a solution.

Suppose an XML file sample.xml has the following structure:

<root>
<p n='1'>Hi, my name is <fname>Mickey</fname> <lname>Mouse</lname>, and what about yourself?</p>
<p n='2'>Nice meeting you, <fname>Mickey</fname>! I am <fname>Minnie</lname>!</p>
</root>

This is a Python solution:

# leave empty if there are no tags that should not be interpreted as named entities; or add more
ignoretags = ['root', 'p']

# dictionary, in case some named entities have to be mapped; or just a list of tags that represent NEs
replacetags = {
    "fname": "PERS",
    "lname": "PERS"
}

# read content
content = open('sample.xml', encoding='utf-8').read()

# output files for BRAT: txt and annotations
f_txt = open('sample.txt', 'w')
f_ann = open('sample.ann', 'w')

# from txt file remove NE tags
clean_content = content
for replacetag in replacetags:
    clean_content = clean_content.replace('<{}>'.format(replacetag), '')
    clean_content = clean_content.replace('</{}>'.format(replacetag), '')

# write content to file
f_txt.write(clean_content)

# char by char
n = len(content)
i = - 1

# token id
tid = 0
# other - for output
start = -1
end = - 1
token = None
tag = None

# let's start parsing! character by character
skipped_chars = 0
number_of_tags = 0

token_buffer = ''
while i < n - 1:

i += 1
    c = content[i]

# beginning of an entity
    if c == '<':

# probably the most important part: always track the count of skipped characters
        start = i - skipped_chars

# get name of the entity
        tag_buffer = ''
        i += 1
        while content[i] != '>':
            tag_buffer += content[i]
            i += 1
        tag = tag_buffer

# skip tags that are not NEs
        if tag not in replacetags:
            continue

# get entity itself
        ent_buffer = ''
        i += 1
        while content[i] != '<':
            ent_buffer += content[i]
            i += 1
        token = ent_buffer

# determine positions
        end = start + len(token)
        number_of_tags += 1

# <fname></fname> i.e. 1 + len('fname') + 1 + 1 + 1 + len('fname') + 1
        skipped_chars += 1 + len(tag) + 1 + 1 + 1 + len(tag) + 1
        tid += 1

# write annotation
        f_ann.write('T{}\t{} {} {}\t{}\n'.format(tid, replacetags[tag], start, end, token))

# navigate to the end of the entity span, e.g. go behind <fname>... </fname>
        i += 1 + len(tag) + 1

The contents of the sample .txt

<root>
<p n='1'>Hi, my name is Mickey Mouse, and what about yourself?</p>
<p n='2'>Nice meeting you, Mickey! I am Minnie!</p>
</root>

Contents of sample.ann:

T1  PERS 31 37  Mickey
T2  PERS 38 43  Mouse
T3  PERS 101 107    Mickey
T4  PERS 114 120    Minnie

Visual effects in BRA:

BRAT visualization of sample.xml

In the case of attributes, a small adjustment is required (I added another key ‘att’ to the replacetags dictionary, i.e. a pair would be “fname": {"tag": "PERS", "att": "attribute value"
Then write another line if the label has attributes.

Hope someone will find this useful!

Related Problems and Solutions