Using Python to check EIDR Token Status

Image for post
Image for post

Before you start reading this article, please read my previous article Using Python to access the EIDR registry with EIDR REST APIs.

Pre-requisites

  1. You have a Text editor installed on your machine. There are many python friendly editor, which helps writing Python code easier.

For Mac, you can use SublimeText, Atom, TextWrangler

For Windows, you can use Notepad, Notepad++, SublimeText, Atom

3. You can also install Jupyter, which is a web-based interactive Python development environment. You can download from https://jupyter.org/

Assumption

  1. You are a member of EIDR and have credentials to access EIDR Registry

EIDR Rest APIs

Here is the link to the EIDR Rest API document

Note: Public services do not necessarily mean open access. Ingesting or registering data into EIDR is controlled, while reading data from EIDR is generally not restricted.

EIDR Status Lookup Service API

Let’s get started…

  1. Install the required python packages
  2. Authenticate and authorize to access EIDR Rest APIs
import requests, base64, hashlib
from bs4 import BeautifulSoup
import pandas as pd

UserID = '10.5238/xxxxxxxx'. # enter your EIDR User Id
Pwd = '************' # enter your EIDR password
PartyID = '10.5237/xxxxxxxxx' # enter your EIDR party ID
url = 'https://sandbox1.eidr.org:443/EIDR/' # EIDR Registry URL
#Encrypt the credentials
PasswordShadow = base64.b64encode(hashlib.md5(pwd.encode('utf-8')).digest()).decode('utf8')
auth_str = '%s:%s:%s' % (UserID, PartyID, PasswordShadow)headers = {'Authorization' : 'Eidr {}'.format(auth_str), 'Accept': 'text/xml', 'Content-Type': 'text/xml'}

Now we write a function to get the status of the EIDR token. Refer to 2.3.4 Status lookup service (page 27) in the EIDR Rest API document for the request call and parameters.

def getTokenStatus(token):
req = url + 'status/token/' + token
resp = requests.get(req, headers=headers)
return resp.content

Let’s test the function by passing a token and seeing the result (or response)

#get the token status
token_status_response = getTokenStatus('1597356462419640618')
#display the token status response xml
soup = BeautifulSoup(token_status_response, 'xml')
print(soup.prettify())

Here is the response XML

<?xml version="1.0" encoding="utf-8"?>
<Response version="2.6.0" xmlns="http://www.eidr.org/schema">
<Status>
<Code>
0
</Code>
<Type>
success
</Type>
</Status>
<RequestStatus>
<Token>
1597356462419640618
</Token>
</RequestStatus>
<RequestStatusResults>
<CurrentSize>
1
</CurrentSize>
<TotalMatches>
1
</TotalMatches>
<OperationStatus>
<Token>
1597356462419640618
</Token>
<Status>
<Code>
0
</Code>
<Type>
success
</Type>
</Status>
<ID>
10.5240/7331-690C-D843-BE10-3CF2-X
</ID>
</OperationStatus>
</RequestStatusResults>
</Response>

The response XML shows that the token is resolved successfully and an EIDR ID (10.5240/7331–690C-D843-BE10–3CF2-X) is generated for that token. We can extract the EIDR Id using the BeautifulSoup package.

#get the token status
token_status = soup.Response.Status.Type.contents[0]
#get the operation status, which could be different from the main token status. Examples are duplicate, validation error, bad id...
token_operation_status = soup.Response.RequestStatusResults.OperationStatus.Type.contents[0]
#get the EIDR Id, if the token is resolved
token_status_ID = soup.Response.RequestStatusResults.OperationStatus.ID.contents[0]
#display the status
print('Status: {} OperationStatus: {} EIDR ID: {}'.format(token_status, token_operation_status, token_status_ID))

The above token resolved into an EIDR ID, which is great, but that is not always the case. There are other scenarios, where we see different results.

Duplicate

#get the token status
token_status_response = getTokenStatus('1591648141088364253')
#display the token status response xml
soup = BeautifulSoup(token_status_response, 'xml')
print(soup.prettify())

Here is the response XML

<?xml version="1.0" encoding="utf-8"?>
<Response version="2.6.0" xmlns="http://www.eidr.org/schema">
<Status>
<Code>
0
</Code>
<Type>
success
</Type>
</Status>
<RequestStatus>
<Token>
1591648141088364253
</Token>
</RequestStatus>
<RequestStatusResults>
<CurrentSize>
1
</CurrentSize>
<TotalMatches>
1
</TotalMatches>
<OperationStatus>
<Token>
1591648141088364253
</Token>
<Status>
<Code>
1
</Code>
<Type>
duplicate
</Type>
</Status>
<Duplicate>
<ID>
10.5240/1209-DB55-259A-32DB-568A-B
</ID>
</Duplicate>
<Duplicate>
<ID>
10.5240/E30E-7657-63B8-3CAB-5060-R
</ID>
</Duplicate>
<Duplicate>
<ID>
10.5240/0EF7-8253-4365-2869-0327-3
</ID>
</Duplicate>
<Duplicate>
<ID>
10.5240/B3B8-FE2D-8F36-6C50-AD96-N
</ID>
</Duplicate>
<Duplicate>
<ID>
10.5240/124E-38C5-038A-656A-CCDC-V
</ID>
</Duplicate>
</OperationStatus>
</RequestStatusResults>
</Response>

In the above response, the main token status is successful, but the operation status type is duplicate. Here we have multiple duplicate IDs for this token. Typically for a content analyst this would require further investigation on the token. We can extract the duplicate IDs using the BeautifulSoup package.

#create an empty list to store the tokens
token_status_Ids = []
#check if the operation status is duplicate
if token_operation_status == 'duplicate':
#loop thru each duplicate ID and add it to the list
for id in soup.Response.RequestStatusResults.OperationStatus.findAll('Duplicate'):
token_status_Ids.append(id.get_text('ID'))

#display the duplicate Id list
print(token_status_Ids)

Validation Error

#get the token status
token_status_response = getTokenStatus('1592869486580440852')
#display the token status response xml
soup = BeautifulSoup(token_status_response, 'xml')
print(soup.prettify())

Here is the response XML

<?xml version="1.0" encoding="utf-8"?>
<Response version="2.6.0" xmlns="http://www.eidr.org/schema">
<Status>
<Code>
0
</Code>
<Type>
success
</Type>
</Status>
<RequestStatus>
<Token>
1592869486580440852
</Token>
</RequestStatus>
<RequestStatusResults>
<CurrentSize>
1
</CurrentSize>
<TotalMatches>
1
</TotalMatches>
<OperationStatus>
<Token>
1592869486580440852
</Token>
<Status>
<Code>
4
</Code>
<Type>
validation error
</Type>
<Details>
Another episode exists with the same Parent and 'HouseSequence' or 'DistributionNumber': 10.5240/59F5-64D4-E54C-D579-1F4A-Y
</Details>
</Status>
</OperationStatus>
</RequestStatusResults>
</Response>

In the above response, the main token status is successful, but the operation status type is validation error. To extract this information, we can use the below code

token_status = soup.Response.Status.Type.contents[0]
token_operation_status = soup.Response.RequestStatusResults.OperationStatus.Type.contents[0]
token_operation_status_details = soup.Response.RequestStatusResults.OperationStatus.Status.Details.contents[0]
print('Status: {} OperationStatus: {} Details: {}'.format(token_status, token_operation_status, token_operation_status_details))

Other Errors

#get the token status
token_status_response = getTokenStatus('159286948658044085')
#display the token status response xml
soup = BeautifulSoup(token_status_response, 'xml')
print(soup.prettify())

Here is the response XML

<?xml version="1.0" encoding="utf-8"?>
<Response version="2.6.0" xmlns="http://www.eidr.org/schema">
<Status>
<Code>
6
</Code>
<Type>
bad token error
</Type>
</Status>
</Response>

In the above response, the main token status type is bad token error. You can extract it as below

token_status = soup.Response.Status.Type.contents[0]
print('Status: {} '.format(token_status))

For more details on other status codes and operation codes, please refer to 2.1.4 Codes and Descriptions (page 9) in the EIDR Rest API document.

Considering the above token responses, we need to conditionally extract token status. To do so, let’s write another function to extract the token status descriptions.

def getTokenStatusDesc(tokenStatusXML):

#initialize a variable to hold the status description
tokenStatus = ''

#read the token status response xml
soup = BeautifulSoup(tokenStatusXML, 'xml')

#get the token status code and convert into an integer
token_status_code = int(soup.Response.Status.Code.contents[0])
#check the token status
if token_status_code == 0:

#get the token operation status code and convert to integer
token_operation_status_code = int(soup.Response.RequestStatusResults.OperationStatus.Code.contents[0])
#check the token operation status code
if token_operation_status_code == 0:
resolved_eidr = soup.Response.RequestStatusResults.OperationStatus.ID.contents[0]
tokenStatus = resolved_eidr

elif token_operation_status_code == 1: #duplicate
#list to store the duplicate EIDRs found
token_status_Ids = []
#loop thru each duplicate EIDR Id and save them into a list
for id in soup.Response.RequestStatusResults.OperationStatus.findAll('Duplicate'):
token_status_Ids.append(id.get_text('ID'))
#covert the list into a comma separated string and concatenate with the status message
tokenStatus = 'Possible Duplicates: ' + ', '.join(token_status_Ids)

elif token_operation_status_code == 4: #validation error
val_error = soup.Response.RequestStatusResults.OperationStatus.Status.Details.contents[0]
tokenStatus = val_error

else:
#print('Token resolution error')
tokenStatus = 'Token resolution error'
else:
token_status = soup.Response.Status.Type.contents[0]
tokenStatus = token_status

return tokenStatus

The above function captures all the different scenarios discussed above. If you encounter any other status, you can modify the above function to accommodate it.

Let’s test the function to see if it returns the right status description

resp = getTokenStatus('1592869486580440852')
getTokenStatusDesc(resp)
Image for post
Image for post
Image for post
Image for post

It returns the correct description. The above code test’s only one token. A real use case would be to extract status for multiple tokens.

Let’s write code to read a text file with multiple token and get the status for each and export the output into an excel file.

Create a text file in a text editor and paste the below tokens or add your own tokens.Save the file as input_tokens.txt

1597356462419640618
1597696316125748968
1597696762145749158
1597697995996749652
1597697996415749654
1597697996840749656
1597698762875749902
1597698763326749904
1597698763763749905
1597698764201749908
1597698764638749912
1592869486580440852
1591648141088364253
159286948658044085

Below code does the following -

  1. Set’s a variable with the filename. In our case, it is input_tokens.txt.
  2. Open the file and split the tokens in a list of tokens
  3. Loop thru each token and call the functions that we created i.e. getTokenStatus and getTokenStatusDesc
  4. Save the results in a list
  5. Create a new pandas dataframe using the results list.
#set the inputfile variable with the token filename
inputFile = 'input_tokens.txt'
#open the tokenfile and split each token into a list object
with open(inputFile) as f:
tokens = f.read().splitlines()
#create an empty list to store the token status
output_list = []
#loop thru each token to get the status
for token in tokens:
resp = getTokenStatus(token)
status = getTokenStatusDesc(resp)

print('Token: {} Status: {}'.format(token, status))
output_list.append([token, status])
df_output = pd.DataFrame(output_list, columns =['Token', 'Status'])
df_output.head()

Here is the output of the df_output.head()

Image for post
Image for post

Once the dataframe is created, you can export it to excel

df_output.to_excel('token_status.xlsx')

Here is the complete python code

import requests, base64, hashlib
from bs4 import BeautifulSoup
import pandas as pd
UserID = '10.5238/xxxxxxxx'. # enter your EIDR User Id
Pwd = '************' # enter your EIDR password
PartyID = '10.5237/xxxxxxxxx' # enter your EIDR party ID
url = 'https://sandbox1.eidr.org:443/EIDR/' # EIDR Registry URL
#Encrypt the credentials
PasswordShadow = base64.b64encode(hashlib.md5(pwd.encode('utf-8')).digest()).decode('utf8')
auth_str = '%s:%s:%s' % (UserID, PartyID, PasswordShadow)headers = {'Authorization' : 'Eidr {}'.format(auth_str), 'Accept': 'text/xml', 'Content-Type': 'text/xml'}def getTokenStatus(token):

req = url + 'status/token/' + token
resp = requests.get(req, headers=headers)
return resp.content
def getTokenStatusDesc(tokenStatusXML):

#initialize a variable to hold the status description
tokenStatus = ''

#read the token status response xml
soup = BeautifulSoup(tokenStatusXML, 'xml')

#get the token status code and convert into an integer
token_status_code = int(soup.Response.Status.Code.contents[0])
#check the token status
if token_status_code == 0:

#get the token operation status code and convert to integer
token_operation_status_code = int(soup.Response.RequestStatusResults.OperationStatus.Code.contents[0])
#check the token operation status code
if token_operation_status_code == 0:
resolved_eidr = soup.Response.RequestStatusResults.OperationStatus.ID.contents[0]
tokenStatus = resolved_eidr

elif token_operation_status_code == 1: #duplicate
#list to store the duplicate EIDRs found
token_status_Ids = []
#loop thru each duplicate EIDR Id and save them into a list
for id in soup.Response.RequestStatusResults.OperationStatus.findAll('Duplicate'):
token_status_Ids.append(id.get_text('ID'))
#covert the list into a comma separated string and concatenate with the status message
tokenStatus = 'Possible Duplicates: ' + ', '.join(token_status_Ids)

elif token_operation_status_code == 4: #validation error
val_error = soup.Response.RequestStatusResults.OperationStatus.Status.Details.contents[0]
tokenStatus = val_error

else:
#print('Token resolution error')
tokenStatus = 'Token resolution error'
else:
token_status = soup.Response.Status.Type.contents[0]
tokenStatus = token_status

return tokenStatus
#set the inputfile variable with the token filename
inputFile = 'input_tokens.txt'
#open the tokenfile and split each token into a list object
with open(inputFile) as f:
tokens = f.read().splitlines()
#create an empty list to store the token status
output_list = []
#loop thru each token to get the status
for token in tokens:
resp = getTokenStatus(token)
status = getTokenStatusDesc(resp)

print('Token: {} Status: {}'.format(token, status))
output_list.append([token, status])
df_output = pd.DataFrame(output_list, columns =['Token', 'Status'])#export the data to an excel file
df_output.to_excel('token_status.xlsx')

This completes the tutorial on how to get EIDR token status for multiple tokens and export to excel.

Also check out:

how to get EIDR data from Alternate Ids

Data Management and Operations Strategy

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store