raw_attributes

The extra instrument attributes each broker publishes, under one shared vocabulary.

A broker's instrument file carries far more than the mapping needs to place an order. Dhan sends an ISIN, a freeze quantity and a price band; Kotak sends margin percentages; Groww says whether an instrument may be bought and sold at all. None of that reaches unified.broker_mappings, because the mapping only keeps the token, the symbols, the lot size and the tick size.

This module names the columns worth keeping and gives every broker's spelling of them one unified name, so that an ISIN is isin whether the broker called it isin, pisin or isin_code. The attributes are read at mapping time, where the raw row is already in hand, because there is no stable key to find that row again afterwards: a broker's token is not unique within a snapshot for seven of the ten brokers, and the symbol columns the mapping stores change from segment to segment.

Every attribute in ATTRIBUTE_NAMES is present for every broker, and a broker that does not publish one has None there, which is the same rule the tick, order and position contracts follow.

Classes:

Name	Description
RawAttributes	The map from each broker's raw instrument columns onto the shared attribute vocabulary.

RawAttributes

The map from each broker's raw instrument columns onto the shared attribute vocabulary.

Attributes:

Name	Type	Description
COLUMNS	dict	Mapping of broker name to a dict of unified attribute name to that broker's column name. A broker's dict names only the attributes it publishes.

Example programs

Reading a dhan rowWho publishes what

examples/stock_brokers/instruments/mapping/utilities/raw_attributes/RawAttributes/example_1_reading_a_dhan_row.py

"""Reads the extra attributes out of one raw Dhan instrument row and expands them for a caller.

A broker's instrument file carries more than an order needs. `extract` picks out the columns `RawAttributes` knows for that broker and renames them to the shared vocabulary, so Dhan's `sm_freeze_qty` becomes `freeze_quantity` and its `asm_gsm_category` becomes `surveillance_category`. Values are kept as text, stripped of spaces, and a column that is empty in the row is left out rather than stored as null.

`fill` then turns the stored attributes back into the full vocabulary of sixteen names, with None for anything Dhan does not publish, which is the shape the REST API answers with. `columns_for` shows the column map the extraction followed.

The row is a made-up but realistic Dhan row for INFY on NSE. No database is needed.

Run it from the project root:

    python examples/stock_brokers/instruments/mapping/utilities/raw_attributes/RawAttributes/example_1_reading_a_dhan_row.py
"""

from stock_brokers.instruments.mapping.utilities.raw_attributes import (
    RawAttributes,
)


class ReadingADhanRowExample:
    """Extracts, prints and fills the attributes of one Dhan row.

    Attributes:
        raw_attributes (RawAttributes): The column map being shown.
        raw_row (dict): One raw row from Dhan's instrument table.
    """

    def __init__(self):
        """Builds the column map and the raw row.

        Returns:
            None: This method returns nothing.
        """
        self.raw_attributes = RawAttributes()
        self.raw_row = {
            'security_id': '1594',
            'exch_id': 'NSE',
            'symbol_name': 'INFOSYS LIMITED',
            'isin': 'INE009A01021',
            'display_name': ' Infosys ',
            'instrument_type': 'ES',
            'series': 'EQ',
            'sm_freeze_qty': '',
            'sm_upper_limit': '1709.60',
            'sm_lower_limit': '1398.80',
            'underlying_security_id': None,
            'asm_gsm_category': 'NA',
            'mtf_leverage': '4.00',
        }

    def run(self):
        """Prints Dhan's column map, the extracted attributes and the filled answer.

        Returns:
            None: This method returns nothing.
        """
        columns = self.raw_attributes.columns_for('dhan')
        print('Dhan column map:')
        for attribute_name in columns:
            print(f'  {attribute_name} <- {columns[attribute_name]}')
        attributes = self.raw_attributes.extract('dhan', self.raw_row)
        print(f'Stored: {attributes}')
        filled = self.raw_attributes.fill(attributes)
        print('Answered:')
        for attribute_name in filled:
            print(f'  {attribute_name}: {filled[attribute_name]}')


if __name__ == '__main__':
    ReadingADhanRowExample().run()

It prints:

Dhan column map:
  isin <- isin
  display_name <- display_name
  instrument_type <- instrument_type
  series <- series
  freeze_quantity <- sm_freeze_qty
  price_band_high <- sm_upper_limit
  price_band_low <- sm_lower_limit
  underlying_token <- underlying_security_id
  surveillance_category <- asm_gsm_category
  margin_trading_leverage <- mtf_leverage
Stored: {'isin': 'INE009A01021', 'display_name': 'Infosys', 'instrument_type': 'ES', 'series': 'EQ', 'price_band_high': '1709.60', 'price_band_low': '1398.80', 'surveillance_category': 'NA', 'margin_trading_leverage': '4.00'}
Answered:
  isin: INE009A01021
  display_name: Infosys
  instrument_type: ES
  series: EQ
  freeze_quantity: None
  price_band_high: 1709.60
  price_band_low: 1398.80
  multiplier: None
  underlying_token: None
  surveillance_category: NA
  permitted_to_trade: None
  buy_allowed: None
  sell_allowed: None
  intraday_leverage: None
  margin_trading_leverage: 4.00
  pledge_eligible: None

examples/stock_brokers/instruments/mapping/utilities/raw_attributes/RawAttributes/example_2_who_publishes_what.py

"""Finds which brokers publish an attribute, and shows how missing and messy values are cleaned.

`publishers_of` answers questions such as "which brokers can tell me an ISIN?" from the column map, in the map's order. `columns_for` returns an empty map for a broker the map does not know, so extracting from such a broker gives nothing rather than an error.

`clean` is the rule every value goes through: None, a pandas missing value and blank text all become None, and anything else becomes stripped text. The program also extracts a Zerodha row, whose file has only two extra columns, and a Groww row with a pandas missing value in it, to show those rules at work. The rows are made up; no database is needed.

Run it from the project root:

    python examples/stock_brokers/instruments/mapping/utilities/raw_attributes/RawAttributes/example_2_who_publishes_what.py
"""

import math

import pandas as pd

from stock_brokers.instruments.mapping.utilities.raw_attributes import (
    RawAttributes,
)


class WhoPublishesWhatExample:
    """Lists the publishers of three attributes, cleans a few values and extracts two rows.

    Attributes:
        raw_attributes (RawAttributes): The column map being shown.
    """

    def __init__(self):
        """Builds the column map.

        Returns:
            None: This method returns nothing.
        """
        self.raw_attributes = RawAttributes()

    def run(self):
        """Prints the publishers, the cleaned values and two extractions.

        Returns:
            None: This method returns nothing.
        """
        attribute_names = [
            'isin',
            'freeze_quantity',
            'buy_allowed',
        ]
        for attribute_name in attribute_names:
            print(f'{attribute_name}: {self.raw_attributes.publishers_of(attribute_name)}')
        values = [
            None,
            pd.NA,
            math.nan,
            '   ',
            ' 1800 ',
            25,
        ]
        for value in values:
            print(f'clean({value!r}) = {self.raw_attributes.clean(value)!r}')
        print(f'Columns for an unknown broker: {self.raw_attributes.columns_for("upstox")}')
        zerodha_row = {
            'instrument_token': '408065',
            'tradingsymbol': 'INFY',
            'name': 'INFOSYS',
            'instrument_type': 'EQ',
            'lot_size': '1',
        }
        print(f'Zerodha: {self.raw_attributes.extract("zerodha", zerodha_row)}')
        groww_row = {
            'exchange_token': '1594',
            'isin': 'INE009A01021',
            'name': 'Infosys',
            'instrument_type': 'EQ',
            'series': 'EQ',
            'freeze_quantity': pd.NA,
            'underlying_exchange_token': '',
            'buy_allowed': '1',
            'sell_allowed': '1',
        }
        print(f'Groww: {self.raw_attributes.extract("groww", groww_row)}')


if __name__ == '__main__':
    WhoPublishesWhatExample().run()

It prints:

isin: ['dhan', 'kotak', 'groww', 'stoxkart', 'fyers', 'wisdom_capital', 'indmoney']
freeze_quantity: ['dhan', 'kotak', 'groww', 'wisdom_capital', 'indmoney']
buy_allowed: ['groww']
clean(None) = None
clean(<NA>) = None
clean(nan) = None
clean('   ') = None
clean(' 1800 ') = '1800'
clean(25) = '25'
Columns for an unknown broker: {}
Zerodha: {'display_name': 'INFOSYS', 'instrument_type': 'EQ'}
Groww: {'isin': 'INE009A01021', 'display_name': 'Infosys', 'instrument_type': 'EQ', 'series': 'EQ', 'buy_allowed': '1', 'sell_allowed': '1'}

Methods:

Name	Description
columns_for	Finds the raw columns one broker publishes attributes in.
publishers_of	Finds every broker that publishes one attribute.
extract	Reads one broker's extra attributes out of one raw instrument row.
clean	Turns one raw column value into the text stored for it, or None.
fill	Expands stored attributes back to the full vocabulary, for an answer a caller reads.

columns_for

columns_for(broker)

Finds the raw columns one broker publishes attributes in.

Parameters:

Name	Type	Description	Default
broker	str	The broker name, as it appears in unified.broker_mappings.	required

Returns:

Type	Description
dict	Mapping of unified attribute name to that broker's column name, empty when the broker publishes nothing beyond what the mapping already keeps.

Example usage

From example_1_reading_a_dhan_row.py

columns = self.raw_attributes.columns_for('dhan')

From example_2_who_publishes_what.py

print(f'Columns for an unknown broker: {self.raw_attributes.columns_for("upstox")}')

Source code in stock_brokers/instruments/mapping/utilities/raw_attributes.py

def columns_for(self, broker):
    """Finds the raw columns one broker publishes attributes in.

    Args:
        broker (str): The broker name, as it appears in unified.broker_mappings.

    Returns:
        dict: Mapping of unified attribute name to that broker's column name, empty when the broker publishes nothing beyond what the mapping already keeps.
    """
    return self.COLUMNS.get(broker, {})

publishers_of

publishers_of(attribute_name)

Finds every broker that publishes one attribute.

Parameters:

Name	Type	Description	Default
attribute_name	str	A name from ATTRIBUTE_NAMES.	required

Returns:

Type	Description
list[str]	The broker names, in the order the column map lists them.

Example usage

From example_2_who_publishes_what.py

print(f'{attribute_name}: {self.raw_attributes.publishers_of(attribute_name)}')

Source code in stock_brokers/instruments/mapping/utilities/raw_attributes.py

def publishers_of(self, attribute_name):
    """Finds every broker that publishes one attribute.

    Args:
        attribute_name (str): A name from ATTRIBUTE_NAMES.

    Returns:
        list[str]: The broker names, in the order the column map lists them.
    """
    brokers = []
    for broker in self.COLUMNS:
        if attribute_name in self.COLUMNS[broker]:
            brokers.append(broker)
    return brokers

extract

extract(broker, raw_row)

Reads one broker's extra attributes out of one raw instrument row.

Only the attributes this broker actually published are in the answer. An attribute it does not publish at all, and one whose column is empty in this row, is left out rather than stored as null, because writing all sixteen names for every broker on every instrument costs about three times what the values themselves do. The endpoint fills the missing names back in from ATTRIBUTE_NAMES, so a caller still sees every name.

Values are kept as the broker sent them, stripped of surrounding whitespace and turned into text, because a raw instrument table stores every column as text and nothing here is a number to compute with.

Parameters:

Name	Type	Description	Default
broker	str	The broker name, as it appears in unified.broker_mappings.	required
raw_row	dict	One raw row from that broker's instrument table.	required

Returns:

Type	Description
dict	Mapping of unified attribute name to a string value, holding only the attributes this broker published for this row.

Example usage

From example_1_reading_a_dhan_row.py

attributes = self.raw_attributes.extract('dhan', self.raw_row)

From example_2_who_publishes_what.py

print(f'Zerodha: {self.raw_attributes.extract("zerodha", zerodha_row)}')

From example_2_who_publishes_what.py

print(f'Groww: {self.raw_attributes.extract("groww", groww_row)}')

Source code in stock_brokers/instruments/mapping/utilities/raw_attributes.py

def extract(self, broker, raw_row):
    """Reads one broker's extra attributes out of one raw instrument row.

    Only the attributes this broker actually published are in the answer. An attribute it does not publish at all, and one whose column is empty in this row, is left out rather than stored as null, because writing all sixteen names for every broker on every instrument costs about three times what the values themselves do. The endpoint fills the missing names back in from ATTRIBUTE_NAMES, so a caller still sees every name.

    Values are kept as the broker sent them, stripped of surrounding whitespace and turned into text, because a raw instrument table stores every column as text and nothing here is a number to compute with.

    Args:
        broker (str): The broker name, as it appears in unified.broker_mappings.
        raw_row (dict): One raw row from that broker's instrument table.

    Returns:
        dict: Mapping of unified attribute name to a string value, holding only the attributes this broker published for this row.
    """
    columns = self.columns_for(broker)
    attributes = {}
    for attribute_name in ATTRIBUTE_NAMES:
        column_name = columns.get(attribute_name)
        if column_name is None:
            continue
        value = self.clean(raw_row.get(column_name))
        if value is not None:
            attributes[attribute_name] = value
    return attributes

clean

clean(value)

Turns one raw column value into the text stored for it, or None.

Parameters:

Name	Type	Description	Default
value	object	The value as the raw instrument table returned it, which may be None, a pandas missing value, or text.	required

Returns:

Type	Description
str | None	The value as stripped text, or None when it is missing or empty.

Example usage

From example_2_who_publishes_what.py

print(f'clean({value!r}) = {self.raw_attributes.clean(value)!r}')

Source code in stock_brokers/instruments/mapping/utilities/raw_attributes.py

def clean(self, value):
    """Turns one raw column value into the text stored for it, or None.

    Args:
        value (object): The value as the raw instrument table returned it, which may be None, a pandas missing value, or text.

    Returns:
        str | None: The value as stripped text, or None when it is missing or empty.
    """
    if value is None:
        return None
    if not isinstance(value, (list, tuple, dict)) and pd.isna(value):
        return None
    text_value = str(value).strip()
    if text_value == "":
        return None
    return text_value

fill

fill(attributes)

Expands stored attributes back to the full vocabulary, for an answer a caller reads.

Storage keeps only what a broker published; a caller should not have to check whether a name is there before reading it. This is the one place the two shapes meet.

Parameters:

Name	Type	Description	Default
attributes	dict	The attributes as stored, holding only the names the broker published.	required

Returns:

Type	Description
dict	Mapping of every name in ATTRIBUTE_NAMES to its stored value or None.

Example usage

From example_1_reading_a_dhan_row.py

filled = self.raw_attributes.fill(attributes)

Source code in stock_brokers/instruments/mapping/utilities/raw_attributes.py

def fill(self, attributes):
    """Expands stored attributes back to the full vocabulary, for an answer a caller reads.

    Storage keeps only what a broker published; a caller should not have to check whether a name is there before reading it. This is the one place the two shapes meet.

    Args:
        attributes (dict): The attributes as stored, holding only the names the broker published.

    Returns:
        dict: Mapping of every name in ATTRIBUTE_NAMES to its stored value or None.
    """
    filled = {}
    for attribute_name in ATTRIBUTE_NAMES:
        filled[attribute_name] = attributes.get(attribute_name)
    return filled