Stoxkart mapping adapter.
Stoxkart needs more override logic than any other broker in this build. Its raw file carries several rows for the same instrument, marks calendar spreads and reference rows inside the same instrument type as real contracts, and scales strikes and ticks differently across its own segments.
The overrides fall into four groups:
- Cross-row winner sets, computed once per run because no row-local rule can express them: NSE equities and BSE equities pick one row per symbol by series priority, BSE fixed income picks one row per ISIN (the file carries a normal market-lot row plus odd-lot duplicates), and MCX options pick one row per contract by highest token.
- Cross-broker and cross-segment lookups: the BSE exchange traded fund allowlist, and the NSE index name resolution against the day's already-written master rows.
- Exclusions and redirects no equality rule can express: the "SP-" calendar spread prefix on NSE derivatives, regular expression single-leg matching on the BSE currency and fixed income derivative segments, the index-name exclusion lists on MCX and NCDEX spot rows, and the ISIN checks that separate funds from equities and bonds.
- Value scaling that is not uniform across the broker's own segments: strikes and ticks divide by 100 on the equity family and the commodity options, and by ten million on every currency and fixed income derivative segment.
Expiry dates arrive as DD-MM-YYYY text on every date-bearing segment, so each one names the day_month_year_date transform explicitly rather than relying on the generic parser, which would read any day of 12 or less as a month.
Classes:
| Name | Description |
|---|---|
StoxkartMappingAdapter |
Mapping adapter for Stoxkart, covering 39 real segments plus the uncategorised buckets. |
Functions:
| Name | Description |
|---|---|
normalize_index_name |
Reduce an index name to uppercase alphanumerics so that differently-spelled broker names collide. |
isin_missing |
Check whether an ISIN value is absent, which pandas may represent as either None or NaN. |
token_sort_key |
Build a sort key for a broker token that orders numerically where the token is a number. |
Bases: BrokerMappingAdapter
Mapping adapter for Stoxkart, covering 39 real segments plus the uncategorised buckets.
Example programs
"""Classifies Stoxkart rows through the checks that run before and after its rules, and shows its uneven scales.
Stoxkart lists some instruments more than once, marks calendar spreads with the same instrument type as real contracts, and scales strikes and ticks differently across its own segments. `StoxkartMappingAdapter.classify` therefore runs code checks first (winner tokens, other brokers' fund lists, spread patterns), then the rules, then filters on the rules' answer (a fund ISIN makes an NSE equity an exchange traded fund, an "SP-" description marks an NSE spread, a row with no ISIN is an index), and finally fallbacks for segments that have no rules.
`to_identity` names NSE indices from the description and brings them to the spelling shared with other brokers, and stores bonds under their ISIN. `to_broker_fields` drops the tick size that Stoxkart puts on every index row whatever the real tick, and the rules divide strikes by 100. `run` normally computes the winner tokens from the whole day's file and gathers the fund list and index lookup from the database; this program sets them by hand, so no database is needed. `uncategorised_exchange` shows where a dropped row is filed.
Run it from the project root:
python examples/stock_brokers/instruments/mapping/stoxkart/StoxkartMappingAdapter/example_1_checks_around_the_rules.py
"""
from stock_brokers.instruments.mapping.stoxkart import (
StoxkartMappingAdapter,
)
class ChecksAroundTheRulesExample:
"""Classifies eight hand-written Stoxkart rows and prints each one's segment, identity and sizes.
Attributes:
adapter (StoxkartMappingAdapter): The adapter being shown.
raw_rows (list): The raw rows, as dictionaries of text columns.
"""
def __init__(self):
"""Builds the adapter, sets the sets `run` would compute, and writes the rows.
Returns:
None: This method returns nothing.
"""
self.adapter = StoxkartMappingAdapter()
self.adapter.bse_etf_symbols = set()
self.adapter.nse_index_master_lookup = {}
self.adapter.nse_equities_winning_tokens = {
'2885',
}
self.adapter.bse_equities_winning_tokens = set()
self.adapter.bse_fixed_income_winning_tokens = {
'950001',
}
self.adapter.mcx_options_winning_tokens = set()
self.raw_rows = [
self.raw_row('NSE', '2885', 'RELIANCE', 'RELIANCE INDUSTRIES LTD', 'EQ', None, 'INE002A01018', None, None, None),
self.raw_row('NSE', '10576', 'NIFTYBEES', 'NIPPON INDIA ETF NIFTY 50 BEES', 'EQ', None, 'INF204KB14I2', None, None, None),
self.raw_row('NSE', '26009', 'Nifty Bank', 'Nifty Bank', 'EQ', None, None, None, None, None),
self.raw_row('BSE', '950001', '762GS2036', 'GOI 7.62% 2036', 'G', None, 'IN0020160019', None, None, None),
self.raw_row('NFO', '52168', 'NIFTY', 'NIFTY26OCTFUT', None, 'FUTIDX', None, '27-10-2026', None, None),
self.raw_row('NFO', '99001', 'NIFTY', 'SP-NIFTY26OCT26NOVFUT', None, 'FUTIDX', None, '27-10-2026', None, None),
self.raw_row('NFO', '40215', 'NIFTY', 'NIFTY26OCT25000CE', None, 'OPTIDX', None, '27-10-2026', '2500000', 'CE'),
self.raw_row('NCDEX', '71003', 'GUARSEED10', 'GUARSEED10NOVDEC2026', None, 'FUTCOM', None, '20-11-2026', None, None),
]
def raw_row(self, exchange, token, symbol, description, series, instrument_type, isin, expiry, strike, option_type):
"""Builds one raw row in the shape of `stoxkart.instruments`.
Args:
exchange (str): The exchange code, such as NSE, NFO, BSE or NCDEX.
token (str): Stoxkart's token for the row.
symbol (str): The `symbol` column.
description (str): The `symbol_description` column.
series (str | None): The exchange series, for cash rows.
instrument_type (str | None): The instrument type, for derivatives.
isin (str | None): The `isin_code` column.
expiry (str | None): The expiry as `DD-MM-YYYY` text.
strike (str | None): The strike in hundredths of a rupee.
option_type (str | None): CE or PE for an option.
Returns:
dict: The raw row, every column as text as the raw table stores it.
"""
return {
'exchange': exchange,
'token': token,
'symbol': symbol,
'symbol_description': description,
'series': series,
'instrument_type': instrument_type,
'option_type': option_type,
'expiry_date': expiry,
'lot_size': '1',
'strike_price': strike,
'isin_code': isin,
'tick_size': '5',
}
def run(self):
"""Prints each row's segment, identity and sizes, or where an unmatched row is filed.
Returns:
None: This method returns nothing.
"""
for raw_row in self.raw_rows:
label = f'{raw_row["exchange"]} {raw_row["symbol_description"]}'
segment_configuration = self.adapter.classify(raw_row)
if segment_configuration is None:
exchange = self.adapter.uncategorised_exchange(raw_row)
print(f'{label} -> no segment, filed under {exchange}_uncategorised')
continue
identity = self.adapter.to_identity(raw_row, segment_configuration)
broker_fields = self.adapter.to_broker_fields(raw_row, segment_configuration)
print(f'{label} -> {segment_configuration["segment"]}')
print(f' identity {identity}')
print(f' lot size {broker_fields["lot_size"]}, tick size {broker_fields["tick_size"]}')
if __name__ == '__main__':
ChecksAroundTheRulesExample().run()
It prints:
NSE RELIANCE INDUSTRIES LTD -> nse_equities
identity {'symbol': 'RELIANCE'}
lot size 1.0, tick size 0.05
NSE NIPPON INDIA ETF NIFTY 50 BEES -> nse_exchange_traded_funds
identity {'symbol': 'NIFTYBEES'}
lot size 1.0, tick size 0.05
NSE Nifty Bank -> nse_equity_indices
identity {'symbol': 'BANKNIFTY'}
lot size 1.0, tick size None
BSE GOI 7.62% 2036 -> bse_fixed_income
identity {'symbol': 'IN0020160019'}
lot size 1.0, tick size 0.05
NFO NIFTY26OCTFUT -> nse_equity_index_futures
identity {'underlying_symbol': 'NIFTY', 'expiry_date': datetime.date(2026, 10, 27)}
lot size 1.0, tick size 0.05
NFO SP-NIFTY26OCT26NOVFUT -> no segment, filed under nse_uncategorised
NFO NIFTY26OCT25000CE -> nse_equity_index_options
identity {'underlying_symbol': 'NIFTY', 'expiry_date': datetime.date(2026, 10, 27), 'strike_price': 25000.0, 'option_type': 'CE'}
lot size 1.0, tick size 0.05
NCDEX GUARSEED10NOVDEC2026 -> no segment, filed under ncdex_uncategorised
"""Maps one day of Stoxkart rows, choosing one row wherever the file lists the same instrument more than once.
Stoxkart's file lists some instruments several times: an NSE symbol under two series, a BSE bond as a normal row plus odd-lot copies, an MCX option contract under two tokens. `StoxkartMappingAdapter.run` reads the day's rows once to pick a winning token for each of those, gathers the BSE exchange traded fund list and the NSE index names already stored, and then hands over to the shared mapping run, which reads the rows again, classifies them, and writes `unified.instruments` and `unified.broker_mappings` in one transaction. Rows that lose are filed in the uncategorised buckets rather than written over the winners.
The adapter normally talks to PostgreSQL, so this program replaces its `engine` with a stand-in. The stand-in connection answers both of Stoxkart's own queries on `stoxkart.instruments`, the narrow one for the winners and the full raw-row one, with the same six hand-written rows, answers every cross-broker query with no rows, and records every write instead of running it. pandas only reads through something it recognises as an SQLAlchemy connectable, and it recognises one by the `ConnectionEventsTarget` base class, so the stand-in connection inherits from that class.
Notice which tokens win: the EQ series over BE for RELIANCE, the normal-lot G series row over its GC odd-lot copy, and the higher of the two tokens for the MCX option.
Run it from the project root:
python examples/stock_brokers/instruments/mapping/stoxkart/StoxkartMappingAdapter/example_2_mapping_one_day.py
"""
import collections
import datetime
from sqlalchemy.engine import interfaces
from stock_brokers.instruments.mapping.stoxkart import (
StoxkartMappingAdapter,
)
class StandInResult:
"""A stand-in for an SQLAlchemy result holding a fixed set of rows.
Attributes:
columns (list): The column names.
rows (list): The rows, as dictionaries.
"""
def __init__(self, rows):
"""Holds the rows.
Args:
rows (list): The rows, as dictionaries that all share the same keys.
Returns:
None: This method returns nothing.
"""
self.rows = rows
self.columns = []
if rows:
self.columns = list(rows[0])
def keys(self):
"""The column names, as pandas asks for them.
Returns:
list: The column names.
"""
return self.columns
def fetchall(self):
"""Every row as a plain tuple, as pandas asks for them.
Returns:
list: The rows as tuples in column order.
"""
tuples = []
for row in self.rows:
tuples.append(tuple(row.values()))
return tuples
def all(self):
"""Every row as a named tuple, so a column can be read as an attribute.
Returns:
list: The rows as named tuples.
"""
row_type = collections.namedtuple('StandInRow', self.columns)
named_rows = []
for row in self.rows:
named_rows.append(row_type(**row))
return named_rows
class StandInConnection(interfaces.ConnectionEventsTarget):
"""A stand-in for a PostgreSQL connection that answers queries from prepared rows and records writes.
Attributes:
answers (list): Pairs of (a piece of query text, the rows to answer with).
unanswered_queries (int): How many queries matched no prepared answer and got no rows.
writes (list): Each write statement's first words and its parameters.
"""
def __init__(self, answers):
"""Holds the prepared answers and starts with nothing recorded.
Args:
answers (list): Pairs of (a piece of query text, the rows to answer with).
Returns:
None: This method returns nothing.
"""
self.answers = answers
self.unanswered_queries = 0
self.writes = []
def __enter__(self):
"""Opens the connection for a `with` block.
Returns:
StandInConnection: This connection.
"""
return self
def __exit__(self, exception_type, exception, traceback):
"""Closes the connection at the end of a `with` block.
Args:
exception_type (type | None): The type of any exception raised in the block.
exception (BaseException | None): Any exception raised in the block.
traceback (object | None): The traceback of any exception raised in the block.
Returns:
bool: False, so any exception carries on.
"""
return False
def execute(self, statement, parameters=None):
"""Answers a query from the prepared rows, or records a write.
Args:
statement (sqlalchemy.sql.elements.TextClause): The statement to run.
parameters (dict | list | None): Its bound parameters.
Returns:
StandInResult: The prepared rows whose text matches, otherwise an empty result.
"""
sql = ' '.join(str(statement).split())
if not sql.startswith('SELECT'):
self.writes.append((' '.join(sql.split()[:3]), parameters))
return StandInResult([])
for query_text, rows in self.answers:
if query_text in sql:
return StandInResult(rows)
self.unanswered_queries += 1
return StandInResult([])
class StandInEngine:
"""A stand-in for an SQLAlchemy engine that always hands out the same stand-in connection.
Attributes:
connection (StandInConnection): The connection handed out.
"""
def __init__(self, connection):
"""Holds the connection to hand out.
Args:
connection (StandInConnection): The connection handed out.
Returns:
None: This method returns nothing.
"""
self.connection = connection
def connect(self):
"""Hands out the connection for reading.
Returns:
StandInConnection: The connection.
"""
return self.connection
def begin(self):
"""Hands out the connection for one transaction.
Returns:
StandInConnection: The connection.
"""
return self.connection
class MappingOneDayExample:
"""Maps six Stoxkart rows for one date and prints the winning tokens and what would be written.
Attributes:
mapping_date (datetime.date): The snapshot date being mapped.
connection (StandInConnection): The stand-in connection.
adapter (StoxkartMappingAdapter): The adapter being shown, using the stand-in engine.
"""
def __init__(self):
"""Builds the adapter and swaps its engine for the stand-in.
Returns:
None: This method returns nothing.
"""
self.mapping_date = datetime.date(2026, 9, 28)
raw_rows = [
self.raw_row('NSE', '12885', 'RELIANCE', 'RELIANCE INDUSTRIES LTD', 'BE', None, 'INE002A01018', None, None, None),
self.raw_row('NSE', '2885', 'RELIANCE', 'RELIANCE INDUSTRIES LTD', 'EQ', None, 'INE002A01018', None, None, None),
self.raw_row('BSE', '950002', '762GS2036', 'GOI 7.62% 2036 ODD LOT', 'GC', None, 'IN0020160019', None, None, None),
self.raw_row('BSE', '950001', '762GS2036', 'GOI 7.62% 2036', 'G', None, 'IN0020160019', None, None, None),
self.raw_row('MCX', '440107', 'GOLDM', 'GOLDM26NOV120000CE', None, 'OPTFUT', None, '25-11-2026', '12000000', 'CE'),
self.raw_row('MCX', '440001', 'GOLDM', 'GOLDM26NOV120000CE', None, 'OPTFUT', None, '25-11-2026', '12000000', 'CE'),
]
answers = [
(
'SELECT token, symbol, series, exchange, instrument_type',
raw_rows,
),
(
'SELECT * FROM stoxkart.instruments',
raw_rows,
),
]
self.connection = StandInConnection(answers)
self.adapter = StoxkartMappingAdapter()
self.adapter.engine = StandInEngine(self.connection)
def raw_row(self, exchange, token, symbol, description, series, instrument_type, isin, expiry, strike, option_type):
"""Builds one raw row in the shape of `stoxkart.instruments`.
Args:
exchange (str): The exchange code, such as NSE, BSE or MCX.
token (str): Stoxkart's token for the row.
symbol (str): The `symbol` column.
description (str): The `symbol_description` column.
series (str | None): The exchange series, for cash rows.
instrument_type (str | None): The instrument type, for derivatives.
isin (str | None): The `isin_code` column.
expiry (str | None): The expiry as `DD-MM-YYYY` text.
strike (str | None): The strike in hundredths of a rupee.
option_type (str | None): CE or PE for an option.
Returns:
dict: The raw row, every column as text as the raw table stores it.
"""
return {
'exchange': exchange,
'token': token,
'symbol': symbol,
'symbol_description': description,
'series': series,
'instrument_type': instrument_type,
'option_type': option_type,
'expiry_date': expiry,
'lot_size': '1',
'strike_price': strike,
'isin_code': isin,
'tick_size': '5',
'download_date': self.mapping_date,
}
def run(self):
"""Maps the day and prints the winning tokens, the summary and the rows written.
Returns:
None: This method returns nothing.
"""
summary = self.adapter.run(self.mapping_date)
print(f'NSE equity winners: {sorted(self.adapter.nse_equities_winning_tokens)}')
print(f'BSE fixed income winners: {sorted(self.adapter.bse_fixed_income_winning_tokens)}')
print(f'MCX option winners: {sorted(self.adapter.mcx_options_winning_tokens)}')
print(f'BSE exchange traded funds from other brokers: {sorted(self.adapter.bse_etf_symbols)}')
print(f'Cross-broker queries that found nothing: {self.connection.unanswered_queries}')
print(f'Summary: matched {summary["matched"]}, uncategorised {summary["uncategorised"]}, errors {summary["errors"]}')
segments = {}
for statement, parameters in self.connection.writes:
if not isinstance(parameters, list):
continue
for row in parameters:
if 'segment' in row:
segments[row['instrument_id']] = row['segment']
else:
print(f' token {row["broker_token"]} -> {segments[row["instrument_id"]]}')
if __name__ == '__main__':
MappingOneDayExample().run()
It prints:
stoxkart: 3/6 raw row(s) classified, 3 uncategorised, 6 segment membership(s), 6 distinct instrument(s), 0 error(s).
NSE equity winners: ['2885']
BSE fixed income winners: ['950001']
MCX option winners: ['440107']
BSE exchange traded funds from other brokers: []
Cross-broker queries that found nothing: 12
Summary: matched 3, uncategorised 3, errors []
token 950002 -> bse_uncategorised
token 2885 -> nse_equities
token 440001 -> mcx_uncategorised
token 440107 -> mcx_commodity_options
token 950001 -> bse_fixed_income
token 12885 -> nse_uncategorised
Load and validate the broker's rules file and create the database engine.
Raises:
| Type | Description |
|---|---|
NotImplementedError
|
If the subclass did not set BROKER_NAME. |
ValueError
|
If the rules file fails validation. |
FileNotFoundError
|
If the broker has no rules file. |
Example programs
"""Classifies three raw instrument rows using nothing but a broker's rules file.
`BrokerMappingAdapter` is the base class every broker's mapping adapter subclasses. A subclass only has to name its broker in `BROKER_NAME`, and the base class loads the matching rules file from `stock_brokers/instruments/mapping/utilities/rules/`, checks it against the canonical segment vocabulary, and classifies rows by plain equality rules. This program's subclass sets `BROKER_NAME` to Dhan and overrides nothing, so every answer below comes from the base class and Dhan's rules file alone, without the extra filters the real `DhanMappingAdapter` adds.
The rows are written out by hand in the shape of `dhan.instruments`, where every column is text. The adapter builds a database engine when it is constructed, but an engine does not connect until it is used, and this program never asks it to, so nothing touches PostgreSQL.
For each row the program prints the segment the rules chose, any extra segments (the base class never adds any), the identity fields that go into the instrument id, the broker's own fields, and the deterministic instrument id. The third row matches no rule. The base class cannot tell which exchange it belongs to, so it falls into the unprefixed `uncategorised` segment, and the run would store its token as its symbol.
Run it from the project root:
python examples/stock_brokers/instruments/mapping/base/BrokerMappingAdapter/example_1_classifying_rows_by_rules.py
"""
from stock_brokers.instruments.mapping import base
from stock_brokers.instruments.mapping.base import (
BrokerMappingAdapter,
)
class DhanRulesOnlyAdapter(BrokerMappingAdapter):
"""A mapping adapter that uses Dhan's rules file and nothing but the base class's behaviour."""
BROKER_NAME = 'dhan'
class ClassifyingRowsByRulesExample:
"""Runs three hand-written Dhan rows through the base class's rules engine and prints the result.
Attributes:
adapter (DhanRulesOnlyAdapter): The adapter being shown.
raw_rows (list): The raw rows to classify, as dictionaries of text columns.
"""
def __init__(self):
"""Builds the adapter and the three raw rows.
Returns:
None: This method returns nothing.
"""
self.adapter = DhanRulesOnlyAdapter()
self.raw_rows = [
{
'exch_id': 'NSE',
'segment': 'E',
'security_id': '1594',
'isin': 'INE009A01021',
'instrument': 'EQUITY',
'underlying_symbol': 'INFY',
'symbol_name': 'INFOSYS LIMITED',
'display_name': 'Infosys',
'instrument_type': 'ES',
'series': 'EQ',
'lot_size': '1.0',
'sm_expiry_date': None,
'strike_price': None,
'option_type': None,
'tick_size': '10.0',
},
{
'exch_id': 'NSE',
'segment': 'D',
'security_id': '40215',
'isin': None,
'instrument': 'OPTIDX',
'underlying_symbol': 'NIFTY',
'symbol_name': 'NIFTY-Oct2026-25000-CE',
'display_name': 'NIFTY 27 OCT 25000 CALL',
'instrument_type': 'OP',
'series': None,
'lot_size': '75.0',
'sm_expiry_date': '2026-10-27',
'strike_price': '25000.00000',
'option_type': 'CE',
'tick_size': '5.0',
},
{
'exch_id': 'NSE',
'segment': 'X',
'security_id': '990001',
'isin': None,
'instrument': 'SPREAD',
'underlying_symbol': '',
'symbol_name': 'TEST SPREAD',
'display_name': 'Test spread',
'instrument_type': 'SP',
'series': None,
'lot_size': '1.0',
'sm_expiry_date': None,
'strike_price': None,
'option_type': None,
'tick_size': '0',
},
]
def run(self):
"""Prints how the rules file classifies each row.
Returns:
None: This method returns nothing.
"""
equities = self.adapter.segment_config('nse_equities')
print(f'Rules file: {self.adapter.config["raw_table"]}, {len(self.adapter.config["segments"])} segments')
print(f'nse_equities is a {equities["shape"]} on {equities["exchange"]} with {len(equities["rules"])} rules')
print(f'Unconfigured segment: {self.adapter.segment_config("nse_bonds")}')
for raw_row in self.raw_rows:
print()
print(f'Row {raw_row["security_id"]} ({raw_row["symbol_name"]})')
segment_configuration = self.adapter.classify(raw_row)
if segment_configuration is None:
exchange = self.adapter.uncategorised_exchange(raw_row)
print(f' No rule matched, and the base class finds no exchange: {exchange}')
segment_configuration = self.adapter.segment_config('uncategorised')
extra_segments = self.adapter.classify_extra(raw_row)
identity = self.adapter.to_identity(raw_row, segment_configuration)
broker_fields = self.adapter.to_broker_fields(raw_row, segment_configuration)
identifier = base.instrument_id(
segment_configuration['exchange'],
segment_configuration['segment'],
segment_configuration['shape'],
identity,
)
print(f' Segment: {segment_configuration["segment"]} ({segment_configuration["shape"]})')
print(f' Extra segments: {extra_segments}')
print(f' Identity: {identity}')
print(f' Broker fields: {broker_fields}')
print(f' Instrument id: {identifier}')
if __name__ == '__main__':
ClassifyingRowsByRulesExample().run()
It prints:
Rules file: dhan.instruments, 32 segments
nse_equities is a security on nse with 2 rules
Unconfigured segment: None
Row 1594 (INFOSYS LIMITED)
Segment: nse_equities (security)
Extra segments: []
Identity: {'symbol': 'INFY'}
Broker fields: {'broker_token': '1594', 'broker_symbol': 'Infosys', 'order_symbol': None, 'lot_size': 1.0, 'tick_size': 0.1}
Instrument id: f310e1b0-e082-5b16-808a-25f741abb9de
Row 40215 (NIFTY-Oct2026-25000-CE)
Segment: nse_equity_index_options (option)
Extra segments: []
Identity: {'underlying_symbol': 'NIFTY', 'expiry_date': datetime.date(2026, 10, 27), 'strike_price': 25000.0, 'option_type': 'CE'}
Broker fields: {'broker_token': '40215', 'broker_symbol': 'NIFTY-Oct2026-25000-CE', 'order_symbol': None, 'lot_size': 75.0, 'tick_size': 0.05}
Instrument id: fe85c704-ce49-5190-b068-d4d3646ed162
Row 990001 (TEST SPREAD)
No rule matched, and the base class finds no exchange: None
Segment: uncategorised (security)
Extra segments: []
Identity: {'symbol': ''}
Broker fields: {'broker_token': '990001', 'broker_symbol': 'Test spread', 'order_symbol': None, 'lot_size': 1.0, 'tick_size': None}
Instrument id: 55a8952d-9071-5670-a4fe-a5384d3876bd
"""Maps one day's raw rows into the two unified tables, against a stand-in database.
`run` is what the daily mapping job calls for each broker. It reads the broker's raw rows for one date with `read_raw_rows`, classifies each row, and then, inside one transaction, deletes that broker's mappings for the date and upserts one row per instrument into `unified.instruments` and one per broker mapping into `unified.broker_mappings`. It returns a summary and prints a one-line report.
The adapter normally reads and writes PostgreSQL, so this program replaces its `engine` with a small stand-in. The stand-in connection answers the raw-row query with four hand-written rows in the shape of `dhan.instruments`, and records every other statement it is sent instead of running it. pandas only reads through something it recognises as an SQLAlchemy connectable, and it recognises one by the `ConnectionEventsTarget` base class, so the stand-in connection inherits from that class and nothing else.
The subclass uses Dhan's rules file and overrides nothing. The fourth row has a strike price that is not a number, so building its identity fails; `run` records that as an error and carries on with the other rows rather than stopping. The unmatched row is stored under its token as its symbol, and its tick size of 5.0 is kept as written, because Dhan's rules divide the tick size by 100 only in the real segments and not in the uncategorised one. The last lines show that a subclass which forgets to set `BROKER_NAME` cannot be built.
Run it from the project root:
python examples/stock_brokers/instruments/mapping/base/BrokerMappingAdapter/example_2_mapping_one_day.py
"""
import datetime
from sqlalchemy.engine import interfaces
from stock_brokers.instruments.mapping.base import (
BrokerMappingAdapter,
)
class DhanRulesOnlyAdapter(BrokerMappingAdapter):
"""A mapping adapter that uses Dhan's rules file and nothing but the base class's behaviour."""
BROKER_NAME = 'dhan'
class UnnamedAdapter(BrokerMappingAdapter):
"""A mapping adapter that forgets to name its broker."""
class StandInResult:
"""A stand-in for an SQLAlchemy result holding a fixed set of rows.
Attributes:
columns (list): The column names.
rows (list): The rows, as tuples in column order.
"""
def __init__(self, columns, rows):
"""Holds the columns and rows.
Args:
columns (list): The column names.
rows (list): The rows, as tuples in column order.
Returns:
None: This method returns nothing.
"""
self.columns = columns
self.rows = rows
def keys(self):
"""The column names, as pandas asks for them.
Returns:
list: The column names.
"""
return self.columns
def fetchall(self):
"""Every row, as pandas asks for them.
Returns:
list: The rows.
"""
return self.rows
def all(self):
"""Every row, as the adapter's own queries ask for them.
Returns:
list: The rows.
"""
return self.rows
class StandInConnection(interfaces.ConnectionEventsTarget):
"""A stand-in for a PostgreSQL connection that serves one raw table and records every other statement.
Attributes:
raw_table (str): The raw table whose rows this connection serves.
raw_rows (list): The raw rows, as dictionaries of column values.
statements (list): Each recorded statement's first line and its parameters.
"""
def __init__(self, raw_table, raw_rows):
"""Holds the raw rows and starts with no statements recorded.
Args:
raw_table (str): The raw table whose rows this connection serves.
raw_rows (list): The raw rows, as dictionaries of column values.
Returns:
None: This method returns nothing.
"""
self.raw_table = raw_table
self.raw_rows = raw_rows
self.statements = []
def __enter__(self):
"""Opens the connection for a `with` block.
Returns:
StandInConnection: This connection.
"""
return self
def __exit__(self, exception_type, exception, traceback):
"""Closes the connection at the end of a `with` block.
Args:
exception_type (type | None): The type of any exception raised in the block.
exception (BaseException | None): Any exception raised in the block.
traceback (object | None): The traceback of any exception raised in the block.
Returns:
bool: False, so any exception carries on.
"""
return False
def execute(self, statement, parameters=None):
"""Answers the raw-row query from the held rows and records anything else.
Args:
statement (sqlalchemy.sql.elements.TextClause): The statement to run.
parameters (dict | list | None): Its bound parameters.
Returns:
StandInResult: The raw rows for the raw-row query, otherwise an empty result.
"""
sql = ' '.join(str(statement).split())
if sql.startswith(f'SELECT * FROM {self.raw_table} '):
columns = list(self.raw_rows[0])
rows = []
for raw_row in self.raw_rows:
values = []
for column in columns:
values.append(raw_row[column])
rows.append(tuple(values))
return StandInResult(columns, rows)
self.statements.append((sql, parameters))
return StandInResult([], [])
class StandInEngine:
"""A stand-in for an SQLAlchemy engine that always hands out the same stand-in connection.
Attributes:
connection (StandInConnection): The connection handed out.
"""
def __init__(self, connection):
"""Holds the connection to hand out.
Args:
connection (StandInConnection): The connection handed out.
Returns:
None: This method returns nothing.
"""
self.connection = connection
def connect(self):
"""Hands out the connection for reading.
Returns:
StandInConnection: The connection.
"""
return self.connection
def begin(self):
"""Hands out the connection for one transaction.
Returns:
StandInConnection: The connection.
"""
return self.connection
class MappingOneDayExample:
"""Maps four raw rows for one date and prints what would be written.
Attributes:
mapping_date (datetime.date): The snapshot date being mapped.
connection (StandInConnection): The stand-in connection holding the raw rows.
adapter (DhanRulesOnlyAdapter): The adapter being shown, using the stand-in engine.
"""
def __init__(self):
"""Builds the adapter and swaps its engine for the stand-in.
Returns:
None: This method returns nothing.
"""
self.mapping_date = datetime.date(2026, 9, 28)
raw_rows = [
self.raw_row('1594', 'NSE', 'E', 'EQUITY', 'ES', 'INFY', 'INFOSYS LIMITED', None, None, None),
self.raw_row('40215', 'NSE', 'D', 'OPTIDX', 'OP', 'NIFTY', 'NIFTY-Oct2026-25000-CE', '2026-10-27', '25000.00000', 'CE'),
self.raw_row('990001', 'NSE', 'X', 'SPREAD', 'SP', '', 'TEST SPREAD', None, None, None),
self.raw_row('40216', 'NSE', 'D', 'OPTIDX', 'OP', 'NIFTY', 'NIFTY-Oct2026-XX-PE', '2026-10-27', 'not a number', 'PE'),
]
self.connection = StandInConnection('dhan.instruments', raw_rows)
self.adapter = DhanRulesOnlyAdapter()
self.adapter.engine = StandInEngine(self.connection)
def raw_row(self, security_id, exchange, segment, instrument, instrument_type, underlying_symbol, symbol_name, expiry, strike_price, option_type):
"""Builds one raw row in the shape of `dhan.instruments`.
Args:
security_id (str): Dhan's token for the row.
exchange (str): The `exch_id` column.
segment (str): Dhan's one-letter segment code.
instrument (str): The `instrument` column.
instrument_type (str): The `instrument_type` column.
underlying_symbol (str): The `underlying_symbol` column.
symbol_name (str): The `symbol_name` column.
expiry (str | None): The `sm_expiry_date` column.
strike_price (str | None): The `strike_price` column.
option_type (str | None): The `option_type` column.
Returns:
dict: The raw row, every column as text as the raw table stores it.
"""
return {
'exch_id': exchange,
'segment': segment,
'security_id': security_id,
'isin': None,
'instrument': instrument,
'underlying_security_id': None,
'underlying_symbol': underlying_symbol,
'symbol_name': symbol_name,
'display_name': symbol_name,
'instrument_type': instrument_type,
'series': None,
'lot_size': '1.0',
'sm_expiry_date': expiry,
'strike_price': strike_price,
'option_type': option_type,
'tick_size': '5.0',
'download_date': self.mapping_date,
}
def short_row(self, row):
"""Shortens one written row to its filled-in columns, leaving out the dates the example already names.
Args:
row (dict): One row the adapter asked to insert.
Returns:
dict: The row's columns that hold a value, with the instrument id cut to its first eight characters.
"""
shortened = {}
for column, value in row.items():
if value is None or isinstance(value, datetime.date):
continue
if column == 'instrument_id':
value = value[:8]
shortened[column] = value
return shortened
def run(self):
"""Reads the raw rows, maps them, and prints the summary and the statements sent.
Returns:
None: This method returns nothing.
"""
raw = self.adapter.read_raw_rows(self.connection, self.mapping_date)
print(f'read_raw_rows returned {len(raw)} rows with {len(raw.columns)} columns')
print()
summary = self.adapter.run(self.mapping_date)
print()
for key in summary:
print(f'{key}: {summary[key]}')
print()
print('Statements sent inside the write transaction:')
for sql, parameters in self.connection.statements:
print(f' {sql[:60]}...')
if isinstance(parameters, list):
for row in parameters:
print(f' {self.short_row(row)}')
else:
print(f' {parameters}')
print()
try:
UnnamedAdapter()
except NotImplementedError as error:
print(f'NotImplementedError: {error}')
if __name__ == '__main__':
MappingOneDayExample().run()
It prints:
read_raw_rows returned 4 rows with 17 columns
dhan: 3/4 raw row(s) classified, 1 uncategorised, 3 segment membership(s), 3 distinct instrument(s), 1 error(s).
broker: dhan
mapping_date: 2026-09-28
raw_rows: 4
matched: 3
uncategorised: 1
memberships: 3
instruments_upserted: 3
errors: [('nse_equity_index_options', "could not convert string to float: 'not a number'")]
Statements sent inside the write transaction:
DELETE FROM unified.broker_mappings WHERE broker = :b AND ma...
{'b': 'dhan', 'd': datetime.date(2026, 9, 28)}
INSERT INTO unified.instruments AS master (instrument_id, ex...
{'instrument_id': '8fdcd38e', 'exchange': 'unknown', 'segment': 'uncategorised', 'shape': 'security', 'symbol': '990001'}
{'instrument_id': 'f310e1b0', 'exchange': 'nse', 'segment': 'nse_equities', 'shape': 'security', 'symbol': 'INFY'}
{'instrument_id': 'fe85c704', 'exchange': 'nse', 'segment': 'nse_equity_index_options', 'shape': 'option', 'underlying_symbol': 'NIFTY', 'strike_price': 25000.0, 'option_type': 'CE'}
INSERT INTO unified.broker_mappings (instrument_id, broker, ...
{'instrument_id': '8fdcd38e', 'broker': 'dhan', 'broker_token': '990001', 'broker_symbol': 'TEST SPREAD', 'lot_size': 1.0, 'tick_size': 5.0, 'attributes': '{"display_name": "TEST SPREAD", "instrument_type": "SP"}'}
{'instrument_id': 'f310e1b0', 'broker': 'dhan', 'broker_token': '1594', 'broker_symbol': 'INFOSYS LIMITED', 'lot_size': 1.0, 'tick_size': 0.05, 'attributes': '{"display_name": "INFOSYS LIMITED", "instrument_type": "ES"}'}
{'instrument_id': 'fe85c704', 'broker': 'dhan', 'broker_token': '40215', 'broker_symbol': 'NIFTY-Oct2026-25000-CE', 'lot_size': 1.0, 'tick_size': 0.05, 'attributes': '{"display_name": "NIFTY-Oct2026-25000-CE", "instrument_type": "OP"}'}
NotImplementedError: UnnamedAdapter must set BROKER_NAME
Methods:
| Name | Description |
|---|---|
run |
Map Stoxkart's raw rows for one date, precomputing the four winner sets and the two lookups first. |
classify |
Classify a raw row, running the code-driven checks around the rules engine. |
to_identity |
Build the unified identity fields, applying Stoxkart's per-segment identity sources. |
to_broker_fields |
Build the per-broker mapping fields, applying Stoxkart's per-segment lot and tick handling. |
uncategorised_exchange |
Determine the canonical exchange for an unmatched Stoxkart row from its exchange column. |
segment_config |
Look up one segment configuration by its segment value. |
classify_extra |
Return additional segments this raw row also belongs to beyond the primary match. |
read_raw_rows |
Read the broker's raw rows for one date. |
Source code in stock_brokers/instruments/mapping/base.py
Map Stoxkart's raw rows for one date, precomputing the four winner sets and the two lookups first.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
mapping_date
|
date
|
The raw snapshot date to map. |
required |
Returns:
| Type | Description |
|---|---|
dict
|
The run summary, as described in BrokerMappingAdapter.run. |
Example usage
Source code in stock_brokers/instruments/mapping/stoxkart.py
Pick one NSE equity row per symbol, by series priority.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
raw_rows
|
list[dict]
|
Every raw Stoxkart row for the date. |
required |
Returns:
| Type | Description |
|---|---|
set
|
The winning tokens, as strings. |
Source code in stock_brokers/instruments/mapping/stoxkart.py
Pick one BSE equity row per symbol, preferring a series other than NS or NT.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
raw_rows
|
list[dict]
|
Every raw Stoxkart row for the date. |
required |
Returns:
| Type | Description |
|---|---|
set
|
The winning tokens, as strings. |
Source code in stock_brokers/instruments/mapping/stoxkart.py
Pick one BSE fixed income row per ISIN, preferring the normal market lot and then the lowest token.
Stoxkart's file carries two or three rows for the same bond: one normal F or G series row, plus FC and GC odd-lot duplicates with lot sizes in the hundreds of thousands. Without this choice the stored token flips between dates depending on which row the database happens to write last.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
raw_rows
|
list[dict]
|
Every raw Stoxkart row for the date. |
required |
Returns:
| Type | Description |
|---|---|
set
|
The winning tokens, as strings. |
Source code in stock_brokers/instruments/mapping/stoxkart.py
Pick one MCX option row per contract, keeping the highest token where the file lists a contract twice.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
raw_rows
|
list[dict]
|
Every raw Stoxkart row for the date. |
required |
Returns:
| Type | Description |
|---|---|
set
|
The winning tokens, as strings. |
Source code in stock_brokers/instruments/mapping/stoxkart.py
Classify a raw row, running the code-driven checks around the rules engine.
The early checks run first, so a segment whose real condition is a winner set, a crossref allowlist, or a regular expression always wins over a broader rule listed for a different segment. The rules engine runs next, and its match is then post-filtered. Anything still unmatched goes through the fallback checks for the segments that have no rules at all.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
raw_row
|
dict
|
One raw row from stoxkart.instruments. |
required |
Returns:
| Type | Description |
|---|---|
dict | None
|
The matched segment configuration, or None when no segment matches. |
Example usage
Source code in stock_brokers/instruments/mapping/stoxkart.py
Check the segments whose real condition cannot be a rule and must win over any rule.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
raw_row
|
dict
|
One raw row from stoxkart.instruments. |
required |
Returns:
| Type | Description |
|---|---|
dict | None
|
The matched segment configuration, or None to leave the row to the rules engine. |
Source code in stock_brokers/instruments/mapping/stoxkart.py
Apply the exclusions and redirects that follow a rules-engine match.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
raw_row
|
dict
|
One raw row from stoxkart.instruments. |
required |
segment_configuration
|
dict
|
The segment the rules engine matched. |
required |
Returns:
| Type | Description |
|---|---|
dict | None
|
The final segment configuration, or None to leave the row uncategorised. |
Source code in stock_brokers/instruments/mapping/stoxkart.py
Check the exclusions that separate a real NSE index row from a fund net asset value or bond index row.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
raw_row
|
dict
|
One raw row from stoxkart.instruments. |
required |
Returns:
| Type | Description |
|---|---|
bool
|
True when the row is a real NSE equity index. |
Source code in stock_brokers/instruments/mapping/stoxkart.py
Classify the segments that have no rules at all, for logic no equality rule can express.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
raw_row
|
dict
|
One raw row from stoxkart.instruments. |
required |
Returns:
| Type | Description |
|---|---|
dict | None
|
The matched segment configuration, or None when the row stays uncategorised. |
Source code in stock_brokers/instruments/mapping/stoxkart.py
Build the unified identity fields, applying Stoxkart's per-segment identity sources.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
raw_row
|
dict
|
One raw row from stoxkart.instruments. |
required |
segment_configuration
|
dict
|
The segment configuration whose identity mapping applies. |
required |
Returns:
| Type | Description |
|---|---|
dict
|
Identity fields, as described in BrokerMappingAdapter.to_identity. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If an NCDEX option row has an unparseable or missing expiry, strike, or option type, which the run counts as a per-row error rather than writing a mis-identified instrument. |
Example usage
identity = self.adapter.to_identity(raw_row, segment_configuration)
Source code in stock_brokers/instruments/mapping/stoxkart.py
Build the per-broker mapping fields, applying Stoxkart's per-segment lot and tick handling.
The NSE cash half of nse_fixed_income carries real sizes with the tick in hundredths, while the NSECD rate-underlying half and the currency and rate index segments carry no meaningful sizes at all. The NSE and BSE equity indices carry a tick of 1 on every row whatever the index's real tick, so their tick is dropped.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
raw_row
|
dict
|
One raw row from stoxkart.instruments. |
required |
segment_configuration
|
dict
|
The segment configuration whose broker field mapping applies. |
required |
Returns:
| Type | Description |
|---|---|
dict
|
Keys "broker_token" (str), "broker_symbol", "lot_size" (float | None), and "tick_size" (float | None). |
Example usage
broker_fields = self.adapter.to_broker_fields(raw_row, segment_configuration)
Source code in stock_brokers/instruments/mapping/stoxkart.py
Determine the canonical exchange for an unmatched Stoxkart row from its exchange column.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
raw_row
|
dict
|
One raw row from stoxkart.instruments. |
required |
Returns:
| Type | Description |
|---|---|
str | None
|
"nse", "bse", "mcx", or "ncdex", or None for any other value. |
Example usage
Source code in stock_brokers/instruments/mapping/stoxkart.py
Check the loaded rules file against the canonical vocabulary.
Every configured segment must be a real canonical segment for its exchange with the matching shape, and the segments must appear in the canonical asset-class order. The file must end with the unprefixed uncategorised entry on exchange unknown, which is the guaranteed fallback for rows whose exchange cannot be determined; unknown is allowed for that one entry only, since it is the value stored in the exchange column of the mapped table for those rows.
Raises:
| Type | Description |
|---|---|
ValueError
|
On any invalid exchange, segment, shape, ordering, or missing or misplaced uncategorised entry. |
Source code in stock_brokers/instruments/mapping/base.py
Look up one segment configuration by its segment value.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
segment
|
str
|
The exchange-prefixed segment value, for example "nse_equities". |
required |
Returns:
| Type | Description |
|---|---|
dict | None
|
The matching segment configuration, or None when the rules file does not configure that segment. |
Example usage
segment_configuration = self.adapter.segment_config('uncategorised')
print(f'Unconfigured segment: {self.adapter.segment_config("nse_bonds")}')
Source code in stock_brokers/instruments/mapping/base.py
Read one raw column value, applying a named transform when the specification asks for one.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
raw_row
|
dict
|
One raw row from the broker's instrument table. |
required |
specification
|
str | dict
|
Either a bare column name, or a dict with "column" and optionally "transform" keys. |
required |
Returns:
| Type | Description |
|---|---|
Any
|
The column value, transformed if requested. |
Source code in stock_brokers/instruments/mapping/base.py
Check one equality rule against a raw row.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
raw_row
|
dict
|
One raw row from the broker's instrument table. |
required |
match
|
dict
|
Mapping of raw column name to expected value, or to a list of acceptable values. |
required |
Returns:
| Type | Description |
|---|---|
bool
|
True when every column in the rule matches. |
Source code in stock_brokers/instruments/mapping/base.py
Return additional segments this raw row also belongs to beyond the primary match.
A raw row can genuinely satisfy more than one segment's criteria at once — a BSE row can be both a real equity and a crossref-confirmed exchange traded fund. The base implementation returns an empty list; the brokers where dual membership was confirmed override it.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
raw_row
|
dict
|
One raw row from the broker's instrument table. |
required |
Returns:
| Type | Description |
|---|---|
list[dict]
|
Additional segment configurations, empty by default. |
Example usage
Source code in stock_brokers/instruments/mapping/base.py
Pick the uncategorised segment configuration a row falls back into.
The per-exchange bucket is used when uncategorised_exchange resolves an exchange and the rules file configures that bucket; otherwise the unprefixed uncategorised entry is used.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
raw_row
|
dict
|
One raw row from the broker's instrument table. |
required |
Returns:
| Type | Description |
|---|---|
dict
|
The uncategorised segment configuration. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If no uncategorised configuration exists, which validation should have made impossible. |
Source code in stock_brokers/instruments/mapping/base.py
Read the broker's raw rows for one date.
Adapters override this where the fetch itself is part of the mapping: to impose a row order the classification depends on, or to bring in rows the plain query would miss.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
connection
|
Connection
|
An open SQLAlchemy connection. |
required |
mapping_date
|
date
|
The raw snapshot date to read. |
required |
Returns:
| Type | Description |
|---|---|
DataFrame
|
The raw rows for that date. |
Example usage
raw = self.adapter.read_raw_rows(self.connection, self.mapping_date)
Source code in stock_brokers/instruments/mapping/base.py
Write one run's rows into the mapped tables inside a single transaction.
Both row lists are sorted by instrument id before the insert so concurrent writers acquire locks in the same order, which is the standard fix for the multi-writer deadlock this design was observed hitting when adapters ran in parallel.
The seen dates widen rather than overwrite, so they do not depend on the order dates happen to be mapped in. Assigning the run's date to the last seen date directly looks equivalent while a backfill walks forward, and is wrong the moment any older date is mapped afterwards: it drags the last seen date backwards, and leaves the first seen date frozen at whichever date was written first rather than the earliest the instrument was actually seen.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
instrument_rows
|
dict
|
Mapping of instrument id to the unified.instruments row. |
required |
broker_rows
|
dict
|
Mapping of instrument id to the unified.broker_mappings row. |
required |
mapping_date
|
date
|
The mapping date the run covers. |
required |
Source code in stock_brokers/instruments/mapping/base.py
Reduce an index name to uppercase alphanumerics so that differently-spelled broker names collide.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name
|
str
|
The raw index name from a broker table. |
required |
Returns:
| Type | Description |
|---|---|
str
|
The name with every non-alphanumeric character removed and uppercased. |
Source code in stock_brokers/instruments/mapping/stoxkart.py
Check whether an ISIN value is absent, which pandas may represent as either None or NaN.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
value
|
str | float | None
|
The raw ISIN value from a Stoxkart row. |
required |
Returns:
| Type | Description |
|---|---|
bool
|
True when there is no usable ISIN. |
Source code in stock_brokers/instruments/mapping/stoxkart.py
Build a sort key for a broker token that orders numerically where the token is a number.
Stoxkart's tokens are stored as text, so a plain string sort would order "800178" before "9" and break the lowest-token tie-break the fixed income winner set depends on.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
token
|
str
|
The raw token value. |
required |
Returns:
| Type | Description |
|---|---|
tuple
|
A (is_not_numeric, numeric_value, text_value) triple usable as a sort key. |