Shared base class for the per-broker mapping adapters.
Each broker has one subclass of BrokerMappingAdapter in its own module under stock_brokers/instruments/mapping/ and one rules file under stock_brokers/instruments/mapping/utilities/rules/. The rules sit under utilities/ rather than beside the modules because a broker package here holds only __init__.py, base.py and one <broker>.py per broker, so listing it answers "which brokers are implemented" and nothing else. The subclass sets BROKER_NAME and overrides methods only where the broker's raw file needs logic that equality rules cannot express.
The design: an instrument's unified identity is the composite natural key (exchange, segment, shape, identity fields) — a security is its symbol, a future is its underlying_symbol and expiry_date, an option adds strike_price and option_type. Every adapter computes instrument_id independently as a UUID5 over that key, so the same real-world instrument from any broker converges on the same row in unified.instruments with no matching step. ISIN and broker tokens are used only as classification aids, never as merge keys.
Rows that no segment's rules match are not dropped. They land in the uncategorised catch-all segments — nse_uncategorised, bse_uncategorised, mcx_uncategorised, ncdex_uncategorised, or the unprefixed uncategorised when the exchange itself cannot be determined — so classification coverage stays measurable and the raw row stays recoverable through the broker table by broker, date, and token.
Classes:
| Name | Description |
|---|---|
BrokerMappingAdapter |
Base class for per-broker mapping adapters, holding the rules engine, the identity computation, and the database writes shared by all ten brokers. |
Functions:
| Name | Description |
|---|---|
divide_by_100 |
Convert a value expressed in hundredths into its plain form, used for brokers that report prices and sizes multiplied by 100. |
divide_by_10_thousand |
Convert a value expressed in ten-thousandths into its plain form, used for the commodity option strikes that report a scale of four decimal places. |
divide_by_10_million |
Convert a value expressed in ten-millionths into its plain form, used for the currency and fixed income derivative segments that report strikes and ticks scaled by 10**7. |
day_month_year_date |
Parse a broker's |
unix_epoch_date |
Convert a broker's expiry timestamp expressed as seconds since the 1970 Unix epoch into a date. |
day_month_name_year_date |
Parse a broker's |
kotak_expiry_epoch |
Convert Kotak's expiry timestamps, which count seconds from 1980-01-01 rather than the 1970 Unix epoch. |
strip_exchange_prefix |
Remove an |
canonical_identity_value |
Stringify one identity field for hashing, the same way regardless of the numeric type the caller happens to pass. |
instrument_id |
Compute the deterministic instrument id as a UUID5 over the natural key. |
Base class for per-broker mapping adapters, holding the rules engine, the identity computation, and the database writes shared by all ten brokers.
Load and validate the broker's rules file and create the database engine.
Raises:
| Type | Description |
|---|---|
NotImplementedError
|
If the subclass did not set BROKER_NAME. |
ValueError
|
If the rules file fails validation. |
FileNotFoundError
|
If the broker has no rules file. |
Example programs
"""Classifies three raw instrument rows using nothing but a broker's rules file.
`BrokerMappingAdapter` is the base class every broker's mapping adapter subclasses. A subclass only has to name its broker in `BROKER_NAME`, and the base class loads the matching rules file from `stock_brokers/instruments/mapping/utilities/rules/`, checks it against the canonical segment vocabulary, and classifies rows by plain equality rules. This program's subclass sets `BROKER_NAME` to Dhan and overrides nothing, so every answer below comes from the base class and Dhan's rules file alone, without the extra filters the real `DhanMappingAdapter` adds.
The rows are written out by hand in the shape of `dhan.instruments`, where every column is text. The adapter builds a database engine when it is constructed, but an engine does not connect until it is used, and this program never asks it to, so nothing touches PostgreSQL.
For each row the program prints the segment the rules chose, any extra segments (the base class never adds any), the identity fields that go into the instrument id, the broker's own fields, and the deterministic instrument id. The third row matches no rule. The base class cannot tell which exchange it belongs to, so it falls into the unprefixed `uncategorised` segment, and the run would store its token as its symbol.
Run it from the project root:
python examples/stock_brokers/instruments/mapping/base/BrokerMappingAdapter/example_1_classifying_rows_by_rules.py
"""
from stock_brokers.instruments.mapping import base
from stock_brokers.instruments.mapping.base import (
BrokerMappingAdapter,
)
class DhanRulesOnlyAdapter(BrokerMappingAdapter):
"""A mapping adapter that uses Dhan's rules file and nothing but the base class's behaviour."""
BROKER_NAME = 'dhan'
class ClassifyingRowsByRulesExample:
"""Runs three hand-written Dhan rows through the base class's rules engine and prints the result.
Attributes:
adapter (DhanRulesOnlyAdapter): The adapter being shown.
raw_rows (list): The raw rows to classify, as dictionaries of text columns.
"""
def __init__(self):
"""Builds the adapter and the three raw rows.
Returns:
None: This method returns nothing.
"""
self.adapter = DhanRulesOnlyAdapter()
self.raw_rows = [
{
'exch_id': 'NSE',
'segment': 'E',
'security_id': '1594',
'isin': 'INE009A01021',
'instrument': 'EQUITY',
'underlying_symbol': 'INFY',
'symbol_name': 'INFOSYS LIMITED',
'display_name': 'Infosys',
'instrument_type': 'ES',
'series': 'EQ',
'lot_size': '1.0',
'sm_expiry_date': None,
'strike_price': None,
'option_type': None,
'tick_size': '10.0',
},
{
'exch_id': 'NSE',
'segment': 'D',
'security_id': '40215',
'isin': None,
'instrument': 'OPTIDX',
'underlying_symbol': 'NIFTY',
'symbol_name': 'NIFTY-Oct2026-25000-CE',
'display_name': 'NIFTY 27 OCT 25000 CALL',
'instrument_type': 'OP',
'series': None,
'lot_size': '75.0',
'sm_expiry_date': '2026-10-27',
'strike_price': '25000.00000',
'option_type': 'CE',
'tick_size': '5.0',
},
{
'exch_id': 'NSE',
'segment': 'X',
'security_id': '990001',
'isin': None,
'instrument': 'SPREAD',
'underlying_symbol': '',
'symbol_name': 'TEST SPREAD',
'display_name': 'Test spread',
'instrument_type': 'SP',
'series': None,
'lot_size': '1.0',
'sm_expiry_date': None,
'strike_price': None,
'option_type': None,
'tick_size': '0',
},
]
def run(self):
"""Prints how the rules file classifies each row.
Returns:
None: This method returns nothing.
"""
equities = self.adapter.segment_config('nse_equities')
print(f'Rules file: {self.adapter.config["raw_table"]}, {len(self.adapter.config["segments"])} segments')
print(f'nse_equities is a {equities["shape"]} on {equities["exchange"]} with {len(equities["rules"])} rules')
print(f'Unconfigured segment: {self.adapter.segment_config("nse_bonds")}')
for raw_row in self.raw_rows:
print()
print(f'Row {raw_row["security_id"]} ({raw_row["symbol_name"]})')
segment_configuration = self.adapter.classify(raw_row)
if segment_configuration is None:
exchange = self.adapter.uncategorised_exchange(raw_row)
print(f' No rule matched, and the base class finds no exchange: {exchange}')
segment_configuration = self.adapter.segment_config('uncategorised')
extra_segments = self.adapter.classify_extra(raw_row)
identity = self.adapter.to_identity(raw_row, segment_configuration)
broker_fields = self.adapter.to_broker_fields(raw_row, segment_configuration)
identifier = base.instrument_id(
segment_configuration['exchange'],
segment_configuration['segment'],
segment_configuration['shape'],
identity,
)
print(f' Segment: {segment_configuration["segment"]} ({segment_configuration["shape"]})')
print(f' Extra segments: {extra_segments}')
print(f' Identity: {identity}')
print(f' Broker fields: {broker_fields}')
print(f' Instrument id: {identifier}')
if __name__ == '__main__':
ClassifyingRowsByRulesExample().run()
It prints:
Rules file: dhan.instruments, 32 segments
nse_equities is a security on nse with 2 rules
Unconfigured segment: None
Row 1594 (INFOSYS LIMITED)
Segment: nse_equities (security)
Extra segments: []
Identity: {'symbol': 'INFY'}
Broker fields: {'broker_token': '1594', 'broker_symbol': 'Infosys', 'order_symbol': None, 'lot_size': 1.0, 'tick_size': 0.1}
Instrument id: f310e1b0-e082-5b16-808a-25f741abb9de
Row 40215 (NIFTY-Oct2026-25000-CE)
Segment: nse_equity_index_options (option)
Extra segments: []
Identity: {'underlying_symbol': 'NIFTY', 'expiry_date': datetime.date(2026, 10, 27), 'strike_price': 25000.0, 'option_type': 'CE'}
Broker fields: {'broker_token': '40215', 'broker_symbol': 'NIFTY-Oct2026-25000-CE', 'order_symbol': None, 'lot_size': 75.0, 'tick_size': 0.05}
Instrument id: fe85c704-ce49-5190-b068-d4d3646ed162
Row 990001 (TEST SPREAD)
No rule matched, and the base class finds no exchange: None
Segment: uncategorised (security)
Extra segments: []
Identity: {'symbol': ''}
Broker fields: {'broker_token': '990001', 'broker_symbol': 'Test spread', 'order_symbol': None, 'lot_size': 1.0, 'tick_size': None}
Instrument id: 55a8952d-9071-5670-a4fe-a5384d3876bd
"""Maps one day's raw rows into the two unified tables, against a stand-in database.
`run` is what the daily mapping job calls for each broker. It reads the broker's raw rows for one date with `read_raw_rows`, classifies each row, and then, inside one transaction, deletes that broker's mappings for the date and upserts one row per instrument into `unified.instruments` and one per broker mapping into `unified.broker_mappings`. It returns a summary and prints a one-line report.
The adapter normally reads and writes PostgreSQL, so this program replaces its `engine` with a small stand-in. The stand-in connection answers the raw-row query with four hand-written rows in the shape of `dhan.instruments`, and records every other statement it is sent instead of running it. pandas only reads through something it recognises as an SQLAlchemy connectable, and it recognises one by the `ConnectionEventsTarget` base class, so the stand-in connection inherits from that class and nothing else.
The subclass uses Dhan's rules file and overrides nothing. The fourth row has a strike price that is not a number, so building its identity fails; `run` records that as an error and carries on with the other rows rather than stopping. The unmatched row is stored under its token as its symbol, and its tick size of 5.0 is kept as written, because Dhan's rules divide the tick size by 100 only in the real segments and not in the uncategorised one. The last lines show that a subclass which forgets to set `BROKER_NAME` cannot be built.
Run it from the project root:
python examples/stock_brokers/instruments/mapping/base/BrokerMappingAdapter/example_2_mapping_one_day.py
"""
import datetime
from sqlalchemy.engine import interfaces
from stock_brokers.instruments.mapping.base import (
BrokerMappingAdapter,
)
class DhanRulesOnlyAdapter(BrokerMappingAdapter):
"""A mapping adapter that uses Dhan's rules file and nothing but the base class's behaviour."""
BROKER_NAME = 'dhan'
class UnnamedAdapter(BrokerMappingAdapter):
"""A mapping adapter that forgets to name its broker."""
class StandInResult:
"""A stand-in for an SQLAlchemy result holding a fixed set of rows.
Attributes:
columns (list): The column names.
rows (list): The rows, as tuples in column order.
"""
def __init__(self, columns, rows):
"""Holds the columns and rows.
Args:
columns (list): The column names.
rows (list): The rows, as tuples in column order.
Returns:
None: This method returns nothing.
"""
self.columns = columns
self.rows = rows
def keys(self):
"""The column names, as pandas asks for them.
Returns:
list: The column names.
"""
return self.columns
def fetchall(self):
"""Every row, as pandas asks for them.
Returns:
list: The rows.
"""
return self.rows
def all(self):
"""Every row, as the adapter's own queries ask for them.
Returns:
list: The rows.
"""
return self.rows
class StandInConnection(interfaces.ConnectionEventsTarget):
"""A stand-in for a PostgreSQL connection that serves one raw table and records every other statement.
Attributes:
raw_table (str): The raw table whose rows this connection serves.
raw_rows (list): The raw rows, as dictionaries of column values.
statements (list): Each recorded statement's first line and its parameters.
"""
def __init__(self, raw_table, raw_rows):
"""Holds the raw rows and starts with no statements recorded.
Args:
raw_table (str): The raw table whose rows this connection serves.
raw_rows (list): The raw rows, as dictionaries of column values.
Returns:
None: This method returns nothing.
"""
self.raw_table = raw_table
self.raw_rows = raw_rows
self.statements = []
def __enter__(self):
"""Opens the connection for a `with` block.
Returns:
StandInConnection: This connection.
"""
return self
def __exit__(self, exception_type, exception, traceback):
"""Closes the connection at the end of a `with` block.
Args:
exception_type (type | None): The type of any exception raised in the block.
exception (BaseException | None): Any exception raised in the block.
traceback (object | None): The traceback of any exception raised in the block.
Returns:
bool: False, so any exception carries on.
"""
return False
def execute(self, statement, parameters=None):
"""Answers the raw-row query from the held rows and records anything else.
Args:
statement (sqlalchemy.sql.elements.TextClause): The statement to run.
parameters (dict | list | None): Its bound parameters.
Returns:
StandInResult: The raw rows for the raw-row query, otherwise an empty result.
"""
sql = ' '.join(str(statement).split())
if sql.startswith(f'SELECT * FROM {self.raw_table} '):
columns = list(self.raw_rows[0])
rows = []
for raw_row in self.raw_rows:
values = []
for column in columns:
values.append(raw_row[column])
rows.append(tuple(values))
return StandInResult(columns, rows)
self.statements.append((sql, parameters))
return StandInResult([], [])
class StandInEngine:
"""A stand-in for an SQLAlchemy engine that always hands out the same stand-in connection.
Attributes:
connection (StandInConnection): The connection handed out.
"""
def __init__(self, connection):
"""Holds the connection to hand out.
Args:
connection (StandInConnection): The connection handed out.
Returns:
None: This method returns nothing.
"""
self.connection = connection
def connect(self):
"""Hands out the connection for reading.
Returns:
StandInConnection: The connection.
"""
return self.connection
def begin(self):
"""Hands out the connection for one transaction.
Returns:
StandInConnection: The connection.
"""
return self.connection
class MappingOneDayExample:
"""Maps four raw rows for one date and prints what would be written.
Attributes:
mapping_date (datetime.date): The snapshot date being mapped.
connection (StandInConnection): The stand-in connection holding the raw rows.
adapter (DhanRulesOnlyAdapter): The adapter being shown, using the stand-in engine.
"""
def __init__(self):
"""Builds the adapter and swaps its engine for the stand-in.
Returns:
None: This method returns nothing.
"""
self.mapping_date = datetime.date(2026, 9, 28)
raw_rows = [
self.raw_row('1594', 'NSE', 'E', 'EQUITY', 'ES', 'INFY', 'INFOSYS LIMITED', None, None, None),
self.raw_row('40215', 'NSE', 'D', 'OPTIDX', 'OP', 'NIFTY', 'NIFTY-Oct2026-25000-CE', '2026-10-27', '25000.00000', 'CE'),
self.raw_row('990001', 'NSE', 'X', 'SPREAD', 'SP', '', 'TEST SPREAD', None, None, None),
self.raw_row('40216', 'NSE', 'D', 'OPTIDX', 'OP', 'NIFTY', 'NIFTY-Oct2026-XX-PE', '2026-10-27', 'not a number', 'PE'),
]
self.connection = StandInConnection('dhan.instruments', raw_rows)
self.adapter = DhanRulesOnlyAdapter()
self.adapter.engine = StandInEngine(self.connection)
def raw_row(self, security_id, exchange, segment, instrument, instrument_type, underlying_symbol, symbol_name, expiry, strike_price, option_type):
"""Builds one raw row in the shape of `dhan.instruments`.
Args:
security_id (str): Dhan's token for the row.
exchange (str): The `exch_id` column.
segment (str): Dhan's one-letter segment code.
instrument (str): The `instrument` column.
instrument_type (str): The `instrument_type` column.
underlying_symbol (str): The `underlying_symbol` column.
symbol_name (str): The `symbol_name` column.
expiry (str | None): The `sm_expiry_date` column.
strike_price (str | None): The `strike_price` column.
option_type (str | None): The `option_type` column.
Returns:
dict: The raw row, every column as text as the raw table stores it.
"""
return {
'exch_id': exchange,
'segment': segment,
'security_id': security_id,
'isin': None,
'instrument': instrument,
'underlying_security_id': None,
'underlying_symbol': underlying_symbol,
'symbol_name': symbol_name,
'display_name': symbol_name,
'instrument_type': instrument_type,
'series': None,
'lot_size': '1.0',
'sm_expiry_date': expiry,
'strike_price': strike_price,
'option_type': option_type,
'tick_size': '5.0',
'download_date': self.mapping_date,
}
def short_row(self, row):
"""Shortens one written row to its filled-in columns, leaving out the dates the example already names.
Args:
row (dict): One row the adapter asked to insert.
Returns:
dict: The row's columns that hold a value, with the instrument id cut to its first eight characters.
"""
shortened = {}
for column, value in row.items():
if value is None or isinstance(value, datetime.date):
continue
if column == 'instrument_id':
value = value[:8]
shortened[column] = value
return shortened
def run(self):
"""Reads the raw rows, maps them, and prints the summary and the statements sent.
Returns:
None: This method returns nothing.
"""
raw = self.adapter.read_raw_rows(self.connection, self.mapping_date)
print(f'read_raw_rows returned {len(raw)} rows with {len(raw.columns)} columns')
print()
summary = self.adapter.run(self.mapping_date)
print()
for key in summary:
print(f'{key}: {summary[key]}')
print()
print('Statements sent inside the write transaction:')
for sql, parameters in self.connection.statements:
print(f' {sql[:60]}...')
if isinstance(parameters, list):
for row in parameters:
print(f' {self.short_row(row)}')
else:
print(f' {parameters}')
print()
try:
UnnamedAdapter()
except NotImplementedError as error:
print(f'NotImplementedError: {error}')
if __name__ == '__main__':
MappingOneDayExample().run()
It prints:
read_raw_rows returned 4 rows with 17 columns
dhan: 3/4 raw row(s) classified, 1 uncategorised, 3 segment membership(s), 3 distinct instrument(s), 1 error(s).
broker: dhan
mapping_date: 2026-09-28
raw_rows: 4
matched: 3
uncategorised: 1
memberships: 3
instruments_upserted: 3
errors: [('nse_equity_index_options', "could not convert string to float: 'not a number'")]
Statements sent inside the write transaction:
DELETE FROM unified.broker_mappings WHERE broker = :b AND ma...
{'b': 'dhan', 'd': datetime.date(2026, 9, 28)}
INSERT INTO unified.instruments AS master (instrument_id, ex...
{'instrument_id': '8fdcd38e', 'exchange': 'unknown', 'segment': 'uncategorised', 'shape': 'security', 'symbol': '990001'}
{'instrument_id': 'f310e1b0', 'exchange': 'nse', 'segment': 'nse_equities', 'shape': 'security', 'symbol': 'INFY'}
{'instrument_id': 'fe85c704', 'exchange': 'nse', 'segment': 'nse_equity_index_options', 'shape': 'option', 'underlying_symbol': 'NIFTY', 'strike_price': 25000.0, 'option_type': 'CE'}
INSERT INTO unified.broker_mappings (instrument_id, broker, ...
{'instrument_id': '8fdcd38e', 'broker': 'dhan', 'broker_token': '990001', 'broker_symbol': 'TEST SPREAD', 'lot_size': 1.0, 'tick_size': 5.0, 'attributes': '{"display_name": "TEST SPREAD", "instrument_type": "SP"}'}
{'instrument_id': 'f310e1b0', 'broker': 'dhan', 'broker_token': '1594', 'broker_symbol': 'INFOSYS LIMITED', 'lot_size': 1.0, 'tick_size': 0.05, 'attributes': '{"display_name": "INFOSYS LIMITED", "instrument_type": "ES"}'}
{'instrument_id': 'fe85c704', 'broker': 'dhan', 'broker_token': '40215', 'broker_symbol': 'NIFTY-Oct2026-25000-CE', 'lot_size': 1.0, 'tick_size': 0.05, 'attributes': '{"display_name": "NIFTY-Oct2026-25000-CE", "instrument_type": "OP"}'}
NotImplementedError: UnnamedAdapter must set BROKER_NAME
Methods:
| Name | Description |
|---|---|
segment_config |
Look up one segment configuration by its segment value. |
classify |
Find the first segment whose rules match the raw row. |
classify_extra |
Return additional segments this raw row also belongs to beyond the primary match. |
to_identity |
Build the unified identity fields for a raw row under one segment. |
to_broker_fields |
Build the per-broker mapping fields for a raw row under one segment. |
uncategorised_exchange |
Determine the canonical exchange for a row that matched no segment, so it can be filed into the right per-exchange uncategorised bucket. |
read_raw_rows |
Read the broker's raw rows for one date. |
run |
Classify every raw row for one date and write the mapped tables. |
Source code in stock_brokers/instruments/mapping/base.py
Check the loaded rules file against the canonical vocabulary.
Every configured segment must be a real canonical segment for its exchange with the matching shape, and the segments must appear in the canonical asset-class order. The file must end with the unprefixed uncategorised entry on exchange unknown, which is the guaranteed fallback for rows whose exchange cannot be determined; unknown is allowed for that one entry only, since it is the value stored in the exchange column of the mapped table for those rows.
Raises:
| Type | Description |
|---|---|
ValueError
|
On any invalid exchange, segment, shape, ordering, or missing or misplaced uncategorised entry. |
Source code in stock_brokers/instruments/mapping/base.py
Look up one segment configuration by its segment value.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
segment
|
str
|
The exchange-prefixed segment value, for example "nse_equities". |
required |
Returns:
| Type | Description |
|---|---|
dict | None
|
The matching segment configuration, or None when the rules file does not configure that segment. |
Example usage
segment_configuration = self.adapter.segment_config('uncategorised')
print(f'Unconfigured segment: {self.adapter.segment_config("nse_bonds")}')
Source code in stock_brokers/instruments/mapping/base.py
Read one raw column value, applying a named transform when the specification asks for one.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
raw_row
|
dict
|
One raw row from the broker's instrument table. |
required |
specification
|
str | dict
|
Either a bare column name, or a dict with "column" and optionally "transform" keys. |
required |
Returns:
| Type | Description |
|---|---|
Any
|
The column value, transformed if requested. |
Source code in stock_brokers/instruments/mapping/base.py
Check one equality rule against a raw row.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
raw_row
|
dict
|
One raw row from the broker's instrument table. |
required |
match
|
dict
|
Mapping of raw column name to expected value, or to a list of acceptable values. |
required |
Returns:
| Type | Description |
|---|---|
bool
|
True when every column in the rule matches. |
Source code in stock_brokers/instruments/mapping/base.py
Find the first segment whose rules match the raw row.
Subclasses override this when the broker's classification cannot be expressed as equality rules — Zerodha is the fully custom case — and call this base implementation only for the segments their rules do cover.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
raw_row
|
dict
|
One raw row from the broker's instrument table. |
required |
Returns:
| Type | Description |
|---|---|
dict | None
|
The matched segment configuration, or None when no segment matches. |
Example usage
Source code in stock_brokers/instruments/mapping/base.py
Return additional segments this raw row also belongs to beyond the primary match.
A raw row can genuinely satisfy more than one segment's criteria at once — a BSE row can be both a real equity and a crossref-confirmed exchange traded fund. The base implementation returns an empty list; the brokers where dual membership was confirmed override it.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
raw_row
|
dict
|
One raw row from the broker's instrument table. |
required |
Returns:
| Type | Description |
|---|---|
list[dict]
|
Additional segment configurations, empty by default. |
Example usage
Source code in stock_brokers/instruments/mapping/base.py
Build the unified identity fields for a raw row under one segment.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
raw_row
|
dict
|
One raw row from the broker's instrument table. |
required |
segment_configuration
|
dict
|
The segment configuration whose identity mapping applies. |
required |
Returns:
| Type | Description |
|---|---|
dict
|
Identity fields, with "symbol" for securities and "underlying_symbol", "expiry_date", "strike_price", "option_type" for derivatives. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If an expiry value is numeric with no explicit transform, since the epoch unit cannot be guessed safely. |
Example usage
identity = self.adapter.to_identity(raw_row, segment_configuration)
Source code in stock_brokers/instruments/mapping/base.py
Build the per-broker mapping fields for a raw row under one segment.
A tick size of zero or below is stored as None, because no instrument moves in steps of nothing, and brokers send 0 or -1 where they have no tick for a row.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
raw_row
|
dict
|
One raw row from the broker's instrument table. |
required |
segment_configuration
|
dict
|
The segment configuration whose broker field mapping applies. |
required |
Returns:
| Type | Description |
|---|---|
dict
|
Keys "broker_token" (str), "broker_symbol", "order_symbol", "lot_size" (float | None), and "tick_size" (float | None). |
Example usage
broker_fields = self.adapter.to_broker_fields(raw_row, segment_configuration)
Source code in stock_brokers/instruments/mapping/base.py
Determine the canonical exchange for a row that matched no segment, so it can be filed into the right per-exchange uncategorised bucket.
The base implementation returns None, which routes the row to the unprefixed uncategorised segment. Brokers whose raw rows carry a usable exchange column override this.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
raw_row
|
dict
|
One raw row from the broker's instrument table. |
required |
Returns:
| Type | Description |
|---|---|
str | None
|
Canonical lowercase exchange name, or None. |
Example usage
Source code in stock_brokers/instruments/mapping/base.py
Pick the uncategorised segment configuration a row falls back into.
The per-exchange bucket is used when uncategorised_exchange resolves an exchange and the rules file configures that bucket; otherwise the unprefixed uncategorised entry is used.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
raw_row
|
dict
|
One raw row from the broker's instrument table. |
required |
Returns:
| Type | Description |
|---|---|
dict
|
The uncategorised segment configuration. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If no uncategorised configuration exists, which validation should have made impossible. |
Source code in stock_brokers/instruments/mapping/base.py
Read the broker's raw rows for one date.
Adapters override this where the fetch itself is part of the mapping: to impose a row order the classification depends on, or to bring in rows the plain query would miss.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
connection
|
Connection
|
An open SQLAlchemy connection. |
required |
mapping_date
|
date
|
The raw snapshot date to read. |
required |
Returns:
| Type | Description |
|---|---|
DataFrame
|
The raw rows for that date. |
Example usage
raw = self.adapter.read_raw_rows(self.connection, self.mapping_date)
Source code in stock_brokers/instruments/mapping/base.py
Classify every raw row for one date and write the mapped tables.
The run is self-correcting: this broker's rows for the mapping date are deleted before the insert, so a re-run after a fix leaves no stale rows from an earlier buggy run.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
mapping_date
|
date
|
The raw snapshot date to map. |
required |
Returns:
| Type | Description |
|---|---|
dict
|
Summary with keys "broker", "mapping_date", "raw_rows", "matched", "uncategorised", "memberships", "instruments_upserted", and "errors". "matched" and "uncategorised" count raw rows and together equal "raw_rows"; "memberships" counts the rows written, which is higher wherever a row belongs to more than one segment. "errors" is a list of (segment, message) pairs. |
Example usage
Source code in stock_brokers/instruments/mapping/base.py
517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 | |
Write one run's rows into the mapped tables inside a single transaction.
Both row lists are sorted by instrument id before the insert so concurrent writers acquire locks in the same order, which is the standard fix for the multi-writer deadlock this design was observed hitting when adapters ran in parallel.
The seen dates widen rather than overwrite, so they do not depend on the order dates happen to be mapped in. Assigning the run's date to the last seen date directly looks equivalent while a backfill walks forward, and is wrong the moment any older date is mapped afterwards: it drags the last seen date backwards, and leaves the first seen date frozen at whichever date was written first rather than the earliest the instrument was actually seen.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
instrument_rows
|
dict
|
Mapping of instrument id to the unified.instruments row. |
required |
broker_rows
|
dict
|
Mapping of instrument id to the unified.broker_mappings row. |
required |
mapping_date
|
date
|
The mapping date the run covers. |
required |
Source code in stock_brokers/instruments/mapping/base.py
Convert a value expressed in hundredths into its plain form, used for brokers that report prices and sizes multiplied by 100.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
value
|
str | float | None
|
Raw value from a broker table row, or None. |
required |
Returns:
| Type | Description |
|---|---|
float | None
|
The value divided by 100, or None when the input is None or empty. |
Source code in stock_brokers/instruments/mapping/base.py
Convert a value expressed in ten-thousandths into its plain form, used for the commodity option strikes that report a scale of four decimal places.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
value
|
str | float | None
|
Raw value from a broker table row, or None. |
required |
Returns:
| Type | Description |
|---|---|
float | None
|
The value divided by 10000, or None when the input is None or empty. |
Source code in stock_brokers/instruments/mapping/base.py
Convert a value expressed in ten-millionths into its plain form, used for the currency and fixed income derivative segments that report strikes and ticks scaled by 10**7.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
value
|
str | float | None
|
Raw value from a broker table row, or None. |
required |
Returns:
| Type | Description |
|---|---|
float | None
|
The value divided by 10000000, or None when the input is None or empty. |
Source code in stock_brokers/instruments/mapping/base.py
Parse a broker's DD-MM-YYYY expiry text into a date, used where the generic parser would read a day of 12 or less as a month instead.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
value
|
str | float | None
|
Raw expiry text from a broker table row, or None. |
required |
Returns:
| Type | Description |
|---|---|
date | None
|
The expiry date, or None when the input is None, empty, or unparseable. |
Source code in stock_brokers/instruments/mapping/base.py
Convert a broker's expiry timestamp expressed as seconds since the 1970 Unix epoch into a date.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
value
|
str | float | None
|
Raw seconds value from a broker table row, or None. |
required |
Returns:
| Type | Description |
|---|---|
date | None
|
The expiry date, or None when the input is None or empty. |
Source code in stock_brokers/instruments/mapping/base.py
Parse a broker's DD-MON-YYYY expiry text, such as "26-SEP-2026", into a date.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
value
|
str | float | None
|
Raw expiry text from a broker table row, or None. |
required |
Returns:
| Type | Description |
|---|---|
date | None
|
The expiry date, or None when the input is None, empty, or unparseable. |
Source code in stock_brokers/instruments/mapping/base.py
Convert Kotak's expiry timestamps, which count seconds from 1980-01-01 rather than the 1970 Unix epoch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
value
|
str | float | None
|
Raw seconds value from the Kotak table, or None. |
required |
Returns:
| Type | Description |
|---|---|
date | None
|
The expiry date, or None when the input is None or empty. |
Source code in stock_brokers/instruments/mapping/base.py
Remove an EXCHANGE: prefix from a broker's own ticker column, for example Fyers' symbol_ticker of BSE:AXISBANK26SEPFUT.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
value
|
str | float | None
|
Raw ticker value, or None. |
required |
Returns:
| Type | Description |
|---|---|
str | None
|
The ticker after the last colon, or None when the input is None or empty. |
Source code in stock_brokers/instruments/mapping/base.py
Stringify one identity field for hashing, the same way regardless of the numeric type the caller happens to pass.
str(Decimal('340')) is '340' but str(float('340')) is '340.0' — two different hash inputs for the same real strike price depending on whether the value arrived from Postgres or from a float cast. Every numeric value is therefore routed through Decimal first, so the same real-world value always hashes the same way.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
value
|
str | float | None
|
One identity field value of any type. |
required |
Returns:
| Type | Description |
|---|---|
str
|
The canonical string form used as hash input. |
Source code in stock_brokers/instruments/mapping/base.py
Compute the deterministic instrument id as a UUID5 over the natural key.
Two different brokers carrying the same real-world instrument compute the same id independently, which is the entire merge mechanism: upserts converge on one unified.instruments row with no lookup or registry step.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
exchange
|
str
|
Canonical lowercase exchange name, for example "nse". |
required |
segment
|
str
|
Exchange-prefixed segment value, for example "nse_equities". |
required |
shape
|
str
|
One of "security", "future", or "option". |
required |
identity
|
dict
|
The identity fields for the shape, for example {"symbol": "RELIANCE"}. |
required |
Returns:
| Type | Description |
|---|---|
str
|
The UUID5 instrument id. |